Construct Validity

NSSE Construct Validity Study

Angie L. Miller, Shimon A. Sarraf, Amber D. Dumford, and Louis M. Rocconi

Factor analyses provide evidence of construct validity for NSSE’s ten Engagement Indicators (EI). By examining EI factor structures through exploratory and confirmatory factor analysis (EFA/CFA), quantitative evidence can support claims that the EIs actually measure what they intend to measure. This NSSE Psychometric Portfolio article documents the results of NSSE’s CFA & EFA analyses and concludes that EIs have sufficiently strong construct validity evidence to support their use for college and university assessment efforts.

Data

Prior to conducting the EFA and CFA, we randomly divided all NSSE 2013 respondents into two groups. One group provided data for EFA while the other group provided data for CFA. EFAs were run separately for first-year (n=32,374) and senior (n=46,259) students as well as for students taking all their coursework online. Due to the small number of first-year students taking all their coursework online, we examined only senior online students (n=3,464) in the EFAs. For the CFAs, we developed separate models for all first-year students, seniors, online first-year students, and online seniors1.

Exploratory Factor Analysis

Methods. Given the ordinal nature of the data, the EFAs used polychoric correlations instead of Pearson’s correlations (Drasgow, 2006). The EFA included all engagement items on the survey, excluding the high-impact practice items, perceived gains, and two of the “time spent” items. We included time spent preparing for class and reading due to their relationship to academic work, but excluded the remainder of the “time spent” items because they relate to personal demographics (e.g., working for pay, commuting, caring for dependents). In addition, we included a composite score for amount of writing in lieu of individual writing items. EFA models used principal component analysis with direct oblimin rotation (oblique) in order to allow factors to correlate. We identified all valid components with eigenvalues of 1.0 or greater and reported all factor loadings and cross-loadings of 0.4 or greater.

Results. The EFA for first-year students, seniors, and online seniors suggested twelve, thirteen, and eleven distinct components explaining 65%, 69%, and 71% of the variance, respectively. The Kaiser-Meyer-Olkin statistic was .94 in all three analyses indicating “meritorious” factorability of the item set (Kaiser, 1974). In addition, the Bartlett’s test of sphericity was significant (p < .001) for all three analyses. For each subpopulation, the first ten components aligned with items in the ten EIs and explained over 60% of the variance. For factor loadings for each of the three models, see Appendices A, B, and C. These results informed the CFAs through examination of factor loadings for conceptually similar items, providing evidence that items were grouping in ways that made sense and were statistically appropriate. 

Confirmatory Factor Analysis

Methods. Using the second group of randomly selected respondents, we conducted CFAs using the ten EIs organized by four primary content area themes: Academic Challenge, Learning with Peers, Experiences with Faculty, and Campus Environment. We developed separate models for all first-year students, all seniors, online first-year students, and online seniors, including conceptually related EIs together in the same model and allowing them to correlate.

Given its larger size, we used the senior sample to develop an initial set of four models with the EIs grouped into the conceptually related areas. After building these models, we used modification indices to determine whether model fit could be improved by correlating the error terms of factor indicators (or individual survey items). Once the final models with the greatest number of paths for each content area had been created using the senior population, we then proceeded by estimating model fit indices, standardized regression weights, and factor correlation estimates for first-year student, online first-year student, and online senior populations. 

To assess model fit, we considered five different indices: CMIN/DF (chi-square divided by degrees of freedom), GFI (goodness of fit index), CFI (comparative fit index), RMSEA (root mean square error of approximation), and PCLOSE (p-value for test of close fit). Traditional good model fit criteria for CMIN/DF is a value of 5 or less; however, this statistic is very sensitive to sample size and likely to be inflated with large samples. For the other fit indices, good model fit criteria (as recommended by Hu & Bentler, 1999) are as follows:

GFI: .85 or higher

CFI: .90 or higher

RMSEA: .06 or lower

PCLOSE: .05 or higher

Standardized regression weights determined the strength of factor loadings. These values could range between 0 and 1, with higher values being more desirable. We considered values under .40 unacceptably low (Kline, 2002). Correlations between content area factors of .80 or greater indicate multicollinearity and the potential for factor indicators to load on more than one factor.

CFA Results

Academic Challenge.  CFA results for the Academic Challenge theme, including Reflective & Integrative Learning (RI), Higher-Order Learning (HO), Quantitative Reasoning (QR), and Learning Strategies (LS) EIs, demonstrated very good model fit overall, with all model fit indices meeting the cutoff criteria (see Table 1a). All four factors correlated between .37 and .63 for first-year students, .33 and .65 for seniors, .29 and .67 for online first-year students, and .36 and .67 for online seniors, suggesting that the factors are related, but do not pose overwhelming multicollinearity concerns. The standardized regression weights for all factors across all four groups were strong, ranging from approximately .6 to .9 (see Table 1b). Overall, fit indices, factor correlations, and regression weights provided sufficient construct validity evidence for RI, HO, QR, and LS.

 

Table 1a. Academic Challenge: CFA Model Fit Indices

All

Seniors

All

First-Year Students

Online Seniors

Online

First-Year Students

N

80,144

52,744

9,588

2,278

CMIN/DF

187.862

92.424

18.496

5.248

GFI

.971

.979

.976

.972

CFI

.972

.976

.981

.980

RMSEA

.048

.042

.043

.043

PCLOSE

1.00

1.00

1.00

.999

 

 

Table 1b. Academic Challenge: Standardized Regression Weights

 

All

Seniors

All

First-Year Students

Online Seniors

Online

First-Year Students

Reflective & Integrative Learning

RIintegrate

.587

.608

.585

.592

RIsocietal

.716

.691

.711

.668

RIdiverse

.691

.659

.697

.644

RIownview

.749

.711

.782

.775

RIperspect

.734

.696

.775

.751

RInewview

.717

.686

.725

.697

RIconnect

.711

.696

.734

.731

Higher-Order Learning

HOapply

.645

.648

.787

.791

HOanalyze

.770

.768

.849

.862

HOevaluate

.844

.827

.897

.889

HOform

.805

.780

.832

.825

Quantitative Reasoning

QRconclude

.773

.735

.803

.791

QRproblem

.884

.862

.928

.915

QRevaluate

.844

.843

.835

.865

Learning Strategies

LSreading

.609

.596

.675

.702

LSnotes

.754

.738

.765

.820

LSsummary

.865

.846

.856

.881

 

Learning with Peers.  CFA results for the Learning with Peers theme, including Collaborative Learning (CL) and Discussions with Diverse Others (DD) EIs, showed very good model fit overall, with all model fit indices meeting the cutoff criteria (see Table 2a). The factors were correlated at .29 for first-year students, .28 for seniors, .29 for online first-year students, and .30 for online seniors, suggesting that the factors are related to some extent but not to the point where multicollinearity would be a concern. The standardized regression weights for both factors were strong, ranging from approximately .6 to .9 (see Table 2b). Overall, fit indices, factor correlations, and regression weights provided sufficient construct validity evidence for CL and DD.

 

 

Table 2a. Learning with Peers: CFA Model Fit Indices

All

Seniors

All

First-Year Students

Online Seniors

Online

First-Year Students

N

85,106

56,325

10,229

2,451

CMIN/DF

106.731

58.536

13.064

2.545

GFI

.995

.996

.995

.996

CFI

.995

.996

.997

.998

RMSEA

.035

.032

.034

.025

PCLOSE

1.00

1.00

1.00

1.00

 

 

Table 2b. Learning with Peers: Standardized Regression Weights

 

All

Seniors

All

First-Year Students

Online Seniors

Online

First-Year Students

Collaborative Learning

CLaskhelp

.642

.633

.587

.598

CLexplain

.676

.640

.750

.721

CLstudy

.803

.819

.678

.724

CLproject

.684

.713

.636

.632

Discussions with Diverse Others

DDrace

.839

.816

.927

.930

DDeconomic

.897

.879

.960

.968

DDreligion

.749

.716

.804

.793

DDpolitical

.742

.723

.844

.832

Experiences with Faculty.  CFA results for the Experiences with Faculty theme, including Student-Faculty Interaction (SF) and Effective Teaching Practices (ET) EIs, showed very good model fit overall, with all model fit indices meeting the cutoff criteria (see Table 3a). The factors were correlated at .21 for first-year students, .25 for seniors, .19 for online first-year students, and .20 for online seniors, suggesting that the factors are related to some extent but not to the point where multicollinearity would be a concern. The standardized regression weights for both factors were strong, ranging from approximately .6 to .9 (see Table 3b). Overall, fit indices, factor correlations, and regression weights provided sufficient construct validity evidence for SF and ET.

Table 3a. Experiences with Faculty: CFA Model Fit Indices

All

Seniors

All

First-Year Students

Online Seniors

Online

First-Year Students

N

89,391

59,976

10,296

2,449

CMIN/DF

183.533

98.421

30.266

8.357

GFI

.993

.995

.990

.988

CFI

.993

.993

.989

.989

RMSEA

.045

.040

.053

.055

PCLOSE

1.00

1.00

.094

.174

 

 

Table 3b. Experiences with Faculty: Standardized Regression Weights

 

All

Seniors

All

First-Year Students

Online Seniors

Online

First-Year Students

Student-Faculty Interaction

SFcareer

.758

.691

.670

.683

SFotherwork

.765

.720

.746

.731

SFdiscuss

.814

.764

.739

.730

SFperform

.783

.788

.780

.784

Effective Teaching Practices

ETgoals

.804

.763

.855

.846

ETorganize

.812

.772

.843

.863

ETexample

.798

.794

.780

.770

ETdraftfb

.577

.561

.557

.646

ETfeedback

.672

.626

.701

.734

  

Campus Environment.  CFA results for the Campus Environment theme, including Quality of Interactions (QI) and Supportive Environment (SE) factors, showed adequate model fit overall (see Table 4a). The only model fit indices that did not meet the criteria were RMSEA and PCLOSE for online seniors (RMSEA and PCLOSE are more conservative indices of model fit).  The factors were correlated at .42 for first-year students, .49 for seniors, .44 for online first-year students, and .52 for online seniors, suggesting that the factors are related to some extent but not to the point where multicollinearity would be a concern. The standardized regression weights for both factors were strong, ranging from approximately .5 to .9 (see Table 4b). Overall, fit indices, factor correlations, and regression weights provided sufficient construct validity evidence for QI and SE (but slightly less adequate for online seniors).

Table 4a. Campus Environment: CFA Model Fit Indices

All

Seniors

All

First-Year Students

Online Seniors

Online

First-Year Students

N

61,495

43,221

4,910

1,277

CMIN/DF

144.994

88.825

21.493

4.724

GFI

.980

.983

.962

.968

CFI

.977

.981

.972

.981

RMSEA

.048

.045

.065

.054

PCLOSE

.999

1.00

.000

.149

 

 

Table 4b. Campus Environment: Standardized Regression Weights

 

All

Seniors

All

First-Year Students

Online Seniors

Online

First-Year Students

Quality of Interactions

QIstudent

.451

.464

.557

.620

QIadvisor

.561

.650

.763

.813

QIfaculty

.622

.742

.760

.848

QIstaff

.829

.845

.862

.861

QIadmin

.794

.832

.849

.868

Supportive Environment

SEacademic

.632

.629

.648

.648

SElearnsup

.597

.579

.623

.609

SEdiverse

.712

.722

.758

.799

SEsocial

.799

.804

.841

.854

SEwellness

.808

.803

.873

.859

SEnonacad

.669

.666

.740

.744

SEactivities

.684

.709

.681

.675

SEevents

.674

.682

.695

.717

Evidence-Based Improvement in Higher Education

Center for Postsecondary Research
Indiana University
School of Education
201 N. Rose Avenue
Bloomington, IN 47405-1006
Phone: 812.856.5824
Email: