NSSENSSENSSE
Angie L. Miller, Shimon A. Sarraf, Amber D. Dumford, and Louis M. Rocconi
Factor analyses provide evidence of construct validity for NSSE’s ten Engagement Indicators (EI). By examining EI factor structures through exploratory and confirmatory factor analysis (EFA/CFA), quantitative evidence can support claims that the
Prior to conducting the EFA and CFA, we randomly divided all NSSE 2013 respondents into two groups. One group provided data for EFA while the other group provided data for CFA. EFAs were run separately for first-year (n=32,374) and senior (n=46,259) students as well as for students taking all their coursework online. Due to the small number of first-year students taking all their coursework online, we examined only senior online students (n=3,464) in the EFAs. For the CFAs, we developed separate models for all first-year students, seniors, online first-year students, and online seniors1.
Methods. Given the ordinal nature of the data, the EFAs used polychoric correlations instead of Pearson’s correlations (Drasgow, 2006). The EFA included all engagement items on the survey, excluding the high-impact practice items, perceived gains, and two of the “time spent” items. We included time spent preparing for class and reading due to their relationship to academic work, but excluded the remainder of the “time spent” items because they relate to personal demographics (e.g., working for pay, commuting, caring for dependents). In addition, we included a composite score for amount of writing in lieu of individual writing items. EFA models used principal component analysis with direct oblimin rotation (oblique) in order to allow factors to correlate. We identified all valid components with eigenvalues of 1.0 or greater and reported all factor loadings and cross-loadings of 0.4 or greater.
Results. The EFA for first-year students, seniors, and online seniors suggested twelve, thirteen, and eleven distinct components explaining 65%, 69%, and 71% of the variance, respectively. The Kaiser-Meyer-Olkin statistic was .94 in all three analyses indicating “meritorious” factorability of the item set (Kaiser, 1974). In addition, the Bartlett’s test of sphericity was significant (p < .001) for all three analyses. For each subpopulation, the first ten components aligned with items in the ten EIs and explained over 60% of the variance. For factor loadings for each of the three models, see Appendices A, B, and C. These results informed the CFAs through examination of factor loadings for conceptually similar items, providing evidence that items were grouping in ways that made sense and were statistically appropriate.
Methods. Using the second group of randomly selected respondents, we conducted CFAs using the ten EIs organized by four primary content area themes: Academic Challenge, Learning with Peers, Experiences with Faculty, and Campus Environment. We developed separate models for all first-year students, all seniors, online first-year students, and online seniors, including conceptually related EIs together in the same model and allowing them to correlate.
Given its larger size, we used the senior sample to develop an initial set of four models with the EIs grouped into the conceptually related areas. After building these models, we used modification indices to determine whether model fit could be improved by correlating the error terms of factor indicators (or individual survey items). Once the final models with the greatest number of paths for each content area had been created using the senior population, we then proceeded by estimating model fit indices, standardized regression weights, and factor correlation estimates for first-year student, online first-year student, and online senior populations.
To assess model fit, we considered five different indices: CMIN/DF (chi-square divided by degrees of freedom), GFI (goodness of fit index), CFI (comparative fit index), RMSEA (root mean square error of approximation), and PCLOSE (p-value for test of close fit). Traditional good model fit criteria for CMIN/DF is a value of 5 or less; however, this statistic is very sensitive to sample size and likely to be inflated with large samples. For the other fit indices, good model fit criteria (as recommended by Hu & Bentler, 1999) are as follows:
GFI: .85 or higher
CFI: .90 or higher
RMSEA: .06 or lower
PCLOSE: .05 or higher
Standardized regression weights determined the strength of factor loadings. These values could range between 0 and 1, with higher values being more desirable. We considered values under .40 unacceptably low (Kline, 2002). Correlations between content area factors of .80 or greater indicate multicollinearity and the potential for factor indicators to load on more than one factor.
Academic Challenge. CFA results for the Academic Challenge theme, including Reflective & Integrative Learning (RI), Higher-Order Learning (HO), Quantitative Reasoning (QR), and Learning Strategies (LS) EIs, demonstrated very good model fit overall, with all model fit indices meeting the cutoff criteria (see Table 1a). All four factors correlated between .37 and .63 for first-year students, .33 and .65 for seniors, .29 and .67 for online first-year students, and .36 and .67 for online seniors, suggesting that the factors are related, but do not pose overwhelming multicollinearity concerns. The standardized regression weights for all factors across all four groups were strong, ranging from approximately .6 to .9 (see Table 1b). Overall, fit indices, factor correlations, and regression weights provided sufficient construct validity evidence for RI, HO, QR, and LS.
Table 1a. Academic Challenge: CFA Model Fit Indices | ||||
All Seniors | All First-Year Students | Online Seniors | Online First-Year Students | |
N | 80,144 | 52,744 | 9,588 | 2,278 |
CMIN/DF | 187.862 | 92.424 | 18.496 | 5.248 |
GFI | .971 | .979 | .976 | .972 |
CFI | .972 | .976 | .981 | .980 |
RMSEA | .048 | .042 | .043 | .043 |
PCLOSE | 1.00 | 1.00 | 1.00 | .999 |
Table 1b. Academic Challenge: Standardized Regression Weights | ||||
| All Seniors | All First-Year Students | Online Seniors | Online First-Year Students |
Reflective & Integrative Learning | ||||
RIintegrate | .587 | .608 | .585 | .592 |
RIsocietal | .716 | .691 | .711 | .668 |
RIdiverse | .691 | .659 | .697 | .644 |
RIownview | .749 | .711 | .782 | .775 |
RIperspect | .734 | .696 | .775 | .751 |
RInewview | .717 | .686 | .725 | .697 |
RIconnect | .711 | .696 | .734 | .731 |
Higher-Order Learning | ||||
HOapply | .645 | .648 | .787 | .791 |
HOanalyze | .770 | .768 | .849 | .862 |
HOevaluate | .844 | .827 | .897 | .889 |
HOform | .805 | .780 | .832 | .825 |
Quantitative Reasoning | ||||
QRconclude | .773 | .735 | .803 | .791 |
QRproblem | .884 | .862 | .928 | .915 |
QRevaluate | .844 | .843 | .835 | .865 |
Learning Strategies | ||||
LSreading | .609 | .596 | .675 | .702 |
LSnotes | .754 | .738 | .765 | .820 |
LSsummary | .865 | .846 | .856 | .881 |
Learning with Peers. CFA results for the Learning with Peers theme, including Collaborative Learning (CL) and Discussions with Diverse Others (DD) EIs, showed very good model fit overall, with all model fit indices meeting the cutoff criteria (see Table 2a). The factors were correlated at .29 for first-year students, .28 for seniors, .29 for online first-year students, and .30 for online seniors, suggesting that the factors are related to some extent but not to the point where multicollinearity would be a concern. The standardized regression weights for both factors were strong, ranging from approximately .6 to .9 (see Table 2b). Overall, fit indices, factor correlations, and regression weights provided sufficient construct validity evidence for CL and DD.
Table 2a. Learning with Peers: CFA Model Fit Indices | ||||
All Seniors | All First-Year Students | Online Seniors | Online First-Year Students | |
N | 85,106 | 56,325 | 10,229 | 2,451 |
CMIN/DF | 106.731 | 58.536 | 13.064 | 2.545 |
GFI | .995 | .996 | .995 | .996 |
CFI | .995 | .996 | .997 | .998 |
RMSEA | .035 | .032 | .034 | .025 |
PCLOSE | 1.00 | 1.00 | 1.00 | 1.00 |
Table 2b. Learning with Peers: Standardized Regression Weights | ||||
| All Seniors | All First-Year Students | Online Seniors | Online First-Year Students |
Collaborative Learning | ||||
CLaskhelp | .642 | .633 | .587 | .598 |
CLexplain | .676 | .640 | .750 | .721 |
CLstudy | .803 | .819 | .678 | .724 |
CLproject | .684 | .713 | .636 | .632 |
Discussions with Diverse Others | ||||
DDrace | .839 | .816 | .927 | .930 |
DDeconomic | .897 | .879 | .960 | .968 |
DDreligion | .749 | .716 | .804 | .793 |
DDpolitical | .742 | .723 | .844 | .832 |
Experiences with Faculty. CFA results for the Experiences with Faculty theme, including Student-Faculty Interaction (SF) and Effective Teaching Practices (ET) EIs, showed very good model fit overall, with all model fit indices meeting the cutoff criteria (see Table 3a). The factors were correlated at .21 for first-year students, .25 for seniors, .19 for online first-year students, and .20 for online seniors, suggesting that the factors are related to some extent but not to the point where multicollinearity would be a concern. The standardized regression weights for both factors were strong, ranging from approximately .6 to .9 (see Table 3b). Overall, fit indices, factor correlations, and regression weights provided sufficient construct validity evidence for SF and ET.
Table 3a. Experiences with Faculty: CFA Model Fit Indices | ||||
All Seniors | All First-Year Students | Online Seniors | Online First-Year Students | |
N | 89,391 | 59,976 | 10,296 | 2,449 |
CMIN/DF | 183.533 | 98.421 | 30.266 | 8.357 |
GFI | .993 | .995 | .990 | .988 |
CFI | .993 | .993 | .989 | .989 |
RMSEA | .045 | .040 | .053 | .055 |
PCLOSE | 1.00 | 1.00 | .094 | .174 |
Table 3b. Experiences with Faculty: Standardized Regression Weights | ||||
| All Seniors | All First-Year Students | Online Seniors | Online First-Year Students |
Student-Faculty Interaction | ||||
SFcareer | .758 | .691 | .670 | .683 |
SFotherwork | .765 | .720 | .746 | .731 |
SFdiscuss | .814 | .764 | .739 | .730 |
SFperform | .783 | .788 | .780 | .784 |
Effective Teaching Practices | ||||
ETgoals | .804 | .763 | .855 | .846 |
ETorganize | .812 | .772 | .843 | .863 |
ETexample | .798 | .794 | .780 | .770 |
ETdraftfb | .577 | .561 | .557 | .646 |
ETfeedback | .672 | .626 | .701 | .734 |
Campus Environment. CFA results for the Campus Environment theme, including Quality of Interactions (QI) and Supportive Environment (SE) factors, showed adequate model fit overall (see Table 4a). The only model fit indices that did not meet the criteria were RMSEA and PCLOSE for online seniors (RMSEA and PCLOSE are more conservative indices of model fit). The factors were correlated at .42 for first-year students, .49 for seniors, .44 for online first-year students, and .52 for online seniors, suggesting that the factors are related to some extent but not to the point where multicollinearity would be a concern. The standardized regression weights for both factors were strong, ranging from approximately .5 to .9 (see Table 4b). Overall, fit indices, factor correlations, and regression weights provided sufficient construct validity evidence for QI and SE (but slightly less adequate for online seniors).
Table 4a. Campus Environment: CFA Model Fit Indices | ||||
All Seniors | All First-Year Students | Online Seniors | Online First-Year Students | |
N | 61,495 | 43,221 | 4,910 | 1,277 |
CMIN/DF | 144.994 | 88.825 | 21.493 | 4.724 |
GFI | .980 | .983 | .962 | .968 |
CFI | .977 | .981 | .972 | .981 |
RMSEA | .048 | .045 | .065 | .054 |
PCLOSE | .999 | 1.00 | .000 | .149 |
Table 4b. Campus Environment: Standardized Regression Weights | ||||
| All Seniors | All First-Year Students | Online Seniors | Online First-Year Students |
Quality of Interactions | ||||
QIstudent | .451 | .464 | .557 | .620 |
QIadvisor | .561 | .650 | .763 | .813 |
QIfaculty | .622 | .742 | .760 | .848 |
QIstaff | .829 | .845 | .862 | .861 |
QIadmin | .794 | .832 | .849 | .868 |
Supportive Environment | ||||
SEacademic | .632 | .629 | .648 | .648 |
SElearnsup | .597 | .579 | .623 | .609 |
SEdiverse | .712 | .722 | .758 | .799 |
SEsocial | .799 | .804 | .841 | .854 |
SEwellness | .808 | .803 | .873 | .859 |
SEnonacad | .669 | .666 | .740 | .744 |
SEactivities | .684 | .709 | .681 | .675 |
SEevents | .674 | .682 | .695 | .717 |
Drasgow, F. (2006). Polychoric and polyserial correlations. Encyclopedia of statistical sciences. Retrieved from http://onlinelibrary.wiley.com/book/10.1002/0471667196
Hu, L.T., & Bentler, P.M. (1999). Cutoff criteria for fit indices in covariance structure analysis: Conventional versus new alternatives. Structural Equation Modeling, 6, 1-55.
Kaiser, H.F. (1974). An index of factorial simplicity. Psychometrika, 39, 31-36.
Kline, P. (2002). An easy guide to factor analysis. New York, NY: Routledge.
Exploratory Factor Analysis Loadings: Seniors | |||||||||||||
| Items | Component Number | ||||||||||||
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 |
RIownview | 0.839 |
|
|
|
|
|
|
|
| -0.423 |
|
|
|
0.831 |
|
|
|
|
|
|
|
| -0.427 |
|
|
| |
0.816 |
|
|
|
|
|
|
|
|
|
|
|
| |
RIperspect | 0.815 |
|
|
|
|
|
|
|
|
|
|
|
|
0.812 |
|
|
|
|
|
|
|
| -0.487 |
|
|
| |
0.779 |
|
|
|
|
|
|
|
| -0.467 |
|
|
| |
RIintegrate | 0.71 |
|
|
|
|
|
|
|
| -0.403 |
|
|
|
0.459 |
|
|
|
|
|
| 0.404 |
|
|
|
|
| |
| -0.839 |
|
|
|
|
|
|
|
|
|
|
| |
| -0.838 |
|
|
|
|
|
|
|
|
|
|
| |
| -0.823 |
|
|
|
|
|
|
|
|
|
|
| |
| -0.798 |
|
|
|
|
|
|
|
|
|
|
| |
SEnonacad |
| -0.752 |
|
|
|
|
|
|
|
|
|
|
|
| -0.746 |
|
|
|
|
|
|
|
|
|
|
| |
| -0.733 |
|
|
| -0.496 |
|
| 0.429 |
|
|
|
| |
SElearnsup |
| -0.725 |
|
|
|
|
|
|
|
|
|
|
|
|
| 0.823 |
|
|
|
|
|
|
|
|
|
| |
|
| 0.823 |
|
|
|
|
|
|
|
|
|
| |
CLaskhelp |
|
| 0.786 |
|
|
|
|
|
|
|
|
|
|
CLexplain |
|
| 0.742 |
|
|
|
| 0.452 |
|
|
|
|
|
present |
|
| 0.616 |
|
|
|
|
|
|
|
|
|
|
|
|
| -0.917 |
|
|
|
|
|
|
|
|
| |
|
|
| -0.899 |
|
|
|
|
|
|
|
|
| |
|
|
| -0.886 |
|
|
|
|
|
|
|
|
| |
|
|
| -0.886 |
|
|
|
|
|
|
|
|
| |
|
|
|
| 0.821 |
|
|
|
|
|
|
|
| |
|
|
|
| 0.794 |
|
|
|
| -0.42 |
|
|
| |
0.477 |
|
|
| 0.582 | -0.402 |
|
|
| -0.448 |
|
|
| |
drafts |
|
|
|
| 0.581 |
|
|
|
|
| 0.406 |
|
|
|
|
|
| -0.535 |
|
|
|
|
|
|
| 0.403 | |
|
|
|
|
| -0.88 |
|
|
|
|
|
|
| |
|
|
|
|
| -0.859 |
|
|
|
|
|
|
| |
|
|
|
|
| -0.85 |
|
|
|
|
|
|
| |
|
|
|
|
| -0.843 |
|
|
|
|
|
|
| |
ETdraftfb |
|
|
|
|
| -0.792 |
|
|
|
|
|
|
|
QRproblem |
|
|
|
|
|
| -0.918 |
|
|
|
|
|
|
QRconclude |
|
|
|
|
|
| -0.907 |
|
|
|
|
|
|
|
|
|
|
|
| -0.905 |
|
|
|
|
|
| |
|
|
|
|
|
|
| 0.876 |
|
|
|
|
| |
SFotherwork |
|
|
|
|
|
|
| 0.839 |
|
|
|
|
|
|
|
|
|
|
|
| 0.833 |
|
|
|
|
| |
|
|
|
|
|
|
| 0.8 |
|
|
|
|
| |
|
|
|
|
|
|
| 0.449 |
|
|
|
|
| |
|
|
|
|
|
|
|
| 0.859 |
|
|
|
| |
|
|
|
|
|
|
|
| 0.852 |
|
|
|
| |
QIfaculty |
|
|
|
|
| -0.543 |
|
| 0.746 |
|
|
|
|
QIadvisor |
|
|
|
|
|
|
|
| 0.733 |
|
|
|
|
|
|
|
|
|
|
|
| 0.622 |
|
|
|
| |
0.434 |
|
|
|
|
|
|
|
| -0.902 |
|
|
| |
0.481 |
|
|
|
|
|
|
|
| -0.862 |
|
|
| |
0.506 |
|
|
|
|
|
|
|
| -0.86 |
|
|
| |
|
|
|
|
|
|
|
|
| -0.807 |
|
|
| |
|
|
|
|
|
|
|
|
|
| 0.695 |
|
| |
|
|
|
|
|
|
|
|
|
| 0.641 | 0.423 |
| |
|
|
|
|
|
|
|
|
|
|
| 0.772 |
| |
| -0.416 |
|
|
|
|
|
|
|
|
| 0.555 |
| |
challenge |
|
|
|
| 0.442 | -0.411 |
|
|
| -0.445 |
| 0.46 |
|
memorize |
|
|
|
|
|
|
|
|
|
|
|
| 0.75 |
Extraction Method: Principal Component Analysis; Rotation Method: Oblimin with Kaiser Normalization | |||||||||||||
1. Ns varied for each CFA model, since due to AMOS specifications, students were excluded from analyses if they had any missing data for survey items used with any particular model. First-year CFA n’s ranged from 43,000 to 59,000; senior n’s ranged from 61,000 to 89,000; online first-year n’s ranged from 1,200 to 2,400; online senior n’s ranged from 4,900 to 10,200.