Student engagement is defined by educational practices that are associated with learning and development during college. Consequently, providing evidence of NSSE’s ability to predict related measures is one of many important psychometric components to warrant the instrument’s use for institutional assessment and improvement. To provide predictive validity evidence, the current study addressed the following research questions:
What is the relationship between NSSE measures (engagement indicators and high impact practices) and retention of first-year students?
Do these relationships exist for different standardized test score groups, and if so, which groups demonstrate a stronger relationship than others?
Sample & Data
The data used for this study included 12,976 first-year respondents to the 2012 NSSE 2.0 pilot survey, an instrument that closely resembles the updated NSSE survey launched in 2013. Of this sample, approximately 68% were female, 97% were enrolled full-time, 63% were White, 12% were Hispanic, 9% were African-American/Black, 6% were Asian (with the rest classifying themselves as another race/ethnicity, multiracial, or unknown). About 98% were traditional age students (younger than 24). These students came from 45 institutions of various types, including 15 doctoral /research universities, 18 master’s institutions, and 12 baccalaureate colleges. They had an average institution-level retention rate of 87%, ranging from 69% to 100%. Eight additional institutions participated in the pilot but did not provide retention or standardized test scores, and thus were excluded from this study.
We supplemented NSSE data with institution reported standardized test scores and retention information (based on spring 2012 to fall 2012 persistence behavior). Using this combined data set, we focused on the relationship between retention and nine NSSE scales (what are now called Engagement Indicators or EIs) as well as three high-impact practices (HIPs) that had particular relevance to the first-year population. The scales included Higher Order Learning (HO), Reflective & Integrative Learning (RI), Learning Strategies (LS), Quantitative Reasoning (QR), Collaborative Learning (CL), Student-Faculty Interaction (SF), Effective Teaching Practices (ET), Quality of Interactions (QI), and Supportive Environment (SE); the three HIPs included learning community (LS), research with faculty (RF), and service learning (SL) participation. The Discussions with Diverse Others scale on the pilot instrument varied considerably from the current version we began using in 2013, thus it was excluded from these analyses; Effective Teaching Practices was also measured somewhat differently but not enough to exclude it from further analysis. All NSSE measures used in this study had a four point scale, with the exception of QI that had a seven point range and the three HIPs that were coded 1 or 0 for having completed or not completed an HIP, respectively.
We addressed our research questions using a combination of descriptive analysis and statistical modeling. As an initial step, we categorized students by engagement scale quartiles and three SAT groups using 2011 national norm percentiles [lowest quartile (SAT ≤ 860), interquartile (861 ≤ SAT ≤ 1,179), and upper quartile (SAT ≥ 1,180)]. We then analyzed retention rates by engagement level with and without the potential moderating influence of standardized test scores, the most powerful predictor of retention available to us. More specifically, we compare the overall difference in retention rate between the top and bottom engagement indicator quartile group, and do the same comparison within each SAT group. Following this step, we developed logistic regression models using STATA that predicted retention status using each NSSE measure individually plus SAT score, a NSSE measure - SAT score interaction term, student sex, and student enrollment status (full- or part-time status). We weighted all results by student sex and enrollment status and adjusted standard errors to reflect the clustering of students within schools.
As shown in Table 1, we relied on average marginal effects (AMEs)for each NSSE measure to estimate the average percentage point change in retention rate for all students given either a one point increase in engagement scale or HIP participation, controlling for other model variables. Referred to as the “range” in tables 1 and 2, we also estimated the impact of moving from the lowest to the highest level of each scale to estimate the ceiling effect of each NSSE measure. To investigate how this relationship varies or not by SAT score, we also present AMEs based on a one-point or maximum change assuming either a 10th, 50th, or 90th national percentile SAT score (equivalent to a raw score of 730, 1,020, and 1,330). Because of missing data associated with NSSE measures, models used between approximately 12,600 and 9,400 students (an average of about 11,460 per model).
Table 1. Predicted Percentage Point Change in Retention Probability for Engagement Indicators and High Impact Practices
Engagement Indicator or HIP
Probability Change (+1)
Lower Limit
Upper Limit
Probability Change (Range)
Higher Order Learning
3%***
2%
3%
10%***
Reflective & Integrative Learning
2%***
1%
3%
7%***
Learning Strategies
2%***
1%
3%
7%***
Quantitative Reasoning
1%***
1%
2%
4%***
Collaborative Learning
3%***
2%
4%
10%***
Student-Faculty Interaction
3%***
1%
4%
8%***
Effective Teaching Practices
3%***
2%
4%
12%***
Quality of Interactions
2%***
1%
2%
14%***
Supportive Environment
3%***
2%
4%
12%***
Learning Community
4%**
1%
6%
-
Research with Faculty
3%*
1%
5%
-
Service Learning
2%*
0.3%
3%
-
Table 2. Predicted Percentage Point Change in Retention Probability for Engagement Indicators and High Impact Practices by SAT Percentile Group
Probability Change (+1)
Probability Change (Range)
SAT Percentile Group
10th
50th
90th
10th
50th
90th
Higher Order Learning
3%*
3%***
2%***
11%*
11%***
9%**
Reflective & Integrative Learning
3%+
2%***
2%**
9%+
8%***
6%**
Learning Strategies
n.s.
2%***
2%***
n.s.
7%***
8%**
Quantitative Reasoning
2%*
1%***
1%+
7%*
5%***
3%+
Collaborative Learning
n.s.
3%***
3%***
n.s.
10%***
11%***
Student-Faculty Interaction
n.s.
3%***
3%**
n.s.
9%***
9%**
Effective Teaching Practices
4%***
3%***
3%***
15%**
13%***
10%**
Quality of Interactions
n.s.
2%***
2%***
n.s.
13%***
15%***
Supportive Environment
5%***
4%***
2%**
20%***
14%***
9%**
Learning Community
6%*
4%**
n.s.
-
-
-
Research with Faculty
8%*
4%*
n.s.
-
-
-
Service Learning
3%+
2%*
n.s.
-
-
-
Results
As with all the NSSE measures assessed in this study, a positive relationship exists between HO and retention from the 2nd to 3rd semester. The average retention rate difference between those in the bottom and top HO quartiles, whether that be in the aggregate or within SAT group, is approximately 5%. Our model estimates a student’s retention probability increases about 3 percentage points given a 1 point increase in HO or about 10 percentage points given a maximum HO change of 3 points. Reviewing results by the three SAT percentile groups, a 1 point change in HO appears to impact students with different SAT scores similarly (about 2 or 3 percentage points). No evidence of a meaningful interaction between SAT and HO scores is suggested by the data. Each SAT percentile group shows a change of 9 to 11 percentage points given a change of 3 points.
Students in the top RI quartile have approximately a 4 percentage point greater retention rate than those in the bottom quartile. However, some variation by SAT groups exists though. A 7 percentage point difference can be seen between the RI bottom and top quartiles within the lowest SAT quartile group, whereas only a 3 percentage point difference is found within the middle and top SAT groups. Furthermore, our model estimates a student’s retention probability increases about 2 percentage points given a 1 point increase in RI or about 7 percentage points given a maximum RI change of three points. A 1 point change in RI appears to impact students with different SAT scores about the same, however with a shift of 3 points in RI a student with a 10th or 50th percentile SAT score appears to gain the most from increases in RI (8 or 9 percentage point change in retention probability versus 6 percentage point change for the 90th percentile).
Students in the top LS quartile have a 5 percentage point greater retention rate than those in the bottom quartile. The bottom SAT group shows a negligible difference with more pronounced differences of about 5 percentage points for the two higher SAT groups. Our LS model estimates a student’s retention probability increases about 2 percentage points given a 1 point increase in LS or about 7 percentage points given a maximum LS change of 3 points. A 1 point change in LS appears to impact students with different SAT scores similarly (about 2% for the middle and top groups), however with a shift of 3 points in LS a student with a 50th or 90th SAT percentile score appears to gain the most from increases in LS compared to students with a 10th percentile score (a 7 or 8 percentage point change compared to a 5 percentage point change).
Students in the top QR quartile have about a 3 percentage point greater retention rate than those in the bottom quartile. Looking within SAT groups, nearly the same difference is found for both the middle and top groups, however the bottom group has a more noticeable difference of about 7 percentage points. Our QR model estimates a student’s retention probability increases about 1 percentage point given a 1 point increase in QR or about 4 percentage points given a maximum QR change of 3 points. A 1 point change in QR appears to impact students with different SAT scores about the same (about 1 or 2 percentage points), however with a 3 point QR change the model predicts that a student with a 10th percentile SAT score gains more than a student with a 50th or 90th percentile score (a 7 percentage point change in retention probability versus 5 or 3 percentage points).
Students in the top CL quartile have about a 5 percentage point greater retention rate than those in the bottom quartile. Looking within each SAT group, nearly the same difference is found. Our CL model estimates a student’s retention probability increases about 3 percentage points given a 1 point increase in CL or about 10 percentage points given a maximum CL change of three points. A 1 point change in CL appears to impact students with different SAT scores about the same for a student in the 50th or 90th percentile (3 percentage points). Our model also predicts that with a 3 point CL change a student with a 50th or 90th percentile SAT score would increase their retention probability by 10 or 11 percentage points. With either a 1 point or maximum CL change, the effects for students at the 10th percentile were not statistically significant.
Students in the top SF quartile have about a 6 percentage point greater retention rate than those in the bottom quartile. Looking within each SAT group, nearly the same difference is found. Our SF model estimates a student’s retention probability increases about 3 percentage points given a 1 point increase in SF or about 8 percentage points given a maximum SF change of three points. A 1 point change in SF appears to impact students with different SAT scores about the same for a student in the 50th or 90th percentile (3 percentage points). Our model also predicts that with a 3 point SF change a student with a 50th or 90th percentile SAT score would increase their retention probability by 9 percentage points. With either a 1 point or maximum SF change, the effects for students at the 10th percentile were not statistically significant.
Students in the top ET quartile have approximately a 5 percentage point greater retention rate than those in the bottom quartile. Some variation by SAT groups exists though. A nearly 8 point difference can be seen between the ET bottom and top quartiles within the lowest SAT quartile group, whereas we only see a 4 or 6 point difference with the middle and top SAT groups, respectively. Furthermore, our model estimates a student’s retention probability increases about 3 percentage points given a 1 point increase in ET or about 12 percentage points given a maximum ET change of 3 points. A 1 point change in ET appears to impact students with different SAT scores about the same (3 or 4 percentage points), however with a shift of 3 points in ET a student with a 10th percentile SAT score appears to gain the most from increases in ET (15 percentage points compared to 13 and 10 point changes for those with 50th and 90th percentile scores, respectively).
Students in the top QI quartile have more than a 6 percentage point greater retention rate than those in the bottom quartile. Looking within each SAT group, we see a difference among students within the top SAT quartile of almost 9 percentage points, whereas the middle and bottom SAT groups show about a 5 percentage point gap between their top and bottom QI quartile students. Our QI model estimates a student’s retention probability increases about 2 percentage points given a 1 point increase in QI or about 14 percentage points given a maximum QI change of six points. A 1 point change in QI appears to impact students with different SAT scores about the same for a student in the 50th or 90th percentile (2 percentage points). Our model also predicts that with a 6-point QI change a student with a 50th or 90th percentile SAT score would increase their retention probability by 13 or 15 percentage points, respectively. The effects for students at the 10th percentile were not statistically significant.
Students in the top SE quartile have about a 7 percentage point greater retention rate than those in the bottom quartile. Looking within each SAT group, we see a difference within the bottom SAT quartile of almost 13 percentage points, whereas the middle and top SAT groups show about a 6 to7 percentage point gap between their top and bottom SE quartile students. Model estimates indicate a student’s retention probability increases about 3 percentage points given a 1 point increase in SE or about 12 percentage points given a maximum SE change of three points. A 1 point change in SE appears to impact students with varied SAT scores differently. A one point increase in SE is associated with a 4 to 5 percentage point change for 10th and 50th SAT percentile scores, but only 2 percentage points for the 90th percentile. With a 3 point SE change the model estimates a 20, 14, and 9 percentage point change for a student with either a 10th, 50th or 90th SAT percentile score, respectively.
Students that participate in learning communities have about a 3 percentage point greater retention rate than those who did not participate. Looking within each SAT group, we see a difference within the bottom SAT quartile of 7 percentage points compared to 4 and 2 percentage points for the middle and top SAT groups, respectively. Model estimates show a student’s retention probability increases about 4 percentage points on average given LC participation. A student with a 10th percentile SAT score (730) has an even greater estimated change of 6 percentage points compared with 4 percentage points for a student with a 50th percentile score (1,020). The effect for a student at the 90th SAT percentile (1,330) is not statistically significant.
Students that participate in research with faculty have a 3 percentage point greater retention rate than those who do not. Looking within SAT groups, we see a difference of 6 percentage points for the bottom SAT group compared to 4 and 2 percentage points for the middle and top SAT groups, respectively. Model estimates show a student’s retention probability increases about 3 percentage points on average given RF participation. A student with a 10th percentile SAT score (730) has an even greater estimated change of 8 percentage points compared with 4 percentage points for a student with a 50th percentile score (1,020). The effect for a student at the 90th percentile (1,330) is not statistically significant.
Students that reported taking classes with a service learning component have a 2 percentage point greater retention rate than those who did not. Looking within SAT groups, we see a larger difference of about 7 percentage points for those in the bottom SAT quartile, which is in contrast to the 1 to 2 percentage point difference for the other two groups with higher SAT scores. Model estimates show a student’s retention probability increases about 2 percentage points on average given SL participation, and a student with a SAT 10th percentile (730) or 50th percentile (1,020) score has about a 2 to 3 percentage point estimated change. The effect for a student at the 90th percentile (1,330) is not statistically significant.
Summary
These results suggest a meaningful, positive relationship exists between various NSSE measures and first-year student retention. A 1 point change in each NSSE measure studied here corresponds to a 2 to 4 percentage point increase in retention rates; overall effects associated with larger changes for NSSE measures are more notable, ranging from 4 to 14 percentage points. The largest effects are seen with Effective Teaching Practices, Quality of Interactions, and Supportive Environment engagement indicators while Quantitative Reasoning shows the weakest effects. Our data also suggests engagement impacts students with SAT scores of various levels differently. Students with relatively lower SAT scores appear to be more positively affected by Reflective and Integrative Learning, Quantitative Reasoning, Effective Teaching Practices, Supportive Environment, learning community participation, and research with faculty than students at the opposite end of the spectrum. In contrast, our models suggest that retention for this population is not strongly influenced by higher levels of Learning Strategies, Collaborative Learning, Student-Faculty Interaction, and Quality of Interactions. Though our study suggests that high impact practices do not influence student retention for those with very high SAT scores, the results show that this population’s behavior is strongly correlated with all engagement indicators except for Quantitative Reasoning and Reflective and Integrative Learning.
Appendix
List of Study Institutions (n=45)
Albany State University
Alma College
Averett University
Baldwin-Wallace College
California State University, Fullerton
California State University, Northridge
Cornell College
DePauw University
Henderson State University
Indiana University Bloomington
Indiana University-Purdue University Indianapolis
Johnson State College
Kenyon College
Marquette University
Oakland University
Philander Smith College
Roger Williams University
Saint Anselm College
San Diego State University
Savannah State University
Slippery Rock University of Pennsylvania
SUNY Potsdam
Sweet Briar College
Taylor University
Texas Christian University
Texas Lutheran University
Texas State University-San Marcos
Truman State University
University of Alabama
University of Miami
University of Nebraska at Kearney
University of Nebraska at Lincoln
University of North Carolina at Charlotte
University of North Carolina Wilmington
University of San Francisco
University of South Florida
University of Southern Mississippi
University of the Incarnate Word
University of Wisconsin-Eau Claire
University of Wisconsin-Green Bay
Utah State University
Virginia Commonwealth University
Weber State University
Winthrop University
Xavier University of Louisiana
Evidence-Based Improvement in Higher Education resources and social media channels
