Brief Summary of Results
Analysis of preliminary 2020 data collected as of May 1 were conducted in response to concerns about the impact of the COVID-19 pandemic that disrupted operations at nearly all participating institutions. While the COVID-19 pandemic disrupted operations at nearly every institution, as of March 18, 96% of NSSE 2020 institutions had already launched the survey and 92% had sent all but one or two of their scheduled reminder messages. Analyses to explore the effect of the disruption on results, show generally trivial effects on response rates and responses to engagement items.
We addressed three broad questions: (a) Did the pandemic change response rates or the types of students who responded to the survey, (b) did the trend of responses change after disruptions to campus operations, and (c) do Engagement Indicators differ pre- and post-disruption?
Effects on Response Rates
Given the small number of institutions that administered NSSE after late March, analysis of the effect on response rates is inconclusive. If there was an effect, COVID-19 most likely increased response rates. A difference-in-difference causal analysis—leveraging data from matched institutions that participated in recent years—showed no significant differences due to COVID for any of the five recruitment messages. In addition, we were pleased to see that overall response rates for the 2020 administration were 2 to 3 percentage points higher on average over past years!
Descriptive Response Patterns by Date
In much of the NSSE, students are asked to respond about their experiences during the course of the entire academic year. So, to understand how responses may have been influenced by disruptions related to the pandemic, we examined trends throughout the administration (February through April) to identify potentially altered responses after, say, mid-March. After reviewing all core items, there were hardly any changes that were likely due to the disruption. Nearly all changes that were present among later respondents could be explained by institutional type and student demographics (online course taking being the exception).
Multivariate Analysis of EIs
Multivariate analysis of Engagement Indicators included three distinct examinations: (a) How early and late responders compare overall and across schools, (b) whether COVID-19 disruptions changed schools’ EIs, and (c) how much average EIs changed over the course of the survey administration. The result? COVID-19’s effect on EIs appears minimal. Being a late responder or late administration school corresponds with slightly higher scores for Learning Strategies, Effective Teaching Practices, Quality of Interactions, and Supportive Campus Environment.
Most institutions completed their NSSE recruitment campaigns prior to disruptions, generating the large majority of respondents, and analyses show trivial impact on responses. This provides some assurance of data quality and continuity of results. For these reasons, NSSE advises against excluding post-COVID respondents from reports.
NSSE will post data files to the Institution Interface around mid-July. With your own data in hand, consider using the covid variable to compare results before and after the disruption and in relation to findings in NSSE’s analysis.
At any time you may also download your disposition file from the Interface (see “Download your disposition file” below the Administration Snapshot). The disposition file is an Excel spreadsheet of every student in your population file and their response status, including the date and time of response.
Again, assessment activities like NSSE and FSSE matter even more in a year of turbulence. Our preliminary analyses should provide additional assurances that the data collected will contribute to your ongoing assessment plans. They will also help you assess your students’ experience in these unprecedented circumstances and study the impact of disruption to inform future planning.