NSSE has experimented continuously over the years to discover the best ways to increase student response rates. In this spirit, we explored the impact on response rates of various factors related to student recruitment email messages during NSSE 2018 with a diverse set of eight U.S. colleges and universities. We aimed to discover if recommended practices by the survey research literature mattered or not relative to our default message templates that have been used by so many schools over the years.
Our experiment included four specific things that we wanted to learn more about:
- Revised message content that conveyed a sense of community, expressed the school’s dependency on students for help, described how data has been used in the past, and emphasized the rare “opportunity” to respond (“we participate once every three years, so this is your chance…”);
- Pre-survey announcements (PSA) to establish NSSE’s credibility before a message with an actual survey link is included; intended to reduce skepticism we have seen from students that the recruitment message with a survey link is inauthentic;
- A promise to provide an end-of-survey comparative report showing individual student responses in relation to the previous year's national norms group; and
- A visually enhanced message that included the school logo and a "take NSSE" survey button.
We tested the above features in an incremental fashion after randomly assigning close to 60,000 survey sample members to one of five recruitment message experimental conditions:
- Standard, default NSSE message (control group)
- Revised message
- Revised message; pre-survey announcement (PSA)
- Revised message; PSA; reference to end-of-survey report
- Revised message; PSA; reference to end-of-survey report; visually enhanced message
What did the results show?
Many different questions could be answered with our data, but we highlight the three most fundamental for this Engagement Insights piece:
In relation to NSSE's default messaging approach, do any of the changes described above increase or decrease the probability of responding to NSSE?
Answer: Based on statistical model results, students in the experimental conditions showed no significantly different propensity to respond relative to the default message group.
Does the impact of the different messaging approaches vary by whether someone is a first-year or senior student, male or female, or minority or non-minority?
Answer: Underrepresented minority status appears to be the only significant factor whereby the various messaging approaches we tested impacted response differently. Class level and sex were not found to interact with the experimental conditions. As the graph below reflects, condition two (revised text) had a positive effect for underrepresented minority students relative to condition one but not for non-minorities.
Survey response probability by experiment condition and minority status
Do experimental conditions have different effects (in relation to the default messaging group) across the eight institutions that participated in the experiment?
Answer: Yes, in some cases (see table below), but overall there are a minimal number of significant differences between the different approaches to messaging and the default option (except for one school, #8). Though our first research question concludes there are no significant differences between the default message approach and the others, one sees that some approaches may work better or worse than others for certain institutions.
Logistic regression results by institution
Recruitment message change for NSSE 2019 and 2020!
What do these results mean for NSSE participating institutions? Given the occasional positive impact of some conditions at schools that participated in this experiment (as well as it just being time to refresh things), we decided to change our default message template text for NSSE 2019 and beyond. And starting with NSSE 2020, we plan to use our visually enhanced messaging approach. In both cases, we anticipate that response rates will improve at a number of schools that use our default message templates or not change much at all.