Can Shorter Surveys Motivate Nonrespondents to Respond? A Randomized Controlled Experiment with College Students
Sarraf, Shimon
During the spring 2023, NSSE conducted a randomized controlled experiment with nonrespondents to assess the impact of administering shorter surveys on response rates and question completion. This poster documents the experiment's major findings and concusions.
Full version

How important are high response rates for college surveys?
Fosnacht, K., Sarraf, S., Howe, E., & Peck, L. K.
Surveys play an important role in understanding the higher education landscape. However, declining survey participation rates threaten this source of vital information and its perceived utility. Although survey researchers have long assumed that the best way to obtain unbiased estimates is to achieve a high response rate, many survey researchers have begun to question the widely held assumption that low response rates provide biased results. Due to the prevalence of survey data in higher education research and assessment efforts, it is imperative to better understand the relationship between response rates and data quality. This study investigates this assumption with college student assessment data. It utilizes data from hundreds of samples of first-year and senior students with relatively high response rates using a common assessment instrument with a standardized administration protocol. It investigates how population estimates would have changed if researchers put forth less effort when collecting data and achieved lower response rates and respondent counts.
Full version

An alternative approach: Using panels to survey college students
Sarraf, S. A., Hurtado, S., Houlemarde, M., & Wang, X.

Survey lottery incentives and institutional response rates: An exploratory analysis
Sarraf, S., & Cole, J.S.
Many institutional and educational researchers are well aware that response rates for assessment surveys have been declining over the past few decades (Dey, 1997; Laguilles, Williams, & Saunders, 2011). As a result, many researchers have noted that our ability to adequately assess student academic experiences, satisfaction, engagement, use of campus resources, and other important topics in higher education are at risk (Pike, 2008). Consequently, use of incentives are one tool that many institutional researchers have come to rely on to boost or hold steady their response rates for various campus student surveys. Though research regarding the efficacy of incentives to boost survey response rates in higher education is scant, the research that does exist suggests that incentives are an effective way to boost institutional response rates (Heerwegh, 2006; Laguilles, Williams, & Saunders, 2011). The purpose of this study is to investigate the efficacy of lottery incentives (the mostfrequently used incentive approach) to boost responses rates for institutions using the National Survey of Student Engagement (NSSE).

How important are high response rates for college surveys?
Fosnacht, K., Sarraf, S., Howe, E., & Peck, L.
How important are high survey response rates for estimating population statistics related to the college experience? Given a general decline in survey participation rates among college students, the answer to this question has broad implications for institutional researchers who often use surveys. Survey methodologists have found that low response rates do not necessarily bias results. This study tests this proposition using results from about 250 colleges and universities that administered NSSE. Findings indicate that survey population estimates based on simulated low response rates are very similar to those based on actual high response rates.

How much effort is needed? The importance of response rates for estimating undergraduate behaviors
Fosnacht, K., Sarraf, S., Howe, E., & Peck, L.
Survey methodologists have found that low response rates do not necessarily bias results. This study tests this proposition using results from several hundred colleges that administered a student survey. Findings indicate that survey population estimates based on low response rates are very similar to those based on high response rates.

Understanding and Improving Survey Data Quality: Key Insights from NSSE
Sarraf, Shimon
What has NSSE learned over the past ten years related to understanding and improving survey data quality? This presentation covers various topics, including the effect of survey incentives, learning management systems, and campus promotions on survey response, the importance of optimally formatting long surveys for small screen devices, the effectiveness of panel data collection, and how to evaluate response rates.

How are survey response rates changing? Findings from NSSE
Sarraf, S.
How much have survey response rates changed at colleges and universities over the last decade? Using a National Survey of Student Engagement longitudinal data set (2010 to 2018) based on approximately 1,000 institutions, this study investigates how much response rates have changed over the years, the degree of variability in response rates within any given year, and what factors influence these outcomes, such as school undergraduate enrollment and the use of survey incentives. While reviewing this poster presentation, attendees will also gain an appreciation for one statistical method well suited for understanding change over time-latent growth curve modeling.

Assessing small populations: Recognizing everyone counts in your counts
BrckaLorenz, A., & Hurtado, S.
Quantitative and survey research depends heavily on large sample sizes, but a focus on the “average student” in quantitative analyses often hides diverse voices. Participants in this session will discuss common issues and solutions associated with giving voice to small populations of college students (e.g., gender variant, multiracial, LGBQ+). Participants will discuss administration issues related to small populations such as increasing response rates, identifying special subpopulations, and writing more inclusive survey questions. Tips for disaggregating, responsibly aggregating, and choosing inclusive comparative information will be provided. Additionally, participants will discuss strategies for analyzing and communicating about the results from small populations as well as approaches for communicating about the validity and data quality from small sample sizes.

Assessing small populations: Recognizing everyone counts in your counts
BrckaLorenz, A., Fassett, K., & Hurtado, S.
Quantitative and survey research depends heavily on large sample sizes, but there are a variety of reasons why larger sample sizes may not be possible. Participants in this presentation will discuss common issues and solutions associated with assessing small populations of college students and instructors. Examples will focus on the experiences of gender variant and LGBQ+ students and faculty. Participants will also learn about and discuss administration issues related to small populations such as increasing response rates and identifying special subpopulations. Next, participants will learn about and discuss strategies for analyzing and communicating the results from small populations. Finally, participants will learn about and discuss approaches for communicating the validity and data quality from small sample sizes.

Improving survey data quality through experimentation
Sarraf, S., & Cole, J.
With help from a large and diverse group of colleges and universities over the past decade, the National Survey of Student Engagement has conducted various randomized experiments aimed at improving survey response rates and minimizing missing data. This session provides an overview of this effort and a summary of major findings. Attendees will gain a better understanding of various issues that could help with developing their own surveys, including smartphone optimization, email subject lines, progress indicators, survey page length, and using learning management systems for recruitment.

Improving survey data quality through experimentation
Sarraf, S., & Cole, J.
With help from a large and diverse group of colleges and universities over the past decade, the National Survey of Student Engagement has conducted various randomized experiments aimed at improving survey response rates and minimizing missing data. This session provides an overview of this effort and a summary of major findings. Attendees will gain a better understanding of various issues that could help with developing their own surveys, including smartphone optimization, email subject lines, progress indicators, survey page length, and using learning management systems for recruitment.

Collecting, analyzing, and reporting on data from small populations
BrckaLorenz, A., Hurtado, S., & Nelson Laird, T.
Quantitative and survey research depends heavily on large sample sizes, but there are a variety of reasons why larger sample sizes may not be possible. Participants in this session will discuss common issues and solutions associated with assessing small populations of college students and instructors, with considerations for special subpopulations (gender variant, multiracial, etc.) as well as considerations for small institutions. Participants will also learn about and discuss administration issues related to small populations such as increasing response rates and identifying special subpopulations. Next, participants will learn about and discuss strategies for analyzing and communicating about the results from small populations. Finally, participants will learn about and discuss approaches for communicating the validity and data quality from small sample sizes.

NSSE & Coronavirus 2020: Preliminary Analysis Results and Recommendations
Bob Gonyea, Brendan Dugan, Shimon Sarraf, and Kevin Fosnacht
Recording

NSSE & FSSE 2020: Guidance on COVID-19 Disruptions
Alex McCormick, Jillian Kinzie, Bob Gonyea, Allison BrckaLorenz, Shimon Sarraf and Jennifer Brooks, NSSE and FSSE Project Staff
Recording

Promoting in a Crunch: Increasing NSSE Response Rates
Dajanae Palmer & Bridgette Holmes
Recording

Survey Recruitment Using Student Portals and Learning Management Systems
Shimon Sarraf, Assitant Director for Survey Operations and Project Services
Recording

Improving student participation rates : What we’ve learned about incentives and promotion
Shimon Sarraf, Assistant Director
Recording

Encouraging student participation in NSSE
Brian McGowan, NSSE Institute Project Associate
Recording

Improving student response rates
Tony Ribera and Brian McGowan
Recording