Shimon Sarraf, Jennifer Brooks, and Jim Cole —
If you're in the business of collecting survey data from college students (or anyone else for that matter), you've probably heard a lot of people talk about optimizing surveys for smartphones. This conversation isn't a surprise, since over 90% of today's college students are estimated to be walking around with smartphones. At NSSE, we've noticed exponential growth in this area, too. In 2011, four years after the first iPhones hit the market, only 4% of NSSE respondents used a smartphone to complete our survey. In 2016, that proportion had jumped to about 42%.
So why does this matter to us at NSSE, and why should it matter to you if you survey college students? Two reasons quickly come to mind: response rates and survey item completion. Among other things, we've discovered firsthand through randomized experimentation that unoptimized smartphone survey formats yield fewer student respondents, and the students who do respond are generally less likely to complete the survey.
Our most recent results from this experimentation come from about 8,000 respondents who completed NSSE either by using a smartphone-optimized version, an un-optimized smartphone version, or a desktop version of the survey. A review of the data shows that optimized smartphone respondents
- were very unlikely to abandon the survey once they saw the first survey page
- were less likely to drop out of the survey or skip individual survey questions, and
- completed NSSE faster - in about 80% of the time it took students using the desktop format
Are all survey optimization formats equally effective? Probably not. One needs to take into account the length of the survey as well as the survey question types being used. NSSE has traditionally used a lot of question grids with the same question stem for groups of items. With a survey with over 100 core questions, it is critical to give respondents the feeling that they are making speedy progress. Strategic use of progress bars, auto-advancing, and making wise use of limited screen space are all things to consider when optimizing surveys for mobile formats.
For additional information about our experiment, see our paper What Is the Impact of Smartphone Optimization on Long Surveys?