Have NSSE response rates been declining over the last decade? Many might assume they have been declining, but as with most questions involving hundreds of distinct colleges and universities, the answer is not straightforward. True, average institutional response rates in 2010 were 38% and in 2018 they were 30%. However, the biggest decline--about 8 percentage points--occurred between 2010 and 2013. Since 2013, average rates have actually gone up and down by a point or two with no discernable downward or upward trend. Averages tell you only so much though.
What about response rate variation within any given year? Generally speaking, the amount of variation in rates has decreased as fewer institutions overtime are seeing very high response rates. For instance, about 17% of all schools in 2010 had achieved a 50% or higher response rate versus about 5% in 2018; the proportion of schools with a 10% response rate or lower, on the other hand, has not changed much at all (between 1% and 2% annually). There is still tremendous variation in institutional response rates each year, no doubt. The box-plot graph in Figure 1 shows the distribution of response rates between 2010 and 2018 across four undergraduate enrollment size categories (showing 5th, 25th, 75th, and 95th percentile values). One also sees from this graph that rates generally declined for all four categories until 2013 at which point things pretty much stabilized (though institutions with between 5,000 and 10,000 students show signs of a slight upward trend).
The results above answer the question posed in the title of this blog post--How have response rates changed over the past decade?--but we can dig deeper into the data to better understand the response rate trends for individual institutions. This is where things get really interesting, and complicated! Figure 2 shows the 59 public and private US institutions that participated in every NSSE administration between 2010 and 2018 below. What do we see? Just about everything imaginable as far as trend lines go. Certainly almost every institution had their response rate fall in the first two or three years. But after this initial period, institutions begin to take different paths. Some continue to decrease (sometimes precipitously), while others either level off or increase (sometimes in dramatic fashion).
Most would agree that succinctly characterizing these 59 institutions would be a somewhat difficult task. How about trying to characterize many, many more? Almost 1,100 colleges and universities have participated in NSSE at least three times (the minimum needed for a trend line) during this period. Luckily, using some advanced statistical modeling techniques suited for longitudinal analyses, we can calculate both aggregate and institution-specific results related to all their response rate trends.
Somewhat similar to the descriptive results already presented above, our statistical model suggests that, on average, response rates declined about 3 percentage points between 2010 and 2011 followed by a decelerating rate of change (about .3 percentage points less each year, amounting to 2.7 points between 2011 and 2012, 2.4 points between 2012 and 2013, and so on). The model also confirms, unsurprisingly based on the graph above, the existence of significant variation in individual school growth rates as well as a negative relationship between the initial response rate in 2010 and school specific growth rates (meaning those with higher initial response rates in 2010 have seen their response rates decline at a faster rate than those with lower initial response rates). The age-old adage "the higher you climb, the harder you fall" rings true. Though our model explains over 80% of the variation in any single year's set of response rates, we have not yet found powerful variables to explain the variance in growth rates. Size of institution, public/private status, and other institutional characteristics such as the proportion of full-time students and various racial/ethnic groups, do not explain the amount of decline or growth in individual school response rates. The one exception is the proportion of African-American students enrolled; the greater the proportion, the slower the decline in response rates have been.
Interested in learning more and planning to attend the 2019 AIR Annual Forum in Denver? Come to Shimon Sarraf's poster presentation (#61) on Wednesday, May 29th.