Documenting Pandemic-Era Conditions

Using NSSE to Document Pandemic-Era Conditions for Teaching and Learning

Photo courtesy of Indiana University-Purdue University Indianapolis

Below...

What Are Institutions Learning from Their NSSE & FSSE 2021 Results?
Participating Institutions Find New Ways to Use NSSE Data
Recommendations for Contextualizing Results

Students’ engagement in practices that matter for their learning and the conditions for faculty teaching were significantly disrupted during the pandemic. As discussed in the first installment of Annual Results, The Pandemic and Student Engagement: Trends, Disparities, and Opportunities, we saw stark changes to the student experience including unprecedented numbers of students taking courses in online and hybrid formats and significant disparities in mental health and emotional well-being. This second Annual Results story, Flexible and Adaptable: Teaching and Learning in A Year of Disruption, focuses more squarely on the influence of the pandemic on critical dimensions of teaching and learning.

Although the 2020-21 academic year is deserving of an asterisk for its unprecedented circumstances, it remains important to document the conditions for education and to invite student and faculty input on the quality of their experience. Institutions participating in NSSE and FSSE can learn a lot from their data. To support institution’s 2021 data use, we sponsored a webinar, “COVID Times and Student Engagement: Using and Interpreting NSSE 2021 Results” (available for viewing anytime) and reflect on some of the concerns raised by our campus contacts here.

The use of student engagement and other assessment results is critical as we work through pandemic-era conditions, and as we strive to simultaneously advance equity and ensure high-quality learning for all students.

What Are Institutions Learning from Their NSSE & FSSE 2021 Results?

Photo courtesy of Washington State University

Institutions that participated in NSSE and FSSE 2021 have first-hand information about what their students and faculty experienced during this challenging year. These data offer insights about how the institution successfully stretched and where things fell short, and provide clues to the dimensions of undergraduate education that deserve attention moving forward. For example, some institutions noticed that student-faculty interaction was consistent with years past, and students’ comments suggested the importance of course instructors who reached out and faculty who stepped up to support students. In another example, evidence of low first-year engagement—such as on involvement in high-impact practices—identifies areas where institutions may want to invest during the sophomore year and beyond. Students’ open-ended comments were particularly revealing, with many mentioning specific people and services that helped or hindered their learning. For example:

“Instructors such as [NAME], are prime examples on how online delivery should be, amazing interactive lectures, recorded lectures, and recorded Panopto's on Blackboard…these instructors made online learning very practical and useful.”

“Say thank you to all the institution members for the great work and devotion.”

And a shout out to a Chemistry department:

“My professors have been really understanding (special thanks to the chemistry department). I find that they truly do care about our mental and physical well-being and I cannot thank them enough.”

“COVID college was not what I expected it to be. [UNIVERSITY] made a great effort to give the students a high-quality education and it showed…I will forever be thankful I chose [UNIVERSITY] as my university.”

Notable in 2021 comments were students expressing appreciation to their institution for asking for their input. It’s indeed rare to see comments like,

“Thanks for the survey”

and even more pointed,

“Thanks for making me take the time to reflect on my college experience.”

Despite the challenges of the pandemic, it was clearly beneficial to ask students about their experience.

Participating Institutions Find New Ways to Use NSSE Data

The variation among students during the pandemic prompted many participating institutions to examine their results based on student populations of interest. Washington State University mentioned that their first-generation and Pell-eligible students struggled more so they took a closer look at these students’ results. Other institutions noticed large shifts in engagement behaviors including more students indicating that they worked more hours for pay off campus and spent more hours caring for dependents than in the past. Variations among student populations are important to document and to identify where follow-up interventions might be needed with particular student populations.

Some participating institutions were understandably apprehensive about reviewing their 2021 results, worrying about anticipated drops in scores or how results would be interpreted (or misinterpreted). Attaching a “collected during COVID-19” note to the datafiles and summary reports is warranted, but should not flag the data as unusable. That said, concern that unprecedented circumstances make results aberrant from past years, or that comparisons among institutions with different course modalities or in states with varied pandemic protocols render peer comparisons useless, are valid. Yet, institutions like Lewis and Clark College commented that these data are necessary to the routine process of iterative assessment to make determinations of quality, effectiveness, and mission fulfillment required of accreditation.

Uncertainty about the influence of pandemic-induced shifts in instructional modality and in particular, if institutions in comparison groups were similar in terms of course modality, is worthy of consideration. Washington State University noticed that among their 21 peer comparison institutions only two were like them, fully remote during Spring 2021. They wondered if their standard peer comparison groups were appropriate given these new ranges of instructional modality. Some institutions requested more customized analyses, forming more refined groups based on knowledge of similar course modality, while others chose to focus on multi-year trends, noting where results were stable or saw a downturn, and then asking if pandemic circumstances explained results that dipped.

Some institutions mentioned their focus on NSSE’s four new effective teaching items, added as an expanded measure of teaching effectiveness during the pandemic to be more responsive to online learning and reflective of innovative pedagogical practice. Institutions can compare their results to the aggregate findings in this Annual Results story, Students: Positive Perceptions of Faculty Teaching. One salient data point this year is that 56% of students indicated their faculty taught in ways that students preferred to learn. This result might prompt reflection among faculty on the flexibility experienced by students, and the value of carrying forward the instructional practices associated with this perception.

Indiana University Purdue University Indianapolis (IUPUI) administered the Coping with COVID Topical Module and found these results illuminated a better understanding of students’ mental health. These results helped IUPUI move the needle on some internal mental health initiatives and to reinforce the positive things that faculty can do to help students. In addition, in the year prior, IUPUI convened a task force to examine university policies with a diversity mindset. Results from the Inclusiveness & Engagement with Cultural Diversity Topical Module, and in particular the ability to break out individual items by race, ethnicity, sexual orientation and gender identity, contributed to IUPUI’s diversity study. Finally, IUPUI looked to their NSSE perceived gains results to examine concerns about perceptions of learning loss during the pandemic. They mapped these items to their institutional learning outcomes, and noticed a decline in some areas that were also identified as dropping in secondary education and higher education during the mostly remote experience. The perceived loss in learning is troubling and NSSE results allowed them to try and understand this phenomenon a little better.

Recommendations for Contextualizing Results

Overall, despite the varied and unusual circumstances during which NSSE 2021 was administered, these data are valuable to institutional assessment efforts. Here are some recommendations for contextualizing results:

"Improvement efforts at colleges and universities are more likely to succeed when they emerge from a shared understanding of the evidence and priorities for action."

Review NSSE 2021 results with an eye toward insights about how the institution successfully stretched, where things fell short, and for clues to the dimensions of undergraduate education that deserve attention moving forward;
Read students’ open-ended comments for unique perspectives on the pandemic, what worked, and who deserves appreciation;
Examine results by student populations, such as first-generation students, students working more than 20 hours per week off campus, or others who may have had different pandemic-era learning experiences;
Place notes and caveats on NSSE 2021 data and reports, but know that these data are useful for routine assessment and accreditation;
Focus on NSSE’s four new effective teaching items, and explore these results with faculty to prompt reflection on instructional flexibility and the potential for carrying forward the teaching practices associated with these perceptions;
Value ways the results might be relevant to current initiatives on campus, such as diversity and equity efforts, attention to mental health, and perceptions of learning loss during the pandemic.

The use of student engagement and other assessment results is critical as we work through pandemic-era conditions, and as we strive to simultaneously advance equity and ensure high-quality learning for all students. Improvement efforts at colleges and universities are more likely to succeed when they emerge from a shared understanding of the evidence and priorities for action. We look forward to working with NSSE participating institutions to advance this imperative.