One student admitted to not completely reading the stem of the questions. Some students had a hard time figuring out how to classify infrequent activities. Many of the online students mentioned that the survey seemed like it was written for traditional college students, not online students. Despite this perception, online students were still able to answer most of the questions easily.

Students had some difficulty distinguishing between the different items in this indicator. HOapply seemed to be more difficult for students to answer and required some time to think about. A few students were stuck on the word "theories" and did not feel like it pertained to them because their major did not focus on theory. Other students only considered when they were applying theories to real world and contemporary problems (e.g., not counting looking at different historical events using a particular theory).

No significant problems with RIsocietal were identified for on-campus or online students. A few students thought a parenthetical with examples of societal problems or issues would be helpful.

RIdiverse was challenging for some students and students asked for clarification on what we were asking about.

A small number of students expressed uncertainty about LSsummary and were not sure if the question was asking about if they actually wrote out a summary or if they just thought about summarizing course material.

Overall, these items were more challenging for students in non-STEM majors. Students seemed to spend more time thinking about these questions but were generally able to answer them. Interviews with online students had similar results. For item QRproblem, there may be a problem with underestimating frequency because some students stated that they would answer “never” because they initially just thought of math classes. When prompted to think about other classes, they were able to come up with examples where they had done this. In general, the parentheticals seemed helpful. Students found QRevaluate slightly confusing and their interpretation varied more. Students were not sure who "others" were (e.g., other students or just researchers). They also were not sure what constituted "evaluating." Some counted any time they read something with numerical information on it, thought about it, or were tested on it. Other students only counted times that they actually challenged assumptions in the research or how the numerical information was calculated.

Students generally understood the questions about writing (wrshort, wrmed, and wrlong). A number of students mentioned that it was easier to accurately report the number of larger papers, and harder to remember all of the short papers. Students also differed on what they counted as a "paper" in the "less than 5" pages category. Some students were counting their daily reflections as less than 1 page assignments and some student did not count these as papers. Students also were not sure if they should count assignments like lab reports or just formal papers. A couple of students noted that they were not sure if they should count all the papers they would write or just the ones they had actually written up to that point in time (e.g., should they include their senior thesis in the paper count). Interviews with online students showed similar results.

A version of tmread was tested asking students about how many pages they read instead of how much time they spend reading. When asked about the number of pages read, students spent a significant amount of time thinking about the question and adding up reading assignments from each class. It was difficult for students to report the amount of pages actually read instead of the number of pages assigned. Students also reported the number of pages they read in books or printed articles but were not counting reading on an e-reader or online. One student admitted that she would skip this question and a few other students just picked a response based on the scale provided instead of really thinking about the number of pages they read. For online students, there were similar problems with this question but this population had more extreme under-reporting of online readings and e-book readings since a larger portion of their reading was not from printed material. Based on these responses, this question was changed for the updated survey to focus on how much time students spend reading to make the question easier to answer and to include readings that did not have page numbers (e.g., online readings).

These items were not specifically tested during every round of interviews, but no problems with answering items emerged when students were completing the survey.

Students had significant problems with the original wording of these questions (DDrace, DDeconomic, DDreligion, DDpolitical). With the original wording of the question, rather than answering the question thinking about having serious conversations with people that are different from them, several students answered the question only thinking about having serious conversations about that topic. Students seem most likely to think of talking to someone who is different from them when answering the question about race, so that item was moved up to appear first in the list. A number of students went back and forth between answering the question from these two different perspectives. To further clarify that the question is about interacting with people who are different from yourself, the items were changed to specifically reference “people” instead of discussion topics (e.g., “People of a different race…).

Other issues appeared with both versions of these items. The question takes a longer time to think about and answer, and some students were confused about what counts as a "serious conversation" and varied in what they considered a serious conversation. Some students only counted heated conversations and arguments as "serious conversations" while other students thought anything beyond superficial chit-chat counted. The question wording was changed to “discussions” instead of “serious conversations” to address this issue.

Students also differed widely in how different was “different,” and this seemed to be related to the type of institution they attended. In general, students at larger schools had a higher standard for identifying differences whereas some students from small, more homogeneous institutions had a lower

standard. For example, a number of students from a small religious school with very little diversity still answered that they often interacted with people who had different religious beliefs because, even though everyone at the school was an Evangelical Christian, they all still had slightly different religious beliefs. At other schools students thought of someone from another religion when answering this question. At the same small religious school, they also counted very small differences in economic and social background as being different (e.g. having a stay-at-home parent or not) where students at other schools only counted more obvious differences (e.g. white collar vs. blue collar).

Similar issues appeared with online students, but these students struggled more with knowing whether or not their classmates were different from them. Students only knew details about their classmates from the bios posted at the beginning of class, so they had limited knowledge about their classmates. They also felt like it was more challenging to have serious conversations since they were just talking through posts on a discussion board.

No significant problems were found with SFotherwork. One student answered the question thinking about getting help on coursework, but this seemed like an error from reading too quickly. Another student was not sure how to account for interactions with his faculty advisor on this question. Online students did not feel like these questions applied to them. They all mentioned that they were not supposed to contact their instructors outside of class or about non-course related issues. 

One online student was confused by SFdiscuss. They were not sure what counts as “in class” or “out of class” for online classes. Is talking to people in the class chat room beyond what is required for class, considered discussing course topics outside of class? Or talking to a teacher over email or chat? The student stated that they felt like it was the same as going to office hours, but was not sure if that was what the question was asking.

Students did not have problems responding to SFperform. When answering this item, most students answered thinking about grades. One student included getting her grades on Blackboard as discussing academic performance. Some students saw this as something negative and something they would only do if they were concerned about their grades.

A few students wrote in comments on the pilot about item QIadvisor being difficult to answer because they had multiple advisors. These students seemed to pick a response option that represented experiences with both advisors, or averaging between the two.

Students generally understood item QIstaff. Most students only thought of the people listed in the parenthetical. Some students also thought of dining commons staff or the custodial staff. A couple students considered student workers when answering this question. A couple of students reported on the frequency of the interaction, rather than the quality. When prompted, to think of quality not quantity, they changed their responses. For online students, this question was more difficult. Most of the online students thought of people in the parenthetical and selected “not applicable.” One student had interacted with career services and answered the question based on those interactions. Some online students were not sure who we were asking about and answered this based on their interactions with their academic advisor or financial aid counselor. Those two students thought QIstaff and QIadmin were redundant and did not understand the difference.

Students generally understood item QIadmin, and most students answered thinking of the offices listed in the parenthetical. A few students still answered on quantity of interactions, rather than quality. Online students thought of who was in the parenthetical and were able to answer the question based on interactions with financial aid office and the registrar's office.

Students generally understood these questions. A few online students felt that some of these items were less applicable to them. For example, an online student thought that SEsocial was referring to getting together with people but was not sure if they should count connecting with other students on Facebook or through other social media. A few online students mentioned that SEactivites and SEevents were also less applicable to their experiences.

Students at schools where service-learning or community projects are often required in classes, such as small, private, and religious institutions, had no problems with servcourse. Students at other schools had a harder time understanding the item and answered with more variation. Some students were not sure what "community based project" meant or what “service-learning” meant. Some students also counted any community service they did regardless of whether it was tied to a class or not. None of the online students had done service learning and felt like it would be less common at an online school. One student did question whether this would mean doing any sort of community service or just as part of a class.  Students generally understood the other high-impact practice questions.

Generally speaking, students disclosed no problem understanding what was intended by the various perceived gains items, with the exception of one item. On the first and second pilot instruments, one item referred to “acquiring a broad general education.” Interview results suggested that students had problems with understanding what exactly a broad general education includes, so the item was eventually dropped. A few students brought up that they answered “very little” on gains items because they already were strong in those areas so they did not change a lot during college. Other students marked “very little” because they felt that they grew in those areas, but not because of their institution.

One online student was confused by item attendart because they were not sure if the question was only asking about an exhibit or performance at the school or just any exhibit or performance. This student answered the question as “never” because their school did not have their own arts performances.

A few students mentioned that item tmcare was difficult to answer because you cannot put a number of hours on being a parent, “you're always a parent.”

With the item present, some online students were not sure how to answer how many presentations they gave. Students mentioned that they turned in PowerPoints as a "presentation" but have never actually given a presentation.

One student was confused by item tmprep and did not know what to count as preparing for class, wondering if it was just preparing right before class, or doing homework, too.

Students were generally able to report how many courses they took in item coursenum. A few students were not sure if they should count non-academic courses (yoga), or independent study classes. A few students also were not sure if academic term meant the semester or the academic year, but all decided on semester. All of the students, including online students, were able to easily answer the number of courses they took online, using item onlinenum.

Students did not have difficulty reporting their educational aspirations in item edaspire. One student was not sure if it meant at the current institution or ever.

A student who used to be an athlete at the school had difficulty with the item athlete. He was not sure if he should answer yes or no because he used to be on an athletic team at his institution but was not currently, and the question does not specify if it means “currently” or have you “ever been.”