• Skip to Content
  • Skip to Main Navigation
  • Skip to Search

NSSE logo

NSSENSSENSSE

Open Search
  • NSSE
    • About NSSE
      • Conceptual Framework
      • Positions and Policies
      • Advisors
      • Partners
      • Employment
    • Survey Instruments
      • Topical Modules
      • Engagement Indicators
      • High-Impact Practices
    • Registration Details
      • Pricing
      • Campus Contacts
      • Administration Checklist
    • Administering NSSE
      • Population File
      • Customizing NSSE
      • Recruitment Method
        • Folder Name
      • Encouraging Participation
      • Terms of Participation
      • Accessibility
      • Data Security
    • Reports & Data
      • NSSE Administration Overview
      • Data Files
      • Sample Report
      • Institutional Report Guide
      • Report Builder
      • Data Summaries & Interactive Displays
    • Working with NSSE Data
      • Data Codebooks
      • Syntax
    • Psychometric Portfolio
      • Response Rates
    • NSSE Shorts (New)
      • Pricing and Registration
      • Administration Instructions
      • Item Sets
      • Reports & Data
      • Dashboard LogIn
  • FSSE
    • FSSE Portal Log-in
    • NEW Customizable Pathways
    • About FSSE
    • Survey Instruments
      • Content Modules
      • FSSE Scales
      • Disciplinary Areas
      • Consortium Questions
      • FSSE Classic (Pre 2026)
        • Main Survey
        • Topical Modules
    • Registration & Pricing
    • Administering FSSE
      • Administration Overview
      • Confidentiality
      • Customization
        • Preparing for Message Delivery
        • Population File Instructions
      • Data Security
      • IRB Protocol
        • Informed Consent
      • Locating Your Data & Results
      • Sample Reports
        • Administration Summary
        • FSSE Respondent Profile
        • FSSE Topical Module Report
        • FSSE Disciplinary Area Report
        • FSSE-NSSE Combined Report
        • FSSE Frequency Report
        • FSSE Snapshot Report
      • Terms of Participation
    • Findings, Data, & Reports
      • FSSE Overview
      • Content Summaries
      • Data Visualizations
      • Data Use Examples
    • Working with FSSE data
      • Using FSSE Data
      • Analysis Resources
      • Data User's Guide
    • Psychometric Portfolio
  • BCSSE
    • About BCSSE
    • BCSSE Survey
      • BCSSE Scales
    • Fall Check-In
    • Registration & Pricing
    • Administering BCSSE
      • Administration Protocol and Procedures
      • BCSSE Contacts
      • Demonstration Dashboard Portal
      • IRB and Participation Agreement
      • Data Security and Accessibility
    • Reports & Data
      • BCSSE Overview
      • Accessing BCSSE Data
      • Summary Tables
    • Working with BCSSE data
      • Additional Resources
    • Dashboard Log-in
  • Support & Resources
    • For Participating Institutions
      • How Institutions Use Their Data
        • Lessons from the Field
          • Institution Examples
        • NSSE Data Use in Brief
        • Search for examples
        • Displaying Results
        • Data Use Teams
      • Data & Results Guides
        • Student Success Mapping
        • Tips for More Inclusive Data Sharing and Analysis
        • Data User's Guide: Sense of Belonging
        • Accreditation Toolkits
        • Sharing and Disseminating NSSE Results
        • NSSE Data User’s Guide
        • Campuswide Mapping
        • Contextualizing NSSE Effect Sizes
        • Custom Analysis
      • Workshops and Webinars
    • For Partnerships
      • Special Projects
    • For Students & Parents
      • Pocket Guide
        • English
        • Spanish
    • For All Audiences
  • Research
    • Annual Results
      • Annual Results 2023
        • Special Report 1
        • Special Report 2
      • Annual Results 2022
        • 1. Rebounding Engagement
        • 2. Digging Deeper Into HIP Quality
        • 3. Hot Topics in Higher Ed
      • Past Annual Results
    • Publications & Presentations
      • Foundational Publications
      • Featured Publications
      • Recent Presentations
      • Lessons from the Field
      • DEEP Practice Briefs
      • Search
    • NSSE Essentials
    • NSSE Sightings (blog)
      • Search Posts
  • Institution Login
  • BLOG
  • Contact Us

Our Research: Projects,
Publications, and More

  • Home
  • NSSE
    • About NSSE
    • Survey Instruments
    • Registration Details
    • Administering NSSE
    • Reports & Data
    • Working with NSSE Data
    • Psychometric Portfolio
    • NSSE Shorts (New)
  • FSSE
    • FSSE Portal Log-in
    • NEW Customizable Pathways
    • About FSSE
    • Survey Instruments
    • Registration & Pricing
    • Administering FSSE
    • Findings, Data, & Reports
    • Working with FSSE data
    • Psychometric Portfolio
  • BCSSE
    • About BCSSE
    • BCSSE Survey
    • Fall Check-In
    • Registration & Pricing
    • Administering BCSSE
    • Reports & Data
    • Working with BCSSE data
    • Dashboard Log-in
  • Support & Resources
    • For Participating Institutions
    • For Partnerships
    • For Students & Parents
    • For All Audiences
  • Research
    • Annual Results
    • Publications & Presentations
    • NSSE Essentials
    • NSSE Sightings (blog)
  • Search
  • Institution Login
  • BLOG
  • Contact Us
  • Home
  • Research
  • NSSE Sightings (blog)

Transparent Quality: Building Trust with Psychometrics

Justin Paulsen and Allison BrckaLorenz

Tuesday, January 22, 2019

Photo courtesy of Wabash College

Whatever the data might be, someone is always going to ask, "Can you really trust those findings?" Effectively, that is a question about the validity of the data. There are a few different validity frameworks out in the literature (Borsboom, Mellenbergh, & van Heerden's, 2004; Lissitz & Samuelson, 2007; Messick, 1989;  etc.), but at the Center of Postsecondary Research, we subscribe to Messick's framework of Unified Validity (see figure below) and provide such evidences for our NSSE, FSSE, and BCSSE surveys. We provide some examples in this blog, but more can be found at our NSSE and FSSE psychometric portfolio websites. These web pages are dedicated to studies of validity, reliability, and other indicators of data quality.

Recently researchers presented guidance on building and maintaining a portfolio of data quality at the 2018 Association for Institutional Research Forum in Orlando, FL. Examples of evidence to fill a portfolio are below.

Evidence  based on Test Content

Evidence used to show that the content of the survey is representative of the content domain.

NSSE Example: In validating the Civic Engagement Topical module we examine key scholars in the literature (e.g., Ehrlich, Jacoby), ask experts in the area questions (e.g., Are we measuring what is important? Are these items actionable?), and examine existing associations' standards (e.g., Institute for Democracy & Higher Education).

Evidence based on Response Process

Evidence that indicates the relationship between the construct and the actual performance on the item.

NSSE Example: In developing the 2013 revised survey, we conducted cognitive interviews with college students to see how they interpreted the questions. An initial draft included items that asked students about both "Synthesizing an idea, experience, or line of reasoning" and "Analyzing an idea, experience, or line of reasoning." Students were unable to differentiate between synthesizing and analyzing, and so the "Synthesizing..." item was dropped, as it was a more difficult item to read.

Evidence based on Internal Structure

Evidence indicating the degree to which items relate to the conceptual framework.

Examples: NSSE and FSSE provide CFA model fits, inter-item correlations, internal consistency measures, and differential item functioning estimates of the various scales included in those surveys. These measures help users to understand the degree to which hypothesized scale items actually share common variance and measure the same latent construct.

Evidence based on Relations to Other Variables

Evidence indicating the degree to which relationships with other constructs follow expectations.

NSSE Example: NSSE examined the extent to which the NSSE scales related to second to third semester retention. The study found that after conditioning on pre-college ability, those who rated active and collaborative learning scales the highest had significantly higher retention rates than those who rated their active and collaborative learning experience lower.

Evidence based on Consequences of Testing

Evidence indicating the degree to which consequences of test score use are related to the conceptual framework of the test.

NSSE Example: NSSE solicits colleges and universities to share how they use their data and to what effect. The University of Nebraska at Lincoln shared:

NSSE findings called attention to the need to revisit UNL's learning outcomes and the structure of its general education program. UNL provided each college with a detailed report of their students' NSSE responses. Some colleges shared the results with other constituent groups (students, alumni, faculty members), and all colleges used the results as benchmark data.

  • Annual Results
  • Publications & Presentations
  • NSSE Essentials
  • NSSE Sightings (blog)
    • Search Posts

Evidence-Based Improvement in Higher Education resources and social media channels

  • Twitter
  • LinkedIn
  • Facebook
  • Instagram
  • YouTube

RELATED SITES

  • Center for Postsecondary Research
  • Indiana University Bloomington School of Education

Evidence-Based Improvement in Higher Education

Center for Postsecondary Research
Indiana University School of Education
201 N. Rose Avenue
Bloomington, IN 47405-1006
Phone: 812.856.5824
Contact Us


NSSE, FSSE, BCSSE, and the column logo are registered with the U.S. Patent and Trademark Office.
Accessibility | Privacy Notice | Copyright © 2021 The Trustees of Indiana University

Center for Postsecondary Research | Indiana University Bloomington School of Education