WHAT IS WEIGHTING?

Weighting is the process of adjusting data to reflect differences in the number of population units that each respondent represents. For example, if a student population is 50% male but respondents are only 33% male, then male respondents are given more weight and female respondents are given less weight in the data so that the results more accurately reflect the student population. In practical terms, a weight is a number in a data file assigned to each respondent, and is used as a multiplier to adjust the number of cases used in a calculation.

WHY DOES NSSE WEIGHT?

Weighting is necessary when two conditions are present: (a) The proportion of respondents within a particular demographic variable (e.g., gender, students of color, adult students) differs substantially from their population percentages and (b) students within the subgroups differ substantially in the variables of interest (e.g., men and women show different patterns of engagement).

Analysis of these conditions compels NSSE to weight by gender, enrollment status and institutional size. Males and part-time students consistently have lower response rates than women and full-time students. From the literature and past NSSE results we know that men and women have different engagement patterns, and part-time students differ to a great extent from full-timers.

In terms of institutional size, we know that the numbers of students attending institutions within comparison groups are disproportionate to their actual population sizes. This is caused by uneven sampling and variable response rates between institutions. NSSE has also consistently observed that enrollment size is related to engagement levels.

HOW NSSE COMPUTES WEIGHTS

For each institution, sets of weights are computed separately for first year and senior students using gender and enrollment status information taken from submitted population files. Since two categories exist for each key background characteristic (e.g., male/female and full-time/part-time students), NSSE calculates specific weights for four types of students: (1) full-time males, (2) full-time females, (3) part-time males, and (4) part-time females. We refer to each of these four categories as “cells.”

The first step in NSSE’s weight calculation is represented by the following formula where pc  equals the overall population size and rc equals the number of respondents for any given cell.

This initial weight calculation serves two purposes: (1) corrects for any disproportionate representation within cells and (2) adjusts the number of respondents to original population counts during statistical calculations. Institutional data files delivered each summer include the results of this calculation under the variable WEIGHT2.

The second and final step of NSSE’s weight calculation is represented by the following formula where avg(wc-all) equals the average of WEIGHT2 for all respondents across all four cells at a particular institution whose reports are being developed.

When developing any particular school’s means, frequency or benchmark report, recognize that the value for avg(wc-all) used to divide WEIGHT2 values does not vary by institution. In essence, we divide by this constant to preserve the original respondent count for any given institution for which reports are being created, but also to ensure proportional representation of institutions within selected peer groups. In summary, when looking at reports, readers should understand that the weighted count for the institution equals the number of respondents and that the weighted count for institutions in selected peer groups reflects proportional representation, albeit an arbitrary respondent count.

In an institutional data file, WEIGHT1 contains the respondent weights as described above for the institution. Thus, those individuals at an institution wanting to reproduce their report numbers should use WEIGHT1. Using WEIGHT1 or WEIGHT2 will lead to essentially the same frequencies and means, however the counts upon which these results are based will differ (i.e., respondent versus population).

ADVANTAGES TO THIS WEIGHTING APPROACH

To summarize, this weighting approach has a few major advantages when compared to weighting processes from prior years. As mentioned previously, weights ensure that selected peer groups used to evaluate an institution’s NSSE results will not be disproportionately represented by any given institution(s), regardless of their sampling approach and response rate. Furthermore, the original respondent count is preserved for the institution whose report is being created; statistical significance tests are no longer unduly influenced by fluctuating weighted respondent counts, as occasionally occurred at large institutions with low response rates and small institutions with high response rates in 2006. Lastly, NSSE continues to weight all three major reports (means, frequency, and benchmark reports) to promote transparency, consistency of interpretation, and result validity.

CAUTION ABOUT USING NSSE WEIGHTS FOR INTRA-INSTITUTIONAL ANALYSES

Using NSSE weights is generally not appropriate for intra-institutional comparisons, such as analyzing engagement results by Honors College participation status. Weights included in institutional data files are designed to give the best estimate of the entire institution, not any particular campus sub-group. We encourage schools interested in looking more closely at their data to use a more sophisticated or locally-derived weighting approach that would address any disproportionate representation and engagement result variation by different student types.

REFERENCES AND OTHER RELATED NSSE PSYCHOMETRIC RESOURCES

References

Cohen, B. H. (2001). Explaining psychological statistics (2nd ed.). New York: John Wiley & Sons.

Höfler, M., Pfister, H., Lieb, R., & Wittchen, H. (2005). The use of weights to account for non-response and drop-out. Social Psychiatry and Psychiatric Epidemiology, 40, 291-299.

Little, R. J. A. (1993) Post-Stratification: A Modeler's Perspective. Journal of the American Statistical Association, 88, 1001-1012.

For additional information about NSSE weight calculations and weighting in general, see the following:

Chen, P. D., & Sarraf, S. (2007, June 4). Creating weights to improve survey population estimates. Workshop presented at the 47th Annual Forum of the Association for Institutional Research, Kansas City, MO.