1997 AERC Proceedings

The measurement of participation in adult education.

Mary A. Collins, J. Michael Brick, and Kwang Kim

Abstract: This paper examines the disparate rates of participation in adult education that have been reported by the Current Population Survey and the National Household Education Survey. The authors examine issues including population coverage and sampling, survey nonresponse, the use of proxy respondents, and survey context effects.

 

Reliable estimates of participation in adult education (AE) are important for a number of reasons. One of the roles of these participation rates is to inform the public about the National Educational Goal that "By the year 2000, every adult American will be literate and will possess the knowledge and skills necessary to compete in a global economy and exercise the rights and responsibilities of citizenship." Participating in adult education may support achieving this objective. However, the measurement of particiption presents a number of challenges. Zemsky and Shapiro (1993), referring to training estimates, equate this effort to "measuring a mirage" and note that estimates vary "not just widely, but wildly."

This study is an assessment of reasons that estimates from the National Household Education Survey (NHES) and the Current Population Survey (CPS), two important studies that measure adult education participation, are so different. As noted in table 1, the estimates from the CPS are substantially lower than those from the NHES. The report examined a number of methodological, definitional, and operational factors that might have had an impact on the differences in the estimates. Some of these factors could be quantified so as to estimate the potential impact they had on the estimates, while other factors evade measurement given the available data. Relevant findings are summarized later in table 3 and discussed below.

The 1984 CPS estimate of participation in adult education excluding full-time college was 14 percent and for 1992 was 20 percent (table 1). An increase of 6 percent in adult education over the 8 years is reasonable given the efforts made by government and business to increase the skills of adults during this time. However, the two CPS studies used very different approaches, in particular, different approaches to selecting adults, that should have had a substantial influence on the participation estimates. The NHES indicates that some adults identified as nonparticipants at a screening stage are later found to be participants when interviewed themselves (referred to here as "switching"). If the CPS:84 had sampled all adults regardless of the responses of their participation status at the screening stage, it is estimated that the ‘switching rate’ would have caused the participation rate in 1984 to be 23 percent. This is close to the 26 percent switching-adjusted estimate from the CPS:92. This very modest increase in participation rates in adult education between 1984 and 1992 is not consistent with what experts in adult education expect, nor is it consistent with the higher rates estimated in the NHES. This raises some questions about the validity of the CPS:92 estimates.

Table 1.--Estimated number of adults age 16 and older¹ and percent participating in adult education: 1984-1995

¹Data from the 1984 survey (Hill 1987) include persons age 17 and older.
²Not available from the published report (Hill 1987).
NOTE: Standard error in parentheses.
SOURCES: 1984 and 1992 Current Population Survey (CPS), 1991 and 1995, National Household Education Survey (NHES),

Comparing the CPS and the NHES estimates, a number of factors were considered. One of the most obvious factors is the bias in the NHES estimates due to the failure to sample adults in households without telephones. However, the empirical evidence clearly indicates that the undercoverage bias in the NHES estimates is very small (less than one percent), especially when compared to the differences between the CPS and NHES estimates that range from 13 to 20 percent (Brick 1996).

A second methodological difference is the estimated bias due to nonresponse that affects both surveys, but has a larger impact on the NHES because of the lower response rate in this study. The nonresponse biases could not be estimated directly because of the lack of data on the differences between respondents and nonrespondents to the surveys. Nevertheless, by making some statistical assumptions about these differences, an upper bound on the differential nonresponse bias was estimated. The bound shows that the NHES may overestimate the participation rate by up to 5 percent more than the CPS. This is a potentially important source of bias, but it is an upper bound and the actual bias due to nonresponse is probably less than 5 percent.

A third potential source of error was the use of proxy respondents, particularly the practice in the CPS where other adults could respond for the sampled adult. In the NHES, the sampled adult was the only person who could complete the extended interview and the responses of other adults in the household were only used for sampling purposes. The use of proxy respondents was found to be an important predictor of AE participation of the adults in the sample, with those adults who responded themselves being more likely to report AE participation (table 2). By using logistic regression analysis, the potential impact of proxy reporting was estimated for the CPS:92. These findings show the estimates from the CPS:92 are biased downward by about 3 percent as a result of proxy reporting; adjusted rates are shown in table 3. No corresponding bias exists in the NHES estimates, since proxy reporting was not permitted.

Table 2.--Percent of adults participating in adult education activities, by source of report: 1992

NOTE: Excludes about 8 percent of the sample, for whom the source of the report (self/proxy) was not identified.
SOURCE: Special tabulations from the 1992 Current Population Survey (CPS).

Table 3.--Observed and predicted participation rates if all adults had self-reported, by type of participation: CPS:92

SOURCE: Special tabulations from the 1992 Current Population Survey (CPS).

The next aspect of the surveys considered was the overall context of the surveys. The literature on context effects is diverse and addresses aspects of wording, sponsorship, and survey context (Dillman and Tarnai 1991; Schwartz and Hippler 1991; Sudman and Bradburn 1974). In some cases context effects can be substantial. The first two aspects of context that were considered were the introductory approach that sets the groundwork for the survey and questionnaire wording. The NHES:95 Splice sample, a methodological study especially designed and implemented to address these issues, reveals that these two factors are not a source of the differences between the NHES and CPS estimates.

Other context issues were considered but could not be quantified. One of these was the fact that the adult education questions in the CPS were a supplement to the core CPS survey. This clearly resulted in different operational approaches that may have influenced the interviewers and the respondents (Shapiro 1987). The training of interviewers for the adult education interviews was also obviously different. The question is whether this affected the estimates and, if so, by how much. In addition, supplemental status may result in the questions holding a place of lesser importance or focus in the minds of respondents, and perhaps interviewers. The second related issue is the sponsorship of the study. Sponsorship has been shown to have an influence on the responses to a survey, but again this cannot be quantified for these studies.

Putting these sources of error together and assuming the errors are additive, much of the difference between the NHES and CPS estimates of participation could be explained. For example, the NHES:91 estimate of 33 percent may be an overestimate by as much as 5 percent due to undercoverage and nonresponse bias, while the CPS:92 estimate of 21 percent may be an underestimate by as much as 3 percent due to proxy reporting and nonresponse. The remaining difference of 4 percent (28% - 24% = 4%) could easily be accounted for by context effects.

Table 3.--Summary of factors associated with possible biases in estimates of adult education participation, by survey and possible size of bias

¹Bias is the difference between the survey estimate and expected population value. A negative bias indicates the survey estimate is less than the population value.
²Biases estimated for the CPS:92 and NHES:95.
³Bias estimated for the CPS:92.
⁴These context effects can not be estimated from the available data, but could be substantial.
SOURCES: 1984 and 1992 Current Population Survey (CPS), 1991 and 1995 National Household Education

While this approach does account for much of the difference, it is not a very satisfying solution for several reasons. First, based on the analysis of the impact of the sampling approach, the CPS:92 estimates appear to be out of line with the other estimates, including the CPS:84 estimates. Adjusting the CPS:84 estimates for not sampling persons screened as nonparticipants who might have reporting participating themselves (i.e., "switchers") results in an estimate of 23 percent, only very slightly lower than the 26 percent switching-adjusted CPS:92 estimate. Second, the differential nonresponse in the two surveys is really an upper bound and the difference between the NHES and CPS due to this source is probably much less than 5 percent. Third, it is unlikely that the error sources are actually completely additive. The errors due to coverage, nonresponse, use of proxy respondents, and context effects almost certainly interact in some complex ways. However, approximating the influence of these effects as additive is all that can be done, given the data, and appears to be a reasonable approach.

The evidence that most conflicts with this simple approach to summing the errors is the response bias study conducted in the NHES:95 (Brick et al. 1996). This study involved an intensive reinterview with a sample of NHES:95 respondents. All of the sampled participants (based on the original NHES:95 interview) who were included in that intensive reinterview study proved to be participants, while 41 percent of the sampled nonparticipants reported an adult education activity that they had not reported in the original interview. The activities reported by these nonparticipants were examined and did appear to be real activities undertaken during the time frame of the survey. The findings of the NHES:95 response bias study strongly suggest that even the NHES:95 estimates, which are larger than any previous estimates of participation, may still underestimate participation. That study suggests that the estimates of participation rates might be substantially higher than 40 percent.

These issues make it difficult to arrive at a conclusion without some ambiguity, although some points appear to be based on solid foundation. The NHES estimates are probably subject to some bias due to nonresponse, but there is little bias coming from surveying only telephone households. The evidence from an NHES:95 response bias study indicates that respondents are actually underreporting, so the overall estimates of participation in adult education are probably even larger than the estimates produced from the NHES. The CPS estimates from 1984 and earlier were biased downward due to several sources, most notably the failure to interview adults classified as nonparticipants by the household respondent, proxy nonresponse, and context effects associated with the survey being a supplement to a labor force survey. The underreporting of activities noted in the bias study for the NHES:95 is probably even greater in the CPS due to the lack of training with respect to the adult education supplement.

Although the results of this study are subject to other interpretations, a reasonable conclusion is that the NHES estimates of participation are more accurate representations of the percentage of adults who participate in some type of adult education than the CPS estimates. As the NHES response bias study suggests, the participation rate is likely to be even higher than the NHES estimates due to respondent underreporting activities. The CPS estimates are depressed by the series of factors noted above. Furthermore, the NHES:95 Splice sample shows that change in the participation rate between 1991 and 1995 is a reflection of an actual change in the rates and not an artifact of the survey methodology (Kim et al. 1995).

Since the results of this study are not definitive, it is interesting to consider options that could be used to further investigate the differences between the NHES and CPS estimates if future studies were to be conducted. A study of the CPS might be most useful to test the hypothesis that the CPS is severely biased downward. To conduct such a study, another supplement to the CPS could be conducted follwed by an intensive reinterview of a subsample of the respondents (both proxy and sampled adults) to ascertain the size of the bias in the estimates. This approach is rather expensive and should only be considered if future adult education surveys might be conducted as supplements to the CPS.

The biggest potential source of error in the NHES, other than respondent underreporting which has already been at least partially examined in the response bias study, is unit nonresponse. This source of error is a significant problem for all telephone surveys, which often have lower response rates than in-person surveys. The importance of conducting the surveys using techniques to increase the response rates should always be considered, and studies of nonresponse might be undertaken. Different study designs should be considered, such as record check studies, where appropriate and feasible.

REFERENCES

Brick, J.M. (1996). Undercoverage Bias in Estimation of Characteristics of Adults and 0- to 2-Year-Olds in the 1995 National Household Education Survey. NCES Working Paper. Washington, DC: U.S. Department of Education, National Center for Education Statistics.

Brick, J.M., Kim, K., Nolin, M.J., and Collins, M.A. (1996). Estimation of Response Bias in the NHES:95 Adult Education Survey. NCES Working Paper 96-13. Washington, DC: U.S. Department of Education, National Center for Education Statistics.

Dillman, D.A., and Tarnai, J. (1991). Mode Effects of Cognitively Designed Recall Questions: A Comparison of Answers to Telephone and Mail Surveys. In P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz, and S. Sudman. (Eds.). Measurement Error in Surveys. New York: John Wiley & Sons.

Hill, S. (1987). Trends in Adult Education: 1969-1984. Washington, DC: U.S. Department of Education, Center for Education Statistics.

Kim, K., Collins, M.A., Stowe, P., and Chandler, K. (1995). Forty percent of Adults Participate in Adult Education Activities: 1994-95. NCES Publication No. 95-823. Washington, DC: Department of Education, National Center for Education Statistics.

Schwarz, N., and Hippler, H.J. (1991). Response Alternatives: The Impact of Their Choice and Presentation Order. In P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowetz, and S. Sudman. (Eds.). Measurement Error in Surveys. New York: John Wiley & Sons.

Shapiro, G. (1987). Interviewer-Respondent Bias Resulting from Adding Supplemental Questions. Journal of Official Statistics, 3:155-168.

Sudman, S., and Bradburn, N.M. (1974). Response Effects in Surveys. Chicago: Aldine.

Zemsky, R., and Shapiro, D. (1993). On Measuring a Mirage: Why U.S. Training Numbers Don't Add Up. Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement.

[Home] [Search] [Conference Proceedings] [Back] [Top]