The Outcomes and Impact of
Adult Literacy Education in the United States
Hal Beder
Rutgers University
Abstract. This study analyzed twenty-two of the most credible outcome/impact studies in adult literacy education conducted since that late 1960s to make reasoned conclusions about program effectiveness and to identify common conceptual and methodological problems.
Objectives
This study examines the outcomes and impacts of adult literacy education in the United States. Outcomes are defined as changes in learners that result from participation in adult literacy education; impacts represent effects on the community and society in general. The study as reported here had two objectives. 1. To make reasoned inferences about the effectiveness of adult literacy education in the United States, 2. To identify common conceptual, design and methodological problems inherent in the studies reviewed.
Methodology
Based on an earlier analysis of the outcome literature on adult literacy education conducted by the author, it was not anticipated that a definitive outcome study (or studies) of adult literacy outcome/impact existed from which logical conclusions regarding program effectiveness could be inferred. When the literature was reviewed in conjunction with this study, this, in fact, proved to be the case. Consequently, the strategy here was to analyze a wide range of outcome/impact studies in order to make reasonable inferences about effectiveness from patterns of findings while taking research limitations into account. The analysis is qualitative in orientation. Although it was hypothetically possible to conduct a quantitative meta-analysis for some outcome variables, tested learning gain for example, it was determined that valid data from a sufficient number of studies did not exist. Furthermore, statistical information critical to a quantitative meta-analysis was generally not reported for the existing studies.
The first step was to identify a pool of research studies conducted since the inception of the federal adult literacy program that were available in the public domain and potentially included an outcome/impact component. ERIC and other abstracting services were searched using numerous descriptors and state and national policy makers were consulted regarding studies the search might have missed. The initial search identified approximately 120 studies. Next, abstracts were reviewed to determine which studies in the initial pool did, in fact, include outcome/impact components. These studies were ordered in hard copy when available and secured in microfiche when hard copy was not available. Seventy studies that included an outcome component were identified and acquired for assessment. Subsequently, each study was abstracted and evaluated according to the following criteria:
- The study included an outcome/impact component.
- The report was adequately documented in respect to design and methods.
- There was an adequate N.
- The sampling plan was adequate ( i.e. could and did result in external validity).
- Data collection procedures were adequate. (i.e. were not tainted by substantial attrition or biased by other factors).
- Objective measures, rather than self-report, were used to measure outcomes.
- Measures, especially tests, were valid and reliable.
- The research design included a control or comparison group.
- Inferences logically followed from the design and data.
No study fully met these criteria. Finally, those studies that were assessed as being the most credible based on the above criteria were selected for in-depth case analysis. While it was originally anticipated that ten to twelve studies would be selected for in-depth analysis, 22 studies were eventually selected. Studies were organized into six contextual categories for presentation: national studies (n=4), state-level studies (n=9), workplace literacy (n=5), welfare (n=2), and family literacy (n=2).
The data for the 22 case studies were acquired from published research reports which varied in completeness and clarity. Hence the case studies necessarily varied accordingly. For many of the studies, measuring outcome/impact was but one of several research objectives. In such cases, only the portions that pertained to outcome/impact were reflected in the case studies.
Studies selected for case analysis are listed in the reference section.
Findings
The case studies themselves were too lengthy to recapitulate here, although details of the most important among them will be discussed during the presentation. Studies varied considerably in their strengths and limitations. While the three of the national evaluations of the federally-funded adult literacy program were extensive in their scope, all were so flawed in either design or execution that generalized conclusions regarding outcome/impact were problematic. Two large scale studies of state welfare systems conducted in California and Texas, one evaluation of the state workforce readiness system conducted in Washington and the National Evaluation of the Even Start Program were well designed and included data on the outcomes and impact of adult literacy education. However, all three were conducted on special populations of learners (welfare recipients, participants seeking employment or Even Start enrollees), and thus the results could not be generalized to adult literacy education in general. Only nine state-level studies were identified as being credible, and all were methodologically limited.
Program Effectiveness
Conclusions regarding program effectiveness are "reasoned" inferences based on the extent to which the 22 studies showed consensus on specific variables and weighed by the credibility of the individual studies. That is to say, studies with large samples that included control or relevant comparison groups were weighted more heavily in making inferences than less credible studies. Findings are represented by Table 1.
Based on the data, it was concluded:
- In general, it is likely that participation in adult literacy education produces gains in employment.
- In general, participants in adult literacy education believe that their jobs improve over time. However, there is insufficient evidence to conclude that participation causes job improvement.
Table 1: Results of the Case Studies.
| empl = gains in employment. | write = self-reported gains in writing |
| bjob = acquisition of a better job | math = self-reported gains in math |
| inc = increased income | TLG = tested learning gain |
| ced = continued education | GED = GED acquisition |
| wel = termination of or reduction in public assistance | slf-c = self-confidence, self-esteem or self-concept |
| read = self-reported gains in reading | chld = impact on children’s education |
| write = self-reported gains in writing | pgoal = attainment of learner’s personal goals |
| STUDY | empl | Bejob | inc | ced | Welf | read | writ | math | TLG | GED | slf-c | chld | pgoal | METH |
| 1N | y | y | y | y | y | y | y | PP- L | ||||||
| 2N | ? | y | y | y | y | y | y | P | ||||||
| 3N | y | y | y | Y | y | y | PP-L | |||||||
| 4N | y | n | n | y | PP-L-C | |||||||||
| 5S | n | y | n | PP-L-C | ||||||||||
| 6S | y | y | y | y | PP | |||||||||
| 7S | y | y | y | PP | ||||||||||
| 8S | y | Y | y | y | y | y | y | y | y | y | y | P | ||
| 9S | y | Y | y | y | y | ? | y | ? | y | PP-C | ||||
| 10S | y | y | y | y | ? | ? | ? | ? | y | y | PP-C | |||
| 11S | y | Y | y | y | y | y | y | y | P | |||||
| 12S | y | P | ||||||||||||
| 13S | ? | ? | y | y | y | y | y | ? | ? | ? | y | P-L | ||
| 14W | n | y | PP-C | |||||||||||
| 15W | y | y | y | PP-C | ||||||||||
| 16WO | PP | |||||||||||||
| 17WO | P | |||||||||||||
| 18WO | P | |||||||||||||
| 19WO | N | y | y | y | y | P | ||||||||
| 20WO | y | n | PP | |||||||||||
| 21FL | ? | y | PP-L-C | |||||||||||
| 22FL | y | Y | y | y | y | P | ||||||||
| Total | y=10 ?=2 n=1 |
y=4 ?=0 n=1 |
y=5 ?=1 n=0 |
y=9 ?=0 n=0 |
y=5 ?=0 n=2 |
y=8 ?=1 n=0 |
y=7 ?=1 n=0 |
y=8 ?=1 n=0 |
y=4 ?=1 n=2 |
y=3 ?=2 n=0 |
y=8 ?=2 n=0 |
y=6 ?=2 n=0 |
y=6 ?=0 N=1 |
Study codes: N=national study, S=state, W=welfare, WO=workplace, FL=family literacy.
For Methods (METH), P = post measurement only, PP = pre-post, L = longitudinal measurement, C = control or comparison group.
y = study found impact on this variable, ? = inconclusive findings, n = study found no or negative impact.
- In general, it is likely that participation results in earning gain.
- In general, participation in adult literacy education has a positive influence on continued education
- Although the evidence suggests that participants in welfare-sponsored (JOBS program) adult literacy education do experience a reduction in welfare dependence, the evidence is inconclusive whether adult literacy education in general reduced welfare dependence.
- Learners perceive that participation improves their skills in reading, writing and mathematics.
- As measured by tests, the evidence in inconclusive that participants gain in reading, writing and mathematics.
- In general, adult literacy education provides gains in GED acquisition for learners entering at the adult secondary level.
- Participation has a positive impact on learners’ self-image.
- Based exclusively on self-report data, participation has a positive impact on parent’s involvement in their children’s education.
- Learners are generally able to achieve their personal goals through participation in adult literacy education.
Conceptual Design and Methodological Problems
With few exceptions, the studies analyzed for this report were flawed in ways that severely compromised the validity and utility of their findings. At best, public funds have been wasted. At worst, important planning and policy decisions have been made on inaccurate and misleading data. There are at least six causes of the flaws which are inherent in the studies reviewed for this report: inaccurate or incomplete data; over-reliance on self-report data; lack of adequate controls; lack of valid, reliable, appropriate tests; poor quality research reports and lack of relevant standards. Clearly, these problems must be avoided in future outcome/impact research if useful knowledge is to result.
Inaccurate and incomplete data. An overwhelming majority of the studies, including all those conducted at the national and state levels, collected learner data through adult literacy education programs. Common problems with data collected through programs were inaccurate learner records, failure to pre- and post-test at specified intervals, administration of inappropriate levels of tests, failure to test, high attrition of subjects between pre and post data collection, program’s withdrawal from the study before data collection was complete, and failure to forward data to researchers in a timely fashion.
Self-report data. Most studies relied on self-report data for their findings rather than on objective data. Lacking pre-data, studies that used a post-only design had little choice. For all studies, collecting self-report data was the obvious option given that hard data on such variables as employment, earnings, welfare reduction and continued education were simply not available. The exceptions were the welfare and workforce studies for whom state welfare and employment records were obtained.
Lack of controls. The most logically defensible way is through the use of an experimental design in which subjects are randomly assigned either to instruction or a control group. Because random assignment insures that participant and control groups are equivalent in all respects except participation, differences in group pre- post- gains can be solely attributed to the impact of adult literacy education. Although when inferring causality is a goal an experimental design is ideal, of all the studies reviewed, only one was able to employ one. As a result, the extent to which adult literacy education causes impact is a very difficult question to answer.
Valid and reliable tests. The most commonly used tests for basic skills gain were the TABE and the CASAS tests. Use of these tests and others was fraught with problems which included the appropriateness of the test, ceiling and floor effects, testing after reasonable intervals of instruction, test validity and reliability and high experimental mortality.
Poor quality reports. Reports of outcome/impact research must contain certain information about design and methodology for researchers and policy makers to assess the credibility of the research. This critical information includes basic research design, sampling procedures, data collection procedures, response rates, test validity and reliability, and time intervals between pre- and post-testing. Of the studies reviewed for this report, many percent lacked this vital information and had to be eliminated from further consideration. Since the credibility of their findings could not be assessed, they were essentially worthless.
Lack of standards. There are two equally important processes involved in credible outcome/impact research. The first requires competent design, execution and reporting so that valid findings result while the second entails judgment regarding whether findings represent program success. There are at least to bases for judgment and both are problematic in adult literacy education. The first is comparative judgment in which the gains of one study are compared to the gains of similar studies. However, because the outcome/impact studies of adult literacy education vary greatly in design, procedures and populations, meaningful comparative judgment is confounded. The second is normative judgment. In this case findings are assessed in relation to established standards. However, for adult literacy education such standards simply do not exist.
References (Reports used in the analysis)
Becker, W. J., Wesselius, F., and Fallon, R. (1976). Adult basic education follow-up study 1973-75. Kenosa, WI: Gateway Technical Institute.
Boggs, D. L., Buss, T. F., & Yarnell, S. M. (1979). Adult basic education in Ohio: Program impact evaluation. Adult Education. 29(2),123-140.
Center for the Study of Human Resources. (1994). Texas JOBS program evaluation: Final Report. Austin, TX: Center for the Study of Human Resources, Lyndon B. Johnson School of Public Affairs, University of Texas at Austin.
Cohen, J, Garet, M., & Condini, L. (1996). Reanalysis of data from the National Evaluation of Adult Education Programs: Methods and results. Washington, D.C.: Pelavin Research Institute.
Darkenwald, G. & Valentine, T. (1984). Outcomes and impact of adult basic education. New Brunswick, NJ: Rutgers University, Center for Adult Development.
Development Associates. (1992). National Evaluation of Adult Education programs first interim report: profiles of service providers. Arlington, VA: author.
Development Associates. (1993). National Evaluation of Adult Education programs second interim report: profiles of client characteristics. Arlington, VA: author.
Development Associates. (1994). National Evaluation of Adult Education programs third interim report: Patterns and predictors of client attendance. Arlington, VA: author.
Fisk, W. (1994). Workplace literacy across the three phases of textile manufacturing. National workplace literacy demonstration project. Performance report. Clemson, SC: Clemson University.
Greenleigh Associates, Inc. (1968). Participants in the field test of four adult basic education systems: A Follow-up study. New York: author.
Gross, A. L., & Feldman, S. (1990). Evaluation report: The workplace education program of the Central Labor Council and Consortium for Worker Education. New York, NY: Center for Advanced Study in Education: The Graduate School and University Center, City University of New York.
Hayes, A., E. (1997). The power of Even Start family literacy: A summary of findings from a follow-up study. Louisville, KY: The National Center for Family Literacy.
Jones, P., L. and Petry, J., R. (1980). Evaluation of adult basic education in Tennessee, 1980. Memphis TN: Memphis State University, College of Education.
Kent, William P. (1973). A longitudinal evaluation of the Adult Basic Education Program. Falls Church, VA: System Development Corporation.
Mahaffy, J. E. (1983). Impact evaluation of adult basic education program outcomes: ABE Evaluation Project final report. Bozeman MT: Montana State University Department of Educational Services.
Martinson. K., and Friedlander, D. (1994). GAIN: Basic education in a welfare to work program. New York, NY: Manpower Development Research Corporation.
Merrifield, J., Smith, M., Rea, K., & Shriver, T. (1993). Longitudinal study of adult literacy participants in Tennessee: Year one report. . Knoxville TN: Center for Literacy Studies, The University of Tennessee.
Merrifield, J., Smith, M., Rea, K., & Shriver, T. (1994). Longitudinal study of adult literacy participants in Tennessee: Year two report. Knoxville TN: Center for Literacy Studies, The University of Tennessee.
Mickulecky, L., and Lloyd, P. (1996). Evaluation of workplace literacy programs: A profile of effective instructional practices. Philadelphia, PA: University of Pennsylvania, National Center on Adult Literacy.
Mrowicki L., Jones, D., Locsin, T., Olivi, L. & Poindexter, C. (1995). Workplace literacy in a total quality management environment for the manufacturing and financial services industry. Final performance report. Des Plaines, IL: The Center-Resources for Education.
Paris, K. A. (1992). Evaluation of the third year of the Wisconsin Workplace Partnership Training Program, March 1, 1991 through August 31, 1992. Madison, WI: Center for Education and Work, University of Wisconsin, Madison.
Schexnayder, D. T., and Olson, J. A. (1995). Texas JOBS program evaluation. Second year impacts. Austin, TX: Center for the Study of Human Resources, Lyndon B. Johnson School of Public Affairs, University of Texas at Austin.
Slolorozano, R. (1989b). An analysis of learner progress from the second reporting cycle of the CALPEP field test: A report to the California State Librarian. Pasadena, CA: Educational Testing Service.
Solorozano, R. (1989a). An analysis of learner progress from the first reporting cycle of the CALPEP field test: A report to the California State Librarian. Pasadena, CA: Educational Testing Service.
St. Pierre, R., Swartz, J., Gamse, B., Murray, S., Deck, D., & Nickel, P. (1995). National evaluation of the Even Start Family Literacy Program: Final Report. Washington, D.C. : U.S. Department of Education, Office of Policy and Planning.
St. Pierre, R., Swartz, J., Murray, S., Deck, D., & Nickel, P. (1993). National evaluation of the Even Start Family Literacy Program: Report on effectiveness. Washington, D.C. : U.S. Department of Education, Office of Policy and Planning.
Walker, S. M., Ewart, D. M., and Whaples, G. C. (1981). Perceptions of program impact: ABE/GED in Maryland. [Paper presented at the Lifelong Learning Research Conference, College Park, MD, February 6 and 7, 1981]
Washington State Training and Education Coordinating Board. (1997). Workforce training results net impact and benefit cost evaluation. [Meeting NO. 52, March 25, 1997]. Author.
Young, M. B., Fitzgerald, N., & Morgan, M. A . (1994b). National Evaluation of Adult Education: Executive Summary. Arlington, VA: Development Associates.
Young, M. B., Fitzgerald, N., & Morgan, M. A., (1994a). National Evaluation of Adult Education programs fourth interim report: learner outcomes and program results. Arlington, VA: Development Associates.
Young, M., Hipps, J., Hanberry, G., Hopstoch, P., & Golsmat, R. (1980). As assessment of the state-administered program of the Adult Education Act: Final report. Arlington, VA: Development Associates.
[Home] [Search] [Conference Proceedings] [Back] [Top]