2000 AERC Proceedings

Mary Beth Bingman
The University of Tennessee, USA

Abstract: An action research project with three adult basic education programs focused on ways to document the invisible outcomes in learners’ lives. It has led to increased understanding of how programs might identify and measure those outcomes not captured by traditional assessments.

 

Adult Basic Education (ABE) programs across the United States are confronting new demands to document the outcomes of their work. The Workforce Investment Act (1998), which funds ABE, mandates the development of performance accountability systems with a few core indicators. Many states had already developed data collection systems that are now being adapted to meet the Workforce Investment Act (WIA) mandates. At the local program level teachers faced with increased data collection requirements feel that much of what their students accomplish is not captured by any of the measures now in use. This paper discusses the findings of an action research project conducted by the Center for Literacy Studies (CLS) to increase understanding of how programs might identify and document the outcomes of adult education participation in learners’ lives, both for local program needs and for purposes of program accountability at every level. The project created new documentation methods, and perhaps more importantly, has given the researchers (ABE teachers and university-based facilitators) new insights into the processes at work in ABE programs.

Studies of ABE outcomes have rarely provided the information hoped for and have often been "seriously flawed" (Beder, 1999, p. 116). Current efforts to build a National Reporting System (NRS) for Adult Education by the U.S. Department of Education Division of Adult Education and Literacy should help remedy this situation, but the data collected in many states will be limited to that mandated by WIA. The NRS will collect data on employment, learning gains measured by standardized instruments, and continuing education and training. What one teacher referred to as the "invisible" outcomes in learners’ lives (for examples, see Bingman and Ebert, 2000) will not be documented. These outcomes include changes in what people are able to do in their lives as a result of new skills or credentials as well as changes in their sense of self. This project attempted to develop ways to document these outcomes.

Action research, described by Kuhne and Quigley (1997) as "a form of inductive, practical research that focuses on gaining a better understanding of a practice problem or achieving a real change or improvement in the practice context" (p. 23), was chosen as a way to look for new approaches to developing outcomes documentation processes for ABE. This method of inquiry allows refinement as the project progresses over time. As this project evolved the teams moved from taking part in activities that were designed by the facilitators to designing and implementing their own documentation strategies.

The project was conducted by three teams of three to six teachers and administrators from adult education programs working with staff from a university-based literacy center. The three project sites are state-funded programs located in the southern Appalachian region of the United States, in the states of Kentucky, Virginia, and Tennessee. They offer instruction in basic skills and prepare students for the General Educational Development (GED) examination.

Teams from each program met regularly with facilitators and took part in activities designed to analyze their current documentation processes and to develop new ones. The facilitators began by asking the teams to identify the various outcomes that they had seen in students and the outcomes they hoped to achieve. Examples included: arranged child care, filled out a job application, organized class notes, reading level improved, writing letters to children, built his self-esteem. The outcomes the team members named were rarely being documented by the programs.

Each team completed a "documentation matrix." They gathered the various forms and assessments used for documentation in their programs and analyzed each by answering the questions: Who does it? For whom? How often? How is it used? Key items reported? Completing the matrix enabled the teams to identify how they were collecting and using information and instances in which they were not. For example there was often duplication of student demographic information, but no program had an effective way of documenting the outcomes in learners’ lives outside the program.

An Inputs to Impacts Grid was developed, first with the Tennessee team, to help the team (and facilitators) clarify what was meant by the terms "input, output, outcomes, impact." The team determined that these applied somewhat differently to students and programs and developed a list for each. This activity clarified the processes programs used to achieve their goals – the hoped for outcomes. The Kentucky and Virginia programs added their examples to the model and used the model as they developed their documentation processes.

After several meetings doing orientation activities, the teams began to experiment with outcomes documentation. They began with activities suggested by the facilitators. These documentation processes were intended to combine instructional activities with outcomes documentation. For example, students were given a series of stem sentences such as "I used a computer to...." or "I learned to....." Another suggested activity was to hold "story circles" during which students talked about how their lives were changing as a result of participation in adult education. The teacher recorded these stories on newsprint or notes. For the most part, the teams did not find these activities effective, and they reported that their students did not want to spend the time on activities that they did not see as relevant to their learning. The teams began to work with their students to develop, test, and revise their own documentation processes.

The research of each team was impacted by various factors. The Workforce Investment Act requirements as implemented under various state plans was a factor for all three programs. However, because Virginia was in the process of putting into place a process called Quality Works that implements the WIA requirements, that team addressed government reporting requirements most directly in their work. The Kentucky and Tennessee teams were aware of the coming changes, but did not deal with them as directly as Virginia. The Tennessee team viewed the action research project as part of their ongoing process of program planning and improvement based on the Malcolm Baldrige Education Criteria for Performance Excellence, a business performance quality/program improvement framework adapted for use in educational organizations.

All three teams utilized the Equipped for the Future (EFF) framework to some extent. EFF is a national standards-based system reform initiative sponsored by the National Institute for Literacy and provides a common framework for defining, tracking, and reporting results to policymakers, as well as to students and their local programs. Developed through a multi-year field-based research process, the EFF framework consists of:

• four purposes for learning, defined originally by adult learners and validated by a wide range of adults

• three "maps" that define successful performance of the roles of worker, citizen, and family member

• thirteen activities that are common across these three roles

• sixteen skill standards, derived from the role maps, which provide specific and measurable statements of what adults need to know and be able to do, clustered in four categories: communications skills, interpersonal skills, decision-making skills, and lifelong learning skills.

The Tennessee team had been an EFF development site and had adopted EFF as the "centerpiece" of their program. EFF now structures their instructional work and frames documentation processes. The Virginia team reviewed the EFF framework and used the roles as they developed their outcomes list. The Kentucky team used the framework to identify the Common Activity that most of their students wanted to address.

The five Kentucky team members worked together in one community adult learning center. Their students included many women who were part of a welfare-to-work project. The team was intrigued by the EFF framework and saw it as a way to integrate some of the life issues their students were facing with the academic skills that were the focus of the program. They discussed the EFF role maps with learners and eventually the group (staff and learners) determined that the parent role was the one they all shared. The learners listed areas of concern and parenting issues that were of interest. The action research team sorted this list and identified an overall goal to be a better parent and a subgoal that matched one of the EFF Common Activities: "supports and encourages child’s education." They used the Inputs to Impacts grid to analyze the activity and determined that they would focus on reading to children as an activity to support children's education and that they would have parents document this on a weekly calendar.

Throughout the summer a group of parents read to their children and recorded what they read and the amount of time on a calendar. Ten students participated, and six advanced to another reading level on a standardized test. The Kentucky team found that adult learners’ self-confidence increased, family relationships improved, and learners' desire for their children to be readers has increased.

After the summer reading program ended, the team met with the parents and found the parents were enthusiastic about focusing on their children’s educations. They wanted to continue to encourage their children's reading and had ideas about how to do so. The staff designed a new form on which parents were asked to record instances of reading to their children, helping with homework, school attendance, children's use of the public library, and meeting with their children's teachers. The staff planned to collect these forms monthly and record the data in a computer database. This effort was not as successful as the summer reading program. The staff thought that the form was too complicated and not relevant to everyone. They revised the form, scheduled more parent meetings, and loaned parents cameras to use as another way to document educational activities at home. The team plans to continue the documentation effort for the rest of the school year. With parents' cooperation, they hope to have access to children's school records and use the data to document impacts for children as well as parents' activities.

From the beginning of the project, the Virginia team, whose program covered a seven-county area, focused on connecting their action research work with their efforts to develop a process for reporting required data to the state. They reviewed Virginia state documentation requirements and the forms they had been using locally and then developed a form to be used at student intake to collect the information required by the state. This included demographic information, the source of the student's information about the program, reasons for enrolling, the student's goals, test scores, and other information. They also added a list of "learner achievements," based on a short (5 item) checklist of personal, social, and academic learning skills. These were chosen as a way to begin to document more than test scores. Some were outcomes (e.g., "helped child with homework") and some were classroom activities (e.g., "worked on assigned tasks"). After discussion and several revisions, the team decided to move their outcomes list to a separate document.

The team had decided that using activity-based documentation, such as story circles and stem sentences was too time intensive. Instead they developed a checklist using some of the items from their "learner achievement" list. They first conceived of the checklist as an exit instrument, a kind of supplement to the other reporting form. It was organized around the three EFF roles, plus the category of "self." The original draft had a space to check when something was accomplished and space for comments. Examples were: use library, ask for directions, read help-wanted ads, use a computer, volunteer in child's school, and pay bills. The team decided to add a column to use at intake (for goal-setting). The learner could note which items she or he already did and those which would be indicators of progress toward one's goals. The focus would be on items relevant to the student's goals, but other accomplished items could be checked as well. As these are written, they are decontextualized competencies or activities, but could be contextualized by the learner's goals. The team talked about having a space for the goals on the form and also about giving the student a copy of the form as well, both to facilitate documentation and as motivation.

The team took this new form and tried it with several students. They noted student reactions, how long it took, and whether students could give evidence of their accomplishments. They found that most students liked the form even though it took some time to complete and that they were able to describe their accomplishments. The students added items and suggested language changes. The team found that using the list helped both the teacher and the students think about goals and outcomes. The form was revised to include student suggestions. In its current iteration the form lists forty-one items with three possible responses to each item: currently do (DO); would like to do (SET), and now can do (MET). The form has been used by several teachers who were not part of the team and they have found it useful, and it has been requested for use by the Virginia Department of Human Services office as part of their intake procedure for new clients.

The Tennessee program is an urban program operating literacy classes in an adult learning center. The program began to use the EFF as a tool for identifying learner goals and learners’ plans for achieving goals. The action research team teachers tried various ways of documenting student performance on specific standards: Listen Actively, Take Responsibility for Learning, Use Math Concepts, and Plan. One teacher conducted regular interviews with a few students; another used a teacher log. While the teachers were able to get useful information using these processes, they found the documentation to be too time intensive to be implemented program-wide. They decided to focus on one EFF standard, Take Responsibility for Learning, and to develop and test processes to document performance on this standard in the program and outcomes in learners' lives outside program.

The process that the team developed to document Take Responsibility for Learning (TRL) involved both instruction and documentation of performance. Teachers gave students a pre-survey on TRL, asking them to write briefly on the meaning of the words, why it is important, and how they might use it. A post-survey asked similar questions. For the month between the survey, teachers talked about TRL using a model story and asking students to read, write about, and discuss the story. Students then were asked to keep a journal of events in their daily lives that indicated they were taking responsibility for learning and report to the teacher. The teachers also kept logs of their observation of TRL in the class room. The team found that while the student journals were useful as a writing activity, and the teacher log helped with planning, it was the pre and post surveys that were most useful in identifying and documenting instances of students taking responsibility for learning both in class and in their everyday lives.

This action research project made progress toward its original objective: to develop new measures of impact of participation in adult basic education on the lives of students. The project also yielded important understandings about the possibilities for action research in adult education and the context in which the performance accountability mandated by the Workforce Investment Act is to be implemented. What was learned by the participants in this project has implications for practitioners and policy makers in adult basic education.

Each team developed processes that have enabled their programs to more closely examine and document certain areas of impact. They have focused on particular parts of students’ lives that were identified by the program and/or the students as areas in which they hoped to find change, and they have developed ways to document changes reported by students and observed by teachers. The teams, their programs, and their students have found these documentation efforts useful as a tool for instructional planning and for learner and project assessment. The programs plan to continue to use these documentation tools and to develop others.

The work of understanding and developing outcomes documentation also created opportunities for improvement in classroom practice. As they participated in this action research, the team members changed their understanding about aspects of their practice. Developing processes to document changes in learners’ lives meant talking to learners about topics and at a depth that was new to many team members. They found that they gained in their understanding of their students’ lives. The process of identifying desired outcomes lead to increased instructional focus on meeting learners’ goals and achieving the desired outcomes. Thinking through program processes led to increased appreciation of how different program aspects – goal setting, instruction, outcomes documentation – can be aligned. The university-based researchers, who had approached this problem as primarily one of measurement, gained new appreciation of the importance of having this alignment clearly explicated.

Practitioner research in which the teacher identifies a question of concern is used for professional development in several states. This project indicates that action research in which the question and methodologies are determined by others can also serve as a valuable professional development experience.

This action research project has led to increased understanding of how programs might identify and document the outcomes of adult education participation in learners’ lives for local program needs. It has not developed processes that are acceptable for purposes of program accountability. While some of what the Virginia team developed will serve as documentation for meeting learner goals (a program performance indicator in Virginia) most of what the teams are documenting will not be reported beyond the program level, at least not at present. The federal National Reporting System and most state systems require standardized measures of only a few outcomes. Increased flexibility on the part of state and federal policy makers in needed so that locally developed processes of a wide variety of outcomes can count as measures for program accountability. And more complex and nuanced information reporting systems must be used to report the kinds of data collected by the teams who took part in this project. While the national legislation focuses on the economic outcomes of adult education, learners have a variety of goals. Programs need to have the ability to focus on these goals as well as national mandates.

Beder, H. (1999). The outcomes and impacts of adult literacy education in the United States.

Cambridge, MA: National Center for the Study of Adult Learning and Literacy.

Bingman, M.B. & Ebert, O. (2000). ‘I've come a long way’; Learner-identified outcomes of participation in adult literacy programs. Cambridge, MA: National Center for the Study of Adult Learning and Literacy.

Kuhne, G. W. & Quigley, B. A. (1997). Understanding and using action research in practice settings. In B.A. Quigley, G.W. & Kuhne, (Eds.). Creating practical knowledge through action research: Posing problems, solving problems, and improving daily practice. New Directions for Adult & Continuing Education, No. 73. San Francisco, CA: Jossey-Bass.

Stein, S. (2000). Equipped for the Future content standards: What adults need to know and be able to do in the 21st century. Washington: National Institute for Literacy Workforce Investment Act - Title II. (1998), Public Law 105-220 - Aug. 7, 1998. 112 Stat. 936. Washington, DC: U.S. Congress.

[Home] [Search] [Conference Proceedings] [Back] [Top]