Are Resources and Support Necessary or
Just Nice in Post-program Application?
Judith M. Ottoson
University of British Columbia
Abstract. This study uses matched survey data (n=1356) to explore the relationship between post-educational application and five contextual variables. Significant positive associations were found between contextual and outcome variables; significant differences were found between post and follow-up ratings of contextual variables. What happens after adult education programs may have more effect on post-educational application than what happens during educational programs.
Study Purpose and Significance
This study explores the relationship between selected practice variables and the application of learning following short-term educational programs. Is there a relationship between contextual variables in practice and post-training application? At the end of training, to what extent do learners anticipate post-training contextual supports? Is there a difference between anticipated contextual supports and those reported following training? Do post-training contextual supports vary by participant or training characteristics? While most adult education intends an impact beyond organized programs, adult education is not the sole influence on whether and how learning is applied following educational programs. By understanding the post-educational experience, learners and educators can better prepare for it and evaluators can use more appropriate variables to assess the effects of adult education. "Knowing only that intended effects were not achieved is not instructive for future program planning" (Weiss, 1972, p.39). An understanding of effects needs to be informed by an understanding of process.
Background
The opportunity to study resources and other contextual variables as obstacles or supports to post-educational application grew out of an evaluation of the Center for Substance Abuse Prevention (CSAP) Training System (CTS). Over 9000 participants attended one of 250 separate events held across the U.S. over a five-year period. The 25 different kinds of programs offered by the CTS ranged from one to five days in length. They were clustered into three general types of training: (1) community partnerships, e.g., local coalitions, multi-sector agencies, (2) cultural-specific, e.g., Gathering of Native Americans, Hispanic/Latino Workshops, and (3) health professionals, e.g., doctors, nurses, counselors. Despite difference among trainings, all intended to facilitate post-educational application: "The intent of all CTS services is that people who participate in training and technical assistance will apply what they learn on the job or in their community" (US Department of Health and Human Services, 1995). Evaluation of the CTS, therefore, focused in part on post-educational application of learning, as well as the factors that influence this process.
Theoretical Framework and Literature Review:
The literature review sought to identify factors and variables which facilitate or hinder practice change in the post-educational environment. In adult education, the Cervero framework proposed four factors that influence performance following continuing professional education (CPE): individual professional, CPE program, proposed change, and social system. The CPE program and individual professional factors were found to have the strongest influence on performance; the social system the least (Cervero & Rottet, 1984). Because the authors acknowledge limitations with the social system variables used in that study, their findings encourage rather than discourage further exploration of the role of the social system in post-educational application of learning.
Other literatures identifying post-educational influences on practice change – transfer of training, diffusion, implementation, knowledge utilization, and application – have been reviewed elsewhere (Ottoson, 1997a) and are only summarized here. The transfer of training literature suggests multiple influences on post-training application including resources, supervisor and peer responses, context similarity, and opportunity for transfer. The diffusion literature indicates that the adaptation or rejection of an innovation or idea is determined not just by the nature of the innovation, but by its cultural context. The implementation literature identifies organizational structures, sources and distribution of power, opportunity to act, and resource distribution as influences on change. Current research shows differences between men and women in the kinds of contextual supports associated with application. For example, in a study of health professionals resources were associated with application among female respondents, but not among males (Patterson, forthcoming).
The Application Process Framework (APF) (Ottoson, 1995) was used to guide the survey methodology of the national CTS evaluation. The APF identifies five factors which influence post-educational application, including characteristics of the (1) educational program, (2) the idea or innovation to be applied, (3) the predisposition of the learner, and the (4) enabling and (5) reinforcing factors of the practice context. In this study, contextual variables are understood to be enabling and reinforcing influences on practice.
Research Design and Methodology:
This study draws on data from questionnaires administered to CTS participants over a two-year period and matched across time. These questionnaires were developed in a participatory process among program developers, policy makers, and evaluators and analyzed for content validity. The dependent variable was the extent to which participants I ncreased substance abuse prevention activities following the training. This measure of application aligned with policy intent of training outcomes. Five independent variables, drawn from the literature and negotiated among stakeholders, were studied: sufficient resources, encouragement from others, opportunity to apply learning, organizational support, and authority to act. The extent to which these five variables were anticipated in the practice environment was asked at the end of CTS programs; the extent to which these variables were experienced in practice was asked again on the two-month follow-up questionnaire. All six variables were measured on a five-point scale were 1= not at all and 5= substantially.
Questionnaires were administered to all CTS participants at the beginning of training (n=7915), at the immediate end of training, and mailed two or more months following training. Overall the matched rate for pre/post questionnaires was 78% (n=6171); the matched rate for pre/follow-up questionnaires was 20% (n=1553); the matched rate for pre/post/follow-up questionnaires was17% (n=1356). Matched rates varied by time and type of training, with the health professional training having the highest matched rates and cultural specific trainings the lowest.
The representativeness of the sample of matched respondents across three points in time was tested against all those who answered the pre-questionnaire on nine variables, i.e., education, gender, team attendance, extent of prevention focus in their job, and perceptions of the following: usefulness of the training, anticipated resources, intent to apply learning, extent of knowledge and skill gained from training, and extent to which the need to do their job differently was a reason for participation.
Following descriptive analyses, T-tests were used to determine differences between end of program and follow-up ratings of the five independent variables and the dependent variable. Correlations were used to explore the relationship between the independent variables and the dependent variable. One-way ANOVA was used to determine whether the availability of post-training contextual supports varied by gender or type of training. Level of significance was set at p=.01 for all tests.
Limitations of the study include survey response rate, the narrowly defined application outcome variable, and the potential ambiguity of some contextual items, such as "resources."
Findings and Conclusions
The matched sample of respondents at pre/post/follow-up was found to be significantly different from unmatched respondents on two of the nine variables tested, i.e., they had a higher level of education, c2 (4, n=7895) =145.6, p = .00, and they were more likely to have attended as teams or with others, c2 (2, n=7688) =11.74, p = .00.
Ratings on dependent and independent variables at post and follow-up are found in Table 1, along with T-test results. Correlations among and between independent and dependent variables are found in Table 2.
Table 1.
Post and follow-up ratings and T-tests for dependent and independent variables
Variable |
n matched |
post |
sd |
fl-up | sd |
T |
df |
sig |
| Increase substance abuse prevention activities | 1041 | 4.03 | .98 | 3.03 | 1.25 | 25.55 | 1041 | .000 |
| Sufficient resources | 1283 | 3.53 | 1.03 | 3.28 | 1.07 | 7.52 | 1282 | .000 |
| Authority to act | 1278 | 3.76 | 1.01 | 3.47 | 1.07 | 8.91 | 1277 | .000 |
| Encouragement from others | 1114 | 3.67 | 1.01 | 3.43 | 1.10 | 6.57 | 1113 | .000 |
| Opportunity to apply | 1219 | 3.93 | .90 | 3.64 | 1.01 | 9.32 | 1218 | .000 |
| Organizational support for changes implied by workshop | 1234 | 3.66 | 1.00 | 3.35 | 1.06 | 9.37 | 1233 | .000 |
Table 2.
Correlations among and between independent and dependent variables
IncATOD |
Resources |
Authority |
Encourage |
Opportunity |
Org. Support |
|
| IncATOD | 1.00 |
.176** |
.319** |
.281** |
.344** |
.413** |
| Resources | .410** |
.564** |
.475** |
.437** |
||
| Authority | .561** |
.709** |
.672** |
|||
| Encourage | .630** |
.615** |
||||
| Opportunity | .635** |
|||||
| Org.Support |
** = correlation is significant at the .01 level (2-tailed)
One-way ANOVA revealed no significant differences between male and female respondents on the extent to which the five contextual variables existed in the post-training context: sufficient resources F (1,1279) = 1.80, p = .18, authority to act F (1,1280) = 1.81, p = .18, encouragement from others F (1,1199) = 1.35, p = .25, opportunity to apply F (1,1285) = 1.89, p = .17, and organizational support F (1,1246) = 3.28, p = .07. In contrast, significant differences were found on all five contextual variables between participants in different types of training: sufficient resources F (2,1313) = 6.66, p = .001, authority to act F (2,1315) = 14.33, p = .000, encouragement from others F (2,1231) = 14.33, p = .000, opportunity to apply F (2,1320) = 10.17, p = .000, and organizational support F (2,1280) = 31.74, p = .000. The Bonferroni post-hoc test was used to determine where differences in post-training contextual variables existed among the three general types of training. Health professionals gave significantly lower follow-up ratings to all five contextual variables than did those participating in the cultural specific or community trainings.
Discussion and Implications
Evaluation of short-term educational programs needs to consider variables other than program characteristics and participant satisfaction in determining whether and how post-educational application occurs. This study explored five contextual variables with potential to influence post-training application.
While all correlations among and between independent and dependent variables were found to be statistically significant, the weakest associations were found between the independent variables and the dependent variable. Most notable is the weak association between sufficient resources and the dependent variable. This finding seemingly contradicts previous research in which lack of resources was identified as a barrier to application (Ottoson, 1997b). One possible explanation for these conflicting findings is the same problem Cervero and Rottet (1984) experienced in the instrumentation of practice variables. "Resources" may be too broad a term to find meaningful results and more specific items, such as money or materials, may yield different results. Another possible explanation for these findings is that resources are weakly linked with application outcomes and that other influences, such as reasons for participation, may be more strongly tied with post-training application (Cividen & Ottoson, 1995). The strong correlations among independent variables lend promise for their combination as a factor to be tested against other factors in the Application Process Framework.
Anticipated practice change and contextual supports at the end of training contrast with reported change and supports following training. While the mean rating of increase in substance abuse prevention activities was a point lower on the five point scale between post and follow-up, all rating changes on contextual variables also were significant. In general, CTS respondents left training with high ratings for the programs they had attended and intent to do something differently about substance abuse prevention (Ottoson, 1996). Practice experiences did not meet these expectations. Adult educators can prepare participants for post-training application by enabling the assessment of available post-training contextual supports. Future research may explore the extent to which educators share participant anticipation for post-training supports. Educators and evaluators might consider how inclusion of questions about post-program contextual supports can be used as a point of discussion and preparation for post-training application, not only as assessment items on an evaluation instrument.
This study barely scratched the surface in exploring the relationship between participant characteristics and post-training contextual supports. The strongest correlation was found between those with authority and opportunity for application. This study found no differences between the male and female respondents on the extent to which contextual variables existed in the post-training context. This finding needs to be followed with additional research to explore the characteristics of the male and females responding to the survey. Lastly, participants attending different types of training perceived different levels of contextual supports in the post-training context. Health professionals, found to have the lowest rated contextual supports in this study, may have been disadvantaged in terms of their usual treatment focus in this prevention-oriented training in which community-based coalitions had more contextual advantages relative to primary prevention. While contextual supports may be necessary for application, they may differ for type of training, participant, and practice context.
References
Cervero, R.M. & Rottet, S. (1984). Analyzing the effectiveness of continuing professional education: An exploratory study. Adult Education Quarterly, 34(3), 135-146.
Cividen, T. & Ottoson, J.M. (1997). Linking reasons for CPE participation with post
program application. Journal of Continuing Education in the Health Professions, 17(1), 46-55.
Ottoson, J.M. (1997a) Beyond transfer of training: Using multiple lenses to assess community education programs. In Rose, A.D. & Leahy, M.A. (eds). Assessing adult learning in diverse settings: Current issues and approaches, New Directions in Adult and Continuing Education, 75, 87-96.
Ottoson, J.M. (1997b). After the applause: Exploring multiple influences on application
following adult education programs. Adult Education Quarterly, 17(1), 46-55.
Ottoson, J.M. (1996). Training evaluation report of 1994-5 profile, feedback, and follow-up data. (Contract #277-91-2004), Pacific Institute for Research and Evaluation and the University of British Columbia, Faculty of Education. Rockville, MD: USHHS, SAMHASA, Center for Substance Abuse Prevention, Division of Community Prevention and Training, Training and Evaluating Branch.
Ottoson J.M. (1995). Use of a conceptual framework to explore multiple influences in the application of learning following a continuing education program. The Canadian Journal for the Study of Adult Education, 9(2), 1-18.
Patterson, I.M. (forthcoming). Contextual influences and post-educational application of learning among health professionals. Forthcoming Master’s thesis, University of British Columbia.
US Department of Health and Human Services, Public Health Service, Substance Abuse and Mental Health Services Administration. CSAP, 1995.
Weiss, C.H. (1972). Evaluation research. New York: Prentice-Hall.
[Home] [Search] [Conference Proceedings] [Back] [Top]