Tracking Learner Outcomes: One-Year versus Multi-Year Reporting Periods
From LiteracyTentWiki
Back to Assessment Information
Back to Using Data for Program Improvement
Back to Full Discussion by Threads
Tracking Learner Outcomes: One-Year versus Multi-Year Reporting Periods
Larry, and others,
Tina Luffman, and many other program administrators have observed patterns like this that suggest that a one-year time frame, a funding year, may not be the best unit of time in which to measure learner gains, except for those who are doing basic skills brush-up or who have very short-term goals like preparing for a drivers license test. I wonder if there is a possibility that the NRS might be adjusted, perhaps in a pilot at first, so that a longer period of learning, say three years, might be used to demonstrate learner gains. Of course, there would need to be intermediate measures, but accountability -- for programs and states -- might be based on a longer period of time.
It seems to me that the one-year time frame within to measure learning gains or goals accomplished comes not from K-12 or higher education, but rather Congressional expectations for job skills training. Would you agree?
Also I wonder if you or others have some examples of programs that track and report learner outcomes over several years, and use the data for program improvement.
David J. Rosen
djrosen@comcast.net
I wonder if there is enough data to even show that adult basic and ESL students stay with a program in large enough numbers to track over a longer period? The conventional wisdom of those outside of the adult basic skills network is that basic skills programs have little impact because students do not stay long to make a difference. Do we have any evidence that shows we work with the same students more than one year and that we work with a high enough number of students more than one year to make a significant difference?
Dan Wann
Professional Development Consultant
IN Adult Education Professional Development Project
dlwann@comcast.net
Dan, I know that's the perception, but I also know that we roll over about half of our students from one year to the next. . . and some of those students had rolled over the previous year as well. We've actually had to put a three year limit on some students (especially ESL).
I'm having our data person look this up as well as we can. Unfortunately, our data tends to be divided, as David indicated, in discrete yearly "lumps." We can get the information, but it's time-consuming because the data bases are designed for accountability over a contract year.
We certainly do have a lot of students who come in with short term goals and leave when these are accomplished. We also have a lot of stop out students, who have to put goals on the back burner while they work out other issues. I think, however, we do keep a significant number of students over time. I think for my own little research project, I'm going to investigate gains over multiple years.
Karen Mundie
Associate Director
Greater Pittsburgh Literacy Council
100 Sheridan Square, 4th Floor
Pittsburgh, PA 15206
412 661-7323 (ext 101)
kmundie@gplc.org
The Longitudinal Study of Adult Learning has been following a target population of ABE learners over a long period of time. It’s finding exactly the pattern that others have been describing – many adults participate in programs over a series of “episodes” which often span multiple years (NRS accounting periods). When we’ve presented these data, we’ve suggested that NRS will not capture all of the impact that programs have on learning in part because of its short-term focus for measuring both participation and outcomes. I wonder if states could get waivers on a pilot basis to experiment with longer reporting periods as David Rosen suggested.
Steve Reder
Last year sometime, I remember hearing Ajit talk about how CT was tracking students over a period of years. They had some very interesting information about the percent of students who did and did not come back the second year. I hope he will share it here.
Kathy Olson
Training Specialist
Our oldest few:
Intake 1997 Grade level equivalent 2, now 11
Intake 1998 GLE K, now 7
Intake 1998 GLE 2, now 7
Intake 1998 GLE 3, now 11
Intake 2000 GLE K, now 4
And all of them have been "stop-out" students, with very uneven progress, at times even going down on the CASAS scale. They tend not to improve as quickly as NRS likes, but they do improve.
There is another set of students that need to continue coming just to keep whatever skills they have. They can be really hard on NRS statistics!
Mary G. Beheler
Tri-State Literacy
455 Ninth Street
Huntington, WV 25701
304 528-5700, ext 156
We too see this happening. Just now we posted the official GED test scores (passing!) for a student who started in June 2003 (65 hours) and stopped out in July 2003. She came back March 2004 and was in and out through March 2005. This March and April (2007!) she took and passed all of the official GED tests. So this is a success story! BUT, we get no credit as far as NRS is concerned because the student is not enrolled this program year. YES, we support efforts to report results over multi-year periods. Thanks.
Barbara Arguedas
Santa Fe Community College Adult Basic Education
Hi Dan, David, Kathy, and others,
I think that many states do have longitudinal data systems where a single student identifier is used across providers and across fiscal years. In those cases, states should be able to look at persistence and success rates across multiple years. We have been studying this issue for some months and find that the number (percent) of learners who return to adult education in a future fiscal year is fewer than we had expected. We are also beginning to notice that this return rate of non-graduates varies significantly (between 35% and 65%) among the three secondary completion options available in CT: the GED Preparation, the Adult High School Credit Diploma Program and the National External Diploma Program. Typically, the GED preparation reflects the lowest return rates. We are now beginning to look at the success rates of those students who persist for more than one fiscal year.
With regard to David’s initial question about tracking learning gains across fiscal years, in the adult education system, many learners begin in January/February or even later in a fiscal year but are held to same expectation of having to demonstrating a learning gain by June 30. These learners have significantly less time within which to achieve that learning gain as compared to those who started in the fall. As an example, 42% of students who started ESL in Connecticut by October 2005 completed an NRS level by June 30, 2006 as compared to 33% among those who started after October.
By contrast, the U.S. Department of Labor’s implementation of NRS as part of its Common Measures policy for out-of-school youth allows the persistent learner, 12 months from the start date before being considered in calculations relative to learning gains, even if that 12-month period spans two fiscal years.
Ajit Gopalakrishnan
Connecticut Department of Education
25 Industrial Park Road
Middletown, CT 06457
Phone: (860) 807-2125
Fax: (860) 807-2062
Email: ajit.gopalakrishnan@ct.gov
We have been fortunate with continuity of our students. Our adult ESL program covers all NRS levels, and we have had many students who began in the literacy or low beginner level and progressed through the years through the low advanced level. We are allowed to keep them if they show one EFL gain in 450 hours. Although we try to get follow-up information, we don't always learn why every student leaves, but frequently we receive the information on students who leave our program and go on to higher education programs. Often these students work, but are able to schedule classes around their job times or vice versa.
Jo Pamment
Director Adult Ed. ESL
Haslett Public Schools
1118 S. Harrison
East Lansing, Michigan 48823
TEL: 517 337-8353
FAX: 517 337-3195
E-Mail: pammenjk@haslett.k12.mi.us
I'd like to respond to David Rosen's question regarding programs which track student gains beyond the bounds of the fiscal year. At my program, students stay for an average of 15 months, and they are permitted to stay for five years if they wish (and many of them do), so we track them from start to finish. We are a small program, serving about 250 students a year, so we take a fairly low-tech approach. After pretesting with BEST "Plus" and BEST Literacy, we plot their scores on line graphs (two separate graphs - one for each of the two tests) and continue to plot their scores for each subsequent test. As time goes by, the lines representing a student's progress slope gradually upward across the pages. This is a graphic that the students can easily understand, and it enables us to see which skill areas require emphasis for individual students. Having a complete picture of a student's testing history also allows us to show students that an occasional disappointing test score may be an aberration from a generally upward trend.
Kate Diggins
Director of Adult Education
Guadalupe Schools
(801) 531-6100 (x-1107)
Kate,
This is a good example of the type of analyses I suggested in my previous post on this topic. Thanks, Kate!
Larry Condelli
Kate--
How do you account for the long retention rate? Other programs report this as a difficulty.
Thanks.
Andrea Wilder
Well, I think I would attribute it to quite a bit of individual attention. Our students are placed in small groups and work with volunteer mentor-tutors. The groups are maintained and supported by the professional staff; so this means that there's a web of supportive relationships to help students with problems that may lead to "stop-out". If a student is absent twice without calling in, we call to find out what's happening. We try, too, to assist in overcoming the most typical barriers to education, so we have childcare and a small shuttle bus that can do door-to-door service to 15 people, if they live within a couple miles of the school. To be honest, I've never been sure that our retention was good, because I've never had data to compare it with. Does anyone know what a typical retention rate is for a non-intensive adult ESL program?
Kate Diggins
Director of Adult Education
Guadalupe Schools
Dan, Karen, Steve and David,
You all have raised the issue of changing the NRS reporting period from one year to multiple years. While this is off the topic of using data, I will give a quick response.
First, the mandate is to have an annual reporting system so some information is required each year top report to Congress. Beyond this, this topic has come up and been considered multiple times and there is some flexibility with ED to make some changes to the reporting period, if there is a compelling reason that can be demonstrated. Our analyses of several states' data (not NRS reported data but individual student data from over several years), however, including some very large states, is that there are proportionally very few students who continue year to year (on the order of 5 percent or less in some states) and it does not appear at this time that it would make a difference in performance data at the national level, as Dan Wann suggested.
NRS is a national system so with some local programs (such as Karen's) or other states, there may be large numbers of students who continue year to year and in those instances it might be advisable to look at and report multi-year data. To bring us back to our topic of using data, this would be a good analysis a state or local program to pursue-- to look at returning and continuing students and see how they differ in outcomes and other factors from students who stay a short time. We also can rely on research, such as Steve Reder's study to look at long-term relationships, which if compelling, could result in a change to the reporting period in the future.
Larry Condelli
Thanks for those points, Larry. Besides giving us a broader view of more complex patterns of participation, multi-year data frames will probably do a better job at revealing program impacts on longer term outcomes such as postsecondary education and employment. It’s good to hear that there may be flexibility within ED for experimentation such as this.
Steve Reder
Larry and Steve,
I agree. There are important gains that are missed when one only looks at data within a year. Longitudinal data would give us a much better view of participant progress, both for those programs in which a significant number of adults continue more than one term, as well as those in which participants stop out and return.
Jodi Crandall
Dan, you have raised some critically important questions. NRS data indicates that only about 36% of ESL students "complete a level" each year. This is cause for concern, because the same data show that the vast majority of ESL students are at the lowest levels of proficiency and have low levels of education in their native countries. However, the NRS data is not definitive for a number of reasons -- such as low rates of re-test in many programs, the use of tests that do not measure the full range of English language skills, and the fact that data is reported only for a single year (students may persist in programs long enough to achieve much larger learning gains).
As a first step toward finding out more about the learning gains and persistence of ESL students, Jodi Crandall and I worked with the faculty and staff at 5 highly regarded community college programs to use student record data as a means of determining both learning gains and persistence rates. At several of the colleges we were able to track the learning gains and persistence of students for as long as seven years. At most of the colleges, the measures of learning gains used was completion of one or more additional levels AS THE COLLEGE DEFINED THE LEVELS. Both the definition of levels and the standards of completion took account of test scores (of the sort reported to the NRS), but they also took account of other measures of student achievement (including proficiency in all core ESL skills).
Needless to say, our findings were fairly complex and cannot be adequately set forth here. In summary, however, we found that at most 30% of students persist for 2-3 college terms and complete more than 2-3 levels over a seven year period. More than 40-50% of students do not complete a level or complete only a single level at any time over a seven year period. Although we could not be sure, it appears that low level students were more likely to persist than higher level students. About 10-15% of adult education ESL students enrolled in credit ESL at these colleges, and the number who eventually enrolled in academic credit courses was in the single digits.
We also found that all the colleges we examined employ strategies that significantly improve the rate of learning gains and retention. Among these were high intensity/managed enrollment classes (more than 3-6 hours per week), strategies to encourage learning outside the classroom, appropriate uses of technology for instruction, co-enrollment of adult education ESL students in vocational programs taught in English, curricular designs that insure instruction is relevant to the interests of students (such as Freirian approaches), enriched guidance/counseling/support services, setting high expectations, and VESL programs. Unfortunately, only small numbers of students have access to most of these strategies at most colleges, because they are far more expensive on a per student basis than is standard ESL instruction. Conversely, it appears that large numbers of students would like to make the commitment to enhanced programs, if they were available.
The results of our research were published by the Council for the Advancement of Adult Literacy (under the auspices of which the research was conducted) in February as the report " Passing the Torch: Strategies for Innovation in Community College ESL." This is available at the CAAL website: www.caalusa.org. CAAL will be publishing more of the data we gathered later this spring.
Among the "take away" messages we gathered from our work were: 1) The use of longitudinal (multi-year) data and holistic assessments of learning gains are essential for understanding and improving the effectiveness of ESL programs. In many programs it is feasible to gather and use longitudinal data in this way, but few programs do so due to a variety of perceived constraints and/or a lack of support for data analysis by their host institutions. 2) Research can be very helpful in program improvement, but it requires a substantial commitment on the part of programs to gather relevant data and tease out its lessons on an on-going basis. Programs should receive far more support for this. 3) It is possible to greatly improve ESL program outcomes using a variety of strategies, but these require a larger investment in instruction per student -- an investment that we believe is well worth the cost. 4) Numbers do not speak for themselves. For example, low rates of learning gains must be read in the context of the goals that both students and programs set for ESL instruction. It may be that some portion of students legitimately wish to use ESL programs as an initial platform to learn SOME English, and that their learning gains after separating from programs are substantial. Too little is known about this. Conversely, we found that the more students learn, the more ambitious their learning goals become. Because numbers do not speak for themselves, it is all the more important for individual programs and state agencies to invest in the use of research for program improvement and to ACTUALLY USE IT for these purposes. Too often over-burdened ESL faculty and staff consider research an after-thought. They need the time, encouragement, resources, and training to development "continuous program improvement" models to their work.
Forrest Chisman
Council for the Advancement of Adult Literacy
