Over What Period of Time Should We Measure Gains?
From LiteracyTentWiki
Back to Assessment Information
The following discussion took place on the National Literacy Advocacy (NLA) Discussion List from October 11 - 17.
View the full archives of this discussion at [1]
Over What Period of Time Should We Measure Gains?
NLA Discussion List
[2]
October 11-17, 2006
As we discuss standards and performance measurements in adult education, I think it important to recognize that adults often make educational gains and achieve educational goals in time periods that extend well beyond the fiscal year, not uncommonly to years. In both of our most recent graduations (September 2006 and November 2005), we counted among our graduates students who spent six years studying to earn their high school credentials. Every term we welcome back students who attended Academy of Hope in a previous term or year or even decade. If we extended our time-frame for measuring educational gain to two or three years, we might gain a better perspective on successes in the field.
Patricia DeFerrari
Academy of Hope
patricia@aohdc.org
Patricia and others,
If we had to pick a period of time during which to measure student achievement, I think almost no one would pick one year. Some would pick six-nine months (September to June), some would pick annually over three-five years, and some might pick annually over ten years. We have one year performance measurements because that is the way Congress -- and states -- appropriate funds, a year at a time.
But if, as you suggest, one year is not enough time to measure progress for adult learners, if as a field we agree that this is the wrong timeframe, I believe we could unhitch assessment from one-year funding cycles, just as K-12 and higher education do. I believe we could persuade Congress -- and could then re-design the NRS to capture a student's learning over multiple years. After all, that would provide greater, not less accountability. However, then we would have a new set of problems: follow-up.
Here's an opportunity to think about a policy question. I would like to know if you agree with Patricia (and me) that measuring student progress within a one year span is inappropriate for adult literacy education, and if so, whether or not you think we can change the policy, and if so, how.
David J. Rosen
Adult Literacy Advocate
DJRosen@theworld.com
This would depend on the number of hours of instruction.
Maureen Hoyt
Basic Education Manager
ACYR
602-252-6721ext 223
fax: 602-252-2952
www.azcallateen.k12.az.us
www.az-aall.org
I agree with Maureen. If a learner has received 80 - 100 hours of instruction within two or three months, the learner will most likely demonstrate a learning gain. If the learner has received 300 - 400 hours within an 8 to 10 month period, the learner will most likely complete an educational functioning level.
Of course, we all realize that there are some learners who can demonstrate measurable progress with fewer hours of instruction while there are learners who will need more hours of instruction before they can demonstrate a learning gain.
In Kansas, the NRS has confirmed what we had suspected for years--the intensity of instruction is a critical factor in achieving learning gains. While we have long recognized the importance of quality classroom instruction and the need for a correlation between curriculum and assessment, and while adult education has recognized the direct correlation between hours of instruction and learning gains, adult education has been reluctant to inform adult learners about the direct correlation between the intensity of instruction and learning gains. A group of learners who receives 100 hours of instruction spread out over nine months is less likely to achieve the level of progress as a group of learners who receives 100 hours of instruction within a three-month time frame.
What we have discovered in Kansas is that a learner who participates for less than 8 hours per week in an instructional environment spends a significant amount of "class" time in review. A learner who participates for a minimum of 10 hours per week spends minimal time in review and maximum time in "new" learning. (We encourage the minimum of 10 hours per week because a minimum of 10 hours per week results in significant progress beyond even the minimum of 8 hours per week.) In other words, 100 hours of instruction spread out over six months is not the same of 100 hours of instruction received in three months.
Of course, every adult learner cannot participate for 10, or even 8, hours per week. However, adult learners should be informed that the intensity of their involvement will directly impact their progress in the program. In Kansas, we believe adult learners deserve to be "informed customers."
Dianne S. Glass
Director of Adult Education
Kansas Board of Regents
1000 SW Jackson Street, Suite 520
Topeka, KS 66612-1368
785.296.7159
Phone: 785.296.7159
FAX: 785.296.0983
dglass@ksbor.org
David, I have not thought about this in the past, but it seems logical that significant educational gains take longer than a year in most cases. The programs at Creative Workplace Learning are measured in hours. The CWL Adult Diploma Program consisting of 12 courses that meet for a total of 240 hours takes longer than a year for people to graduate. The CWL Adult Diploma Program at Polaroid was part of the Harvard study on adult learning development which was published as a NCSAAL Report #19 in August 2001. The English as a Working Language classes meet for 60 hours each cycle and usually run for more than 1 cycle, since often 60 hrs is not enough time to show significant gains. Finally the Computers for the Office Training Program combining Microsoft Office skills with ABE also runs for 240 hours followed by an internship that goes for 60 hours. This program has been offered in a computer center in a public housing development. Around 70% of the students who enrolled completed the program with more than half getting jobs or continuing their education in a community college or technical school. I think that what is important in the end is the curriculum and the methods used for teaching. I am not certain how that is shown in some of these reporting systems.
Lloyd David, EdD.
Creative Workplace Learning
311 Washington Street
Brighton, MA 02135
Tel : 617-746-1260
FAX: 617-782-0136
I have just been reading a British NRDC report on finding in relation to earnings and participation. They say "The returns for adults participating in an English or basic mathematics course appear to be greatest when looking at earnings three or more years after attending the course."
John Benseman PhD
Director of Research & Evaluation, Upskilling NZ
(based at Department of Labour)
DDI 04 915 4195 - Cell: 021 0489 143
Faculty of Education, The University of Auckland
PB 92019, Auckland, NZ
mailto:j.benseman@auckland.ac.nz>
John,
How does accountability reporting work in New Zealand? Are programs funded a year at a time there? Are student outcomes reported within that one-year fiscal period? Are any data (learning level gains, learner goals attained, qualifications or licenses obtained, admissions to apprenticeships or higher education, work status, job advancement, income, social inclusion) collected on those students in subsequent years? How many hours must a student be enrolled before s/he is "counted" in the reporting system? (In the U.S. it's 12 hours, which I have never understood. Anyone know why it's 12 hours? What was the original rationale for that? Basic skills "brush-up" for the GED?)
John, If you could design a reporting system that made sense for New Zealand, what would be some of the elements?
Others: If you could design a reporting system that captured and reported learner outcomes (and impact) over time, what would it look like?
David J. Rosen
Adult Literacy Advocate
DJRosen@theworld.com
Maybe I'm missing something here, but I think current ED/state policy is NOT that adult learners be assessed once a year, but that they be assessed after X hours of instruction, with the X determined on the basis of the test developer's recommendations and maybe state policy. The cumulative results are reported annually to ED. Instructional hours, though, are the driver, not calendar days.
Regular assessment is critical to good instruction. I can't imagine instructors or adult learners agreeing that they'd prefer to wait a year or two before learning gains are assessed. Both want regular snapshots of how well they are doing.
The more important issue, I think, is that states and local programs need to develop richer, longitudinal data systems that follow adult learners across fiscal (and calendar) years. This is something that ED has been promoting in adult ed and K-12 for the last few years, but I don't have any idea if states are moving in this direction. As has been pointed out previously, adult learners don't enroll at the start of a fiscal year and exit when it ends. Longitudinal data systems that track learners' participation in adult education month-by-month, their instructional hours, periodic assessment results, and other data points (e.g., what time of day was the class offered?) etc. would give states and local programs much more useful and "actionable" data about how to improve their programs. Longitudinal data systems also open up some new possibilities for accountability. This needs to be a priority, I think, for the field.
If the objection to annual reporting is really drive more by concern that it makes adult ed "look bad," I think it's safe to set aside that worry right now. Spend some time looking closely at the NRS data, and data for other similar programs, if you can find it. Adult educators are doing an extraordinary job. I hope they soon receive the recognition they merit--and I think they will.
Run Spot Run
Run Spot Run,
What programs have you compared with adult education? K-12? Higher ed? And could you summarize how adult ed programs compare with each?
Our goal should be, as you have suggested, to get an accurate short-term and long-term picture of what adult learners are -- or are not -- learning, and how this affects their work status, income, and perhaps social inclusion. To do that, we need both short-term post- assessments (perhaps after 50-60 hours, and after 100 hours) and the longitudinal data. I believe that what we have now, in most states, is a situation where we post-test once, within the fiscal year. I don't think that's capturing very well what programs and learners are -- or are not -- accomplishing. If we had a reporting system that did capture the short and long term learning gains and other outcomes, we might learn some useful things to improve programs. For example, we might learn that people who enroll 12-20 hours are wasting their time, or alternatively, we might learn that for many this is a "toe in the water" experience and although they don't make gains the first or second time, they do return later and do eventually -- over years -- make gains. We have no way to track that now. We might be able to confirm, for example, fin other states the findings Dianne Glass reports from Kansas.
One objection might be that this would call for more assessment, and take more resources away from instruction, thus resulting in fewer learning gains. I would agree that assessment needs to be costed out accurately and paid for. Another objection might be that, if our goal is data for program improvement and for capturing what outcomes programs -- or the state -- achieves, we don't need to test every student each time. We could do a random sample of programs each year. This makes sense to me, and may be the only way we could afford to collect the data. I wonder if anyone has attempted to cost out a model like this.
David J. Rosen
Adult Literacy Advocate
DJRosen@theworld.com
Dear Friends:
I'd like to offer one program's perspective on the issue of "over what period of time" should we measure goals. I do not know if our experience can be extrapolated to others, or if others have had like experiences. At the moment I am struggling to figure out if we are unusual, or if I am just more willing to share our "dirty linen". You tell me. 30% of our learners are ESL. Of the remaining 70% (ABE learners), half of those score below 4th grade at entry-level. 40% have identified disabilities, and 30% are over 55.
Effective this fiscal year, my program has "disconnected" from the AELS precisely because the NRS standard that reports results every year has proven to be a standard we cannot meet. The standard requires that a learner must complete one NRS level in a year. There is no tracking of learners in our state to count how many advance a measurable amount within an NRS level.(More than 80% of our learners do, but the NRS doesn't want to know about it, at least not as structured in SC.) If the learner begins at the bottom of that NRS level, this means the learner must achieve two grade equivalents in a year, twice as much as a K-12 student is expected to achieve. For FY 06-07, the "achievement target" in SC for Beginning ABE Literacy is 55%, for Beginning Basic Education learners 60%. (No rationale exists for the two different standards, by the way, and the 60% standard is the HIGHEST for any level. Go figure.) These targets lump together computer literacy and numeracy with reading, that is, the programs report whichever post-test score shows the greatest advance. A high percentage of the scores are math or computer literacy scores. But our focus is reading. There is no question that it takes MUCH longer to show a measurable improvement in reading than in math or computer skills.
That is not to say we have failed. We chose to "disconnect" by just not applying for the funding, and in the process, we have returned to what was our status in 1993 when we received no federal or state funding and set our own benchmarks for success. But the resulting lack of funding has slashed the number of learners we can serve by over half, and our service center is now open only 2 1/2 days per week. We "disconnected" anyway, taking a considerable risk that we may not be able to survive, because we want to continue to focus on reading and ESL, and to serve the hardest to serve. We want to do this because others do not serve this group as well, (so our learners tell us) and because the need for ESL is overwhelming. We want to support the school districts' efforts to move higher-level learners to a GED or diploma instead of duplicating that service. This is, of course, a local decision. But the NRS system, and especially the issue of "over what period of time", may in the end provide our undoing. We simply cannot find funding to replace the lost WIA money, at least not fast enough.
If the "period of time" issue can be resolved, we would likely re-apply for WIA funding. (I daresay there are those who are pleased not to share WIA funding with non-profits, anyway, and who will come forward to block efforts to change policy that would encourage providers of reading instruction to apply.)
But David Rosen, my muse, asks what should be the field's position. So here is what I think is needed:
1-A different, "slower" standard specifically for improvement of READING skills, teasing out this data using specifically reading sub-test scores. What should the standard be? How can anyone know that without baseline data? Such a policy change would be "back to square one" to establish a reasonable standard, unless there are states that are already tracking student performance this way. I'd suggest start with collecting data every year, but hold programs accountable for "progress" over two or more years. That means collected data shouldn't even be published until the specified period has ended. Publication of "preliminary" data has an "accountability" effect even when it is incomplete.
2-An accommodation to allow measured progress within a level to "count".
If there is anything I can do to join with anyone to get policies changed so funding will flow to the hardest-to-serve as well as to those who are "marketable", I will do it. Maybe someone on this list can point me in the right direction.
However, I am not optimistic that this will happen. I think the entire WIA and the NRS are deliberately designed to focus taxpayers' resources on those adults who are most likely to become employable the fastest. That is not the group I serve. Until literacy (read that reading ability) is framed as a civil right, not a marketable skill, and until this country focuses on what is right, instead of what is expedient, we will continue to have dismal "below basic" numbers reflected in studies like the NAAL.
I have just returned from the ProLiteracy conference, where I was heartened to hear Robert Wedgeworth speak of literacy not as a set of skills but a "predictor of quality of life." He is right. Unfortunately he is retiring.
Because adult ed is so marginalized, we have been forced, especially the last six years, to tell people what they want to hear ("adult ed is a good investment") instead of telling them the truth. One untold truth is that many providers, including my 2 1/2 day a week program, pay little more than lip service to reading instruction for adults, all adults, because there are so few resources from anywhere to apply to the problem. Dirty linen indeed. But no one will own up to this, least of all my colleagues. I am willing to speak up about it because, at this point, my learners and I have very little left to lose.
"Turning Pages into Possibilities", Debbie
Deborah W. Yoho
Director, Turning Pages
a community service of Volunteers of America Carolinas
formerly the Greater Columbia Literacy Council
2728 Devine Street, Columbia, SC 29205
803-765-2555
Fax 803-799-8417
dwyoho@earthlink.net
Dear Patricia, David, Diane, Lloyd, Run Spot Run, and others,
I’ve followed this discussion with great interest. I think that the issue of when to test, how often, and what type of testing is one of our greatest concerns in Adult Literacy in terms of policy and practice. I would definitely agree that measuring progress in one year intervals is not supportive of the work that needs to be done, and is indeed a mechanism not based on the learning process, but on the funding distribution process, which in turn, is tied directly to the results produced (or not produced) via these very tests. An unfortunate paradox.
I have some questions and comments for folks who have posted throughout the discussion, and so I’ll take a quote out of their email and add my reply. I also am preparing this discussion to post on the Assessment Discussion List (http://www.nifl.gov/mailman/listinfo/Assessment) and further it along; please join us there if you like as well.
David: in your reply to Patricia’s initial email, you note: “I believe we could persuade Congress -- and could then re-design the NRS to capture a student's learning over multiple years. After all, that would provide greater, not less accountability. However, then we would have a new set of problems: follow-up.”
Marie: can you tell us a bit more about what you see as potential problems with follow-up were our system changed to multi-year funding?
Maureen and Dianne: you both mention hours of instruction (as do others) in terms of a key to predicting gains. I would agree – especially with Dianne’s point that the “intensity of instruction” is a critical factor for achievement.
Marie: everyone: are there ways to think about “intensity of instruction” and “number of hours” differently? For example, does intensity equal hours/timeframe, or could it be that intensity can mean that in addition to hours of instruction and over what period of time (3 months vs. 9 months, etc), other factors come into play in order to help people achieve their goals? I’m thinking about activities that learners might do on their own outside of class, or work provided by the program for learners to do outside of class, that adds to the intensity (“outside of class strategies”). Lloyd noted that his program also includes an internship component – this for me is different than hours and intensity. What do you think?
I believe that we are hinged on hours and intensity because our system is already set up that way and it’s difficult to imagine another way to approach or structure our programming. Mastery of skills/abilities should be measured by performance, not by an expectation that after so many hours, you’ll clinch the test (which may or may not be a performance-based test).
Run Spot Run: you note that “ED/state policy is NOT that adult learners be assessed once a year, but that they be assessed after X hours of instruction, with the X determined on the basis of the test developer's recommendations and maybe state policy. The cumulative results are reported annually to ED. Instructional hours, though, are the driver, not calendar days.”
Marie: this may be true but I think it’s prudent to understand where this has come from initially. These cycles have all been set up to serve funding cycles – and test publishers follow policy they do not influence it, at least not at this time. In many ways, calendar days DO in fact drive this – because so many adult students cannot achieve their goals within the timeframe of the grant period – which is marked in calendar days.
Also, you note: “Regular assessment is critical to good instruction. I can't imagine instructors or adult learners agreeing that they'd prefer to wait a year or two before learning gains are assessed. Both want regular snapshots of how well they are doing.”
Marie: absolutely. The difficulty here is that these are two types of assessment that you are talking about. When you say that regular assessment is critical to good instruction, you are talking about formative assessment, which is not tracked by any systems at this time (except of course within classrooms and programs). Incremental gains and formative assessment are what inform the classroom and allows teachers and students to understand where they are and where they need to go. It should be the cornerstone of our structure, but it is not even on the radar screen. What IS tracked is the summative assessment piece – summative assessment is what we report on and what is used to determine our worth as educational programming – by year. This is not useful for the classroom, its purpose is for accountability for funding. So in fact, there are no ‘snapshots’ for policy to examine at this time – annual reporting has no choice but to end up highlighting the extremes through summative testing. This is a fundamental struggle for programs.
Finally, you and others have suggested that we examine data to make some determinations about what is working or not within and across states and programs. Unfortunately, this is not possible at this time. We do not have a national set of standards, nor a national framework against which to judge. Nor do states use the same assessment tools, processes, or data collection and interpretation. In other words, there is nothing standardized about our entire system, therefore, we simply cannot make comparisons or determinations on anything other than very small scales.
In my state of Massachusetts, we are trying very hard to build a system based upon the stated goals of the students. Capturing this achievement is difficult, to put it mildly. One of the most important pieces of reaching your goals is by recognizing the tiny steps along the way that provide a foundation upon which to build. We are not there yet in my state but we’re trying. We are not yet good at capturing ‘short term goals’ – never mind how to collect this info and report it as outcome data. (I actually would go beyond this and note that short term goals are also made up of mini steps – something that could prove to be hugely powerful if we could figure out how to track, collect, interpret, and report on this type of outcome.) As well, this state requires 3 assessments per year: pre-, post-, and mid-. At first glance this would appear a great step in the right direction for attempting to collect shorter term goals (and it is in many ways); however, the types of tools we are using to do this are still summative assessment tools. In addition, the reason that the mid-year test was instituted was not in order to capture incremental gains, but rather to make sure that people got tested before they dropped out so that the post-test data yields more positively. Not a bad thing in and of itself, but clearly a response to policy, not educational practice.
Back to David: you note: “Another objection might be that, if our goal is data for program improvement and for capturing what outcomes programs -- or the state --achieves, we don't need to test every student each time. We could do a random sample of programs each year. This makes sense to me, and may be the only way we could afford to collect the data. I wonder if anyone has attempted to cost out a model like this.”
Marie: yes, I highly agree with this approach. It removes the burden of accountability from the shoulders of the teachers and students (i.e.: the program) and actually places it on the system. And I also agree that this is probably the only way to truly capture incremental learning.
Finally finally: regarding the random sample approach. Here is an exchange between myself and Janet Looney, who headed up a study on formative assessment in Europe (taken from the Special Topics Discussion List, 9-7-06):
Yesterday, Marie wrote: "You noted that in Finland, they use “random sample evaluations of different subjects” – does this mean that they study test data in subjects on an aggregate level and make determinations from this? If so, this would mean that accountability is spread more broadly, as opposed to studying individual performance/test data and then linking success or failure to either individuals (the student or the teacher) or smaller entities (one school as opposed to a whole school system). I would think that such an approach would necessitate studying not only what is happening in the classroom and school, but also what types of educational materials and assessments are being used. Would that be a correct assumption? "
Janet Looney: You are correct in your assumptions - the test data are used to track general trends in student learning, not as a mechanism for holding individual schools accountable. Schools being evaluated do receive their results, but the results are not published more widely. The Finnish National Board of Education uses the results of the random evaluations for ongoing development of the education system and core curricula.
Sorry to be so long-winded. Assessment is my passion.
Marie Cora
Assessment Discussion List Moderator
Hello Marie,
On Oct 17, 2006, at 11:54 AM, Marie Cora wrote:
David: in your reply to Patricia’s initial email, you note: “I believe we could persuade Congress -- and could then re-design the NRS to capture a student's learning over multiple years. After all, that would provide greater, not less accountability. However, then we would have a new set of problems: follow-up.”
Marie: can you tell us a bit more about what you see as potential problems with follow-up were our system changed to multi-year funding?
With our present system, getting follow-up data is difficult. It is especially difficult for programs in states that do not use data matching to find and get job-related information from students who have left the program. If we want to track outcomes and impact over a period of time longer than a year, follow-up efforts would be even more difficult. Many adult learners frequently change or lose telephone and email access and change their residence. Some do not wish to make it easy for others to find them, especially government agencies.
However, we could make some changes which might make follow-up easier. For example, if everyone who enrolled in any publicly-funded adult education and literacy service (e.g. classes. tutoring, distance education) were assigned a unique identifier which would be entered when using these services, if every learner's pre-post test scores were attached to the number, if each time a person enrolled and unenrolled in a service this were recorded, and if the data were aggregated we would have a picture of learning outcomes over time.
For example, we could answer questions such as "Given all the students who got a GED this year how many entered their first adult education service at level three, and for those what was the average and range of time it took them to get a GED diploma?" In other words, we could design an MIS to capture student information not for a one-year period, but for a multi-year period so we could look at what happens to students enrolled in the system over time.
For those who enroll once and never enroll again, follow-up would be just as difficult; but for those who re-enroll in the same or a different service, re-enrollment would also be an opportunity for follow up data gathering.
David
David J. Rosen
DJRosen@theworld.com
