Literacy Needs
From LiteracyTentWiki
Back to Assessment Information
The following discussion took place on the National Institute for Literacy Assessment discussion list ("Listserv") between June 22, 2005 and June 24, 2005. The discussion covers a variety of topics on the subject of literacy needs and the dichotomy between accountability and instructional needs.
Summary
The discussion began with a post from an Arizona teacher describing the processes in her program that add to the discouragement of students when they realize how much learning they need to accomplish to meet their goals. This causes a problem with retention in the program. She describes some strategies that her program is employing to combat this issue, and asks for other suggestions in terms of motivating student retention. Initial responses included taking great care with testing at in-take, possibly avoiding it altogether, and instead using a variety of other procedures at in-take such as collecting full information on a student’s background. Some of this information is very helpful, such as educational background, while other information is not at all useful, such as whether the student earned a high school diploma. One respondent noted that the funding requirements at her program forced them to test students immediately and that although this is detrimental, it probably produces lower scores from the beginning (due to student discomfort) and thus, it would be more likely to show higher educational gains in post-test. This discussion quickly turned to one of values, in which many respondents argued that the appropriate mission in ABE is one of serving the student population, not the funding requirements. However, it was noted that this dichotomy poses a large problem because programs need funding in order to serve the students, and in particular federal funding requires very strict tools in order to meet National Reporting System (NRS) accountability.
Several states’ and programs’ processes were outlined by respondents, noting how they tackled these issues. It was noted that the process of “gaming numbers” is not surprising in light of the mismatch between needs for accountability and needs to serve the population. The discussion turned to questioning the use of GLEs (Grade Level Equivalents), the lack of appropriate standardized testing tools, and the lack of standard interpretations of data across the states. Again, it was noted that ABE should not allow instruction to be driven by the NRS data system, not just because of the accountability/instruction dichotomy, but because there are no tools that align curricula with the descriptors in the NRS.
One person described his program processes of assessing students by using alternate ways of collecting information, such as lengthy questionnaires. He noted that his program does not receive federal funding precisely because this would require the program to prioritize accountability purposes over instruction. One respondent noted once again the difficulty of matching accountability with instructional purposes, but also pointed out that assessment is crucial in terms of providing fair programming. He noted that a large part of the problem is that much of the present accountability system lacks the sound theoretical underpinnings needed for appropriate development of educational programming, not only in ABE but also at the K-12 public school level.
One post noted that CASAS uses a consortia of service providers within ABE, and thus its assessments are informed directly by the field, making it a more appropriate assessment tool. CASAS is then described in some detail.
Discussion Thread
Hi there,
I am writing from sunny Arizona to share some concerns we have here with low-level literacy students. We find that in our literacy programs in Yavapai County, students come and start working on computers and then realize that the software is too difficult. Then when they realize they have to do one-on-one tutoring, they feel discouraged and drop out. In the GED program where I work, I find that students who come as ABEI seldom return for class or stay with us if they do. I sense that these students realize that they have a long way to go to get their GED after taking the TABE assessment and give up on themselves.
Yesterday I was speaking with the literacy program coordinator, and we agreed to start using a referral form to send students between our programs. When I get lower students, I will refer them to her, and when she gets higher students, she will keep them for a short time until they feel successful in her program and then she will send them to me.
What successful motivators do any of you have that could help us with student retention? We would be glad to hear of your success stories.
Tina Luffman
GED Instructor
Instructional Specialist, ABE-GED
Verde Valley Campus
634-6544
tina_luffman@yc.edu
Greetings!
I find that if, during the intake process, you gather demographic information about a prospective student, you'll have predictors to work with in terms of their literacy levels. Perhaps the most important clue is a student's formal educational background. Another is experiencing a student fill out an intake form: how fluently and accurately he or she fills it out.
Unfortunately, having a high school diploma from the U.S. is much less predictive of literacy levels than graduation from high schools in foreign countries. Indeed, we often experience high school graduates with reading levels at 3rd or 4th grade-level equivalency. Thankfully, high school graduates will generally not also enter a GED program, although many would be extremely well-served by doing so.
My suggestion is to test "upwards" and not "downwards." It's much more motivating to test a student again using a more difficult instrument as a result of out-of-accuracy range scores on the high end using an easier test, than "demoting" a student who doesn't do well on the TABE by then having him or her take an easier test.
Finally, I am EXTREMELY cautious about testing on enrollment, because there are too many variables amongst adult learners that come into play when their first experience is the need to take a test. You could administer a "locater" test first (such as those offered by CASAS and AMES), while keeping in mind that the limited number of test questions, especially in the AMES (Adult Measure of Essential Skills) have unreliable predictive value.
Bottom line, all that said, is to administer the easier test first, and to acknowledge the psychological domain that plays such a key role with students who return to school as adults. A lot has to do with how you help "package" the experience from the student's point of view.
I hope this helps!
Michael A. Gyori, Educational Linguist
Language Development & Technology Director
Language and Literacy Resource Center
Hui Malama Learning Center, Inc.
375 Mahalani Street
Wailuku, Maui, Hawaii, 96793, U.S.A.
Hi Michael,
I agree with you that giving the TABE test on the first day of orientation is a discouragement for students. Because our funding requires that we test initially and then after a period of study, and then show educational gains, we feel it is necessary to maximize our potential for showing student progress and maintaining our grant program by giving the exam first. I can see good rationale behind waiting a few weeks and having students begin with group lessons to create a community atmosphere, especially for those with a negative educational background. I will consider your ideas and find out if my program is willing to try this out.
Tina
Hi Tina,
If you need to demonstrate gains among a percentage of students being pre- and post-tested, it's good to test them on intake, as you will likely get lower scores than you would once they're comfortable. Under the Adult Education & Family Literacy Act (AEFLA), students can receive up to 10 hours of instruction before they are tested.
Michael
Hi Michael,
I agree with you that the reason our program will probably decide to continue giving the TABE at intake is not only to locate the students into curriculum as soon as possible, but also because we are likely to get lower TABE scores from them when they are first entering the process rather than later when they are back into school mode.
Tina
Hi Tina, Michael and everyone,
I’m forwarding some technical info on the TABE from the Mass. SABES website (System for Adult Basic Education Support) which outlines the Locator, when to re-test, testing tips, etc. I’ve posted this before but thought it would be useful given this discussion. Go to:
http://www.sabes.org/assessment/tabe.htm
The folks at UMass, Amherst’s Center for Educational Assessment helped the state to hone in on some of the ‘hiccups’ if you will with TABE – esp. the test/re-test timeframe.
Like Michael’s programs, here in MA we also avoid testing upon entry to the program – pretty much for all the reasons Michael noted. We go thru an in-take process as well in order to gauge personal background, experience, and particularly the student’s goals, and administer program-made placement tests in order to determine generally which level folks should be in. I think the requirement here states that you actually cannot administer the selected learning gains assessment until something like 2 weeks after the start of the program (Mass. folks – help me out on that bit if that’s not correct). I believe the TABE Locator counts as the program placement test as well (again, Mass practitioners help me if I’m not quite right on this).
The other testing detail in this state is that although the NRS requires only 2 test administrations (one at entry, one at exit), Mass. requires 3 – one at entry, one half-way thru the program year, one at exit. We do this for 2 reasons: the first being that some students do not stay the entire year, but have done a great deal of work. If we don’t have an exit score for them, they can’t be counted. We can use the mid-year administration as their final test score. Also, for folks that do stay the whole year, that’s a lot of time gone by without checking in to see where one might have advanced, or where one might still need to be working.
How about other programs out there? What’s your timeline like for administering the tests and how does that affect your program?
Marie
Dear Michael and Tina et all,
I have a real problem with the value system that drives the process that you both implement.
I am a literacy provider who doesn't test the community-based literacy-level registrants who come to us for help. Instead I evaluate their reading, writing and life skills. I do not have GED students either. So perhaps that's where my philosophy goes off-to-the-left of many instructors/administrators like you.
Michael commented: "If you need to demonstrate gains among a percentage of students being pre-and post-tested, it's good to test them on intake, as you will likely get lower scores than you would once they're comfortable." To which Tina replied: "I agree with you that the reason our program will probably decide to continue giving the TABE at intake is. because we are likely to get lower TABE scores from them when they. are first entering..." Have either of you considered the inhumanity behind your decision? Have either of you thought about the learners' self-image and what it does to their confidence failing immediately at intake? And all for the sake of a number that can be placed in a report. Where a post-test won't even show a significant level change (much less 2 grade levels in literacy level students.) I get the literacy-level adults who are, for the sake of numbers, tested like your learners are. They come to our program from GED-prep programs like yours where, after the student has been subjected to the above process, leap/jump/flee ship !!! They've come with their heads hung low, thinking they cannot succeed in *this* program either. I have even had men and women (mostly men) say to me *directly*, "I must be dumber than I thought I was because I couldn't even pass their TABE test." And they don't come immediately! It can be months (up to a year) later before they "dare" try again somewhere else. It takes a very long time for the Testing Wounds to heal after standardized, timed tests knock the pins right out from under them. I realize that funding drives *you* because testing is required by the fed's. And there may be a sensible reason to test GED students. But do all of the adult education entrants come wanting to achieve a GED and enroll in classes to do so? Don't you have men and women who want to increase their personal capabilities and improve their life skills? Their needs are not being met by giving them a low score that they have to raise in the post-testing timeframe. They know they've received a poor score. And my belief is they hurt because they are ashamed of that number their practitioner has just given them. It's way more than "discouragement" as Michael puts it. It's a loss of self-esteem. I feel so strongly that there needs to be a broad base of advocates who value people more than numbers among those who have the power to change the assessment systems in our adult education field. Is that a fantasy? Must be. There are more like you than like me who come here to chat.
Nancy Hansen
Executive Director
Sioux Falls Area Literacy Council
Sioux Falls, SD
sfallsliteracy@yahoo.com
605) 332-BOOK
Given a supply-side performance system, I'm not at all surprised at the candid discussion of how to game the numbers--it happens in workforce. Frankly, I've never understood why we are using grade-level equivalency as a proxy for what adults needs to know to be able to function at work, home and the community, and how well they are able to do those tasks as a result of receiving instruction--as if learning stops at the 12th grade. I can only imagine the sheer horror of being classified as being at the 9th-grade level in reading, having failed or done poorly in K-12, when I'm 35.
Varshna Jackson
[varshna.jackson@twc.state.tx.us]
Hi Varshna, thanks for this.
Yes, I will show my colors (not the first time, eh?) that I also think using GLEs (Grade Level Equivalents) is not a great thing for our field and the population we serve.
Another post (apologies that I don't recall who all of you posted it - I usually like to call you by name) also noted that it's time to put pressure on our system to come together as a voice to influence some of the accountability issues that lead to funding and classroom to be at odds with one another.
So some comments:
-some states are in fact moving away from GLEs and using Scale Scores instead (see http://www.sabes.org/assessment/scalescores.htm for how Mass. is using these) -I'll raise my "vision" here in response to making change on a national scale: that practitioners need to (learn how to) get involved with test development, construction, implementation, and interpretation. Until we all do, we don't have enough knowledge base, or practitioner research data, to make a case for change. An alternative must be suggested in order for someone to sit up and listen. That said, I know that small pieces of this change are indeed happening in some states already.
And questions:
If your state or your program is using Scale Scores, or is using some other way of reporting testing scores, can you let us know what and how you are doing it?
If you don't get federal funding, what does your accountability system look like?
If you are an EFF (Equipped for the Future) state, for example, what does your accountability system look like?
Thanks,
marie
Arizona uses the TABE scale scores. We have also identified "out of range" scores for each level test. If you were to line up the scales scores of all the tests in writing for example, you would find that a person taking an E level test could score over 600 putting them in an ASE II level. However, the test is not written at that level. We use the level and the scale score. No way would I say that learner is at that level.
Assessments help us find out where our students are, and in reflecting on Varshna's comment, using scale scores can be helpful - a learner can get a 502 on an E, M, D or A level TABE; we're not talking GLE's. A 502 on an E says - this material is too easy, and 502 on an M says the learner can handle this level of material, on a D or A it may be somewhat to,it is, challenging. With that in mind, the teacher can then make a better choice of material to use.
One of the problems I keep seeing in our discussion is that we talk about the TABE test. However - the TABE has tests. What happens is we end up with students who are one level in reading, another in math, and a third level in writing. If gains aren't made in the lowest level, but perhaps they are made in one or both of the others, then no gain is made. It would appear that there is no consistency among states as to which tests are being reported for accountability purposes. Then again, the NRS materials clearly state that the assessment policies are to be defined by the state. There needs to be a "standardized process". That there are few, or perhaps too many, "standardized processes" and few "standardized tests" for adults may be a problem.
Something else that I think we need to be careful about. We can't let a data system drive our instruction. Our assessments, and the NRS, should be like a periscope - they help us to get a bearing on where our learners are, but they shouldn't be our final destination.
Is there agreement that the descriptors for NAAL and the NRS are accurate? If so, how does what you teach align with these descriptors? How can you verify that a learner has moved from one level descriptor to another? Because we say so just doesn't hold water any more.
-Miriam Kroeger
Arizona
Quick comment to add to what Nancy says. We don't pre-test students either, and we do have a large Spanish GED program. We welcome anyone interested and we've had students who had 3rd and 4th grade education from Mexico enrolling. The instructors assess students while in the class. Once students have been in class for a while the instructors give them the sample tests in the Spanish GED books. Not using any pretest has not been a deterrent for us. In fact, we have very mixed classes with students with various academic levels. They work in groups and help each other a lot. We have high retention and completion rates.
What we do, is we ask the students to fill out an entry form with questions such as name, address, telephone number, personal interests, background, etc. these types of forms are very useful and much more precise at assessing the literacy level of a person than a commercial standardized test. By looking at penmanship, use of capitals, punctuation, syntax, orthography, etc. you can get a very accurate picture.
I do not have any ABE funding for GED nor I want any for the reasons articulated by Tina and Michael. NRS has reduced learners to producers of meaningless numbers. We have a small EL civics grant and we must report gains for NRS. The testing detract us from focusing on important things like language competency gains and retention issues and it takes precious time away from students and teachers. We do it cause we have to, not because of any pedagogical benefit.
Andres Muro
Hi Nancy and all,
Yes, I do understand the inhumanity of what we do and do understand the reason for change. However, our state requires that we use the TABE to measure educational gains or else we lose funding, and then students are in worse shape because there is no program at all for the students.
And I certainly understand why the state requires that we use the TABE. Otherwise we would have unsuccessful programs receiving funding, quality of education may lag significantly without accountability, and students may be cheated one more time. There is no perfect solution, just as there is no perfect way to assess. The TABE is totally objective--multiple choice. Having a subjective portion such as writing an essay in response to a specific question, would improve the ability of our instructors to measure student ability.
I have indeed have students TABE higher than their ability as well as lower when comparing their designated score to the list of competencies that go along with such labels as pre-lit, ABEI, ABEII, and so on. Labeling is a major topic I studied in grad school as a Rhetoric major, and I do understand completely the inhumanity of telling a 47 year old city employee that he is third grade level, for example. These are the bits of information that I try not to share, but the students are smarter than we give them credit for.
I am really glad that others are seeing the need to revise the way we do things, but change must be taken seriously because there is, I still believe, no perfect way to do what we do. Consequences for making change will always affect those we chose to teach and to encourage on the path to success.
Tina
Aloha Nancy and all other readers,
The most poignant response I can provide to your comments is that I agree with you 100%. My comments (and I suspect Tina's as well) do not reflect my value system, the nature of my and my fellow teachers' interactions with our students, nor an endorsement of practices that are mandated if one chooses to receive funding with questionable accountability requirements.
In my case, I have been blessed by being able to design an ESL program on Maui back in 1986 under the umbrella of a very established and reputable 32-year old CBO that generally serves individuals who otherwise have fallen or would fall between the cracks of more conventional educational institutions and approaches. Most of my students are immigrants, as am I (English is my third language and I have been living in California and Hawaii but 22 of my 53 years in this incarnation). My main purpose is to support immigrant communities as they struggle to maneuver and succeed in their adopted surroundings. ESL, L1 and L2 literacy, as well as ABE and ASE are key components of the support system. Further, with respect to all of our programs, I, my fellow teachers, as well as my colleague who runs our Literacy Outreach Program with her cohorts, are actively engaged in framing our instruction in standards such as EFF to link learning with what is important in life for our students' own perspective. As you know, appropriate assessment is a major project still in its early stages.
The Adult & Family Literacy Act (AEFLA) is at its core the No Child Left Behind (NCBE) of Adult Education. Ironically, it only funds one of the four components of Family Literacy as defined by the very same federal government: adult education. Both, however, demand accountability. Accountability in and of itself a noble goal, for without it, as Tina seems to imply, many instructors out there would not be able to demonstrate accountability, because they might have little to account for. The "theoretical" and "best practice" underpinnings of accountability as defined by both AEFLA and NCBE are wherein the problem lies. One could argue that there is very little sound and tested theory and best practice that guides these "educational" policies that are (re)enforced by legislation.
We're dealing with a double-edged sword in that a lot of "dirty laundry" has been brought to the surface. At least many more students who truly need more help are getting it, while measures of what constitutes adequate progress, and of the construct itself, are what will derail this legislation. The validity of the measures (with respect to what our students wish to achieve in life) is difficult, at best, to discern, let alone subject to a successful validation process.
All that said, AEFLA has helped me realize by biggest dream: to become involved with the public school system in Hawaii, which ranks very low by national ratings, all the more so because Hawaii has one of the highest "standards" in the nation. My CBO is, for the first time, on equal footing with the state Adult Education system. Services for immigrant students in Hawaii are in their infancy, in part because the school system has not nearly caught up with the demographic upheaval this state has experienced in the past 30 years. I now have an opportunity to help develop services by means of an emergent family literacy program within the schools, even if the funds won't allow me to provide parenting classes, direct services to children, and PACT time.
In my interactions with the Hawaii DOE, I have repeatedly emphasized that as educators we need to educate not only those who are our students, but also those who attempt to define and legislate what constitutes best educational practice. I am now reaping the benefits of federal funding, while in my advocacy I have entered a pertinent domain to voice my views that have emerged in my 30 years of teaching and administrative experience and 19 or so years of formal education.
I hope that you consider this discussion with these thoughts in mind.
Michael
Michael A. Gyori, Educational Linguist
Language Development & Technology Director
Language and Literacy Resource Center
Hui Malama Learning Center, Inc.
375 Mahalani Street
Wailuku, Maui, Hawaii, 96793, U.S.A.
Hi everyone,
Nancy and Andres, thanks so much for your comments.
Andres, you point out that you avoid applying for federal dollars precisely for the reasons that are the heart of this discussion: that the present accountability system and its accompanying assessments do not necessarily meet the needs of the student, classroom, and program. However, many programs are dependent on federal funding in order to serve their populations.
So that's quite a bind for some: no money, no service.
Nancy, it's not clear in your reply if you receive federal funding that mandates use of the NRS system - if you do, can you describe for us how you work within the accountability structure? If not, can you let us know what types of things you do in your programs to provide services without federal dollars?
And to you all: if you do receive federal funds that require you to respond to the NRS, what types of strategies do you use to address the "testing fall-out" if you will, that Nancy describes in her post?
Thanks,
marie
Marie and everyone,
I would like to point out that CASAS uses a National Consortium composed of State Directors of Adult Education, practitioners in the field of adult ed, ESOL, MH, and Corrections to guide the development of its assessment instruments.
Their approach is a good model of field informed test design. Targeted at needs identified by the consortium members.
Alan Toops
More thoughts from my esteemed colleague, Joanna Barnes, who runs our Literacy Outreach Program at Hui Malama Learning Center on Maui. I am forwarding this to the list with her permission. Michael
From: Joanna Barnes mailto:joannareads@gmail.com
Sent: Thursday, June 23, 2005 8:43 PM
To: Michael & Sunay Gyori
Subject: Re: [NIFL-ASSESSMENT:1129] FW: RE: Literacy needs
What a spirited discussion and I really loved your rebuttal. It's funny though that all parties talk in terms of 'tests', which I regard as a deficit approach, rather than assessment, which validates what a person knows. When I bring up the subject of assessment with my learners, I talk in terms of what they, as adults, bring to the table: a lifetime of knowledge resulting in an array of expertise and competencies that I have no way of knowing without assessment.
When they come to us to improve their reading/writing skills, we establish together that they already know a lot, usually much more than they think they know. The last thing we want to do is to reinvent their wheel. We want to establish their areas of competence so we don't waste each other's time reintroducing what they already know. They have 'pukas' in their knowledge. Through the CASAS assessment, we can pinpoint those pukas and proceed accordingly.
The adult learners I work with often will grumble that process is less than fun, but once we start looking at the assessment results, they always become engaged in the detective part of figuring out the competencies they need to master to achieve their goals.
Alan,
Would you describe the CASAS for me?
Thanks.
Andrea Wilder
Andrea,
I feel obligated to state my bias up front. I have been a member of the CASAS National Consortium for a number of years and have watched it evolve and adapt to the assessment needs of its users. So I speak about CASAS as a believer and an advocate.
CASAS (Comprehensive adult Student Assessment System) was formed in 1980 by a consortium of California programs, adult educators and members of the California Department of Education (all of this history is available on the CASAS web site (http://www.casas.org).
Essentially, CASAS designs assessment instruments based upon the needs of the field in Adult Education, Developmental Disabilities, Correctional Education, ESOL among others.
CASAS is a system that attempts to link assessment with instruction with appropriate resources. It is not a curriculum but a way to assess the skills of adults to perform real life tasks. I see it as a strength model not a weakness model since CASAS is looking at how adults use the skills they all ready have to perform a given series of competencies.
CASAS has over 140 assessment instruments in its catalog. Certainly the most often used would be the Life Skill and Work Series, ESL Listening series and the Employability Competency Series.
Because CASAS is driven by its consortium to provide field based adult assessments, and operates as a non profit, only those assessment needs that meet the highest priority ranking of the consortium make it into development.
I won't go into test psychometrics but CASAS uses a scale score approach to reporting results.
There is much more about CASAS than I can describe here.
Alan Toops
