Educational Gain, NRS Functioning Levels, and Post-Testing
From LiteracyTentWiki
Back to Assessment and WIA Reauthorization
Back to Assessment Information
Back to WIA Community Conversations
WIA Community Conversations on the Assessment Discussion List
February 11 – 15, 2010
Educational Gain, NRS Functioning Levels, and Post-Testing
Good morning,
Under the current WIA regulations, accurately representing educational gain is a challenge because of the NRS requirement that an educational functioning level gain can only be counted if it is in the subject area with the lower pre-test level or in literacy if the pre-test levels are identical. Pre-test scores may vary by as little one point and both math and literacy skills are necessary for success. If level increases in either subject could be counted as an educational gain (rather than only in the lower one), we would more accurately represent the progress that our students make.
It would also help us more accurately represent educational gain if the NRS educational functioning levels each represented two years of educational progress, rather than the current system that defines level 4 (High Intermediate Basic Education) as grades 6-8.9, when the other levels represent one year (level 6) or two years (all other levels).
Another challenge we have with accountability for WIA is following up with students once they leave our program. Our students move frequently and phone numbers are often out of service when we call. This is especially challenging for “retained employment” when we are attempting to contact them three quarters after they exit the program. I would appreciate any ideas on how to contact a higher percentage of our students.
Amy Peritsky
Greece Central School District
North Greece, NY
I agree completely! We face these same challenges. It seems particularly unfair that the middle level requires more improvement to "count" than other levels. It also takes more skills learning to advance from lower levels than from higher ones because the students who score at the lower levels often don't have the strategies to improve as rapidly as students who are already somewhat successful.
They must learn strategies before they can learn content.
[awayman1@kirksville.k12.mo.us]
The previous subscriber said:
They must learn strategies before they can learn content.
Amen to that! I've found that in transitioning to post-secondary education also.
Bonnie Odiorne
Post University, Waterbury, CT
I also agree with the remarks with the skills attainment in regard to literacy. Many times one or two points increase on testing is a major achievement and limiting how success is measured gives an inaccurate report as to the success of students. Many of the students we work with require intensive work to even move a couple of levels and it seems that once the mid-level is achieved they are penalized for not having the skills to move up rapidly. In addition, we are also finding it difficult with WIA accountability and youth. This population moves frequently and follow-up is difficult because of address and telephone number changes. Many times forwarding addresses do not exist and telephone numbers are not an option.
Cindy Casler
Very true. Many readers and math students who score in the 6-9 GE range actually function mathematically at a lower level--they guesstimated well--rarely is the reverse true, so it does often take much longer to bring these students up to speed in math. Over the years and with lots of data crunching, we have found that some students actually range up to two grade levels up or down from what their assessment indicates (the TABE).
Stephanie Moran
Hello All,
We, as an Adult Literacy Center, serve a representative number of the lowest literacy levels and therefore do not see the gains made with our learner population that the state requires. Many students do not have the skills to "enter employment, retain employment or improve employment" such as full time homekeepers educating their infant children, or those that are disabled. Neither will they "enroll in secondary education, obtain a GED, or obtain a high school diploma" which are two accountability measures that are given by our state.
Could an individual's own personal goals such as reading to their children or attending parent teacher conferences for the first time be used as a measure by which to judge progress being made rather than employment or schooling? Not everyone is made to fit into those pegs, and we see such simple progress as reading labels on supermarket shelves and writing a check for the first time as progress made. Also, could it be feasible to assess those lowest learners by the completion of the individual's book levels which contain a sequential development with pre and post testing? This suggested method has been documented over the past 20 years of operating our literacy center in Detroit, and is reported and validated by individual tutors.
It is our hope that the progress made through our own longstanding curriculum published through New Reader's Press, a division of ProLiteracy, could be used to gauge progress being made for those lowest literacy levels, as well as the learner's own individual goals. Is this too simplistic to hope for? What are your thoughts?
Marcia Cole
Adrian Rea Literacy Center
To Assistant Secretary Dann-Messier: Thank you for the opportunity to provide our honest feedback.
To the discussion list: It has been an interesting week reading all of your comments. I would like to add my two cents worth, looking at the issues from a state-level perspective.
GED tests: In my opinion, the NRS should count the successful passing of the GED tests as an educational gain (as well as a goal) regardless of the starting level of the student. Yes, it is lengthy, costly, and stressful as one person commented, but our students who already have a high school diploma are not going to be interested in it, and they are not even eligible to take it. For these students, the way to show gain will be to post test with TABE or CASAS until they reach the ASE High level, and then do not need to post test anymore because there is not a federal benchmark to meet at that level. For many of our other students (although certainly not all! especially ESL students), obtaining the GED credential is their primary goal for coming to adult education and literacy programs! As mentioned in earlier comments in this discussion, our students know which test is which -- and they give it their all when it comes to the GED tests. We should reward them for their effort and achievement, no matter which level they started at.
EFLs: In my opinion, the twelve ABE, ASE, and ESL Educational Functioning Levels (EFL) should be changed. As many people have already commented, the EFLs are not evenly distributed, and students at the bottom of a level have a much harder time showing an educational gain than students at the top of a level. (This does not take into effect the variability of test takers and the seriousness of the test taking as mentioned elsewhere.) Why are EFLs two grade level equivalents (or three for ABE Intermediate High) in the first place? All other educational agencies and institutions consider an educational gain as moving up ONE grade level. The NRS should have a fixed, scale score point spread to determine a gain (as previously mentioned by Ajit Gopalakrishnan [see thread entitled ‘Responses to Assessment Questions and Comments’), and my suggestion is that this point spread should be the equivalent of ONE grade level, not two. This would more accurately reflect the educational gains made by all of our students, regardless of their starting point within a level. (NOTE: I do not know if our currently approved assessments will work this way along the full range of the scale scores.) And one more logical suggestion: the gain should be counted in any content area, whether it is in the students lowest EFL or not. If we are going to reward our students and our programs, then why not reward them for any content area gain.
Post testing: This is a difficult issue. The federal office had turned to the test publishing companies for guidance and guidelines but ended up forcing the states to follow strict requirements, instead of the more flexible recommendations. 40 hours (CASAS) and 60 hours (TABE and BEST) were the RECOMMENDED minimum number of hours a student should have before post testing, but these have become fast and firm state policies, with very few and limited exceptions. Some states have more exceptions than others which also tips the scales so that we are no longer comparing apples with apples when looking at different state results. As several others in this discussion have mentioned, our students are sometimes ready to post test much sooner than 60 hours! And finally, why does a student need to wait until he or she has 60 contact hours in the classroom or lab to post test, when they should be, and hopefully are, practicing their reading, math, and language skills outside the classroom for many hours that are never counted in the system? I know that some of the test publishing companies are looking at data on students who have successfully passed post tests prior to 60 hours. They may make new recommendations to the federal office. I would hope that we can trust the teachers to be the best judges of when their students are ready to post test or not - regardless of how many hours they have spent in class.
Dr. Philip Less
Arkansas Department of Career Education
Little Rock, Arkansas
The following 5 subscribers sent posts in agreement or support in response to Dr. Philip Less’ post above:
Marie Bruno
Arkansas Literacy Councils, Inc.
Little Rock, AR
Amy Peritsky
Greece Central School District
North Greece, NY
Joanie Rethlake
Philip Anderson
Florida Department of Education
[Awayman]
I'm just curious...with CASAS, students only need 40 hours of instruction before post-testing. With TABE, they need 60 hours of instruction. Are the students who test with CASAS smarter than those who test with TABE, since they need 20 hours less instruction?
Bev Dye
Casper, WY
As has been mentioned, each state creates its own assessment policy. CASAS recommends 80-100 hours of instruction between pre-and post-test. Some states or programs test after 60, yours apparently after 40. Intensity and duration of instruction are key, as is student effort. Are you seeing a lot of learner gains? If so, are students being held the 1 hour time limit? Also, just curious about that last.
Allison Pickering
Escondido Adult School
We use TABE here in Wyoming, so actually we're 60 hours. We do see significant gains...exceeding the performance measures in both ABE & ASE. I wrote a previous email stating some of our statistics re: the post-testing requirements. I am opposed to requiring 60 hours before being allowed to posttest students, as I agree with you in one aspect...intensity (and quality of instruction) is the key. I disagree that duration has anything to do with progress. Not sure about what you meant about 1 hour time limits (unless that's something CASAS requires (I'm not real familiar with CASAS). Thanks for your comments... :)
Bev Dye
We see a high percentage of gains for students who persist in the program. Yes, they are held to the one hour time limit.
Laura J. Chardiet
What types of accountability measures are needed to successfully gauge whether adult learners are achieving their goals? What are some solutions?
In California, we have a very efficient system of reporting learner gains through CASAS. Tests used for ESL include the listening and reading tests. The problem is that the tests don't measure the gains made in language development, which is the primary focus of most ESL classes. Level completion statistics are based solely on test score gains only. We have no way to report the actual rates of level completion - number of those students who move to the next level, based on listening, speaking, reading, and writing skills. What we need are standardized level exit tests based on our content standards that assess more than just reading and listening. Instead of depending on pre and post tests, level completion rates should be based on passing level completion tests. For this purpose WIA funds should support the development of these tests. It would be nice to receive benchmarks for those gains. Similarly in Citizenship classes, gains are only reported on reading pre-post test results, not on students actually passing the Naturalization exam. It would be nice to receive benchmarks for this outcome. Because the wait for Naturalization interviews is shorter now, some students leave the class before they have had a chance to take pre and post tests and so are not counted for learning gains or goal attainment.
Another problem is the inability to report the gains of pre-literate or non-literate students when they are too low to take a pre-test. Pre-post gains are only measured from test scores. It would be great to earn learner gains for those students who cannot test and then later can test. A score of "0" needs to be a valid pre-test score.
As of now, ESL students that score over 236 on the CASAS tests are not counted for learning gains. As we focus on transitioning our advanced ESL students to vocational training or post secondary education, it would be nice to receive funding for learning gains made by these higher level students.
Gretchen Bitterlin
Mid City Campus
San Diego, CA
First of all, thank you for allowing us to have these discussions through this format...it's interesting to see how other centers operate, and how they perceive things.
I am writing to voice my concerns over the required minimum number of instructional hours a student must put in before being allowed to posttest. We use the TABE, therefore our students are required to put in 60 hours.
Many of our students are in situations where obtaining a GED is a
priority, and to hold them back just so they can obtain the number of
hours required by the federal government is ludicrous. Consider a few of
the examples of students we are currently serving at our ABE/GED
Center:
● Unemployed, seeking a job which requires a GED
● Homeless
● Court-ordered
● Entering military
Our students don’t all fit into the square box that the folks who are making these rules would like to put them. There are reasons why students left school to begin with; replicating a situation they were previously in (i.e., traditional high schools where they failed in the first place) does no good for anyone. Requiring someone to put in seat time does no good for anyone.
Educational gain has little to do with the number of hours a student spends in a program. Students should be evaluated on a case-by-case basis to determine whether or not they are ready to posttest. Instructors who have worked with students know when their students are ready. Our own ABE/GED Center instructors have a combined total of 92 years of teaching experience in ABE. Teacher-made tests and chapter reviews, not seat time, tell instructors when a student has mastered the material and is ready to move on and posttest.
Many factors determine student progression rates: one-to-one or small groups vs. classroom instruction; quality and intensity of instruction; intellectual capability – some students learn more quickly than others; and motivation, just to name a few.
Some students can move from one Educational Functioning Level (EFL) to the next in 15 hours, while others might take 1,500 hours. A student who pretests with the TABE at an NRS Low Adult Secondary Education level (NRS Level 5) with a scaled score of 594 in math (10.9 grade equivalent using the TABE) only needs to increase their scaled score by 1 point in order to be in the next level (595 = 11.0 grade equivalent). Do we really need to hold this student back for 60 hours before we posttest him? Chances are pretty good that he can increase his score to a 12.9 (much less an 11.0 grade equivalent) in much less than 60 hours. A student who pretests (TABE) at an NRS Beginning Basic Education level with a scaled score of 368 in reading could take hundreds of hours to bump to the next level. In my experience, students with difficulties in math can progress very quickly from one level to the next. Students with difficulties in reading tend to take much longer. The policy of 60 instructional hours before being allowed to posttest takes none of these factors into consideration.
What curriculum is used to determine that 60 (or 50 or 30) hours is the number at which students can move from one NRS level to the next? We would like to see the research that proves a student must have 60 hours of instruction before showing educational gains; our center’s federal tables on educational gain show otherwise! We have been operating for years with no minimum hours required between a pre- and a posttest. We also exceed the state’s federal Negotiated Performance Levels for both ABE and Adult Secondary Education (ASE).
A research article, “Federal Accountability System Fuels the Decline of the Adult Education & Literacy System of the United States” written in 2003 by Thomas G. Sticht, an International Consultant in Adult Education, compares attendance and progress of Wyoming ABE programs with Florida ABE programs. After reading the article, we certainly question why OVAE would mandate all students to put in the same number of instructional hours before being allowed to post-test
Some centers offer very limited hours of instruction during a program year. Many centers are unable to offer 60 hours of classroom instruction, even during a full program year. As you may know, Wyoming is a very rural state; many, if not most, sites instruct on a part-time basis. ABE/GED instruction is offered at detention and treatment centers, again on a very part-time basis. How will students be able to access 60 hours of instruction when it isn’t even available to them during a year’s time?
Every center in Wyoming has experienced federal and state budget cuts this year. How can we offer any more instruction while we are already understaffed? Our facilities are bursting at the seams with students. Where are we supposed to put students who need to attend all those extra hours?
OVAE states that centers must follow test publisher guidelines. CTB McGraw-Hill, the publisher of TABE, states on its website (under Frequently Asked Questions on the TABE): “There is no recommended minimum number of hours of instruction that must occur if a different form or different level of the TABE pre-test is administered as a post-test.” We have operated under these guidelines for years, and have yet to see a revised TABE publisher’s manual that states post-testing should occur only after 60 hours. OVAE, however, insists that CTB McGraw-Hill requires 60 hours before students be allowed to posttest.
It should be noted that Wyoming’s assessment policy allows for a limited number of students to posttest before 60 hours of instruction; however, these exceptions are only allowable for 20% of the students whose pretest places them at one of the 2 highest Educational Functioning Levels (i.e., ASE Low and ASE High). ABE students (EFL levels 1 – 4) are never allowed to posttest before 60 hours (according to the policy).
Our ABE/GED Center compared statistics for Table 4b (which tallies educational gain for students who have been both pre-and post-tested), on students who had more than 12 hours of attendance with students who had more than 60 hours of attendance. Students who were in the program for 60 or more hours made less educational gain than students who had less than 60 hours. Generally, students who spend more than 60 hours in the program are the students who take longer to progress (i.e., students with learning disabilities, slower learners, students with ADHD, etc.). In 2008-2009, our center had 328 ABE/GED students who were both pre- and post-tested. There were only 33 students who had greater than 60 hours. A total of 173 of our ABE/GED students earned their GED in 2008-2009. Of the 173 GED graduates: 165 (95%) earned a GED with under 60 hours of instruction; only 8 (5%) earned a GED with 60 or more hours of instruction.
We set high standards for our students, but there are times when it is not realistic for students to spend 60 hours in a program. Seat time does not put food on the table! The policy requiring all students to put in the same number of hours before being post-tested needs to be eliminated.
Thanks again for listening.
Bev Dye
I am in100% agreement with the posting of Bev Dye of Wyoming. Standardization is all well and good assuming that the playing field is level and balanced. Unfortunately, across the country there is tremendous disparity in funding, and access to adult literacy programming due to the rural nature and/or lack of transportation.
I have made the assumption that the current NRS is an attempt to apply similar psychometric standards of NCLB to the adult literacy programming. It was a good start, but having lived with it for the past 12 years it is very apparent that it needs significant tweaking to provide a realistic perspective of the impact of our programs and for programs to be able to report and get credit for the many lives they have improved.
Another discouraging facet of NRS came to light yesterday, as we were finishing up the updates for the 176 GED graduates in our center between July through December. Twenty-six (15%) of these students were enrolled in the 2008-2009 program year and started their testing, but finished up in July and August of the 2009-2010 program year without having attended beyond taking their GED test and were therefore, not re-enrolled in the program. As a result, unless they had post-tested and gained a level in the spring, AND attended 60 hours our program got nothing for the efforts that were made not only to assist them in earning their GED, but also in transitioning many of them to college. Where is the rationality in this reality that is but one of dozens of ways that the current NRS is stacked against learners and the programs that are doing all they can to assist them in fulfilling their goals.
Thanks again for providing this opportunity to share our perspectives on this topic. I am hopeful that Assistant Secretary Dann-Messier was very recently one of us, and that these comments will serve a greater purpose than therapeutic venting from the field.
Sincerely,
Jim Schneider
Davenport, IA
I am in100% agreement with the postings of Bev Dye and Jim Schneider. "The current NRS is stacked against learners and the programs that are doing all they can to assist them in fulfilling their goals." *We* know that we are making a difference in our student's lives and in our communities; but given the current NRS rules and regulations for assessing and reporting our student successes, we are unable to show accurately just what a difference we *are* making. This can be very discouraging for those of us who work hard to see that our students and our programs succeed.
Jean Light Kinyon
Rockingham County Literacy Project
Eden, NC
End transcript
