Responses to Questions and Comments
From LiteracyTentWiki
Back to Assessment and WIA Reauthorization
Back to Assessment Information
Back to WIA Community Conversations
WIA Community Conversations on the Assessment Discussion List
February 11 – 15, 2010
Responses to Assessment Questions and Comments
I offer the following suggestions for consideration in order to make the National Reporting System (NRS) reflect more completely the outcomes achieved by our learners.
- Evaluate learner progress based on ‘significant scale score gain’ instead of ‘level completion’;
- Include the attainment of a secondary school credential as an indicator of success at least for the ABE High Intermediate and Adult Secondary Low levels (in addition to the Adult Secondary High level);
- Evaluate learner gain across fiscal years (where applicable);
- Display the post-test rate alongside the NRS results.
I concur with others who have already mentioned the importance of sharing standardized test results with learners. This feedback can be more meaningful to the learner if research can highlight the relationship between learner performance on the NRS levels and the successes they aspire to after adult education (e.g. GED attainment, employment outcomes, postsecondary success, etc.). We have done some research along these lines in Connecticut but much more is needed.
A piece I wrote for the Literacy and Numeracy Studies Journal titled Extending Accountability: From compliance to learning discusses these and other issues in more detail. It can be accessed online at http://utsescholarship.lib.uts.edu.au/epress/journals/index.php/lnj/issue/view/111/showToc.
Thank you.
Ajit Gopalakrishnan
Connecticut State Department of Education
Ajit,
Thanks for these wise words and for the link to your article and those of others.
Excellent resources, thanks.
Janet Isserlis
I have been involved with TABE, CASAS, and BEST Plus for more than 15 years. In addition, most CBI programs included pre-assessments and post-assessments. The pre assessments are sometimes used too automatically select lessons.
These are my thoughts on assessment.
1. Proper assessment is vitally important for adult learners who do not want
to cover ground unnecessarily.
2.The assessment tool must have reliability and validity.
3. The assessment must be done in a proper environment.
4. with a trained monitor.
1. QU - What types of accountability measures are needed to successfully gauge whether adult learners are achieving their goals?
1. ANS - Employability skills, workplace skills assessments should be mandatory under WIA. Writing assessments should be mandatory under WIA. Assessment tools must match the ABE/ASE/ESL program goals under NRS levels. Reading assessment tools are good. Math assessment tools are fair.
2. QU - What do you see as some of the greatest challenges with our present accountability measures? What are some innovative solutions to those challenges?
2. ANS: Coordinating the post assessment tool with the ABE/ASE/ESL program goals to complete the NRS level or GED exam. SOLUTION: I base each education program for the next NRS level / GED exam / BEST level and also the assessment post test. This takes longer, sometimes too long for the June 30 deadline or semester end, but it is the most successful one that I have found.
3. QU - How might WIA reauthorization support improving our present accountability measures? What are your hopes?
3. ANS: More funding for research that includes nation-wide data gathering. Funding for program grants to purchase assessment tools for use on the computer instead of the paper.
Barbara J. Struble
Because the CASAS does not give specific literacy needs and because it doesn't give grade levels, we must give a diagnostic reading and language battery written for adults. Instructors need a graded reading level to locate instructional material by grade level and they need to know specific needs such as ability to decode multi-syllabic words. The CASAS is an extra burden on staff and students. It takes up an extra hour and half. Instead of giving a monopoly to one or two test publishers, the government could set up criteria for diagnostic tests such as: a K-12-Adult test from a major publisher, used in universities and in at least the third edition. Tests as these always yield the same grade levels regardless of publisher and they have validity and reliability established by blind reviews. In addition they give information useful in identification of learning disabilities. (Many adult learners do have undetected disabilities.)We have a successful GED program, but for years we have been following our students and in many cases cannot locate them. They move without a forwarding address or a phone number. In our area jobs are difficult to find. At one time we offered a $10 gift certificate for a response and only seven out of seventy responded. Tracking people by Social Security Numbers seems invasive and punitive. Identity theft may increase. When we take a person who can't read and write the alphabet and move him to the second grade level, it doesn't count as a gain in the CASAS system. If one of our students obtains a GED or learns English, they should be counted as a success. Is it our fault that Michigan has a significant unemployment rate and they student can’t find work? Does it count if they move out of the state looking for work elsewhere?
Di Clark
Capital Area Literacy Coalition
Lansing, MI
I recommend that NRS change the definition of a "participant" from students with 12+ hours of instruction to students with 20 or 30 hours of instruction.
Currently NRS defines "participants" as those students with 12 or more hours of instruction, while testing manufacturers and states recommend post testing after many more hours of instruction. Minnesota requires 40-60 hours of instruction and CASAS claims that most programs post-test between 70-100 hours of instruction. This creates a huge gap between obtaining "participant" status for NRS reporting purposes (12+ hours of instruction) and when a student actually takes a post-test.
In addition we have a highly mobile population and students with life, work and children commitments, we know that students often leave our programs well before 60 or 70 hours of instruction.
Increasing the hours of instruction required to define a student as a "participant" would close the gap a bit and allow us to report on students who have more hours of instruction, have had more opportunity to learn and are more likely to have post-tested.
Jenny Schlukebier
St. Paul Public Schools
Hi, I manage a GED preparation program at a community college in Michigan. Our goal is for our students to complete their GED, and then transition to post-secondary education, hopefully, with us.
Our measures of success are simple: GED completion, college transition. We are a GED testing center so we are able to get actual completion numbers and transitions. Ideally, those would be our measures of success.
We are also asked to TABE pre and post test our students and call them five times after completion or leaving our program to see what they are doing now. The TABE testing especially the post testing is a distraction. Our students take five practice GED tests (at least) five actual GED tests and they take COMPASS before transitioning to college. Regardless of what we tell them, they see TABE post-testing as a nuisance. With all this testing, you can see why.
Also, in Michigan, our WIA information is stored but not shared with adult education entities. As a result, we are asked to call each student who leaves our program to see if they have gained or improved their employment. We are asked to call four times. We simply do not have the staff to complete this request. I would like to see if the two recordkeeping systems in the state could talk to each other so this information could be retrieved electronically like it is in some other states.
Ideally, I would like to be measured against the purpose of our program. I think outcomes should match assessments. Any thoughts?
Anne Greashaber
Washtenaw Community College
Ann Arbor, Michigan
Assisting adults who were left behind as children is my passion. I was somewhat resistant to the accountability process back in 1998, but soon bought into it. Having worked with it since the inception I am VERY aware of the shortcomings of the system. The following is by no means comprehensive, but hits most of the major issues as I see them. My apologies for the following lengthy diatribe. Thanks for this opportunity.
Q: What types of accountability measures are needed to successfully gauge whether adult learners are achieving their goals?
R: More authentic accountability measures that reflect the realities of and adult literacy student as oppose to imposing NBLB-like educational standards created for the K-12 system.
Respect adults literacy students have a myriad of responsibilities to fulfill and their own unique goals when they enroll in a program. As a result, the accountability measure requirements need to respect their time and not force them to spend extensive time hopping through hoops that have little to do with their personal goals.
Q: What do you see as some of the greatest challenges with our present accountability measures? What are some innovative solutions to those challenges?
The present accountability measures have too many provisions stacked against the programs and learners. For example:
- holding programs accountable for any learner who attends over 12 hours, but not allowing post-testing to be done until after 40 or more hours.
o Solutions: Make accountability & required instruction prior to post-testing the same #. If accountability starts at 12 hours, post-testing should be allowed at 12 hours. If post-testing isn't "authentic" or "realistic" until 40 hours, then programs should only be held accountable for learners who attend instruction for this amount of time.
- Artificial & arbitrary "educational functioning levels" that require broad learning gains before recognizing the learning that has occurred. A learner at the lowest point of a level has to make significant gain to achieve the next level.
o Solution: Eliminate EFL's. Recognize the learning gains that occur irrespective of the artificial & arbitrary levels.
- Artificial & arbitrary "educational functioning levels" that recognize a learning gain when minimal learning has been attained. A learner at the highest possible point in an EFL only has to make a 1 point improvement for a gain to be recognized.
- The GED and official practice tests probably do not meet all the psychometric requirement for accountability measures as defined by a NCLB system. Nevertheless, they are meaningful for a large portion of the learners we serve and ought to be recognized and utilized
- Limiting the use of the GED for only the top EFL is ridiculous. Limiting the program from "counting" a student with less than xx hours is ridiculous. Anyone who has actually worked with adults knows that there are a myriad of reasons that people might not do as well on an assessment at orientation. It is a frequent occurrence that we help an adult overcome their anxiety and insecurities and they progress much more quickly than would have been expected by the all-knowing standardized assessment. This is yet another example of how the current system is stacked against the realities of the learners and programs
How might WIA reauthorization support improving our present accountability measures? What are your hopes?
Like most adult literacy professionals my wildest dreams would be that reauthorization recognizes the tremendous cost to society of the adults who were left behind as children and establish meeting their educational needs as a priority. Adults who were left behind as children have little hope of competing in the modern workforce. This lack of hope affects everyone around them from society at large, their local community, and the children. Educated adults will be contributing adults who are committed to the education of their children.
I lie awake at night dreaming of reauthorization in which legislators would fund adult literacy programs at the level where part-time, low pay instructors/staff would be the exception rather than the rule.
Reauthorization ought to establish a maximum number of students and managed enrollment to be served by the available funding similar to what TRIO Programs are subjected to. As it stands, some programs operate with very little to no state or local funds, subsisting entirely on the federal allocation, spreading it as far and wide as humanly possible. As a result, programs are over-enrolled, learners are under-served, and effective outcomes are miraculous. Or, require a local/state funding level so that local program coordinators aren't trying to meet a Cadillac accountability system with a Yugo budget working with a never-ending stream of students.
Below are relevant thoughts that I wrote one day when I was particularly frustrated with the accountability measures and marginalization of my program throughout all levels local, state, and federal.
Does anyone remember any of those wacky Doctor Dolittle characters from their childhood books? As many as there were, the one I remember the most was the Pushmi-pullyu (pronounced Push-Me-Pull-you). The Pushmi-pullyu was the two-headed llama whose two halves were so connected that they had to work together to go anywhere. I bring this up because it seems to me that it is an appropriate metaphor for the state of Adult Literacy Education during my 16 years as a local program coordinator.
There are a myriad of competing forces that we face on a daily basis.
GED Completion vs. adequate academic preparation for Postsecondary Education
Research based instructional practices vs. part-time, low pay, limited instruction
Enrollment based goals of heads & hours vs. educationally sound practices or focus on outcomes
Student Want What They Want When They Want It vs. part-time, limited instruction
Students want what they want vs. NRS Benchmarks & Goals
Narrowly defined NRS measures vs. Adult Literacy Education Realities.
Learner-based instruction vs. Accountability Measures - Benchmarks
What we have vs. What we want
Who we are vs. Who we could be
What we accomplish vs. What we could accomplish
Jim Schneider
Career Assistance Center
Davenport, IA
Jim,
I agree with most of your post but this part:
Jim Schneider wrote:
The present accountability measures have too many provisions stacked against the programs and learners. For example:
- holding programs accountable for any learner who attends over 12 hours, but not allowing post-testing to be done until after 40 or more hours.
o Solutions: Make accountability & required instruction prior to post-testing the same #. If accountability starts at 12 hours, post-testing should be allowed at 12 hours. If post-testing isn't "authentic" or "realistic" until 40 hours, then programs should only be held accountable for learners who attend instruction for this amount of time.
If you are using the TABE for instance...it takes approximately 4 hours of "instruction time" to get through the 4 main tests. If you complete the full TABE it would take longer. And you want to post test after 8 hours of additional instruction?
1. Students (like ours) doing 3 4-hour days a week, would have to post test every 4 days...they would soon memorize the TABE...we have enough problems with that now. 30 to 40 hours for post testing gives them some actual study time between assessments.
2. 12 hours keeps programs from getting "no" credit for their honest efforts with students who drop in and drop out.
3. some of our "graduates" don't spend 40 hours with the program before they take the GED test...that to me is why the GED test (if taken voluntarily by the student) should count as a post test...and should count for the progress measured against their TABE or whatever beginning assessment is given them. (Again IF getting a GED is a student goal)
Dave Fowler
Assessment
- What types of accountability measures are needed to successfully gauge whether adult learners are achieving their goals?
- What do you see as some of the greatest challenges with our present accountability measures? What are some innovative solutions to those challenges?
- How might WIA reauthorization support improving our present accountability measures? What are your hopes?
WIA reauthorization should maintain a high accountability system and measures for student outcomes. The success of adult education is valued because of what data is kept on outcomes. It's a necessary evil sometimes, but it works. Reauthorization needs to look at the development of a secondary system for those learners who make significant progress but do not advance from one ELF to another (i.e., ESL). While it's a positive growth, if it isn't from one EFL to the next, then it doesn't count and that doesn't seem right. Each state will continue to have those learners and a secondary system would allow states to report gains in another manner that can actually expand the success of adult education. Lastly, there must be flexibility for testing before 60 hours - if we could do so, we'd have better data to report, especially in the situations where students leave, etc.
Valerie Fischer
ND Department of Public Instruction
Bismarck, ND
We are a small community organization which supports about 100 student/tutor pairs at any given time. Most of the students are low functioning, and all of the tutors are volunteer. All but a few of the pairs meet weekly, and some students utilize the computer lab in addition. In past years, on average, our students have consistently met state targets for advancement, which means that the one-to-one, weekly model is an effective practice for some adult learners.
The new assessment requirements - which require a minimum number of hours for post-testing - do not take into account the effectiveness of the one-to-one, once a week model, which derives its benefit from something other than hours. With the new requirement, in fact, only a few of our 100 students will qualify for post-testing within a year, even though many will have progressed (according to past record).
Our suggestion is that we return to the practice of using the minimum hours for post-testing as guidelines rather than requirements, so that we can continue to use the one-to-one, weekly model where it is clearly effective. Otherwise, the lowest functioning adults, and adults who are working two and three jobs, will eventually find themselves without access to basic education, because there isn't adequate funding to support that model of community-based service.
Susan Sherard
Moore County Literacy Council
Southern Pines, NC
What types of accountability measures are needed to successfully gauge whether adult learners are achieving their goals? Adult learning theory tells use that adults need to be able to see progress toward their goal(s) if they are going to persistence in any program. Therefore, we need to be able to provide some system of benchmarks that will show the learner as well as the instructor that progress is being made. Instructional programs need to be continuously reviewed to determine if they are meeting these goals by using the benchmarks.
What do you see as some of the greatest challenges with our present accountability measures? What are some innovative solutions to those challenges? Unfortunately, our present system of standardized tests is inadequate as a benchmark tool in measuring progress in short enough periods to satisfy most learners. They are designed more as summative assessments. We have begun using a series of formative assessments (designed by instructors) to aid in measuring progress. Again, unfortunately, these benchmarks do not serve as "pay points" for WIA funding and they are time consuming and labor costly to develop and use. Because we, in California, are not presently driven by attendance as the principal determination for funding with general adult education funds, we were able to move to a managed enrollment in the majority of our WIA program classes. This has allowed instructors to use these formative assessments in a more managed environment (and typically with slightly smaller class sizes), but funding resources still drive the number of classes/sections we can operate, so they may be many learners who are forced to wait to enter into instruction.
How might WIA reauthorization support improving our present accountability measures? First, we need to connect the WIA programs. As an example Title I and Title II programs are entirely disconnected, but in many ways are serving the same populations. If we are to better provide transitions for adult learners we need this connectivity among all WIA programs.
What are your hopes? Our regional WIA networking group met today and I asked this question to them. The previous response was based on some of that feedback. Additionally, if we are going to emphasize and require transitions through program levels as a principal WIA goal, we are going to need funding to develop tracking systems across a variety of programs and we are going to need start up funding to create benchmark assessments (much the same way that EL Civics was initially funded at start up). Funding based on level/program completion would come at too large a time period to be able to provide adequate instructional programs for most adults.
David Williams
Beaumont Adult School
Beaumont CA.
Hello all, In California we receive "pay points" for statistically significant learner gains, two-level completions in ESL(on CASAS), and for those who set it as a goal regardless of entry level, GED completion. I would like to see a national policy. What is reasonable? I do not object to pre- and post-testing with a valid and reliable, standardized assessment four times a year to capture most of the students who pass through our program. Alas, also in California, in order to get all the WIA funds available we also have to conduct EL Civics performance-based assessments. These are designed by each agency and are in no way standardized, reliable or valid; yet we have to expend HUGE amounts of staff time and energy in creating and administering the assessments. Although we do make some money on it, the cost / benefit ratio is very low. Also, no other division of education has to test so much to get their money. The EL Civics carrot (or stick depending on perspective) means classroom-based, teacher-created assessments get omitted because we are chasing dollars. Some agencies try to meet the intent of the law; others are less conscientious. A slap dash approach to accountability is easier with these agency created assessments than with one that has been field tested and the results passed through psychometric analysis.
What types of accountability measures are needed to successfully gauge whether adult learners are achieving their goals? Learner educational gains and goals can be two different things. Learning is one of a student's goals. As in other divisions of education a simple, reliable and valid standardized assessment, such as CASAS or other similar instrument, meets accountability needs. It is easy to administer can be compared across programs, and is far less burdensome than performance-based or one-on-one assessments. As far as post-AE goals are concerned, funding for counseling and transition classes will assist in tracking students' success in attaining goals--provided we can track them down as has been pointed out.
How might WIA reauthorization support improving our present accountability measures? What are your hopes? I would hope for a simple, national assessment policy. Although the original law mandates multiple assessments, only one, perhaps from a menu of choices, or not, --even if administered several times in a year-- should be THE accountability instrument for receiving the WIA dollars. GED completion regardless of entry level should count. Agencies should not have to create and administer EL Civics assessments.
Allison Pickering
Escondido Adult School
I agree with all of your comments, Allison, especially those related to the lack of standardization in EL Civics implementation and assessments across the state. This is not due to lack of effort on the part of CASAS. They've done a stellar job to provide pre-approved assessment plans, networking opportunities, training, etc. to promote standardization. There is just no feasible method to really check up on what each and every agency is actually doing in their EL Civics programs. I'd like to note that California is the only state in which agencies are required to do additional assessments in order to access EL Civics funding.
Jan Forstrom
Continuing Education San Diego Community College District
Allison,
Just wanted to clear up some confusion. Agencies do not have to participate in EL Civics to receive WIA funds. There are many agencies in California that did not submit for the EL Civics grant, but still participate in the WIA grant. While it is true that in order to qualify for an EL Civic pay-point a learner must have a complete SOD (Student Outcome Data set) which includes an appropriate pre and post CASAS test pair, they do not have to have an EL Civic assessment for a WIA pay point.
David Williams
Beaumont Adult School, CA
David,
Of course, we don't HAVE to participate in EL Civics. I object to having to do all that testing to get *available* funds. The EL Civics money should be rolled back into regular WIA, Title II funding and distributed to all agencies through the regular pre- and post-testing not this ridiculously burdensome, invalid hoop jumping.
Allison Pickering
Escondido Adult School
End Transcript
