Searching for Usable Data/Dissatisfaction with Available Tools
From LiteracyTentWiki
Back to Assessment Information
Back to Using Data for Program Improvement
Back to Full Discussion by Threads
Searching for Usable Data/Dissatisfaction with Available Tools
We tutor adults. No children.
Almost all our students are at Beginning Literacy to High Intermediate ABE level. Almost no high or even low adult secondary. At the secondary level we only get the students that can't (or won't) tolerate study in a regular ABE classroom. ESL instruction is done by a different organization, with paid teachers.
We net 20 to 25 students with more than 12 hours of study each year. We are so small that sometimes an entire FFL will have only one student in it. When that happens the only question is, "How many advanced a level: 0 or 100%?"
We deal with students on a highly individualized basis. One may need to learn to read again after having a stroke or a fever. Another may have taught himself to sight read at a very high level, but neglected to teach himself any spelling or writing skills. A high school graduate may not have learned even his ABCs, for whatever reason. One or two students a year might have an employment or higher education goal. (Then WV can't verify it, if the student works or studies out of state.) I can safely say that no two students have been alike in the nearly ten years that I have been here.
I genuinely *like* statistics and know they can be very useful, and don't mind gathering data to be put in a bigger pool if what comes back is helpful. However, if a level has only 3 students, is the data even "statistically significant" if just 2 of them are available for both pre and post assessment? 2 of 4?
Some things are better seen by microscopes and others by telescopes. Right now neither NRS nor CASAS seems especially useful at a local level. Maybe all I need is to find out how to focus them. Maybe they should be just trashed. They may cost more to use than they return in terms of time and money and *stress*, on us and especially on our students.
I'm from West Virginia, not Missouri, but, "Show me!" (Please.)
Mary G. Beheler
Tri-State Literacy
455 Ninth Street
Huntington, WV 25701
304 528-5700, ext 156
Mary, I think you raise some valid concerns. When you’re working with a small pool of students like this, aggregate statistics can be rather meaningless. I would think the most helpful data for you would be individual diagnostic reading assessment and progress monitoring data. Are you familiar with the Adult Reading Component study and the work related to using reading profiles? You might want to check out [1]. I’m wondering if the component reading assessments wouldn’t go a long way toward “focusing” the reading instruction you offer based on each learner’s profile. That doesn’t get you off the hook for NRS reporting, but it does provide a mechanism for meeting the highly individual needs of your learners. Just a thought.
Sandy Strunk
Program Director for Community Education
Lancaster-Lebanon Intermediate Unit 13
1020 New Holland Avenue
Lancaster, PA 17601
(717) 606-1873
Yes, WV would "consider our mitigating circumstances" if we had missed a few of our goals. But we would also get a warning about, "Money is tight, and if you don't make your goals...."
We have made our goals, but sometimes completing the nagging and pleading it takes to get the one last student in to post test (and succeed) has been frighteningly close to the fiscal year deadline. We had a group of 5. Three post-tested: 2 "improved a level" and 1 did not (even though he had actually gained more CASAS points than the others). Therefore, our NRS score at that level was only 40%. We needed something above 50%. We had to get another student assessed!
One missing student had moved hundreds of miles away. The other had a new job providing him many over-time hours. He finally came in on the last possible day to post test and fell across the FFL line with a 1 or 2 point gain. Our improvement rate for that level suddenly jumped from 40 to 60%! That's silly (IMHO).
To complicate things a bit, other funders, such as United Way, look at these statistics as well. Explaining levels and how some students can improve quite a bit and not be a "success" is hard to do without sounding like we are just whining. Fortunately, they have allowed us to use any 5 point CASAS gain as a measure of success instead of using the NRS brackets. And if one student comes up 20 points, we get 4 United Way credits! (On the NRS report it counts only one gain in his entry level, though he may have crossed more FFL lines.
Doesn't using the different level brackets, instead of total points gained by each individual, distort the results, even in larger groups? Why are they used?
Since the statistics for small groups can be changed so drastically by even one individual, why not pool the results from groups like ours? (Call the pool "The Long Tail" if you must: [2]) The pool might give researchers an idea of whether classrooms or one-on-one tutoring is most effective at the lowest literacy levels.
So, if you have any influence, please try to persuade NRS to describe the goals for small groups in terms that make more sense for our situation. Right now we are like little kids clomping around in Mom's high heels. It is hard to work that way, and very far from useful.
All of which is not to say that individual assessments are not quite helpful for spotting individual problems and successes. They do help.
Mary G. Beheler
Tri-State Literacy
455 Ninth Street
Huntington, WV 25701
304 528-5700, ext 156
Hi Mary,
Using the CASAS 5 point gains and United Way credits are good ways to help your program get the recognition and credit for helping learners. We encourage states and local programs to use such strategies to demonstrate gain for small programs such as yours and others that serve students at the lower literacy levels. This is not a distortion at all but a more accurate way to look at your program gains in this case.
The NRS is a national system and for that reason, there has to be some generalizations and standardization so that cross-program and cross-state aggregation is meaningful. But there is flexibility at the state and local level to use other measures to show gains and progress and to meet the accountability requirements we all have to face. We hope states and local programs take advantage of this flexibility at the local level, where there is no need for national aggregation. Your state or local program in your area could pool your results for your own use.
Larry Condelli
Does GIGO apply to the data all of us, big and small, are gathering?
The information gleaned from the CASAS Life Skills (or any other) assessment tool can be useful for spotting problems and successes, but because of the *multiple layers of skills involved in answering any one question* the specific question(s) missed must be looked at very carefully. But, getting the student feedback that makes looking carefully possible gets into problems of assessment confidentiality.
Example: A excellent student may understand *everything* else about a certain CASAS map question, but if he or she has heard the expression "X marks the spot," and assumes X means the goal, not the beginning, this question will be nonsense. I don't think the check-off sheet of demonstrated skills specifies, "Knows that on *this* map X means, "Start here." If the student's tutor or I can't discuss a missed question with the student, how will I discover that?
How do we know that the question a student answers or misses actually assesses the skill the assessment manual tells us it does?
I was scoring an answer sheet and was dismayed at how poorly a student was doing, when I noticed I was using the math answers, not reading. I switched to the correct set, and the student made even a worse score! Both scores were in the "valid" range, too! How much confidence should I place in that assessment?
Having the questions in a booklet and marking the answers to multiple choice questions on a separate sheet may be a skill even somewhat advanced adult students do not have. Because our student workbooks don't use multiple choice questions, we have actually created lists of number-letter pairs to see if a student can mark the letter in the appropriate column of each numbered row on a separate answer sheet. That's all. (Did that after a strong level 2 student marked the answer sheet by page number, not question number, with answers to 2 and 3 questions marked in the same row.)
Has anyone made an effort to see if the lower level literacy students want to learn what the CASAS or other accepted NRS assessments wants us to teach? Lots of our students are on SSI. They don't see the point in learning about employment applications. That often means any question about employment forms isn't important enough to take seriously, even if the ones about other forms are.
Colleges have sense enough to make all freshman year classes pretty generic, and leave the "major" study to later years. Is it useful for assessment of beginning literacy to get so specific so soon? Whatever happened to "Learn to read; then read to learn"?
We used to use the quick and unintimidating SORT-R for student assessment. Even on that simple test almost all men missed word "dainty," no matter what their reading level. Does anyone know if any specific question(s) on the assessments used for gathering NRS data is answered incorrectly by most students at any one level? Or if a significant number of students in the Laubach series misses different questions than students in Challenger or Voyager, or another series?
Mary G. Beheler
Tri-State Literacy
455 Ninth Street
Huntington, WV 25701
304 528-5700, ext 156
Mary,
Where did the valid score place the excellent student who was doing poorly? Since we work from the scale score and not the number correct I do not know how to interpret the doing so poorly. When working with teachers and students, we try to place an emphasis on the where the student falls on the scale or NRS level. In training with teachers, we always stress that students are not a single level but students have range of strengths and needs and we discuss about how to present this information to the student.
I agree that a missed question does not tell us why the student was not able to correctly answer the question; however, by taking “multiple measures” of student performance over time with a variety of formats tells me if the student has the concept and if the student can transfer what is known to different contexts.
In the example cited below, in most exercises that use maps the X is also marked with the words such as “you are here.” It would seem that the student using “X marks the spot” and not “X you are here” would indicate a reading/literacy problem. As a teacher I would watch that student read directions and try to perform tasks. Does the student ask others to help explain the task? Does the student ask me, the teacher, for help? Observing the student’s behavior and patterns of work is part of the assessment process, and from your other posts I think that is what you seem to do in your program with such a variety of learners at vastly different levels.
The last point you make about application forms when the student does not intend to seek employment presents a challenge that we always face. If not application for work, all ESL students are faced without filling out forms and as a teacher I want to have students learn to “transfer” knowledge so I think I might use your quote “learn to read and then read to learn.” That is I would use the application form for work as a transition to other forms and so filling out forms is the concept that I am teaching as well as helping students understand that there are many forms that they will have to fill out in English and forms have certain questions and vocabulary in common. That way there is a relationship between the assessment and what is taught. The context of the question is not as important as the ability to read and fill forms. I explain to students that the employment form for them is not important but I would try and brainstorm with them when they might need to help someone with an employment form.
Since you mention using CASAS life skills test you might want to look at the new tests CASAS has developed such as the Life and Work. If your state has other approved assessments for ESL then you might look at those assessments to see if you think it is a better match for your curriculum. I have found that CASAS Literacy tests and level A reading tests are very sensitive instruments in tracking lower level student learning gains.
Dan Wann
Professional Development Consultant
Indiana Adult Education Professional Development Project
dlwann@comcast.net
We have no ESL students. They are all basic literacy students, primarily at FFL levels 1-4. We use the Life Skills assessment because the one for Employment Skills is just too irrelevant to our many disabled and retired students. I have not seen the new series, but the term "work" makes me suspect it might bring up some of the same issues. And right now the both the time and money budgets are too tight to experiment.
The map question, which does *not* include a note that X means "you are here," has baffled many students, not just the excellent ones. Hate to "teach to the test" but now I try to remind our tutors to mark a "You are here" X on at least one of the maps they use in practice. It is just one of those things you either know or you don't. ("Everything is intuitive, once you know how," is one of my favorite quotes about learning a yet another computer application.)
I wanted to see which questions our students missed most, and over the years the map question with the unmarked X has been prominent, though other map questions were not. (The one with the left pointing north arrow is the second most missed map question.)
The person who did worse after I began comparing his answers to the correct master is a very artistic student who once just filled in the bubbles on the answer sheet in a pretty pattern. (Had to have a serious talk with him about that.)
Scaled scores are derived from the raw score. If one is low, so is the other. I used the term "doing so poorly" because I knew he had previously answered more than half the questions correctly on a parallel assessment. This time I was seeing very few correct answers, even at the very beginning of the assessment, where the questions tend to be easier for most students.
Sometimes he is fully engaged in what he is doing and we get a good assessment. When he is in a "lets get this over with" mood, anything can happen. The day he did better with the wrong answer sheet was a day he was doing random guesses. Even so, marked enough of them correctly for his scaled score to be in the valid range. However, the ones he hit correctly were scattered all over the place. As a human being could see the randomness. I just junked what he'd done that day and gave him a different assessment a couple of months later, when his attitude was more suitable. Glad the random marking didn't happen when we were up against the end of the fiscal year deadline!
Mary G. Beheler
Tri-State Literacy
455 Ninth Street
Huntington, WV 25701
304 528-5700, ext 156
Folks,
I hope that this is not a duplicative posting, but one implicit assumption being made here is that NRS is itself valid. Many of the postings assume that it is, and work from there to elaborate usage for various accountability and program improvement purposes.
I work in Ohio and have noted the requirements promulgated by DAEL that other systems must align to NRS (most recently last fall with the Federal register outline of the periodic review process) and in various letters from the AIR psychometricians to states or vendors.
When a national system is above all of the state systems, I would think that we should strive to keep it on the quality control line as well as all of the underlings.
In my reading of the history of NRS, I have not seen the sorts of construct validity studies that seem to be requested / required for state tests and vendor tests. Consider the width of the levels and the statements that comprise them.
There was a thread of work coming out of the CRESST organization a few years ago (Eva Baker's name comes to mind) called Standards for Educational Accountability (SEA) which might be useful if folks wanted to think about validation of NRS. Another useful framework is the Data Quality Campaign (DQC) that Achieve, Inc. and other organizations are using to improve the seamlessness of transitions in P-16 systems. In Ohio, for example, we have ABLELink for adult basic education, EMIS for K-12, and other systems for One-Stops and post-secondary. Many of these systems end up in front of the "tower of babel" when they try to communicate, as noted in some of our "Data Match" to unemployment wage records.
JTA
CoreComm Webmail.
http://home.core.com
Maybe aggregate figures for large groups can give some indication are doing better or worse than they used to. Maybe.
But NRS gives surely gives meaningless results for small groups like ours. (Total 20-25 students with 12 or more hours per year) One individual can vary the group results far too drastically.
My first posting on the 16th asked a question about the statistical validity of the figures for a five member level containing only 3 or 4 post-tested individuals. If these figures has real meaning for our group, would someone please explain? (I'm not a statistician, but I have had some college math classes, including calculus.)
The standard for a level with only one member could be 2% or 98%, it still amounts to pass/fail. At level 4 (high intermediate ABE) the student could improve more than 10 CASAS points and the whole level, and therefore our group, would not be an NRS success!
Yes, I know, the state will take our smallness into consideration when looking at our figures, but the rules are not written that way.
It seems like it would be useful for small groups to have their results bundled for comparison to the big guys. Maybe invent a standard for a whole organization, if it is big enough. If level-by-level analysis of small groups is as nonsensical as I think it is, why burden us with it? Let us just report.
And you are right, JTA, the whole NRS system might be useless in the long run. Maybe even harmful.
Mary G. Beheler
Tri-State Literacy
455 Ninth Street
Huntington, WV 25701
304 528-5700, ext 156
