NASDiscussion
From LiteracyTentWiki
Back to Measuring Literacy: Performance Levels for Adults, Interim Report
Date of post: June 20, 2005
From: Marie Cora
I'm going to start us right off by asking our guests to respond to the first item that appeared in the "suggested preparations" section of the announcement, which I've copied below:
"Questions concerning the lowest and the highest categories of the literacy levels; specifically, questions concerning the difference between the Non-Literacy in English and Below Basic categories, and comparing the percentages of folks falling within these categories between the 1992 survey and the 2003 survey;"
I know that the 2003 survey included additional components to try to gauge the lower ESOL levels, so I'm interested in hearing about that. I'm also interested in hearing about the Non-Literacy versus Below Basic categories.
marie cora
Moderator, NIFL Assessment Discussion List, and
Coordinator/Developer LINCS Assessment Special Collection at
http://literacy.kent.edu/Midwest/assessment/
Date of post: June 21, 2005
From: Judy Koenig
Hello, I'm writing to reply to Marie's opening questions about the differences between the Nonliterate in English and Below Basic Literacy performance level categories.
When making plans for the 2003 NAAL, the Dept of Ed decided to design a special supplemental assessment (the ALSA) that was intended to evaluate skills of low-literate adults. The Dept. developed a set of screening questions that were to be used to get an initial idea of a participant's literacy skills. Based on their responses to the screening questions, participants were either assigned to take ALSA or the main NAAL assessment. The NRC Committee developed the Nonliterate in English category as a means for documenting the percentage of adults who's skills were so low that they could not "pass" the screening questions. This includes both the individuals whose skills were so low that they could not even attempt the screening questions as well as those who tried the questions but could not "pass" them and were assigned to ALSA.
The Below Basic Literacy category is intended to document the percentage of adults whose literacy skills were sufficient to "pass" the screening questions but were still extremely low. This category includes the individuals who were assigned to take the main NAAL but whose scores were lower than the cut score for the Basic category.
Having the two categories allows for finer distinctions among low literate adults than were possible with the 1992 NALS results. In the discussions that the NRC Committee had with stakeholders, some indicated a desire to know the percentage of adults in the U.S. whose literacy skills were so low that they would be regarded as "not literate." The Nonliterate in English category was designed to convey this type of information, since adults who are classified into this category have extreme difficulty reading in English. Stakeholders also indicated that it would be useful to know the percentage of adults who could read a little but not very well, and the Below Basic Literacy category is intended to respond to this information need.
Because the ALSA was not available in 1992, it is not possible to identify the group of adults who should be classified as Nonliterate in English. So, for the 1992 results, all of the individuals whose scores were lower than the cut score for Basic were classified as Below Basic (that is, the lowest category for NALS is the Below Basic category). Therefore, when making trend comparisons (between the 1992 NALS results and the 2003 NAAL results), the 2003 percentages in Nonliterate in English and in Below Basic need to be combined. This will allow measuring growth over time in the percentages of individuals in the lowest category of literacy.
I hope that this provides additional information to explain what the Committee intended by forming these two performance levels. I welcome this opportunity to further discuss this and other topics about our report.
Sincerely, Judy Koenig
NRC Study Director for the Committee on Performance Levels for Adult Literacy
Date of post: June 22, 2005
From: Debbie Yoho
Judy, a primary concern of many direct providers of literacy services like me is the accountability standards of the National Reporting System in relation to the lowest level learners. Do you know if the new data will be used or could be used to establish two very low levels in the NRS system? Many providers feel it is not reasonable to expect learners who score that low to achieve the equivalent of two grade levels per year in order to meet NRS standards. For too many, progress is much slower. I believe the current system creates a disincentive to serve the lowest level learners because the accountability pressures are related to funding. In short, unless something changes, many feel that serving "too many" very low level learners could jeopardize an entire program.
Deborah W. Yoho
Co-moderator, NIFL-Health Listserv
Executive Director, Greater Columbia Literacy Council
Past President, SC Adult Literacy Educators
2728 Devine Street, Columbia, SC 29205
803-765-2555 Fax 803-799-8417 dwyoho@earthlink.net
Date of post: June 23, 2005
From: Judy Koenig
Hello, I'd like to respond to [Debbie Yoho’s comments from yesterday]:
The Committee was well aware of the NRS and the challenges the NRS poses for adult educators. We tried to keep the NRS levels and legislative requirements in mind as we made decisions about the performance levels. We address this on page 4-12 and 4-13 of the report, where we say:
"In identifying these levels, we were conscious of the fact that one of the chief audiences for NAAL results is adult education programs, which are guided legislatively by the Workforce Investment Act of 1998. Title II of this act mandates an accountability system for adult education programs, known as the [NRS] that specifies a set of education functioning levels used in tracking the progress of enrolless. Feedback from stakeholders emphasized the usefulness of creating levels for NAAL aligned with the NRS levels. Although it was not possible to establish a clear one-to-one correspondence between NAAL performance levels and the NRS levels, there appears to be a rough parallel between Nonliterate in English and the NRS beginning literacy level; between below basic and the NRS beginning basic and low intermediate levels; and between basic and the NRS high intermediate level."
I hope that this helps a bit, although I recognize that it doesn't fully answer your question. The committee did not make any recommendations about changes to be considered for the NRS levels since this would have been beyond the scope of their charge.
Sincerely, Judy Koenig
NRC Study Director for the Committee on Performance Levels for Adult Literacy
Date of post: June 22, 2005
From: Tom Sticht
Questions Regarding the NAS/NRC/BOTA Performance Level Report and the Adult Literacy and Lifeskills (ALL) Report
Recently the OECD, Statistics Canada, and the U.S. NCES released a report of the Adult Literacy and Lifeskills (ALL)survey. In this report the authors have overstated the extent of adults with literacy problems in Canada, the U.S. and other countries because they used a methodology which the new National Academy of Science report on performance levels for adult literacy states is too stringent and exaggerates the problem of adult literacy.
The ALL authors claim that people without literacy skills at (NALS/IALS) ALL Level 3 or above lack skills to cope in contemporary society but the new NAS/NRC report says that is not demonstrated by their methodology.
Also, the idea that people at a lower level of literacy, Levels 1 or 2 can only perform simple literacy tasks is also wrong according to the NAS/NRC report, because people at the lower levels can perform tasks that are at higher levels of difficulty but with less than an 80 percent probability of success.
One of my concerns is that by overstating the extent of adult literacy problems (e.g., crying "wolf" too many times) policymakers become unbelievers to a large extent as do the public in general so funding is not provided for the really poorly literate adults to the extent it is needed.
Given the fairly widespread understanding of the shortcomings of the NALS and IALS, and given that the National Academies of Sciences, National Research Council, Board of Testing and Assessment has released its report on adult literacy performance levels confirming the many technical and arbitrary decisions that have gone into the NALS, IALS and now the ALL, I was stunned to read the ALL report that simply ignored all these various concerns. What bothers me is that these various government agencies are widely disseminating what they know are misleading statistics.
I'm wondering what responsibility the National Academy of Sciences has in addressing the shortcomings of the new ALL and informing policymakers, news agencies, and the general public about the misleading ALL report. This is particularly important since the ALL report also disseminates faulty data about the U.S. I'm also wondering what the responsibilities of the larger community of adult literacy researchers and scientists are in trying to bring the misleading statements of the ALL report to the attention of the general public.
Does anyone from the NAS/NRC/BOTA report on performance levels for adult literacy have any thoughts about any of this?
Tom Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht@aznet.net
Date of post: June 24, 2005
From: Judy Koenig
A quick response to Tom Sticht's request in which he asked if anyone from the NRC's Committee or the NRC would be willing to comment on the recently released ALL report. Unfortunately, it would be very difficult for me or anyone on the Committee to comment on the ALL report. The Committee's charge was to make recommendations about performance levels for NALS and NAAL, not to review and make suggestions for ALL. In order to come up with the recommendations for NALS and NAAL, the Committee undertook an intensive review of the assessment, its purposes and design, its uses, and the other factors, as detailed in the report. The Committee did not do a similar sort of intensive review of ALL, so it would really be inappropriate for them (or me) to make any comments about ALL. There may be a few messages in the report that could be applied to ALL, however, as the Committee tried to make suggestions that would be broadly useful to those making decisions about assessments. For instance, the Committee emphasizes that the conclusions drawn about test results should be justifiable, given the procedures used for developing the assessment and the intent of the assessment. The Committee also tried to highlight that there are a number of judgment calls involved with designing tests and setting standards. While there are professional guidelines that layout best practices, many of the decisions about test development and standard setting require making judgments, and reasonable people might disagree about what are the right choices.
Sincerely, Judy Koenig
NRC Study Director for the Committee on Performance Levels for Adult Literacy
Date of post: June 22, 2005
From: Debbie Yoho
Tom Sticht is to be congratulated, once more, for his tireless stand for quality and accuracy in everything we do. However, I think that those of us in the field have little to contribute about the ongoing issues of validity regarding the NALS and ALL instruments. From a practical standpoint, it is an argument among statisticians, researchers and policy makers, and no literacy practicitioner I know is equipped to discuss the issue intelligently, including me. But I do think his comments underscore a serious problem we have throughout the field of adult ed: the need to come to a place of common ground in defining concepts such as "literacy", "reading", "adult education", "proficiency", and "measurement". This is a reflection of the challenge of putting research into practice, and of practice informing research.
Rightly or wrongly, the statistics from the NALS are in wide use and have been for some time, and from where I sit have helped, not hindered, the case for more services. I expect and hope the ALL will do the same thing. Having said that, because I respect Tom's expertise, I always use the NALS numbers with the caveat that the figures may be too high or too low, stressing that a person's proficiency is always a matter of context, and therefore shifts across a continuum. No single measurement or instrument can capture those dynamics.
On the other hand, a major practical problem for practitioners, as alluded to in a previous post I offered in this discussion, is that the NALS certainly masked the prevalence and seriousness of the reading problem many adults struggle with. Hence the good news of two categories "below basic". It seems to me the ALL has been constructed with a lot of input from the field on this issue, and therefore must be an improvement, not a continued muddying of the waters.
Deborah W. Yoho
Co-moderator, NIFL-Health Listserv
Executive Director, Greater Columbia Literacy Council
Past President, SC Adult Literacy Educators
2728 Devine Street, Columbia, SC 29205
803-765-2555 Fax 803-799-8417 dwyoho@earthlink.net
Date of post: June 23, 2005
From: Marie Cora
Hi Debbie,
I wanted to counter your idea that literacy practitioners are not equipped to join such discussions. It may in fact be true that in general practitioners are not educated in highly technical issues of test development and interpretation. But in my opinion, they should be. I also believe that little by little in this age of accountability, more and more practitioners ARE in fact becoming knowledgeable and skilled in test development, construction, and interpretation of results.
I believe that if some amount of pre- or in-service training around these issues were a mandatory part of ABE teacher preparation, as an entire field we would be better equipped to work with the present selection of commercial tests, respond more accurately to the NRS, identify the short-comings of both, and then contribute to the improvement of our system. Teachers 'home-grown' classroom assessments would also increase in quality, accuracy, and interest. It would make the practitioner base a much stronger voice for influencing the way that policies are developed and implemented.
I also agree to an extent that the results of the NALS raised awareness of adult literacy in our country, and that some amount of action regarding increased services has occurred. But as stated in the Interim Report, the interpretations of the tests were actually based on a misconception in the first place - that the NALS somehow is able to make judgments about what are adequate or inadequate skills/abilities to function at some level in society.
So my concern is actually the same case you make for saying that the NALS has helped the field: I think it has not allowed us to have a clear understanding of the picture of literacy in the U.S. today because literacy IS in fact way more complex that can be captured in a test. I'm not sure whether these tests have hidden the picture of literacy, or have uncovered the true problem. As you note, you can say that the statistics tell a particular story, but not without your caveat that perhaps the statistics are not quite on.
I think we do need to figure out how to gauge literacy across the nation - for our children, for our economy, for us to better understand and interact with the world. And to bring my comments to full circle, I think we as practitioners need to be right in there from the beginning because we are the ones who know what happens with the students.
The Report does indicate in the Recommendations section that the results were interpreted the way that the media, politicians, and the general public wanted them to. Where are the practitioners? They are not there - maybe for 2 reasons - because the interpretations didn't exactly ring true with the workforce? Or maybe because of what you said in your first paragraph Debbie: that practitioners are not prepared to be involved in the technical test venue. I'm not sure. But it did not escape me in the least that the Report's Recommendations for developing definitions of concepts of literacy must involve practitioners.
marie cora
Moderator, NIFL Assessment Discussion List, and
Coordinator/Developer LINCS Assessment Special Collection at
http://literacy.kent.edu/Midwest/assessment/
Date of post: June 23, 2005
From: Debbie Yoho
[In response to Marie’s post]:
"I wanted to counter your idea that literacy practitioners are not equipped to join such discussions. It may in fact be true that in general practitioners are not educated in highly technical issues of test development and interpretation. But in my opinion, they should be. I also believe that little by little in this age of accountability, more and more practitioners ARE in fact becoming knowledgeable and skilled in test development, construction, and interpretation of results."
I think Marie is absolutely right. We all should be knowledgeable in test development, etc., and many are working to develop their professional knowledge base to include this information. Unfortunately, I believe that so long as funding is so low that many programs have to operate with part-time and/or volunteer instructors, Marie's vision of a truly well-trained professional cadre is unrealistic.
>From another of her posts: "...I'm not sure if we should be defining literacy as a concept (as noted in the Report) or as a set of skills and abilities..."
Ahh, here is the rub. Defining literacy as either one or the other is highly problematic. But defining literacy as BOTH a concept (or set of concepts) AND a set of skills is even more problematic for test developers, or so I'm told by people who do understand test formulation.
Marie has done a service reminding us that literacy and reading are not the same thing, however one defines terms, and that the TABE is not a reading test. Effective July 1, our state will accept ONLY the TABE (and the BEST for ESL) for NRS accountability purposes. Suppose for a moment that a program offers and delivers ONLY reading instruction, which is pretty near the case in many programs, including mine. Clearly the TABE-only policy isn't fair to those programs. I hope the NAAL (ALL) data will help us make the case at the state level that if the AELS system is to include a wide variety of providers, both specialized and comprehensive, states must allow for more than one instrument as documentation for the NRS. In fact, it seems to me the feds should mandate that states may not demand the use of only one instrument for accountability purposes.
Deborah W. Yoho
Co-moderator, NIFL-Health Listserv
Executive Director, Greater Columbia Literacy Council
Past President, SC Adult Literacy Educators
2728 Devine Street, Columbia, SC 29205
803-765-2555 Fax 803-799-8417 dwyoho@earthlink.net
Date of post: June 22, 2005
From: Andrea Wilder
One point Tom made some posts ago (or maybe on another web site) was about the technology/format (what you will) of literacy problems and presentation. The difficulty he pointed out was the problem of extracting information when the format was different, for example, a bus schedule. This is a literacy problem, not a reading problem--it is the set-up for completion of a task: understanding, extracting meaning, from a bus schedule. So the format can trip up a naive reader, and if they have trouble reading words, the difficulty is compounded. Anyone who has trouble reading a guide for installing software knows what I mean.
5 terms to define, and we have gone around lots of time on these, I don't know that there is much more to say:
1) literacy 2) reading 3) adult education 4) proficiency 5) measurement
In my mind, there is considerable overlap between literacy and reading--literate has the root of "letter," and also suggests "to be educated." I take this last to mean some skill in understanding a topic through the mastery of terms, concepts, words, associated with that topic (domain). Literate without reading skill? Nope.
Reading I take to mean skill in decoding, that is knowledge of English (or other) spelling patterns and the ability to pronounce and "read" and understand words up to say a 6th grade level. This is where reading and literacy may divide--there are lots of words that the adult may be able to read easily which do not fit into a domain, are in fact scattered all over the place in different domains. And not all words are nouns and verbs--many are "function" words that have to be mastered in the "reading" process.
Proficiency and measurement. If I were judging the proficiency of a person's use of software manuals (an art form), then I would measure that proficiency by seeing the person follow the manual--or maybe translate it into usable English (which would imply proficiency with the words and concepts).
"Reading" implies more general skills, like finding a core idea, knowing vocabulary, writing in full sentences, using quotation marks, etc. I'd toss "inference" in there, too, and other skills associated with skilled reading.
Various kinds of standardized tests could be used to measure just reading. I say that based on Art LaChance's observation on various lists that progressively higher TABE scores correlated well with ability to pass the GED--so these are basic reading skills and schooled skills. I would put in this domain (reading) the Wilson program, and some LVA materials, for example.
"Adult Education" is too mushy a term to be able to define well, though I use it myself as a catch all when I want to avoid more precise definitions.
Tom did interesting work which"translated" grade levels into FCE levels--I hope I've got that right, Tom. Maybe not, time for review?
EFF is kind of a hybrid, tosses in other elements.
Andrea Wilder
AWilder106@aol.com
Date of post: June 24, 2005
From: Tom Sticht
Andrea has brought up the idea of the distinction between literacy and reading. In a paper entitled Teaching Reading With Adults (www.nald.ca/fulltext/sticht/jan02/cover.htm) I discuss the merging of alphabetics with graphics technology to create graphic displays for reading of the written (graphic) language as a second signaling system for the spoken language.
The use of the three primary properties of graphic displays, ie., their use of light, their ability to be arrayed in space, and their relative permancence, when combined with the unique properties of language make possible the many types of literacy tasks assessed in the YALS/NALS/IALS/ ALL and forthcoming NAAL.
Tom Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht@aznet.net
Date of post: June 23, 2005
From: Marie Cora
Hi Andrea, I think you are hitting on a number of the issues that the Interim Report also gets at.
First off, the issue of defining literacy. I feel like if this were actually done, so many of the difficulties the field grapples with could be corralled. For example, being able to clearly state who we are, what we do, how we do it, how effective we are, etc. so that EVERYONE could understand us. So information and communication would change dramatically. There are many examples, and I'm sure you don't need a laundry list. But after reading the Report, I feel the need to take a step back from even the list of 5 terms to define that you note below - because I'm not sure if we should be defining literacy as a concept (as noted in the Report) or as a set of skills and abilities. Maybe both? For different purposes or reasons?
High Stakes tests don't really allow for It Takes a Village notion - but if you think about what you do at home and in your daily life, I bet you share your literacy with others (and borrow from others) more often than you don't. And if we think about our classrooms - how many of us purposefully construct group work because we know that this type of scaffolding really enhanced the educational process.
I think it's absolutely good to gauge the individual. I just worry that if all the emphasis lies on the individual, then that is not reflected in our reality of life.
So back to the definitions: so do we need to define literacy in both ways? Again: for different purposes and needs?
Andrea - I need to put on my educational assessment hat now: neither the TABE nor the GED (unless you are specifically talking about the reading portion of the GED) are reading tests. They were not developed that way, constructed that way, and they are not structured that way. The TABE may touch on just what you said: reading skills. But reading skills assessed in discreet ways on a multiple choice test is not at all assessing a person's ability to read. Progressively higher TABE scores correlating with passing the GED does not speak to a person's reading ability at all. It speaks to the person's ability to do well enough on the TABE test so that they can move on to master the GED tests.
marie cora
Moderator, NIFL Assessment Discussion List, and
Coordinator/Developer LINCS Assessment Special Collection at
http://literacy.kent.edu/Midwest/assessment/
Date of post: June 23, 2005
From: Andrea Wilder
I agree with Debbie that how on earth the field can become professionalized (people with lots of specialized knowledge) is beyond our scope right now. Multiple reasons.
OK--TABE. I read what Marie said--it is not a reading test--and I thought: of course it isn't! Because you never hear a person reading! Then I rethought that. If it isn't a reading test, what is it? It does seem to meet the criteria I advanced for "reading" as a mostly taught-in-school subject. Some illumination here would be useful.
Literacy. It can include both dimensions--concept and set of skills (these need to be defined somehow), it depends where you look on a vertical continuum. I can imagine looking at a set of skills that I am teaching people, and that are different from straight reading. As I go on looking at this set of skills, it occurs to me that they can be bundled together as a concept, literacy, as I defined it in my earlier post.
I really appreciate the informed minds that are brought together this week on the list serv, and thanks to Marie and respondents for doing this.
Andrea Wilder
AWilder106@aol.com
Date of post: June 23, 2005
From: Debbie Yoho
Andrea, I'll take a stab at your question about TABE because I have the time right now. However, the main thread on this list right now is on the NALS-NAAL discussion, so I hope we don't go off too far on just TABE. A lot has been written about it here anyway.
The TABE is essentially an achievement test, in my view, although Contemporary makes a stab at setting it up to be "diagnostic". It is available in a Survey version or a longer Full Battery version. It has several subtests, including a Reading Subtest, a Spelling Subtest, and a Language Subtest. In my view, the test questions stress the APPLICATION of some skills, and not even very many skills at that. The reading subtest is primarily a comprehension test, including summarizing, cause and effect, identifying details, identifying opinions as opposed to facts, context skills, and some vocabulary, etc. The test does not check mastery of the many skills that the skill-and-drill people would identify as essential to reading (long/short vowels, blends, diagraphs, etc. although this is arguable.) It also includes questions similar in format to those on the NALS, using "real life" materials such as bus schedules, or a receipt from a store purchase. Other questions are more "story oriented". I feel it places great stress on political correctness for a vastly diverse market. For example, one passage is about Eskimos and another about a Chinese family. That's not a bad thing, by the way, but my southern Black learners have had no exposure to Chinese or Eskimo names, and are confused when confronted in a test environment with terms and names they've never seen before, hence my pet peeve that I think the TABE is culturally biased because of its attempt not to be culturally biased! But that's another discussion...
I think most practitioners would agree that the TABE is a compromise in the search for an instrument that is useful in the broadest possible sense, and this is why it is so widely used. At least it is designed specifically for adults. And as an achievement test, it is valid and reliable. It just isn't what instructors need to plan customized instruction, it isn't nearly sensitive enough to pick up gradual gains, it doesn't assess mastery, it can be very quirky in its results especially with the lowest level learners, and in my opinion it is culturally biased unless you live in a highly diverse neighborhood. Is it a reading test? Well, not the whole TABE. The reading subtest is an assessment of reading achievement if you define reading as the Elementary and Secondary Schools Act does (!), but not if you include useage and spelling as reading skills.
Deborah W. Yoho
Co-moderator, NIFL-Health Listserv
Executive Director, Greater Columbia Literacy Council
Past President, SC Adult Literacy Educators
2728 Devine Street, Columbia, SC 29205
803-765-2555 Fax 803-799-8417 dwyoho@earthlink.net
Date of post: June 23, 2005
From: Andrea Wilder
Thanks also for your cut on the TABE. Yes, I'd say a reading achievement test, with an assumption that the person knows how to read the words presented.
Andrea Wilder
AWilder106@aol.com
Date of post: June 24, 2005
From: Marie Cora
Hmmm...I still don't find the TABE to be a test of reading. Even with the thoughtful ideas on it presented by folks. You need to be able to read to do the test, true. So the way I interpret that is to say that one has enough skills to be able to tackle (or master) a selected-response test (multiple choice test).
What do others think?
marie cora
Moderator, NIFL Assessment Discussion List, and
Coordinator/Developer LINCS Assessment Special Collection at
http://literacy.kent.edu/Midwest/assessment/
Date of post: June 24, 2005
From: Amy R. Trawick
Marie, earlier you said that the TABE and GED are not reading tests. Specifically, you commented "They were not developed that way, constructed that way, and they are not structured that way." I understand your comment in terms of the GED, but I'm curious why you say this about the TABE Reading test. Are you saying that the TABE developers had no *intention* of measuring reading, or that they failed in the attempt?
Amy R. Trawick
North Wilkesboro, NC
atrawick@charter.net
Date of post: June 24, 2005
From: Marie Cora
Hi Amy, Thanks for your question.
I cannot speak to what the TABE developers' intentions are or were. I am referring to comments made earlier this week that it can be used to assess reading.
The reading section may indeed capture some of a person's ability to read. I think I noted that some reading skills can be determined. However, what I often hear from folks in the field is that the TABE is the only test that can be used by a state or programs to report on reading levels and reading gains of the students they serve. People just don't seem very satisfied with this. Yes, you need to read, but are you examining the person's ability to take the test, or a person's ability to read? That's really my question here. I don't necessarily have an answer, just opinions.
As for failing in an attempt to make a reading test (just my opinion here) - I don't think so. The TABE was developed to try and capture stuff that is extremely hard to capture. At some point (and others would know the history of TABE much better than I), folks were wise enough to understand that we need to build a coherent system of education for our adult population - which is only a good thing. Do I think that it's time to re-examine our tests and our system so that it better meets our students' and teachers' needs: yes. And I believe that we can do this if we do it together with all the stakeholders that are involved in our field.
marie cora
Moderator, NIFL Assessment Discussion List, and
Coordinator/Developer LINCS Assessment Special Collection at
http://literacy.kent.edu/Midwest/assessment/
Date of post: June 24, 2005
From: Amy R. Trawick
Marie, I appreciate the clarification. I thought you were somehow privy to information that the TABE Reading test hadn't been intended to measure reading. I think it would be fair to say that the TABE developers and users, for the most part, consider it a reading test. But you and Andrea and others are raising an important question: how is the TABE defining reading? And, relatedly, is that a useful/meaningful definition of reading?
The "reading" that is assessed on the TABE seems to be implicitly defined as something that one does to written text in order to find or extract meaning (that is "in" the text, if you know how to find it). I think a more useful definition of reading is found in the EFF Content Standard Read With Understanding. I understand the development process of this standard to be somewhat like the "job analysis" that Judy described in her last response. That is, through use of focus groups including adults from various walks of life, consensus was reached about the key responsibilities that adults have in their major roles as workers, family members, and community members. Next the skills required to fulfill these responsibilities were identified and *then* what these skills looked like in application were described to create the content standards. Thus, reading is defined in the Content Standard Read With Understanding (http://eff.cls.utk.edu/fundamentals/standard_read_with_understanding.htm) as a problem-solving process that originates from a meaningful purpose and requires the application of cognitive and metacognitive strategies to achieve that purpose.
This definition is much broader than the implied one used in the TABE. If I were using EFF's definition of reading to design instruction, I would want to make sure students knew that we read for different purposes, that we apply different strategies based on those purposes, that figuring out the author's meaning is one part of the process, and that what I already know may help me figure out what the author's meaning is but then I have to figure out what that means to me (related to my purpose). I would structure learning assignments that required them to apply this knowledge for meaningful, often life-based, as opposed to just academic, purposes (though learning how to read "academically" would certainly be a part of instruction. And I would ensure that they encountered/used/read multiple kinds of texts, not just workbooks. If I were using this definition of reading to design assessment, the same would apply. I would want to measure to what degree students are able to identify their own purposes for reading, employ strategies to problem-solve, and construct meaning that addresses their purposes for reading. The TABE just doesn't do this.
So, to your statement:
"Do I think that it's time to re-examine our tests and our system so that it better meets our students' and teachers' needs: yes. And I believe that we can do this if we do it together with all the stakeholders that are involved in our field."
I absolutely agree!
Amy R. Trawick
North Wilkesboro, NC
atrawick@charter.net
Date of post: June 24, 2005
From: Andrea Wilder
Guests,
I did some home work last night and am now up to speed, sorry for going off track yesterday.
First, I am interested in knowing how you came up with the definition of literacy in the first paragraph of the Executive summary. Are these "common sense" definitions, or are you citing conclusions from other work?
Also, it looks to me, going over the NAAL levels, TABE levels, NRS levels, there is an attempt to make these congruent; I hope I have been reading accurately. Is this true?
I read on ES-6 that it was not possible to establish a one-to-one correspondence between the NRS and NAAL levels, but that there was a "rough parallel," I think this is what I am getting at.
If there were this correspondence, then the literacy description (NAAL) would key into one of the major proficiency tests, and then the reporting/accountability requirements.
Andrea Wilder
AWilder106@aol.com
Date of post: June 24, 2005
From: Judy Koenig
Response to Andrea's Question 1: The Committee did not intend this to be a "definition" of literacy but simply some examples of the variety of ways in which literacy skills are used. We wrote this paragraph to alert readers (particularly readers who have not thought about literacy in the way that you and others on this listserv have) to the importance of literacy skills. The ideas are the committee's based on their own experiences and the materials that they reviewed.
Andrea's question 2: Also, it looks to me, going over the NAAL levels, TABE levels, NRS levels, there is an attempt to make these congruent; I hope I have been reading accurately. Is this true? I read on ES-6 that it was not possible to establish a one-to-one correspondence between the NRS and NAAL levels, but that there was a "rough parallel," I think this is what I am getting at. If there were this correspondence, then the literacy description (NAAL) would key into one of the major proficiency tests, and then the reporting/accountability requirements.
Response: You are reading the text as we intended. We tried to develop performance levels that would be useful to a variety of audiences, but particularly to adult educators who most address the requirements of NRS. Given the scope of what is assessed by NALS/NAAL (e.g., the test frameworks, specifications), it wasn't possible to completely align NALS/NAAL levels with NRS levels, but we did the best that we could. And, we provided the mapping from one to the other on page ES-6 to assist with this.
Sincerely, Judy Koenig
NRC Study Director for the Committee on Performance Levels for Adult Literacy
Date of post: June 24, 2005
From: Andrea Wilder
Did the Committee discuss a Spanish language track for literacy?
In our state, Massachusetts, there is a need for many more ESL classes for immigrants; I have also heard (have not verified) that ESL is a health risk factor. I understand that our national immigrant policies are chaotic, and I wonder how this chaos may influence decisions made for descriptive literacy assessment, specifically the NAAL.
Andrea Wilder
AWilder106@aol.com
Date of post: June 24, 2005
From: Judy Koenig
The committee discussed the issue of immigrants and non-English speakers at length. While we did not call specifically for assessments in languages other than English, we did provide a number of suggestions for ways to collect information about the literacy levels (in English and in the native language) of non-English speakers. See pages 6-3 to 6-5 and pages 7-8 to 7-10 for these discussions.
Sincerely, Judy Koenig
NRC Study Director for the Committee on Performance Levels for Adult Literacy
Date of post: June 22, 2005
From: Miriam Kroeger
Not quite sure where to jump in with this question, so will start at the beginning. Have been following the discussion, and have also gotten a little sidetracked with an off shot to one of the participants, but our side discussion prompt me to ask this question. Has there been any correlations of the "standard" assessments (TABE, CASAS,) and the GED Tests, including the English Proficiency test (Test 6) to the 2003 NAALS, or for that matter to the NRS?
Miriam Kroeger
Arizona
MKroege@ade.az.gov
Date of post: June 23, 2005
From: Howard Dooley
Miriam -- CASAS has a research paper which shows the (high) correlation between their assessments and the GED tests. You can find it on their website, www.casas.org. On the left look for "Research & Reports", click for the sub-menu and then click on "Assessment Research", and then again click on "CASAS/GED 2002". There is also a CASAS/GED Writing report, but the info is outdated. There are interesting things happening with the CASAS writing assessments, though, if you are interested...
At my program, we use the scores in this report as one indicator that a learner is ready to take a GED Official Practice Test, a pre-requisite to referral to a GED Test Center. We find this helps us in getting a post-test for learners in the GED level classes -- and our experience aligns well with what is reported in this brief, which helps sell its usefulness to instructors and learners. (It's not just "one more thing to do" for Howard...)
Howard, Project RIRAL
hdooley@riral.org
Date of post: June 23, 2005
From: Judy Koenig
Hello, I'd like to respond to [Miriam Kroeger’s post from yesterday]:
Our report discusses the performance levels we recommend be used for reporting the 2003 NAAL results, and for purposes of exemplification, we applied the levels and cut scores to the 1992 NALS results. Our report doesn't include any actual date from the 2003 NAAL because the Department of Education has not yet publically released them. The Department was awaiting our recommendations for performance levels and cut scores before reporting the 2003 results, and they are now in the process of preparing their reports. Once the 2003 NAAL results are released, I would expect that analyses that had been conducted with the 1992 NALS (such as the GED study) would be replicated with the new results.
Sincerely, Judy Koenig
NRC Study Director for the Committee on Performance Levels for Adult Literacy
Date of post: June 23, 2005
From: Marie Cora
I am posting several questions that I have for our guests. These are all based on the Recommendations Chapter of the Interim Report. Please feel free to jump in and add to the questions, or make your own comments.
1. Please define "Demand-Side Analysis"; give us an example, perhaps one that compares demand-side analysis to other types of analyses. What sort of analysis was utilized in determining the original results of the NALS and the NAAL? Do you know why?
2. The way the test was developed does not support standards-based inferences being made about the data. However, when the data was reported, inferences were in fact based on a some set of standards. Is it possible to re-interpret the data based on a different set of analyses? Do you feel that the results would be very different if they were? Would this be a useful exercise or not?
3. The NALS and NAAL are examples of assessments that try to get at evaluating program needs. So how do the results relate to the individuals that took the tests? Can interpretations be made of the individuals then, if the assessments were designed to examine the larger program needs? And if literacy is a collaborative process in many instances, and many programs structure the students in groups (groupwork), wouldn't the validity of testing individuals be problematic? What is the relationship between testing the individual and inferring that programs who serve some particular population might be adequate or not?
4. Why was the health section done differently than the other sections? Wouldn't that affect the validity and reliability of the rest of the test?
marie cora
Moderator, NIFL Assessment Discussion List, and
Coordinator/Developer LINCS Assessment Special Collection at
http://literacy.kent.edu/Midwest/assessment/
Date of post: June 24, 2005
From: Judy Koenig
[Responses to Marie’s questions]:
Response to Question 1:
One way to think about this is to think about the way that licensing and certification tests are developed (e.g., medical licensing exams). The development process for such tests first involves something called a "job analysis" to determine the knowledge, skills, and competencies a licensed or certified professional should know and be able to do. This is not an easy task and involves surveys, discussions with stakeholders and professionals working in the particular profession, and judgments about what licensed professionals "should" know to function well (and safely). Often the process involves convening a variety of expert panels to make these sorts of judgments. Once there's been consensus on what a licensed or certified professional should know and be able to do, test questions are developed that measure these skills, and the rest of the test development process (field testing question, item analyses, etc.) can take place. Standard setting (the process of determining the score required to "pass" the exam; that is, the "cut score") is based on the judgements made by the expert panels. This process is quite different from the one that was used to develop NALS and NAAL. While NALS/NAAL did involve a variety of expert panels (such as, the literacy definition panels), the panels were not asked to make judgments about what adults should know and be able to do in order to function in society. (This is explicitly stated in the documentation about NALS.) Such a process would have been a vast undertaking. What the committee was suggesting by a "demand side analysis" is that a process like this be undertaken but on a somewhat smaller scale than what might be implied by "functioning in society." NALS and NAAL use stimulus materials and questions drawn from 6 context areas. The committee suggested that surveys and discussions (like the job analyses procedures described above) be undertaken that would delineate the types of demands that are put upon adults to function in each of the context areas. The Committee proposed that this sort of analysis be done first, and the assessments designed to measure the skills adults need to have to meet these demands. The committee recognized that this was no easy task, but still thought that this approach to test development would provide the basis for the sorts of inferences policy makers and the public wanted to make about NALS and NAAL results.
Response to Question 2:
The answer to this question is partly addressed in the above response about demand-side analyses. What we meant by "standards-based inferences," is that NALS and NAAL were not designed to provide information about what adults should know or need to know. When the assessments were developed, no expert panel was involved in deciding what is "adequate" literacy and what is "inadequate" literacy to function in life. Therefore, it isn't appropriate to draw conclusions like this about the test results. The Committee selected performance level categories that were intended to be descriptive of performance; in doing so, they do make judgments about what level of performance adults need in order to be classified into the levels (e.g., what should adults be able to do in order to be classified as "basic" or as "intermediate."). However, they did not make any recommendations about which of the levels would constitute adequate or inadequate levels of literacy to function well in life (e.g., Is basic literacy enough? Is intermediate Literacy enough?). This is a fine, but important, distinction.
Response to Question 3:
NALS and NAAL were designed to provide information about groups of individuals, in the same way that NAEP does. That is, the assessment doesn't report scores for individuals (although NALS and NAAL have statistically derived scores for individuals that can be used for research purposes), but they do report scores for groups, such as gender or ethnic groups. The assessments also weren't designed to evaluate programs. For instance, an assessment designed to evaluate a program would need to select a sample of adults from that program (e.g., adults who participate in ABE). That's not what NALS and NAAL did. NALS and NAAL were household surveys administered to a stratefied random sample of adults. Instead, NALS and NAAL results can be used to evaluate program needs. That is, the results can be used to estimate the need for ABE programs, for health literacy programs, or the like.
Response to Question 4:
The inclusion of additional health literacy questions should not affect the reliability and validity of the scores; that is, the meaning of the 2003 scores should not change, and the inclusion of such questions should not affect the comparability of results from 1992 and 2003. To explain: NALS included some questions that were based on stimulus materials drawn from health and safety contexts. The questions assessed the same sorts of skills as other questions on the assessment, and the questions contributed to scores in the Prose, Document, and Quantitative areas. For NAAL, additional questions were developed that were based on stimulus materials from health and safety contexts, and enough of these questions were developed that a "health literacy" score can be reported. However, the questions still measure the same skills as measured by questions that use other stimulus materials; and responses to these questions contribute to the Prose, Doc, and Quant scores.
Sincerely, Judy Koenig
NRC Study Director for the Committee on Performance Levels for Adult Literacy
Date of post: June 24, 2005
From: David Rosen
Some questions for our guests:
The NALS, and now the NAAL, as I understand them, attempt to measure levels of literacy of a representative sample of the U.S. population. The audiences for this information may include practitioners, but they also include policy makers, researchers, the general public and others.
1. Do we have any evidence that the NALS data have been used by any of these audiences to improve adult literacy education services? My impression is that practitioners do not (perhaps cannot) use the NALS data to improve instruction. And my experience with policy makers is that the NALS findings -- the large numbers of Americans in need of literacy skills -- has been dismissed as an exaggeration, or has resulted in a throwing up of hands of hopelessness. What exactly, do you think, is the added value of these studies? For example, do you think the NAAL will be more useful to practitioners and policy makers than the NALS? If so, why?
2. In addition to literacy levels, "need, for literacy in our society," there are two other related questions whose answers might be of more use to practitioners and policy makers: a) what is the actual unmet demand for services to improve basic skills? That is, how many people who need these services have stepped forward and are on waiting lists? and b) what is the "latent demand," that is, the number of people in need who -- if they knew adult literacy education services were available -- would step forward to enroll in adult literacy education programs? Have there been -- or is anyone planning -- national studies of demand or latent demand?
David J. Rosen
djrosen@comcast.net
Date of post: June 24, 2005
From: Judy Koenig
Response to David's Question #1:
I can only address this from the aspect of the performance levels, since this is what the committee’s report is about. The committee intentionally designed the performance levels to address specific policy and programmatic questions, and the report is pretty explicit about this (see bullets on ES-4 and supporting discussion on pgs. 4-11 and 4-12). I think that the committee hopes (and anticipates) that reporting NAAL results using these performance levels will enhance their usefulness to those making policy and programmatic decisions. I’m not sure that the levels could be used by adult educators to improve instruction per se, but would be interested in hearing your (and other listserve participants') ideas about this.
Response to David's Question #2:
The NAAL results, once they are released, should provide some answers to your second question. Based on the percentages of adults in the bottom three performance levels (non-literate in English, below basic, and basic), it should be possible to estimate the percentages of adults in this country who are in need of adult literacy education programs. As far as the percentages (or numbers) of adults who are already being served, I believe the Dept of Education collects statistics on this (see website referenced on pg. 6-4 of report). So, it’s likely that part of your question will be addressed by the release of NAAL results combined with information that’s already collected.
I think that you’re actually asking a much more difficult question, however; namely, how many people need the kind of assistance that adult literacy services can provide but who don’t even know such services exist or that they can receive such services. I don't know of any national studies that would target this question, but perhaps others on the listserve have information to share about this topic.
Sincerely, Judy Koenig
NRC Study Director for the Committee on Performance Levels for Adult Literacy
Date of post: June 24, 2005
From: Marie Cora
Hi everyone,
A lot of information and terms have been used with the discussions this week. I would like to point out a couple of resources where you can find some further info on tests (someone inquired about the CASAS), and where you can see some definitions of assessment terms as well.
You can go to the LINCS Special Collection in Assessment at http://literacy.kent.edu/Midwest/assessment/ At the Collection, you can find a section on Published Tests that will take you to the homepages of some of the commonly used assessments. You can also find the Assessment Glossary, and on-line resources for further reading.
You can also go to the Adult Literacy Education Wiki (ALEWiki) at http://wiki.literacytent.org/index.php/Main_Page and click on Assessment Information (number 3.7). At the ALE Assessment Wiki you can also find info on commercially available tests, a Glossary, on-line links for assessment, and past discussion excerpts from this List. At the ALE Wiki, YOU can contribute to the information there - a wiki is an interactive site where you can add information as well as go get it.
Here are two examples, taken from the ALE Assessment wiki:
http://wiki.literacytent.org/index.php/Grade_Level_Equivalent
http://wiki.literacytent.org/index.php/Scale_scores
Thanks!
marie cora
Moderator, NIFL Assessment Discussion List, and
Coordinator/Developer LINCS Assessment Special Collection at
http://literacy.kent.edu/Midwest/assessment/
