Summary
From LiteracyTentWiki
Back to Measuring Literacy: Performance Levels for Adults, Interim Report
Summary of Discussion
NIFL-Assessment Listserv
Measuring Literacy: Performance Levels for Adults, Interim Report
June 20 – 24, 2005
Guests: Judy Koenig and Stuart Elliot, Project Directors
The Guests provided List members with several questions to both get people thinking and to indicate areas of the report that they wished to elaborate on or clarify. The questions focused on several areas including explanations of various research procedures; a discussion of the range of levels; and recommendations that the Committee put forth.
The questions posed by List members ranged broadly and focused on the levels of literacy, particularly non-literacy levels and how to capture such information on the NRS; correlations among commonly used ABE tests and the surveys and/or NRS; the Committee’s response to questions concerning other present-day literacy surveys and their results; the interpretation and use of the survey results; the usefulness of such surveys; information on the demand and/or need for literacy services in the U.S.; and definitions of literacy.
In discussing the levels, the Committee noted that stakeholders indicated they wanted information on the lowest literacy levels of adults. They developed 2 categories to try and capture data at the lowest ends of the survey: “non-literate in English” for those people who were unable to pass the initial screen for the ALSA (for low-literate adults), and “below basic” for those who were unable to score at the Basic Level in the main assessment. The 1992 NALS did not include the ALSA, thus it is not possible to make the same fine distinctions at the lowest levels of literacy in the NALS as is possible with the NAAL.
In response to questions regarding the pressures of the NRS, and the lack of ability to capture gains in very low level learners, the Committee noted that while the NAAL and the NRS are not aligned, “there appears to be a rough parallel between Non-literate in English and the NRS beginning literacy level; between below basic and the NRS beginning basic and low intermediate levels; and between basic and the NRS high intermediate level." The Committee was not charged with, and did not make, any recommendations to changes with the NRS.
The Committee was not able to comment on other present-day surveys as they were charged with focusing on the NALS and the NAAL only. However, the Committee did note that some of their recommendations can apply to other situations. For example, any conclusions drawn must be justifiable, based on the test or survey’s development procedures and its purpose. Also, the Committee highlighted the fact that a certain amount of judgment goes into test design, and can therefore be reasonably disagreed with.
Listmembers carried on with a discussion that focused on the usefulness of the NAAL and other large literacy surveys, and whether they served to paint an accurate picture or not of adult literacy, and whether or not they clarified the demand/need for literacy services or whether they “muddied the waters”. Various opinions were expressed.
Listmembers discussed the role of the classroom teacher in the landscape of assessment and testing, and whether or not they can or should become involved in the research and development of tests, and the interpretation and reporting of results. Opinions expressed ranged from full support to skepticism.
Members began to discuss reading versus literacy in the context of assessing reading, focusing on the TABE. Several posts discussed the TABE as a test of reading or not. This thread also led to a discussion of definitions – particularly the definition of reading and the definition of literacy. One post discussed the EFF definition of reading and noted that it is more useful because it allows for a much broader definition than the TABE.
In framing the way literacy is viewed in the Interim Report, the Committee provided some examples of typical ways that literacy skills are used – they did not attempt to define literacy per se.
The Committee tried to develop performance levels that would be useful to stakeholders – particularly for teachers and administrators who must respond to the NRS requirements. Although the NALS/NAAL and NRS are not aligned as noted above, the Committee did their best to map the levels as closely as possible.
One List member asked questions regarding correlations among some of the common commercial assessments and the NAAL, but the results of the 2003 NAAL have not been released yet. More information concerning these questions will be forthcoming upon the release of the 2003 NAAL in the fall of 2005.
In terms of questions regarding interpretation of results, the use of “demand-side analysis” would have provided a better basis for making the types of inferences that stakeholders evidently wanted. The survey was not designed to collect information on what adults should know or need to know in order to function in society (which is an enormous undertaking to determine); thus, inferences that indicate a judgment about an individual’s ability to function in society are not valid. As well, the survey was not designed to evaluate a program; however, it can be used to estimate program needs.
The Committee designed the performance levels for reporting the NAAL results with the intention that they would be more useful for policy and programming decisions. It is unclear whether these levels will also be helpful to the classroom teacher in her instruction.
Once the results of the 2003 NAAL are released in the fall, it may indicate the numbers of people who are in need of adult literacy services in the U.S. versus the number who are being served. However, the results will not reveal how many people are in need of services who are not aware that the services exist.
The discussion concludes with suggestions of further resources and sources of definitions.
