Assessment NAAL
From LiteracyTentWiki
NAAL Discussion on the NIFL Assessment Discussion List, assessment@dev.nifl.gov
From: marie.cora@hotspurpartners.com
Subject: [Assessment] NAAL release
Date: December 20, 2005 3:46:24 PM EST
Dear List Members,
Last Thursday, Dec. 15 the results of the NAAL (National Assessment of Adult Literacy) were released, and some interesting discussions ensued on the NLA List (http://lists.literacytent.org/mailman/listinfo/aaace-nla).
To find information on the NAAL and its background, as well as discussion threads from the NLA, go to the ALE Wiki, Public Policy area at: http://wiki.literacytent.org/index.php/Public_Policy
If you would like to hold discussions on the NAAL, please do not hesitate.
Thanks,
marie cora
Assessment Discussion List Moderator
From: marie.cora@hotspurpartners.com
Subject: [Assessment] Your questions re: the NAAL
Date: December 21, 2005 1:09:52 PM EST
Dear List Members,
Mark Kutner of American Institutes of Research (AIR) is on the List and would be glad to answer any questions people may have regarding the NAAL. One of Mark’s many projects is the NAAL study – you are director of that project, isn’t that right Mark? Folks, please take this opportunity to ask any questions you might have.
For information on adult education related projects at AIR, go to: http://www.air.org/projects/projects_ehd_adult_ed.aspx
Thanks,
marie cora
Moderator, The National Institute for Literacy Assessment Discussion List, and
Coordinator/Developer LINCS Assessment Special Collection at
http://literacy.kent.edu/Midwest/assessment/
From: Ajit.Gopalakrishnan@po.state.ct.us
Subject: Re: [Assessment] Your questions re: the NAAL
Date: December 21, 2005 2:02:40 PM EST
Hello Mark,
Hope you are well.
I have two questions: The first is about state level data. Will there be any specific state level data for those states that did not pay to have their sample sizes increased? I presume it is inappropriate to apply the national percentages to a states’ adult population. I am not sure how this was handled with the 1993 NALS but it appears that some extrapolation was done in the late 90s to arrive at state level estimates.
My second question is about the four literacy levels. I understand that these levels are different from the NRS levels but is there any way to establish some comparison? For example, can the descriptors for these four levels be compared with the NRS level descriptors? Knowing this would help to establish more precisely the need for the adult education system.
Thanks.
Ajit
Ajit Gopalakrishnan
Education Consultant
Connecticut Department of Education
25 Industrial Park Road
Middletown, CT 06457
Tel: (860) 807-2125
Fax: (860) 807-2062
ajit.gopalakrishnan@po.state.ct.us
From: MKutner@air.org
Subject: Re: [Assessment] Your questions re: the NAAL
Date: December 21, 2005 2:46:14 PM EST
Hi Ajit.
Now that the first NAAL report is out, all is well :) We are busy finalizing the next report which is scheduled for release in the Spring.
A model for developing small area estimates to estimate state level data for non-participating states, as well as sub-state data is under development. The work is being supported by NCES in response to state concerns. When this project is completed, non-participating states will have state-level and sub-state level estimates of prose literacy. Also, for the states that purchased a state sample their sub-state estimates will be more statistically precise. This work is being conducted by Westat, with AIR as a subcontractor. We will have a better idea about the time frame for the release of these small area estimates in the next couple of months, although as an NCES funded activity the work is subject to stringent technical review which can be somewhat time consuming.
Trying to link the NRS and NAAL levels is a complicated task. However, I strongly believe that the NAAL data in of itself certainly demonstrates the need for the adult education system. While a lot of media attention has focused on the adults with Below Basic literacy, many adult education clients are likely in the Basic literacy level. The abilities associated with below Below Basic literacy certainly places limits on adults as they undertake activities related to their roles as parents, workers, and members of the community.
I hope this helps.
Mark
Ajit.Gopalakrishnan@po.state.ct.us
Subject: Re: [Assessment] Your questions re: the NAAL
Date: December 21, 2005 3:04:11 PM EST
Thanks Mark for the prompt response.
I appreciate your clarification with respect to the level descriptors and the challenge in aligning them with the NRS descriptors. I agree that media attention has focused on the Below Basic level. In addition to that level, adult ed’s role relative to basic skill development for individuals in the Basic level is pretty clear.
After reading the descriptors and seeing some sample test items, it seems to me that the adult education role may extend into the higher levels, especially in terms of document and quantitative literacy. However, without some formal alignment with the NRS, I wonder how the adult education system will be able to make the argument for the great extent of need for its services.
Thanks.
Ajit
From: djrosen@comcast.net
Subject: [Assessment] Has U.S. Literacy increased, declined or stayed the same?
Date: December 21, 2005 4:18:29 PM EST
Hello Mark,
A colleague, on another list, writes: …in a decade in which the use of computer technology in our schools and colleges, and in the culture at large, has expanded significantly. . . the general level of literacy has declined.
He cites as evidence that the general level of literacy has declined this text:
Here are the findings as summarized by Grover Whitehurst of the Department of Education: his words, not mine.
Educational Attainment: 1992-2003
I will now present the results on change in scores between 1992 and 2003 for selected educational attainment levels. There were no increases in literacy in any of any of the educational attainment levels. Prose literacy decreased among adults at every level of education. This decrease calls out for more research. On the quantitative scale, there were no changes in literacy at any level of educational attainment. For document literacy, those with higher levels of education showed a decline while those with less education had no change. With scores dropping in prose literacy for every level of education, you might wonder why there was no overall decline in the average score for this type of literacy. This is because adults with higher educational levels tend to outperform those with lower educational levels, and the percentage of adults with high educational levels-those with "some college" or more-has been increasing, while the percentage with low levels of education has been declining. We have more higher-scoring adults with high levels of education, and fewer lower scoring adults with low levels of education, which offsets the fact that average scores for highly educated adults are declining.
and my colleague adds:
So: the point stands. In a decade of massive growth in the use of computers and the Internet in and out of school and college there has been no improvement in the literacy level of the nation's adults. Prose and document literacy have declined.
Leaving aside who said or wrote the paragraph on Educational Attainment 1992-2003 (I think it may have been Mark Schneider ) I am wondering if this is the right conclusion to draw from the NAAL results. On the DOE press release, it seems to me, there was a different overall conclusion:
The National Assessment of Adult Literacy (NAAL), released today by the National Center for Education Statistics (NCES), found little change between 1992 and 2003 in adults' ability to read and understand sentences and paragraphs or to understand documents such as job applications.
African Americans scored higher in 2003 than in 1992 in all three categories, increasing 16 points in quantitative, eight points in document and six points in prose literacy. Overall, adults have improved in document and quantitative literacy with a smaller percentage of adults in 2003 in the Below Basic category compared to 1992. Whites, African Americans and Asian/Pacific Islanders have improved in all three measures of literacy with a smaller percentage in 2003 in the Below Basic category compared to 1992.
Hispanic adults showed a decrease in scores for both prose and document literacy and a higher percentage in the Below Basic category. The report also showed that five percent of U.S. adults, about 11 million people, were termed "nonliterate" in English, meaning interviewers could not communicate with them or that they were unable to answer a minimum number of questions.
Other report highlights:
- White adults' scores were up nine points in quantitative, but were unchanged in prose and document literacy.
- Hispanic adults' scores declined in prose and document literacy 18 points and 14 points, respectively, but were unchanged in quantitative literacy.
- Asian/Pacific Islanders' scores increased 16 points in prose literacy, but were unchanged in document and quantitative literacy.
- Among those who spoke only Spanish before starting school, scores were down 17 points in prose and document literacy between 1992 and 2003.”
[ Press Release from the U.S. Department of Education http://www.ed.gov/news/pressreleases/2005/12/12152005.html ]
Mark, has the general level of literacy in the U.S. from 1992 – 2003 declined, increased or stayed the same?
Thanks,
David J. Rosen
Adult Literacy Advocate
DJRosen@comcast.net
From: MKutner@air.org
Subject: Re: [Assessment] Has U.S. Literacy increased, declined or stayed the same?
Date: December 22, 2005 4:12:17 PM EST
Hi there David,
Your provide a lot of interesting and intriguing comments in your email below. At the end of your email, however, you ask the question which I believe is what you would like me to answer:
"Mark, has the general level of literacy in the U.S. from 1992 - 2003 declined, increased or stayed the same?"
Our report shows that the general level of literacy has remained the same on the prose and document scales, and has increased on the quantitative scale. NAAL is able to detail literacy levels, but not literacy requirements, which I believe is the point of the earlier part of your email.
NAAL, as in most comprehensive studies, provides really good news and news that is not so good. There has been a statistically significant increase in the scores of Black adults over the past decade on all three scales. Data cannot provide causal inferences about why the literacy of Black adults has increased, but additional NAAL analyses will be looking at possible reasons, including increased educational opporutnities.
The decline in scores for adults whose first language is Spanish is of course very disturbing. Additional analyses that we hope to conduct will explore in greater depth the why this decline might be taking place. In addition to increased immigration, the 2003 NAAL also provides what we believe is more accurate data about the literacy of Spanish speaking adults. As detailed on page 18 of the report, the NAAL allowed Spanish speaking adults to read and answer the easy literacy tasks at the beginning of the assessment in Spanish (although the materials from which they needed to find the answer was in English). It seems that fewer adults were excluded from the assessment than were in 1992. Our next report will provide more in-depth information about these adults through the Adult Literacy Supplmental Assessment, as well as a fluency assessment.
Quantitative literacy scores have increased over the past decade, and I find this quite promising given the computer age we live in. As we write on page 18 of the report, respondents in 2003 were allowed to use calculators unlike in 1992 so that the assessment better reflects the demands of every day life. The fact that quantitative scores increased may mean that adults in this country are becoming familiar with the technical tools that they will need to succeed I the coming years. Yes I certainly am aware that calculators are not computers, and that there is a digital divide in the country; the next NAAL report will provide more detail about computer use and literacy levels.
Pardon me if I have gone off on a tangent and not completely addressed your question. I am very excited about the information that is available in NAAL, as well as the additional data that will be available through future reports. Please let me know if I can provide any additional clarifications.
Regards,
Mark
From: djrosen@comcast.net
Subject: Re: [Assessment] Has U.S. Literacy increased, declined or stayed the same?
Date: December 22, 2005 7:32:14 PM EST
Thanks, Mark. That affirms my understanding.
Now the only puzzle is that "Educational Attainment: 1992-2003" paragraph. I can't figure out how it fits with the overall conclusions. It would seem to be saying the opposite. Mark Schneider warned that this might be tough to understand at the session with the press on 12/15. I think he said about that last sentence that it took him six years to understand it. Can you shed any light on what the paragraph means and why it doesn't seem to support the overall findings?
David
David J. Rosen
djrosen@comcast.net
From: EJacobson@air.org
Subject: Re: [Assessment] Has U.S. Literacy increased, declined or stayed the same?
Date: December 23, 2005 1:17:23 PM EST
I have to admit that it is difficult for me to keep up with who said what, and whose conclusions are whose. Yet the results don't seem that puzzling to me. For example, these two statements:
1) There was no overall decline in the average score for prose literacy
2) The score for prose literacy dropped at every level of education
To me, the first statement precludes one from suggesting that prose literacy has declined. In fact, given the reports of the scores of African Americans (increases in all three), Asians and White (a mix of increases and unchanged), it would seem that things are not so bad. The reduced average scores of Hispanic adults is troubling, but I think the combination of a more inclusive study and changes in demographics might have more to do with it than any generalized decline in the nation's literacy (defined by this instrument).
I also think that more than likely the second statement does not provide evidence of a reduction in standards or of a reduced impact of higher education. To measure the impact of higher education by this instrument, you would have to track individuals' progress through the school system. That is, what score did you have before you went to school and what did you have after? This is also true at the societal level, if one is trying to measure the impact of college spending on general literacy levels. Again, since the study did not involve identifying what literacy level was required by colleges, inferences about changes in the nature of those requirements would not be supported by the data.
I think that framing it in terms of standards might confuse the analysis of the decline at the different educational levels. Imagine that there was a group of people in 1992 who had no college experience despite having literacy skills that could possible provide entry. They didn't have the high scores in literacy typically associated with college, but were more on the line between going and not going. Now that more members of this group are in college (the report notes more people falling in that category), it would seem to do two things. First, it would reduce the literacy levels of "those without college experience" because the higher scorers from that category have been removed. A good thing, since that means they are getting a chance to go to college. Second, the average scores of those with college experience might be reduced because this newly entering group is not beginning college with the same scores of those that had been there in the past. Again, since there is no preset score that indicates that one has met the standards for entering college, I don't think you can automatically say that standards for admission have declined.
The increased number of people who might show up as "high-education with lower scores" (and therefore reduce the average score of the high-education group) would only be a bad thing if once they were in college their literacy scores did not go up, but this report cannot address that because it is a not a longitudinal study - it is snapshot. It is possible that although their scores are lower compared to past averages, the college experience has increased their personal score quite a bit. Of course it is also possible that it didn't, but we don't know. Because of this ambiguity, I don't think one can jump to conclusions about lower standards or a lowered impact of higher education based on the results of this study.
Erik Jacobson
