AAACE-NLA NAAL2

From LiteracyTentWiki

NAAL Discussion on AAACE-NLA List, Page 2 of 2

From: gdemetrion@msn.com
Subject: Re: [AAACE-NLA] Media and the NAAL
Date: December 17, 2005 9:18:52 AM EST
To: aaace-nla@lists.literacytent.org

Whether or not we have the details nailed as to how to go about what Bob suggests, I believe the focus he points to is the right one. If that is the case perhaps the more reasoned response among field participants is a reflective absorption of what Bob is saying before expecting him (or anyone else) to come up with the solutions. Diagnosis is one thing and it's crucial that we do that well. The identification of the right problem situation is a perquisite to any proposed problem solutions. As a field we are pretty good at organized defensive measures to combat negative legislation. What we haven't been so good at is in the hard work of shaping public opinion on our own terms.

The reports on the new NAALS that I've seen have also been stressing the 11 million, which in effect becomes a "non-problem" in the public and policy sector. That's a far cry from the 90 million figure used previously, and in any event makes the observant outsider wonder how reliable any of these figures are.

George Demetrion

From: comingjo@gse.harvard.edu
Subject: Re: [AAACE-NLA] NAAL -- What % of the adult literacy population is the adult literacy population
Date: December 17, 2005 9:59:11 AM EST
To: aaace-nla@lists.literacytent.org

I haven't checked but I think the "high school" in this case is finished high school but did not go on to postsecondary education. Transition into some form of post secondary education or its equivalent in skilled job training is essential to compete in today's job market.

--On Thursday, December 15, 2005 3:11 PM -0500 Hal Beder <hbeder@rci.rutgers.edu> wrote:

If you look at the average scores [p14] for high school and GED they are virtually the same, 262-260 for prose. These scores fall at the high end of the basic scale. Thus if we want to get them up to secondary education proficiency we have to shoot for both below basic and basic. So the new NAAL makes the case even better than NALS which labeled the levels with numbers.

John Comings, Director
National Center for the Study of Adult Learning and Literacy
Harvard Graduate School of Education
7 Appian Way
Cambridge MA 02138
(617) 496-0516, voice
(617) 495-4811, fax
(617) 335-9839, mobile
john_comings@harvard.edu
http://ncsall.gse.harvard.edu

From: comingjo@gse.harvard.edu
Subject: RE: [AAACE-NLA] Media and the NAAL
Date: December 17, 2005 10:12:28 AM EST
To: aaace-nla@lists.literacytent.org

Bob Bickerton said something interesting at the MA news conference on the release of the NAAL and MA SAAL, which I paraphrase as: The field in MA didn't get immediate benefit from the release of NALS in 1994 but several years later the data was used to build a case for increased funding in MA that was successful. We should take this approach. We now have new data, and we should begin using it to make our own case, not worry about how it is pitched by politicians who have temporary jobs.

Intermediate and Proficient are the levels needed by workers, parents and citizens. Adults can succeed without these skills but success is more difficult and will only become more difficult in the future.

From: tsticht@znet.com
Subject: [AAACE-NLA] MAAL Method Issues
Date: December 17, 2005 5:57:50 PM EST
To: aaace-nla@lists.literacytent.org

December 17, 2005

Some Methodological Issues for the NAAL

Tom Sticht
International Consultant in Adult Education

The 2003 National Assessment of Adult Literacy (NAAL), with first results released on December 15th, 2005, is a follow-up to the 1992 National Adult Literacy Survey (NALS). Like the NALS, the NAAL is based upon many decisions about how to represent the literacy abilities of adults. In reading the report of the Key Concepts involved in the methodology used in the NAAL several decisions that the test developers took raised questions to me about the validity of the methods for representing the literacy abilities of adults. Following are some quotes from the Key Concepts report and my comments about the concerns the report raised for me.

Quote: "Like other adults, NAAL participants bring to literacy tasks a full range of backgrounds, experiences, and skill levels. Like real-life tasks, NAAL tasks vary with respect to the difficulty of the materials used as well as the complexity of the actions to be performed. However, in order to be fair to all participants, none of the tasks require specialized background knowledge, and all of them were reviewed for bias against particular groups." [p.3, Key Concepts]

Comment: In this quote I see two problems with the NAAL methodology for assessing adult’s literacy abilities. First, the extent to which the so-called "real world" tasks are actually representative of tasks each of the respondents are familiar with is not documented. This raises a question of the "ecological validity" of the tasks, especially when the materials being used and the tasks being called for are used with adults with geological and cultural differences as far apart as Native Americans on remote, rural enclaves as well as with residents of Manhattan in New York Cty.

The methodology as described also violates one of the most important findings about reading that much research has established: ones specialized background knowledge about things read has a large effect on the person’s comprehension and ability to perform tasks with the material read. A hallmark of adulthood is that following formal schooling, adults go on to specialize in areas related to their work and out-of-school activities, such as participating in the governance of a social organization, engaging extensively in hobbies and so forth. Hence to insist on tasks for which adults lack "specialized background knowledge" violates a major aspect of the development of literacy in adulthood, that is, the development of specialized knowledge, and can lead to greatly underestimating adults’ abilities as readers in their particular life contexts. This would tend to be more and more important the longer adults have been out of formal schooling.

Finally, the methodology of applying tasks in which the complexity of actions called for, coupled with the failure to honor adults’ contextual knowledge, seems likely to account to a significant extent for the sudden drop in performance of adults over the age of 50 years. Research has shown that the NALS/NAALS-type tasks are positively correlated with working memory, and decades of psychological and geriatrics research has demonstrated that the older adults get the less efficient their working memories become. Hence it is questionable whether the NAAL possesses ecological validity across the age range, as well as across "real world" tasks, geological, and sub-cultural contexts. ….. Quote: "Each performance level represents a continuum of abilities Although certain tasks can be characterized as typical of each performance level (as shown in table 2, on the previous page), it is important to remember that the tasks at each level extend across a certain range of difficulty and therefore require a continuum of abilities. For example, the new Basic level of document literacy encompasses scores ranging from 205 to 249. Adults with a score of 205 (the lowest score included in the level) have a 67 percent rate of success with the easiest task at that level, while adults with a score of 249 (the highest score included in the level) have a 67 percent rate of success with the most difficult task at that level. This means that adults at the high end of the Basic level have an even higher rate of success with some of the level’s easiest tasks. Moreover, these adults have a fairly high rate of success with some of the tasks at the low end of the Intermediate level, even though the rate is below 67 percent. "[p.17 Key Concepts] ….. Quote: Regardless of the specific criteria used to establish performance levels, adults at every level have some probability of performing any task correctly. Therefore, it is not correct to say that adults at a certain performance level are "not able to do" tasks at higher levels. These adults are, however, less likely to succeed with such tasks." [p.18 Key Concepts]

Comment: The two preceding quotes indicate that even though adults may be classified as being in one level, such as Below Basic, they may actually be able to do tasks at higher levels with some greater than zero probability. This raises the question as to just how we are to think about adult literacy competence. In a person’s mind, one probably does not have literacy ability segregated by "tasks" into "levels" and hence if they could do some fairly complex task, even though they could not do 67 out of a hundred of such tasks, they might think themselves fairly competent at reading. This, plus the considerations given above regarding "ecological validity" may account for why some 95 percent of adults in the earlier NALS thought they read well or very well. In turn, this may account why so few show up at basic skills programs for help with their reading. Maybe they actually are more competent than these assessments indicate.

A final comment: The NAAL provides three different scales: Prose, Document, and Quantitative literacy and each scale has different score ranges defined for the four different ability levels: Below Basic, Basic, Intermediate, and Proficient. This arrangement tells us little about how we should think about an individual’s overall literacy ability summed across all three of these scales. Conceivably someone could be just below the Intermediate level on the Prose scale but above the Intermediate level on the Document scale, and perhaps even above the Intermediate level on the Quantitative scale. How should this person’s overall literacy ability be thought about?

Thomas G. Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht@aznet.net

From: merleayres@hotmail.com
Subject: Re: [AAACE-NLA] NAAL -- What % of the adult literacy population...
Date: December 17, 2005 10:24:48 PM EST
To: aaace-nla@lists.literacytent.org

I wouldn't like to throw a wrench in all this statistics but, scores and goals are meaningless in adult literacy if the outcome is only the score results. Can a basic GED or diploma help the generation xers work a 8 or 9 hour day? Our work ethic is missing in the younger generation. College does not give an automatic work ethic. I always encouraged this in school that working on something hard was good. I don't see the stick-to-it- iveness today in the new work force.

Merle Ayres
412 8th st. North
Humboldt,Iowa 50548
Tel.1-515-332-4630
Fax 515-332-1738

From: tsticht@znet.com
Subject: [AAACE-NLA] NAAAA..L This Can't Be Right
Date: December 19, 2005 4:26:18 PM EST
To: aaace-nla@lists.literacytent.org

December 18, 2005

NAAAA….L, This Can’t Be right?

Tom Sticht
International Consultant in Adult Education

You studied hard in the primary, middle, and secondary grades of our nation’s K-12 school system, graduated with a high school diploma, applied for and passed tests to get into college, spent four more years studying and finally graduated with a college undergraduate degree. Then, perhaps, you went on to graduate school, took and passed some graduate level courses and even went on and obtained an advanced degree. Then the results of the 2003 National Assessment of Adult Literacy (NAAL) were released in December of 2005 and the federal government indicated that there was a 60 to 75 percent probability that you, -you with a high school diploma, 4-year college degree and perhaps even an MD, JD, or Ph.D - are not Proficient in literacy. What do you say? Naaaaa…, that can’t be right!

Yet that is what the developers of the NAAL have decided and reported. Among our nation’s most highly educated adults, those with graduate study and advanced degrees, only some 31 percent performed at the Proficient level on the Document literacy scale, meaning that some 70 percent of the most highly educated adults in the nation are not Proficient in Document literacy. Sixty-four percent are not Proficient in Quantitative literacy and some 60 percent are not proficient in Prose literacy. Can this be right?

Suppose, on the other hand, instead of having gotten through elementary, middle and secondary school you had goofed off, decided to drop out in the 11th grade and then, after a few years, re-thought the benefits of education a little deeper and went to an adult education program. You studied hard and finally passed the high school diploma equivalency exam, got your General Educational Development (GED) certificate and enrolled in a local community college. After three years of working at a job and going to college part time you finally get your Associate of Arts degree. With that degree, the chances are 80 to 85 out of hundred that you would be declared to be less than Proficient in literacy on the federal government’s NAAL tests of Prose, document, and Quantitative literacy. Can this be right?

It seems to me that these kinds of findings should raise questions about the validity of the NAAL for accurately characterizing the literacy abilities of adults. Is it reasonable to conclude that 60, 70 or even 85 percent of two- or four-year college graduates lack Proficiency in literacy, no matter how it is assessed?

How can unemployment for those in the labor force be less than 6 percent if 70 to 80 percent of the workforce lacks Proficiency in literacy? How can the United States have one of the highest rates of productivity in the world if 70 or 80 percent of adults lack Proficiency in literacy? How can the United States’ economy be the envy of the world, with foreign investment in the U. S. surpassing such investment in each of the other nations of the world, if 70 to 80 percent of adults lack Proficiency in literacy? Wouldn’t this cause investors to bolt to nations with more capable workforces?

Earlier in 2005 the National Center for Education Statistics (NCES) released the international Adult Literacy and Life Skills (ALL) report. This report used methods for assessing the extent to which adults aged 16 to 65 worked in jobs for which their literacy skills matched, exceeded, or were below the skills needed for their jobs. They reported that some 60 percent of adults possessed literacy skills that matched the literacy required for their jobs, while an additional 20 percent had literacy skills that actually surpassed the demands for such skills in their jobs. Only 20 percent were thought to be working in jobs for which their skills were actually deficient. In other words, some 80 percent of U.S. adults were proficient in the literacy abilities needed for their jobs. Once again, these data raise questions about the validity of the NAAL when it represents 70 to 80 percent of adults as below the Proficient level of literacy.

Setting aside for the moment the 5 percent of adults in the NAAL who could not take part in the testing, the NAAL report acknowledges that all adults in the four levels had some non-zero probability of performing literacy tasks above the level they were defined into. This means that it is not appropriate to refer to the levels as "basic" or "below basic" since this implies that the tasks included within those levels are either "basic" or "below basic" when an examination of the released tasks indicates clearly that there is nothing "basic" upon which other, higher level, tasks build. That is, the higher level tasks are not "based on" the lower level tasks. Further, the NAAL report indicates that adults in one level could be expected to perform tasks at higher levels, hence the person cannot be said to be at a "basic" level because the person is able to perform above the "basic’ level with some non-zero probability.

Perhaps the NAAL would have characterized the literacy ability of the American workforce better if instead of calling the four literacy levels of adults Below Basic, Basic, Intermediate, and Proficient, it had acknowledged that adults at all four levels were proficient to differing degrees and named the four literacy levels Below Average, Average, Advanced, and Superior Proficiency. Labels do make a difference in people’s understanding of things, including their understanding of the literacy abilities of America’s adults.

In labeling 60 to 80 percent of our nation’s most highly educated adults as less than Proficient, the NAAL defames the skills of one of the most productive workforces on earth. It also deflects attention away from those adults whose literacy abilities are truly incapacitating and who are in need of serious educational services by the Adult Education and Literacy System (AELS) of the United States. The last time the NALS came out in 1993 and declared almost half of American adults functionally illiterate, cartoonists had a field day poking fun at this ridiculous conclusion, and for each of the next three years the federal funds for adult literacy education went down.

Then, under the present Bush administration, the U.S. Department of Education continued to disseminate the idea that some 90 million adults, almost half the adult population, were below the literacy skill levels needed to cope in contemporary society, Despite this stated drastic national crisis of adult functional illiteracy, the present Bush administration never asked for increased funds for adult literacy education, and in the most recent Bush request for adult literacy education in fiscal year 2006, the administration actually asked for the budget to be cut by some $375 million, leaving only around $200 million for the nation to combat this adult literacy crisis. This suggests to me that the present administration does not actually believe its own survey data for 1993, and I have read nothing about the administration revising its fiscal year 2006 budget request for adult literacy education in the face of its recent 2003 survey in which it has declared as many as 80 percent of adults as below Proficient in literacy. Does this seem right?

Naaaaa…this can’t be right!

Thomas G. Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht@aznet.net

From: EJacobson@air.org
Subject: [AAACE-NLA] NAAL
Date: December 21, 2005 11:17:49 AM EST
To: aaace-nla@lists.literacytent.org

Two discussions in the media of the NAAL results:

1) There is a brief article in Education Week Online (www.edweek.org)

2) Today at 9 AM P.S.T. there will be a discussion of adult literacy on KQED (public radio in San Francisco). You can listen live over the internet, and the program should be available in podcast form sometime after the fact.

Erik Jacobson

From: tsticht@znet.com
Subject: [AAACE-NLA] FW: Literacy of College Graduates Is on Decline
Date: December 27, 2005 2:25:19 PM EST
To: aaace-nla@lists.literacytent.org

Aaace-NLA Colleagues: If the NAAL is correct then some 60 to 75 percent of college graduates, including graduate students and advanced graduate degree holders are not Proficient in literacy. This being the case, it could mean that as many as 60 to 75 out of a 100 newspaper journalists are not Proficient at literacy. When I read the newspaper articles that have been written about the NAAL, without exception, not one single writer has raised any questions about the validity of the assessment.

I am also distressed to note that, aside from the messages I have posted, I have read no messages on the aaace-nla list questioning the validity of the NAAL (or the earlier NALS). This list is for adult literacy educators who are supposed to be interested in helping their students "think critically" yet I have read no critical thinking about the NAAL. Even the researchers who have posted messages have not raised any critical questions about the validity of the NAAL. But perhaps 60 to 75 percent of college educated adult literacy educators and researchers are not Proficient in literacy and can not understand the NAAL report.

In an earlier message (NAAAA..L This Can't Be Right) I raised questions about the NAAL and its representation of the literacy of college grads and said, "Perhaps the NAAL would have characterized the literacy ability of the American workforce better if instead of calling the four literacy levels of adults Below Basic, Basic, Intermediate, and Proficient, it had acknowledged that adults at all four levels were proficient to differing degrees and named the four literacy levels Below Average, Average, Advanced, and Superior Proficiency. Labels do make a difference in people’s understanding of things, including their understanding of the literacy abilities of America’s adults." Notice that just by renaming the levels on the NAAL the discussion of the proficiency of college grads would have changed dramatically.

Regarding the drop in Proficiency from 1992 to 2003, it is important to realize that there are differences in the demographics of the samples in 1992 and 2003, and the test items are different too. Finally, even if everything in the two assessments were comparable, it is not possible to discern a trend with just two data points. The changes from one assessment to the next may just be random fluctuations.

The believability of the NAAL results would be enhanced had the U. S. Department of Education announced it was going to reinstate the 65 percent cut in funding that it had made in the adult education budget for 2006. But no funding was forthcoming in 1993 when the NALS results lead to newspaper articles saying that half the U. S. population of adults were functionally illiterate, and I don’t think we should count on seeing very much of a response from the federal government with the announcement of the results of the NAAL. So far all I have read about is that the NAAL adds research support to the President’s high school initiative and the U. S. Department of Education and other government agencies are going to coordinate their adult education and training better. Now that’s a big help!

Tom Sticht

From: amlandoll@yahoo.com
Subject: Re: [AAACE-NLA] FW: Literacy of College Graduates Is on Decline
Date: December 27, 2005 4:48:45 PM EST
To: aaace-nla@lists.literacytent.org

Well, how does the NAAL compare to the SAT, ACT, GRE, PRAXIS, etc.? You are right, we should discuss it. And how are students able to do ok on the other tests?

You are also correct about the fed dept of education. I don't think the folks from Texas will be satisfied until every public school student of age 4-18 is tested annually. (Using useless criterion-referenced tests of course...

Anita
www.learntoreadnow.com

From: DJRosen@theworld.com
Subject: [AAACE-NLA] NAAL and the Media: The Story We Should Be Telling
Date: December 27, 2005 7:27:08 PM EST
To: aaace-nla@lists.literacytent.org

AAACE-NLA Colleagues,

I wonder what you think of the articles you have seen about the NAAL results.

I don't like the ones I have seen. Most are hand-wringers. They bemoan the loss of standards in higher education, where according to the NAAL results, relatively few are proficient in literacy. They focus on the 13% of Americans at the "below basic" level. They despair at how immigrants are the problem. (They aren't.) And so far, there has been almost no analysis, no synthesis, not even much hypothesis.

In the past two weeks I have seen only one sensible article about adult literacy education, and it's not about the NAAL. It's about our smallest state and how an adult learner and his family have benefited from adult classes. It's about how he made an investment in his education, and how an adult education program made an investment in him, and the return on that investment for him, his family and Rhode Island. It's about a Governor who believes that this kind of investment is essential for the Rhode Island economy. It's about a new state adult education director who is determined to build a system of adult education. And it's about the importance of a public and private investment to make that happen. It's about an immigrant who came to this country with a sixth grade education, who was determined to use adult literacy education classes to get the skills he needed to advance, how he became a worker, a taxpayer, an American citizen, and recently a home buyer. It's about how, through their words and actions, he and his wife show their children how important education is.

This is the kind of article that I want to see in every newspaper in the country. We have had enough articles about the funding crisis, enough articles deploring the decline of American literacy. We need articles about how investing -- this appears to be the right word -- in adult literacy and English language learning pays off, that it's a good investment. As a colleague recently pointed out to me, this is the investment that makes other social investments (welfare reform, health education, job skills training, early childhood education, children's literacy, and others) succeed.

The article, in the Providence Journal, will be found at:

http://www.projo.com/education/content/projo_20051227_adulted.da3f44b.html

Please let us know when you get an article, editorial, op ed, or letter to the editor -- one that shows how adult learners, adult learner leaders, practitioners and public policy leaders are investing in adult education -- in _your_ local newspaper.

David J. Rosen
Adult Literacy Advocate
DJRosen@theworld.com

From: merleayres@hotmail.com
Subject: Re: [AAACE-NLA] FW: Literacy of College Graduates Is on Decline
Date: December 27, 2005 7:10:13 PM EST
To: aaace-nla@lists.literacytent.org

Anita: Does this mean the Iowa Test of Basic Skills is not a good benchmark either. I remember that through some conversation with other teachers that the test was gone over to make the schools look good and " no child left behind act" was met and the principal was not reprimanded. I simply called it cheating and got scolded. I was taught by university profs that tests were to find strengths and weaknesses and maybe that has changed over the years. If its used to help kids then tests are ok. but not as a end all to look good for gov agencies and parents who end up using scores for watercooler conversations. Im a retired teacher but may not be up to par and all of this.

Merle Ayres
412 8th st. North
Humboldt,Iowa 50548
Tel.1-515-332-4630
Fax 515-332-1738

Return to NAAL Discussion on AAACE-NLA List Pange 1 of 2 AAACE-NLA_NAAL
Go back to NAAL_Discussion page

Retrieved from "http://wiki.literacytent.org/index.php/AAACE-NLA_NAAL2"

AAACE-NLA NAAL2

From LiteracyTentWiki

Views

Personal tools

Navigation

Search

Toolbox