EbaeAssessmentInstruments

From LiteracyTentWiki

Assessment Instruments Comments

• Tools for measuring outcomes should include CASAS [Message 8, Part One]

• Scores on achievement tests won’t capture the breadth of what practitioners and learners do in programs. “Victoria Purcell-Gates's work with NCSALL and Sheehan-Holt and Smith (Reading Research Quarterly, 2000 Vol. 35) offer an example of conflicting conclusions on what counts and how to count it. I'd like to see preliminary steps added that include quantitative, exploratory research to address these epistemological issues.” [Message 12, Part One]

• Some tools for measuring student gain are not good and will not align with the curriculum of a program. “So how would that affect the Baseline Program Models?” [Message 14, Part One]

• More interesting models of measuring gain (EFF and family literacy) are not fully developed, do not have neat measurement packages like 'you are an SPL 4 according to the BEST test'. How can we develop a new (evidence-based) system when (measurement) components of ABE are flawed, or those flaws are being worked on? Lack of a cohesive system is a problem, but if we try to build a new one, “we'll get really hung up on fundamental components that need to be very different but aren't. Can both efforts happen simultaneously?” Have the authors considered some timeframes for such an endeavor? [Message 14, Part One]

• It would be important to use all assessments that are approved for NRS reporting if the results of this Evidence-Based Adult Education System are to reflect the real world of adult education in the field. [Message 17, Part One]

• We should be careful not to “teach to the test” if the test is not aligned to students’ goals [Message 25, Part One]

• It is important to measure literacy behaviors like “using an aspect of the health care system, helping children with homework, ordering a prescription online, understanding a prescription, voting, using the internet, going into a chat room, transitioning to college, getting a job, eating healthier, donating blood, moving to the Shelter for Battered Women, applying for political asylum.” [Message 26, Part One]

• “How can I measure these? The only way I can think of, is by documenting them if and when they are reported. However, different students may engage in different behaviors. Also, some may not report any, but that does not mean that they are not engaging in them.” How can programs do this and document it? [Message 26, Part One]

[Reply to Message 26, Part One] • “What I think we can do is to help adults develop and apply the knowledge and skills they need in order to effectively engage in one or some or all of the literacy behaviors you list. Once we identify what those adults need to know and be able to do in order to meet their real-life purposes for learning, like the ones you mention, we can teach this ‘important content’ in contexts that give learners opportunities to apply and practice what they are learning in meaningful activities (reading with understanding some information about diabetes treatment options in order to prepare for a visit to the doctor? Speaking in a simulated doctor visit so the doctor can understand one's concerns about getting the best treatment?). In fact, the evidence from cognitive research into how people learn and develop expertise tells us that teaching in this way makes it more likely that learners will be able to transfer what they learn into real-life situations, to meet real-life needs. And here's the really good news: it is possible, and in fact important, to measure this kind of contextualized knowledge/skill development and application in an assessment system that includes performance-based assessments. These assessments allow us to collect -- in sufficiently standardized ways -- evidence that learners have mastered the knowledge and skills needed to successfully perform some meaningful task. So in the examples above, we would be measuring ability to Read with Understanding, or to Speak so Others can Understand, in the context of real-life health-related activities. The focus for instruction and assessment is the same important content, and the evidence of learning speaks to adult purposes….with this kind of evidence in hand, I think we have a sound basis for linking learner achievement to the[se] kinds of important outcomes.” [Message 27, Part One]

David Rosen, the discussion moderator, asked: • Do we need better assessment instruments as part of an evidence-based adult education system? Is the current state of the art -- the range and quality of the assessments -- an obstacle to our doing good research? If so, what kinds of new instruments or improvements in current instruments are needed? [Message 21, Part One]

• Many forms of assessment are used in nursing education and certification, including performance-based assessments.

“Nursing assistant students take a written test AND must perform certain skills such as bed-making or measurement of vital signs. These assessments are done through observation using checklists that are standardized to assure validity, reliability. Performance-based checklists are a pragmatic way to accomplish standardized assessment through observation of applied knowledge. They allow for context-specific application of learning. Another advantage of observational assessments such as checklists is that they do not place unrealistic demands in terms of time at the program level - a burden that many practitioners may find unworkable. This type of testing requires training to assure that assessments are uniformly administered. Testing of nursing assistants is, in fact, administered independent of their educational programs, further assuring that bias does not enter into the assessment process.” [Message 31, Part One]

• Graduates should demonstrate high achievement scores on standardized tests like the TABE and CASAS, but should also demonstrate that they are good readers. These are not the same. [Message 34, Part One]

• “ 'what works' is not often measurable by the tools we currently have or are mandated to use…. The adult education standardized assessments we have used in our programs do not measure what students are learning; they oftentimes just add frustration to the student/teacher relationship; and they lead to program decisions that are responding to a test rather than the education of whole people with big lives.” [Message 44, Part One]

• “Someone brought up the notion of using observational tools and more classroom-based types of assessment. Without the rigor of structured tools and common ways of scoring it would be difficult to use these types of measures in an evaluation study. As research sites are selected and begin implementing program models it will be essential that common tools are used to assess the variety of outcomes that define student success. How else can we combine data to gather the large number of students necessary to make valid statements about student achievement?” [Message 57, part One]