Authentic Accountability Tools
From LiteracyTentWiki
Return to Accountability
From: rrunyon@mail.unomaha.edu
Subject: [NIFL-AALPD:1867] Re: accountability (from the ed. and learning thread)
Date: Wednesday, January 05, 2005 11:44 AM
Colleagues I think the recent contributions by Eileen and Sally have hit upon essential elements of assessment and accountability reporting. We need guidelines, procedures, checklists, etc. for naturalistic or ethnographic reporting of learning outcomes from the learners' perspectives. Last fall, I conducted interviews with 15 adult learners and recorded some fascinating interview data. These were self-reports of personal experiences and program benefits as perceived by our learners. I concur with the emphasis placed by Sally on the learner's goal/objectives framework. I wonder if there are any social scientists out there who can give us guidance or published references to formal procedures (possibly checklists and interview guides) that would assist us in colleting this sort of data from our learners?
Bob Runyon
Literacy Center for the Midlands (www.midlandsliteracy.org)
From: rkmcknight@comcast.net
Subject: [NIFL-AALPD:1868] Re: accountability (from the ed. and learning thread)
Date: Wednesday, January 05, 2005 12:06 PM
Hi Bob,
I agree that Sally and Eileen have shared some excellent insights. To answer your question below, the field of nursing has used checklists and varied types of assessments for some time now. Since so much of nursing involves psychomotor skills, these types of assessments are necessary.
For example, part of the certification process for CNAs (Certified Nursing Assistants) is a demonstration of skills - performance of nursing skills such as reading a thermometer and the like. Evaluators are trained to evaluate candidates and ensure reliability (consistency of evaluation processes) prior to administering the exams.
Nursing also created the first CAT (Computerized Adaptive Test) which is used for nursing licensure. I'm sure there are similar examples in other fields. You can view some info about this at the National League for Nursing website:
http://www.nln.org/testprods/index.htm
While these process are not perfect, trememdous effort and resources have been invested in developing and continually refining them.
Cheers!
Roberta McKnight
Healthcare Multimedia Design
http://www.hcmmdesign.net
From: AWilder106@aol.com
Subject: [NIFL-AALPD:1869] Re: accountability (from the ed. and learning thread)
Date: Wednesday, January 05, 2005 12:13 PM
Bob,
There are two NCSALL publications that might be useful to you, #13 and #2. Beth Bingham who worked on #13 can be reached at the University of Tennessee. Erik Jacobson who worked on #2 can be reached at: Jacobson@air.org.
Both Beth and Erik are experienced social scientists, and both support (through their work) the inclusion of student perspectives in evaluations.
If someone invited them, I think they might both participate in the discussion.
Andrea
From: marie.cora@hotspurpartners.com
Subject: [NIFL-AALPD:1870] Re: accountability (the learners' perspectives)
Date: Thursday, 6 January 2005 6:33 a.m.
Yes, the Bingman papers came immediately to mind:
I've come a long way: learner-identified outcomes of participation in adult literacy programs http://gseweb.harvard.edu/~ncsall/research/report13.pdf The Learner Identified Outcomes study brings learners' perspectives to the ongoing research conversation on outcomes of participation in adult literacy education.
Outcomes of participation in adult basic education: the importance of learners' perspective http://gseweb.harvard.edu/~ncsall/research/op_bingman.pdf This paper suggests that learners have a different perspective on performance than the authors of the Workforce Investment Act.
And I know this isn't from this century (wow, that sounds funny!), but Ways With Words by Shirley Brice Heath cannot be overlooked when thinking about this from the learners' perspective.
About a year ago on the NIFL assessment list, there was a discussion of capturing advancement and progress in students that is not literacy-based, but rather the social pieces - the archives have those posts and many people listed experiences, tools, and processes that they found helpful. You can check out the archives at http://www.nifl.gov/lincs/discussions/nifl-assessment/assessment.html or if people feel it's necessary, I can dig up some of them to forward.
marie cora
Moderator, NIFL Assessment Discussion List, and Coordinator/Developer LINCS Assessment Special Collection at http://literacy.kent.edu/Midwest/assessment/
From: eileeneckert@hotmail.com
Subject: [NIFL-AALPD:1871] assessment, from accountability (from the ed. and learning thread)
Date: Wednesday, January 05, 2005 12:54 PM
Bob, I'd be interested in hearing your interview findings. Could you summarize some themes you found, or powerful examples?
There are many techniques in Angelo and Cross' book, "Classroom Assessment Techniques" that can be easily adapted for ABE/ESOL classes and programs of many types.
I've been exploring development and use of rubrics to make aspects of skilled performance explicit--several advantages to rubrics are: 1. They can be developed for as small or large a group as needed; they don't have to be applicable across program types but can be tailored to the individual class' needs if necessary (or they can be applied more broadly). 2. The process of creating and developing them makes the teacher(s) who are doing it think about exactly what's important, what different levels of performance look like, etc. Creating them is an ongoing learning experience for teachers (PD with immediate applicability) 3. When shared with students (as they should be!) they can be a tool to faciliatate teacher-student dialogue about learning and progress, and to focus attention on things to work on--they are a learning tool as well as assessment 4. Students who use them for self-assessment as part of an overall assessment strategy develop greater metacognitive skills and capacity for self-direction; they don't have to always wait for teacher feedback and direction 5. A rubric that's done well can be used for assessment of a broad range of tasks and situations that require use of the skill, so it can be used to assess performance on <real> real-life tasks, not just simulations carried out in the classroom or lab.
When rubrics are used to assess a range of performances of real importance to learners, then the assessments and documents (where there are written documents) can be collected in portfolios and be the subject of reflection, self-evaluation by students, and dialogue with instructors. They are tangible evidence of the outcomes of learning. Students, teachers, and anyone else can see how effective instruction and learning are--accountability!
When teachers and program leaders understand assessment as more than testing, and learn to create their own assessment tools, they don't have to rely on products created by others that may not be just right for them, or at least they can be more effective in adapting and using them. It's along the lines of giving someone a fish vs. teaching them how to fish.
I do understand Sally's reminder that adult ed. is underfunded, but there are already activities going on, and they should be continually moving toward greater effectiveness and responsiveness to learners' needs. Also, until we articulate our own vision, values, and practices consistent with those, we're always going to be in a position of reacting to and responding to, and defending against the imposition of someone else's agenda. Sally's in the position of dealing with the reality that is, and I respect that and think she's in a place where there's a community of practitioners making the best of difficult circumstances, and doing good work in those circumstances. I have decided, for now, to advocate primarily from outside the system and not experience as much pressure as some experience to not make waves, and I think both positions are needed.
OK. I'll stop now, as I've been dominating this thread for days and it's time to shut up for a while. I may have to tie my hands behind my back to keep them from the keyboard.
Eileen
From: sreid@workbase.org.nz
Subject: [NIFL-AALPD:1872] Re: accountability (the learners' perspectives)
Date: Wednesday, January 05, 2005 03:08 PM
Good morning I have been following this discussion and am thankful that at this stage we do not have standardised tests in New Zealand although I am sure there are others in the field here who would welcome them. Thank you Marie for your succinct summary of the differences between assessment, evaluation and accountability. Although these reports from NIACE( National Institute of Adult and Community Education) in the UK are not specifically literacy focussed they are useful because they acknowledge confidence of learners to do or try to do something as an acceptable outcome.
http://www.niace.org.uk/funds/aclf/Catching-Confidence-Final-Report.pdf
http://www.niace.org.uk/funds/aclf/Catching-Confidence-Summary.pdf
The main report is 75 pages long so the summary might be a good place to start
Susan Reid Workbase the New Zealand Centre for Workforce Literacy Development
www.workbase.org.nz
www.nzliteracyportal.org.nz
From:bingman@utk.edu
Subject:Accountability catch-up
Date:Monday, January 17, 2005 03:27 PM
This is a somewhat belated response to a thread from a week or so ago. I’m not sure I am a social scientist, but I have thought about the questions you all have been exploring in the accountability thread ever since I decided as a county program coordinator (in the previous century) that our program should use a standardized test to measure the effectiveness of our program and then saw the effects on both students and teachers when some students did worse on a post-test. The more recent work I did with NCSALL included the papers Marie referred to and also some work with local programs reported in NCSALL Report # 20. In that action research project three programs developed ways to measure or document their progress and proposed to their states that these be used for accountability reporting (this was early WIA). The states were interested, but did not in the end agree to use these approaches. We did however develop a guide for developing local outcomes documentation that can help programs be accountable to their own constituents even if the documentation is not acceptable to their primary funder. The guide is called How Are We Doing? and can be downloaded from NCSALL at http://gseweb.harvard.edu/~ncsall/teach.html
I do not have or know of a checklist or tool for using student interviews for accountability purposes that will meet federal guidelines. However, the work that EFF has done provides tools that could be used both for accountability as defined by our current system (learning gains by level) and accountability re. learners’ and programs’ varied goals. The EFF Assessment Resource Collection found at http://eff.cls.utk.edu/ describes the performance continua and level descriptors for the EFF Standards. Performance measures based on the continua can be used to determine progress (or level gain) on a variety of literacy skills. Look at the Guides and the ARC Library for the level descriptors for eleven of the 16 EFF Standards and for information on how they were developed and how they can be used to design as well as assess learning.
Beth Bingman
Return to Accountability
