Accountability Full Discussion
From LiteracyTentWiki
Return to Accountability
From: eileeneckert@hotmail.com
Subject: [NIFL-AALPD:1852] Re: education and learning
Date: Monday, January 03, 2005 07:38 PM
Andrea, you said it sounds like I was getting at organizational learning when I wrote about examining finance, policy, administration, etc. in light of the extent to which they are organized around and support learning as a goal and as a process. A couple of comments.
"Organizational learning" as a term becomes a kind of shorthand for a whole set of assumptions, theories, and practices. This list regularly features discussions in which someone uses a word or phrase, assuming it will mean the same thing to all the readers as it does to the writer, only to find him/herself caught in a storm of controversy because that innocuous bit of shorthand means something different to everyone and everyone assumes their meaning is "the" meaning--I exaggerate, but probably not by much. It's certainly happened to me enough, and probably will again with this message.
That said, many of the individuals and teams in an organization, or different organizations in a system, can focus their efforts on a common goal or issue and they might demonstrate "organizational learning" around that issue, but that content (the goal or issue) doesn't have to be learning. I'm talking about all the different people and departments and areas of organizations (and the systems of which they are a part) focusing on learning--organizational learning about learning--META-learning.
Let me give you an example. There are others, but this one comes to mind. In the last several years, especially since the Workforce Investment Act of 1998, but it started before that, there has been a lot of effort--and, I think--organizational learning, focused on the concept of accountability, as defined by the government and developers of the NRS and research and training activities. However, this effort around accountability is not an effort around learning; in fact, I'd argue it's detrimental to a focus on learning learning by siphoning money, time, and energy away from learning activities and into testing and record-keeping activities. A whole lot of smart and practical program leaders have become very savvy (demonstrated organizational learning) about playing the accountability game, but are they and their programs correspondingly better than they used to be at focusing on, fostering and supporting learning? Has that organizational learning led to more learning for students? Has it made instructors better at teaching? Has it made program directors more expert at setting up the conditions to support learning on the part of students, teachers, and themselves?
So no, I'm not just getting at organizational learning, I'm talking about making learning the focus of all an organization's, and a system's, efforts. If we had that at the top, if WIA and the NRS were about learning, it would make it much easier at every other level to focus on learning; but that's still not an excuse not to focus on learning.
Eileen
From: AWilder106@aol.com
Subject: [NIFL-AALPD:1854] Re: education and learning
Date: Monday, January 03, 2005 08:43 PM
Eileen,
I'm still not sure if you are talking about organizatinal learning or student learning--if student learning then I am with you all the way in your observation. Or organizational learning bent to student learning. I think that is the point of what you are saying? If so, yes,the student is being given short shrift.
Andrea
From: eileeneckert@hotmail.com
Subject: [NIFL-AALPD:1855] Re: education and learning
Date: Monday, January 03, 2005 10:07 PM
Andrea, I'm talking about learning as the <content or focus> of effort and organizational learning--certainly student learning; that's what adult basic skills, ESOL, all adult ed. programs are there for. But we shouldn't ignore the power of making our activities supportive of staff, teachers, and administrators learning, individually and together, <about learning>--that also has an effect on student learning (see the West Ed report, Teachers Who Learn, Kids Who Achieve). Learning and meta-learning.
Another example: it's kind of a cliche to say that feedback facilitates learning, but Kluger and DeNisi found that the effects of feedback can vary widely. Kluger and DeNisi’s (1996) meta-analysis examined the effects of feedback interventions (FIs), that is, “actions taken by (an) external agent(s) to provide information regarding some aspect(s) of one’s task performance” (p.255). Kluger and DeNisi found that on average, feedback had a moderate positive effect on performance (ESsm = .41). They also reported that over 38% of calculated effect sizes were negative. So in almost 40% of studies, people did worse with feedback than without it! I'm quoting my own dissertation here, which went on to review research on how feedback affects the development of proficiency.
It helps to know something about what research shows of the effects of feedback on learning, and what characteristics of feedback support learning (and what kind has a negative effect). It also helps to look at other disciplines that address learning and performance, as well as education, to find out what the research shows. And it helps to apply it, whether to student learning or to supervision of teachers or even to performance/compliance reviews of programs (there's an interesting study by Pajak and Glickman on how teachers received feedback with slight changes in wording). And to try it out, and to have time to reflect and evaluate how well it works when you try it, and revise based on that reflection...
I didn't pick that example because everyone should know about feedback--it's just one example of how the research is much more informative than the best practice of giving feedback on student work.
Eileen
P.S. Here are the references if you want to look them up: Kluger, A. N. & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254-284.
Pajak, E. & Glickman, C. D. (1989). Informational and controlling language in simulated supervisory conferences. American Educational Research Journal, 26(1), 93-106.
From: rkmcknight@comcast.net
Subject: [NIFL-AALPD:1856] Re: education and learning
Date: Monday, January 03, 2005 10:53 PM
Good points, Eileen!
Thanks for raising these questions. I will add some more:
You are right on target asking if learning is the focus of our efforts. On a broad scale, ARE publicly funded organizations accountable to educators who work in these systems and to the public? Who monitors publicly funded programs? What results do they report? Do they even bother, or is it like insider trading - only known to a privileged few? Do the results demonstrate learning? Are they concerned with validity?
Perhaps its time to take a more proactive position - to ask for an accounting of publicly funded initiatives. Like the bills before congress, are we reading the annual reports? Where are the annual reports? Perhaps we need something like a "three day rule"? Here in Virginia, as in many states, technology and other initiatives have evaded the scrutiny of accountability. Though everyone is aware of this, nobody is willing to put their head on the chopping block by asking questions. As the saying goes, "never bite the hand that feeds you", and asking questions can have a negative effect on job security.
So I ask you now, is this how learning works? Does intimidation have a place in learning? Is it possible for learning to take place when questions are repressed? As surely as these questions need to be asked, publicly funded organizations need to be accounable, particularly those that are outsourced. An illustrative example can be found amid the clamor for charter schools. Though few results have been reported, and the integrity of some reports have come into question, a steady flow of funding and legislation continues to promote vouchers.
Who is minding the store? When funding is allocated, what results are achieved? Has public accountability fallen by the wayside? Must we wait for a crisis before we ask for public accountability? With so much banter about evidence, what evidence demonstates organizational accountability - organizational learning? Are we afraid to ask these questions? Whatever happened to learning organizations? This concept, so widely described, has not been part of my recent experience - how 'bout you? Are publicly funded organizations, learning organizations? Should they be? And, whatever happened to integrity - has it become an outmoded concept, too?
With hopes for accountablility in 2005,
Roberta McKnight Healthcare Multimedia Design http://www.hcmmdesign.net
From: rboone@vineland.org
Subject: [NIFL-AALPD:1857] Re: education and learning
Date: Tuesday, January 04, 2005 10:46 AM
>>Has public accountability fallen by the wayside? <<
Working in Adult Education an enormous amount of time is spent generating reports. The problem being, the amount of time actually "teaching" is inversely proportional to the volume of reports that must be submitted to various agencies each month. But in the name of accountability, we persevere. It seems that the reports are an end to themselves. It feels like no one even reads the reports. This is not just a problem with Adult Education, but with a ubiquitous information overload in the "computer age."
My question would be, is the accountability accountable?
What is the ultimate goal of accountability? Is it to help the student or to
build job security into an administrative layer that can demonstrate the
ability to quantify results?
richard boone
From: James.Parker@ed.gov
Subject: [NIFL-AALPD:1858] Re: education and learning
Date: Tuesday, January 04, 2005 10:54 AM
Richard, an excellent question. I think Public Accountability should travel both ways (up and down).
Parker, James
From: eileeneckert@hotmail.com
Subject: [NIFL-AALPD:1859] accountability (from the ed. and learning thread)
Date: Tuesday, January 04, 2005 01:25 PM
Roberta, Richard, James, and others, I think Roberta articulated many of my feelings and concerns, and I hope others will discuss the issues she raised (I know I'll come back to them). To pull one thread from the many in the messages, though, can we consider what we mean by accountability? It's another of those terms that has been so overused (in general, not so much on the list) and so contorted that it has lost much of its meaning. Instead of guiding and ensuring progress toward better meeting the needs of learners, it's evolved into a jumping-through-hoops set of empty formalities that often takes educators' attention away from a focus on learning. Several years ago I began trying to formulate my ideas on the subject, stemming from the thought that there's got to be a better way to structure accountability so that it is meaningful and consistent with improving learning and instruction to meet learners' needs. The result was an article with my colleague Sandy Bell of the UConn Adult Learning Program that will be published in Adult Basic Education, and I'm going to copy an excerpt here:
"Instead of outside interventions and requirements that structure accountability as a matter of responding to external standards, efforts in keeping with a complex systems view of literacy education would produce generative activity on the part of individuals and programs; that is, activity that contributes to the 'emergence of a coherent collective identity' (Davis & Sumara, 2001, p. 88). This activity weaves together learning and the documentation, assessment, and evaluation of that learning—at the individual, class, and program levels. Such activity would produce what we call 'authentic accountability.' "
We go on to describe how that could be done, and from the conclusion:
"We believe that authentic accountability efforts ... are congruent with the work of improving instruction and student learning in ways that conventional accountability efforts ... are not. However, authentic accountability activities cannot be imposed in addition to current conventional accountability requirements. Each approach requires different actions. Policymakers should not expect teachers and programs to fit the actions required for authentic accountability and those required for conventional accountability into their already overstretched budgets of time, energy, and money. Policymakers need to choose their paradigm. Educators need to analyze the factors influencing policymakers. With that analysis in mind, they can plan how best to advocate for use of an authentic accountability framework. Though we have described and explained authentic accountability from a complex systems and naturalistic research perspective, it is congruent with a Freireian approach to literacy as praxis, embodying both reflection and action (Freire, 1970/2000). It is profoundly political, and therefore, to some, dangerous. With an informed understanding of authentic accountability and the factors supporting and hindering its being understood, adopted, and implemented, educators can advocate for the approach to accountability that will meet their learners’ needs."
I think that rather than extending meaningless accountability busywork upwards (I don't think anyone was advocating for that, but I add it just to be clear), we need to have "authentic accountability" at every level.
Roberta, Richard, James, others, what would it "look like" if the agencies and organizations "above" you in the hierarchy were accountable to you and to learners? How would you know?
Thanks for the thought-provoking discussion.
Eileen
From: rkmcknight@comcast.net
Subject: [NIFL-AALPD:1860] Re: accountability (from the ed. and learning thread)
Date:Tuesday, January 04, 2005 03:02 PM
Yes Eileen,
We must clarify the distinction between assessment and accountability. Both should be free of bias.
Accountability is devised to ensure that funding is not wasted or
controlled by political agendas. This is why we have Sunshine Laws that
allow access to meeting minutes - to ensure that things are on the up
and up.
Accountability also ensures that political agendas do not usurp the
integrity of education's mission to serve the needs of learners and educators.
Assessment and accountability are not synonymous terms - they are quite distinct. I hope this helps.
My Best, Roberta McKnight Healthcare Multimedia Design http://www.hcmmdesign.net
From: rboone@vineland.org
Subject: [NIFL-AALPD:1861] Re: accountability (from the ed. and learning thread)
Date: Tuesday, January 04, 2005 03:55 PM
The assessment of Accountability is political by nature. Someone must set a "standard" by which we say there is a measure of Accountability. This standard is set politically by arbitrarily defining the assessments that measure Accountability. Ultimately accountability would just be the dependant variable applied to a function on assessment. A = f(a)
richard boone
From: eileeneckert@hotmail.com
Subject: [NIFL-AALPD:1863] Re: accountability (from the ed. and learning thread)
Date: Tuesday, January 04, 2005 09:26 PM
Roberta, I appreciate this statement that you made: "Publicly funded education is a public trust. How do we account for tax funded expenditures? What benefit has been derived from this funding?"
But I think you are boxing yourself in with your definitions of assessment and accountability. Assessment, even in conventional terms, goes well beyond the role you give it: "Assessment is devised to improve and serve instruction, teachers and learners. This is where we test learners with all the various means at our disposal - authentic, etc. - we assess what was learned in order to improve instruction." Assessment certainly includes assessment of learner performance--beginning and ending and the progress in between (although I don't <test> students at all even when I'm fortunate enough to be teaching or tutoring). Assessment can also be used to improve programming beyond instruction.
Assessment is at the heart of accountability--assessing and demonstrating the outcomes of instruction and programming, what's been learned and what's been gained. But I am <not> talking about standardized assessments. I think the outcomes should be measured first and most importantly in terms of quality-of-life and empowerment of learners, by their own perception, and how their learning in a program contributed to that. And instruction and programming should be focused on supporting that learning.
I'm not trying to operate within the conventional definitions of assessment and accountability; I'm proposing a legitimate (more legitimate, I think) alternative. It's not "free of bias" as I don't think there's any such thing; rather it tries to compensate for individual biases through triangulation of data and other naturalistic methods.
I don't think there's any reason to accept the current assessment and accountability frameworks even if the assessments in the accountability system actually met their own standards for validity and reliability. Those ideas suited the Cold War mentality that anything quantifiable was scientific and therefore better than anything that...wasn't, or didn't try to be.
I hear that the journal issue with my article is in the fall 2004 issue of Adult Basic Education (I didn't get a copy, so I didn't even know it was out!), so maybe you'll find it worth reading what I can't fully articulate in a post to the list. My point is simply that we can organize all aspects of education around learning, and we should.
Eileen
From: sgabb@bristol.mass.edu
Subject: [NIFL-AALPD:1864] Re: accountability (from the ed. and learning thread)
Date: Tuesday, January 04, 2005 10:59 PM
Roberta, Eileen (and all ) - I too thank you both for your thoughtful responses to the important questions of assessment and accountability. I too agree in spirit with your statement and questions - "Publicly funded education is a public trust. How do we account for tax funded expenditures? What benefit has been derived from this funding?"
I would remind our colloquia, however, that the Adult Basic Education delivery systems in this country are inadequately funded, therefore a betrayal of the hope that such a poorly supported system can in fact meet the exploding need for such services with quality and assurance of reasonable 'benefits' as defined by those with the power of the purse. In my 32 years with this amazing endeavor, I have learned over and over about the strength and determination of adults who choose to extend their formal education and build their basic skills. I have seen an increased awareness of this need in both the public and private sectors, and an increased willingness to provide 'tax funded expenditures'. However, the increase in funding has been meagher when compared to the expanding demand. Nevertheless, we have seen an increasing cry for 'accountability' from a field still condemning those within it to part time work and poverty wages. (Following is my longish response to this conversation. Read on at your own risk.)
I do embrace totally our responsibility, despite our limited funding, our need to be accountable to our learners. I agree with Eileen that it is our responsibility to engage in dialogue about goals, assessment and outcomes with our learners, in order to evaluate whether participation in the ABE services have indeed enabled the learners to frame their goals, identify needed skills and knowledge, and demonstrate competency and knowledge attainment in ways that are meaningful in their lives. In my experience, adult learners always know why they returned to school: each learner can spell out the reasons for entering our programs.
Each learner has defined purposes: to read better so I can enjoy a book, read to my children, feel like I am smart; to get a GED so I can complete my education, show my children it's important to finish what you start, feel like I am smart; to improve my English fluency in speaking, reading and writing so I can communicate with anyone who requires that I speak in English, fill out forms, write letters, understand what I am hearing.
The 'benefits' outlined by those who frame 'accountability to the tax payers' are listed in terms of numbers, certificates, certified employment. These 'countable outcomes' are considered by many the only legimate 'proof' of our accountability. Accountability to the learners - their evaluation of the services - is not considered legitimate.
Certainly we provide assessment to measure our own performance as well: are learners succeeding in attaining the skills and knowledge we have introduced? But as Eileen indicated, we need to assess with equal importance whether the knowledge and skills we introduce are really in line with the purposes and goals of our learners. Once we are sure our instructional content is in line with learner purposes and goals, we should indeed assess our own competency as instructors, guides for learning, partners in educational dialogue. In this context, as Eileen says, assessment with our learners enables us to evaluate and improve our entire schema for adult education programming.
I am also a realist. I recognize that currently, in order to work with and for adult learners in the current climate for funding and programming, I must comply with the limited, even destructive definitions of 'outcomes', 'benefits' and 'accountability'. I hope to continue engaging both practitioners and learners in dialogue about why this is important for survival in the short run, and about how we can - and must change and expand these definitions and practices in the long run. Through this dialogue we can elevate the voices of our learners as equal partners in advocacy for a system that is truly 'public adult education' - not the few bread crumbs we are spared, but rather the true investment in the natural resource of our learners, whose outcomes will continue to benefit us all.
Sally Gabb, SABES SE, Bristol Community College, Fall River, MA sgabb@bristol.mass.edu
From: rkmcknight@comcast.net
Subject: [NIFL-AALPD:1865] Re: Accountability (from the ed. and learning thread)
Date: Wednesday, January 05, 2005 01:07 AM
Thanks so much Eileen,
You are right on all counts. When I refer to testing, I'm referring to all forms of assessment. Assessment can take the form of asking questions or simply talking with a student. But, it gives us some place to begin and focus the learning process. I think of assessment as a microlevel process, and accountability as a macrolevel process:
Assessment let's us know what a person needs to learn - at the individual level. Its a circular concept - a feedback loop - to guide the continuous process of learning. It helps us discern, not only what needs to be learned, but also whether learning occurred from our efforts. Using myself as an example, I check (test, if you will) myself to see whether the direction of my inquiry is fruitful. If I'm not learning what I hoped to learn through a particular process, then I must decide where to direct my learning/inquiry to fill in any gaps in my knowledge/understanding.
From an accountability perspective, if I find that I'm not learning what I wanted in a particular program, then I'll scout around to find a program to help me. I might learn that the program isn't helping anyone - I don't know until I ask. Accountability exists at this macrolevel. It tells us things like - how a whole class/program is doing, whether particular teaching methods are useful. At the program/state/national level we evaluate the merits of programs and methods. By exchanging information about macrolevel approaches/issues at conferences and listservs like this one, we can make informed decisions about funding - to ensure the public trust.
For the purposes of this discussion, let's use the TABE as an example of the dynamics between assessment and accountability (micro/macro). The TABE has a limited scope - it assesses a particular range of skills for each individual (micro level). We know its validity is limited because it does not measure all that we wish to measure. Its use as a tool for accountability (macro level) is similarly limited. The magnitude of these limitations is increased when it is used as a tool for accountability to the exclusion of a broader range of measures.
We need more assessment tools to provide a better picture of what is occuring at the program level. Stites and many others have been saying this for a long time. To get at the issue of accountability and discern whether funding decisions are worthwhile, we have to ask questions like:
What is the process for approving new assessment instruments? This has not been made public by the NRS.
What content are we measuring with existing/approved instruments? This is a source of appropriate concern.
What benefit is derived from particular funding initiatives? This should be public information.
The issue here is public disclosure. Administators must address these and other questions. They must account for whether and how programs and expenditures for adult education serve learners and educators. In the process, perhaps we can link assessment and accountability in a more useful and meaningful way.
My Best, Roberta McKnight Healthcare Multimedia Design http://www.hcmmdesign.net
Go on to Accountability Full Discussion 1
Return to Accountability
