LPRPArchivesTheJury

From LiteracyTentWiki


4/6/05

John, George, and others,

How shall we define "professional wisdom" in our field? Do we have an agreed upon definition for the term as used in the way you , John, are using it here?

I ask because we need to wrestle with this on the ALE Wiki, where (we claim) researchers and practitioners can find -- and discuss -- important questions in our field which are addressed by "research and professional wisdom." We will also need to define research, of course. So far, we haven't tackled either.

As I was sitting in a Boston courtroom this week, waiting to be empaneled to perform my civic duty, it occurred to me that life and death issues, as well as all of the other concerns which affect the well-being of people and corporations, are decided in our society using a different "evidence-based" process, one where a broad range of evidence --including testimony -- is offered, but then judged by a jury. I wondered: if this is good enough for life and death issues, why isn't evidence, decided by a jury (of experts) good enough for most decisions about education practice. Maybe it is, and maybe "expert juries" from our field should be part of the definition of "gold standard" professional wisdom. In any case, I think the legal metaphor offers an interesting alternative to the medical metaphor we have been using recently in discourse about standards for adult education practice based on evidence.

David J. Rosen

--

4/6/05

David,

This is an idea that I have been promoting to federal officials. I wasn't calling it a jury, but I like this concept a lot and will start using it. A jury develops a consensus after weighing evidence of many different types, but the jury is given direction as to the hierarchy of that evidence -- an eye-witness account is much better than circumstantial evidence, for example, and hearsay is not admitted.

Our field has very little in the way of high quality experimental or quasi-experimental evidence, but it does have some high quality corrlational evidence and quite a lot of high quality qualitative evidence. In addition, much of the high quality evidence from K-12, postsecondary, and training research is useful for filling in the gaps. Even so, many gaps, some of which may be crucial to success, remain. However, even these gaps are addressed by a significant body of professional wisdom literature, past efforts of groups of experts who were acting as a jury.

An extremely useful thing for our field to do would be:

1. IES, OVAE, NIFL, NICHD, etc. agree to abide by the recommendations of a jury of experts 2. IES, OVAE, NIFL, NCHD, etc. articulate a hierarchy of evidence for the jury to employ 3. Someone funds a jury of experts and a group of research assistants to help the jury. 4. The jury develops a framework for describing good programs to structure their deliberations. 5. The jury weighs all the evidence and makes decisions on what constitutes good ABE, adult ESOL, and ASE programs by employing the hierarchy of evidence. This means that if the only evidence is professional wisdom, then for now it is the best available evidence. 6. The jury describes good programs (probably along a continuum from sufficient to excellent). 7. The jury identifies research questions that are most likely to resolve important questions or develop new knowledge that will improve program performance.

This would give our field a consensus on what we believe are services that are worthy of the time and lost opportunities our students invest in our programs, would define programs in which research might be productive, and provides an agenda for useful research.

John

--

4/6/05

John,

The process you have described makes sense to me. And understanding a hierarchy of evidence should be part of the jury's deliberation.

What do others think about this?

From a teacher perspective, would this be useful?

What other elements would make a jury process respected by practitioners and by policy makers?

John, what would it take to make this process a reality for our field?

David

--

4/08/05

Erik and all,

You ask: >For the practitioners who were asking questions about research terminology, who may feel less conversant than they want to be - what kinds of things do they have to incorporate into their vocabulary to get a seat at the table?

One answer that comes to mind is that the people at the table stand up, move over and/or add a chair to make room. Inevitably this conversation becomes polarized. At the adult education interest section academic session last week, Heide Wrigley made the wise observation that there is a continuum of research practice, undertaken by different sorts of people for different sorts of reasons.

Some voices and language are privileged more than others, particularly in work funded by the current administration in the US. Amongst ourselves we try, one hopes, to accord each other mutual respect. Part of that respect, maybe, has to do with those with power in the academy recognizing it, acknowledging it, and using it proactively to push at the edges of the kinds of research that is fundable. We also need to recognize that certain approaches to and kinds of research serve different purposes.

Janet Isserlis

--

4/08/05

Greetings!

So many ideas/ comments have come through this listserv this past week, that I must leave lesson planning and writing of an academic research paper on hold to respond to some of these comments. (Some background first: I am both a practitioner and a research associate at the NCSALL ESOL labschool in Portland, Oregon)

" The judgment that individuals acquire through experience increased professional wisdom is reflected in numerous ways, including the effective identification and incorporation of local circumstances into instruction"

Whether we agree with it or not is irrelevant. That is how the people who take professional wisdom into consideration define it. Just like hearsay in the court of law, that is what it is, unless you can prove it otherwise. ( Technicality is a beautiful thing.)

  • On hierarchy of evidence: I personally don't think there is a hierarchy of evidence. I think all the evidence brought to prove a point, and here it could be to justify a particular "best practice", complement each other. Here is an example: we just finished the analysis on our year-long reading experiment, that is the quantitative analysis. It is beautiful but it only shows one side of the research: it is one dimensional. As such, once it is presented to practitioners, it gives them very little to go by to make a decision on what to do Monday morning. I think it leaves curriculum writers pensive, and professional developers on the edge. What to do? with what to think as an after thought, is often what is expected from educational research.

Which brings 2 points:

1) Unless we complement the qualitative data with quantitative data, which should include tape recording, video recording, field notes, teacher reflections, student reflections, we are ignoring the richness and the depth that learning is; learning here being, of course, the learning of the students as well as the practitioners an the researchers.

Got to quote Primo Levi, from " the Periodic Table"( originally published in 1975) . Levi was a chemist by trade, and knew all about scientifically based evidence... This is what he writes about working in a lab on the quantitative analysis of rock samples.

" Quantitative analysis, so devoid of emotion, heavy as granite, came alive, true, useful, when part of serious and concrete work. It was useful: it was part of a plan, a tessera in a mosaic. The analytical method I followed was no longer a bookish dogma, it was put to the test every day , it could be refined , made to conform with our aims, by a subtle play of reason, of trial and error. To make a mistake was no longer a vaguely comic accident that spoils an exam for you or affects your marks: to make a mistake was similar to when you go climbing- a contest, and act of attention, a step up that makes you worthy and fit. ( 1984 English translation, Schocken books inc, p. 71) These words have kept me going as both a practitioner and a researcher, giving reason to both as whole. If I don't reflect on my trials and errors as a practitioner, I don't advance in my practice. If we only look at the hard facts of scientifically based research, we are missing out the human component of teaching and learning, which we are constantly reminded of, in our work, is vital to establishing our community of learners.


2) Can we ever breach the gap between practice and academic research? There are two languages and there is no common language. It takes a while to become truly bilingual in both. As in any conversation, one must use the appropriate register to address whoever the audience is.

I said "academic research" not "action research": I believe, ( professional wisdom accumulated over the past 4 years wearing 2 hats) that action research can easily go from practitioner language to academic research language. However, it is much more difficult to go from an academic language to practitioner language. I am in the process of doing just that, and even with practitionese as my first language, I have to change the whole approach in order for it to have somewhat of a "practical" direction. You might say: then, maybe you're not doing the "right" kind of research. There is no right kind. As Janet says: "We also need to recognize that certain approaches to and kinds of research serve different purposes."

Back into lurking ( and planning, and writing.)

Dominique Brillanceau

--

4/15/05

I think a panel like this could be really useful if it included adult school students, teachers, and administrators, as well as researchers because all these perspectives are valid and bring a different view. I am thinking of the movie Roshomon (I think that is how you spell it) that I rented from Netflix recently. In it you get all the perspectives of a crime and are left to decide what really happened. But unless you have all the perspectives you really have a hard time getting to the truth because everyone has a point of view and sometimes it is hard for us to be objective about what our point of view is. I have noticed this recently on the GED taskforce committe I am serving on. It includes members from Social Services, local businesses, my school and others, the community college system, and the local state university. All of us approach the idea of why the GED is important and what we want to "fix" from totally different perspectives. It has been a real eye opener for me in terms of other people's beliefs about what I do and what the GED represents to them.

In taking the courtroom analogy a bit further, and in an attempt to define this slippery term "professional wisdom." It seems to me that by observing adult school teachers more as they teach (which they regularly do when I teach at the community college), you would get to know both by student evaluations and by teaching evaluations which teachers actually know what they are doing. I always welcomed these evaluations as it helped me to improve my teaching. Yet while these practices are standard in the university and college systems in which I've taught, I have never had anyone formally observe my teaching or collect student evaluations. So my question is, how do you identify a teacher whose observations result in actual productive teaching vs. just someone who "talks the talk?" How do you identify those master teachers who have the wisdom you seek?

These are the people, who once identified, should be our "expert witnesses." After all, they are witnessing and observing daily things that cause them to ask questions and seek answers. But even non-expert teachers who are really interested in the art of teaching make observations and ask questions. How do researchers currently assess the burning questions adult school teachers want answered? Or do they?

Michele Craig

Retrieved from "http://wiki.literacytent.org/index.php/LPRPArchivesTheJury"