Professional Wisdom
From LiteracyTentWiki
Return to Evidence_Based_Adult_Education
Background
This discussion on professional wisdom in the field of adult education and literacy, between John Comings and David J. Rosen, took place by e-mail in November 2007. Dr. John Comings was the Director of the National Center for the Study of Adult Learning and Literacy (NCSALL) at Harvard, and is now an independent consultant. Dr. David J. Rosen, for many years the Director of the Adult Literacy Resource Institute at the University of Massachusetts Boston, is also an independent consultant. This discussion grows in part from an observation that David made to John a few years ago, when he had been called for jury duty, that in our society given that juries make life and death decisions based on an examination of evidence, perhaps a jury model might enable our field to judge professional wisdom in adult literacy education. Through this discussion they hope to promote greater interest in how our field might build a model for achieving a body of professional wisdom that teachers and administrators might use to improve basic skills teaching, learning and programs for adults.
DAVID: John, I am looking forward to our discussion about professional wisdom in adult literacy education, how it should be defined, why it is important to the field, what its relationship is -- or should be -- to research, how professional wisdom should be developed, judged and shared, and where professional wisdom fits in a hierarchy of evidence that teachers could consider and test out in their own practice.
Let's start with the name, professional wisdom, also referred to as practitioner wisdom and practitioner knowledge. What does professional wisdom mean? Why should it be important to teachers and other practitioners? What does it have to do with evidence-based practices? Why is important to the U.S. Department of Education? Why is it important to you as a researcher?
JOHN: David, here are my answers to your questions:
What does professional wisdom mean?
Professional wisdom, which is also called practitioner wisdom and practitioner knowledge, has been defined by the Department of Education as the judgment that practitioners acquire through experience, consensus views, and the effective identification and incorporation of local circumstances into instruction (http://www.ed.gov/nclb/methods/whatworks/eb/edlite-slide004.html).
Why should it be important to teachers and other practitioners?
Professional wisdom is a component of evidence-based practice, and the Department of Education is requiring practitioners to engage in evidence-based practice.
What does it have to do with evidence-based practices?
The Department of Education defines evidence-based practice as the integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction (http://www.ed.gov/nclb/methods/whatworks/eb/edlite-slide003.html ). Empirical evidence is defined by the Department of Education as evidence coming from scientific research studies that are rigorous, systematic, objective, empirical, and peer reviewed and that rely on multiple measurements and observations, preferably through experimental or quasi-experimental methods (which are referred to as gold standard methods) (http://www.ed.gov/policy/elsec/leg/esea02/pg107.html). The field of adult education has very little gold standard research to base decisions on, and its body of non-gold standard scientific research is also small. This leads the field to draw on K-12 research and to make decisions without the help of empirical evidence. Professional wisdom, therefore, plays a bigger role in adult education than in K-12 education or in fields, such as medicine and public health, that have a large body of gold standard research. Professional wisdom is what practitioners learn when they a. put empirical evidence into practice, b. adapt K-12 evidence to adult education, and c. fill in a gap in evidence through building a consensus, through action research, or through experience.
Why is it important to the U.S. Department of Education?
This evidence-based approach to making decisions is meant to insure that participants in adult education programs receive services that provide them with the best possible opportunity to gain the skills and credentials they need to be successful in the labor market and in their roles as parents and citizens.
Why is it important to you as a researcher?
Researchers have two reasons to be interested in professional wisdom. The first is that our research usually produces "findings" (that is, we might say that when practitioners do X in their programs or classes students learn more efficiently or persist longer) but practitioners are looking for "practices" (that is, how do I do X in my classroom?). Professional wisdom can be the result of practitioners employing research findings in their programs and classes and describing how they did it, and this takes what researchers produce and makes it useful to practitioners. The second is that when practitioners put research into practice they often generate hypotheses on the strengths or weaknesses of that finding. When there is no research on a subject, the consensus views and experience of practitioners provide likely hypotheses about what might work. Professional wisdom provides researchers with good advice on where they should next take their research.
DAVID: I find those answers very helpful and clear, John. We now have a context in which to explore professional wisdom in more depth.
A great deal of attention has been paid by the U.S. Department of Education to empirical evidence. I am interested in our field now paying the same kind of attention to professional wisdom. Given how little empirical evidence we have now, practitioners will need to rely on professional wisdom, yet it seems to me that there are many questions that need to be explored before that is possible. I would be interested to know how you think the field of adult education and literacy should elaborate on the Department's definition of professional wisdom. Specifically, how do you think professional wisdom should be developed in our field? How should it be judged? How should professional wisdom be shared?
JOHN: David, here are answers to your new questions:
How do you think professional wisdom should be developed in our field?
We need a continuous collaboration between researchers and practitioners, and both will need training to ensure that the relationship is successful. Unfortunately, our field no longer has a national research center. We do have many fine individual researchers, but they do not have the leadership and support to engage in a continuous relationship with practitioners that would improve research, practice, and policy. So, first, we need to fund a national research center. Then we need to put that funding in the hands of researchers who are committed to a broad definition of evidence-based practice and to working in collaboration with practitioners. The first step for this new research center would be to ask practitioners and policy makers to clearly state the subjects they need advice on, and then begin reviewing all the existing empirical research and professional wisdom that might provide good advice. Examples of these subjects are how can a program best support persistence, help participants transition to postsecondary education and training, or support students to develop their numeracy skills?
Since some of the research and professional wisdom on one subject might be contradictory, the research center should establish a jury that would review the existing empirical evidence and professional wisdom and then propose evidence-based practice for the many different components of adult education. As in a trial, the jury would weigh the evidence and make decisions. The ideal members of such a jury would be experienced practitioners who have significant knowledge of research and experienced researchers who have significant knowledge of practice. The field has many such people. The jury would have to meet several times a year to review new research and professional wisdom in order to update its advice. Out of this review would come the best available advice for practitioners and research questions that could be pursued by the research center.
Practitioners could then put this advice into practice and report on their experience. This would be professional wisdom.
How should it be judged?
Professional wisdom should be judged in the same way that research is judged, with a peer review process. Peer review is a process in which qualified researchers judge the quality of a study's methodology and the analysis of the study's data. Peer reviewers decide if a study should be published and often ask for changes in a study's report. Professional wisdom needs a similar peer review process, though like the jury it should have both researcher and practitioner members.
How should professional wisdom be shared?
Peer reviewed professional wisdom should be shared through an online resource that would also have, for each subject, the synthesis of research and professional wisdom completed by the jury and information on new research findings or research underway that focus on the question.
DAVID: As you know, I also favor the idea of an adult education jury weighing the evidence of professional wisdom and research. In a court of law, there is a hierarchy of evidence and the jury is instructed to consider some kinds of evidence as more valuable than others. I wonder if you have given any thought to what a hierarchy of evidence might look like for a jury of expert practitioners and researchers weighing the evidence -- the professional wisdom and research -- that would be collected to answer questions about persistence, transition to higher education, numeracy, English language teaching, uses of technology, and many other questions that practitioners need to know the answers to. We would want the experience or reflection of a single teacher, I suppose, at the bottom of the hierarchy, and gold standard research at the top. Could you fill in some of the other levels of evidence in this hierarchy? And could you comment on why we shouldn't just put all of our efforts and resources into gold standard research? Why is a hierarchy useful to those in the field of adult literacy education?
I have found that teachers are most often driven to read research when they have a burning question from their own practice. I have been experimenting with organizing studies for practitioners on the Adult Literacy Education Wiki: http://wiki.literacytent.org so that there are three levels of responses to a question posed by a practitioner. A level one response is a short answer based on the research. At level two one finds a summary of the research. At level three is a link to the study. A level four response might be a jury review of the research, or a meta-research study that identifies, classifies, analyzes and evaluates the research when there are multiple studies. These levels of answers enable practitioners to dig as deeply as they wish but also, if they have limited time, to get a good answer to their question. Does this sound to you like a good way to share professional wisdom as well as research?
JOHN: David, you ask:
Could you fill in some of the other levels of evidence in this hierarchy?
Here is a possible hierarchy:
• Adult education gold standard research integrated with professional wisdom developed when practitioners put that research into practice.
• K-12 gold standard research integrated with professional wisdom developed when practitioners put that research into practice.
• Adult education non-gold standard scientific research integrated with professional wisdom developed when practitioners put that research into practice.
• K-12 non-gold standard scientific research integrated with professional wisdom developed when practitioners put that research into practice.
• Consensus views of a group of practitioners and researchers
• Practitioner knowledge developed through action research
• Practitioner judgment developed through experience
Could you comment on why we shouldn't just put all of our efforts and resources into gold standard research?
Gold standard research is only worth doing once researchers and practitioners have developed a theory through observation and exploratory research. Until that time, something like a consensus view may be the best advice.
Why is a hierarchy useful to those in the field of adult literacy education?
It provides a way to make decisions. Of course, it is possible that a practitioner could develop, through individual experience, better advice than gold standard adult education research. If the gold standard research isn't working for a practitioner’s class or with some of his or her students, then other sources of advice should be tried. However, practitioners should be looking for evidence that what they are doing is working, and working better than what they have tried before.
Does what I have proposed sound to you like a good way to share professional wisdom as well as research?
The teacher training faculty at my school believe that to be a good teacher, one of their students must learn the content they are teaching, learn teaching techniques and approaches for that content, and learn to be reflective. That is, they have to have the core competencies to do their jobs, but they must also engage in continuous improvement. As practitioners come up against problems, they should have easy access to the best available empirical evidence and professional wisdom in ways that make it easy for them to incorporate that advice into their practice. Multiple layers of detail should be part of that system of access. This is another task for a national research center.
DAVID: What would a consensus view of a group of practitioners and researchers look like? Do we have some examples of consensus views in our field now? Would the jury you described earlier be a primary way in which a consensus view was reached? (If not, where would the jury fit in the hierarchy?) Are there other ways in which a consensus view might be reached? How would we know when we have a consensus view? What would that look like? Are you aware of examples of consensus views from other professional fields (medicine, law, engineering or others)? Would any of these models be useful to our field?
JOHN: Here are the answers to your questions:
What would a consensus view of a group of practitioners and researchers look like?
We have some consensus views that have been around for a while. For example, that our classes should be open entry open exit because our students have a difficult time maintaining consistent attendance. That consensus has been changing to one that favors managed enrolment. The jury suggested earlier could resolve this consensus view by reviewing all the evidence (I don't know of any research on this question but there is data from programs or states that have changed from one to the other approach that could be reviewed) and gather professional wisdom. The outcome of this review would probably not be that everyone should do one or the other, but, rather, a statement of the strengths and weaknesses of each approach and, maybe, the types of students who might benefit from one or the other approach.
Do we have some examples of consensus views in our field now?
In 2000, TESOL published "Program Standards for Adult Education ESOL Programs". This is a good example of an effort to pull together a consensus among experts as to what constitutes good practice.
Would the jury you described earlier be a primary way in which a consensus view was reached?
Yes, the value of the jury approach is that 1. The field would have a chance to pick the people who would be on the jury and 2. The jury could function under a hierarchy of evidence.
Are there other ways in which a consensus view might be reached?
Yes, but a jury's authority derives from who is on it, how they are chosen, and the rules they function under. One ESOL expert could have written the program standards and claim that it describes the field's consensus, but that publication would have less authority than the TESOL document, since the ESOL field knows TESOL and the people who participated in the drafting of that document. Adding a process for the field to participate in choosing the jurors and rigorous criteria for weighing evidence would give it even more authority.
How would we know when we have a consensus view?
The consensus view is just a starting point. It articulates the best available advice for its time. But, that advice should then be put under scrutiny and new research and professional wisdom should test the advice and learn more about when it works and when it doesn't, who it works for and who it doesn't work for, and ways it can be improved. The jury would meet again and review this new evidence and change the consensus. This should be a continuous process.
What would that look like?
The Internet provides us with new options of how the consensus would be presented. Most likely, it would have three forms: one for practitioners, one for policy makers, and one for researchers. For practitioners it might provide direct advice. For policy makers it might provide a framework for effective programs. For researchers it would chronicle the changing theory that is evolving through this process.
Are you aware of examples of consensus views from other professional fields (medicine, law, engineering or others)?
A famous one was a Delphi study done by the Rand Corporation for the US Department of Defense during the Cold War. DOD wanted to know how much retaliatory power the US would have after a first strike nuclear attack by the Soviet Union. Not the kind of question that lends itself to a gold standard research test. So Rand wanted to ask experts their opinions, but as we all know when a group of experts come together an interpersonal dynamic comes into play with the more senior experts intimidating the more junior experts. So, Rand asked each expert individually and then analyzed that data to come up with a range of projected outcomes. Rand sent these back to the experts and asked them to narrow the range. This went on until Rand had a final outcome. Billions of dollars were spent based on this consensus.
The hospital industry is undergoing an international consensus building process about how to make hospitals safer, led by a nonprofit here in Boston.
Would any of these models be useful to our field?
I think we can look at these and other examples, but we should develop our own approach. We have much less empirical evidence than medicine, and our questions are more complicated than nuclear retaliation. We should start a process and then improve it over time.
DAVID: These answers are very helpful, John. I especially liked your examples of consensus.
We have discussed how professional wisdom could be judged and, to some extent, how it could be shared. I would like to look more closely at how it could be developed, specifically how we could build and strengthen relationships between researchers and practitioners to undertake the development of professional wisdom and to share their findings with each other.
Over the past year, as the moderator of the National Institute for Literacy Special Topics Discussion list, I have had the opportunity to look at the interaction of researchers and practitioners in several discussions. I have been pleased to see, in some of these discussions, not only that practitioners have been influenced by researchers' findings but that researchers, considering new questions in their areas of their research, have said they have learned what questions practitioners care about, and also have learned about promising practices that they might examine. I saw this same engagement in the two Research and Practice conferences sponsored by NCSALL and CALPRO in California. I have seen some sharing of research, and some sharing of practice, too, on the Adult Literacy Education Wiki. I wonder if there are other opportunities for researchers and practitioners to share their findings that we should explore, and if it is possible, as researchers and practitioners, to set the priorities for further research together.
JOHN: David, you wrote:
I wonder if there are other opportunities for researchers and practitioners to share their findings that we should explore, and if it is possible, as researchers and practitioners, to set the priorities for further research together.
There are some places, California being one example, where researcher-practitioner collaboration is taking place. This is not enough. The field has to make a commitment to pursuing "evidence-based practice", not as something that is imposed on them by Washington DC, but as something they want to do to improve the services they provide to their students. Once there is a new administration in Washington, adult education will receive a little bit more attention, but there will be no money to expand funding until the nation's fiscal house is put back into order. So, the field will probably be level funded for several more years. In relation to what we have been talking about, this is actually an opportunity. The field, through its 50+ state directors, should say to the new administration that we are willing to reallocate some of the existing national funding (say $5 to $10 million a year) for a national research center that is a true collaboration between researchers and practitioners. However, the field wants to have some say in how this center is designed, how the funds are allocated, and how the center is managed. Then, when additional funding becomes available in a few years, the field will be able to argue for additional funding for services and for research based on its commitment to evidence-based practice.
DAVID:
What are your thoughts about a jury model through which we could synthesize and integrate empirical evidence and professional wisdom? Who would select the jury? How often would it meet? How would it conduct its work? For example, would it look at persistent practitioner and researcher questions? After reviewing the available evidence and professional wisdom, would it propose evidence-based practice for the classroom, tutorial, online learning, for program quality? For other areas? What are some steps that we, as a field could take to create this model?
JOHN: David, I feel this next step is something the field, or representatives of the various stakeholder groups, should take on as a consensus building effort. However, I could suggest a design to begin with, since I have been thinking about how this might work.
The first step might be to form a board that would oversee the establishment of this system and then monitor it. That board should have members from the various stakeholder groups: adult students, practitioners, policy makers, and researchers. That group should select, through a competitive process, a person and institution to manage this system, which would become the national research center. The research center would then undertake a broad consultation with the field to identify the components of a program (literacy instruction, math instruction, counseling, and transition are examples of components) and a system (professional development, data systems, and monitoring are examples of components) that supports programs. These components would be the "subjects" or areas of interest for the research center. The board would consider the findings from the consultation and finalize the components and prioritize them. The research center would then draft a hierarchy of evidence, a jury process, and list of jury members. The board would review and finalize these. The research center would then begin reviewing all of the research and professional wisdom on the first several subjects on the priority list. The jury would read the drafts and then meet with the research center staff to make decisions before finalizing the review and posting it on the web. Once the initial work is done on each component, the jury would meet three or four times a year to review new findings and add them to the existing reviews.
The reviews would suggest the next steps in research to improve the knowledge base on that subject. The reviews would be a source for other institutions and professionals who would use them in professional development, degree programs, program design, program management, policy formation, and other activities that build and support the field. The research center should assist the sharing of experience among these institutions and professionals as they put the evidence into practice.
This would allow the whole field to work together to implement and improve the same basic framework. If the framework has flaws, they will show up almost everywhere, can be easily identified, and the resources of the research center could then be redirected by the board to look at those flaws and find ways to address them.
This all looks logical and neat when written down here. In practice, this is going to be a difficult and messy process. But, if we get started on it and commit to making it work, it will have an immediate benefit in those states and programs that are struggling to raise the quality of their services and allow high quality programs to share what they are doing within a common framework. After a few years, we will have a national system of continuous improvement and will be able to argue for greater funding since the system is based on the best available empirical evidence and professional wisdom.
As I said before, I think the only way to move ahead with this is for the field to agree to give up some of the funding for program services and put it into the establishment and operation of this system. That way, it can happen quickly without increased funding, and it might more likely be managed by an independent board that serves the interests of the field.
Return to Evidence_Based_Adult_Education
