Scientific-basedEdResearch
From LiteracyTentWiki
Evidence-Based Educational Research
An AAACE-NLA Electronic Forum Colloquy
January 28 to February 7, 2005
What is viewed as legitimate educational research? What counts as evidence? What role does scientific analysis play in a field that is grounded in cultural practice? What about the role of political culture? What is the relationship between theory and rigorous scientific analysis? Can conflicting perspectives over various definitions of educational research and theory be reconciled? What about the relevance of educational research and theory to the dynamics of everyday practice? These issues are probed in some depth here, but they are far from resolved. Whether or to what extent they can be, is a significant question in itself. While definitive resolution of complex educational matters like how adults learn to read and the relationship of reading to that of knowledge acquisition is likely not possible at all, there is at least hope that substantial progress can be made in our understanding that can (and must) have practical as well as theoretical application. Are these issues are of interest to you? If so, you are likely to find value in the following discussion. As an interactive format, you can add to this discussion, and we hope you will. To learn how to do this go to AleDirections .
The discussion has arbitrarily been divided into four parts, on four pages. Read down each page and there is a link at the bottom to the next part. There is also a page with pertinent writings from the American philosopher and educator, John Dewey. You will find this at ConceptsinInquiryandLearning .
The following three articles, two of which are directly referenced in the messages below examine the relationship between science and educational research from various vantage points, and help frame the discussion on Evidence Based Educational Research. As you read through the messages, you may want to consider how each participant draws, implicitly or explicitly from one of these articles, or how what participants say challenges one or more of the arguments made by the three sets of authors.
Overview statement of Paula J. Stanovich and Keith E. Stanovich (2003). Using Research and Reason in Education: How teachers can use scientifically based to make curriculum and instructional decisions. Partners in Reading: National Institute for Literacy, National Institute for Child, Health and Human Services, U.S. Department of Education, U.S. Department of Health and Human Services. http://www.nifl.gov/partnershipforreading/publications/pdf/Stanovich_Color.pdf
"Education is informed by formal scientific research through the use of archival research-based knowledge such as that found in peer-reviewed educational journals. Preservice teachers are first exposed to the formal scientific research in their university teacher preparation courses (it is hoped), through the instruction received from their professors, and in their course readings (e.g., textbooks, journal articles). Practicing teachers continue their exposure to the results of formal scientific research by subscribing to and reading professional journals, by enrolling in graduate programs, and by becoming lifelong learners.
Scientific thinking in practice is what characterizes reflective teachers--those who inquire into their own practice and who examine their own classrooms to find out what works best for them and their students. What follows in this document is, first, a 'short course' on how to become an effective consumer of the archival literature that results from the conduct of formal scientific research in education and, second, a section describing how teachers can think scientifically in their ongoing reflection about their classroom practice.
Being able to access mechanisms that evaluate claims about teaching methods and to recognize scientific research and its findings is especially important for teachers because they are often confronted with the view that 'anything goes' in the field of education--that there is no such thing as best practice in education, that there are no ways to verify what works best, that teachers should base their practice on intuition, or that the latest fad must be the best way to teach, please a principal, or address local school reform. The "anything goes" mentality actually represents a threat to teachers' professional autonomy. It provides a fertile environment for gurus to sell untested educational 'remedies' that are not supported by an established research base."
'"Summary of Evidence on 'What Works': An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs'," by Madhabi Chatterji. Educational Researcher, Vol. 33, No 9/02, 3-13. http://www.aera.net/uploadedFiles/Journals_and_Publications/Journals/Educational_Researcher/Volume_33_No_9/02_ERv33n9_Chatterji.pdf
The article begins:
"Federal policy tools for gathering evidence on “What Works” in education, such as the What Works Clearinghouse’s (WWC) standards, emphasize randomized field trials as the preferred method for generating scientific evidence on the effectiveness of educational programs. This article argues instead for extended-term mixed method (ETMM) designs. Emphasizing the need to consider temporal factors in gaining thorough understandings of programs as they take hold in organizational or community settings, the article asserts that formal study of contextual and site-specific variables with multiple research methods is a necessary prerequisite to designing sound field experiments for making generalized causal inferences. A theoretical rationale and five guiding principles for ETMM designs are presented, with suggested revisions to the WWC’s standards."
Summary of Postpositivist Scientific Philosophy: Mediating Perspectives (2004). George Demetrion. http://www.the-rathouse.com/Postpositivism.htm
In highlighting the matter of scientific philosophy, this article raises the issue of epistemology (how we can come to know) on the assumption that how we can come to know is powerfully influenced by the mental models (or paradigms) that undergird them. In drawing out a postpositivist philosophy of science, the following hypotheses are assumed:
(a) That the focal point of an investigation is a problem orientation of significant enough scope worthy of the time and resources needed to work toward a reasonably viable provisional resolution. What constitutes a "worthy" problem and a "provisional resolution" are debatable matters and are culturally determined. Still, the point holds.
(b) A scientific-based approach to a cultural phenomenon like adult literacy education is based on a quest for truth as a regulative ideal. It is the quest for truth that separates postpositivist projects from constructivist and postmodern research designs. It is the ideal nature of the quest, even to the extent of truth being defined as a creative fiction (following, Karl Popper), that separates it from the positivist and neo-positivist research tradition in its quest for exacting certitude.
(c) That in a problem-focused approach to scientific investigation, methodologies are drawn upon as relevant to the study. On this assumption, there is no hierarchy (gold standard) of methodologies even as the nature of the study needs to be highly competent according to the disciplinary canons that undergird the study.
The messages in part one below and continued in the next two parts were originally posted on the National Literacy Advocacy electronic list (AAACE-NLA). To subscribe to the AAACE-NLA, or to view the archives of messages posted there, go to http://lists.literacytent.org/mailman/listinfo/aaace-nla , scroll down to the bottom of the page to enter your email address, name and pick a password. Then select subscribe. You will be sent an email to confirm that you want to do this. Reply to that message and you will be subscribed.
Part 1
Message One
1-28-05
I need clarification from the experts
Yesterday I attended a workshop on evidence based practice at the Texas Association of Literacy and Adult Education Conference (TALAE). The workshop was given by a NCSALL researcher who shared results of NCSALL's research on student persistence.
The presenter explained that according to the Department of Ed's office of scientific research (don't remember the precise name), educational programs must engage in evidence based practice. According the Dept of Ed, evidence based practice is: (I' am paraphrasing) "the combination of Scientific research and teacher's wisdom". I think that it may be needless to say that I have no idea of how to interpret this.
Based on the presentation by the NCSALL researcher I learned that students persist in programs because of good program quality among other factors. When I inquired about what this meant, the presenter said that she was reporting the data on persistence but she didn't know what defined program quality. So, in my wisdom (of an experienced teacher ;-)) I deduced that scientific research tells me that students want quality, something that I may have not been able to figure out on my own. I can combine this with my wisdom. In my wisdom, I could decide that quality program is to teach people phonics extensively. So, would this mean that I am applying evidence based research to my practice?
I also could deduce that quality means getting rid of WIA and NRS in my program. Again would this satisfy DOE?
Andres Muro
Message Two
1-28-05
Andres and Partners:
I had the opportunity to see the presentation on the subject of student persistence, in April 2004, in Boston. John Comings (NCSALL at Harvard) was sharing his research on this topic. He and his people are a well-respected source of information, able to take research and translate it into something we can use.
As Hal Beder (NCSALL-Rutgers) once said to me, and I am paraphrasing him, "The purpose of research is not to give answers, but to give practitioners ideas for their work." I didn't like the answer much, but it was honest and acceptable. I stopped looking for answers and have worked harder to be a translater. Like so many others, I am doing what I can to convert research information into promising practices.
One source defines evidence-based practice as... "a total process beginning with knowing what clinical questions to ask, how to find the best practice, and how to critically appraise the evidence for validity and applicability to the particular care [instructional] situation. The best evidence then must be applied by a clinician [teacher] with expertise in considering the patient's [students] unique values and needs. The final aspect of the process is evaluation of the effectiveness of care [instruction] and the continual improvement of the process" (DePalma, 2000).
The way I read this, your presenter's definition is consistent with the above source.
For the cause...clarity is a good thing!
Jose L. Cruz California Literacy
Message Three
1-28-05
Andres
As a practitioner, your "wisdom" would only be part of a picture. You would need to place that under the tutelage of a scientific researcher who would conduct an experimental design. Ideally, your wisdom would add confirmation to the research. If not, your wisdom might be factored in to the construction of a more perfect design, which perhaps a more rigorously trained scientist would conduct. There is a singular gold standard and that has already been defined as "experimental design." Your wisdom would play a role, but it would fall perhaps under the category of the clay standard along with case study ethnographic research and folk psychology. Otherwise, your "wisdom" could too easily be conflated with the "guru principle." It says so in some of the literature.
Socrates (A.K.A. George Demetrion)
Message Four
1-28-05
Dear Jose & others,
While I don't care for the medical analogy between education and medicine (the teacher dispenses treatment to the student/patient), I do appreciate the general tenor of the definition of "evidence-based practice" as you describe below. On that objective methodologies stem from the particular purpose of a given research project. That's one thing and that's reasonable. Where the problem comes in for me is when experimental design is interpreted as the gold standard. In my view, that's putting methodology ahead of the particular research question at hand. It's like saying that we seek the best approaches to assessment that can get at whatever it is we're trying to figure out about learning, and then saying the standards have to be measurable, uniform, and quantifiable. In defining the parameters of assessment (or research) thusly, one limits the focus to what the methodologies allow one to see. There may be good practical/political reasons to do so, but that's not synonymous with good science.
George Demetrion
Message Five
1-28-05
Colleagues,
Let's say I want to explore the operating premises of the New Literacy Studies as identified by David Barton, Brian Street, Sylvia Scribner (sort of a precursor), and Juliet Merrifield. The short definition focuses on the ways in which "literacy practices" are entwined in the ways in which adults draw on print-based texts (however defined) in relation to various tasks and projects that are of interest to them whether the "domain" is the home, the market, the workplace, the community, the child's school, or the local church. The object here is to obtain the needed information to achieve what one is setting to accomplish. Literacy, from this vantage point is viewed as a metaphor for knowledge acquisition and meaning making, in which the reading of print-based texts typically is (or can be) one of the intervening variables that facilitate the learning goals of the individual, or it could be a group.
A literacy program founded on the premises of the New Literacy Studies, while, obviously, including the teaching of the basic skills of reading and writing, wouldn't be exclusively defined by the methodologies (there we go again) of what might be viewed as pure reading instruction (whatever that may mean). It would include some type of prompt (whether through codes, direct text, photos, audio-or video tapes, field trips, etc) that has some correlation to the overall objective at hand (preparing for a meeting with one's child's school teacher). In a literacy program for low-reading level adults, specific practice in learning how to read and write would also be included. Depending on the type of program and objective of the students, the reading and writing component may be central, may be balanced with the more pressing issue of content analysis of the situation under examination, or under certain (rare) circumstances incidental.
Given this, what if the question I want to explore is the nature of the relationship between learning how to read and write and mastery of critical issues of concern to students in the various life domains, say the EFF role maps, so that we can draw on a commonly known example? What if also in my probing I don't view learning to read and domain-based knowledge mastery for beginning level readers as an either/or proposition, but, using the language of pragmatic philosopher Nicholas Rescher I am seeking a "duly-hedged synthesis. The goal might be, to use the language of EFF, to "strengthen and express a sense of self that reflects personal history, values, beliefs, and roles in the larger community." Say that I am a 25 male African American high school drop-out who has entered a GED program who seeks to become a full time manager at Burger King. Say the learning task is that of students exploring the sources of their motivation in whatever career paths they are thinking about. This student, say, still has basic problems with writing, reading vocabulary, and with comprehension beyond the literal level even as he has substantial knowledge about the dynamics of his own personal thinking process and of many of the environmental contexts that he is and will be confronting in the seeking of his goal.
The question becomes, how do I as a program manager construct a viable instructional plan with and for my teachers who are working with this student and others (including the 78 year old grandmother who is going back to school because "it's her time in life to do something for herself") who have various reasons for getting their GED connected to the various domains identified in the EFF project and other areas? To what extent do I focus on directly mastering the five content areas through continuous work with the GED books, and to what extent do I broaden the instruction, including even bringing in a Burger King regional manager to talk about career options at BK? Et cetera.
In short, how do I go about establishing a duly-hedged synthesis between mastering the basic skills of the text and broadening the instructional plan to include areas that student will want to more effectively engage after they obtain their GED, or after, say, they learn to read at the 5-th grade reading level as determined by readability formulas, to complicate our discussion just a bit?
Specifically, in what ways will research help to inform the making of a sound decision.
1. Will I wait for an experimental design that complexly factor in all the variables over a significant enough sample that will be sufficiently reliable so that I can apply the findings to my situation, knowing that my decision is "research-based?"
2. Given the practical impossibility of that will I rely instead on a quasi-experimental sample that factor in at least many of the variables similar to my situation?
3. As a seasoned practitioner, will I place more faith in my own wisdom and examine in depth a few highly-descriptive ethnographic studies even with considerable differences "speak" sufficiently to my own situation based on my imaginative and critical capacity in utilizing the texts to make my own reasoned inferences?
4. Will I seek out other practitioners in the formation of a teacher-research design in which, over a period of time we engage in what Cochran-Smith and Lytle (Inside/Outside) refer to a continuous oral inquiry investigation of our mutual classrooms?
In the field of adult literacy there are many provocative issues to explore at various micro and macro levels. There is nothing intrinsic about the size of a study or even its scope which makes it a better research project than another. What is critical, in my view is the importance of the problem identified (including intensive explorations of even single learner's learning history that would have much applicability to the learning history of other students, even if, to use the sanitizing language of positivism, the individual case study is not replicable) and in the nature of the set up of that problem.
We learn a lot about wars by studying individual wars and examining causation and effect as well as even individual battles in detail, perhaps more so than wars in general or through a macro study that would encompass all wars, in which it would be difficult to have more than a survey. In studying three or four wars over a period of time, even if the factors in each are not replicable, we learn a great deal. We learn a great deal particularly both in teasing out the various factors involved as well as through a critical narratively-based analysis of similarities and differences. Quantitative information would be factored in, in, typically, its subordinate function of supplying additional information to the case at hand based on the question under investigation. So it is with adult literacy if we have the collective wherewithal to make it so--a provocative field of sustained investigation in the willingness to probe the interstices of the critically informed imagination in the connection to a wide array of problems that are both interesting and important.
Final two questions:
1. Who owns the field of adult literacy research? 2. Who should?
George Demetrion
Message Six
1-29-05
Functional Context Education: January 29, 2005 An Evidence-Based Approach to Adult Literacy Education Integrating Professional Wisdom With Scientific Evidence
Tom Sticht International Consultant in Adult Education
In 2002, Dr. Grover (Russ) Whitehurst, Director of the Institute of Education Sciences, defined evidence-based education as: "the integration of professional wisdom with the best available empirical evidence in making decisions about how to deliver instruction." He went on to define professional wisdom as "the judgment that individuals acquire through experience" and "consensus views." He noted that "Increased professional wisdom is reflected in numerous ways, including the effective identification and incorporation of local circumstances into instruction."
Functional Context Education is an approach to adult literacy education that had it beginnings in professional wisdom of teachers of adult literacy and later acquired an empirical base of research that has validated and extended the professional knowledge of teachers. Following is a brief outline of this transition of Functional Context Education from being based solely on professional wisdom to being based more on empirical research.
A Basic Tenet of Functional Context Education: The approach to teaching adults to become literate should be based on the life circumstances of adults, not children.
Origins:
1861-1870 The education of freedmen during reconstruction following the Civil War. The teachers developed special materials that tried to reflect the post-slavery circumstances and aspirations of both children and adults. Though aspects of the materials reflected a middle-class conception of how freed slaves should live, the attempt was made to make the materials relevant to the current life circumstances of freedmen rather than impart a typical childhood education in reading and writing using the primers of the schools.
1911 Cora Wilson Stewart started the Moonlight Schools of Kentucky and explicitly stated that one should not teach adults as though they were children. She developed functional materials for teaching reading, writing, and math in the contexts of health, family care, farming, banking, citizenship, etc. for adults.
1917 World War I soldiers were taught reading, writing, and math using materials that incorporated aspects of camp life and military circumstances to make it easier for the men to related their experiential knowledge to the new knowledge they were to gain from book reading.
1943 World War II soldiers were taught reading, writing, and math using military contexts and two fictional characters, Private Pete and his buddy Daffy to help men relate to literacy learning during war time. Testing was introduced to measure learning progress, which was found to occur.
Late 1940’s Frank Laubach created materials for teaching reading in India which incorporated adult themes and concerns such as health and citizenship.
1960’s Paulo Freire developed methods for teaching reading in the functional contexts of adult’s lives and lead them to critical consciousness about their life circumstances and how they might go about changing their situations.
All the foregoing were based on professional wisdom without the benefit of much by way of what would be considered empirical research.
Late 1960’s into 1970’s Army’s Functional Literacy (FLIT) R & D program was first research program that introduced systematic methods for studying literacy practices of personnel in various jobs and job training programs, incorporated these practices into the design of job-related literacy programs, and compared the effectiveness of general literacy programs to job-related programs and found that the latter produced as much improvement in general literacy but three to five times the improvements in job-related literacy, which was what the programs were supposed to do. The FLIT program was not only based on the professional wisdom of earlier adult literacy educators it also incorporated concepts from cognitive science in formulating the practices of "reading to do" versus "reading to learn" based on research in psychology on a human cognitive system with various memory systems, and it incorporated both direct instruction based on behavioral principles of systematic instruction, pre-and post-testing of learning, and progression based on mastery, and instruction of a constructivist nature based on an extensive review of linguistic, computer science (e.g., artificial intelligence), developmental psychology, and experimental studies of reading. The program was externally and independently evaluated by the American Institutes for Research and implemented in several states indicting that the methods were generalizable beyond the R & D site.
1970’s – Present. Various research projects in cognitive science reinforced the ideas making up Functional Context Education principles that were based on professional wisdom at the turn of the 20th century. The principles were officially formulated in 1987 in a book entitled Cast-off Youth: Policy and Training Methods From the Military Experience. Research by Victoria Purcell-Gates and colleagues at NCSALL in the late 1990s confirmed the principle of transfer formulated in FCE and found that programs that used materials from the lives that adults live outside the classroom were more likely to stimulate the transfer of literacy from the classroom to the "real world" of the adults. Numerous projects in India and other nations have confirmed that making materials relevant to the lives of adults promotes greater participation and retention in programs than do academic oriented programs. Recent research in the 2000s using pre and post tests indicates that integrating basic literacy instruction with health knowledge development can produce gains in both general literacy and health literacy.
The foregoing, although an incomplete and informal compilation of a wider base of research relevant to Functional Context Education, illustrates the accumulation of educational knowledge about the teaching of adult literacy based on professional wisdom and scientific research over a period of more than a century.
This body of knowledge has produced at its most generally applicable level a series of principles, not particular materials or techniques, that can guide the development of adult literacy programs. To transform these research-based principles back into sound adult literacy education requires solid professional knowledge, skill, and, yes, wisdom. The cycle of knowledge generation then begins again.
Thomas G. Sticht International Consultant in Adult Education 2062 Valley View Blvd. El Cajon, CA 92019-2059 Tel/fax: (619) 444-9133 Email: tsticht at aznet.net
Message Seven
1-30-05
The findings from the NCSALL Persistence study are exploratory, that is we
are trying to understand this critical issue and then suggest a way to
address it. Once we get to that point, later this year, we will suggest a
series of experiments that would test these approaches to supporting
persistence. At that time our advice would move from "things for you to
try" to "practices that we have evidence for."
On quality, the measure is per-student cost that manifests itself as trained teachers, smaller class size, class grouping based on student needs and goals, managed enrollment, and other instructional components that only become possible when per-student cost is high enough. We found that persistence can increase when programs are improved and provided with sufficient funding, but they also need to provide support services, such as counseling and referral to social services. However, there is a limit to what increasing quality can do for persistence. We found that sometimes increasing quality inputs leads to greater intensity (hours per month) but not duration (months of engagement). So we will suggest two experiments. One that adds a specific set of support services to programs and one that redesigns the approach to participation. That is, our research suggests that a program that allows students to put together episodes of program participation and facilitated self-study might lead to greater persistence and greater learning gains.
Once these experiments are completed (or even now since that will take many years), programs can look at this advice (the best available empirical evidence) and employ their professional wisdom to make this advice work in their program. NCSALL would like to have a way to document and share that professional wisdom. Rather than thinking about the medical model as a pill, I feel the way for our field to think about is as surgery. Two surgeons using the same evidence-based practice can have two very different outcomes (a patient who is dead rather than alive, for example). Good surgeons get together with their peers and share experiences of putting research evidence into practice, and this sharing of experience has led to improved patient outcomes.
Unfortunately, I'm not sure the Department of Education or any of the National Foundations are willing to spend the money to implement the experimental studies that could benefit our field nor to spend the additional funds needed to train practitioners to use that research. However, I think the DOE would be willing to support a limited number of experiments, and a good discussion on this list might be what questions are most important to answer.
John Comings, Director
National Center for the Study of Adult Learning and Literacy
Harvard Graduate School of Education
7 Appian Way
Cambridge MA 02138
(617) 496-0516, voice
(617) 495-4811, fax
(617) 335-9839, mobile
john_comings at harvard.edu
http://ncsall.gse.harvard.edu
Message Eight
1-31-05
"However, I think the DOE would be willing to support a limited number of experiments, and a good discussion on this list might be what questions are most important to answer."
John (and others),
In a recent article titled Postposititivist Scientific Philosophy: Mediating Convergences ,I established a theory of literacy based on a "duly-hedged synthesis" of learning how to read and the progressive mastery of the knowledge that matters gained as a result of learning how to read. The article can be accessed here: http://www.the-rathouse.com/Postpositivism.htm
This definition includes the following four hypotheses:
1. Literacy facilitates knowledge acquisition in the grappling with and mastery of print-based texts.
2. Literacy is enhanced to the extent to which individuals gain the capacity to read and write print-based texts.
3. Growth in literacy is experienced to the extent to which readers progressively comprehend and draw meaning from texts and appropriate them into their lives.
4. Literacy has a technological component in the mastery of reading, writing and the comprehension of texts and a metaphorical dimension that resides in transactions between the reader and the text in which meaning making and significance lies beyond the text into that of appropriation, however variously that may be defined.
For these four hypotheses I have articulated 19 supportive statements, which I would refer to as propositions that require analysis, including that coming from good experiments. All of these statements along with the four hypotheses can be turned into questions, which I will leave to the reader. So as not to get caught up in the details or to dismiss what is here as irrelevant, keep in mind that the largest question I am positing is that on the relationship between learning how to read and learning how to appropriate emergent reading and writing capabilities within the various life contexts in which adult literacy students are engaged. In fact, forget about the 19 statements if you like, but keep the focus on the relationship between learning how to read and the utilization of reading and writing across the continuum of adult literacy provision as an investigative query worthy of much thoughtful probing.
For those who are interested in the 19 statements, see below.
George Demetrion
Message Nine
1-31--05
Debbie Yoho writes: “As a practicing teacher, I found the document insulting, especially the overview and introduction...”; in response to reading Using Research and Reason in Education: How teachers can use scientifically based research to make curricular and instructional decisions, written by Paula & Keith Stanovich, of OISE, and distributed by the Partnership for Reading.
Although I wouldn't presume to speak for Keith, I'm sure that he'd be distressed to learn that you felt insulted. Keith has been one of the premier scholars in literacy over the past two decades, and has won numerous awards and recognitions from the International Reading Association, American Educational Research Association, American Psychological Association, and others. He's been one of those top- drawer researchers who do what they do in order to make a meaningful difference in terms of instructional practices that make a real difference in learners' lives.
He may be best known for his work about a decade back in which he developed a series of measures, most notably, the Author Recognition Test (ART), to assess children, adolescents', and adults' knowledge based upon their reading practices. Across a series of studies, Stanovich showed that those persons who engaged in wide reading (e.g., those who read a lot are more likely to recognize the names of authors) demonstrated larger vocabularies, better comprehension etc. than comparable individuals whose reading was relatively limited. He controlled for differences in exposure to information by assessing what people learned from viewing TV.
His work has been very influential in regards to the significant role of practice in literacy acquisition and has certainly influenced how I think about literacy practices--and what results from engaging in”practice.”
As for the “Using Research and Reason in Education...” paper, I use it in my research methods course (largely taken by practitioners) because I think it puts research and “evidence-based practices” into language that my students can understand. My students seem to agree.
Cecil Smith
M Cecil Smith, Ph.D
Professor of Educational Psychology
Northern Illinois University
DeKalb, IL 60115-2854
mcsmith@niu.edu
www.cedu.niu.edu/~smith/
Message Ten
1-31-05
Cecil,
I'm aware of the expertise that Keith Stanovitch brings both to reading research and scientific-based educational theory. Moreover, I saw a recent book he wrote at Barnes and Noble, then wondered about my ability to critique such a seasoned professional. (I am saying this without an iota of sarcasm). Nonetheless, I have raised some questions about that paper very much along the lines that Debbie had mentioned, which I have placed in an on-line review that can be accessed at: (http://www.nald.ca/FULLTEXT/George/review/cover.htm)
I believe my analysis is balanced in the raising of certain concerns both about the polemics the authors use, and also their definition of science. Although I am clearly an amateur when it comes to scientific theory, I've taken a crack at offering an alternative vision to the neo-positivist perspective in a paper titled Postpositivist Scientific Philosophy: Mediating Convergences, which can be accessed here: (http://www.nald.ca/fulltext/George/post/01.htm).
Unfortunately neither my review of the Stanovitch & Stanovitch paper nor my own essay was subject to peer review, though for what it's worth I did put my best thinking in both pieces.
George Demetrion
Message Eleven
1-31-05
Persistence in English as a Second Language (ESL) Programs:
Research Using the Method of Natural Variations
Tom Sticht
International Consultant in Adult Education
Testing hypotheses derived from theory using experimental research designs with random assignment are extremely difficult to implement in operational settings such as adult literacy education programs. It may also be prohibitively expensive to conduct this sort of "gold standard" research.
For these reasons colleagues and I explored a different approach when we set out to test hypotheses from Functional Context Education theory in an operational context. In this case we followed a method used in sciences such as astronomy and biology and set out to test a hypotheses from FCE theory using the method of examining naturally occurring variations in adult literacy programs.
A basic tenet of Functional Context Education (FCE) theory is that adult literacy education ought to be based on what is relevant to the contexts of adults’ lives. In research on the persistence of adults in Vocational English as a Second Language (VESL) programs, colleagues and I considered the foregoing and hypothesized that the reasons why adults frequently enroll in a course of study and then quickly drop out might have something to do with mismatches between what the adults want to learn and what programs actually offer. So we tested this hypothesis by studying persistence in VESL programs that had only a general focus on work, or a focus on a specific career field, or a focus on a specific job.
Data on persistence showed that , in general, the closer the match between the reasons of the adult students for taking the VESL course, in this case to get a job, and the focus of the program, in this case focusing directly on vocational training and finding jobs for students, the more likely the students were to complete the course.
The method of studying naturally occurring variations in adult literacy programs is considerably less expensive and more feasible to implement than experimental designs. Though not as rigorous as true experiments, the method of natural variation can be used in testing hypotheses derived from adult literacy theory. The paucity of such research may result at least in part from the lack of adult literacy theory that has been formulated explicitly enough to derive hypotheses that are testable.
Theory of this sort is crucial for the development of validated knowledge for the general advance of adult literacy education because it provides guidance beyond the specifics of particulars techniques, materials, technologies, management practices and other operational details of programs. The latter are too numerous and change too often to make it practicable to conduct experimental tests of the relative effectiveness of programs comprised of combinations of such particulars.
As Kurt Lewin said, "There is nothing so practical as a good theory. "
Reference
Sticht, Thomas G.; McDonald, Barbara A.; Erickson, Paul R. (1998, January). Passports to Paradise: The Struggle To Teach and To Learn on the Margins of Adult Education. Online at: http://SearchERIC.org/scripts/texis.exe/scripts/asearch1?db=ericft&expan=no&disp=snote&proximity=rank&lmt=nul&nsz=20&arg=passports+to+passports
Thomas G. Sticht
International Consultant in Adult Education
2062 Valley View Blvd.
El Cajon, CA 92019-2059
Tel/fax: (619) 444-9133
Email: tsticht@aznet.net
Message Twelve
2-1-05
Hello Tom Sticht, and colleagues:
With regard to your note on theory and research, I suggest that you might want to take a look at D. Mertens new edition of "Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative, Qualitative, and Mixed Methods" (2005) from Sage Publications.
Here you will be able to see your own theories, which as drawn from methods commonly applied to natural science data, and which comes from the positivist and post-positivist traditions, juxtaposed to other paradigms and philosophical assumptions underlying research.
These are important to know about for those of us who are doing theory in adult education because they are about human beings, and of course adult human beings, rather than about natural data (in the sense of biology, physics, etc.) and as such require different approaches which are now being explored, and have been for some time, in educational theory in general.
Catherine King
Adjunct Instructor
National University
Department of Education
San Diego, CA
End of Part 1. Continue on at Scientific-basedEdResearchPart2
