LPRPArchivesScienceBasedResearch
From LiteracyTentWiki
- Back to LPRPArchivesResearchParadigms
3/30/05
Here is a short article that ran in last week’s Education Week about “scientifically based practice.” It covers a lot of ground in four short pages, and I recommend it (though of course that doesn’t mean I endorse any or all of the viewpoints expressed). There is no mention of adult education, so it may be productive to take up certain points with regards to areas of adult education you have interest in and post them to the list.
For example, during a training I was doing a few weeks ago that was based on a NCSALL study of student persistence, participants in the workshop wondered about the student population in the study. Not out loud, for the most part - it was more of a subdued buzz about using research based on students from Massachusetts in the context of adult education California. I didn’t have a lot of time to go into it (since it wasn’t the point of the workshop), but perhaps this list is the right place to bring this buzz out into the open. What kinds of questions do you have about student samples used in research? What kind of sample do you think you need to make it applicable in your own state? What kinds of things would make you think the student sample is too different to have any relevance to your own students?
Erik Jacobson
--
3/30/05
It still depends on the nature of the problem one is focusing on. Consequently, I would not make an essential correlation between scientifically-based practice or research and that of sampling. Depending on what the question is, random sampling may be very important, for example, how many adults between 25-44 with incomes between $30,000-$55,000 watched the last program of NYPD Blues. On the other hand, if the question was to describe the major changes in personality and character development of Detective Andy Sipowitz during the 12 year history of the program and account for the major reasons for these changes, that wouldn't require sampling at all. Moreover, a valid sample requires a tight control over intervening variables, which shouldn't be many, which works against the inclusion of complexity in a given study.
There may be some questions of a quantitative type that may be relevant to our field where a random sample is the key factor. But say the question is, how and in what ways do adult students at varying levels of literacy aptitude learn to read, I would rather rely on in-depth and comparative (but not at the level of a sample or even quasi-sample) analyses of case study study--in short, critical ethnography. Then I would want to subject whatever findings and analysis emerge to even wider scrutiny. The more pressing research task (IMHO) would be in-depth analysis of about 10-15 typologies consisting of different constellations of students supported by a wide array of relevant documentation and types of scrutiny called for by the nature of the problem. As one broke down the key components of the problem, there may be a sub-facet or two of the broader question wherein most likely, a quasi-random sampling would be of value, but if one is going to probe into the interstices of the problem this type of positivist research may be of the least of importance. Then if the knotty issue of motivation is a significant factor in a particular problem-focus that even compounds matters further from any predominant positivist /neo-positivist perspective.
In terms of scientific-based reasoning and methodology, a few words from scientific philosopher, Karl Popper (1959) from The Logic of Scientific Discovery may be in order. Here's the bullet statement, which Popper applies to science as well as to philosophy:
"Philosophers [and scientists] are free as others to use any method in searching for truth There is no method peculiar to philosophy" or science (p. xix)
"The method I have in mind is that of stating one's problem clearly and of examining its various proposed solutions critically (ibid.).
In short, the question, in my view, is not what type of samples used in research, but first and foremost, what is the problem that you are researching and secondarily, why are you defining that problem as significant? Then, depending on the problem focus, methodologies, the validity of data, types and levels of analyses, types of hypotheses formations designed to push the research forward, follows.
What I would be most concerned about is only selecting problems that are susceptible to sampling and defining only that as legitimate science. Without even looking at the politics behind any such schema, from a scientific point of view alone, that is at best an impoverished type of science and at worse voodoo science masking something even more pernicious.
I know I've highlighted this piece elsewhere, but I think there is at least some merit in the postpositivist approach that various researchers, including me, are advocating for. My contribution, a theoretical focus, to be sure, can be obtained here: http://www.the-rathouse.com/Postpositivism.htm
George Demetrion
--
3/30/05
I read this article and, on the whole, thought it made sense and was appealingly short and clear. I want to explore one sentence in the article. What I am about to write has been simmering awhile and this sentence set it to boil. The sentence is:
"We need to create an appetite for research findings."
That suggests a "marketing" or "dissemination" approach. I see these having had small success in our field for many reasons. Our federal and state public agencies don't seem to provide much support for dissemination or marketing, even of products whose development they have funded. Even if they did, it's tough to sell "research" to practitioners in our field. But most important, this sentence -- unlike some other parts of the article -- seems to imply that researchers set the questions, then spend a lot of time, money and expertise on doing the research, and _then_ "create an appetite" (presumably teachers' appetities) for the research.
But do not adult literacy education teachers already have a good appetite for knowledge about teaching? Every day, teaching surfaces questions about how to be more effective in helping students learn. Every day teachers want answers -- funded knowledge or solid professional wisdom would be ideal, but it would help to even good advice from someone who has more experience -- advice on how to think about a particular student's learning difficulties, what to try, what works and doesn't work with what kind of student under what kind of circumstances, and how to know if it's working. The kind of research that the Administration is promoting -- gold standard -- may ultimately be useful -- but it won't answer many of the hundreds or thousands of important questions that teachers have. If the goal is to significantly improve teaching practice -- to get better results for students -- and if we are relying on gold standard research as the solution to that problem, we may have to wait several centuries to see practice improve.
In our field we don't often have in place all the necessary things in between front-line teaching or tutoring and gold standard research. We don't have much in the way of research at all, and the way things look we might have less. In most states we don't have many adult literacy education resources devoted to gold standard supervision or gold standard technical assistance (i.e. one-on-one in-service training) not to mention gold standard professional training in the basic skills we should expect of adult literacy education teachers: teaching reading, writing, and numeracy, and teaching students who have learning disabilities. I wonder how many reading this could say that at the program where you work -- or have worked -- inexperienced teachers routinely have the training they need to teach reading, writing and numeracy (and science, social studies and other adult secondary education areas, and...ESOL... and .... How many could say that routinely there was experienced supervision in place for new teachers in all these areas?
In adult literacy education we have big missing pieces of a system for reaching high standards of quality. And although gold standard research may be one missing piece, it isn't the most important one. The first one I think of is how to help teachers get answers to the burning questions they have about students who aren't learning. In my experience, teachers are ravenous for that knowledge, and if we have professional wisdom based on some good evidence (not necessarily gold standard research) and if that knowledge, when applied, proves to work -- that is, the students learn -- then we are more likely to improve the quality of teaching in this century.
So, instead of looking at how to create an appetite for research I would ask: How can we help teachers get good answers to their "frequently burning questions" (FBQ's) about effective teaching?
There is an important role for research in this, but whatever research might be planned will be more useful if it answers teachers' and tutors' FBQs.
David J. Rosen
-- 3/31/05
Thanks David,
To highlight just one paragraph of your thought provoking message:
"But do not adult literacy education teachers already have a good appetite for knowledge about teaching? Every day, teaching surfaces questions about how to be more effective in helping students learn. Every day teachers want answers -- funded knowledge or solid professional wisdom would be ideal, but it would help to even good advice from someone who has more experience -- advice on how to think about a particular student's learning difficulties, what to try, what works and doesn't work with what kind of student under what kind of circumstances, and how to know if it's working. The kind of research that the Administration is promoting -- gold standard -- may ultimately be useful -- but it won't answer many of the hundreds or thousands of important questions that teachers have. If the goal is to significantly improve teaching practice -- to get better results for students -- and if we are relying on gold standard research as the solution to that problem, we may have to wait several centuries to see practice improve."
I can't think of a single better way to probe into these issues than through critical listserv discussion, and then subjecting that to broader analysis that draws in more traditionally-based research in a a manner that fosters a continuous stream of critical and thoughtful engagement between practitioners and more formal academic scholars of whatver disciplines relevant to the problem at hand. Assuming there is an open-ended nature to human understanding and knowledge acquisition, the interface between more academic-like theorists and researchers, and critically informed practitioners, needs to be at the core of the probing, ideally, in my view, through a relatively small set of highly important questions which then become subject to a more intensive systematic investigation than that which typically takes place.
On the former, the current discussion on the relevance of the GED taking place on NIFL-FOB is the type of finely honed problem that then can be subjected to this broader type of investigation which I think is crucial, particularly if those on or close to the ground are going to have a significant influence on (a) problems identified for invsestigation, (b) and on the definition on what consists legitimate research and theory construction in which there is considerable contestability. On any gold standard, the only gold standard that I can conceive of is the articulation of a highly significant problem and the utilization of whatever methodologies, sources of evidence, tentative hypotheses, experiements, etc. that shed the most light on the hard work of proximately resolving the problem.
BTW, on my earlier post, I was not referring specifically to the article mentioned, but to the broader issue as to the definition of scientific-based research and the extent to which experimental/quaisi-experimental design is a core and essential dimension to the definition, which, obviously I argue, that it isn't. On that assumption I reject any association of the gold sstandard with the rigors of random sampling, which may or may not be relevant to any given problem.
George Demetrion
--
4/03/05
The only part I thought was at all worth while was the section that discussed how teaching and research are both neccesary for scientifically based research. But is this news? I mean the scientific method is that you create a hypothesis and test it. Isn't the classroom the educational laboratory?
I have to say that this article is obviously not written for teachers, but rather for those who have resposibility for whipping us into shape. Thus the condescending tone of the article in general towards teachers which implies that research is written in "forms that only other researchers comprehend" and that we need our "appetite for research findings" whetted (sounds kinda like the tobacco companies to me). Also, the ludicrous idea that "basing teaching on research or evidence is not part of the culture of teaching," is absurd. What do teachers do if not learn from each other? Teaching is one of the few professions where you actually watch someone doing it well (and badly if your student teacher placement folks are on the ball) in order to learn how to do it. I discuss my teaching practice daily with people (and so do most of the teachers I know). However, I think these guys are actually describing the teaching practice of many university professors and academics -- which is why I got out of academia and returned to "regular" teaching.
And please...is it news that the educational system is product driven?
I don't think the article says anything new. It is just more pumping up the academy at the expense of teachers who are seen as ignorant. Until the academy realizes that they have something to learn from teachers, we won't get anywhere.
Michele Craig
