Discussion on Evaluating the Effectiveness of PD
From LiteracyTentWiki
Subject: [ProfessionalDevelopment 483] Levels of PD evaluation
From: Taylor, Jackie jataylor at utk.edu
Date: Fri Aug 11 14:55:42 EDT 2006
Professional Development Colleagues:
>From a continuous improvement perspective, I'm curious to know what it takes in order to measure the impact of professional development (PD) in terms of student outcomes and achievement, and to what degree this type of evaluation is even feasible.<br>
In evaluating the effectiveness of professional development, some suggest that the real measure of impact should be on student learning outcomes and achievement, based on what the teacher learns and changes from the professional development in which he or she has participated. Yet, the higher the level of evaluation needed to capture student outcomes of teacher professional development, the less often it is carried out due to factors such as lack of knowing how this can be done, expense, system and organizational factors, and other critical concerns.
For Example: One evaluation model to consider might be Kirpatrick's Four Levels of Evaluation. Some of you may follow other models, and if so, please tell us about your work with them.
Kirpatrick's Four Levels of Evaluation
http://coe.sdsu.edu/eet/articles/k4levels/index.htm
Below you will find a terse sentence describing each level. Whether you are a teacher, program administrator, professional development or state staff, please offer your insights on any of the following questions:
- Do you evaluate at any of these levels? If so, how?
- What recommendations do you have for evaluating the effectiveness of professional development at each level?
- What are specific considerations for each level?
- What specific, additional resources are needed at each level?
- What are the concerns for evaluation at each level?
Level 1 Evaluation - Reactions: Evaluates participants' reaction to the training
Level 2 Evaluation - Learning: Assesses the participants' learning that has occurred due to the training (pre/post)
Level 3 Evaluation - Transfer: Measures the changes in participants' behavior that has occurred due to the training program
Level 4 Evaluation - Results: Measures the bottom line (from a business standpoint): Does teacher PD affect learner outcomes and achievement?
Thanks for your thoughts on this ~ Best, Jackie
Jackie Taylor Adult Literacy Professional Development List Moderator, jataylor at utk.edu
Subject: [ProfessionalDevelopment 484] Re: Levels of PD evaluation
From: David Rosen djrosen at comcast.net
Date: Fri Aug 11 15:12:57 EDT 2006
Jackie,
You have raised an important but frustrating issue for professional
developers and researchers. Of course, as professional developers,
we want to know if and how what we do with teachers makes a positive
difference in their teaching, and in learner outcomes. But as the
limited amount of research on PD in K-12, and the dearth of research
in adult literacy education PD suggest, we don't have many good answers.
The Kirpatrick model, while well-known, in my view misses an
important assessment step in the teacher change process, perhaps
between Levels 2 and 3, what I would call assessment of changes in
thinking, approach or paradigm. High quality professional
development often challenges teachers to think differently, and then
to make changes in what they do. Training focuses on changes in what
teachers do, not necessarily on why to do things differently.
David J. Rosen
newsomeassociates.com
djrosen at comcast.net
Subject: [ProfessionalDevelopment 484] Re: Levels of PD evaluation
From: Lynda Ginsburg ginsburg at rci.rutgers.edu
Date: Fri Aug 11 16:12:11 EDT 2006
David and Jackie,
You both make good points. I also agree that often, one of the
educational goals (rather than "training goals") of professional
development is for teachers to walk away with a new or revised vision of
instruction, a new lens through which to look critically at what they
have been doing. Evaluating the development of such growth is not easy
but may show impact in ways that are not otherwise evident.
You may also be interested to know that Kirkpatrick's evaluation model, which was designed for business and industry, has also been modified by Thomas Guskey who added a level between #2 (Participants' learning) and
- 3 (Participants' use of new knowledge and skills). He says that in
educational settings, teachers' implementation of change may well need
to be preceded by Organization Support and Change. While he was
addressing K-12, the same holds true for adult education. A culture of
support, trust, a willingness to experiment, availability of resources
and respect for teacher knowledge will go far towards enabling growth
and change.
Lynda
Subject: [ProfessionalDevelopment 485] Re: Levels of PD evaluation
From: David Rosen djrosen at comcast.net
Date: Fri Aug 11 17:01:08 EDT 2006
Hello Lynda,
Thanks for the tip about Guskey. I looked up his work on the Web,
and found this:
http://www.gse.harvard.edu/hfrp/eval/issue32/qanda.html
Guskey has this interesting thought about the five levels:
"...many educators are now finding how useful it can be to reverse
these five levels in professional development planning. In other
words, the first thing people need to do when they plan professional
development is to specify what impact they want to have on student
learning. They begin planning by asking, “What improvements in
student learning do we want to attain and what evidence best reflects
those improvements?” Then they step back and ask, “If that's the
impact we want, what new policies or practices must be implemented to
gain that impact?” Next, they consider what types of organizational
support or change are needed to facilitate that implementation, and
so forth. This planning process compels educators to plan not in
terms of what they are going to do but in terms of what they want to
accomplish with their students. All other decisions are then based on
that fundamental premise."
Professional development and program improvement should go hand-in-
hand. To get improved learning outcomes, a very important purpose of
an education program, we need capable teachers; a teaching
environment which supports professional growth; a learning
environment which provides good learning and teaching resources; good
student support services like counseling, childcare, and regular
access to state-of the-art technology; a climate of experimentation;
and a process for systematically collecting and using data on program
improvement and professional development goals.
One might argue that it is a waste of resources to evaluate -- or
even offer -- professional development when, after top notch, in-
depth professional (expensive) development courses teachers return to
programs where what they have learned to do is not respected,
welcomed, or supported, where it is not an integral part of planned
program development. We need to join program improvement and
professional development, so they walk hand in hand in measured
steps toward improved learning outcomes.
I would like to see a national foundation or government agency make a
serious commitment to multi-year adult literacy program improvement,
a commitment which would include selecting programs that had a solid,
basic level of resources and whose administrators and teachers wanted
to make significant changes to attain significantly improved learning
outcomes. This commitment would need to provide the additional
resources they needed to identify their program improvement and
professional development goals, make the needed changes, and annually
evaluate their progress. The Kirkpatrick/Guskey five level
assessment model -- and the sixth level I have suggested, where
changes in teacher perceptions are also assessed -- might then make
sense for our field, and might give us some good answers to our
questions about improving professional development and program
improvement strategies.
David J. Rosen
djrosen at comcast.net
Subject: [ProfessionalDevelopment 486] Re: Levels of PD evaluation
From: Corley, Mary Ann MCorley at air.org
Date: Fri Aug 11 18:43:36 EDT 2006
Hi, David:
Don't know whether this list will accept attachments, but I'm going to
try to send this out. We use Guskey's five levels of evaluation in
CALPRO work--and find it quite helpful.
I'm attaching a chart of the five levels--taken from an issue of
Educational Leadership a few years back. It's a pretty nifty chart. Hope
it's useful to you.
Thanks,
-Mary Ann Corley
CALPRO Director
American Institutes for Research
Subject: [ProfessionalDevelopment 491] Guskey's Levels and the "bottom line"
From: Taylor, Jackie jataylor at utk.edu
Date: Tue Aug 15 15:52:53 EDT 2006
Hi David, Lynda, Mary Ann and All,
David, I also find that evaluating the effectiveness of professional development can be frustrating. But I feel it is important to raise the matter and explore the issues - both on the surface and underneath -- so as a list community, we might benefit from each others' expertise. While Kirpatrick's model provides a starting place, I did not intend to imply that business training models of evaluation were the end-all for evaluating learning from professional development. That said, I also felt similarly as you when I looked at Kirpatrick's model, and I'm glad you raised the issue - it examines changes in behavior without considering changes in thinking.
In looking at Guskey's model (and thanks, Mary Ann, for the grid!) it appears Guskey does intend to capture changes in thinking, where (under the column header "How Will Information Be Gathered"), it suggests to capture written reflections of teachers at both the level of learning (Level 2) and * level of transfer/use of new knowledge and skills (Level 4). But I like placing the emphasis on changes in thinking that David suggests -- by making it a separate level of its own. So I wonder where others think such a level might belong? And whether such a level would be placed before or actually after student learning (when do you suggest transformative * teacher changes in thinking really occur? And in what ways might that bring the bottom line of professional development back into question)?
Hmm...so my initial questions to the group remain the same. In looking at Guskey's Levels of Evaluation:
Level 1 - Participants' Reactions
Level 2 - Participants' Learning
Level 3 - Organizational Support and Change
Level 4 - Participants' Use of New Knowledge and Skills
Level 5 - Student Learning Outcomes
- Proposed Level - Changes in Thinking
- Do you evaluate at any of these levels? If so, how?
- What recommendations do you have for evaluating the effectiveness of professional development at each level?
- What are specific considerations for each level?
- What specific, additional resources are needed at each level?
- What are the concerns for evaluation at each level?
Thanks in advance for your thoughts on any of this ~ Jackie
Jackie Taylor, Adult Literacy Professional Development List Moderator, jataylor at utk.edu <mailto:jataylor at utk.edu>
Subject: [ProfessionalDevelopment 492] Re: Guskey's Levels and the "bottom line"
From: Judy Judy at ripolicy.org
Date: Tue Aug 15 16:07:22 EDT 2006
Here are two additional articles that I found really interesting on
measuring the effectiveness of PD looking at the link between teacher
professional development and student outcomes. The Whitehurst article is
particularly interesting around what impacts student outcomes the most.
Perhaps 'thinking' was left explicitly off the list because it seems
that 'what have you learned' and 'what are you using' are far easier
questions to answer and to study than 'what are you thinking'.
Meiers, M. & Ingvarson, L. (2003). Investigating the links between teacher professional development and student learning outcomes. Australian Council for Educational Research. Retrieved at: http://www.dest.gov.au/sectors/school_education/publications_resources/profiles/teacher_prof_development_student_learning_outcomes.htm
Whitehurst, G.J. (2003). Research on Teacher Preparation and Professional Development. Presented at the White House Conference on Preparing Tomorrow's Teachers. Retrieved at http://www.ed.gov/admins/tchrqual/learn/preparingteachersconferenc/whitehurst.html
Judy Titzel
Providence, RI
Subject: [ProfessionalDevelopment 493] Re: Guskey's Levels and the "bottomline"
From: Grebow, David David_Grebow at ctb.com
Date: Tue Aug 15 17:56:07 EDT 2006
Hi All -
I look at the issue from a very different perspective.
First, I think the levels are not really "levels" but points along a
continuum. The continuum is broken into 3 key chunks: I Know; I Can Do;
I Can Adopt and Adapt.
I Know is about 5% of the way there; I Can Do goes another 20%; and I
Can Adopt and Adapt takes the rest of the time on the learning journey.
That's why I measure the effectiveness of any PD program as a person
progresses over time from new learning to ability to tutor - the point
at which they can teach the new learner (and hopefully add new learning
back into the mix). It's more time consuming that using any of the
levels and looking for outcomes in the short term, yet I think that is
at the heart of the problem - outcomes really occur and can be truly
measured over a much longer period of time than many of us have time
for.
As for measuring thinking ... measuring thinking is like looking at the
wind. You can't see IT but you can watch the leaves move - an outcome of
IT. So I believe that the outcome of changes in thinking are not
necessarily changes in behavior, but the observable and measurable
ability to adopt and adapt what one has learned in the I Know and I Can
Do stages under a wide variety of ever-changing circumstances. Your
behavior may not necessarily reflect what you are thinking, but your
thinking will always indicate whether or not you can adopt and adapt
what you learned. By inference, your thinking has changed and you have
learned something new.
If this helps there's a whitepaper I wrote that goes into more detail at
http://www.agelesslearner.com/articles/watercooler_dgrebow_tc600.html .
Cheers,
David Grebow
Subject: [ProfessionalDevelopment 494] Re: Guskey's Levels and the "bottomline"
From: Marie Crecca-Romero creccaromero at cox.net
Date: Tue Aug 15 23:38:00 EDT 2006
Judy,
I was not able to open either link. Do you have copies?
Marie
Marie Crecca-Romero
Director
Transition to College
401 722-9800
www.transitiontocollege.org
"Our mission is dedicated to improving the lives of adult learners through the pursuit of higher education."
Subject: [ProfessionalDevelopment 496] Re: Guskey's Levels and the "bottomline"
From: Taylor, Jackie jataylor at utk.edu
Date: Wed Aug 16 17:07:38 EDT 2006
Hi Judy and all!
Judy -- thanks so much for the resources, I look forward to exploring them. Interestingly, both web sites have since moved the documents (go figure). So I've done a quick search and here are the new links I've found. Please let us know if these are not the correct links.
Meiers, M. & Ingvarson, L. (2003). Investigating the links between teacher professional development and student learning outcomes. Australian Council for Educational Research.: http://www.dest.gov.au/sectors/school_education/publications_resources/profiles/teacher_prof_development_student_learning_outcomes.htm
Whitehurst, G.J. (2003). Research on Teacher Preparation and Professional Development. Presented at the White House Conference on Preparing Tomorrow's Teachers: http://www.ed.gov/admins/tchrqual/learn/preparingteachersconference/whitehurst.html?exp=0
Do others have resources to share that you have found especially valuable on evaluating the effectiveness of professional development? Please post them to the listserv, and let's start a compilation that can serve as a resource for us all ...Jackie
- ADD YOUR THOUGHTS HERE!
