Descriptions of Data Use by Guest Panelists
From LiteracyTentWiki
Back to Assessment Information
Back to Using Data for Program Improvement
Back to Full Discussion by Threads
Descriptions of Data Use by Guest Panelists
Good morning, afternoon and evening to you all.
Today begins our discussion on Using Data for Program Improvement. I have pasted the announcement below – please note that there have been some edits to Guest bios. Also, I am trying to send 4 attachments (they are power points) but I’m having a tough time getting them through the server. For now, you have the announcement below and as soon as I arrange access to the Power Points, I’ll let you know. If you received the original announcement that I sent, you have one of the attachments already (“Using Data Effectively DCornellier”). Thank you for your patience with this.
Also, I would like to acknowledge that today is Patriot’s Day and is celebrated in many corners of the United States. Some of our guests, as well as subscribers, may not be present on-line with us today and this is fine – they will catch up with us tomorrow. For anyone ready to begin, please feel free to post your messages.
I’ll start us off by asking our guests to briefly describe how they use data in their work to improve literacy services. Subscribers, please post your questions and share your own experiences using data. What type of data would you like to track and why?
Thanks!
Marie Cora
Assessment Discussion List Moderator
Good morning Marie. Thanks for giving me the opportunity to share my ideas with the members of this list. You asked, “I’ll start us off by asking our guests to briefly describe how they use data in their work to improve literacy services.”
I will begin my comments from my perspective as state agency staff member. In my opinion, if we (at the state office) want local programs to utilize data for program improvement, we have to continuously model those behaviors at the state level. We have used data to:
- inform the development of our policies and recommendations;
- highlight program practices that need attention;
- target providers for on-site monitoring;
- identify high-performing providers to learn from, about strategies that work;
- provide technical assistance and feedback; and
- enhance our professional development model.
Ajit Gopalakrishnan
Connecticut Department of Education
25 Industrial Park Road
Middletown, CT 06457
Phone: (860) 807-2125
Fax: (860) 807-2062
Email: ajit.gopalakrishnan@ct.gov
Hi Everyone,
It is a pleasure to be a guest on the list this week and my thanks to Marie for asking me and organizing this.
There is a strong federal initiative to promote use of data for program improvement at the state and level. Through the National Reporting system project which I direct, we have conducted several training and technical assistance activities over the past 4 years on this topic, including two general training seminars on using data and more specific ones on promoting adult education programs, monitoring, developing state and local report cards. All of the training materials and other information on the topic, including sample work from states, is available on the NRSWeb website, which Marie has referenced.
All of the other guests have done a great deal of interesting work and many of them attended our training (and Sandy Strunk served as a trainer for us a few years back).
I will be interested to get your questions and learn of your experiences, as well as the responses from the other guests.
Larry Condelli
Good afternoon, everyone, and many thanks to Marie for putting this panel together. Using data for decision-making has been a passion of mine for some time. I think by nature, I’m just one of those people who always has questions about things – and I love testing my assumptions about how the world works. It started some years ago when Pennsylvania first began its work with Project EQUAL (Educational Quality for Adult Literacy). Judy Alamprese was our professional developer and she challenged each of our sites to pose a question related to program improvement, collect some data related to the question, analyze that data, and then develop a program improvement plan based on our data analysis. I took to the process like a fish to water.
I still remember the first teacher I walked through this process with. She was a beginning ESL teacher – one of our best. She wanted to know if she was teaching the sorts of things her students were most interested in learning. This sounds fairly basic, but when none of your students speak English, it’s a challenge to know if what you’re teaching is what they most want to learn. To collect our data, we partnered with the Advanced ESL class and had them translate some basic questions into nine languages. At that time, our beginning curriculum was based on work skills and basic communication for the workplace. What we learned was that this particular cohort of students wanted to learn about shopping and healthcare – work skills were at the bottom of their list! More importantly, we learned that our assumptions about what students want and need to learn are not always accurate. We were teaching job prep when they wanted to know how to order a quarter pounder with fries (we’ve since started a health literacy unit, as well ;-).
That was the start of a program improvement process that we have used in our local program ever since. We have a program improvement team – which most years, is representative of all facets of our adult education program. I say most years because this year, for the first time, the program improvement team is limited to site managers and supervisors who have been working together to develop an ongoing progress monitoring system. Our progress monitoring system provides our learners with individual written feedback on a quarterly basis related to their actual attendance compared to the number of class hours available, achievement on standardized assessments they have completed, and goal attainment related to what they told us they wanted to achieve when they enrolled. This is the first year that we have shared written progress data with our learners in this manner. If using data for decision-making is a powerful tool for changing what we do, we wondered what impact the data might have on our learners. Our assumption is that it will have a positive impact on student retention, but implementation isn’t stable enough yet to do a comparison study. I’d love to know if anyone else is doing this sort of written student feedback and what they’ve learned as a result.
Other ways we use data? Well, we always have one or two action research or inquiry projects running. Last year, two of our staff did a great action research project on the question, “Why do some of our students complete our orientation process but never reach enrollment status?” When this question came up in our program improvement meetings, we all had opinions (our team is never at a loss for opinions). Some speculated that “childcare and transportation” are the issues. Some of us were quick to suggest that “childcare and transportation” are the universal retention scapegoats for our field. Others felt that quality teaching was the issue. Still others suggested that the problem rests with the motivation of our learners. Without data for decision-making, we would have no mechanism for moving this discussion beyond the opinion stage.
We also collect customer satisfaction data twice a year. One day in the Fall and one day in the Spring we survey everyone who is in class with a simple instrument that rates the student’s satisfaction with the classroom environment, the teacher, instructional materials, and goals. In fact, we just finished our Spring cycle and last week I got the report comparing our Spring 2007 student satisfaction numbers to our Spring 2006 numbers. Here’s an interesting snippet – 69% of our family literacy students (N=42) strongly agree that they can use what they learn in class at home or at work. 58% of our ESL students (N=263) strongly agree. 27% of our ABE/GED students (N=150) strongly agree. Another interesting tidbit – 76% of our family literacy students strongly agree that the teacher starts class on time. 86% of our ESL students strongly agree. 59% of our ABE/GED students strongly agree. Well, as usual, one question always leads to another. I’m not exactly sure how to make sense of these numbers, but my next step will be to look at the disaggregated data by classroom. The good news is that all of these percentages are up from last year. Either we’re doing better or we have an especially agreeable cohort of learners.
I think the point I want to make is – for us, data for decision-making is tied to our constant curiosity about the work we do. Yes, we routinely look at our performance against state standards – but that’s a routine part of our work. The more interesting investigations tend to stem from something we notice and wonder about. Like – do ABE/GED teachers in our program really start class later than ESL teachers or does it just seem that way because so many of them start the day with individual work rather than group lessons?
Sandy Strunk
Program Director for Community Education
Lancaster-Lebanon Intermediate Unit 13
1020 New Holland Avenue
Lancaster, PA 17601
(717) 606-1873
I am, like Sandy Struck, a product of Pennsylvania's program improvement process, Project Equal, and I was for a number of years a trainer in using data for decision making. The heart of this training was to help programs to describe, in detail, an area for improvement within their program, to ask a question based on that particular area of concern, to look at program data that related to the problem area, to come to some conclusions based on the data, and then to take actions that would result in program improvement. This sounds very simple, but in fact, it was a hard road for all of us, and there were difficulties at every step.
In the beginning people (most of us) tended to ask questions that were too broad (or too narrow) or too vague, and we tended to look at aggregated data only --and even when there was data enough to draw conclusions from, our action plans often seemed to have little relation to those conclusions. In short, learning to use data for program improvement was a surprisingly slow process, and involved the creation of a habit of mind that was not at all "second nature" to most of our program directors and their staffs.
I think that our tendency now is to think that using data for making program decisions is just common sense--but I think it's a more complex issue. And one that has implications for professional development at all levels.
I would say that over time some of the programs I worked with developed the habit of using data for decision making, and that others reverted back to decision making by intuition--as Sandy said there's rarely a dearth of opinions in our programs.
Karen Mundie
Associate Director
Greater Pittsburgh Literacy Council
100 Sheridan Square, 4th Floor
Pittsburgh, PA 15206
412 661-7323 (ext 101)
kmundie@gplc.org
Good morning from Rain soaked Boston. One area that we use data to improve literacy services in our program is looking at attendance data. In Massachusetts, we have a DOE web based system that allows us to view class attendance data, as well as other pertinent data that gives us tools to improve literacy services. I review the data and look at each individual class to see how attendance is. If attendance is low one month I then review what happened. For example: Was there inclement weather? Or have any natural disasters occurred in homelands of students? Was the teacher absent for a period of time? If yes, I know that is an outside factor. But if I see that attendance is low for more than a month, I investigate. Ask the program advisor to review calls to students when absent to find out the reasons given for dropping out and share the student feedback with the teacher and ask for her/his opinion. If the overriding case involves students feeling lost over what is being taught, the teacher opens discussion about the lessons and works with the students on the topics. Students feel empowered in their learning and attendance improves.
Toni F. Borge
Adult Education & Transitions Program Director
Bunker Hill Community College
Chelsea Campus
175 Hawthorne Street
Chelsea, MA 02150
Phone: 617-228-2108
Fax:617-228-2106
E-mail: tborge@bhcc.mass.edu
Good Morning Everyone!
I am sorry to be so late joining the group. I am one of the panelists, Rosemary Matt. Just recently, I accepted the position of NRS Liaison for New York State so monitoring data and providing technical assistance to programs in need is now my entire focus. We also have a large contingent of programs that provide service through a volunteer network in a one to one tutoring arrangement. Your concerns Mary (see posts by Mary G. Beheler in “Searching for Usable Data”) regarding the inability for programs such as these to meet performance benchmarks is shared by New York programs as well. Our state department has thoroughly considered the population these folks serve and consider that to be a mitigating circumstance when assessing their performance. The value of these organizations serving some of our lowest skilled readers is well known and appreciated. In a state the size of New York it is possible to absorb the lack of educational gain increments from these agencies as they are balanced by other programs serving students for whom gain is eminent.
At the same time however we have worked closely with these programs and their statewide leadership team to provide technical assistance in the area of assessment. As they learn more about the strategies and nuances that evolve around the NRS accountability system, they are better able to show whatever gain is possible from their students.
As some of you are aware, New York also utilizes the program level Report Card. We attended the training two years ago that was provided by Larry and his staff at AIR. I would strongly recommend this training to any state considering this accountability tool for programs. We have made incredible advances in terms of identifying high performing programs and targeting those in need of technical assistance through our Report Card Rubric. Marie has posted three power points that I offer in training built around this rubric. To further support our volunteer programs, our state department has chosen to rank these programs among themselves providing a homogeneous category specific to their needs. They are not measured against the cohort of traditional adult education programs.
Another strategy we have recently embarked upon is through our statewide data system, we have introduced Collaboration Metrics. Many students working first with these volunteer programs while they are at minimal skill levels will eventually move into traditional programming and continue to succeed through the educational levels. To ensure the volunteer programs remain tied to the student's success, they are informed of the students’ progress through the data system and can subsequently report on that gain as well.
These few methods of support have been well received by our volunteer affiliates. Hope they may give you and your state some thoughts for the future.
Rosemary I. Matt
NRS Liaison for NYS
Literacy Assistance Center
12 Meadowbrook Drive
New Hartford, NY 13413
315.798.1026
Hi,
In Massachusetts we have developed and implemented a plan for ongoing staff development on using data to promote continuous improvement. Our state-wide professional development system, (SABES) has developed a program planning process that incorporates NRS and other data to promote continuous improvement. They utilized the following approaches to providing support for program planning: (1) a comprehensive 12-hour course, offered in all five regions of the state, on planning for program improvement, including a module on types and sources of data, data quality, and data analysis. The course culminates with presentations by participants on their program planning activities. (2) On-site coaching to selected programs in need of a tailored approach (3) A separate data module from the planning course presented twice as a separate workshop (4) Program and staff development sharing groups to provide forums for directors and practitioners to share experiences. Follow-up for all participants who attended this training is provided. Programs are now required to submit program improvement plans that are tied to their performance in attendance, average attended hours, pre- and post- testing, learning gains and eventually, the achievement of student goals. SABES provides ongoing support to programs in developing their continuous improvement plans. We have found that offering more intensive courses for several staff members at local programs has been very helpful for our local programs.
Donna Cornellier
SMARTT ABE Project Manager
Adult and Community Learning Services
Massachusetts Department of Education
350 Main Street
Malden, MA 02148
781 338-3814
Hi all:
I wanted to chime in about our program’s use of data since this is the focus of our discussion. Coincidentally, I am in the process of writing our proposal for next year, so I am knee-deep in data even as we speak!
The use of data takes many forms in our program. We look at what most people consider the “hard data” -- the raw numbers with regard to attendance, learner gains, retention, goal attainment, etc. We believe; however, that the numbers alone provide an incomplete picture of what is happening, so we use the numbers as a basis for discussion, not decision making. After analyzing the numbers, we begin to look at additional sources of data that we find essential in informing our planning---meetings with staff, classes, our student advisory board, and focus groups.
Here’s an example we’re currently working on---we did a two year analysis of learner retention, and began to document why students did not persist. We found that the retention for students who enrolled after January 1 (our programs runs on a school calendar year from September to June) was significantly lower than the retention for students who began in September. Even more compelling, we learned that the retention for students who began after March 1 was 0%.
We met with staff and students, and did some research around student retention issues. After a year-long process, we decided to pilot a “managed enrollment” approach. In Massachusetts, our grantor (MA DOE) allows us to “over-enroll” our classes by 20%, so we enroll 20% more students in the fall. When students leave, we “drop” the overenrolled students into funded slots. This allows us to keep the seats filled even with the typical attrition that occurs.
In January, when we do our mid-point assessments; we move students to the higher level who are ready to progress….that typically leaves several openings in the beginner levels and we begin students in February as a cohort. This year, we implemented new orientation programs including a requirement that new students observe a class before enrolling.
While it is still too early to tell if these new procedures will have a positive impact, we are hopeful and we know anecdotally that the transition seems to be easier for some of these students. We are eager to look at the data at the end of the year to analyze the effectiveness of this plan.
As we begin to look at our data, we are finding that there seem to be a unique set of issues for our beginner ESOL students. We suspect that the lack of effective English communication skills to advocate for themselves with employers is influencing their attendance and persistence. This is an issue that we are beginning to tackle in terms of policy. Do we need to have a more flexible, lenient policy for beginner students? Is there a way to support students in addressing these employment issues? How can we empower students more quickly? Are there other issues for these beginner level students that affect their participation? As we enter these discussions, the numbers will provide a basis for developing strategies, but the students themselves with be our greatest source of valuable data.
Luanne Teller
Director of ABE (ESOL) and Transitions to College Programs
Massasoit Community College
Stoughton, MA
Hi Luanne,
I find it interesting that what you are finding in data seems to be consistent with what we see in our GED classes here in Arizona. Often the last group who enter in March are the least likely to stay with the program until posttesting, and the August group seem to have the highest posttesting and retention rate.
Tina Luffman
Coordinator, Developmental Education
Verde Valley Campus
928-634-6544
tina_luffman@yc.edu
A few posts ago, Luanne spoke to her concerns with retention particularly as students entered at the later portion of the fiscal year. Luanne, you mention a student retention rate of 0% for those entering after March 1st, I am curious, what is your benchmark beyond which you expect students to remain in programming. Is there an hour allocation or are you basing your calculations on a completion of the session only?
Also, I wondered if Massachusetts employed any distance learning for students leaving a program for employment. Larry, I am sure you have heard New York voice our concern previously regarding our data indicating what appears to be a disincentive for programs encourage students to enter employment as that often results in the student leaving the literacy program prematurely and not showing educational gain. Have other state’s data shown this trend?
Rosemary I. Matt
NRS Liaison for NYS
Literacy Assistance Center
12 Meadowbrook Drive
New Hartford, NY 13413
315.798.1026
Good questions....
For us, it's sometimes a question of poor attendance, and then we have to get to the heart of the problem. More often however, when it came to late-year enrollment, students simply stopped attending at all.
It was extremely difficult to get feedback of any kind from these students, so we had to piece together bits of information we could gather from students, teachers, and other students in the program who were friendly with the departing students.
Our classes end in June, and begin again in September. I reviewed all the data for the year for students who left and never returned; I didn't specifically set March 1 as a cut point---it revealed itself to be significant upon review of the dates when students enrolled and then left.
When compiling the data into charts, it became apparent that any enrollment after the first half of the year was compromised; but all enrollments after March 1 were not successfully retained.
We are just beginning to work with one of my programs on distance learning as a supplement to classroom instruction...so new that I have no idea where it's going!
Hope this clarifies...
Luanne Teller
