Getting Participants' Feedback

From LiteracyTentWiki

Subject: [NIFL-AALPD:1280] Getting Participants Feedback
From: David Rosen (djrosen_at_comcast.net)
Date: Wed Mar 31 2004 - 13:53:53 EST

Heide and others,

The problem for a presenter of getting feedback (information from the audience which allows you to know whether you are hitting the mark or not) is, as Heide pointed out (Being Funny Online), not limited to the online medium. I have been a speaker and performer (in a musical group, outdoors, at the Hatch Shell on the Esplanade in Boston) where the audience was seated so far away, and sitting in the dark, that I could not get tell if they were awake or even there. It usually isn't as big a problem face-to-face as it can be online. But it can be a big problem face-to-face, and I have talked with online PD course facilitators who say they get more feedback, much deeper and more personal feedback, from students online than in face-to-face classes.

Recently I gave a keynote address to several hundred people at a conference. It was in a university chapel auditorium: very beautiful but also quite formal. I was on a stage behind a podium, very removed from those I wanted to dialogue with. And I didn't have as much time as I needed. I anticipated this problem and came prepared with hundreds of mini-questionnaires on purple slips of paper. These were passed out for me at the beginning, and people were asked to list 5 top trends in our society and 5 top trends in adult literacy from their perspective. They passed these (purple) slips to the end of the row where I had asked that they be tabulated. (Yellow) row tabulations were passed up to a gracious graduate student who had agreed to tabulate the yellow tabulations while I presented. When he was done I was given the feedback, and I read the results back to those assembled. Then we had a short discussion about what they and I saw as trends. (There was, of course, some overlap.) This worked for me as a presenter, giving me useful feedback for our discussion. And I think it worked for those in the audience in several ways: 1) it helped them focus on the topic - - trends in adult literacy - - by thinking about it from their perspective first, then listening to mine in the context of what they had been thinking about: how does this guy reinforce my thinking or challenge it? 2) it gave them something to do while latecomers were straggling in; and 3) it was amazing to all of us that we could get nearly 100% participation in a large group and get it boiled down so quickly. Maybe some of you also do this in your face-to-face staff development classes and workshops.

So, what's the online distance learning equivalent of the face-to-face mini-survey?

One thing I am planning to try, using Blackboard, is asking online participants to do a short survey. I will ask them to rate each of the course objectives in terms of their expertise -- from "don't know anything about this" to "am an expert at this." Then I will post the (automatically tabulated) results so everyone can see where the class is on each objective -- not individuals, but the group. This gives us all an idea of what we may need to spend more time on, and what's pretty much in everybody's grasp already.

I am looking for other ideas like this that are quick and easy ways for online participants to give facilitators feedback.. Have you seen or tried something that will give the online facilitator this kind of feedback and also possibly the participants, too?

David J. Rosen
djrosen@comcast.net

Subject: [NIFL-AALPD:1284] RE: Getting Participants Feedback
From: Jane Mencer (jmencer_at_famlit.org)
Date: Wed Mar 31 2004 - 15:03:34 EST

What a great idea, David. I've also seen online polls used effectively for a similar purpose. I have experienced online facilitators during synchronous sessions (I think that we were using NewMeeting) who, after introducing and covering a learning point, using a poll feature of the software posed a one-question multiple-choice question which we, as learners, all answered. Based on the percentage of learners who answered correctly, the facilitator would know whether what he had presented was understood.

Jane

Subject: [NIFL-AALPD:1287] RE: Getting Participants Feedback
From: Christy Nelson (cnelson_at_nwm.cog.mi.us)
Date: Wed Mar 31 2004 - 15:33:05 EST

I also think this is a great idea. There is nothing worse than taking part in a workshop/training/class that is so far off from what you thought it would be when registering. It would great to have the opportunity to voice opinions and perhaps have the chance to gain the information you were looking for when you enrolled.

Christina Luckey-Nelson
Adult Education Coordinator
TBAISD Adult Education
1209 S. Garfield - Suite C
Traverse City, Michigan 49686
231-922-3710

Subject: [NIFL-AALPD:1290] RE: Getting Participants Feedback
From: Mary Russell (russell_at_literacy.upenn.edu)
Date: Wed Mar 31 2004 - 15:54:52 EST

I'd like to second David's mini-survey idea. We have had considerable success using the "instant survey." We took the information and fed it into a spreadsheet. That way you can see where eveyone is on various issues/topics. Blackboard has a survey tool that looks good (it will collate results, too), that I am going to try with my classes this summer.

Subject: [NIFL-AALPD:1292] Re: Getting Participants Feedback - webex??
From: Marian Thacher (mthacher_at_otan.us)
Date: Wed Mar 31 2004 - 16:18:17 EST

Has anyone used WebEx, or something like it, to do professional development? The California Distance Learning Project recently purchased a license to do some training of teachers on new online high school courses that are being piloted. I got to lurk in one of the trainings, and it seems to me that this model could be helpful in terms of getting a sense of your audience.

For one thing, it's synchronous. When you log on you get a phone number to call, and everyone is on the phone together at the same time they are online together, so you hear laughter, grumbling, snoring, whatever. There is also a way to post an emoticon next to your name on the participant list (although I never did find out how to get rid of it once it was there). There is also an instant message screen to post questions or comments. All this while we were watching the presenter's browser so he could show us some information, how to log on to the courses, what's there, etc.

I think there is also a survey function. You can ask a question and have everyone respond, and the results are aggregated immediately. (Does that mean that in some ways this beats face-to-face??)

It seems like having this capacity could be a great way to introduce people to an online course, because you can talk back and forth and coach while you are showing them. They can also have a separate browser window open where they are doing whatever you are demonstrating, although that involves toggling which could be challenging for beginners.

Of course, there is a cost factor. I'm not sure of the cost but I know it's not cheap, but on a statewide level it might be really cost-effective.

I'm interested to hear the opinions of experienced users.

Marian Thacher
OTAN
www.otan.us

Subject: [NIFL-AALPD:1331] BlackBoard Surveys vs Tests
From: Duren Thompson (solveig_at_utk.edu)
Date: Mon Apr 05 2004 - 08:25:53 EDT

David,

One of the things that was in the Florida course that Sandra and I duplicated in our first course was a multiple choice "pre-test" and a "post-test" covering key points in the course. We used BlackBoard's "Test" feature to duplicate these in our course - being very careful to use the exact same wording on both "tests."

Oh man. We got *lot's* of feedback - they *stressed* over not getting them all right. We told them over and over that it was just a pre-test, used to help us know what they know, etc. and still they complained that it wasn't fair, that, "They could have gotten the rest right if..." Sandra and I knew right away that we'd do something different next time.

For the January 2002 course, we implemented the Survey option in Blackboard. we also shifted the questions from a more traditional multiple choice "knowledge of facts" quiz to a competency-based survey built off of the course objectives. Course objectives include "practitioners will implement a cooperative learning strategy in their AE classroom." So the pre and post Surveys ask, "Have you implemented a cooperative learning strategy in your AE classroom in the last 6 weeks?" If they complete the course - they will answer this yes - period. Some questions were short answer, some multiple, multiple choice and some true/false - but all about "Do you already know/do the things we will talk about and do in this course?"

In Blackboard the survey answers are *not* available to the course participants naturally - and this has been our one complaint. Practitioners have told us in passing, "I'd like to know what others said the survey." We have contemplated collating the answers and publishing them for each course's participants - but there simply hasn't been enough time yet - and BlackBoard doesn't make it particularly easy to get the data out into another format (copy and paste has been our only option.) Oh - and the surveys are anonymous (no other option) - although when you get to know your students - you can make pretty good guesses as to who said what.

I think you'll like the survey option - we found that it helped us to get a "feel" for the course participants as a "whole group" and allowed us to tailor discussions and chats to needs we saw in the survey responses. It also lets us know how many "personalities" we have - ringers, skeptics, enthusiasts, the timid, etc.

So, back to the idea of "tests." We really marveled over how these AE instructors reacted. We *know* they have said the same things about their standardized AE "placement" tests - like the TABE - to their own learners. What is it about us as "instructors" or "learners" that makes us all upset when we don't do well even on a "test" that doesn't count? One of the reasons I've hesitated to publish the results of the surveys is that I worried that some folks would "fret" over having answered "differently" from others - or think that they were *wrong.* Are our practitioner egos as fragile as those of our learners? And if so, how does this affect our online course development (or our pd development altogether?)

Duren Thompson
Center for Literacy Studies

Subject: [NIFL-AALPD:1333] Re: Centra vs webex
From: Duren Thompson (solveig_at_utk.edu)
Date: Mon Apr 05 2004 - 08:27:37 EDT

Marian,

This sounds a lot like Centra E-meeting which we use here at the Center for Literacy Studies (or more truthfully, are *beginning* to use for cross country and cross-state meetings and trainings coordinated by the Center). Michelle Meilleur mentions that literacy practitioners in Ontario have used Centra for training as well.

Currently Centra e-meeting is a service that the Southern LINCS RTC provides to NIFL and the southern regional states as they request it. We've used it for our Portfolio pilot project practitioner meetings twice this year here in TN. We didn't have funding for even quarterly face-to-face meetings and it was *way* cheaper than renting a phone bridge/conference call. It also allowed them to "share" a document they were working on and edit it as a group. I'm looking forward to using it to assist some of our far-flung presenters for our Summer Academy to plan their 3 hour joint presentations together.

The biggest barriers that we've encountered are:

1) Training - it really does take about 30 minutes of "non-work/content" time to accustom practitioners to using it.
2) Headphones/microphones. Centra runs completely through your computer - no phone charges - and so requires a standard $15 headset/mic combo (although headphones and a standing mic will do). Folks just don't have these things and when they get them or borrow them they have the darndest time getting them plugged in right and then setting their systems to use them. Particularly in K-12 situations where the computers are so locked down the practitioner can't change any settings without a tech support person over their shoulder.
3) "Consistent" bandwidth. Centra really doesn't take up a lot of

bandwidth (as long as you leave the video turned off) for a simple meeting. But if you are in a system with "jumpy" internet service (really slow, then ok, then fast, then dropped for a second, then slow, etc.) - it really messes with the participant's audio reception. Unfortunately, many of our rural areas have the strangest, small company phone service monopolies - with lousy phone service and thus lousy internet connections.

4) You *really* need a facilitator *pair* in order to effectively group edit a document. One to do the mark -up and one to "facilitate" the group. Working the technology to edit and working the technology to facilitate AND editing and facilitating is just too much for one person to do and be effective for the group.

Our center librarian, Beth Ponder, has used it to conduct LINCS cataloging training for a group of 8 - like most new technologies, she said, "We needed to have designed our presentation for the tool - not just assumed our "in-person" PowerPoint would be effective unmodified." But her trainings was successful - just a little bumpy.

I like Michelle's comments about how to "train" folks to use the technology. We tend to train folks to use Centra via a distance - with at least one or two tech folks to help straighten out the "glitches" via phone during that first set-up and training. The tech folks have to be separate from the presenter - and the step-by-step guides we have written do help!  ;)

Duren Thompson
Center for Literacy Studies

Subject: [NIFL-AALPD:1347] Re: Centra vs webex
From: Marian Thacher (mthacher_at_otan.us)
Date: Mon Apr 05 2004 - 12:03:42 EDT

Thanks, Duren, this is very helpful. I'm actually planning to use WebEx with a couple of co-presenters to revise our powerpoint, so I will plan in some time to get used to the tool first. Your idea about having two people at my end, one to facilitate the conversation and one to actually make the changes as we talk is great too.

WebEx uses the phone, so we don't have the microphone problem. I have a headset for my phone, but I could see that if you were trying to hold the phone with your shoulder and type at the same time for any length of time it would get pretty uncomfortable.

About the bandwidth issue, we shall see. Tune in next week...

Marian Thacher
OTAN

Subject: [NIFL-AALPD:1352] Re: Centra vs webex
From: Rejoicer_at_aol.com
Date: Mon Apr 05 2004 - 17:44:10 EDT

We use WebEx at my office, and it's pretty simple to use. A word for the wise presenter is to set up two computers, and log into one as you the teacher, and another as a student so you can watch the latency. It used to be a much bigger problem with speed, but that appears to be corrected. It is also really helpful to see what they see regarding buttons, since the instructor view is a little different. I have the early arrivals play on the whiteboard to get comfortable with the environment and the tools. That appears to take away some of the nervousness with the new system.

I use the chat area like I would use a flip chart to document what people expect out of the class, and like we sould do in a training class, go back over the list at the end to make sure we have addressed everything.

Do warn the students that they will log in and see a white setup screen first, then it goes into the meeting room. They also may have a plugin to download in the beginning.

It's a great tool.
Jean Marrapodi

Retrieved from "http://wiki.literacytent.org/index.php/Getting_Participants%27_Feedback"