• We tend to conduct observations because we want or need to detect / witness behaviors that imply particular attitudes or patterns of thinking. According to Fitzpatrick, et al. (2005), observations allow the evaluator to see programs in action. By contextualizing the frame of reference, he or she is able to better understand program operations and outcomes, participant behaviors, patterns of interaction, and other vital factors (p. 375).
• Critical steps in the process … with which you’ll want to be familiar
– Identify the specific information you hope to capture. Here are some examples of data collection activities from the recent past.
Sean Glithero focused on language arts teachers learning how to better align their writing instruction with the state’s Language Arts framework. Altogether, he attended nine training events – and witnessed three different facilitators.
Natalie Mingura focused on a train-the-trainer design, where teachers “learned” how to model/demonstrate technology fundamentals to peers; she attended three training sessions – each conducted by a different team leader!
Susan Wilson’s teachers boned up on math and science content; she attended four sessions – but the facilitation was consistent.
Mary McGibbons and Cindy Davis were in a corporate environment. They observed several modules of a complex and highly theoretical program designed to improve the leadership and managerial skills of key Contexant staff.
Elements of Sean Glithero’s semi-structured protocol are reprised below:
Room configuration
Organization of participants
Facilitation – in particular, the balance between “lecture”/presentation and active learning
The types of activities in which participants were engaged
How activities were “scored”/how feedback was generated
Grade level appropriateness of workshop activities
Transitions/segues that linked elements of a workshop; transitions/segues that linked the different workshops in the series
Etc.
[In May 2003, Caroline Willi completed detailed observations of student engagement for Dr. Sharon Lightner—who taught ACCTNG 620 in the Experimental Classroom. (Don’t let me forget to hand out copies of Caroline’s protocol in class.) Her goal was to watch “stable” teams work on accounting assignments of two specific types: those that were intellectually challenging and those focused on application skill-building (Microsoft® Excel).]
– Decide on the most appropriate way to obtain the information.
The more structured the protocol, the more quantifiable the information. Checklists help you record the presence, absence or frequency of selected behaviors. They’re less useful, however, when you need to collect data relative to duration or quality.
Once way to ease the burden is to develop codes that represent behavior types as well as groups of people.
For example, Victor Kiuhan (who during the Fall ’01 semester completed a tracking study at the San Diego Zoo) used a mix of abbreviations to represent exhibit areas as well as visitor types.
Sean Glithero’s protocol featured a coding schema, too: T to represent participating teachers, F the facilitator, and G any guest speaker or presenter; GQ to represent questions posed within teams or groups of participants and GH for help or assistance that team members offered offer one another. Codes help you capture patterns of personal interchange.
A delayed report instrument is completed immediately after an observation. It’s best employed by someone experienced with this form of data gathering, since it requires some personal judgment / inference.
– Select and prepare the observers and – if you’re using an instrument, develop it.
Training is essential when someone is “hired” to conduct an
observation for you. What might be some
components of effective observer training? In what ways might you eliminate …
or greatly reduce … the risk of bias?
– Decide who will
be observed … and when.
Here’s where observations
often go awry – because you must control (through defensible methods) whom to
observe and key logistics (when, how often). Among the questions to consider:
• Is preselection of subjects possible … or appropriate? What are the consequences of preselection if you yourself are not conducting the observation?
• Are you staying with an entire group – or selected individuals – for the entire observational period. If not, when is it ‘appropriate’ to switch your focus?
• How many observations must be conducted to provide convincing evidence that the findings represent actual (recurring) behavior?
• When during a program’s rollout are observations of a program most feasible to conduct?
• What is the optimum time of day for observation?
• What’s an appropriate observational length?
– Make any necessary arrangements – if the situation warrants … and if failure to do so does not violate any evaluation standards.
Food for thought: What are secret shoppers (at Vons, Nordstrom, or the airport) looking for?