Educational Technology

On-line chat: October 24, 1996, 4-5pm

Erik: Hello! Is anybody home?

Bob: Hi Erik, and everyone!

Erik: I received your email with attachments. Which one are we going over?

Bob: We may have a pretty "small" chat today -- One person is ill and another had a family emergency. But we'll forge ahead, anyway. Let's talk test items!

Bob: Well, what say we look at Activity 1. Let's take the first Objective, "define the word...". Which test item at the right matches it best?

Erik: I am really interested in this because I think it will apply to certain elements of my current job.

Traci: Hi...

Bob: Hi Traci! We're practicing test items. Do you happen to have the practices I sent handy? Actually, maybe its easier if I paste 'em in.

Traci: I have them here. Seems like defining the word microcomputer is an appropriate test item.

Erik: Sorry for the delay. My ppp connection dropped off. The last one I believe

Bob: Exactly. These are too easy for you over-achiers. But let's try fixing one.

Bob: 1. Objective: Given a flow chart of media usage, the instructional designer will be able to choose the most effective medium to use the when the objective concerns teaching a procedure and you are required to keep production cost down. Test item: List the two most effective media for teaching procedures when keeping cost down is a concern. How would you fix the test item to match the objective?

Erik: I think you need to put in the conditions/givens. Add something about the flow chart of media usage.

Bob: That's one thing, Erik. Anything else?

Traci: Because the objective is looking for the most effective method,shouldn't the test ask for only one?

Bob: Exactly, Traci. OK, brainiacs, try this one. What would be an appropriate test item for this objective? Given the sales reference manual, the new employee will be able to state the three most common printers used with the DEC line of products.

Erik: This is kind of restating, but how about: "By referring to the sales reference manual, state the three most common printers used with the DEC line of products."

Bob: You hit the nail on the head, Erik. If you've written good objectives, in some cases you can simply rephrase them as test items. Let's do one more -- the class had fun with this one last night. Within one day of being hired, the new employee shall be able to use the security entry code with 100% accuracy.

Erik: Would it be better to put, "state(write down)..."

Traci: How about: Enter the building without setting off the alarm.

Bob: Yes, Erik, you could make it more explicit that way. Yes, Traci, that would work. One solution yesterday was, "Welcome to your second day on the job. Try to get in." A bit facetious, of course, but its captures the idea.

Bob: This one gave people fits last night. See what you can do with it. The secretary will choose to recycle used computer paper each day when access to a recycling bin is within 50 feet of the work desk.

Erik: Short, simple & effective unless s/he is a cat burgular.

Erik: Given a recycling bin within 50 feet of your work desk, determine(write down on your test paper) whether to recycle used computer paper each day. I don't think I like this psuedo-objective.

Erik: Help me, Traci:)

Traci: I see why it gave them fits...

Bob: Yeh, this is a tough one. Its what we call an affective objective, as compared to a psychomotor (doing something with your hands) or cognitive (with your head). Affective objectives deal with attitudes. They need indicator performances, of course. Is there some way you could test this?

Erik: Perhaps you could test an attitude by testing a reaction to a situation.

Bob: Um, I'm hearing deafening silence! Here's what some of the students came up with last night. What about weighing the amount of paper in the recycling bin before and after instruction? Not a test item in the traditional sense, but a practical way of doing it.

Bob: Yes, Erik, that would work in some cases -- give people a simulated situation and see how they respond. A colleague of mine here once had a program to help get kids to eat more vegetables. They measured success by going through the garbage in the school lunch room and weighing the amount of vegetables thrown away, before and after the program. Yucky, but effective!

Erik: So you could test the effects of your instruction via a third party-a recycling bin. Bob, how would you approach affective objectives?

Erik: I mean how could you approach them without going through the garbage?

Traci: Oh, I see.

Bob: OK, let's tackle the REALLY hard stuff. So far, we've been dealing with "construct" type test items, where you ask people to do a performance or list something, etc. These are easier to write, but sometimes harder and more expensive to grade -- they take lots of time. Witness your projects in this class. If I had to grade each one carefully, myself, it would take me about 15 hours a week to do them even minimal justice!

Bob: Erik, you could give them an attitudinal survey, but those are fairly tricky and expensive to put together, and still may not tell you how they're going to behave. No easy answer to this one.

Erik: I guess it's clothespin on the nose time.

Bob: So, anyway, let's try our hand at a multiple choice item. These are a little tricky to put together (see Chapter 9), but are very easy to score once they're constructed. And they're highly underrated. Well-constructed multiple choice or "selection" type items can test so-called "higher level thinking." See some of the examples in Chapter 9. So want to try your hand at one? Take one of your own objectives and try writing a multiple choice item for it. I'll hang out here.

Erik: The main elements of animation timing theory are: a)frames per minute & movement rotation rate, b)frames per minute & pixel shift, or c)movement rotation rate & frames per minute.

Traci: One of my objectives might be: Given access to the style guide, writers will correctly format an index entry in FrameMaker. The multiple choice test could be: Select the correctly formatted index entry: A. Entering information about drivers B. Driver information:entering C. Information about drivers;entering D. About drivers, entering information

Bob: These are both really good. You've done well with the most difficult part, writing good distractors.

Bob: Do you see how you can test higher level thinking here, like analysis and synthesis?

Erik: Yes. It's funny, but I always looked forward to multiple-guess being easy. Not anymore.

Bob: Let me ask you both a question (Traci, I don't know if you saw the beginning, two of us are out with illness or family emergency). Do you like this format of my posing specific questions, or do you like the more free-for-all sessions, or a mixture, or...?

Erik: I like having everyone answer questions because I use the chat transcripts as a study aid.

Bob: So right, Erik. I made a pretty decent MC test for one of my EDTEC classes for some Health Science Folks. They thought it was going to be a piece of cake -- weren't expecting to have to think. They did! I'll try to do some of the same for the quiz in a few weeks, but there will only be a VERY few items. And, as I said, these are tricky to write, so you may criticize the quiz. Maybe that should BE the quiz!

Erik: Though, I do miss the freeform analysis.

Traci: Yes, I saw. I like a mixture, but after a full day, I'm better at answering specific questions than general discussion.

Erik: That would be a different approach. By the way, are there resources you can point to for Task analysis?

Bob: OK, that's good feedback. "See" you'all next week. Email me if you have issues during the week!

Erik: Otay! Later.

Bob: Erik, I think the references in the book are probably the best ones. There was a good talk on Cognitive Task Analysis at the ISPI meeting last week -- sorry I forgot to mention in to you. But we do Cog Task Analysis in 544 pretty thoroughly.

Traci: I like the idea of criticizing the quiz *being* the quiz! See ya!

Erik: Groovy. Thanks.

page author: Bob Hoffman
Post a question to the class forum.