This isn’t going to be a tirade of criticism like the last post but more of a reflection of how I think most people uses the Four Levels of Kirkpatrick, namely wrong. I’ve heard the following quite a few times:
“Yeah, we do course evaluation but we’ve only come as far as Level 1 and 2.”
So what’s wrong with this? Of course it’s very much easier to just measure what people think of the course (Level 1) and what they’ve learned (Level 2) than mess with change of behavior (Level 3) and result (Level 4). And of course people tend to do what’s easy instead of doing what they should do.
I’m amazed that people look at the four levels and interpret the “1” as “do this first and do the rest when you’re ready”. (Or maybe the numbering issue is the Kirkpatrick’s fault but hey, I promised, no more criticism!) There’s no other way of measuring and analyzing a change program (i.e. a course) than to start with what result your company is after.
Ask yourself the following question: Would your CEO ask you about what the target group participants felt about the course if you got the result you were after? Of course not! She’d applaud you and tell you to move on to another project! Would your CEO worry about how many things your participants learned? Nope. If they changed their behavior? Well, not really, no. It’s the end result that matters in the end, am I right? NB! This is not to say that you shouldn’t focus on behavior though. Quite the opposite and I’ll come back to that in a bit.
Will Thalheimer is famous for his straightforwardness about the pointlessness of using smile sheet ratings as a barometer of how effective your learning intervention is, citing a correlation at about 0,09 – which basically is no correlation at all.
Thus, forget about Level 1 I say! Ask these questions in the classroom while you have a chance to do something about it! Level 2? While it’s interesting to know what level the participants acquire what you’re offering (not only knowledge of course) the only things that’s really interesting is their commitment and confidence. How much they’ve learned, heard or read is totally beside the point so skip that or make it a secondary item of that evaluation. What I suggest is a much larger focus on Level 3 and 4, i.e. behavior needed to reach the result wanted.
- Frame and formulate your change program as to generate and incorporate the participants’ ideas and suggestions.
- Provide a wealth of content that supports the critical behavior of Level 3 but not anything else. We want them to do stuff and if they have to learn things for this to be possible, fine, but if not, that’s even better!
- Create checklists
- Idea databases with crowdsourcing, i.e. up-voting for good ideas and down-voting for the opposite
- Provide job-aids that can be printed out if needed but always are available when at work
- Make sure your organization really supports the change program by doing their share of key required drivers:
- Have managers coach and act as role models
- Do extensive follow-up of results and make the results really visible
- Share best practices and reward the practitioners.
- Get a good foundation on where you are today to get a realistic and necessary goal, i.e. measure properly and analyze your data to make sure you know where you really want to go. From there you create your leading indicators – part goals that will get you to the top of that mountain in the end.
Update: Since my blog got hacked and went down over the weekend I didn’t have the possibility to post this until today. What’s interesting is that Jim Kirkpatrick is thinking what I’m thinking as shown by today’s tweet:
A false belief about the 4 levels: you should first become experts in 1 & 2 before moving to 3 & 4. Really, you should quickly master 3 & 4.
— Jim Kirkpatrick (@Jim_Kirkpatrick) August 20, 2013