Skip to content


Book Review-Kirkpatrick’s Four Levels of Training Evaluation

Sometimes, you say a thing and it just catches on. It’s a moment of insight that gets frozen in time like a mosquito in amber, and later you realize just what you have. Kirkpatrick’s Four Levels of Training Evaluation is like this. It’s a simple framework for evaluating the efficacy of your training program. Don Kirkpatrick uttered the words: reaction, learning, behavior, and results. His son and his son’s bride take up these words and refine the meaning that the industry gave to the words and adjust them back towards their original intent.

The Levels

Despite the fact that Don never uttered the words as levels, others added them to the descriptors, and eventually people began calling it the “Kirkpatrick Model.” It stuck. Today, professionals speak about the levels of evaluation like this:

  • Level 1: Reaction – Did the students report that they enjoyed the learning experience, including the materials, the instructor, the environment, and so on?
  • Level 2: Learning – Did the students learn the material that was delivered? This is the typical assessment process that we’re used to having to complete to be able to report successful completion of a course, but it’s more than that. It’s did we learn anything that we can retain after the class and the test are long over?
  • Level 3: Behavior – Ideally when we’re training, we’ve identified the key behaviors that we want to see changed. Level 3 is the measurement of the change in the behavior.
  • Level 4: Results – Did the change in behaviors create the desired outcome? Are we able, as training professionals, to demonstrate that what we’re doing has value to the organization in a real and tangible way?

The Process called ADDIE

Many instructional designers use a design process called ADDIE after the steps in the process:

  • Analysis – What results do we want, what behaviors need to change to support that, and what skills need to be taught to change the behaviors? (Here, I’d recommend looking at The Ethnographic Interview and Motivational Interviewing for tools you can use.)
  • Design – What kinds of instructional elements and approaches will be used to create the skills and behaviors that are necessary to accomplish the goal? (Here, Efficiency in Learning, Job Aids and Performance Support, The Art of Explanation, and Infographics are all good resources.)
  • Development – The long process of developing each of the individual elements of the course.
  • Implement – Implementation is the execution of the training, either instructor led or in a learning management system.
  • Evaluate – Assess the efficacy of the program – and, ideally, revise it.

If you’re unfamiliar with the course development process or you’d like to explore it in more detail, our white paper, “The Actors in Training Development,” can help you orient to the roles in the process and what they do.

The untold truth is that, in most cases, the processes is rushed, hurried, and many of the steps are skipped or given insufficient attention. Rarely does an organization even have someone with instructional design training much less the time to do the process right. There’s always more training that needs to be developed and never enough time. The Kirkpatricks are driving home an even more telling point. The evaluation process – how you’re going to assess the efficacy – needs to be planned for during the analysis and design phases. The development and implementation phases need to consider the conditions that will be necessary to get good evaluation results. Evaluation isn’t something that can be bolted on at the end with good results.

It’s sort of like Wile E. Coyote strapping a rocket to his back and hoping to catch the roadrunner. It always seems to end badly, because he never seems to think through the whole plan.

Leading and Lagging Indicators

I learned about the horrors of metrics through The Tyranny of Metrics but learned real tools for how to create metrics through How to Measure Anything. However, it was Richard Hackman who really got me thinking about leading and lagging indicators in his book, Collaborative Intelligence. He was focused not just on how to make teams effective in the short term but how to create teams where their performance remains good and keeps getting better. He was talking about the results as a lagging measure, an outcome from the creation of the right kind of team. Influencer picked up and reinforced the concept. We need to look not just at the outputs that we want but the behaviors that we believe will drive those outcomes.

It’s all too easy, as you’re working on developing the metrics for your training, to focus on the lagging metrics and say that you don’t have enough influence on them. After all, you can’t take responsibility for sales improvement. Some of that’s got to be up to the sales manager. And you certainly don’t want to say that your training sucked if sales dropped after salespeople took the course. As a result, training professionals too often shy away from the very metrics that are necessary to keep the organization when there’s a downturn. Instead of being seen as an essential ingredient to success, they’re seen as overhead.

By focusing on a mixture of both leading indicators and lagging indicators, training professionals can get to an appropriate degree of accountability for end performance. Leading indicators are – or at least should be – behaviors. They should be the same behaviors that were identified as a part of the analysis phase as needing to be changed. These should be very highly impacted by the training. The lagging measures are the business outcomes that also should have been a part of the analysis process – but are further from the learning professionals’ control.


While it’s not true that we need to hope for good outcomes, there’s a bit we can learn from The Psychology of Hope with regard to training’s role in the process of changing behaviors. In The Psychology of Hope, Snyder explains that hope is made of two components: willpower and waypower. Willpower is what you’d expect. It’s the desire, perseverance, or grit to get things done. (See Willpower or Grit for more.)

Waypower is different. It’s the knowledge of how to get to the other side. It’s the knowledge of the how that learning professionals can help individuals with. It’s waypower that training professionals give to their students every day. This may be used for the purposes of some corporate objective, but in the end, it’s a way of creating hope in the minds of the students that they can get it done if only they try. (Here, a proper mindset is important, too, as explained in Mindset.)


There’s nearly zero research on the relationship between overall performance on the job and well trained, knowledgeable people. The problem is that we don’t really know how much training does really matter. What we do know, however, is that the application of the skills and behaviors that are taught in the classroom don’t always happen. The problem is called “far transfer,” and it’s a relative secret that what we teach in classrooms doesn’t always get applied to the real world. (If you’re interested in some other relative secrets in the training industry, check out our white paper, “Measuring Learning Effectiveness.”)

There’s an absolute essential need to consider how the skills that are being taught in the course can – and will – be applied by the student in the real world. Discussions, case studies, and conversations make for learning experiences that tend to be more used long after the training has been completed.

About the Questions

The book wouldn’t be complete without some guidance on how to write actual evaluation questions, including avoiding superlatives and redundant adjectives when evaluating in a scale – and ensuring that the scale matches the type of question being asked. Question authors are encouraged to keep the questions focused on the learning experience rather than the instructor or environment to get better answers.

The real question for you is will you read Kirkpatrick’s Four Levels of Training Evaluation and apply it to the way you evaluate your training?

Book Review-Critical Knowledge Transfer: Tools for Managing Your Company’s Deep Smarts

What is it that makes one person more valuable to an organization than another? Take two engineers with the same degrees from the same universities and even the same grade point average. One is invaluable to the organization, and the other is just a solid contributor. One just seems to know things the other doesn’t. When considering how to make the knowledge of the organization more accessible, it’s in the organization’s best interest to highlight the more knowledgeable of the two. However, how can you determine that?

Critical Knowledge Transfer: Tools for Managing Your Company’s Deep Smarts is designed to help solve this problem, both from the point of view of identifying which employees hold critical knowledge and from the perspective of learning techniques to transfer those deep smarts to other members of the organization.

Identifying Deep Smarts

Before you can put a plan together about how to transfer deep smarts, you must first be able to identify where those deep smarts might be. While the deep smarts themselves may not be easy to isolate and convert into explicit knowledge, there are precursors that you can look for that may indicate that deep smarts exist.

There are cognitive, behavioral, and physical indicators that, when you find them, can indicate a deeper level of thinking and processing is happening – and therefore deep smarts might be nearby. In the physical dimension, the ability to quickly, accurately, and precisely predict and respond to physical touch and other senses can lead to deeper “feels” that may be hard to articulate. From the mundane riding a bike to the subtle skill of a master illusionist, there are observable physical traits that indicate there’s more happening.

In the behavioral category are a cluster of skills that seem to be formed around communicating and relationship-building with others. We’ve all met the master connector who can make friends with anyone and who can plug people together. Whether it’s for personal gain or in service to others, master connectors can communicate and connect when others can’t. Their deep smarts involve the map of the other people they know and what they’re capable of as well as their own deep smarts about how to operate in social environments with people who have such a variety of interests, backgrounds, and goals.

The most commonly focused on category is the cognitive area, with systems thinking (see Thinking in Systems for more), critical “know-how” and “know-what” skills, wise judgement, context awareness, and pattern recognition being the contributors to this kind of deep smarts.

Comfortable with Disagreement

One of the curious entries in the list is the capacity to be comfortable with disagreement. Those with deep smarts often have a characteristic capacity for being comfortable with disagreement. The fact that other people don’t agree with them doesn’t threaten their perception or their identity. These folks have accepted that others will have different perspectives than them, and that is OK. Either they’ll learn to adjust their perception about reality – or they’ll accept that someone else has a different perspective.

We learn through well-managed conflict – through dialogue. Dialogue with another person grants us the gift of revealing our inconsistencies. (See Dialogue for more.)

Creative Abrasion

When John Gottman first did research on couples, his ideas were odd. Why would you intentionally ask couples to fight? What would it reveal about who would stay married and who would divorce? The answer was in the way the couples fought. Some couples would fight and get nasty and personal. Others would fight in ways that recognized the disagreement and acknowledged the value of the person. (See The Science of Trust for more.)

Similarly, it’s generally believed that, in teams, there shouldn’t be conflict. Conflict is seen negatively about the ability of the team members to get along. However, the problem isn’t that the team is fighting. It’s an opportunity see how they’re disagreeing. If the team can be respectful of their colleagues’ perspectives and seek to better understand each other, the creative abrasion can fuel excellence and performance. Ed Catmul, in Creativity, Inc., explains that, at Pixar, the brain trust (a specific meeting) reviews and criticizes early drafts of movies. The way it’s done is constructive rather than destructive, and this allows the meetings to empower directors rather than to demotivate them.

A Monkey’s Expertise

Unfortunately, what appears to be deep smarts isn’t so deep after all. Some environments don’t give good feedback, and people aren’t interested in it anyway, since it would expose that they’re not good at their job. A bunch of monkeys could pick stocks as well as the expert investors. This sad proclamation is almost true. With dart-throwing, when humans stood in for their evolutionary cousins, the assertion was proved false. However, the expert investors only beat the Dow Jones Industrial Average 51% of the time. That’s not a particularly impressive record.

A single percentage point difference can be important when there are large amounts of money involved. However, for most people, the expertise isn’t worth much. Sometimes, “expertise” really isn’t. And, sometimes, it can lay dormant without notice.


When it comes to recognizing deep smarts after it is too late, posthumous recognition is the ultimate. Once someone is dead, they can, obviously, no longer produce additional works. History is littered with experts whose genius was recognized after their death. Edgar Allan Poe, Emily Dickerson, Henry David Thoreau, Claude Monet, and dozens of others’ expertise was never recognized while they lived.

In business and knowledge management, this highlights a key problem that plays out every day with less grisly consequences. Organizations don’t recognize the key smarts an employee had. The employee leaves, and the organization discovers what they lost. While this cannot be prevented, it does represent a challenge for organizations that don’t want to find they wasted precious opportunities to capture the knowledge that someone has.

Tactical with an Overarching Theme

I’d call it strategic and tactical. It’s the ability to be both fox and hedgehog. (See Range and Should You Be the Fox or the Hedgehog?) It’s rare to find people who have both a “vague understanding about everything” and a detailed understanding about some things. More importantly, it’s powerful. Sometimes, the most important knowledge doesn’t stand out as the most detailed expertise in a single area; sometimes, it’s hidden in the valleys between the peaks of the disciplines. It’s in the connections between one thing and another. These sorts of deep smarts are particularly hard to identify and transfer, since developing them requires developing expertise in multiple areas, which takes time.

Leaving a Legacy

There may be no one formula for identifying or capturing the deep smarts that folks in your organization possess. Techniques like those detailed in Lost Knowledge are all options for capturing the knowledge that has been identified. But identifying those deep smarts – particularly those smarts that will be useful to others – isn’t as easy as it looks. However, there may be a way to leverage mentoring opportunities to empower the expert with the drive to share what they know.

It starts with leaving a legacy. We all, at some level, want to leave our mark on the world. We know that some day we’ll die, no matter how much our ego tries to shield us from this fact. When we believe that we can leave a lasting change beyond our lifetime we’re motivated to try. While most of us won’t contribute to the library system like Andrew Carnegie, we can leave our mark through the way that we educate, support, and mentor others.

When you’re working at an organization to encourage experts to share what they know, consider connecting this sharing to their being able to leave a legacy at the organization and in the world.

Maybe, just maybe, this will be what it takes to accomplish Critical Knowledge Transfer.

Book Release: Secret SharePoint

We interrupt our normal schedule of book reviews to announce that we’ve released our latest book – Secret SharePoint. We’ve been leaking some content from the project for months on our blog, through a free subscription to an email series, and through partners like SPTechCon. This has been in preparation for our launch of the Secret SharePoint book – and that launch is today!

The book is over 200 pages of special tips that I’ve learned over the last 18 years of working with SharePoint. It’s solutions to hard problems, and they work whether you’re using SharePoint on-premises or SharePoint Online.

From birthday calendars and navigation to search and Word Quick Parts, Secret SharePoint has the solutions you need for the problems you’re facing.

More than that, we’re still offering the free email course that you can sign up for to get part of the content from the book and a newly-released, self-paced online course. Both of these offer not only the written text but videos and screen casts that walk you through the entire process of building the solutions that we’ve created for other customers.

As a special thank you for following us, you can get the book directly from us for $15 + Shipping. That’s half the retail price. Order yours today.

How We Learn: The Surprising Truth About When, Where, and Why It Happens

Book Review-How We Learn: The Surprising Truth About When, Where, and Why It Happens

Our ability to learn ranks right up there with our ability to coordinate our activities as the chief weapon that we’ve used to become the dominant species on the planet. As anthropologists John Tooby and Irven DeVore have commented, we carve out the “cognitive niche”. Despite our cognitive capacity being so essential to our survival that it literally drives us to be born before we’re fully prepared to take on the world, relatively little is understood in science about how we learn — and substantially less of what we have learned has become common knowledge. How We Learn: The Surprising Truth About When, Where, and Why It Happens is designed to change the public’s awareness of what little we do know about learning. (If you want more on our ability to coordinate and its importance, you can see The Righteous Mind or Mindreading.)


While learning is essential to our current world, much of what we think about as learning is new from an evolutionary sense. Even reading, writing, and arithmetic are relatively new creations. Consider that before the invention of the printing press, literacy meant the ability to write your own name — and there weren’t that many people that were literate. Today, we view literacy differently: it’s the ability to read and write in our native language.

We expect — rightly or wrongly — that our children should be able to have basic fluency in their native language by the time they’re ten. We expect even more from them as they progress through schools. Where calculus was the domain of specialized mathematics only a few decades ago, it’s an assumption for most professions today. You’re expected to understand the basics of a branch of mathematics that was until recently a speciality — and much, much more.


In evolutionary terms, the human being we know is a newcomer. Written history extends back a few thousand years, and fossil evidence goes back ten thousand years or so. That’s a blink in evolutionary time. We evolved from hunter-gatherers to the masters of agriculture, and with that we developed a caloric surplus, which allowed us to start to pursue more abstract thoughts than worrying about the next meal and avoiding becoming one.

During this rapid conversion from a nomadic existence following berries and buffalo to one with deep roots in agriculture and our subsequent adaption into a sedentary and highly intellectual experience, we’ve moved faster than our genes can keep up. We’ve moved into a world where our shared knowledge is so much more than we as humans have ever encountered.

Some have estimated that we experience in a single year more information and data than our grandparents experienced in their lifetimes. That’s an increase in information of 50-100 to 1 in just the last 100 years, and it’s getting faster.

Information Management

When I speak to audiences about information management, I share how the advances in our ability to share knowledge is growing at a breathtaking pace. Until Gutenberg’s printing press in 1450, if you wanted something copied, you gave it to a scribe or a monk. Gutenberg made it possible to take important texts and make copies efficiently, thereby reducing the barriers to having books. In 1870, we got typewriters. This allowed us to standardize the appearance of text and to provide a standard structure. In 1959, Xerox created the xerographic process for photocopying content. Suddenly, the bar for replication was dramatically lowered. In the 1970s, computers made the processing and replication of data easier. In the 1980s, computers became personal, and suddenly everyone was able to store and share their information. In the 1990s, computers were networked, so sharing between people became automatic. By the 2000s, we shared images as well. In the 2010s, we started delivering video.

The upshot of this is that it took us thousands of years to get to writing and then a few thousand to get to the ability to replicate content. Now we’re looking at innovations in our ability to share information about every decade. How can you possibly keep up with all the knowledge being created? The answer is that you can’t — however, to even keep up with a portion of what we need to know, we need to be efficient and effective with our learning. Unfortunately, our learning innovations haven’t kept up.

Brain Science

There are two distinct branches of science that study how the brain works. One branch is psychology, which is largely concerned with the proper functioning of our minds as it relates to the behavioral outcomes. The other branch is neurology, which is focused on understanding how our brains perform the wonders that they do.

I’ve shared in my reviews of The Cult of Personality Testing, Warning: Psychiatry Can Be Hazardous to Your Health, The Heart and Soul of Change : Delivering What Works in Therapy, and other books how little we actually know about psychology. In truth, the correlations of outcomes for the counseled and uncounseled states are horrifyingly similar. There’s great arguments in this field about what is and isn’t effective. Psychoactive drugs are prolifically prescribed, and yet seem to have very little effect.

On the neurology front, we’ve got some knowledge about the regions that are active for various thinking and behaviors, but there’s more that we don’t know than we do know. We’re looking into an opaque gray matter hoping to tease out how the magic works — and we’ve been largely unsuccessful. (See Incognito, The Rise of Superman, The End of Memory and Emotional Intelligence for some on neurology.)

Along the way, we’ve found some answers from brave and insightful (and sometimes lucky) scientists who stared at a result and scratched their heads until they could come up with plausible hypothesis about what is going on inside our heads. These answers have not been adopted by those who lead the charge for better education for everyone – they’re marching the same old beat to the same old drum. (See Helping Children Succeed and Schools without Failure for alternative views.)

Myths and Legends

Old myths about how we learn that were garnered from limited experience and observation sometimes run directly counter to the research generated by prestigious universities. Good science is saying some of the things that we’re doing aren’t the right things. We’re not optimizing the learning experience. What we thought we knew about how to teach and learn is being turned on its head — and some is being validated as fundamentally correct.

Some of the myths like having to “keep your nose to the grindstone” are being dispelled by compelling evidence that taking a break can increase retention and free up the cognitive resources necessary to generate the innovations to drive the next generation of business leaders forward.

Forgetting is Your Friend

The nemesis of learning has been the forgetting curve. Ebbinghaus precisely documented the decay of memory using nonsense — in an exacting way. The forgetting curve has long been the enemy of professional trainers and teachers. It’s seen as failure of learning. However, it might be the result of an active process where our brain is trying to cope with the onslaught of information that it wasn’t ever designed to handle. It could be that our mental systems that were designed to consolidate memories trimmed them from our consciousness, so we could focus on things that are more urgent and more relevant.

Losing memories – forgetting – is a painful experience for all of us. It’s frustrating to forget a name or a word when we feel like we need it most. However, this process isn’t one measure but is instead two. Moments after we “need” the information, we may find that we suddenly rediscover what we lost in an annoying but normal aspect of how our memories work.

Memory as Two Separate Measures

One way to consider memory is that it can be measured by two separate attributes. The first measure is the measure of storage. Did we encode the memory and keep it in our brains? Even if we did manage to keep it in our brains, that says relatively little about our ability to recall the information at will. There are things that I know, that when prompted I can recall but for which there are few paths in my brain to be able to recall.

This model for memory is the brainchild of Robert Bjork of UCLA and his wife, Elizabeth Ligon Bjork. Their hypothesis is that we evolved with systems that allow us to forget as a natural part of the process. If we had too much in the front of our mind — that is, with high retrieval — we’d never be able to get anything done. The thoughts would constantly be competing with one another. The retrieval paths for some of our memories are trimmed so that they can only be recalled with very specific stimulus.

Desirable Difficulty

Some research points to a desirable difficulty in learning that causes the brain to more intensely link a memory, and this seems to happen with things that were learned once, then “forgotten” or dramatically unlinked for retrieval and relinked. They are so hard to find that our brain seems to not want to make the same mistake of unlinking again. As a result, ideas that are difficult to learn — or relearn — are given special priority for relinking.

In a strange way, forgetting isn’t the enemy of learning; it may be the tool that our brains use to ensure that we’re able to retrieve the right memories at the right times, even if it doesn’t always guess correctly.

Memory is Context Dependent

Have you ever heard that if you study drunk, you should take the test drunk? As crazy as this sounds, it may be correct. Studies with marijuana proved that when someone studied something while under the influence, their performance was better on a test if they were also under the influence. It seems like, somehow, the state of the person got encoded along with the information and the retrieval was linked to that state.

It’s a well-established fact that behavior is a function of both the person and the environment (see Leading Successful Change for more on Lewin’s function). It’s further a researched fact that people’s opinions are related to where they’re asked questions. If you put them in an environment that feels like home, they’ll give more accurate answers about their home life than if they’re placed in an office or at a college. (See Loneliness for more.) It seems that the web of neural connections is shaped by where we are.

The Importance of Sleep

Historically, sleep was viewed as wasted time. However, from an evolutionary standpoint, we find that most animals sleep at times and lengths that serve them. Koalas survive on a very low-calorie diet of eucalyptus leaves and sleep 20 hours a day. The brown bat similarly sleeps all day but during dusk and dawn, when their adaptation of echolocation is most powerful at allowing them to feed on mosquitos, and they are least likely to be struck down by predatory birds. So, too, there must be an evolutionary reason for our sleep cycle. Some of the evidence seems to be repair of our bodies; but more interestingly, it’s essential for the development of long-term memories and learning.

There has been a great deal of research on sleep now, but it wasn’t always that way. In December 1951, Armond, the son of a young graduate student, Eugene Aserinsky, was hooked up to the predecessor of the EEG, and REM sleep was observed for the first time. Aserinsky thought it was a fluke, but test after test confirmed high levels of brain activity during specific periods of sleep – and more than was expected all the time.

Since then, research has progressed. We now know there are various stages of sleep, and these different phases of sleep seem to be performing different kinds of maintenance. Stage 2 is all about motor memory, stages 3 and 4 are for building retention, and REM helps us build pattern recognition. (If you want more on the research into sleep, see The Rise of Superman.)

Trying It Out – Testing as Studying

One of the challenging things about assessing the efficacy of training (see Efficiency in Learning) is that each assessment changes the learning. Assessing retention after a day increases the probability that someone will remember more when tested two weeks later. The finding is relatively easy to explain. They see a greater relevance in the information, because they’ve been tested on it. (See The Adult Learner for more on the importance of relevance.) What’s harder to explain is how, after two weeks, the average performance will climb when compared to the test just one day later. Even without additional studying, performing an assessment will cause the student to retain more than they remember at the first assessment.

There’s not clear consensus on exactly how or why this happens – but it does happen. We don’t know whether the assessment creates desirable difficulty in the learning process, it increases awareness and therefore elevates memories of related topics that can be used to navigate back to the original idea, or whether sleep continues to reintegrate old memories. Whatever the cause, we learn, in part, based on the way that we’re tested. The more that we’re tested on simple recall, the more that we’ll remember things that require simple recall. The more we provide complexity in our testing, the more likely we are to encourage complex storage of facts.

The real test is the test of life. What will you retain from How We Learn – and why?

cash register

Cost Effective Training

There’s a lot of disruption in the training industry – there’s always a lot of disruption in the training industry. However, this disruption sits along the edges and rarely penetrates to the core. The core of what training does – or, rather, is supposed to do – is improve human performance. It’s a tool, like coaching and productivity aids, that is designed to make humans more productive, happier, and healthier.

We’ve got decades of solid research on how people learn – and how they don’t. (See Efficiency in Learning, The Adult Learner, and The ABCs of How We Learn for a start.) We’ve got good strategies for reducing the gap between what we want people to know and what they actually do. (See Job Aids and Performance Support for an example.) Unfortunately, few practitioners have done much research on what does work and what doesn’t. Instead, they rely on their experience and how they were taught. The thinking goes like this: “If it worked for me to learn, it will work for other people.” Accepting that this is true for the moment, that’s not the point. The point in today’s information overload, high-speed, rapid-change environment isn’t whether it can accomplish the objective. (See The Information Diet and The Organized Mind for more on information overload.) The question is whether it’s the most effective way to improve the performance of humans.


Efficacy is measured on whether the humans are able to perform the skills or behaviors that the training is designed to enhance. This is balanced against the cost, both in terms of the individual human learner and the effort in producing the training, including its distribution. The largest shift in corporate training over the last two decades (which is a short time in learning terms) has been the shift from instructor-led classroom training to electronic-based training.

This shift is due to the substantial reduction in cost by eliminating room logistics, flights for the parties involved, and the instructor for every delivery. These costs are substantial, and because they are so large, it’s acceptable in many kinds of training to accept lower learning retention rates through electronic learning and still have greater efficacy. So even though we don’t get as far down the road to our goal of total learning, its cost reduction is so significant it has a higher efficacy.

With electronic learning in place, the primary remaining costs are the cost to develop the course and the cost for the consumers to go through it. Unfortunately, the distributed nature of the cost for people to go through the course makes this portion of the educational cost less tangible to managers and leaders who are looking at the costs of a training program. Thus, the primary constraint on costs becomes the cost to develop the course.

Build vs. Buy

This leads to the classic build vs. buy decision. When should an organization build their own content, and when should they buy existing courses developed by others to leverage economies of scale? The rather simplistic answer is that you build when the training needs to be customized to your organization. The problem is that the lines are rarely clear between the need to customize and the ability to accept mass-market training.

Certainly, when training on the processes inside the organization, it’s necessary to develop the content internally. On the opposite extreme, few learning organizations would believe that customizing the introduction to Microsoft Word course makes sense. The rub comes in when we move to the gray areas like customer relationship management (CRM) software or even advanced Microsoft Word. In the CRM example, you may want to teach the skill (adding an opportunity) with the details of the organization’s rules. For instance, you may need to discuss the specific rules for how to rate the likelihood of closing the opportunity based on your organization’s rules. In the Microsoft Word example, you may have a specific location where templates must be stored or a specific set of styles that should be used for larger documents. In these cases, the skills are infused with the particulars of the organization.

Buy and Customize

A strategy for addressing this need is to buy a baseline set of content and customize it. While this strategy sounds good in theory, in practice it can be difficult to do, as content producers are reluctant to share their source materials with corporations to allow them customization. It also requires a set of skills that many learning professionals don’t have. We have SCORM and TinCan, but there’s not one way of doing things that a learning professional can learn to understand how to customize the content. There’s always conventions of the content producer that the corporate trainer must learn ad hoc.

Ultimately, the most effective answer for organizations is to buy content and customize it, but the market isn’t ready to make this a reality for every organization. For the time being, many organizations are going to settle for buying some content and creating other content. Solutions like the SharePoint Shepherd’s Guide, which offers simple ways to replace screen shots and edit the items, aren’t common, and they’re likely to not be common for a while.

analog volume meter

Training Myth: Feedback is Always Good (or Bad)

In developing training for people, we make several assumptions. Those assumptions are based on our own experience, the environment, or folklore surrounding training. Assumptions aren’t bad. They allow us to cope with a complex set of variables that impact learning. However, challenging these assumptions can help us to create training that is more effective and sustained.

The myth of feedback is a twisting tale. Early research seemed to indicate that feedback depressed learning, while common sense says that feedback is essential to learning. With the advent of more eLearning courses, careful designers are looking for ways to provide meaningful feedback to students and in the process stumbling on age-old controversies.

Research on Delayed Feedback

Learning researchers started investigating the impact of feedback and discovered something curious. Delaying feedback created better learning and long-term retention. One of the conclusions from this research was that feedback was bad for learning. However, it’s only when you look under the covers to the experimental design that you begin to see that what they were testing isn’t exactly the same thing that we’re looking at in our training courses.

The design of the test was such that the feedback was delayed – but it was delayed less than 10 seconds. Most of the time when we’re considering feedback, we’re not considering such short periods of time. We’re evaluating whether we should show a student their result after they have answered a question (hint: you shouldn’t) or after the test (hint: you should). In the computerized learning world, we’re talking about whether we provide effectively instant feedback or whether we delay that feedback.

We wouldn’t typically think of delaying feedback for only a few seconds – but we should use this as a clue.

Training or Productivity Aid

One reason that instant feedback may indeed depress learning is the perception that it’s going to be permanently available. If you can always ask your mother, or turn to your colleague or a resource, why would you learn it? If the resource is always available and easier than learning the material, then you shouldn’t learn the material. Learners are leveraging this phenomenon more and more frequently as the quip “just Google it” flows freely from our mouths on a wide variety of topics.

It may be that when we make getting the correct answer (even through random guessing) too easy, we reduce the ability of the learner to justify the mental expense of committing it to memory. Consider another change in learning over the last 20 years. Twenty years ago, we would be amazed by people who remembered phone numbers. Today, few people know the numbers of their closest friends, because they don’t have to. They select the name in their contacts on their phone and the number is automatically dialed – without you even having to know it for a moment.

As we’re designing training feedback, we need to be cognizant that we want to create a small barrier to getting the feedback so that learners don’t use the training program as a crutch and use that crutch instead of learning.

Learning or Performance

Before leaving the topic of learners using the training as a crutch for not learning something, it’s important to realize that this is a valid strategy when the training isn’t training instead is a performance aid. That is, the content produced is intentionally designed to be a sidekick to the learner when they’re performing the actual task. (See Job Aids and Performance Support for more.)

The fact that it is a valid strategy relies on the awareness that our goal isn’t learning. Though we’re all in the training business, that isn’t what the organization wants. The organization wants productivity and effectiveness. They expect that they can get productivity and effectiveness through training.

When we can bypass the learning process and make employees productive without it, we should do that even if it doesn’t officially match our titles.

When Feedback is Good

While we’ve explored when feedback can be bad, it’s most frequently good. In fact, the lack of feedback strongly inhibits learning. If you can’t see the results of your actions, then you have no way of improving. The psychological concept of flow requires tight feedback loops. (See Flow, Finding Flow, and The Rise of Superman for more on flow.) Why does flow matter? Well, because it is a mental state which generally produces five times the results of other mental states. Generating flow states also improves mood well after the flow state has ended. Even the residual effects help lubricate organizations to better interaction.

High-performance athletes create scenarios where they receive expert and timely feedback so that they can improve their performance to its peak (see Peak). Deliberate practice drives improvement, and feedback drives deliberate practice. The more we can give meaningful feedback, the more we can create opportunities for learning and deeper learning.

In Sum

Give feedback – every time. Give feedback with enough of a barrier that learners won’t use ease of access as an excuse not to learn – unless that’s your goal.

The ABCs of How We Learn: 26 Scientifically Proven Approaches, How They Work, and When to Use Them

Book Review-The ABCs of How We Learn: 26 Scientifically Proven Approaches, How They Work, and When to Use Them

I’m always trying to find ways to better teach and train. I, just like you, have seen plenty of bad training courses, where you want to stab pencils in your ears and gouge out your eyes just to stop the pain of listening and seeing the training session. While not every teaching engenders this response, far too many of them do. My goal is that no one will ever feel that way in my teaching. I desire to create an experience that’s aligned with how adults learn and is based on everything we know about learning through research.

Getting back to the fundamentals is important. When I saw The ABCs of How We Learn: 26 Scientifically Proven Approaches, How They Work, and When to Use Them it felt like I could get back to the fundamentals and review what I knew about learning.

Levels of Learning

If you’ve read this blog for a while you may have seen my review of Efficiency in Learning, which I felt was a powerful book about the process of learning design. It was primarily focused on the detailed level of what strategies to use to minimize cognitive load. In this way, it was focused on the instruction component of the learning process. I believe that these tools are as essential as learning your multiplication tables. It teaches the fundamentals you need to know no matter what strategies you use.

The ABCs of How We Learn looks at the problem from a much higher level. Instead of the fundamental skills of managing cognitive load, The ABCs of How We Learn is more focused on which tool to pull out of the toolbox when teaching. When the question is whether you use an analogy or a worked example, The ABCs of How We Learn has the answer. When you’re looking for what are the barriers outside the training which may prevent learning, The ABCs of How We Learn has the answers.

This is a still different dimension of looking at learning from Bloom’s Taxonomy. Instead of looking at the kind of thinking that is desired after the training, the view is from the perspective of how to ensure that the training stays with the student long after the training is over. It’s a dirty little secret in the training industry that, without reinforcement, 80% of the training a student receives will be gone within two weeks. Not even in baseball would a 20% success rate be acceptable – but in most training situations it is.

We’re All Adults Here, Right?

Most of my training work is with adults. While I support programs that educate children and teens, this isn’t the primary focus of my work. One of the questions that I ask when looking at materials is whether the target is for teaching adults or teaching children. Adults learn differently (see The Adult Learner). There are programs that work well for teaching children – but their child focus makes them not effective at teaching adults. (See “G” is for Growing: Thirty Years of Research on Children and Sesame Street.)

I was pleasantly surprised to see that most of the approaches and techniques described apply whether we’re talking about adult or child learners. Certainly, things like reward take on different context when working with adults compared to children. (Though food seems to be a universal motivator: children are more interested in candy, and adults are more interested in donuts.)

Admittedly, most of the research cited was with children; however, this is to be expected, since most of the educational research being done in the world is done for children rather than adults.

Spelling Learning

The order of the approaches was established by the alphabetic reference. They’re delivered in a strictly alphabetical sequence; however, what struck me is that some of the approaches precede teaching, and some of them are necessary after training. For instance, analogy is a tool used during the teaching. Belonging is used prior to training. Contrasting cases is another teaching approach and deliberate practice is a post-instruction item. Elaboration is something that is done as a part of education process. Feedback is what’s done to help the practice work effectively.

To see how the 26 approaches might look separated into categories of pre-, during, and post-learning, I’ve arranged the approaches in the following table. Note that the “pre” items are mostly setting the conditions for learning either in the environment directly or in the student’s perspective on learning. (See Mindset for more about how a perspective can impact outcomes.)

Pre During Post
  • B – Belonging
  • N – Norms
  • O – Observation
  • X – eXcitement
  • Y – Yes I Can
  • A – Analogy
  • C – Contrasting Cases
  • E – Elaboration
  • H – Hands On
  • J – Just-in-Time Telling
  • L – Listening and Sharing
  • Q – Question Driven
  • S – Self-Explanation
  • T – Teaching
  • V – Visualization
  • W – Worked Examples
  • D – Deliberate Practice
  • F – Feedback
  • G – Generation
  • I – Imaginative Play
  • K – Knowledge
  • M – Making
  • P – Participation
  • R – Reward
  • U – Undoing
  • Z – Zzzzz… (Sleep)

Obviously, ordering the items in this way destroys the neat ordering of the alphabet. However, it allows you to think about learning in a way that’s more connected to the way that students learn. The above table could be further refined by ordering the approaches within these three buckets. For instance, contrasting cases are particularly effective when paired with worked examples.

The Alphabet Song

Despite the relatively short length of this review, The ABCs of How We Learn is a great book to help improve your teaching. Each of the approaches is covered in a bite-sized chunk that you can easily read in a few minutes. That means in less than a month you can get through it. If you’re an educator – formally or informally – it’s worth learning The ABCs of How We Learn.

Recent Posts

Public Speaking