Profile of Connie Karlsson
When Connie Karlsson cites familiar training industry maxims, she does it with relish, as if they were her own.
Take the one about formal training having an impact on just 25 per cent of performance. Karlsson likes to revise it by a few percentage points because her own numbers tell her so.
Karlsson is head of Learning Outcomes, a unit of six people inside the learning and development department at TD Bank Financial Group. Their job is to measure the value of the bank’s training spending, which easily adds up to about $50 million in investment a year (with 51,000 employees worldwide, that’s just short of $1,000 per employee).
With an accumulation of training research and data in her tool kit, Karlsson is among those at the forefront of the industry’s transformation, from training to strategizing on business solutions (see story above).
Like many other training and development professionals, Karlsson started out in adult education. She joined the bank 10 years ago as a consultant in interface design. Around that time, Catherine Chandler-Crichlow, the bank’s associate vice-president education and training at the time, decided that the bank had to “get ahead of the game” by showing the value of the training department. She remembered Karlsson had a background in applied social research, and recruited Karlsson as an analyst in a newly formed measurement group.
“It was a good marriage, because I had adult-learning principles. I had interface design. I had instructional design. And I had this applied social research background,” said Karlsson, who’s also a member of the ROI network at the American Society for Training and Development.
The group started out by picking an evaluation model that would work. “Six Sigma was not going to work because we were in a sales and service environment and it didn’t make sense to be that analytical.” They preferred Jack Phillips’ return-on-investment methodology but stopped short of doing the actual cost-benefits analysis known as ROI.
“Everything had to be simple,” said Karlsson. “You take out of it what works for you. You don’t go into this thinking that you have to do everything in the model. So we took a much different approach from how Jack Phillips probably would have recommended. We did not aspire ever to go into ROI. We aspired to become institutionalized in measuring training value.”
The group’s first goal was, “Let’s get really good at Level 1,” said Karlsson, referring to the questionnaire learners fill out at the end of each training session. The questionnaire the group developed was unique in that it went after indications of all four levels of measurement outlined in Don Kirkpatrick’s 1959 training evaluation model (see sidebar).
“It’s not a traditional smile sheet. It’s got really good indicators of business results — Level 4 — job application — Level 3 — isolation of training factors, knowledge gain, and so forth. So that’s all the four levels contained in Level 1 data.”
More than a measurement, Karlsson also sees the questionnaire as a reinforcement tool.
“It was a form of subliminal coaching. It said: ‘You’re here on training to affect your performance, to affect your ability to contribute to the business goals.’ How powerful is that?”
Rolling out the same set of questions across the organization helped lay the ground for benchmarking. Soon, training programs had to meet minimum scores or they had to go back to the drawing board.
“We backed up everything with research. Everything was scrutinized. Why did we use the five-point Likert scale? Or how can we say that coaching really helps? We backed it up with research and put a footnote in. So to me, we weren’t just showing the value of training. We had become a research house.”
The level of rigour that the group established went a long way in institutionalizing the value of measurement and evaluation at the bank. It also helped prime the business lines for the difficult work ahead, particularly for Level 4 measurements of how training supports business goals.
“Level 4, the business owns. We in learning and development don’t own Level 4. That’s just a conversation with the business line where we say, ‘That business objective... how are you measuring that?’” said Karlsson. “Getting the business to tell you what the objectives are at the level of granularity we need to map that to training — it’s a lot of work.”
It’s in helping business units figure out this level of measurement that Karlsson discovers the true value of her work.
“We’ve become business analysts. We’ve created partnerships. The people on my team are not perceived as training designers or facilitators. They’re treated as business partners in helping the business align their business reporting with training. That was the real business win.”
Four measurement levels (plus one)
Donald Kirkpatrick’s four levels of evaluating learning, developed in 1959, are still widely used today. They are:
1. Reaction: How did the trainees like the program? This is typically measured with “smile sheets” at the end of the training event.
2. Learning: What did the trainees learn? This can be measured with formal tests, various forms of informal assessments or a combination of both.
3. Behaviour: How did the training result in a change in behaviour on the job?
4. Results: What were the long-term results that the training event had — on effectiveness, efficiency, quality?
5. Return on investment: Jack Phillips added this fifth level of measurement to derive the cost-benefit ratio of training. In simple terms, measuring ROI involves converting data compiled at Level 4 into monetary values and comparing the sum of these benefits against the monetary cost of the training.
Take the one about formal training having an impact on just 25 per cent of performance. Karlsson likes to revise it by a few percentage points because her own numbers tell her so.
Karlsson is head of Learning Outcomes, a unit of six people inside the learning and development department at TD Bank Financial Group. Their job is to measure the value of the bank’s training spending, which easily adds up to about $50 million in investment a year (with 51,000 employees worldwide, that’s just short of $1,000 per employee).
With an accumulation of training research and data in her tool kit, Karlsson is among those at the forefront of the industry’s transformation, from training to strategizing on business solutions (see story above).
Like many other training and development professionals, Karlsson started out in adult education. She joined the bank 10 years ago as a consultant in interface design. Around that time, Catherine Chandler-Crichlow, the bank’s associate vice-president education and training at the time, decided that the bank had to “get ahead of the game” by showing the value of the training department. She remembered Karlsson had a background in applied social research, and recruited Karlsson as an analyst in a newly formed measurement group.
“It was a good marriage, because I had adult-learning principles. I had interface design. I had instructional design. And I had this applied social research background,” said Karlsson, who’s also a member of the ROI network at the American Society for Training and Development.
The group started out by picking an evaluation model that would work. “Six Sigma was not going to work because we were in a sales and service environment and it didn’t make sense to be that analytical.” They preferred Jack Phillips’ return-on-investment methodology but stopped short of doing the actual cost-benefits analysis known as ROI.
“Everything had to be simple,” said Karlsson. “You take out of it what works for you. You don’t go into this thinking that you have to do everything in the model. So we took a much different approach from how Jack Phillips probably would have recommended. We did not aspire ever to go into ROI. We aspired to become institutionalized in measuring training value.”
The group’s first goal was, “Let’s get really good at Level 1,” said Karlsson, referring to the questionnaire learners fill out at the end of each training session. The questionnaire the group developed was unique in that it went after indications of all four levels of measurement outlined in Don Kirkpatrick’s 1959 training evaluation model (see sidebar).
“It’s not a traditional smile sheet. It’s got really good indicators of business results — Level 4 — job application — Level 3 — isolation of training factors, knowledge gain, and so forth. So that’s all the four levels contained in Level 1 data.”
More than a measurement, Karlsson also sees the questionnaire as a reinforcement tool.
“It was a form of subliminal coaching. It said: ‘You’re here on training to affect your performance, to affect your ability to contribute to the business goals.’ How powerful is that?”
Rolling out the same set of questions across the organization helped lay the ground for benchmarking. Soon, training programs had to meet minimum scores or they had to go back to the drawing board.
“We backed up everything with research. Everything was scrutinized. Why did we use the five-point Likert scale? Or how can we say that coaching really helps? We backed it up with research and put a footnote in. So to me, we weren’t just showing the value of training. We had become a research house.”
The level of rigour that the group established went a long way in institutionalizing the value of measurement and evaluation at the bank. It also helped prime the business lines for the difficult work ahead, particularly for Level 4 measurements of how training supports business goals.
“Level 4, the business owns. We in learning and development don’t own Level 4. That’s just a conversation with the business line where we say, ‘That business objective... how are you measuring that?’” said Karlsson. “Getting the business to tell you what the objectives are at the level of granularity we need to map that to training — it’s a lot of work.”
It’s in helping business units figure out this level of measurement that Karlsson discovers the true value of her work.
“We’ve become business analysts. We’ve created partnerships. The people on my team are not perceived as training designers or facilitators. They’re treated as business partners in helping the business align their business reporting with training. That was the real business win.”
Four measurement levels (plus one)
Donald Kirkpatrick’s four levels of evaluating learning, developed in 1959, are still widely used today. They are:
1. Reaction: How did the trainees like the program? This is typically measured with “smile sheets” at the end of the training event.
2. Learning: What did the trainees learn? This can be measured with formal tests, various forms of informal assessments or a combination of both.
3. Behaviour: How did the training result in a change in behaviour on the job?
4. Results: What were the long-term results that the training event had — on effectiveness, efficiency, quality?
5. Return on investment: Jack Phillips added this fifth level of measurement to derive the cost-benefit ratio of training. In simple terms, measuring ROI involves converting data compiled at Level 4 into monetary values and comparing the sum of these benefits against the monetary cost of the training.
