Organizations are paying more attention to ROI when it comes to training leaders, but the focus should stay on long-term results
Measuring the value of any type of learning initiative is a challenge, even for the most knowledgeable measurement and evaluation experts. Even when training proves to contribute to organizational results, many will claim bias in the measurement process.
This skepticism, which applies even to the training of hard, tangible skills, is magnified when it comes to soft-skills training. Although the scrutiny and skepticism most soft-skills training faces are often absent when it comes to the application of leadership development programs, this is changing as organizations continue to make significant investments in leadership development.
Increasingly, management is asking important questions such as:
•What is the impact of our leadership development on our organization?
•Are we reaping the benefits expected from the process? and
•Is the investment we are making worth the return we expect?
Is it possible to actually measure the impact of leadership training or is this type of development too ethereal to define in any tangible way? For those responsible for training and HR, answering this question is the challenge. The traditional approach to evaluating leadership impact is to simply wait to see the outcomes and hope for the best. But this type of thinking is no longer acceptable.
A modified framework
It is possible to measure the results of leadership development programs, as long as the program is aligned with organizational objectives. However, existing evaluation models do not necessarily provide an appropriate framework to measure the effectiveness of training for soft skills such as leadership. The impact of a leadership program is very intangible and is often dependent on the individual who receives the training. This in turn has a direct correlation on the application of the newly acquired skills and its impact on the organization.
What this means is, using the methodology developed by Donald Kirkpatrick and refined by Jack Phillips, measuring the effectiveness of leadership development requires a different emphasis on how each level is applied. Kirkpatrick’s four levels of evaluating learning, developed in 1959, are: reaction, learning, behaviour and results. Phillips later added a fifth level, return on investment (ROI).
An ROI measurement may be appropriate for financial justification in the short term, but does not carry as much weight compared to more qualitative measures such as skills application and business impact. Here’s a look at the different levels of evaluation and their relative significance when it comes to leadership development:
•Level one — measuring the satisfaction: Although this is the easiest level to evaluate, it tends to be the most discounted since many participants typically respond favourably to a course. However, since participants in a leadership development program (mid- to senior-level managers) are also generally responsible for organizational outcomes, a level one measurement will usually receive honest and critical feedback. Measuring the satisfaction of these participants will provide an early and solid indication of whether the program is working and headed in the right direction.
•Level two — measuring the learning: In traditional training programs, the acquisition and retention of the content are important and, at this level, some form of “testing” is required. But in leadership development programs, how the knowledge is applied is more relevant than retention. It’s not that level two is inconsequential, but there must be more of an emphasis on how the knowledge is applied. The important point here is to measure the knowledge gained by the participant compared to what she already knew. In the context of leadership development, level two measurement is not as critical as level three.
•Level three — measuring the application of the knowledge: The application of new leadership knowledge and skills is important to the growth and success of the organization. But the application of newly acquired leadership skills is only as effective as their connection to the business strategy. Level three is therefore critical, but it’s also interdependent with the impact on the business and strategy (level four).
•Level four — measuring the business impact: Strictly defined, this level seeks direct changes in business outcomes resulting from some form of learning. In a leadership context, changes in the business are dependent on how the participant applies the new knowledge and how the objective for change is connected to the leadership development strategy. These issues are not mutually exclusive. If any change is to take place, the leadership development program must be directly connected to the organizational strategy. It is essential to connect level four evaluation to a leadership program.
•Level five — measuring ROI: This is one of the most requested measures in training evaluation, especially by the same individuals participating in the leadership development process. Essentially, measuring ROI involves determining the financial benefits compared to the costs of the program. Even though measuring financial impact is nice to have, it is not the most crucial measure for the effectiveness of leadership development programs. That’s because, in this context, ROI is a short-term measure and dependent on tangible outcomes or returns. Leadership development is intangible. It is a long-term process, as the bottom-line impact may not show up until months, or years, later. So in the short term, when financial managers attempt to measure the return on a leadership program investment, the results often make the program appear ineffectual, not providing any return compared to the significant investment.
Since leadership is primarily a non-financial performance-based skill, financial measures such as ROI are incompatible with the nature of the skills application. Even though it can be measured through some financial results, leadership development requires a more non-financial performance-based approach.
A needs analysis just as critical
When making the decision to move forward with a leadership training strategy, start by conducting a gap analysis to determine the shortfall in talent needed to execute the strategy.
Forcing management to go through the process of determining what gets measured is often an eye-opening experience. This is the first step before any type of evaluation starts and will provide a benchmark to compare results against.
From the gap analysis, create development plans for the managers and executives. Many executive development programs are still “bottom up.” They start at the individual level and ask, “What else does the participant need to become a well-rounded and respected leader?”
Instead, the process should start with the organizational needs and then determine which skills and capabilities need to be developed. This in turn should drive program design, identify the leadership skills the organization requires to attain its objectives and help develop a focused measurement and evaluation process to ensure leadership development remains on track.
When implementing a leadership development program it is critical to find out what is important to the organization and build a credible process in which key members agree on the expected outcomes.
When deciding to measure the effectiveness of leadership development programs, decide what is important to the organization and develop a strategy around these needs.
When a training or HR professional can get all parties to agree on what a good investment looks like, leadership development has a much better chance of surviving the next round of budget cuts.
Ajay Pangarkar and Teresa Kirkwood are partners at CentralKnowledge.com, a Laval, Que.-based consultancy specializing in delivering training and measuring training impact. They may be reached at (866) 489-7378 or at [email protected].
This skepticism, which applies even to the training of hard, tangible skills, is magnified when it comes to soft-skills training. Although the scrutiny and skepticism most soft-skills training faces are often absent when it comes to the application of leadership development programs, this is changing as organizations continue to make significant investments in leadership development.
Increasingly, management is asking important questions such as:
•What is the impact of our leadership development on our organization?
•Are we reaping the benefits expected from the process? and
•Is the investment we are making worth the return we expect?
Is it possible to actually measure the impact of leadership training or is this type of development too ethereal to define in any tangible way? For those responsible for training and HR, answering this question is the challenge. The traditional approach to evaluating leadership impact is to simply wait to see the outcomes and hope for the best. But this type of thinking is no longer acceptable.
A modified framework
It is possible to measure the results of leadership development programs, as long as the program is aligned with organizational objectives. However, existing evaluation models do not necessarily provide an appropriate framework to measure the effectiveness of training for soft skills such as leadership. The impact of a leadership program is very intangible and is often dependent on the individual who receives the training. This in turn has a direct correlation on the application of the newly acquired skills and its impact on the organization.
What this means is, using the methodology developed by Donald Kirkpatrick and refined by Jack Phillips, measuring the effectiveness of leadership development requires a different emphasis on how each level is applied. Kirkpatrick’s four levels of evaluating learning, developed in 1959, are: reaction, learning, behaviour and results. Phillips later added a fifth level, return on investment (ROI).
An ROI measurement may be appropriate for financial justification in the short term, but does not carry as much weight compared to more qualitative measures such as skills application and business impact. Here’s a look at the different levels of evaluation and their relative significance when it comes to leadership development:
•Level one — measuring the satisfaction: Although this is the easiest level to evaluate, it tends to be the most discounted since many participants typically respond favourably to a course. However, since participants in a leadership development program (mid- to senior-level managers) are also generally responsible for organizational outcomes, a level one measurement will usually receive honest and critical feedback. Measuring the satisfaction of these participants will provide an early and solid indication of whether the program is working and headed in the right direction.
•Level two — measuring the learning: In traditional training programs, the acquisition and retention of the content are important and, at this level, some form of “testing” is required. But in leadership development programs, how the knowledge is applied is more relevant than retention. It’s not that level two is inconsequential, but there must be more of an emphasis on how the knowledge is applied. The important point here is to measure the knowledge gained by the participant compared to what she already knew. In the context of leadership development, level two measurement is not as critical as level three.
•Level three — measuring the application of the knowledge: The application of new leadership knowledge and skills is important to the growth and success of the organization. But the application of newly acquired leadership skills is only as effective as their connection to the business strategy. Level three is therefore critical, but it’s also interdependent with the impact on the business and strategy (level four).
•Level four — measuring the business impact: Strictly defined, this level seeks direct changes in business outcomes resulting from some form of learning. In a leadership context, changes in the business are dependent on how the participant applies the new knowledge and how the objective for change is connected to the leadership development strategy. These issues are not mutually exclusive. If any change is to take place, the leadership development program must be directly connected to the organizational strategy. It is essential to connect level four evaluation to a leadership program.
•Level five — measuring ROI: This is one of the most requested measures in training evaluation, especially by the same individuals participating in the leadership development process. Essentially, measuring ROI involves determining the financial benefits compared to the costs of the program. Even though measuring financial impact is nice to have, it is not the most crucial measure for the effectiveness of leadership development programs. That’s because, in this context, ROI is a short-term measure and dependent on tangible outcomes or returns. Leadership development is intangible. It is a long-term process, as the bottom-line impact may not show up until months, or years, later. So in the short term, when financial managers attempt to measure the return on a leadership program investment, the results often make the program appear ineffectual, not providing any return compared to the significant investment.
Since leadership is primarily a non-financial performance-based skill, financial measures such as ROI are incompatible with the nature of the skills application. Even though it can be measured through some financial results, leadership development requires a more non-financial performance-based approach.
A needs analysis just as critical
When making the decision to move forward with a leadership training strategy, start by conducting a gap analysis to determine the shortfall in talent needed to execute the strategy.
Forcing management to go through the process of determining what gets measured is often an eye-opening experience. This is the first step before any type of evaluation starts and will provide a benchmark to compare results against.
From the gap analysis, create development plans for the managers and executives. Many executive development programs are still “bottom up.” They start at the individual level and ask, “What else does the participant need to become a well-rounded and respected leader?”
Instead, the process should start with the organizational needs and then determine which skills and capabilities need to be developed. This in turn should drive program design, identify the leadership skills the organization requires to attain its objectives and help develop a focused measurement and evaluation process to ensure leadership development remains on track.
When implementing a leadership development program it is critical to find out what is important to the organization and build a credible process in which key members agree on the expected outcomes.
When deciding to measure the effectiveness of leadership development programs, decide what is important to the organization and develop a strategy around these needs.
When a training or HR professional can get all parties to agree on what a good investment looks like, leadership development has a much better chance of surviving the next round of budget cuts.
Ajay Pangarkar and Teresa Kirkwood are partners at CentralKnowledge.com, a Laval, Que.-based consultancy specializing in delivering training and measuring training impact. They may be reached at (866) 489-7378 or at [email protected].
