Evaluation of Executive Training
by Glenn Crone
THE ULTIMATE TEST OF EXECUTIVE TRAINING IS AN EFFECTIVE ORGANIZATION. BUT HOW DO YOU MEASURE IF YOU ARE ON THE RIGHT TRACK? POST-COURSE RATINGS PROVIDE AN ASSESSMENT OF HOW ENJOYABLE THE COURSE WAS, BUT NOT HOW USEFUL IT WILL BE. WE ASKED THE CENTRE OF EXCELLENCE FOR EVALUATION AT TREASURY BOARD CANADA SECRETARIAT FOR ADVICE.
It is important to take stock of whether executive development programs are effective – even more so with the cohort about to move into management ranks who have had less on-the-job development due to the baby boomers clogging the top spots.
Evaluation is the application of systematic methods to periodically and objectively assess the effectiveness of programs in achieving expected results, their impacts, both intended and unintended, continued relevance and alternative or more cost-effective ways of achieving expected results.
In the federal government, evaluations should be conducted in accordance with the TB Evaluation Policy and the evaluation standards contained within it. The full range of evaluation issues should be considered in the planning of evaluations: relevance (does the policy, program or initiative continue to be consistent with departmental and government-wide priorities and does it realistically address an actual need?); success (is the policy, program or initiative effective in meeting its objectives, within budget and without unwanted outcomes?); and cost-effectiveness (are the most appropriate and efficient means being used to achieve objectives, relative to alternative design and delivery approaches?).
A crucial step in planning for an evaluation is to ensure that a logic model or results chain is developed to depict the program. It is an analysis of the causal or logical relationships between activities, inputs, outputs and the outcomes of a given policy, program or initiative. Outcomes are usually further qualified as immediate, intermediate or ultimate (final).
The figure below shows a potential generic logic model for a training program: In the past, the public sector focused on the measurement of inputs, activities and outputs as measures of success, e.g., how many days of training were delivered. However, in today’s results-based culture, the focus is on the measurement of outcomes. This is a critical role for evaluation. In the example provided above, this means that organizations should look at the impacts of executive training on the level of knowledge of participants and, to the extent possible, the impact on the participants’ use of this new knowledge in their job and the extent to which this improves their job performance.
A training evaluation model used by many (including the Canada School of Public Service) is one developed by Donald Kirkpatrick in 1959. It requires four separate levels of evaluation:
1) reactions
2) learning
3) transfer
4) results
Evaluations should begin with the first level and progressively move towards the 2nd, 3rd and 4th as time and budgets allow. The information gathered within each level provides an increasingly accurate measure of the training's effectiveness and serves as a basis for the following level of evaluation. The information becomes progressively more rigorous and takes more time/effort to gather and analyze.
The levels of the Kirkpatrick model correspond to the levels of outcomes shown on the logic model. To demonstrate the impact of the program, organizations should strive to implement the highest level of evaluation that is feasible.
Evaluation Results
Over the past few years, there have been several evaluations of executive development programs conducted in the Canadian federal government. This includes an evaluation by the Canadian Centre for Management Development (now Canada’s School of Public Service) of the educational component of the Career Assignment Program (CAP), an evaluation of the Accelerated Executive Development Program (AEXDP) by the Public Service Commission of Canada, and an evaluation by Treasury Board Secretariat of the Management Trainee Program
(MTP).
By and large, these evaluations confirmed the effectiveness of these executive development programs. However, some recommendations were made to fine-tune the programs by modifying participant expectations, better matching of candidates to assignments and providing central coordination of assignments. Canada’s School of Public Service reports that they broadly apply Level 1-2 evaluations to their training programs.
They suggest that evaluations at higher levels are difficult in the public sector context but that they have attempted such higher-level evaluations, including their evaluation of the Accelerated Executive Development Program (AEXDP).
Summary
The evaluation of executive training programs should provide evidence to demonstrate the relevance, success and cost-effectiveness of the programs. Evaluators of federal executive training programs should refer to the Treasury Board Evaluation Policy and ensure that the Evaluation Standards are met in the planning and implementation of evaluations of executive training. The Kirkpatrick model of evaluation is a widely used model for the evaluation of training programs.
Regardless of the evaluation model employed, evaluators of executive training programs should strive to measure the achievement of intended training outcomes as close as possible to the intended ultimate outcome level.
Glenn Crone is a senior analyst with the Centre of Excellence for Evaluation at
Treasury Board Canada Secretariat (www.tbs-sct.gc.ca/eval/).
