Survey, developed by IWH, measures leading indicators of OHS performance at workplaces
It had a lengthy and complex tool to assess the safety behaviours, processes and knowledge of New Brunswick employers selected for focused intervention, but it was time consuming.
Normally, an organization’s injury and illness rates are used to measure health and safety performance, but because they represent injuries that have already occurred, they are known as lagging indicators. WorkSafeNB was interested in finding a tool that measured leading indicators. Leading indicators are organizational characteristics and practices that point to possible areas of improvement before workplace accidents or illnesses occur.
Since 2001, WorkSafeNB had been using a questionnaire to identify gaps in a firm’s OHS management system, but because it had to be administered to all employers, supervisors and managers within an organization, it was a lengthy time commitment for organizations — and for WorkSafeNB.
They wanted to make a change, but also wanted to be sure the change they made was effective.
"If we’re going to work towards improving the IRS (internal responsibility system) in New Brunswick employers, then how are we going to measure whether we’re being successful or not," said Anne Lise Albert, assistant director of program development and evaluation at Saint John-based WorkSafeNB. "So, we talked about needing some kind of a survey or tool to measure the (IRS) in a firm."
Albert and her colleagues started their search.
"We emailed our counterparts across the country looking for any experience they had with a survey trying to measure internal responsibility," she said. "And we came up with quite a few jurisdictions."
The team found tools being used in Nova Scotia, Ontario and Saskatchewan. They also found a tool developed by the Institute for Work and Health (IWH) called the Organizational Performance Metric (OPM). The OPM was developed in 2009 by the IWH in response to a call from Ontario’
s workplace injury prevention system.
The tool did just what WorkSafeNB was looking for — it measured leading indicators of OHS performance in workplaces.
"We ended up really liking the work that the IWH had done and how they had developed the OPM," said Albert. "But we were uncertain about the ranking system."
The IWH’s eight-item OPM ranked activities according to the percentage of time they took up in a day.
"Percentage of time is not something that we were used to in terms of the scale. We’re used to scales like, ‘completely agree’ or ‘somewhat disagree,’" she said.
WorkSafeNB decided to develop its own tool. The result was an 18-item survey called the Internal Responsibility System Questionnaire (IRSQ).
Because the IWH had experience validating its OPM, WorkSafeNB asked the Institute to assist with the a validation study of the IRSQ.
When the IWH’s OPM was validated, it was administered through a safety association as opposed to Ontario’s Workplace Safety and Insurance Board (WSIB), which was responsible for both workers compensation and prevention at the time. Because WorkSafeNB is responsible for occupational health and safety enforcement and workers’ compensation, there was concern that a survey administered by the OHS enforcement agency might affect the answers and render the tool invalid.
"It’s kind of like the cops sending you a survey saying, ‘Did you speed?’" said Albert. "So a way to verify whether our presence in this exercise was going to impact the results was to also send out the IWH survey."
Both the OPM and IRSQ were sent out to about 800 employers and more than 300 responded. They were asked to have an employee, as well as a senior manager or a supervisor, fill out both questionnaires. WorkSafeNB then made the survey results and its de-identified historical claims data available to the IWH to assess how well the results of each tool were associated with past injury claims.
The results indicated a strong correlation between the OPM and the province’s five-year historical claims rates.
"Firms that had better OPM scores also had lower claims rates," said Albert.
In contrast, the results found a weak correlation between the IRSQ and five-year workers’ comp claims rates.
"Our own survey was not validated, which is really quite interesting because the questions are very similar — some of them are exactly the same," Albert said.
While more research would be needed to determine why the OPM tracked well with compensation claims and the IRSQ did not, WorkSafeNB was happy with the results.
"Our goal was to have an easy-to-use tool to measure IRS-type questions — and we have it," she said. "Our distribution method — mailing it out by the regulator, sending it back to the regulator — didn’t impact the validation of the (OPM)… so that’s what we’re moving forward with in New Brunswick."
WorkSafeNB plans to begin using the OPM this fall in its Focus Firm program, which identifies firms that have a high accident frequency for their industry.
"We’re going to use the OPM at the beginning of the exercise — before we start working with them," she said. "Then we will measure them three years later at the completion of the program and again in three years after that to measure longer-term impacts."
WorkSafeNB will also incorporate the OPM into its annual client surveys where a random selection of employers and workers are asked to respond to a survey.
"We will incorporate the OPM questions in that survey so that we can get a province-wide average and use that to compare it year after year to see whether we get movement on that score," she said.
IWH perspective
The IWH continues to research the effectiveness of the OPM in Ontario.
"What we really want is a leading indicator for people who score high on it to actually have the lowest claim rates two or three years down the road," said Benjamin Amick, a senior scientist at IWH and head of the research team that worked with WorkSafeNB.
The IWH has data from its work in Ontario from two years ago to compare any new data with, but enough time may not have passed between a firm’s initial response to the questionnaire and what its answers are now.
The IWH is also conducting cognitive interviews with individuals responding to the survey to determine whether respondents are considering the same safety issues when answering the questions.
"We’re literally sitting down with people and as they’re answering the questions, we’re having them think aloud and talk to us about it to find out what are they thinking," Amick said, adding that results indicate similarities so far.
"For the most part, if there are problems, everybody is finding them," he said. "The good news is consistency. The bad news is people are finding problems. We’ll see what happens."
The Organizational Performance Metric (OPM) developed by the Institute for Work and Health (IWH) seeks to determine how often each of the following practices occurs within the workplace:
• How frequently formal safety audits take place
Employer Resource — IWH’s OPM
• How much safety improvements are valued
• How safety compares to other processes, like production
• How much safety information is readily available
• How frequently employees are consulted in safety decision making
• How much power safety supervisors have to make changes
• How positively safety is recognized
• Whether or not the tools/equipment to complete work safely are available.
Have your say on LinkedIn: Does your workplace measure leading indicators?
What tool does your workplace use to measure leading indicators? Is it effective? Join the discussion in our LinkedIn group.
Not a member of Canadian Safety Reporter's LinkedIn group? Head to LinkedIn in and search for "Canadian Safety Reporter."
