Employee engagement surveys: Junk science?

5 reasons those surveys HR obsesses over may be somewhat unreliable
By Robert Gerst
|Canadian HR Reporter|Last Updated: 12/16/2013

There is a gap between what science knows and what business does, according to American business and management author Daniel Pink — but he was being kind.

When measuring employee engagement, that gap is more like a canyon. This is because employee engagement surveys are largely based on pseudo-science.

The engagement survey industry entered this rabbit hole with the 1998 publication of “The employee-customer profit chain at Sears” in the Harvard Business Review.

It claimed to have found HR’s Holy Grail: A predictive, causal link between 10 employee survey questions and demonstrated corporate performance including customer satisfaction and revenue growth.

Any doubts were removed a few months later when the Wall Street Journal provided corroborating evidence that Nortel Networks had increased customer satisfaction rates and financial returns by working on specific employee satisfaction metrics.

While both studies had their faults, in the 1841 book Extraordinary Popular Delusions and the Madness of Crowds, Charles Mackay could have been describing the response of the HR community when he wrote:

“Whole communities suddenly fix their minds upon one object, and go mad in its pursuit; that millions of people become simultaneously impressed with one delusion, and run after it, till their attention is caught by some new folly more captivating than the first.”

Organizations were soon flooded with engagement hyperbole, with various providers touting the benefits of high engagement, such as greater productivity, profitability and customer satisfaction, reduced absenteeism and higher-quality products and services.

There are five reasons why the claims around these surveys may not always be true:

Meaningless models

Engagement models are built using regression analysis. These look scientific to non-statisticians but the analyses are entirely post hoc. This means:

•these models cannot predict or determine the level of anything, let alone engagement

•the factors or drivers in the engagement model have no causal relationship with engagement

•factors or drivers in the regression equation are not even those most correlated with engagement.

Cherry-picking factors that drive engagement

Engagement models cherry-pick the factors said to drive engagement. For example, one company removed questions highly correlated to engagement but thought to be not “actionable,” including: “Are you proud to be working for your company?’”

Not actionable? Shouldn’t building an organization people are proud to work for be someone’s responsibility?

Measurement scale manipulation

Questions on employee engagement surveys employ small scales — six points or fewer. Smaller scales take advantage of positive response bias, producing overly positive results, an effect amplified when data is analyzed using top two box scores.

Combining four and five on a five–point scale to describe “satisfied,” for example, includes people who are neutral or even slightly dissatisfied. This creates a warm and fuzzy blanket of positive results for an employer that may be covering up serious organizational issues.

Statistical significance is statistical incompetence

The purpose of statistical analysis is separating signals from the noise — identifying what’s important and what isn’t.

One department may have an employee engagement score of 90 per cent and another of 82 per cent, but is this difference important?

Should we expend organizational development effort on the second department and not the first? On both? Or neither?

What if 77 per cent of your employees are “engaged” compared to 85 per cent of a benchmark comparison group, or if 85 per cent of your employees were engaged last year and only 77 per cent this year?

Are these signals of something important? Should we do something?

Employee engagement surveys typically use statistical significance to identify important differences between departments, organizations and time periods.

Doing so is statistical incompetence — it confuses the statistical meaning of significance (detectable) with the everyday meaning (important). This makes employee engagement analysis little more than a bad pun.

Even worse, when statistical significance fails to produce sufficient phony findings, ranking is employed to manufacture more. Ranking doesn’t identify important differences.

It takes advantage of the truism that in any ranked data set, there will be at least one person at the bottom to take the fall. If greater numbers of guilty are required, the lower quartile can be used instead.

The all-encompassing single metric

There’s also the reduction of complex concepts such as engagement down to a single number.

Asking the question “What is the meaning of life?” for example, guarantees a meaningless answer because the concept is more complicated than the question allows.

Boiling employee engagement down to a single measure means you don’t understand engagement, people or measurement.

Overcoming engagement pseudo-science

Pseudo-science is doing real damage to employee engagement, organizational performance, human resources credibility and the reputations of people.

But does this mean we should dump employee surveys? No, they are a valuable source of employee feedback — but only when pseudo-science is dumped for the real thing.

This requires:

•replacing meaningless computer-generated models with relevant questions that can help inform organizational development

•using a mix of open and closed-ended questions, with the latter having seven- to 11-point scales

•replacing statistical significance and ranking with thoughtful data analysis (based on statistician Walter Andrew Shewhart’s rules and physicist W. Edwards Deming’s distinction between enumerative and analytic studies)

•not pretending that complex concepts are reducible to a single number — start understanding what matters to your people instead.

Scientist Richard Feynman said the purpose of science is to keep us from fooling ourselves — and we are always the easiest ones to fool.

To build credibility in the executive suite, HR could begin by ridding itself of fairy tale delusions.

In the meantime, there is a new, more captivating folly emerging on the horizon — predictive analytics. It has a cooler sounding name and a bigger price tag.

Robert Gerst is a partner in charge of operational excellence, research and statistical methods at Converge Consulting Group in Calgary. He can be reached at (403) 266-0061 or visit www.converge-group.com.

Add Comment

  • *
  • *
  • *
  • *