Performance indicators are most successful when they originate from professionals within an industry
.jpg)
By Edward Chancellor
LONDON (Reuters Breakingviews) - Once upon a time, there was a factory in the Soviet Union that made nails. Planners in Moscow set quotas on nail production. When the quotas involved quantity, the factory churned out many small, useless nails. When the bureaucrats realised their error and set a quota by weight instead, the factory produced big, equally useless nails that weighed a pound each.
This much-repeated tale of Soviet industrial inefficiency is an urban legend. But it contains a large grain of truth. Communism failed in large measure because central planners had inadequate knowledge of conditions on the ground and their attempts at control were generally thwarted. It would be nice to think that we have learned from the mistakes of Stalin’s Russia. As Jerry Muller explains in The Tyranny of Metrics, this is not the case. The world remains in thrall to what Muller calls “metric mania".
It has long been recognised that attempting to manage institutions - whether factories, companies or public bodies - by reference to quantitative indicators has its limitations. When a Victorian politician, Robert Lowe, proposed rewarding Britain’s state schools by their progress in reading, writing and maths, the essayist Matthew Arnold responded that education should be aimed more broadly at fostering “general intellectual cultivation.” The trouble is that performance criteria are generally chosen from that which is most easy to quantify. Fuzzy stuff gets left aside.
There’s another problem exemplified by the tale of the Soviet nail factory. As the U.S. social scientist Donald Campbell pointed out in the 1970s, “the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” Muller takes the reader on a whistle-stop tour of performance indicators as recently applied in academia, schooling, medicine, policing and business. It is a sorry tale.
Muller is a professor of history who has witnessed at first hand the corrosive effects that performance metrics have had in academia. When colleges are ranked by their graduation rates, they tend to lower standards or promote courses with lower dropout rates. When academics are judged by the number of papers they publish, they flood the world with unoriginal - and quite often unreadable - articles. The number of article citations is another widespread academic performance metric. But as Muller points out, a citation in itself does not inform whether a piece of research is valuable or not. Besides, academics have responded with mutual back-scratching.
Recent attempts to measure the performance of state schools have also been fraught. In 2001, U.S. President George W. Bush passed the No Child Left Behind Act. The laudable intention was to close the gap in achievement between schools. The unfortunate result was that educational resources were diverted from history and other subjects towards maths and English, whose performance was being measured. Schools began “teaching to the test”. Across the country, teachers turned to outright cheating to get the required results. Police forces have done much the same.
Healthcare is another victim of metric mania. Surgeons whose success rate is recorded have been known to turn away tricky cases, a practice known as “creaming.” In Britain, Prime Minister Tony Blair’s government was an enthusiastic adopter of performance targets for the National Health Service. Waiting times for operations were targeted, which resulted in resources being taken away from non-targeted treatments. Several hospitals faked their waiting lists. When the performance of Britain’s ambulance service was measured according to response times, call centres falsified the data. Ambulances started to arrive at emergency scenes before their calls were logged.
Nor is finance immune. The creed of shareholder value, which appeared in the 1980s, was designed to encourage corporate executives to act in the interest of investors. Stock-based compensation schemes would align the incentives of senior managers and shareholders. The stakes were high, since individual pay packages could be worth hundreds of millions of dollars. Inevitably, over the years every metric included in executive compensation packages has been manipulated. Today, companies routinely engage in financial engineering to enhance their share price - boosting the executive’s pay package.
There have been many harmful consequences from various attempts to measure corporate performance: companies pursue short-term profit at the expense of long-term investment; excess debt undermines financial stability; practices such as “rank and yank” - a management technique favoured by the infamous Enron - corrodes morale by weeding out employees deemed to be underperforming their peers. Muller suggests that the gaming of corporate performance metrics may be responsible for the slowdown in US productivity. He has a point.
This is not to say that performance metrics and targets should be rejected outright. We need to know how our public services and companies are doing. We live in a world where the interests of agents – whether doctors, teachers or corporate executives – often differ from those they are supposed to be serving. Having said that, metrics should not be allowed to completely undermine institutional autonomy or to promote risk aversion. Furthermore, quantitative indicators should never take precedence to informed judgment when assessing institutional performance.
Performance indicators are most successful when they originate from professionals within an industry, rather than imposed by outsiders. Those wishing to exercise external control over institutions should be aware of the dangers. Goodhart’s Law (as formulated by the British economist Charles Goodhart) states that any measure used for control becomes unreliable. Or as Muller says “anything that can be measured and rewarded will be gamed.” Too many people appear oblivious to this basic fact of life. A close reading of Muller’s excellent, if somewhat brief, introduction to the pitfalls of quantitative measurement should set them right.
