‘We’re going to continue working to do our bit to eliminate bias’
An ambitious plan to use name-blind recruitment to combat discrimination in the federal public service has come back with mixed results.
Removing identifying candidate information from job applications had no effect on the screening decisions of managers when it came to applications from visible minority groups, according to the Public Service Commission of Canada (PSC), which carried out the pilot project.
“We wanted to test it, we know other jurisdictions had done similar types of pilots or tests, and we were aware in some cases it did lead to different outcomes,” said Patrick Borbey, president of the PSC. “So we went in with an open mind, and wanted to make sure the pilot was going to be as sound as possible.”
“It’s opened up some further work, further avenues we want to pursue, and it’s also raised the level of sensitivity to this issue among managers, so that’s a good thing,” he said.
“I know there will continue to be people who feel frustrated they’re getting screened out, and we can’t guarantee there’s not discrimination, there’s not bias known or unknown, but we’re going to continue working to do our bit to eliminate that as much as possible.”
But it’s not clear there was a problem to begin with, according to Philip Oreopoulos, professor of economics and public policy at the University of Toronto.
“It was a really nice attempt to follow up on some of this literature to try to do something, and they did a great job at trying to execute a pilot and randomized control trial… but given the infrastructure that they already have in place to try to avoid discrimination… I wouldn’t be surprised if there was minimal name discrimination going on to begin with.”
The results weren’t that surprising because it’s not clear the government, based on the way it was running the process, fully understood what it was trying to achieve, said Michael Bach, CEO of the Canadian Centre for Diversity and Inclusion in Toronto.
“They’re trying to eliminate bias in hiring, and I just feel like the methodology they used was very scientific, but the reality is it doesn’t necessarily take into consideration the human condition.”
“Just as an example, they looked at groups based on employment equity categories, which are arguably very dated. And the whole point of name-blind recruiting is around the bias found in names… but if your name is John Smith, and you’re black, well the bias there is not in your name; but if your name is Dimitri Metonev, the bias is in your name but you’re white.”
The project launched nearly one year ago, in April, with federal departments wiping out some of the applicants’ information during screening, with hopes of strengthening diversity and inclusion. The pilot examined results from 27 external job processes across 17 participating organizations and involved more than 2,200 candidates, of which 685 self-declared as visible minorities. Overall, 54 independent reviewers participated in the pilot, resulting in 4,452 independent screening decisions.
Applications requiring anonymization were assigned to trained anonymizers for redaction, and these applications were then quality controlled via a second trained anonymizer. The information redacted was: names, citizenships and country of origin; mailing addresses and phone numbers; educational institutions; references to organizations, businesses and establishments where general training and professional experience were acquired; languages spoken and written; references to geographical locations; references to employment equity groups; references to religion; and references to publications.
In the end, 47.3 per cent of visible minorities were screened in using traditional methods, compared to 46 per cent of visible minorities using name-blind recruitment. However, 48.7 per cent of all other candidates were screened in using traditional methods, compared to 42 per cent using name-blind recruitment.
The government admitted to a few challenges during the study. For one, reviewers were aware they were participating in the project, which could have potentially affected their assessments.
The process of anonymizing the information also proved to be very labour-intensive, taking between 15 and 20 minutes for each application, found PSC.
“It was more work than expected; in fact, we had some managers who were quite worried about the extra burden,” said Borbey. “We identified 20 minutes on average per application was required in order to anonymize, so if you’re a manager and you’ve received 200 applications, that’s kind of daunting.”
As a result, PSC is looking at whether there are some technology solutions that could eventually make the process easier, he said, “so there could be digital screening, but that’s going to require a fair amount of work and we are looking at whether other jurisdictions have some good lessons learned there, and haven’t found good examples of where digital screening works perfectly.”
There have been attempts at software-based solutions that are intended to do exactly this, to remove names and identifiers, with varying degrees of success, said Bach.
“There will never be a perfect technological solution that will address bias in hiring — you just can’t get a computer to do this work, there will always be an opportunity for a person who has a bias, whatever that bias may be, to eliminate a candidate based on whatever that may be, whether it’s race or sexual orientation, who knows. There will always be a human component to it.”
PSC also admitted some identifying information may have been inadvertently left unconcealed, while some tangible, skill-related information may have been concealed.
“That’s a risk, that there’s so much that’s taken out that you lose some context, key context,” said Borbey.
“We tried to be as thorough as possible in terms of taking out any information that would give a hint of origins, so that leads to a lot of information being excised.”
It’s a Swiss cheese approach, he said, and as a result, “managers in some cases don’t have as much information to be able to make a judgment as to whether this is a person that should be allowed to go to the next phase.”
It would have been a lot easier to just start by stripping out the name, said Oreopoulos.
“It may not get rid of personal information and all the identifiers but, in a lot of cases, it may remove a large part of the ability to identify whether a person is coming from an ethnic background or immigrant background or not; but even if it didn’t, one thing it would suggest is it’s the first impression that might matter, the first thing you see on a resumé is the name and before everything else, you have an initial subconscious reaction that may make you more prone to one type of decision versus another,” he said.
“And before you look at any part of a resumé, you make a decision, so just blinding the name may actually have some impact.”
The government’s process name-blinded a lot of items, said Oreopoulos, “anything that may have indicated their background, and that may have thrown out the baby with the bath water in someone trying to understand whether this resumé was worth looking at. Basically, you strip everything out, including where they got their education, which may be valuable information to the person trying to make the decision, so they may have complicated it… and that process may have impacted the ability to actually make good hiring decisions.”
The problem is that name-blind hiring doesn’t address the systemic bias that exists, said Bach.
While a racialized person or a newcomer to Canada might get an interview, she could still face bias around her lack of Canadian experience or because she speaks with an accent, he said.
“I would rather see addressing of those more systemic issues that are present in the recruiting process, and far more difficult to really deal with,” said Bach.
“That’s why I don’t think hiring should be done in an individual format, I think you should always do panel interviewing — it should be a weighted decision and that way you can start to identify biases that are at play that you don’t necessarily see. You need to be looking at the hiring process in a very holistic manner as opposed to these one-offs of ‘Let’s take off the name and make it work.’ Well, no, that’s one piece of the puzzle — you’ve got to look at the whole process.”
To complement the recent findings, the PSC said it will undertake audit work, starting in May, to explore the success rates of applicants at key stages of the appointment process.
It will also explore how name-blind principles could be included in the design of any future technology changes to the government’s recruitment systems.