'Use of AI to provide those answers is not going to be acceptable'
As AI becomes more of a workplace reality, there is a new worry emerging for some HR professionals.
Tools such as ChatGPT might be used to fake personality tests and a new study was recently conducted that shows just how easy this might be.
“We administered a personality assessment to these large language models (LLMs), and instructed them to provide the best response faking a given job descriptions and then compared those results to the previously tested group of students who were also instructed to fake and our outcomes were quite surprising,” says Jane Phillips, master of science, management at Wilfrid Laurier University in Kitchener, Ont.
The research, entitled Can a computer outfake a human? was conducted by Phillips, and Chet Robie, professor, organizational behaviour and human resources management, also with Wilfrid Laurier.
The researchers deployed ChatGPT-3.5, ChatGPT-4, Google Bard and Jasper against two personality assessments, she says.
One of the tests was fairly easy to fake, while the second one was a bit more challenging.
“A transparent personality assessment, which is really the most prevalent in these kinds of assessments, you’re asked to just simply rate the degree in which a certain behavioural tendency is descriptive of you. Those are normally on scales that sound like somewhat agree or agree or neutral, those are actually fairly easy to fake, and then other assessments ask you to choose between several behavioural tendencies. Those are a little bit more difficult to fake. They just require a higher level of technical sophistication to develop and are more likely to evoke negative candidate reactions, so they’re not used as often,” says Phillips.
One of the biggest surprises from the study came from one of the more well-known tools.
“ChatGPT 4 her was really good at faking compared to the student population and it performed within the 85th and 98th percentile when looking at extraversion and conscientiousness respectively. It performed very well,” says Phillips.
Many employees are feeling overwhelmed by the pace of AI in the workplace, found a recent survey.
The end of personality tests?
So does this mean that organizations and HR departments should stop using personality assessments?
“I think that they are a great way to assess more ingrained aspects of a candidate; they definitely should be paired with multiple hiring approaches so don’t rely on them alone. But definitely, it’s good for HR practitioners to be aware of how easy it can be to use LLMs to fake assessments administered online,” she says.
With all of the recent advances in AI, “it’s not overly surprising that within that there’s going to be some pretty good writing,” says Jenson Leung, an employment lawyer at KSW Lawyers in Vancouver.
“And depending on the nature of those personality tests, especially if it’s relatively formulaic, which usually would be the case because in order to have that standardized test, you basically have ideal responses and you have certain things that it’s going to be looking for so it’s not surprising that if the AI has been trained on a similar data set that it is going to be able to spit out those ideal answers.”
There are some tangible steps HR and employers can easily implement to prevent this from happening, he says.
“To deal with the possibility of people using AI to generate responses to interview questions is in potentially having language built into their job postings that makes it clear that use of AI to provide those answers is not going to be acceptable and that if it is discovered that a candidate has used AI, then that could be cause for termination,” says Leung.
While this study has shown what might be possible, it’s not really coming up in many discussions with employers, says Leung.
“I’ve heard anecdotally of instances where people are using AI for uses such as drafting correspondence, or drafting cover letters, that type of thing. People are certainly becoming increasingly aware of it and especially as more articles come out it’s going to be more and more on candidates minds. I haven’t heard of a significant trend towards this but I do think that it’s a matter of time.”
The danger to organizations is real, says Phillips in making flawed hiring decisions.
“Putting the wrong person in a job is a huge risk for organizations: the increased turnover, training that they’re investing in hiring decisions can cost an organization a lot of money, and time and resources so inaccurate measures would increase the likelihood of hiring the wrong person and candidates might think, ‘I can use LLM to make my score on this assessment better,’ but that doesn’t really help the individual or the organization.”
Organizations should become overly reliant on AI’s accuracy and utility, says another expert.
The right way to make tests effective
In order to fight this potential, there are some ways to present these tests for candidates, she says.
“Suggestions that are easy to make changes in are things like timed assessments or live assessments; in-person interviews and combining personality assessments with other ways that they’re measuring candidates as well,” says Phillips.
Should employers consider dropping personality tests?
“Hiring the right person for the job, which can lead to better retention, overall morale and a number of other factors if you just get the right person in the right job, that’s a home run for an HR professional and personality tests have been helping with that for a long time and there’s a lot of support for personality assessments in past research, personality assessments and testing for certain personality traits, is directly linked to higher performance in the right job,” says Phillips.
HR should be aware of the limitations inherent in these tests, says Leung.
“I’ve personally been skeptical of personality tests partially because, particularly for people with a sufficient degree of familiarity with personality tests or even just a higher level of education usually, it’s not that hard to determine from the personality test, what answers are ideal answers.”
“For example, if there’s a prompt that asks you what you would do in a team-based scenario, it’s generally quite obvious to most candidates that they should come across as a team player in that prompt. I personally think that there’s limited utility although of course, there’s going to be studies regarding the extent to which these are actually effective in hiring,” he says.
