The old adage “saving the best for last” may have more resonance when it comes to judging talent contests, as new research has found the order of contestants subconsciously influences how the judges vote.
In the spotlight: Kate Miller-Heidke is competing in the first semi-final heat on Tuesday ahead of Saturday's grand final. Picture credit: Jo Duck.
Macquarie economics researcher Associate Professor Jordi McKenzie looked back over 25 seasons of Dancing with the Stars on NBC in the US and 15 of Strictly Come Dancing on the BBC in the UK, to assess the data on the order in which contestants appeared and how a panel of judges and audiences ranked them.
The research - carried out with British colleagues Professor Alan Collins and Professor Leighton Vaughan Williams from the economics department at Nottingham Trent University - may have implications for many other situations in which people are judged on their performance.
As the crowd in the dancing talent show cheers more, the judges’ scores get higher over the course of an episode
Before starting the project, McKenzie says he hadn’t really watched the popular series, where a celebrity mentors an ordinary person and teaches them how to dance.
“I actually had to get on to YouTube and see a couple of episodes myself just to find out what it was all about. Alan and Leighton definitely were big fans,” he says. “I was having a sabbatical in the UK so we came up with the idea of doing the research together.”
The trio of academics built on the findings of Queensland University of Technology researchers in their study of the international Idol talent quests - where judging is done by audience vote.
Judges replicate audience votes
“They observed there was this J curve of bias comprised of what they call a 'primacy bias'; the first mover effect bias and a 'recency bias; where you remember the last contestants because you saw them most recently,” he says. “Those who performed first and later in the series were less likely to be eliminated.”
The reason why the first contestant on the episode rates well could also be attributed to “producer bias”. The show’s producers choose a performer they think has talent to ensure that viewers stick with the episode.
McKenzie and his colleagues decided to test these bias hypotheses on dancing talent contests. Their research would also push it a step further, because these contests combined two scores, one from the audience and another from a panel of judges.
“We were interested to look at whether this sort of primacy and recency bias translated to a different format and if it had wider implications,” McKenzie says. “We thought it wouldn’t be the same because we thought that the judges would use their expertise. It’s more of a criterion-based system and the judges also announce the results after every performance.”
But that’s not what their analysis of the data revealed. “When we dissected the data, we looked specifically at the judges’ scores during the shows and observed that they also had the same biases as the audiences. The judges’ scores almost replicated the audience votes,” McKenzie says.
Follow the crowd: The research found the mood of spectators at sports matches can also influence referee and umpire decisions.
He points to other studies in sport that demonstrate that calls of umpires and referees can also be influenced by the excitement of the crowd. In this case, as the crowd in the dancing talent show cheers more, the judges’ scores get higher over the course of an episode.
This is a type of “grade inflation”, he says, a phenomenon which usually happens in academic contexts and over a longer period.
The team’s findings, published this month in Economic Letters, could have wide implications wherever people are being assessed on their performance, such as when a panel is assessing job applicants, examiners are marking written or oral exams or judges are assessing sports figures to win a trophy.
And perhaps it’s something to keep in mind when you watch the Eurovision song contest in May. It might not just be about talent - the first and final performers may be the ones that win.
Jordi McKenzie is an Associate Professor in the Department of Economics.
