I knew I needed to build some kind of survey to see if dropping the time limit from the code test would have any measurable impact on time spent or pressure. But I wasn’t sure if it should be anonymous or not.

On one hand, I assumed the data for an anonymous survey would be more reliable as people would be more honest. On the other, we could get more info about the outcomes of the candidate if we knew who sent it.

To figure out the best path, I asked myself two questions:

• What am I measuring?
• Are people more honest in an anonymous survey?

What am I measuring?

I wanted to see if removing the time limit had an impact on:

• Time spent taking the test
• Pressure felt from the test

In my instance, knowing the outcome of the test (did they pass or not, do they end up being hired, etc), did not influence either of those pieces of data. While that extra info would be interesting, it would not help me answer my core questions. As a result, I felt anonymous was the best choice.

Are people actually more honest in an anonymous survey?

This decision relied on my assumption that people were more honest in an anonymous survey. I figured someone had thought about and researched this before.

A quick search turned up the The Impact of Anonymity on Responses to Sensitive Questions by Anthony D. Ong and David J. Weiss, published in the Journal of Applied Social Psychology in 2000.

They designed a study where they knew if people had cheated on a test or not, then asked them if they cheated under confidentiality vs anonymity. In confidentiality only 25% told the truth, while 75% told the truth under anonymity.

The really interesting (and funny) part is how they designed the study. Basically, they wanted to see if people would self-report cheating in a scenario where they could tell if a person had actually cheated or not. 😈

They told people they’d get $25 if they got a score better than 17/20 on a test with really difficult words. There was a dictionary amongst some books set out that the participant could access, but they didn’t mention this. They would know if the person cheated based on if the dictionary was moved or if a bookmark in it ended up in a different spot.

Then, the pièce de résistance:

In order to ensure that the words would be difficult enough to inspire cheating, we made up the last three words.

The Impact of Anonymity on Responses to Sensitive Questions. p. 1698

The whole study is quite clever and funny. It’s well worth a read.

Anonymous is Best for Honesty

In the end, I went with an anonymous survey because I needed to be able to trust the self-reported time and pressure results as much as possible. Anonymous surveys are more reliable in this sense, and the extra info gleaned from a confidential survey would not have helped me determine the core goal of the study.