Rod Hilton has a blog post where he argues against using puzzler type questions during an interview. I agree with him in spirit, but feel that puzzler questions have a place in interviews, provided they are used correctly.
Hilton’s piece is very well written and I recommend it, especially if you are in a position where you interview candidates. Hilton is strongly against Puzzler questions because he feels they add additional stress to an already stressful situation and only detect if a candidate has heard the question before and memorized the answer.
I generally agree with Hilton’s sentiments, but, if you form a Union of questions/strategies that different people consider unsuitable for judging job candidates, you will end up with nothing left to ask or look at. This suggests that interviewing is an art: a complex human activity that can be learned through observation and practice, but is probably hard to convey in writing.
When I was interviewing for jobs I experienced a variety of interview styles. I am lucky to report that I enjoyed every interview. I interviewed at three places, and only in one did I get a puzzler kind of question.
In one firm, in the first round I got a very easy ETL (industry speak for scratching out usable, neat data, from a raw data file) problem, which I solved in Python and guessed they were using to kick out folks who could not program at all. In the next round I gave a talk and faced a panel of interviewers most of whom asked me straight technical questions related to statistics (though one of the interviewers clearly had no interest in interviewing me and went through my CV in front of me and asked random questions from it).
At the next firm the first round was simply getting to know me and them (which I guessed they were using to kick out people who could not communicate/were socially unsuitable). The next round I found immensely enjoyable: they gave me some of their data and just let me lose on it with a very open ended question. I had a lot of fun playing with it and ended up making an IPython notebook of my results. They were very thoughtful in giving guidelines: they said they were looking at an applicant’s ability to condition data (detect and toss out bad data) and an applicant’s ability to present analysis.
This was the interview I enjoyed the most and I suspect the interview that took me and the company folks the most time to do. When I was done with my notebook I was very satisfied and happy and eager to go work for these folks. I was invited over for a site interview, but by that time I had already received and accepted a job offer. This brings me to the puzzler question.
The job offer that I finally accepted was from the company where I was presented with a puzzler question. It was a statistical puzzler, rather than a programming/algorithm one. It caught me off guard. I worked through it the best I could, but I could tell the answer wasn’t quite there. My interviewer would listen to me think, wait for my partial answer and then give me a hint. I used the hints to move forward, but I felt I was failing.
At the end, I was pretty sure I would not hear from the company again, so I was surprised when I was asked in for a second interview. This was a broader discussion of work and interests and I felt that went really well, and indeed, I received an offer shortly.
A few months into the job, I spoke about the puzzler question with the colleague who had interviewed me. His reasoning for the puzzler, which I found satisfactory, was that it was a way to test how people thought when faced with a tough problem and whether they thought at all, since the work we do if often difficult and often does not have clear cut answers. I also realized, on my first day on the job, that my colleagues had been reading one of my blogs (this one) and that formed part of their opinion about me.
Hilton’s points are well taken. I feel, however, that there is great value in seeing whether people are willing to think. When I was preparing for my Ph.D. defense one of my advisors told me that some questions during a defense are thrown at the candidate simply to see how they react. The panel assumes – the question being so bizarrely out of scope of the thesis – that the candidate doesn’t know the answer, and they want to see how the candidate thinks through novel situations.
When the job you are hiring for requires people to take initiative and solve problems on their own, given a minimum of guidance, it is very effective to be able to see how a candidate reacts to such situations: do they seem eager to tackle open ended situations, or are they unhappy when directions are vague and outcomes not clear? It is possible that a Puzzler question will be able to unearth some of these qualities.
If I were to use puzzler questions during an interview, I would open by being honest: I would say that this is an intentionally difficult question, it is not meant to test knowledge but rather approach. I would then pick a case from my own work that I had to think through, so I would be familiar with the topic AND it would be a relevant kind of puzzler question.
What, of course, you can not judge from this kind of puzzler question is staying power: the person was willing to spend 10min thinking about a problem, but will this person come back to the problem day after day, week after week, until they crack it?
There are also many other factors at work here: perhaps the person does not like to expose their vulnerabilities to strangers – when faced with the same question with familiar colleagues the same person that choked at the interview could be spectacular to brainstorm with.
Looking back, the interview I liked the most – the one where I got to play with some data analysis – was actually a puzzler question. It was open ended, it did not have a set answer and it was designed to see how I thought. But it was not a cookie cutter puzzler question that you would get out of an interviewing book – it was a real life test, that was relevant to the position they were interviewing for.
Taking this further, I think the only way firms should interview candidates is to have paid internships or trial periods where they get to see a person in their “natural” state, working through the kind of problems they would actually face, and interacting as they normally would.
The problem of course, like many other things we don’t do well, is that we don’t have that kind of time …