A note on “The Worst Programming Interview Question”

Rod Hilton has a blog post where he argues against using puzzler type questions during an interview. I agree with him in spirit, but feel that puzzler questions have a place in interviews, provided they are used correctly.

Hilton’s piece is very well written and I recommend it, especially if you are in a position where you interview candidates. Hilton is strongly against Puzzler questions because he feels they add additional stress to an already stressful situation and only detect if a candidate has heard the question before and memorized the answer.

I generally agree with Hilton’s sentiments, but, if you form a Union of questions/strategies that different people consider unsuitable for judging job candidates, you will end up with nothing left to ask or look at. This suggests that interviewing is an art: a complex human activity that can be learned through observation and practice, but is probably hard to convey in writing.

When I was interviewing for jobs I experienced a variety of interview styles. I am lucky to report that I enjoyed every interview. I interviewed at three places, and only in one did I get a puzzler kind of question.

In one firm, in the first round I got a very easy ETL (industry speak for scratching out usable, neat data, from a raw data file) problem, which I solved in Python and guessed they were using to kick out folks who could not program at all. In the next round I gave a talk and faced a panel of interviewers most of whom asked me straight technical questions related to statistics (though one of the interviewers clearly had no interest in interviewing me and went through my CV in front of me and asked random questions from it).

At the next firm the first round was simply getting to know me and them (which I guessed they were using to kick out people who could not communicate/were socially unsuitable). The next round I found immensely enjoyable: they gave me some of their data and just let me lose on it with a very open ended question. I had a lot of fun playing with it and ended up making an IPython notebook of my results. They were very thoughtful in giving guidelines: they said they were looking at an applicant’s ability to condition data (detect and toss out bad data) and an applicant’s ability to present analysis.

This was the interview I enjoyed the most and I suspect the interview that took me and the company folks the most time to do. When I was done with my notebook I was very satisfied and happy and eager to go work for these folks. I was invited over for a site interview, but by that time I had already received and accepted a job offer. This brings me to the puzzler question.

The job offer that I finally accepted was from the company where I was presented with a puzzler question. It was a statistical puzzler, rather than a programming/algorithm one. It caught me off guard. I worked through it the best I could, but I could tell the answer wasn’t quite there. My interviewer would listen to me think, wait for my partial answer and then give me a hint. I used the hints to move forward, but I felt I was failing.

At the end, I was pretty sure I would not hear from the company again, so I was surprised when I was asked in for a second interview. This was a broader discussion of work and interests and I felt that went really well, and indeed, I received an offer shortly.

A few months into the job, I spoke about the puzzler question with the colleague who had interviewed me. His reasoning for the puzzler, which I found satisfactory, was that it was a way to test how people thought when faced with a tough problem and whether they thought at all, since the work we do if often difficult and often does not have clear cut answers. I also realized, on my first day on the job, that my colleagues had been reading one of my blogs (this one) and that formed part of their opinion about me.

Hilton’s points are well taken. I feel, however, that there is great value in seeing whether people are willing to think. When I was preparing for my Ph.D. defense one of my advisors told me that some questions during a defense are thrown at the candidate simply to see how they react. The panel assumes – the question being so bizarrely out of scope of the thesis – that the candidate doesn’t know the answer, and they want to see how the candidate thinks through novel situations.

When the job you are hiring for requires people to take initiative and solve problems on their own, given a minimum of guidance, it is very effective to be able to see how a candidate reacts to such situations: do they seem eager to tackle open ended situations, or are they unhappy when directions are vague and outcomes not clear? It is possible that a Puzzler question will be able to unearth some of these qualities.

If I were to use puzzler questions during an interview, I would open by being honest: I would say that this is an intentionally difficult question, it is not meant to test knowledge but rather approach. I would then pick a case from my own work that I had to think through, so I would be familiar with the topic AND it would be a relevant kind of puzzler question.

What, of course, you can not judge from this kind of puzzler question is staying power: the person was willing to spend 10min thinking about a problem, but will this person come back to the problem day after day, week after week, until they crack it?

There are also many other factors at work here: perhaps the person does not like to expose their vulnerabilities to strangers – when faced with the same question with familiar colleagues the same person that choked at the interview could be spectacular to brainstorm with.

Looking back, the interview I liked the most – the one where I got to play with some data analysis – was actually a puzzler question. It was open ended, it did not have a set answer and it was designed to see how I thought. But it was not a cookie cutter puzzler question that you would get out of an interviewing book – it was a real life test, that was relevant to the position they were interviewing for.

Taking this further, I think the only way firms should interview candidates is to have paid internships or trial periods where they get to see a person in their “natural” state, working through the kind of problems they would actually face, and interacting as they normally would.

The problem of course, like many other things we don’t do well, is that we don’t have that kind of time …


3 thoughts on “A note on “The Worst Programming Interview Question”

  1. I apologize if I misunderstood you but are you saying that hard puzzles could be some sort of a candidate’s character testing. Well that could be true, but nonetheless I must say I agree with Hilton. Testing the candidates reaction like this is rather time consuming so to me that does not justify this kind of testing if the reaction is the only value here.
    I mean the interview by itself is stressful and you add more with this so I would guess that quite a few developers could freeze at this point. But is this really the environment that he will be working in? Is the position in that job so stressful that you need to see how the candidate will behave in that environment? I really doubt it…

    Also the way I see is that if a candidate is successful at solving the puzzle there can be two reasons for it:
    – First one is that he actually came up with the conclusion on the spot.
    – Second one is that he already encountered this so he draws the conclusion from his memory.
    The problem is that I believe the first one will occur on very rare occasion and probably by the candidates that enjoy solving puzzles in free time and the second one does not provide any value to the candidate’s insight (and he can even lie and present his conclusion like it was on the spot).

    Note that I as well believe that the candidate should be presented with a problem (for example like those data analysis tasks that you mentioned) and that he needs to resolve it by himself so that you can observe his way of approaching the problem at hand and his way of thinking about the solution.
    But the problem needs to be something relevant to the position because if it’s not then what it the point? The candidate can solve something that he will not encounter when working for you… well that is a great skill to have, right…
    Anyway we actually like to craft the coding tests based on the position’s requirements and we then invite the candidates to do them online:
    Then we follow-up with an interview in which the candidate is presented with a more complex task that requires enhancing the solution from the coding test that he passed.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s