I'm using the same technique. Taking MRs from Junior Devs together with the Ticket description, reducing the scope a bit and sprinkling different errors or incomplete implementations here and there.
In the interview, I'm checking of problems that the applicant notices and see how he approaches the problem.
So far I've been satisfied with all hired who passed that test. And they liked not having to solve stupid whiteboard problems.
My experience was that whiteboard type interviews are often conceptual and fairly shallow (at least in terms of amount of code) to make it easy to review.
A bug in any piece of non trial software could involve a ton of investigation / research as the candidate doesn't know that code base...
You could accidentally give one candidate an absolute nightmare of a bug, another fixing a fairly lame typo.
Could you give some insight as to how you perform interviews?
How do you get the candidate to clearly show they have the broad knowledge and creativity required to be a good developer? What kinds of whiteboard coding problems do you usually give?
Yeah I've considered doing this a bunch of times but my biggest problem is coming up with small chunks of work that require little-to-no domain knowledge and can be done in a day/weekend/week.
We've switched to having candidates do two take home problems that are derived from actual problems we faced and it's worked very well. It cuts down on the number of bad candidates that get to the office and waste an hour of time and it gives a chance to candidates who are good coders but bad whiteboarders.
I did the reading of existing code thing in interviews a lot a while ago (and somehow forgot about it when switching teams, this reminded me to reintroduce it), with the probably common twist of having intentional mistakes at various levels in the code, and it was absolutely great.
I feel it really gives you a good impression of the level of experience the candidate has with actual coding, and opens up for a lot of related discussion, such as security implications. Without, as the article mentions, the tediousness and artificiality of whiteboard coding (which I don't want to completely dismiss, however).
We're going to be working on a lot of code together, so it makes sense to have the interview about doing exactly that.
It is not only a far more pleasant experience for the interviewee,
but it also represents the candidates real world programming skills more accurately.
I agree with #2 a lot, although when I start interviewing not sure that would make the first couple of interviews go smoothly. Usually takes me 3 or 4 interviews to get warmed up to the type of problems people like to solve in whiteboard interviews.
When I just randomly interview a single time, I feel like a dolt, do pretty miserably. After a few interviews all the whiteboarding starts to take effect and I do much, much better.
When I interview people I usually explain a problem I'm working on and see if they can think through it, then ask a lot of questions about what they've learned from their past failures. People that can go into detail about something they failed at and learned from usually impress me, as I suspect they won't make the same mistakes over and over. Also, if you haven't failed spectacularly with something very hard and you've been a software dev for 10+ years, you might not be doing it right.
Obviously, I've been a dev for 10+ years, and I can't pass whiteboard tests very easily, so I suppose maybe I'm not doing it right, either...
Is whiteboard interviewing so bad that people would go out of their way to avoid it? I mean, I've encountered / can imagine horror stories for all the usual alternatives: a) take this 3-day long mini-project, do it using all of our preferred technologies (some of which you'll be using for the first time) and get it back to us - it should only take you 4 hours! Or b) do this on site, using our machines and environments which will be unfamiliar to you. Or c) speak to some random invigilator over Skype. Or d) do a written test, or e) take us through your GitHub collection, which may or may not exist, etc.
My preferred way of testing people was usually to try to get them to show me that they have thought about why they develop software in the fashion that they do. I would prefer to do this in person, at a very high level, but it often ended up being a written test (where necessary, the language was your choice, or pseudocode). Is this wrong?
Random fun idea: pair whiteboarding, have one of the reviewers start improvising a solution (or put on a show of not having done it dozens of times before) and see how the candidate will contribute.
If the problem is not one of the standard ones (that will be perfectly memorized by desperate candidates, while rightfully confident ones may struggle), it will demonstrate understanding, not the irrelevant ability to sequentially write out code in an unfamiliar medium. After all, you probably don't want to hire those who have already been rejected so often that they can code on a whiteboard as if they did it all day.
we do this at my current company.
I think it is a great alternative to whiteboard tests as long as it is reasonably short.
For mobile devs, we ask them to write a simple app solving a problem they should be familiar with and that we have had to deal with in the past.
I agree. It's not a great test. The first and only time I had to do it - we had a huge round table with all sorts of people. It was not natural, and I was not informed ahead of time that it would be occurring.
The main challenge is that you have to write what may be incorrect in order to get to what is correct in most problem solving. When you have people watching and potentially judging you, you try to program in your mind which is difficult. Then it shakes you because time is passing and you haven't written any code. They're drinking beverages and you're sweating, haha.
I will say however, that the company that had me do a whiteboard was very professional in how they handled disarming all these things you mentioned from my mind when they sensed them.
I got offered a job but I had several other offers. But I still had the sense that they were a good company.
I don't think whiteboards are fair, but I also don't think it's fair to write a company off for using them in their interview process.
If you're going to do them, inform the candidate ahead of time and also help put them at ease when you sense the inevitable anxiety. Let them know that you really want to see the approach not the end result and writing bugs is ok.
I agree with the code test, the discussion and whiteboard I would say depends on the type of developer. The former member of my team that produced negative work did well with discussions and popular little whiteboard problems. Our normal policy has become to have applicants do a mini project that represents actual work that we are doing. So far it has worked flawlessly. This guy didn't have to do that because he was so experienced. He ended up creating over-complicated, very buggy code that took more time for us to review than it would have taken for someone else to write. And when we needed to make even minor changes to his code the fastest way to do it was to just delete it all and start from scratch, everything was so complicated and codependent.
He had an appearance of technical skill, was very well spoken, knew all the interview questions and answers, but had really bad judgment and somewhat poor logic when he had to come up with novel solutions. On the contrary, I taught myself and didn't have the CS background to do well in purely technical interviews. But I was given the practical challenge and crushed it. I knew what I needed to know for my job. Of course I had a lot to learn (and still do), but I was productive from the start, or so I have been told.
I guess there is nothing wrong with whiteboard tests, but I think a practical coding challenge where the applicant makes part of an application is much more valuable. You can see that they have basic coding ability, but also how they organize code and break down larger problems. Can they get the big picture right while making clean, maintainable code?
Gainlo (http://www.gainlo.co/) does mock interviews with actual Big N / unicorn developers for a price.
Triplebyte (https://triplebyte.com/iv/afyYdFu/cp , referral link), if you can make it to their final video interview, will also give you full feedback whether or not you pass, for free. They don't do whiteboarding but they do live coding and debugging.
There are a few general tips though... the biggest of which is to talk out loud. They're not necessarily wanting a full solution to the problem (although it helps); the interviewer wants to know how you think, whether or not you can code decently, and if you test. (You prove the last part by running an example through your code while talking about it to find and fix any bugs.)
Indeed, I don't write off whiteboard interviewing altogether - if you know exactly what you are hiring for, it can be very helpful to have e.g. a systems engineer write out some C code or go through tree problems. The issue comes in that a lot of people aren't entirely sure what they are hiring for, just that they need a warm body who can code to some arbitrary ability. That's why I always make sure to ask the interviewer what the burning problems are and what kind of action they are looking for on Day One.
I love this suggestion, and thank you for elaborating! FWIW, the interviews that I've talked about taking as a candidate actually used this approach. And, when I give interviews, I use this approach too.
The only thing I'm really defending here is using a 2nd stage screening process that only takes 30 minutes, before I go to the 3rd stage that takes a half day or full day. @BitL seems to be fighting the screening process, which is a necessary practicality and isn't going away.
The first stage is reading resume and glancing briefly at any side projects, which is ~5 minutes. The first stage is invisible, and filters out more people than any other stage. If anything unfair is happening, it's happening here, and there's little a candidate can do about it.
My thoughts on your outline:
A - getting rid of the whiteboard is of course optional. In my case, having graded exercises doesn't mean I stop using the whiteboard. I only provide a whiteboard for people who want one, and I don't ask people to whiteboard things better done in an IDE.
B - Yes! I do like having exercises that range from super easy to not solvable. It helps to know where people get stuck.
C - Yes! That's very helpful for language specific jobs. Personally, I also like exercises that are language agnostic, to let the candidate pick the language they're most comfortable with. I've loved doing coding quizzes in Python during interviews, and would prefer that over, say, C++.
Interesting. I've done a fair number of interviews for my last few companies, and while I've found that whiteboarding isn't terribly useful, likewise I've found that I really do need people to do some coding. I've definitely encountered people that talk a great story but don't have the chops behind a keyboard.
My usual process (which isn't to say it's perfect, but it's a process I regularly edit to address problems I see) is to give a few, simple problems, and have them actually code it on a system.
* I know that the computer and environment are unfamiliar to you, so I expect there to be chopiness and typos
* I know that most people are nervous during interviews, so I don't fail people if they freeze on one of the questions or miss a concept
* I don't give "trivia" questions - the problems tend to be fairly easy ones that cover your ability to approach basic problems. Example: "here is a nested data structure (like an array of objects). Write a method to pull these sorts of elements out."
* I'm actually hoping you'll screw up - I'm far more interested in seeing how you deal with a bug than if you can dash out an algorithm flawlessly. Nonetheless, I keep the questions simple because I want to minimize the impact of nervousness
* I inform people they are absolutely allowed to use Google, StackOverflow, etc - I'm trying to mimic the actual work experience as much as I can. I WILL judge people on their searches, but I'm pretty loose - I can only recall one person that I dinged for searching, and that's because they went to about.com and copied the answer there without trying to understand it (and it subsequently didn't work). Usually this ends up GIVING people points, because if they demonstrate comfort with finding solutions, I expect that we can hire them and know they'll improve over time.
Despite this, I still ask a few questions that are purely verbal. I want to see if you are someone that will force your preferences on others or are willing to bend. (I'm not looking for a doormat, but I've never met one, so that's a bit moot) I want to see that you are continuing to learn things, because the skills you have now will just not be the skills we need in a year or two.
but overall - I have to judge candidates based on the limited info I can glean in an interview. Of:
* resume
* whiteboarding skills
* discussion skills
* coding skills
...I find the latter to be the best measurement of the options, even allowing that it won't be fully accurate.
One of the best things you could do in a whiteboard interview is just start writing effective unit test cases without being prompted into it.
When I joined my current company, it was still tiny and sketchy and I got shanghaied into a surprise full day interview by the promise of an informal two hour tour. A two hour tour... At one point I was given 1.5 hours to write an implementation of a very core library that most software engineers take for granted every day. Trying to have fun with it, I spent about 20 minutes writing a design document, about 30 minutes coding the implementation, and then the last 40 minutes writing a dozen or so unit test cases (as well as a simple standalone unit test framework). When I ran out of time, only maybe 2/3rds of my test cases passed, and I had just narrowed the problem down to a flaw in one of the assumptions I had enumerated in my design document. All the interviewers seemed pretty impressed by the documentation and test coverage, and didn't really care about the implementation, mostly because it was relatively boring by design.
I ask most candidates to write code on a whiteboard in front of me and I’m not apologizing for it!
The problem is usually fairly easy and if the candidate cannot get to a “describe solution in words” milestone then I will guide them to one. After that I ask them to write code. I will also tell them “the code is the part I care about”.
The specific competency I am evaluating here is “can you turn thoughts into code”. I repeat that phrase in interview training so much I think people make fun of me for it.
Imagine I give you a python list with 100 elements and I ask you to write code that will find all instances of the number “5” and move them to the front of the list. I don’t care about performance.
You know you have to write a loop and whenever you see a 5, remove it and put it at the front. Easy. Not even really an “algorithm”. Some people can make code happen very easily and accurately. But some people really need to really think about it - what control flow to use, off by one errors, bounds issues - and make mistakes. Some people don’t see their own bugs. Some people do weird stuff that makes me think they haven’t seen a lot of code before.
I teach interviewers to evaluate the act of the writing as much as the end product, kind of like when airport security asks you “where did you stay in New York” and doesn’t really care about the answer so much as how shifty you look when answering it. It doesn’t mean you have to materialize perfect code on the board to pass - not even close! But this exercise provides information, and when other exercises corroborate that information we use it to make a hire/nohire decision.
Anyway, bottom line is whiteboard code is a completely reasonable technique to deploy in an interview setting and if you do this you shouldn’t feel bad about it. Much more depends on (a) whether the interviewer is trained and calibrated and (b) whether the company knows what it is even trying to evaluate than the question format.
I wasn’t referring to the in-office vs. take-home aspect, just the “stress test” nature of the white boarding test. (And I was admittedly unclear about that in my post and I’m sorry.)
Think of ways you can de-stress the candidate while extracting useful information. Ideas:
1. Have them bring something they wrote with them and describe it in the interview. A lot of developers will “come out of their shell” when they start explaining something that they worked on.
2. Show them some code and have them explain it to you. I had a manager open a C book once and had me explain some code.
3. Pair program on a problem that the interviewer doesn’t know the answer to. This situation is a lot more like a true work situation.
In the interview, I'm checking of problems that the applicant notices and see how he approaches the problem.
So far I've been satisfied with all hired who passed that test. And they liked not having to solve stupid whiteboard problems.
reply