Jason Thomas here, CTO at CoScreen. Deep inside I had always suspected coding interviews were in large part bullshit.
Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google.
I was the first full-time engineer hired at the video interviewing pioneer HireVue, and have a decade of experience in the HR and interviewing technology space. This experience made me a firm believer in the structured interview. Line up a set of candidates, give them a set of standard questions in a structured fashion and look for genius in the answer of routine questions.
What could be fairer and more organized than lining up candidates one by one, question by question, rating each, and just hiring the best rated out of all?
Read my post for what I learned and how to set up a truly fair and objective interviewing framework.
No offense, but what could be fairer than offering college admissions to people based on lining them up and having them take a standardized test?
Honesty, there is no perfect solution. There’s a better solution than the status quo but there is no “fairest solution because there are 68,000 ways to define fair
So, this is about coding interviews for hiring developers, and it's more about the goal of hiring a good engineer.
Obviously evaluating them dynamically will be more heuristic, and less fair, but it has worked far better for me than using standard uniform async interviews.
Possibly the best analysis of this I have yet seen is the video Why our generals were most successful in World War II that in Korea, Vietnam or Iraq/Afghanistan: https://www.youtube.com/watch?v=AxZWxxZ2JGE
Essentially, the answer is the hiring methodology. If someone seemed like a best fit for a job then they were hired. They then had two weeks to demonstrate basic functional capacity and six weeks to make significant progress. If at any point things did not work out then they were immediately dismissed and replaced with someone else. The high rate of this process meant that there were always open opportunities even for those who had been dismissed from other positions, so things worked out for both employers and employees.
Great, if not for the healthcare and retirement savings being tied to employment. Having left a role after only 4 months my SO is not happy with so many changes.
I don't think even the employers like it. It costs a lot and it's a pain in the ass to even sign up for if you're a small company. To add an insult to an injury, costs go up every year like clockwork in a largely out-of-control fashion. I worked for several companies now which started from scratch. "Retention" was never a part of the conversation, but "pain in the ass" always was. I think employers provide health insurance simply because employees expect it, as well as to pay you less than they would have to pay you otherwise. It's tax advantaged on their end. If your accountant is any good, it can even be tax advantaged for single person LLCs. Most people don't know this, though. Why this is not tax deductible for "regular" people I have no idea. IMO it should either be tax deductible for everyone, or no one. To have this in-between situation is idiotic.
That's not really the employers problem. I'm surprised more companies don't do this, at least in America where there seems little legislation in the way.
FYI: in WW2, unsuccessful US generals were not dismissed, they were usually just rotated out.
Although US leadership in WW2 was amazing overall, the problems were:
- Pearl Harbor (top 2 military commanders there were relieved)
- Italian amphibious landing failures (slow to breakout)
- B-29 campaigns against Japan slow to ramp up (Curtis LeMay replaced the previous leader, then lowered altitude from 15,000' to 5,000')
- Patton slapped a couple of ill GIs and got disciplined. German intelligence thought it was some kind of disinformation campaign since they had generals shooting their own soldiers with impunity.
In later wars, leaders were hampered with limited rules of engagement, or interference from the White House.
In one of Rick Atkinson's trilogy on the US Army in WW II, some general is quoted as remarking that under WW II rules, Robert E. Lee and some other prominent general--Grant or Sherman would probably qualify for the example--would have been removed from command and sent off to training duties.
Orthogonal to the app being a cool way to do interviews, am I the only one perplexed by the example this person chose to highlight problems with coding interviews?
If I understand correctly, they hired someone who seemed to have a good grasp of algorithms but when it came time to code, took forever to write code and took forever to type something out?
Isn't that the exact thing that code interviews are good for? Finding out if someone can write legible, working code quickly?
Code interviews are terrible for finding out if someone can do deep thinking on a very difficult problem over a long period of time, but the exact problem scenario this hire had should have been caught.
I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly catch people like this?
Jason (article writer) here on my normal ycombinator account.
> but the exact problem scenario this hire had should have been caught.
I think if people weren't taking courses specifically training for engineering interviews, and common interview challenges, maybe this wouldn't be a problem. The truth is gamified programming challenges (commonly used for interviews) do not have a strong relationship to actual day to day programming skills, like reading code, understanding documentation, and doing research.
I suppose I'm just reiterating your point, but I'm not sure why you come to a different conclusion regarding the person I hired being slow reading our code, doing research, and setting up basic tooling being something such an interview process would weed out.
It didn't, and hasn't like I said, any more than a coin flip, in my personal (anecdotal of course) experience.
> I'm not sure how having them write/debug some slightly more complex code in IDEA vs an online Vim approximation is going to suddenly catch people like this?
It won't, if that's all I was suggesting.
I was suggesting good engineers may be being filtered due to duress in the interview being caused by poor tooling.
The real focus was having them do a small semi-real project of some kind, and then having a solid engineer pair-program with them.
This has proven to be a successful strategy for us.
>I was suggesting good engineers may be being filtered due to duress in the interview being caused by poor tooling.
Ah, well said, and I agree. If I don't have Vim keys I'm about 5x slower, so I can imagine how someone used to a certain setup is equally handicapped in an artificially constrained interview process.
Someone called this to my attention in terms of making engineering interviews more fair, and there are some overlapping points, particularly with not biasing your interviews based on tooling:
Website and person look fake AF to me. Let me prove it:
"Requiring engineers to dress in any specific way."
"I've been writing software for over 20 years. "
Old habits die hard. In 1995 you would still be wearing a collar shirt to work. So either he's been amazingly stubborn about the habit or has never worked corporate for a significant amount of time in the old days.
"A huge part of software development is seeing how things are done, and then continuing the pattern. "
A good senior engineer would have added "and breaking that pattern too."
"Success criteria is hidden"
Bullshit. Companies standardize their loops and ask their interview group to grade along dimensions and make people justify their decisions to other engineers.
"Suppose you have a developer who has written code in 7 different languages spread over 20 years. ( me ) Average length of time coding any specific language then is 3 years. Now suppose you have a developer who has written code in only one language for 5 years. Which developer do you think will impress more at a coding interview? See the problem?"
No I really don't. Because if someone says they know Java, they better know the damn in and out of it. I would love to see them talk about how JVM garbage collects as well and other beneficial APIs. You can either know 7 things poorly or 1 thing very well. Any competent engineer would know that persistence and master of 1 thing would mean an insane depth of knowledge: a depth that can't be replicated easily and would make a engineer highly valuable.
"The skills required to create effective software that makes the company money are very different from those skills that are used to pass a coding interview. "
Again, bullshit. What are you and your 7 languages over 20 years going to do make the company money? How is that Lisp, Haskell, and PHP knowledge working out for you?
"Senior engineers who have been writing code for 10+ years may either hate coding interviews or simply be terrible at them because they have focused on delivering business value"
No, because there is something called the System Design interview that they would have a better affinity for. In fact, at the Senior Level, its 50/50 (or 30/70 depending where you go) weight between DS&A and SD. This guy is just plain uninformed.
"They are a form of random chance"
If its truly random, then you should see an equal mix of competent and incompetent engineers hired at all engineering levels. If this was the case, why aren't people being fired en mass all the time? If it was random, then its equally likely that shit engineers would get hired at the same rates as good engineers and you would be forced to believe that Google is full of incompetent morons that magically ship amazing products. I would be stunned to learn that a turd flinging monkey could keep Google Search Engine online.
I could keep going but bottom line: this guy is a fucking fake who would suck 2 dicks to get into FAANG in an instant and has no idea what coding interviews really entail.
Sure, I'll engage with OP. All he needs to do is reveal his identity on the website. After all, a majority of his points are based on his own personal experience so I certainly hope that his own personal experience isn't made up.
I know who the owner is, he isn't fake and reached out to me personally, but I didn't feel comfortable doxing him without his explicit permission.
I don't necessarily agree or subscribe to all of these points, just found it interesting there was someone very passionate about the process of software engineering interviews who had made an entire website dedicated to it.
The author posits that the best way to interview is to simulate an actual work environment for the candidate as if he was working in an actual code environment. In theory, this would be good. After all, the best way to understand a candidate is to see how he actually works.
The reality is that the Leetcode style coding interview, as we know it today, is best approached the same way you would approach the engineering process in general: prototype, refine the idea, test the idea, revise based on inefficiencies, code it. Any semi-competent moron can do this if he gives a damn about his career because this is actually what people should be doing day to day: thinking about how and why the approach to code would be effective. This is logical thinking that can take years to perfect and adjust, especially when your career progression means tackling increasingly difficult problem areas (from deterministic products to nondeterministic infrastructure issues).
Which is why most of the internet thinks the interview is BS: because they think like code monkies, not engineers. Most people believe software engineering is copy-pasting stuff from Github and then moving on when its using software and code to solve a problem.
In the author's eyes, he thinks that more weight should be given to the soft skills. Soft skills that can easily be learned on the job. I mean, how long does it take for someone to dive into the code? Just follow the history of changes of that line. A day or 2 of practice. How long does it take for someone to learn new tools like ripgrep? 2 days. These are things that can easily be picked up with less than a year of industrial programming.
Which is effectively what onboarding is: getting a person used to the tools, nuances, and implicit culture of a company.
And different companies have different needs. When you're a startup, you need to code fast and quick, ship the product, and itterate. In a larger corporate environment, you need to code intelligently, be careful of the leverage you are given, and truly understand what you're writing.
In each of the cases, the dimensions you're judged against are going to be different. You artificially restrict your own candidate pool to culture fits instead of people who can think like actual engineers. For big tech companies, it doesn't matter: they have so many people coming through the door that they couldn't care less. If anything, this hurts startups (ironically) because now they have a smaller group of people to work with. Or a larger one if you're dismissing all the people who can actually thinking like a proper engineer.
I mean, if a startup wants to hire someone who says "I don't know why the code works, I copied it from stack overflow and moved on with my day" then by all means, have him write the Robinhood options platform.
Ceter paribus, I would take the former candidate: someone who's clear in logical thinking, then ram him into reality and the company until he learns. At least he would learn quicker. On the other hand, it would take someone who's the opposite, it would take me at least 5 years for him to even begin to understand to not be a code monkey.
Secondly, the Leetcode style interview is actually scalable. Any engineer can give it and can evaluate the problem solving strategies that the candidate uses. That's what allows companies to be efficient with their hiring: people are interchangeable. On the other hand, doing something as nebulous as judging the soft skills of engineering is going to be way too subjective and give more noise-to-signal.
Example: if an infrastructure engineer is judging a product engineer, there's going to be a big disconnect considering that the problem space of infra engineers is more nebulous and the solution is more unknown. A product engineer would shit his pants if he was asked how to migrate a codebase at scale.
Of course, you could get around this by standardizing the test and pairing engineers with engineers of similar backgrounds. But along what dimensions would you standardize? What if the candidate can solve the issue without tracing the code history? How can you create a test where a candidate has to go through the code history without the answer being blatantly telegraphed (ie. when did this bug occur?)? And how could you do this effectively at scale without screwing over your own company's resources (again, think startups. Only so many engineers to go around.)
A more honest way to achieve the same thing (if the author truly wanted to see if the candidate was well suited for the company) without going into Leetcode would be the company to put a probationary period on new-hires. It may cost more but you might as well filter hard and fast instead of using a easy/wrong filter simply because a bunch of newgrad Reddit idiots who can't pass a coding interview are crying their collective heads off about it being unfair.
But if a bunch of smart and successful engineers can consistently land offers, then why can't other smart and successful engineers? Even Max Howell, creator of homebrew, who flamed the coding interview for being shit (his tweet about Homebrew and inverting a binary tree), still got an Senior Apple offer. So the fact of the matter is that it works. It might not be efficient, yes. But you cannot deny it works.
TL;DR: Author is full of shit about Leetcode interviews and his thesis would actually screw over smaller companies.
Hi, Jason here, CTO of CoScreen, and author of this article.
Thanks for your candid opinion Richard.
> The reality is that the Leetcode style coding interview, as we know it today...
I'd love a guide, or other details about how to best pose a "Leetcode" style interview, and refine it. It sounds like a lot of work, but I'm sharing my personal experience interviewing engineers over the last 20+ years. I don't pose just hackerrank like challenges as the only method, but also whiteboard/and ad-hoc in person interviews.
I'm willing to believe there are ways to get better results using these async type methods. But, for my last several companies, including being in management/hiring positions for several startups, I have found them to be very poor at predicting on the job performance.
> In the author's eyes, he thinks that more weight should be given to the soft skills.
I never stated this. I'm not sure where you are getting this from.
Being able to read code isn't a soft skill, nor is being able to navigate large code bases, debug, read documentation, and research. Absolutely these are skills that you can learn on the job, but not by any means easily.
> And different companies have different needs. When you're a startup, you need to code fast and quick, ship the product, and iterate.
I definitely agree with this.
> That's what allows companies to be efficient with their hiring: people are interchangeable.
Have you been working for software companies where you found engineers immediately fungible and replaceable?
This is really different than my own personal experience.
> And how could you do this effectively at scale without screwing over your own company's resources (again, think startups. Only so many engineers to go around.)
My experience hiring has primarily been for startups, and in both Director/CTO, and other positions, have had to spend an inordinate amount of time hiring developers. Scaling to three, or four engineering hires isn't difficult for a startup.
The cost on the other hand, to the point I made in this article, of making those two or three hires bad hires, is extremely high.
> without going into Leetcode would be the company to put a probationary period on new-hires.
Most good programmers are gainfully employed, and won't agree to a probationary period when being hired... Why would they leave a stable role for one that is tentative --- I wouldn't...
I think the onboarding project (could be paid/contracted) like I suggested in the article, as well as a pair-programming session, is what I am suggesting here. You could think of it as a commitment free probationary period on both sides, without requiring the engineer leave their current position.
> But if a bunch of smart and successful engineers can consistently land offers, then why can't other smart and successful engineers?
I'm not sure, really, I'm just offering advice for startups, like mine, that is earnest and based on my own experience.
>Leetcode interviews and his thesis would actually screw over smaller companies.
I don't really mention leetcode once in my article, and mention other forms of traditional interviews--- like in person whiteboard interviews... I call out doing hackerrank challenges.
I'm not sure why you're so focused on leetcode.
If I would have based my hiring on hackerrank alone, I wouldn't have hired one of the best engineers I have worked with in my career.
These coding challenges really don't prove competency in software engineering, again in my personal opinion, based on a large amount of experience in this area.
It's fine for you to disagree with that, but as the Dude would say: that's your opinion man.
Coding assessments are largely used to filter unqualified candidates and save time. You can't reasonably interview every applicant, so you set a small barrier in place and those who can cross it get the time. Coding in interviews is more about understanding your logical decisions and brain flow, whether you ask the right questions or approach problems in a reasonable way.
Full disclosure, we still use hackerrank for just this internally.
I just think if a candidate scores a 70%, I don't expect them to be better or worse than a candidate who scores 100%.
Having criteria to perform an interview is one thing, but if your interview is just to go in with another toy challenge on a whiteboard, I think it would be much better to do a demo project, and immediately pair-program before making the decision to hire.
Originally I based my ambitions on the collective wisdom of the Internet (what could go wrong?), and of course looked towards the FANG interview process, and especially Google.
I was the first full-time engineer hired at the video interviewing pioneer HireVue, and have a decade of experience in the HR and interviewing technology space. This experience made me a firm believer in the structured interview. Line up a set of candidates, give them a set of standard questions in a structured fashion and look for genius in the answer of routine questions.
What could be fairer and more organized than lining up candidates one by one, question by question, rating each, and just hiring the best rated out of all?
Read my post for what I learned and how to set up a truly fair and objective interviewing framework.
reply