The "audition project" is a trend story with, from what I can see, very little empirical evidence to back it up. When the most notable example of a company routinely employing audition is Treehouse --- all due respect to Treehouse, but still --- that's a red flag.

The bigger red flag is that the paid audition project method has obvious flaws. The top 20% of software developers are almost uniformly employed. Prospective new employers for these people court them actively; in fact, the problem of companies luring top developers out of other companies is so challenging that large SFBA tech companies have entered into illegal compacts to stifle the practice. Companies are so hard-up for talent that they'll "buy" startups just to get access to their teams.

In this environment, why would a top developer, who has their choice among tens of different high-paying interesting jobs, moonlight for a prospective new employer just so they can make sure the relationship's going to work?

Most technical interviews are terribly flawed. They aren't standardized and they aren't rigorous so you can't compare candidates on any apples-apples basis and you can't correlate them to job performance to make them more predictive. Most of the developers tasked with conducting them suck at interviewing; many use interviews as a sort of hazing ritual, and most use them as opportunity to project their own subjective views about how software should be built onto candidates. Many technical interviews are trivia games punctuated by awkward attempts at working through code on a whiteboard or a piece of paper in a high-pressure environment.

The solution to this problem is to improve technical interviews. It's not to pretend that the whole market for devs has suddenly embraced temp- to- perm hiring. It hasn't. It's getting harder to find good developers, not easier, and the notion that companies have the luxury of inflicting "audition projects" on candidates is counter to reality.

All are essential, day-to-day situations any developer will find themselves in. It's difficult to replicate that in anything other than a technical interview.

Having said that, tech employers don't hold the same power they once did. Decent developers with people skills know they can always find work or just make work for themselves if need be. We're entering an era where these guys will set their own hours and pay and the employer will be the interviewee.

My favorite interviews were when we were trying to hire a senior level engineer, and we were only interviewing people with 10+ years of development experience. We'd ask people to build a priority queue. Items with a higher priority need to be handled before those with a lower priority, but otherwise items should be handled in order.

Can you guess what the most common methodology people chose was?

Did you guess an XML document?

These are people with degrees in comp sci, some claiming 15 years of experience as a developer, trying to get a role at a top 10 bank in New York City.

Of course technical interviews shouldn't be the only test used for hiring, but they absolutely should be included. Maybe it's demeaning that you have to jump through these hoops. Or maybe you're bad at them. Or they're "beneath you" in some sort of way. But don't blame the interviewer, blame the hundreds of other people trying to get a job that they aren't qualified for.

The Stanford course, for the curious: https://web.stanford.edu/class/cs9/

The product we're working on: https://devscreen.io

That's fine - some jobs do require that - but it does mean that any 'best and brightest' rhetoric should be shelved. Everyone knows that tech hiring is broken, but still the trumpets sound about how it really means something to get through one of these processes. It is almost a counter-signal.

Look at past experience, and ask the candidate to talk about that experience to verify that they aren't lying.

If someone has 5 years experience working at Google, and they can talk coherently about technical aspects of the projects they worked on, there is no reason to ask them to code on a whiteboard.

The way it currently works is this: I see you have a CS, degree and 10 years experience at a well known company. Excellent. Now you have 2 eggs and a 100 story skyscraper. Tell me the minimum number of egg drops you'll have to perform to discover the minimum height that will break the egg.

Every engineer no matter how experienced has to answer questions like they just left school yesterday, no other industry does this.

>Personally, I know that to be false. I have worked with incompetent engineers, who literally couldn't code FizzBuzz.

99 times out of 100 you could catch that engineer by talking to him about technical aspects of past projects he worked on.

> some people fail / lose

This comes off as a condescending way of thinking about the hiring process. Even if the candidate is not qualified for the position I want to leave them feeling like I helped them a little in their career or at the very least in their interviewing skills and confidence. You can find a good fit for a role without leaving the other candidates feeling like they failed. And I think that starts to get to the crux of the problem at some companies : the process is meant to boost the ego of the interviewer. Almost every senior dev has a few stories about how the interviewer just wanted to feel superior. We can do far better than that.

My belief has become that the only way to hire the traits that I'm looking for (high technical ability, sufficient education, predisposition to rigor, and -- most importantly -- indefatigable persistence) is by judging a candidate on their work, not their performance in an interview. (After all, software engineering isn't done via pop quiz -- and it's not even a timed test.)

The problem then becomes: how do you judge the works of someone you've never worked with before? Three ways that have worked for me:

(1) Rolodex. This is an easy way to hire reliably: someone on the team has worked with the person in the past, and vouches for them as one of the tribe. Assuming that you trust the person vouching for them, the interview consists of you selling them, not them selling you. This method has high fidelity (though is still fallible), but also suffers from obvious limitations in terms of scale and breadth.

(2) Known curricula. There are some schools where I know (or someone on the team knows) the computer science curriculum very well, and can assess a student simply by the courses that they have taken (or, better, TA'd), the labs they have worked in, or the professors that they have worked for. The fidelity of this method will depend on the school, how well one knows the curriculum, etc. -- and it has all of the strengths and weaknesses of hiring university grads. (It also suffers from serious scale problems and runs the risk of creating monoculture.)

(3) Open source. If you lead open source projects that attract community contributors, you may find it surprisingly easy to coax the best of those contributors into working for you full-time. While I know that this isn't a fit for every company, it has become my preferred way to hire: you are hiring someone who can obviously do the work (because, um, they're already doing it) and who has demonstrated interested in the problem space (especially if their contributions have come during their free time). Importantly, you are also not limiting yourself to a particular geographic area or school or company history or whatever; the people I have hired via open source are people I never would have found otherwise. And, it must be said, they have proven (without exception, actually) to be great hires. There are obvious problems with this as well in terms of scale and (especially) predictability, but I view open source as the farm system for software engineering: it's a way to find and develop talent at lowest opportunity cost.

Edit: I forgot a fourth method that has actually worked for me.

(4) Homework. When interviewing someone who I don't know and who is otherwise a stranger, I will ask them some exceedingly basic questions to assess technical viability, and then proceed to discuss the problems that we're solving and why I personally find them exciting. If they are sufficiently interested at the end of this conversation (which is really more me selling them than them selling me), I will assign homework -- which is generally to take some fixed amount of time (I have usually said no more than eight total hours) to build something fun in one of the technologies that we have built or otherwise use. (When node.js was new, this was to build something in node.js -- but more recently, it's been to build something fun with Manta.[1]) If a candidate comes back with completed homework, you each know something about the other: they were sufficiently interested in you to actually play around (and they like playing around to begin with -- which is essential for us) and you now have some works by which to judge them.

[1] http://www.joyent.com/blog/introducing-kartlytics-mario-kart...

Here's the core of the article right here. The "regurgitate LeetCode-style problems" interview fails to be relevant to the day to day of pretty much any SWE job (unless, maybe, you're applying to work for LeetCode, and one of the duties is to create solutions to problems in under 40 minutes, without outside resources).

Take the author's golden rule and layer on a structured interview process, and now you've got an interview process that will be more predictive of whether a candidate is a good hire than any "throw a random engineer in a room with the candidate and have them answer random algorithm problems" type interview. Even better, since you have an actual process, you can tweak it to be even more predictive of a good hire.

Why don't companies understand this?

Some of the worst candidates have had the best resumes. I'd bet good money they could have talked coherently about their projects, made it sound like they'd cured cancer. But when I asked them to do something not much harder than FizzBuzz on a whiteboard, they choked. Ask them to intersect two lists? They'd use Excel. Ask them how Excel would implement that? They freeze, the interview is over. This was a normal pattern. Thankfully, our recruiters started asking pulse-checking questions, and the number of these interviews dropped.

Interviews need something that can't be directly prepared for and directly rehearsed. Unfortunately, I don't think your solution has provided that. I think it would tend to hire silvertongued people who cut and paste.

I respected that decision but it none of those engineers every got hired by companies I was working with. They all had jobs already so clearly that was fine with them but it seems like there needs to be a better alternative.

One developer would ask what the role entailed, then find a relevant job/project that showed he had done similar work. That was helpful but obviously only works if you have a longer career with many projects to pull from.

Doesn't everyone complain about how technical interviews only measure how well the candidate is prepared for technical interviews? With that in mind, I'd argue that the willingness to prepare for a technical interview has become just another signal. Also you go against your argument just after:

> My first company that hired me out of college got a much better deal waiting for me to complete my senior year. Yes, each year isn't worth the same. This makes sense. Senior year was my capstone -- building valuable communication skills in a team environment.

Firms can not measure soft skills accurately, so this returns right back into the value of the college degree as a signal. You get points for attending a school which provided a teamwork-oriented final course. It's your final grade that matters, not what you learned in that course. You might have developed better skills than the rest of the cohort, but it's meaningless if you didn't get a better grade then them. Everyone would rightfully assume that the others have better developed skills than you, because that's what the current system is able to measure.

One final unrelated point:

> Would you pay 50% the price for a half-finished version of Microsoft Word? Probably not. It'd be far less useful of a piece of software.

The utility of added features is diminishing. It'd be worth a lot more than half, or Microsoft would just keep on piling features and raise the price quadratically (or with some power above one).

I remember when I graduated from a "Top School" and interviewed at "hot startups" from the valley. I aced a lot of the interviews - why? Because I had just taken classes on LinkedLists, Binary Trees, HashMaps, etc... So when they asked me to whiteboard a "shortest path algorithm" it was just rehashing what I did in school.

Years later, looking back, I fail to see the relevance in most of the technical questions. In fact, if I had to do those questions over again today I would probably fail miserably. Yet, I have been in the industry for a while now and have worked with countless more technologies and have accomplished far more than my younger self.

Just because someone performs well in a technical interview doesn't mean they will do a good job. That is the data that really matters. I've interviewed hundreds of candidates as a hiring manager for some big startups, and from my experience technical interviews are not a great indicator of success.

I'm saying this coming from someone who has gone to a "Top School" and done multiple Coursera/Udacity/etc classes.

Yes, someone might be able to whiteboard a random forest or write a merge sort, but do they know how to engineer a system? Can the candidate:

> Communicate well with others in a group?

> Solve unique technical problems?

> Research and learn new technologies effectively?

> Understand how to push back to product owners if there's scope creep?

etc...

These are all things that are not really analyzed in many technical interviews.

As I'm reading this analysis all I can think of is that it is pretty useless - if not dangerous for the industry.

What I've found is that it is critically important that someone knows how to code at some basic level. But their ability to code and explain algorithms on the fly, while probably relevant in academia/research, is such a minor part of the day-to-day of a programmer - At least from my experience.