I am compelled to write all the list of things why it will not work :) unfortunately I don't have that much time in my life. Don't take it as putting you down, I would love to see such thing working and live in real world, so look at it as problems that would need to be solved while aiming for grand solution.
First, there are wrong incentives at play, good guys currently don't need that as they mostly are employed and look for jobs passively. Good guys don't care about certificates and you want to certify good guys so your certificates mean something and can build trust. Weak guys will be spending money to cheat their way around just to get passing grade, which will cost money to prevent.
Then you have hiring managers that will not use this exam as good enough just to hire a person unless there is big company that would do the same. So you have chicken and egg problem, because no one will trust your certification and to get the trust someone would have to use it.
My idea is that you cannot effectively hire at scale. I could hand pick maybe 5 devs/year for my company, getting to real numbers in my small company I can see that we can hire maybe 2 developers per year. When We hire someone it worked for me with simple questions and attitude check. I don't see a way to hire 100 good developers in 1 year, that is not possible. Maybe at a scale you can hire 10-20 good developers a year but you have to have 3-5 people involved full time - and ideally those 3-5 people should be nice and technical - where in the end you don't want to use them only for hiring because those are probably your best developers with soft skills.
For that competency matrix - that is something you can do when you have developers inside of the company and you can observe them for at least 6 months. Good luck getting that in practical testing setup for 2 or less hours. I also find those kind of matrices terrible and not useful for evaluating people or projects. I work on a project where we were not fitting in anything of what was in the matrix .. yet we were delivering good quality for business all the time, while teams that were stuck on their "competency matrix" trying to improve meaningless points ended up disbanded and not working anymore in the organization.
Let me first start by saying that I, too, dislike the prevailing interview process for software development jobs.
While I agreed with many of your points, I could not stop thinking what a huge time burden of implementing something like what you propose, at scale. For a growing company with several work streams and projects hungry for talent, the interview approach you posit would never work.
Another thing that came to mind is the fact that educators and cognitive scientists have been working for decades in what constitutes a good and effective test. Here you seem to claim that somehow you and your folks have "cracked" the system and have come up with a test process that's guaranteed to yield good long-term and performant workers?
Finally, I feel this approach while intended to alleviate some of the most common pain points of the interview process, it tries to reduce it to a number. While it would seem a number is plain and objective, much like a test score, it neglects to "tell the story" and unless all you want are gifted and highly skilled human automatons, you need a way to gauge "soft skills", which one would argue are as important as their core technical competency.
You do understand that there is a very real non-hypothetical cost to doing something like that right? Each of these people needs to be paid, added to payroll by our HR, shadowed by a more experienced dev to give them a chance to ramp up. If you hired 5+ devs that are potentially "diamonds in the rough" you need to have IT give them their hardware and do the onboarding. In hiring it's always quality over quantity and absolutely not the opposite because of precisely this reason.
Another fallacy is basing your hiring on test results. This is silly because you also discriminate by assuming everyone has time to do a take home test which is certainly not the case. If you meant a quick test in-office then there is no way you can get deep enough insight for it to be worth your time.
I wonder if the better approach (for a company like this one) would be to hire 4-5 good programmers, and 4-5 bad programmers, and let candidates write the tests; assign the candidates that most successfully identify the on-staff good coders from the bad ones the highest score?
It sounds about as silly as this scoring does in the first place anyway...
We use a coding challenge as well but I like your addition of "auditing" the coding part and using existing open source or side project work. Bonus points if it uses some of the target technologies too!
The author doesn't appear to understand the economics of the hiring company; it's about time & effort to filter and the cost of making a bad hire. I cannot give every application 1-2 hours of review if I get 100+ resumes. I can't give every interviewee a half-day assessment to get the very poor signal/noise ratio answers to questioning methods suggested. I can't hire easy and then fire easy. I can't invest weeks of work from myself and other mentors on non-ideal potentials. I know this means I pass over great candidates and that really sucks and is totally unfair. I'm sure that's happended to me as well. The problem is nothing presented here is a better alternative.
Work-hire test works for positions that requires failry narrow skill sets. You need iOS 2D game developer or nodejs to MongoDB plumber or single page JS app developer? Sure it would work beautifully. However most companies are not looking for such very narrow skill sets. Companies look for people who are generalists at heart and can be specialized in given area with little ramp up on demand . One week you might write build script and other week debugging Rails and yet another week fixing javascript in UX. When doing these tasks, it's not expected that you already know all these tools and platforms. You will have some ramp up time of 1 day to a week. But the key is that you should be able to move around stack, across projects and get shit done given reasonable ramp up times.
Also a lot of experienced programmers are generalists as well. You might have been C++ system guy in your past life, worked on JavaScript couple of years back and now working on Hadoop backend. You may not remember all the nitty gritty details of past platforms and tools but you can ramp up on demand.
So this 2 hour work-hire test that narrowly focuses on specific tools and platform won't work for vast majority of programmers who haven't branded themselves in to one narrow area. Also remember that a lot of programmers at places like Google, FB and Microsoft work in highly proprietory environment with their own internal build system, platforms and tools. They are likely not familiar with everything that's latest and in fashion. However they are very smart people, can change gear easily and, most importantly, ramp up on demand very quickly.
Yet another big issue is that work-hire test usually fail to measure hardcore computer science problem solving abilities. Sure, you are good at fixing some iOS bugs but can you figure out how to do driving directions on maps or may be scale email system to 100 million users? Now I do agree that lot of jobs don't require hard core CS but many do. And for a startup, it's usually impossible to tell if your future 6 month down the line will hing on having someone with hard core CS skills.
There are many reasons why I think this process is broken that others have written so much at length about enough that I'm not going to bother taking the time.
If it's enough for you to say that because you have a test that everyone takes and it filters your candidates so that's proof of ability, without even going into the results and asking if your test is optimizing for the correct things, there's nothing I can say to convince you otherwise.
Anecdotal, but I know about a half dozen people with experience and CS degrees who would blow your filter but can really get shit done and don't really need any training.
Hiring is hard and the real shame of this industry is that collectively we don't approach this problem with the same level of rigor that we do everything else because by and large we consider this sort of work to be beneath us and not worth a significant investment of our time.
You are often weeding through a large number of applicants, so spending 100 hours on each candidate is just not feasible, and, generally, not necessary. Plus, high quality, experienced developers are not going to want to spend 100+ hours doing real work for companies just to see if they can get a job.
I'm more than happy to spend an hour here or there for a phone screen or coding test to show that I understand the basics of data structures and algorithms and that I can use that information and my reasoning ability to solve problems I haven't seen before.
The core problem IMHO is that a great many of the people hiring developers are not themselves developers and have little to no technical knowledge. Companies depend on these ridiculous hiring processes because many decision makers believe it allows them evaluate candidates that they have no other way to evaluate. They see Google using these approaches and figure if it is good enough for Google it is good enough for them -- not realizing that Google is an entirely different situation with needs that don't match their own (Note to 99% of companies in existence, you are NOT Google and Facebook). I speak from experience. I have been hiring and managing IT staff for 20 years. I also happen to have 25 years of experience in development and infrastructure. One of my priorities when taking over an IT division is to get rid of these useless tests and I have received push back from internal recruiters, HR, and non-technical management every time I have attempted to do this. It requires moving a good percentage of of the hiring process from non-technical management to the IT staff and its not an easy change. Recruiters and non-technical management see it as a threat to their positions. Until there is a realization that you have to know something about a topic to judge the skillsets of others I don't see this improving (I expect that to happen... never).
The problem with this approach is that it is unnecessary time consuming on the applicants side and that the tests aren't language agnostic. The goal to reduce the in-house effort is reasonable, however it can be achieved a lot easier. You can filter the 90% of applicants that are not qualified in tests that only take a couple of minutes. In my opinion the goal of automation in the hiring process should be limited to filtering out the obvious misfits, otherwise you risk to miss hidden talents among your applicants.
In the end you rather want to hire a brilliant engineer with little experience in the desired technology stack than a mediocre developer that is able to pass your automated tests because he has worked with the technology for years.
Assuming the test project isn't something complicated, good devs wouldn't need to spend an entire day fixing it. It definitely would take more time than just sending CVs around, but other screening techniques take time as well, and can be a lot less effective (eg phone interviews).
You're right, it would be pretty difficult to make a test that actually gauges competency. That shouldn't necessarily be a reason not to try though.
Honestly, it might be easier to adjust the hiring process. If the hiring process were able to easily select driven individuals who retain "software engineer" as part of their identity, and people that take pride in their work, seek constructive criticism and want to learn, then that would be good enough.
I should also clarify that being self taught isn't necessarily a bad thing (one of the best coworkers I had was self taught), lacking any of the above traits is.
What these companies have successfully done is weed out all the great devs who value their time very much. Those who are desperate for a job will spend the 8hrs it takes on these silly tests and make it thru the process. Good developers who can take their choice of jobs will just ignore these companies who think they are being clever to require a test, and just move to some other company what is not showing signs that they will be a pain in the ass to work for. IMO, the more difficult the interview process, the more of a pain in the ass the company is going to be once you're hired. If an interviewer is not capable of determining my knowledge level by a normal interview that tells me right away something up, and I probably don't want to work with them because they aren't too smart themselves.
Create a fully automated and objective system to gauge the ability of developers. Then people and companies wouldn't have to waste time performing interviews.
I'm not sure I entirely agree. Having been on the "other side" I can say with confidence that at least some employers truly dislike the hiring process and would be eager to hire the first candidate who shows signs of competence. Unfortunately, at least in the tech sector, competence can be more difficult to ascertain than in other industries. Even lengthy resumes often come affixed to very incompetent candidates. I suppose one could subject them to the indignity of an actual "coding test," but the whole process of finding a test that accurately reflects competence is itself quite difficult. The result is that we've ended up doing "trial runs" with contractors with the promise of full employment after some time, many of whom clearly demonstrate a real lack of preparedness after the first two weeks at the cost of a few thousand dollars.
There is powerful opposition in our line of work to this type of thing, particularly the standardized tests. I don't fully understand it because I think that certification exams, etc. have clear, if limited value.
I think it stems from an intrinsic starting point among many developers that programming is a true meritocracy and that all hiring questions can be settled with a judicious study of what the person is truly capable of (eg. the fabled personal github account filled with side projects). And that anything other than "hacking on the code" is a waste of time.
Of course this isn't how the real world works. In fact almost no one has the time, energy, or inclination to do impressive side projects. Gauging skills on the basis on resumes and very short interviews is quite hard and time-consuming. Hiring decisions are made on the basis of essentially social interactions and personal references.
I do think there is room for an informal credentialing process in principle, but there are longstanding mental blocks that will need to be skirted. There will need to be leadership from the top to make this happen, ie. the best, most-respected programmers will need to take it seriously first.
From the employee's point of view this will always be a hated service because people hate being tested. Especially the majority who are not naturally gifted but only got by because of cramming stuff into their heads. These are perfectly serviceable programmers who will learn what's needed and do a good job in the industry. In addition, you're not forcing employees to put too much emphasis on a single test for a potentially (hopefully?) large group of employees. Does this become a stressor like an SATs for coders? If the employee doesn't do well are they screwed? Or do you allow them to retest? If so, you've become just like any number of certification services that have come and gone over the years.
From the employers point of view you appear to have the opinion that most jobs are in startups. In reality, the huge quiet slowly moving employer base is below the visible portion of iceberg. These are the businesses that just need coders to keep things moving from day to day. They are not startups, never will be, and don't even exist in the world where founders are wanted or needed. There needs in a coder is not a superstar, it's someone who can learn to tow the line, get things done, and keep their heads down and code. A average coder is fine here as long as they fill a niche.
Finally, how could you develop a standardized test for all these below the waterline businesses? If a business needs to fill a slot for a person to crank out Java reports for a few years do you test the same as one who needs to fill a position interfacing C++ to POS/IoT devices?
Not that idea is a bad way to make money. It's just that you'll likely need to move into a niche and then you'll need to drive test retakes, release cram books, and become another paper certificate mill. Remember when paper CNAs and such were all the rage.
One thing that I always thought would really help, which "senior developers" as a group seem to be largely against, would be a basic competency test like the medical board exam or law bar exam. Do it once, and demonstrate that you can know some minimum standard amount of knowledge and can do the bare minimum of software development. It wouldn't test every development skill out there. It wouldn't test particular language competency. It wouldn't test for business domain knowledge. And it would not replace the interview or prove that you were the right fit for any particular job. But it could weed out 90% of the fakers who literally cannot program or know nothing about software that all the hiring managers in these threads are complaining about. It would help to prevent our current mess where every company in the world has to waste time doing FizzBuzz again and again for every candidate.
I don't understand why we are against this basic, minimal gate. We have this unreasonable fear of regulation and gatekeeping in our profession, which is ultimately to everyone's collective detriment. I'm not calling for 20,000 pages of federal regulations on software engineering. A minimal gate. Nobody benefits from the current world where anyone with no knowledge or skills can simply claim to be a software engineer. Nobody except the fakers.
I think that the problem entirely rests with #1. First, consider that this central third party already exists: Universities. I have a PhD, so I should be hired over someone who has only a BS and we should both be hired over someone who never went to college. Except a large number people on HN who do hiring don't agree with this ranking (and rightly so - I'm only an average programmer). Therefore, we need a new organization to do this certification.
We got that new organization. Microsoft did certifying with the MCSE. Of course, I've heard MCSE used more often as a punchline than as a job requirement, so that hasn't helped us too much, either.
Thus, we move into the tests companies actually perform while hiring. Here, we still have the problem that no one agrees on the smartness test. On the one hand, puzzles that I've considered trivial logic puzzles that have been declared obscure memorization tests by people who are certainly more qualified than I am. On the other hand, I've seen programmers who literally couldn't tell a hard drive from a network card breeze through interviews on the basis of rote memorization. Even if we could find a good rubric for grading programmers, we'd need to sell the businesses on this test.
I agree that hiring is completely broken and that outsourcing it to a company who makes it their expertise is pretty much the future of business. Honestly, the reason I've written so much here is because I've spent a lot of time thinking about this idea, too, and these are the issues I've run into. However, until problem #1 is solved, I don't see this going forward.
First, there are wrong incentives at play, good guys currently don't need that as they mostly are employed and look for jobs passively. Good guys don't care about certificates and you want to certify good guys so your certificates mean something and can build trust. Weak guys will be spending money to cheat their way around just to get passing grade, which will cost money to prevent.
Then you have hiring managers that will not use this exam as good enough just to hire a person unless there is big company that would do the same. So you have chicken and egg problem, because no one will trust your certification and to get the trust someone would have to use it.
My idea is that you cannot effectively hire at scale. I could hand pick maybe 5 devs/year for my company, getting to real numbers in my small company I can see that we can hire maybe 2 developers per year. When We hire someone it worked for me with simple questions and attitude check. I don't see a way to hire 100 good developers in 1 year, that is not possible. Maybe at a scale you can hire 10-20 good developers a year but you have to have 3-5 people involved full time - and ideally those 3-5 people should be nice and technical - where in the end you don't want to use them only for hiring because those are probably your best developers with soft skills.
For that competency matrix - that is something you can do when you have developers inside of the company and you can observe them for at least 6 months. Good luck getting that in practical testing setup for 2 or less hours. I also find those kind of matrices terrible and not useful for evaluating people or projects. I work on a project where we were not fitting in anything of what was in the matrix .. yet we were delivering good quality for business all the time, while teams that were stuck on their "competency matrix" trying to improve meaningless points ended up disbanded and not working anymore in the organization.
reply