Do you have any hard evidence that this is actually a skill that some people have and others don't? How do you even measure it to verify?

I'm genuinely curious -- I've been thinking in these terms myself recently, too, but my literature search came up empty.

Have they specified that you should be testing your code? If they haven't specified then it is wrong to expect some kind of arbitrary outcome.

People write code in many different ways, some do it very chaotically, some have very organized process.

I try to figure out which of these behaviors are fundamental to writing good code and which are not.

For example, I believe a habit of writing readable code IS fundamental to writing good code. If you don't have a habit of writing readable code, on average your code is going to be worse than if you had.

On the other hand, I believe writing unit tests IS NOT fundamental. Unit tests is one possible way of working with the code, but people were writing reliable code before unit testing became popular. Unit testing takes time and effort that can be allocated differently.

And so I try to look (I can't call it measuring because it is subjective) only at the behaviors that I believe are fundamental and dismiss others.

In the end, if you are smart person that can program well, you will be able to learn those other behaviors. But if you are not smart or you lack certain fundamental abilities, it is going to be much more difficult to fix it.

Right, and having that algorithmic intelligence is key part of me being a good developer. But testing how someone implements an algorithm in time-scarce circumstances is not a good test of that kind of intelligence because it is likely to favour people who have been exposed to that particular algorithm before (even if they only rote-learnt information about it - as many interviewees seem to do in practice) over people who have the capability to think deeply and reason correctly about it, but have not previously been exposed to that problem.

This is exactly how I feel about TDD, but it always feels like you're not supposed to say it. Even in environments where features are described, planned, designed, refined, written as ACs, and then developed, there are still almost always pivots made or holes filled in mid-implementation. I feel like TDD is not for the vast majority of software in practice - it seems more like something useful for highly specialist contexts with extremely well defined objective requirements that are made by, for, and under engineers, not business partners or consumers.

Most places I've ever worked have made automated testing impossible but then gone ahead and made up for it by just never testing anything and then blaming the programmers when bugs show up in production.

Am I supposed to trust you more or less from this introduction?

> But neither codebase has tests, for example, that take a long-ass time to run.

Is this for real? Any hint of tongue-in-cheek? The two projects have tests. But they don’t have long-ass-running tests? It sounds like they might test the core data structures and core logic. But maybe they don’t “integrate” by setting up and tearing down all application state and friends. So maybe what is ostensibly missing are those integration tests to complement the lean core-logic tests.

And it’s surprising that the heavy integration tests leave little return? Why? They are long-ass-running, they are hard and messy to set up, and in the worst case they just work as a smoke test, which you can do manually once in a while anyway.

Losing faith on testing for what? They already do testing.

Exactly. We're (basically) saying the same thing here. I kept my comment short-and-sweet as not to get into the nitty-gritty of everyone's different work situations/requirements/stances on different testing practices/building an app vs a library or service/programming paradigm/etc.

To expand ever so slightly: "These days I'm all about 'dirty integration' first to help inform my ultimate design."

In enterprise software, the pattern i see is specifically coupling rather than garden variety complexity. There’s a really pervasive idea that unit testing means 1:1 mapping of test class to production code class. To put it another way, there’s this really common but terrible idea that tests should map to your current implementation rather than the behaviour you want.

Worse, some check in test cases that should only exist locally during development, they do things like directly test private implementations. A related issue is no one ever deletes tests.

Testing done well, is utterly transformational. I took a piece of enterprise software, it wasn’t badly written per se but it was hard to release. 80%+ of releases were rolled back due to issues found in production. For each release, 1 of 7 “experts” were required to review your change to this component (to make it worse it changed fairly frequently). The problem was only 2 of those 7 had a track record of only approving good changes. I’m ashamed to say i was one of the other 5 - i reviewed and approved a couple of changes that had to be rolled back from prod.

So, i thought sod this and i grabbed the stingy nettles. The code was not written to be testable but fast forward 16,000 (i had a small team) test cases which execute in around 16 minutes and over the past years that software continues to be released as frequently but has NEVER had a prod rollback since. The 7 experts idea is disbanded, it’s just a normal pull request these days. Lead time on changes is about 1/3 what it was - all the faffing about exploring model output in excel for hours if not days is gone, it’s just all captured as tests.

It wasn’t cheap to do this but back of the envelope calculations, it repaid within 3 months just on lack of prod outages.

Right. If you don't know what it needs to do, why in the HECK would you be writing any code at this point??

If you're just hacking / prototyping, there's no conceivable reason to involve tests.

Well said. I’ve experienced an interesting anti pattern first hand in my company. They hired “Uncle Bob” to give a talk on TDD. Then after that managers are talking about how we need 100% coverage and comprehensive unit tests. Most of these tests did not matter and did not give me any confidence in the safety of the program. Integration tests have immensely more value.

I honestly can't imagine any software product with test automation that would require more than a real day to pass, never mind a quarter. Not unless some person or bureaucratic process was deliberately slowing it down.

>But all this has been at least partially resolved. What we are still sorely lacking is simulation. Everyone who ever worked with a staging environment knows how limited this usually is. What I would like to work with before I retire is a tool that lets me standup a complete copy of the production environment, isolated from the real world of course, with all external services automatically mocked. Now that would help to improve the quality.

There are plenty of tools that do this. I run simulators to catch sent mails, pretend to be browsers, mock REST APIs, etc. This is the bread and butter of test automation. The actual scenario writing I find to be the easiest part.

The real problem is that this stuff is usually done really badly by cheap, poorly skilled people who will never earn as much as a dev and who almost never do this part of the job properly.

The actual solution is in my username.

Has anyone ever actually found this to be true?

I work in a place with 99% test coverage requirements and it's honestly still a super brittle system that everyone is afraid to make big changes to

In my experience, automated tests don't promote quality engineering, they just calcify whatever quality exists

And before you get into it, TDD is not a panacea for this problem

When test-driven development is applicable, it’s so damn nice. If for no other reason, just because it requires so many fewer keystrokes and clicks, just have a process watching for changes and rerunning the test suite. Less chance of developing RSI, it’s good for developers.

I added the caveat “when applicable” because for straight-up UI behaviors it’s not, but the longer I write code, the more I see ways to separate out pure logic from UI and other external effects and I’ve never regretted doing it for reasons including: general reasoning, readability, TDD, future changes, and bug fixes.

> There is an unwavering belief that knowing the corner cases and weak points of your own code makes you somehow better at writing the tests. We also seem to be obsessed with testing that is half-way between unit level testing and system level testing. We test APIs and features more or less in isolation.

You're both right. Devs should write tests for their own code, but they shouldn't be the only ones writing them.

One of the problems (at least at my employer) is QA is considered a place for second-rate developers. Some of them are so bad that they rely on the devs to basically write their tests for them, by requiring extremely detailed and specific acceptance criteria. Only once have I worked with a tester that was engaged enough to really understand the requirements on their own and call out weird behavior or corner cases that weren't spelled out in advance. It was great.

> Hopefully we'll arrive at a reasonable middle ground at some point, and unit tests will be valued (and prioritized) neither too much nor too little.

My coding style has very much gone through a similar evolution. When I first learned about TDD, I went in whole hog - test everything, ui tests, request tests, unit tests - tests for everything.

Then I started to notice that the development costs associated with extreme coverage did not, in fact, pay off.

The number of bugs found was vastly outnumbered by the time wasted dealing with the peculiarities of various ui testing frameworks, let alone the amount of time wasted waiting for those tests to run.

My metric is now that I've written enough tests to feel confident that the code works. It's an extraordinarily qualitative metric, and I cannot find a way to objectively quantify it, yet it very much works for me.

Really, it all comes down to visibility. Is there somewhere that is up to date, that will tell you what the code is supposed to do? Do you have tooling in production that will alert you when it's not doing what it's supposed to do? Do you feel confident that the code works?

The tricky thing is that no matter what test you contrive, it's more likely to say something about the developer's recent experience than about their competency in general.

For example, I'd say I have pretty good intuition for when to just read code or sprinkle printfs or fire up valgrind/gdb/asan when debugging C. Which I guess is to be expected given that I've been doing C almost exclusively for many many years. I'd do pretty bad with Haskell; the last time I really used it was around 13 years ago. The next guy might be a bit lost with gcc's error messages since the last time they used C in anger was 5+ years ago for a small project, but they'd do well if you hand them Python code that uses a well known unit test framework or whatever. I guess that's fine if you're a run of the mill crud company looking for "senior <foo-language> developer" but not if you're after general competency.

You can try hard to make the debugging be more about the system than about the implementation but it's not easy to separate the two. You can make different tests for people with different backgrounds but that only makes calibration harder.

One trick I've seen a company do is deliberately pick a very obscure language that most people have never heard of. That can eliminate some variables but not all of them (I took the test and did well but I also spent a fair amount of time studying the language to figure out if it's suitable for a purely functional solution before handing in a very boring piece of imperative code). Ultimately it wasn't much more than a fizzbuzz.

And if there's puzzling involved, I'd say there's an element of luck involved. At least that's how I perceive the subconscious mind to work when you're thinking about a problem that isn't immediately obvious or known to you beforehand. Which path does your mind set you on? Are you stupid or incompetent if it happened to pick the wrong one today and you spent 10 minutes thinking about it too hard? Are you super smart if the first thing that came to mind just happened to be the right one and you didn't doubt yourself before blurting out an answer?

If you're lucky and know the problem beforehand, you can always fake brilliance: https://news.ycombinator.com/item?id=17106291

That is to say, test calibration is hard and there are so many variables involved. It follows that there's no obvious right way to conduct interviews. And I guess it follows that companies who need people (and aren't necessarily experts at interviewing) effectively outsource the problem by conducting the same type of interviews they've seen elsewhere. Maybe that's less cargo culting and more just doing whatever seems popular and good enough?

They can line up, but I'm not sure there's a causation. There are many different ways to write tests the wrong way. It's taken a lot of trail and error for me to get better at it (I can't even say I'm "great" yet). I'd have given up long ago if I wasn't a passionate developer. A less passionate developer would think the whole concept is BS and give up long ago if he took my path.

One of the first mistakes I made was using a mocking framework to test my implementation. The test read like, "first call this method, then call this, then call this". Needless to say it was worthless. I was just writing my code twice.

Then you can use mocks inside-out and end up writing code you don't even need to call, etc.

I'm sure there's many ways to do TDD right, but from in my personal experience, there's more ways to do it wrong.

3. It's worth it.

I work at (relatively) low levels, and I would absolutely love to have extensive tests (plus more, e.g. TLA+ models to prove critical properties of the systems I work on).

The pushback comes from stakeholders. They don't want to invest time and money into automated testing.

And when no automated testing has been done yet, you can guess that the system hasn't been architected to be easily testable. Figuring out how to add useful tests without massive (time-consuming and expensive, potentially error-prone) re-architecting is also something that requires quite a bit of investment.

Of course a part of is just lack of experience. If someone who knows how it's done could lead by example and show the ropes, that'd probably help. Getting the framework off the ground could be the key to sneaking in some tests in the future, even when nobody asks for them.

I agree that testing is a valuable tool for making good software, but I think the idea that all code categorically requires testing to be considered good is overzealous.

I think the most ardent TDD proponents underestimate the costs of testing. Tests are also code which can have bugs and has to be maintained, and having a well-developed test-suite can act as a type of inertia which makes it harder for an organization to make necessary architectural changes.

Don't get me wrong - I think testing is important, but it is just one tool which has to be balanced against a number of other factors.