It's often impractical to fully test every single aspect of all code. An example that I've seen several times is bottlenecks in unexpected places. Once was due to false sharing, another due to worse standard library QoI.
Though I think we could derive some more realistic scenarios, like evolving requirements and switching developers mid project that would be more enlightening.
Yeah, having running software forces you to prove a certain level of correctness and lets people test against it.
You can call a spec done arbitrarily and it's much harder to test that spec against any number of edge cases; and harder to visibly inspect "do these two specs get along" than "did this API call to this other place succeed?" "It doesn't compile" or "it throws an error" or "it doesn't do what it should" are all much more concrete and force you to confront "we might not understand the problem as well as we thought."
I prefer to fix bugs before they hit production. It is almost impossible to do that without tests. Also testable code is usually much more readable than untestable god objects, tightly coupled modules and spagetti code.
It's not all tests, it's just the API tests. I'm not sure why that was unclear to so many people. You can have hundreds of tests, thousands even, only the API tests are special.
If there's no stable behavior because the software is still at that stage of development, it's 0.x software still. That's true in SemVer as well as this refinement of it.
Contrariwise, if you think software is ready for 1.0 and you can't come up with any tests which display guaranteed behavior which won't change without that major version bump, then no, it's not ready.
I think another thing to consider (and reason that isn't brought up in this article) is that the longer a piece of software has been available to the public, the more chances there'll be for people to find bugs in it by doing things its creators would never have expected.
Think about it. Let's say you're working on a CMS system for a shop, and that your startup has both automated testing processes and a specialist QA testing team. You'll find and fix quite a few bugs, but you'll then find thousands more coming in when the public gets their hands on it.
Why?
Because the sheer number of use cases people have will push your software to its limits. You simply can't test every 'case' someone may have for the software.
Some might be using an outdated version on a newer scripting language version. On an estoteric choice of operating system. On an estoteric choice of emulated operating system via a virtual machine. With third party or self developed plugins meant to integrate with other random software. With interesting themes that may not have well thought out. You may have simply made a screw up that's only noticeable if you do ten specific things in a certain order. Or if your connection cuts out midway through a process. Or any number of factors that are incredibly hard to predict.
Eventually, the reason comes down to 'it's not plausible to test every possibility and predict everything your user could theoretically do at any one time'. Heck, look at it in games terms. All those games with arbitary code execution setups only have them because someone was bored enough to try and programmatically break the game by doing about fifty seemingly random actions in ten seconds or so. Most speedrun techniques similarly only 'work' because the people who found them spend weeks or months testing random things on the off chance a wall might not hold.
If there's a large enough audience, anything will be buggy.
Almost no software is tested on a proper suite of different architectures. If you don't want to test it, the proper answer is to emit valid code without any shenanigans.
he might be partially right, because I have not yet found time to write tests for certain functions of the transpiler (I have been effectively unemployed for a few months now and making ends meet is my only priority at the moment.)
that said, I too believe sheer popularity may not always be a great heuristic to judge the stability of a piece of software with .....
That’s what I’m saying though. At some point, you have to write those tests and there will be bugs. There will be things that aren’t ideal. It’s like the first time you write the CI/CD pipelines and you have to commit 100 stupid things directly to main to test it and make sure it works.
It's true that in the future I would do some more thorough testing, but never-ever for this legacy application can I build a bullet-proof automated testing solution - there will not be a budget for that for sure. However, even with a fancy solution mistakes will happen and you still can't stop release propagating. It's just a matter of time when it happens.
I wish that was an option for at least half the codebases I've worked on. Tangent: you'd be surprised how many life threatening codebases have little to no tests at all.
Let's not pretend that people here in extremely permissive environments don't build hugely buggy things that wouldn't survive a similarly superficial test.
reply