You have to have tests for all combinations though. At least those combinations that you actually want to use. You get the same problem when your code is a big ifdef-hell.
Testing is important, for sure, but just because you have two parameters with n choices each, does not mean you have to test n^2 combinations. You can aim to express parameterization at a higher level than ifdefs.
For example, template parameters in C++. The STL defines map<K, V>. You don't have to test ever possible type of key and value.
Lets say you've got a complicated piece of code that has a bunch of options (either input or configuration, doesn't matter). You can't test it exhaustively because that's exponential in the number of options. But you can test all combinations of each pair of options, in ~O(A x B) where A and B are two largest option cardinalities.
To be concrete about this, suppose there are 10 option variables each having at most 5 possible values, a,b,c,d,e,f,g,h,i,j. Suppose that d and e are have the largest cardinality (5). So you can test all pairs of assignments to d and e in 25 tests. But with the same 25 tests (or epsilon extra ones) you can test all pairs of assignments to every pair of variables - you need hardly any more tests.
Of course, coming up with the table of values is a bit tricky, especially if there are some constraints. But there are libraries to do it.
Takeaway: with a very small number of tests you can find corner cases that are very hard to locate with a standard approach.
If you wanted to test all combinations possible, you would have to brute-force until the end of the universe
You only need to test for a combination of all the input variables that affect the execution flow. However, to do that properly, you need to know the flow of the methods your methods calls. (E.g. you would need to know that substring throws an exception if the string in shorter than expected.) This kind of information could be captured in some kind of meta-data that could be propagated up the call chain for testing.
It's also worth looking at all-pairs testing.
The insight is that you can cover all pairs of a set of combinations in only O(N^2) tests rather than O(N^D) irrespective of the number of dimensions. The number of tests ends up being the product of the size of the two largest dimensions. So in your example you can cover all combinations of each pair:
* (lib A, lib B)
* (lib A, language runtime)
* (lib A, browser)
* (lib B, language runtime)
* (lib A, browser)
* (language runtime, browser)
in 50*30 = 1500 tests (or a little more to make the analysis tractable)
They have a check clause which is detached from the functions themselves. This is where they would want you to do tests with multiple functions interacting.
Or even worse you start writing code for "testability" and it becomes a bloated mess of one- or few-liner functions that are only called in one location by some other function.
Tests do not have to guarantee that the behavior is correct for all possible inputs. If testing all possible inputs is easy, do it. Otherwise, you write the tests which you think are most likely to catch errors. Try to exercise different code paths to catch errors in different parts of the code. Try to use different logic in your test from the logic in the code under test.
You can’t test everything. To believe we can is just hubris; even if we get every function to have coverage, we won’t have coverage of the full range of inputs unless all that’s being done is some very simple programming. The combinatorial explosion is fast.
Testing definitely has value, but when testing you always have to make the assumption that you can’t test everything, and then the trick is deciding what to test versus what not to. I think people forget this and focus on having every function tested, every interface having coverage, etc.
Yes. But what's striking me is that one woud ever want to test only one input state. If I have to test function f(x) and see if its output is as expected, I always want to test many inputs (including "silly" ones like wrong type, nulls, extremes). Writing one test for each of them is absurd, just give a list of inputs, a list of expected outputs and check them all.
I think your comment just made me understand what was nagging me about testing: writing tests is a form of decomposition. The method/function is complex/complicated, and therefore hard to reason about. The unit tests limit the scope of input so that the in particular instances, the method/function becomes more simple to reason about, enough that a test can be written.
Sadly, I feel there are things for which tests can't be written. If a function takes two integers, and adds them together (the typical 2+2=4 therefore n+n=n). Does this work when the integers are near-infinity large (trillions of zeros -- theoretically possible with python)? How would a unit test validate this?
If you wanted to test all combinations possible, you would have to brute-force until the end of the universe, or until the machine runs out of hard drive space (or SAN space), whichever comes first. If you wanted to take a statistically significant sample, you would only have an elevated level of certainty, not an absolute level of certainty.
I think that what the author is pointing at is that like mathematics, which, let's face it, the human brain is much better at than computers, programming is best done in the human brain, and that once the human brain has satisfied itself of the proper of the program, the coding becomes simple.
And the program no longer needs to be decomposed, because it is understood as a whole.
True. I just run into these situation too much because I functional style code where there is very little internal state and try to use pure functions as much as possible. This narrows down the code that has to be written on a more testable way.
No. It's the same. If you have a function with a large if statement in it, you'd want to test all conditions within that if statement to make sure it was working, right?
You'd do the exact same thing if you were testing a series of pattern-matched functions. The tests would be unchanged either way.
Both the fib and prime factor examples are shit. The tests are individual cases which is treating the input as a Cartesian product (a point on a line since these are single-variable functions). What you want are sets. The inputs for both are all natural numbers. However, this doesn't tell you much, there's still a lot of state to work with. So what you want are end conditions, but again that doesn't tell you much.
Tests are no substitute for reasoning and TDD is just another fad.
My beef against testing is that I often see it adopted (and often subtly pushed) as an alternative to thinking through the logic carefully -- Why mess with the if statements, my tests will catch the errors if any. Should I index the array with i or i+1 ? I will just test what works. Should I loop till n-1 or n-2. Let me just stick with n-2 and add a test...
The problem is that to write tests that really ensure that the logic is correct you really need to think through the logic really hard. Just spraying the code with a few tests that happen to come to the mind is not enough.
I am yet to see a convincing argument that shows that coming up with a sufficient number of (unit)tests from a specification is any easier than getting the logic right.
Testing is well intentioned but often abused as an excuse to be sloppy.
Which would need even more tests, because now there a set of types that will work (e.g. numbers, arrays of numbers), with code to make them work and tests on that; and other types do not work, and tests on that.
A lot of those if-/case-blocks are precisely where I'd put functions :)
If you changed a bunch of those to separate, pure (i.e. side-effect-free) functions it would if nothing else make unit testing a breeze, and then you'd be free to fix bugs in the logic without fear. As it is, if I had a bug in that huge function I'd be really worried about breaking some edge-condition or implied-state 500 lines up etc.
reply