Hacker Read

duped · 2022-01-29 17:14:14

My guess is bad regression testing based on subjective qualifiers at best or incentivize poor results that promote as revenue at worst.

theodpHN | karma 584 | avg karma 0.89 · | 2023-07-28 07:52:40

Regression testing.

heavenlyblue | karma 2264 | avg karma 0.77 · | 2021-10-26 04:47:00

They also call them regression testing

jacquesm | karma 227883 | avg karma 3.76 · | 2016-02-04 17:19:20+00:00

I think we can take that as evidence that regression testing is hard.

empath75 | karma 15699 | avg karma 2.9 · | 2019-12-22 17:14:21+00:00

A/B testing gone horribly wrong is my guess.

pydry | karma 11003 | avg karma 2.18 · | 2020-09-24 13:55:09+00:00

Main reason it it tests unrealistically, I think. The other issues (doesnt catch bugs, hard to maintain) tend to flow from that.

brightball | karma 15608 | avg karma 3.36 · | 2020-08-26 17:35:54

The way the tests were written in that case, it was hard coded to how the work was being done and not the result produced. Both the code and the tests were bad.

Normally, I’d agree with you though.

reply

wnkrshm | karma 1173 | avg karma 1.92 · | 2023-03-27 02:25:53

The reason being tests/constraints specified by someone designing the software.

jbreckmckye | karma 1861 | avg karma 4.81 · | 2018-02-12 17:40:22

It's more than that. Optimizely fear that poorly delivering tests reflect badly on their other products.

millstone | karma 6895 | avg karma 3.52 · | 2018-02-14 08:34:19+00:00

In this sort of scenario, the bugs lie in the expectations themselves. Tests that don’t account for that are dead weight.

garethadams | karma 545 | avg karma 3.19 · | 2013-03-25 15:07:26+00:00

A 'bad test' isn't one that is failing, it's one that doesn't test anything useful, or worse tests something useful but incorrectly

em3rgent0rdr | karma 5777 | avg karma 2.68 · | 2017-12-01 23:59:52

maybe there wasn't any regression testing...

The_Colonel | karma 5205 | avg karma 2.55 · | 2022-08-25 06:08:42

Most of those tests are generated. It's wrong to focus on this metric IMO.

astrange | karma 14132 | avg karma 1.32 · | 2021-06-19 12:20:09+00:00

Was that the point of the tests, or just a coincidence?

Now that they've failed once they've become regression tests (one of the most useful kind of test), but if you set out wanting to test the platform under you I think you'd want to do a lot more work than that.

reply

gcp | karma 5363 | avg karma 3.26 · | 2012-05-17 04:08:03

The problem is that the metrics can be gamed, and there's incentive to do so.

Nobody is arguing against testing. They're arguing about doing tests where you know the results will be deeply flawed.

reply

drewg123 | karma 13875 | avg karma 6.95 · | 2020-06-01 20:30:17+00:00

So much for regression tests.

mikeash | karma 74524 | avg karma 3.52 · | 2012-06-29 21:54:20+00:00

The fact that the tests would be easy to fix makes it worse, no better. The problem the tests and the security hole was not that the tests indicated severe problems, but that all these simple failing tests masked the presence of a new, serious failure. If your test output is filled with junk due to failing tests because of minor bugs, it makes it much harder to notice when your tests uncover a major regression.

stfwn | karma 830 | avg karma 6.59 · | 2019-01-04 23:41:29+00:00

Can you elaborate a bit on how such testing is done, or share a good article on the topic? It sounds like a hard problem to need to get things right this bad, or else.

Temporary_31337 | karma 413 | avg karma 2.11 · | 2023-01-14 06:06:03

Do you simply mean regression testing?

jakobegger | karma 8283 | avg karma 4.45 · | 2018-07-24 08:32:40

Relying on tests is naive. Your tests can't cover every case. The article even mentions this -- their tests passed, but it failed in production.