Hacker Read

aidenn0 · 2021-09-25 18:54:20

Presumably that happens in production as well, and the test can determine that the system does the proper thing when that happens?

chrishacken | karma 1191 | avg karma 5.34 · | 2016-07-20 13:34:03+00:00

Exactly. You can only test for these scenarios to a certain extent and even when everything works as expected during tests, things still tend to break during production when shit hits the fan.

apotheon | karma 1217 | avg karma 1.54 · | 2009-09-02 22:00:45+00:00

Thanks for the information. I think I'll give that a look on a test system, if only to see whether I care enough to bother.

cryptonector | karma 9732 | avg karma 1.52 · | 2020-01-17 05:03:46

But you also have to test the whole system.

captainkrtek | karma 539 | avg karma 3.74 · | 2023-11-28 20:33:59

That’s a good point, would be curious to understand more what the testing setup is like for these kinds of systems.

analog31 | karma 17962 | avg karma 2.56 · | 2016-01-08 01:05:53+00:00

I have worked in manufacturing plants. One thing I've observed is that a unit will be tested after assembly. If it passes, it gets shipped. If it fails, it's tested again, perhaps with some token adjustment or swapping of components with other units.

lazylizard | karma 230 | avg karma 0.31 · | 2014-08-23 03:27:31+00:00

um. just wondering. they have to test for failure conditions too, yes?

gburt | karma 1600 | avg karma 3.77 · | 2016-04-06 17:34:41+00:00

Any chance you could point me to a reference? I'm doing research in this space and currently working on a paper which does exactly this for diagnostics of testing processes.

DavidSJ | karma 6348 | avg karma 5.22 · | 2023-02-27 13:48:00

This sort of tests can be useful when you change things under the hood in such a way that the output shouldn’t have changed.

bcrosby95 | karma 9525 | avg karma 3.64 · | 2022-09-15 17:09:21

Yes. When interfacing with a complex system, especially one you don't control, always do at least one live test.

hulitu | karma 3686 | avg karma 0.58 · | 2021-08-13 06:26:57

In the automotive industry when all your devices fail the test, the test is wrong.

markburns | karma 264 | avg karma 2.22 · | 2015-12-02 18:57:20

Another possible conclusion: fix this several years from now, the first time it causes a test failure and still probably before it causes any production problems.

mattlondon | karma 8349 | avg karma 4.42 · | 2019-10-18 20:06:25+00:00

Yeah absolutely but if you are running a simulation/test would you deliberately inject some random sensor failure if you are doing your tests for something else?

It is not clear what they were testing - perhaps they were indeed testing the MCAS system with sensor failures, but if so I probably wouldn't have expected such a surprised resction from them. It seemed like it was totally unexpected and unexplained, which is not a reaction I would expect if they were testing this.

reply

01100011 | karma 7985 | avg karma 3.41 · | 2019-01-28 18:13:20

I wonder how deep the pipeline is between inputs and failing tests on the output? I'd guess they'd have quite a bit of inventory in flight before they realized there was a problem with the output, but hell, I don't know.

edwintorok | karma 5758 | avg karma 5.47 · | 2014-07-15 21:10:00+00:00

Is it guaranteed to at least be different for different processes? They could use that in addition to the PID test to know when to reseed.

toasterlovin | karma 4624 | avg karma 1.92 · | 2017-10-27 14:12:35

Yeah, but usually an integration test tells you what line it failed on, which gets you fairly close to knowing the exact reason it failed.

thinkingkong | karma 3798 | avg karma 5.51 · | 2018-08-12 23:20:38+00:00

Wouldnt the test just need to apply to all modes? If its a matter of automatically tuning through a series of conditions then its reasonable that those could be documented and tested by the facility / standards body.

Jtsummers | karma 23473 | avg karma 2.62 · | 2021-04-05 03:58:11+00:00

That's typically configurable with the test system.

grandalf | karma 11860 | avg karma 2.47 · | 2015-09-23 21:53:54+00:00

Was the software designed only to detect tests? Or is detecting tests (and other cases where limiting performance has no adverse consequences) a side-effect of the system's intended behavior.

For instance, suppose the vehicle was programmed to shut off its engine when stopped at a stoplight. Suppose the test scenario measures emissions while stopped at a stoplight...

reply

varjag | karma 13715 | avg karma 2.89 · | 2019-08-31 16:48:06

My understanding was if the function is interrupted on discharge, you fail the test. At least that's how our equipment was tested by an independent lab.