That's exactly the problem I was trying to reference. The algorithms and data models are black boxes - we don't know wat they learned or why they learned it. That setup can't be intentionally fixed, and more importantly we wouldn't know if it was fixed because we can only validate input/output pairs.
I am not an ML engineer (except when I program in ML, of course ;)). But this sounds a lot like the following:
1. We had a model that worked in principle, but the search space was practically infeasible.
2. We made an observation that a different model might exist that makes the search space irrelevant.
3. We threw ML at it.
4. Now we might have a model that fulfills (2) but we cannot be sure because we used a black-box approach.
5. Somehow the results are exciting. Better results would be really exciting.
6. We hope that more data yields these better results.
Is that correct? Am I the only one to lament these black-box approaches? Should there not be a bunch of people now studying the learned models to figure out if much better results can actually exist?
Thanks for explaining that perspective. I was seeing things a little too narrow-mindedly for sure.
Based on your phrasing of the issue, it seems like we could think of the problem as: can we reduce the number of parameters in an ML model to the point where humans can understand all of the parameters? That's related to an active research area - minimizing model size. ML models can have billions of parameters. It's unfeasible for a human to evaluate all of them. Research shows that (sometimes) you can reduce the number of parameters in a model 97% without hurting it's accuracy [0]. But, 3% of a billion parameters is still way too many for people to evaluate. So I think the answer so far is no, we can't create human-understandable models that perform as effectively as black boxes.
The problem comes when you need to peer inside the black box because it's working counter to your expectations of it: With most systems you can do so by reading code / documentation etc. With ML you're kind of stuck. You can conduct experiments to see what changes about the behavior of the model, but even that proves tricky when the model has thousands of inputs.
Yeah there are definitely ways they would have avoided that, but it's just one example of many. The whole point of ML is that it picks up on learned patterns. The problem is that it can be difficult to identify what it is learning from- this paper itself says they do not know what is causing it to make these predictions. As a result it is difficult to validate that the model is doing what people think it is.
Interesting, I suppose what you're proposing is that models could, in some abstract way, extrapolate research results taking ideas A and B that it "knows" from its training, and using them to create idea AB. Then, we assert that there is some "validation system" that can be used to validate said result, thus creating a new data point, which can be retrained on.
I can see how such a pipeline can exist. I can imagine the problematic bit being the "validation system". In closed systems like mathematics, the proof can be checked with our current understanding of mathematics. However, I wonder if all systems have such a property. If, in some sense, you need to know the underlying distribution to check that a new data point is in said distribution, the system described above cannot find new knowledge without already knowing everything.
Moreover, if we did have such a perfect "validation system", I suppose the only thing the ML models are buying us is a more effective search of candidates, right? (e.g., we could also just brute force such a "validation system" to find new results).
Feel free to ignore my navel-gazing; it's fascinating to discuss these things.
The point here is that checking their work is actually quite hard. Using your analogy to libraries, it’s easy to see something like an unexplained base64 blob being exec()-ed and ask why they did it but it’s much harder to notice a subtle logic flaw which only affects, say, a numeric value unlikely to occur in normal usage. Very, very few organizations have the resources to catch problems like that and so everyone should be thinking about how they could contain the damage or otherwise make the system easier to trust, which in the case of ML might mean being a lot more careful to vet training data.
This makes me wonder: is it possible for ML models to be provably correct?
Or is that completely thrown out the window if you use a ML model rather than a procedural algorithm?
Because if the model is a black box and you use it for some safety system in the real world, how do you know there isn’t some wierd combination of inputs that causes the model to exhibit bizzare behaviour?
The thing is even many ML researchers don't believe that interpretable models are possible without losing accuracy. There's a good episode of Data Skeptic [0] with one of the co-authors of a paper [1] delving into this. They talk about real-world ML models used to predict recidivism as well as medical expert systems.
I wasn't sure whether to post my edit as a separate comment or not, but I significantly expanded my comment just now, that helps explain my position.
I'd be very interested in your thoughts on that position, because if it's mistaken, I shouldn't be saying it. It represents whatever small contribution I can make to fellow new ML researchers, which is roughly: "watch out."
In short, for two years, I kept trying to implement stated claims -- to reproduce them in exactly the way you say here -- and they simply didn't work as stated.
It might sound confusing that the claims were "simply wrong" or "didn't work." But every time I tried, achieving anything remotely close to "success" was the exception, not the norm.
And I don't think it was because I failed to implement what they were saying in the paper. I agree that that's the most likely thing. But I was careful. It's very easy to make mistakes, and I tried to make none, as both someone with over a decade of experience (https://shawnpresser.blogspot.com/) and someone who cares deeply about the things I'm talking about here.
It takes hard work to reproduce the technique the way you're saying. I put all my heart and soul into trying to. And I kept getting dismayed, because people kept trying to convince me of things that either I couldn't verify (because verification is extremely hard, as you well know) or were simply wrong.
So if I sound entitled, I agree. When I got into this job, as an ML researcher, I thought I was entitled to the scientific method. Or anything vaguely resembling "careful, distilled, correct knowledge that I can build on."
Half of data scientists or ML engineers I work with can implement and train models but do not necessarily understand the underlying principles or math.
Most people just care that it works and not why.
Really dangerous depending on the problem and field tbh
I wonder if ML can someday be employed in deciphering such black box problems; a second model that can look under the hood at all the number crunching performed by the predictive model, identify the pattern that resulted in a prediction, and present it in a way we can understand.
That said, I don’t even know if ML is good at finding patterns in data.
Count me among those who thought validation was a thing you just had to do when training ML algorithms. After all, the most beautiful theoretical model in the world is of no use if the predictions it delivers are terrible.
The real trick (for most algorithms) is to select the correct features to train against. This really is more of a black art than an exact science, so I think labeling it a trick is justified.
Your description sounds like you're training your reviewers to rubberstamp the system's results as correct.
That is a horrible outcome.
By the time something comes along they really need to question, they'll already have adapted to the system being pretty good and they won't even notice there was an issue.
This problem is endemic to ML systems and cannot be patched away, so far as I can tell. We need fundamentally different approaches to avoid this pit.
Please make sure decision-makers understand this unavoidable gotcha their ML systems must necessarily have.
Let's be really clear here. It doesn't just make ML solutions a challenge. It makes them impossible.
Even with all the compute power and automation of google, you can't blindly create an ML model and say yes it works without a data scientist actually looking at the guts with row-wise access to the training/validation data.
You can't do analytics without giving engineers access to prod data. Not at even at Google.
Explainable models do not preclude the systemic problems you highlight. Plenty of systems before the advent of non-explanatory ML models had those defects. One option is to define test and validation sets and encourage 3P validation, somewhat like how accreditation works in other contexts.
ML systems do not guarantee valid output. If you have a system that can prove a given transformation is behavior-preserving, then sure you can let the AI have a go at it and then check the result, but having such a checker is much harder than it sounds.
ML engineers do not realize frequently enough that many of their algorithms, as beautiful as they are, still essentially fit a conditional expectation. For example the last logit layer that undoubtedly decided to ban the op here.
I always wonder why almost all papers and essentially all tutorials and courses spend literally zero time discussing whether that is the right thing to estimate.
In the future, when these algorithms decide everything about our lives, people will either conform and uniformize themselves to the expectation,
or they will find themselves secretly banned, silenced, left out, rejected or whatever it is, always without knowledge why and without recourse.
It’ll be strange for people, if we even hear about it.
reply