Yawn. Contrarianism is easy and this article offers little. The real world application you’re speaking of has a comically small amount of data (a few million miles?). You hear about a handful of accidents that still average to better than human performance and suddenly the sky is falling.
When machine learning stops successfully solving new problems daily, then maybe a thread like this will be warranted.
It would be dumb to say that it's not worth it to pursue ML research and apply it to engineering problems to improve well-established solutions.
What bothers me the most however and is the reason I was tongue-in-chick in my first comment was the insane overrepresentation machine learning receives in virtually every interaction related to tech, considering its success to non-physical problems. I get that hypes come and go and will continue so, but it's another thing having a trend spreading out in almost every scientific discipline advertising it as a discovery by big brain computer people that have come to salvage their poor inferiors (that its ardent proponents have zero experience in engineering design is another story). You may think this is hyperbole but it's really a common sentiment among ML skeptics. And it's especially frustrating when it's presented as a subtitute, and not as a supplement, to well established mathematical ways, e.g. machine learning vs control theory in AVs as you've said.
On the other hand, although SV startup culture is definitely to blame for all this to an extent, I can definitely understand some subfields being way too conservative to trendier topics just for the sake of not blending in with the hype. There have been some decent attempts lately to bridge the numerous chasms each discipline has and get the best out of each world. Hopefully something useful comes out of it.
Sorry, but Machine Learning and statistical AI is way over-hyped in my experience working in the industry. There are definitely very cool things coming from it. But the hype implies it will solve way more cases than it currently can.
Modeling complexity/cost is worse than exponential. The low hanging fruit is already taken. This doesn't mean we should stop doing it. But we should curve our expectations.
Machine Learning algorithms are only as good as the data they are trained on.
Machine Learning algorithms are never better than the data they're trained-on. But they can easily be "worse".
Specifically, an ML algorithm trained on one data set can have a hard time operating on a different data set with some similarities and some fundamental differences. This is related to algorithms generally not having an idea how accurate their predictions/guesses are. This in turn relates to current ML as being something like statistical prediction with worries about bias tossed-out (which isn't to dismiss it but to illuminate it's limitations).
For tasks like self-driving or spotting cancer in x-rays, they are producing novel result because these kinds of tasks are amenable to reinforcement. The algorithm crashed the car, or it didn't. The patient had cancer, or they didn't.
Ironically, both those applications have been failures so-far. Self-driving is far more complex than a binary crash-or-not scenario (usually the road is predictable but just about anything can wander into the roadway occasionally and you need to deal with this "long tail"). It also requires extreme safety to reach human levels. Diagnosis by image has problems of consistence, of doctors considering more than just an image and other things possibly not understood.
The main problem is human overconfidence in the resulting functions, which are often measuring things humans didn't realize was embedded in the data (eg with tumor recognition), or are utterly hopeless at representing the infinite edge cases that the real world can present (eg self-driving cars).
It's one thing to have hoped that these methods could solve these problems when the improvements were coming rapidly, but there will always be a limit to how well these systems can perform. And the problem is that they fail in entirely non-intuitive ways, making human oversight to correct for errors very difficult or impossible as well.
Yeah. Andrej Karpathy said in a recent interview that at this point it's almost completely a data problem, instead of a machine learning research problem. Most of their work is in figuring out the best way to collect and annotate lots of good data. Waymo has a few hundred cars and Tesla has millions, and in this problem space, scale wins over more precise sensors.
Professional Tetris players and people who classify pictures of cats and dogs.
Machine learning is not a mature or robust field. Real world applications are brittle and extremely narrow in scope. The past few years has seen a burst funding and hype, as a result of deep learning advancements, but that will gradually wane as all the low hanging fruit is picked. Many applications that are promising (like self-drive cars and medical classification) will face major regulatory hurdles. It will take a much more radical (and, frankly, unexpected) breakthrough before most humans have to worry.
I'd clarify that there is a specific delusion that any data scientist straight out of some sort of online degree program can go toe to toe with the likes of Andrej Karpathy or David Silver with the power of "teh durp lurnins'." And the predictable disappointment arising from the craptastic shovelware they create is what's finally creating the long overdue disappointment.
Further, I have repeatedly heard people who should know better, with very fancy advanced degrees, chant variants of "Deep Learning gets better with more data" and/or "Deep Learning makes feature engineering obsolete" as if they are trying to convince everyone around them as well as themselves that these two fallacious assumptions are the revealed truth handed down to mere mortals by the 4 horsemen of the field.
My personal (and admittedly biased) belief is that if you combine DL with GOFAI and/or simulation, you can indeed work magic. AlphaZero is strong evidence of that, no? And the author of the article in this thread is apparently attempting to do the same sort of thing for self-driving cars. I wouldn't call this part of the field irrational exuberance, I'd call it amazing.
Couple of years ago, there's been an article posted here: How to Recognize AI Snake Oil. https://readhacker.news/c/4cSun It looks like this was the case with Zillow.
Most of commenters here insist ML and data science have nothing to do with this failure, and put all blame on the management.
I don't by it's not failure of ML. In the last several years I consistently see Tech leads oversell the possibilities of ML and theorizing that ML could solve many problems with deep learning.
The result is always modest. Error margins for real world metrics that involve location data are around 20-30%, which is somewhat better than analysts predictions in Excel, but not enough to rely on them. I saw very competent ML engineers try to improve a model, take a huge amount of satellite imagery, tracking data, etc., train their neural networks for couple of months, and only improve error margin by 1 percent point! (e.g. 25% -> 24%)
That’s the fundamental problem with machine learning. You get no control over what the machine has actually learned.
It’s like GPT-3. In 9 cases out of ten, it responds in a way that lets you think the model has “grokked” facts of the world, and a human like sense of logic. In truth, it hasn’t. It’s just pretending. It’s taking the easy path by parroting back what it has seen before.
They say that ML requires a re-thinking on the part of the developer. Whereas with traditional programming it would be his job to “structure the problem and the path to its solution”, in ML, he should hold back his human notions and let the network discover its own structure.
To use an analogy: Whereas we teach our children concepts step by step - you and me, identifying objects, gasping objects, referring to objects, etc., all the way up in complexity to writing thoughtful comments on HN - the neural network is to be bombarded by unfiltered real world data. And the hope is that, if you just do this long enough, the AI will come up with the important underlying concepts by itself. That is to say, for GPT-3, the idea that words are really describing a real world out there, with objects that have certain properties, and that claims may be true or false. Or, for Tesla, that the car is a physical object in space, and that it can collide with other objects.
This is very unintuitive. It might be the right approach. But if I was building a self driving car, I would want to build that system block by block. Learn the basics. And only proceed to the next level of complexity once I have verified that the previous step has been properly learned.
1. Let the car learn to predict it’s own acceleration, steering and braking.
2. Repeat on different road surfaces.
3. Add other objects to avoid.
4. Train ability to tell objects from non-obstacles (fog, raindrop on camera)
5. Train lane markings, traffic lights on empty roads.
...
X. Throw real world data at it. Now you can hope (and validate) that the model will have the foundation to make sense of it. Like a kid after Highschool. It will still make dumb mistakes. But at least you can talk about them and understand why they happened and take corrective action. Because, while you do not know the other’s mind, you do speak the same language.
this article is of course reasonable, but what’s depressing is we genuinely have made really incredible strides in machine learning. like enough that it shouldn’t be possible to overhype it. of course people manage though.
Maybe this problem is too hard for humans to solve. We've spent a long time trying to crack this nut. I am more bullish on Machine learning solving this problem quickly in the next 5 years.
This is really only an argument that the example in the article is not realistic (which it doesn't have to be, it might be expository). There are in fact countless applications of machine learning in actual daily use, such as detecting credit card fraud, where simpler manual methods would perform measurably worse in terms of money lost.
I'm not arguing that current machine learning technologies are not useful. I'm just arguing that progress is based on increasing some metric, usually depending on a trade-off of computation. This can even make ML-techniques applicable to some new fields, but it's not what is holding back autonomous driving, the often touted parade example which also brings in a lot of employment for machine learning.
This article clearly sits on the peak of inflated expectations in the hype cycle.
I would say that we should start by critiquing the implicit claim by Google, Facebook, Waymo, and others that new means of computational statistical analysis can serve as a reasonable replacement for human intuition and judgment. Turns out some of the world’s most valuable companies are betting on lots of poor assumptions and easily-gamed algorithms, and we’re just starting to see the fallout of our failure to call BS on the ever-louder hype machine.
I certainly agree that deep learning has delivered some impressive results, but the acceptance by VCs and the media that his new wave of AI will bring a revolution in Internet time to established real-world infrastructure and processes is a major failure of basic critical thinking on a broad scale.
A lot of people are building things [with big data] hoping that they work, and sometimes they will ... Eventually, we have to give real guarantees. Civil engineers eventually learned to build bridges that were guaranteed to stand up. So with big data, it will take decades, I suspect, to get a real engineering approach, so that you can say with some assurance that you are giving out reasonable answers and are quantifying the likelihood of errors.
It's seems like the idea is that machine learning and data driven inference have to grow up and become a real scientific discipline. "Why can't you be more like Civil Engineering?" This isn't the best way to look at it. Machine learning is designed for situations where data is limited and there are no guarantees. Take Amazon's recommendation engine for example. It's not possible to peer into someone's mind and come up with a mathematical proof that states whether they will like or dislike John Grisham novels. A data driven model can use inference to make predictions based on the person's rating history, demographic profile, etc. It's true that many machine learning approaches don't have the scientific heft of civil engineering, but they are still very useful in many situations.
I'm not disagreeing with the eminence of Michael I. Jordan. I think this is a philosophical question with no correct answer. Is the world deterministic, can we model everything with rigorous physics style equations? Or is it probabilistic, are we always making inferences based on a limited amount of data? Both of those views are valid, especially in different contexts. Some of the most interesting problems are inherently probabilistic, such as predicting the weather, economic trends and the behavior of our own bodies. "Big Data" is obviously a stupid buzzword, but the concept of data driven decision making is very sound. We should put less focus on media hype terms and continue to encourage people to make use of large amounts of information. Get rid of the bathwater, keep the baby.
True, but many people jump on the hype wagons and assume that they need machine learning, or whatever the flavor of the day is when in fact they need something far simpler.
Who could've ever predicted this scenario? A great example of something that's trivial for human beings to understand but where ML training sets will come up short.
When machine learning stops successfully solving new problems daily, then maybe a thread like this will be warranted.
reply