Yes, I've seen what neural nets are now capable of- they are capable of
exactly what they were always capable of, except "now" (in the last few years)
we have more data and more compute to train them to actually do it. Says Geoff
Hinton [1].
I have also seen what neural nets are incapable of. Specifically,
generalisation and reasoning. Says François Chollet of Keras [2].
AI, i.e. the sub-field of computer science research that is called "AI" and
that consists of conferences such as AAAI, IJCAI, NeurIPS, etc, and assorted
journals, cannot progress on the back of a couple of neural net architectures
incapable of generalisation and reasoning. We had reasoning down pat in the
'80s. Eventually, the hype cycle will end, the Next Big Thing™ will come
around and the hype cycle will start all over again. It's the nature of
revolutions, see?
So hold your horses. Deep learning is much more useful for AI researchers who
want to publish a paper in one of the big AI conferences, and to the FANG
companies who have huge data and compute, than it is to anyone else. Anyone
else who wants to do AI will need to wait their turn and hope something else
comes around that has reasonable requirements to use, and scales well. Just as
the original article suggests.
Geoffrey Hinton: I think it’s mainly because of the amount of computation
and the amount of data now around but it’s also partly because there have
been some technical improvements in the algorithms. Particularly in the
algorithms for doing unsupervised learning where you’re not told what the
right answer is but the main thing is the computation and the amount of
data.
Say, for instance, that you could assemble a dataset of hundreds of
thousands—even millions—of English language descriptions of the features of
a software product, as written by a product manager, as well as the
corresponding source code developed by a team of engineers to meet these
requirements. Even with this data, you could not train a deep learning model
to simply read a product description and generate the appropriate codebase.
That's just one example among many. In general, anything that requires
reasoning—like programming, or applying the scientific method—long-term
planning, and algorithmic-like data manipulation, is out of reach for deep
learning models, no matter how much data you throw at them. Even learning a
sorting algorithm with a deep neural network is tremendously difficult.
After watching 'Recent Developments in Deep Learning' by Geoff Hinton [1] a while ago I did come away believing that there does seem to be quite a bit of untapped potential in deep neural nets.
Do you feel that talk is overly optimistic/misleading? Or just that you don't think there's that much money to be made with it?
For starters, I said modern. Most of the big recent breakthroughs in AI have come from neural nets. There is a inherent problem with deep learning with neural nets. Neural nets are exceptional at finding patterns within a huge dataset, but they are REALLY, REALLY bad at predicting new patterns that they haven't seen. Even something as simple as the pattern a^2b is impossible for a neural network to generalize past whatever length of the pattern the neural network has seen without modifications (check out stack-rnn by facebook).
Even if an AI is trained for years, I don't see this limitation being overcome by pure magnitude of data with our current techniques. You say I don't have enough "future vision", but all you are doing is basically guessing that AI researchers will make some huge breakthrough that significantly changes the direction of the entire field. You are playing the lottery with your guesses, I'm simply extrapolating current trends.
>Almost every recent A.I. advance has come from one tiny corner of the field, machine learning. Machine learning exposes a set of connected nodes, known as neural nets, to mass amounts of labeled real-world data in an attempt to give those nodes tacit knowledge. The breakthrough example was software that was able to identify cat pictures.
>So far, these neural nets have given us some great demos but mostly niche real-world applications. We don't have self-driving cars quite yet!
What's the author's threshold for "real-world applications"?
- Google's Youtube algorithm for recommendations uses neural nets[1]. So ~2 billion viewers being affected by it doesn't seem like a "niche" application.
- Google language translation uses neural net[2]
- Apple Siri voice recognition uses neural net
It doesn't seem like neural nets are analogous to the joke that "graphene is the wonder material that can do everything except escape the research lab".
In contrast, deep learning neural nets have escaped the research lab and are widely used in production systems today.
The author's blog post is recently dated August 2021 so it seems like he's not kept up-to-date on this topic since the experimental neural net winning ImageNet in 2012. Yes, that was an artificial contest but things have progressed quickly and there is real-world commercial deployment of NN trained models.
The point is that no one could train deep nets 10 years ago. Not just because of computing power, but because of bad initializations, and bad transfer functions, and bad regularization techniques, etc.
These things might seem like "small iterative refinements", but they add up to 100x improvement. Even when you don't consider hardware. And you should consider hardware too, it's also a factor in the advancement of AI.
Also reading through old research, there is a lot of silly ideas along with the good ones. It's only in retrospect that we know this specific set of techniques work, and the rest are garbage. At the time it was far from certain what the future of NNs would look like. To say it was predictable is hindsight bias.
Yes. The advances in Deep Learning are very impressive and are already proving useful. Still there seem to be no real breakthroughs in actual intelligence, reasoning or something that resembles consciousness.
Precisely. When I first looked at neural nets 20 years ago or so it was a "Well Known Result"[tm] that deep networks of any kind were pointless and anything that could be done with a deep network could be done in a single layer. Like lots of tech procnostications that didn't age well.
Exactly. The history of NN is full of ups and downs and it's becoming increasingly popular again the form of Deep Learning thanks to increasing cloud processing power and advancements by Hinton and others. Most to of the traditional criticism of NN is related to shallow nets. But deeper and far more complex structures like those in the animal brains are not explored enough.
I think it's more a demonstration than anything, but unlike these demonstrations, no one expected just how powerful deep nets could be. No one expected they could win Go, by contrast, everyone (in DL) expected that they could do these things.
I do agree though that meta-learning is the future and very interesting, especially interested in the fact that DRL can now construct nets better than humans (and you can actually speed it up with an in-house technique I'm not allowed to share :( ).
IMO, wavenet and pixelcnn/rnn demonstrate the power of convolutions as a general purpose AI and broke the reign of RNNs/LSTMs for many tasks.
This has been said about neural nets two times already. Sadly, they did never deliver.
There are still applications where e.g. random forests beat the crap out of all kinds of deep learning algorithms in (a) training time (b) predictive quality (c) prediction time.
We should stop hyping this. I am a researcher working in deep learning myself, but the current deep learning hype is actually what makes me worry that I will have trouble getting a job because industry will be disappointed a third time.
Deep Learning was a noticeable improvement over previous neural models, sure. But deep learning is not the entire field of AI and ML. There has been more stuff going on like neural turing machines and differentiable neural computers.
Well, yes, mostly, but there also have been genuine discoveries in the last 10 years. We can now train deep networks because we learned how to regularize - before it was impossible because of vanishing gradients. We can have even 1000-layer deep nets, which would have been unthinkable. Also, there are some interesting approaches to unsupervised learning like GANs and VAEs. We learned how to embed words and not only words, but multiple classes of things into vectors. Another one would be the progress in reinforcement learning, with the stunning success of AlphaGo and playing over 50 Atari games. Current crop of neural nets do Bayesian statistics, not just classification. We are playing with attention and memory mechanisms in order to achieve much more powerful results.
This should not be downvoted, he is essentially correct.
'AI' has always been around, but the 'big leap' we've seen recently is entirely due to researchers finally able to make Neural Nets actually work :).
'Deep Learning' really refers to a specific kind of Neural Network.
The first application was image recognition. It's also used a lot now in voice recognition - and they're trying to jam it into everything possible.
I think it's basically fair to say that at least in 2016 AI pretty much boils down to Deep Learning / Neural Nets to move something more classical along in terms of capability.
What 'AI' means changes over time, but I think it's safe to say the renewal and hype is based on the 'big eureka' in Neural Nets and their relevant applications.
So this is what confuses me about what is 'new' with respect to deep learning. Neural nets are not new - I was aware of the existence of, and some of the basic ideas behind, neural nets as a technology in the 1990s and I wasn't even involved in computer science, so I assume that means that even then they were a mainstream AI technique.
When I read content about 'deep learning' neural nets today I don't see anything especially different to what my (admittedly shallow) understanding of neural nets was back then. So what I'm missing mainly is - what changed? Is it just that advances in compute power mean that problems for which neural nets were impractical have now become practical? Is there something different about the way neural nets are employed in 'deep learning' that is different than the neural nets that were discussed in the past?
Deep neural networks are quite powerful because they don't require someone to design the network specific to the task. Large deep nets will generally be able to learn feature representations by themselves, which makes them extremely powerful for tasks where we aren't quite sure of exactly what features we want.
However, deep nets require large amounts of training data and computational power. The fairly recent widespread adoption of general purpose GPUs has allowed much faster training. Combine this with the popularity of "big data", and you've got a perfect storm for deep neural nets.
Of course, the hype may be overvaluing deep nets as the future of AI. DNNs work well in practical applications, but they're poorly defined theoretically and the AI community suspects that we're still bad at training them -- a recent paper showed that a simpler shallow net can perform as well as a deep net if a deep net is trained first[1]. We're also fairly certain that deep nets are not how the brain actually works, and thus we'll need a different architecture in order to achieve human level performance on some tasks.
The current round is probably that deep learning can do some impressive things through some combination of CPU/GPU/TPU computing advances, the ability to collect and process a lot of data, and algorithmic/neural net architectural improvements in more or less that order. However, even some of the leading researchers are effectively cautioning that deep learning can probably only take you so far. [1] And we've arguably made very little progress on cognitive science generally.
And the answer is an obvious no, not even close, with the current state-of-the-art in deep nets. I could forgive an opaque model if it actually performed well.
Nah. Since neural networks are called deep learning it became very fashionable.
There's some well-funded startup (SkyMind maybe?) who declares on their homepage that deep learning is sooo much better than machine learning (first wtf). Then they explain that neural nets with less than 3 hidden layers is machine learning, with more than 3 layers it's deep learning.(second)
I didn't know if I should laugh or cry.(Not that what they actually seem to be doing is bad, but this newly found hype is just wrong.)
This fits perfectly into the narrative of yesterday's discussion on HN [1].
Deep Neural Nets are somewhat of a brute force approach to machine learning. Training efficiency is horrible as compared with other ML approaches, but hey, as long as we can trade +5% of classification performance for +500% of NN complexity and throw more money at the problem, who cares?
I see a dystopian future where much better and much more efficient approaches to ML exist, but nobody's paying attention because we have Deep Neural Nets in hardware and decades of infrastructure supporting it.
I have also seen what neural nets are incapable of. Specifically, generalisation and reasoning. Says François Chollet of Keras [2].
AI, i.e. the sub-field of computer science research that is called "AI" and that consists of conferences such as AAAI, IJCAI, NeurIPS, etc, and assorted journals, cannot progress on the back of a couple of neural net architectures incapable of generalisation and reasoning. We had reasoning down pat in the '80s. Eventually, the hype cycle will end, the Next Big Thing™ will come around and the hype cycle will start all over again. It's the nature of revolutions, see?
So hold your horses. Deep learning is much more useful for AI researchers who want to publish a paper in one of the big AI conferences, and to the FANG companies who have huge data and compute, than it is to anyone else. Anyone else who wants to do AI will need to wait their turn and hope something else comes around that has reasonable requirements to use, and scales well. Just as the original article suggests.
_________________
[1] http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-...
[2] https://blog.keras.io/the-limitations-of-deep-learning.htmlreply