>We've got a winner. Tensors "do" something; they're functions.
And a neural network is just a stack of linear transforms (the "tensors") and nonlinear transforms (the activations). So in a lot of cases, these "tensors" do do something. Granted the abstraction for storing the active tensors is also a great abstraction for storing high-dimensional data (like a 128x128x4x60 2-second video).
> This is something I am interested and considering as a possible future for my career, coming from a math background and starting to dip my toes into ML.
This is a great time and place to be! Neural networks are the twenty-first century Fourier series. It's just that we don't yet understand them. We can easily run them (synthesis) but we are missing the analysis. There's a lot of math to do here.
> Advanced math is mostly useless because of the dimensionality of neural nets.
It depends what you mean by advanced math. There is a lot of math that only really comes into play because of the high dimensionality! For example math related to tensor wrangling, low rank approximations, spectral theory, harmonic theory, matrix calculus derivatives, universality principles, and other concepts that could be interesting or bewildering or horrifying depending how you react to it. Of course some of it is only linear algebra of the 'just high school math' kind but that's not how I would normally describe it. If you look at the math in the proofs in the appendices of the more technical AI papers on arxiv there is often some weird stuff in there, not just matrix multiply and softmax.
> The only numerical linear algebra related to machine learning, in particular deep learning, is matrix multiplication
I suspect the OP is not talking about how deep nets are implemented, but rather how people are trying to understand how and why they work so well, or how to reverse-engineer knowledge out of a trained net, or how to train them faster, etc ...
In that space, you need quite a bit more than matrix multiplication.
>Do people really understand what complex neural networks do
From a biological sense, the only neural net we have a decent understanding of is the C elegans nematode worm with a few hundred neurons. Large neural networks like that of our brain are beyond our understanding and I suspect the same is the case for ANNs.
>This is a rather intuitive mathematical proof that neural nets can emulate all functions in many dimensions.
Yes, but the problem is we don't know what function we want to approximate. So it is like interpolation you give it some points and you get an approximation that is good enough around those points but probably very wrong if you are far away of the points.
We don't even know how many layers and neurons are needed so it seems to be a brute force approach, throw more data and more hardware and when stuff goes wrong maybe throw more data. But one day a Tesla might go on a different environment and maybe people have different hats and goes to shit because those hats were not in the training data, we don't have like in other sciences bounded errors.
> The other is that current neural networks are probably very inefficient algorithms. They are fundamentally based on linear algebra but there is some evidence that the non-linearity is the key.
Deep neural networks, though they use linear algebra, are definitely non linear, as a result of both their architecture and their use of non linear activation functions.
Their non-linearity is part of the reason they can classify phenomena that linear techniques like simple regression cannot.
> What so-called neural networks do should not be confused with thinking, at least not yet.
I disagree:
I think neural networks are learning an internal language in which they reason about decisions, based on the data they’ve seen.
I think tensor DAGs correspond to an implicit model for some language, and we just lack the tools to extract that. We can translate reasoning in a type theory into a tensor DAG, so I’m not sure why people object to that mapping working the other direction as well.
>> As far as we know, neural networks indeed are at the core of human intelligence.
Right.
Quick intro to neural models, for the only very slightly mathematically inclined.
Neural networks solve this equation:
Y = W · X
Where X a matrix of inputs to a function ƒ, W a matrix of its coefficients and Y
a matrix of its outputs.
Once that's done once for all N instances of X in our dataset, a time period which we term an epoch, Y is used to calculate a measure of error, typically mean squared error:
MSE = 1/N ? (Y' - Y)²
Where Y' the true value of Y that we're trying to approximate. This is repeated
until the cows come home, or until the network has converged, whichever comes
home first (er, hint).
And that's neural networks in a nutshell. There's nothing much more complicated
than that (er, in theory), there's no magic and certainly nothing "like the
brain" in them at all. They're just optimisation over systems of functions. We
learned how to solve systems of functions with matrix multiplication when we
were at high school. I'm pretty sure we did - I've personally repressed the
memory but I'm pretty sure I did once have it.
Why do people insist on calling them "neural"? Who the fuck knows. Way back in
the 50s, the perceptron was based on a model of neural activation, where a
neuron "fires" when its electrical charge crosses over a threshold. This was
originally represented with a binary step function (the very one used in
Perceptron classification). That was, at the time, deemed to be "a good model"
of neural activation. A good model, my foot. Soon people moved on to sigmoid
functions which are "a better model" of neural activation. Whatever.
I've also heard it that, say, conv nets (convolutional neural networks) are
"like the brain", presumably because if you squint at the layer connection
diagrams long enough, conv nets start to look a bit like idealised models of the parts of the brain that handle vision.
But if you squint long enough, everything starts to look like everything else.
Hell, you might suggest that neural networks are based on the effin Tree of Life
from Cabala, and Geoff Hinton is Alistair Crowley reincarnated.
The whole "brain" thing is just an abstraction. You should not read too much
into it. And don't listen to the people who try to convince you neural networks
are magic- they're not. It's high school maths and a lot of very expensive
processing power.
P.S. If my notation doesn't make sense to you, you may need to use a different font in your browser; sorry about that.
>Common Misconceptions: Neural networks work like brains. In fact, real neurons are much more complex than the simple units used in artificial neural networks; there are many different types of neurons; real neural connectivity can change over time; the brain includes other mechanisms, besides communication among neurons, that affect behavior; and so on.
I really have to disagree with this statement because it's actually broad and misleading. Yes, real neurons are more complex, but that doesn't mean that artificial neural networks are not capturing the important functional/computational properties or real neurons. In fact, you might even say that real neural networks are approximating the computational properties of artificial neural networks via a somewhat Rube Goldberg like process. If you actually look at the mathematics that functionally describe artificial neural networks it's the same math that describes gene regulatory networks. This is not a coincidence. This mathematical abstraction like a platonic computational system. In effect, evolution converged on this computing paradigm twice using the processes available at its disposal. The first was a chemical reaction network, the second was neurons. Moreover, if you actually study the computational properties of these networks you discover that most of the implementation details are actually irrelevant to the functional output and it's the topology of the network that's primarily driving the function. Not unlike how an engineer might recognize the A 8-bit added from the circuit-logic diagram. The problem that people have with artificial neural networks is that they're not spending enough time reverse engineering the salient properties of biological systems and instead they're trying to brute force AI with their mathematical brilliance. A lot of these questions disappear if you study the network architecture.
> as I scientist I am very uninterested in AI based on neural nets because of the lack of explication
Neural nets are more like reflexes than reasoning. Most of them are feed forward and some don't even have memory and can't solve references. So it's unfair to expect a job that is best done based on graphs or on memory-attention to be done by a rudimentary system that only knows to map X to y.
But they are not totally unexplainable - you can get gradients on the data and see what parts of the input data most influenced the output, then you can perturb the inputs to see how the output would change.
>I believe the human mind is specialised to think in terms of serial solutions
The brain is massively parallel - 80 billion neurons operating asynchronously at only 100hz.
>is having a neural network the end problem that needs solving?
Absolutely! Neural networks are a general-purpose way to solve a huge number of problems, especially problems that involve raw data or the messy real world.
Yes, it's multiplying and adding matrices. That and mapping some simple function over an array.
Neural networks are only that.
reply