Deep learning is fundamentally linear algebra. Spiking networks are fundamentally event-based processors. The two concepts don’t play well together.
Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs).
Generally, spiking networks will likely have an edge when the signals they are processing are events in time. For example, when processing signal streams from event based sensors, like silicon retinas. There’s also evidence that event-based control has advantages over their periodically-sampling equivalents.
Neuromorphic hardware has been usable for 10 years now. Since then, algorithms for neuromorphic hardware (i.e. spiking networks) always performed 'almost as good' as ANN solutions on GPUs (meaning: inferior). But each year a new generation of GPUs comes out, using modern processes, with excellent toolchains. In a direct comparison of power efficiency, GPUs win over NMHW most of the time.
I would love to see Spiking Networks and NMHW take over machine learning but it has such a long way to go. And I seriously doubt the strategy, followed by most players, to try to beat good old ANNs at their own game.
Unless we identify a problem set where event-based computing with spikes is the inherently natural solution, I find it hard to imagine that spiking networks will ever outcompete ANN solutions.
> Spiking Neural Networks (SNNs) have been an attractive option for deployment on devices with limited computing resources and lower power consumption because of the event-driven computing characteristic.
But training them requires new techniques compared to continuous neural networks — SNNs aren't differentiable and therefore you can't back-propagate (as I understand it; please correct me if I'm off here).
> Unless we identify a problem set where event-based computing with spikes is the inherently natural solution, I find it hard to imagine that spiking networks will ever outcompete ANN solutions.
i'd guess that domain would be real time (unbuffered / unbatched) processing of raw sensory data. it seems reasonable that biological neural systems evolved for optimal processing of sensory information encoded temporally in spike trains, yet the few papers on neuromorphic computing i've seen tend to try and hammer spiking neural networks into a classic batch based machine learning paradigm and then score them against batch based anns.
Spiking networks are to Machine Learning like Haskell to C++ and Python: While not really used to solve many real-world problems, they are extremely interesting academically, and important concepts have ended up in the mainstream, like event-based sensing and control, or event-driven signal processing.
Spiking neural networks are not software, they are usually built directly into silicon chips because they are using pulse timing to encode information instead of multiple bits. The problem is that training them is difficult because they operate over time, not that they don't work. As of now, scaling training infrastructure is more important than theoretical power efficiency.
I don't get the appeal of spiking networks. They struggle to solve problems that have already readily been solved by ANNs and they don't offer much in terms of biological realism - they don't account for neuronal geometry and dendritic nonlinear phenomena, nor do they explain the protein dependence of LTP.
These are two disparate applications of spiking networks. 1. Machine learning—yes, many in the field seem to be trying to reinvent ANNs with spikes. Not very useful in my opinion. 2. Modeling biological processes. A lot of progress has been made in neuroscience research thanks to spiking network models, coupled with dendritic computation and all other sorts of biological detail. But one would not normally use neuromorphic hardware if the end goal is biological realism.
Only if biological realism is required in real time and on a constrained power budget, such as on a robot, is Neuromorphic hardware the weapon of choice.
Spiking neural networks do have many advantages over conventional rate based networks. They can, for example, handle time which a convnet cannot do without resorting to frames.
They also have a lot of disadvantages, the main one being that no one has any idea of how to use them properly yet.
I don’t know too much about their technology and the website isn’t giving away too much detail. It doesn’t look like they are using spiking networks, so no event-based neuromorphic tech, but perhaps good old linear algebra/ANN ML. They’re using analog computation which is attractive power-wise, but in the past has always suffered from variability due to device mismatch. Unless they have some really revolutionary process or algorithm that magically makes the downsides of mismatch disappear, they’ll have a hard time going beyond what has been tried in analog computing before (and which had its heyday in the 70s).
From what little I've gathered about the subject, I think so. Spiking neural networks would be orders of magnitude more energy efficient than deep neural networks for image and audio processing. The main hurdle lies in designing network structure and weights; the cost function of a classical neural network can be differentiated using basic calculus, but optimizing a spiking neural network is not as easy.
Spiking nets are an academic curiosity. They do not work currently, and might never work. Pretty much zero progress since Carver Mead proposed them in the late 80s. Intel Loihi is a joke, it's a useless product pushed by some exec as a pet project. Considering the mess Intel has been lately, I'm not surprised something like that got approved.
Algorithms to successfully perform any practical tasks with spiking nets do not exist yet. We have very little clue how a brain uses spikes. As a general rule, you don't want to build hardware to accelerate something that does not work in software.
Source: I worked in a lab where we have built memristor based chips to run spiking nets for image classification and other tasks (my role was developing training algorithms).
You are totally right. The algorithm itself is a potential game-changer. I guess I was carried away by the pitch in the abstract that starts off with deep learning.
Personally I think that way too many resources were wasted on trying to make better deep networks with spikes. In my opinion it is much more promising to apply spiking networks on problems that are inherently event-based.
Having a functional backpropagation algorithm such as the one provided can help with that, obviously.
There are neuromorphic deep learning algorithms. From what I read, one promise of these spiking neural networks is higher efficiency than that of typical neural nets, which would enable learning from much fewer data samples.
If anybody here works with SNNs, can you share if you think this claim is true? Also, are there any good entry points for people interested in learning more about SNNs?
I'm pretty surprised by all the focus on spiking neural networks too; they are really still much more of an academic topic and I have never heard of any real practical applications (yet).
There are a number of methods of training these kinds of networks like Spike Timing Dependent Plasticity (STDP) which is essentially a reinforcement learning algorithm that increases the weights between neurons that spike often together and modulates the increases by a reward signal. However, all these methods are really more focused on replicating and modeling the biological phenomena and not being performant.
In theory, spiking networks should be much more efficient, real neurons are able to take advantage of many non-linear effects to create incredibly complex analog to digital (spikes) computation in every cell using incredibly little energy - our brains only use about 20 watts of power.
But, the current approach to CPUs and even GPUs means simulating all this stuff is ridiculously inefficient and ultimately looks nothing like the real thing.
I strongly believe the next generation of models will be based upon spiking neural concepts wherein action potentials are lazily-evaluated throughout the network (i.e. event-driven). There are a few neuron models that can be modified (at some expense to fidelity) in order to tolerate arbitrary delays between simulation ticks. Using actual latency between neurons as a means of encoding information seems absolutely essential if we are trying to emulate biology in any meaningful way.
Spiking networks also lend themselves nicely to some elegant learning rules, such as STDP. Being able to perform unsupervised learning at the grain of each action potential is really important in my mind. This gives you all kinds of ridiculous capabilities, most notably being the ability to train the model while it's live in production (learning & use are effectively the same thing).
These networks also provide a sort of deterministic, event-over-time tracing that is absent in the models we see today. In my prototypes, the action potentials are serialized through a ring buffer, and then logged off to a database in order to perfectly replay any given session. This information can be used to bootstrap the model (offline training) by "rewinding" things very precisely and otherwise branching time to your advantage.
The #1 reason I've been thinking about this path is that low-latency, serialized, real-time signal processing is somewhat antagonistic to GPU acceleration. I fear there is an appreciable % of AI research predicated on some notion that you need at least 1 beefy GPU to start doing your work. Looking at fintech, we are able to discover some very interesting pieces of technology which can service streams of events at unbelievable rates and scales - and they only depend on a handful of CPU cores in order to achieve this.
Right now, I think A Time Domain Is All You Need. I was inspired to go outside of the box by this paper: https://arxiv.org/abs/2304.06035. Part 11 got me thinking.
But is sparse activation sufficient to motivate the use of SNNs? In my opinion one needs a temporal component as well.
Sparse activations that don't also have a time component (i.e. are sparse in space and time) can be very well implemented without events.
Granted, SNN processors can handle sparse activations better than matrix accelerators. But then again, SNN accelerators might carry lots of SNN overhead that is not required for sparse activations alone.
Edit: A good example for a non-spiking sparse activation accelerator is the NullHop architecture [1].
The learning mechanism[1] is actually relatively well understood (at least as far as neuroscience goes) but it isn't used for ANNs because Spiking Neural Networks(SNNs) don't map well onto conventional computing hardware whereas the matrix multiplication used by conventional ANNs does.
I agree that it's worth researching but currently it is quite difficult to compete with conventional ANNs due to the commonality of processors well suited for ANNs.
Many researchers have been trying hard to shoe-horn deep ANNs into spiking networks for the last 10 years. But this doesn’t change the fact that linear algebra is best accelerated by linear algebra accelerators (i.e. GPUs/TPUs).
Generally, spiking networks will likely have an edge when the signals they are processing are events in time. For example, when processing signal streams from event based sensors, like silicon retinas. There’s also evidence that event-based control has advantages over their periodically-sampling equivalents.
reply