Neuromorphic computing basically uses individual "neurons", represented with either analog or digital circuits, which communicate using asynchronous pulses called "spikes". Unlike the human brain, neuromorphic chips are 2D, but we can replicate a good amount of neural dynamics in silicon.
It's unclear how they managed to use this to run LLMs, though. Getting GPT-2 running with SNNs is a legitimate achievement, because SNNs have traditionally lagged significantly behind conventional deep learning architectures.
Neuromorphic computing is cool, but not new tech. However, using a neuromorphic spiking architecture to run LLMs seems new. Unfortunately, there doesn't seem to be a paper associated with this work, so there's no deeper information on what exactly they're doing.
“In neuromorphic computing, however, a "spike input" — a set of discrete electrical signals — is fed into the spiking neural networks (SNNs), represented by the processors. Where software-based neural networks are a collection of machine learning algorithms arranged to mimic the human brain, SNNs are a physical embodiment of how that information is transmitted. It allows for parallel processing and spike outputs are measured following calculations.
Like the brain, Hala Point and the Loihi 2 processors use these SNNs, where different nodes are connected and information is processed at different layers, similar to neurons in the brain.”
My impression is that there hadn't been much success with SNN's yet.
Neuromorphic Computing is already a thing, and Intel's already developing chips (Loihi 2).
But it's not as powerful as GPU's yet, and it only runs spiking neural networks.
I'm a PhD student working with neuromorphic computing. I like to think about SNNs as RNNs with discretized outputs. The neurons themselves may have some complicated nonlinear dynamic (currents integrating into the membrane voltage somehow etc.) but they are essentially just stateful transfer functions. The notion of spikes is a crippling simplification, but it's power efficient and you can argue for numerical stability in the limit. So I tend to consider spikes as an annoying engineering constraint in some neuromorphic systems. Brains function perfectly well without them, although in smaller scales (C. elegans).
The true genius of neuromorphics in my view, is that you can build analog components that performs neutron integration for free. Imagine a small circuit that "acts" like the stateful transfer function, with physical counterparts to the state variables (membrane voltage, synaptic current, etc.). In such a circuit you don't need transistors to inefficiently approximate your function. Physics is doing the computation for you!
This gives you a ludicrous advantage over current neural net accelerators. Specifically 3-5 orders of magnitude in energy and time, as demonstrated in the BranScaleS system https://www.humanbrainproject.eu/en/science-development/focu...
Unfortunately, that doesn't solve the problem of learning. Just because you can build efficient neuromorphic systems doesn't mean that we know how to train them. Briefly put, the problem is that a physical system has physical constraints. You can't just read the global state in NWN and use gradient descent as we would in deep learning. Rather, we have to somehow use local signals to approximate local behaviour that's helpful on a global scale. That's why they use Hebbian learning in the paper (what fires together, wires together), but it's tricky to get right and I haven't personally seen examples that scale to systems/problems of "interesting" sizes. This is basically the frontier of the field: we need local, but generalizable, learning rules that are stable across time and compose freely into higher-order systems.
Regarding educational material, I'm afraid I haven't seen great entries for learning about SNNs in full generality. I co-author a simulator (https://github.com/norse/norse/) based on PyTorch with a few notebook tutorials (https://github.com/norse/notebooks) that may be helpful.
I'm actually working on some open resources/course material for neuromorphic computing. So if you have any wishes/ideas, please do reach out. Like, what would a newcomer be looking for specifically?
I don’t know too much about their technology and the website isn’t giving away too much detail. It doesn’t look like they are using spiking networks, so no event-based neuromorphic tech, but perhaps good old linear algebra/ANN ML. They’re using analog computation which is attractive power-wise, but in the past has always suffered from variability due to device mismatch. Unless they have some really revolutionary process or algorithm that magically makes the downsides of mismatch disappear, they’ll have a hard time going beyond what has been tried in analog computing before (and which had its heyday in the 70s).
Neuromorphic computing is running some known ANN model directly in hardware. Why do we want it? Because ANN models in software work well for pattern matching, and we want to speed it up/make it more efficient.
There are neuromorphic deep learning algorithms. From what I read, one promise of these spiking neural networks is higher efficiency than that of typical neural nets, which would enable learning from much fewer data samples.
If anybody here works with SNNs, can you share if you think this claim is true? Also, are there any good entry points for people interested in learning more about SNNs?
I don't understand if those chips are designed to function like real neurons in the brain, or just designed to run artificial neural networks (ie matrix multiplication).
In the beginning it seems that they do mimick real neurons, but later the article says that it runs deep learning benchmarks. How can they run the benchmark, how did they train their chips ?
The main problem with SNNs is not that they don't work well. It's not that they are slow to run on CPUs/GPUs. Those would indeed be temporary problems. The fundamental problem is we don't know the right abstraction level to imitate brain operations. It's far from clear that simulating spikes is necessary to implement brain's "learning algorithms". There are two main arguments for using spikes: "biological realism", and "energy efficiency". Neither is convincing: if you want to simulate a conventional CPU you don't want to simulate IV curves for transistors. You don't want to simulate CMOS gates. No, because the lowest abstraction level necessary to understand the computer operation is boolean logic. Anything below that is completely irrelevant. I strongly suspect that simulating spikes is below the lowest relevant abstraction level. By the way, Jeff Hawkins of Numenta agrees with me, and he's pretty strict about "biological plausibility" of his neocortex algorithms. As for energy efficiency - sure, spikes might be the most efficient way to compute and encode information given the constraints of biological brains. But why should we care about those constraints? We are building computing machinery in silicon, using vastly different technology, circuits, and tricks from what's available in "wetware". It does not make any sense to copy biological evolution tricks to improve efficiency when building things in silicon. Neuromorphic hardware papers always mention energy efficiency and then proceed to compare their analog spiking chips to digital (!) chips. That's ridiculous! I can't think of a single good argument why would spiking analog circuits be more energy efficient than non-spiking analog circuits, if we are talking about any existing computing hardware technology (or one that's likely to be developed in the foreseeable future).
Deep learning took off in 2012 not because faster hardware allowed us to develop good algorithms. The algorithms (gradient descent optimization, backpropagation, convolutional and recurrent layers) have been developed -- using slow hardware -- long before (Lecun demonstrated state of the art on MNIST back in 1998). The fast hardware allowed us to scale up the existing good algorithms. I don't see any good algorithms developed for SNNs. Perhaps these algorithms can be developed, perhaps faster hardware is indeed necessary, but as I argue above, the motivation to pursue this research is just not obvious to me.
Note that we shouldn't confuse this SNN research (such as the papers you cited), to efforts like Human Brain Project, where they're actually trying to derive higher level brain algorithms from accurate simulations of low level mechanics. Emphasis on accurate, because as any neuroscientist will tell you (e.g. [1]), these SNNs have very little to do with what's actually going on in a brain on any abstraction level.
A neuromorphic processor is a processor that tries to function as the brain does. I say tries to because the models are still very simplified versions of how we think that a neuron works. The spiking part comes from one of these models. A neuron will only send an electrical signal to another neuron once a certain threshold is met, this is what is meant by a spike. What this allows us to do is to add a temporal/time component to a neural network. You've probably heard of "neurons that wire together, fire together". If you have a lower threshold it means that the neuron you are sending the signal to is more relevant for whatever thought process is going on right now and vice versa. New input can change these thresholds.
The biggest promise of spiking neuromorphic computing seems to be a massive reduction in energy usage while still offering decent accuracy. So for example you could use it to train a neural network to get to 80% accuracy after which you'd let another type take over to get to 95%. This field is still in its infancy though, so expect things to change/improve fast.
PhD student here doing neuromorphic computing and running deep learning models on analog hardware. The 'neuromorphic computing" is a vaguely defined term, which can mean anything from developing novel Spiking NN models to the design of hardware capable to run those models (e.g. TrueNorth). Wetware computing is probably even lesser understood area. Neither has much to do with deep learning field, which does not use spiking NN computation.
On the other hand, there are very promising research directions precisely in the intersection of neuroscience and deep learning (e.g. biologically plausible credit assignment problem, learning algorithms of the brain, objective functions in the brain, sensori-motoric integration, etc). It's rare to find a person with solid understanding of both fields, especially in academia.
If I were you I would look at residency programs at big companies (Google Brain, Facebook AI fellowship, etc). Also, there's a company called Numenta which is probably looking for people with your skills (they are trying to reverse engineer neocortex' algorithms). From what I heard, those places give you some freedom to choose your own problems, encourage publishing papers, and provide very bright people as mentors.
I dont understand why pursuing "neuromorphic hardware". These are like asics running the most rudimentary (and wrong) simulation of human neurons, which are not known to solve any problem very well, in order to ... what exactly?
Artificial neural networks with backprop is all we need for AI
There is very little information about how the two types of networks, neural and neuromorphic, are trained, which is an important issue for either one separately, and even more important when they're combined. Pei and colleagues write, in the "Methods" section of the paper, that they trained the deep learning part in the normal way, and that for the neuromorphic part, they relied on a method introduced last year by some of the researchers, called "Spatio-temporal backpropagation," a version of the backpropogation approach common in deep learning.
There are also some missing details about the chip's fabrication. For example, the part is said to have "reconfigurable" circuits, but how the circuits are to be reconfigured is never specified. It could be so-called "field-programmable gate array," or FPGA, technology or something else. Code for the project is not provided by the authors as it often is for such research; the authors offer to provide the code "on reasonable request."
More important is the fact the chip may have a hard time stacking up to a lot of competing chips out there, says analyst Gwennap. The specs seem underwhelming, in his view. "Tianjic's reported 1.28 TOPS/watt [trillions of operations per watt, a common measure of performance] is similar to today's GPUs," he notes, referring to graphics processing unit chips made by Nvidia and Advanced Micro Devices. However, the performance is "well behind more advanced architectures" of newer chips, he notes.
Gwennap's colleague, Mike Demler, concurs. He notes some inaccuracies in the paper, such as the contention that spiking neurons require "extra high-precision memory" circuits for some functions. Demler's review of a neuromorphic chip by chip giant Intel, called "Loihi," shows that such is not the case. A chip developed by startup Brainchip, Inc., also proves the claim false, he says. Moreover, since the Loihi chip has already shown that conventional neural networks, such as a convolutional neural network, or CNN, can be implemented as a spiking neuron, there's no need for the kind of "unified" chip that the Tsinghua authors claim.
They have been designed, and are being used either for more efficient pattern matching, or to speed up brain simulations (again, using known neuronal models).
You seem to expect something else from neuromorphic computing, why?
It's unclear how they managed to use this to run LLMs, though. Getting GPT-2 running with SNNs is a legitimate achievement, because SNNs have traditionally lagged significantly behind conventional deep learning architectures.
https://web.stanford.edu/group/brainsinsilicon/documents/ANe... https://web.stanford.edu/group/brainsinsilicon/documents/Ben...
reply