Hacker Read

earthnail | karma 965 | avg karma 3.73 · 2023-06-05 16:34:20

The Neural engine is for inference, not training.

sillysaurusx | karma 21456 | avg karma 5.32 · 2023-06-05 17:25:50

There’s no difference between inference and training. Training is simply ~4x more expensive inference.

The submitted article also talks about training models.

bufo | karma 1591 | avg karma 10.97 · 2023-06-06 09:03:42

There is a difference. We train with large batch sizes these days. The ANE silicon size is tiny and can't do the large matrix multiplications for big LLMs with or without a batch size higher than 1. Meaning that it cannot saturate the RAM bandwidth and that you're better using off the much bigger GPU on the Apple die.