The idea was always appealing, but the implementation has always remained challenging.
For over a decade, "Mythic AI" was making accelerator chips with analog multipliers based on research by Laura Fick and coworkers. They raised $165M and produced actual hardware, but at the end of 2022 have almost gone bankrupt and since then there has been very little heard from them.
Much earlier, the legendary chip designers Federico Faggin and Carver Mead founded Synaptics with an idea to make neuromorphic chips which would be fast and power efficient by harnessing analog computation. Carver Mead published a book on that in 1989: "Analog VLSI and Neural Systems", but making working chips turned to be too hard, and Synaptics successfully pivoted to touchpads and later many other types of hardware.
Of course, the concept can be traced to an even older and still more legendary Frank Rosenblatt's "Perceptron" -- the original machine learning system from 1950s. It implemented the weights of the neural network as variable resistors that were adjusted by little motors during training. Multiplication was simply input voltage times conductivity of the resistor producing the current -- which is what all the newer system are also trying to use.
The idea of analog neural networks is appealing. I bought Analog VLSI and Neural Systems in 1989 and still have it as a trophy on my bookshelves. My gut feeling says one day analog neural networks will be a thing, if only for the reason of considerable lower power consumption.
I’m not saying that life is analog, DNA is two bits. IMHO life is a mix of Analog & Digital.
I know of only one real world successful product using analog computation in place of expensive high end micro. It was the first proper (no dedicated special mousepads) Optical Mouse designed and build by HP->Agilent->Avago and released by Microsoft in 1999 as IntelliMouse Optical. https://gizmodo.com/20-years-ago-microsoft-changed-how-we-mo... Afaik Microsoft bought 1 year explosivity for the sensor. Avago HDNS-2000 chip did all the heavy lifting in analog domain.
"each array element had nearest neighbor connectivity so you
would calculate nine correlations, an autocorrelation and eight cross-correlations, with each of your eight
nearest neighbors, the diagonals and the perpendicular, and then you could interpolate in correlation
space where the best fit was."
"And the reason we did
difference squared instead of multiplication is because in the analog domain I could implement a
difference-squared circuit with six transistors and so I was like “Okay, six transistors. I can’t do
multiplication that cheaply so sold, difference squared, that’s how we’re going to do it.”
"little chip running in the 0.8 micron CMOS could do the equivalent operations per second to 1-1/2 giga operations per second and it was doing this for under 200 milliwatts, nothing you could have approached at that time in the digital domain."
Faster than an H100 for solving 128x128 matrices. But it’s not clear to me how they tested this, code is only available on request.
> We have described a high-precision and scalable analogue matrix
equation solver. The solver involves low-precision matrix operations,
which are suited well to RRAM-based computing. The matrix operations
were implemented with a foundry-developed 40-nm 1T1R RRAM array
with 3-bit resolution. Bit-slicing was used to guarantee the high preci-
sion. Scalability was addressed through the BlockAMC algorithm, which
was experimentally demonstrated. A 16 × 16 matrix inversion problem
was solved with the BlockAMC algorithm with 24-bit fixed-point preci-
sion. The analogue solver was also applied to the detection process
in massive MIMO systems and showed identical BER performance
within only three iterative cycles compared with digital counterparts
for 128 × 8 systems with 256-QAM modulation.
But what do we do about bottleneck operating systems like Windows 11. You can give them a chip 10000x faster but they find way to add more telemetry, more bloat and thus render those gains meaningless. Let us talk about this from the perspective of a gamer, a guy who solely depends on Windows for Visual C++, .NET SDK etc etc (the versions for these go back all the way to 2000s) We need an OS capable of running games all the way from good old Quake 2 to modern games but the GPU isnt the bottleneck anymore
This looks like one of many ideas for more efficient compute chips for machine learning. I'm waiting for the day some chip gets mass produced and works at scale for some large model and with sufficient reliability, but until then, I don't think there's anything particularly newsworthy here. I do think it'll eventually happen at some point maybe within a decade, but surely some alternative computing paradigm to the GPU will succeed. The analog chip in the article only seems to be a research prototype for now.
The idea was always appealing, but the implementation has always remained challenging.
For over a decade, "Mythic AI" was making accelerator chips with analog multipliers based on research by Laura Fick and coworkers. They raised $165M and produced actual hardware, but at the end of 2022 have almost gone bankrupt and since then there has been very little heard from them.
Much earlier, the legendary chip designers Federico Faggin and Carver Mead founded Synaptics with an idea to make neuromorphic chips which would be fast and power efficient by harnessing analog computation. Carver Mead published a book on that in 1989: "Analog VLSI and Neural Systems", but making working chips turned to be too hard, and Synaptics successfully pivoted to touchpads and later many other types of hardware.
Of course, the concept can be traced to an even older and still more legendary Frank Rosenblatt's "Perceptron" -- the original machine learning system from 1950s. It implemented the weights of the neural network as variable resistors that were adjusted by little motors during training. Multiplication was simply input voltage times conductivity of the resistor producing the current -- which is what all the newer system are also trying to use.
The idea of analog neural networks is appealing. I bought Analog VLSI and Neural Systems in 1989 and still have it as a trophy on my bookshelves. My gut feeling says one day analog neural networks will be a thing, if only for the reason of considerable lower power consumption.
I’m not saying that life is analog, DNA is two bits. IMHO life is a mix of Analog & Digital.
It is very difficult to scale digital-analog hybrids, because of amount of DAC-ADC components required.
I know of only one real world successful product using analog computation in place of expensive high end micro. It was the first proper (no dedicated special mousepads) Optical Mouse designed and build by HP->Agilent->Avago and released by Microsoft in 1999 as IntelliMouse Optical. https://gizmodo.com/20-years-ago-microsoft-changed-how-we-mo... Afaik Microsoft bought 1 year explosivity for the sensor. Avago HDNS-2000 chip did all the heavy lifting in analog domain.
Travis Blalock Oral History https://www.youtube.com/watch?v=wmqa9XJED-Q https://archive.computerhistory.org/resources/access/text/20...:
"each array element had nearest neighbor connectivity so you would calculate nine correlations, an autocorrelation and eight cross-correlations, with each of your eight nearest neighbors, the diagonals and the perpendicular, and then you could interpolate in correlation space where the best fit was."
"And the reason we did difference squared instead of multiplication is because in the analog domain I could implement a difference-squared circuit with six transistors and so I was like “Okay, six transistors. I can’t do multiplication that cheaply so sold, difference squared, that’s how we’re going to do it.”
"little chip running in the 0.8 micron CMOS could do the equivalent operations per second to 1-1/2 giga operations per second and it was doing this for under 200 milliwatts, nothing you could have approached at that time in the digital domain."
Extra Oral History with inventor of the sensor Gary Gordon: https://www.youtube.com/watch?v=TxxoWhCzIeU
Faster than an H100 for solving 128x128 matrices. But it’s not clear to me how they tested this, code is only available on request.
> We have described a high-precision and scalable analogue matrix equation solver. The solver involves low-precision matrix operations, which are suited well to RRAM-based computing. The matrix operations were implemented with a foundry-developed 40-nm 1T1R RRAM array with 3-bit resolution. Bit-slicing was used to guarantee the high preci- sion. Scalability was addressed through the BlockAMC algorithm, which was experimentally demonstrated. A 16 × 16 matrix inversion problem was solved with the BlockAMC algorithm with 24-bit fixed-point preci- sion. The analogue solver was also applied to the detection process in massive MIMO systems and showed identical BER performance within only three iterative cycles compared with digital counterparts for 128 × 8 systems with 256-QAM modulation.
But what do we do about bottleneck operating systems like Windows 11. You can give them a chip 10000x faster but they find way to add more telemetry, more bloat and thus render those gains meaningless. Let us talk about this from the perspective of a gamer, a guy who solely depends on Windows for Visual C++, .NET SDK etc etc (the versions for these go back all the way to 2000s) We need an OS capable of running games all the way from good old Quake 2 to modern games but the GPU isnt the bottleneck anymore
This looks like one of many ideas for more efficient compute chips for machine learning. I'm waiting for the day some chip gets mass produced and works at scale for some large model and with sufficient reliability, but until then, I don't think there's anything particularly newsworthy here. I do think it'll eventually happen at some point maybe within a decade, but surely some alternative computing paradigm to the GPU will succeed. The analog chip in the article only seems to be a research prototype for now.
This group has had some success turning machine learning algorithms into low power analog chips:
https://sites.dartmouth.edu/odame/
Not the same as general purpose training type computations though.
Wonderful, can’t wait to run Crysis with this chip.
Using all analog signal, why non analogue multiplying cells (operation amplifier)!
Seems a bit too good to be true.
Huge if true, room temperature semiconductor if false
Semi or supra conductor ?
Now put it in a guitar pedal!
Wo Fat has you covered: "Analog Man" => https://open.spotify.com/track/6KcM6et6Pn6UIna1o8Vl07?si=qFu...
What’s this good for?
Computing, they propose
Fear