Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a neural network's clever solution to binary addition

cprimozic.net

Post reply on HN
Print view View original

131–140 of 160 posts

Re: Reverse engineering a neural network's clever solution to binary addition

#131

Post by garganzol »

em3rgent0rdr wrote: post #119
garganzol wrote: post #56

Earlier quoted context omitted.

It can be implemented in hardware but the implementation would be more complex than a digital adder based on logical gates.

Analog addition is actually really easy if you can tolerate noise and heat and can convert to/from representing the signal as an analog current. Using Kirchhoff's Current Law, addition of N currents from current sources is achieved by joining those N wires to a shared exit wire which contains the summed current.

Addition is easy, DAC is relatively easy, but ADC is not.

Re: Reverse engineering a neural network's clever solution to binary addition

#132

Post by shagie »

djmips wrote: post #129
elcritch wrote: post #128

Earlier quoted context omitted.

Nifty, makes one wonder if logarithmic or sigmoid functions for ML could be done using this method. Especially as we approach the node size limit, perhaps dealing with fuzzy analog will become more valuable.

There are a few startups making analog ML compute. Mythic & Aspinity for example https://www.eetimes.com/aspinity-puts-neural-networks-back-t...

Veritasium has a bit of an video on Mythic - Future Computers Will Be Radically Different (Analog Computing) - https://youtu.be/GVsUOuSjvcg

Re: Reverse engineering a neural network's clever solution to binary addition

#133

Post by nomel »

pjc50 wrote: post #50
qikInNdOutReply wrote: post #44

Earlier quoted context omitted.

Ah, the good old radio component you can program into fpgas.

Very easy to design a radio component into electronics, much harder to design it out.

I mistakenly made a two AM radios in my electronics classes, from trying to make an amplifier and a PLL. The FM radio was mistakenly made while trying to make an AM radio. :-|

Re: Reverse engineering a neural network's clever solution to binary addition

#134

Post by mgliwka »

WalterBright wrote: post #123
convolvatron wrote: post #113

Earlier quoted context omitted.

you can mention Massalin by name. she's pretty brilliant, idk where she is now

I don't recall who wrote the paper, thanks for the tip, enabling me to find the paper: H. Massalin, “Superoptimizer - A Look at the Smallest Program,” ACM SIGARCH Comput. Archit. News, pp. 122–126, 1987. https://web.stanford.edu/class/cs343/resources/superoptimize... The H is for "Henry".

It’s Alexia Massalin now: https://en.m.wikipedia.org/wiki/Alexia_Massalin

Re: Reverse engineering a neural network's clever solution to binary addition

#135

Post by monkpit »

teknopaul wrote: post #76
amelius wrote: post #33

Earlier quoted context omitted.

So that's how savants do it ...

There is one savant that can do crazy math but is otherwise normal. Afgter an epileptic fit. His system is base 10000, he can add any two numbers below 5000 and come up with a single digit response in one loop. Then covert to base10 for the rest of us. Each digit in his base 10000 system has a different visual representation, like we have 0-9. It's integer accurate so I don't think it uses sine wave approximations. G…

I was interested in reading more about this, but the only google result is your comment. Could you remember a name or any other details that might help find the story?

Re: Reverse engineering a neural network's clever solution to binary addition

#136

Post by swyx »

gnfargbl wrote: post #117
primis wrote: post #72

Earlier quoted context omitted.

Watts are actually a time independent measurement, note that the TWh has "hour" affixed to the end. This is 1 Tera Watt over the course of one hour, not one second. Your numbers are off by a factor of 3600. 1TWh / 20 Watt brain = 50,000,000,000 (50 Billion) Hours. 50 Billion Hours / (24h * 365.25) = 5,703,855.8 Years

Thanks for explaining the calculation! There is a huge error though, which is that I mis-read the units in the second link I posted. The actual power estimate for GPT-3 is more like 1 GWh (not 1TWh), so about 6000 years and not 6 million...!

lol, this is more like the other estimate’s ballprk… 1 TWh sounded way off and i was like “wait i gotta check this”

Re: Reverse engineering a neural network's clever solution to binary addition

#137

Post by abernard1 »

sweezyjeezy wrote: post #38
aargh_aargh wrote: post #13

The essay linked from the article is interesting: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

In the years since this came out I have shifted my opinion from 100% agreement to kind of the opposite in a lot of cases, a bitter lesson from using AI to solve complex end-to-end tasks. If you want to build a system that will get the best score on a test set - he's right, get all the data you possibly can, try to make your model as e2e as possible. This often has the advantage of being the easiest approach. The prob…

Your understanding that problem solving has to do with "steps" is the correct one in my opinion.

Also, with data driven approaches, the model isn't necessarily learned in any meaningful way. If you train with certain inputs to get certain realistic looking outputs, you can build sophisticated parrots or chameleons. That's what GPT and stable diffusion are: compressed knowledge bases to get an output without a knowledge model. (No, language models are not knowledge models).

Thinking, rational or otherwise, requires causal steps. Since none of these data driven approaches have even fuzzy causal models, they require memorizing infinite universes to see if search can find a particular universe that's seen this before. That's why they're not intelligent and never will be. An intelligent animal only needs one universe and limited experiences to solve a problem because it knows the latent structure of the problem and can generate approaches. Intelligent entities, unlike these autistic Rain Man automatons, do not need to memorize every book in the library to multiply two large numbers together.

Re: Reverse engineering a neural network's clever solution to binary addition

#138

Post by nine_k »

agapron1 wrote: post #11
greenbit wrote: post #3

That's awesome - the network's solution is essentially to convert the input to analog, perform the actual addition in analog , and then convert that back to digital. And the first two parts of that all happened in the input weights, no less.

An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results. Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.

A sequential one-bit adder, like in pocket calculators or earliest computers with delay-line memory, could likely be invented in the process.

Re: Reverse engineering a neural network's clever solution to binary addition

#139

Post by GeorgeTirebiter »

082349872349872 wrote: post #35
greenbit wrote: post #3

That's awesome - the network's solution is essentially to convert the input to analog, perform the actual addition in analog , and then convert that back to digital. And the first two parts of that all happened in the input weights, no less.

Sometimes when doing arithmetic with sed I use the unary representation, but usually it's better to use lookup tables: https://en.wikipedia.org/wiki/IBM_1620#Transferred_to_San_Jo...

If you want to have some fun, implement code to do arbitrary precision arithmetic the way the CADET did it. It's remarkably simple and reliable.

Re: Reverse engineering a neural network's clever solution to binary addition

#140

Post by GeorgeTirebiter »

Small nit: the analog DAC circuit has MSB and LSB reversed. That is, the LSB goes into the top 128k resistor. If you think of it one bit at a time, the gain of the circuit = -Rf/R --- where R is one of the resistors in the ladder. (Rf is the feedback resistor, which is unlabeled and un-valued in the article.) Clearly, you get bigger output when R is smaller.
Post reply on HN