Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a neural network's clever solution to binary addition

cprimozic.net

Post reply on HN
Print view View original

141–150 of 160 posts

Re: Reverse engineering a neural network's clever solution to binary addition

#141

Post by sebzim4500 »

MontyCarloHall wrote: post #43

A couple questions: 1. How much of this outcome is due to the unusual (pseudo) periodic activation function? Seems like a lot of the DAC-like behavior is coming from the periodicity of the first layer’s output, which seems to be due to the unique activation function. 2. Would the behavior of the network change if the binary strings were encoded differently? The author encodes them as 1D arrays with 1 corresponding to…

1. The author has prior blog posts talking about this activation function, and apparently it does help to learn binary logic tasks.

2. I doubt this matters at all here. For some architectures having inputs be 0 on average is useful, so the author probably just picked it as the default choice.

Re: Reverse engineering a neural network's clever solution to binary addition

#142

Post by IIsi50MHz »

TeMPOraL wrote: post #94
dsign wrote: post #59

Earlier quoted context omitted.

Thanks for those numbers! They are frankly scary. Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. With that said, I think we should apply ML and the results of the bitter lesson to things which are more intractable than searching the web or pla…

> Six millions years of power for a single brain is one year of power for six million brains. The knowledge that ChatGPT contains is far higher than the knowledge a random sample of six million people have produced in a year. On the other hand, I don't think if it would be larger than that of six million people selected more carefully (though I suppose you'd have to include the costs of selection process in the tally…

> On the other hand, > … > On the other other hand,

See: "on the gripping hand" (-:

Re: Reverse engineering a neural network's clever solution to binary addition

#145

Post by eru »

adwn wrote: post #74
eru wrote: post #49

Earlier quoted context omitted.

Are you doing the equivalent of repeated squaring here? Otherwise, you'd need up to 255 (or so) additions to multiply two 8 bit numbers, I think?

An 8x8 bit multiplication only requires 7 additions, either in parallel or sequentially. Remember long-form multiplication? [1] It's the same principle. Of course, high-speed digital multiplication circuits use a much more optimized, much more complex implementation. [1] https://en.wikipedia.org/wiki/Multiplication_algorithm#Examp...

OK. It sounded to me like the comment I originally replied to (https://news.ycombinator.com/item?id=34400296) suggested to implement multiplication as naive repeated addition.

Re: Reverse engineering a neural network's clever solution to binary addition

#146

Post by westurner »

shagie wrote: post #132
djmips wrote: post #129

Earlier quoted context omitted.

There are a few startups making analog ML compute. Mythic & Aspinity for example https://www.eetimes.com/aspinity-puts-neural-networks-back-t...

Veritasium has a bit of an video on Mythic - Future Computers Will Be Radically Different (Analog Computing) - https://youtu.be/GVsUOuSjvcg

From "Faraday and Babbage: Semiconductors and Computing in 1833" https://news.ycombinator.com/item?id=32888210 and then "Qubit: Quantum register: Qudits and qutrits" https://news.ycombinator.com/item?id=31983110:

>>> The following is an incomplete list of physical implementations of qubits, and the choices of basis are by convention only: [...] Qubit#Physical_implementations: https://en.wikipedia.org/wiki/Qubit#Physical_implementations

> - note the "electrons" row of the table

According to this Table on wikipedia, it's possible to use electron charge (instead of 'spin') to do Quantum Logic with Qubits.

How is that doing quantum logical computations with electron charge different from from what e.g. Cirq or Tequila do (optionally with simulated noise to simulate the Quantum Computer Engineering hardware)?

FWIU, analog and digital component qualities are not within sufficient tolerance to do precise analog computation? (Though that's probably debatable for certain applications at least, but not for general purpose computing architectures?) That is, while you can build adders out of voltage potentials quantified more specifically than 0 or 1, you might shouldn't without sufficient component spec tolerances because noise and thus error.

IMHO, Turing Tumble and Spintronics are neat analog computer games.

(Are Qubits, by Church-Turing-Deutsch, sufficient to; 1) simuluate arbitrary quantum physical systems; or 2) run quantum logical simulations as circuits with low error due to high coherence? https://en.wikipedia.org/wiki/Church%E2%80%93Turing%E2%80%93... )

>> See also: "Quantum logic gate" https://en.wikipedia.org/wiki/Quantum_logic_gate

Analog computers > Electronic analog computers aren't Electronic digital computers: https://en.wikipedia.org/wiki/Analog_computer#Electronic_ana...

Re: Reverse engineering a neural network's clever solution to binary addition

#147

Post by dilippkumar »

kens wrote: post #104

The trick of performing binary addition by using analog voltages was used in the IAS family of computers (1952), designed by John von Neumann. It implemented a full adder by converting two input bits and a carry in bit into voltages that were summed. Vacuum tubes converted the analog voltage back into bits by using a threshold to generate the carry-out and more complex thresholds to generate the sum-out bit.

> designed by John von Neumann.

I can strongly recommend an excellent biography:

The Man from the Future: The Visionary Life of John von Neumann by Ananyo Bhattacharya

I knew of von Neumann because his name shows up many many times when studying computers. But I had no idea he had several equally monumental bodies of work in such a wide range of subjects.

It’s that rare biography that helps understand the history of multiple different disciplines: quantum mechanics, game theory, computer science and more.

Re: Reverse engineering a neural network's clever solution to binary addition

#148

Post by barrenko »

greenbit wrote: post #3

That's awesome - the network's solution is essentially to convert the input to analog, perform the actual addition in analog , and then convert that back to digital. And the first two parts of that all happened in the input weights, no less.

Can someone please explain, I don't get here "converted the input to analog", converted how, where?

Re: Reverse engineering a neural network's clever solution to binary addition

#149

Post by gnfargbl »

swyx wrote: post #136
gnfargbl wrote: post #117

Earlier quoted context omitted.

Thanks for explaining the calculation! There is a huge error though, which is that I mis-read the units in the second link I posted. The actual power estimate for GPT-3 is more like 1 GWh (not 1TWh), so about 6000 years and not 6 million...!

lol, this is more like the other estimate’s ballprk… 1 TWh sounded way off and i was like “wait i gotta check this”

Yeah, the other thing that I should have noticed was the completely unreasonable cost of 1 TWh of energy. If 1 kWh cost about $0.15 at the time they trained the model, 1TWh would have been like $150M. Probably twice that after accounting for cooling. Doh.

Re: Reverse engineering a neural network's clever solution to binary addition

#150

Post by imtringued »

I don't see how this is surprising. It's cool but come on, you have a network consisting of addition and multiplication and you expect it to not use them? What else is it supposed to do? Memorize every input output pair like a look up table?
Post reply on HN