Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse engineering a neural network's clever solution to binary addition

cprimozic.net

Post reply on HN
Print view View original

151–160 of 160 posts

Re: Reverse engineering a neural network's clever solution to binary addition

#151

Post by imtringued »

TeMPOraL wrote: post #90
vletal wrote: post #58

Earlier quoted context omitted.

Why it works here is that the "analog" representation is not influenced by noise, because it's simulated on digital hardware. On the other hand, why we use digital hardware precisely because it's robust against the noise present in the hardware analog circuits.

So in other words, this is analog computation on a digital hardware on analog substrate - the analog-to-digital step eliminates the noise of our physical reality, and the subsequent digital-to-analog reintroduces a certain flexibility of design thinking. I wonder, is there ever a case analog-on-digital is better to work with as an abstraction layer, or is it always easier to work with digital signals directly?

Sorry but those signals aren't analog. Floating point numbers are still digital.

Re: Reverse engineering a neural network's clever solution to binary addition

#152

Post by alltheworlds »

jccalhoun wrote: post #41

These stories remind me of a story from Discover Magazine https://www.discovermagazine.com/technology/evolving-a-consc... A researcher was using a process to "evolve" a FPGA and the result was a circuit that was super efficient but worked in ways that were unexpected: part of the circuit seemed unconnected to the rest but if removed the whole thing stopped working and it would only work at a specific temperature.

I believe I remember reading about that in the book "The Emperor's New Mind" by Roger Penrose back in the late 90s. It was something that has stuck with me ever since, even though I didn't follow up on the AI route.

Re: Reverse engineering a neural network's clever solution to binary addition

#153

Post by crapaud23 »

djmips wrote: post #129
elcritch wrote: post #128

Earlier quoted context omitted.

Nifty, makes one wonder if logarithmic or sigmoid functions for ML could be done using this method. Especially as we approach the node size limit, perhaps dealing with fuzzy analog will become more valuable.

There are a few startups making analog ML compute. Mythic & Aspinity for example https://www.eetimes.com/aspinity-puts-neural-networks-back-t...

Analog pid controllers: https://www.youtube.com/watch?v=Ps9iD738rUg Soviet IR tracking missile heads too. Stumbled upon one on ebay once, was surprised the control system was analog. Not sure which one it was and I'm very surprised to find a Raytheon AIM 9 Sidewinder missile kit there: https://www.ebay.com/itm/273992271867

Re: Reverse engineering a neural network's clever solution to binary addition

#154

Post by lostmsu »

lgas wrote: post #125
lostmsu wrote: post #30

Earlier quoted context omitted.

Humans are not trianed on words.

What do you mean? That we are not trained ONLY on words? Or that the training we receive on words is not the same as the training of a NN? Or something else?

We are not trained only on words.

In fact, we are not trained on words at all. We are trained on images, sounds, tactile, and a few more types of sources.

Re: Reverse engineering a neural network's clever solution to binary addition

#155

Post by gyorgytamas »

torginus wrote: post #16
agapron1 wrote: post #11

Earlier quoted context omitted.

An 8 bit adder is too small and allows such 'hack'. He should try training a 32 or 64 bit adder, decrease the weights accuracy to bfloat16, introduce dropout regularization (or other kind of noise) to get more 'interesting' results. Addendum: another interesting variation to try is a small transformer network, and feeding the bits sequentially as symbols. This kind of architecture could compute bignum-sized integers.

A 32-bit adder is just 4 8-bit adders with carry connected. I don't see why it'd be significantly more difficult.

[flagged]

Re: Reverse engineering a neural network's clever solution to binary addition

#156

Post by blurker »

kordlessagain wrote: post #70
jccalhoun wrote: post #41

These stories remind me of a story from Discover Magazine https://www.discovermagazine.com/technology/evolving-a-consc... A researcher was using a process to "evolve" a FPGA and the result was a circuit that was super efficient but worked in ways that were unexpected: part of the circuit seemed unconnected to the rest but if removed the whole thing stopped working and it would only work at a specific temperature.

Yeah, IIRC it grew an antenna that was utilizing the clock in the computer it was sitting next to.

You're referring to [this](https://www.researchgate.net/publication/3949367_The_evolved...)

Which was also really cool!

> As with Thompson's FPGA exploiting some subtle physical properties of the device, Layzell found that evolved circuits could rely on external factors. For example, whilst trying to evolve an oscillator Bird and Layzell discovered that evolution was using part of the circuit for a radio antenna, and picking up emissions from the environment [18]. Layzell also found that evolved circuits were sensitive to whether or not a soldering iron was plugged in (not even switched on) in another part of the room[19]. ...

However, OP was actually referring to an experiment by Dr Adrian Thompson which was different but also sort of similar. The FPGA evolved by Thompson ended up depending on parts of the circuit that were disconnected from the main circuit but still affected its operation. It probably relied on electromagnetic properties, so was sort of a radio, but it did not rely on the clock of a nearby computer.

[damninteresting.com did a really interesting writeup about this](https://www.damninteresting.com/on-the-origin-of-circuits/)

Both were really cool, unexpected behaviors of evolved hardware systems.

Re: Reverse engineering a neural network's clever solution to binary addition

#157

Post by kordlessagain »

blurker wrote: post #156
kordlessagain wrote: post #70

Earlier quoted context omitted.

Yeah, IIRC it grew an antenna that was utilizing the clock in the computer it was sitting next to.

You're referring to [this]( https://www.researchgate.net/publication/3949367_The_evolved... ) Which was also really cool! > As with Thompson's FPGA exploiting some subtle physical properties of the device, Layzell found that evolved circuits could rely on external factors. For example, whilst trying to evolve an oscillator Bird and Layzell discovered that evolution was using part of the circuit for a radio antenna, a…

That's right! Thanks for tracking it down...

Re: Reverse engineering a neural network's clever solution to binary addition

#158

Post by glitchc »

kens wrote: post #104

The trick of performing binary addition by using analog voltages was used in the IAS family of computers (1952), designed by John von Neumann. It implemented a full adder by converting two input bits and a carry in bit into voltages that were summed. Vacuum tubes converted the analog voltage back into bits by using a threshold to generate the carry-out and more complex thresholds to generate the sum-out bit.

There's a limit to this process. At a certain number of bits, the sum of voltages will be too high to represent safely. Yet adding more bits on a digital computer is trivial.

Re: Reverse engineering a neural network's clever solution to binary addition

#159

Post by stabbles »

canadianfella wrote: post #126
stabbles wrote: post #124

Earlier quoted context omitted.

It's better to say the NN "converged to" something than "came up with" something.

Why?

There is no intelligence or reasoning involved in stochastic gradient descent

Re: Reverse engineering a neural network's clever solution to binary addition

#160

Post by lgas »

lostmsu wrote: post #154
lgas wrote: post #125

Earlier quoted context omitted.

What do you mean? That we are not trained ONLY on words? Or that the training we receive on words is not the same as the training of a NN? Or something else?

We are not trained only on words. In fact, we are not trained on words at all. We are trained on images, sounds, tactile, and a few more types of sources.

Seems odd to say we're trained on sounds but not words, especially when many of the sounds we hear while we are developing are words being spoken.
Post reply on HN