Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

2 times 3 can sometimes equal 7 with Android's Neural Network API

alexanderganderson.github.io

Post reply on HN
Print view View original

61–70 of 92 posts

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#61

Post by cercatrova »

HotVector wrote: post #55
Avalaxy wrote: post #7

Using a neural network for things that have clear cut rules is wrong. When you know the exact rules, implement them as such, instead of bruteforcing a guesstimation. This is also why I'm sceptical of the usr of GPT-3 for all sorts of purposes where accuracy is important. Think of the code generation case. Bugs may be very subtle and may go unnoticed.

How is it that human brains can follow deterministic logic but neural networks can't? What's the missing piece? Is it just that people are a lot more complex than algorithms like GPT-3?

This is a very interesting question that I've been thinking of as well. Why can't neural network based AI learn deterministic logic? I think because it doesn't have internal representations inside its "head" so to speak. It does not know that it can manipulate such structures as we can.

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#62

Post by Agentlien »

Avalaxy wrote: post #7

Using a neural network for things that have clear cut rules is wrong. When you know the exact rules, implement them as such, instead of bruteforcing a guesstimation. This is also why I'm sceptical of the usr of GPT-3 for all sorts of purposes where accuracy is important. Think of the code generation case. Bugs may be very subtle and may go unnoticed.

I once (2010) had an oral exam in neural networks during which I had to design a system to solve a particular task. My solution used two neural networks connected by a simple logic circuit. The professor, who was actually a neuro scientist with little understanding of the technical, asked why not a third neural network. This was a neural network course, after all.

Thinking this was a trick question I excitedly explained how stupid it would be to build and train a network to approximate a function which could easily be precisely described with a tiny circuit or code statement.

The professor was not amused. He said that's what he would have done. After a few similar incidents he concluded the exam giving me an 8/10 saying my answers were perfectly correct but he didn't like my attitude.

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#63

Post by tsimionescu »

HotVector wrote: post #55
Avalaxy wrote: post #7

Using a neural network for things that have clear cut rules is wrong. When you know the exact rules, implement them as such, instead of bruteforcing a guesstimation. This is also why I'm sceptical of the usr of GPT-3 for all sorts of purposes where accuracy is important. Think of the code generation case. Bugs may be very subtle and may go unnoticed.

How is it that human brains can follow deterministic logic but neural networks can't? What's the missing piece? Is it just that people are a lot more complex than algorithms like GPT-3?

That's essentially like asking "how is it that human feet can walk up mountain slopes, but car wheels can't?"

That is, there is no relation between human brains and artificial neural networks, other than them serving similar purposes in particular environments.

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#64

Post by p1esk »

shoyer wrote: post #56
aga_ml wrote: post #47

Earlier quoted context omitted.

Thanks for your comments. Regarding determinism, potentially a fair point. Here are a few comments: (1) A driver which randomly produces different output when running the network would be valid according to these restrictions. (2) It is conceivable that a driver would produce non-deterministic input with the same hardware. One commonly known example is that tensorflow will run multiple different convolution kernels a…

All good points! "Non-deterministic" behavior within the same program/process is still a bridge I would not want to cross. This could result in subtle glitches, e.g., when a user hits "refresh" with the same inputs, and could make reproducing bugs impossible. I am a strong believer in always using a seed for random number generation for exactly these sorts of reasons. (Side note: deterministic RNGs is one of my favor…

You are paying with performance for the determinism. Any DL framework can be made deterministic (just add few lines of configuration), not just JAX.

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#65

Post by anonytrary »

My pencil can sometimes be in China according to quantum mechanics, but the probability is extremely low. I think the fact that neural networks are almost right is not really concerning at all. As long as your network can produce a result within an acceptable error boundary, who cares? That is literally how nature works.

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#67

Post by visarga »

Avalaxy wrote: post #7

Using a neural network for things that have clear cut rules is wrong. When you know the exact rules, implement them as such, instead of bruteforcing a guesstimation. This is also why I'm sceptical of the usr of GPT-3 for all sorts of purposes where accuracy is important. Think of the code generation case. Bugs may be very subtle and may go unnoticed.

GPT-3 is a generator, that's just half the equation. It needs a discriminator (critic) to check out its outputs and in the case of program synthesis and realistic physics, a simulator. That's how humans do creative stuff - generate silly ideas then check them out.

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#68

Post by hyko »

HotVector wrote: post #55
Avalaxy wrote: post #7

Using a neural network for things that have clear cut rules is wrong. When you know the exact rules, implement them as such, instead of bruteforcing a guesstimation. This is also why I'm sceptical of the usr of GPT-3 for all sorts of purposes where accuracy is important. Think of the code generation case. Bugs may be very subtle and may go unnoticed.

How is it that human brains can follow deterministic logic but neural networks can't? What's the missing piece? Is it just that people are a lot more complex than algorithms like GPT-3?

Human brains make logic mistakes all the time.

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#69

Post by visarga »

nmfisher wrote: post #53
dealforager wrote: post #10

Earlier quoted context omitted.

For code, I could see it being super useful for a beefed up auto-complete. There are many times I find myself searching for things like "how do I do X in Y language" to copy a snippet that I'm sure has been written 10000x times before. I can review the code and verify its correctness by writing tests.

Either OpenAI or Microsoft demoed something similar to this some time in theist 12-18 months. The fact that it’s not released (and also the fact that GPT-3 etc are still not publicly available) makes me suspect that these models are far too unstable for actual production use. It’s also why I’m getting a bit tired of these overhyped cherry-picked samples with seemingly nothing solid to ever back it up.

They are too expensive to run - hundreds of GB of GPU memory - so they can't be deployed for the public at large yet, kind of like the SGI workstations from 20 years ago. You can do that and more for cheap today, but not then.

I think we can get models about 1/100th the size for general use. That's also the main reason Google is developing TPUs.

Re: 2 times 3 can sometimes equal 7 with Android's Neural Network API

#70

Post by ZephyrBlu »

visarga wrote: post #67
Avalaxy wrote: post #7

Using a neural network for things that have clear cut rules is wrong. When you know the exact rules, implement them as such, instead of bruteforcing a guesstimation. This is also why I'm sceptical of the usr of GPT-3 for all sorts of purposes where accuracy is important. Think of the code generation case. Bugs may be very subtle and may go unnoticed.

GPT-3 is a generator, that's just half the equation. It needs a discriminator (critic) to check out its outputs and in the case of program synthesis and realistic physics, a simulator. That's how humans do creative stuff - generate silly ideas then check them out.

Sounds like maybe a GAN-like ML algo could be a good choice for attempting to do code generation.
Post reply on HN