Andrej Karpathy talks about how Tesla's NNs are structured and trained [video] • Hacker News Board

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#111

Post by konschubert » Sun, Nov 10, 2019, 10:01 AM UTC

jacquesm wrote: post #105

The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants t…

I think that there is still a need for some brilliant insights and breakthroughs, it isn't just a matter of getting the work done.

So actually, I think it's one of these situations where having a lot of independent efforts might be worth it.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#112

Post by arketyp » Sun, Nov 10, 2019, 10:04 AM UTC

jacquesm wrote: post #105

The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants t…

You can apply this line of reasoning on many markets, like the pharma or food industry which also have safety concerns. It strikes me as the kind initiatives EU attempts nowadays when realizing we are running behind on some tech and want to leverage the one possible advantage we have as a great centralizing power. Not too different from communist states, actually. I agree with the sentiment that redundancy seems wasteful, but it seems to me a necessary evil as a driving force in development, as with the right to private property in general.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#113

Post by jacquesm » Sun, Nov 10, 2019, 10:19 AM UTC

arketyp wrote: post #112

jacquesm wrote: post #105

The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants t…

You can apply this line of reasoning on many markets, like the pharma or food industry which also have safety concerns. It strikes me as the kind initiatives EU attempts nowadays when realizing we are running behind on some tech and want to leverage the one possible advantage we have as a great centralizing power. Not too different from communist states, actually. I agree with the sentiment that redundancy seems wast…

I'm reminded of the Manhattan project and I don't think they would have succeeded in their goal if they had tried to run 10 of those at once. There just aren't that many really great scientists in a space this narrow.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#114

Post by antpls » Sun, Nov 10, 2019, 10:23 AM UTC

modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Watch the presentation from 6 months ago, where they explain the decision to build their own hardware for inferring : https://youtu.be/Ucp0TTmvqOE?t=4309

It's not surprising that they also build the hardware for training. Correct me if I'm wrong, but Google use the same TPUs for training and inference, because the underlying operations are the same : multiply then add numbers. Once Tesla built the hardware for inferring, the design of the hardware for training is probably similar.

Unlike Google's TPUs, Tesla have a specific use case for the hardware (computer vision for automotive), and maybe than means they can further optimize the computation pipeline with their own specialized hardware.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#115

Post by jacquesm » Sun, Nov 10, 2019, 10:28 AM UTC

paraschopra wrote: post #108

jacquesm wrote: post #105

The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants t…

Standards shouldn't emerge too soon. I think for self driving tech, at the current stage, competition is good because there are lots of unsolved questions. Competition will ensure the best tech is ultimately available to consumers. Of course, it's not a binary choice. Things like data should probably be pooled but the use of data in tech should compete.

Ditto validation tech frameworks. If those are not standardized then people will not be able to make an informed choice about which solution is the safest other than to wait a decade and do a bodycount.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#116

Post by jacquesm » Sun, Nov 10, 2019, 10:29 AM UTC

roystonvassey wrote: post #99

modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Also, the software part of it (NNs and their algorithms) have been so widely researched and published that competitive advantages here are harder to come by than in hardware RD. Also, vendor lock-in is a huge challenge in the cloud space. I don’t think Tesla would be comfortable with the fact that all their training data sits on a potential competitor’s datacenter.

A car is a hardware device as well, and an electric car does not have the kind of power budget that allows you to throw oodles of standard pieces at it without paying a severe penalty in range.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#117

Post by antpls » Sun, Nov 10, 2019, 10:32 AM UTC

eanzenberg wrote: post #62

One thing I didn't quite understand is how training sub-graphs in parallel works. If you are editing a sub-graph of a monolith type model, aren't you affecting other graphs that have dependencies on the one you're editing? If these are independent graphs, then what's a "sub-graph" even mean?

First time I read about "sub-network" is in this AI Google blog post : https://ai.googleblog.com/2019/09/recursive-sketches-for-mod...

They talk about the concept of "modular network". The article itself links to the Wikipedia page : https://en.wikipedia.org/wiki/Modular_neural_network

Not sure it's exactly the same idea, but it looks similar.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#118

Post by atoav » Sun, Nov 10, 2019, 10:32 AM UTC

jacquesm wrote: post #105

The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants t…

There was a time in the medival ages where alchemists were kidnapped by kings and held in chambers so they would only generate knowledge for them. This obviously lead to a similar duplication to the one you describe, right up to calculus where Newton kept the thing hidden in a drawer and then Leibnitz had the same idea.

Once that kind of secrecy was gone our whole technical progress was accelerated, because people could build on the discoveries of other people.

Right now we are going back to the alchemist model in some ways (the highest profile people work for the big companies and don’t share their discoveries). This makes progress slower.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#119

Post by arketyp » Sun, Nov 10, 2019, 10:33 AM UTC

jacquesm wrote: post #113

arketyp wrote: post #112

Earlier quoted context omitted.

You can apply this line of reasoning on many markets, like the pharma or food industry which also have safety concerns. It strikes me as the kind initiatives EU attempts nowadays when realizing we are running behind on some tech and want to leverage the one possible advantage we have as a great centralizing power. Not too different from communist states, actually. I agree with the sentiment that redundancy seems wast…

I'm reminded of the Manhattan project and I don't think they would have succeeded in their goal if they had tried to run 10 of those at once. There just aren't that many really great scientists in a space this narrow.

Fair point. And the moon mission. Autonomous driving is such a consumerist issue though, seems quite well suited for a free market dynamic. I'd rather see a great joint effort on fusion energy or something.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#120

Post by modeless » Sun, Nov 10, 2019, 10:38 AM UTC

atoav wrote: post #118

jacquesm wrote: post #105

The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants t…

There was a time in the medival ages where alchemists were kidnapped by kings and held in chambers so they would only generate knowledge for them. This obviously lead to a similar duplication to the one you describe, right up to calculus where Newton kept the thing hidden in a drawer and then Leibnitz had the same idea. Once that kind of secrecy was gone our whole technical progress was accelerated, because people co…

> the highest profile people work for the big companies and don’t share their discoveries

I have to strongly disagree with this for the specific case of AI/ML. The big company labs are publishing open access papers non-stop, often with code and sometimes even datasets. They're more open than some areas of academia, in fact.