Meanwhile Waymo is way ahead.
Except, well, Waymo doesn't actually build cars, and has no plans to do so.
Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
81–90 of 249 posts
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#82Earlier quoted context omitted.
Elon belittles lidar saying it is doomed and will never work yet Waymo and Cruise will probably be operating self driving taxi fleets in California next year. Tesla deserves getting dumped on for those comments because they are no where near self driving.
Andrej Karpathy just started working on Tesla's software 2 years ago, before what Chris Lattner did was a mess (he wanted to just have 1 task that learns magically everything), Andrej had to start everything from scratch. Waymo had a 20 year advantage, but Google lost many key people there in the meantime as Larry Page didn't want to launch partial self driving. I think both approaches are great and I wouldn't want t…
> before what Chris Lattner did was a mess (he wanted to just have 1 task that learns magically everything), Andrej had to start everything from scratch
any source for that?
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#83Earlier quoted context omitted.
Did they say they were building their own training hardware? I thought it was just their inference hardware (the boards on the teslas)?
Yes https://youtu.be/oBklltKXtDE?t=572
Oops, didn't catch that. For some reason I thought it was a orchestration engine like Stripe's Railyard.
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#84Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…
Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs. Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would…
Stuff like their TPU and Waymo's Honeycomb Laserbear (something along those lines... their lidar naming system is pretty long) shows that Google is making good products for a limited reach of people.
TPU? Seems like it has a lot of potential, but not for people directly competing with them.
Waymo's Laserbear lidar? Seems like it has a lot of potential, but not for AV companies directly competing with them.
Google's playing this game pretty fiercely... which given their size is pretty bad/daunting.
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#85Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…
Stay ahead? Are they actually ahead?
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#86Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…
At Tesla's scale and priorities, they'd probably be less keen on using external cloud providers. Using TPUs at their scale would certainly require Google's AI consultants to supervise which isn't ideal for Tesla.
Not agreeing or disagreeing with their decisions, but if you have the resources, you can certainly design a custom chip that performs a specific type of task very well that beats other competitors. Nvidia's GPUs are have to be reasonably good at training across different NNs. You could have a chip that's exceptional good at training one/two specific types of tasks.
For most companies, this would be a bad idea. However, Tesla knows how to produce hardware.
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#87Earlier quoted context omitted.
Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs. Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would…
Stuff like their TPU and Waymo's Honeycomb Laserbear (something along those lines... their lidar naming system is pretty long) shows that Google is making good products for a limited reach of people. TPU? Seems like it has a lot of potential, but not for people directly competing with them. Waymo's Laserbear lidar? Seems like it has a lot of potential, but not for AV companies directly competing with them. Google's p…
[deleted]
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#88Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…
Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs. Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would…
> Google TPU TOS prohibits the use of TPUs for stuff that competes with Google
I don't think this is true. If you're talking about https://news.ycombinator.com/item?id=19855099, it doesn't apply to TPU hardware, as is explained in the comments there.
> TPU-like stuff is ~10x the energy efficiency of GPUs
10x is probably overstating it when talking about newer GPUs because they have ML hardware in them now. Also, that still doesn't make it a good idea to build your own chips because there will soon be many third party options to choose from. Doing your own chips is a bet that you will out execute dozens of companies ranging from startups to industry giants. Simply taking your pick of the best commercially available options is likely to be a better choice in the near future.
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#89Just listening to this talk scares me. The amount of errors - even in a seemingly normal, sunny day - is mind boggling to think people trust this crap. How can we rely on the output of eight cameras? This is not a kid's science project. It's all fancy neural networks until someone dies. Pretty callous and Silicon valley-mindset for such an important and critical function of the car. Will never buy a Tesla after havin…
> mind boggling to think people trust this crap It's also mind boggling to think we currently trust organic tissue to do this crap, some of which is bathed in psychoactive chemicals. And yet we do, and as a result, horrendous catastrophes occur every minute of every day. > It's all fancy neural networks until someone dies No, that can't be the standard, not when people are dying right now in the current regime. Unles…
No, it needs to be a lower rate since it will kill at random. Today’s rate includes drunk drivers, people on their phone and other “unsafe” drivers. If you are an attentive driver your chance of death would actually go up if the overall death rate was the same.
Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]
#90Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…
At Tesla's scale and priorities, they'd probably be less keen on using external cloud providers. Using TPUs at their scale would certainly require Google's AI consultants to supervise which isn't ideal for Tesla. Not agreeing or disagreeing with their decisions, but if you have the resources, you can certainly design a custom chip that performs a specific type of task very well that beats other competitors. Nvidia's…
> At Tesla's scale and priorities, they'd probably be less keen on using external cloud providers.
By hardware-hours, Tesla is hardly one of the top companies training deep networks. Planet Labs (satellite imaging), Netflix, Pornhub, to name a few.
What's the info they'd leak to the Google consultants? How much data or TPUs they're using? This is practically public information.