Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

youtube.com

Post reply on HN
Print view View original

91–100 of 249 posts

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#91

Post by xiphias2 »

kayoone wrote: post #82
xiphias2 wrote: post #75

Earlier quoted context omitted.

Andrej Karpathy just started working on Tesla's software 2 years ago, before what Chris Lattner did was a mess (he wanted to just have 1 task that learns magically everything), Andrej had to start everything from scratch. Waymo had a 20 year advantage, but Google lost many key people there in the meantime as Larry Page didn't want to launch partial self driving. I think both approaches are great and I wouldn't want t…

> before what Chris Lattner did was a mess (he wanted to just have 1 task that learns magically everything), Andrej had to start everything from scratch any source for that?

Sorry, I can't find the source, I'm reading Elektrek all the time and often there are people in the comments section with knowledge about what happens inside Tesla.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#92

Post by vintermann »

m0zg wrote: post #79
modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs. Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would…

> Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google

Can it be true? Then again, Apple's app store behaviour seems to suggest such demands are tolerated. Antitrust is really asleep in the US, isn't it.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#93

Post by m0zg »

modeless wrote: post #88
m0zg wrote: post #79

Earlier quoted context omitted.

Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs. Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would…

> Google TPU TOS prohibits the use of TPUs for stuff that competes with Google I don't think this is true. If you're talking about https://news.ycombinator.com/item?id=19855099 , it doesn't apply to TPU hardware, as is explained in the comments there. > TPU-like stuff is ~10x the energy efficiency of GPUs 10x is probably overstating it when talking about newer GPUs because they have ML hardware in them now. Also, tha…

Yep that's the clause. The clause itself is not that problematic for Tesla. What's problematic is that it can be changed over time, and it'd be foolish to single-source something as important as deep learning compute without the option to go elsewhere. Not to mention the rather extravagant Cloud pricing. So Tesla is taking a page out of Steve Jobs' playbook and it will control its own core tech. That's smart, especially considering that they already have bits and pieces of the IP that they'll need.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#94

Post by spectramax »

visarga wrote: post #77
spectramax wrote: post #5

Earlier quoted context omitted.

Without meaning offense to Sam, I thought he was an investor / YC head. What credentials does he have to be at OpenAI?

Teaching the CS231n course at Stanford, for one.

Thanks, that makes sense. So now the question is - how did Sam become a YC/Angel investor from someone who taught a class at Stanford? I think we need an interview with Sam.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#95

Post by joenathanone »

modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

>Also, I'm really curious whether the custom hardware in the cars is benefiting them at all yet. Every feature they've released so far works fine on the previous generation hardware with 1/10 the compute power.

The latest OTA finally brings a hardware v3 only feature, traffic cone visualization, and traffic cone automatic lane change.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#96

Post by modeless »

m0zg wrote: post #93
modeless wrote: post #88

Earlier quoted context omitted.

> Google TPU TOS prohibits the use of TPUs for stuff that competes with Google I don't think this is true. If you're talking about https://news.ycombinator.com/item?id=19855099 , it doesn't apply to TPU hardware, as is explained in the comments there. > TPU-like stuff is ~10x the energy efficiency of GPUs 10x is probably overstating it when talking about newer GPUs because they have ML hardware in them now. Also, tha…

Yep that's the clause. The clause itself is not that problematic for Tesla. What's problematic is that it can be changed over time, and it'd be foolish to single-source something as important as deep learning compute without the option to go elsewhere. Not to mention the rather extravagant Cloud pricing. So Tesla is taking a page out of Steve Jobs' playbook and it will control its own core tech. That's smart, especia…

That clause doesn't apply at all though. It's not even for the same product.

As for single-source, the models are written in PyTorch, not TPU machine code, and they're pretty standard models anyway (e.g. Resnet-50). They can easily transfer to other hardware if necessary. There's not a ton of lock-in there. It doesn't justify the massive costs of ASIC development just to avoid this nonexistent lock-in and imaginary TOS clause.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#97

Post by sgt101 »

m0zg wrote: post #79
modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs. Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would…

>Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs.

could you share the stats on this? Google told me to use a K-80 for training.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#98

Post by newnewpdro »

natch wrote: post #64
newnewpdro wrote: post #25

Earlier quoted context omitted.

Same. Watching the AI visualization of summons in-action was horrifying, and made clear why many have reported summons mode as resembling a drunk person navigating a parking lot.

Yes it's not perfected yet, which is why it requires human supervision for now. Having it operate in the wild as it is now (again, under human supervision) will help it become less horrifying, which I think you would agree is what we want.

I don't agree, as I don't particularly value/want a future with self-driving vehicles.

I also think the way Tesla is going about this is utterly idiotic and reckless. It upsets me that I'm sharing roads with vehicles having these dysfunctional immature systems.

Point this crap at video game engines and don't let it anywhere near real people until it can drive millions of virtual miles in something like GTA without hitting anyone/anything and without behaving like a drunk driver who lost their glasses.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#99

Post by roystonvassey »

modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Also, the software part of it (NNs and their algorithms) have been so widely researched and published that competitive advantages here are harder to come by than in hardware RD.

Also, vendor lock-in is a huge challenge in the cloud space. I don’t think Tesla would be comfortable with the fact that all their training data sits on a potential competitor’s datacenter.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#100

Post by drchewbacca »

throwaway010718 wrote: post #15
timzaman wrote: post #11

His team is hiring; https://www.tesla.com/careers/job/software-engineerdeeplearn... https://www.tesla.com/careers/job/machine-learninginfrastruc... https://www.tesla.com/careers/job/machine-learningscientista...

Any guess what the compensation is like for these positions ?

If you get rejected just edit 10 words on your resume and then resubmit. Do this a 100hz and you'll get in eventually :)
Post reply on HN