Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

youtube.com

Post reply on HN
Print view View original

71–80 of 249 posts

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#71

Post by sdan »

natch wrote: post #56
sdan wrote: post #12

Really liked this talk. Looks like they are really nicely orchestrating workloads and training on numerous nets asynchronously. As a person in the AV industry I think Tesla's ability to control the entire stack is great for Tesla... maybe not for everyone who can't afford/doesn't have a Tesla.

>maybe not for everyone who can't afford/doesn't have a Tesla. Affordability is not as much of an issue as some make it out to be. Cost-wise it's like owning a Camry or an Accord, if you go for the lower end models. If you mean not everyone can afford a new car, then sure I agree with you. Edit: if you think I'm wrong about this, please explain or ask me to clarify anything?

As a small anecdote, my parents couldn't afford/didn't want to spent over $30k for a car. Surely we could've gotten a Tesla for $5k+ more, but given the relatively new infrastructure with electric charging stations (and the fact that none are available in the apartment I live in) my parents didn't find all the new cool features appealing and instead got a regular Toyota Sienna that has nothing fancy, just enough to take the family around.

Similarly, the infrastructure around electric charging stations I believe hasn't fully matured yet and as a result many people who've already owned a car, I believe will stick with gas cars since there's no huge incentive to change, unless it becomes easier to charge (faster, more convenient).

Do note that I don't have a drivers license. I never intend on getting one (I believe in what I do in the AV industry). I'm just guessing on the habits of people, not that I have any real experience in buying a gas/electric car.

Also note I didn't think you were wrong, not sure why the downvotes.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#72

Post by ojn »

adamnemecek wrote: post #7

The trick for level 5 is learning the mapping between the lidar point cloud and the video stream. It’s the best of both worlds.

That falls apart as soon as the map and the real world deviates and you need to drive based on what’s in front of you.

Lidar helps you spot obstructions, but won’t tell you what they are and won’t help you figure out what to do to avoid them.

Want an example? Cruise’s first real world demo got stuck behind a simple taco truck in downtown SF.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#73

Post by sdan »

modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Did they say they were building their own training hardware? I thought it was just their inference hardware (the boards on the teslas)?

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#74

Post by modeless »

sdan wrote: post #73
modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Did they say they were building their own training hardware? I thought it was just their inference hardware (the boards on the teslas)?

Yes https://youtu.be/oBklltKXtDE?t=572

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#75

Post by xiphias2 »

adrr wrote: post #66
grecy wrote: post #36

Earlier quoted context omitted.

Do you belittle everyone that gets second place in the Olympics because the winner is "way ahead"? Your comment just reeks of anger and hostility. It seems like you'd rather Tesla didn't try at all, and instead we all just give up and go back to the status quo.

Elon belittles lidar saying it is doomed and will never work yet Waymo and Cruise will probably be operating self driving taxi fleets in California next year. Tesla deserves getting dumped on for those comments because they are no where near self driving.

Andrej Karpathy just started working on Tesla's software 2 years ago, before what Chris Lattner did was a mess (he wanted to just have 1 task that learns magically everything), Andrej had to start everything from scratch.

Waymo had a 20 year advantage, but Google lost many key people there in the meantime as Larry Page didn't want to launch partial self driving.

I think both approaches are great and I wouldn't want to choose between the 2, just be a happy user of the end result of the competition.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#76

Post by jeffshek »

kegan wrote: post #16

Anyone knows why Andrej's team chooses PyTorch (as oppose to say TensorFlow?)

Some potential reasons:

- TensorFlow is great at deployment, but not the easiest to code. PyTorch isn't frequently used in production until recently.

- If you have the resources for great AI engineers and researchers, your team will be good enough to build and deploy both frameworks.

- Preference toward the easier framework your tech leads prefer.

- Lots of new academic research is coming in PyTorch

- TensorFlow is undergoing a massive change from 1.1x to 2.0; if you choose TensorFlow, write on 1.1x just to then refactor to TF 2.0? Or write on TF 2.0 now and deal with all new edge cases? Or write in PyTorch (easier) but handle the more difficult deployment process.

- ML code quickly rots. Bad PyTorch code is just bad Python code. Bad TensorFlow code can be a nightmare to debug.

- PyTorch's eager execution makes coding NNs much easier to prototype and build.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#77

Post by visarga »

spectramax wrote: post #5
mkagenius wrote: post #2

Oh he's no longer with OpenAI? Sam Altman must be worried about this..

Without meaning offense to Sam, I thought he was an investor / YC head. What credentials does he have to be at OpenAI?

Teaching the CS231n course at Stanford, for one.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#78

Post by meheleventyone »

ars wrote: post #60
soulslicer0 wrote: post #18

Earlier quoted context omitted.

lol i just did the interview and failed. had to find shortest path between tesla chargers. all in C++. completed it but failed

How does that work? Where you given the algorithm to use? Or was it really a data science question, rather than a programming question?

Without more details it sounds like an algorithm problem you'd be expected to solve from prior knowledge. Stuff like a breadth first search from the start point, up through various path finding algorithms to applying heuristics (I believe route finding on roads exploits the road topology).

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#79

Post by m0zg »

modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Nothing crazy about it. TPU-like stuff is ~10x the energy efficiency of GPUs and several times the speed. When you're spending megawatt-hours and days to train a single model, it adds up in both real and opportunity costs.

Also, Google TPU TOS prohibits the use of TPUs for stuff that competes with Google (and I'm assuming with other companies under Alphabet umbrella), at Google's sole determination. Not that it would be a good idea to upload Tesla's proprietary data into Google Cloud even if it did not. Cloud, after all, is just somebody else's computer.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#80

Post by m0zg »

kegan wrote: post #16

Anyone knows why Andrej's team chooses PyTorch (as oppose to say TensorFlow?)

Because PyTorch literally triples researcher productivity. Imagine a deep learning framework which you can actually debug when something goes wrong and which you don't have to fight every step of the way to do even simple things. That's PyTorch.
Post reply on HN