Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

youtube.com

Post reply on HN
Print view View original

221–230 of 249 posts

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#221

Post by codeulike »

mrep wrote: post #163
codeulike wrote: post #136

Earlier quoted context omitted.

Compared to the energy needed to move the car, everything else is pretty irrelevant. Power hungry features like Heating/AC only makes a few % difference to range.

From autonomy day hacker news comment: "Pegasus consumes about 500Watts, compared to under 100 Watts for Tesla's FSD computer. Elon in particular emphasized the performance per watt (as it's always possible to cram more chips to increase performance if you ignore cost and power consumption). The comparison made in the video: 500Watts for an hour consumes about 2-3 miles of range. In a city in slow traffic, going 12mp…

Wow, I stand corrected

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#222

Post by dna_polymerase »

anonu wrote: post #177
londons_explore wrote: post #176

I'm still amazed that Teslas team isn't using a map... I know maps get outdated and are sometimes wrong, but having inaccurate knowledge of what's around the corner is far far more helpful than not having any clue whats around the corner. The smart solution would be to consider a map a probabilistic thing, which neural networks are really good at handling.

I'm still amazed Tesla has decided not to use lidar and instead just stick with cheap cameras. Better sensors are there, they're available, they're cheap and they can probably "see" better than plain old cameras... it doesn't make too much sense not to use them IMHO. But then again, I am not coding NNs for Tesla...

I recommend watching George Hotz's take on this: https://www.youtube.com/watch?v=IxuU5L2MEII

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#223

Post by sdan »

natch wrote: post #180
sdan wrote: post #71

Earlier quoted context omitted.

As a small anecdote, my parents couldn't afford/didn't want to spent over $30k for a car. Surely we could've gotten a Tesla for $5k+ more, but given the relatively new infrastructure with electric charging stations (and the fact that none are available in the apartment I live in) my parents didn't find all the new cool features appealing and instead got a regular Toyota Sienna that has nothing fancy, just enough to t…

Thanks for the reply. Living in an apartment is not a big issue for me... we have two, live in an apartment with no charger. That's interesting that you relate something you do in the AV industry to not ever getting a car... what's that about? I do think that it's possible in the future the majority of people will never need to own a car.

I think for a good majority of people especially in America will go along the lines of "If it ain't broke, don't fix it" for gas cars.

Regardless, I think Ghost Locomotion and Comma.ai have a lot of potential for what they're doing now. I think they'll coincide with fully driverless cars like Cruise, Waymo, or Aurora. Regardless of if they're electric or not (I think electric cars will be more heavily adapted if we use Cars as a Service)

Also, yes, I don't ever intend of getting a license. This problem is really fascinating and I'm excited to play a little part of it. By not having a license, I can spend more time having the perspective of a person in 20-30 years when drivers licenses will become less common and utilize it in my work.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#224

Post by maxerickson »

mlyle wrote: post #219
maxerickson wrote: post #172

Earlier quoted context omitted.

Good ol' velocity-cubed. Increasing their compute per watt allows them to get more compute per watt, so they will pursue it lots, low speed energy savings are a way of making the range extension seem like a big deal when it isn't (maintaining highway speeds requires tens of kilowatts).

It was a big deal for Tesla, though, because they had to fit into a tiny power budget. HW2.0 wasn't enough for what they wanted to do, and to retrofit existing cars they had to consume a similar amount of power. Power efficiency does make a range difference and is worth seeking, but Tesla is exaggerating this, IMO, to conceal one of the ways that the churn-heavy cycle for FSD has imposed organizational costs (an enti…

I would expect them to seek compute/watt basically until they stopped making progress.

An actual working system might slow them down, but even then, better hardware is one of the ways to make it better.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#225

Post by diveanon »

Andrej Karpathy is such a treasure.

He is an excellent presenter who really has a passion for teaching.

Im not really involved with the industry, so I cant really speak to how he holds up to other experts. However he is by far the most digestable resource I have found for learning about NN and science behind them.

If you are just discovering him now, google his name and just start reading. His work is truly binge worthy in the most meaningful way.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#226

Post by millettjon »

chronic739i wrote: post #90
jeffshek wrote: post #86

Earlier quoted context omitted.

At Tesla's scale and priorities, they'd probably be less keen on using external cloud providers. Using TPUs at their scale would certainly require Google's AI consultants to supervise which isn't ideal for Tesla. Not agreeing or disagreeing with their decisions, but if you have the resources, you can certainly design a custom chip that performs a specific type of task very well that beats other competitors. Nvidia's…

> At Tesla's scale and priorities, they'd probably be less keen on using external cloud providers. By hardware-hours, Tesla is hardly one of the top companies training deep networks. Planet Labs (satellite imaging), Netflix, Pornhub, to name a few. What's the info they'd leak to the Google consultants? How much data or TPUs they're using? This is practically public information.

Do those really take more than 70,000 gpu hours to train a model?

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#227

Post by p1esk »

h5rea4he wrote: post #216
p1esk wrote: post #207

Earlier quoted context omitted.

Same can be said about level 5 autonomy!

To me, the fatal flaw of HBP (or USA's competing HBI) is that they are akin to "cargo cult science", the idea that we can replicate the superficial structures to a significant enough degree that they system they impart will suddenly somehow become activated. But just like the Melanesians with their coconut-shell headsets, there won't be anyone listening on the other end...

If we replicate a car with a sufficient accuracy it will start. I don’t see why this wouldn’t apply to any other piece of machinery, including a brain.

HBP should have tried a simpler task first, e.g. replicate a fruit fly’s brain.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#228

Post by wstrange »

xiphias2 wrote: post #138
jacquesm wrote: post #110

Earlier quoted context omitted.

> Larry Page didn't want to launch partial self driving That's the responsible thing to do.

If you look at self driving automation as a black box, sure. But at the same time people understand that on a highway Tesla Autopilot is safe enough to be used on a long boring road, and dricers generally feel less tired (and can focus more on the harder parts of the road).

There have been notable Tesla fatalities on "long boring roads". The software has a long way to go..

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#229

Post by adrr »

deboflo wrote: post #178
adrr wrote: post #66

Earlier quoted context omitted.

Elon belittles lidar saying it is doomed and will never work yet Waymo and Cruise will probably be operating self driving taxi fleets in California next year. Tesla deserves getting dumped on for those comments because they are no where near self driving.

I live in San Francisco. Every time I see Cruise vehicles, the driver has his hands on the steering wheel.

Cruise vehicles go thousands of miles without human intervention and they have a license to run their cars without a driver like Waymo.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#230

Post by spyder »

antpls wrote: post #114
modeless wrote: post #70

Awesome presentation. Crazy that they're developing their own training hardware too. It's going to be a very crowded space very soon. Can they really stay ahead of everyone else in the industry? Can it really be cheaper to staff up whole teams to design chips for cutting edge nodes, fabricate them, build supporting hardware and datacenters and compilers, than to just rent some TPUs on Google Cloud? I can see the case…

Watch the presentation from 6 months ago, where they explain the decision to build their own hardware for inferring : https://youtu.be/Ucp0TTmvqOE?t=4309 It's not surprising that they also build the hardware for training. Correct me if I'm wrong, but Google use the same TPUs for training and inference, because the underlying operations are the same : multiply then add numbers. Once Tesla built the hardware for inferr…

Very good video, it contains answers to many of the questions that people are speculating here about and other interesting things about Tesla's custom chip.

- It's under 100W so they can retrofit into old cars

- lower part cost, so they can do full redundancy with doubling the parts

- they estimated that 50 TOPS is needed for self-driving

- lower latency with batch size of 1 compared to TPU's 256

+ GPU for post-processing

- security: only code signed by Tesla can run on the chip

- at the time (2016) there was no neural net accelerator chips

- some part's are built from bought IP (so not reinventing them) Probably things like the 12 ARM CPUs, LP DDR4 memory, video encoder, maybe the separate post-processing GPU too...

- physical size of the board is small

- performance example: on CPU 1.5 FPS, on GPU (600 gflop) 17 FPS, on Tesla's NN accelerator 2100 FPS

- Besides the convolution even the ReLU and Pooling is implemented in hardware

- Paying attention to the energy efficiency down to the arithmetic and data type usage.

- The silicon cost is less than their previous hardware (HW 2.5)

- old hardware 110 FPS new one 2300 FPS

- 144 TOPS compared to NVidia's Drive Xavier 21 TOPS

Post reply on HN