Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

youtube.com

Post reply on HN
Print view View original

211–220 of 249 posts

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#211

Post by eanzenberg »

punnerud wrote: post #69
eanzenberg wrote: post #62

One thing I didn't quite understand is how training sub-graphs in parallel works. If you are editing a sub-graph of a monolith type model, aren't you affecting other graphs that have dependencies on the one you're editing? If these are independent graphs, then what's a "sub-graph" even mean?

In PyTorch you have full control on the graph and weight, everything feels like Python. So feeding some of the learning between “sub-graph” is easy. Not sure if this is possible on Tensorflow/Keras? He describes the sub-graph training in the context that they they have all the predictors in one big model, and with control of the network can feedforward and train sub-graph (read sub-parts) of the model.

This is possible in keras, just drive new models that are functions of a monolith model and train independently. I still don’t understand the point though. If you train a “subgraph”, the other tasks dependent on the part of the graph will have to get retrained anyways, since those edits will affect the other tasks.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#212

Post by eanzenberg »

jacquesm wrote: post #105

The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants t…

Competition is good

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#213

Post by semi-extrinsic »

tim333 wrote: post #137
semi-extrinsic wrote: post #102

Earlier quoted context omitted.

To really be a measurable improvement over humans, and especially an improvement that is statistically distinguishable from just letting the safety tech of 2019 percolate into the average car, self-driving needs to achieve fatality rates around 0.1 per billion miles. Current averages for the US are around 10 per billion miles and decreasing; best countries in Europe are already below 5 per billion miles. There's some…

I'd quibble with your stats that self driving has to be 100x better than human to be a measurable improvement. You can estimate fairly well if a human driver is safe or not from a few hours as a passenger by seeing if they notice everything and if they have near misses. Also it's not to hard on published data to see the Tesla autopilot seems a fair bit worse than humans under the same conditions. I imagine they will…

I'm not saying it has to be 100x better than a human.

What I'm saying is, the averages today are across all vehicles on the road, some of which are old and have fairly low survival probability in an accident. And the deployment of technology to handle/avoid distracted driving is still in its infancy. If we just wait 10-15 years, I believe the safety rates for humans will increase by 10x over current US statistics.

Then self-driving has to manage another 10x improvement over that, in order to be worth it.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#215

Post by tim333 »

anonu wrote: post #177
londons_explore wrote: post #176

I'm still amazed that Teslas team isn't using a map... I know maps get outdated and are sometimes wrong, but having inaccurate knowledge of what's around the corner is far far more helpful than not having any clue whats around the corner. The smart solution would be to consider a map a probabilistic thing, which neural networks are really good at handling.

I'm still amazed Tesla has decided not to use lidar and instead just stick with cheap cameras. Better sensors are there, they're available, they're cheap and they can probably "see" better than plain old cameras... it doesn't make too much sense not to use them IMHO. But then again, I am not coding NNs for Tesla...

Musk doesn't like it:

>“Lidar is a fool’s errand,” Elon Musk said. “Anyone relying on lidar is doomed. Doomed! [They are] expensive sensors that are unnecessary. It’s like having a whole bunch of expensive appendices. Like, one appendix is bad, well now you have a whole bunch of them, it’s ridiculous, you’ll see.” https://techcrunch.com/2019/04/22/anyone-relying-on-lidar-is...

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#216

Post by h5rea4he »

p1esk wrote: post #207
jacquesm wrote: post #206

Earlier quoted context omitted.

The human brain project was DOA from day #1. Unrealistic goals, no clear reason why more money would lead to better results and no concrete deliverables that anybody needed.

Same can be said about level 5 autonomy!

To me, the fatal flaw of HBP (or USA's competing HBI) is that they are akin to "cargo cult science", the idea that we can replicate the superficial structures to a significant enough degree that they system they impart will suddenly somehow become activated.

But just like the Melanesians with their coconut-shell headsets, there won't be anyone listening on the other end...

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#217

Post by nfoz »

eanzenberg wrote: post #212
jacquesm wrote: post #105

The competition in this space is great but I can't help but wonder what would happen if instead all these companies pooled their resources and went after the goal collectively. There is so much duplication going on and the paths do not seem to me - as an outsider - to be all that divergent, which is usually a pre-condition for having a lot of independent efforts one of which will succeed. It's as if everybody wants t…

Competition is good

Choice is good, alternative implementations are good. But I think competition is bad. It is wasteful and antisocial.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#218

Post by mlyle »

dzdt wrote: post #191
SloopJon wrote: post #166

Earlier quoted context omitted.

Their own page says, "our recommendation algorithms ... learning characteristics that make content successful ... optimize the production of original movies and TV shows ... optimize video and audio encoding, adaptive bitrate selection, and our in-house Content Delivery Network ... and advertising". https://research.netflix.com/research-area/machine-learning Here's another article on the subject: https://becominghuma…

Netflix put a lot of work into recommendation back in the dvd delivery days when their catalog was absolutely massive. Now in the streaming space the catalog they license is much smaller so recommendation is less important; basically they just advertise the popular stuff for your demographic. Its a bit odd that legally one can rent out physical disks, but there is no corresponding way to legally get permission to ren…

> Its a bit odd that legally one can rent out physical disks, but there is no corresponding way to legally get permission to rent out streaming content without negotiating with the rightsholder. But thats how it is...

It's hard to think of a good fair regime to do this under.

At least with a physical disk, there's a maximum reasonable rate that you can turn the disk around between users and you need to have enough copies for whatever the lifecycle peak demand is.

We do have an audio compulsory licensing system for things that are purely songs. But with video works, there's not a clear boundary for "how big" the work is-- how do you treat 30 hour anime series vs a 5 minute Pixar short? How do you treat continuing medical education videos vs. fluff amateur made content? Etc.

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#219

Post by mlyle »

maxerickson wrote: post #172
mrep wrote: post #163

Earlier quoted context omitted.

From autonomy day hacker news comment: "Pegasus consumes about 500Watts, compared to under 100 Watts for Tesla's FSD computer. Elon in particular emphasized the performance per watt (as it's always possible to cram more chips to increase performance if you ignore cost and power consumption). The comparison made in the video: 500Watts for an hour consumes about 2-3 miles of range. In a city in slow traffic, going 12mp…

Good ol' velocity-cubed. Increasing their compute per watt allows them to get more compute per watt, so they will pursue it lots, low speed energy savings are a way of making the range extension seem like a big deal when it isn't (maintaining highway speeds requires tens of kilowatts).

It was a big deal for Tesla, though, because they had to fit into a tiny power budget. HW2.0 wasn't enough for what they wanted to do, and to retrofit existing cars they had to consume a similar amount of power.

Power efficiency does make a range difference and is worth seeking, but Tesla is exaggerating this, IMO, to conceal one of the ways that the churn-heavy cycle for FSD has imposed organizational costs (an entire chip-level hardware program, in this case).

Re: Andrej Karpathy talks about how Tesla's NNs are structured and trained [video]

#220

Post by logicallee »

kortex wrote: post #193
logicallee wrote: post #147

Earlier quoted context omitted.

>Imagine every car manufacturer having a completely different take on what a car should be like from a safety perspective. We have standards bodies for a reason Roads are also governed by public bodies. Road signs are standardized and public. I think the government should take a much larger role in defining self driving cars. For example, rather than using computer vision to recognize signs, signs could be active sta…

> Would it be absurd for the crosswalk to know roughly how many people are at it, then broadcast this to the car Yes, completely absurd. For one, many places are too sparse, poor, or unstandardized for this to be remotely economical. Two, people often don't cross at crosswalks (jaywalking). Level 5 AV is a thing where 99% coverage isn't good enough. You need a lot of nines. That's why many think lidar systems are a c…

In sparsely populated areas smart roads could inform the car that there hasn't been any movement whatsoever in the whole area for hours. (Along the entire road and adjacent to the roads). How the car responds (driving somewhat faster, perhaps) could mean a large measure of safety as compared to when there are a large group of people about to cross a rural road at night that the car may or may not see visually.

Anyway it's just one example. RFID or similar embedded beacons in the road paint would make roads much easier for cars to follow. They are expensive for any one company to do but cheap for the government to do if it is done everywhere at once and lasts 5-10 years. Road signs that broadcast what they are (instead of needing to be "seen" and interpreted visually instead of through radio by the car), are similar.

Finally, a government standard could coordinate cars into a caravan, avoiding pileups for example, and giving the participating cars several advantages you can find by Googling "car caravan". (Though this might be achieved by industry.)

Post reply on HN