Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular self-driving car dataset is missing labels for hundreds of pedestrians

blog.roboflow.ai

Post reply on HN
Print view View original

141–150 of 202 posts

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#141

Post by tomnipotent »

perl4ever wrote: post #140
tomnipotent wrote: post #134

Earlier quoted context omitted.

> you'd have to convert the number of fatalities to an accuracy percentage Telsa's early results for their very limited "self-driving" technology has shown a huge reduction in accidents for any given period of time the vehicles are on the road.

That seems like it incorporates a lot of assumptions. I think it's best to slow down and realize that comparisons don't mean much if you're comparing the wrong things. The first step is to determine the first thing that you are comparing and exactly what it is. Then you can move on to the other half and determine whether it is appropriate. Humans are much safer than people on average, when driving in conditions suita…

> Humans are much safer than people on average

This makes zero sense and isn't how "average" works. For the same 1000 hours on the road, a Tesla car with Autopilot will have fewer accidents than a car driven for 1000 hours by humans. This changes as driving conditions get worse, and humans outperform Autopilot.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#142

Post by pavel_lishin »

rlayton2 wrote: post #113
TeMPOraL wrote: post #81

Earlier quoted context omitted.

> No one is putting an actual self-driving car on the market using this specific data set. Disingenuous to pretend this is any indication of the data using by serious companies in the space Are you sure about that? What about "non-serious" companies? Various fly-by-night self-driving startups? I mean, this sounds like the machine learning equivalent of "no serious business is pulling random bits of code from StackOve…

I've mentioned to a few people now that the first generation of self-driving cars don't scare me that much. It will be weird, but I assume that special care will be taken and the cars themselves will be quite risk-averse. I'm scared about the next generation of self-driving cars when we have "solved it" and the race is on to get cars out quickly and at low cost (and things like in-depth testing are out of budget). Ho…

> I assume that special care will be taken and the cars themselves will be quite risk-averse.

Bold assumption that's already got a counterpoint: https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#143

Post by gdsdfe »

ClumsyPilot wrote: post #139
gdsdfe wrote: post #130

Do they really think people are using these datasets for commercial applications?

Do you really think there is any possible fuckup that won't happen sooner or later? Will staff of a nuclear silo forget to lock the door, and then fall asleep? Because that happened in US Do you really think someone will put a plane in production with a single safety critical sensor with no backup or fall back? Do you really think someone will pour dissolved uranium down the drain, starting a nuclear reaction and dyi…

None of these is similar! The off the shelf dataset won't help you achieve nothing commercially, maybe learning or tinkering at best.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#144

Post by neurobro »

jwandborg wrote: post #40
tomnipotent wrote: post #24

Earlier quoted context omitted.

It does not make use aware of a new class of errors. Labeling issues is nothing new, but plenty of systems trained on them continue to work just fine. This is FUD.

Is there any statistical/mathematical tool to completely eradicate or greatly diminish the effects of bad labeling? Is there any reason - other than the combination of pure circumstance and gut feeling of the Data Scientist in charge of saying that it's good enough to deploy - that ~33% insanity in training doesn't become ~33% insanity in the system?

Not sure about bad labels, but semi-supervised learning is the term for training on data with a lot of missing labels. Essentially the algorithm makes predictions on the unlabeled data and uses its highest confidence predictions as additional training data. Generative models can also "dream up" entirely new training examples. There is a risk of amplifying the confidence in bad predictions, but it works well overall (better than using only the labeled portion of the data).

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#145

Post by drusepth »

BigJono wrote: post #117
TeMPOraL wrote: post #81

Earlier quoted context omitted.

> No one is putting an actual self-driving car on the market using this specific data set. Disingenuous to pretend this is any indication of the data using by serious companies in the space Are you sure about that? What about "non-serious" companies? Various fly-by-night self-driving startups? I mean, this sounds like the machine learning equivalent of "no serious business is pulling random bits of code from StackOve…

> these days, JS ecosystem is getting a bad rap for having such bad code wrapped in neat and easy-to-install NPM packages.) Is it? IME nobody seems to give a flying fuck outside of HN. Much like the PHP days. Or maybe I've never been in the right workplaces to notice.

I work in Ruby on Rails (currently with a React frontend, but previous jobs used Angular) in the Midwest. IME almost everything around the "recent" JS ecosystem (npm, npx, node, yarn, grunt, react, jsx, etc) are used (begrudgingly) when necessary, but are just the butt of jokes 99% of the time.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#146

Post by a_bonobo »

pavel_lishin wrote: post #142
rlayton2 wrote: post #113

Earlier quoted context omitted.

I've mentioned to a few people now that the first generation of self-driving cars don't scare me that much. It will be weird, but I assume that special care will be taken and the cars themselves will be quite risk-averse. I'm scared about the next generation of self-driving cars when we have "solved it" and the race is on to get cars out quickly and at low cost (and things like in-depth testing are out of budget). Ho…

> I assume that special care will be taken and the cars themselves will be quite risk-averse. Bold assumption that's already got a counterpoint: https://en.wikipedia.org/wiki/Death_of_Elaine_Herzberg

Wikipedia already has a list: https://en.wikipedia.org/wiki/List_of_self-driving_car_fatal...

Along with your pedestrian death there were five (!!!) driver deaths.

There's a reason why places like Germany don't want Tesla to use the term 'autopilot' because it's reckless endangerment.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#147

Post by mycall »

yeldarb wrote: post #2

This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…

I would think you want some images that cut off vehicles since this is how cameras will work at times, no?

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#148

Post by perl4ever »

tomnipotent wrote: post #141
perl4ever wrote: post #140

Earlier quoted context omitted.

That seems like it incorporates a lot of assumptions. I think it's best to slow down and realize that comparisons don't mean much if you're comparing the wrong things. The first step is to determine the first thing that you are comparing and exactly what it is. Then you can move on to the other half and determine whether it is appropriate. Humans are much safer than people on average, when driving in conditions suita…

> Humans are much safer than people on average This makes zero sense and isn't how "average" works. For the same 1000 hours on the road, a Tesla car with Autopilot will have fewer accidents than a car driven for 1000 hours by humans. This changes as driving conditions get worse, and humans outperform Autopilot.

Deleting half a sentence and saying it doesn't make sense?

The way "average" works is that you average over something - a population or set. It is very important to be clear about what that something is and whether it's appropriate.

Why do you believe that Autopilot outperforms humans in comparable conditions? If this is based on Tesla marketing, I'm extremely prejudiced against them, and assume out of hand that they simply aren't making the right comparison and don't care. However, if you think that is incorrect, you could elaborate on why you have the opinion you do.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#149

Post by notatoad »

mycall wrote: post #147
yeldarb wrote: post #2

This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…

I would think you want some images that cut off vehicles since this is how cameras will work at times, no?

I think the problem is not that the car is cut off, its that the cut-off portion of the car isn't labeled. If that's your training data, then you're training the system to interpret cut-off bits of cars as not cars.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#150

Post by tomnipotent »

perl4ever wrote: post #148
tomnipotent wrote: post #141

Earlier quoted context omitted.

> Humans are much safer than people on average This makes zero sense and isn't how "average" works. For the same 1000 hours on the road, a Tesla car with Autopilot will have fewer accidents than a car driven for 1000 hours by humans. This changes as driving conditions get worse, and humans outperform Autopilot.

Deleting half a sentence and saying it doesn't make sense? The way "average" works is that you average over something - a population or set. It is very important to be clear about what that something is and whether it's appropriate. Why do you believe that Autopilot outperforms humans in comparable conditions? If this is based on Tesla marketing, I'm extremely prejudiced against them, and assume out of hand that they…

The average of something cannot be more than the average of... itself. Thus, "Humans are much safer than people on average" is nonsensical.

> Why do you believe that Autopilot outperforms humans in comparable conditions?

Because they have the data that proves it?

> I'm extremely prejudiced against them

And I've chosen to take them at face value with a grain of salt, and to believe that for the data they've collected from the hundreds of thousands of Tesla's with millions of hours of data using Autopilot, it's fair to say they have a large enough sample to draw conclusions about the safety of their cars vs. any incident rates from pretty much any other distribution.

Post reply on HN