Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular self-driving car dataset is missing labels for hundreds of pedestrians

blog.roboflow.ai

Post reply on HN
Print view View original

81–90 of 202 posts

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#81

Post by TeMPOraL »

tomnipotent wrote: post #15
yeldarb wrote: post #2

This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…

> This is really scary. No, it's not even remotely "really scary". No one is putting an actual self-driving car on the market using this specific data set. Disingenuous to pretend this is any indication of the data using by serious companies in the space or is represented of the impact a few mislabelled samples have on the ability of these systems & algorithms to generalize.

> No one is putting an actual self-driving car on the market using this specific data set. Disingenuous to pretend this is any indication of the data using by serious companies in the space

Are you sure about that? What about "non-serious" companies? Various fly-by-night self-driving startups?

I mean, this sounds like the machine learning equivalent of "no serious business is pulling random bits of code from StackOverflow or whatever tutorial popped up first in the search results", and yet I think everyone who worked with software for more than few years can confirm that companies absolutely do it, and a lot.

(Tangentially, PHP got a bad rap in large part because of this - people not knowing how to do things correctly copying code from tutorials written by people who didn't know it either; these days, JS ecosystem is getting a bad rap for having such bad code wrapped in neat and easy-to-install NPM packages.)

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#82

Post by SolaceQuantum »

jacquesm wrote: post #61
yeldarb wrote: post #2

This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…

You would hope that the software would be resilient enough that missing or even faulty labels would not be the cause of problems. It is by the way relatively easy to - once you reach some arbitrary threshold of precision - to detect such missing or wrong labels automatically, a simple suggestion UI with an accept / reject button could take care of supplying the bulk of the missing labels and correcting the bulk of th…

How would one reach a threshold of precision if the data on which precision is determined is missing or faulty?

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#83

Post by kozd »

keenmaster wrote: post #70
mumblemumble wrote: post #31

Earlier quoted context omitted.

I'm saying that nuclear power could have been a safer energy option, but, in practice, the whole enterprise has been scuttled by a bunch of regrettably bad decisions that have pretty much destroyed everyone's trust. So now it doesn't really matter if it's safer, because it can no longer realistically be considered an option.

If everyone believes that everyone else will be irrational, then they themselves will not throw their support behind nuclear power, rendering their stance on nuclear de facto irrational. As Baby Boomers age, and Millennials/GenZ form a greater percentage of the voting population, we have an opportunity to press the reset button on nuclear. The younger generations don’t really have a solid opinion on the matter, and p…

Is it a tautology or just an unfortunate nash equilibrium?

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#84

Post by jasonwatkinspdx »

yeldarb wrote: post #2

This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…

No it's not scary in the slightest. Missing or scrambled labels are routine in ML and the algorithms are able to handle it.

This was a huge paper a couple years ago that demonstrates deep nets will still find the structure in the data even with totally randomized labels: https://arxiv.org/abs/1611.03530

There are also various well known techniques to verify whether you've over or underfit.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#85

Post by jasonwatkinspdx »

SolaceQuantum wrote: post #82
jacquesm wrote: post #61

Earlier quoted context omitted.

You would hope that the software would be resilient enough that missing or even faulty labels would not be the cause of problems. It is by the way relatively easy to - once you reach some arbitrary threshold of precision - to detect such missing or wrong labels automatically, a simple suggestion UI with an accept / reject button could take care of supplying the bulk of the missing labels and correcting the bulk of th…

How would one reach a threshold of precision if the data on which precision is determined is missing or faulty?

The simple idea is to do stuff like train your model on randomized subsets of your data and then compare its performance to using all the data you have.

This doesn't overcome a true "black swan", but that's not what NN are meant to be doing anyhow.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#86

Post by jasonwatkinspdx »

jwandborg wrote: post #40
tomnipotent wrote: post #24

Earlier quoted context omitted.

It does not make use aware of a new class of errors. Labeling issues is nothing new, but plenty of systems trained on them continue to work just fine. This is FUD.

Is there any statistical/mathematical tool to completely eradicate or greatly diminish the effects of bad labeling? Is there any reason - other than the combination of pure circumstance and gut feeling of the Data Scientist in charge of saying that it's good enough to deploy - that ~33% insanity in training doesn't become ~33% insanity in the system?

We have a lot more than the gut feeling you're assuming: https://arxiv.org/abs/1611.03530

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#87

Post by SolaceQuantum »

jasonwatkinspdx wrote: post #85
SolaceQuantum wrote: post #82

Earlier quoted context omitted.

How would one reach a threshold of precision if the data on which precision is determined is missing or faulty?

The simple idea is to do stuff like train your model on randomized subsets of your data and then compare its performance to using all the data you have. This doesn't overcome a true "black swan", but that's not what NN are meant to be doing anyhow.

"The simple idea is to do stuff like train your model on randomized subsets of your data and then compare its performance to using all the data you have."

How do you do this when you cannot verify that your data, in subset or in whole, is accurate? And furthermore you don't know how inaccurate it is?

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#88

Post by TeMPOraL »

Florin_Andrei wrote: post #27
mumblemumble wrote: post #21

Earlier quoted context omitted.

It's scary when you consider two other factors: First, the AI hype train. People think that calling something "Artificial Intelligence" implies that it is artificial, yes, but also, critically, that it is intelligent. Many enthusiastic people, and also many policymakers, don't fully realize the extent to which machine learning is constrained by both the quality and nature of its training data, and the capabilities of…

> People think that calling something "Artificial Intelligence" implies that it is artificial, yes, but also, critically that it is intelligent. I mean, the exact same argument could be made about people. Just because you belong to Homo Sapiens doesn't imply you're an intelligent being at all times - e.g. drink a bottle of vodka and the sapiens part is gone.

The thing with people is that we all share the same brain architecture - we mostly think alike. As a species, we have a hundred thousand plus years of experience dealing with each other; as a civilization, a couple thousand. We've explored most corner cases, designed our infrastructure around those, and built systems protecting us from outliers. For instance, people deemed too unpredictable will not be given driving licenses (and extremely unpredictable people tend to get locked up). And DUI is severely punished; while the execution isn't perfect, the threat of consequences goes a long way towards reducing it.

That's the bar an AI system has to meet today. And one big hurdle is that, viewed as minds, AI systems are completely unlike our own. They're extremely unpredictable. On top of that, they lack self-preservation instinct (this may be indirectly a side effect of how we structure companies - the level of effort put into an AI system is a reflection of how much people working on it care about getting it right vs. getting it to market).

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#89

Post by bo1024 »

jasonwatkinspdx wrote: post #84
yeldarb wrote: post #2

This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…

No it's not scary in the slightest. Missing or scrambled labels are routine in ML and the algorithms are able to handle it. This was a huge paper a couple years ago that demonstrates deep nets will still find the structure in the data even with totally randomized labels: https://arxiv.org/abs/1611.03530 There are also various well known techniques to verify whether you've over or underfit.

Strongly disagree. ML is traditionally deployed in low-risk scenarios like image labeling or recommendations where 1% or 5% error rate is totally "routine" and acceptable. Doesn't mean this kind of ML pipeline can translate to cases where errors are literally fatal.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#90

Post by sheepstrat »

My knee-jerk is to be upset because the risk factor of a poorly functioning pedestrian classifier is obviously higher than something like a sentiment analyzer, but at the same time, this dataset is just for educational purposes, right? Is Udacity actively recommending people use this in production settings?
Post reply on HN