Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular self-driving car dataset is missing labels for hundreds of pedestrians

blog.roboflow.ai

Post reply on HN
Print view View original

91–100 of 202 posts

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#91

Post by [deleted] »

kozd wrote: post #83
keenmaster wrote: post #70

Earlier quoted context omitted.

If everyone believes that everyone else will be irrational, then they themselves will not throw their support behind nuclear power, rendering their stance on nuclear de facto irrational. As Baby Boomers age, and Millennials/GenZ form a greater percentage of the voting population, we have an opportunity to press the reset button on nuclear. The younger generations don’t really have a solid opinion on the matter, and p…

Is it a tautology or just an unfortunate nash equilibrium?

[deleted]

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#92

Post by sukilot »

jasonwatkinspdx wrote: post #84
yeldarb wrote: post #2

This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…

No it's not scary in the slightest. Missing or scrambled labels are routine in ML and the algorithms are able to handle it. This was a huge paper a couple years ago that demonstrates deep nets will still find the structure in the data even with totally randomized labels: https://arxiv.org/abs/1611.03530 There are also various well known techniques to verify whether you've over or underfit.

> deep nets will still find the structure in the data

Yes, but in the training data, not in the non-training data! We want the cars to avoid real pedestrians, not only the ones labelled in the training set!

From the paper:

> When trained on a completely random labeling of the true data, neural networks achieve 0 training error. The test error, of course, is no better than random chance as there is no correlation between the training labels and the test labels.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#93

Post by jacquesm »

myself248 wrote: post #51
PeterisP wrote: post #29

Earlier quoted context omitted.

A big thing is not that the example is missing, but that it counts as a negative example. I.e. if during training a ML system notices the ambiguous combination (i.e. a woman pushing a baby stroller or a crowd) and marks it as a pedestrian, then it gets penalized in a manner that teaches it to ignore these ambigious combinations and treat it as nothing; while in practice it should probably treat such ambiguous combina…

Likewise, I have yet to see one of these things that can recognize potholes and swerve to avoid them lest a wheel be ripped off. Sooner or later it'd be nice to be able to drive one of these things in a place that isn't southern California.

That's a feature, not a bug. Swerving for potholes can be very dangerous, more dangerous than having undercarriage damage. If a pothole surprises you enough that you have to swerve you were either not paying attention to the road or you are following too close.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#94

Post by dmix »

yters wrote: post #54
pathseeker wrote: post #30

Earlier quoted context omitted.

>Would you bet a family member? "think of the children!" Lives at stake don't change anything here. The question is whether self-driving cars, even with the errors, are safer for people than regular drivers on average. If so, then absolutely yes everyone should bet their lives and their families'. Thousands of people are dying every day in cars. This is not something we need to wait for it to be perfect. It only need…

I think people intuitively ascribe a moral dimension to whether people are involved in accidents, hence why they are more worried about dying on an airplane than in a car, even though the former is much less likely than the latter. If I die in a plane, I have no control over the matter. If I die in a car, then at least I may have had control, and it could be chalked up to my inattentiveness, bad driving behavior, etc…

The great part of self-driving cars is that just like when there are major airplane accidents there is a thorough learning process afterwards and all flights become safer as a result on a frequent basis.

This is the same reality for self-driving cars. The edge cases will happen but they won't keep happening with rigorous improvements in the models and trained behaviour.

Flight safety has improved dramatically over the past century and the ability for self driving cars to adapt will be even faster, as it's mostly just software.

Of course this implies that an accident had to happen for such an improvement to take place. But we have both simulation improving drastically to help alleviate that plus the alternative is the current situation where the same types of accidents keep happening again and again with only the occasional improvement to car technology and safety features.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#95

Post by runarberg »

mumblemumble wrote: post #21
tomnipotent wrote: post #15

Earlier quoted context omitted.

> This is really scary. No, it's not even remotely "really scary". No one is putting an actual self-driving car on the market using this specific data set. Disingenuous to pretend this is any indication of the data using by serious companies in the space or is represented of the impact a few mislabelled samples have on the ability of these systems & algorithms to generalize.

It's scary when you consider two other factors: First, the AI hype train. People think that calling something "Artificial Intelligence" implies that it is artificial, yes, but also, critically, that it is intelligent. Many enthusiastic people, and also many policymakers, don't fully realize the extent to which machine learning is constrained by both the quality and nature of its training data, and the capabilities of…

> First, the AI hype train.

Yes! I really wished we would call this something more descriptive of the boring statistics involved. Artificial Intelligence is just too cool sounding name not to get excited about it. And what people don’t realize is that this is really just a statistical inference model, nothing intelligence. Machine Learning is better, but it is still not descriptive enough.

I think if we called it something like Computational Reinforcement Modelling or Iterative Weight Inference people might stop having the idea that there are machines making “smart” choices involved and finally seeing that really we are just inferring based some computations on provided data.

Really, “Machine Learning” deserves no more hype than the boring sounding Kalman Filter.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#96

Post by notahacker »

cameronbrown wrote: post #76
craftinator wrote: post #73

Earlier quoted context omitted.

I'm voting for the "less dead people" option. Mostly because I'm a selfish person, been in automobile accidents caused by lapsing human attention, and I want it to be less likely that I'll die in a car crash.

But it's not just about quantity. It's also _different_ people who will die. That radically alters things from an ethical perspective.

Yep. Medical professionals have been aware of this dilemma for millennia: many people die from an ailment if no treatment is attempted, but bad approaches to treatment can kill people that would have survived otherwise. And setting 'better average accident rates' as the threshold for self driving vehicle software developers to be immune from the consequence of their errors is like setting 'better than witch doctors' as the threshold for making doctors immune from claims of malpractice.

Move fast, break different things, is not the answer.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#97

Post by JoeAltmaier »

notahacker wrote: post #96
cameronbrown wrote: post #76

Earlier quoted context omitted.

But it's not just about quantity. It's also _different_ people who will die. That radically alters things from an ethical perspective.

Yep. Medical professionals have been aware of this dilemma for millennia: many people die from an ailment if no treatment is attempted, but bad approaches to treatment can kill people that would have survived otherwise. And setting 'better average accident rates' as the threshold for self driving vehicle software developers to be immune from the consequence of their errors is like setting 'better than witch doctors'…

What if its very much better average accident rates? This isn't black-and-white.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#98

Post by shoeffner »

While I am happy about their efforts, it's interesting that in the bottom left image of their example I can clearly see another unlabeled car in the lower left half (standing at the sidewalk of the street). Also, I am not sure, but it seems like there's a cyclist on the sidewalk, visible between the stroller and the car (the wheel and hands are more clearly visible). The google image marks a car and a few pedestrians, but completely misses the traffic lights at the junction.

So I guess even their fixed dataset still misses many labels, if already their showcases miss some.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#99

Post by notahacker »

jacquesm wrote: post #93
myself248 wrote: post #51

Earlier quoted context omitted.

Likewise, I have yet to see one of these things that can recognize potholes and swerve to avoid them lest a wheel be ripped off. Sooner or later it'd be nice to be able to drive one of these things in a place that isn't southern California.

That's a feature, not a bug. Swerving for potholes can be very dangerous, more dangerous than having undercarriage damage. If a pothole surprises you enough that you have to swerve you were either not paying attention to the road or you are following too close.

tbf there's still the evaluate whether slowly changing your lane position to avoid the pothole will impede or confuse other road users, and do so only if that isn't the case option to avoid potholes, and I don't imagine [semi]autonomous driving systems do that either?

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#100

Post by strbean »

rifung wrote: post #38
pathseeker wrote: post #30

Earlier quoted context omitted.

>Would you bet a family member? "think of the children!" Lives at stake don't change anything here. The question is whether self-driving cars, even with the errors, are safer for people than regular drivers on average. If so, then absolutely yes everyone should bet their lives and their families'. Thousands of people are dying every day in cars. This is not something we need to wait for it to be perfect. It only need…

> Lives at stake don't change anything here. The question is whether self-driving cars, even with the errors, are safer for people than regular drivers on average. If so, then absolutely yes everyone should bet their lives and their families'. Maybe logically that makes sense but from an ethical perspective I argue it's much more complicated than that (e.g. the trolley problem) In the current system if a human is at…

Do we gain something from placing blame? Who do we blame for people who die from natural disasters? Freak occurrences?

Are deaths where blame can be placed preferable to deaths where it cannot? By what factor? Should we try to exchange one of the latter for two of the former?

Post reply on HN