Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A popular self-driving car dataset is missing labels for hundreds of pedestrians

blog.roboflow.ai

Post reply on HN
Print view View original

121–130 of 202 posts

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#121

Post by yeldarb »

a_t48 wrote: post #80

Hold on a second here - are unlabeled pixels used in training a NN to do detection? Will a typical NN get trained to label those pixels as “not a human”? I agree that they should be labeled, but it’s the difference between needing to throw more data at the problem (because you aren’t getting as much learning per image as you could) and actively training the car to do something bad.

Yes, if you don’t “punish” incorrect predictions in your loss function your neural net could just get “perfect” accuracy by putting a giant bounding box around the entire image.

Technically that’s “right”; it did put a box around all the obstacles just like you asked it to. But that “solution” is not useful. You want it to find what it’s looking for and only what it’s looking for.

In this case, if it detects an unlabeled pedestrian the loss function will penalize it a bit for that “wrong” answer and it will slightly deviate to try to not find that pedestrian but still find the correctly labeled examples. It’s trying to fit the examples you give it best as possible.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#122

Post by perl4ever »

runarberg wrote: post #95
mumblemumble wrote: post #21

Earlier quoted context omitted.

It's scary when you consider two other factors: First, the AI hype train. People think that calling something "Artificial Intelligence" implies that it is artificial, yes, but also, critically, that it is intelligent. Many enthusiastic people, and also many policymakers, don't fully realize the extent to which machine learning is constrained by both the quality and nature of its training data, and the capabilities of…

> First, the AI hype train. Yes! I really wished we would call this something more descriptive of the boring statistics involved. Artificial Intelligence is just too cool sounding name not to get excited about it. And what people don’t realize is that this is really just a statistical inference model, nothing intelligence. Machine Learning is better, but it is still not descriptive enough. I think if we called it som…

...I like the Kalman Filter even if I might not be quite up to programming one. Every time I drive a modern car with a shitty automatic transmission that glitches and stutters and fails to shift right, I think "Is this thing using a Kalman filter, or what fundamentally wrong thing was done with the programming here?" I wish someone would reveal to me whether the algorithm is not the panacea I wish it were, or else what a lousy programmer does when they don't know about it.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#123

Post by shoeffner »

yeldarb wrote: post #120
shoeffner wrote: post #98

While I am happy about their efforts, it's interesting that in the bottom left image of their example I can clearly see another unlabeled car in the lower left half (standing at the sidewalk of the street). Also, I am not sure, but it seems like there's a cyclist on the sidewalk, visible between the stroller and the car (the wheel and hands are more clearly visible). The google image marks a car and a few pedestrians…

Hey, OP here, yeah you’re correct. The dataset doesn’t label any obstacles that small/far in the distance. I zoomed in on the region with errors for the sake of the screenshot. Here’s the original run through Google Vision AI. They actually don’t get the pedestrian either: https://imgur.com/a/84IVTV6 (I fired up the labeling tool I use and grabbed a recording of the few seconds of video around that frame to give an i…

Nice, thank you for the upload and thank you for clarifying that small objects are not labeled, that explains it. I was suprised because other images do contain rather small labels for traffic lights or even cars, but I guess it's always in the eyes of the person who labels the data.

I think you did an amazing amount of work and huge improvements over the original, have you considered contributing the changes back upstream?

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#124

Post by sneak »

> REDACTED accelerates your computer vision workflow through automated annotation quality assurance, universal annotation format conversion (like PASCAL VOC XML to COCO JSON), team sharing and versioning, and exports directly to file format, like TFRecords.

This is an ad, posing as a sensationalistic blog post.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#125

Post by perl4ever »

jwandborg wrote: post #32
ummonk wrote: post #28

Earlier quoted context omitted.

Given that nuclear power is significantly safer than other forms of power, are you asserting that the risks of self-driving cars are more about PR and perception than actual risk?

I think the point is rather that it's an unpredictable-stakes game. That is, if you make few enough mistakes, you'll be fine for a long time until you make the wrong mistake at the wrong time and it kills somebody. Maybe a better analogy is driving a car, that is, nuclear power and self-driving cars are both vaguely like driving a car. However, my intuition is that nuclear power has less variables than driving a car.…

"However, my intuition is that nuclear power has less variables than driving a car."

During the cold war they launched fission reactors into orbit. That probably had more variables. Then again, since the nuclear airplane project was unsuccessful, maybe that had the most variables of all.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#126

Post by perl4ever »

thaumasiotes wrote: post #49
semi-extrinsic wrote: post #45

Earlier quoted context omitted.

Nobody gets killed by Google Translate etc. messing up, so no sweat, throw together something that works 99% or the time. But for safety critical systems it's six nines reliability or GTFO.

> Nobody gets killed by Google Translate etc. messing up, so no sweat It's not so much that "nobody gets killed by Google translate messing up" as "when people get killed by Google Translate messing up, I can't tell".

I think Google Maps is more relevant to self driving. Google Maps has become, in my opinion, the best way to navigate, far superior to any other GPS-enabled option I know of. But the better it gets, the more obvious it is that you can't rely on it totally. If self-driving was feasible tomorrow or in the next decade, navigation would be much better. And if anyone could do it, Google would.

Think of the trope about idiots driving into lakes or off the road or whatever because they were following navigation. And now think about unconditionally trusting Google Maps or anything else, for say, a year. You know it's not up to that standard.

I think you can debate whether it needs to be an order of magnitude better, or multiple orders of magnitude better.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#127

Post by ClumsyPilot »

paxys wrote: post #119
rlayton2 wrote: post #113

Earlier quoted context omitted.

I've mentioned to a few people now that the first generation of self-driving cars don't scare me that much. It will be weird, but I assume that special care will be taken and the cars themselves will be quite risk-averse. I'm scared about the next generation of self-driving cars when we have "solved it" and the race is on to get cars out quickly and at low cost (and things like in-depth testing are out of budget). Ho…

IMO to be "scared" of self driving cars they need to be more dangerous than any other random car on the street today, and the bar for that is pretty low.

Nope, with other cars you know the factors that increase your risks (fog, drunk driving, distractions). You can make decisions, like not getting into a car with a drunk friend.

With machine learning you never know when it might mistake the back of a semi for an overpass or soemthing.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#128

Post by yeldarb »

shoeffner wrote: post #123
yeldarb wrote: post #120

Earlier quoted context omitted.

Hey, OP here, yeah you’re correct. The dataset doesn’t label any obstacles that small/far in the distance. I zoomed in on the region with errors for the sake of the screenshot. Here’s the original run through Google Vision AI. They actually don’t get the pedestrian either: https://imgur.com/a/84IVTV6 (I fired up the labeling tool I use and grabbed a recording of the few seconds of video around that frame to give an i…

Nice, thank you for the upload and thank you for clarifying that small objects are not labeled, that explains it. I was suprised because other images do contain rather small labels for traffic lights or even cars, but I guess it's always in the eyes of the person who labels the data. I think you did an amazing amount of work and huge improvements over the original, have you considered contributing the changes back up…

I plan to. They use a custom CSV format that my labeling tool can't work with so I converted everything to VOC XML. I need to write a script to convert back to their format to submit a PR.

Not sure if they'll accept the PR though; the original data had a "visualization link" back to the labeling company on each line which I can't reproduce.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#129

Post by perl4ever »

tomnipotent wrote: post #78
jwandborg wrote: post #69

Earlier quoted context omitted.

> Yes it's called statistics and probability theory. My understanding of statistics is: - I can halve the % insanity by adding another 100% of good labels. - If I want to reduce the insanity of labels to 1/33th of ~33% I need to add another 3200% of good labels. - If I want to reduce the insanity to 0% I need to balance the bad labels with an infinite amount of good labels. Is there anything I'm missing entirely exce…

You don't reach 0%, that's a straw man. The goal is better than human, and the 35,000+ vehicle-related fatalities that happen in the U.S. each year.

There's a disconnect here.

People who talk about the danger of humans driving cars always seem to talk about the raw numbers, because humans drive cars a lot and the raw numbers are rather large.

But when we talk about automated driving, it's in percentages, because it's not being done on the same scale.

So to compare apples to apples, you'd have to convert the number of fatalities to an accuracy percentage. Have you considered trying? There is certainly more than one way to do it, but it would greatly contribute to the discussion if you made some attempt.

Post reply on HN