It is a self-correcting problem: these pedestrains won’t be present in the next dataset.
A popular self-driving car dataset is missing labels for hundreds of pedestrians
11–20 of 202 posts
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#12This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…
I understand your concern and share it myself. This is an important time and we should be really careful training these things. However, training as used in the real world isn't on a still frame only basis, it's used in sequence. And while a single frame might be missing a label, I bet that at-speed most everything important gets labeled correctly enough to be better than a distracted human driver, or the average hum…
>And while a single frame might be missing a label, I bet that at-speed most everything important gets labeled correctly enough to be better than a distracted human driver, or the average human driver for that matter.
Would you bet a family member? That a distracted driver is a hazard does not mean other drivers are safe, or even safer.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#13Well, if an autonomous vehicle outfit were running unmonitored Level 4 vehicles on public roads using only an open source data set I'd be worried. Even if it was labelled thoroughly and correctly, there isn't nearly enough data in any open source dataset to train an autonomous vehicle perception system that can operate safely without human supervision. This is not a safety critical issue.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#14This seems like the kind of problem that is suitable for and important enough to throw hundreds or thousands of (volunteer?) people at. Imagine assembling a massive public corpus of human verified training data - we'd collectively be that much closer to a technology which will change society! This problem is screaming for a consortium effort on behalf of major corporate entities in the space, who could each benefit without revealing internal secrets.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#15This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…
> This is really scary.
No, it's not even remotely "really scary". No one is putting an actual self-driving car on the market using this specific data set. Disingenuous to pretend this is any indication of the data using by serious companies in the space or is represented of the impact a few mislabelled samples have on the ability of these systems & algorithms to generalize.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#16This seems like the kind of problem that is suitable for and important enough to throw hundreds or thousands of (volunteer?) people at. Imagine assembling a massive public corpus of human verified training data - we'd collectively be that much closer to a technology which will change society! This problem is screaming for a consortium effort on behalf of major corporate entities in the space, who could each benefit w…
If you've ever had recaptcha ask you to identify stop signs, traffic lights, buses etc you're helping improve Waymo's data set for free :) Maybe it'll be shared with the public instead of used for profit in a decade or two.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#17I work in the AV space. There's a lot of ambiguity in labeling. How should a crowd of people be annotated? A line of parked cars? A photograph of a car? Examine the training set!
A line of parked cars absolutely needs to be labeled as individual cars. Any one of them could pull out in front of you at any moment.
Indeed, whilst additionally presenting the chance of a door opening or an obscured pedestrian stepping out from between them.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#18These are manually tagged stills, right? Not video? That's a data set for training CAPCHA breakers, not self-driving. You need to use video, where you get to see the same objects at different ranges. Recognition gets better as you get closer. Then track the recognized objects backwards to when they first appear, and try to recognize them at smaller sizes.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#19This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…
I understand your concern and share it myself. This is an important time and we should be really careful training these things. However, training as used in the real world isn't on a still frame only basis, it's used in sequence. And while a single frame might be missing a label, I bet that at-speed most everything important gets labeled correctly enough to be better than a distracted human driver, or the average hum…
At what framerate? With 30 fps, a 60mph vehicle advances about 5 feet per frame. Every incorrectly labeled frame delays the reaction by a nontrivial amount.
My intuition is that your trained accuracy will not exceed the accuracy of the training set. This is literally a matter of life and death; every frame matters.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#20It is a self-correcting problem: these pedestrains won’t be present in the next dataset.
People should learn not to go outside if they're not labelled.