A popular self-driving car dataset is missing labels for hundreds of pedestrians
1–10 of 202 posts
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#2This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images.
Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not being labeled) but some are egregious (like the woman in the crosswalk with a baby stroller).
I think this really calls out the importance of rigorously inspecting any data you plan to use with your models. Garbage in, garbage out... and self-driving cars should be treated seriously.
I went ahead and corrected by hand the missing bounding boxes and fixed a bunch of other errors like phantom annotations and duplicated boxes. There are still quite a few duplicate boxes (especially around traffic lights) that would have been tedious to fix manually, but if there's enough demand I'll go back and clean those as well.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#3Until self driving is possible with purely unsupervised data, self driving cars and other algorithms relying on massive datasets are unlikely to get really strong because label inaccuracies like this are inevitable
Also, far worse has been found in large scale datasets. Pretty sure there was CP found somewhere in imagenet
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#4I work in the AV space. There's a lot of ambiguity in labeling.
How should a crowd of people be annotated? A line of parked cars? A photograph of a car?
Examine the training set!
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#5What's the dataset size? If there's billions of pedestrians tagged and a couple hundreds are missing, would it actually have a big impact on the training? Also looks like a lot of the people in those images are off the road. What's the current standard in AV? Is everything tagged on the sidewalk? (Genuine questions)
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#6This is really scary. I discovered this because we're working on converting and re-hosting popular datasets in many popular formats for easy use across models... I first noticed that there were a bunch of completely unlabeled images. Upon digging in, I was appalled that fully 1/3 of the images contained errors or omissions! Some are small (eg a part of a car on the edge of the frame or a ways in the distance not bein…
I understand your concern and share it myself. This is an important time and we should be really careful training these things.
However, training as used in the real world isn't on a still frame only basis, it's used in sequence. And while a single frame might be missing a label, I bet that at-speed most everything important gets labeled correctly enough to be better than a distracted human driver, or the average human driver for that matter.
Then, of course, my intuition might be incorrect.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#7What's the dataset size? If there's billions of pedestrians tagged and a couple hundreds are missing, would it actually have a big impact on the training? Also looks like a lot of the people in those images are off the road. What's the current standard in AV? Is everything tagged on the sidewalk? (Genuine questions)
From the article:
>Perhaps most egregiously, 217 (1.4%) of the images were completely unlabeled but actually contained cars, trucks, street lights, and/or pedestrians.
This gives us a data set size of 15500.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#8What's the dataset size? If there's billions of pedestrians tagged and a couple hundreds are missing, would it actually have a big impact on the training? Also looks like a lot of the people in those images are off the road. What's the current standard in AV? Is everything tagged on the sidewalk? (Genuine questions)
If they are reporting a 33% image error rate, I would expect a large effect on accuracy no matter what the individual annotation error rates were.
I am unfamiliar with the detailed tagging standards but that also seems irrelevant until the more egregious problems are resolved. Get everything untagged first, then look for smaller scoped issues. And thanks Roboflow for doing whatever amount of this.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#9I work in the AV space. There's a lot of ambiguity in labeling. How should a crowd of people be annotated? A line of parked cars? A photograph of a car? Examine the training set!
A line of parked cars absolutely needs to be labeled as individual cars. Any one of them could pull out in front of you at any moment.
Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians
#10I work in the AV space. There's a lot of ambiguity in labeling. How should a crowd of people be annotated? A line of parked cars? A photograph of a car? Examine the training set!
Are there any studies on how this affects recognition? I'm labeling some vehicles and will my neural net learn better (for say vehicle identification) from a row of identical vehicles, individual vehicles with overlap, non-overlapping parts only?