A popular self-driving car dataset is missing labels for hundreds of pedestrians • Hacker News Board

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#191

Post by notahacker » Thu, Feb 13, 2020, 10:37 AM UTC

JoeAltmaier wrote: post #188

notahacker wrote: post #180

Earlier quoted context omitted.

I mean, the 'half a dozen reasonable measures' is a problem, not a solution, when they're not all saying the same thing. And sure, it only takes days before we know the latest version of the software actually isn't safer than the average human. And a lot of unnecessary deaths, and the likelihood the fix will cause other unnecessary deaths instead [maybe more, maybe less]. It's frankly sociopathic to dismiss the possi…

Straw man? There are many phases to testing a new piece of software, short of deploying everything to the field indiscriminately. Some of us believe (perhaps wrong but there it is) that the human error rate will be trivially easy to improve upon. That's not sociopathic. It would be unhelpful to dismiss this innovation (self-driving cars) because of FUD.

Some of us believe, based on the evidence that the human fatal error rate is as low as 3 per billion miles driven in many countries, and some people actually are better than average drivers. Might be trivially easy to improve upon human ability to not to dent cars whilst parking or observe speed limits, but you're going to struggle to argue that improving on the fatal error rate is trivially easy for AI, or that the insurance cost of the dents matters more than the lives anyway.

People who actually want initiatives to succeed are going to have to do better than sneering dismissal in response to anybody people pointing out obvious facts that complex software seldom runs for a billion hours without bugs and successfully overfitting to simulation data in a testing process doesn't mean a new iteration of software will handle novelty it hasn't been designed to solve less fatally than humans over the billions of real world miles we need to be sure.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#192

Post by dTal » Thu, Feb 13, 2020, 1:17 PM UTC

TomVDB wrote: post #101

SolaceQuantum wrote: post #87

Earlier quoted context omitted.

"The simple idea is to do stuff like train your model on randomized subsets of your data and then compare its performance to using all the data you have." How do you do this when you cannot verify that your data, in subset or in whole, is accurate? And furthermore you don't know how inaccurate it is?

My intuitive take on it: You train on a subset of the initial data. Even if the data has a certain number of incorrect frames, it should still do a decent job getting a lot of things right. Then you manually loop through all the images of the data set for which the network has detected something that isn't present in the annotations (and vice versa). If the network correctly identified a missing item that wasn't in t…

I've been thinking a lot about this sort of thing lately, and isn't it the case that ideally you shouldn't need to manually confirm or reject mismatches? If the learning program maintains a probability density for "training labels are wrong", definitive ground truth should be unnecessary - eventually it will figure out the mismatches by itself.

As I understand it, this is the core of recursive Bayesian estimation. At the end of the day we don't really have ground truth for anything - it's all filtered through senses with error bars. So any learning process needs to be robust to that.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#193

Post by JoeAltmaier » Thu, Feb 13, 2020, 3:54 PM UTC

notahacker wrote: post #191

JoeAltmaier wrote: post #188

Earlier quoted context omitted.

Straw man? There are many phases to testing a new piece of software, short of deploying everything to the field indiscriminately. Some of us believe (perhaps wrong but there it is) that the human error rate will be trivially easy to improve upon. That's not sociopathic. It would be unhelpful to dismiss this innovation (self-driving cars) because of FUD.

Some of us believe, based on the evidence that the human fatal error rate is as low as 3 per billion miles driven in many countries, and some people actually are better than average drivers. Might be trivially easy to improve upon human ability to not to dent cars whilst parking or observe speed limits, but you're going to struggle to argue that improving on the fatal error rate is trivially easy for AI, or that the…

People CAN drive well. But understand in my rural state the highway department has signs over the road, showing fatalities for the year. It averages one a day. I don't think the cancer patients in the hospital die that frequently.

So you can name-call all you like and disparage dialog because you disagree or whatever. But I don't think a billion miles between accidents is anywhere close to what I see every day.

FUD isn't a position, its got no place in this public-safety discussion.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#194

Post by jwandborg » Thu, Feb 13, 2020, 7:06 PM UTC

tomnipotent wrote: post #77

jwandborg wrote: post #66

Earlier quoted context omitted.

That's correct. I know what goes in and what comes out, not what happens in the middle. How does ~33% insanity in become Edit: Parent was edited, was previously (paraphrased) > I'm guessing you have no technical understanding of how this works

How does making up something ridiculous like "33% insanity" give you anything that's resembles a subject that we can discuss? Hyperbole in, hyperbole out.

I'm 33% insane myself. I believe that's part of what makes me human.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#195

Post by jwandborg » Thu, Feb 13, 2020, 7:51 PM UTC

tomnipotent wrote: post #78

jwandborg wrote: post #69

Earlier quoted context omitted.

> Yes it's called statistics and probability theory. My understanding of statistics is: - I can halve the % insanity by adding another 100% of good labels. - If I want to reduce the insanity of labels to 1/33th of ~33% I need to add another 3200% of good labels. - If I want to reduce the insanity to 0% I need to balance the bad labels with an infinite amount of good labels. Is there anything I'm missing entirely exce…

You don't reach 0%, that's a straw man. The goal is better than human, and the 35,000+ vehicle-related fatalities that happen in the U.S. each year.

It's hard to reach 0% bad labels because:

1. You can't have an infinite amount of good labels 2. Humans are in charge of labeling too.

The question is if you can reliably overcome the number of bad labels in your training set, so that 33% of bad labels equates to <33% "insanity" in the system.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#196

Post by jwandborg » Thu, Feb 13, 2020, 7:55 PM UTC

diffeomorphism wrote: post #167

jwandborg wrote: post #69

Earlier quoted context omitted.

> Yes it's called statistics and probability theory. My understanding of statistics is: - I can halve the % insanity by adding another 100% of good labels. - If I want to reduce the insanity of labels to 1/33th of ~33% I need to add another 3200% of good labels. - If I want to reduce the insanity to 0% I need to balance the bad labels with an infinite amount of good labels. Is there anything I'm missing entirely exce…

Your understanding is wrong for anything nonlinear. The whole reason machine learning is useful is because it is nonlinear.

How nonlinear are we talking? My understanding is probably closer to the truth than to the opposite of the truth. I'm looking for an estimate of how far from the truth I am.

How would a system reliably discredit missing labels while still learning from good labels? The simplest solution would be that system is able to spot the bad/missing labels itself with some certainty, but that seems like a catch 22.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#197

Post by jwandborg » Thu, Feb 13, 2020, 7:58 PM UTC

perl4ever wrote: post #125

jwandborg wrote: post #32

Earlier quoted context omitted.

I think the point is rather that it's an unpredictable-stakes game. That is, if you make few enough mistakes, you'll be fine for a long time until you make the wrong mistake at the wrong time and it kills somebody. Maybe a better analogy is driving a car, that is, nuclear power and self-driving cars are both vaguely like driving a car. However, my intuition is that nuclear power has less variables than driving a car.…

"However, my intuition is that nuclear power has less variables than driving a car." During the cold war they launched fission reactors into orbit. That probably had more variables. Then again, since the nuclear airplane project was unsuccessful, maybe that had the most variables of all.

Well, nuclear power isn't harder because you put it in a car, but it's a great way to add to the complexity of the system as a whole.

Putting a nuclear reactor in space (as we still do, I think the most recent Mars rover had one) greatly reduces the risk of it directly causing deaths, since it's really far away from humans for most of its service life.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#198

Post by MauranKilom » Sat, Feb 15, 2020, 2:23 AM UTC

perl4ever wrote: post #157

tomnipotent wrote: post #150

Earlier quoted context omitted.

The average of something cannot be more than the average of... itself. Thus, "Humans are much safer than people on average" is nonsensical. > Why do you believe that Autopilot outperforms humans in comparable conditions? Because they have the data that proves it? > I'm extremely prejudiced against them And I've chosen to take them at face value with a grain of salt, and to believe that for the data they've collected…

"Thus, "Humans are much safer than people on average" is nonsensical." Does it make more sense as "Humans, when driving in conditions suitable for Autopilot, are much safer than people on average"? "I've chosen to take them at face value with a grain of salt, and to believe that for the data they've collected from the hundreds of thousands of Tesla's with millions of hours of data using Autopilot, it's fair to say th…

> Humans, when driving in conditions suitable for Autopilot, are much safer than people on average

The immaterial distinction between "humans" and "people" still makes that sentence confusing. I take it that you mean "driving a mile (either as human or autopilot) is safer in conditions that are good for autopilot than driving a mile in average conditions"? Or more directly, isn't your question really "are the conditions the same for the averaged human drivers and the averaged autopilots"?.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#199

Post by MauranKilom » Sat, Feb 15, 2020, 2:32 AM UTC

dTal wrote: post #192

TomVDB wrote: post #101

Earlier quoted context omitted.

My intuitive take on it: You train on a subset of the initial data. Even if the data has a certain number of incorrect frames, it should still do a decent job getting a lot of things right. Then you manually loop through all the images of the data set for which the network has detected something that isn't present in the annotations (and vice versa). If the network correctly identified a missing item that wasn't in t…

I've been thinking a lot about this sort of thing lately, and isn't it the case that ideally you shouldn't need to manually confirm or reject mismatches? If the learning program maintains a probability density for "training labels are wrong", definitive ground truth should be unnecessary - eventually it will figure out the mismatches by itself. As I understand it, this is the core of recursive Bayesian estimation. At…

Sure, you can modify your model to better account for bad training data. But you could also fix the training data. The previous comment pointed out that fixing the training data (to a high degree at least, using the predictions of an intermediate version) is significantly faster than the initial labeling.

Re: A popular self-driving car dataset is missing labels for hundreds of pedestrians

#200

Post by perl4ever » Sat, Feb 15, 2020, 6:15 AM UTC

MauranKilom wrote: post #198

perl4ever wrote: post #157

Earlier quoted context omitted.

"Thus, "Humans are much safer than people on average" is nonsensical." Does it make more sense as "Humans, when driving in conditions suitable for Autopilot, are much safer than people on average"? "I've chosen to take them at face value with a grain of salt, and to believe that for the data they've collected from the hundreds of thousands of Tesla's with millions of hours of data using Autopilot, it's fair to say th…

> Humans, when driving in conditions suitable for Autopilot, are much safer than people on average The immaterial distinction between "humans" and "people" still makes that sentence confusing. I take it that you mean "driving a mile (either as human or autopilot) is safer in conditions that are good for autopilot than driving a mile in average conditions"? Or more directly, isn't your question really "are the conditi…

"isn't your question really "are the conditions the same for the averaged human drivers and the averaged autopilots"?"

My implication is that I severely doubt the conditions are the same, when somebody touts a comparison, and I would need clear and convincing evidence otherwise to change my mind. As well as strong evidence of good intent and trustworthiness by the source of the information.

It's not just about being intentionally deceitful, but about the fact that it's hard to do the right comparison, so people feel justified in giving up on it.