Live data from Hacker News

Scenarios in which Tesla FSD Beta 9.0 fails

twitter.com

271–280 of 502 posts

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#271
post #183

Earlier quoted context omitted.

Imagine if algorithmic accidents had biases. For example, let's say a car tended to crash into children (maybe they are harder to detect with cameras), more often than adults. This type of algorithmic bias would be unacceptable no matter how safe FSD were on aggregate. So you're right, the only bar to reach is perfection (which is impossible), because algorithmic errors have biases that will likely deviate from human…

> let's say a car tended to crash into children (maybe they are harder to detect with cameras), more often than adults This is already true today of human drivers because of the tall SUVs that are so popular. Do you think matching biases will be acceptable?

That's not true.

First, the vast majority of pedestrian deaths are adults. In 2018, a total of 206 age 15 or younger were killed by cars. Compare that to 5,965 killed who were age 16 or older.[1] Both in absolute numbers and relative to population, children are far less likely to be run over and killed than adults.

Second, while light trucks (vans, SUVs, & pickups) are 1.45x more deadly to pedestrians than cars, buses are far more dangerous than either. Motorcycles (which have excellent visibility) are particularly deadly to child pedestrians. From United States pedestrian fatality rates by vehicle type[2]:

> Compared with cars, the RR of killing a pedestrian per vehicle mile was 7.97 (95% CI 6.33 to 10.04) for buses; 1.93 (95% CI 1.30 to 2.86) for motorcycles; 1.45 (95% CI 1.37 to 1.55) for light trucks, and 0.96 (95% CI 0.79 to 1.18) for heavy trucks. Compared with cars, buses were 11.85 times (95% CI 6.07 to 23.12) and motorcycles were 3.77 times (95% CI 1.40 to 10.20) more likely per mile to kill children 0–14 years old. Buses were 16.70 times (95% CI 7.30 to 38.19) more likely to kill adults age 85 or older than were cars. The risk of killing a pedestrian per vehicle mile traveled in an urban area was 1.57 times (95% CI 1.47 to 1.67) the risk in a rural area.

All else equal, being hit by a larger vehicle does increase the risk of severe injury or death, but all else isn't equal. Larger vehicles tend to be more visible, louder, and slower than their smaller counterparts. Different types of vehicles are driven in different environments with different propensities for mingling with pedestrians. If vehicle mass and blind spots were the main factors in pedestrian deaths, we should have seen deaths skyrocket over the past 40 years (as cars got bigger and bulkier for greater passenger safety). Instead we saw pedestrian deaths decrease.

1. https://docs.google.com/spreadsheets/d/e/2PACX-1vRqGqodKkWkS...

2. https://injuryprevention.bmj.com/content/11/4/232

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#272
post #11

I don't understand how something this broken is allowed to operate on public roads. If I drove like Tesla's FSD seems to based on the videos I've seen, I'd be pulled out of my car and arrested on (well founded) suspicions of "driving while hammered." After a decade of work, it's not capable of doing much beyond "blundering around a city mostly without hitting stuff, but pay attention, because it'll try to hit the mos…

> I don't understand how something this broken is allowed to operate on public roads. It's important to point out that this software is currently only offered as a private beta to deliberately selected testers. Now, maybe they shouldn't be using it on public roads either, but at least it's not available to the general public.

As long as all the people in traffic with this experiment signed this agreement as well, all is good.

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#273

Earlier quoted context omitted.

Right, so what happens when one says wall, and one says clear air? You brake? That is what happens now with phantom braking. Approaching a bridge? camera sees air, radar sees wall, slam on brakes. You want the best sensor type over a high-fidelity sensor and a lower fidelity sensor. The Tesla system has 8 cameras (3 forward), so they def have overlap between what they are considering better cameras. Time will prove w…

The post I was responding to said lidar, not radar. But if you want to switch to radar, we can talk about that too. > camera sees air, radar sees wall, slam on brakes Seeing bridges as walls is not a fundamental property of radar. That's an implementation problem. If cars are doing radar poorly, maybe the fix is to start doing it less poorly instead of throwing it away entirely.

Well, that was specifically about blending two different sensors with different characteristics. For you walking, it would be like blending your eyes with your nose. If your eyes tells you the floor is safe, and your nose smells something bad, do you stop? Anytime you have two different sensors with different characteristics, you want "the best". Your body uses your eyes to see where to walk, and your nose to test if pizza is rotten. Blending multiple sensor types is tricky.

So back to LIDAR.. same difference. Camera and LIDAR have different profiles. I think it's fine to use either, but I think trying to blend the two is a sub-optimal solution to the problem.

Again, this is my guess from what I know. I could be wrong, and the winning technology could use 12 different sensors (vision + radar + lidar + smell + microphones), and blend them all to drive. Cool if someone pulls it off! But if I had to do it myself or place a bet, I would put it on a single sensor type.

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#274

Earlier quoted context omitted.

Why not have redundant sensors and crosscheck them? Boeing 737 Max had only one AOA sensor [1], and that wasn't a great idea. https://www.cnn.com/2019/04/30/politics/boeing-sensor-737-ma...

Right, so what happens when one says wall, and one says clear air? You brake? That is what happens now with phantom braking. Approaching a bridge? camera sees air, radar sees wall, slam on brakes. You want the best sensor type over a high-fidelity sensor and a lower fidelity sensor. The Tesla system has 8 cameras (3 forward), so they def have overlap between what they are considering better cameras. Time will prove w…

If you have redundant AOA sensors on a plane and they disagree what do you do? Alert the pilot. You have to do the same on a self-driving car as well. You can't just ignore a serious malfunction, or pretend to not see it just because you don't know to handle it!

To be truly redundant you have to use different technologies, such as camera and lidar.

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#275

Earlier quoted context omitted.

Right, so what happens when one says wall, and one says clear air? You brake? That is what happens now with phantom braking. Approaching a bridge? camera sees air, radar sees wall, slam on brakes. You want the best sensor type over a high-fidelity sensor and a lower fidelity sensor. The Tesla system has 8 cameras (3 forward), so they def have overlap between what they are considering better cameras. Time will prove w…

> Approaching a bridge? camera sees air, radar sees wall, slam on brakes. That's simplifying the situation a bit too much. The camera can give more results than air/not-air. Specifically in this case it could detect a bridge. Same applies to the radar really - you'll get measurements from multiple heights which would tell you that it may be an inclined street, not a wall.

I think you are missing parts. Have you watched this video from someone actually working in the field?

https://www.youtube.com/watch?v=eOL_rCK59ZI&t=28286s

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#276
post #77

Earlier quoted context omitted.

>This really isn't beta grade software, as it isn't feature complete as the failure scenarios in the video clearly show. I think it depends what they actually are trying to accomplish. This is Beta for a glorified cruise control overhaul; not a beta for promised RoboTaxi. Musk/Tesla tend to talk about RoboTaxi then slip seemlessly into/out of 'but today we have low engagement cruise control!'. Fair bit of hucksterism…

> I think it depends what they actually are trying to accomplish Good point. "Full Self Driving" in my mind paints a picture beyond "a better cruise control". But maybe they meant that and just named it wrong.

From Tesla's webpage:

> Full Self-Driving Capability

>

> All new Tesla cars have the hardware needed in the future for full self-driving in almost all circumstances. The system is designed to be able to conduct short and long distance trips with no action required by the person in the driver’s seat.

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#277

Earlier quoted context omitted.

Right, so what happens when one says wall, and one says clear air? You brake? That is what happens now with phantom braking. Approaching a bridge? camera sees air, radar sees wall, slam on brakes. You want the best sensor type over a high-fidelity sensor and a lower fidelity sensor. The Tesla system has 8 cameras (3 forward), so they def have overlap between what they are considering better cameras. Time will prove w…

If you have redundant AOA sensors on a plane and they disagree what do you do? Alert the pilot. You have to do the same on a self-driving car as well. You can't just ignore a serious malfunction, or pretend to not see it just because you don't know to handle it! To be truly redundant you have to use different technologies, such as camera and lidar.

> If you have redundant AOA sensors on a plane and they disagree what do you do? Alert the pilot.

Right, which means it's not a solution for L4/L5 autonomy, only for L2. Tesla is trying to reach L4/L5, so just alerting the pilot is not satisfying the design goal.

> To be truly redundant you have to use different technologies, such as camera and lidar.

I think that is an opinion and not a fact. Watch a video such as

https://www.youtube.com/watch?v=eOL_rCK59ZI&t=28286s

from someone working on this problem

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#278

Earlier quoted context omitted.

The monorail video is jaw-dropping. Nine versions in, I would expect ongoing challenges with things like you mention. But continued failure to even see large, flat obstacles is no longer something that needs to be fixed – that it has persisted this long (even after killing someone as in the case of T-boning a semi trailer at highway speeds) is an indictment of the entire approach Tesla has been taking to FSD. I used…

The monorail video is jaw-dropping. Yes. Pause the video and look at the car's screen. There's no indication on screen of the columns. A car on the other side of the row of columns is recognized, but not the columns. It's clear that Tesla has a special-purpose recognizer for "car". The columns are a solid obstacle almost impinging into the road, one that doesn't look like a car. That's the standard Tesla fail. Most h…

> As I've been saying for years now, the first job is to determine that the road ahead is flat enough to drive on. Then decide where you want to drive. I did the DARPA Grand Challenge 16 years ago, which was off-road, so that was the first problem to solve. Tesla has lane-following and smart cruise control, like other automakers, to which they've added some hacks to create the illusion of self-driving. But they just don't have the "verify road ahead is flat" technology.

This matches my perception as well and it continues to blow my mind. Like it just seems like it would require full-on incompetence to use only specific pedestrian/car/sign/lane classifiers rather than splitting the world first into ground/obstacle by reconstructing geometry and then assigning relative velocity, acceleration, and curvature to coherent obstacle segments irrespective of their identification. But the videos always make it look like this is exactly what's happening. And worse it always appears to be happening on an instantaneous frame-by-frame basis with things flickering in and out of existence as if "thing there" and "nothing there" are somehow equally safe guesses.

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#279

Earlier quoted context omitted.

They should be trivially handled by stereopsis and structure from motion as well. Stereo+time photogrammetry has been solved well enough to not steer directly towards large obstacles for decades. Overreliance on machine learning pixel models to classify everything in view is the real problem.

Why not have redundant sensors and crosscheck them? Boeing 737 Max had only one AOA sensor [1], and that wasn't a great idea. https://www.cnn.com/2019/04/30/politics/boeing-sensor-737-ma...

>Boeing 737 Max had only one AOA sensor

Just a small nit-pick but it makes the case against Boeing worse. The airframe had multiple AOA sensors but the base software only used one sensor reading. Note the image in [1] shows readings from both a "left" and "right" AOA. From your link:

>software design for relying on data from a single AOA sensor

Boeing sold a software upgrade to read both AOA devices. (This still leaves the problem that if the two AOAs disagree there might be cases where you don't know which is bad). The fact that they listed MCAS as 'hazardous' rather than 'catastrophic' means it was allowed to have a single point of failure. It also means they may not have fully understood their own design.[1]

[1] https://www.seattletimes.com/business/boeing-aerospace/black...

Re: Scenarios in which Tesla FSD Beta 9.0 fails

#280
post #120

Many of the comments here seem a bit... unfair to me, considering that these clips were handpicked. I watched (and fast-forwarded) through a few of the original, full-length videos from which these clips were taken. The full-length videos show long drives (in some cases, hours long) almost entirely without human intervention, under conditions explicitly meant to be difficult for self-driving vehicles. One thing I rea…

If this happened all in one month of constant driving, I'd say it isn't fit even for limited closed testing in public traffic. It should be back at the closed circuit with inflatable cars. If it was cut down from just one or a few days of driving that's horrifying.
Post reply on HN