Live data from Hacker News

Former Tesla Firmware Engineer Discusses the System

twitter.com

321–330 of 589 posts

Re: Former Tesla Firmware Engineer Discusses the System

#321
post #175

I have a Model S, and Tesla pushes buggy firmware to it last week. It causes the entire instrument cluster, including the speedometer, to disappear periodically while driving. Tesla knows about the bug and it’s apparently a “high priority”. It’s not entirely clear to me that they are capable of rolling back the update.

I join the III family this coming week. I am really interested in how well the updates are managed as the idea of cratering my car isn't something anyone is keen about.

can you as an owner delay updates?

Re: Former Tesla Firmware Engineer Discusses the System

#322
post #256

Earlier quoted context omitted.

As a person who used to write navigation and control software for autonomous vehicles who occasionally gets downvoted for telling people that you really don't need a fancy RTOS for this stuff, you really don't need a fancy RTOS for this stuff. Linux is a very common platform for highly responsive robotic systems. I promise that their pid control isn't an Electron app.

Is that like, you really don't need a fancy RTOS for this stuff 99.999% of the time, although sometimes you do? Or truly, despite the life safety element, there is never any need for RTOS.

In every application I can think of off the top of my head, but mostly in the ones that apply to Tesla, you truly don't, except when the law or a contract says otherwise. I'm sure there are exceptions for things that don't apply to Tesla (or for that matter SpaceX).

Here's what's inside of every autonomous vehicle ever made: a message-passing subsystem, sensors, fusers, navigation, dynamic control, actuator device drivers, and thruster device drivers.

Sensors measure things and emit readings. Your most expensive, highest frequency general purpose sensors emit new readings at something too fast for a human but hella slow for a computer, like 100Hz-5KHz. Your common sensors, a video camera for instance, don't get even close to that. These sensors are often connected, even today because milspec companies hate modernity, via RS-232 serial cables. For those younger than 30, RS-232 is what non-Apple computers used for non-keyboard/mouse peripherals prior to the introduction of the first iMac in 1998 because USB didn't really take off until then.

Sensors send their readings via the message-passing subsystem to fusers.

Fusers take the readings from the sensors and, hur hur, "fuse" them together into a description of where the vehicle is and what the environment is like. This usually involves something like a kalman filter. Fusing even your very fastest sensors, the 5KHz IMUs of the world, is just a small bit of math and basically takes no time at all.

Fusers send their fused states via the message-passing subsystem to navigation.

Navigation takes the fused sense of self and the world and decides which direction to head and how fast to go. The objective could be something like hitting route waypoints or it could be something like staying in a lane and not being rear-ended and avoiding obstacles. Car navigation probably doesn't act on new input more frequently than 100Hz, you certainly can't act on new input more frequently than 100Hz, and it takes basically no time at all.

Navigation sends its directives via the message-passing subsystem to dynamic control.

Dynamic control takes navigation's "which way" and "how fast" directives and turns them into more realistic short-term goals accounting for hysteresis and other physical limitations of the system like minimum turn radius. This is just a small bit of math and basically takes no time at all.

Dynamic control sends its directives via the message-passing subsystem to the actuator and thruster drivers.

Actuator drivers convert dynamic control's "go more left" message into trying to go more left.

Thruster drivers convert dynamic control's "go more fast" message into trying to go more fast.

Actuator and thruster drivers send readings (hopefully) from the actuators and thrusters, because those are also sensors, back to dynamic control and fusion.

Sensors feed into fusers, fusers feed into nav, nav feeds into dynamic control, dynamic control feeds into actuation and thrust. When you have new data, you do something new with it which is technically doing the same old thing with it and just producing new output.

Now there aren't that many sensors. There are way fewer fusers. There's only one navigation. There's probably only one dynamic control, though there could be a couple.

Anything else that I haven't already described, like Waymo's machine learning object classifying 4D mustache adding hotdog detectors, are just sensors and fusers sitting on their own computers feeding new lat/lng/heading/speed to navigation at a rate that is hella slow for a computer. And for sure Waymo's convolutional neural network middle-out jaywalking yoga mom detector takes a lot of processing, but it's running on its own computer, not competing for resources, and emitting its fused readings at some hella slow for a computer rate.

Re: Former Tesla Firmware Engineer Discusses the System

#323
post #307

Let's look at the facts: a) Tesla is the most shorted stock in history, b) 1000s of hackers are working hard to break things like this, c) if it was possible to break it and damage the company someone would have done it already. This just feels like an disgruntled employee.

Let's work the opposite way too

a) car manufacturing is one of the most competitive industries, b) 1000s of engineers are working on making better cars, c) if it was possible to make profitable, affordable, self-driving electric cars, somebody would have done it. Comes off as a snake oil salesman.

Re: Former Tesla Firmware Engineer Discusses the System

#324

Earlier quoted context omitted.

resume-driven development?

everything is revenue driven development

he said resume, i.e. working with tech you want to use at your next gig not the tech you need at this gig.

Re: Former Tesla Firmware Engineer Discusses the System

#325
post #191

The most disturbing part of this, for me, is this line in the very first tweet: "... caused almost the entire fleet to reboot loop ..." I am not interested in being part of someone's fleet . The fact that they use this language at all to describe an end-user who has purchased an automobile suggests that their expectations and my own - of what it means to purchase and operate a car - are in deeply (possibly dangerousl…

Your post conveniently ignores the fact that over the last 20 years cars and trucks have become physical delivery vehicles for software. In fact, the complexity of the software running in any manufacturer's modern car dwarfs the complexity of the hardware in the car itself. So, if you have a car manufactured any time in the last 25 years, you're running software that hasn't likely been patched in years, has a ton of…

This is why I do not have a car that was manufactured any time in the last 25 years. You can't trust software, and you definitely can't trust software developers.

Re: Former Tesla Firmware Engineer Discusses the System

#326

Earlier quoted context omitted.

> the rockets fly with a heavily customized Linux install My jaw hit the floor the first time I heard this. Why Linux instead of an RTOS?? Apparently Tesla's autopilot also runs Linux, which seems like a huge accident waiting to happen (pun intended).

> Why Linux instead of an RTOS? Same reason SpaceX eschews radiation-hardened processors for redundant off-the-shelf cores: supplier competition. There aren't many RTOS engineers on the market; there are many Linux engineers. Once they got over the cost of hardening the kernel, SpaceX found itself at a scaling advantage versus RTOS-based competitors.

> There aren't many RTOS engineers on the market; there are many Linux engineers.

There aren't many Linux engineers who have experience with resource-constrained systems or real-time programming requirements.

Re: Former Tesla Firmware Engineer Discusses the System

#327

Content aside, what is the deal with images of screenshots of a forum, then posted to Twitter? Why not include a fax machine in there while you’re at it? Just link to the forum and we can all read the text.

They periodically have times when the forum or at least some subforums are viewable by registration only.

Re: Former Tesla Firmware Engineer Discusses the System

#328

Earlier quoted context omitted.

The iPhone has to reliably connect for emergency calls and provide accurate location. That can be life or death. In fact, "Apple isn't ready to engineer phones at a life-or-death standard" was a fairly common critique of the iPhone in the early days. In response, Apple ran a little PR campaign around their radio engineering efforts--they had a web page that showed all the cool-looking rooms for testing radios, which…

Didn't the iPhone use a dual processor set up like the other manufacturers at the time? One ui/app processor and a separate radio processor stack?

There's a lot of EE black magic around antenna design.

Re: Former Tesla Firmware Engineer Discusses the System

#329

Earlier quoted context omitted.

Agree completely. Hard real time is possible with Linux, we use it for sub millisecond control of Traffic Lights. The only issue we ever hit is proving that the code running is the stuff we expected to run.

On a tangent, but still germane to the thread: Why arent their efforts to create "autonomous only" traffic managment scenarios, where people drive into a given, known area, and the area then takes control of managing the traffic and vehicles. Such that you relinquish control of the vehicle to that area's control system, with your destination stated and then your vehicle is managed accordingly. For example, a parking…

If it were that easy to setup tech-based infrastructure, we would have had positive train control implemented years ago.

Re: Former Tesla Firmware Engineer Discusses the System

#330
post #307

Let's look at the facts: a) Tesla is the most shorted stock in history, b) 1000s of hackers are working hard to break things like this, c) if it was possible to break it and damage the company someone would have done it already. This just feels like an disgruntled employee.

Counterpoint: when you ignore Musk’s Twitter tirades about shorts, and his uberfans’ subsequent aping of those tirades a simple truth remains: Tesla is one of the most overvalued stocks in history, and far from being mustache-twirling villains with limitless resources and no morals, people shorting TSLA are doing so in anticipation of reality catching up with the listed price. They’re not undermining Tesla, they’re not hoping for it to fail, they’re just waiting for the stock price and the reality of the company to intersect. It would be a lot less frustrating talking to hardcore Tesla/Musk fans if you didn’t have to wade through conspiracy theories to do it.
Post reply on HN