Tesla can SSH into a customer's car? What could possibly go wrong?
Former Tesla Firmware Engineer Discusses the System
171–180 of 589 posts
Re: Former Tesla Firmware Engineer Discusses the System
#172This was the most intense and compelling piece of writing I've read all year. I love hearing this gnarly warts & all reporting from the front lines. Guy sounded really smart, able to wrap his head around so many types of systems and roles in the company.
Re: Former Tesla Firmware Engineer Discusses the System
#173Earlier quoted context omitted.
> Why Linux instead of an RTOS? Same reason SpaceX eschews radiation-hardened processors for redundant off-the-shelf cores: supplier competition. There aren't many RTOS engineers on the market; there are many Linux engineers. Once they got over the cost of hardening the kernel, SpaceX found itself at a scaling advantage versus RTOS-based competitors.
Not just that but holy shit some of those commercial RTOSes have major issues. I work in aerospace and we recently used one where the whole system would crash after ~230 days of uptime. At least with Linux you're getting a system that's been used so much that all major issues like that are ironed out. Nothing beats a few million testers.
Ubuntu Linux has millions of testers, your custom version has only been tested by you and your customers :)
Re: Former Tesla Firmware Engineer Discusses the System
#174Earlier quoted context omitted.
> the rockets fly with a heavily customized Linux install My jaw hit the floor the first time I heard this. Why Linux instead of an RTOS?? Apparently Tesla's autopilot also runs Linux, which seems like a huge accident waiting to happen (pun intended).
As a person who used to write navigation and control software for autonomous vehicles who occasionally gets downvoted for telling people that you really don't need a fancy RTOS for this stuff, you really don't need a fancy RTOS for this stuff. Linux is a very common platform for highly responsive robotic systems. I promise that their pid control isn't an Electron app.
Automotive ADAS systems generally require ASIL-D certification, which is much easier with an RTOS than Linux. I don't have much experience with real-time embedded Linux, but my understanding is that it is very difficult or impossible to certify to ASIL-D. Can someone correct me?
Re: Former Tesla Firmware Engineer Discusses the System
#175I have a Model S, and Tesla pushes buggy firmware to it last week. It causes the entire instrument cluster, including the speedometer, to disappear periodically while driving. Tesla knows about the bug and it’s apparently a “high priority”. It’s not entirely clear to me that they are capable of rolling back the update.
Re: Former Tesla Firmware Engineer Discusses the System
#176Earlier quoted context omitted.
Also, an RTOS is susceptible to priority inversion[0] so it’s not necessarily a panacea. 0. which infamously occurred on the Mars Pathfinder. https://www.rapitasystems.com/blog/what-really-happened-to-t...
There are solutions to priority inversion. This is an old limitation that is now days even taught in school. Any decent RTOS should have Priority inheritance that should avoid this. Pointing to this one things as RTOS issue isn't really an accurate portrayal of current RTOS capabilities.
That’s good to know. Indeed, the Pathfinder’s OS (VxWorks) had priority inheritance but it wasn’t enabled on a particular mutex and enabling it was the fix.
Priority inversion had been known about since the 70s. Priority inheritance seems to have first been proposed in 1990:
https://www3.nd.edu/~dwang5/courses/spring18/papers/real-tim... (Priority Inheritance Protocols: An Approach to Real-Time Synchronization)
The Pathfinder engineers were apparently unaware of the priority inheritance option available in VxWorks until they had to debug the issue live from a few hundred million km away.
Re: Former Tesla Firmware Engineer Discusses the System
#177Earlier quoted context omitted.
Tesla’s infotainment and IT infrastructure is unrelated to their safety. If this guy worked on motor control or braking system firmware then that would be scary, but he didn’t.
> Tesla’s infotainment and IT infrastructure is unrelated to their safety Because drivers can't get distracted and crash because of a failure in the infotainment and IT infrastructure?
If that's the standard, then almost everything becomes safety critical. Drivers can easily get distracted by malfunctioning smartphones or apps. (Or, for that matter, properly functioning smartphones or apps.) Yet the prior discussion was based on the idea that things like iPhones and Facebook aren't safety critical the way this is.
Re: Former Tesla Firmware Engineer Discusses the System
#178Earlier quoted context omitted.
Well said, though I think the GP (and many others) blames the platform, not the language. I find C# and .NET runtime (before it meets windows) quite nice. I'm not a big C#-er myself though.
The .NET CLR and core C# runtime libraries are really nice to work with. But things become somewhat Microsoft-y (that is, nice-looking but amazingly half-assed in the most unexpected ways) when you start to do things like writing GUIs.
I've been learning WPF, and I want to shoot myself. It looks pretty, and is very flexible, but it's just so much damn typing. Plus the errors you get out of it are often pretty useless.
Maybe my brain just doesn't get it, but the documentation makes me crazy too. I just hate everything about it.
Re: Former Tesla Firmware Engineer Discusses the System
#179Earlier quoted context omitted.
Very little off-the-shelf software was in use. The whole thing was custom.
Custom as in customized Oracle/SAP, or custom as in from-the-ground up custom? The former is generally a given for a company of any significant size (employees or business activity). The latter is unheard of for most backoffice functions (other than specialized accounting and finance functions) since it's a waste of money and would place the company at significant legal and regulatory risks--it would require effectiv…
There isn't actually that many sap/oracle like packages in .net
Even dynamics isn't .net
Re: Former Tesla Firmware Engineer Discusses the System
#180Earlier quoted context omitted.
Yes, but he's already marked as "former", so recovery is limited. See the Waymo lawsuit.
So, after separating from company, you can release all proprietary information and you'd have limited liability? I thought there would be larger recourse. Not familiar with law though.
If you release corporate secrets you can be successfully sued, regardless.