Live data from Hacker News

Former Tesla Firmware Engineer Discusses the System

twitter.com

51–60 of 589 posts

Re: Former Tesla Firmware Engineer Discusses the System

#51
They've been going 100mph for years now. It doesn't come as a surprise to me that they haven't gone back and made it pretty.

While not even remotely on the same scale, I just spent 3 weeks building an app as fast as possible because my client's old one was causing 75% of their support calls. The code isn't pretty, but it works and it works a lot better than the old one. I know deep down that code is going to stay ugly for a while but that isn't what's important right now.

Re: Former Tesla Firmware Engineer Discusses the System

#52

I used to work at SpaceX on the team that did a piece of software called "WarpDrive". It was a massive monolithic ASP.NET application, with large swaths done with ASP.NET WebForms and a slow frontier of ASP.NET MVC gradually growing when I was working there. This application was responsible for practically everything that ran the factory: inventory, supply chain management, cost analysis, etc. Elon is a big Windows f…

> the rockets fly with a heavily customized Linux install

My jaw hit the floor the first time I heard this. Why Linux instead of an RTOS?? Apparently Tesla's autopilot also runs Linux, which seems like a huge accident waiting to happen (pun intended).

Re: Former Tesla Firmware Engineer Discusses the System

#53

I used to work at SpaceX on the team that did a piece of software called "WarpDrive". It was a massive monolithic ASP.NET application, with large swaths done with ASP.NET WebForms and a slow frontier of ASP.NET MVC gradually growing when I was working there. This application was responsible for practically everything that ran the factory: inventory, supply chain management, cost analysis, etc. Elon is a big Windows f…

Only naive ppl blame the language instead of the codes based on the language.

Well said, though I think the GP (and many others) blames the platform, not the language.

I find C# and .NET runtime (before it meets windows) quite nice. I'm not a big C#-er myself though.

Re: Former Tesla Firmware Engineer Discusses the System

#54

Re QA: To quote Dodge (via Deming), "You can not inspect quality into a product." This poster seems to think well of the QA team they had. However, It doesn't matter how great your QA team is, if the quality isn't in the system the QA team can't put it there. All they can do is tell you that you're making poor quality products, and attempt to inform management (as QA rarely has real authority) who should then act on…

Isn't the point of inspection so that you can perform a REJECT, based on some rejection rate parameter ( which is chosen by business economic reality ) This is the core of the TQM/Six Sigma theory

If your QA is only doing inspections, they're only doing a fraction of the job. Rejecting the work and performing rework or starting over does not address the causal factors in the system of the rejection in the first place.

You will always have some product that needs to be reworked or rejected. The goal is to ensure quality at every stage in order to reduce the amount of rework. You don't want to go the way of the old US car companies who had to suffer major setbacks in the market because they were spending gross amounts of time reworking defective cars before they could ship them. I do not have a copy of it in front of me, but one of Womack's books had some numbers. Double digit percentage of time for a car's production was spent reworking the car after it had been made to make it suitable for sale.

Now, a premise of Agile (and Lean) is to improve the feedback loop. This is where testing and other inspections come into play. By doing them more often (run unit tests on check in, reject if any formerly passing tests start failing, for instance) you can address some quality concerns earlier.

But you still have to address the cause of the rejections. If Stage 10 of production consistently requires rework, it's great that you're catching, addressing, and doing the rework right then. At least it's better than after Stage 20. But management has to work with QA to not just inspect and accept/reject, they have to address the systemic causes of the failure.

Re: Former Tesla Firmware Engineer Discusses the System

#55

Earlier quoted context omitted.

Cars weigh thousands of pounds and routinely drive upwards of 60 miles per hour. The success of the first iPhone or Facebook app didn't depend on using it to navigate through life-and-death situations not only for the users but also for everyone around them. There are places for 'move fast and break things'. But cars move fast already, and they can really break things.

I think this is a naive perspective. I've heard many similar stories from friends that work in the German automotive industry, and I think you would be surprised how many payment systems are tied together (e.g. (insecure) FTP servers to sync daily payments). Every organization has these types of things internally. As an engineer, I don't like it, but it's a fact of life.

Banks can reverse transactions if they want to. Tesla's cavalier attitude towards manufacturing has yielded a 14% first pass through rate on the Model 3 line (which is abysmal and may bankrupt the company along with declining M3 demand) vs. the industry standard of ~80% FPT rate.

Source - https://www.businessinsider.com/tesla-hit-model-3-target-by-...

We're at the point in the Tesla story similar to that of 2008 where Dr. Berry et el were watching the housing market collapse around them and the banks wouldn't re-price the swaps. Tesla is already bankrupt, most people don't know this yet. But they will.

Re: Former Tesla Firmware Engineer Discusses the System

#56
post #44

Earlier quoted context omitted.

>> butter clenching Excuse me, I am a native English speaker and I do not understand this reference

When autocorrect meets "buttock-clenching", I'd guess.

You are correct sir or madam! I’m leaving it though, I like the sound of it.

Re: Former Tesla Firmware Engineer Discusses the System

#57

Earlier quoted context omitted.

Yes, but he's already marked as "former", so recovery is limited. See the Waymo lawsuit.

So, after separating from company, you can release all proprietary information and you'd have limited liability? I thought there would be larger recourse. Not familiar with law though.

[deleted]

Re: Former Tesla Firmware Engineer Discusses the System

#58
post #31

Earlier quoted context omitted.

Remember the Toyota uncontrolled acceleration bug and the report that came as a result of the trial. It seems like software is a bit of a second thought for some incumbents too.

Not sure if you're being sarcastic or not, but the Toyota issues were not caused by any software bug. Most cases were caused by people hitting the wrong pedal. The others were caused by floor mats that were a bit too long, causing the gas pedal to stick. https://www.caranddriver.com/features/its-all-your-fault-the...

Regardless of whether the acceleration were caused by the software or not, the testimonies for the software experts called in to review Toyota's source code for the case were eye-popping:

http://www.safetyresearch.net/blog/articles/toyota-unintende...

> Skid marks notwithstanding, two of the plaintiffs’ software experts, Phillip Koopman, and Michael Barr, provided fascinating insights into the myriad problems with Toyota’s software development process and its source code – possible bit flips, task deaths that would disable the failsafes, memory corruption, single-point failures, inadequate protections against stack overflow and buffer overflow, single-fault containment regions, thousands of global variables. The list of deficiencies in process and product was lengthy.

>There are a large number of functions that are overly complex. By the standard industry metrics some of them are untestable, meaning that it is so complicated a recipe that there is no way to develop a reliable test suite or test methodology to test all the possible things that can happen in it. Some of them are even so complex that they are what is called unmaintainable, which means that if you go in to fix a bug or to make a change, you're likely to create a new bug in the process. Just because your car has the latest version of the firmware -- that is what we call embedded software -- doesn't mean it is safer necessarily than the older one….And that conclusion is that the failsafes are inadequate. The failsafes that they have contain defects or gaps. But on the whole, the safety architecture is a house of cards. It is possible for a large percentage of the failsafes to be disabled at the same time that the throttle control is lost.

etc. etc.

Re: Former Tesla Firmware Engineer Discusses the System

#59

I used to work at SpaceX on the team that did a piece of software called "WarpDrive". It was a massive monolithic ASP.NET application, with large swaths done with ASP.NET WebForms and a slow frontier of ASP.NET MVC gradually growing when I was working there. This application was responsible for practically everything that ran the factory: inventory, supply chain management, cost analysis, etc. Elon is a big Windows f…

> the rockets fly with a heavily customized Linux install My jaw hit the floor the first time I heard this. Why Linux instead of an RTOS?? Apparently Tesla's autopilot also runs Linux, which seems like a huge accident waiting to happen (pun intended).

I'm not familiar with this space but I'm pretty sure there are hard realtime kernels for linux. I'd hope they're using one.

Re: Former Tesla Firmware Engineer Discusses the System

#60
post #22

Re QA: To quote Dodge (via Deming), "You can not inspect quality into a product." This poster seems to think well of the QA team they had. However, It doesn't matter how great your QA team is, if the quality isn't in the system the QA team can't put it there. All they can do is tell you that you're making poor quality products, and attempt to inform management (as QA rarely has real authority) who should then act on…

QA done well is more than testing. It's managing risk and ensuring the process matches your risk tolerance. Car entertainment systems and Car breaking systems can be handled differently because the results of failure are different.

I agree. QA has to inform management, who ought to work with the teams producing (software, widget, service) to address the causal factors of the discovered defects and deficiencies.

By inform, I don't mean "rat out". I mean, management has to have a learning objective. To understand the system that they're managing (because no one understands it fully, their mental model is different than what's actually happening). QA, along with other sources, inform the model of management who can then work with teams to improve the overall system.

Post reply on HN