Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

131–140 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#131

Poor title; physical repair is not required. Physical presence is required.

The article doesn't really state what is required to repair the vehicle. I'd assume if it was as simple as loading a flash drive and plugging it in, then Rivian would have provided a way for customers to self-fix. The second a single body panel is removed to gain access to the headunit, it is a physical repair.

So without more info we cannot know if it is accurate or not.

Re: Rivian software update bricks infotainment system, fix not obvious

#133
post #43
post #27

What a nightmare. This is where software engineering meets "real" engineering, where a "bug" has potentially life threatening consequences.

> where a "bug" has potentially life threatening consequences. What are you referring to? That is not relevant to this story, and would require a deep understanding of the system to make such a claim of negligence. “The issue impacts the infotainment system. In most cases, the rest of the vehicle systems are still operational ...” Also, you can't do an update while driving.

> What are you referring to?

Not the specifics of this article, but more generally about the gravity of the situation car makers (and their software engineers) operate under. The very idea that an OTA software update that causes a bug within more critical features of a car could be life threatening. So my point isn't about the specifics of this particular bug, rather the capacity for a bug that could kill.

Re: Rivian software update bricks infotainment system, fix not obvious

#134
post #63

Earlier quoted context omitted.

A/B gets complicated in the real world. BL1 may not support A/B for example, so to implement A/B bootloaders you may need a shim that can read/write NVM to handle that. Your HSM may not have slots for multiple keys to have different signatures, so upgrading one may trample the other if your update code doesn't check that. Lots of ways to screw this up, especially in automotive where you're likely to be dealing with T…

With all due respect, that all sounds like Programmer Induced Problems(tm). Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardw…

I agree here. A partition is just a partition. It's taking one disk and abstracting it into two. This is not hard.

The rovers on Mars, the Voyager, these problems have been solved for a long time. The compute in a Tesla these days can probably run Crysis.

You can do this OTA to a Raspberry Pi running Nerves via remote SSH and it works really well. The Nerves runtime utilizes A/B partitions for OTA updates.

Re: Rivian software update bricks infotainment system, fix not obvious

#135
post #60

I used to work for a company that built satellite receivers that would be installed in all sorts of weird remote environments in order to pull radio or tv from satellite and rebroadcast locally. If we pushed a broken update it might mean someone from the radio company would have to make a trip to go pull the device and send it to us physically. Our upgrader did not run as root, but one time we had to move a file as r…

> Turns out we had a separate process running that listened on a local socket and would run any command it received as root. Nobody remembered building or releasing it but it made my work quick. No offense, but what a shit show. It makes me assume no source control, and a really good chance that state actors made their way into your network/product. This almost happened at a communication startup I know, with three l…

oh, yeah, this place was a total shit show. BUT we were ISO9001 certified!! So we had source control (CVS) and a Process (with a capital P) to follow. In this case that code was added in a previous development iteration because someone needed to run something as root when a user pressed a certain button on the LCD panel in front and this was the decoupled solution they wrote intentionally. Somehow I feel like that makes it worse than if it was a malicious three letter agency lol.

Re: Rivian software update bricks infotainment system, fix not obvious

#136
post #63

Earlier quoted context omitted.

A/B gets complicated in the real world. BL1 may not support A/B for example, so to implement A/B bootloaders you may need a shim that can read/write NVM to handle that. Your HSM may not have slots for multiple keys to have different signatures, so upgrading one may trample the other if your update code doesn't check that. Lots of ways to screw this up, especially in automotive where you're likely to be dealing with T…

With all due respect, that all sounds like Programmer Induced Problems(tm). Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardw…

Let's use two of the examples I gave above. How would you go about modifying the silicon to support new features in the boot rom, and how do you get around the pigeonhole principle when that silicon vendor doesn't ship enough bits of OTP memory to store multiple keys? M-x butterfly doesn't quite get there, but maybe there's another Emacs command I could use?

Re: Rivian software update bricks infotainment system, fix not obvious

#137
post #63

Earlier quoted context omitted.

With all due respect, that all sounds like Programmer Induced Problems(tm). Cars are a long solved problem, being around for over a century. Telephony and computing hardware and infrastructure today are in the realm of the ludicrously good compared to even just a few decades ago, even if we consider bottom of the barrel worst case scenarios. If software somehow can't work a solved problem using ludicrously good hardw…

Let's use two of the examples I gave above. How would you go about modifying the silicon to support new features in the boot rom, and how do you get around the pigeonhole principle when that silicon vendor doesn't ship enough bits of OTP memory to store multiple keys? M-x butterfly doesn't quite get there, but maybe there's another Emacs command I could use?

You choose different silicon? You get a different vendor? You build your own boards and chips?

For a vehicle that costs $100k+ it shouldn't be hard to double or triple the budget for onboard compute considering it is vital to the operation of the entire vehicle.

Re: Rivian software update bricks infotainment system, fix not obvious

#138
post #50

My 2019 car is not connected to the internet. Instead, I use Apple CarPlay for everything. Is there any reason not to do it this way?

Then automakers can’t sell your information for fat stacks of cash.

..And authoritarian governments can't take complete custody of your movements. Can't shutdown your car either. Seems to be a win-win for both sides (private and public industry).

https://qz.com/1522309/how-chinas-electric-car-surveillance-...

https://apnews.com/article/4a749a4211904784826b45e812cff4ca

Re: Rivian software update bricks infotainment system, fix not obvious

#139
post #15

Earlier quoted context omitted.

The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).

Rollouts don't solve problems, they limit who they effect.

Is not reducing the effective cost of a bad update by 10x or more worthwhile?

Sure, but if you are rolling out to 1% of users per hour, you detect the problem in a couple of hours and much fewer than 2% of users will have applied the update. This is a relatively small support problem.

While if you roll out to everyone at once, you'll detect the problem sooner (within an hour) but have 10x as many affected.

Re: Rivian software update bricks infotainment system, fix not obvious

#140
post #89
post #84

Earlier quoted context omitted.

When a car company is losing money on every car sale. C level execs going to cut corners

This is a bad take.

Rivian layoffs earlier this year [2] combined with reports of $33K loss per sale [1]. Rivian is hemorrhaging money right now.

RVN IPO’d at $150/share. Now it’s trading at $16/share.

All of these indicators of poor leadership to me. No sustainability. Burning cash. Poor company outlook. Poor products.

[1] https://tfltruck.com/2023/10/rivian-financial-results-losses...

[2] https://www.theverge.com/2023/2/1/23581642/rivian-layoff-ev-...

Post reply on HN