Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

41–50 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#41

Post by AlotOfReading »

whalesalad wrote: post #28
mlyle wrote: post #15

Earlier quoted context omitted.

The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).

A/B partitions tends to solve that. You will only switch to the new partition when the update is 100% verified installed. If it doesn't complete in an atomic manner, your device will just boot into the previous healthy partition.

A/B gets complicated in the real world. BL1 may not support A/B for example, so to implement A/B bootloaders you may need a shim that can read/write NVM to handle that. Your HSM may not have slots for multiple keys to have different signatures, so upgrading one may trample the other if your update code doesn't check that.

Lots of ways to screw this up, especially in automotive where you're likely to be dealing with TI and their (in)secure boot.

I've solved this problem god only knows how many times now and I've rarely found an automotive board that doesn't introduce fun, new edge cases. OTA can't exceed x kilobytes of memory, the processor isn't fast enough to verify signatures and write the image in < x seconds, can't write the image to flash unless the signature is verified, but the image doesn't fit in RAM, the server delivering the update is 3+ networks away from the device receiving the update, etc.

Re: Rivian software update bricks infotainment system, fix not obvious

#42

Post by jbott »

samsquire wrote: post #18

This is actually a topic that I think about from time to time: how to do aggressive changes to software while they are running. In Ruby world you have monkeypatching. And Linux kernel has livepatching. For example, if you have a distributed system and you want to upgrade a component that every caller uses: you have a large exercise on your hands where you might have to roll out a change over time and then clean up yo…

You might be interested in learning about Erlang – it supports hot code reloads natively: https://oozou.com/blog/understanding-elixir-otp-applications...

Re: Rivian software update bricks infotainment system, fix not obvious

#43

Post by nomel »

qudat wrote: post #27

What a nightmare. This is where software engineering meets "real" engineering, where a "bug" has potentially life threatening consequences.

> where a "bug" has potentially life threatening consequences.

What are you referring to? That is not relevant to this story, and would require a deep understanding of the system to make such a claim of negligence.

“The issue impacts the infotainment system. In most cases, the rest of the vehicle systems are still operational ...”

Also, you can't do an update while driving.

Re: Rivian software update bricks infotainment system, fix not obvious

#44

Post by cozzyd »

galangalalgol wrote: post #37
martin8412 wrote: post #33

Stuff like this is why I don't want OTA updates in my cars. Let the car dealership deal with it during regular maintenance. They'll be on the hook for fixing it before handing the car back to me.

Don't even need OTA. A seattle radio station bricked a bunch of mazdas. https://www.autoblog.com/2022/02/09/seattle-radio-station-br...

Amazing. Can't wait for some car software stack to be so poorly designed that an FM transmitter can remote takeover.

Re: Rivian software update bricks infotainment system, fix not obvious

#45

Post by bri3d »

sturza wrote: post #17
adolph wrote: post #8

The vehicles are drivable but software and displays go black. It appears that the 2023.42 software update hangs at 90% on the vehicle screen or 50% on the app screen and then the vehicle screens black out. All systems appear to still work except for the displays. This is what I do with my Prius to get a comfortably distraction-free driving environment. Sounds like a feature not a bug.

Instrument cluster display going black is a functional safety/QM issue. No blinker, transmission direction, speed etc confirmations.

It looks like they correctly isolated the safety critical components on the instrument cluster and they are still functional without infotainment: https://twitter.com/RivianSoftware/status/172443804967573962...

Re: Rivian software update bricks infotainment system, fix not obvious

#47

Post by galangalalgol »

cozzyd wrote: post #44
galangalalgol wrote: post #37

Earlier quoted context omitted.

Don't even need OTA. A seattle radio station bricked a bunch of mazdas. https://www.autoblog.com/2022/02/09/seattle-radio-station-br...

Amazing. Can't wait for some car software stack to be so poorly designed that an FM transmitter can remote takeover.

It quite possibly could have with a well formed digital fm payload.

Re: Rivian software update bricks infotainment system, fix not obvious

#48

Post by jawns »

nomel wrote: post #43
qudat wrote: post #27

What a nightmare. This is where software engineering meets "real" engineering, where a "bug" has potentially life threatening consequences.

> where a "bug" has potentially life threatening consequences. What are you referring to? That is not relevant to this story, and would require a deep understanding of the system to make such a claim of negligence. “The issue impacts the infotainment system. In most cases, the rest of the vehicle systems are still operational ...” Also, you can't do an update while driving.

Based on the photo included in the article, what they're calling an infotainment system is actually two separate components, one of which appears to be taking the place of a traditional dashboard. If that's the case and there's no other way to monitor speed, fuel levels, engine temperature, warning lights, etc., I'd say that's quite a bit more worrisome than just not being able to play your favorite music while driving.

Re: Rivian software update bricks infotainment system, fix not obvious

#49

Post by fabianlindfors »

samsquire wrote: post #18

This is actually a topic that I think about from time to time: how to do aggressive changes to software while they are running. In Ruby world you have monkeypatching. And Linux kernel has livepatching. For example, if you have a distributed system and you want to upgrade a component that every caller uses: you have a large exercise on your hands where you might have to roll out a change over time and then clean up yo…

The database aspect of this problem is particularly interesting to me. I’ve previously built Reshape [0], a zero-downtime migration tool for Postgres, and am now working on ReshapeDB [1], a full database designed from the ground up to tackle this problem.

[0] https://github.com/fabianlindfors/reshape [1] https://reshapedb.com

Post reply on HN