Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

241–250 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#241

Post by lttlrck »

aftbit wrote: post #86
cozzyd wrote: post #54

Earlier quoted context omitted.

We'll hopefully no car manufacturer is dumb enough that the FM radio software can affect vehicle control functions but I wouldn't be surprised ..

I'm pretty sure they are all dumb enough for this. This is why people can steal (modern) cars by ripping a headlight out and accessing the CAN bus through the gap. https://kentindell.github.io/2023/04/03/can-injection/ AIUI the military uses "CAN bus firewalls" to prevent this. There is also some sort of encrypted/authenticated CAN bus protocol in the works. Neither are common in production cars as of 2022 or so, tho…

Encrypted/authenticated CAN sounds like "Apple component pairing coming to car near you". No thanks.

Re: Rivian software update bricks infotainment system, fix not obvious

#242

Post by p_j_w »

Lightbody wrote: post #234

I have preorders in for the R1S, the Volvo EX90, and the Kia EV9. I passed once already on buying the R1S when they had one in town available for immediate purchase, simply because they refuse to adopt CarPlay. This incident does NOT give me confidence that Rivian is likely to offer a better alternative to CarPlay, despite their statements otherwise. I suspect the EX90 will be what I land on eventually.

>This incident does NOT give me confidence that Rivian is likely to offer a better alternative to CarPlat,

I have complete faith that, 5 and maybe even 10 years from now, no auto maker will have delivered anything that can compete with either CarPlay or Android Auto. The fact that an auto maker thinks they can do better is a sign of a really high level of either arrogance or outright greed. Complete deal breaker.

Re: Rivian software update bricks infotainment system, fix not obvious

#243

Post by foobiekr »

latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

Rivian is an embedded use case, though, which is not at all like a fleet of servers.

Having worked for companies that produce network devices - including devices that are unreachable for example for 6 months of the year - and on software installation and upgrade, I am baffled how this bricking is possible. For one thing, you generally use some kind of confirmed boot mechanism - you upgrade a standby partition, set an ephemeral boot value that causes device to boot the alternate image, and reboot - only when the image is declared "up" does that get persisted (and then the alternate is upgraded, in order to prevent rollback in the event of a media error). You use watchdogs that are tied to actual forward progress (and not just some demon that the kernel schedules and bangs on the watchdog even if the rest of the system is hung) and if they fail, the WD reboots you. (This is one of the reasons that event driven programming is somewhat preferred - actually processing events from a single dispatch thread makes it easier to reason about the system.)

On top of that, you make sure that the core system is an immutable filesystem so that you can validate the _offline_ alternate image before rebooting (write-and-read-back-uncached) and periodically scrub the alternate image (same).

Like.. this is all embedded 101, stuff people have been widely doing since the mid 1990s and I think I can find examples going back to the 70s. Sometimes you get a little more sophisticated (allow sub-packages or overlays and use a manifest to check the ensemble instead of just a single image), but it's very standard.

Re: Rivian software update bricks infotainment system, fix not obvious

#244

Post by brewdad »

FireBeyond wrote: post #184
BlueTemplar wrote: post #175

Earlier quoted context omitted.

Sounds ridiculous. How is that even road legal ?!?

Teslas occasionally need to reboot / hard reset their software too, when driving no less, and during that period all that information, and most of the controls, are unavailable (like windshield wipers, etc.)

As someone who lives in a place where it rains almost every day for 9 months of the year, this reinforces my decision to never buy a Tesla. Does this kill the headlights too?

Re: Rivian software update bricks infotainment system, fix not obvious

#245

Post by lotsoweiners »

CamperBob2 wrote: post #178
hef19898 wrote: post #144

Earlier quoted context omitted.

Funny, our 2020 MY Jaguar controls car functions from the digital screen in front of the driver, the middle console screens only control AC, entertainment, phone, navigation and other non-car related stuff. No idea how the architecture looks behind all that so. But seriously, even if on the same bus, just don't the media player, radio and connected phone access to the systems actually running the car from engine to b…

And please, please, finish developing the embedded software running on car before shipping said car. Sorry, but things no longer work that way, and never will again. This is a good thing, as long as processes are improved to avoid situations like this one.

> This is a good thing, as long as processes are improved to avoid situations like this one.

How do you figure. I can’t think of a single thing in my vehicle that could be improved by software. When I buy a car I’d prefer it was done.

Re: Rivian software update bricks infotainment system, fix not obvious

#246

Post by karaterobot »

ezfe wrote: post #194
karaterobot wrote: post #80

What kinds of changes are generally included in these over the air updates? I have this sudden urge to shake my fist at a cloud and tell the gods that cars shouldn't need updates in the first place, if the car was ever deemed ready for production and then sold to customers for money. But, maybe I'm wrong, and it makes perfect sense. All I can think of would be something like a periodic update to navigation data, is t…

It’s possible to deem software ready to sell but find improvements later. Simple example: my Subaru was sold to me with an interesting design decision that caused the radio to come on whenever the car was started. This was not a bug. Every Subaru worked this way for years. A year into ownership I received an OTA update that added a “not playing” state on startup. This was never a safety issue and was likely not a def…

I wish my Mazda had this option! But I would still say that I'd expect them to have included this option before selling the car, especially since radios and user preferences around radio UI are pretty well established.

Re: Rivian software update bricks infotainment system, fix not obvious

#248

Post by brewdad »

neoromantique wrote: post #196
qmarchi wrote: post #111

Earlier quoted context omitted.

I understand the sentiment, but think about the alternatives. There are a few different kinds of updates that can be applied, each with their own protective layers. Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else. Then there's system updates, which is where danger noodle things…

>Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else. Also speedometer, which is hardly a convenience feature.

Losing air con in Phoenix in July might not kill you but you'd wish you were dead.

Re: Rivian software update bricks infotainment system, fix not obvious

#249

Post by spaceywilly »

qmarchi wrote: post #5

It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.

Yeah... I worked on an embedded project with literally 2 engineers, and we had an A/B partitioning scheme, and a recovery partition (we fully qualified the recovery image and it was flashed to the units on day 1, it was guaranteed to boot and it would just sit and wait for the user to initiate a firmware load). The app on the device would reset a U-boot variable once it was successfully loaded, so U-boot could check the number of failed boot attempts. If it was >= 5 reboot attempts without booting successfully, it would go into the recovery partition.

There's really no excuse from Rivian on this, this is shoddy

Re: Rivian software update bricks infotainment system, fix not obvious

#250

Post by brewdad »

LastMuel wrote: post #206
worik wrote: post #98

Earlier quoted context omitted.

What amazes me is that any grown up person thinks it is a good idea to update vehicles as if they were telephones Owners should have to bring the vehicle into a shop to have changes made, and they should be very rare. This lazy, control freakery of the worst kind Something very bad is going on happen and people will die before we realize that it is a stupid dangerous practice

On the other hand, we update irreplaceable spacecraft billions of miles away with new software. It should be fine to push software updates out, as long as the correct safety and fallback procedures are in place. It simply has to be designed to handle failure and procedures need to be in place to mitigate risks. It sounds like that wasn't the case here. Also, why wouldn't you have a small initial release pool when you…

NASA's philosophy is the polar opposite of "move fast and break things".
Post reply on HN