Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

251–260 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#251

Post by jdechko »

latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

As a non-developer, the whole situation with a bad software update to the Voyager spacecraft really puts things into perspective as far as how bad remote updates can be.

It’s also a testament to the way that the system was designed that they were able to get it back online.

Re: Rivian software update bricks infotainment system, fix not obvious

#252

Post by pard68 »

gunapologist99 wrote: post #222

This is why I don't really want my car to have any antenna (that receives/interprets code) or receive OTA updates, ever. I'd like to please force any attackers to at least be within 50 feet of my TPMS, instead of being literally anywhere on the planet. A car doesn't need data updates, and definitely not code updates[1] 1. source: every car built in previous century.

My insistence on only driving cars made prior to 2005 keeps making more and more sense.

(2005 is just an arbitrary date I settled on, nothing significant about it)

Re: Rivian software update bricks infotainment system, fix not obvious

#253

Post by AlotOfReading »

foobiekr wrote: post #239
aaronbeekay wrote: post #186

Earlier quoted context omitted.

As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.

do you guys not have confirmed boot and swizzling to fallback images?

Automotive varies widely between "basically modern Linux systems with proper updates" and the most janky, home-grown update systems imaginable, sometimes even within the same components and teams.

Re: Rivian software update bricks infotainment system, fix not obvious

#254

Post by serf »

FireBeyond wrote: post #184
BlueTemplar wrote: post #175

Earlier quoted context omitted.

Sounds ridiculous. How is that even road legal ?!?

Teslas occasionally need to reboot / hard reset their software too, when driving no less, and during that period all that information, and most of the controls, are unavailable (like windshield wipers, etc.)

some ex-Boeing engineer probably came up with that 'fix'.[0]

[0]: https://www.seattletimes.com/business/boeing-aerospace/faa-o...

Re: Rivian software update bricks infotainment system, fix not obvious

#255

Post by dagmx »

VyseofArcadia wrote: post #68
cs702 wrote: post #13

It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…

Forget updates entirely. My car is one of the few places I expect to get software that works the first time. If you absolutely must have updates, then at least not OTA updates. Have them done at the dealership or service center so any issues can be dealt with immediately. Come on, is this engineering or hacking? This is a car, not a CRUD app. Get. It. Right.

That’s how things used to be and it resulted in lots of long standing bugs because the update rates were low, and so manufacturers didn’t push updates. Many people don’t live near dealers or service centers or can afford the continued cost (it’s not free usually unless it’s a recall)

OTA is better for consumer when done properly. Other manufacturers manage it fine, and one bad example shouldn’t be what we base things on. It’s what we should learn from and improve on.

Re: Rivian software update bricks infotainment system, fix not obvious

#256

Post by ezfe »

karaterobot wrote: post #246
ezfe wrote: post #194

Earlier quoted context omitted.

It’s possible to deem software ready to sell but find improvements later. Simple example: my Subaru was sold to me with an interesting design decision that caused the radio to come on whenever the car was started. This was not a bug. Every Subaru worked this way for years. A year into ownership I received an OTA update that added a “not playing” state on startup. This was never a safety issue and was likely not a def…

I wish my Mazda had this option! But I would still say that I'd expect them to have included this option before selling the car, especially since radios and user preferences around radio UI are pretty well established.

Sure, but they didn't - and it's not because the software is complicated. The Subaru headunits are very basic (and more-so a few years ago).

When they make an improvement, I like getting the trivial things on my older vehicle. It's better for everyone involved, so long as they do it responsible.

Re: Rivian software update bricks infotainment system, fix not obvious

#257

Post by ClumsyPilot »

dalyons wrote: post #233
VyseofArcadia wrote: post #68

Earlier quoted context omitted.

Forget updates entirely. My car is one of the few places I expect to get software that works the first time. If you absolutely must have updates, then at least not OTA updates. Have them done at the dealership or service center so any issues can be dealt with immediately. Come on, is this engineering or hacking? This is a car, not a CRUD app. Get. It. Right.

eh i guess i disagree. We had that (& still do for some cars) for decades, and it universally resulted in terrible software that you were stuck with for the life of the car. Hard to update == hard to iterate == bad software.

bad software is the one that kills people. Ugly software that works is fine

Re: Rivian software update bricks infotainment system, fix not obvious

#258

Post by akira2501 »

latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

When possible, I used a fail back mechanism. If the update failed to fully come up, then the watchdog timer would catch it, the bootloader would notice the incomplete boot, and attempt to boot from the previous known working image in that case.

Re: Rivian software update bricks infotainment system, fix not obvious

#259

Post by foobiekr »

AlotOfReading wrote: post #253
foobiekr wrote: post #239

Earlier quoted context omitted.

do you guys not have confirmed boot and swizzling to fallback images?

Automotive varies widely between "basically modern Linux systems with proper updates" and the most janky, home-grown update systems imaginable, sometimes even within the same components and teams.

Yah, I know from friends at ford and vw that there's still vxworks and qnx, but even there, good grief, a-b with confirmed boot is about as basic as you can get.

I confess I've seen incredible sloppiness about when a confirmation is done (too early, including in the initial init stages which is way too soon) and watchdogs (spawn off a process that has a while loop stroking the wd - just absolutely pointless).

Re: Rivian software update bricks infotainment system, fix not obvious

#260

Post by intern4tional »

AlotOfReading wrote: post #220
DannyBee wrote: post #181

Earlier quoted context omitted.

It's probably closer to: The test vehicles accept test/prod signed versions Regular vehicles only accept prod signed versions They are otherwise identical. The test vehicles were sent test signed versions The prod vehicles were sent the exact same update, signed with test. This would not be uncommon since the test vehicles probably occasionally run test releases for debugging. Further, the update is probably multiple…

It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle. You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.

Likely not their code.

OTA is generally developed by tier 1, so this is probably a bug in the tier 1's code. (Samsung, Panasonic, Sony, etc are common tier 1s in this space.)

Post reply on HN