Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

381–386 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#381

Post by paledot »

qmarchi wrote: post #100
michaelt wrote: post #56

Earlier quoted context omitted.

To me it's all-too-understandable how this is possible. Maybe they've got a test fleet, but it accepts code signed with the test build key. Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process. Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process. I wouldn't be surprised…

So, using Polestar as a reference as it's both a vehicle that I've worked on, and one that I personally drive. > Maybe they've got a test fleet, but it accepts code signed with the test build key. Polestar solves this by only delivering signed updates to their vehicles. The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone fro…

> The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone from flashing a malicious update, as well as corruption detection.

And of course preventing people from modifying and controlling hardware that they own, having paid 6 figures for (in the case of the Polestar 3 anyway). But that's table stakes for embedded systems in this day and age. Security for me, not for thee.

Re: Rivian software update bricks infotainment system, fix not obvious

#382

Post by mlyle »

AlotOfReading wrote: post #322
DannyBee wrote: post #278

Earlier quoted context omitted.

Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about. IE OTA is a signed package, inside package are also signed binaries. OTA itself is properly signed, a single binary (infotainment) is signed with wrong key. While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time,…

That's not how I've ever implemented OTA, but I'll grant that it's possible if it was designed by someone with no idea what they were doing. Certainly not a good OTA process though, for this among many other reasons.

It makes sense for each component to check signatures of its code to prevent various kinds of attacks -- e.g. someone coming and reflashing just infotainment or motor controllers with something malicious.

So, OTA update comes in, containing a bundle of software for different subsystems. It's sent to different subsystems. Then, those subsystems check integrity at startup, but one subsystem's bootloader isn't happy because the firmware looks to be invalid.

You can only prevent this if the OTA knows how to do equivalent verification for every subsystem in the car that checks integrity. (And, of course, even if you do this, there's other ways you can go wrong that aren't specific to integrity checks).

Re: Rivian software update bricks infotainment system, fix not obvious

#383

Post by DannyBee »

AlotOfReading wrote: post #322
DannyBee wrote: post #278

Earlier quoted context omitted.

Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about. IE OTA is a signed package, inside package are also signed binaries. OTA itself is properly signed, a single binary (infotainment) is signed with wrong key. While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time,…

That's not how I've ever implemented OTA, but I'll grant that it's possible if it was designed by someone with no idea what they were doing. Certainly not a good OTA process though, for this among many other reasons.

Uh, this is a perfectly normal ota process when dealing with multiple embedded systems.

It seems silly to off the cuff assert that it is a bad process or only done by those who have no idea what they are doing.

Re: Rivian software update bricks infotainment system, fix not obvious

#384

Post by DannyBee »

mytailorisrich wrote: post #297
DannyBee wrote: post #279

Earlier quoted context omitted.

It is great to live in an ideal world, and in fact, in most software, you can do what you are suggesting quite cheaply. But once you get past the sort of "quip on hacker-news" level of thinking about this, or trivial and cheap production testing scenarios, people have to make real tradeoffs because it's never that simple. Talking about those is much more interesting than just asserting that everything should be a cer…

I am not sure why you are being so defensive. It would be interesting if you could develop about what you mean by real world constraints and how cost of units affect what I wrote, which I did from real world experience.

I'm not defensive, I just find your extreme position remarkably silly.

You included nothing about how costs affect anything. You simply assert that you should always test prod on pristine production units.

There are plenty of times outside of software where production units cost millions or you can only produce them so quickly, or both, and where your extreme take would result in remarkable cost or a competitor eating your lunch.

Which is precisely why its not done, and in the real world tradeoffs are made between what really needs 100% assurance and not. Spending money or losing customers for 5 9s of reliability through testing when two are needed is not a best practice, and is often explicitly called out as such.

In the case of rivian, maintaining a significant fleet of expensive, pristine, exact customer spec (ie not debuggable) cars just to try to get 100% prod ota success assurance is unlikely to provide value vs getting 98% assurance and not doing that (by rough calculation, it stands at 98% after this incident).

Re: Rivian software update bricks infotainment system, fix not obvious

#385

Post by mytailorisrich »

DannyBee wrote: post #384
mytailorisrich wrote: post #297

Earlier quoted context omitted.

I am not sure why you are being so defensive. It would be interesting if you could develop about what you mean by real world constraints and how cost of units affect what I wrote, which I did from real world experience.

I'm not defensive, I just find your extreme position remarkably silly. You included nothing about how costs affect anything. You simply assert that you should always test prod on pristine production units. There are plenty of times outside of software where production units cost millions or you can only produce them so quickly, or both, and where your extreme take would result in remarkable cost or a competitor eatin…

My position is neither silly nor extreme. It's the way it is usually done and other comments here have been along the same lines.

In fact you are trying to spin what I wrote to an extreme to make your point.

By the way, it is not about 100% success assurance but assurance that failure does not brick the unit. This is an assurance that should be, and can be, close to 100%, indeed a good number of 9s because, obviously you cannot brick 2 cars out of 100 for every software upgrade!

Re: Rivian software update bricks infotainment system, fix not obvious

#386

Post by fransje26 »

Terr_ wrote: post #378
fransje26 wrote: post #372

Earlier quoted context omitted.

You never had Windows force-reboot your system during an unrequested background update? Happened twice to me.

I've lost work to it because it killed my running VirtualBox sessions.

I feel you. I've had my machine blocked for 5 hours as it was pushing an update down my throat. Luckily it gave me a 2 minute warning before the initial reboot so that I could close everything down.

As that was during a client emergency, I had a very happy client that day.

Post reply on HN