Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

321–330 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#321

Post by iso8859-1 »

matrss wrote: post #66
cs702 wrote: post #13

It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…

> All things considered, physical buttons and dials are probably easier and cheaper, because they don't require software updates! I am pretty sure there is a market for a dumb modern car, but no one is building it. I am thinking of an electric car without anything "smart" in it. Modern safety features can stay, if they work completely self contained and without requiring an external connection ever over the lifespan…

I wonder if it is somehow possible to use an open source battery management system to build a car like this. See https://foxbms.org/

Re: Rivian software update bricks infotainment system, fix not obvious

#322

Post by AlotOfReading »

DannyBee wrote: post #278
AlotOfReading wrote: post #220

Earlier quoted context omitted.

It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle. You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.

Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about. IE OTA is a signed package, inside package are also signed binaries. OTA itself is properly signed, a single binary (infotainment) is signed with wrong key. While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time,…

That's not how I've ever implemented OTA, but I'll grant that it's possible if it was designed by someone with no idea what they were doing. Certainly not a good OTA process though, for this among many other reasons.

Re: Rivian software update bricks infotainment system, fix not obvious

#323

Post by ThePowerOfFuet »

singleshot_ wrote: post #169
gravitronic wrote: post #31

I used to work for a company that built satellite receivers that would be installed in all sorts of weird remote environments in order to pull radio or tv from satellite and rebroadcast locally. If we pushed a broken update it might mean someone from the radio company would have to make a trip to go pull the device and send it to us physically. Our upgrader did not run as root, but one time we had to move a file as r…

The person who built and released this might not have ever worked for your company, which might be why no one remembers building or releasing it.

This is one of the very finest comments I have ever seen on HN (or anywhere else, for that matter).

Re: Rivian software update bricks infotainment system, fix not obvious

#324

Post by fargle »

worik wrote: post #98
qmarchi wrote: post #5

It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.

What amazes me is that any grown up person thinks it is a good idea to update vehicles as if they were telephones Owners should have to bring the vehicle into a shop to have changes made, and they should be very rare. This lazy, control freakery of the worst kind Something very bad is going on happen and people will die before we realize that it is a stupid dangerous practice

> What amazes me is that any grown up person thinks it is a good idea to update vehicles as if they were telephones

What amazes me is that any grown up person thinks it is a good idea to update telephones as if they were software and not phones.

Or rather that it is a good idea to have phones that need updates? Either way, we're all one 1/2 assed push update to a fridge, vacuum, washing machine, phone or car away from a really annoying day.

Re: Rivian software update bricks infotainment system, fix not obvious

#325

Post by vel0city »

fma wrote: post #311
vel0city wrote: post #152

Earlier quoted context omitted.

> GM is supposed to drop CarPlay in favor of whatever they are doing on their EVs IIRC its not just their EVs its all cars.

Shouldn't make such a bold IIRC statement correcting someone without first doing a quick Google to verify. https://www.caranddriver.com/news/a43488135/gm-apple-carplay... Specifically it's for the new Ultium EVs. Bolts, Lyriqs, Hummers will still have it.

If I had known it for a fact I wouldn't have prefaced it with "IIRC".

Guess I didn't recall correctly. Thanks for pointing that out.

Re: Rivian software update bricks infotainment system, fix not obvious

#326

Post by quailfarmer »

cjbprime wrote: post #280
latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

> This ran in CI and would fail the build if it didn't pass. I don't mean to be pedantic, but since we're talking about what should happen instead, this is insufficient. It works until the day you realize you made some kind of manual change to your CI infra, or that CI has some non-standard configuration that makes it work for you but not some significant fraction of the fleet. People should do what you described in…

A good way to handle the who-updates-the-updater issue is to use a triple partition updater. A updates B, and then B updates C, then C updates A. If anything about the new version prevents it from properly updating its neighbor, that neighbor won't be able to close the loop, and you'll fall back to A. This simplifies the FSBL, because it just boots the three partitions in a loop, no failure detection required. You don't need to triplicate the full application either, just the minimum system needed to perform an update, and then have the "application" in it's own partition to be called by the updater.

Re: Rivian software update bricks infotainment system, fix not obvious

#327

Post by kalleboo »

jacquesm wrote: post #299
aaronbeekay wrote: post #186

Earlier quoted context omitted.

As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.

Thank you. At least you're honest about it, the other day someone was trying real hard to convince me that software developers at automakers are made of magic fairy dust.

I'm amazed anyone would argue that after the Toyota firmware analysis.

Re: Rivian software update bricks infotainment system, fix not obvious

#328

Post by onion2k »

postalrat wrote: post #281
psychlops wrote: post #230

Earlier quoted context omitted.

Having worked on 25K machines, I can assure you that it never deployed to every single machine and failed to do so in interesting ways all the time.

As a frontend web developer I'm constantly deploying software to many thousands of machines. And you know what? It's pretty damn simple.

I'm not really a frontend dev any more but I was for a long time. I can assure you that the only reason you think your code works is because no one tells you it's broken. If you use an error logging or telemetry service (Sentry, Rollbar, New Relic, etc) you will be aware that errors happen in frontend code all the time. It's just that most of the time bugs don't crash the app, and the user doesn't know what to expect so they see a broken feature and think it's meant to be like that.

Re: Rivian software update bricks infotainment system, fix not obvious

#329

Post by onion2k »

jrumbut wrote: post #304
postalrat wrote: post #281

Earlier quoted context omitted.

As a frontend web developer I'm constantly deploying software to many thousands of machines. And you know what? It's pretty damn simple.

It's simple because we tolerate certain limitations in the web platform. If you had a hard requirement that a page load could never take more than 100ms, regardless of network conditions, you'd have quite a challenge on your hands.

The laws of physics are definitely very challenging. If you've got a solution please write a blog post.
Post reply on HN