Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

181–190 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#181

Post by DannyBee »

MichaelZuo wrote: post #22
mlyle wrote: post #15

Earlier quoted context omitted.

The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).

The 'early release tests' weren't testing an identical copy of the actual update?

It's probably closer to:

The test vehicles accept test/prod signed versions

Regular vehicles only accept prod signed versions

They are otherwise identical.

The test vehicles were sent test signed versions

The prod vehicles were sent the exact same update, signed with test.

This would not be uncommon since the test vehicles probably occasionally run test releases for debugging.

Further, the update is probably multiple signed pieces, and the only part accidentally signed with test was likely infotainment software.

Or something like this.

It's hard to believe they wouldn't test sending badly signed updates, so i have to imagine it's a particularly weird badly signed update.

In other words, i would not assume they are idiots.

Re: Rivian software update bricks infotainment system, fix not obvious

#182

Post by FireBeyond »

cs702 wrote: post #13

It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…

> So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's neither easy nor cheap.

Tesla, whose computer systems quite regularly need to be hard rebooted while the car is driving? That Tesla?

Re: Rivian software update bricks infotainment system, fix not obvious

#183

Post by FireBeyond »

As annoying as this, I find this laughable, too. Rivian updated users on the situation. Then, whines Electrek:

> That’s the last update we had over 10 hours after Rivian customer vehicles were fed the bad software update.

"Over 10 hours"!

I suppose it isn't Tesla, who yeets updates over the fence, that break new things, yeets another update that fixes that problem but introduces another one, then reverts back to two versions prior, before the issue. The Tesla that gets firmware fixes from vendors that have a test harness that should take 36+ hours to run, but says YOLO and flashes it onto a random car they have lying around and emails the vender back 3 hours later saying "LGTM, WFM, thanks!"

Re: Rivian software update bricks infotainment system, fix not obvious

#184

Post by FireBeyond »

BlueTemplar wrote: post #175
Gud wrote: post #123

Earlier quoted context omitted.

To be honest, I don't think Polestar set a very high bar for software quality. I am currently renting a Polestar 2 from Hertz, and sometimes the HUD doesn't work(it's 50/50 if it will turn on). That means, I don't see speed, battery charge, etc, while driving. Infotainment system is working though.

Sounds ridiculous. How is that even road legal ?!?

Teslas occasionally need to reboot / hard reset their software too, when driving no less, and during that period all that information, and most of the controls, are unavailable (like windshield wipers, etc.)

Re: Rivian software update bricks infotainment system, fix not obvious

#185

Post by BlueTemplar »

ct0 wrote: post #82
nomel wrote: post #43

Earlier quoted context omitted.

> where a "bug" has potentially life threatening consequences. What are you referring to? That is not relevant to this story, and would require a deep understanding of the system to make such a claim of negligence. “The issue impacts the infotainment system. In most cases, the rest of the vehicle systems are still operational ...” Also, you can't do an update while driving.

You've never been to death valley without air conditioning Or Russia without heat. I think the infotainment system in this case has a broken climate control function. There are workarounds, but why if you don't have your phone?

I have been talking recently to someone whose job involves sometimes driving to other continents, and they mentioned that cars more recent than ~2003 were out of the question because outside of the EU you cannot expect random mechanics to have the computers required to interface with the car's computers - required for repairs.

Re: Rivian software update bricks infotainment system, fix not obvious

#186

Post by aaronbeekay »

latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.

Re: Rivian software update bricks infotainment system, fix not obvious

#187

Post by liminalsunset »

gitfan86 wrote: post #97
nomel wrote: post #39

Earlier quoted context omitted.

I think this shows more of a fundamental flaw in their update mechanism, than anything. I don't think a botched update is a big deal. It happens, and should be expected , in a sane design. The fact that the customer noticed is a big deal. There are many implementations that could be used for an "auto rollback" feature. They either failed to implement that in a sane way, or they were goobers, and assumed things would…

The Tesla update is slow probably for this reason. It is probably verifying that it can rollback at any point of failure.

I believe one of the reasons it is slow is because it is also updating the firmware on any number of connected ECUs over the CAN bus. This typically means the image has to be sent over a 500kbit/s bus so there is a limit to how long it has to take.

Re: Rivian software update bricks infotainment system, fix not obvious

#188

Post by jandrese »

DannyBee wrote: post #180
qmarchi wrote: post #5

It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.

Rivian does have a test fleet, and they test it for weeks before releasing. This particular issue is because they apparently distributed the firmware signed with the wrong cert. Not a bug in the software itself. That is independent of testing the software, but still a distribution issue.

Yeah, but how did the vehicle not just reject the wrong cert and refuse to flash the update?

Re: Rivian software update bricks infotainment system, fix not obvious

#189

Post by FireBeyond »

xgbi wrote: post #23
cs702 wrote: post #13

It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…

From what I read somewhere, Tesla was able to do that because they have remote ssh capability. In at least one instance, they fixed the cars manually by running a massive remote command on all cars after a messed up update: https://lobste.rs/s/v42zil/former_tesla_employee_ssh_d_as_ma... I wouldn’t call that very reliable , but they indeed do it regularly

And it's not like they'd ever abuse that ability, like when someone pokes around in their car and discovers references to a new unannounced model, and then Tesla reaches in, force downgrades the vehicle to older software with no references, and then disables the ethernet port on the vehicle, and for a final fuck you disables its ability to ever get another update.

They'd never do that, except when they did do that.

Re: Rivian software update bricks infotainment system, fix not obvious

#190

Post by donmcronald »

latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

> While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me.

I feel like it's going to happen to someone that makes network devices eventually. I'm always scared to update my (several hundred) UniFi devices. Their update process isn't foolproof and they push auto-updates via the UI pretty hard.

Several years ago they caused some people's devices to disconnect from the management controller when they enabled 'https' communication. Prior to that, if you were pointing devices at 'https://example.com:8080...' they would ignore the 'https' part and do an 'http' request to port '8080'. Then they pushed their 'https' update which expected an 'https' connection and didn't fall back to the old behavior for anyone that was mistakenly using 'https' in their URL initially. Some people on their forums complained about having to manually SSH to every device to fix the issue.

It was caused by an end-user mistake, but they knew it was a potential issue. AFAIK, their attitude on it hasn't changed and a lot and at the time their response was that they knew it would break some people, but that it wouldn't be that many (lol).

IMO, the issue with those systems is that basic communication back to the update / config server is part of the total package which is too complex (ie: a full Debian install). I'd rather see something like Mender (mender.io) where the core communications / updates come from a hardened system with watchdog, recovery, rollback logic.

Think of how crazy it is to have something like pfSense doing package based updates rather than slice based updates. At least with boot environments they could add some watchdog and rollback type logic, but it'll still be part of the total system instead of something like a hardened slice based setup where the most critical logic is isolated from everything else and treated like a princess.

Do you have any insight on package vs slice based systems for updates? Did you isolate update logic from the rest of the system or am I out of touch with that opinion?

Post reply on HN