Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

121–130 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#121

Post by refulgentis »

mlyle wrote: post #15
qmarchi wrote: post #5

It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.

The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).

Rollouts don't solve problems, they limit who they effect.

Re: Rivian software update bricks infotainment system, fix not obvious

#122

Post by theandrewbailey »

jdminhbg wrote: post #96
babypuncher wrote: post #77

Earlier quoted context omitted.

I would be pretty pissed if I went out to my garage to head to work one morning and found that a damn software update bricked my car overnight. This shouldn't even be a thing, why does a car need regular software updates to keep functioning?

There's never been any car that 100% will work in the morning when you go to the garage. It's all tradeoffs.

Now we can add 'bad software update' to the list of things that can go wrong with cars. We didn't use to have that.

Re: Rivian software update bricks infotainment system, fix not obvious

#123

Post by Gud »

qmarchi wrote: post #100
michaelt wrote: post #56

Earlier quoted context omitted.

To me it's all-too-understandable how this is possible. Maybe they've got a test fleet, but it accepts code signed with the test build key. Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process. Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process. I wouldn't be surprised…

So, using Polestar as a reference as it's both a vehicle that I've worked on, and one that I personally drive. > Maybe they've got a test fleet, but it accepts code signed with the test build key. Polestar solves this by only delivering signed updates to their vehicles. The vehicle headunit will refuse to flash a partition that isn't signed by the private key held by Polestar. Pulls double duty to prevent someone fro…

To be honest, I don't think Polestar set a very high bar for software quality. I am currently renting a Polestar 2 from Hertz, and sometimes the HUD doesn't work(it's 50/50 if it will turn on). That means, I don't see speed, battery charge, etc, while driving. Infotainment system is working though.

Re: Rivian software update bricks infotainment system, fix not obvious

#124

Post by enragedcacti »

hef19898 wrote: post #81
mlyle wrote: post #15

Earlier quoted context omitted.

The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).

I am still not sure why I would update software on car, a piece of hardware that, IMHO, shoupd be able to run air gapped 24/7. Exceptions: recurring bugs, GPS maps and security updates. All of which can be done either during service (preferred, if they brick it, they are liable) or by plugging in something. OTA updates just seem completely pointless... Edit: Also, why the heck isn't the entertainment system completel…

Rivian consistently ships a lot of new features and improvements, you can see the changelogs here [1]. I think you can pretty fairly critique a lot of them with: They are just solving a problem that they created by making it too techy, or they are shipping stuff they should have completed before releasing the product. I do think its hard to argue that the updates aren't adding anything of value though. There's convenience stuff like pet mode or bird's eye camera view that were added after release, but there are also things like new driving modes (soft sand and snow) or improved DC charging curves and smarter battery conditioning that legitimately improve the quality of the product as a vehicle.

> Edit: Also, why the heck isn't the entertainment system completely air gapped from the software running the car?

As for this, the entertainment system can control basically every feature of the car and is often the primary or only way to accomplish certain things. Even in much much dumber cars the infotainment is still part of the CAN bus and is able to interact with the rest of the vehicle.

https://rivian.software/category/public/

Re: Rivian software update bricks infotainment system, fix not obvious

#125

Post by margalabargala »

bhauer wrote: post #109
xyst wrote: post #94

Tesla. Rivian. All cut from the same cloth. A car should be simple. Yet we are stuffing all of this tech junk into it and trying to repackage is as something else to pump the numbers. Car companies suck at tech. Let’s be realistic. They should stay their lane and focus on improving the car and physical aspects (safety, reducing carbon output, longevity, ease of repairability, reducing supply chain issues)

> Tesla. Rivian. All cut from the same cloth. I'm not aware of any Tesla OTA updates bricking the infotainment system. At least since I've been paying attention. I don't see them quite as similar as you suggest.

There have been plenty.

https://www.reddit.com/r/TeslaLounge/comments/112oqln/new_te...

https://teslamotorsclub.com/tmc/threads/failed-software-upda...

Two examples of many.

I'm not aware of any fleet-wide issues that accidentally bricked Teslas, but as one-offs they do happen; and unlike this Rivian update, a botched Tesla OTA generally leaves the car undriveable and needing to be towed. These Rivians will at least still drive, as long as you don't need fancy extraneous luxury features like a...speedometer.

Re: Rivian software update bricks infotainment system, fix not obvious

#126

Post by ralmidani »

Move fast and break things that move fast…

I don’t really like or trust most (if not all) of the established automakers, but there is something to be said for having several decades (over a century in some cases) of experience building potential killing machines vs. a company that’s not even 15 years old. The established players have put out cars which suffered freak malfunctions, but Rivian (and Tesla) seem to be struggling more with QA.

Non-rhetorical question: do companies have safeguards for critical components like braking systems, or are they also prone to catastrophic failure if a software engineer pushes a bad commit?

Re: Rivian software update bricks infotainment system, fix not obvious

#127

Post by Someone1234 »

I wonder if the way Microsoft's XBox is designed may be something to look towards in terms of hardware reliability/fallback. Specifically they utilize a Hypervisor which rarely needs updates, running different operating environments which need frequent updates.

- Better isolation of different parts of the system (e.g. infotainment unit, instrument cluster, et al).

- Better isolation for updates (e.g. run a "beta" update, and a "stable" update side-by-side).

- Automatic error detection and rollback (e.g. if a VM keeps restarting after an update).

- Ease of offering features like rollbacks to end-users.

- Rare hypervisor updates can be held to a much higher standard relative to other VM updates.

The only downside of hypervisor-based systems is slightly higher hardware costs. But even that is largely mitigated by modern architectures that natively support virtualization.

PS - You can also look to any containerization. I specifically brought up the XBox because it is a hardware product, just like a vehicle.

Re: Rivian software update bricks infotainment system, fix not obvious

#128

Post by theandrewbailey »

BuckRogers wrote: post #87
babypuncher wrote: post #77

Earlier quoted context omitted.

I would be pretty pissed if I went out to my garage to head to work one morning and found that a damn software update bricked my car overnight. This shouldn't even be a thing, why does a car need regular software updates to keep functioning?

It doesn’t. People and these tech companies are tools. And do it largely in search of ways to take more of your money. It’s not a favor.

If ICE tech was the hot new thing in cars, things like spark plugs would have a chip so that it would fire n times then break, but don't worry, there's a subscription for new ones and they will be automatically ordered when the car says so. If the credit card on file expires, your spark plugs won't work anymore, even if you just replaced them.

Re: Rivian software update bricks infotainment system, fix not obvious

#129

Post by 0xffff2 »

SoftTalker wrote: post #92
martin8412 wrote: post #33

Stuff like this is why I don't want OTA updates in my cars. Let the car dealership deal with it during regular maintenance. They'll be on the hook for fixing it before handing the car back to me.

Also worth considering that a manufacturer like Rivian is pretty small. Every town has a Ford dealer. There are many states, however, that don't even have a single Rivian service center.

Coverage is even thinner than I would have guessed. California has 6 Rivian service centers, but they're strongly clustered in the Bay area and LA/OC/SAN. Even in California if you live in Fresno/Bakersfield/Santa Barbara etc you're looking at several hours round trip to visit an official service center.

Re: Rivian software update bricks infotainment system, fix not obvious

#130

Post by 1234letshaveatw »

scardycat wrote: post #24

Bringing CI/CD mindset to cars is probably not a great idea. Software updates to commuter vehicles should have a high bar for operational standards, and a simple thing such as an expired certificate should have never been deployed. Having isolated networks in vehicles helps but doesn't prevent broken updates from, eventually, bricking the cars.

From a few days back- Its software has been a “key differentiator” https://electrek.co/2023/11/10/rivian-using-software-to-scal... kind of humorous in hindsight
Post reply on HN