Lexus did the very same thing about 8 years ago:
https://www.consumerreports.org/lexus/what-to-do-if-your-lex...
Rivian software update bricks infotainment system, fix not obvious
21–30 of 386 posts
Re: Rivian software update bricks infotainment system, fix not obvious
#22It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.
The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).
The 'early release tests' weren't testing an identical copy of the actual update?
Re: Rivian software update bricks infotainment system, fix not obvious
#23It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…
From what I read somewhere, Tesla was able to do that because they have remote ssh capability.
In at least one instance, they fixed the cars manually by running a massive remote command on all cars after a messed up update: https://lobste.rs/s/v42zil/former_tesla_employee_ssh_d_as_ma...
I wouldn’t call that very reliable , but they indeed do it regularly
Re: Rivian software update bricks infotainment system, fix not obvious
#24Bringing CI/CD mindset to cars is probably not a great idea. Software updates to commuter vehicles should have a high bar for operational standards, and a simple thing such as an expired certificate should have never been deployed. Having isolated networks in vehicles helps but doesn't prevent broken updates from, eventually, bricking the cars.
Re: Rivian software update bricks infotainment system, fix not obvious
#25It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…
This isn't a bunch of Windows PCs home-built from a hodgepodge of components. They designed, built, and shipped all the hardware. There is ABSOLUTELY NO excuse for not having a database of the exact hardware configs by serial number. They have the ability to test every single shipped configuration. If they don't, they have already failed as a car company.
I guarantee they have a database with the hardware configs. It's required by NHTSA to do recalls and notices. They'll undoubtedly be using that to inform the right people to come in.
The update servers almost certainly don't talk to that system though.
Re: Rivian software update bricks infotainment system, fix not obvious
#26It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…
> So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's neither easy nor cheap.
My Jeep Grand Cherokee has OTA for over 5+ years. BMW has been doing it since 2018.
I’m almost positive a family member had it with GMC on star back in the late 2000s.
Re: Rivian software update bricks infotainment system, fix not obvious
#27What a nightmare. This is where software engineering meets "real" engineering, where a "bug" has potentially life threatening consequences.
Re: Rivian software update bricks infotainment system, fix not obvious
#28It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.
The code went through early release tests successfully; the problem came with how it was more broadly released. They should have had further staging of the rollout (randomizing when it is offered to users).
A/B partitions tends to solve that. You will only switch to the new partition when the update is 100% verified installed. If it doesn't complete in an atomic manner, your device will just boot into the previous healthy partition.
Re: Rivian software update bricks infotainment system, fix not obvious
#29It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…
My Volvo XC90 gets regular OTA updates without issue, and so did my Land Rover Discovery before it.
Re: Rivian software update bricks infotainment system, fix not obvious
#30It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…
> All things considered, physical buttons and dials are probably easier and cheaper, because they don't require software updates!
Almost all automotive control modules have firmware, whether that firmware is parsing touchscreen inputs or a rotary encoder.