What a nightmare. This is where software engineering meets "real" engineering, where a "bug" has potentially life threatening consequences.
critical safety systems/functions appear to be unaffected by this outage.
Rivian software update bricks infotainment system, fix not obvious
271–280 of 386 posts
Re: Rivian software update bricks infotainment system, fix not obvious
#272/r/Rivian is a class act. I expected a wall of screaming, but instead entered a relatively calm room. People are upset, but there's no seething or flamewars, which is kind-of surprising given the cost of these trucks ($80k+, Range Rover territory).
I think the reason is because they're $80k trucks, not $400/month Tesla leases. Also, they're first generation and I think most of the buyers understand that.
Re: Rivian software update bricks infotainment system, fix not obvious
#273Earlier quoted context omitted.
Teslas occasionally need to reboot / hard reset their software too, when driving no less, and during that period all that information, and most of the controls, are unavailable (like windshield wipers, etc.)
>and most of the controls, are unavailable All the controls related to driving are available. You can use turn signals, wipers, change lights, honk, shift gears, etc. But you are correct that you don't see your speed during the reboot.
That's not true. You cannot shift gears in the Model X, S, or 3 without the touchscreen - the only way to change into or out of park, drive, or reverse is to swipe on the touchscreen. Only the Model Y has a stalk and it is being removed in the next version.
Also even on older Teslas lights are controlled exclusively via touchscreen.
Re: Rivian software update bricks infotainment system, fix not obvious
#274Earlier quoted context omitted.
Sounds ridiculous. How is that even road legal ?!?
Teslas occasionally need to reboot / hard reset their software too, when driving no less, and during that period all that information, and most of the controls, are unavailable (like windshield wipers, etc.)
There are some comments from Tesla developers in previous posts about what a shitshow the code is that runs Tesla cars and what the OTA process is.
Re: Rivian software update bricks infotainment system, fix not obvious
#275It's easy to underestimate how hard and expensive it is to build, deploy, and remotely upgrade software that runs reliably on a fleet of diverse cars (different models, different years, slightly different components from batch to batch, etc.). It makes updating a mobile phone OS look trivial in comparison. So far, only Tesla seems to be able to update car software remotely, regularly and reliably. I'm certain it's ne…
Tech junk shouldn’t go in cars, period. Cars shouldn’t be as pervasive and prevalent in society (at least in USA). Yet here we are. Car manufacturers have spent an insane amount of money over decades to get to this point (buying legislators, forcing highway infra, subsidies, profit driven strategy over sustainability)
Decades? Try almost a century. For better or worse, our cities and various economies were built around the automobile.
It's still a free market - these companies could choose not to put tech into their product. But look at the backlash against GM when they announced they wouldn't support Apple Car Play or Android Auto. Consumers want it.
Re: Rivian software update bricks infotainment system, fix not obvious
#276Earlier quoted context omitted.
Automotive varies widely between "basically modern Linux systems with proper updates" and the most janky, home-grown update systems imaginable, sometimes even within the same components and teams.
Yah, I know from friends at ford and vw that there's still vxworks and qnx, but even there, good grief, a-b with confirmed boot is about as basic as you can get. I confess I've seen incredible sloppiness about when a confirmation is done (too early, including in the initial init stages which is way too soon) and watchdogs (spawn off a process that has a while loop stroking the wd - just absolutely pointless).
I've seen kicking and petting the watchdog, but this is my first time seeing stroking
Re: Rivian software update bricks infotainment system, fix not obvious
#277Earlier quoted context omitted.
>and most of the controls, are unavailable All the controls related to driving are available. You can use turn signals, wipers, change lights, honk, shift gears, etc. But you are correct that you don't see your speed during the reboot.
That's not true. You cannot shift gears in the Model X, S, or 3 without the touchscreen - the only way to change into or out of park, drive, or reverse is to swipe on the touchscreen. Only the Model Y has a stalk and it is being removed in the next version. Also even on older Teslas lights are controlled exclusively via touchscreen.
There are captive buttons to change gears/direction in the X (2023) where the hazards are, as outlined in the docs: https://www.tesla.com/ownersmanual/modelx/en_us/GUID-E9B387D...
Re: Rivian software update bricks infotainment system, fix not obvious
#278Earlier quoted context omitted.
It's probably closer to: The test vehicles accept test/prod signed versions Regular vehicles only accept prod signed versions They are otherwise identical. The test vehicles were sent test signed versions The prod vehicles were sent the exact same update, signed with test. This would not be uncommon since the test vehicles probably occasionally run test releases for debugging. Further, the update is probably multiple…
It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle. You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.
Sure, but it could also be signed binaries inside the OTA, a variant of what you are talking about.
IE OTA is a signed package, inside package are also signed binaries. OTA itself is properly signed, a single binary (infotainment) is signed with wrong key.
While most OTA verifiers will verify the OTA signature (which this would pass), most don't verify the individual inside-package binary signatures at install time, only at runtime.
Re: Rivian software update bricks infotainment system, fix not obvious
#279Earlier quoted context omitted.
The software package probably is signed right but contains multiple signed binaries of which one is signed wrong. Or is multiple signed packages and one is wrong. Or the test cars accept prod and test certs. Or some combo of the above. There are lots of ways this could have broken that doesn't amount to rivian not being able to write software
I am not criticising Rivian, not least because I don't know the details. That being said, "signed wrong", including all your hypotheses, that results in a bricked unit is definitely a serious oversight in general. This also might highlight why production tests should be run on exactly production units. No tweaks allowed.
It is great to live in an ideal world, and in fact, in most software, you can do what you are suggesting quite cheaply. But once you get past the sort of "quip on hacker-news" level of thinking about this, or trivial and cheap production testing scenarios, people have to make real tradeoffs because it's never that simple.
Talking about those is much more interesting than just asserting that everything should be a certain way, without any consideration for real world constraints, like cost of units, etc.
Re: Rivian software update bricks infotainment system, fix not obvious
#280I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…
> This ran in CI and would fail the build if it didn't pass.
I don't mean to be pedantic, but since we're talking about what should happen instead, this is insufficient. It works until the day you realize you made some kind of manual change to your CI infra, or that CI has some non-standard configuration that makes it work for you but not some significant fraction of the fleet.
People should do what you described in CI, but as well as that, you need phased rollout, where e.g. the build can only be rolled out to the next percentage point of randomly selected users in a specific segment (e.g. each hardware revision and country as independent segments) after meeting a ratio of successful check-ins, in the field, from the new build by production customers in that segment. That's the actual metric for proceeding with the rollout: actual customers are successfully checking in from the new version of the software.
Except, that's actually not sufficient either. What if the new build is good, but it contains an update to the updater which bricks the updater? Now you're getting successful check-ins from the new version in the field, but none of those customers will ever successfully auto-update again. So, test the new updater's ability to go forwards successfully, too.