Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

341–350 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#341

Post by vmladenov »

theandrewbailey wrote: post #116
babypuncher wrote: post #77

Earlier quoted context omitted.

I would be pretty pissed if I went out to my garage to head to work one morning and found that a damn software update bricked my car overnight. This shouldn't even be a thing, why does a car need regular software updates to keep functioning?

Cars 20 years ago, even most of them 10 years ago, never got any updates unless they got recalled. Nothing broke, nothing got hacked, and most are probably still working fine. What happened to cars today? I refuse to believe that it's solely because these are electric cars, as if the way the car stores and uses energy dictates that it must be part of the internet of things. Edit: there were electric cars over 100 yea…

> Nothing broke, nothing got hacked

This needs to be heavily qualified or else it is outright false.

Re: Rivian software update bricks infotainment system, fix not obvious

#342

Post by vmladenov »

martin8412 wrote: post #88
dilyevsky wrote: post #38

Earlier quoted context omitted.

There is nearly zero regular maintenance to be done on EVs though. No oil, no belts, no fuel filter, spark plugs etc. Even the brakes will likely last entire lifetime of the car

Bullshit. BEVs eat through tires because they're heavy. The air cabin filter and pollen filter need frequent replacement. In the case of Tesla, you better check the undercarriage so you can potentially spot control arms soon to fail. The shield for the battery should be inspected.

BEVs don't eat through tires because they're heavy. They eat through tires because they have fast acceleration and their drivers make use of said acceleration. A Tesla Model 3 weighs around what a BMW M4 weighs and less than all trims of an M5.

Re: Rivian software update bricks infotainment system, fix not obvious

#343

Post by ahartmetz »

jacquesm wrote: post #300
bozhark wrote: post #211

Earlier quoted context omitted.

What's the priority then, telemetry data? Why is it rough out there?

Relatively crappy pay, complex toolchains, long build times, layer upon layer of (really bad) legacy code, badly specified (if they're specified) protocols between subsystems, subsystems that are completely opaque (no source code provided), homegrown OS's or older RTOS's, subset-of-C to keep it safe(r), tricky debugging environments and if you're really unlucky anemic hardware. I hope I didn't miss anything but I wou…

Yeah, I think you missed something. The "software architects, heavy enterprisey tooling, and minions" approach to development where some of the architects could be good developers, but they don't develop, and the minions are often not that good and also not given any autonomy, so they are in a state of learned helplessness and just do what they're told without much thinking or initiative. It results in over-abstracted, over-complicated, slow, unreliable, and sometimes just stupid code.

Re: Rivian software update bricks infotainment system, fix not obvious

#344

Post by ahartmetz »

foobiekr wrote: post #259
AlotOfReading wrote: post #253

Earlier quoted context omitted.

Automotive varies widely between "basically modern Linux systems with proper updates" and the most janky, home-grown update systems imaginable, sometimes even within the same components and teams.

Yah, I know from friends at ford and vw that there's still vxworks and qnx, but even there, good grief, a-b with confirmed boot is about as basic as you can get. I confess I've seen incredible sloppiness about when a confirmation is done (too early, including in the initial init stages which is way too soon) and watchdogs (spawn off a process that has a while loop stroking the wd - just absolutely pointless).

From experience, QNX is actually very nice. I wouldn't say "still using QNX" like it's some crap that nobody would want.

Re: Rivian software update bricks infotainment system, fix not obvious

#345

Post by KingMachiavelli »

foobiekr wrote: post #243
latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

Rivian is an embedded use case, though, which is not at all like a fleet of servers. Having worked for companies that produce network devices - including devices that are unreachable for example for 6 months of the year - and on software installation and upgrade, I am baffled how this bricking is possible. For one thing, you generally use some kind of confirmed boot mechanism - you upgrade a standby partition, set an…

Did you just use standard Yocto or similar tools to build such images? Are there standard daemons for managing hardware watchdogs (besides systemd since that's too simple as you say)? I think there's a lot of niche knowledge in the embedded space and many programmers are used to cloud systems and at most target. The most embedded experience most programmers have is likely iOS/Android development where all of the actual embedded concerns are handled for you. Even Google (soft)bricked a bunch of phones with the latest Android 14 update [1].

IMO there's not a lot of regular OSS for building embedded systems that comes with A/B partitioning, watchdogs, secure and verified boot - it's all custom at every org and tailored for individual products.

[1] https://arstechnica.com/gadgets/2023/11/android-14-patches-r...

Re: Rivian software update bricks infotainment system, fix not obvious

#346

Post by MarkSweep »

KingMachiavelli wrote: post #345
foobiekr wrote: post #243

Earlier quoted context omitted.

Rivian is an embedded use case, though, which is not at all like a fleet of servers. Having worked for companies that produce network devices - including devices that are unreachable for example for 6 months of the year - and on software installation and upgrade, I am baffled how this bricking is possible. For one thing, you generally use some kind of confirmed boot mechanism - you upgrade a standby partition, set an…

Did you just use standard Yocto or similar tools to build such images? Are there standard daemons for managing hardware watchdogs (besides systemd since that's too simple as you say)? I think there's a lot of niche knowledge in the embedded space and many programmers are used to cloud systems and at most target. The most embedded experience most programmers have is likely iOS/Android development where all of the actu…

I quit my job before I got to deploy this, but RAUC looked like it would handle this for Yocto:

https://github.com/rauc/rauc https://github.com/rauc/meta-rauc

For microcontrollers, Memfault had a good article:

https://interrupt.memfault.com/blog/device-firmware-update-c...

Re: Rivian software update bricks infotainment system, fix not obvious

#347

Post by jacquesm »

ahartmetz wrote: post #343
jacquesm wrote: post #300

Earlier quoted context omitted.

Relatively crappy pay, complex toolchains, long build times, layer upon layer of (really bad) legacy code, badly specified (if they're specified) protocols between subsystems, subsystems that are completely opaque (no source code provided), homegrown OS's or older RTOS's, subset-of-C to keep it safe(r), tricky debugging environments and if you're really unlucky anemic hardware. I hope I didn't miss anything but I wou…

Yeah, I think you missed something. The "software architects, heavy enterprisey tooling, and minions" approach to development where some of the architects could be good developers, but they don't develop, and the minions are often not that good and also not given any autonomy, so they are in a state of learned helplessness and just do what they're told without much thinking or initiative. It results in over-abstracte…

Fair enough, yes. That's hopefully not all of them though but I don't doubt that many of the older companies work like that.

Re: Rivian software update bricks infotainment system, fix not obvious

#348

Post by mips_r4300i »

ahartmetz wrote: post #344
foobiekr wrote: post #259

Earlier quoted context omitted.

Yah, I know from friends at ford and vw that there's still vxworks and qnx, but even there, good grief, a-b with confirmed boot is about as basic as you can get. I confess I've seen incredible sloppiness about when a confirmation is done (too early, including in the initial init stages which is way too soon) and watchdogs (spawn off a process that has a while loop stroking the wd - just absolutely pointless).

From experience, QNX is actually very nice. I wouldn't say "still using QNX" like it's some crap that nobody would want.

Indeed, a good RTOS from 10-20 years ago works just as good now as it did back then. The only things that change are dev environments and the driver support.

Re: Rivian software update bricks infotainment system, fix not obvious

#349

Post by jacquesm »

kalleboo wrote: post #327
jacquesm wrote: post #299

Earlier quoted context omitted.

Thank you. At least you're honest about it, the other day someone was trying real hard to convince me that software developers at automakers are made of magic fairy dust.

I'm amazed anyone would argue that after the Toyota firmware analysis.

Check out the thread a couple of days ago:

https://news.ycombinator.com/item?id=38244149

Re: Rivian software update bricks infotainment system, fix not obvious

#350

Post by neoromantique »

fyrn_ wrote: post #209
neoromantique wrote: post #196

Earlier quoted context omitted.

>Infotainment updates, like what happened to Rivian aren't that dangerous. You lose "convienience features" like maps, air con, etc, but generally nothing that could kill you or someone else. Also speedometer, which is hardly a convenience feature.

The dashboard panel is working, at least according to Twitter updates. "Only" affects the infotainment console.

The reddit thread in official subreddit suggests otherwise, with people suggesting using 'speedometer apps' on their phones as a workaround.

I don't know, I have car from 1991 so I'm just observing the trainwreck.

Post reply on HN