Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

291–300 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#291

Post by LoganDark »

michaelt wrote: post #56
qmarchi wrote: post #5

It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.

To me it's all-too-understandable how this is possible. Maybe they've got a test fleet, but it accepts code signed with the test build key. Maybe they've got a watchdog timer, but it doesn't get configured until later in the boot process. Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process. I wouldn't be surprised…

> Maybe they've got A/B boot partitions, but trouble counting their boot attempts - maybe they don't have any writable storage that early in the boot process.

You do not report a successful boot until and unless the entire system loads up successfully. You will definitely have writable storage by then.

Re: Rivian software update bricks infotainment system, fix not obvious

#292

Post by LoganDark »

lotsoweiners wrote: post #245
CamperBob2 wrote: post #178

Earlier quoted context omitted.

And please, please, finish developing the embedded software running on car before shipping said car. Sorry, but things no longer work that way, and never will again. This is a good thing, as long as processes are improved to avoid situations like this one.

> This is a good thing, as long as processes are improved to avoid situations like this one. How do you figure. I can’t think of a single thing in my vehicle that could be improved by software. When I buy a car I’d prefer it was done.

It's supposedly a good thing because more features can be added over time, but if they were features worth adding in the first place, the car should have shipped with them already.

Re: Rivian software update bricks infotainment system, fix not obvious

#293

Post by LoganDark »

bdamm wrote: post #218
phpisthebest wrote: post #59

Earlier quoted context omitted.

Well then we need to ask why is their infotainment systems so complex? and does it need to be? I want my infotainment systems go connect to Android Auto. That is is. Make it do that, and only that. This drive to make EV's as complex as possible is one of the reason i am not planning on buying one EV's are suppose to be SIMPLER than ICE. Make me a Simple Car with simple controls, and just replace the ICE with a batter…

Well, good news, Bollinger has made your product!

No they haven't. Bollinger only makes commercial trucks.

Re: Rivian software update bricks infotainment system, fix not obvious

#294

Post by dalyons »

ClumsyPilot wrote: post #257
dalyons wrote: post #233

Earlier quoted context omitted.

eh i guess i disagree. We had that (& still do for some cars) for decades, and it universally resulted in terrible software that you were stuck with for the life of the car. Hard to update == hard to iterate == bad software.

bad software is the one that kills people. Ugly software that works is fine

in cars it is often is ugly AND doesnt work.

Re: Rivian software update bricks infotainment system, fix not obvious

#295

Post by xeromal »

mckn1ght wrote: post #177
jdminhbg wrote: post #156

Earlier quoted context omitted.

Cars 20 years ago didn't have realtime traffic on big touchscreens that you can use to look up your destination and plan out a route that also lets you schedule fueling/charging stops, oh and also stream humanity's entire library of recorded music, books, and podcasts. It's a tradeoff that the vast majority of people want.

All that stuff should be done via smartphones and the screen in the car should be a dumb display for it.

Requiring a cell phone to replicate features a car should have just makes more problems IMO.

Re: Rivian software update bricks infotainment system, fix not obvious

#296

Post by kccqzy »

FireBeyond wrote: post #189
xgbi wrote: post #23

Earlier quoted context omitted.

From what I read somewhere, Tesla was able to do that because they have remote ssh capability. In at least one instance, they fixed the cars manually by running a massive remote command on all cars after a messed up update: https://lobste.rs/s/v42zil/former_tesla_employee_ssh_d_as_ma... I wouldn’t call that very reliable , but they indeed do it regularly

And it's not like they'd ever abuse that ability, like when someone pokes around in their car and discovers references to a new unannounced model, and then Tesla reaches in, force downgrades the vehicle to older software with no references, and then disables the ethernet port on the vehicle, and for a final fuck you disables its ability to ever get another update. They'd never do that, except when they did do that.

The Twitter thread linked by the link posted GP actually contained a reasonable explanation of why that happened. https://x.com/atomicthumbs/status/1032939644621545473

Re: Rivian software update bricks infotainment system, fix not obvious

#297

Post by mytailorisrich »

DannyBee wrote: post #279
mytailorisrich wrote: post #225

Earlier quoted context omitted.

I am not criticising Rivian, not least because I don't know the details. That being said, "signed wrong", including all your hypotheses, that results in a bricked unit is definitely a serious oversight in general. This also might highlight why production tests should be run on exactly production units. No tweaks allowed.

It is great to live in an ideal world, and in fact, in most software, you can do what you are suggesting quite cheaply. But once you get past the sort of "quip on hacker-news" level of thinking about this, or trivial and cheap production testing scenarios, people have to make real tradeoffs because it's never that simple. Talking about those is much more interesting than just asserting that everything should be a cer…

I am not sure why you are being so defensive.

It would be interesting if you could develop about what you mean by real world constraints and how cost of units affect what I wrote, which I did from real world experience.

Re: Rivian software update bricks infotainment system, fix not obvious

#298

Post by cjbprime »

latchkey wrote: post #283
cjbprime wrote: post #280

Earlier quoted context omitted.

> This ran in CI and would fail the build if it didn't pass. I don't mean to be pedantic, but since we're talking about what should happen instead, this is insufficient. It works until the day you realize you made some kind of manual change to your CI infra, or that CI has some non-standard configuration that makes it work for you but not some significant fraction of the fleet. People should do what you described in…

> It works until the day you realize you made some kind of manual change to your CI infra, or that CI has some non-standard configuration that makes it work for you but not some significant fraction of the fleet. Nah, my CI process was solid. This was proven in the field over the course of years. > I don't mean to be pedantic... you need phased rollout You don't need to be pedantic, but better to ask the question rat…

I think there's a minor confusion here, where you think the purpose of my response involves doubting whether your system was successful. I understand it was successful. My response is to the sense in which your comment can be interpreted as advice to other people on what they should build.

I think the fact that you were able to survive with CI-only doesn't mean that we should encourage others to skip implementing a phased rollout based on verified customer successes, including testing of their new updaters before the first time they accidentally brick all the updaters, rather than afterwards. That's what I was hoping to help avoid, through my comment.

Re: Rivian software update bricks infotainment system, fix not obvious

#299

Post by jacquesm »

aaronbeekay wrote: post #186
latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.

Thank you. At least you're honest about it, the other day someone was trying real hard to convince me that software developers at automakers are made of magic fairy dust.

Re: Rivian software update bricks infotainment system, fix not obvious

#300

Post by jacquesm »

bozhark wrote: post #211
aaronbeekay wrote: post #186

Earlier quoted context omitted.

As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.

What's the priority then, telemetry data? Why is it rough out there?

Relatively crappy pay, complex toolchains, long build times, layer upon layer of (really bad) legacy code, badly specified (if they're specified) protocols between subsystems, subsystems that are completely opaque (no source code provided), homegrown OS's or older RTOS's, subset-of-C to keep it safe(r), tricky debugging environments and if you're really unlucky anemic hardware.

I hope I didn't miss anything but I wouldn't be surprised if I did.

Post reply on HN