Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

211–220 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#211

Post by bozhark »

aaronbeekay wrote: post #186
latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

As somebody currently working at an automaker on software systems, the amazing thing to me is that a mess up of this level doesn’t happen weekly. It’s rough out here.

What's the priority then, telemetry data? Why is it rough out there?

Re: Rivian software update bricks infotainment system, fix not obvious

#212

Post by nomel »

gravitronic wrote: post #135
nomel wrote: post #60

Earlier quoted context omitted.

> Turns out we had a separate process running that listened on a local socket and would run any command it received as root. Nobody remembered building or releasing it but it made my work quick. No offense, but what a shit show. It makes me assume no source control, and a really good chance that state actors made their way into your network/product. This almost happened at a communication startup I know, with three l…

oh, yeah, this place was a total shit show. BUT we were ISO9001 certified!! So we had source control (CVS) and a Process (with a capital P) to follow. In this case that code was added in a previous development iteration because someone needed to run something as root when a user pressed a certain button on the LCD panel in front and this was the decoupled solution they wrote intentionally. Somehow I feel like that ma…

Package this up and send it to https://thedailywtf.com

It's beautiful.

Re: Rivian software update bricks infotainment system, fix not obvious

#213

Post by fsckboy »

I wish the economics of mass production didn't turn pennies into millions that need to be eliminated, because I've always thought the "don't disconnect from power" and "update bricks it" type problems could be solved by having extra EPROM to download into, the way linux keeps the previous kernel around after an update.

Or at least the ability to re-init/download from scratch, like a borked macbook disk. And hey, not the extra ability to do that, make it "the way it works" so you're always testing it.

Re: Rivian software update bricks infotainment system, fix not obvious

#215

Post by DannyBee »

mytailorisrich wrote: post #198
DannyBee wrote: post #180

Earlier quoted context omitted.

Rivian does have a test fleet, and they test it for weeks before releasing. This particular issue is because they apparently distributed the firmware signed with the wrong cert. Not a bug in the software itself. That is independent of testing the software, but still a distribution issue.

My 2c based on your comment: * " signed with the wrong cert " should mean the software package is rejected before it it is installed. * software upgrades are tricky and there should be at least 2 versions available so that fallback to the previous is possible and automatic in case of issues.

The software package probably is signed right but contains multiple signed binaries of which one is signed wrong.

Or is multiple signed packages and one is wrong.

Or the test cars accept prod and test certs.

Or some combo of the above.

There are lots of ways this could have broken that doesn't amount to rivian not being able to write software

Re: Rivian software update bricks infotainment system, fix not obvious

#216

Post by latchkey »

code_runner wrote: post #193
latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

out of morbid curiosity.... how long did it take to ssh into and fix all of those servers? I imagine even automating a fix (if possible) would still take a good amount of time.

gnu parallel and sshpass is your friend.

The way I built my app was that I could install it cleanly via a curl | bash.

So, I just had a simple shell script that iterated through the list of IP addresses (from the DHCP leases), ran curl | bash and that cleaned up the mess pretty quickly.

Re: Rivian software update bricks infotainment system, fix not obvious

#217

Post by PeterisP »

BlueTemplar wrote: post #185
ct0 wrote: post #82

Earlier quoted context omitted.

You've never been to death valley without air conditioning Or Russia without heat. I think the infotainment system in this case has a broken climate control function. There are workarounds, but why if you don't have your phone?

I have been talking recently to someone whose job involves sometimes driving to other continents, and they mentioned that cars more recent than ~2003 were out of the question because outside of the EU you cannot expect random mechanics to have the computers required to interface with the car's computers - required for repairs.

If that's the only problem, then that would be fixed simply by taking "computers required to interface with the car's computers" i.e. a Also, since many of these places have a lot of used cars that once were in the EU or other places with legally required computers, this is changing as now most of the mechanics even in relatively poor remote places have to have these devices as now many of their local customers don't have "computerless" cars any more, and the existing pre-2003 cars won't last forever even there.

Re: Rivian software update bricks infotainment system, fix not obvious

#218

Post by bdamm »

phpisthebest wrote: post #59
willio58 wrote: post #32

Earlier quoted context omitted.

I don't think the Jeep or BMW infotainment systems are nearly as fleshed out or complex as Rivian's, especially not Tesla's. Maybe I'm wrong!

Well then we need to ask why is their infotainment systems so complex? and does it need to be? I want my infotainment systems go connect to Android Auto. That is is. Make it do that, and only that. This drive to make EV's as complex as possible is one of the reason i am not planning on buying one EV's are suppose to be SIMPLER than ICE. Make me a Simple Car with simple controls, and just replace the ICE with a batter…

Well, good news, Bollinger has made your product!

Re: Rivian software update bricks infotainment system, fix not obvious

#219

Post by r00fus »

post_break wrote: post #192
wannacboatmovie wrote: post #9

Interesting to note that Ford's approach of updating software is far more conservative and car-like. It can be done fully offline via USB, but requests that you kindly upload the log files written to the memory stick back to them when complete, in the instructions as a necessary step. Presumably so they can track and stop incidents like this before they happen fleet-wide. Rivian seems more like a "ship it and we'll f…

Fords approach is flawed however. You can still update sync with a bad update and bork it over usb. Ask me how I know.

Pray tell, how painful was your discovery?

Re: Rivian software update bricks infotainment system, fix not obvious

#220

Post by AlotOfReading »

DannyBee wrote: post #181
MichaelZuo wrote: post #22

Earlier quoted context omitted.

The 'early release tests' weren't testing an identical copy of the actual update?

It's probably closer to: The test vehicles accept test/prod signed versions Regular vehicles only accept prod signed versions They are otherwise identical. The test vehicles were sent test signed versions The prod vehicles were sent the exact same update, signed with test. This would not be uncommon since the test vehicles probably occasionally run test releases for debugging. Further, the update is probably multiple…

It'd be pretty silly to implement an OTA scheme that didn't check signatures before installing updates. That would mean any random attacker could soft-brick the module by sending an invalid image, which a development image should be to a production vehicle.

You could get this situation if the application code accepted signatures the bootloader does not though. I can imagine that accidentally occurring.

Post reply on HN