Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rivian software update bricks infotainment system, fix not obvious

electrek.co

Post reply on HN
Print view View original

331–340 of 386 posts

Re: Rivian software update bricks infotainment system, fix not obvious

#331

Post by yreg »

ClassyJacket wrote: post #273
yreg wrote: post #267

Earlier quoted context omitted.

>and most of the controls, are unavailable All the controls related to driving are available. You can use turn signals, wipers, change lights, honk, shift gears, etc. But you are correct that you don't see your speed during the reboot.

That's not true. You cannot shift gears in the Model X, S, or 3 without the touchscreen - the only way to change into or out of park, drive, or reverse is to swipe on the touchscreen. Only the Model Y has a stalk and it is being removed in the next version. Also even on older Teslas lights are controlled exclusively via touchscreen.

I don't know what to tell you, you are wrong. I drive a Tesla, I have specifically checked all of what I mentioned when I had this (in my opinion highly problematic) issue happen to me. Have you actually tried it?

Even the new stalkless cars have gear selectors.

Re: Rivian software update bricks infotainment system, fix not obvious

#332

Post by jrumbut »

onion2k wrote: post #329
jrumbut wrote: post #304

Earlier quoted context omitted.

It's simple because we tolerate certain limitations in the web platform. If you had a hard requirement that a page load could never take more than 100ms, regardless of network conditions, you'd have quite a challenge on your hands.

The laws of physics are definitely very challenging. If you've got a solution please write a blog post.

No blog post required! You just install the whole app on a dedicated piece of hardware on site.

But then deployment becomes more challenging ;)

Re: Rivian software update bricks infotainment system, fix not obvious

#333

Post by ugh123 »

latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

Please tell me you scripted that ssh into across your 25k servers!

Re: Rivian software update bricks infotainment system, fix not obvious

#334

Post by latchkey »

ugh123 wrote: post #333
latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

Please tell me you scripted that ssh into across your 25k servers!

https://news.ycombinator.com/item?id=38270986

One thing my little control process did on the box was to always set the password to be the same... user/1.

None of these boxes needed inbound connections, so it wasn't a big deal to do that.

Re: Rivian software update bricks infotainment system, fix not obvious

#335

Post by yreg »

xenadu02 wrote: post #302
brewdad wrote: post #244

Earlier quoted context omitted.

As someone who lives in a place where it rains almost every day for 9 months of the year, this reinforces my decision to never buy a Tesla. Does this kill the headlights too?

FWIW in two years of ownership we have never seen the touchscreen reboot while driving.

I had it happen two times this year, but the original comment is wrong, you can use the wipers (and also the headlights) while the touch screen is rebooting.

Re: Rivian software update bricks infotainment system, fix not obvious

#336

Post by hef19898 »

LoganDark wrote: post #292
lotsoweiners wrote: post #245

Earlier quoted context omitted.

> This is a good thing, as long as processes are improved to avoid situations like this one. How do you figure. I can’t think of a single thing in my vehicle that could be improved by software. When I buy a car I’d prefer it was done.

It's supposedly a good thing because more features can be added over time, but if they were features worth adding in the first place, the car should have shipped with them already.

That is the big difference between hardware and software engineering: Once hardware is shipped, there is nothing you can do besides repaurs and retrofits requiring a workshop. Software can updated and changed today anytime.

My biggest issue with modern cars, and it seems this is spreading to other embedded systems than cars, is treating those as software: connect them to the web and run OTA updates everytime you need to fix a bug. That requires some form of inzernet cinnection, which requires regular security updates, which require OTA update capability. All that because, bluntly, software devs cannot be bothered with just finishing software running on non-connected hardware that just works, not can they be bothered following hardware development in case hardware is the main component, as in cars. And no, no car manufacturer is a software company, nor phone maker like Apple.

The proper way of fixing software bigs in automotive used to be, again thank you Tesla for breaking something that just worked, to recall the affected cars to a workshop to conduct the software update. Honestly, the only thing on a car that should be updated by the owner is maps for the GPS unit.

Re: Rivian software update bricks infotainment system, fix not obvious

#337

Post by rekoil »

reneberlin wrote: post #266

When will humans be crazy enough to update the firmware of artificial hearts OTA? Updating cars with new features OTA, even "just" an Infotainment can possibly cost lives, because the driver might get confused and isn't putting eyes on the streets. It should be forbidden and every change should be made clear to the driver, shown in detail, and should need verification twice before being accepted. There must not be an…

Not updating cars OTA (yes, even "just" the infotainment) can potentially cost lives as well, as security holes would not get patched until the next service appointment.

Re: Rivian software update bricks infotainment system, fix not obvious

#338

Post by kalleboo »

barbazoo wrote: post #165
BobaFloutist wrote: post #74

Earlier quoted context omitted.

Does it not have an aux port? USB C to Aux cables are pretty cheap and simple.

I know I know but then there's a cable from where the phone is to the middle console where the aux port is. But true, it'd for sure be better sound quality.

You could pull the AUX to a Bluetooth receiver stuck in the 12V port, then you can route the cable a bit neater and still have convenience.

Re: Rivian software update bricks infotainment system, fix not obvious

#339

Post by vmladenov »

fma wrote: post #312
pcchristie wrote: post #305

Earlier quoted context omitted.

Tesla doesn't do that, you're misunderstanding. You can do a hart reboot on the computer by yourself, by holding down both steering will scroll wheels for 10 seconds. Fixes any glitches with the screen.

So the commenter is driving through intersections without blinkers and supposed to do that? I'm not sure if this is a joke or not. I need to teach my grandma how to hard reboot a car without crashing?

A lot of silliness has been tossed around about Teslas in this thread so let's be clear about how they work. A Tesla vehicle have two main independent systems:

- The "car" that operates the motors, blinkers, shifting, Autopilot, etc. It also supplies standard readouts via OBD that you can monitor independently e.g. with a phone app. As far as I'm aware, it is impossible to turn this machine off except when the car is stationary, in park, and the brake is fully pressed (or by disconnecting the 400V battery leads under the rear seat).

- The infotainment computer is a small Linux machine that displays the map, speedometer, music, plays the sound of the blinkers through the speakers, and random other stuff. This system can be reset at any time by holding the two scroll wheels.

There are valid things to criticize about this setup e.g. by not having a standard fallback display of speedo, but you can still read speed from OBD at all times. And Tesla has broken things in OTA updates before, but their rollouts are heavily staged. The last major revert I can remember was FSD beta 10.3 in 2021 back when it had reached a few 10s of testers.

Re: Rivian software update bricks infotainment system, fix not obvious

#340

Post by dcow »

foobiekr wrote: post #243
latchkey wrote: post #174

I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…

Rivian is an embedded use case, though, which is not at all like a fleet of servers. Having worked for companies that produce network devices - including devices that are unreachable for example for 6 months of the year - and on software installation and upgrade, I am baffled how this bricking is possible. For one thing, you generally use some kind of confirmed boot mechanism - you upgrade a standby partition, set an…

Assuming Rivian does know embedded 101, my guess is that the infotainment system is running Android and the watchdog reported all green once the system services all came online and that it doesn't actually check whether the application layer is really working because, as you know, that would require the watchdog to run a full regression suite before giving the okay, which isn’t practical. Since the update swapped the system to an internal dev cert, they cant push an immediate update to change the boot args because the management plane daemon won’t connect to the C&C server, or it can but the blob they push wouldn’t pass signature validation, or the TEE won’t unlock the device keys because the roots changed. Whatever the case, someone has to go blow a fuse and re-flash the thing, or at least rewrite the boot args via serial. Just a guess.

If it is the most likely “management plane TLS certs” issue, I bet the watchdog won’t confirm the new boot args until the command dispatch daemon gets a pong from the C&C server moving forward (:

Post reply on HN