371–380 of 386 posts
Earlier quoted context omitted.
Having worked on 25K machines, I can assure you that it never deployed to every single machine and failed to do so in interesting ways all the time.
As a frontend web developer I'm constantly deploying software to many thousands of machines. And you know what? It's pretty damn simple.
Earlier quoted context omitted.
Teslas occasionally need to reboot / hard reset their software too, when driving no less, and during that period all that information, and most of the controls, are unavailable (like windshield wipers, etc.)
> Teslas occasionally need to reboot / hard reset their software too, when driving no less Move over, Microsoft, I think I've found an update policy I hate even more...
Earlier quoted context omitted.
It always deployed. It was eventually consistent. Any failure would automatically be resolved after a period of time.
Interesting. At any point in time, I had errors from hardware, software and networking. Even the racks would be getting overwhelmed at certain times. Simply being able to ssh into every host wasn't guaranteed. I'm not sure how you did it.
I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…
Rivian is an embedded use case, though, which is not at all like a fleet of servers. Having worked for companies that produce network devices - including devices that are unreachable for example for 6 months of the year - and on software installation and upgrade, I am baffled how this bricking is possible. For one thing, you generally use some kind of confirmed boot mechanism - you upgrade a standby partition, set an…
That made me think, imagine NASA bricking up the voyager with a SW update.
I built a whole remote software update mechanism for a control binary that ran on 25k+ servers across multiple data centers. Rest assured that after the first time I messed it up (which required ssh into each box individually), I wrote a lot of unit and integration tests to make sure that it never failed to deploy again. One of the integration tests ensured that the app started up and could always go through the inte…
> While I fully understand that this is hard to get right 100% of the time, a mess up of this level by a car manufacturer is pretty amazing to me. I feel like it's going to happen to someone that makes network devices eventually. I'm always scared to update my (several hundred) UniFi devices. Their update process isn't foolproof and they push auto-updates via the UI pretty hard. Several years ago they caused some peo…
Earlier quoted context omitted.
But average quality was a bit higher because nobody thought 'oh, we'll fix that next week in the OTA update'.
I don’t think so. This is the same thought people apply to cartridge based games for example, before software updates. But the reality is that the same types of bugs would likely exist. Nothing about the engineering aspects has really changed other than more features. All that would happen is future revisions would have it fixed and early buyers would be stuck.
But you're not looking at near endless lists of known issues and 'wontfix' wasn't a thing back then.
Earlier quoted context omitted.
Interesting. At any point in time, I had errors from hardware, software and networking. Even the racks would be getting overwhelmed at certain times. Simply being able to ssh into every host wasn't guaranteed. I'm not sure how you did it.
+1 to this, we have a 0.1% hardware failure rate every time we do a rolling restart (40-50k nodes). Some just never come back, in the best case, but actively misbehave in the worst. If the node is unresponsive we remove it from the cluster and fix it async.
If the machine had crashed, it would start up, start my daemon, and that daemon would start the ping/update process all over again.
A large portion of the machines were iPXE booted... so, just reboot was one option and it would all start from scratch again.
Yes, some of the boxes had flaky power supplies or would fail an ssd, and that would cause a technician to go out and manually fix things.
I found it was critical to think of everything as eventually consistent because my hardware was boxes with 12 GPUs and they were flaky and would crash the whole box randomly. I got used to boxes rebooting hundreds of times. My process would also auto-tune the GPU for stability too, changing clock/power settings until the individual cards would become stable and stop the crashing.
The only time I had problems was when the daemon was dead. I had a dashboard where I could see which machines hadn't reported their status. It was easy to pick those off by hand.
Earlier quoted context omitted.
> Teslas occasionally need to reboot / hard reset their software too, when driving no less Move over, Microsoft, I think I've found an update policy I hate even more...
You never had Windows force-reboot your system during an unrequested background update? Happened twice to me.
It's crazy to me that this is possible in the first place. Standard practice is to have a fleet of test vehicles that are effectively production except in an early release group. Or, you know, having an A/B boot partition scheme with a watchdog. Things that have been around for decades at this point. Disclaimer: Former Googler, Worked closely with Automotive.
Yeah... I worked on an embedded project with literally 2 engineers, and we had an A/B partitioning scheme, and a recovery partition (we fully qualified the recovery image and it was flashed to the units on day 1, it was guaranteed to boot and it would just sit and wait for the user to initiate a firmware load). The app on the device would reset a U-boot variable once it was successfully loaded, so U-boot could check…
That just felt like a massive product to build and maintain for what really could have been backed by AWS iam. GCP IAM if they really really needed hierarchy. I guess I'm not surprised at this outage.