That is amazing
The August 17 outage
431–440 of 804 posts
Re: The August 17 outage
#432Earlier quoted context omitted.
You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
For resolution of a problem you can’t prevent categorically, you have to spend a good bit of time pretending to be the computer before you can trust the computer to do it. A run book is a draft of a requirements doc for a program to do the same thing. And you can’t afford to test that program in production without loads of simulations first. And that’s running the process manually and checking aggressively for any signs of problems.
Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO.
They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went.
A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else).
A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before.
Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
Re: The August 17 outage
#433Earlier quoted context omitted.
This is why you have circuit breakers upstream. Not on every individual instance.
In highly distributed microservice architecture, there's almost never a single upstream. In some cases you may have a couple of customer-facing entry-points (a global API gateway, and a couple of BFFs), but these are not the only paths that need to be protected. There are client-side retries (which have broken GitHub in this case) and server-side initiated API calls between microservices that don't pass through any o…
I completely agree, I currently manage a fleet of microservices that handles a few trillion requests a month. It’s about defense in layers to these sorts of things. All the way through the stack if possible starting at the edge.
At least it should be required for critical level services in production.
Re: The August 17 outage
#434Earlier quoted context omitted.
This is why you have circuit breakers upstream. Not on every individual instance.
Doesn't do you any good if the outage is in the circuit breaking layer, which it was for GitHub (this started as a load balancer outage).
Ideally you have levers further up from your local load balancers as well. Even at the edge. Granted you never want those to trigger but it’s better than fighting a storm while you fix things.
Re: The August 17 outage
#435Earlier quoted context omitted.
AI finding issues in code and reporting them so that an AI can review and triage them for another AI to fix.
I mean, isn't that the dream? I don't know if that's sarcasm or not. I know it doesn't work , but that's the future we've been promised, right?
> I mean, isn't that the dream?
Unironically: no.
Re: The August 17 outage
#436Earlier quoted context omitted.
Jittered exponential backoff. You don't want the whole herd to come back at the same time, you have to add timing jitter to the clients.
Possibly dumb/silly question.... Are there any sorts of reverse proxies out there that provide their own layer of jittered/exponential backoff based on patterns? (i.e. requesting IP, cookie, etc.) I suppose the main reason I think it might be a bad idea, is that it would add complexity to the reverse proxy (i.e. now it's having to track whatever thing is being used and that complexity itself becomes a potential failu…
In general, helping prevent retry storms is one of the functions of a service mesh. E.g. Istio lets you configure exponential backoff with jitter, linkerd provides a retry budget and loadshedding based on backpressure.
“Adding complexity to the reverse proxy” is kind of the central feature of service meshes, but the point is you pick one that’s battle tested and solves more problems than it creates.
Re: The August 17 outage
#437"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.
So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?
You already know the answer to that question
Re: The August 17 outage
#438Earlier quoted context omitted.
> I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. That knowledge will be worthless 12-18 months from now when AI does everything better than you, including “understanding”. If you’re one of the world’s best programmers, it might be 24 months instead, but the writing is on the wall for everyone. I wonder if people were behavi…
It will be interesting to see if LLM can push beyond the intelligence embedded in language. General intelligence may still be a ways off.
General intelligence is a convenient myth perpetuated by the marketing departments of AI corpos to help further their pursuit of regulatory capture. Don't fall for it.
Re: The August 17 outage
#439Earlier quoted context omitted.
For some context, in June they said commits "commits nearly doubled year over year, crossing 1.4 billion per month". Now, it has more than doubled that in just a few months. https://github.blog/news-insights/product-news/github-copilo...
Makes sense given the ubiquity of agentic coding. I made a joke to my coworker today that all we do is make sure AI agents can communicate with other AI agents.
It’s funny how the quality of the average commit message has gone up, now that we rarely read or write them directly anymore.
Re: The August 17 outage
#440Earlier quoted context omitted.
Bonkers is right. Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better? Software and apps in particular are getting worse, normies hate AI more than ever because they're even less likely to get their desired outcome when calling their doctor or trying to get their online order refunded when chatting with a cutely named chatbot, wages for (most…
This comment comes up over and over again and it's incredibly ignorant. To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before. I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public
Why is it incredibly ignorant? Where is he wrong? We have definitely lost the plot.