"The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR " - silly question but github uses github for their own PRs and deploy right? Do they have a special dedicated system just for them so they can fix github with a code change even if the rest of us can't?
The August 17 outage
661–670 of 804 posts
Re: The August 17 outage
#662Re: The August 17 outage
#663Earlier quoted context omitted.
This comment comes up over and over again and it's incredibly ignorant. To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before. I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public
> I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public On the other hand, famine is coming. Come back and read this next year. Was your dalliance worth these food shortages?
Hang on! What's the proposed causal chain from AI to famine here? Inference -> energy prices -> fertilizer/food prices? Or are you referring to the war?
Re: The August 17 outage
#664> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…
So, in all of your software, you have introduced randomness in your retries so that the billions of your clients avoid retry synchronization dances?
I'm not asking for randomness, I'm only asking it to stop retrying after say 10 seconds and report it to the user as a fatal error.
Re: The August 17 outage
#665Earlier quoted context omitted.
On the topic of translation/ language learning, I still find Google Translate better for when I need to type emails in Russian (my older family who only speak it) as far as grammar is concerned than what LLMs spit out.
I'd be surprised if Google Translate isn't powered by a specialized, Transformer-powered model we would now recognize as a "small" language model heavily constrained to a specific task.
that already happened in like 2016
Re: The August 17 outage
#666Earlier quoted context omitted.
Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...
actually, I think this time it is the other way around, Azure is more the solution than the problem. Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead ti…
A company suffering a spam attack would normally choose to deal with it by banning, and optionally prosecuting, the spammers.
Re: The August 17 outage
#667Earlier quoted context omitted.
actually, I think this time it is the other way around, Azure is more the solution than the problem. Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead ti…
Agreed on not hosting things yourself, but, I am not arguing against cloud hosting at all - just Azure. If they just came out and admitted it is a disaster and swallowed their pride and moved their services towards anything else at all - GCP, AWS or whatever else - I think their uptime would significantly improve. Of course, the fundamental problem here is the culture of the company itself. That's harder to fix.
Related: inside perspective on why it's a shitshow
Re: The August 17 outage
#668Earlier quoted context omitted.
Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...
I read the article and found it interesting. It's wild to me that when I think MS I think this article, Vista, Windows Phone, Zune, Metro, ads in the start menu, the pleated pants meme and so-on, but I have coworkers that are total evangelists for MS technologies and platforms. Clearly a lot of devs get value from the ecosystem.
A lot of their platforms (not Azure) used to just work. Visual Studio and .NET were solid. The Windows NT line was pretty good for servers too (but different from Linux, and not as flexible by virtue of not being a bazaar). It was at Windows 8 era they turned it into shit, first by designing a UI for tablets (fine) and then putting that UI on desktops (wtf) and silently upgrading Windows 7 users without their consent.
Re: The August 17 outage
#669Earlier quoted context omitted.
I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding. And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, b…
You understand it if you think of engineering infinites rather than mathematical infinites. The capacity of a full single datacenter can be treated as infinite for most customers. I explained how its defined in the original comment. Amount of places in engineering where you treat even a 3 order of magnitude difference as infinite is a lot, but the number of order of magnitudes varies depending on context.
Maybe if you're the only customer, but it seems you've not yet gotten a "no capacity" error from AWS, which happens sometimes.
Re: The August 17 outage
#670Earlier quoted context omitted.
Yes, just absolutely crazy way of doing load shedding. > why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway? Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario. > Application backends? I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only wo…
Website being down is not worst-case scenario.
The errors were pushed to the client just like you suggested...