> Since April, monthly commits have grown from 1.4 billion to 2.9 billion Bonkers. You can tell the entire industry is in a "productivity panic" and here's more proof. There's a velocity zealot crying tears of joy somewhere.
The August 17 outage
191–200 of 804 posts
Re: The August 17 outage
#192Earlier quoted context omitted.
No doubt. ...You can have non-zero revenue and still be loss-making though.
Conversely, what suggests GitHub has a huge operating cost? Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right? It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitH…
Re: The August 17 outage
#193Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…
Re: The August 17 outage
#194> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded). If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go gre…
Re: The August 17 outage
#195Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…
Re: The August 17 outage
#196Here's Vladimir Fedorov's GitHub contribution graph, as linked to as the author of this post: https://imgur.com/a/zIbT0Gi It shows zero contributions in the past year, on this account. This is a huge, huge red flag.
I strongly disagree. GitHub, a year ago, acknowledged the fundamental problems and began work on them. We all agree with the diagnosis and strategy: stop building new things, bring stability. Why would whether the CTO codes have any bearing on the correctness of this strategy? GitHub’s problem isn’t that leadership don’t understand the product, or that they don’t know what they should be doing, it’s that they’re batt…
Where I have a problem with the positioning of their GitHub profile is, he's the CTO of GitHub, arguably the defacto standard for open-source version control systems. His GitHub profile is linked to as the author for the post, and his GitHub profile simply tells me: "this guy doesn't code."
I don't care if this guy doesn't work on GitHub itself, I hardly would expect that, but IMO, any CTO of a company like GitHub should eat, breathe, and sleep code. He might, but his profile, which is being published as if it means anything, tells me he doesn't.
Re: The August 17 outage
#197Earlier quoted context omitted.
I'm not following your line of questioning. Without ever using github as an online editing platform, you can do one push with two new commits.
Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push. You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.
A push pushes commits and blobs and trees and tags. It’s an interesting metric to track, but the core unit of complexity (and expense) worth tracking on GitHub’s side is obviously the commit.
There’s a difference between pushing 1 commit and 100.
Re: The August 17 outage
#198Re: The August 17 outage
#199Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…
In my experience, errors which go away within a few seconds are quite rare, and are mainly due to flaws which are usually caught in testing.
I think a very careful cost/risk/benefit analysis should be done when adding automatic retries to things. As well as potentially causing cascading failures, it is a degraded user experience when it doesn't succeed.
As a user I would rather see an error straight away than see many seconds of spinning while something silently retries, and THEN an error.
Re: The August 17 outage
#200Earlier quoted context omitted.
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded). If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go gre…
Anyone who designs such a system should know to use an exponential backoff to avoid the thundering herd. Maybe copilot missed that while it was reviewing its own PR