Earlier quoted context omitted.
I don't know how to delete things.
And what would be the expected result (other than spending $20 a month)?
Automated slop?
The August 17 outage
181–190 of 804 posts
Re: The August 17 outage
#182Earlier quoted context omitted.
I'm not following your line of questioning. Without ever using github as an online editing platform, you can do one push with two new commits.
Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push. You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.
>You can rig up a local ide to pathologically commit+push per save
The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder, then does a commit on the temp folder and pushes to a an intermediary repo in github which then triggers an action that causes the 3rd party system to update from the intermediary repo. This way we don't pollute our main source repo with a commit every time we save an update to a source file.
It's not my favorite way to develop but it's caused us no real problems except when github goes down.
Re: The August 17 outage
#183The trend doesn't seem sustainable.
Re: The August 17 outage
#184"... these incidents make clear that we must accelerate this work."
It feels like GitHub maybe needs to slow down? 'We must change things faster' is a wild way to start off an eight hour hard-down postmortem.
Re: The August 17 outage
#185Earlier quoted context omitted.
Exponential backoff is your friend... too few people use it.
Jittered exponential backoff. You don't want the whole herd to come back at the same time, you have to add timing jitter to the clients.
Possibly dumb/silly question.... Are there any sorts of reverse proxies out there that provide their own layer of jittered/exponential backoff based on patterns? (i.e. requesting IP, cookie, etc.)
I suppose the main reason I think it might be a bad idea, is that it would add complexity to the reverse proxy (i.e. now it's having to track whatever thing is being used and that complexity itself becomes a potential failure point...)
(To be clear, the clients should have their own backoff procedures, but I'm thinking about cases involving naughty clients, which are sometimes a harder problem to correct for...)
Re: The August 17 outage
#186Are retries bad?
These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic.
I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.
Re: The August 17 outage
#187Great read - I'm glad they realize there's work ahead but what I'm missing is: * Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value. * Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust. * Paid customer…
This. I own a small company with 5 people. I pay Github $250/m. I'm sorry but the narrative of, "look at this burden we have, it's hard to take care of all of this code!" is pretty insulting when I'm paying $50 per person per month to host code and run CI pipelines. If they do not want my money, I'll find a company who does.
Re: The August 17 outage
#188> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded). If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go gre…
CAP theorem. Pick one of those.
Re: The August 17 outage
#189> Since April, monthly commits have grown from 1.4 billion to 2.9 billion
Bonkers.
You can tell the entire industry is in a "productivity panic" and here's more proof. There's a velocity zealot crying tears of joy somewhere.
Re: The August 17 outage
#190Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…
It seems like retries are sometimes best left to the human being in front of the screen. Works well enough.