Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

111–120 of 804 posts

Re: The August 17 outage

#111

Post by verdverm »

lenerdenator wrote: post #13

We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization. GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when th…

I'm particularly fond of the concepts within git-appraise, gits back to the federated foundations

https://github.com/google/git-appraise (unmaintained)

Re: The August 17 outage

#112

Post by seizethecheese »

shimman wrote: post #89
seizethecheese wrote: post #81

Earlier quoted context omitted.

Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).

Is it impressive? All it's doing is decaying the services. 15 years ago never have imagined I would go to the lengths to host a github alternative on a VPS but after doing just this (also being the last one in the my professional group to do so), GitHub is giving a master class in destroying their reputation in pursuit of advocating for hostile entities. Not all growth is good, especially growth that is actively hurt…

This growth is surely good for github. If another company becomes the "github for AI agents", they'll lose not just their business for AI but also human coders. (Sure, maybe there will be a human coder only github, but it will be quite small.)

Re: The August 17 outage

#113

Post by r3trohack3r »

Aachen wrote: post #93
r3trohack3r wrote: post #75

Earlier quoted context omitted.

Don’t forget jitter!

I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference. Do you know of a documented instance w…

Have experienced it, but didn’t document.

Downstream database of our edge serverless platform went down. A tonne of requests failed all at once. Every service in the microservice request path, and the client, had their own retry policy.

Clients all retried at the same time. Retries amplified in our microservice graph (1 request at the front door ended up with like 10s of retries internally as each downstream microservice along the path retried requests). Request queues backed up and couldn’t drain fast enough. Clients all timed out at roughly the same time. All waited the same time. All retried again at the same time.

It was a pulsing thundering herd of many hundreds of thousands of requests at the front door that was amplified by internal retries.

Had to tune up load shedding to 100% after the database outage was mitigated until the backend recovered then tune it down in increments to restore service.

Added jitter to clients and turned off retries on the serverless platform.

Re: The August 17 outage

#114

Post by dhruvrrp »

yipinwong wrote: post #21

AWS CloudWatch has an option to show the trend and what it will be like after x-period. Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem

The problem is there are a class of problems that only appear after you go over the tip of what your system can handle, which are very difficult to predict or model.

Re: The August 17 outage

#115

Post by spongebobstoes »

ryanisnan wrote: post #105
mvdtnz wrote: post #103

Earlier quoted context omitted.

Mine would look the same if you didn't have access to the private repositories I contribute to at work.

Don't contributions to private repositories simply show as: "N contributions in private repositories"? Here's me: https://github.com/ryanisnan In other words, I think his private contributions should still manifest on the contribution graph. And for being the CTO of an organization like GitHub, with no open-source contributions... Not a great look.

you have to opt in to private contributions being visible like that

Re: The August 17 outage

#116

Post by [deleted] »

xyzsparetimexyz wrote: post #52
brookst wrote: post #34

Earlier quoted context omitted.

It really is. I know I've gone from tens a month to thousands a month. They have to be projecting >100B/month in the next year or two.

wow. they should really institute a maximum amount of individual pushes per-month per-user.

[deleted]

Re: The August 17 outage

#117

Post by airstrike »

Aachen wrote: post #87
seizethecheese wrote: post #81

Earlier quoted context omitted.

Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).

How is it impressive if we all know it's autogenerated? There's no more people there than there were before. Heck, at ~2x growth that's possibly a decrease in real humans there since bots generate loads of them per person

It's an impressively large quantity of X, not an impressively good X.

Re: The August 17 outage

#118

Post by IanCal »

Aachen wrote: post #93
r3trohack3r wrote: post #75

Earlier quoted context omitted.

Don’t forget jitter!

I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference. Do you know of a documented instance w…

It’s not hard to get started, it’s a case of adding small amounts of randomness.

If you have, say, a long poll then kick off all users due to a deploy or error (or a broadcast message) then you can have a situation where you’ve got a huge clustering of connections at 1 minute, which spreads very slowly out as real life issues give you jitter for free. You can avoid this or at least return to normal much quicker by adding some jitter.

It might happen if all your users back off at the same rate too, if the clustering causes a bunch of errors. Error -> lots reconnect 1 minute after -> fail -> lots reconnect 2, 4…

More likely to occur in cases where there’s a way you can have people all connecting at the same time - synchronisation to a real world event is one case and then connecting again at the same time after.

Re: The August 17 outage

#119

Post by SAI_Peregrinus »

pixl97 wrote: post #44
k33P1Tr3aL wrote: post #29

Earlier quoted context omitted.

the 'ol thundering herd problem...

Exponential backoff is your friend... too few people use it.

Jittered exponential backoff. You don't want the whole herd to come back at the same time, you have to add timing jitter to the clients.

Re: The August 17 outage

#120

Post by duped »

sajithdilshan wrote: post #62

[flagged]

I mean, a lot of us have paid GitHub a lot of money for CI on private repos. And when GitHub themselves encourages the insane behavior of vibe coders and agents instead of just charging or rate limiting access of bots, it's hard to give them sympathy.
Post reply on HN