Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

91–100 of 804 posts

Re: The August 17 outage

#91

Post by jdm2212 »

yipinwong wrote: post #88
jdm2212 wrote: post #35

Earlier quoted context omitted.

How would any of what you're saying help with this? > The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually…

load balancer failure? rate limit woudl address concurrency limits? throttle or queue up messages. auto-scale failed cause was misconfiguration policy, which i admit cannot be handled by my suggestions. The cascade? it's downstream service degradation, which I mentione should have had been prevented with queues. One of the jobs that queues/kafka solve is to prevent these downstream outages.

If your LB is down, you're just kind of screwed. You can't enqueue things if requests aren't getting through at all. Same deal with authn/authz issues, which they also had. If you can't answer the question "is this message allowed to be added to the queue" you can't enqueue stuff.

GitHub does use queueing for all kinds of stuff internally, though, because they're not morons.

Re: The August 17 outage

#92

Post by [deleted] »

arn3n wrote: post #78

Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI. I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.

[deleted]

Re: The August 17 outage

#93

Post by Aachen »

r3trohack3r wrote: post #75
pixl97 wrote: post #44

Earlier quoted context omitted.

Exponential backoff is your friend... too few people use it.

Don’t forget jitter!

I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference.

Do you know of a documented instance where it would have helped?

Re: The August 17 outage

#94

Post by shimman »

ivraatiems wrote: post #11
mort96 wrote: post #5

Earlier quoted context omitted.

If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set. If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.

The issue with these systems is that they lack Github's sophistication for issue tracking, knowledge transfer, and automation. I think Gitlab is a mature product in its own space and unlikey to change, for instance, at this point. Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but…

Dude no one uses github tracking for anything serious, come on. The only thing github has over gitea/forgejo is discussions. That's the only real social "innovation" github has contributed to open source development and it's just a shitty tacked on forum.

Also for accuracy, Codeberg has a pro-human and anti-corporation stance. Both of which are definitely en vogue at the moment.

Much better than GitHub's pro slop sentiment, which is doing nothing but destroy their reputation.

Re: The August 17 outage

#95

Post by Aachen »

lysace wrote: post #86
sajithdilshan wrote: post #62

[flagged]

"Most people use GitHub and features for free" Do you have a source for that factoid? (I suspect the vast majority of Github resource usage is paid. And we are upset.)

That sounds unusual for a free platform (not a limited trial but an actual free tier). Isn't it usually the case that only some small percentage can be convinced to pay?

Re: The August 17 outage

#96

Post by DANmode »

bpavuk wrote: post #39
ivraatiems wrote: post #3

"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products." Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak. Github is ripe for disruption and I hope it is disrupted soon.

I'm betting on Tangled and Codeberg. Tangled has a better press and in general is a dark horse, Codeberg has the "brand" and some network effects from projects that moved to there. (famously, Zig.) I heard that Sourcehut is having a moment as well, and I love the idea of email-based workflow and not having to have an account to contribute to someone's project hosted there, but I'm not maintaining anything worthwhile…

What if it was $2?

Re: The August 17 outage

#97

Post by trymas »

mikert89 wrote: post #84
blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

alot of distributed systems in big companies grow at this rate, its not exponential, its mundane

Eh, I would be more empathic in this situation[0].

Github isn’t small startup, where other 10x threshold is as cheap as buy bigger box in your IaaS.

When you are already biggest player in the ecosystem and you suddenly get 10x persisted traffic, with at least 30x+ forecast “soon” - I am not surprised they have issues.

[0] even at current MS owned github

Re: The August 17 outage

#100

Post by christophilus »

Aachen wrote: post #93
r3trohack3r wrote: post #75

Earlier quoted context omitted.

Don’t forget jitter!

I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference. Do you know of a documented instance w…

If you’re talking about internet clients, I think the real world provides sufficient jitter. If you’re talking about a fleet of clients on your 10gbps network, jitter might be useful.
Post reply on HN