Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

81–90 of 804 posts

Re: The August 17 outage

#81

Post by seizethecheese »

toephu2 wrote: post #77
blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

Not that impressive when you realize it's mostly due to AI slop edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.

Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).

Re: The August 17 outage

#83

Post by anvuong »

cyberax wrote: post #53
bluedino wrote: post #33

Earlier quoted context omitted.

> We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure. That can't be cheap.

A server box now has around 256 CPU cores. So that's about 12000 servers. If each one is $10k that's $120 million. Not a lot compared to Github's income.

$10,000 is only enough for CPU cost (2x AMD EPYC 9754). The rest of the servers (mainboard, RAM, storage, power supply, network card, rack, cooling, etc.) can easily triple or quadruple the cost.

Re: The August 17 outage

#84

Post by mikert89 »

blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

alot of distributed systems in big companies grow at this rate, its not exponential, its mundane

Re: The August 17 outage

#87

Post by Aachen »

seizethecheese wrote: post #81
toephu2 wrote: post #77

Earlier quoted context omitted.

Not that impressive when you realize it's mostly due to AI slop edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.

Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).

How is it impressive if we all know it's autogenerated? There's no more people there than there were before. Heck, at ~2x growth that's possibly a decrease in real humans there since bots generate loads of them per person

Re: The August 17 outage

#88

Post by yipinwong »

jdm2212 wrote: post #35
yipinwong wrote: post #27

Earlier quoted context omitted.

What they can implement is to slowdown the commit rate, rate limt or just queue-up messages not to overburden their downstream service. I don't think GH has any of those, but just keep scaling, but that scaling failed. Just bad architectural decisions from the postmortem. -- It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource. They can scale but not scalable in ter…

How would any of what you're saying help with this? > The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually…

load balancer failure? rate limit woudl address concurrency limits? throttle or queue up messages. auto-scale failed cause was misconfiguration policy, which i admit cannot be handled by my suggestions. The cascade? it's downstream service degradation, which I mentione should have had been prevented with queues. One of the jobs that queues/kafka solve is to prevent these downstream outages.

Re: The August 17 outage

#89

Post by shimman »

seizethecheese wrote: post #81
toephu2 wrote: post #77

Earlier quoted context omitted.

Not that impressive when you realize it's mostly due to AI slop edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.

Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).

Is it impressive? All it's doing is decaying the services. 15 years ago never have imagined I would go to the lengths to host a github alternative on a VPS but after doing just this (also being the last one in the my professional group to do so), GitHub is giving a master class in destroying their reputation in pursuit of advocating for hostile entities.

Not all growth is good, especially growth that is actively hurting the company.

There is an equilibrium in both nature and software. Purposely designing systems that mimic the effects of cancer is going to benefit who exactly?

Re: The August 17 outage

#90

Post by tracerbulletx »

mikert89 wrote: post #84
blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

alot of distributed systems in big companies grow at this rate, its not exponential, its mundane

When a service that was already the primary git hosting provider for most of the world for 20 years grows at that rate its not mundane and its not comparable to any example.
Post reply on HN