Live data from Hacker News

The August 17 outage

github.blog

631–640 of 804 posts

Re: The August 17 outage

#631

Distributing across different services wouldn't be a bad idea.... I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but... Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (exce…

[dead]

Re: The August 17 outage

#632
post #520

Earlier quoted context omitted.

This I don't understand. If it's not your job, then just ignore the reports. If it's actually critical, someone will put money on the table and then it's a business. And then it's about scheduling and resourcing - also should not burn anyone out. Just because many people have false sense of entitlement as soon as they get a free offering, it does not mean anyone needs to accommodate them.

> If it's not your job, then just ignore the reports. If you have a highly conscientious personality, this is easier said than done.

" highly conscientious "

Just doing what others wish is not conscientous in itself! It _may_ be depdending on situation but it can be just pathological towards the self.

When it's psyhocologically hard to do things you imagine will dissapoint someone that's probably not concientousness. It's more like low self-esteem or codependency.

It's very hard for someone to tell these apart themselves. Hence when this topic pops out it's good idea to remind that being super-accomodating may in fact be a personality flaw - that can be healed if acknowledged.

There is very large spectrum between "trying not to dissapoint anyone" and doing what you know is the right thing.

Re: The August 17 outage

#633
post #630

Earlier quoted context omitted.

They could do this in a way that lessens external pull requests. For example, bolting co-pilot on to github, or a Codex in the web kind of thing that gives unlimited check ins. It's like how Grok Heavy gives the user X premium or whatever. You charge for the tokens and give the unlimted premium access as a bonus. Basically, bundle it.

> bolting co-pilot on to github They already did this, no?

I meant as a prerequisite to unlimited checkins.

Right now anyone can publish to public repos in an unlimited manner. They could choose to limit that and elevate unlimited to a paid co-pilot of codex bundled plan.

Re: The August 17 outage

#634

Earlier quoted context omitted.

Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler. > basically just one for system and majority of serving workload ran on another priority and the rest was for batch RCA blames in-house load-balancing services (HAProxy) that rea…

Yes, just absolutely crazy way of doing load shedding. > why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway? Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?

The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario.

> Application backends?

I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only workloads continue to succeed.

Re: The August 17 outage

#635
post #621

Earlier quoted context omitted.

You're not helping your case by questioning the usefulness of software produced with AI in the same comment section, or complaining about other people supposedly compromising their own privacy in overusing AI. The reason it doesn't matter is because the environmental impact is moderate, and the benefit obviously tremendous.

The environmental impact is anything but moderate and the benefits are not obviously tremendous at all. I'm happy using Claude Code as much as the next guy, but saying that the impact has been "tremendous" is vastly overstating the actual results.

I disagree, with the projected doubling by 2030 we're looking at 3% of global electricity consumption or 3.4 EJ, less than 1% of final energy consumption.

That is moderate. Energy-intensive industry is around 130 EJ, and global final energy consumption > 450 EJ.

Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035.

Now that was about operational energy consumption. Someone might bring up manufacturing and construction.

From what I could find the climate impact of those are estimated somewhere between 10-35% of the total climate impact of data centers, so relatively small compared to the operational energy consumption.

It is very hard to justify more than moderate environmental impact here, in my opinion.

For the benefits of AI, my personal results have been great, so I am quite optimistic. And objectively, I find it hard to ignore recent results in mathematics and security research.

Re: The August 17 outage

#636
I recently received a PR fully automated by Claude for an 8 years old repo. The bug is legit and the scope it affects is larger than what that PR addressed, but I no longer care too much about that legacy code anyway, so I also let Claude run free for the first time in my life, from handling that PR to fixing all related bugs.

I walked away for half an hour and back, found Claude opened and merged 9 more PRs and added a comprehensive CI for testing for all platforms. It will likely take me months to reach this level of output, but only half an hour for a capable agent.

No wonder why GitHub is down all the time.

Re: The August 17 outage

#637
post #560

Earlier quoted context omitted.

Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle. Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.

I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding. And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, b…

You understand it if you think of engineering infinites rather than mathematical infinites. The capacity of a full single datacenter can be treated as infinite for most customers. I explained how its defined in the original comment. Amount of places in engineering where you treat even a 3 order of magnitude difference as infinite is a lot, but the number of order of magnitudes varies depending on context.

Re: The August 17 outage

#638
post #602
post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...

actually, I think this time it is the other way around, Azure is more the solution than the problem.

Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead times of many weeks, especially in the current situation, datacenter capacity is even longer and more difficult, especially right now.

Choosing not to deal with scaling the hardware is a valid choice in this situation. Yes, Azure is a bunch of servers held together with glue, duct tape and a lot of hope, but I think, the github hardware is not much better at the moment.

Re: The August 17 outage

#639
Linux didn't (yet) kill Microsoft. Microsoft absorbed that shot. Then the Git arrow went straight to cold black heart of Microsoft. The next few months will determine if they survive it. If they do, what will we see from the third draw out of Linus' quiver?

Re: The August 17 outage

#640
post #602

Earlier quoted context omitted.

Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...

actually, I think this time it is the other way around, Azure is more the solution than the problem. Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead ti…

Agreed on not hosting things yourself, but, I am not arguing against cloud hosting at all - just Azure. If they just came out and admitted it is a disaster and swallowed their pride and moved their services towards anything else at all - GCP, AWS or whatever else - I think their uptime would significantly improve. Of course, the fundamental problem here is the culture of the company itself. That's harder to fix.
Post reply on HN