Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

601–610 of 804 posts

Re: The August 17 outage

#601

Post by Xunjin »

vincnetas wrote: post #579
brookst wrote: post #462

Earlier quoted context omitted.

I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.

Best code is the one that you have not written :) Because the goal is not the code , it is the things that code does, and if it can be done without code, its the best code. Also if you produce lots of code that does not do anything in reality, then its worst code. And yes, we can rebut that with "time you enjoy wasting is not wasted" except of course some externalities, like boiling earths oceans.

But what about my pet project? The one that I just do for the fun o seeing AI go brrrr, do you mean now I should care about the Earth?

Note: It's sarcasm.

Re: The August 17 outage

#602

Post by neya »

afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it.

Worth reading:

https://isolveproblems.substack.com/p/how-microsoft-vaporize...

Re: The August 17 outage

#603

Post by menaerus »

sph wrote: post #559
brookst wrote: post #462

Earlier quoted context omitted.

I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.

It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in…

Hey, traditional hand coder fellow ... the times have past and the future is already a present. I never have been this productive before, and it's been ~20 years that I spent coding. System level programming. Few years back I would have said the bottleneck is not in spelling out the code so we wouldn't see that much AI impact but boy I was wrong. Writing code has never been this cheap, both in terms of time resources and $$$. Now I can iterate over the ideas I didn't have the capacity before, both intellectual and time-wise.

Re: The August 17 outage

#604

Post by Brian_K_White »

nosefrog wrote: post #599
vasco wrote: post #560

Earlier quoted context omitted.

Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle. Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.

I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding. And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, b…

It is infinitely larger, because there is no distinction between more than you use, and infinity.

Re: The August 17 outage

#605

Post by dilyevsky »

solatic wrote: post #591
dilyevsky wrote: post #571

Earlier quoted context omitted.

What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.

Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler. > basically just one for system and majority of serving workload ran on another priority and the rest was for batch RCA blames in-house load-balancing services (HAProxy) that rea…

Yes, just absolutely crazy way of doing load shedding.

> why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?

Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?

Re: The August 17 outage

#606

Post by xnorswap »

d0vs wrote: post #561
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

GH didn't collapse so it's not impossible that they already implement these measures: > At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. https://www.githubstatus.com/incidents/zkxwbgr0cnmx

It completely collapsed, 20% is meaningless, as I said in: https://news.ycombinator.com/item?id=49333089

    If you load an issue page, you'll see 1 failed request to:
    
        /project/product/issues/
    
    And sure, that's what you care about, but consider the working requests to:
    
        /in-product-messaging/copilot-budget-request-banner
        /in-product-messaging/code-scanning-ai-findings-preview-banner
        /github-copilot/chat
        /_private/browser/stats
Those are actual endpoints and results.

Re: The August 17 outage

#607

Post by asdfsa32 »

asdfsa32 wrote: post #455
bentt wrote: post #397

Earlier quoted context omitted.

This is like letting someone stay in your house for free while charging them to burn it down.

If you can sell them fuel, it might be a lucrative business.

Thinking more, the object of the "house" is to extract value, whatever it is by rent or letting them burn it down, as long as the correct dues are paid, the mission is accomplished.

Re: The August 17 outage

#608

Post by TiredOfLife »

neya wrote: post #602
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...

So for the past 6 years the bad uptime was because of less than 12% of github being on azure?

Re: The August 17 outage

#610

Post by Superfud »

lbrandy wrote: post #411
Breakthrough wrote: post #357

Earlier quoted context omitted.

You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?

Some anecdotes 1. People add retries w/o backoff. Now we have a retry storm. 2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure. 3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns. 4. People find ways…

"Oh no, I hit a timeout" -> "I've increased the timeout and added a retry" -> "Oh no."
Post reply on HN