Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

571–580 of 804 posts

Re: The August 17 outage

#571

Post by dilyevsky »

solatic wrote: post #558
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected many teams) on what our PriorityClasses should be and to migrate workloads to have priorities. And this is something relatively easy to implement - no developer work required, and practically no YAML to write. Why not? Because sadly, fundamentally, most workplac…

What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.

Re: The August 17 outage

#572

Post by rfgplk »

sph wrote: post #559
brookst wrote: post #462

Earlier quoted context omitted.

I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.

It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in…

> I could have accepted people saying "I'm 20% more productive", which is an incredible achievement by itself, but not the 10x, 20x, 100x I keep hearing about. I think I've read 200x this week.

The ratios are factual though. Just look at the "Insights" tab of any LLM written project. https://github.com/oven-sh/bun/pulse

This kind of velocity is impossible to achieve manually.

Re: The August 17 outage

#573

Post by ieie3366 »

madrox wrote: post #327

I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free. I've been saying this for a while: https://news.ycombinator.com/item?id=47534499

They are effectively getting spammed / ddosed by vibe coders pushing slop nobody uses.

A majority of this increased traffic is the codex/claude in auto mode used by people who don’t even know what git is.

Re: The August 17 outage

#574

Post by darkwater »

teiferer wrote: post #549
0x696C6961 wrote: post #511

Earlier quoted context omitted.

This is anecdotal, but I know that a lot of my coworkers and coasting and putting up one AI generated PR per day which they've hardly even self reviewed.

In a good company that will come back to bite them next performance review. So, if it doesn't then you learned something about your workplace (and it's not good).

> In a good company that will come back to bite them next performance review.

In a good company that should be discussed in the next 1:1s so actual change can happen meanwhile. If it just waits for the end of year review, then it's not a good company.

Re: The August 17 outage

#575

Post by rfgplk »

ozgung wrote: post #508
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

Has anyone noticed an increase in the quality, performance or capabilities of the software they use?

I have. In fact, some of my hand rolled stuff is actually _provably_ faster and more secure then even heavily battletested and widely used "industry standard" solutions. This is mainly due to them passing an exhaustive barrage of millions of lines of code of tests (literally, in fact I just did a pass over all my tests/ dirs and it's sitting at 6.3 million as of today) ranging from adversarial CVE probing attacks to fuzz tests. Ironically, my same testing suite has caught _numerous_ bugs in production stacks (openSSL/libuv) particularly, literal hard SIGSEGVs and the like.

On CVE probing, and I haven't really seen anyone describe/use it (or I may be oblivious), but the way you do it is you curate a list of CVEs for the class of software you're writing, say a web server. Then you take this list in chunks and hand them off to your agents to devise and implement adversarial technically analogous attacks against your codebase. If it's red, report and patch. Ironically (even with Fable 5) it's never complained/refused to do it.

Re: The August 17 outage

#576

Post by user43928 »

croes wrote: post #564
user43928 wrote: post #552

Earlier quoted context omitted.

Nobody is concerned about the costs and environmental impact of data centers. When someone in a discussion about the benefits of AI goes "did you think about the environment?!", it's always performative. The real motivation is disliking AI itself or doubt about the government's ability to offset the labor market impact. Discussions that start with feigned concerns being raised are nearly always going to be unproducti…

Wrong. I don’t dislike AI but really of mine are negatively affected by climate change and AI isn’t helping what is easily observed when Google and MS scrapped their CO2 reduction targets. So every time I use AI I think about the necessity and usefulness of what I‘m doing with AI and if the use outweighs the costs. Since the rise of AI the environmental impact doesn’t seem to matter anymore. I guess because it’s the…

You're not helping your case by questioning the usefulness of software produced with AI in the same comment section, or complaining about other people supposedly compromising their own privacy in overusing AI.

The reason it doesn't matter is because the environmental impact is moderate, and the benefit obviously tremendous.

Re: The August 17 outage

#577

Post by vincnetas »

fsloth wrote: post #520
yread wrote: post #512

Earlier quoted context omitted.

Other projects get unmaintained with maintainers burnt out by a torrent of vulberability reports

This I don't understand. If it's not your job, then just ignore the reports. If it's actually critical, someone will put money on the table and then it's a business. And then it's about scheduling and resourcing - also should not burn anyone out. Just because many people have false sense of entitlement as soon as they get a free offering, it does not mean anyone needs to accommodate them.

Not everyone acts rationally even when knowing that they act irrationally.

Re: The August 17 outage

#578

Post by cube00 »

tclancy wrote: post #377
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour Not to join the parade, but what would a unit test that confirms a cycling behavior across all the instances in-flight even look like? I mean, besides "Not a unit test".

Mock out the network call (which you'd be doing anyway because unit tests never connect to the network) so it always returns a failure. Mock out the timer.

Call the function multiple times and ensure it'd passing the expected wait durations in for each time it's called followed by a fatal error after say 30 seconds.

Re: The August 17 outage

#579

Post by vincnetas »

brookst wrote: post #462
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.

Best code is the one that you have not written :) Because the goal is not the code , it is the things that code does, and if it can be done without code, its the best code. Also if you produce lots of code that does not do anything in reality, then its worst code.

And yes, we can rebut that with "time you enjoy wasting is not wasted" except of course some externalities, like boiling earths oceans.

Re: The August 17 outage

#580

Post by cube00 »

bluerooibos wrote: post #302
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour That wouldn't be a unit test - that's more like an end-to-end or integration test. Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.

I'd disagree it needs end-to-end. Testing that proper sleep calls are made followed by an error can be done in unit tests.

Unit tests already mock network calls so create mocks to return errors instead.

https://news.ycombinator.com/item?id=49384968

Post reply on HN