Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

591–600 of 804 posts

Re: The August 17 outage

#591

Post by solatic »

dilyevsky wrote: post #571
solatic wrote: post #558

Earlier quoted context omitted.

My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected many teams) on what our PriorityClasses should be and to migrate workloads to have priorities. And this is something relatively easy to implement - no developer work required, and practically no YAML to write. Why not? Because sadly, fundamentally, most workplac…

What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.

Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler.

> basically just one for system and majority of serving workload ran on another priority and the rest was for batch

RCA blames in-house load-balancing services (HAProxy) that reached capacity limits. Even if autoscaling is not working correctly because it didn't take Istio into account - why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?

Re: The August 17 outage

#592

Post by sfdfsdfsd »

georgel wrote: post #420
peab wrote: post #414

Earlier quoted context omitted.

This comment comes up over and over again and it's incredibly ignorant. To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before. I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public

On the topic of translation/ language learning, I still find Google Translate better for when I need to type emails in Russian (my older family who only speak it) as far as grammar is concerned than what LLMs spit out.

I'd be surprised if Google Translate isn't powered by a specialized, Transformer-powered model we would now recognize as a "small" language model heavily constrained to a specific task.

Re: The August 17 outage

#593

Post by bblb »

ozgung wrote: post #508
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

Has anyone noticed an increase in the quality, performance or capabilities of the software they use?

I play modded Starfield. There's been a clear increase in mods lately. Some of them are from self-proclaimed non-programmers who are using the LLM's to reverse engineer the game or other abandoned mods, and they've started to create new cool mods or they've fixed various engine limitations. Can confirm that these actually work and I can finally have my 1000+ modlist.

There was a huge exodus of existing programmers/modders ~two years ago, due to paid mods and what not. The gamers took over with their LLM tools.

Re: The August 17 outage

#594

Post by Agentlien »

sph wrote: post #559
brookst wrote: post #462

Earlier quoted context omitted.

I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.

It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in…

I think using AI often feels faster than it is because you put less thought and effort into the problem yourself.

Another possibility is that the people who experience these 100x productivity increases are honest, correct, and simply had abysmal productivity which has now been increased to near-average junior levels thanks to AI.

Re: The August 17 outage

#595

Post by magicalhippo »

sph wrote: post #559
brookst wrote: post #462

Earlier quoted context omitted.

I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.

It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in…

The variance is extreme though. Thanks to Claude Code and Codex I've been able to make several non-trivial internal tools and libraries without writing much in terms of code, just some reviews here and there.

I spent a couple of days on those, and its would have taken me months to write manually I am sure, so in that regards it's close to 50x.

I've also had Claude track down some logic issue in a module I was unfamiliar with which had very large and complicated flows. Would have taken me many days, since I did not have a reproducible case, so had to go by logs and customer description alone. I spent 5 minutes writing a prompt and when I checked back, Claude had identified the issue. The fix I had to implement myself, but was fairly easy. So there Claude definitely was a 100x increase in productivity.

Then there are cases where they're much more modest, or where they might even be negative, when they think they're fixing stuff but actually are introducing more bugs.

Re: The August 17 outage

#597

Post by reactormonk »

bigiain wrote: post #470
amoshi wrote: post #441

Earlier quoted context omitted.

It's AI all the way down. An issue reported by a person account but post made by AI. https://github.com/oven-sh/bun/issues/39800 AI (robobun) responds and creates PR. https://github.com/oven-sh/bun/pull/37459 AI (coderabbit, claude, github actions) review the PR, AI (robobun) applies the fixes. Some AI back and forth. A human finally merges the PR. Not gonna lie, it's kind of beautiful.

Some guy, profiling his linux distro, wonders why ssh connections are a few hundred milliseconds slower than expected, finds there PR introduced and RCE backdoor and p0wns the whole project.

ref: https://en.wikipedia.org/wiki/XZ_Utils_backdoor

Re: The August 17 outage

#598

Post by bob1029 »

I wonder what the ratio of repositories to physical machines is these days. I'd also be curious to see this as change over time.

I have a hard time with the premise that a mere doubling of git ops would be especially crippling for any particular repository. GitHub runs like ass because it's oversubscribed by a huge factor. Not because git is inherently constraining at scale.

Re: The August 17 outage

#599

Post by nosefrog »

vasco wrote: post #560
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle. Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.

I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding.

And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, but it significantly lessens the negative effects of it.

Re: The August 17 outage

#600

Post by jillesvangurp »

madrox wrote: post #327

I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free. I've been saying this for a while: https://news.ycombinator.com/item?id=47534499

That's a business decision for Github of course. They are getting some value out of being the goto place for source code hosting for what is essentially a shockingly high percentage of OSS projects and a really large amount of enterprise projects. It's essentially the largest and most complete network of developers and code in the world. That kind of influence and reach is valuable in itself to Microsoft. And of course it's a gold mine for AI training data as well. Which is presumably why they sponsor it.

But it does raise the question for especially commercial users of Github whether it's time to reconsider the relationship with Github and maybe not put all our eggs in one basket. Basically, this wiped out a whole workday for many companies.

I'm not that eager to start self hosting my stuff. But I am considering it. Besides availability, CI build performance is also becoming a blocker for us. My AI coding jobs creating lots of PRs are making that a bottleneck. Fixing that in Github would require switching to a paid plan. And at that point, self hosting might be the more cost effective option. There are a few tradeoffs here of course. But I like the idea of throwing more memory/cpu at this to get blazingly fast builds.

Post reply on HN