Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

651–660 of 804 posts

Re: The August 17 outage

#651

Post by prennert »

Why does Github not segregate the free offerings from the enterprise or even better, all paid offerings?

It is unacceptable that enterprise plans get impacted by traffic on free and public repos. Our repos are neither on the free plan nor are they open. We have not had more AI stuff happening in the last weeks. Our traffic is stable. I would wager that most enterprises did not spike the traffic all of the sudden. Even if they were, we are paying for our quotas. Still our Github actions were breaking and our PRs not viewable at some times.

I am hoping this instability is going to cause a Cambrian explosion of forges and if that is happening, Github will be the first victim of the AI revolution.

I am working on a truly decentralized / local first code review right now, and a big part of my motivation for this is how bad Github has become. I dont know if I have enough time to build CI as well, but I am hoping others do. Otherwise I will just fall back onto Jenkins.

Re: The August 17 outage

#652

Post by toasty228 »

gabriel-uribe wrote: post #473
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

More done + more leisure.

Really? https://newsroom.haas.berkeley.edu/ai-promised-to-free-up-wo...

Re: The August 17 outage

#653

Post by dastbe »

solatic wrote: post #634
dilyevsky wrote: post #605

Earlier quoted context omitted.

Yes, just absolutely crazy way of doing load shedding. > why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway? Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?

The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario. > Application backends? I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only wo…

are these separate services? Are you going to integrate how services preempt into overall system design?

My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.

Re: The August 17 outage

#654

Post by jascha_eng »

mikert89 wrote: post #351
jeremyjh wrote: post #292

Earlier quoted context omitted.

Tell me about the time that an app at Meta grew at 10% a year for a decade and then began doubling every few months.

its probably happened on half the core teams, idk what you want me to say, it sounds like you dont have alot of experience

Instagram, Facebook and even threads all had much more mundane growth rates and definitely no unexpected jumps like GitHub is experiencing. I'm sure if suddenly the solar system had 10 more earths with each about 10 billion people and they would all start using Instagram tomorrow we would have exactly the same growing pains and outages that GitHub has today.

Luckily for Meta agents are not yet as much into doomscrolling as humans are.

Re: The August 17 outage

#655

Post by klm127 »

neya wrote: post #602
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...

The article doesn't mention any Azure outages, but a client had an Azure hosted container restart that day which caused problems due to configs being stored out of persistant storage.

Re: The August 17 outage

#656

Post by noir_lord »

mey wrote: post #308
noir_lord wrote: post #289

Earlier quoted context omitted.

That was the final straw that stopped me using it. I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.

What do you use now and what is your primary use case?

neovim, general text editing/small scripts/markdown where the task is short/light enough I just want to do it and keep moving.

Intellij for basically all software development that is more than 2 minutes.

Re: The August 17 outage

#657

Post by brovonov »

lietuvis wrote: post #612
menaerus wrote: post #603

Earlier quoted context omitted.

Hey, traditional hand coder fellow ... the times have past and the future is already a present. I never have been this productive before, and it's been ~20 years that I spent coding. System level programming. Few years back I would have said the bottleneck is not in spelling out the code so we wouldn't see that much AI impact but boy I was wrong. Writing code has never been this cheap, both in terms of time resources…

> I never have been this productive before So, what are you doing/ have done with all your productivity?

writing code by hand of course /s

Re: The August 17 outage

#658

Post by noir_lord »

bodge5000 wrote: post #649
hirvi74 wrote: post #382

Earlier quoted context omitted.

I always advocate for the "boomer" editors when possible. It doesn't matter which one you choose. People can say what they want about them, but very little software survives almost 60 years. I sure as Hell hope VS Code doesn't.

For me, boomer editors (mostly nvim to be honest) join a long list of other things like tiling window managers as something I genuinely want to learn, but that I can never justify learning over either learning something else or doing "real work". I've been slowly introducing both and making gradual progress, but its extremely slow

Gradual progress is still progress.

Just pick a specific task that you do routinely (say markdown editing) and just use neovim for that one task - eat the elephant one bite at a time and markdown is generally small enough/light enough that you won't get out into the weeds trying to make neovim the perfect IDE in the process.

Re: The August 17 outage

#659

Post by f6v »

winrid wrote: post #467
shdtabasum wrote: post #154

Earlier quoted context omitted.

LOL, this kind of things will happen when projects like Bun ( https://github.com/oven-sh/bun ) are running on auto. :)

This will result in pretty low quality software. I've had opus 5 along with its AI code reviewer agree to do some pretty stupid shit.

Wait till you see what some people agree to.

Re: The August 17 outage

#660

Post by gilrain »

peab wrote: post #414
ethagnawl wrote: post #386

Earlier quoted context omitted.

Bonkers is right. Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better? Software and apps in particular are getting worse, normies hate AI more than ever because they're even less likely to get their desired outcome when calling their doctor or trying to get their online order refunded when chatting with a cutely named chatbot, wages for (most…

This comment comes up over and over again and it's incredibly ignorant. To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before. I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public

> I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public

On the other hand, famine is coming. Come back and read this next year. Was your dalliance worth these food shortages?

Post reply on HN