Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

751–760 of 804 posts

Re: The August 17 outage

#751

Post by lionkor »

But GitHub were one of the first and strongest to advocate for using AI, so excessively in fact that it turned people away from the platform. At every turn on the website it suggested to use Copilot, or edit it right now, and commit right now, and do this and that right now(!). The platform effectively begs you to use the free GitHub Actions, too. Not sure what the business model is here.

There are a lot of ways to avoid exponentially more commits, issues, and PRs breaking your backend down, and begging every visitor and user to please use AI to write 40x more code that needs 40x more fixes is not one of them.

Re: The August 17 outage

#752

Post by mrguyorama »

darkwater wrote: post #574
teiferer wrote: post #549

Earlier quoted context omitted.

In a good company that will come back to bite them next performance review. So, if it doesn't then you learned something about your workplace (and it's not good).

> In a good company that will come back to bite them next performance review. In a good company that should be discussed in the next 1:1s so actual change can happen meanwhile. If it just waits for the end of year review, then it's not a good company.

Middle management was all cut so all managers have 30 direct reports and "Don't have time" to do more than a couple 1:1s per year.

Re: The August 17 outage

#753

Post by charleslmunger »

inigyou wrote: post #675
vermilingua wrote: post #276

Earlier quoted context omitted.

More than 50% of the recent commits are from robobun, jesus

Here's the most recent nontrivial one as of this comment: https://github.com/oven-sh/bun/commit/d4de65e9a43224a14591ad... The code change makes no sense and should do nothing. The commit message described a very deep investigation into garbage collection on the C++ side. Some object is being kept alive when the test requires it to be collected, and changing the code in this way allegedly prevents that. But wouldn't y…

The first is, annoyingly, a relatively common problem and solution when dealing with GC lifetimes in tests. Few interpreters/JITs want to generate extra instructions to null out stack slots or pre clobber registers to ensure something becomes collectible at a specific point. Eager nulling of a variable often gets removed by dead store elimination or even just from being a disconnected SSA node. I've written extra nested scopes or wrappers in Java to deal with this in tests.

Don't know anything about the second one.

Re: The August 17 outage

#754

Post by DerArzt »

amazingamazing wrote: post #8

Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.

I wish I had say in our git forge decisions at work, but I don't and I can either tolerate this or quit my job. So I will continue to disparage one of the world's biggest tech companies not being able to manage GitHub properly.

Re: The August 17 outage

#755

Post by yladiz »

fooster wrote: post #721
sph wrote: post #559

Earlier quoted context omitted.

It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in…

You can accept what you like, but it's true. Our team and our business is incredibly more productive. The number of new valuable customer facing features, and the number of PRs (which represent REAL work, not taking a PR and splitting it into 200 PRs game) have all increased dramatically. We've shipped something like ~10x more PRs so far this year than all last year. And we've done that without increasing the number…

Great, number go up. Have the features led to actual customer growth, or just increased productivity?

Re: The August 17 outage

#756

Post by pseudosavant »

asdfsa32 wrote: post #451
pseudosavant wrote: post #439

Earlier quoted context omitted.

It’s funny how the quality of the average commit message has gone up, now that we rarely read or write them directly anymore.

You may be confusing articulate and confident with useful and accurate.

Not really. In my experience, the average human written commit messages were almost always useless 1-3 word "Should work non" kind of messages. The agents had a low bar to clear.

Re: The August 17 outage

#757

Post by frollogaston »

wat10000 wrote: post #427
frollogaston wrote: post #374

Earlier quoted context omitted.

The Github outage was about internal clients. Phone apps are a reasonable place to say things are known to be unreliable and can't be fixed. Your IP address changes when you leave the house. Btw, PWAs added offline capabilities to websites. I hate how the only thing that got used for was these stupid pages that look like you were able to reach the site but it's actually just saying you have no internet, like YouTube.

I’m sure some retries are helpful there too (TCP is doing them, at the very least) but yeah, different approaches for different situations. Maybe you retry but you don’t spend many seconds hoping for it to work.

I'm ok with TCP retries generally. But the assumption at L4 is that L3 can and will drop/reorder packets at random, and retrying is cheap. Also a lottt of tuning has gone into TCP already.

Re: The August 17 outage

#758

Post by abustamam »

afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Isn't this exactly the kind of systems design questions they ask during interviews???

Makes me wonder how many folks at GH and MSFT could even pass their own interviews.

Thanks for sharing though. I learn more about systems design from HN comments than anything else

Re: The August 17 outage

#759

Post by e12e »

solatic wrote: post #634
dilyevsky wrote: post #605

Earlier quoted context omitted.

Yes, just absolutely crazy way of doing load shedding. > why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway? Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?

The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario. > Application backends? I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only wo…

If git push and pull go down - that truly is a worst case scenario. The website being down is just an annoyance by comparison.

Re: The August 17 outage

#760

Post by augunrik »

I don't know, this blog post and the lasts read like organisational and human failures to me. Their system doesn't degrade, they even have infinite retries and no meaningful quotas anywhere I can see (but I'm not a heavy github user).

I feel like this becomes a lesson on how NOT to design and operate a SaaS.

Post reply on HN