Live data from Hacker News

The August 17 outage

github.blog

61–70 of 804 posts

Re: The August 17 outage

#61
Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.

> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations

How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.

Re: The August 17 outage

#64
post #53

Earlier quoted context omitted.

> We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure. That can't be cheap.

A server box now has around 256 CPU cores. So that's about 12000 servers. If each one is $10k that's $120 million. Not a lot compared to Github's income.

Does that $10k server not need RAM?

Re: The August 17 outage

#65
post #58
post #41

Sorry to suggest this but if they charged everyone say $1/mo. it would absolutely help the massive surge from AI coding they seem to have had. I don't like paying for free stuff but gh certainly worth it.

Unfortunately time and time again, the overwhelming majority of people show that they would rather deal with an unreliable scummy company for "free" than pay $1/mo for a reliable service which treats them with respect :(

Surely that's the point. Shed users.

Re: The August 17 outage

#66
post #45
post #38

Earlier quoted context omitted.

This absolutely can happen in large systems. If some part of the system is at capacity, then slightly increasing the load can cause it to fall behind and start accumulating a backlog. These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.

I believe their point is that "system is at capacity" is something they ought to start fixing before the capacity is exceeded

Sure. But you might not even be realizing that something is just at the cusp if the load is spiky enough.

The art of large system design is to identify and avoid these kinds of chokepoints. And when something happens, propagate the "backpressure" up the stack to avoid queuing.

AWS got a fair share of similar outages, so the newer SDKs now try to not exacerbate these kinds of issues: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr...

The original AWS EBS outage is probably the canonical example: https://aws.amazon.com/message/65648/

Re: The August 17 outage

#68
Related recently:

GitHub has alternatives, but no replacement

https://news.ycombinator.com/item?id=49135365

Why developers are ditching GitHub for Codeberg and self-hosting alternatives

https://news.ycombinator.com/item?id=48842611

and new entry:

Cursor Origin Code Hosting

https://news.ycombinator.com/item?id=49334209

Post reply on HN