Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

301–310 of 407 posts

Re: Incident with Actions and Pages

#301

If you don't want to self-host Gitea/Forgejo, I recommend SourceHut for private repos and Codeberg for public ones. Happy to answer any questions you might have for either based on my experience!

For private repos I just have a folder on my NAS that I run `mkdir && git -C init --bare` in. Works great.

Re: Incident with Actions and Pages

#302
post #82
post #55

Earlier quoted context omitted.

Same. It's weird how I always find out that GitHub is down before GitHub does. Took 15 minutes before it appeared on githubstatus.com

> It's weird how I always find out that GitHub is down before GitHub does No, it's not. Official updates = potential SLA penalties. Always requires approval.

This is the most plausible reply.

Re: Incident with Actions and Pages

#304

Earlier quoted context omitted.

Shouldn't Github be large enough to not have anyone on-call, but just rotate the responsible team around the world?

At least when I worked at a Bigcorp a lot of that was being cut to save costs.

I've worked in large orgs where we could (at at some times did) have around the world rotations. They don't work well. It've very hard to maintain real team cohesion, and you end up with really superficial operations. People tend not to dig in really deep, find good fixes, etc. Lots of superficial bandages.

Re: Incident with Actions and Pages

#305
post #255

Earlier quoted context omitted.

It's an eye opener. Think about it - today, it was a mistake. But, what if it really happened? What if you really lost access to all your years of hard work? It's a wake up call. A blessing in disguise to store what matters to you the most locally, backed up offline. Never trust any single provider. Be it MS or Google or Apple. RAID is the way.

People should use something that keeps a local copy of their code and just copies it to Github and to other contributors with a sync process to push and pull changes. Some sort of 'distributed source control system' maybe. Then people would only need a 'hub' to connect to people, and it'd be easier to move somewhere else.

What you just described is Fossil. It has an auto-sync feature that makes everything feel distributed.

Just set up a Kubernetes deployment and you’re set.

But as others mention, GitHub’s primary strength is collaboration. If you want decentralized, solve this by creating a decentralized collaboration tool on top of fossil and/or git.

For example, how to do pull requests and code reviews?

Re: Incident with Actions and Pages

#307

This is your periodic reminder that Github is growing at ~14x (1400%!) annually. This would be incredible growth for a young, unprofitable, VC-funded startup, even Uber never achieved more than ~3x AFAIK. For a widely-established company that was already very well known and a market leader in its niche for many years? Absolutely unprecedented. This is a conservative estimate assuming linear growth, the actual number…

Growing what exactly. Seems like a spam issue or some sort of automatic circular commit chain between alot of projects.

Re: Incident with Actions and Pages

#308

In my mind there's no doubt Github datacenters can't handle the recent load that came after agentic AI. They just need to get new servers. It's simple as that.

“Just” “simple as that”

Reminds me of the occasional “JavaScript developer tries to vibe debug a Linux kernel issue” comments we get here.

Re: Incident with Actions and Pages

#309

Is it about funds? Why Github is not catching up with the traffic? I know there's a mass rush on Github recently specially due to Claude Code leading users to use Github. sometimes even persuasive.

Because scaling complex systems is not trivial

Sometimes it is. There are some incredibly brute force yet simple and elegant pattern that power some of the biggest scale system you could think of.

It is relatively easy to scale a collection of simple things to extreme and exhibit complex behavior together. It is a lot harder to scale something complex to extreme. But too many times the latter is the default - designed wrong from the ground up and stuck in scaling hell.

Re: Incident with Actions and Pages

#310
post #296

Earlier quoted context omitted.

> Even a synthetic probe needs a few failures to trigger an alert. It doesn't "need" that. That just how most people set it up because it’s an easy sane default that allows for network jitter without inexperienced engineers thinking about different conditions triggering different types of responses. If you’re measuring internal APIs from an observablity solution that’s has nodes already inside you’re network enclave,…

Ah, yes, I misunderstood. And I have seen cases where a direct CPU alert makes sense, but 99 times out of 100 times I see it, it's nothing but trouble. Worse, I tend to see the cpu alert when there are no end to end synthetic alerts, 500 alerts, queue depth alerts, etc. If your requests are fast and cheap, you can probe frequently relative to your goals, but often that's not really possible (think, long SQL queries,…

Yeah very true.

Talking about long SQL queries, I quite like throwing CPU alerts on database servers. They'll be a low priority alert (ie no out of hours "pagers") so just something that goes into a slack channel. But they're a good indicator of when developers have poorly optimized SQL, or the DB schema is poorly defined (eg missing indexes), or the DB server itself is poorly sized.

This wouldn't be something you'd expect to need in production and definitely not something you'd rely on as a notice of a production outage. But it is an example of one of those 1% occasions where a CPU alert does add value to the overall observability of the application.

But this also ties into your excellent point about how you'd use CPU and other data points to build a picture of what's happening in your application.

Post reply on HN