Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

181–190 of 407 posts

Re: Incident with Actions and Pages

#181
post #6

Apparently they deleted the Github Actions account as it shows up as ghost in PR comments.

if that's the case, such a thing is so absurd that it goes around and it becomes almost hilarious

I would love to know how much of their internal workflows are being handled by AI workflows. Because this seems like the kind of thing your agent might do.

Re: Incident with Actions and Pages

#182
post #164

Earlier quoted context omitted.

Yep, they just need to improve their reliability by 2%! https://mrshu.github.io/github-statuses/

This page tells a very different story from GitHub own status page. What is different here?

Github measures/reports the SLA of the individual services.

The external page linked above goes the other extreme and considers it a bad status whenever any individual service is degraded.

In reality the majority of people only use 3 or 4 of the core services the majority of the time but since there's no "core services" SLA/uptime the usability of github for the majority of people is slightly obfuscated.

Re: Incident with Actions and Pages

#183
post #169

Earlier quoted context omitted.

Im curious about this: because in my experience (working on smaller services though), a small number of errors is always there, as a "baseline". Recently there was this: https://news.ycombinator.com/item?id=47252971 "10% of Firefox crashes are caused by bitflips" Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on…

Bitflips are something that can happen in consumer-grade RAM, so that tracks (and it's comforting that wayward cosmic rays are a substantial reason for an application's crashes!), but on enterprise servers, they will run ECC RAM that is very resistant to bit flips. This is why data hoarders who have NASes with lots of space insist on running their servers with ECC RAM despite it being significantly more expensive. Be…

Bitflips specifically may not be; things like network issues, noisy neighbors, row/rack/host maintenance (leading to a downed and migrated host) absolutely are things that happen at high frequency at scale and cause your background level of errors to be more than 0.

Re: Incident with Actions and Pages

#184
post #169

Earlier quoted context omitted.

Im curious about this: because in my experience (working on smaller services though), a small number of errors is always there, as a "baseline". Recently there was this: https://news.ycombinator.com/item?id=47252971 "10% of Firefox crashes are caused by bitflips" Which makes me think a small amount of random issues which happen even though nothing is broken, is normal everywhere. Especially once move things around on…

Bitflips are something that can happen in consumer-grade RAM, so that tracks (and it's comforting that wayward cosmic rays are a substantial reason for an application's crashes!), but on enterprise servers, they will run ECC RAM that is very resistant to bit flips. This is why data hoarders who have NASes with lots of space insist on running their servers with ECC RAM despite it being significantly more expensive. Be…

You've completely missed the point - It's not about bitflips it's about errors that are outside the scope of what's fixable.

Re: Incident with Actions and Pages

#185
post #33

Earlier quoted context omitted.

Same here. You’d think they could at least separate out the GitHub-hosted and self-hosted runners, so you’re still able to dispatch jobs if the self-hosted runners are down.

If the job queue is down, that wouldn't help, would it? On my repo the jobs do not get scheduled on the PRs at all, so I assume that separation wouldn't help for todays issue.

They have the github enterprise domain separated out and its working fine right now https://us.githubstatus.com/posts/dashboard

Re: Incident with Actions and Pages

#186

Earlier quoted context omitted.

Yes, Thais can be be really frustrating when you’re trying to get work done. There needs to be more competition and better alternatives and the LLMs need to offer easier connection to these alternatives.

What do the Thai people have to do with this? :(

[deleted]

Re: Incident with Actions and Pages

#187
post #16

Why do they go down so often? Is it true that the reason is that they've incorporated too much AI without human review?

It's (a) they're under massively increased load because everyone's vibing up new projects these days, (b) they've been in a weird frankenstein "on azure but also we have our own control plane" state for years and they're pushing to no longer have that be the case. I don't think vibecoding at Github has much to do with it.

I started using an agent (Codex) on my repo and it went from a a few dozen clones to thousands (3383 this week). I dunno what the agents are doing to clone the repo so many times -- I'm not running 3000 agents or prompts, maybe 10 or so this week. But if this is typical, a 1000x increase in usage across the board can't be good on the system.

Re: Incident with Actions and Pages

#190
If you want an alternative to GitHub Actions, you could self-host Forgejo Actions, but I'm not that happy with the design.

I much prefer Woodpecker CI, which is an open source fork of Drone.io. It supports multiple Git backends like GitHub, Gitea, Forgejo, Gitlab, Bitbucket. It supports running jobs locally, on Docker, and on Kubernetes. And there's autoscalers built in for AWS, Hetzner, Linode, Vultr, and Scaleway. There's a bunch of 3rd party plugins (https://woodpecker-ci.org/plugins) for custom integrations. The UX is also very simple, with OAuth used not only for authentication/authorization but also setting up & accessing repos. The system architecture is great, with separate components that run stateless connected to a database, and a custom plugin is any program that takes environment variables and does stdio. The config file is a good balance of ugly YAML and convenience syntax like shell-style parameter expansion variables.

It probably takes less than 15 minutes to install, set up, and run WoodpeckerCI for a small team, so it's not a big investment to try out or host. With the autoscaling plugins it lets you scale your workload up to whatever size. Honestly you could run it on a laptop since it's written Go.

(to clarify for beginners: the config file docs are found in a section called "workflow syntax" (https://woodpecker-ci.org/docs/usage/workflow-syntax) and variable parameter expansion is buried deep in an environment variables page called "string operations" (https://woodpecker-ci.org/docs/usage/environment#string-oper...). poorly organized docs aside, the system itself works well)

Post reply on HN