Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

131–140 of 407 posts

Re: Incident with Actions and Pages

#131
post #16

Why do they go down so often? Is it true that the reason is that they've incorporated too much AI without human review?

It could be many things. Microsoft mismanaging stuff. Azure. Vibe-coded Github. So much AI slop being committed it adds an extra burden on the servers, etc.

Re: Incident with Actions and Pages

#133

Whilst you're waiting for it to come back, try out AGENT-CI (which is a project I built.), which runs GitHub Actions on your machine: https://agent-ci.dev . (Open source, etc.) No, it's not like "act," because it uses the standard Github runner, the difference is that the control plane is an emulation of api.github.com, because of this we can do all kinds of nice things: Caching in ~0 ms. Pause on failure, so you can…

"Its not like act, because we can add AI" Is what it boils down to. > codex "Fix this pipeline, use `act` to verify your changes"

I did not say that, what I said was: It's not like `act` because it's not a rewrite of the runner. It's the standard runner... So the one that actually runs GitHub Actions.

I have tried to use act many times, and many times I've failed.

P.S. pause on failure is also helpful for humans, but I'm trying to be realistic about where the future of programming is going...

Re: Incident with Actions and Pages

#134
post #83

Earlier quoted context omitted.

In a high performance service with good maintenance and upkeep, you page for all 500s. A noisy pager forces the team to fix the 500s. Maybe the Github Actions infrastructure isn't run like that. edit: my oncall rotation notified on all 500s, 24/7, not just rates - https://news.ycombinator.com/item?id=48279262

Do you know of a single service at a single company that actually does that? I know all of Gmail, every GCE service I can think of, every AWS service I can think of, Amazon.com, Netflix, and Github all do not page on just a single 500. I know none of those are particularly "high performance" though. Curious where your experience is coming from.

I worked at a large fintech moving billions of dollars in volume a day.

I had a fairly long tenure, where I maintained multiple key services in critical online payments flow. Authentication, authorization, core business and risk data, as well as some cross-cutting control plane stuff, etc. You needed one or more of our services to take a payment, serve any request from the employee dashboard - pretty much everything hit our services. The entire company ground to a halt without my team.

We paged for every single 500. In instances where a particular class of 500 was spurious or not worth fixing, we would leave it acked or mark it as noise. But typically we'd just put in a fix as soon as possible so we didn't page.

Our graceful shutdown and traffic shaping stack was great, but occasionally we'd get a few pages during deploys or failovers.

Oncall was typically not bad, but when it did get bad it was terrible. I've been involved in huge outages that cost hundreds of millions of dollars. Usually it was the fault of multiple teams having compounding runaway failures rather than one service or bug in particular.

It's inexcusable to have a customer's payments not go through. We engineered around resilience. We had strict five nines SLAs and p99 targets and evaluated our adherence with even the smallest partial outage. Hundreds of other services depended on ours, and downstream impacts were huge, so we had to keep a tight ship.

We didn't have "business hours"-only paging either as our platform was available globally, including a heavy install base in Asia.

Re: Incident with Actions and Pages

#136

Earlier quoted context omitted.

Founder of Depot here. To my knowledge, we are the first engine to support different syntaxes in this compatible way via Depot CI [0]. Great time to try it out and let us know your thoughts! We’ve built a lot of cool stuff into it like parallel steps, custom images, and a full CLI/API interface so you can literally everything without going into the web app. [0] https://depot.dev

Are you able to bring your own runners? Our org is heavily invested in self-hosted runners at this point and have gotten a pretty tremendous value from it. I think we'd be wise to get away from GitHub's control plane but keep running jobs in our own infra.

Yes, we support this via Depot Managed for all of our products including the latest one: Depot CI [0].

[0] https://depot.dev/products/ci

Re: Incident with Actions and Pages

#137
post #11

Wasn’t my fault this time! I haven’t started work yet. https://news.ycombinator.com/item?id=47237377

Sorry guys it might be me.

I vibe coded a script that interacts with both Gitlab and Github via their APIs and I've been using it pretty heavily since this morning. I crossed the streams! Goodness, I didn't know it would be _this_ bad!

Re: Incident with Actions and Pages

#139
post #6

Apparently they deleted the Github Actions account as it shows up as ghost in PR comments.

if that's the case, such a thing is so absurd that it goes around and it becomes almost hilarious

The team got greenlight to more tokens and the problem should be fixed soon. Fingers crossed. /s

source: voices in my head. Not affiliated with MSFT.. anymore.

Re: Incident with Actions and Pages

#140
post #68

Too many times we've been bitten by this - it has been an issue too many times to count. This is why we don't use Github Actions, kids. Seriously, its a proprietary build service that puts the keys to the kingdom in someone elses' control. Just: No! Print this status page to PDF so you've got it handy next time someone castigates you for not using Github Actions, folks.

So, what do you use?
Post reply on HN