Live data from Hacker News

Incident with Actions and Pages

githubstatus.com

371–380 of 407 posts

Re: Incident with Actions and Pages

#371
post #322

Earlier quoted context omitted.

What did you switch to, and what do you like about it?

Running tests locally. It's primitive, but incredibly reliable, and a breeze to debug if (big if) there is any dependency issue.

I have Gitlab with a runner on a notebook I have running as a server. Pretty solid and if you need to bail on Gitlab SaaS you can BYOI and selfhost. Plus the CI is many streets ahead of GitHub in terms of pretty much everything.

Re: Incident with Actions and Pages

#372
post #363

Earlier quoted context omitted.

> But effective monitoring is harder than people assume. Who says public status page equals internal monitoring. They likely know faster than you. Whether they post it publicly is a different issue (hint: SLA penalties, news impacting stock etc)

I never mentioned anything about status pages. Are you sure you’re replying to the right comment?

> I never mentioned anything about status pages.

For context, the parent comment you replied to started with status page.

Then are you talking about internal leaks or just guessing? Otherwise besides what's public how do you know they don't know?

Re: Incident with Actions and Pages

#373
post #329

Earlier quoted context omitted.

People should use something that keeps a local copy of their code and just copies it to Github and to other contributors with a sync process to push and pull changes. Some sort of 'distributed source control system' maybe. Then people would only need a 'hub' to connect to people, and it'd be easier to move somewhere else.

> Some sort of 'distributed source control system' maybe The day it broke away and became centralized was when we had a PR + mandatory "Required actions" to merge to main.

That’s only mandatory on the “hub”. I can do that locally anytime.

Re: Incident with Actions and Pages

#375
post #261

Earlier quoted context omitted.

Deploying SQL changes? Why not just let the application do that on startup. Ofcourse be backward and forward compatible. SQL change only deploy. "Update something in the cloud" <- What do you mean?

> Why not just let the application do that on startup. That only works on extremely simple setups and has risks. If you have only a single server, you can stall it. Now, how to roll back?

We try to keep things simple. Everything has risks. No stall, run async, backward compatible. DB handles rollback via transactions. Happy to expand if interested.

Re: Incident with Actions and Pages

#377
post #19

My action failed with "Unexpected error fetching GitHub release for tag refs/heads/master: HttpError: Sorry. Your account was suspended" Which certainly made me shit myself, briefly.

> Which certainly made me shit myself, briefly.

Can you sue companies for inducing such anxiety?

Re: Incident with Actions and Pages

#378
post #367
post #365

Earlier quoted context omitted.

So you switched to nothing? That’s not the purpose of github actions or remote ci/cd. Anyone can run tests/builds locally.

I think it comes down to risk tolerance. For an established company that wants to avoid upsetting users at all costs, CI/CD makes sense. But for a nimble 'move fast and break things' startup, it can steal dev time for very little upside. Say a disaster happens and someone pushes to main without running tests, 9 times out of 10 it will be of ~zero consequence (either the code works first time, it was a cosmetic change…

That's not the purpose of a remote CI/CD. Your pipeline can be as strict or as loose as you wish. It's there to show you a log of the execution as it happened in a neutral environment (remote server).

Basically, what you are suggesting is that everyone advertises their tests/builds run on slack? Also when two devs merge their changes, who compile/tests the master branch?

Re: Incident with Actions and Pages

#379
post #19

My action failed with "Unexpected error fetching GitHub release for tag refs/heads/master: HttpError: Sorry. Your account was suspended" Which certainly made me shit myself, briefly.

> Which certainly made me shit myself, briefly. Can you sue companies for inducing such anxiety?

IANAL, but I can probably imagine a case being made if a person really got so stressed that for example any health condition got invoked from the stress. It might be up to the lawyer to explain how exactly the service caused the stress and its direct relation to health condition though and up to the judge.

but I suppose that there might be some terms of conditions within using github (ahem Microsoft) that you can probably not sue them for something like this.

It really depends upon the severity of situation (imo)

For example, if a person had any heart condition and they got so stressed because of an error at github (which to be fair, I can understand the stress part, imagine losing some part of your software because it was on github and the amount of direct damage to livelihood if your income depended on it)

and I think that the judge might have to be in just the right technical know-spot as well and someone who can understand the situation from programmer's perspective hopefully.

Then I can see a case being made.

once again not a lawyer but an interesting question, would love reading other replies to your comment.

also for what its worth, you can sue any company for X,Y or Z. The question worth asking is if you can win such lawsuit.

Personally I believe it might be hard but not impossible but for all practical use cases it might as well be but the only answer can probably be found in court. I am just guessing at this point.

Re: Incident with Actions and Pages

#380
post #169

Earlier quoted context omitted.

Bitflips are something that can happen in consumer-grade RAM, so that tracks (and it's comforting that wayward cosmic rays are a substantial reason for an application's crashes!), but on enterprise servers, they will run ECC RAM that is very resistant to bit flips. This is why data hoarders who have NASes with lots of space insist on running their servers with ECC RAM despite it being significantly more expensive. Be…

You've completely missed the point - It's not about bitflips it's about errors that are outside the scope of what's fixable.

I suppose I misunderstood what the "random error" was supposed to mean. I wouldn't call a network error a "random error" because it's caused by things that are internal to the system (entities using a network). A bit flip is caused by an external factor: cosmic radiation. To me, that's what a "random" error is.

If your network goes down because of a DDOS, or part of your system overheating, that's an internal issue you had control over.

If a bit flips because of cosmic radiation, you can't really do anything about that, and it's utterly unpredictable. That's "random" to me.

Post reply on HN