Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub is down again

githubstatus.com

Post reply on HN
Print view View original

391–400 of 425 posts

Re: GitHub is down again

#391

Post by zinodaur »

amonith wrote: post #387
nosman wrote: post #309

Earlier quoted context omitted.

What do you mean by public services? All kinds of companies lose millions of dollars of revenue per day if not hour if their sites are not stable.... apple, amazon, google, Shopify, uber, etc etc. Those companies have decided the extra complexity is worth the reliability. Even if you're operating a tech company that doesn't need to have that kind of uptime, your developers probably need those services to be productiv…

By public services I mean only important things like healthcare, law enforcement, fire department. Definitely not stores and food delivery. You can wait an hour or even a couple of hours for that. > Those companies have decided the extra complexity is worth the reliability. Companies always want more money and yes it makes sense economically. I'm not disagreeing with that. I'm just saying that nobody needs this. I gr…

Eh, if I'm paying someone to host my git webui, and they are as shitty about it as github has been recently, I'd rather pay someone else to host it or go back to hosting it myself. It is not absolutely required, but it's a differentiating feature I'm happy to pay for

Re: GitHub is down again

#392

Post by wizzwizz4 »

paulddraper wrote: post #378
wizzwizz4 wrote: post #252

Earlier quoted context omitted.

So… create your shadow system piecewise? There is no reason to have "explode production" in your workflow, unless you are truly starved for resources.

Does this shadow system have usage? Does it handle queries, trigger CI actions, run jobs?

If you test it, yes.

Of course, you need some way of producing test loads similar to those found in production. One way would be to take a snapshot of production, tap incoming requests for a few weeks, log everything, then replay it at "as fast as we can" speed for testing; another way would be to just mirror production live, running the same operations in test as run in production.

Alternatively, you could take the "chaos monkey" approach (https://www.folklore.org/Monkey_Lives.html), do away with all notions of realism, and just fuzz the heck out of your test system. I'd go with that, first, because it's easy, and tends to catch the more obvious bugs.

Re: GitHub is down again

#393

Post by oldge »

showerst wrote: post #7

If you'd have asked me a few years ago if anything could be an existential threat to github's dominance in the tech community I'd have quickly said no. If they don't get their ops house in order, this will go down as an all-time own goal in our industry.

It’s not so much an op’s issue as an architecture and code quality issue. If you have ever dug into the GitHub enterprise self hosted product you get an idea of the mess.

Re: GitHub is down again

#394

Post by meccar »

Does anyone have issue like Workflow cannot be canceled? It keeps displaying "Failed to cancel workflow." when I tried to cancel it and the flow has been stuck in status "In Progress" for the last 5 hours and still counting

Re: GitHub is down again

#395

Post by chickenpotpie »

wizzwizz4 wrote: post #392
paulddraper wrote: post #378

Earlier quoted context omitted.

Does this shadow system have usage? Does it handle queries, trigger CI actions, run jobs?

If you test it, yes. Of course, you need some way of producing test loads similar to those found in production. One way would be to take a snapshot of production, tap incoming requests for a few weeks, log everything, then replay it at "as fast as we can" speed for testing; another way would be to just mirror production live, running the same operations in test as run in production. Alternatively, you could take the…

So just double your cloud bill for several few weeks, costing site like GitHub millions of dollars?

How do you handle duplicate requests to external services? Are you going to run credit cards twice? Send emails twice? If not, how do you know it's working with fidelity?

Re: GitHub is down again

#396

Post by antonvs »

joshstrange wrote: post #321
imglorp wrote: post #316

Earlier quoted context omitted.

I'm pretty sure they don't GAF about GH uptime as long as they can keep training models on it (0.5 /s), but Azure is revenue friction so might be a real problem. Something this week about "oops we need a quality czar": https://news.ycombinator.com/item?id=46903802

> (0.5 /s), Does this mean you are only half-sarcastic/half-joking? Or did I interpret that wrong?

I’m Planck’s constant serious about this

Re: GitHub is down again

#397

Post by [deleted] »

wizzwizz4 wrote: post #392
paulddraper wrote: post #378

Earlier quoted context omitted.

Does this shadow system have usage? Does it handle queries, trigger CI actions, run jobs?

If you test it, yes. Of course, you need some way of producing test loads similar to those found in production. One way would be to take a snapshot of production, tap incoming requests for a few weeks, log everything, then replay it at "as fast as we can" speed for testing; another way would be to just mirror production live, running the same operations in test as run in production. Alternatively, you could take the…

[deleted]

Re: GitHub is down again

#398

Post by paulddraper »

chickenpotpie wrote: post #395
wizzwizz4 wrote: post #392

Earlier quoted context omitted.

If you test it, yes. Of course, you need some way of producing test loads similar to those found in production. One way would be to take a snapshot of production, tap incoming requests for a few weeks, log everything, then replay it at "as fast as we can" speed for testing; another way would be to just mirror production live, running the same operations in test as run in production. Alternatively, you could take the…

So just double your cloud bill for several few weeks, costing site like GitHub millions of dollars? How do you handle duplicate requests to external services? Are you going to run credit cards twice? Send emails twice? If not, how do you know it's working with fidelity?

> several few weeks

*many months

Re: GitHub is down again

#399

Post by nextaccountic »

bartread wrote: post #148
showerst wrote: post #7

If you'd have asked me a few years ago if anything could be an existential threat to github's dominance in the tech community I'd have quickly said no. If they don't get their ops house in order, this will go down as an all-time own goal in our industry.

Yeah, I'm literally looking at GitLab's "Migrate from GitHub" page on their docs site right now. If there's a way to import issues and projects I could be sold.

I wish more people used git-bug. It stores issues and PRs in git (not as branches or tags but as special refs that don't pollute the output when you list branches), and have bridges from/to github and other sites

It's like the fossil model, but on Git

https://github.com/git-bug/git-bug

If everyone used it, it would eliminate this specific form of lock in

Re: GitHub is down again

#400

Post by LittleBox »

petetnt wrote: post #150

GitHub has had customer visible incidents large enough to warrant status page updates almost every day this year ( https://www.githubstatus.com/history ). This should not be normal for any service, even at GitHub's size. There's a joke that your workday usually stops around 4pm, because that's when GitHub Actions goes down every day. I wish someone inside the house cared to comment why the services barely stay up and…

How strange. We have that joke too in our office for Azure pipelines. Do they use the same agents perhaps?
Post reply on HN