Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub having issues [resolved]

githubstatus.com

Post reply on HN
Print view View original

91–100 of 150 posts

Re: GitHub having issues [resolved]

#91

Post by nadirollo »

shykes wrote: post #47

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun. It can be a pain to setup a break-glass, especially if you have a lot of l…

This is a must when your systems deal with critical workloads. At Fastly, we process a good chunk of the internet's traffic and can't afford to be "down" while waiting for the CI system to recover in the event of a production outage.

We built a CI platform using dagger.io on top of GH Actions, and the "break glass" pattern was not an afterthought; it was a requirement (and one of the main reasons we chose dagger as the underlying foundation of the platform in the first place)

Re: GitHub having issues [resolved]

#92

Post by sidewndr46 »

LollipopYakuza wrote: post #65
cpfohl wrote: post #9

I swear this is my fault. I can go weeks without doing infra work. Github does fine, I don't see any hiccups, status page is all green. But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....

Plot twist: cpfohl works at Github and actually messes with the infra.

Second plot twist: cpfohl actually works at Microsoft on Copilot

Re: GitHub having issues [resolved]

#93

Post by wiether »

littlestymaar wrote: post #26

In many companies I worked for, there were a bunch of infrastructure astronauts who made everything very complicated in the name of zero downtime and sold them to management as “downtime would kill pur credibility and our businesses ”, and then you have billion dollar companies everyone relies on (GitHub, Cloudflare) who have repeated downtime yet it doesn't seem to affect their business in any way.

It's a multitude of factors but basically they can act like that because they are dominant on the market.

The classic "nobody ever gets fired for buying IBM".

If you pick something else, and there's issue, people will complain about your choice being wrong, should have gone with the biggest player.

Even if you provide metrics showing your solution's downtime being 1% of the big player.

Something like Cloudflare is so big and ubiquitous, that, when there's a downtime, even your grandma is aware of it because they talk about it in the news. So nobody will put the blame on the person choosing Cloudflare.

Even if people decides to go back (I had a few customers asking us to migrate to other solutions or to build some kind of failover after the last Cloudflare incidents), it costs so much to find the solutions that can replace it with the same service level and to do the migration, that, in the end, they prefer to eat the cost of the downtimes.

Meanwhile, if you're a regular player in a very competitive market, yes, every downtime will result in lost income, customers leaving... which can hurt quite a lot when you don't have hundreds of thousands of customers.

Re: GitHub having issues [resolved]

#94

Post by aezart »

cpfohl wrote: post #9

I swear this is my fault. I can go weeks without doing infra work. Github does fine, I don't see any hiccups, status page is all green. But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....

Sounds like my Dad, who used to have an uncanny ability to get stuck in elevators. Even got stuck in one with his claustrophobia therapist.

Re: GitHub having issues [resolved]

#95

Post by cperciva »

wolfi1 wrote: post #44
cpfohl wrote: post #9

I swear this is my fault. I can go weeks without doing infra work. Github does fine, I don't see any hiccups, status page is all green. But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....

do you know the Pauli-Effect? https://en.wikipedia.org/wiki/Pauli_effect

Related: In FreeBSD we used to talk often about the Wemm Field. Peter Wemm was one of the early FreeBSD developers and responsible for most of the early project server cluster, and hardware had a phenomenal habit of breaking in his vicinity. One notable story I heard involved transporting servers between data centers and hitting a Christmas tree in the middle of a highway... in March.

Re: GitHub having issues [resolved]

#96

Post by adithyareddy »

duckkg5 wrote: post #6

I would so very much love to see GitHub switch gears from building stuff like Copilot etc and focus on availability

The #1 priority at GitHub for this year is migrating from their own data center to Azure, any other work that gets in the way of this is being deprioritized: https://thenewstack.io/github-will-prioritize-migrating-to-a...

Re: GitHub having issues [resolved]

#97

Post by duggan »

A directory over SSH can be your git server. If your CI isn't too complex, a post-receive hook looping into Docker can be enough. I wrote up about self hosting git and builds a few weeks ago[1].

There are heavier solutions, but even setting something like this up as a backstop might be useful. If your blog is being hammered by ChatGPT traffic, spare a thought for Github. I can only imagine their traffic has ballooned phenomenally.

1: https://duggan.ie/posts/self-hosting-git-and-builds-without-...

Re: GitHub having issues [resolved]

#98

Post by Night_Thastus »

joecool1029 wrote: post #77
Imustaskforhelp wrote: post #68

Earlier quoted context omitted.

At this point I am thinking of creating a 0 days until github outage website similar to how we had the running joke of 0 days until JS framework dropped.

Too slow: https://github-incidents.pages.dev/

That site could use a little more. Maybe a count of how many in the current month and year, tallies for each year, maybe even trends. Could be nice. :)

Re: GitHub having issues [resolved]

#99

Post by esseph »

coffeebeqn wrote: post #51
duckkg5 wrote: post #6

I would so very much love to see GitHub switch gears from building stuff like Copilot etc and focus on availability

This is an absurd state they are at! Weekly outages in 2025 and 2026. From developer beloved and very solid to Microslop went faster than I expected

They may have been Beloved before MS bought them. It takes awhile for technical debt to catch up.

Re: GitHub having issues [resolved]

#100

Post by esseph »

politelemon wrote: post #25
jsheard wrote: post #15

Earlier quoted context omitted.

Top-down mandates to use AI as much as possible, and to rip up their infrastructure and move everything to Azure. https://www.windowscentral.com/microsoft/using-ai-is-no-long... https://thenewstack.io/github-will-prioritize-migrating-to-a...

This is very worrying if their mandate doesn't include quality control.

Wasn't QC fired a decade ago in most companies?
Post reply on HN