Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub having issues [resolved]

githubstatus.com

Post reply on HN
Print view View original

141–150 of 150 posts

Re: GitHub having issues [resolved]

#141

Post by jamesfinlayson »

OtomotO wrote: post #139
dzonga wrote: post #127

I remember the days when it was mostly Gitlab having issues. Github was super stable - then it got shitty once they switched to React on the frontend instead of the server rendered pages, then Co-pilot stuff lately I haven't heard them bragging about the Rails Monolith

The question I ask myself to this day is why they began switching to React. It made no sense at all for me. Like it was a working product, so why would you switch? I get that new developers might be more familiar with React, but then again, as soon as the trade-offs were apparent, I would've pulled the plug. But they said: Buckle up, everyone, let's ruin our product!

Hm, I didn't realise they'd moved to React. I remember reading years ago that it used jQuery for the longest time but they put in some effort to move to pure Javascript (maybe using web components).

Re: GitHub having issues [resolved]

#142

Post by ehnto »

VorpalWay wrote: post #123
shykes wrote: post #47

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun. It can be a pain to setup a break-glass, especially if you have a lot of l…

At times like this is when I'm so happy I don't work with deploying to a production environment, but rather we release software that (after extensive qualification), customers can install in their environment on their airgapped networks. Using a USB stick to cross the air gap. If we miss a release by a day or thrre, there is enough slack in the process before it goes to the customer that no one will be any the wiser.…

I had that revelation for embedded software. After years of live service hosted software, I released an embedded device. It just runs happily, somewhere, who knows, not me.

Re: GitHub having issues [resolved]

#143

Post by ralph84 »

OtomotO wrote: post #139
dzonga wrote: post #127

I remember the days when it was mostly Gitlab having issues. Github was super stable - then it got shitty once they switched to React on the frontend instead of the server rendered pages, then Co-pilot stuff lately I haven't heard them bragging about the Rails Monolith

The question I ask myself to this day is why they began switching to React. It made no sense at all for me. Like it was a working product, so why would you switch? I get that new developers might be more familiar with React, but then again, as soon as the trade-offs were apparent, I would've pulled the plug. But they said: Buckle up, everyone, let's ruin our product!

Promotion-driven development happens at Microsoft just like any other big tech company.

Re: GitHub having issues [resolved]

#144

Post by SAI_Peregrinus »

hinkley wrote: post #118
alex_suzuki wrote: post #52

Earlier quoted context omitted.

100%. We used to design the pipeline a way that is easily reproducible locally, e.g. doesn’t rely on plugins of the CI runtime. Think build.sh shell script, normally invoked by CI runner but just as easy to run locally.

My automation is always an escalation of a run book that has gotten very precise and handles corner cases. Even if I get the idea of an automation before there’s a run book for it.

I like run scripts. Shell or python scripts that do nothing other than prompt the user with what to do, or which choice to make, and wait for them to hit a key to proceed to the next step. Encode the run book flowchart into an interactive script. Then if a step can be automated, the run book script can directly call that automation. Eventually you may end up with a fully automated script, but even if you don't it can still be a significant help.

Re: GitHub having issues [resolved]

#145

Post by littlestymaar »

Krutonium wrote: post #32
littlestymaar wrote: post #26

In many companies I worked for, there were a bunch of infrastructure astronauts who made everything very complicated in the name of zero downtime and sold them to management as “downtime would kill pur credibility and our businesses ”, and then you have billion dollar companies everyone relies on (GitHub, Cloudflare) who have repeated downtime yet it doesn't seem to affect their business in any way.

To be fair - it SUPER does. Being down frequently makes your competition look better. Of course, once you have the momentum it doesn't matter nearly as much, at least for a while. If it happens too much though, people will start looking for alternatives. The key to remember is Momentum is hard to redirect, but with enough force (reasons), it will.

Few companies (and none of the companies I worked for) are “momentum”-based. The typical company grows because incoming cash flow allows to hire more salespeople and develop new features attracting new kinds of customers.

If people tolerate 10 monthly github failures, they can most likely tolerate one hypothetical hour of downtime from one physical server failure for some random Saas product you're selling to them.

Re: GitHub having issues [resolved]

#146

Post by anon7000 »

2001zhaozhao wrote: post #119
anon7000 wrote: post #115

Earlier quoted context omitted.

The problem is that any kind of automatic code change process like CI, PRs, code review, deployments, etc etc are based on having a central git server. Even security may be based on SSO roles synced to GH allowing access to certain repos. A self-hosted git server is trivial. Making sure everything built on top of that is able to fallback to that is not. Especially when GH has so many integrations out of the box

Forgejo has all of the features you mentioned and is completely open source!

That’s awesome, but now we’re talking about moving a big enterprise install with loads of hooks connected to GH and hundreds of repos. Not an easy project.

Re: GitHub having issues [resolved]

#147

Post by vmaffet »

nadirollo wrote: post #91
shykes wrote: post #47

In moments like this, it's useful to have a "break glass" mode in your CI tooling: a way to run a production CI pipeline from scratch, when your production CI infrastructure is down. Otherwise, if your CI downtime coincides with other production downtime, you might find yourself with a "bricked" platform. I've seen it happen and it is not fun. It can be a pain to setup a break-glass, especially if you have a lot of l…

This is a must when your systems deal with critical workloads. At Fastly, we process a good chunk of the internet's traffic and can't afford to be "down" while waiting for the CI system to recover in the event of a production outage. We built a CI platform using dagger.io on top of GH Actions, and the "break glass" pattern was not an afterthought; it was a requirement (and one of the main reasons we chose dagger as t…

Being able to run the exact same pipeline locally and in any CI environment is the most compelling feature of dagger. It frees you from any underlying platform, so you can adapt more easily.

Re: GitHub having issues [resolved]

#148

Post by AlexeyBelov »

lelanthran wrote: post #125
rezonant wrote: post #50

Earlier quoted context omitted.

Because they're moving it to Azure and doing it far too quickly, not taking care to avoid availability issues

Could be. Or could be that the recent 12 months of 100x increase in code and activity is more than they had planned for when they last did capacity planning. Vibe-coders, many of them here, often boast about the insane amount of KLoC/hour they can generate and merge.

I've seen this take in another GitHub thread, but are there any stats confirming this? As far as I know a lot of Github stats are publicly available, and can be queried via Clickhouse.

Re: GitHub having issues [resolved]

#149

Post by hinkley »

SAI_Peregrinus wrote: post #144
hinkley wrote: post #118

Earlier quoted context omitted.

My automation is always an escalation of a run book that has gotten very precise and handles corner cases. Even if I get the idea of an automation before there’s a run book for it.

I like run scripts. Shell or python scripts that do nothing other than prompt the user with what to do, or which choice to make, and wait for them to hit a key to proceed to the next step. Encode the run book flowchart into an interactive script. Then if a step can be automated, the run book script can directly call that automation. Eventually you may end up with a fully automated script, but even if you don't it can…

Someone gave me that idea about eight years ago and I spent the next several trying to look for a nail for that hammer.

I eventually expanded the one I wrote to include URLs to the right places in Bamboo to do things like disable triggers or start manual deployments. By the time I finished that we were doing 10x as many canary deployments as we had been before, and we’re retiring tech debt way faster because of it. 10/10 would do again.

npm publish will open a web browser for you for passcode entry, and I think I’ll do that next time instead of using cut and paste.

Re: GitHub having issues [resolved]

#150

Post by cpfohl »

RGamma wrote: post #80
cpfohl wrote: post #9

I swear this is my fault. I can go weeks without doing infra work. Github does fine, I don't see any hiccups, status page is all green. But the day comes that I need to tweak a deploy flow, or update our testing infra and about halfway through the task I take the whole thing down. It's gotten to the point where when there's an outage I'm the first person people ask what I'm doing...and it's pretty dang consistent....

Surely this would earn you loads of internet street cred.

I'm not going to lie, I have a certain mystique among my coworkers.
Post reply on HN