Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: Git operation failures

githubstatus.com

Post reply on HN
Print view View original

211–220 of 335 posts

Re: GitHub: Git operation failures

#211

Post by noir_lord »

BadBadJellyBean wrote: post #80
cluckindan wrote: post #46

Earlier quoted context omitted.

FYI in an emergency you can SSH to your server and edit files and the DB directly. Where is your god now, proponents of immutable filesystems?!

I love when people do that because they always say "I will push the fix to git later". They never do and when we deploy a version from git things break. Good times. I started packing things into docker containers because of that. Makes it a bit more of a hassle to change things in production.

Depends on the org, the big ones I've worked for regular Devs even seniors don't have anything like the level of access to be able to pull a stunt like that.

At the largest place I did have prod creds for everything because sometimes they are necessary and I had the seniority (sometimes you do need them in a "oh crap" scenario).

They where all setup on a second account in my work Mac which had a danger will Robinson wallpaper because I know myself, far far too easy to mentally fat finger when you have two sets of creds.

Re: GitHub: Git operation failures

#212

Post by tapoxi »

_jab wrote: post #44

GitHub is pretty easily the most unreliable service I've used in the past five years. Is GitLab better in this regard? At this point my trust in GitHub is essentially zero - they don't deserve my money any longer.

Another GitLab self-hosting user here, we've run it on Kubernetes for 6 years. It's never gone down for us, maybe an hour of downtime yearly as we upgrade Postgres to a new version.

Re: GitHub: Git operation failures

#213

Post by hungariantoast »

letrix wrote: post #182
lorenzleutgeb wrote: post #165

Earlier quoted context omitted.

Please check out radicle.dev, helping hands always welcome!

> Repositories are replicated across peers in a decentralized manner You lost me there

"Replicated across peers in a decentralized manner" could just as easily be written about regular Git. Radicle just seems to add a peer-to-peer protocol on top that makes it less annoying to distribute a repository.

So I don't get why the project has "lost you", but I also suspect you're the kind of person any project could readily afford to lose as a user.

Re: GitHub: Git operation failures

#214

Post by nightski »

IshKebab wrote: post #201
jimbokun wrote: post #170

Earlier quoted context omitted.

As long as you didn't go all in on GitHub Actions. Like my company has.

Do you think you'd get better uptime with your own solution? I doubt it. It would just be at a different time.

I mean yes. We've hosted internal apps that have four nines reliability for over a decade without much trouble. It depends on your scale of course, but for a small team it's pretty easy. I'd argue it is easier than it has ever been because now you have open source software that is containerized and trivial to spin up/maintain.

The downtime we do have each year is typically also on our terms, not in the middle of a work day or at a critical moment.

Re: GitHub: Git operation failures

#215

Post by 0x457 »

junon wrote: post #81
SteveNuts wrote: post #67

I have a serious question, not trying to start a flame war. A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. B. If it's becoming more common, what are the reasons? I can think of a few, but I don't kn…

Been on GitHub for a long time. It feels like they're more often. It used to be yearly if at all that GitHub was noticably impacted. Now it's monthly, and recently, seemingly weekly.

Definitely not how I remember. First, I remember seeing unicorn page multiple times a day some weeks. There were also time when webhook delivery didn't work, so circle ci users couldn't kick off any builds.

What change is how many services GitHub can be having issues.

Re: GitHub: Git operation failures

#216

Post by grayhatter »

SteveNuts wrote: post #67

I have a serious question, not trying to start a flame war. A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. B. If it's becoming more common, what are the reasons? I can think of a few, but I don't kn…

End of year, pre-holiday break, code/project completion for perf review rush.

Be good to your Stability reliability engineers for the next few months... it's downtime season!

Re: GitHub: Git operation failures

#217

Post by wavemode »

IshKebab wrote: post #201
jimbokun wrote: post #170

Earlier quoted context omitted.

As long as you didn't go all in on GitHub Actions. Like my company has.

Do you think you'd get better uptime with your own solution? I doubt it. It would just be at a different time.

Uptime is much, much easier at low scale than at high scale.

The reason for buying centralized cloud solutions is not uptime, it's to safe the headache of developing and maintaining the thing.

Re: GitHub: Git operation failures

#218

Post by jakewins »

IshKebab wrote: post #201
jimbokun wrote: post #170

Earlier quoted context omitted.

As long as you didn't go all in on GitHub Actions. Like my company has.

Do you think you'd get better uptime with your own solution? I doubt it. It would just be at a different time.

“Your own solution” should be that CI isn’t doing anything you can’t do on developer machines. CI is a convenience that runs your Make or Bazel or Just or whatever you prefer builds, that your production systems work fine without.

I’ve seen that work first hand to keep critical stuff deployable through several CI outages, and also has the upside of making it trivial to debug “CI issues”, since it’s trivial to run the same target locally

Re: GitHub: Git operation failures

#219

Post by tingletech »

SteveNuts wrote: post #67

I have a serious question, not trying to start a flame war. A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. B. If it's becoming more common, what are the reasons? I can think of a few, but I don't kn…

Years ago on hackernews I saw a link about probability describing a statistical technique that one could use to answer a question about if a specific type of event was becoming more common or not. Maybe related to the birthday paradox? The gist that I remember is that sometimes a rare event will seem to be happening more often, when in reality there is some cognitive bias that makes it non-intuitive to make that decision without running the numbers. I think it was a blog post that went through a few different examples, and maybe only one of them was actually happening more often.

Re: GitHub: Git operation failures

#220

Post by jakub_g »

_jab wrote: post #44

GitHub is pretty easily the most unreliable service I've used in the past five years. Is GitLab better in this regard? At this point my trust in GitHub is essentially zero - they don't deserve my money any longer.

My company self-hosts GitLab. Gitaly (the git server) is a weekly source of incidents, it doesn't scale well (CPU/memory spikes which end up taking down the web interface and API). However we have pretty big monorepos with hundreds of daily committers, probably not very representative.
Post reply on HN