Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: Git operation failures

githubstatus.com

Post reply on HN
Print view View original

321–330 of 335 posts

Re: GitHub: Git operation failures

#321

Post by paulddraper »

PhilippGille wrote: post #305
paulddraper wrote: post #301

Earlier quoted context omitted.

For sure. You can commit, branch, tag, merge, etc and be just fine. Now, if you want to share that work, you have to push.

You can push to any other Git server during a GitHub outage to still share work, trigger a CI job, deploy etc, and later when GitHub is reachable again you push there too. Yes you lose some convenience (like GitHub's pull requests UI can't be used, but you can temporarily use the other Git server's UI for that. I think their point was that you're not fully locked in to GitHub. You have the repo locally and can mirror…

For sure, you don’t have to use GitHub to be that shared server.

It is awfully convenient, web interface, per branch permissions and such.

But you can choose a different server.

Re: GitHub: Git operation failures

#322

Post by array_key_first »

zdragnar wrote: post #277
suddenlybananas wrote: post #243

Earlier quoted context omitted.

To be deliberately provocative, LLMs are being more and more widely used.

Word on the street is github was already a giant mess before the rise of LLMs, and it has not improved with the move to MS.

I don't know anything about githubs codebase, but as a user, their software has many obvious deficiencies. The most glaring being performance. Oh my God, github performs like absolute shit on large repos and big diffs.

Performance issues always scare me. A lot of the time it's indicative of fragile systems. Like with a lot of banking software - the performance is often bad because the software relies on 10 APIs to perform simple tasks.

I doubt this is the case with GitHub, but it still makes you wonder about their code and processes. Especially when it's been a problem for many years, with virtually no improvement.

Re: GitHub: Git operation failures

#323

Post by 1718627440 »

IshKebab wrote: post #201
jimbokun wrote: post #170

Earlier quoted context omitted.

As long as you didn't go all in on GitHub Actions. Like my company has.

Do you think you'd get better uptime with your own solution? I doubt it. It would just be at a different time.

With a build system that can run on any Linux machine, and is only invoked by the CI configuration? Even if all your servers go down, you just run it on any developers machine.

Re: GitHub: Git operation failures

#324

Post by 1718627440 »

CGamesPlay wrote: post #281
jakewins wrote: post #218

Earlier quoted context omitted.

“Your own solution” should be that CI isn’t doing anything you can’t do on developer machines. CI is a convenience that runs your Make or Bazel or Just or whatever you prefer builds, that your production systems work fine without. I’ve seen that work first hand to keep critical stuff deployable through several CI outages, and also has the upside of making it trivial to debug “CI issues”, since it’s trivial to run the…

Yes, this, but it’s a little more nuanced because of secrets. Giving every employee access to the production deploy key isn’t exactly great OpSec.

Every Linux desktop system has a keychain implementation. You can of course always use your own system, if you don't like that. You can use different keys and your developers don't need access to the real key, until all the CI servers are down.

Re: GitHub: Git operation failures

#325

Post by 1718627440 »

paulddraper wrote: post #321
PhilippGille wrote: post #305

Earlier quoted context omitted.

You can push to any other Git server during a GitHub outage to still share work, trigger a CI job, deploy etc, and later when GitHub is reachable again you push there too. Yes you lose some convenience (like GitHub's pull requests UI can't be used, but you can temporarily use the other Git server's UI for that. I think their point was that you're not fully locked in to GitHub. You have the repo locally and can mirror…

For sure, you don’t have to use GitHub to be that shared server. It is awfully convenient, web interface, per branch permissions and such. But you can choose a different server.

If your whole network is down, and you also don't want to connect the hosts with an Ethernet cable, you can even just push to an USB stick.

Re: GitHub: Git operation failures

#326

Post by jakewins »

IshKebab wrote: post #308
jakewins wrote: post #218

Earlier quoted context omitted.

“Your own solution” should be that CI isn’t doing anything you can’t do on developer machines. CI is a convenience that runs your Make or Bazel or Just or whatever you prefer builds, that your production systems work fine without. I’ve seen that work first hand to keep critical stuff deployable through several CI outages, and also has the upside of making it trivial to debug “CI issues”, since it’s trivial to run the…

> should be that CI isn’t doing anything you can’t do on developer machines You should aim for this but there are some things that CI can do that you can't do on your own machine, for example running jobs on multiple operating systems/architectures. You also need to use CI to block PRs from merging until it passes, and for merge queues/trains to prevent races.

Yeah agreed, CI infra provides tons of value.

Ended up expanding this little quip into a blogpost to refer to in the future, feedback welcome! https://tech.davis-hansson.com/p/ci-offgrid/

Re: GitHub: Git operation failures

#327

Post by 1718627440 »

BadBadJellyBean wrote: post #80
cluckindan wrote: post #46

Earlier quoted context omitted.

FYI in an emergency you can SSH to your server and edit files and the DB directly. Where is your god now, proponents of immutable filesystems?!

I love when people do that because they always say "I will push the fix to git later". They never do and when we deploy a version from git things break. Good times. I started packing things into docker containers because of that. Makes it a bit more of a hassle to change things in production.

It's easier, when the deployed instance on the server is also a Git repo, so you can commit the changes there and pull.

Re: GitHub: Git operation failures

#328

Post by raxxorraxor »

HardwareLust wrote: post #241
aeldidi wrote: post #238

I'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actually disrupted my work; this quarter alone I'm already on my fourth, all within the past few weeks. This is, of course, just an anecdote and not evidence of any wider trend (not to mention that I might not have even logged everything last year), but it was enou…

It's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.

And tech hype. Infrastructure to mitigate here isn't expensive. In many cases quite the opposite. The expensive thing is that you made yourself dependent on these services. Sometimes this is inevitable, but to host on GitHub is a choice.

Re: GitHub: Git operation failures

#329

Post by olivia-banks »

luca616 wrote: post #198
olivia-banks wrote: post #153

This is incredibly annoying. I've been trying to fix a deployment action on GitHub for a the past bit, so my entire workflow for today has been push, wait, check... push, wait, check... et cetera.

You should really check out (pun intended) `act` https://github.com/nektos/act

I’ve tried! Most I’ve ever gotten was an inefficient way to fill some disk space and an ‘act’ that didn’t work :-)

Re: GitHub: Git operation failures

#330

Post by Wowfunhappy »

HardwareLust wrote: post #241
aeldidi wrote: post #238

I'm becoming concerned with the rate at which major software systems seem to be failing as of late. For context, last year I only logged four outages that actually disrupted my work; this quarter alone I'm already on my fourth, all within the past few weeks. This is, of course, just an anecdote and not evidence of any wider trend (not to mention that I might not have even logged everything last year), but it was enou…

It's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.

…can I make the case that this might be reasonable? If you’re not running a hospital†, how much is too much to avoid a few hours of downtime around once a year?

† Hopefully there aren’t any hospitals that depends on GitHub being continuously available?

Post reply on HN