Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub: Git operation failures

githubstatus.com

Post reply on HN
Print view View original

311–320 of 335 posts

Re: GitHub: Git operation failures

#311

Post by csomar »

stinkbeetle wrote: post #262
HardwareLust wrote: post #241

Earlier quoted context omitted.

It's money, of course. No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray.

> It's money, of course. 100% > No one wants to pay for resilience/redundancy. I've launched over a dozen projects going back to 2008, clients simply refuse to pay for it, and you can't force them. They'd rather pinch their pennies, roll the dice and pray. Well, fly by night outfits will do that. Bigger operations like GitHub will try to do the math on what an outage costs vs what better reliability costs, and optimi…

> Look at a big bank or a big corporation's accounting systems

Not my experience. Any banking I used, in multiple countries, had multiple and significant outages and some of them where their cards have failed to function. Do a search of "U.S. Bank outage" to see how many outages have happened so far this year.

Re: GitHub: Git operation failures

#312

Post by Akronymus »

rvnx wrote: post #233
collingreen wrote: post #229

Earlier quoted context omitted.

You've ruined something for me. My adult side is grateful but the rest of me is throwing a tantrum right now. I hope you're happy with what you've done.

I read a book about elevators accidents; don't.

With people properly using them or not?

I am fairly certain that the vast majority comes from improper use (bypassing security measures, like riding on top of the cabin) or something going wrong during maintenance.

Re: GitHub: Git operation failures

#314

Post by attendant3446 »

_jab wrote: post #44

GitHub is pretty easily the most unreliable service I've used in the past five years. Is GitLab better in this regard? At this point my trust in GitHub is essentially zero - they don't deserve my money any longer.

We use gitlab.com at work, and it's noticeably bad. Parts of it go down monthly, or sometimes even weekly.

Re: GitHub: Git operation failures

#315

Post by never_inline »

SteveNuts wrote: post #67

I have a serious question, not trying to start a flame war. A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. B. If it's becoming more common, what are the reasons? I can think of a few, but I don't kn…

I think they're becoming more common because AI -> FOMO -> tighter deadlines on projects "since you can use AI to accelerate your work", which is often not how it works, and last 10% of reliability work is forgotten.

Re: GitHub: Git operation failures

#316

Post by never_inline »

pm90 wrote: post #110
SteveNuts wrote: post #67

I have a serious question, not trying to start a flame war. A. Are these major issues with cloud/SaaS tools becoming more common, or is it just that they get a lot more coverage now? It seems like we see major issues across AWS, GCP, Azure, Github, etc. at least monthly now and I don't remember that being the case in the past. B. If it's becoming more common, what are the reasons? I can think of a few, but I don't kn…

Github isn't in the same reliability class as the hyperscalars or cloudflare; its comically bad now, to the point that at a previous job we invested in building a readonly cache layer specifically to prevent github outages from bringing our system down.

I think most systems should not on https://github.com at run time (rather than build time - build failures should not bring your system down)?

Re: GitHub: Git operation failures

#317

Post by goku12 »

__MatrixMan__ wrote: post #250
fishpen0 wrote: post #247

Earlier quoted context omitted.

Not really comparable at any compliance or security oriented business. You can't just zip the thing up and sftp it over to the server. All the zany supply chain security stuff needs to happen in CI and not be done by a human or we fail our dozens of audits

Why is it that we trust those zany processes more than each other again? Seems like a good place to inject vulnerabilities to me...

This isn't really a trust issue. People tend to take shortcuts and commit serious mistakes in the process. Humans are incredibly creative (no, LLMs are nowhere close). But for that, we need the freedom to make mistakes without serious consequences. Automation exists to take away the fatigue of trying to not commit mistakes.

Re: GitHub: Git operation failures

#318

Post by goku12 »

fishpen0 wrote: post #247
__MatrixMan__ wrote: post #209

Earlier quoted context omitted.

This escalator is temporarily stairs, sorry for the convenience.

Not really comparable at any compliance or security oriented business. You can't just zip the thing up and sftp it over to the server. All the zany supply chain security stuff needs to happen in CI and not be done by a human or we fail our dozens of audits

While true, the mistake we made was to centralize them. Just imagine the case if git was a centralized software with millions of users connecting over a single domain? I don't care how much easier it would be, or how flashy it would be, I prefer much to struggle with the current incarnation rather than deal with headaches like these. Sadly, the progress towards decentralized alternatives for discussions, issue tracking, patch sharing and CI is rather slow (though they all do exist) due to the fact that the no big investor invests in them.

Re: GitHub: Git operation failures

#319

Post by WhyNotHugo »

_jab wrote: post #44

GitHub is pretty easily the most unreliable service I've used in the past five years. Is GitLab better in this regard? At this point my trust in GitHub is essentially zero - they don't deserve my money any longer.

GitLab is slow and its pages are heavy. I often have to work on slow/unreliable internet and GitLab pages simply can’t finish loading. Most of them need to load megabytes of JavaScript before they even fetch the data to be shown. The data being usually under 1kB of text.

Re: GitHub: Git operation failures

#320

Post by __MatrixMan__ »

goku12 wrote: post #317
__MatrixMan__ wrote: post #250

Earlier quoted context omitted.

Why is it that we trust those zany processes more than each other again? Seems like a good place to inject vulnerabilities to me...

This isn't really a trust issue. People tend to take shortcuts and commit serious mistakes in the process. Humans are incredibly creative (no, LLMs are nowhere close). But for that, we need the freedom to make mistakes without serious consequences. Automation exists to take away the fatigue of trying to not commit mistakes.

I'm not against automation at all. But if all of the devs build it and get one hash and CI runs it through some gauntlet involving a bunch of third party software that I don't have any reason to trust and out pops an artifact with a different hash, then the CI has interfered with the chain of trust between myself and my user.

Maybe I've just been unlucky, but so far my experience with CI pipelines that have extra steps in them for compliance reasons is that they are full of actual security problems (like curl | bash, or like how you can poison a CircleCI cache using a branch nobody reviewed and pick up the poisoned dependency on a branch which was reviewed but didn't contain the poison).

Plus, it's a high value target with an elevated threat model. Far more likely to be attacked than each separate dev machine. Plus, a motivated user might build the software themselves out of paranoia, but they're unlikely to securely self host all the infra necessary to also run it through CI.

If we want it to be secure, the automation you're talking about needs to runnable as part of a local build with tightly controlled inputs and deterministic output, otherwise it breaks the chain of trust between user and developer by being a hop in the middle which is more about a pinky promise and less about something you can verify.

Post reply on HN