Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

411–420 of 1001 posts

Re: Incident with Github.com [resolved]

#411
post #363

Earlier quoted context omitted.

How is github being down losing you guys money?

If you pay developers x money / day and one of their core tools is down for n hours during the day and they spend their money on HN instead that's pretty straight forward to calculate.

I meant more for their customers specifically

Re: Incident with Github.com [resolved]

#412

Earlier quoted context omitted.

How is github being down losing you guys money?

I was in the middle of a hot fix. Our pipeline goes through Github.

That's understandable, but surely there's a way to bypass it? You need to have a breakglass procedures lol

Re: Incident with Github.com [resolved]

#413

I'm self-hosting Gitea and Woodpecker CI for least 6 years, alongside my own docker and pypi registries. Adding x86 and ARM runners, it costs me circa $30/month to run the whole thing, and I am reasonably sure that I could make the whole thing scale at a lower cost per user than what Github charges. I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github.…

I work at a startup with ~20-30 devs. We're talking about a project like this. I'm not sure weekends would work for an actual company, though maybe it depends. I think there would be an appetite for this. From my POV - the main areas of interest are how you can roll this out to a team gracefully without breaking everyone's workflows for multiple days. Good luck

Re: Incident with Github.com [resolved]

#414
post #256

Earlier quoted context omitted.

Boomer here as well, but I'd add that trusting your own org for critical infra usually also ends in tears. Most everything in IT involves failure, including in well designed systems designed by great engineers. I worked for a few years in an exceedingly well capitalised place which ran everything in their own data centers, money no object, with a truck parked somewhere, ready to go, with a smaller version of our crit…

At this point github is barely managing one nine.

True. I'd be very annoyed if I had anything mission critical on github. While we have everything in the cloud, we do actually self-host that part.

Re: Incident with Github.com [resolved]

#415

Can anyone shed some light why GitHub is having so many problems lately?

Github's, COO Kyle Daigle, claimed that the number of commits has increased 14x since last year. So shear scale has something to do with it. I don't know of too much software already operating a planetary scale that could easily absorb a 14x bump in a year.

14x is barely an order of magnitude

Re: Incident with Github.com [resolved]

#416

Earlier quoted context omitted.

How is github being down losing you guys money?

Imagine Github being critical infrastructure for you... The ineptitude...

We use github quite a bit as well. Github being down currently is not actively losing us money, i.e. having customer impact.

Re: Incident with Github.com [resolved]

#417

Earlier quoted context omitted.

Different set of ideologies completely, not an alternative.

An alternative set of ideologies is part of what makes it ... an alternative! I mean, that plus better uptime.

Its about freedom, which codeberg are supposed to be proponents of. Turns out, they're no better than github :)

See: https://blog.codeberg.org/protecting-our-floss-commons-from-...

Re: Incident with Github.com [resolved]

#418

We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?" No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it. What's everyone else using? Self hosted Gitlab? Gitea?

Worth being upfront with, I'm an employee, but; Buildkite.

Frontier labs and global saas heavyweights run CI through us, plus a crop of open source tooling like Bazel, vLLM, Firecracker, Rails, Ray, Sorbet and Rust.

Shopify, Uber, Airbnb, Canva stresstested the platform ten years ago for scaling issues.

You need to take a little time to design upfront and take advantage of dynamically programmable primitives. It isn't the same ease/convenience as Actions. Bit more available though.

Re: Incident with Github.com [resolved]

#419

Earlier quoted context omitted.

For the amount of money enterprises are paying to GitHub, there is a reasonable (and contractual) expectation of uptime. I don't think my boss would find it to fun if I bailed work to hit the gym just because GitHub was down.

Yes, but GitHub enterprise has different uptime ( https://us.githubstatus.com/posts/dashboard ) and actually does have an SLA. Edit: it seems GHE is also affected by the outage and isn't much better.

This, like all of their dashboards, is complete bullshit. As a GitHub enterprise user i am absolutely affected by the outage today, and have been affected by all of the GitHub Cloud outages.

There is no separate enterprise platform unless you buy their self hosted server product.

Post reply on HN