Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Another GitHub outage in the same day

githubstatus.com

Post reply on HN
Print view View original

71–80 of 298 posts

Re: Another GitHub outage in the same day

#71

Post by tibbar »

noodlesUK wrote: post #3

Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable? I moved my company over to GH enterprise last year (from AzDO) and I'm considering moving us away to another vendor altogether as a result of the constant partial outages. Things that used to "just work" now are slow in the UI, and GH actions fail to schedul…

Github used to publish some pretty interesting postmortems. Maybe they still do. IIRC that they were struggling with scaling their SQL db and were starting to hit the limits. It's a tough position to be in because you have to either to a massive migration to a data layer with much different semantics, or you have to keep desperately squeezing performance and skirting on the edge of outages with a DB that wasn't really meant to handle what you're doing with it now. The OpenAI blog post on "scaling" Postgres to their current scale has much the same flavor, although I think they're doing it better than Github appears to be doing.

Re: Another GitHub outage in the same day

#72

Post by zero_shift »

dylan604 wrote: post #39
zero_shift wrote: post #23

Earlier quoted context omitted.

As an aside, God, Azure DevOps, what a total pile of crap that product is My "favourite" restriction that an Azure DevOps PR description is limited to a pathetic 4000 characters.

Amazon's deprecated CodeCommit is limited to 150 chars like it's an old SMS or Tweet.

Ha! Nice. I never worked with CodeStar / CodeCommit. Was it pretty bad?

Re: Another GitHub outage in the same day

#73

Post by amarant »

HoldOnAMinute wrote: post #21
sobjornstad wrote: post #8

Earlier quoted context omitted.

I second this. GitHub used to be a fantastic product. Now it barely even works. Even basic functionality like the timeline updating when I push commits is unreliable. The other day I opened a PR diff (not even a particularly large one) and it took fully 15 seconds after the page visually finished loading -- on a $2,000 dev machine -- before any UI elements became clickable. This happened repeatedly. It is fairly stun…

The trend of "non-functional software" is happening everywhere. See the recent articles about Copilot in Notepad, failing to start because you aren't signed in with your Microsoft Account. We are in a future that nobody wanted.

Not quite everywhere. There's a common denominator for all of those: Microsoft.

Their business is buying good products and turning them into shit, while wringing every cent they can out of the business. Always has been.

They have a grace period of about 2-4 years after acquisition where interference is minimal. Then it ramps up. How long a product can survive once the interference begins largely depends on how good senior leadership at that product company is at resisting the interference. It's a hopeless battle, the best you can do is to lose slowly.

Re: Another GitHub outage in the same day

#74

Post by easton »

noodlesUK wrote: post #61
zero_shift wrote: post #42

Earlier quoted context omitted.

So I work for a devtools vendor (Snyk) and 6 months ago I signed into Azure DevOps for the first time in my life I couldn't believe it. I actually thought the product was broken. Just from a visual perspective it looked like a student project. And then I got to _using_ the damn thing

It's also completely unloved. Even MSFT Azure's own documentation regularly treats it as a second class citizen to GitHub. I have no idea why they don't just deprecate the service and officially feature freeze it. Honestly that's the case with a lot of Azure services though.

It's the boards. GitHub issues doesn't let you do all the arcane nonsense Azure DevOps' boards let you do.

Re: Another GitHub outage in the same day

#75

Post by reactordev »

zero_shift wrote: post #63
reactordev wrote: post #24

Earlier quoted context omitted.

This is the real scenario behind the scenes. They are struggling with scale.

How much has the volume increased, from what you know?

Over 100x is what I’m hearing. Though that could just be panic and they don’t know the real number because they can’t handle the traffic.

Re: Another GitHub outage in the same day

#76

Post by neilv »

vampiregrey wrote: post #2

At this point, GitHub outages feel closer to cloud provider outages than a SaaS blip. Curious how many people here still run self-hosted Git (GitLab / Gitea) vs fully outsourcing version control.

Yay for GitLab and Forgejo/Gitea.

My previous two startups used GitLab successfully. The smaller startup used paid-tier hosted by gitlab.com. The bigger startup (with strategic cutting-edge IP, and multinational security sensitivity) used the expensive on-prem enterprise GitLab.

(The latter startup, I spent some principal engineer political capital to move us to GitLab, after our software team was crippled by the Microsoft Azure-branded thing that non-software people had purchased by default. It helped that GitLab had a testimonial from Nvidia, since we were also in the AI hardware space.)

If you prefer to use fully open source, or have $0 budget, there's also Forgejo (forked from Gitea). I'm using it for my current one-person side-startup, and it's mostly as good as GitLab for Git, issues, boards, and wiki. The "scoped" issue labels, which I use heavily, are standard in Foregejo, but paid-tier in GitLab. I haven't yet exercised the CI features.

Re: Another GitHub outage in the same day

#77

Post by [deleted] »

noodlesUK wrote: post #3

Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable? I moved my company over to GH enterprise last year (from AzDO) and I'm considering moving us away to another vendor altogether as a result of the constant partial outages. Things that used to "just work" now are slow in the UI, and GH actions fail to schedul…

[deleted]

Re: Another GitHub outage in the same day

#78

Post by terminalbraid »

zer00eyz wrote: post #31
zhouzhao wrote: post #15

Earlier quoted context omitted.

Yeah man. Forgejo (albeit it being a weird name from a language that nobody wants to use), is doing very well in my homelab. When I worked at the univerity we used Gitea. Every job outside of univerity I had used Gitlab self hosted. While I don't like the UI or any aspect of Gitlab a lot, it gets the job done.

I use Gitea already... I haven't seen Forejo before today. Im now curious if it is worth the switch.

Forejo was originally forked from Gitea

Re: Another GitHub outage in the same day

#80

Post by samgranieri »

sobjornstad wrote: post #8
noodlesUK wrote: post #3

Can someone in GitHub senior leadership please start paying attention and reprioritise towards actually delivering a product that's at least relatively reliable? I moved my company over to GH enterprise last year (from AzDO) and I'm considering moving us away to another vendor altogether as a result of the constant partial outages. Things that used to "just work" now are slow in the UI, and GH actions fail to schedul…

I second this. GitHub used to be a fantastic product. Now it barely even works. Even basic functionality like the timeline updating when I push commits is unreliable. The other day I opened a PR diff (not even a particularly large one) and it took fully 15 seconds after the page visually finished loading -- on a $2,000 dev machine -- before any UI elements became clickable. This happened repeatedly. It is fairly stun…

I've been a GitHub user since the very early days. I had a beta invite to the service. I really wish they didn't swap out the FE for a React FE.

They need to start rolling back some of their most recent changes.

I mean, if they want people to start moving to self hosted GitLab, this is gonna get that ball rolling.

Post reply on HN