Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub is once again down

githubstatus.com

Post reply on HN
Print view View original

161–170 of 214 posts

Re: GitHub is once again down

#161

Post by nomilk »

Why don't companies with chronic outages mimic their stack from top to bottom (i.e. starting with a new domain), then before making a change, make the change on the duplicate stack and blast it with mock requests.

Might catch 90% of problems before they make it into the real stack?

E.g. every step of GitHub's migration to Azure could be mimicked on the duplicate stack before it's implemented on the primary stack. Is this just considered too much work? (I doubt cost would be the issue, because even if it costs millions, it would pay for itself in reduced reputational damage from outages).

EDIT: downvotes - why? - I think this is a good idea (I'd do it for my sites if outages were an issue).

Re: GitHub is once again down

#162

Post by 0xbadcafebee »

altairprime wrote: post #17
voidfunc wrote: post #11

Earlier quoted context omitted.

Azure

> Azure To explain this one-word comment for those unfamiliar, see previously: GitHub will prioritize migrating to Azure over feature development (5 months ago) https://news.ycombinator.com/item?id=45517173 In particular: > GitHub has recently seen more outages, in part because its central data center in Virginia is indeed resource-constrained and running into scaling issues. AI agents are part of the problem here. B…

Age-old lesson: change the tires on the moving vehicle that is your business when it's a Geo Metro, not when it's a freight train.

I'm sure the people with the purse strings didn't care, though, and just wanted to funnel the GH userbase into Azure until the wheels fell off, then write off the BU. Bought for $7.5B, it used to make $250M, but now makes $2B, so they could offload it make a profit. I wonder who'll buy it. Prob Google, Amazon, IBM, Oracle, or a hedge fund. They could choose not to sell it, but it'll end up a writeoff if the userbase jumps ship.

Re: GitHub is once again down

#163

Post by ambicapter »

carlosft wrote: post #129
htrp wrote: post #68

Earlier quoted context omitted.

To be fair a bunch of this is because the CEO after Nat Friedman (Thomas Dohmke) was pushed out in August 25.

And a ton of the top end ruby staff have left. Many of them ended up at shopify. There is a growing about of non ruby/rails code at github, but most of the system that people think of when they think github are ruby/rails.

Shopify is on the AI-everything train as well, we'll see how that goes.

Re: GitHub is once again down

#166

Post by NooneAtAll3 »

0x3f wrote: post #76
msandford wrote: post #48

Earlier quoted context omitted.

I once worked at a place with more micro services than engineers. We joked about "we have as many 8s of uptime as you need!"

> I once worked at a place with more micro services than engineers. Currently consulting somwhere with 30 services per engineer. I cannot convince them this is hell. Maybe that makes it my personal hell.

as a person that never touched webdev, I have a question

how is such service spam different from unix "small functions that do one thing only" culture?

why in unix case it is usually/historically seen as nice, while in web case it makes stuff worse?

Re: GitHub is once again down

#168

Post by davebranton »

proc0 wrote: post #71

It's starting to really look like the AI effect. It might be coincidence but I've noticed a lot more downtime and bad software lately. The last Nvidia drivers gave me a blue screen (last week or so), and speaking about Windows, I froze updates last year because it was clear they were introducing a bunch of issues with every update (not to mention unwanted features). I like AI but actually not for coding because code…

Interesting how many people "Like AI" because it's good at all the jobs other than the one they happen to make a living doing.

Did you hear about the screenwriters school in which the professors said to avoid AI for writing, but it's great for storyboards. And the storyboard school where the professors said the opposite?

The reality is that AI isn't actually "good" at anything. It produces passable ersatz facsimiles of work that can fool those not skilled in the art. The second reality of AI is that everyone is busy cramming it into their products at the expense of what their products are actually useful for.

Once people realise (1), and stop doing (2), the tech industry has a chance of recovering.

Re: GitHub is once again down

#170

Post by 0x3f »

NooneAtAll3 wrote: post #166
0x3f wrote: post #76

Earlier quoted context omitted.

> I once worked at a place with more micro services than engineers. Currently consulting somwhere with 30 services per engineer. I cannot convince them this is hell. Maybe that makes it my personal hell.

as a person that never touched webdev, I have a question how is such service spam different from unix "small functions that do one thing only" culture? why in unix case it is usually/historically seen as nice, while in web case it makes stuff worse?

There are so many failures in microservices that just can't happen with a local binary. Inter-service communication over network is a big one with a failure rate orders of magnitude higher than running a binary on the same machine. Then you have to do deploys, monitoring, etc. across the whole platform.

You will basically need to employ solutions for problems only caused by your microservices arch. E.g. take reading the logs for a single request. In a monolith, just read the logs. For the many-service approach, you need to work out how you're going to correlate that request across them all.

Even the aforementioned network failures require a lot of design, and there's no standardization. Does the calling service retry? Does the callee have a durable queue and pick back up? What happens if a call/message gets 'too old'?

Also, from the other end, command line utils are typically made by entirely different people with entirely different philosophies/paradigms, so the encapsulation makes sense. That's not true when you're the one writing all the services, especially not at small-to-mid-size companies.

Plus, you already can do the single-concern thing in a monolith, just with modules/interfaces/etc.

Post reply on HN