Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub is once again down

githubstatus.com

Post reply on HN
Print view View original

171–180 of 214 posts

Re: GitHub is once again down

#171

Post by proc0 »

davebranton wrote: post #168
proc0 wrote: post #71

It's starting to really look like the AI effect. It might be coincidence but I've noticed a lot more downtime and bad software lately. The last Nvidia drivers gave me a blue screen (last week or so), and speaking about Windows, I froze updates last year because it was clear they were introducing a bunch of issues with every update (not to mention unwanted features). I like AI but actually not for coding because code…

Interesting how many people "Like AI" because it's good at all the jobs other than the one they happen to make a living doing. Did you hear about the screenwriters school in which the professors said to avoid AI for writing, but it's great for storyboards. And the storyboard school where the professors said the opposite? The reality is that AI isn't actually "good" at anything. It produces passable ersatz facsimiles…

Yeah, I think I heard about that. Within certain domains it is certainly a useful tool. I would say things like online search are much nicer now (in that asking an AI is equivalent to searching online but it summarizes it for you). Online search fits the strengths of LLMs nicely, but right now it's being sold as a silver bullet, which it's not.

Re: GitHub is once again down

#173

Post by worik »

nomilk wrote: post #161

Why don't companies with chronic outages mimic their stack from top to bottom (i.e. starting with a new domain), then before making a change, make the change on the duplicate stack and blast it with mock requests. Might catch 90% of problems before they make it into the real stack? E.g. every step of GitHub's migration to Azure could be mimicked on the duplicate stack before it's implemented on the primary stack. Is…

Testing? Who needs it when you have Copilot!

Re: GitHub is once again down

#174

Post by 0x3f »

KaiserPro wrote: post #112
0x3f wrote: post #76

Earlier quoted context omitted.

> I once worked at a place with more micro services than engineers. Currently consulting somwhere with 30 services per engineer. I cannot convince them this is hell. Maybe that makes it my personal hell.

"Its like family here!" In that every night you're playing murder mystery, and its never fun.

I would never trust my family with system design either.

Re: GitHub is once again down

#175

Post by drewbug01 »

nomilk wrote: post #161

Why don't companies with chronic outages mimic their stack from top to bottom (i.e. starting with a new domain), then before making a change, make the change on the duplicate stack and blast it with mock requests. Might catch 90% of problems before they make it into the real stack? E.g. every step of GitHub's migration to Azure could be mimicked on the duplicate stack before it's implemented on the primary stack. Is…

> EDIT: downvotes - why? - I think this is a good idea (I'd do it for my sites if outages were an issue).

Because that's a monumental amount of work, and extraordinarily difficult to retrofit into a system that wasn't initially designed that way. Not to mention the unstated requirement of mirroring traffic to actually exercise that system (given the tendency of bugs to not show up until something actually uses the system).

Re: GitHub is once again down

#176

Post by bandrami »

merlindru wrote: post #77
paxys wrote: post #42

Took a full 8 years for a Microsoft acquisition to go to shit, which is probably a record. Kudos to the Github team for holding out this long.

GitHub has always been incredibly outage riddled no? This is not a MSFT thing

I think having a single 9 of uptime is a relatively new thing even for GH

Re: GitHub is once again down

#177

Post by nomilk »

drewbug01 wrote: post #175
nomilk wrote: post #161

Why don't companies with chronic outages mimic their stack from top to bottom (i.e. starting with a new domain), then before making a change, make the change on the duplicate stack and blast it with mock requests. Might catch 90% of problems before they make it into the real stack? E.g. every step of GitHub's migration to Azure could be mimicked on the duplicate stack before it's implemented on the primary stack. Is…

> EDIT: downvotes - why? - I think this is a good idea (I'd do it for my sites if outages were an issue). Because that's a monumental amount of work, and extraordinarily difficult to retrofit into a system that wasn't initially designed that way. Not to mention the unstated requirement of mirroring traffic to actually exercise that system (given the tendency of bugs to not show up until something actually uses the sy…

> that's a monumental amount of work

Agree, but look at the alternative; GitHub is constantly being savaged by users who (quite reasonably) expect uptime. Ignoring impacts on morale and reputation, damage to their bottom line alone might tens (hundreds?) of millions per year.

> mirroring traffic

yeah, I agree that's difficult, but it need to not be exact to still be useful.

Re: GitHub is once again down

#178

Post by justinko »

guywithabike wrote: post #49

The worst part of all this is that GitHub's CTO and VP of Engineering sent out the usual "here's what we'll do to fix things" letter to their larger customers and, without exaggeration, it boiled down to: 1) "Here's a bunch of stuff we already did!" which... clearly isn't working, and 2) "We're continuing our Azure migration." also clearly not working. So needless to say, if you depend on GitHub for critical business…

There’s plenty of alternatives, but people continue to stay. Therefore, it’s not as bad as you think it is.

Re: GitHub is once again down

#179

Post by joshribakoff »

AndroTux wrote: post #86
paxys wrote: post #42

Took a full 8 years for a Microsoft acquisition to go to shit, which is probably a record. Kudos to the Github team for holding out this long.

I’m still baffled that Minecraft is doing so well, despite the whole Bedrock thing. At this point I think Microsoft just forgot that they bought Mojang.

They don’t enforce or even default to 2fa to change the account email. In addition, they have no process to get a human to reverse account takeovers. Just a web form that tells you to call a number that redirects you back to a web form

Re: GitHub is once again down

#180

Post by sathish316 »

How much of this is due to Microsoft Culture of not innovating and buying leading companies with their revenue from windows/office and slowly destroying the aspects of companies that made them great in the first place?

Is all the recent GitHub downtime entirely attributable to GitHub AI Copilot related development? How hard can it be to reduce the blast radius of new AI features to not affect the core parts of hosting repositories? Because of Copilot everywhere, The UX has become bad and I had to click all over the place and on my profile to find repositories.

Post reply on HN