Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

281–290 of 804 posts

Re: The August 17 outage

#281

Post by [deleted] »

b112 wrote: post #133
Macha wrote: post #125

Earlier quoted context omitted.

Well either they can handle this load that Microsoft can't, or they can't. If Microsoft are going to continue to be unreliable in the absence of the rate limit then: If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway. If alternatives aren't able to manage, then user's aren't going to jump since those services won't actua…

I think the number of commits is a red herring, but that said, I wonder how spiky their load is. Imagine a fee over X commits, but only during certain hours. I can imagine 90% of the commits over 6 or 8 timezones, maybe 50% over 4 right now...

[deleted]

Re: The August 17 outage

#282

Post by semiquaver »

kjellsbells wrote: post #43
ivraatiems wrote: post #3

"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products." Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak. Github is ripe for disruption and I hope it is disrupted soon.

Ok, but there's no universe where a major Microsoft-owned property is not being forced to run on Azure. Just like AWS pushing to get off Oracle back in the day. It would be career-destroying to suggest otherwise regardless of technical merit (and tbf, no infrastructure is bulletproof, unless you want to port GitHub to z/OS on mainframe)

LinkedIn gave up after four years of trying: https://www.cnbc.com/amp/2023/12/14/linkedin-shelved-plan-to...

Azure just has very poor performance and reliability characteristics. It’s a particularly bad migration target for a colo-based company that mainly runs on owned hardware (such as GitHub or LinkedIn). Requires much larger architecture changes than (say) a company coming from AWS.

Re: The August 17 outage

#283

Post by lbrandy »

bbarn wrote: post #212
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

Exactly this. I've seen production level trading systems grind to a halt over a simple bug and no matter what tests you have in place, it happens.

I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.

Re: The August 17 outage

#284

Post by [deleted] »

dapperdrake wrote: post #188
jdm2212 wrote: post #169

Earlier quoted context omitted.

A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded). If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go gre…

CAP theorem. Pick one of those.

[deleted]

Re: The August 17 outage

#285

Post by sgarland »

whatsThisBtn4 wrote: post #130
VCFundedGenYer wrote: post #124

Earlier quoted context omitted.

Not really. It's AI commits. Not quality commits.

[flagged]

Serious question: do you think your 6 year old is learning coding from prompting an AI?

I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. This is proven on a nearly daily basis at my job, where, with identical agents and prompts, I see designs being pushed with objectively incorrect facts, sub-optimal code in PRs, and a general explosion of slop. That tells me that it still very much matters if you know how to do the job without the help of robots.

Re: The August 17 outage

#286

Post by Groxx »

blitzar wrote: post #267
dcrazy wrote: post #242

Earlier quoted context omitted.

A push containing 100 commits is more expensive than a push containing 1 commit.

But is it 100x more expensive?

Probably? If you do 100 pushes instead, there is roughly zero additional data. At best you'd be comparing cache costs, which probably are lper for one large push, but there's a ton of calculation and CI that runs per commit regardless of other data being cached.

Re: The August 17 outage

#287

Post by mikert89 »

jeremyjh wrote: post #259
mikert89 wrote: post #241

Earlier quoted context omitted.

im not buying that github is a unique engineering problem, harder than the rest of hyperscalars.

No one said it is unique. But if you take an infrastructure and engineering org that had been growing at 10% a year for a decade, you are going to have a different set of capabilities and practices in place. Adapting to a new reality of doubling every few months will predictably produce failures anywhere. GitHub is not unique in that regard.

ive spent 15 years in big tech companies, this problem is common and its why they pay engineers 500k-1M. this happens at meta literally all the time.

half of engineering in big tech is just rewriting a system to scale

microsoft is incompetent, they havent changed windows/excel/outlook in 30 years

Re: The August 17 outage

#288

Post by CrimsonCape »

bibimsz wrote: post #74
nycpig wrote: post #9

Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post. "If you were trying to ship software that day, we let you down" is classic corporate non-apology speak. I’m done.

thats what i liked about it. its fact and action oriented. what does a "sorry" buy you that the "we let you down" doesn't.

It's funny, I bet you could take any software dev and blind AB test a page written by a corporate manager and a page written by an engineer.

Here's something an engineer writes, loaded with facts:

"I got to the office and we had a huge panic going on, I immediately called our IT in US-2West and they reported on cascading box failures, I checked our load balancer via remote admin and indeed it was failing to. I called my IT managers and learned we had hard resetting in progress for the past 20 minutes with minimal impact on recovery."

Totally missing from the article.

Re: The August 17 outage

#289

Post by noir_lord »

dannyw wrote: post #232
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

It’s been proven that VS Code has product managers who vibe code commits that get merged, including the co-authored with copilot bug. So there’s some problems there.

That was the final straw that stopped me using it.

I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.

Re: The August 17 outage

#290

Post by fragmede »

bpavuk wrote: post #140
DANmode wrote: post #96

Earlier quoted context omitted.

What if it was $2?

that is more manageable but c'mon I can't even keep Google One 100GB up on a consistent basis, that's how poor I am. self-hosting would be a far better option because apparently I find enough people to provide free Hetzner VPSes and stuff as long as I can sell this as mutually beneficial. for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets yo…

Oracle has a free tier that you can self host a git server on, if that's what you're looking for.
Post reply on HN