Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

351–360 of 804 posts

Re: The August 17 outage

#351

Post by mikert89 »

jeremyjh wrote: post #292
mikert89 wrote: post #287

Earlier quoted context omitted.

ive spent 15 years in big tech companies, this problem is common and its why they pay engineers 500k-1M. this happens at meta literally all the time. half of engineering in big tech is just rewriting a system to scale microsoft is incompetent, they havent changed windows/excel/outlook in 30 years

Tell me about the time that an app at Meta grew at 10% a year for a decade and then began doubling every few months.

its probably happened on half the core teams, idk what you want me to say, it sounds like you dont have alot of experience

Re: The August 17 outage

#353

Post by [deleted] »

ryanisnan wrote: post #196
reticulates wrote: post #110

Earlier quoted context omitted.

I strongly disagree. GitHub, a year ago, acknowledged the fundamental problems and began work on them. We all agree with the diagnosis and strategy: stop building new things, bring stability. Why would whether the CTO codes have any bearing on the correctness of this strategy? GitHub’s problem isn’t that leadership don’t understand the product, or that they don’t know what they should be doing, it’s that they’re batt…

Interesting - There's a couple of things that I disagree with here, but I do think this resonates: "We all agree with the diagnosis and strategy: stop building new things, bring stability." Where I have a problem with the positioning of their GitHub profile is, he's the CTO of GitHub, arguably the defacto standard for open-source version control systems. His GitHub profile is linked to as the author for the post, and…

[deleted]

Re: The August 17 outage

#354

Post by fg137 »

eszed wrote: post #261
danudey wrote: post #248

Earlier quoted context omitted.

AI finding issues in code and reporting them so that an AI can review and triage them for another AI to fix.

I mean, isn't that the dream? I don't know if that's sarcasm or not. I know it doesn't work , but that's the future we've been promised, right?

If the same AIs don't hallucinate and create bugs in the first place.

I don't know how much my own time is wasted on Claude imagining API response formats that never existed.

Re: The August 17 outage

#355

Post by lonertecher »

Is git still the best VCS today? I ask because it seems so much effort in the industry has been invested in making git scale, like Cursor's Origin, or the stories in the past with Facebook's monorepo, but they all seem like bandaids to its intended design.

Re: The August 17 outage

#356

Post by [deleted] »

sgarland wrote: post #285
whatsThisBtn4 wrote: post #130

Earlier quoted context omitted.

[flagged]

Serious question: do you think your 6 year old is learning coding from prompting an AI? I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. This is proven on a nearly daily basis at my job, where, with identical agents and prompts, I see designs being pushed with objectively incorrect facts, sub-optimal code in PRs, and a genera…

[deleted]

Re: The August 17 outage

#357

Post by Breakthrough »

lbrandy wrote: post #283
bbarn wrote: post #212

Earlier quoted context omitted.

Exactly this. I've seen production level trading systems grind to a halt over a simple bug and no matter what tests you have in place, it happens.

I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.

You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?

Re: The August 17 outage

#358

Post by 999900000999 »

cube00 wrote: post #228
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

> bar of corporate code quality It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?

Microsoft is very much not FAANG. They’ve been trying to eliminate dedicated QA for years and underpay contractors.

Microsoft can afford to do a lot of things, but why when you can squeeze a bit more profit out.

Re: The August 17 outage

#360

Post by andrekandre »

jdm2212 wrote: post #268
dannyw wrote: post #247

Earlier quoted context omitted.

Your highly available system is probably somewhat important, otherwise you won’t have invested in making it HA. While your premise holds for happy cases, when you do have a cascading series of outages, not using exponential backoff is just adding a self-inflicted DoS to when you do go down. I don’t really follow your premise and can’t really articulate many cases for when you shouldn’t use exponential backoff. Maybe…

When you have an outage, you should not retry at all. Exponential backoff is exactly how you get cascading outages. If service A fails a request to service B and decides to exponentially back off, now service A is holding open an end user request that will claim resources on service A. Fast forward ten minutes and the service B degradation has metastasized into a service A degradation. And even after service B has re…

  > To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients.
basically enumerate your errors, and depending on the type, retry or just return/forward that same "dont retry this" error?
Post reply on HN