Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

331–340 of 804 posts

Re: The August 17 outage

#331

Post by jeremyjh »

sgarland wrote: post #291
jeremyjh wrote: post #259

Earlier quoted context omitted.

No one said it is unique. But if you take an infrastructure and engineering org that had been growing at 10% a year for a decade, you are going to have a different set of capabilities and practices in place. Adapting to a new reality of doubling every few months will predictably produce failures anywhere. GitHub is not unique in that regard.

They have what amounts to an unlimited budget for AI spend. If it’s so fantastic, why can’t they let it crawl over every piece of their codebase, every metric, every log, and spot these problems before they occur? “We misconfigured a sidecar” is something I would think AI could quite easily find and fix.

I don't know who you are responding to. I haven't made any statements about what AI can do for them. I would expect AI is making the situation worse, like it is in many dysfunctional tech organizations.

Re: The August 17 outage

#332

Post by sergiotapia »

Why not identify the lunatic top 1% of free user you know are just abusing the hell out of the system and put severe rate limits across the board for those organizations/accounts?

Why let your entire platform suffer?

Re: The August 17 outage

#333

Post by mananaysiempre »

8cvor6j844qw_d6 wrote: post #329
tecleandor wrote: post #318

Earlier quoted context omitted.

And 5K PRs. I'm crying.

> 5k PRs Wow, is Bun the record holder for number of PRs? I recall GitHub recommends to keep the number of PR to a certain level due things such as GitHub Actions slowing down.

Nixpkgs has 11k open PRs at the moment (a lot of them routine version bumps and such), so no. But then Nixpkgs isn’t a piece of software in the conventional sense: the monorepo does contain a few of those, but most of it is a giant collection of basically-independent build scripts.

Re: The August 17 outage

#334

Post by microscoper »

sergiotapia wrote: post #332

Why not identify the lunatic top 1% of free user you know are just abusing the hell out of the system and put severe rate limits across the board for those organizations/accounts? Why let your entire platform suffer?

Yeah you could probably have a very high free limit that works for 99% of people

Re: The August 17 outage

#335

Post by [deleted] »

CrimsonCape wrote: post #288
bibimsz wrote: post #74

Earlier quoted context omitted.

thats what i liked about it. its fact and action oriented. what does a "sorry" buy you that the "we let you down" doesn't.

It's funny, I bet you could take any software dev and blind AB test a page written by a corporate manager and a page written by an engineer. Here's something an engineer writes, loaded with facts: "I got to the office and we had a huge panic going on, I immediately called our IT in US-2West and they reported on cascading box failures, I checked our load balancer via remote admin and indeed it was failing to. I called…

[deleted]

Re: The August 17 outage

#337

Post by jambalaya8 »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

I think a lot have mostly experienced working for tech companies and do not understand how different that is from working at some other kind of company (particularly something not technically sexy) doing tech (see last parenthetical).

Re: The August 17 outage

#338

Post by fg137 »

cube00 wrote: post #228
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

> bar of corporate code quality It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?

> It's Microsoft

That means nothing, especially with all the layoffs.

One thing that I learned over the years is never mystify "code quality". Maybe you can say a certain team/project produces high quality code, and maybe Google's overall quality is better than my company's (considering their bar of hiring), but you want to avoid generalizing that, especially at scale.

The average software engineer at a large company ships just enough code at good enough quality to earn a salary, nothing fancy about it.

Re: The August 17 outage

#339

Post by p-e-w »

sgarland wrote: post #285
whatsThisBtn4 wrote: post #130

Earlier quoted context omitted.

[flagged]

Serious question: do you think your 6 year old is learning coding from prompting an AI? I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. This is proven on a nearly daily basis at my job, where, with identical agents and prompts, I see designs being pushed with objectively incorrect facts, sub-optimal code in PRs, and a genera…

> I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong.

That knowledge will be worthless 12-18 months from now when AI does everything better than you, including “understanding”.

If you’re one of the world’s best programmers, it might be 24 months instead, but the writing is on the wall for everyone.

I wonder if people were behaving like that for other revolutionary technologies in the past. “I can still run faster than a car can drive in sharp turns on a gravel road…”

Post reply on HN