Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

301–310 of 804 posts

Re: The August 17 outage

#301

Post by gopher_space »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

If you've only worked at a small-to-medium privately held company you'd assume every company ran that smoothly. "Too big to fail" makes sense once you've worked for a behemoth and not before.

Re: The August 17 outage

#302

Post by bluerooibos »

cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour

That wouldn't be a unit test - that's more like an end-to-end or integration test.

Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.

Re: The August 17 outage

#303

Post by tymscar »

eszed wrote: post #261
danudey wrote: post #248

Earlier quoted context omitted.

AI finding issues in code and reporting them so that an AI can review and triage them for another AI to fix.

I mean, isn't that the dream? I don't know if that's sarcasm or not. I know it doesn't work , but that's the future we've been promised, right?

Who promised it?

Re: The August 17 outage

#304

Post by ninkendo »

leptons wrote: post #182
Brian_K_White wrote: post #168

Earlier quoted context omitted.

Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push. You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.

>You can rig up a local ide to pathologically commit+push per save The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder…

> requires a commit and push for every file save

I don’t think I could imagine a stupider idea than this if I tried. To paraphrase Babbage: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a solution.

Re: The August 17 outage

#305

Post by busterarm »

TomBombadildoze wrote: post #250
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

[flagged]

I solve real problems at scale and the engineering practices I see on a daily basis are a clown show.

There's little to no basic understanding of networking, distributed systems, databases, etc. 99% of our engineers were hired from their college internships and never worked anywhere else. Industry hires to improve systems rarely last more than a year and it is almost never their fault.

We're in the next tier down from the biggest tech companies and what we do is hardly uncommon among our peers.

I should be shocked that 99% of engineers I deal with treat all resources as infinite bandwidth, 100% uptime, but I'm not. They NIH super hard and write tons of code for things that a docker container running nginx (or similar) would solve in 5 minutes. There's almost no useful testing and worse documentation.

Welcome to corporate life.

Re: The August 17 outage

#306

Post by 8cvor6j844qw_d6 »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage."

Cloudflare outage on December 5, 2025 [1]:

> However, we have never before applied a killswitch to a rule with an action of “execute”.

[1]: https://blog.cloudflare.com/5-december-2025-outage/

Re: The August 17 outage

#308

Post by mey »

noir_lord wrote: post #289
dannyw wrote: post #232

Earlier quoted context omitted.

It’s been proven that VS Code has product managers who vibe code commits that get merged, including the co-authored with copilot bug. So there’s some problems there.

That was the final straw that stopped me using it. I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.

What do you use now and what is your primary use case?

Re: The August 17 outage

#309

Post by cactusplant7374 »

jjordan wrote: post #207

I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-gh No side projects? Nothing? Just seems odd.

Probably managing and mentoring. I has a boss that wanted to code and be CTO. Just horrible.

Re: The August 17 outage

#310

Post by consumer451 »

b112 wrote: post #133
Macha wrote: post #125

Earlier quoted context omitted.

Well either they can handle this load that Microsoft can't, or they can't. If Microsoft are going to continue to be unreliable in the absence of the rate limit then: If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway. If alternatives aren't able to manage, then user's aren't going to jump since those services won't actua…

I think the number of commits is a red herring, but that said, I wonder how spiky their load is. Imagine a fee over X commits, but only during certain hours. I can imagine 90% of the commits over 6 or 8 timezones, maybe 50% over 4 right now...

I was thinking the same thing at first: ideas to increase product limits on GitHub, to increase reliability given limited infra.

However, there are sharks in the water, and with the diminishing mean of user technical knowledge, the product actually needs to become even more free. GitHub likely needs even lower friction.

"All it takes" is the insanely heavy technical lift to support that. There is no other solution. All the C-Suite needs to do is foster an environment with well-thought through, and possibly over-funded engineering, at the edge of the art. That sounds like an amazing challenge.

Post reply on HN