Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

371–380 of 804 posts

Re: The August 17 outage

#371

Post by wasting_time »

p-e-w wrote: post #339
sgarland wrote: post #285

Earlier quoted context omitted.

Serious question: do you think your 6 year old is learning coding from prompting an AI? I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. This is proven on a nearly daily basis at my job, where, with identical agents and prompts, I see designs being pushed with objectively incorrect facts, sub-optimal code in PRs, and a genera…

> I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. That knowledge will be worthless 12-18 months from now when AI does everything better than you, including “understanding”. If you’re one of the world’s best programmers, it might be 24 months instead, but the writing is on the wall for everyone. I wonder if people were behavi…

It will be interesting to see if LLM can push beyond the intelligence embedded in language.

General intelligence may still be a ways off.

Re: The August 17 outage

#372

Post by stingraycharles »

sqquima wrote: post #243
stingraycharles wrote: post #218

Earlier quoted context omitted.

Retries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering. Source: decades of operational pain.

This is one of the scenarios where I feel we as engineers should have been more diligent in publicly writing down what techniques have worked in which scenarios and what haven't, and the AI could have gathered decades of operational knowledge and provide proper guidance to architects designing new systems. It's also true that we're witnessing unprecedented levels of scale.

Thundering herds and circuit breakers are extremely well documented and practiced in production. AI absolutely knows about this.

But knowing when to use which strategy and when a simple retry suffices is precisely the type of thing humans will remain to be better at than AI for the foreseeable future.

Re: The August 17 outage

#373

Post by normie3000 »

jambalaya8 wrote: post #337
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

I think a lot have mostly experienced working for tech companies and do not understand how different that is from working at some other kind of company (particularly something not technically sexy) doing tech (see last parenthetical).

Is GitHub a tech company? Because that's who they're criticising.

Re: The August 17 outage

#374

Post by frollogaston »

wat10000 wrote: post #262
GeorgeDewar wrote: post #199

Earlier quoted context omitted.

I totally agree with you, I think retries are overused, with the exception of operations that are known to be unreliable and can't be improved. In my experience, errors which go away within a few seconds are quite rare, and are mainly due to flaws which are usually caught in testing. I think a very careful cost/risk/benefit analysis should be done when adding automatic retries to things. As well as potentially causin…

I have the exact opposite view. Way too often, I’ll be presented with an error to the effect of, “something went wrong, please try again” and often the retry works. And I’m left wondering why this machine whose sole purpose is to automate things can’t do that for me automatically. In particular, networks tend to be a LOT less reliable than the typical developer accounts for. And the failures are very often transient.…

The Github outage was about internal clients. Phone apps are a reasonable place to say things are known to be unreliable and can't be fixed. Your IP address changes when you leave the house.

Btw, PWAs added offline capabilities to websites. I hate how the only thing that got used for was these stupid pages that look like you were able to reach the site but it's actually just saying you have no internet, like YouTube.

Re: The August 17 outage

#375

Post by fishfasell »

pseudosavant wrote: post #219
blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

For some context, in June they said commits "commits nearly doubled year over year, crossing 1.4 billion per month". Now, it has more than doubled that in just a few months. https://github.blog/news-insights/product-news/github-copilo...

Makes sense given the ubiquity of agentic coding. I made a joke to my coworker today that all we do is make sure AI agents can communicate with other AI agents.

Re: The August 17 outage

#376

Post by frollogaston »

frollogaston wrote: post #195
Quarrelsome wrote: post #186

Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic. I feel like I'm mildly stupid…

I don't like blind retries. It's different if the server or LB knows it's overloaded and asks clients to retry in X seconds.

Oh and this is already assuming the blind retries are randomized exponential backoff. Thought it went without saying but maybe not.

Re: The August 17 outage

#377

Post by tclancy »

cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour

Not to join the parade, but what would a unit test that confirms a cycling behavior across all the instances in-flight even look like? I mean, besides "Not a unit test".

Re: The August 17 outage

#378

Post by itemize123 »

nycpig wrote: post #9

Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post. "If you were trying to ship software that day, we let you down" is classic corporate non-apology speak. I’m done.

im sorry u feel this way

Re: The August 17 outage

#379

Post by greatgib »

> Copilot services took longer. Errors in those services triggered a client-side retry loop that increased traffic during recovery.

Let's pretend that the scale traffic is with the number of commit/pr and not self-inflicted with all the copilot eye candy features that were vibe-coded-added to GitHub.

In addition they say that they will continue their migration to azure and that azure is supporting their actions run. But GitHub actions is one of the things that was the most constantly broken without multiple outages recently. So I have the feeling that it proves the point that part of the stability issues is also due to their forced usage of azure.

Re: The August 17 outage

#380

Post by tverbeure »

cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

They added 3 million CPUs. You reduce the complexity of their systems to a unit test…
Post reply on HN