Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

471–480 of 804 posts

Re: The August 17 outage

#471

Post by _kidlike »

cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

So, in all of your software, you have introduced randomness in your retries so that the billions of your clients avoid retry synchronization dances?

Re: The August 17 outage

#473

Post by gabriel-uribe »

m4rtink wrote: post #417
blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

More done + more leisure.

Re: The August 17 outage

#474

Post by thisOtterBeGood »

parapsychic wrote: post #452
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

To some extent we are getting more things done as well. In my company (mid-sized startup), they're making us push features every other day now as opposed to maybe 1-2 features per person per sprint. Back when I joined, things were a lot slower. Today, they expect freshers to push new features on day one. But ofc, slop has increased a lot more as well.

That's very fast. Feels like you cannot really review this code so it's all just "working" AI code with few railguards and few human supervision?

Re: The August 17 outage

#475

Post by otterley »

jdm2212 wrote: post #237
wat10000 wrote: post #215

Earlier quoted context omitted.

The whole point of exponential backoff is that the first retry can be quick.

The right answer is for the RPC framework to accurately communicate "try again on another node" vs "don't try again, just hard fail". When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will tim…

This is probably the best and most thorough explanation I’ve seen: https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/...

I’m particularly fond of the token-bucket mechanism for pacing recovery.

Re: The August 17 outage

#476

Post by [deleted] »

llama052 wrote: post #258
jdm2212 wrote: post #237

Earlier quoted context omitted.

The right answer is for the RPC framework to accurately communicate "try again on another node" vs "don't try again, just hard fail". When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will tim…

This is why you have circuit breakers upstream. Not on every individual instance.

[deleted]

Re: The August 17 outage

#477

Post by zombot »

peab wrote: post #414
ethagnawl wrote: post #386

Earlier quoted context omitted.

Bonkers is right. Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better? Software and apps in particular are getting worse, normies hate AI more than ever because they're even less likely to get their desired outcome when calling their doctor or trying to get their online order refunded when chatting with a cutely named chatbot, wages for (most…

This comment comes up over and over again and it's incredibly ignorant. To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before. I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public

Having an app means exactly nothing. How good your language skills will be after N years of learning is the actual measure of whether it's worth anything.

Re: The August 17 outage

#478

Post by otterley »

llama052 wrote: post #258
jdm2212 wrote: post #237

Earlier quoted context omitted.

The right answer is for the RPC framework to accurately communicate "try again on another node" vs "don't try again, just hard fail". When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will tim…

This is why you have circuit breakers upstream. Not on every individual instance.

Close but not quite: https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/...

“Circuit breakers, where calls to a downstream service are stopped entirely when an error threshold is exceeded, are widely promoted to solve this problem. Unfortunately, circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant additional time to recovery. We have found that we can mitigate this risk by limiting retries locally using a token bucket. This allows all calls to retry as long as there are tokens, and then retry at a fixed rate when the tokens are exhausted.”

Re: The August 17 outage

#479

Post by mikae1 »

gabriel-uribe wrote: post #473
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

More done + more leisure.

That's how it's sold, but have you heard about any major tech company that sends their worker home when they've achieved what they used to achieve pre-AI?

EDIT: Instead of simply down-voting, you're welcome to name examples that proves me wrong ;)

Re: The August 17 outage

#480

Post by otterley »

conductr wrote: post #152
OJFord wrote: post #127

Earlier quoted context omitted.

No doubt. ...You can have non-zero revenue and still be loss-making though.

Conversely, what suggests GitHub has a huge operating cost? Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right? It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitH…

> what suggests GitHub has a huge operating cost?

They just added 3 million CPU cores to the fleet. That’s a lot of cheddar.

Post reply on HN