Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

221–230 of 804 posts

Re: The August 17 outage

#221

Post by stingraycharles »

Quarrelsome wrote: post #214
taylor-s wrote: post #211

Earlier quoted context omitted.

Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.

isn't that a bomb with a pair of scissors to cut the fuse that could break down under certain conditions? I feel like they could also hide an issue that might get fixed if there were no retries. Is it slow or is our resource sporadically offline?

Google “thundering herd” and you’ll understand why uncontrolled retries can be / are bad.

Re: The August 17 outage

#222

Post by subarctic »

jjordan wrote: post #207

I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-gh No side projects? Nothing? Just seems odd.

Maybe he's too busy with leading the engineering side of the company to write code these days?

Re: The August 17 outage

#223

Post by Quekid5 »

pixl97 wrote: post #44
k33P1Tr3aL wrote: post #29

Earlier quoted context omitted.

the 'ol thundering herd problem...

Exponential backoff is your friend... too few people use it.

Exponential is also overkill (even with jitter as others have mentioned).

I seem to remember there was a "you failed 5 PIN entries in a row, please wait 500000 seconds before you retry" on Apple phones. So, you probably also want a sensible max... which makes exponential a bit pointless. Just do a basic fixed delay + (large, e.g. 0.5 x the delay) jitter and you'll be fine for most things. You can add a bit of cumulative delay if it's really costly to do retries.

Re: The August 17 outage

#224

Post by jdm2212 »

grim_io wrote: post #205
jdm2212 wrote: post #200

Earlier quoted context omitted.

Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.

Why? You can retry, but there is nothing wrong with increasingly waiting slightly longer if we fail many times.

Try asking Opus or Fable that question. It'll give you a good answer on why microservice architectures work the way they do in order to keep user-facing latency acceptable and minimize downtime. It's a complicated enough topic that I don't feel like explaining it for free to you in a HN comment.

Re: The August 17 outage

#226

Post by felooboolooomba »

jjordan wrote: post #207

I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-gh No side projects? Nothing? Just seems odd.

Can you imagine the flack he'd take if it turns out he'd been moonlighting whilst the GitHub ship is sailing though cat 5 with both the mast and the ship whore on fire?

Re: The August 17 outage

#227

Post by googletron »

jjordan wrote: post #207

I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-gh No side projects? Nothing? Just seems odd.

yeah grinding on side projects, while everything is in flames.

this is fine.

Re: The August 17 outage

#228

Post by cube00 »

ACCount37 wrote: post #208
cube00 wrote: post #132

> Errors in those services triggered a client-side retry loop that increased traffic during recovery Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours. > Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token…

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

> bar of corporate code quality

It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?

Re: The August 17 outage

#229

Post by sandeepkd »

muglug wrote: post #178
sandeepkd wrote: post #172

Earlier quoted context omitted.

Not sure to be honest, from a machine perspective 2X should never be a big deal, unless 1.4 was the threshold or sweet state and no one thought too much about scale and architecture beyond that

If you’re the size of GitHub and you’ve been running your infra for years with very little variation in traffic patterns you have a strong incentive to optimise costs for that existing behaviour.

I am afraid thats now how infrastructure works from what I have seen. The number that really matters is QPS. For any system the QPS varies through out the day and across the week and months. Most design considerations easily absorb any 2X increase. Pick up any company and the chances are that the servers are over provisioned, no one takes chances specially with critical components.

What you have going on with Github is mix of multiple things. Traffic alone is not the cause from what little I know, it does adds to the problem for sure

1. Infrastructure is being moved to use Azure, and overall all the cloud providers are struggling with hardware at the moment (same is going on for linkedin too)

2. The core teams, the people who knew the existing systems have either been laid off or moved from Github

3. Microsoft veterans are brought in to fill the gap across the board, they are trying their best but its a lot of unknown for them

Re: The August 17 outage

#230

Post by dcrazy »

verdverm wrote: post #108
dcrazy wrote: post #54

Earlier quoted context omitted.

Pay per issue and commit. Buy 1,000 commit credits at a time. Might force people to review their slop before pushing it.

I would support time based quotas or limits per tier (free, solo, team, etc) But I'm not going to pay per commit over my monthly plan

I wasn’t being completely serious with my suggestion, but I was thinking of a totally PAYG model. Monthly plans could come with credits, like they already do for Actions.
Post reply on HN