Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

271–280 of 804 posts

Re: The August 17 outage

#271

Post by fragmede »

wegwerper wrote: post #165
toephu2 wrote: post #77

Earlier quoted context omitted.

Not that impressive when you realize it's mostly due to AI slop edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.

Agree - this reminds me very much of the old joke of two economists increasing GDP by taking turns giving the same 200usd back and forth for having each other eat shit. Useful / impressive for whom is the question. Not for us! We pay for Github enterprise, and because GH can't be bothered to separate service tiers for sloplords and actual paying customers we get garbage level performance. They could of course always…

They do. Talk to your account representative for the alternate domain name to hit.

Re: The August 17 outage

#272

Post by brazukadev »

meerita wrote: post #260
blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

It's almost double. If you have scaling prepared, it should be linear, but I doubt they've prepared for this.

they are/were in the middle of a migration

Re: The August 17 outage

#273

Post by cube00 »

dannyw wrote: post #232
ACCount37 wrote: post #208

Earlier quoted context omitted.

"You can't seriously tell me that the unhappy leg of the code path has no test coverage." Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.

It’s been proven that VS Code has product managers who vibe code commits that get merged, including the co-authored with copilot bug. So there’s some problems there.

They also charge ahead to prod with known defects and then change their story when they're caught out https://news.ycombinator.com/item?id=48032310

Re: The August 17 outage

#274

Post by ctvo »

jdm2212 wrote: post #224
grim_io wrote: post #205

Earlier quoted context omitted.

Why? You can retry, but there is nothing wrong with increasingly waiting slightly longer if we fail many times.

Try asking Opus or Fable that question. It'll give you a good answer on why microservice architectures work the way they do in order to keep user-facing latency acceptable and minimize downtime. It's a complicated enough topic that I don't feel like explaining it for free to you in a HN comment.

Why are you pretending this is a harder topic than it is and you have some secret insight. And you're a dick on top of it:

Yes, exponential backoffs alone are insufficient. Yes, adding jitter helps randomize the calls across a fleet and should be the default with exponential backoffs. Yes, both of these may be sufficient for most systems. Yes, you can dive more into circuit breakers and adaptive retries to limit thundering herd.

https://aws.amazon.com/blogs/architecture/exponential-backof...

https://brooker.co.za/blog/2022/02/28/retries.html

Re: The August 17 outage

#275

Post by steve-atx-7600 »

altcognito wrote: post #265

Distributing across different services wouldn't be a bad idea.... I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but... Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (exce…

“ Name another service that does what they do for FREE (and no ads) at this scale” and is reliable is the question

Re: The August 17 outage

#276

Post by vermilingua »

shdtabasum wrote: post #154
blakesterz wrote: post #4

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.

LOL, this kind of things will happen when projects like Bun ( https://github.com/oven-sh/bun ) are running on auto. :)

More than 50% of the recent commits are from robobun, jesus

Re: The August 17 outage

#277

Post by sandeepkd »

Anon1096 wrote: post #255
sandeepkd wrote: post #229

Earlier quoted context omitted.

I am afraid thats now how infrastructure works from what I have seen. The number that really matters is QPS. For any system the QPS varies through out the day and across the week and months. Most design considerations easily absorb any 2X increase. Pick up any company and the chances are that the servers are over provisioned, no one takes chances specially with critical components. What you have going on with Github…

How much infra have you seen of the top 100 sites in the world? I've worked on multiple top 10, and absorbing a 2x increase (and the peak is very likely more than 2x) is a very very hard problem that would cause hundreds of pagers to go off and load shedding to very high degrees. There is just not tons of unused capacity lying around in wait at the scale of github. "No one takes chances with critical components" is a…

I have some good experience and I feel bad about state of these things too specially given that a lot of it could have been prevented. What you have here is not a single service, its a system compromised of hundreds of services, possibly without clear ownership for some of them after these many years and reorgs. There is not a single person or group that understands the whole system from technical standpoint and pressure points. It akin to people trying to plug the holes as the water starts getting under pressure from different joints. This duct taping is present in almost all big enough systems, you name them.

> "No one takes chances with critical components" is also very wrong for the simple fact that you don't know which is the weakest link in the chain until it fails.

  These companies were built and run by people passionate enough for the craft, ones who cared for the systems, who designed them. There is this idea that you can replace people by process and everyone is replaceable. What you have is a classical state where people are just doing their time.

Re: The August 17 outage

#280

Post by blitzar »

Shish2k wrote: post #58
dpweb wrote: post #41

Sorry to suggest this but if they charged everyone say $1/mo. it would absolutely help the massive surge from AI coding they seem to have had. I don't like paying for free stuff but gh certainly worth it.

Unfortunately time and time again, the overwhelming majority of people show that they would rather deal with an unreliable scummy company for "free" than pay $1/mo for a reliable service which treats them with respect :(

> service which treats them with respect

Which mythical tech company - free or paid - does this?

Post reply on HN