1.5 billion commits of worthless slop
The August 17 outage
521–530 of 804 posts
Re: The August 17 outage
#522Earlier quoted context omitted.
So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?
I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.
Same!
Re: The August 17 outage
#523Re: The August 17 outage
#524"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.
So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?
That’s like asking whether fuel consumption was productive or leisure as the number of cars on the road increased. It’s both! I don’t think you can separate one from the other in any reasonable way.
Re: The August 17 outage
#525Earlier quoted context omitted.
Ideally you have levers further up from your local load balancers as well. Even at the edge. Granted you never want those to trigger but it’s better than fighting a storm while you fix things.
I wondered that myself. Curious as to why they couldn’t shed load at the edge to help protect goodput.
Isn't that what they did though? Start returning more-or-less hardcoded 403s for the Copilot endpoint that was causing the issues?
Re: The August 17 outage
#526Re: The August 17 outage
#527Earlier quoted context omitted.
I have some big issues with this technology and the companies behind it, but I know of quite a few people personally who were not previously coders but have now been able to use LLMs to make their own custom software, solving real problems they had.
Do you believe this is a significant part of the increased GutHub traffic, people’s personal software?
Are non-coders these days aware that they should use version control (and push their code to GitHub)? Or does their agent helpfully suggest setting up a GitHub repo?
Re: The August 17 outage
#528> Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.
I operated services at similar scale, and generally we use to put a bit of slack so that you would get an alarm when capacity goes up to 80%+ (or whatever number makes sense)
This allows to check, in the morning, after coffee, why the load balancer fleet didn't scale up automatically.
I am sure there is a good answer to why this is impractical, but it would be nice to know
Re: The August 17 outage
#529> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.
I'm missing in these descriptions the most obvious approach: Resilience. Shedding load so that you can keep services up even though capacity is too low. If you flip over as soon as load exceeds what you can handle then this problem will never go away, unless you always have insane overprovisioning of resources which is uneconomical. There will always be spikes. You need to plan to handle them, no matter how high.
> we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks.
Sorry, but again, that is not good enough. They should ask themselves why they are expecting that trying the same medicine as last time will prevent next time. It won't.
With that mindset I'm not surprised this happened and it will surely happen again.
Edit: In more concrete terms. If you 2x your capacity and in a week you face a burst 2x of what happened last time, you are back in the same seat. If you improve efficiency by 2x, same thing. And after a bottleneck is before a bottleneck. There will always be a bottleneck. The key is to be able to handle a bottleneck. Removing one just pushes the issue to the next one.
Your architecture must be such that your whole system should be able to run on a raspberry pi. Most client requests would be dropped, but those that make it through will be served. If your architecture serves 0% because it crashes when load is 10% over capacity, then capacity increases or efficiency increases or bottleneck removal are not going to prevent the next outage.
Re: The August 17 outage
#530Earlier quoted context omitted.
To some extent we are getting more things done as well. In my company (mid-sized startup), they're making us push features every other day now as opposed to maybe 1-2 features per person per sprint. Back when I joined, things were a lot slower. Today, they expect freshers to push new features on day one. But ofc, slop has increased a lot more as well.
That's very fast. Feels like you cannot really review this code so it's all just "working" AI code with few railguards and few human supervision?
Jumping on the great asymmetry of output between humans and llms, yet still being responsible for the product is pure kafka.