Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The August 17 outage

github.blog

Post reply on HN
Print view View original

671–680 of 804 posts

Re: The August 17 outage

#671

Post by darkwater »

vasco wrote: post #637
nosefrog wrote: post #599

Earlier quoted context omitted.

I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding. And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, b…

You understand it if you think of engineering infinites rather than mathematical infinites. The capacity of a full single datacenter can be treated as infinite for most customers. I explained how its defined in the original comment. Amount of places in engineering where you treat even a 3 order of magnitude difference as infinite is a lot, but the number of order of magnitudes varies depending on context.

You don't have a single Google datacenter (or any other provider / big name) for a single tenant. The bigger the DC, the more the number of tenants/customers reaches "infinity" (giving the same meaning to "infinity" you are giving to it)

Re: The August 17 outage

#672

Post by inigyou »

colechristensen wrote: post #460
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

One of the things I've done is create an entire fully functional GitHub alternative that does more of what I want and hosts all of my other projects, so yes I at least am getting considerably more done.

You could have just installed Forgejo.

Re: The August 17 outage

#673

Post by gilrain »

msephton wrote: post #429
jryan49 wrote: post #336

With all the software being written on github you'd think we were going though a software rennasance. Where are the results? Is it really just all slop?

I can't speak for all of it, obviously, I don't have time to try much of it, but I see tons of amazing new software in my various feeds pretty much daily.

What are the top 3 you remember? You say you can’t speak for all of it… can you speak for any of it?

Re: The August 17 outage

#674

Post by markus_zhang »

neya wrote: post #602
afc wrote: post #531

> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity. This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling fai…

Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it. Worth reading: https://isolveproblems.substack.com/p/how-microsoft-vaporize...

This looks familiar. Microsoft had similar experiences back in the early Word and Windows era. Too many bugs and too late to ship.

Re: The August 17 outage

#675

Post by inigyou »

vermilingua wrote: post #276
shdtabasum wrote: post #154

Earlier quoted context omitted.

LOL, this kind of things will happen when projects like Bun ( https://github.com/oven-sh/bun ) are running on auto. :)

More than 50% of the recent commits are from robobun, jesus

Here's the most recent nontrivial one as of this comment: https://github.com/oven-sh/bun/commit/d4de65e9a43224a14591ad...

The code change makes no sense and should do nothing. The commit message described a very deep investigation into garbage collection on the C++ side. Some object is being kept alive when the test requires it to be collected, and changing the code in this way allegedly prevents that. But wouldn't you think there would be a better way to ensure an object gets collected, like setting the variable to null?

The comments in the code don't make a lot of sense either. Something so obscure and brittle has to be explained extremely clearly.

While the issue might be real, this commit is so far away from the locus of normal that it's sending red alert. Plus a hallucination is very likely with such a long investigation - once an LLM agent starts investigating it just assumes there is a problem. And this is the 1 out of 1 robobun commit that I looked at.

Edit: here's the next one: https://github.com/oven-sh/bun/commit/72ec6e2594892455df0090...

Make sure the fs module keeps working if someone freezes or seals its exports table. I was wondering who was going around freezing random tables from other modules, so I checked the linked issue - robobun reported the issue, too. Why? I'm skeptical of whatever robobun was doing when it decided that it was necessary for code outside of a module to freeze their export tables. It needs a very good justification.

Re: The August 17 outage

#676

Post by solatic »

dastbe wrote: post #653
solatic wrote: post #634

Earlier quoted context omitted.

The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario. > Application backends? I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only wo…

are these separate services? Are you going to integrate how services preempt into overall system design? My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applyin…

> you want to move these decisions into the application tier as much as possible

I actually agree, but this is a luxury that most large companies cannot politically prioritize (it is not Product/Sales-driven, see earlier comment). Especially when the company is large, and there are dozens if not hundreds of developer teams in a polyglot microservice environment, pushing application-level handling of these concerns is virtually impossible without executive support, and because it doesn't move the bottom line in an easily measurable way, you won't get executive support.

Companies much prefer infrastructure-based solutions to these problems, even if they're coarser, because the relatively small number of people who need to be involved makes it politically feasible. Easy example off the top of my head - mutual TLS encrypting east-west traffic has been implementable at the application layer for decades, but it was a pipe dream until service meshes made it easy to deploy (it's still a pipe dream for many orgs that refuse to schedule any infra work not Product/Sales-driven though).

Re: The August 17 outage

#677

Post by goalieca »

ozgung wrote: post #508
m4rtink wrote: post #417

Earlier quoted context omitted.

So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?

Has anyone noticed an increase in the quality, performance or capabilities of the software they use?

Well, many of the saas I use continue to have problem meeting their SLA. And I’m on support just as often as always trying to get through to a human to file a bug that will never get fixed. “It’s in our roadmap”.. no it isn’t. Even with LLMs, product will focus on making new features to push their AI mission.

Re: The August 17 outage

#678

Post by croes »

user43928 wrote: post #576
croes wrote: post #564

Earlier quoted context omitted.

Wrong. I don’t dislike AI but really of mine are negatively affected by climate change and AI isn’t helping what is easily observed when Google and MS scrapped their CO2 reduction targets. So every time I use AI I think about the necessity and usefulness of what I‘m doing with AI and if the use outweighs the costs. Since the rise of AI the environmental impact doesn’t seem to matter anymore. I guess because it’s the…

You're not helping your case by questioning the usefulness of software produced with AI in the same comment section, or complaining about other people supposedly compromising their own privacy in overusing AI. The reason it doesn't matter is because the environmental impact is moderate, and the benefit obviously tremendous.

I'm pretty sure that most of people private data contains data about third parties too. It's more as their own privacy they compromise

Re: The August 17 outage

#679

Post by 27183 »

afc wrote: post #647
concerned_user wrote: post #611

Earlier quoted context omitted.

If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.

+1. This is a very real problem in practice. A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, ra…

It's easier to create a nice worst-case failure UX this way too. Rather than a page that sorta kinda loads but doesn't actually work at all you can unambiguously put up a "oops we're broken right now try again later" page whenever that top-level endpoint returns a non-retryable error.

But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.

Re: The August 17 outage

#680

Post by inigyou »

f6v wrote: post #659
winrid wrote: post #467

Earlier quoted context omitted.

This will result in pretty low quality software. I've had opus 5 along with its AI code reviewer agree to do some pretty stupid shit.

Wait till you see what some people agree to.

Ah the good old "AI isn't bad because people are bad too" argument
Post reply on HN