Bluesky April 2026 Outage Post-Mortem

Post by [deleted] » Fri, Apr 10, 2026, 9:46 PM UTC

slopinthebag wrote: post #44

floating-io wrote: post #41

Earlier quoted context omitted.

You can have this problem with any kind of thread -- including OS threads -- if you do an unbounded spawn loop. Go is hardly unique in this. Goroutines are actually better AFAIK because they distribute work on a thread pool that can be much smaller than the number of active goroutines. If my quick skim created a correct understanding, then the problem here looks more like architecture. Put simply: does the memcached…

Rust's async doesn't have this issue. Or at least, it's the same issue as malloc in an unbounded loop, but that's a more general issue not related to async or threading. 15-20 thousand futures would be trivial. 15-20 thousand goroutines, definitely not.

Post by evbogue » Fri, Apr 10, 2026, 10:06 PM UTC

fiatjaf wrote: post #49

pfraze wrote: post #17

Earlier quoted context omitted.

This writeup is useful for backend engineers: https://atproto.com/articles/atproto-for-distsys-engineers The simple answer is that atproto works like the web & search engines, where the apps aggregate from the distributed accounts. So the proper analogy here would be like yahoo going down in 1999.

Sorry, but this analogy is very misleading, no one browses websites through Google's servers. For example, right now in my URL bar I read "news.ycombinator.com", not "google.com/profile/news.ycombinator.com". If Google goes down now I can keep browsing this website and all the other websites I have in all my other tabs as if nothing had happened.

Post by emidoots » Fri, Apr 10, 2026, 10:07 PM UTC

jonstaab wrote: post #10

nostr never goes down

Post by madeofpalk » Fri, Apr 10, 2026, 10:11 PM UTC

tombert wrote: post #40

drewg123 wrote: post #28

Golang's use of a potentially unbounded number of threads is just insane. I used to be fairly bullish on golang, but this, combined with the fact that its garbage collected, makes me feel its just unsuitable for production use.

Why does garbage collection make it unsuitable for production use? A lot of production software is written in garbage collected languages like Java. Pretty much the entire backend for iTunes/Apple Music is written in Java, and it's not doing any kind of fancy bump allocator tricks to avoid garbage. In my mind, kind of hard to argue that Apple Music is not "production use". There are certainly plenty of projects where…

Post by Night_Thastus » Fri, Apr 10, 2026, 10:16 PM UTC

jandrese wrote: post #27

threecheese wrote: post #2

> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request. That’ll do it.

The incredible part about this is because their backend is all TCP/IP they were literally exhausting the ports by leaving all 65k of them in TIME_WAIT, and the workaround was to start randomizing the localhost address to give them another trillion ports or so.