Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bluesky April 2026 Outage Post-Mortem

pckt.blog

Post reply on HN
Print view View original

41–50 of 85 posts

Re: Bluesky April 2026 Outage Post-Mortem

#41

Post by floating-io »

drewg123 wrote: post #28

Golang's use of a potentially unbounded number of threads is just insane. I used to be fairly bullish on golang, but this, combined with the fact that its garbage collected, makes me feel its just unsuitable for production use.

You can have this problem with any kind of thread -- including OS threads -- if you do an unbounded spawn loop. Go is hardly unique in this.

Goroutines are actually better AFAIK because they distribute work on a thread pool that can be much smaller than the number of active goroutines.

If my quick skim created a correct understanding, then the problem here looks more like architecture. Put simply: does the memcached client really require a new TCP connection for every lookup? I would think you would pool those connections just like you would a typical database and keep them around for approximately forever. Then they wouldn't have spammed memcache with so many connections in the first place...

(edit: ah, it looks like they do use a pool, but perhaps the pool does not have a bounded upper size, which is its own kind of fail.)

Re: Bluesky April 2026 Outage Post-Mortem

#42

Post by numpad0 »

jonstaab wrote: post #13
pfraze wrote: post #11

Earlier quoted context omitted.

All support to other decentralizers but nothing never goes down.

1000x redundancy makes it vanishingly unlikely. Although I know we're due for a pole shift so all bets are off I suppose.

Wasn't aware there are ~2k relays now. Have inter-relay sharing situation improved?

When I tried it long time ago, the idea was just a transposed Mastodon model that the client would just multi-post to dozen different servers(relays) automatically to be hopeful that the post would be available in at least one shared relays between the user and their followers. That didn't seem to scale well.

Re: Bluesky April 2026 Outage Post-Mortem

#43

Post by direwolf20 »

tapoxi wrote: post #15

I don't really understand this architecture, but I thought Bluesky was distributed like Mastodon? How can it have an outage?

It's not really distributed. It's a centralised service that pulls some parts of 0.01% of user profiles from their own servers.

Re: Bluesky April 2026 Outage Post-Mortem

#44

Post by slopinthebag »

floating-io wrote: post #41
drewg123 wrote: post #28

Golang's use of a potentially unbounded number of threads is just insane. I used to be fairly bullish on golang, but this, combined with the fact that its garbage collected, makes me feel its just unsuitable for production use.

You can have this problem with any kind of thread -- including OS threads -- if you do an unbounded spawn loop. Go is hardly unique in this. Goroutines are actually better AFAIK because they distribute work on a thread pool that can be much smaller than the number of active goroutines. If my quick skim created a correct understanding, then the problem here looks more like architecture. Put simply: does the memcached…

Rust's async doesn't have this issue. Or at least, it's the same issue as malloc in an unbounded loop, but that's a more general issue not related to async or threading.

15-20 thousand futures would be trivial. 15-20 thousand goroutines, definitely not.

Re: Bluesky April 2026 Outage Post-Mortem

#45

Post by slopinthebag »

tombert wrote: post #40
drewg123 wrote: post #28

Golang's use of a potentially unbounded number of threads is just insane. I used to be fairly bullish on golang, but this, combined with the fact that its garbage collected, makes me feel its just unsuitable for production use.

Why does garbage collection make it unsuitable for production use? A lot of production software is written in garbage collected languages like Java. Pretty much the entire backend for iTunes/Apple Music is written in Java, and it's not doing any kind of fancy bump allocator tricks to avoid garbage. In my mind, kind of hard to argue that Apple Music is not "production use". There are certainly plenty of projects where…

Everything is understood by comparison. Unsuitable for production use, compared to what is the more apt question.

Re: Bluesky April 2026 Outage Post-Mortem

#46

Post by jonstaab »

numpad0 wrote: post #42
jonstaab wrote: post #13

Earlier quoted context omitted.

1000x redundancy makes it vanishingly unlikely. Although I know we're due for a pole shift so all bets are off I suppose.

Wasn't aware there are ~2k relays now. Have inter-relay sharing situation improved? When I tried it long time ago, the idea was just a transposed Mastodon model that the client would just multi-post to dozen different servers(relays) automatically to be hopeful that the post would be available in at least one shared relays between the user and their followers. That didn't seem to scale well.

Getting clients to do the right thing is like herding cats, but there has been some progress. Early 2023 Mike Dilger came up with the "gossip model" (renamed "outbox model" for obvious reasons). Here's my write-up: https://habla.news/hodlbod/8YjqXm4SKY-TauwjOfLXS

The basic idea is that for microblogging use cases users advertise which relays their content is stored on, which clients follow (this implies that there are less-decentralized indexes that hold these pointers, but it does help distribute content to aligned relays instead of blast content everywhere).

Also, relays aside, one key difference vs ActivityPub is that no third party owns your identity, which means you can move from one relay to another freely, which is not true on Mastodon.

Re: Bluesky April 2026 Outage Post-Mortem

#49

Post by fiatjaf »

pfraze wrote: post #17
tapoxi wrote: post #15

I don't really understand this architecture, but I thought Bluesky was distributed like Mastodon? How can it have an outage?

This writeup is useful for backend engineers: https://atproto.com/articles/atproto-for-distsys-engineers The simple answer is that atproto works like the web & search engines, where the apps aggregate from the distributed accounts. So the proper analogy here would be like yahoo going down in 1999.

Sorry, but this analogy is very misleading, no one browses websites through Google's servers.

For example, right now in my URL bar I read "news.ycombinator.com", not "google.com/profile/news.ycombinator.com".

If Google goes down now I can keep browsing this website and all the other websites I have in all my other tabs as if nothing had happened.

Re: Bluesky April 2026 Outage Post-Mortem

#50

Post by floating-io »

slopinthebag wrote: post #44
floating-io wrote: post #41

Earlier quoted context omitted.

You can have this problem with any kind of thread -- including OS threads -- if you do an unbounded spawn loop. Go is hardly unique in this. Goroutines are actually better AFAIK because they distribute work on a thread pool that can be much smaller than the number of active goroutines. If my quick skim created a correct understanding, then the problem here looks more like architecture. Put simply: does the memcached…

Rust's async doesn't have this issue. Or at least, it's the same issue as malloc in an unbounded loop, but that's a more general issue not related to async or threading. 15-20 thousand futures would be trivial. 15-20 thousand goroutines, definitely not.

I don't know enough about rust to confirm or deny that -- but unless rust somehow puts a limit on in-flight async operations, I don't see how it would help.

The problem is not resource usage in go. The problem is that they created umpteen thousand TCP connections, which is going to kill things regardless of the language.

Post reply on HN