Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bluesky April 2026 Outage Post-Mortem

pckt.blog

Post reply on HN
Print view View original

61–70 of 85 posts

Re: Bluesky April 2026 Outage Post-Mortem

#61

Post by wmf »

evbogue wrote: post #60
wmf wrote: post #58

Earlier quoted context omitted.

Not the original poster but I do have some ideas. Official Bluesky clients could randomly/round-robin access 3-4 different appview servers run by different organizations instead of one centralized server. Likewise there could be 3-4 relays instead of one. Upgrades could roll across the servers so they don't all get hit by bugs immediately.

If multiple personal data servers (pdses) share the same set of posts how would we guarantee that they are tamper resistant to third parties?

PDSes should be sharded not replicated. Your posts live on your PDS which lives in one place (although it can move).

Re: Bluesky April 2026 Outage Post-Mortem

#62

Post by evbogue »

wmf wrote: post #61
evbogue wrote: post #60

Earlier quoted context omitted.

If multiple personal data servers (pdses) share the same set of posts how would we guarantee that they are tamper resistant to third parties?

PDSes should be sharded not replicated. Your posts live on your PDS which lives in one place (although it can move).

What's stopping us from doing both?

Re: Bluesky April 2026 Outage Post-Mortem

#63

Post by kyledrake »

jandrese wrote: post #27
threecheese wrote: post #2

> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request. That’ll do it.

The incredible part about this is because their backend is all TCP/IP they were literally exhausting the ports by leaving all 65k of them in TIME_WAIT, and the workaround was to start randomizing the localhost address to give them another trillion ports or so.

This is a pretty interesting solution. I could see how this could useful for certain kinds of problems (as part of a ddos attack mitigation for example).

Re: Bluesky April 2026 Outage Post-Mortem

#64

Post by wmf »

evbogue wrote: post #62
wmf wrote: post #61

Earlier quoted context omitted.

PDSes should be sharded not replicated. Your posts live on your PDS which lives in one place (although it can move).

What's stopping us from doing both?

Cost and complexity tradeoffs. IMO the relay/appview is the current bottleneck.

Re: Bluesky April 2026 Outage Post-Mortem

#65

Post by evbogue »

wmf wrote: post #64
evbogue wrote: post #62

Earlier quoted context omitted.

What's stopping us from doing both?

Cost and complexity tradeoffs. IMO the relay/appview is the current bottleneck.

This is why I'm hoping fiatjaf has a recommendation here. I have a feeling he might have a proposal that solves this. But doesn't solve all of it, just some of it.

Re: Bluesky April 2026 Outage Post-Mortem

#67

Post by verdverm »

kyledrake wrote: post #63
jandrese wrote: post #27

Earlier quoted context omitted.

The incredible part about this is because their backend is all TCP/IP they were literally exhausting the ports by leaving all 65k of them in TIME_WAIT, and the workaround was to start randomizing the localhost address to give them another trillion ports or so.

This is a pretty interesting solution. I could see how this could useful for certain kinds of problems (as part of a ddos attack mitigation for example).

An oldie and a goodie

https://news.ycombinator.com/item?id=21865715

Re: Bluesky April 2026 Outage Post-Mortem

#68

Post by verdverm »

floating-io wrote: post #50
slopinthebag wrote: post #44

Earlier quoted context omitted.

Rust's async doesn't have this issue. Or at least, it's the same issue as malloc in an unbounded loop, but that's a more general issue not related to async or threading. 15-20 thousand futures would be trivial. 15-20 thousand goroutines, definitely not.

I don't know enough about rust to confirm or deny that -- but unless rust somehow puts a limit on in-flight async operations, I don't see how it would help. The problem is not resource usage in go. The problem is that they created umpteen thousand TCP connections, which is going to kill things regardless of the language.

case in point, an old HN post about scaling Go to 1M websocket connections

https://news.ycombinator.com/item?id=21865715

Post reply on HN