Tell us more about this buggy "new internal service" that's scraping batch data :P
Bluesky April 2026 Outage Post-Mortem
21–30 of 85 posts
Re: Bluesky April 2026 Outage Post-Mortem
#22Re: Bluesky April 2026 Outage Post-Mortem
#23nostr never goes down
If nostr went down would people even notice?
Re: Bluesky April 2026 Outage Post-Mortem
#24Re: Bluesky April 2026 Outage Post-Mortem
#25I don't really understand this architecture, but I thought Bluesky was distributed like Mastodon? How can it have an outage?
This writeup is useful for backend engineers: https://atproto.com/articles/atproto-for-distsys-engineers The simple answer is that atproto works like the web & search engines, where the apps aggregate from the distributed accounts. So the proper analogy here would be like yahoo going down in 1999.
This is a fantastic write-up, thanks for sharing!
Re: Bluesky April 2026 Outage Post-Mortem
#26[dead]
Re: Bluesky April 2026 Outage Post-Mortem
#27> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request. That’ll do it.
The incredible part about this is because their backend is all TCP/IP they were literally exhausting the ports by leaving all 65k of them in TIME_WAIT, and the workaround was to start randomizing the localhost address to give them another trillion ports or so.
Re: Bluesky April 2026 Outage Post-Mortem
#28Golang's use of a potentially unbounded number of threads is just insane. I used to be fairly bullish on golang, but this, combined with the fact that its garbage collected, makes me feel its just unsuitable for production use.
Re: Bluesky April 2026 Outage Post-Mortem
#29> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request. That’ll do it.
And then they fix the issue by using multiple localhost IPs rather than, perhaps, not sending 15-20 thousand URIs at a time
Re: Bluesky April 2026 Outage Post-Mortem
#30I don't really understand this architecture, but I thought Bluesky was distributed like Mastodon? How can it have an outage?
A web interface and home server can have an outage. Bluesky is just a web interface and home server.