Bluesky April 2026 Outage Post-Mortem
1–10 of 85 posts
Re: Bluesky April 2026 Outage Post-Mortem
#2> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request.
That’ll do it.
Re: Bluesky April 2026 Outage Post-Mortem
#3> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request. That’ll do it.
Zero, one, many, many thousands.
Re: Bluesky April 2026 Outage Post-Mortem
#4> What I had missed is that we deployed a new internal service last week that sent less than three GetPostRecord requests per second, but it did sometimes send batches of 15-20 thousand URIs at a time. Typically, we'd probably be doing between 1-50 post lookups per request. That’ll do it.
Ahh, the three relevant numbers in development: 0, 1, and infinity.
Re: Bluesky April 2026 Outage Post-Mortem
#5[flagged]
Re: Bluesky April 2026 Outage Post-Mortem
#6[flagged]
Why?
Re: Bluesky April 2026 Outage Post-Mortem
#7> The timing of these log spikes lined up with drops in user-facing traffic, which makes sense. Our data plane heavily uses memcached to keep load off our main Scylla database, and if we're exhausting ports, that's a huge problem.
I expect this is common.
Re: Bluesky April 2026 Outage Post-Mortem
#8Lite Blue on a dark Blue background. That is a new one, I have seen grey text on lite grey, but blue on blue ?
The article does work in lynx, at least I can read it.
Re: Bluesky April 2026 Outage Post-Mortem
#9Thank you for the post mortem on this outage.
Re: Bluesky April 2026 Outage Post-Mortem
#10nostr never goes down