Cloudflare service outage June 12, 2025
blog.cloudflare.com
Cloudflare service outage June 12, 2025
1–10 of 17 posts
Re: Cloudflare service outage June 12, 2025
#2[deleted]
Re: Cloudflare service outage June 12, 2025
#3It's not clear what the actual issue was. My best guess is that Workers KV was in transition to B2 from a combination of GCS and "a storage provider" before. A step in transitioning to B2 involved removing the "a storage provider" and relying just on GCS for some period of time because of consistency concerns. Today's GCS outage was unfortunately timed and they didn't have coverage to support GCS being down.
Separately, their dependence on their own products results in a SPOF where now, theoretically, if B2 goes down so does over half of Cloudflare's product suite. Ideally those outages can be limited to a single region at a time but that's still a massive blast radius from a single service. I completely understand them not wanting to invest in competing products as a form of redundancy though.
Re: Cloudflare service outage June 12, 2025
#4This post did not mention the "external storage backend."
It's assumed that the "external storage backend" is Google Cloud Storage, since they also use it for Durable Objects storage: https://youtu.be/C5-741uQPVU?si=yOX-6gRzTbIhh34h&t=1725 (this video is worth the watch on their neat arch for Durable Objects SQLite)
Similarly, their data subprocessors lists Google for the developer platform platform: https://www.cloudflare.com/gdpr/subprocessors/cloudflare-ser...
And aligns with the Google Cloud Storage incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1S...
Re: Cloudflare service outage June 12, 2025
#5This post did not mention the "external storage backend." It's assumed that the "external storage backend" is Google Cloud Storage, since they also use it for Durable Objects storage: https://youtu.be/C5-741uQPVU?si=yOX-6gRzTbIhh34h&t=1725 (this video is worth the watch on their neat arch for Durable Objects SQLite) Similarly, their data subprocessors lists Google for the developer platform platform: https://www.clou…
makes no sense that google is behind all this, considering twitch(which uses AWS) and Microsoft outlook(Azure) also were having issues around the same time
Re: Cloudflare service outage June 12, 2025
#6It's not clear what the actual issue was. My best guess is that Workers KV was in transition to B2 from a combination of GCS and "a storage provider" before. A step in transitioning to B2 involved removing the "a storage provider" and relying just on GCS for some period of time because of consistency concerns. Today's GCS outage was unfortunately timed and they didn't have coverage to support GCS being down. Separate…
makes more sense to host your own at such scale. The more you fragment your infrastructure the more likely something is to break. Its the same idea of doing a raid 0 on multile drives: if a single one fails, all your stuff breaks.
Re: Cloudflare service outage June 12, 2025
#7There still seem to be some remaining issues. My Cloudflare Pages site is still giving me 500 errors. I looked up the response headers and realized that when requests are served by a certain data center, they fail, but if processed by another data center, they succeed. I suspected there to be some stale cache data, so I looked around the Cloudflare console but found no way to invalidate the cache in the Pages menu (the one in the domains menu didn't work). I also sent a ticket to their help center, only to be greeted by an AI. Probably waiting more will solve this problem automatically.
Re: Cloudflare service outage June 12, 2025
#8This post did not mention the "external storage backend." It's assumed that the "external storage backend" is Google Cloud Storage, since they also use it for Durable Objects storage: https://youtu.be/C5-741uQPVU?si=yOX-6gRzTbIhh34h&t=1725 (this video is worth the watch on their neat arch for Durable Objects SQLite) Similarly, their data subprocessors lists Google for the developer platform platform: https://www.clou…
makes no sense that google is behind all this, considering twitch(which uses AWS) and Microsoft outlook(Azure) also were having issues around the same time
Twitch and Outlook probably have an indirect dependency on GCP. Say, one of their third parties uses Cloudflare which uses GCP.
Re: Cloudflare service outage June 12, 2025
#9It's not clear what the actual issue was. My best guess is that Workers KV was in transition to B2 from a combination of GCS and "a storage provider" before. A step in transitioning to B2 involved removing the "a storage provider" and relying just on GCS for some period of time because of consistency concerns. Today's GCS outage was unfortunately timed and they didn't have coverage to support GCS being down. Separate…
I don’t think Cloudflare is using B2; Backblaze isn’t listed as a sub-processor.
https://www.cloudflare.com/gdpr/subprocessors/cloudflare-ser...
Re: Cloudflare service outage June 12, 2025
#10It's not clear what the actual issue was. My best guess is that Workers KV was in transition to B2 from a combination of GCS and "a storage provider" before. A step in transitioning to B2 involved removing the "a storage provider" and relying just on GCS for some period of time because of consistency concerns. Today's GCS outage was unfortunately timed and they didn't have coverage to support GCS being down. Separate…
I don’t think Cloudflare is using B2; Backblaze isn’t listed as a sub-processor. https://www.cloudflare.com/gdpr/subprocessors/cloudflare-ser...
Yeah it should be cloudflare R2