Cloudflare outage caused by bad software deploy
blog.cloudflare.com
Cloudflare outage caused by bad software deploy
1–10 of 137 posts
Re: Cloudflare outage caused by bad software deploy
#2I run a service placing bids the last few seconds on eBay. Every time this happens I lose measurable business (we place thousands of bids per day). While it doesn’t affect scheduled bids, they can’t place bids and are likely to move to a competitor. These recent outages have been costly.
Does anyone know a more reliable provider?
Re: Cloudflare outage caused by bad software deploy
#3Re: Cloudflare outage caused by bad software deploy
#4I run a service placing bids the last few seconds on eBay. Every time this happens I lose measurable business (we place thousands of bids per day). While it doesn’t affect scheduled bids, they can’t place bids and are likely to move to a competitor. These recent outages have been costly. Does anyone know a more reliable provider?
From experience, using two services (active/active) is really the only way to avoid downtime. DNS can be trickier, but there are providers that can fallback automatically or split requests (dnsmadeeasy etc)
Re: Cloudflare outage caused by bad software deploy
#5I run a service placing bids the last few seconds on eBay. Every time this happens I lose measurable business (we place thousands of bids per day). While it doesn’t affect scheduled bids, they can’t place bids and are likely to move to a competitor. These recent outages have been costly. Does anyone know a more reliable provider?
Maybe use cloudflare only for the landing page and docs but serve your bidding app frontend and backend directly? Since users are already there it will only affect first load :)
Re: Cloudflare outage caused by bad software deploy
#6Kinda wonder at this point what findings exist on their Availability SOC 2, assuming they've gotten one.
The repeated outages plus the constant malicious advertising by scammy ad providers through cloudflare are slowly turning me off to the service as a potential enterprise customer. Unfortunate too since plenty of superlatively qualified people build great things there (hat tip to Nick Sullivan), but it seems like the build-fast culture may now be impeding the availability requirements of their clients.
This is also a great example of a case where SLAs are meaningless without rigorous enforcement provisions negotiated in by enterprise clients. Cloudflare advertises 100% uptime (https://www.cloudflare.com/business-sla/) but every time they fall over, they're down for what, an hour at a time? Just this one issue would've blown anyone else's 99.99% SLA out of the water -- https://www.cloudflarestatus.com/incidents/tx4pgxs6zxdr
I love the service, but if I'm to consider consuming the service, they'd do well to have the equivalent of a long term servicing branch as its own isolated environment, one where changes are only merged in once they've proven to be hyper-stable.
Re: Cloudflare outage caused by bad software deploy
#7I run a service placing bids the last few seconds on eBay. Every time this happens I lose measurable business (we place thousands of bids per day). While it doesn’t affect scheduled bids, they can’t place bids and are likely to move to a competitor. These recent outages have been costly. Does anyone know a more reliable provider?
Maybe use cloudflare only for the landing page and docs but serve your bidding app frontend and backend directly? Since users are already there it will only affect first load :)
That is exactly how we use Cloudflare, but I appreciate the guidance. I am new to devops.
Re: Cloudflare outage caused by bad software deploy
#8I run a service placing bids the last few seconds on eBay. Every time this happens I lose measurable business (we place thousands of bids per day). While it doesn’t affect scheduled bids, they can’t place bids and are likely to move to a competitor. These recent outages have been costly. Does anyone know a more reliable provider?
Haha, people pay you to bid-snipe on eBay for them?
It just baffles me that manual/third-party bid-sniping is still a thing. eBay has had automatic bidding for more than twenty years. You'll pay the same whether you put in the winning bid a week in advance or 5 seconds. But people see that "you lost this auction" notice and they're irrationally convinced that it would have gone differently if they'd bid at the last minute, somehow.
Re: Cloudflare outage caused by bad software deploy
#9I run a service placing bids the last few seconds on eBay. Every time this happens I lose measurable business (we place thousands of bids per day). While it doesn’t affect scheduled bids, they can’t place bids and are likely to move to a competitor. These recent outages have been costly. Does anyone know a more reliable provider?
Haha, people pay you to bid-snipe on eBay for them? It just baffles me that manual/third-party bid-sniping is still a thing. eBay has had automatic bidding for more than twenty years. You'll pay the same whether you put in the winning bid a week in advance or 5 seconds. But people see that "you lost this auction" notice and they're irrationally convinced that it would have gone differently if they'd bid at the last m…
If other bidders are irrational, then bid-sniping can work. It doesn't give others the opportunity to contemplate, "I've been out-bid, do I actually want this item more than I originally thought?"
Re: Cloudflare outage caused by bad software deploy
#10Cloudflare should write a guide to doing post-event communication. Or perhaps they shouldn’t, as this seems to be a potential differentiator.
This is direct and doesn’t attempt to avoid blame. Well done.