Relax, they just had to install a new beam splitter for VIGIL
VIGIL?
[deleted]
Cloudflare API, dashboard, tunnels down
71–80 of 93 posts
Re: Cloudflare API, dashboard, tunnels down
#72If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment. As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope. luckily this time its just administrative interfaces and supplemental services that as…
The current situation is preferable to the before times, where any website could be nuked from orbit by anyone with a mild grievance. Cloudflare has succeeded in effectively shutting down that attack vector, and that’s worth the risk of occasional brief but broad outages, at least to me.
There were mitigations for this but it was usually on “IT”s budget and IT was considered a cost center.
Really the best innovation in the last 10 years has been the penchant for putting things on development budgets instead which loosened the purse strings considerably.
I wish I had sufficient data to back this up but for context; I was part of a small infrastructure team running a little over 1% of all web traffic at one point in a pair of colos (with DDoS protection) for less than £5MM on infra costs.
Now I can easily be in the 6 digits with just a CRUD app
Re: Cloudflare API, dashboard, tunnels down
#73If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment. As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope. luckily this time its just administrative interfaces and supplemental services that as…
I think the solution here isn’t “don’t use Cloudflare” … IMO if you need high availability and can’t tolerate downtime, then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly) and 2 or more clouds. The centralization around AWS and GCP and Azure is more concerning to me than the centralization around Cloudflare. It’s not obvious to me why lots of people using Cloudflare is any worse than lots…
This. If downtime is that big of an issue, you should be on Cloudflare's business/enterprise plan which allows CNAME DNS, have redundant DNS providers with 30s TTLs that flip over to another provider. Better yet if you can 50/50 the traffic and 100/0 it when something goes bad. Downside is cost and DNSSEC has an issue with multiple keys. I think there was an RFC on that though I'm not sure it's well supported yet.
Re: Cloudflare API, dashboard, tunnels down
#74Re: Cloudflare API, dashboard, tunnels down
#75If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment. As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope. luckily this time its just administrative interfaces and supplemental services that as…
Is it really that bad? Services go down for a few minutes, maybe a few hours? A thousand other companies are down at the same time. Grab lunch, come back, and it'll have fixed itself. Nobody blames you because it's just another provider outage -- whether it's Cloudflare or AWS or Microsoft or Gmail doesn't really matter, it's a convenient outsourcing of not just network admin but also blame. The alternative is, what,…
>Grab lunch
Sorry man, our Stripe terminals aren't working, we can't even accept cash... Um, come back later.
Re: Cloudflare API, dashboard, tunnels down
#76Downtime happens - no one is happy about it but it comes with the job and downtime will happen whether you use cloud or run on prem. Cloudflare at least accurately reflects the incident on their status page. Often dashboards will show everything green despite the service being down. In these moments communication is critical and Cloudflare does it well especially as we all know when it comes to the post mortem. They…
True.
It's also nuts that this downtime according to creation of the post -> resolved in comments, was only 27 minutes...
I've literally been on standby to get an answer what happened with a huge cloud provider for 3 weeks now and today they got back with static assets that weren't reachable ( ico's ???)
They don't even know what happened since it's not logged in their logging, since it's using middleware and because of the sudden weekend downtime, they middleware wasn't triggered...
After a detailed description that we found that it was related to a app service plan, they came back with 404's on static assets...
Only now it got escalated..
Fuck the cloud, outside of cloudflare.
The only time we had a downtime with cloudflare, they gave a detailed description and it was related to our cloud provider and not Cloudflare . Then their was a infra update and it also didn't get logged... Because of that we thought it was the wrong party...
18 hours down, redeployment fixed it, nothing logged on their end. Luckily one service wasn't redeployed ( barely used), so we can prove there was an issue.
Ho boy...
I'm wondering how many more weeks it will take...
Re: Cloudflare API, dashboard, tunnels down
#77If you’re waiting for the inevitable “this is why we shouldn't centralize everything”; this is that comment. As much as I like cloudflare and easy to use stuff, lets be a little bit mindful of the blast radius of highly centralized services which are even more complex to operate due to the scale of the deployment and problem scope. luckily this time its just administrative interfaces and supplemental services that as…
I think the solution here isn’t “don’t use Cloudflare” … IMO if you need high availability and can’t tolerate downtime, then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly) and 2 or more clouds. The centralization around AWS and GCP and Azure is more concerning to me than the centralization around Cloudflare. It’s not obvious to me why lots of people using Cloudflare is any worse than lots…
>then deploy and split load between 2 or more providers (e.g. Cloudflare, Fastly)
How does this work when most edge providers manage your DNS to point at the available servers they have?
Also, if you "HAVE" to be online, then you're the type of target that's at big risk of DDOS which complicates this even further.
Re: Cloudflare API, dashboard, tunnels down
#78Earlier quoted context omitted.
I don't think the problem is a few minutes of down time but all services going down at the same time. You can't order food (Uber down), your email down, your TV down, maybe even the traffic control?
Ok, and...? (Edit: Not to be snarky, but really, so what if that happens? Same thing happens if there's a power outage in your neighborhood. Is it a big deal?)
Are power outages even a thing? My PC at home (germany) is always on, so I'd notice, and I'm involved with energy at our data center (Frankfurt, Germany) and home and external data center had no outage for the last 20 years, at a minimum.
Re: Cloudflare API, dashboard, tunnels down
#79Earlier quoted context omitted.
Is it really that bad? Services go down for a few minutes, maybe a few hours? A thousand other companies are down at the same time. Grab lunch, come back, and it'll have fixed itself. Nobody blames you because it's just another provider outage -- whether it's Cloudflare or AWS or Microsoft or Gmail doesn't really matter, it's a convenient outsourcing of not just network admin but also blame. The alternative is, what,…
>Grab lunch Sorry man, our Stripe terminals aren't working, we can't even accept cash... Um, come back later.
That actually happens with some frequency where I live, and they either use the old-school card imprint machines, or take down your number manually, or some terminals have an offline mode where it'll save the cards and then batch-run them later. It's also rare enough that it's not an issue at all in regular life.
Edit: Just to clarify, obviously this can't be both common and rare... what I meant is that different neighborhoods will frequently go down, but there's usually some neighborhood within a 10-min drive that still has power, or some food truck has a cellular connection off their iPad PoS that still works, or you happen to have cash in your wallet or whatever or know the guy and they'll just put it on your tab for next time, whatever. I live in a semi-rural community in Oregon and we have a lot of small businesses, a lot of wildfires, and not a whole lot of infrastructure. We just kinda make do when things happen.
A couple weeks ago, half the town lost power for the whole day, and the other half of the town just got some increased business and traffic. Life was back to normal the next day.
Re: Cloudflare API, dashboard, tunnels down
#80Earlier quoted context omitted.
Ok, and...? (Edit: Not to be snarky, but really, so what if that happens? Same thing happens if there's a power outage in your neighborhood. Is it a big deal?)
Are power outages even a thing? My PC at home (germany) is always on, so I'd notice, and I'm involved with energy at our data center (Frankfurt, Germany) and home and external data center had no outage for the last 20 years, at a minimum.
[flagged]