I am disappointed to see this article flagged. I thought it was excellent.
Questions for Cloudflare
51–60 of 71 posts
Re: Questions for Cloudflare
#52A lot of these questions bely a misunderstanding of how it works - bot management is evaluated inline within the proxy as a feature on the site (similar to other features like image optimization). So during ingress there’s not an async call to the bot management service which intercepts the request before it’s outbound to origin - it’s literally a Lua script (or rust module in fl2) that runs on ingress inline as part…
Maybe I'm misunderstanding something but it being a blocking call does not make timeouts less important -- if anything they become more important!
Re: Questions for Cloudflare
#53It's a detailed postmortem published within a couple hours of the incident and this blog post is disappointed that it didn't provide a comprehensive assessment of all the procedural changes inside the engineering organization that came as a consequence. At the point in time when this blog post was written, it would not have been possible for them to answer these questions.
Part of my argument in the article is that it does't take long to come to that realisation if using the right methods. It would absolutely have been possible to identify the problem of missing feedback by that time.
It absolutely does take long with the right methods; in fact, the righter the methods, the longer it takes. You're talking about a postmortem that was up within single digit hours of the initial incident resolution. A lot of orgs would wait on the postmortem just to be sure the system is settling back into a steady state!
You were way off here.
Re: Questions for Cloudflare
#54A lot of these questions bely a misunderstanding of how it works - bot management is evaluated inline within the proxy as a feature on the site (similar to other features like image optimization). So during ingress there’s not an async call to the bot management service which intercepts the request before it’s outbound to origin - it’s literally a Lua script (or rust module in fl2) that runs on ingress inline as part…
Maybe I'm misunderstanding something but it being a blocking call does not make timeouts less important -- if anything they become more important!
I don't understand how it is you're doing distributed systems design on a system you don't even have access to. Maybe the issue is timeouts, maybe the issue is some other technical change, maybe the issue is human/procedural. How could you possibly know? The owners of the system probably don't have a full answer within just a few hours of handling the incident!
Re: Questions for Cloudflare
#55Earlier quoted context omitted.
Maybe I'm misunderstanding something but it being a blocking call does not make timeouts less important -- if anything they become more important!
I don't understand how it is you're doing distributed systems design on a system you don't even have access to. Maybe the issue is timeouts, maybe the issue is some other technical change, maybe the issue is human/procedural. How could you possibly know? The owners of the system probably don't have a full answer within just a few hours of handling the incident!
I would be worried if they had all the answers within a few hours! I was just caught off guard by the focus on technical control measures when there seems to have been fairly obvious problems with information channels.
For example, "more global kill switches for features" is good, but would "only" have shaved 30 % off the time of recovery (if reading the timeline charitably). Being able to identify the broken component faster would have shaved 30–70 % off the time of recovery depending on how fast identification could happen – even with no improvements to the kill switch situation.
Re: Questions for Cloudflare
#56Earlier quoted context omitted.
Part of my argument in the article is that it does't take long to come to that realisation if using the right methods. It would absolutely have been possible to identify the problem of missing feedback by that time.
It absolutely does take long with the right methods; in fact, the righter the methods, the longer it takes. You're talking about a postmortem that was up within single digit hours of the initial incident resolution. A lot of orgs would wait on the postmortem just to be sure the system is settling back into a steady state! You were way off here.
To be clear, I'm not expecting a full analysis within hours. I'm hoping for a method of analysis by which the major deficiencies come up at a high level, and then as more effort is spent on it, more details around those deficiencies are revealed.
What otherwise tends to happen, in my experience, is the initial effort brings up some deficiencies which are only partially the major ones, and subsequent effort is spent looking mainly in that same area, never uncovering those major deficiencies which were not initially discovered.
Re: Questions for Cloudflare
#57I am disappointed to see this article flagged. I thought it was excellent.
In defense of your taste, it was updated based on the loud feedback here, so you probably read a slightly better version than that which was flagged.
Re: Questions for Cloudflare
#58A lot of these questions bely a misunderstanding of how it works - bot management is evaluated inline within the proxy as a feature on the site (similar to other features like image optimization). So during ingress there’s not an async call to the bot management service which intercepts the request before it’s outbound to origin - it’s literally a Lua script (or rust module in fl2) that runs on ingress inline as part…
Maybe I'm misunderstanding something but it being a blocking call does not make timeouts less important -- if anything they become more important!
It’s literally in the same process as the router. Performance issues like the layer taking too long are handled differently than just timing out afaik. It’s possible it should time out but safe cancellation in Rust can be tricky depending on how this is architected. Timeouts are irrelevant to this outage and I wouldn’t classify it as a huge issue unless there’s a possibility that a layer has an indefinite blocking call as a misbehavior and forever hangs all requests. My hunch is that this is generally already handled - FL2 is not documented enough to speculate.
Re: Questions for Cloudflare
#59Earlier quoted context omitted.
? I don't remember telling anyone to trust the reviews? I think it is healthy to try alternatives to Cloudflare and then come to your own decision.
I'm not saying you did, but for me things like what I mentioned are red flags. They also use C#—another red flag. There's OVH, Hetzner, DigitalOcean, etc—all are private companies that aren't on Wall Street.
No.
DigitalOcean is owned by Wall Street.
Only Hetzner is a good alternative CDN.
Re: Questions for Cloudflare
#60Earlier quoted context omitted.
I'm not saying you did, but for me things like what I mentioned are red flags. They also use C#—another red flag. There's OVH, Hetzner, DigitalOcean, etc—all are private companies that aren't on Wall Street.
No. DigitalOcean is owned by Wall Street. Only Hetzner is a good alternative CDN.
You're right, DO is public.