111–120 of 149 posts
Earlier quoted context omitted.
"why so" what? not sure what you're questioning/implying here.
I thought you wanted users to flag Cloudflare domains through a browser extension, which is why I got confused.
I assume there's some way to detect that mechanically, which is probably what you described. I just haven't yet seen anything that does this - hopefully it already exists, if not I should probably make it.
Earlier quoted context omitted.
I don't understand what difference bots make. For me, a website (the public part) is a storefront. People walk down the street and see what's inside — that's the purpose. If something should not be available immediately, that's the private part of the store. I've been monitoring bot traffic on digital platforms for over 10 years. Sure, the crawler share is growing, some even with malicious intentions, and those I det…
When most of your server capacity is going to answering the scrapers it matters. It's not that the stuff is hidden, it's that storefront being flooded with 10x as many customers as the fire code allows. And some of them go around asking your employees mindless questions. (Small forum I help moderate: we were getting hammered with what was probably some sort of AI that was taking search queries and feeding them into t…
I've been dealing with the web since the previous century and still haven't managed to build a website that could be hurt by scrapers visiting it.
If you went through the logs, you'd probably see that these bots are on a single IP or subnet, which can be easily detected and blocked instead of closing off search to non-registered users.
Earlier quoted context omitted.
I don't understand what difference bots make. For me, a website (the public part) is a storefront. People walk down the street and see what's inside — that's the purpose. If something should not be available immediately, that's the private part of the store. I've been monitoring bot traffic on digital platforms for over 10 years. Sure, the crawler share is growing, some even with malicious intentions, and those I det…
For efficiently-hosted sites with little media it's not too bad. E.g. hosting a static site just doesn't cost much, even if you're hammered occasionally. That's extremely far from all sites though. It's probably safe to say it's a severe minority, particularly when you ignore personal / non-profit-bringing sites. Tons of small and large sites run stuff like poorly-written wordpress or ruby on rails or thousands of mi…
Even on a very bad day, the server's page load time doesn't go over 1s.
However, it seems like I'm indeed looking at the problem through the wrong prism, as what I've seen from the comments suggests that the initial issue is performance, and the bots are what uncover it.
Earlier quoted context omitted.
I thought you wanted users to flag Cloudflare domains through a browser extension, which is why I got confused.
I want to visibly indicate it on links / visits, for myself, so I can learn what the impact is and what might be worth avoiding. And how tightly bound use of it is - often these kinds of services are used/avoided by pockets that are somewhat closed (spam networks, indie webrings, etc), and I'd like to get a feel for where those pockets might be. I assume there's some way to detect that mechanically, which is probably…
Earlier quoted context omitted.
And assume you have User-agent: meta-externalagent Disallow: /
I have observed the same from Meta's crawler. User-agent: * Disallow: / on e.g. our preproduction site, Meta is the only big-tech crawler that accesses it, at least with an honest user agent. (Meta also accesses disallowed paths on the production site.)
User-agent: *
Earlier quoted context omitted.
For efficiently-hosted sites with little media it's not too bad. E.g. hosting a static site just doesn't cost much, even if you're hammered occasionally. That's extremely far from all sites though. It's probably safe to say it's a severe minority, particularly when you ignore personal / non-profit-bringing sites. Tons of small and large sites run stuff like poorly-written wordpress or ruby on rails or thousands of mi…
The website I mentioned has over 15k webpages and ~200 GB of media, and yet we monitor bots manually and only block them if they're pulling 5k requests in a row. Malicious URLs, multiply 404 are blocked by default. HEAD request rejected. Even on a very bad day, the server's page load time doesn't go over 1s. However, it seems like I'm indeed looking at the problem through the wrong prism, as what I've seen from the c…
Optimization is very very much not a solved problem though, just look at basically all software ever written - it's written for an optimization priority and to a price point (whether commercial $$ or via personal time), and that target's value to its users has shifted rather dramatically.
Earlier quoted context omitted.
The website I mentioned has over 15k webpages and ~200 GB of media, and yet we monitor bots manually and only block them if they're pulling 5k requests in a row. Malicious URLs, multiply 404 are blocked by default. HEAD request rejected. Even on a very bad day, the server's page load time doesn't go over 1s. However, it seems like I'm indeed looking at the problem through the wrong prism, as what I've seen from the c…
I think a good chunk of it is bot-induced performance problems, yea. Whether that's compute or transfer. And advertisement costs. Optimization is very very much not a solved problem though, just look at basically all software ever written - it's written for an optimization priority and to a price point (whether commercial $$ or via personal time), and that target's value to its users has shifted rather dramatically.
I'm working on an open-source tool that could be useful for bot detection, but I'm still not confident that anyone would deploy it on-prem and make the setup/maintenance instead of just routing traffic through the cloud.
Perhaps performance as a KPI could work. Thanks!
Earlier quoted context omitted.
I think a good chunk of it is bot-induced performance problems, yea. Whether that's compute or transfer. And advertisement costs. Optimization is very very much not a solved problem though, just look at basically all software ever written - it's written for an optimization priority and to a price point (whether commercial $$ or via personal time), and that target's value to its users has shifted rather dramatically.
This is really interesting. I indeed looked at this problem from the wrong perspective. I'm working on an open-source tool that could be useful for bot detection, but I'm still not confident that anyone would deploy it on-prem and make the setup/maintenance instead of just routing traffic through the cloud. Perhaps performance as a KPI could work. Thanks!
As to if it'd get enough interest: yea no idea at all. I wish you luck tho! Clearly there's a need for this kind of thing.
Earlier quoted context omitted.
Regulation could conceivably disallow a single company to control such a significant portion of internet traffic. The parent can be interpreted as lamenting the absence of such regulation.
The only thing more annoying than people chanting "regulation is bad" is people chanting "regulation is good". What regulation? Be specific. CloudFlare provides significant utility to me. I chose to use them. Explain why you think someone else needs to butt into this relationship.