[flagged]
Detecting scraper bots through scroll behaviour
11–20 of 25 posts
Re: Detecting scraper bots through scroll behaviour
#12I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse. If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation…
I've already seen a small amount of transition in that direction. For instance:
Adafruit, last year: They blocked bots very heavy-handedly, including those that were there to gather information at the behest of a present-at-the-keyboard human. This made their products nearly invisible for those who were using ChatGPT instead of Google to find particular things to buy. Even when they were discoverable through forum postings, this added friction meant that I was unable to make a swift comparison betwixt their wares and those of others, and I wasn't buying anything from them because they were making deliberate choices that I felt were hostile to my process.
But that's changed.
Adafruit, at least a week or so ago: The bot was able to get in and find details about the stuff they sell. This lack of friction meant that I was able to make a swift comparison between their wares and those of others. I still bought the particular sensor-chip-on-a-board widget that I wanted elsewhere, but at least Adafruit's offerings were in the running this time.
Re: Detecting scraper bots through scroll behaviour
#13[dead]
Re: Detecting scraper bots through scroll behaviour
#14I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse. If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation…
Yeah, as soon as site operators implement that paid traffic scheme the Cloudflare has been pushing. It's all about being to monetize their compute/content.
Re: Detecting scraper bots through scroll behaviour
#15I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse. If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation…
Very few "user facing" applications will have to deal with real users any longer. From an author's point of view let's admit that it will make things easier.
Re: Detecting scraper bots through scroll behaviour
#16That's great! I was experimenting with Google Ads traffic to one of the websites I cared about, and even just looking at the existence of mouse move and scroll events, let alone the patterns, made me understand how many bots, or users who just (mis)clicked the ad and never did any action on my website, were counted as paid clicks in Google Ads. Analyzing for patterns like in this article would've made even more sense…
I alway wondered why so many people (not saying you're one of them) get mad at being charged for misclicks and bots. It all averages out. You bid $3 a click because your conversion ratio is x, and paying $3 is worth it. Even if 90% of the traffic is bots, it works out. If Google filtered out all the bots, the conversion ratio for you and your competitors would go up, and your competitors would be willing to pay $27 p…
The problem is when your competitors get the idea to spam your campaign with fake traffic, which is a real thing that happens.
Re: Detecting scraper bots through scroll behaviour
#17I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse. If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation…
My cgit instance is incredibly friendly to bots; I go out of my way to support git's fast "smart" HTTP protocol, and the clone URL is linked on every project's root page. I even included the full clone command on 429 error pages.
The end result was that I got millions of requests daily trying to download every HTML page (every variation of action, commit hash, branch, whatever else cgit allows) and basically no traffic to the proper clone route.
Turns out the bots just don't care about your bot-friendly access method and will instead throw more illicitly-proxied residential IPs at the problem.
Re: Detecting scraper bots through scroll behaviour
#18This is another one of those horse-blinders style arguments that considers only for-profit/institutional/etc contexts and so doesn't realize their very method of measuring something through javascript program execution is biased or that it's creepy. I'm a human, I don't execute javascript programs and by default my browser does not either. Anyone using this will label me a bot.
Hello! Author of the article here. I am working on this because I had one of my websites (which served downloads related to the game Minecraft) flooded with hundreds of thousands of requests coming from scraper bots from all around the world that even tried to download the files.
This made me hit the bandwith cap of my hosting provider and had to pay extra per TB used that month from my own pocket, since the website was not making a profit at all.
I never thought of increasing profits for any institution, but to stop the waste of bandwith many independent websites suffer.
Re: Detecting scraper bots through scroll behaviour
#19This is another one of those horse-blinders style arguments that considers only for-profit/institutional/etc contexts and so doesn't realize their very method of measuring something through javascript program execution is biased or that it's creepy. I'm a human, I don't execute javascript programs and by default my browser does not either. Anyone using this will label me a bot.
if by "bot" we mean "not economically interesting", aren't websites within their right to filter our hacker-nerds who probably won't click on ads and buy useless objects?
You're absolutely right. The modern meaning of "bot" is "not economically exploitable". I wish they'd just say that rather than using the plausible deniability for what they doing re: "bots".
Re: Detecting scraper bots through scroll behaviour
#20I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse. If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation…
the best economic use-case for the semantic web (text/turtle, text/n3); businesses providing manifests of their stocks and services
plus, AI would also skip search result ads, and, if those have to legally be displayed as such, then there goes the market for them, I'd think