Live data from Hacker News

Detecting scraper bots through scroll behaviour

niki.cat

21–25 of 25 posts

Re: Detecting scraper bots through scroll behaviour

#21
post #12
post #7

I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse. If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation…

I've already seen a small amount of transition in that direction. For instance: Adafruit, last year: They blocked bots very heavy-handedly, including those that were there to gather information at the behest of a present-at-the-keyboard human. This made their products nearly invisible for those who were using ChatGPT instead of Google to find particular things to buy. Even when they were discoverable through forum po…

ssl-3 - phil from adafruit, pt@adafruit.com if email is easier, ladyada and i will both read it.

the blocking of automated attacks or purchase attempts wasn't aimed at you. when a scarce part is in stock here cheaper than everywhere else, scripts that some human decided to go after us with, show up and try to clear the store before a human person can click buy when it's in stock, first fix is a blunt instrument, then filing it down. your bot getting through is the filing working, good to know, most people only post when it's broken. most people are only complaining, most people. but you, us, and others, are not most people - we work here.

you bought the widget elsewhere, fair, that's what comparison shopping is for, is there is a personal axe to grind, could probably help with that too. we'd still like to know where the price, stock, or a spec buried three clicks deep. we can fix just about anything, you can help us.

if you've got a project that keeps stalling on tooling instead of the idea, send that too and ladyada and i will look at it as a problem to solve, and share the results, like we always do. if the answer is nothing adafruit makes, that's a useful answer. and next time the site snags your setup, tell us exactly where.

today is circuitpython day too, it's taken almost 10 years for being able to do anything i want in less than 5 minutes to get started, and then i can spend as much time on the best parts.

Re: Detecting scraper bots through scroll behaviour

#22
post #2

That's great! I was experimenting with Google Ads traffic to one of the websites I cared about, and even just looking at the existence of mouse move and scroll events, let alone the patterns, made me understand how many bots, or users who just (mis)clicked the ad and never did any action on my website, were counted as paid clicks in Google Ads. Analyzing for patterns like in this article would've made even more sense…

I alway wondered why so many people (not saying you're one of them) get mad at being charged for misclicks and bots. It all averages out. You bid $3 a click because your conversion ratio is x, and paying $3 is worth it. Even if 90% of the traffic is bots, it works out. If Google filtered out all the bots, the conversion ratio for you and your competitors would go up, and your competitors would be willing to pay $27 p…

For me, $3 a click would've worked for some projects, but $30 would not work for sure. Knowing that it's $30 and not $3 before the start of some campaigns, I would've saved some money. I instead paid that money to Google for this lesson.

Re: Detecting scraper bots through scroll behaviour

#23
post #12

Earlier quoted context omitted.

I've already seen a small amount of transition in that direction. For instance: Adafruit, last year: They blocked bots very heavy-handedly, including those that were there to gather information at the behest of a present-at-the-keyboard human. This made their products nearly invisible for those who were using ChatGPT instead of Google to find particular things to buy. Even when they were discoverable through forum po…

ssl-3 - phil from adafruit, pt@adafruit.com if email is easier, ladyada and i will both read it. the blocking of automated attacks or purchase attempts wasn't aimed at you. when a scarce part is in stock here cheaper than everywhere else, scripts that some human decided to go after us with, show up and try to clear the store before a human person can click buy when it's in stock, first fix is a blunt instrument, then…

Hi Phil. Well-met.

I remember the days of the Great Pi Shortage very well. You guys were trying very hard to sell Raspberry Pis at sensible prices to real people instead of to computer programs, which was/is a commendable effort.

The part I was trying to find last year was a USB canbus interface to do some automotive hacking with. That was a frustrating experience because trying to find such a widget with Google retrieves endless pages of impossible-to-exclude OBD-II interfaces and "CANBUS" replacement automotive LED lamps -- stuff that obviously wasn't applicable to my hacking project at all.

So I used the bot (ChatGPT) to help sort the good stuff that I might want to buy from the irrelevant stuff that I did not want to buy. It got stalled hard when trying to access adafruit.com, which did not help my process.

But, like I said: It's working now. Last week, when I set forth on a small project to characterize daylight for home automation, the same bot worked well with finding your pages on the AS7341 boards you list. No issues. It all flowed very well and Adafruit was listed prominently in the bot's output as a potential place to buy a board from. :)

Anyway, for full disclosure: I see that this board is out of stock right now and I don't remember if that was the case last week or not. I was nearly broke at that time (insert "in this economy" joke here), so I wound up looking on Aliexpress so that maybe I could move the project forward and also buy a cheeseburger. A seller there had a little board which is supposed to have an AS7341 and appears to have the right interface glue brought out to sensible pins, for like $2.50 -- delivered to my door in the US.

You guys can't compete with that. There's no way. Someone lost money on that transaction.

But! The bot wasn't able to find this on Aliexpress. It got stuck, and didn't recommend that as a potential source at all.

So score one for the team: Your stuff is showing up in the bot's results, while your (ruthless) competition is not! That exposes more people to the things that you guys work so hard to accomplish, and more exposure tends to lead to more sales, which tends to help keep the lights on. :)

It's good stuff. Right now, you're ahead of the game.

And thanks. I will shoot you an email directly if I find an aspect that's broken or buried in the future.

Re: Detecting scraper bots through scroll behaviour

#25
post #7

I'm curious whether we'll hit a point where some sites will want to be as friendly to bots as possible (hopefully they'll just shift to having sane, rate-limited public APIs so everyone can save time and bandwidth), while the rest want the reverse. If you're providing a service, and AI agents truly take off to where people say "Claude, go buy me a new shower hose about 5' long, brushed nickel. Use my usual evaluation…

the best economic use-case for the semantic web (text/turtle, text/n3); businesses providing manifests of their stocks and services plus, AI would also skip search result ads, and, if those have to legally be displayed as such, then there goes the market for them, I'd think

Ngl - Can’t wait for advertising to just fully die if you’re right.

If I need to pay a few bucks here and there for content, or email, or whatever that’s FINE. Just end the ad insanity and its associated pathologies like the “engagement optimizer” algos that are eating out fucking society!

Post reply on HN