A year of fighting scrapers on my 1.5 million-page website

Post by hk__2 » Sat, Aug 08, 2026, 7:48 AM UTC

jt2190 wrote: post #222

Terretta wrote: post #150

Earlier quoted context omitted.

you know what a browser is called by the web site? check the header that tells the version. USER AGENT not user, an agent on behalf of the user. the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT. for a glimpse at one possible…

Yeah it's sort of weird that people are so used to using their computer to interact with other computers across the internet that the whole thing has become invisible to them: They completely forget that they're using a computer at all. Along come "bots" and suddenly everyone remembers that the whole thing involves computers. Driving the computer by hand for hours was never efficient, but is marginally entertaining I…

Post by [deleted] » Sat, Aug 08, 2026, 7:58 AM UTC

drknownuffin wrote: post #401

vector_spaces wrote: post #372

Earlier quoted context omitted.

You are assuming that because you are giving the operator money that you are entitled to use their website however you please -- that's not how it works. If you enter a brick & mortar establishment and break their rules, even as a paying customer, you might risk e.g. being kicked out and banned, depending on the behavior. This is not unique to e-commerce To be fair, I do not generally support wholesale banning of scr…

I think he's saying that his use of the site, via the LLM, is consistent with what the site owner-operator likely desires. To give a concrete example: I am a physician and have a site for my private clinic. I am not only unopposed to being scraped by bots/LLMs, I prefer it. The only value I gain from users being on my site is referral to my practice, and that same information being surfaced by search engines and LLMs…

Post by ccgreg » Sat, Aug 08, 2026, 8:05 AM UTC

knuckleheads wrote: post #93

Previously, I had done a fair amount of research into how Google's monopoly on web crawling further entrenches their monopoly in the search engine market. You can read more about this here, https://knuckleheads.club , there is a long report from ~2020 or so that explains how it worked at the time. The club is mothballed, I am doing other things with my life, and I'm happy to say that we played a very small role in th…

Post by rozumem » Sat, Aug 08, 2026, 8:30 AM UTC

Post by psd1 » Sat, Aug 08, 2026, 9:33 AM UTC

hluska wrote: post #206

matsemann wrote: post #171

Earlier quoted context omitted.

I don't disagree, but there is a sliding scale here. For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an e…

I don’t see the sliding scale - if the website didn’t want a bot they could have blocked you. They clearly don’t mind so what’s the issue? You still used a bot.