Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

blog.cloudflare.com

Post reply on HN
Print view View original

521–530 of 799 posts

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#521

Post by remus »

alexey-salmin wrote: post #496
remus wrote: post #335

Earlier quoted context omitted.

> The no-crawl directives don't really make sense when I'm doing research and want my tool of choice to be able to pull from any relevant source. If you are the source I think they could make plenty of sense. As an example, I run a website where I've spent a lot of time documenting the history of a somewhat niche activity. Much of this information isn't available online anywhere else. As it happens I'm happy to let b…

> As it happens I'm happy to let bots crawl the site, but I think it's a reasonable stance to not want other companies to profit from my hard work. How do you square these two? Of course big companies profit from your work, this is why they send all these bots to crawl your site.

When I said "I think it's a reasonable stance" I meant as in "I think it's a reasonable stance for someone to take, though I don't personally hold that view".

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#522

Post by zzo38computer »

JimDabell wrote: post #26

Their test seems flawed: > We created multiple brand-new domains, similar to testexample.com and secretexample.com. These domains were newly purchased and had not yet been indexed by any search engine nor made publicly accessible in any discoverable way. We implemented a robots.txt file with directives to stop any respectful bots from accessing any part of a website: > We conducted an experiment by querying Perplexit…

Yes, it should stop recursive fetches. Furthermore, excessive unnecessary requests should also be stopped, although that is separate from robots.txt. At least, these are what I intended, and possibly also you.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#523

Post by eddythompson80 »

the_real_cher wrote: post #518
eddythompson80 wrote: post #467

Earlier quoted context omitted.

> http is neutral. Who misled you with that statement?

Http doesnt have emotions or thought last time I checked.

It seems that a 403 makes you sad though.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#525

Post by remus »

crazygringo wrote: post #478
remus wrote: post #335

Earlier quoted context omitted.

> The no-crawl directives don't really make sense when I'm doing research and want my tool of choice to be able to pull from any relevant source. If you are the source I think they could make plenty of sense. As an example, I run a website where I've spent a lot of time documenting the history of a somewhat niche activity. Much of this information isn't available online anywhere else. As it happens I'm happy to let b…

> but I think it's a reasonable stance to not want other companies to profit from my hard work Imagine someone at another company reads your site, and it informs a strategic decision they make at the company to make money around the niche activity you're talking about. And they make lots of money they wouldn't have otherwise. That's totally legal and totally ethical as well. The reality is, if you do hard work and ma…

For me, the point is that the person who has put in the work then has some rights to decide how that information is accessed and re-used. I think it is a reaosnable position for someone to hold that they want individuals to be able to freely use some content they produced, but not for a company to use and profit from that same content. I think just saying "It's public now" lacks any nuance.

Ultimately these AI tools are useful because they have access to huge swaths of content, and the owners of these tools turn a lot of revenue by selling access to these tools. Ultimately I think the internet will end up a much worse place if companies don't respect clearly established wishes of people creating the content, because if companies stop respecting things like robots.txt then people will just hide stuff behind logins, paywalls and frustraing tools like cloudflare which use heuristics to block malicious traffic.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#526

Post by dgshsg »

ghurtado wrote: post #510
dgshsg wrote: post #463

Earlier quoted context omitted.

When lying and cheating doesn't get you ahead, there is no reason to do it.

You seriously think that mankind wasn't lying and cheating long before inventing capitalism?

Sure, but the risk/reward ratio was different.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#527

Post by warkdarrior »

rzz3 wrote: post #145
larodi wrote: post #91

Good they do it. Facebook took TBs of data to train, nobody knows what Goog does to evade whatever they want. the service is actually very convenient no matter faang likes it or not.

Well Cloudflare doesn’t even block Google’s AI crawlers because they don’t differentiate themselves from their search crawlers. Cloudflare gives Google an unfair competitive advantage.

Google claims their AI crawlers have user agents distinct from the search crawlers: https://developers.google.com/search/docs/crawling-indexing/...

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#528

Post by pests »

michaelt wrote: post #453
pests wrote: post #299

Earlier quoted context omitted.

What’s wrong with it downloading documents when the user asks it to? My browser also downloads whole documents and sometimes even prefetches documents I haven’t even clicked on yet. Toss in a adblocker or reader mode and my browser also strips all the ads. Why is it okay for me to ask my browser to do this but I can’t ask my LLM to do the same?

When Google sends people to a review website, 30% of users might have an adblocker, but 70% don't. And even those with adblockers might click an affiliate link if they found the review particularly helpful. When ChatGPT reads a review website, though? Zero ad clicks, zero affiliate links.

So if enough people used adblockers that would make them bad too? It’s just an issue of numbers?

Brave blocks ads by default. Tools like Pocket and reader mode disables ads.

Why is it okay for some user agents but not others?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#529

Post by ch_fr »

bob1029 wrote: post #397

Has anyone bothered to properly quantify the worst case load (i.e., requests per second) that has been incurred by these scraping tools? I recall a post on HN a few weeks/months ago about something similar, but it seemed very light on figures. It seems to me that ~50% of the discourse occurring around AI providers involves the idea that a machine reading webpages on a regular schedule is tantamount to a DDOS attack.…

I mean, it did happen, don't you remember in March when SourceHut had outages because their most expensive endpoints were being spammed by scrapers?

Don't you remember the reason Anubis even came to be?

It really wasn't that long ago, so I find all of the snarky comments going "erm, actually, I've yet to see any good actors get harmed by scraping ever, we're just reclaiming power from today's modern ad-ridden hellscape" pretty dishonest.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#530

Post by k1m »

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

When Yahoo! Pipes was still running (long time ago), their official position was:

> Because Pipes is not a web crawler (the service only retrieves URLs when requested to by a Pipe author or user) Pipes does not follow the robots exclusion protocol, and won't check your robots.txt file.

Post reply on HN