Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

blog.cloudflare.com

Post reply on HN
Print view View original

291–300 of 799 posts

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#291

Post by seydor »

TZubiri wrote: post #264
seydor wrote: post #216

> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…

Websites and any business really, have the right to impose terms of use and deny service. Anyone circumventing bans is doing something shitty and ilegal, see the computer fraud and abuse act and craiglist v 3taps. "And those LLMs didn't ask anyones permission to crawl the entire 'net." False, openai respects robots.txt, doesnt mask ips, paid a bunch of $ to reddit. You either side with the law or with criminals.

Is that also how e.g. antrhopic trained on libgen?

You can't even say the same thing about openAI because we don't know the corpus they train their models on.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#292

Post by dabockster »

sublinear wrote: post #178
Polizeiposaune wrote: post #160

Earlier quoted context omitted.

To stretch the analogy to the breaking point: If you send 10,000 personal shoppers all at once to the same store just to check prices, the store's going to be rightfully annoyed that they aren't making sales because legit buyers can't get in.

Too bad. Build a bigger store or publish this information so we don't need 10,000 personal shoppers. Was this not the whole point of having a website? Who distorted that simple idea into the garbage websites we have now?

> Who distorted that simple idea into the garbage websites we have now?

Corporate America. Where clean code goes to die.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#293

Post by otterley »

rzz3 wrote: post #152
curiousgal wrote: post #129

I am sorry, Cloudafre is the internet police now?

They hate AI it seems. I don’t see them offering any AI products or embracing it in any way. Seems like they’ll get left behind in the AI race.

I don't think they hate AI. I think they're offering a service that their customers want.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#294

Post by jgrall »

caesil wrote: post #181

Cloudflare is an enemy of the open and freely accessible web.

If by "open and freely accessible" you mean there should be no rules of the road, then I suppose yes. Personally, I'm glad CF is pushing back on this naive mentality.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#295

Post by nostrademons »

It's entirely possible that it's not Perplexity using the stealth undeclared crawlers, but rather their fallback is to contract out to a dedicated for-pay webscraping firm that retrieves the desired content through unspecified means. (Some of these are pretty dodgy - several scraping companies effectively just install malware on consumer machines and then use their botnet to grab data for their customers.). There was a story on HN not long ago about the FBI using similar means to perform surveillance that would be illegal if the FBI did it itself, but becomes legal once they split the different parts up across a supply chain:

https://news.ycombinator.com/item?id=44220860

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#296

Post by gkbrk »

blibble wrote: post #161
bob1029 wrote: post #142

"Stealth" crawlers are always going to win the game. There are ways to build scrapers using browser automation tools [0,1] that makes detection virtually impossible. You can still captcha, but the person building the automation tools can add human-in-the-loop workflows to process these during normal business hours (i.e., when a call center is staffed). I've seen some raster-level scraping techniques used in game dev…

> "Stealth" crawlers are always going to win the game. no, because we'll end up with remote attestation needed to access any site of value

Almost no site of value will use remote attestation because an alternative that works will all of your devices, operating systems, ad blockers and extensions will attract more users than your locked-down site.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#298

Post by gruez »

542354234235 wrote: post #137
gruez wrote: post #38

Earlier quoted context omitted.

>I do not really get why user-agent blocking measures are despised for browsers but celebrated for agents? AI broke the brains of many people. The internet isn't a monolith, but prior to the AI boom you'd be hard pressed to find people who were pro-copyright (except maybe a few who wanted to use it to force companies to comply with copyleft obligations), pro user-agent restrictions, or anti-scraping. Now such positio…

There is an expression “the dose makes the poison”. With any sufficiently complex or broad category situation, there is rarely a binary ideological position that covers any and all situations. Should drugs be legal for recreation? Well my feeling for marijuana and fentanyl are different. Should individuals be allowed to own weapons? My views differ depending on if it a switch blade knife of a Stinger missile. Can law…

>There is an expression “the dose makes the poison”. With any sufficiently complex or broad category situation, there is rarely a binary ideological position that covers any and all situations. Should drugs be legal for recreation? Well my feeling for marijuana and fentanyl are different. Should individuals be allowed to own weapons? My views differ depending on if it a switch blade knife of a Stinger missile. Can law enforcement surveille possible criminals? My views differ based on whether it is a warranted wiretap or an IMSI catcher used on a group of protestors.

This seems very susceptible to manipulation to get whatever conclusion you want. For instance, is dose defined? It sounds like the idea you're going for is that the typical pirate downloads a few dozen movies/games but AI companies are doing millions/billions, but why should it be counted per infringer? After all, if everyone pirates a given movie, that wouldn't add up much in terms of their personal count of infringements, but would make the movie unprofitable.

>People’s position just might be that the law should be used to protect the rights of parties when there is a large power asymmetry.

That sounds suspiciously close to "laws should just be whatever benefits me or my group". If so, that would be a sad and cynical worldview, not dissimilar to the stance on free speech held by the illiberal left and right. "Free speech is an important part of democracy", they say, except when they see their opponents voicing "dangerous ideas", in which case they think it should be clamped down. After all, what are laws for if not a tool to protect the interests of your side?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#299

Post by pests »

binarymax wrote: post #288
seydor wrote: post #216

> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…

Here's how perplexity works: 1) It takes your query, and given the complexity might expand it to several search queries using an LLM. ("rephrasing") 2) It runs queries against a web search index (I think it was using Bing or Brave at first, but they probably have their own by now), and uses an LLM to decide which are the best/most relevant documents. It starts writing a summary while it dives into sources (see next).…

What’s wrong with it downloading documents when the user asks it to? My browser also downloads whole documents and sometimes even prefetches documents I haven’t even clicked on yet. Toss in a adblocker or reader mode and my browser also strips all the ads.

Why is it okay for me to ask my browser to do this but I can’t ask my LLM to do the same?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#300

Post by novok »

Taek wrote: post #246
seydor wrote: post #216

> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…

We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. Cloudflare only needs to exist because the server doesn't get paid when a user or bot requests resources. Advertising only needs to exist because the publisher doesn't get paid when a user or bot requests resources. And…

The reason why that didn’t work was because regulations made micropayments too expensive, and the government wants it that way to keep control over the financial system.
Post reply on HN