Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

blog.cloudflare.com

Post reply on HN
Print view View original

311–320 of 799 posts

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#311

Post by benregenspan »

seydor wrote: post #216

> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…

> The bots of Big Tech, namely Google, Meta and Apple are of course exempt from this by pretty much every website and by cloudflare. But try being anyone other than them , no luck. Cloudflare is the biggest enabler of this monopolistic behavior

The Big Tech bots provide proven value to most sites. They have also through the years proven themselves to respect robots.txt, including crawl speed directives.

If you manage a site with millions of pages, and over the course of a couple years you see tens of new crawlers start to request at the same volume as Google, and some of them crawl at a rate high enough (and without any ramp-up period) to degrade services and wake up your on-call engineers, and you can't identify a benefit to you from the crawlers, what are you going to do? Are you going to pay a lot more to stop scaling down your cluster during off-peak traffic, or are you going to start blocking bots?

Cloudflare happens to be the largest provider of anti-DDoS and bot protection services, but if it wasn't them, it'd be someone else. I miss the open web, but I understand why site operators don't want to waste bandwidth and compute on high-volume bots that do not present a good value proposition to them.

Yes this does make it much harder for non-incumbents, and I don't know what to do about that.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#312

Post by [deleted] »

pests wrote: post #299
binarymax wrote: post #288

Earlier quoted context omitted.

Here's how perplexity works: 1) It takes your query, and given the complexity might expand it to several search queries using an LLM. ("rephrasing") 2) It runs queries against a web search index (I think it was using Bing or Brave at first, but they probably have their own by now), and uses an LLM to decide which are the best/most relevant documents. It starts writing a summary while it dives into sources (see next).…

What’s wrong with it downloading documents when the user asks it to? My browser also downloads whole documents and sometimes even prefetches documents I haven’t even clicked on yet. Toss in a adblocker or reader mode and my browser also strips all the ads. Why is it okay for me to ask my browser to do this but I can’t ask my LLM to do the same?

[deleted]

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#313

Post by armchairhacker »

Forgeties79 wrote: post #265
Taek wrote: post #246

Earlier quoted context omitted.

We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. Cloudflare only needs to exist because the server doesn't get paid when a user or bot requests resources. Advertising only needs to exist because the publisher doesn't get paid when a user or bot requests resources. And…

My first reaction: This solution would basically kill what little remaining fun there is to be had browsing the Internet and all but assure no new sites/smaller players will ever see traffic. Curious to hear other perspectives here. Maybe I’m over reacting/misunderstanding.

Depending on the implementation (a big if) it would help smaller websites, because it would make hosting much cheaper. ISPs don’t choose what sites users visit, only what they pay. As long as the ISP isn’t giving significant discounts to visiting big sites (just charging a fixed rate per bytes downloads and uploaded) and charging something reasonable, visiting a small site would be so cheap (a few cents at most, but more likely <1 cent) users won’t weigh cost at all.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#314

Post by crossroadsguy »

I was recently listening to Cloudflare CEO on the Hard Fork podcast. He seemed to be selling a way for content creators to stop AI companies from profiting off such leeching. But the way he laid the whole thing out, adding how they are best placed to do this because they are gatekeepers of X% of the Internet (I don't recall the exact percentage), had me more concerned than I was at the prospect of AI companies being the front of summarised or interpreted consumption.

He went on, upfront — I’d give him that, to explain how he is expecting a certain percentage of that income that will come from enforcing this on those AI companies and when the AI companies pay up to crawl.

Cloudflare already questions my humanity and then every once in a while blocks me with zero recourse. Now they are literally proposing more control and gatekeeping.

Where have we all come on the Internet? Are we openly going back to the wild west of bounty hunters and Pinkertons (in a way)?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#315

Post by skeledrew »

This is why Perplexity is my preferred deep search engine. The no-crawl directives don't really make sense when I'm doing research and want my tool of choice to be able to pull from any relevant source. If a site doesn't want particular users to access their content, put it behind a login. The only way I - and eventually many others - will see it in the first place anyway is when it pops up as a cited source in the LLM output, and there's an actual need to go to said source.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#316

Post by seydor »

wulfstan wrote: post #66
JimDabell wrote: post #26

Their test seems flawed: > We created multiple brand-new domains, similar to testexample.com and secretexample.com. These domains were newly purchased and had not yet been indexed by any search engine nor made publicly accessible in any discoverable way. We implemented a robots.txt file with directives to stop any respectful bots from accessing any part of a website: > We conducted an experiment by querying Perplexit…

Yeah I'm not so sure about that. If Perplexity are visiting that page on your behalf to give you some information and aren't doing anything else with it, and just throw away that data afterwards, then you may have a point. As a site owner, I feel it's still my decision what I do and don't let you do, because you're visiting a page that I own and serve. But if, as I suspect, Perplexity are visiting that page and then…

Perplexity can then just ask the user to copy/paste the page content. That should be legal , it's what the user wants. The cases are equivalent

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#317

Post by OptionOfT »

Taek wrote: post #246
seydor wrote: post #216

> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…

We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. Cloudflare only needs to exist because the server doesn't get paid when a user or bot requests resources. Advertising only needs to exist because the publisher doesn't get paid when a user or bot requests resources. And…

> We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago.

But it's done through a bait and switch. They serve the full article to Google, which allows Google to show you excerpts that you have to pay for.

It would be better if Google shows something like PAYMENT REQUIRED on top, at least that way I know what I'm getting at.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#318

Post by Analemma_ »

Forgeties79 wrote: post #265
Taek wrote: post #246

Earlier quoted context omitted.

We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. Cloudflare only needs to exist because the server doesn't get paid when a user or bot requests resources. Advertising only needs to exist because the publisher doesn't get paid when a user or bot requests resources. And…

My first reaction: This solution would basically kill what little remaining fun there is to be had browsing the Internet and all but assure no new sites/smaller players will ever see traffic. Curious to hear other perspectives here. Maybe I’m over reacting/misunderstanding.

If site operators can’t afford the costs of keeping sites up in the face of AI scraping, the new/smaller sites are gone anyway.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#319

Post by blibble »

gkbrk wrote: post #296
blibble wrote: post #161

Earlier quoted context omitted.

> "Stealth" crawlers are always going to win the game. no, because we'll end up with remote attestation needed to access any site of value

Almost no site of value will use remote attestation because an alternative that works will all of your devices, operating systems, ad blockers and extensions will attract more users than your locked-down site.

tell that to the massive content sites already using widevine

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#320

Post by Forgeties79 »

armchairhacker wrote: post #313
Forgeties79 wrote: post #265

Earlier quoted context omitted.

My first reaction: This solution would basically kill what little remaining fun there is to be had browsing the Internet and all but assure no new sites/smaller players will ever see traffic. Curious to hear other perspectives here. Maybe I’m over reacting/misunderstanding.

Depending on the implementation (a big if) it would help smaller websites, because it would make hosting much cheaper. ISPs don’t choose what sites users visit, only what they pay. As long as the ISP isn’t giving significant discounts to visiting big sites (just charging a fixed rate per bytes downloads and uploaded) and charging something reasonable, visiting a small site would be so cheap (a few cents at most, but…

But users depend on major sites like google [insert service] still and will prioritize their usage accordingly like limited minutes and texts back in the day, right?
Post reply on HN