Perplexity is using stealth, undeclared crawlers to evade no-crawl directives • Hacker News Board

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#471

Post by ndriscoll » Mon, Aug 04, 2025, 8:02 PM UTC

yojo wrote: post #223

shadowgovt wrote: post #119

Earlier quoted context omitted.

> Otherwise there is literally no reason for them to make any of it available on the open web This is the hypothesis I always personally find fascinating in light of the army of semi-anonymous Wikipedia volunteers continuously gathering and curating information without pay. If it became functionally impossible to upsell a little information for more paid information, I'm sure some people would stop creating informati…

Any information that requires something approximating a full-time job worth of effort to produce will necessarily go away, barring the small number of independently wealthy creators. Existing subject-matter experts who blog for fun may or may not stick around, depending on what part of it is “fun” for them. While some must derive satisfaction from increasing the total sum of human knowledge, others are probably blogg…

OpenStreetMap is basically Wikipedia for maps and is quite successful. Over 10M registered users and millions of edits per day. Lots of information is also shared online on forums for free. The hosting (e.g. reddit) is basically a commodity that benefits from network effects. The information is the more interesting bit, and people share it because they feel like it.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#472

Post by hnburnsy » Mon, Aug 04, 2025, 8:03 PM UTC

Respone from Perpelexity to Tech Crunch...

>Perplexity spokesperson Jesse Dwyer dismissed Cloudflare’s blog post as a “sales pitch,” adding in an email to TechCrunch that the screenshots in the post “show that no content was accessed.” In a follow-up email, Dwyer claimed the bot named in the Cloudflare blog “isn’t even ours.”

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#473

Post by Forgeties79 » Mon, Aug 04, 2025, 8:04 PM UTC

brookst wrote: post #449

armchairhacker wrote: post #346

Earlier quoted context omitted.

Networking is so cheap, unless ISPs drastically inflate their price, users won’t care. The average American allegedly* downloads 650-700GB/month, or >20GB/day. 10MB is more than enough for a webpage (honestly, 1MB is usually enough), so that means on average, ISPs serve over 2000 webpages worth of data per day. And the average internet plan is allegedly** $73/month, or That’s cheap enough, wrapped in a monthly bill,…

Wait, so the ISPs do from taking $73/user home today to taking $0/user home tomorrow under this plan?

Yeah same reaction here - there's no world in which ISP's would agree to this and even if they did I don't want to add them to my list of utilities I have to regularly fight with over claimed vs. actual usage like I do with my power/water/gas companies.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#474

Post by Ray20 » Mon, Aug 04, 2025, 8:04 PM UTC

dgshsg wrote: post #463

sublinear wrote: post #395

Earlier quoted context omitted.

Why bring up capitalism? I don't get it. What's stopping people from lying and cheating under any other system?

When lying and cheating doesn't get you ahead, there is no reason to do it.

The problem is that without capitalism ONLY lying and cheating will get you ahead. Look at ANY country that builds its economy on the restriction of people's economic freedom, on the absence of private property rights - these are the most deceitful and disgusting regimes in the world with zero level of public trust.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#475

Post by Forgeties79 » Mon, Aug 04, 2025, 8:05 PM UTC

Analemma_ wrote: post #318

Forgeties79 wrote: post #265

Earlier quoted context omitted.

My first reaction: This solution would basically kill what little remaining fun there is to be had browsing the Internet and all but assure no new sites/smaller players will ever see traffic. Curious to hear other perspectives here. Maybe I’m over reacting/misunderstanding.

If site operators can’t afford the costs of keeping sites up in the face of AI scraping, the new/smaller sites are gone anyway.

Maybe not but we are not realistically in an either/or scenario here.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#476

Post by helloplanets » Mon, Aug 04, 2025, 8:07 PM UTC

vineyardmike wrote: post #302

dabockster wrote: post #266

Earlier quoted context omitted.

> We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. I agree, but your idea below that is overly complicated. You can't micro-transact the whole internet. That idea feels like those episodes of Star Trek DS9 that take place on Feregenar - where you have to pay admission…

> You can't micro-transact the whole internet. I agree that end-users cannot handle micro transactions across the whole internet. That said, I would like to point out that most of the internet is blanketed in ads and ads involve tons of tiny quick auctions and micro transactions that occur on each page load. It is totally possible for a system to evolve involving tons of tiny transactions across page loads.

You could argue that the suggested system is actually much simpler than the one we currently have for the sites that are "free", aka funded with ads.

The lengths Meta and the like go to in order to maximize clickthroughs...

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#477

Post by snihalani » Mon, Aug 04, 2025, 8:09 PM UTC

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

you are paying for LLM but not paying for the website. LLM is removing the power the website had. Legally, that's cause for loss of income

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#478

Post by crazygringo » Mon, Aug 04, 2025, 8:10 PM UTC

remus wrote: post #335

skeledrew wrote: post #315

This is why Perplexity is my preferred deep search engine. The no-crawl directives don't really make sense when I'm doing research and want my tool of choice to be able to pull from any relevant source. If a site doesn't want particular users to access their content, put it behind a login. The only way I - and eventually many others - will see it in the first place anyway is when it pops up as a cited source in the L…

> The no-crawl directives don't really make sense when I'm doing research and want my tool of choice to be able to pull from any relevant source. If you are the source I think they could make plenty of sense. As an example, I run a website where I've spent a lot of time documenting the history of a somewhat niche activity. Much of this information isn't available online anywhere else. As it happens I'm happy to let b…

> but I think it's a reasonable stance to not want other companies to profit from my hard work

Imagine someone at another company reads your site, and it informs a strategic decision they make at the company to make money around the niche activity you're talking about. And they make lots of money they wouldn't have otherwise. That's totally legal and totally ethical as well.

The reality is, if you do hard work and make the results public, well you've made them public. People and corporations are free to profit off the facts you've made public, and they should be. There are certain limited copyright protections (they can't sell large swathes of your words verbatim), but that's all.

So the idea that you don't want companies to profit from your hard work is unreasonable, if you make it public. If you don't want that to happen, don't make anything public.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#479

Post by pixl97 » Mon, Aug 04, 2025, 8:14 PM UTC

bradleyjg wrote: post #131

rjbwork wrote: post #112

Earlier quoted context omitted.

But I can send my personal shopper and you'll be none the wiser.

It’s possible to violate all sorts of social norms. Societies that celebrate people that do so are on the far opposite end of the spectrum from high trust ones. They are rather unpleasant.

[flagged]

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#480

Post by crazygringo » Mon, Aug 04, 2025, 8:17 PM UTC

gcbirzan wrote: post #403

tempfile wrote: post #236

Earlier quoted context omitted.

Do you ask for consent before you visit a website? If I told you, you personally, to stop visiting my blog, would you stop?

I am not told I cannot access. And, yes, I would, because I'd be breaking the law otherwise.

> And, yes, I would, because I'd be breaking the law otherwise.

No you wouldn't be. Even if someone tells you not to visit your site, you have every legal right to continue visiting it, at least in the US.

Under common interpretation of the CFAA, there needs to be a formal mechanism of authorized access. E.g. you could be charged if you hacked into a password-protected area of someone's site. But if you're merely told "hey bro don't visit my site", that's not going to reach the required legal threshold.

Which is why crawlers aren't breaking the law. If you want to restrict authorization, you need to actually implement that as a mechanism by creating logins, restricting content to logged-in users, and not giving logins to crawlers.