Perplexity is using stealth, undeclared crawlers to evade no-crawl directives • Hacker News Board

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#711

Post by tomhow » Tue, Aug 05, 2025, 2:00 PM UTC

pixl97 wrote: post #479

bradleyjg wrote: post #131

Earlier quoted context omitted.

It’s possible to violate all sorts of social norms. Societies that celebrate people that do so are on the far opposite end of the spectrum from high trust ones. They are rather unpleasant.

[flagged]

> Oh, this is a bunch of baloney...

Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.

When disagreeing, please reply to the argument instead of calling names. "That is idiotic; 1 + 1 is 2, not 3" can be shortened to "1 + 1 is 2, not 3."

Please don't fulminate. Please don't sneer, including at the rest of the community.

Eschew flamebait. Avoid generic tangents. Omit internet tropes.

Please don't use Hacker News for political or ideological battle. It tramples curiosity.

https://news.ycombinator.com/newsguidelines.html

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#712

Post by tomhow » Tue, Aug 05, 2025, 2:02 PM UTC

p3rls wrote: post #579

pixl97 wrote: post #479

Earlier quoted context omitted.

[flagged]

[flagged]

You can't comment like this on Hacker News, no matter what you're replying to. It's not what this site is for, and destroys what it is for.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#713

Post by tomhow » Tue, Aug 05, 2025, 2:14 PM UTC

thoroughburro wrote: post #2

[flagged]

> I genuinely hope you feel shame. I would shun you in real life.

HN is not a platform for attacking people, even imagined ones.

Please don't fulminate.

https://news.ycombinator.com/newsguidelines.html

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#714

Post by [deleted] » Tue, Aug 05, 2025, 2:15 PM UTC

egypturnash wrote: post #596

jychang wrote: post #562

Earlier quoted context omitted.

Who cares what Hacker News wants? You’re not obliged to participate in discussion. Am I supposed to spend money on Amazon.com when I visit the website just because Amazon wants me to?

If most people quit spending money on Amazon then Amazon stops being worth running. If most people stop discussing things on HN, and the discussion is indeed one of the major reasons it’s kept running, then HN stops being worth running.

[deleted]

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#715

Post by crazygringo » Tue, Aug 05, 2025, 2:28 PM UTC

Wilder7977 wrote: post #658

crazygringo wrote: post #553

Earlier quoted context omitted.

I don't think it's bleak, just the opposite. If someone writes valuable stuff on a blog almost nobody finds, that's a tragedy. If LLM's can process the information and provide it to people in conversations where it will be most helpful, where they never would have found it otherwise, then that's amazing! If all you're trying to do is help people with the information you've discovered, why do you care if it's delivere…

Because attribution, social recognition and prestige are among the many reasons why people put the information out there, and there is nothing wrong with any of them. This is why I care if my ideas are presented to others by an LLM (that maybe cites me in some % of cases) or directly to a human. There is already a difference between a human visiting my space (acknowledging it as such) to read and learn information an…

If you want attribution and prestige, then publish your stuff in an actual publication -- a journal, a magazine, whatever. Go on podcasts, speak at conferences, and so forth.

Publishing on a personal blog is not the path.

LLM's aren't taking away from your "prestige" or recognition. Any more than a podcaster referencing an idea of yours without mentioning you is. Or anyone else in casual conversation.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#716

Post by danbruc » Tue, Aug 05, 2025, 2:39 PM UTC

shortformblog wrote: post #674

danbruc wrote: post #657

Earlier quoted context omitted.

But the entire reason that the web is so frustrating is that visitors don't want to pay for anything. They are already paying, it is the way they are paying that causes the mess. When you buy a product, some fraction of the price is the ad budget that gets then distributed to websites showing ads. Therefore there is also nothing wrong with blocking ads, they have already been paid for, whether you look at them or not…

Publishers don't get paid a dime if you block the ad unless they are doing a direct ad transaction. Adtech has largely made that transaction a rarity for like 30 years. It's not like newspapers where advertising is paid in full before publishers put stories online. It has not been that way for a long time. Your reasoning for not accessing advertising reminds me of that scene in Arrested Development where, to hide the…

Of course, they will not get paid for me visiting the website if I block the ads, but that was not my point. People have already bought stuff and with that paid for the ad budget. And that money will be spent somewhere. Maybe someone else will see the ad that I blocked, someone who would otherwise not have seen it because the ad budget would have been exhausted. Or maybe the prices for ads go up because there are less impressions to sell. Only if companies would lower their ad budgets in response to ad blocking would there be less money to distribute. If that would be the case, then my argument would fail.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#717

Post by Sam6late » Tue, Aug 05, 2025, 2:41 PM UTC

fxtentacle wrote: post #4

I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…

perplexity is being used to bypass paywalls, I noticed this when I pasted into it a text and it was captured as a hyperlinked text. I will try other websites with paywalls to see if it is a go.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#718

Post by dataflow » Tue, Aug 05, 2025, 2:46 PM UTC

rjbwork wrote: post #112

itsdesmond wrote: post #97

Earlier quoted context omitted.

Some stores do not welcome Instacart or Postmates shoppers. You can shop there. You can shop with your phone out, scanning every item to price match, something that some bookstores frown on, for example. Third party services cannot send employees to index their inventory, nor can they be dispatched to pick up an item you order online. Their reasons vary. Some don’t want their businesses perception of quality to be ta…

But I can send my personal shopper and you'll be none the wiser.

> But I can send my personal shopper and you'll be none the wiser.

They will be quite the wiser if they track/limit how often your shopper enters the store. You probably aren't entering the same store fifteen times every day and neither would be your shopper if they were only doing it on your behalf.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#719

Post by bayindirh » Tue, Aug 05, 2025, 2:47 PM UTC

accrual wrote: post #389

bayindirh wrote: post #218

Earlier quoted context omitted.

As a person who has a couple of sites out there, and witnesses AI crawlers coming and fetching pages from these sites, I have a question: What prevents these companies from keeping a copy of that particular page, which I specifically disallowed for bot scraping, and feed it to their next training cycle? Pinky promises? Ethics? Laws? Technical limitations? Leeroy Jenkins?

Thanks for sharing your experience. A little off-topic but I'd like to start hosting some personal content, guides/tutorials, etc. Do you still see authentic human traffic on your domains, is it easy to discern? I feel like I missed the bus on running a blog pre-AI.

I intentionally doesn't keep detailed analytics on my homepage server and my digital garden, because I respect my users and don't want to push unnecessary Javascript on them. The blog platform I use (Mataroa) keeps rudimentary analytics (essentially page hit counters, nothing more) on index, RSS and per post.

Both my blog homepage and posts see mostly human traffic. Sometimes bots crawl the site and they appear as spikes in the analytics.

Looks like my homepage which doesn't have anything but links is pretty popular with crawlers. My digital garden doesn't get much interest from them. All in all, human traffic on my sites are pretty much alive.

I don't believe in missing the bus in anything actually, because I don't write these for others, first. Both my blog (more meta) and digital garden (more technical) are written for myself primarily, and left open. I post links to both when it's appropriate, but they are not made to be popular. If people read it and learn something or solve one of their problems, that's enough for me.

This is why my software is GPLv3, Digital Garden is GFDL and blog is CC BY-NC-SA 2.0. This is why everything is running with absolutely minimum analytics and without any ads whatsoever.

Lastly, this is why I don't want AI crawlers in my site and my data in the models. This thing is made by a human for humans, absolutely for free. It's not OK somebody to sell something designed to be free and make money over it.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#720

Post by shortformblog » Tue, Aug 05, 2025, 2:47 PM UTC

danbruc wrote: post #716

shortformblog wrote: post #674

Earlier quoted context omitted.

Publishers don't get paid a dime if you block the ad unless they are doing a direct ad transaction. Adtech has largely made that transaction a rarity for like 30 years. It's not like newspapers where advertising is paid in full before publishers put stories online. It has not been that way for a long time. Your reasoning for not accessing advertising reminds me of that scene in Arrested Development where, to hide the…

Of course, they will not get paid for me visiting the website if I block the ads, but that was not my point. People have already bought stuff and with that paid for the ad budget. And that money will be spent somewhere. Maybe someone else will see the ad that I blocked, someone who would otherwise not have seen it because the ad budget would have been exhausted. Or maybe the prices for ads go up because there are les…

Your point is illogical. It’s like you’ve invented a theory as to how companies advertise that has zero tethering to reality.

It’s especially stupid because it doesn’t include publishers in the equation at all. It’s just you looping over yourself attempting to validate your choice for running an ad blocker.

Admit you’re doing it because you want to callously screw over publishers. You certainly haven’t put their thoughts into consideration here.

To be clear: Run an ad blocker if you want, but stop acting as if you bought those ads. The chicken dinner I ate the other night has no say how I live my life after our transaction has ended.