Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

blog.cloudflare.com

Post reply on HN
Print view View original

721–730 of 799 posts

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#721

Post by kldg »

71bw wrote: post #620
kldg wrote: post #566

Earlier quoted context omitted.

if you do analytics, it is not so hard, but then you need to store user data (if not directly, then worse, with a third party), which should be viewed as a liability. I see ~2/3 human traffic, ~1/3 bot traffic (I just parse user agent strings and count whitelisted browsers as human), but my main landing page is all dynamic-populated webgl. I just asked Gemini what it sees on website, and it states "The page appears t…

> I made a stateful Internet implementation in Python earlier for proof-of-concept Is there a repo or some other form of public access? I'd like to see this.

it's not in a shareable state; is unsafe as-is. can share general idea and sample "webpage" files, though.

the server ("lodge") passes JSON to the client from what are called .branch files. the client receives JSON, parses it, then builds the UI and state representation from the JSON, then stored in that client's memory (self.current_doc and self.page_state in python client).

branches can invoke waterwheel (.ww) files hosted on the lodge. waterwheel files on the lodge contain scripts which define how patches (as JSON) are to be sent to the client. the client updates its state based on the JSON patch it receives. sample .branch and .ww from python implementation (in pastebin so to not make everyone have to scroll through this): https://pastebin.com/A0DEZDmR

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#722

Post by bayindirh »

account42 wrote: post #689
bayindirh wrote: post #285

Earlier quoted context omitted.

Many site operators want people to access their content, but prevent AI companies from scraping their sites for training data. People who think like that made tools like Anubis, and it works. I also want to keep this distinction on the sites I own. I also use licenses to signal that this site is not good to use for AI training, because it's CC BY-NC-SA-2.0. So, I license my content appropriately (No derivative, Non-c…

> Many site operators want people to access their content, but prevent AI companies from scraping their sites for training data. That is unfortunately not a distinction that is currently legally enforceable. Until that changes all other "solutions" are pointless and only cause more harm. > People who think like that made tools like Anubis, and it works. It works to get real humans like myself to stop visiting your si…

> It works to get real humans like myself to stop visiting your site

If we talk about Anubis, it's pretty invisible. You wait a couple of seconds in the first visit, and don't get challenged for a couple of weeks, at least. With more tuning some of the sites using Anubis work perfectly well without ever seeing Anubis' wall while stopping AI crawlers.

> And to be clear, what you are advocating for is DRM.

Yes. It's pretty ironic that someone like me who believes in open access prefers a DRM solution to keep companies abusing the small fish, but life is an interesting phenomenon, and these things happen.

> Until that changes all other "solutions" are pointless and only cause more harm.

As an addendum to above paragraph, I'm not happy that I have to insert draconian measures between the user and the information I want to share, but I need a way to signal that I'm not having their ways to these faceless things. What do you propose? Taking my sites offline? Burning myself in front of one of the HQs?

> If AI crawlers cared about that we wouldn't be talking about this issue. A license and only give more permissions than there are without one.

AI crawlers default to "Public Domain" when they find no licenses. Some of my lamest source code repositories made into "The Stack" because I forgot to add COPYING.md. A fork of a GPLv2 tool I wrote some patches also got into "The Stack", because COPYING.md was not in the root folder of the repository. I'd rather add licenses (which I can accept) to things rather than leave them as-is, because AI companies also eagerly grab things without license.

All licenses I use mandate attribution and continuation of license, at least, and my blog doesn't allow any derivations of from what I have written. So you can't ingest it into a model to be derived and remixed with something else.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#723

Post by lonelyasacloud »

In many ways what is going on with Perplexity is reminiscent of the earlier 2000s battles between the p2p music sharing services like Napster and the music industry. Then we had wildly popular services (p2p) where most of the content was being provided illegally without payment to the IP owners.

Which makes it particularly interesting now that Apple is being linked with Perplexity. Because in large part p2p music services were effectively consigned to history by Apple (primarily) negotiating with the music industry so that it could provide easy, seamless purchase and playback of legal music for their shiny new (at the time) mass-market Apple iPod devices: it then turning out that most users are happy to pay for content if it is not too expensive and is very convenient.

Given Apple’s existing relationships with publishers through its music, movies, books, and news services, it’s not hard to imagine them attempting a similar play now.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#724

Post by danbruc »

shortformblog wrote: post #720
danbruc wrote: post #716

Earlier quoted context omitted.

Of course, they will not get paid for me visiting the website if I block the ads, but that was not my point. People have already bought stuff and with that paid for the ad budget. And that money will be spent somewhere. Maybe someone else will see the ad that I blocked, someone who would otherwise not have seen it because the ad budget would have been exhausted. Or maybe the prices for ads go up because there are les…

Your point is illogical. It’s like you’ve invented a theory as to how companies advertise that has zero tethering to reality. It’s especially stupid because it doesn’t include publishers in the equation at all. It’s just you looping over yourself attempting to validate your choice for running an ad blocker. Admit you’re doing it because you want to callously screw over publishers. You certainly haven’t put their thou…

If I buy an iPhone, does some fraction of the price contribute to Apple's ad budget? If so, where does that money end up? What would change if I did not block Apple ads?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#725

Post by account42 »

bayindirh wrote: post #722
account42 wrote: post #689

Earlier quoted context omitted.

> Many site operators want people to access their content, but prevent AI companies from scraping their sites for training data. That is unfortunately not a distinction that is currently legally enforceable. Until that changes all other "solutions" are pointless and only cause more harm. > People who think like that made tools like Anubis, and it works. It works to get real humans like myself to stop visiting your si…

> It works to get real humans like myself to stop visiting your site If we talk about Anubis, it's pretty invisible. You wait a couple of seconds in the first visit, and don't get challenged for a couple of weeks, at least. With more tuning some of the sites using Anubis work perfectly well without ever seeing Anubis' wall while stopping AI crawlers. > And to be clear, what you are advocating for is DRM. Yes. It's pr…

> If we talk about Anubis, it's pretty invisible. You wait a couple of seconds in the first visit, and don't get challenged for a couple of weeks, at least. With more tuning some of the sites using Anubis work perfectly well without ever seeing Anubis' wall while stopping AI crawlers.

It's not invisible, the sites using it don't work perfectly well for all users and it doesn't stop AI crawlers.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#726

Post by bayindirh »

tempfile wrote: post #493
bayindirh wrote: post #285

Earlier quoted context omitted.

Many site operators want people to access their content, but prevent AI companies from scraping their sites for training data. People who think like that made tools like Anubis, and it works. I also want to keep this distinction on the sites I own. I also use licenses to signal that this site is not good to use for AI training, because it's CC BY-NC-SA-2.0. So, I license my content appropriately (No derivative, Non-c…

Of course some people want that. And at the moment they can prevent it. But those methods may stop working. Will it then be alright to do it? Of course not, so why bother mentioning that they are able to prevent it now - just give a justification. Your license is probably not relevant. I can go to the cinema and watch a movie, then come on this website and describe the whole plot. That isn't copyright infringement. E…

> I can go to the cinema and watch a movie, then come on this website and describe the whole plot. That isn't copyright infringement.

This is a false analogy. A correct one would be going to a 1000 movies and creating the 1001th movie with scenes cropped from these 1000 movies and assemble it as a new movie, and this is copyright infringement. I don't think any of the studios would applaud and support you for your creativity.

> But it is wrong-headed (and actually wrong in fact) to try to say someone may or may not use my content for some purpose because it hurts my feelings or it messes with my ad revenue.

Why does it have to be always about money? Personally it's not. I just don't want my work to be abused and sold to people to benefit a third party without my consent and will (and all my work is licensed appropriately for that).

> We should not have laws that say, yes you can read my site but no you can't use it to train an LLM, or to build a search index.

This goes both ways. If big corporations can scrape my material without asking me and resell it as an output of a model, I can equally distill their models further and sell it as my own. If companies can scrape my pages to sell my content as theirs, I can scrape theirs and unpaywall them.

But that will be copyright infringement, just because they have more money. What angers me is "all is fair game because you're a small fish, and this is a capitalist marketplace" mentality.

If companies can paywall their content to humans that don't pay, I can paywall AI companies and demand money or push them out of my lawn, just because I feel like that. The inverse is very unethical, but very capitalist, yes.

It's not always about money.

P.S.: Oh, try to claim that you can train a model with medical data without any clearance because it'd be unethical to have laws limiting this. It'll be fun. Believe me.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#727

Post by shortformblog »

danbruc wrote: post #724
shortformblog wrote: post #720

Earlier quoted context omitted.

Your point is illogical. It’s like you’ve invented a theory as to how companies advertise that has zero tethering to reality. It’s especially stupid because it doesn’t include publishers in the equation at all. It’s just you looping over yourself attempting to validate your choice for running an ad blocker. Admit you’re doing it because you want to callously screw over publishers. You certainly haven’t put their thou…

If I buy an iPhone, does some fraction of the price contribute to Apple's ad budget? If so, where does that money end up? What would change if I did not block Apple ads?

It’s up to them how they spend their money, not you. You can complain if they somehow damaged your product, they got your money unfairly, or were somehow doing something bad with your data, but at some point it is their money to spend how they see fit. They earned it, and they might spend it on advertising.

If I buy stuff at a grocery store, I can’t get a random bagger fired just because I feel like it. At some point the transaction ends and they ultimately continue to operate with or without your input.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#728

Post by bayindirh »

account42 wrote: post #725
bayindirh wrote: post #722

Earlier quoted context omitted.

> It works to get real humans like myself to stop visiting your site If we talk about Anubis, it's pretty invisible. You wait a couple of seconds in the first visit, and don't get challenged for a couple of weeks, at least. With more tuning some of the sites using Anubis work perfectly well without ever seeing Anubis' wall while stopping AI crawlers. > And to be clear, what you are advocating for is DRM. Yes. It's pr…

> If we talk about Anubis, it's pretty invisible. You wait a couple of seconds in the first visit, and don't get challenged for a couple of weeks, at least. With more tuning some of the sites using Anubis work perfectly well without ever seeing Anubis' wall while stopping AI crawlers. It's not invisible, the sites using it don't work perfectly well for all users and it doesn't stop AI crawlers.

I haven't seen any problems with any Anubis enabled site I encountered. Can you give examples? This is interesting.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#729

Post by theptip »

blibble wrote: post #161
bob1029 wrote: post #142

"Stealth" crawlers are always going to win the game. There are ways to build scrapers using browser automation tools [0,1] that makes detection virtually impossible. You can still captcha, but the person building the automation tools can add human-in-the-loop workflows to process these during normal business hours (i.e., when a call center is staffed). I've seen some raster-level scraping techniques used in game dev…

> "Stealth" crawlers are always going to win the game. no, because we'll end up with remote attestation needed to access any site of value

But for the case of Perplexity-User, presumably the user is in the loop to provide their attestation.

This case (“go research this subject for me”) is the grey area here. It’s not the same as simple scraping or search indexing, it’s a new activity that is similar in some ways.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#730

Post by Wilder7977 »

crazygringo wrote: post #715
Wilder7977 wrote: post #658

Earlier quoted context omitted.

Because attribution, social recognition and prestige are among the many reasons why people put the information out there, and there is nothing wrong with any of them. This is why I care if my ideas are presented to others by an LLM (that maybe cites me in some % of cases) or directly to a human. There is already a difference between a human visiting my space (acknowledging it as such) to read and learn information an…

If you want attribution and prestige, then publish your stuff in an actual publication -- a journal, a magazine, whatever. Go on podcasts, speak at conferences, and so forth. Publishing on a personal blog is not the path. LLM's aren't taking away from your "prestige" or recognition. Any more than a podcaster referencing an idea of yours without mentioning you is. Or anyone else in casual conversation.

Are there journals who discuss about personal life and perspectives? Or a big publication about clever homelab configuration? Or the millions of other topics people discuss and publish? Publishing a website is a perfectly fine way to put your ideas out there and expecting to be acknowledged by those who read those ideas.

And yes, a podcaster talking about someone's idea without referencing it is an unethical behavior.

What a bleak view of the world.

Post reply on HN