PS: perplexity is using cloudflare browser rendering to scrape websites
Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
331–340 of 799 posts
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#332Crawling and scraping is legal. If your web server serves the content without authentication, it's legal to receive it, even if it's an automated process. If you want to gatekeep your content, use authentication. Robots.txt is not a technical solution, it's a social nicety. Cloudflare and their ilk represent an abuse of internet protocols and mechanism of centralized control. On the technical side, we could use CRC m…
Well if it continues like this, that's what will happen. And I dread that future.
Noone will care to share anything for free anymore, because it's AI companies profiting off their hard work. And no way to prevent that from happening, because these crawlers don't identify themselves.
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#333Earlier quoted context omitted.
>People never agreed DOSing a site to take copyright material was acceptable. Many people did not have a problem with taking copyright material in a respectful way that didn't kill the resource. Has it be shown that perplexity engages in "DOSing"? I've heard of anecdotes of AI bots gone amuck, and maybe that's what's happening here, but cloudflare hasn't really shown that. All they did was set up a robots.txt and sho…
The guy that runs shadertoy talked about how the hostingcost for his free site shot up because Openai kept crawling his site for training data (ignoring robot.txt) I think that’s bad, and I have also experimented a bit with using BeautifulSoup in the past to download ~2MB of pictures from Instagram. Do you think I’m holding an inconsistent position?
My point is that to invoke the "they're DOSing" excuse, you actually have to provide evidence it's happening in this specific instance, rather than vaguely gesturing at some class of entities (AI companies) and concluding that because some AI companies are DOSing, all AI companies are DOSing. Otherwise it's like youtube blocking all youtube-dl users for "DOSing" (some fraction of users arguably are), and then justifying their actions with "People never agreed DOSing a site to take copyright material was acceptable".
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#334>We conducted an experiment by querying Perplexity AI with questions about these domains, and discovered Perplexity was still providing detailed information regarding the exact content hosted on each of these restricted domains Thats... less conclusive than I'd like to see, especially for a content marketing article that's calling out a company in particular. Specifically it's unclear on whether Perplexity was crawli…
The examples the article cites seem to me that they are merely retrieving content on behalf of the user. I do not see a problem with this.
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#335This is why Perplexity is my preferred deep search engine. The no-crawl directives don't really make sense when I'm doing research and want my tool of choice to be able to pull from any relevant source. If a site doesn't want particular users to access their content, put it behind a login. The only way I - and eventually many others - will see it in the first place anyway is when it pops up as a cited source in the L…
> The no-crawl directives don't really make sense when I'm doing research and want my tool of choice to be able to pull from any relevant source.
If you are the source I think they could make plenty of sense. As an example, I run a website where I've spent a lot of time documenting the history of a somewhat niche activity. Much of this information isn't available online anywhere else.
As it happens I'm happy to let bots crawl the site, but I think it's a reasonable stance to not want other companies to profit from my hard work. Even more so when it actually costs me money to serve requests to the company!
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#336> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…
We're moving progressively in the direction of "pages can't be served for free anymore". Which, I don't think is a problem, and in fact I think it's something we should have addressed a long time ago. Cloudflare only needs to exist because the server doesn't get paid when a user or bot requests resources. Advertising only needs to exist because the publisher doesn't get paid when a user or bot requests resources. And…
402 Payment Required
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/...
Sadly development along these lines has not progressed. Yes, Google Cloud and other services may return it and require some manual human intervention, but I'd love to see _automatic payment negotiation_.
I'm hopeful that instant-settlement options like Bitcoin Lightning payments could progress us past this.
https://docs.lightning.engineering/the-lightning-network/l40...
https://hackernoon.com/the-resurgence-of-http-402-in-the-age...
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#337Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#338> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…
> The bots of Big Tech, namely Google, Meta and Apple are of course exempt from this by pretty much every website and by cloudflare. But try being anyone other than them , no luck. Cloudflare is the biggest enabler of this monopolistic behavior Plenty of site/service owners explicitly want Google, Meta and Apple bots (because they believe they have a symbiotic relationship with it) and don't want your bot because the…
they didnt seem to mind when openai et al. took all their content to train LLMs when they were still parasites that didn't have a symbiotic relationship. This thinking is kind of too pro-monopolist for me
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#339Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#340> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…
> The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall
I don't think it's fair to blame Cloudflare for that. That's looking at a pool of blood and not what caused it: the bots/traffic which predate LLMs. And Cloudflare is working to fix it with the PrivacyPass standard (which Apple joined).
Each website is freely opting-into it. No one was forced. Why not ask yourself why that is?