It really shouldn't be hard to generate gigantic quantities of the stuff. Simulate old forum posts, or academic papers.
Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
231–240 of 799 posts
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#232Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#233I find this problem quite difficult to solve: 1. If I as a human request a website, then I should be shown the content. Everyone agrees. 2. If I as the human request the software on my computer to modify the content before displaying it, for example by installing an ad-blocker into my user agent, then that's my choice and the website should not be notified about it. Most users agree, some websites try to nag you into…
1. To access a website you need a limited anonymized token that proves you are a human being, issued by a state authority 2. the end I am firmly convinced that this should be the future in the next decade, since the internet as we know it has been weaponized and ruined by social media, bots, state actors and now AI. There should exist an internet for humans only, with a single account per domain.
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#234Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#235Earlier quoted context omitted.
It’s possible to violate all sorts of social norms. Societies that celebrate people that do so are on the far opposite end of the spectrum from high trust ones. They are rather unpleasant.
Just the Silicon Valley ethos extended to it's logical conclusions. These companies take advantage of public space, utilities and goodwill at industrial scale to "move fast and break things" and then everyone else has to deal with the ensuing consequences. Like how cities are awash in those fucking electric scooters now. Mind you I'm not saying electric scooters are a bad idea, I have one and I quite enjoy it. I'm sa…
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#236AI companies continuing to have problems with the concept of "consent" is increasingly alarming god help us if they ever manage to build anything more than shitty chatbots
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#237> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#238Earlier quoted context omitted.
It’s possible to violate all sorts of social norms. Societies that celebrate people that do so are on the far opposite end of the spectrum from high trust ones. They are rather unpleasant.
[flagged]
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#239It's ironic Perplexity itself blocks crawlers: $ curl -sI https://www.perplexity.ai | head -1 HTTP/2 403 Edit: trying to fake a browser user agent with curl also doesn't work, they're using a more sophisticated method to detect crawlers.
someone asked this already to the CEO: https://x.com/AravSrinivas/status/1819610286036488625
Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives
#240> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…