Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

blog.cloudflare.com

Post reply on HN
Print view View original

791–799 of 799 posts

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#791

Post by Timwi »

imtringued wrote: post #688
skydhash wrote: post #659

Earlier quoted context omitted.

It's not ads. We have ads in paper magazines and newspapers and no one went around with scissors to remove them. It's obnoxious ads, designed to violently grabs your attention and trackers (malware). It's like a newspapers giving your address to a whole crew of salemens that intrudes on your property at 3am and looking at you sleeping and installing cameras in your bathroom. All so that they can jump at you in the st…

This is one of the dumbest things about ad networks. Google has enough data about your watching habits on Youtube and their algorithm is basically as good as it gets in terms of showing you what you want to watch and getting you hooked on it, but the moment they show you ads, all that technical expertise appears to have vanished into thin air and all they show you is fake mobile ads? People hate obnoxious ads because…

> their algorithm is basically as good as it gets in terms of showing you what you want to watch and getting you hooked on it

It is? Are we talking about the same YouTube? I get absolutely useless recommendations, I get un-hooked within a couple videos, and I even keep getting recommendations for the same videos I've literally watched yesterday. Who in the world gets hooked by this??

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#792

Post by daviddanielng »

Spivak wrote: post #163
cj wrote: post #80

Earlier quoted context omitted.

Doesn't o3 sort of already do this? Whenever I ask it something, it makes it look like it simultaneously opens 3-8 pages (something a human can't do). Seems like a reasonable stance would be something like "Following the no crawl directive is especially necessary when navigating websites faster than humans can." > What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to ga…

You don't middle click a bunch of links when doing research? Of all the things to point to I wouldn't have thought "opens a bunch of tabs" to be one of the differentiating behaviors between browsing with Firefox and browsing with an LLM.

> simultaneously opens 3-8 pages (something a human can't do).

Can't you read?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#793

Post by 542354234235 »

gruez wrote: post #298
542354234235 wrote: post #137

Earlier quoted context omitted.

There is an expression “the dose makes the poison”. With any sufficiently complex or broad category situation, there is rarely a binary ideological position that covers any and all situations. Should drugs be legal for recreation? Well my feeling for marijuana and fentanyl are different. Should individuals be allowed to own weapons? My views differ depending on if it a switch blade knife of a Stinger missile. Can law…

>There is an expression “the dose makes the poison”. With any sufficiently complex or broad category situation, there is rarely a binary ideological position that covers any and all situations. Should drugs be legal for recreation? Well my feeling for marijuana and fentanyl are different. Should individuals be allowed to own weapons? My views differ depending on if it a switch blade knife of a Stinger missile. Can la…

>That sounds suspiciously close to "laws should just be whatever benefits me or my group".

I do not understand how you can make that leap. Saying laws should account for the imbalance of power between parties has nothing to do with "my" group. Police have asymmetric power over citizens, so laws should protect citizens from abuses of that power. Employers have asymmetric power over employees, so laws should protect workers from abuses of that power.

>This seems very susceptible to manipulation to get whatever conclusion you want.

Everything is. That is what bad faith arguments are. But in the real world, in a complex society, no simple rule over something as broadly defined as "intellectual property" can work every time in every situation.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#794

Post by lenkite »

danlitt wrote: post #626
AnthonBerg wrote: post #607

Earlier quoted context omitted.

Who cares what you want?

Most humans place the desires of human beings over the desires of companies.

Indeed. But that is a false equivalence - this is conflict of desires between small companies and creators and an AI-corp where the AI-corp wants to steal their content and give it to users with their shop branding.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#795

Post by hamishmacewan »

Cloudflare sits in a privileged choke-point of the internet, peering into traffic others can’t. Now they’re playing hall monitor, publicly wagging their finger at Perplexity for “stealth crawling.” If Perplexity’s a customer, they should be furious at the breach of trust; if not, this smells like a cheap sales pitch dressed up as public service. Who appointed Cloudflare as Miss Manners of the web, or deputised them as law enforcement?

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#796

Post by icetank »

hamishmacewan wrote: post #795

Cloudflare sits in a privileged choke-point of the internet, peering into traffic others can’t. Now they’re playing hall monitor, publicly wagging their finger at Perplexity for “stealth crawling.” If Perplexity’s a customer, they should be furious at the breach of trust; if not, this smells like a cheap sales pitch dressed up as public service. Who appointed Cloudflare as Miss Manners of the web, or deputised them a…

Cloudflare does allow bots to scrape sites. But in this case cloudflare was alerted by customers that there setting to disallow ai companies to access there site was not working. People pay cloudflare to specifically block perplexity bots.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#797

Post by martin-t »

crazygringo wrote: post #715
Wilder7977 wrote: post #658

Earlier quoted context omitted.

Because attribution, social recognition and prestige are among the many reasons why people put the information out there, and there is nothing wrong with any of them. This is why I care if my ideas are presented to others by an LLM (that maybe cites me in some % of cases) or directly to a human. There is already a difference between a human visiting my space (acknowledging it as such) to read and learn information an…

If you want attribution and prestige, then publish your stuff in an actual publication -- a journal, a magazine, whatever. Go on podcasts, speak at conferences, and so forth. Publishing on a personal blog is not the path. LLM's aren't taking away from your "prestige" or recognition. Any more than a podcaster referencing an idea of yours without mentioning you is. Or anyone else in casual conversation.

I can't believe the hypocrisy of a guy with 76029 internet points (that's a big time investment, would be a shame if someone trained an LLM on it) pretending to not understand that people want recognition for what they say, regardless of where they say it.

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#798

Post by kz20129087 »

andy99 wrote: post #327
seydor wrote: post #216

> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…

Can't agree more, cloudflare is destroying the internet. We've entered the equivalent of when having McAffe antivirus was worse than having an actual virus because it slowed down your computer to much. These user hostile solutions have taken us back to dialup era page loading speeds for many sites, it's absurd that anyone thinks this is a service worth paying for.

yes exactly, cloudflare is just bad tech where the remedy is worse than the disease. I am using a VPN and I get endless loops of please verify you are not a robot, this may take a few second (minutes, hours...)... so basically cloudflare tech must have this primitive code

is_using_vpn? -> bad,abuse,ddos

thanks' cloudflare for saving our internet by destroying it...

Re: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives

#799

Post by kz20129087 »

andy99 wrote: post #327
seydor wrote: post #216

> it is built on trust. This is funny coming from Cloudflare, the company that blocks most of the internet from being fetched with antispam checks even for a single web request. The internet we knew was open and not trusted , but thanks to companies like Cloudflare, now even the most benign , well meaning attempt to GET a website is met with a brick wall. The bots of Big Tech, namely Google, Meta and Apple are of cou…

Can't agree more, cloudflare is destroying the internet. We've entered the equivalent of when having McAffe antivirus was worse than having an actual virus because it slowed down your computer to much. These user hostile solutions have taken us back to dialup era page loading speeds for many sites, it's absurd that anyone thinks this is a service worth paying for.

Cloudflare operates with such biased logic such as "why are we shooting at all men who have a long beard?" really you want the terrorists to kill all your kids then???

"why are we cutting all the trees in the park?" really you want trees to fall on your kid and crushing them to death?? what's wrong with saving kids??

"why are we closing the water in the fountains in the town?" really you want your kids to drown into the fountains or drink contaminated water??

Post reply on HN