Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook was used as a proxy by web scraping bots

datadome.co

Post reply on HN
Print view View original

21–30 of 125 posts

Re: Facebook was used as a proxy by web scraping bots

#21

Post by Kaze404 »

raverbashing wrote: post #20
rainingcatndogs wrote: post #16

Earlier quoted context omitted.

And unfortunately, cloudflare is everywhere. This trend will make it even harder for projects like a new search engine to enter the game.

Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons. It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.

I use Zoho.com and I rarely get spam, if ever.

Re: Facebook was used as a proxy by web scraping bots

#22

Post by nbadg »

bgirard wrote: post #18
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

Long term, a new HTTP META method would be interesting. I wonder if something like that has ever been considered. Providers like Cloudflare would hopefully be more lenient with these requests.

Huh. It's certainly an interesting idea! Strictly speaking, individual people could implement this today, since nonstandard HTTP verbs don't break anything that doesn't know to request with them. (It wouldn't be of much use, because clients wouldn't know to use it, but still -- something that could easily be prototyped).

I don't think FAAANG (or any other big players) would have much interest in making it happen in the standard though, since it would undercut their big-player advantage.

Re: Facebook was used as a proxy by web scraping bots

#23

Post by snazz »

Kaze404 wrote: post #21
raverbashing wrote: post #20

Earlier quoted context omitted.

Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons. It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.

I use Zoho.com and I rarely get spam, if ever.

Zoho isn't Google-size, but it isn't irrelevant, either. Sending mail from a self-hosted email server is far harder since the big providers might put it in spam or drop it even earlier.

Re: Facebook was used as a proxy by web scraping bots

#24

Post by dzhiurgis »

Kaze404 wrote: post #21
raverbashing wrote: post #20

Earlier quoted context omitted.

Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons. It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.

I use Zoho.com and I rarely get spam, if ever.

To add to sibling - running your own mail server is the only way to ensure your email is not read by someone else which is so messed up.

Re: Facebook was used as a proxy by web scraping bots

#25

Post by ddorian43 »

At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ?

Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets.

How many websites actually ping google on new content ?

Re: Facebook was used as a proxy by web scraping bots

#26

Post by sergiotapia »

cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

The cloudflare and google catcha are terrible. It's so bad that at this point I just close the tab if they challenge me with it. I use Brave and always have Shields UP, it seems having it up makes the captchas extremely difficult. Mission accomplished I guess.

Re: Facebook was used as a proxy by web scraping bots

#27

Post by daniellarusso »

tyingq wrote: post #8

That's pretty interesting, Facebook as a "web scale / hundreds of pages per second" batch web page summarizer. I imagine you could build a pretty decent general purpose search engine that way...free crawler.

As long is they are using opengraph meta tags.

Re: Facebook was used as a proxy by web scraping bots

#28

Post by fiddlerwoaroof »

bgirard wrote: post #18
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

Long term, a new HTTP META method would be interesting. I wonder if something like that has ever been considered. Providers like Cloudflare would hopefully be more lenient with these requests.

I wonder if

    Accept: application/json
Would be a reasonable alternative? Wasn't this supposed to be the point of content negotiation?

Re: Facebook was used as a proxy by web scraping bots

#29

Post by stingraycharles »

fiddlerwoaroof wrote: post #28
bgirard wrote: post #18

Earlier quoted context omitted.

Long term, a new HTTP META method would be interesting. I wonder if something like that has ever been considered. Providers like Cloudflare would hopefully be more lenient with these requests.

I wonder if Accept: application/json Would be a reasonable alternative? Wasn't this supposed to be the point of content negotiation?

Maybe, but not really; seems like this thread is more about intent (“I just want a preview”) while content type is more about representation (“I want the content as json”). I can imagine that there will be websites that are actively using the accept parameter to distinguish between “regular visitors” and have their APIs at the same paths (didn’t Reddit do this at some point?), and thus your approach would break in this case.

I guess what this is really about is, I hate to say it, but something in the direction of the semantic web, where web servers (and in this case, CloudFlare et al) actually gain a deeper understanding of the content they serve, and a web browser / crawler being able to query this content directly.

Re: Facebook was used as a proxy by web scraping bots

#30

Post by cblconfederate »

raverbashing wrote: post #20
rainingcatndogs wrote: post #16

Earlier quoted context omitted.

And unfortunately, cloudflare is everywhere. This trend will make it even harder for projects like a new search engine to enter the game.

Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons. It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.

cloudflare can just allow a fair crawl rate instead of a captcha on first request
Post reply on HN