Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook was used as a proxy by web scraping bots

datadome.co

Post reply on HN
Print view View original

31–40 of 125 posts

Re: Facebook was used as a proxy by web scraping bots

#31

Post by cblconfederate »

skybrian wrote: post #17
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

Is this the case for any web crawler?

not sure but it s a very common problem: https://www.google.com/search?q=cloudflare+attention+require...

Re: Facebook was used as a proxy by web scraping bots

#32

Post by stickfigure »

stingraycharles wrote: post #29
fiddlerwoaroof wrote: post #28

Earlier quoted context omitted.

I wonder if Accept: application/json Would be a reasonable alternative? Wasn't this supposed to be the point of content negotiation?

Maybe, but not really; seems like this thread is more about intent (“I just want a preview”) while content type is more about representation (“I want the content as json”). I can imagine that there will be websites that are actively using the accept parameter to distinguish between “regular visitors” and have their APIs at the same paths (didn’t Reddit do this at some point?), and thus your approach would break in th…

Accept: text/preview

Re: Facebook was used as a proxy by web scraping bots

#33

Post by echelon »

ddorian43 wrote: post #25

At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ? Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets. How many websites actually ping google on new content ?

This is so fucked. We've encrusted ourselves into this walled fiefdom, and there's no way to break free.

It's only going to get worse.

Chrome will be the only browser. AMP the only delivery mechanism. Video will require DRM. Eventually, text content will too. Binary blobs with no ad blocking.

Re: Facebook was used as a proxy by web scraping bots

#35

Post by beagle3 »

cblconfederate wrote: post #30
raverbashing wrote: post #20

Earlier quoted context omitted.

Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons. It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.

cloudflare can just allow a fair crawl rate instead of a captcha on first request

The problem is that bad actors can masquerade as a lot of independent clients (The first D in DDoS stands for "distributed").

Figuring out whether a site is under a DDoS attack or getting legitimate requests from many sources is a very hard problem, and can just be worded "telling good actors from bad actors" -- no simple solution works; also, who YOU consider a good actor and who the website owner considers a good actor may be at odds.

Most people (and CloudFlare by default) consider FAcebook a good actor; but as far as I'm concerned, Facebook is an evil an actor as one can be.

Re: Facebook was used as a proxy by web scraping bots

#36

Post by TechBro8615 »

ddorian43 wrote: post #25

At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ? Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets. How many websites actually ping google on new content ?

And don't you dare scrape Google results. That's against their TOS! Rules for thee, not for me.

Isn't it weird there is no machine-readable API to Google search results?

Re: Facebook was used as a proxy by web scraping bots

#37

Post by dmix »

cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR.

But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).

I hope every web admin gives it serious second thought before adopting Cloudflare. Just like for cellphones OS/operator the one thing I’d dream of is a tool that offers a limited set of what Cloudflare does (DDOS protection, hosting privacy layer) but is pro internet and pro privacy. They seem hostile to it in many ways likely because it directly affects their bottom line.

The bigger question is whether such a tool could be created without all the downsides. The two I listed I think yes. But their web app security system is overly strict and bad for the internet IMO.

And I say that knowing they protect some serious defenders of human rights and face a lot of abuse from the ‘bad guys’. I just wished there was a better middle ground.

Re: Facebook was used as a proxy by web scraping bots

#38

Post by dmix »

jedberg wrote: post #12

So this was a very long web page to say: Facebook forgot to rate limit their web scraper on a per user basis, but we told them and they fixed it.

Also it’s not really a scraper. Just a json response with the website preview stuff like meta tags, , and other basic information which could be useful for some bots.

But it does not give you the whole HTML. Or anything close.

Re: Facebook was used as a proxy by web scraping bots

#39

Post by pvg »

dmix wrote: post #37
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

because both the founder and employees of cloudflare are quite popular users on HN.

It seems a lot more likely people aren't finding your argument as convincing as you'd like. Plenty of well-known users (and users who identify their employer) around whose companies' HN-perception fortunes change quite a bit over time.

Re: Facebook was used as a proxy by web scraping bots

#40

Post by hombre_fatal »

dmix wrote: post #37
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

> But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN.

I don't think it gets much traction because you're barking up the wrong tree. Also, suggesting that YC is out to silence you and that nobody actually has a counter argument isn't very good for traction, either.

Until my website can't get taken off by a $5 rental of an internet-of-shit botnet, Cloudflare gives me and my users recourse against the bad actors of the world. (I also enjoy its host cloaking for my privacy)

You simply gloss over bad actors and attack one of the only solutions that works. The biggest threat to the open internet was its naive "there are no bad actors" design, not the people giving us one of the only bulwarks against bad design.

I agree with your last sentence that it would be nice to have a better middle ground, but notice that's not the "cloudflare bad" thesis of your comment.

The internet needs to be improved so that Cloudflare is redundant. It's not Cloudflare's fault that fundamental design oversights (like optional ISP egress filtering) have created a lucrative niche. And things like faster, unlimited data plans accessible by smart toasters and smart doorbells on top of the internet's naive architecture only entrenches Cloudflare further.

Post reply on HN