Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook was used as a proxy by web scraping bots

datadome.co

Post reply on HN
Print view View original

41–50 of 125 posts

Re: Facebook was used as a proxy by web scraping bots

#41

Post by cblconfederate »

beagle3 wrote: post #35
cblconfederate wrote: post #30

Earlier quoted context omitted.

cloudflare can just allow a fair crawl rate instead of a captcha on first request

The problem is that bad actors can masquerade as a lot of independent clients (The first D in DDoS stands for "distributed"). Figuring out whether a site is under a DDoS attack or getting legitimate requests from many sources is a very hard problem, and can just be worded "telling good actors from bad actors" -- no simple solution works; also, who YOU consider a good actor and who the website owner considers a good a…

> sources is a very hard problem

We're talking about virtually unknown blogs that get 1 http request from my server's IP, which is not blacklisted anywhere. It's not hard at all , i just think cloudflare's tech s not that good

Re: Facebook was used as a proxy by web scraping bots

#42

Post by fiddlerwoaroof »

stingraycharles wrote: post #29
fiddlerwoaroof wrote: post #28

Earlier quoted context omitted.

I wonder if Accept: application/json Would be a reasonable alternative? Wasn't this supposed to be the point of content negotiation?

Maybe, but not really; seems like this thread is more about intent (“I just want a preview”) while content type is more about representation (“I want the content as json”). I can imagine that there will be websites that are actively using the accept parameter to distinguish between “regular visitors” and have their APIs at the same paths (didn’t Reddit do this at some point?), and thus your approach would break in th…

It seems to me that what "previews" really want is an API for the page's content in a structured format: OpenGraph tags and other microformats are one representation, but it's annoying to have to load _all_ the HTML just to grab title and the OG tags.

Re: Facebook was used as a proxy by web scraping bots

#43

Post by [deleted] »

cblconfederate wrote: post #30
raverbashing wrote: post #20

Earlier quoted context omitted.

Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons. It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.

cloudflare can just allow a fair crawl rate instead of a captcha on first request

[deleted]

Re: Facebook was used as a proxy by web scraping bots

#44

Post by hombre_fatal »

cblconfederate wrote: post #41
beagle3 wrote: post #35

Earlier quoted context omitted.

The problem is that bad actors can masquerade as a lot of independent clients (The first D in DDoS stands for "distributed"). Figuring out whether a site is under a DDoS attack or getting legitimate requests from many sources is a very hard problem, and can just be worded "telling good actors from bad actors" -- no simple solution works; also, who YOU consider a good actor and who the website owner considers a good a…

> sources is a very hard problem We're talking about virtually unknown blogs that get 1 http request from my server's IP, which is not blacklisted anywhere. It's not hard at all , i just think cloudflare's tech s not that good

You're really pulling a "how hard could it really be??" to DDoS prevention?

You should at least be humbled by how few services can even offer DDoS protection that works against volumetric attacks and isn't just based on null-routing. The people with skin and money in the game might know something you don't.

Re: Facebook was used as a proxy by web scraping bots

#46

Post by qwerty456127 »

neilparikh wrote: post #13
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.

There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index and categorize the web subset I choose for me. I believe this is a perfectly legitimate use. But they will probably try to stop me.

Re: Facebook was used as a proxy by web scraping bots

#47

Post by cblconfederate »

hombre_fatal wrote: post #44
cblconfederate wrote: post #41

Earlier quoted context omitted.

> sources is a very hard problem We're talking about virtually unknown blogs that get 1 http request from my server's IP, which is not blacklisted anywhere. It's not hard at all , i just think cloudflare's tech s not that good

You're really pulling a "how hard could it really be??" to DDoS prevention? You should at least be humbled by how few services can even offer DDoS protection that works against volumetric attacks and isn't just based on null-routing. The people with skin and money in the game might know something you don't.

here's how simple it is :

    if (!website.underDDoS && website.requestedTimesToday[ip]

Re: Facebook was used as a proxy by web scraping bots

#48

Post by cblconfederate »

qwerty456127 wrote: post #46
neilparikh wrote: post #13

Earlier quoted context omitted.

> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.

There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index…

> Blocking crawling should be outlawed

That's overly broad. But maybe it should be illegal to have exceptions only for major monopolies.

Re: Facebook was used as a proxy by web scraping bots

#49

Post by ShamelessC »

dmix wrote: post #37
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

Edit: my bad. Misinterpreted your comment.

Can you elaborate on how Tor is a threat to the open internet? That's a non-obvious statement to me. I'm aware that it's compromisable via controlling exit nodes (NSA, various nations) but that's not really the threat profile for the average person. Are there any other reasons?

Because despite its flaws, afaik TOR is an attempt to make the internet _more_ open to those who are being surveiled.

What am I missing?

Re: Facebook was used as a proxy by web scraping bots

#50

Post by JosephRedfern »

ShamelessC wrote: post #49
dmix wrote: post #37

Earlier quoted context omitted.

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

Edit: my bad. Misinterpreted your comment. Can you elaborate on how Tor is a threat to the open internet? That's a non-obvious statement to me. I'm aware that it's compromisable via controlling exit nodes (NSA, various nations) but that's not really the threat profile for the average person. Are there any other reasons? Because despite its flaws, afaik TOR is an attempt to make the internet _more_ open to those who a…

I think OP is suggesting that Cloudflare is a threat to TOR, not that TOR is a threat to the internet.
Post reply on HN