Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook was used as a proxy by web scraping bots

datadome.co

Post reply on HN
Print view View original

51–60 of 125 posts

Re: Facebook was used as a proxy by web scraping bots

#51

Post by m-ee »

ShamelessC wrote: post #49
dmix wrote: post #37

Earlier quoted context omitted.

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

Edit: my bad. Misinterpreted your comment. Can you elaborate on how Tor is a threat to the open internet? That's a non-obvious statement to me. I'm aware that it's compromisable via controlling exit nodes (NSA, various nations) but that's not really the threat profile for the average person. Are there any other reasons? Because despite its flaws, afaik TOR is an attempt to make the internet _more_ open to those who a…

I read that as Cloudflare is a threat to tools like Tor

Re: Facebook was used as a proxy by web scraping bots

#52

Post by [deleted] »

ShamelessC wrote: post #49
dmix wrote: post #37

Earlier quoted context omitted.

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

Edit: my bad. Misinterpreted your comment. Can you elaborate on how Tor is a threat to the open internet? That's a non-obvious statement to me. I'm aware that it's compromisable via controlling exit nodes (NSA, various nations) but that's not really the threat profile for the average person. Are there any other reasons? Because despite its flaws, afaik TOR is an attempt to make the internet _more_ open to those who a…

[deleted]

Re: Facebook was used as a proxy by web scraping bots

#53

Post by grumpitron »

ShamelessC wrote: post #49
dmix wrote: post #37

Earlier quoted context omitted.

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

Edit: my bad. Misinterpreted your comment. Can you elaborate on how Tor is a threat to the open internet? That's a non-obvious statement to me. I'm aware that it's compromisable via controlling exit nodes (NSA, various nations) but that's not really the threat profile for the average person. Are there any other reasons? Because despite its flaws, afaik TOR is an attempt to make the internet _more_ open to those who a…

My read of it was that CloudFlare is a threat to both the open internet and TOR, not that TOR was also a threat to the open internet.

Re: Facebook was used as a proxy by web scraping bots

#54

Post by londons_explore »

cblconfederate wrote: post #41
beagle3 wrote: post #35

Earlier quoted context omitted.

The problem is that bad actors can masquerade as a lot of independent clients (The first D in DDoS stands for "distributed"). Figuring out whether a site is under a DDoS attack or getting legitimate requests from many sources is a very hard problem, and can just be worded "telling good actors from bad actors" -- no simple solution works; also, who YOU consider a good actor and who the website owner considers a good a…

> sources is a very hard problem We're talking about virtually unknown blogs that get 1 http request from my server's IP, which is not blacklisted anywhere. It's not hard at all , i just think cloudflare's tech s not that good

I got round it by just making sure the user agent is set to the latest version of Chrome rather than a version from a few years ago that I had hardcoded before. It seems Cloudflares protection is pretty much "is your user agent in the top 10 user agents?".

Did you try that?

Re: Facebook was used as a proxy by web scraping bots

#55

Post by seniorgarcia »

cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

I don't get what value link previews add. Someone shares a link with me (on skype, slack, teams... whatever) and I care about the content because the person sharing it with me thinks I could/should care about it, or someone shares a link on an aggregator and then I don't think it is too much to ask for that someone to write a summary. If the link is worth sharing writing 1 sentence to explain why isn't too much to ask.

What is the value a link preview adds? And why should I, as a content provider care about the value you add? Cloudflare does something for me, what is your service doing for me and why should I whitelist you (or care about you)?

Re: Facebook was used as a proxy by web scraping bots

#56

Post by londons_explore »

dzhiurgis wrote: post #24
Kaze404 wrote: post #21

Earlier quoted context omitted.

I use Zoho.com and I rarely get spam, if ever.

To add to sibling - running your own mail server is the only way to ensure your email is not read by someone else which is so messed up.

> running your own mail server is the only way to ensure your email is not read by someone else

But any mail you send to someone else probably ends up read by Google/Microsoft anyway, since that's where their mailbox is.

Also, email security is a joke. It's 2020, and even TLS encrypted SMTP connections tend not to check for a valid certificate, making them trivial to MITM.

Re: Facebook was used as a proxy by web scraping bots

#57

Post by dasil003 »

seniorgarcia wrote: post #55
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

I don't get what value link previews add. Someone shares a link with me (on skype, slack, teams... whatever) and I care about the content because the person sharing it with me thinks I could/should care about it, or someone shares a link on an aggregator and then I don't think it is too much to ask for that someone to write a summary. If the link is worth sharing writing 1 sentence to explain why isn't too much to as…

They're sending you traffic.

Imagine Twitter or Facebook without link preview, it's much harder to use and overall reduces the change I'll click on a link. Do you think only Twitter and Facebook should be allowed publish previews?

Re: Facebook was used as a proxy by web scraping bots

#58

Post by cblconfederate »

londons_explore wrote: post #54
cblconfederate wrote: post #41

Earlier quoted context omitted.

> sources is a very hard problem We're talking about virtually unknown blogs that get 1 http request from my server's IP, which is not blacklisted anywhere. It's not hard at all , i just think cloudflare's tech s not that good

I got round it by just making sure the user agent is set to the latest version of Chrome rather than a version from a few years ago that I had hardcoded before. It seems Cloudflares protection is pretty much "is your user agent in the top 10 user agents?". Did you try that?

I have, iirc it worked some times, but not always. Is it a reliable solution for you?

Re: Facebook was used as a proxy by web scraping bots

#59

Post by einpoklum »

dasil003 wrote: post #57
seniorgarcia wrote: post #55

Earlier quoted context omitted.

I don't get what value link previews add. Someone shares a link with me (on skype, slack, teams... whatever) and I care about the content because the person sharing it with me thinks I could/should care about it, or someone shares a link on an aggregator and then I don't think it is too much to ask for that someone to write a summary. If the link is worth sharing writing 1 sentence to explain why isn't too much to as…

They're sending you traffic. Imagine Twitter or Facebook without link preview, it's much harder to use and overall reduces the change I'll click on a link. Do you think only Twitter and Facebook should be allowed publish previews?

> it's much harder to use

Actually it's easier to use, in that the preview doesn't take up screen real estate. Perhaps you mean the experience is less pleasant?

Re: Facebook was used as a proxy by web scraping bots

#60

Post by def_true_false »

dasil003 wrote: post #57
seniorgarcia wrote: post #55

Earlier quoted context omitted.

I don't get what value link previews add. Someone shares a link with me (on skype, slack, teams... whatever) and I care about the content because the person sharing it with me thinks I could/should care about it, or someone shares a link on an aggregator and then I don't think it is too much to ask for that someone to write a summary. If the link is worth sharing writing 1 sentence to explain why isn't too much to as…

They're sending you traffic. Imagine Twitter or Facebook without link preview, it's much harder to use and overall reduces the change I'll click on a link. Do you think only Twitter and Facebook should be allowed publish previews?

Half the time the link preview picks the wrong picture and sometimes even the quote. Twitter and Facebook would both be improved by disabling it. Hell, it might even stop people from thinking they need a hero image for their 2 paragraph medium shitpost.
Post reply on HN