Live data from Hacker News

Facebook was used as a proxy by web scraping bots

datadome.co

101–110 of 125 posts

Re: Facebook was used as a proxy by web scraping bots

#101

Earlier quoted context omitted.

And don't you dare scrape Google results. That's against their TOS! Rules for thee, not for me. Isn't it weird there is no machine-readable API to Google search results?

I thought this is exactly how DuckDuckGo worked?

You mean startpage?

Re: Facebook was used as a proxy by web scraping bots

#102

I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.

Less technically knowledgeable people would probably do something similar, using Facebook as an even slower (and lossier) "layer 8 proxy" as opposed to your "layer 7 proxy".

Re: Facebook was used as a proxy by web scraping bots

#103

Earlier quoted context omitted.

In The Philippines, my understanding is that they have ample bandwidth via Korea and other countries in the region. But the reason they have such expensive terrible internet is because of a lack of net neutrality and deregulation. The cellphone duopoly sells "YouTube passes", that entitle you to get unthrottled YouTube for brief periods of time.

I wonder if a proxy could be made to encode data as video to put in a YouTube livestream. You'd still need an uplink but the upload bandwidth usage is a fraction of the download one for typical Internet usage.

Yes, that is definitely possible and might make for a fun project. Use stego to make the livestream look innocuous and apply heavy ECC, further resisting censorship and arousal of suspicion. I think this is the closest I've seen to a public implementation of that idea: https://news.ycombinator.com/item?id=12166332

You'd still need an uplink but the upload bandwidth usage is a fraction of the download one for typical Internet usage.

Perhaps the chat/comments (once again with heavy stego/encryption) would work?

Related and older idea: https://en.wikipedia.org/wiki/ArVid

Re: Facebook was used as a proxy by web scraping bots

#104
post #80
post #77

Earlier quoted context omitted.

May I ask which country?

There are a lot of countries where Facebook is “free”, for example India and Philippines.

Free Basics was not allowed in India by the Telecom Regulatory Authority of India[0]. The list of countries where Free Basics is currently operated is listed on Internet.org website[1].

[0] https://www.theverge.com/2016/2/8/10913398/free-basics-india...

[1] https://info.internet.org/en/story/where-weve-launched/

Re: Facebook was used as a proxy by web scraping bots

#105
post #37

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

Cloudflare even puts multiple captcha challenges for any request from the default browser on the Samsung S7 Edge. Granted it's an old phone at this point, and most users install Chrome on their phones, but I end up skipping a lot of websites on my phone rather than participate in furthering the misconception that "Chrome is the only browser".

Re: Facebook was used as a proxy by web scraping bots

#106
post #95

Earlier quoted context omitted.

> running your own mail server is the only way to ensure your email is not read by someone else But any mail you send to someone else probably ends up read by Google/Microsoft anyway, since that's where their mailbox is. Also, email security is a joke. It's 2020, and even TLS encrypted SMTP connections tend not to check for a valid certificate, making them trivial to MITM.

Practically speaking how does one MITM an SMTP connection? For example, from Google to Microsoft. They connect directly to the IP addresses they get from MX records + lookup. What's the actual threat vector/execution here?

Anyone with hardware on the network path can do it... Or anyone who can inject BGP routes can do it too.

Re: Facebook was used as a proxy by web scraping bots

#107

Earlier quoted context omitted.

I got round it by just making sure the user agent is set to the latest version of Chrome rather than a version from a few years ago that I had hardcoded before. It seems Cloudflares protection is pretty much "is your user agent in the top 10 user agents?". Did you try that?

I have, iirc it worked some times, but not always. Is it a reliable solution for you?

It's at least a 95% reliable solution, which seems to be about the same as a real user sees.

Re: Facebook was used as a proxy by web scraping bots

#108

I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.

Any chance you could share that? I think some flights still don't charge for Facebook messenger, so HTTP over messenger might still be useful.

Re: Facebook was used as a proxy by web scraping bots

#109
post #37

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

Any company through which a high percentage of web traffic is not only routed through but fully reverse-proxied of course always should be a significant concern and should be subject to extreme scrutiny. But why explicitly do you think they're anti-internet and anti-privacy? To me it seems like being pro-internet and pro-privacy aligns both with their general incentives and their monetary incentives.

I genuinely think they're a net positive for and supporter of Tor users. Before, site owners and security providers who faced issues with abusive/malicious traffic behind Tor connections (spam, illicit content, security scanning, password struffing) nearly always resorted to outright blocking all Tor exit node IPs, because they had no other feasible option. I've been in that position. Cloudflare at least provides any site owner an ability to easily allow the traffic; just with a fairly quick occasional bot check.

Additionally, as of 2018 they now have an "Onion Routing" option which site owners can enable, which results in Tor users being able to access your site 100% through the Tor network. As a result, Tor users no longer experience any captchas, load your site faster, and never have to touch the clearnet.

>But their web app security system is overly strict and bad for the internet IMO.

Their WAF seems to have a pretty low false positive rate, compared to others I've seen. (Though the flipside of that is it also has a pretty high false negative rate and isn't very helpful against a dedicated non-automated attacker, like many other WAFs.)

>But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN.

They do post a lot here, but I doubt that's really responsible for defensive responses from other HN users. The most common criticism I see here (presenting a captcha for people using Tor, which site owners can now disable) makes me think the majority of people making the criticism have never run large websites or worked infosec for any organization with a large website.

Tor is of course not a threat itself, but anecdotally I'd estimate 90 - 95% of traffic that the average website owner receives from Tor is highly abusive/malicious, and Cloudflare empirically estimated 94% as of 2016 (https://blog.cloudflare.com/the-trouble-with-tor/). And anecdotally, not only is a high percentage of Tor traffic malicious, in many cases a significant percentage of all malicious traffic is Tor traffic. Naturally, due to Tor by design making it impossible to distinguish the ~94% connections from the ~6%, it's extremely difficult to mitigate this without just blocking 100% of Tor traffic. This is obviously not Tor or anyone's fault; it's just a practical reality for website owners. This sort of situation will always be the case for any kind of robust privacy-protecting application.

Cloudflare is possibly the first free service that actually enables anyone to easily allow normal traffic from Tor without much increase in security/abuse risk. They seem explicitly pro-Tor, especially with the explicit Onion Routing feature that lets Tor users access your site 100% through the Tor network without ever experiencing captchas, and statements like in https://blog.cloudflare.com/the-trouble-with-tor/ and https://blog.cloudflare.com/cloudflare-onion-service/

One may certainly have lots of other justified, legitimate concerns regarding the company and their disproportionate control of a huge chunk of the internet and web, but I'm not sure how someone could read those, see how the traffic is handled in practice, and conclude they're anti-Tor or a dangerous threat to Tor.

Re: Facebook was used as a proxy by web scraping bots

#110
post #37

Earlier quoted context omitted.

I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…

> But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. I don't think it gets much traction because you're barking up the wrong tree. Also, suggesting that YC is out to silence you and that nobody actually has a counter argument isn't very good for traction, either. Until my website can't get taken off by a $5 rental of an internet-of-shit…

Your tone shines a light that perhaps you are on of those cloudflaire employees/fanboys who will drown out a warning.

The parent poster had a point and your reply is reenforcing it.

Post reply on HN