Earlier quoted context omitted.
And don't you dare scrape Google results. That's against their TOS! Rules for thee, not for me. Isn't it weird there is no machine-readable API to Google search results?
I thought this is exactly how DuckDuckGo worked?
Facebook was used as a proxy by web scraping bots
101–110 of 125 posts
Re: Facebook was used as a proxy by web scraping bots
#102I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.
Re: Facebook was used as a proxy by web scraping bots
#103Earlier quoted context omitted.
In The Philippines, my understanding is that they have ample bandwidth via Korea and other countries in the region. But the reason they have such expensive terrible internet is because of a lack of net neutrality and deregulation. The cellphone duopoly sells "YouTube passes", that entitle you to get unthrottled YouTube for brief periods of time.
I wonder if a proxy could be made to encode data as video to put in a YouTube livestream. You'd still need an uplink but the upload bandwidth usage is a fraction of the download one for typical Internet usage.
You'd still need an uplink but the upload bandwidth usage is a fraction of the download one for typical Internet usage.
Perhaps the chat/comments (once again with heavy stego/encryption) would work?
Related and older idea: https://en.wikipedia.org/wiki/ArVid
Re: Facebook was used as a proxy by web scraping bots
#104Earlier quoted context omitted.
May I ask which country?
There are a lot of countries where Facebook is “free”, for example India and Philippines.
[0] https://www.theverge.com/2016/2/8/10913398/free-basics-india...
Re: Facebook was used as a proxy by web scraping bots
#105You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…
I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…
Re: Facebook was used as a proxy by web scraping bots
#106Earlier quoted context omitted.
> running your own mail server is the only way to ensure your email is not read by someone else But any mail you send to someone else probably ends up read by Google/Microsoft anyway, since that's where their mailbox is. Also, email security is a joke. It's 2020, and even TLS encrypted SMTP connections tend not to check for a valid certificate, making them trivial to MITM.
Practically speaking how does one MITM an SMTP connection? For example, from Google to Microsoft. They connect directly to the IP addresses they get from MX records + lookup. What's the actual threat vector/execution here?
Re: Facebook was used as a proxy by web scraping bots
#107Earlier quoted context omitted.
I got round it by just making sure the user agent is set to the latest version of Chrome rather than a version from a few years ago that I had hardcoded before. It seems Cloudflares protection is pretty much "is your user agent in the top 10 user agents?". Did you try that?
I have, iirc it worked some times, but not always. Is it a reliable solution for you?
Re: Facebook was used as a proxy by web scraping bots
#108I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.
Re: Facebook was used as a proxy by web scraping bots
#109You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…
I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…
I genuinely think they're a net positive for and supporter of Tor users. Before, site owners and security providers who faced issues with abusive/malicious traffic behind Tor connections (spam, illicit content, security scanning, password struffing) nearly always resorted to outright blocking all Tor exit node IPs, because they had no other feasible option. I've been in that position. Cloudflare at least provides any site owner an ability to easily allow the traffic; just with a fairly quick occasional bot check.
Additionally, as of 2018 they now have an "Onion Routing" option which site owners can enable, which results in Tor users being able to access your site 100% through the Tor network. As a result, Tor users no longer experience any captchas, load your site faster, and never have to touch the clearnet.
>But their web app security system is overly strict and bad for the internet IMO.
Their WAF seems to have a pretty low false positive rate, compared to others I've seen. (Though the flipside of that is it also has a pretty high false negative rate and isn't very helpful against a dedicated non-automated attacker, like many other WAFs.)
>But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN.
They do post a lot here, but I doubt that's really responsible for defensive responses from other HN users. The most common criticism I see here (presenting a captcha for people using Tor, which site owners can now disable) makes me think the majority of people making the criticism have never run large websites or worked infosec for any organization with a large website.
Tor is of course not a threat itself, but anecdotally I'd estimate 90 - 95% of traffic that the average website owner receives from Tor is highly abusive/malicious, and Cloudflare empirically estimated 94% as of 2016 (https://blog.cloudflare.com/the-trouble-with-tor/). And anecdotally, not only is a high percentage of Tor traffic malicious, in many cases a significant percentage of all malicious traffic is Tor traffic. Naturally, due to Tor by design making it impossible to distinguish the ~94% connections from the ~6%, it's extremely difficult to mitigate this without just blocking 100% of Tor traffic. This is obviously not Tor or anyone's fault; it's just a practical reality for website owners. This sort of situation will always be the case for any kind of robust privacy-protecting application.
Cloudflare is possibly the first free service that actually enables anyone to easily allow normal traffic from Tor without much increase in security/abuse risk. They seem explicitly pro-Tor, especially with the explicit Onion Routing feature that lets Tor users access your site 100% through the Tor network without ever experiencing captchas, and statements like in https://blog.cloudflare.com/the-trouble-with-tor/ and https://blog.cloudflare.com/cloudflare-onion-service/
One may certainly have lots of other justified, legitimate concerns regarding the company and their disproportionate control of a huge chunk of the internet and web, but I'm not sure how someone could read those, see how the traffic is handled in practice, and conclude they're anti-Tor or a dangerous threat to Tor.
Re: Facebook was used as a proxy by web scraping bots
#110Earlier quoted context omitted.
I’ve long said Cloudflare is a dangerous threat to the open internet and as well as some privacy tools like TOR. But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. Some have given me brief half assed counter answers that conveniently miss other harder questions like a good PR person does (and which you seem to have gotten in your reply).…
> But it doesn’t always get much traction on here because both the founder and employees of cloudflare are quite popular users on HN. I don't think it gets much traction because you're barking up the wrong tree. Also, suggesting that YC is out to silence you and that nobody actually has a counter argument isn't very good for traction, either. Until my website can't get taken off by a $5 rental of an internet-of-shit…
The parent poster had a point and your reply is reenforcing it.