I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.
Facebook was used as a proxy by web scraping bots
71–80 of 125 posts
Re: Facebook was used as a proxy by web scraping bots
#72At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ? Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets. How many websites actually ping google on new content ?
You don’t crawl unless you have to, Twitter[1], Facebook, WordPress.com[2] and other big services have a firehose you can apply to and get real-time changes. If you’re crawling the web, you’re probably doing it wrong or only servicing a particular niche.
[1] https://developer.twitter.com/en/docs/twitter-api/v1/tweets/...
Re: Facebook was used as a proxy by web scraping bots
#73At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ? Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets. How many websites actually ping google on new content ?
And don't you dare scrape Google results. That's against their TOS! Rules for thee, not for me. Isn't it weird there is no machine-readable API to Google search results?
I thought this is exactly how DuckDuckGo worked?
Re: Facebook was used as a proxy by web scraping bots
#74Earlier quoted context omitted.
here's how simple it is : if (!website.underDDoS && website.requestedTimesToday[ip]
How do you implement "website.underDDoS"? Through a proxy - mind you; CloudFlare makes their decision without access to your CPU or DB metrics, and don't know which page load times are legitimately slow and which aren't supposed to be.
how about "haven't had requests for the past 2 minutes". Again, i m talking about links to obscure blogs that barely anyone reads, let alone DDoSes
I think another comment here may be closer to the truth, CF may only be running heuristics on the user agent
Re: Facebook was used as a proxy by web scraping bots
#75Earlier quoted context omitted.
when you paste a link on reddit and it autocompletes the title update a bookmark title, or check if it exists. is it not self-evident that a link being crawlable is useful?
>when you paste a link on reddit and it autocompletes the title Oh no, you have to copy/paste the title? >update a bookmark title, or check if it exists. I can access the site without a captcha, my browser can fetch the title. >is it not self-evident that a link being crawlable is useful? No, it is not. Maybe a site owner does not want crawlers to index the site? Me being able to access the title and any html meta ta…
i m not sure if you re being serious
CF is so widespread that it breaks a significant part of the web for simple things like getting the page title. That's all. The End.
Re: Facebook was used as a proxy by web scraping bots
#76Earlier quoted context omitted.
And don't you dare scrape Google results. That's against their TOS! Rules for thee, not for me. Isn't it weird there is no machine-readable API to Google search results?
I thought this is exactly how DuckDuckGo worked?
No, DuckDuckGo purchases search results from Bing: https://azure.microsoft.com/en-us/services/cognitive-service...
Re: Facebook was used as a proxy by web scraping bots
#77I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.
May I ask which country?
Re: Facebook was used as a proxy by web scraping bots
#78Earlier quoted context omitted.
>when you paste a link on reddit and it autocompletes the title Oh no, you have to copy/paste the title? >update a bookmark title, or check if it exists. I can access the site without a captcha, my browser can fetch the title. >is it not self-evident that a link being crawlable is useful? No, it is not. Maybe a site owner does not want crawlers to index the site? Me being able to access the title and any html meta ta…
i m not sure if you re being serious CF is so widespread that it breaks a significant part of the web for simple things like getting the page title. That's all. The End.
'I' can get the page title though. That's all. The End.
I don't care about your crawler. Or your ability to post the link to my site to twitter/fb and if I did maybe I'd revise my cloudflare settings.
Re: Facebook was used as a proxy by web scraping bots
#79Earlier quoted context omitted.
The problem is that bad actors can masquerade as a lot of independent clients (The first D in DDoS stands for "distributed"). Figuring out whether a site is under a DDoS attack or getting legitimate requests from many sources is a very hard problem, and can just be worded "telling good actors from bad actors" -- no simple solution works; also, who YOU consider a good actor and who the website owner considers a good a…
> sources is a very hard problem We're talking about virtually unknown blogs that get 1 http request from my server's IP, which is not blacklisted anywhere. It's not hard at all , i just think cloudflare's tech s not that good
Well if you have an easy solution that you think would work, why don't you put up a website, commission a DDOS attack from a skilled actor and try to demonstrate mitigation?
Companies pay big money to CloudFlare. If a simpler and cheaper solution is workable, they'll pay you instead.
Re: Facebook was used as a proxy by web scraping bots
#80I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.
May I ask which country?
There are a lot of countries where Facebook is “free”, for example India and Philippines.