Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook was used as a proxy by web scraping bots

datadome.co

Post reply on HN
Print view View original

111–120 of 125 posts

Re: Facebook was used as a proxy by web scraping bots

#111

Post by arebhaibhai »

Ombudsman wrote: post #71

I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.

Same story here. Data was so expensive and Airtel let users open airtel.in etc at Zero balance. We used to use all kinds of Opera and UC "Handler Mods" with custom HTTP headers like Host or X-Online-Host to fool the ISP. First on Nokia s40 and Symbain and later on Android. Someone made a Handler Mod of pshipon VPN and man, it was slow but so cool. And then Jio happened!

Re: Facebook was used as a proxy by web scraping bots

#112

Post by raxxorrax »

Ombudsman wrote: post #71

I came from a third world country and internet was pretty expensive. For some reason, my provider made Facebook completely free. So in my free college days I used the Facebook developer echo API to make a HTTP proxy so I can browse internet for free. It was terrible, was only HTTP 1, so no web sockets, videos stopped randomly etc, but hey I could read Reddit.

It would also be a decent anti-tracking mechanism.

Re: Facebook was used as a proxy by web scraping bots

#113

Post by beagle3 »

cblconfederate wrote: post #74
beagle3 wrote: post #66

Earlier quoted context omitted.

How do you implement "website.underDDoS"? Through a proxy - mind you; CloudFlare makes their decision without access to your CPU or DB metrics, and don't know which page load times are legitimately slow and which aren't supposed to be.

how about "haven't had requests for the past 2 minutes". Again, i m talking about links to obscure blogs that barely anyone reads, let alone DDoSes I think another comment here may be closer to the truth, CF may only be running heuristics on the user agent

If hardly anyone reads or DDoSes them, why did they go to the trouble of setting up CloudFlare? It’s free for those obscure blogs, but it’s definitely a non trivial hassle. Usually people set it up only after they experienced their first attack.

I get it that you are upset Google gets to scrape them and you don’t. But bad actors really are making it difficult for everyone to just “be” on the internet.

Re: Facebook was used as a proxy by web scraping bots

#114

Post by ddorian43 »

withinboredom wrote: post #72
ddorian43 wrote: post #25

At the best case scenario, google has a monopoly on scraping. Imagine trying to create a global search engine, how can you possibly even crawl sites that are behind cloudflare or just allow google/fb/bing bots ? Can you real-time crawl twitter ? Pretty sure they have a special deal with google to instant ping on new tweets. How many websites actually ping google on new content ?

You don’t crawl unless you have to, Twitter[1], Facebook, WordPress.com[2] and other big services have a firehose you can apply to and get real-time changes. If you’re crawling the web, you’re probably doing it wrong or only servicing a particular niche. [1] https://developer.twitter.com/en/docs/twitter-api/v1/tweets/... [2] https://developer.wordpress.com/docs/firehose/

You missed the pricing. This is what you're doing wrong. And you missed the cloudflare part too.

Re: Facebook was used as a proxy by web scraping bots

#116

Post by nl »

octoberfranklin wrote: post #82
anuila wrote: post #80

Earlier quoted context omitted.

There are a lot of countries where Facebook is “free”, for example India and Philippines.

I'm guessing that in these countries the cost of Internet traffic is dominated by the undersea cables at their borders. Facebook has been paying to have new undersea cables laid. This is done as part of a consortium, but those cables only have 6-12 strands in them (the repeaters are bulky) so owning even just one whole strand of fiber in an undersea cable is still an obscene amount of bandwidth for a single company t…

> the cost of Internet traffic is dominated by the undersea cables at their borders.

This is rarely the case. It's usually monopoly providers and/or people speaking historically when it was expensive because it was rarer.

India and the Philippines both have more than adequate international bandwidth.

Re: Facebook was used as a proxy by web scraping bots

#117

Post by mtberatwork »

bgirard wrote: post #18
cblconfederate wrote: post #7

You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…

Long term, a new HTTP META method would be interesting. I wonder if something like that has ever been considered. Providers like Cloudflare would hopefully be more lenient with these requests.

Doesn't the oembed spec [1] already solve this? I think the OP could solve their problem by simply creating an oembed endpoint with all the necessary meta data.

[1] https://oembed.com/

Re: Facebook was used as a proxy by web scraping bots

#118

Post by sokoloff »

qwerty456127 wrote: post #46
neilparikh wrote: post #13

Earlier quoted context omitted.

> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.

There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index…

Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler?

They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this proposal.

Re: Facebook was used as a proxy by web scraping bots

#119

Post by cblconfederate »

beagle3 wrote: post #113
cblconfederate wrote: post #74

Earlier quoted context omitted.

how about "haven't had requests for the past 2 minutes". Again, i m talking about links to obscure blogs that barely anyone reads, let alone DDoSes I think another comment here may be closer to the truth, CF may only be running heuristics on the user agent

If hardly anyone reads or DDoSes them, why did they go to the trouble of setting up CloudFlare? It’s free for those obscure blogs, but it’s definitely a non trivial hassle. Usually people set it up only after they experienced their first attack. I get it that you are upset Google gets to scrape them and you don’t. But bad actors really are making it difficult for everyone to just “be” on the internet.

i dont know! but they do it, everyone does it because everyone else does it. it s not unusual

Re: Facebook was used as a proxy by web scraping bots

#120

Post by stingraycharles »

stickfigure wrote: post #32
stingraycharles wrote: post #29

Earlier quoted context omitted.

Maybe, but not really; seems like this thread is more about intent (“I just want a preview”) while content type is more about representation (“I want the content as json”). I can imagine that there will be websites that are actively using the accept parameter to distinguish between “regular visitors” and have their APIs at the same paths (didn’t Reddit do this at some point?), and thus your approach would break in th…

Accept: text/preview

In what content type? Json? Xml? Html?
Post reply on HN