Facebook was used as a proxy by web scraping bots • Hacker News Board

Re: Facebook was used as a proxy by web scraping bots

#121

Post by anuila » Fri, Nov 06, 2020, 6:29 PM UTC

navanchauhan wrote: post #104

anuila wrote: post #80

Earlier quoted context omitted.

There are a lot of countries where Facebook is “free”, for example India and Philippines.

Free Basics was not allowed in India by the Telecom Regulatory Authority of India[0]. The list of countries where Free Basics is currently operated is listed on Internet.org website[1]. [0] https://www.theverge.com/2016/2/8/10913398/free-basics-india... [1] https://info.internet.org/en/story/where-weve-launched/

Indeed, I must have confused it with Bangladesh, I visited both in the same month. Thanks for linking to the actual list.

Re: Facebook was used as a proxy by web scraping bots

#122

Post by 67868018 » Sat, Nov 07, 2020, 2:45 AM UTC

mtberatwork wrote: post #117

bgirard wrote: post #18

Earlier quoted context omitted.

Long term, a new HTTP META method would be interesting. I wonder if something like that has ever been considered. Providers like Cloudflare would hopefully be more lenient with these requests.

Doesn't the oembed spec [1] already solve this? I think the OP could solve their problem by simply creating an oembed endpoint with all the necessary meta data. [1] https://oembed.com/

Yea but when your request to fetch the oembed data is blocked by a CAPTCHA...

This is a real problem, we experience it in the Fediverse

Re: Facebook was used as a proxy by web scraping bots

#123

Post by 67868018 » Sat, Nov 07, 2020, 2:48 AM UTC

thatfunkymunki wrote: post #92

seniorgarcia wrote: post #69

Earlier quoted context omitted.

What meta tags do I have to fill and why is Twitters/FBs preview suddenly my problem? > https://developer.twitter.com/en/docs/twitter-for-websites/c... So, I should have to include twitter specific meta tags even though I personally don't care about twitter? Maybe twitter should make it clear which tags they read? Maybe it's SEO bullshit I don't care about? Maybe even even the OG: tags don't work all the time and res…

If you don't want to fill them out, don't... Filling them out lets you customize your link preview on twitter. If you don't care about Twitter, why would this affect you at all?

They're used by instant messengers too: slack, iMessage, WhatsApp, telegram, signal, ...

Re: Facebook was used as a proxy by web scraping bots

#124

Post by beagle3 » Sat, Nov 07, 2020, 7:58 AM UTC

SiempreViernes wrote: post #86

sfifs wrote: post #79

Earlier quoted context omitted.

Well if you have an easy solution that you think would work, why don't you put up a website, commission a DDOS attack from a skilled actor and try to demonstrate mitigation? Companies pay big money to CloudFlare. If a simpler and cheaper solution is workable, they'll pay you instead.

Just like telling if it's raining is easy but stopping rain once has started is hard, the claim is that it's not hard to detect if a site is being ddosed.

It is not at all easy to tell the difference between a DDoS and the slashdot effect (or HN hug of death, depending on your age). At least not without a man in the loop.

Re: Facebook was used as a proxy by web scraping bots

#125

Post by samoa42 » Sat, Nov 07, 2020, 8:14 AM UTC

sokoloff wrote: post #118

qwerty456127 wrote: post #46

Earlier quoted context omitted.

There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index…

Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler? They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this pr…

i think there is a line somewhere along with beeing accountable in a business sense.

ie. if your internet host just hands out information, you are free to block/throttle as you please.

as soon as you are taking money (operate as a business), you are accountable and must not discriminate.

so: > Does a website operator have to handle whatever arbitrary traffic you want to throw at them

absolutely, yes!