The website doesn’t open for me.
Check your DNS, if you are running a pihole or something similar you will need to disable or allow this site.
Enable spamware? Big red flag nokthx
Facebook was used as a proxy by web scraping bots
11–20 of 125 posts
Re: Facebook was used as a proxy by web scraping bots
#12So this was a very long web page to say: Facebook forgot to rate limit their web scraper on a per user basis, but we told them and they fixed it.
Re: Facebook was used as a proxy by web scraping bots
#13You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…
> Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted.
Yup, and exposing just a key pieces of information (title, and some of the meta/og tags) without the body would limit the potential for abuse, while still being fairly useful for legitimate uses.
Re: Facebook was used as a proxy by web scraping bots
#14I've used Yahoo's YQL. While I would hit rate limits and other crap when trying to scape data of some sites directly. YQL would provide me nicely structured data without these stupid limits as many don't see yahoo's bot as a scraper.
Re: Facebook was used as a proxy by web scraping bots
#15Why can’t you just make your web crawler look like fb or googlebot (via user agent)?
Do website owners actually check the ip?
Re: Facebook was used as a proxy by web scraping bots
#16You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…
And unfortunately, cloudflare is everywhere. This trend will make it even harder for projects like a new search engine to enter the game.
Re: Facebook was used as a proxy by web scraping bots
#17You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…
Is this the case for any web crawler?
Re: Facebook was used as a proxy by web scraping bots
#18You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…
Long term, a new HTTP META method would be interesting. I wonder if something like that has ever been considered. Providers like Cloudflare would hopefully be more lenient with these requests.
Re: Facebook was used as a proxy by web scraping bots
#19Why can’t you just make your web crawler look like fb or googlebot (via user agent)? Do website owners actually check the ip?
Sounds like this company is checking that the ip is from facebook. That would probably work on less secure sites though.
Re: Facebook was used as a proxy by web scraping bots
#20You can't create your own link previewer, cloudflare will put a captcha in front of every website. All I want is a a freaking tag. They don't seem eager to fix it either, their proposed solution is to contact every website owner (seriously) to ask them to whitelist you[1]. Frankly, i wish facebook or cloudflare offered their previewer as a free service, since most websites have them whitelisted. 1. https://community.…
And unfortunately, cloudflare is everywhere. This trend will make it even harder for projects like a new search engine to enter the game.
Because if you don't have it some a-hole will go and ddos your site or you want to prevent a hug-of-death because of reasons.
It seems a lot of issues happen because bad players are continued to allowed to thrive, example: everybody uses a big provider because they're the only ones that solved the spam issue.