Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Facebook crawler was making 7M requests per day to my stupid website

coding.napolux.com

Post reply on HN
Print view View original

51–60 of 416 posts

Re: A Facebook crawler was making 7M requests per day to my stupid website

#52

Post by napolux »

k2xl wrote: post #5

This is actually a common tactic from malicious actors - pretend to be other bots to get through to websites. If you reverse lookup the IP address you can see if it part of the facebook network. UA is very untrustworthy

Here is another one https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...

Re: A Facebook crawler was making 7M requests per day to my stupid website

#53

Post by sradman »

Interesting. According to the Facebook Crawler link provided by the OP [1], it makes a range request of compressed resources. I wonder if the response from the OP’s PHP/SQLite app isn’t triggering a crawler bug.

Maybe Cloudflare can step up and act as an intermediary to isolate the problem. Isolating DoS attacks is one of their comparative advantages.

[1] https://developers.facebook.com/docs/sharing/webmasters/craw...

Re: A Facebook crawler was making 7M requests per day to my stupid website

#56

Post by mikece »

systemvoltage wrote: post #13
youeseh wrote: post #7

Earlier quoted context omitted.

When you paste in a URL to share it with your friends, Facebook tries to grab some information from that webpage to provide a summary.

This pattern needs to die. Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise. Messaging apps have no business in looking up the URL. Just let it pass as a link. Fuck everything about this and we need to push back on this nonsense.

A funny story (well, funny because it didn't happen to me) was chronicled on a recent episode of Corey Quinn's "Whiteboard Confessions" -- https://www.lastweekinaws.com/podcast/aws-morning-brief/whit... -- where Slackbot's auto URL unfurling feature "clicked" an SNS alert unsubscribe link when a tech posted the a report including that link into a slack channel. If nobody lost their job over this triggering a SEV-1 alert I'm sure they had a laugh about it later.

Another issue with auto-unfurling links or generating previews of URLs is the potential for stalking. If I'm hunting my ex and I know their cell phone number AND that they have an iPhone I can simply send a link to a website and let iOS generate the preview... which then pings the URL of the website which I set up and I can now run geo-IP resolution and narrow down where my ex is (and maybe I follow up with a well-crafted PDF exploit when I'm ready to narrow down even more on an area).

While systemvoltage was a bit brusque in how the complaint was worded above, these "friendly and helpful" features of Slack, SMS, and other apps can have devastating unforeseen consequences. I question the value of having to opt out of these features instead of making them opt-in instead.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#57

Post by napolux »

hinkley wrote: post #45
spiderfarmer wrote: post #2

Was it a real FB crawler or a random buggy homemade bot masquerading as one? Check here: https://developers.facebook.com/docs/sharing/webmasters/craw...

Someone up to no good using a false flag would not surprise me in the slightest. "who do people already like to hate? I'll pretend to be them."

Doesn't seem the case https://apps.db.ripe.net/db-web-ui/query?searchtext=2a03:288...

Re: A Facebook crawler was making 7M requests per day to my stupid website

#58

Post by elviswolcott »

nitrogen wrote: post #18
systemvoltage wrote: post #13

Earlier quoted context omitted.

This pattern needs to die. Whenever I paste a link in any sort of messaging app - iMessages to Slack - it puts a thumbnail and summary, polluting the entire conversation with tons of noise. Messaging apps have no business in looking up the URL. Just let it pass as a link. Fuck everything about this and we need to push back on this nonsense.

I like link previews. I don't like when they are managed server-side instead of client-side.

The issue with doing them client side (other than the missed cache opportunity of using a server) is that it means you're having every user in the chat send a request to an arbitrary URL from their device. Also CORS makes this impossible 99% of the time.

Re: A Facebook crawler was making 7M requests per day to my stupid website

#60

Post by lixtra »

crazygringo wrote: post #26
nitrogen wrote: post #18

Earlier quoted context omitted.

I like link previews. I don't like when they are managed server-side instead of client-side.

Caching a link's title and thumbnail server-side will save your site potentially millions of requests from Facebook and elsewhere. Seems like a benefit for site owners to me. Not to mention that the server is reducing the image size of a thumbnail, potentially converting from HTTP to HTTPS, and so on.

> Seems like a benefit for site owners to me.

Maybe, but it’s definitely less control for the site owner.

Maybe she wants to do analytics on the previews (how far shared, which ip regions) or delete the content at a specific time. Etc.

Post reply on HN