Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Facebook crawls links in PDFs you send in Messenger

twitter.com

Post reply on HN
Print view View original

21–30 of 165 posts

Re: Facebook crawls links in PDFs you send in Messenger

#23

Post by seanieb »

gruez wrote: post #18
mdasen wrote: post #10

Earlier quoted context omitted.

How do you know if they're malicious if you don't make HTTP requests to them? One of the things that phishers and others do is use link wrapping and other services to hide malicious links. So, I get something.wordpress.com/something-clean. I then put in an HTML or JS redirect on that page to something malicious. Given that browsers don't warn about HTTP, HTML, or JS redirects, it's an easy way for scammers to get aro…

But in this case, that doesn't help at all because facebook's crawler uses a predictable user agent string. You give a clean result to the facebook crawler and a malicious result to everyone else.

There are services to frawl for you from miltipke ips and user agents, just for situations like this.

Re: Facebook crawls links in PDFs you send in Messenger

#24

Post by marksomnian »

idoco wrote: post #20
gruez wrote: post #18

Earlier quoted context omitted.

But in this case, that doesn't help at all because facebook's crawler uses a predictable user agent string. You give a clean result to the facebook crawler and a malicious result to everyone else.

That is a very good point. Security crawlers should probably use a masked user-agent.

I'm fairly sure Google's search crawler already uses a masked UA, to detect when pages serve it different content than they do to users.

Re: Facebook crawls links in PDFs you send in Messenger

#25

Post by tialaramex »

austhrow743 wrote: post #11
crazygringo wrote: post #6

Huh, but why? I can totally understand scanning a PDF for links to look for malicious links to protect users. But that wouldn't involve actual HTTP requests to them. I'm struggling to imagine what purpose this could have.

How do you check if a link is serving up something terrible without http requests to them?

You _could_ ask a service like Google Safe Search

Just in case you didn't follow any of the previous HN discussion of how that's done

consider the URL https://accounts.example.com/tmp/badmojo.exe

You (Facebook in this case) run a hypothetical method SafeSearch('accounts.example.com') and also SafeSearch('example.com') and SafeSearch('accounts.example.com/tmp') and SafeSearch('accounts.example.com/tmp/badmojo.exe')

SafeSearch(string) is defined as, you do SHA(string) and that's your hash, you compare the start of this hash to a huge list of prefixes that Google provides, which you fetch updates for every few minutes. If there's no match, fine, done. If there's a match you ask Google OK, I saw this Prefix you sent me, what hashes should I be scared of? Google gives you a list of hashes with that Prefix. If your hash in this new list, the original URL was scary, warn users not to visit, otherwise continue what you were doing.

Post reply on HN