Live data from Hacker News

Facebook crawls links in PDFs you send in Messenger

twitter.com

11–20 of 165 posts

Re: Facebook crawls links in PDFs you send in Messenger

#11

Huh, but why? I can totally understand scanning a PDF for links to look for malicious links to protect users. But that wouldn't involve actual HTTP requests to them. I'm struggling to imagine what purpose this could have.

How do you check if a link is serving up something terrible without http requests to them?

Re: Facebook crawls links in PDFs you send in Messenger

#12
post #3
post #2

Could someone effectively DOS another site using this method by including a bunch of links that generate a lot of load? Would be interesting to see if Facebook has a maximum number of links it'll follow.

I remember something like this was possible back in the day with Google Sheets. You could embed a URL as if it was an image in each cell of a sheet and it would make thousands of requests. I don't remember the details.

https://news.ycombinator.com/item?id=3890328

Re: Facebook crawls links in PDFs you send in Messenger

#13

Huh, but why? I can totally understand scanning a PDF for links to look for malicious links to protect users. But that wouldn't involve actual HTTP requests to them. I'm struggling to imagine what purpose this could have.

Probably anti-spam, particularly to catch groups of fake accounts sending the same or similar PDF.

Re: Facebook crawls links in PDFs you send in Messenger

#14
post #2

Could someone effectively DOS another site using this method by including a bunch of links that generate a lot of load? Would be interesting to see if Facebook has a maximum number of links it'll follow.

I'm sure you could just point all those links to a domain you own (on some poor, unsuspecting VPS) and see what happens? Might be playing with fire, though.

Re: Facebook crawls links in PDFs you send in Messenger

#15
post #5

Sidenote i wonder why FB doesnt launch a search engine since they crawl most of the web anyways

That would let people leave the Facebook platform and explore the open web.

I remember the sheer awe when I first learned there was a huge open web outside of AOL. I'm sure people nowadays are aware of the rest of the web, but if the draw is minimal, they will likely get stuck in the same loops of well-trodden space.

Re: Facebook crawls links in PDFs you send in Messenger

#18
post #10

Huh, but why? I can totally understand scanning a PDF for links to look for malicious links to protect users. But that wouldn't involve actual HTTP requests to them. I'm struggling to imagine what purpose this could have.

How do you know if they're malicious if you don't make HTTP requests to them? One of the things that phishers and others do is use link wrapping and other services to hide malicious links. So, I get something.wordpress.com/something-clean. I then put in an HTML or JS redirect on that page to something malicious. Given that browsers don't warn about HTTP, HTML, or JS redirects, it's an easy way for scammers to get aro…

But in this case, that doesn't help at all because facebook's crawler uses a predictable user agent string. You give a clean result to the facebook crawler and a malicious result to everyone else.

Re: Facebook crawls links in PDFs you send in Messenger

#19
post #5

Sidenote i wonder why FB doesnt launch a search engine since they crawl most of the web anyways

That would let people leave the Facebook platform and explore the open web.

I dont think they re worried much about that anymore , people always return, they ve established their position. OTOH, it would be nice for google to have some serious competition on the web, esp. considering that FB has a great NLP AI team.

Re: Facebook crawls links in PDFs you send in Messenger

#20
post #18
post #10

Earlier quoted context omitted.

How do you know if they're malicious if you don't make HTTP requests to them? One of the things that phishers and others do is use link wrapping and other services to hide malicious links. So, I get something.wordpress.com/something-clean. I then put in an HTML or JS redirect on that page to something malicious. Given that browsers don't warn about HTTP, HTML, or JS redirects, it's an easy way for scammers to get aro…

But in this case, that doesn't help at all because facebook's crawler uses a predictable user agent string. You give a clean result to the facebook crawler and a malicious result to everyone else.

That is a very good point. Security crawlers should probably use a masked user-agent.
Post reply on HN