Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hello Facebook Crawler

mwmeyer.com

Post reply on HN
Print view View original

31–37 of 37 posts

Re: Hello Facebook Crawler

#31

Post by egfx »

mars wrote: post #8
cddotdotslash wrote: post #5

Why is this even news? Facebook has been crawling links for ages every time you post on the site. The crawler is how the link you paste gets a title, description, and sometimes a thumbnail.

+1. not sure how a post like this can make it to the front page.

Hmm, I thought people mainly knew about this. But I researched this a lot while making http://2fb.me and actually witnessed this myself with google docs but didn't think twice about it. I'm sure the same thought was going through the heads of everybody reading this that knew anything about how the Facebook sharer worked. For those that found this news, it's beneficial to put open graph meta tags on your pages to control the crawler and you can also invoke the crawler (http://developers.facebook.com/tools/debug) when your page changes before Facebook updates it automatically every 2 weeks or so.

Re: Hello Facebook Crawler

#32

Post by drivebyacct2 »

eli wrote: post #23
whalesalad wrote: post #20

Earlier quoted context omitted.

This was before the document had been sent to anyone. It was still being edited, only my friend and I were working on it. Also, the documents were not public.

I would be surprised if Microsoft is intentionally indexing links in private documents, but my point stands: Google et al are remarkably good at indexing the web. If you don't want an otherwise public URL indexed you must use robots.txt or equivalent.

>If you don't want an otherwise public URL indexed you must use robots.txt or equivalent.

Which only blocks bots that respect the file...

Re: Hello Facebook Crawler

#33

Post by mindctrl »

The input box is a keylogger that sends what you type to FB before you ever press enter. That's the interesting story here. Everyone already knew FB had a scraper for retrieving images and meta data. What they probably didn't know is every keystroke inside that box is logged and sent, no Enter required.

Re: Hello Facebook Crawler

#34

Post by k3n »

cddotdotslash wrote: post #5

Why is this even news? Facebook has been crawling links for ages every time you post on the site. The crawler is how the link you paste gets a title, description, and sometimes a thumbnail.

Do note, and I'm not sure if this differs from what you're referring to, but that the link was never even posted to the site; rather, it was placed into a chat box but never sent. Small difference but I think it's an interesting point.

> To my bemusement, not only was the friend I was messaging away, I also hadn't even sent the link; I pasted it into the chat window but forgot to hit enter.

Re: Hello Facebook Crawler

#35

Post by cddotdotslash »

k3n wrote: post #34
cddotdotslash wrote: post #5

Why is this even news? Facebook has been crawling links for ages every time you post on the site. The crawler is how the link you paste gets a title, description, and sometimes a thumbnail.

Do note, and I'm not sure if this differs from what you're referring to, but that the link was never even posted to the site; rather, it was placed into a chat box but never sent. Small difference but I think it's an interesting point. > To my bemusement, not only was the friend I was messaging away, I also hadn't even sent the link; I pasted it into the chat window but forgot to hit enter.

You're right, but it also happens when you paste in the status button before hitting "Post." Try this: go to Facebook, post a link in your status box. Wait a few seconds. Notice that it will populate the link information fields even before you submit the post. It has to get that info from somewhere. That's where the crawler comes in.

Re: Hello Facebook Crawler

#36

Post by k3n »

cddotdotslash wrote: post #35
k3n wrote: post #34

Earlier quoted context omitted.

Do note, and I'm not sure if this differs from what you're referring to, but that the link was never even posted to the site; rather, it was placed into a chat box but never sent. Small difference but I think it's an interesting point. > To my bemusement, not only was the friend I was messaging away, I also hadn't even sent the link; I pasted it into the chat window but forgot to hit enter.

You're right, but it also happens when you paste in the status button before hitting "Post." Try this: go to Facebook, post a link in your status box. Wait a few seconds. Notice that it will populate the link information fields even before you submit the post. It has to get that info from somewhere. That's where the crawler comes in.

Yeah that's expected, but it's not expected (at least for me) to happen in a chat. But, I never use FB chat so I don't know -- does it also create thumbnails for links and such?

Re: Hello Facebook Crawler

#37

Post by cddotdotslash »

k3n wrote: post #36
cddotdotslash wrote: post #35

Earlier quoted context omitted.

You're right, but it also happens when you paste in the status button before hitting "Post." Try this: go to Facebook, post a link in your status box. Wait a few seconds. Notice that it will populate the link information fields even before you submit the post. It has to get that info from somewhere. That's where the crawler comes in.

Yeah that's expected, but it's not expected (at least for me) to happen in a chat. But, I never use FB chat so I don't know -- does it also create thumbnails for links and such?

Hmm, I see what you're saying; no it doesn't appear to create the thumbnail in chat. It would be interesting if Facebook uses different crawlers depending on whether the link is posted in the status box or in a chat. That could lead to some interesting analytics such as "your website was chatted about x number of times and shared via a status update x times."
Post reply on HN