Hello Facebook Crawler
mwmeyer.com
Hello Facebook Crawler
1–10 of 37 posts
Re: Hello Facebook Crawler
#2By looking at the headers you now have a great way of writing some analytics tools to see how much your website is shared on Facebook...
Re: Hello Facebook Crawler
#3By looking at the headers you now have a great way of writing some analytics tools to see how much your website is shared on Facebook...
I would imagine that they cache the page contents and hence hit a URL only once in a certain period of time, thus skewing any analytics built around this.
Re: Hello Facebook Crawler
#4By looking at the headers you now have a great way of writing some analytics tools to see how much your website is shared on Facebook...
I would imagine that they cache the page contents and hence hit a URL only once in a certain period of time, thus skewing any analytics built around this.
Yeah it's cached by Facebook. That's why if you want to change your meta and/or open graph tags info, you need to feed your page to Facebook's Url Linter (https://developers.facebook.com/tools/lint/).
Re: Hello Facebook Crawler
#5Why is this even news? Facebook has been crawling links for ages every time you post on the site. The crawler is how the link you paste gets a title, description, and sometimes a thumbnail.
Re: Hello Facebook Crawler
#6This reminds me of a recent experience I had with the Bing bot.
This most recent YC round, my co-founder and I used Skydrive to edit our application. Skydrive integrates pretty nicely with Word, even on a Mac, to allow for collaborative editing. It's like the best parts of Sharepoint, minus all the crap, and inside of a modern UI. I'm a diehard Apple user, but I also subscribe to the "right tool for the job" principle ... in this case it worked pretty well.
Anyway, inside the document were links to some private areas of our website that contained demo materials for YC. As requested, they were not password protected, but also not linked from anywhere else. While submitting I ensured that our nginx logs would capture visits to these URL's in a separate log, so we'd know when it was being looked at (sidenote, seeing visitors coming from inside justin.tv + the rincon hill towers is kind of exhilarating).
What surprised me was that almost immediately after we began working on the document, the Bing bot was going apeshit exploring the domain and the 'private' URL's. I had to quickly add a robots.txt to deny all on the root. I thought it was pretty interesting. At first I felt almost violated. But then it seems logical that they'd be indexing every URL in every document stored in their datacenter, why not?
Re: Hello Facebook Crawler
#7Why is this even news? Facebook has been crawling links for ages every time you post on the site. The crawler is how the link you paste gets a title, description, and sometimes a thumbnail.
"What to Submit
On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity."
Re: Hello Facebook Crawler
#8Why is this even news? Facebook has been crawling links for ages every time you post on the site. The crawler is how the link you paste gets a title, description, and sometimes a thumbnail.
+1. not sure how a post like this can make it to the front page.
Re: Hello Facebook Crawler
#9Why is this even news? Facebook has been crawling links for ages every time you post on the site. The crawler is how the link you paste gets a title, description, and sometimes a thumbnail.
+1. not sure how a post like this can make it to the front page.
There was a period where Hacker News consisted primarily of people on the right-hand side of the spectrum. People who were working inside of startups or had lots of experience with the web and our industry. Pretty much everyone knew what sharding was, and MongoDB wan't very popular.
These days we've got a lot more people and they show up all across the board.
Clearly if this is on the homepage, it was voted there by your peers. This kind of knowledge is completely obvious to many of us, but not everyone is on your level. Cut 'em some slack.
Re: Hello Facebook Crawler
#10This reminds me of a recent experience I had with the Bing bot. This most recent YC round, my co-founder and I used Skydrive to edit our application. Skydrive integrates pretty nicely with Word, even on a Mac, to allow for collaborative editing. It's like the best parts of Sharepoint, minus all the crap, and inside of a modern UI. I'm a diehard Apple user, but I also subscribe to the "right tool for the job" principl…
Eh, I'm pretty sure you should still feel violated. The fact that they are parsing your private documents for information that they can use to help another business unit is really sketchy. It would make me wonder what else they are scanning my data for.
Personally, I'll never use an MS cloud service because of this anecdote - not that it was that likely to begin with.