Also it would be smart to run malware check on these urls if they don't already doing it.
Hello Facebook Crawler
21–30 of 37 posts
Re: Hello Facebook Crawler
#22This reminds me of a recent experience I had with the Bing bot. This most recent YC round, my co-founder and I used Skydrive to edit our application. Skydrive integrates pretty nicely with Word, even on a Mac, to allow for collaborative editing. It's like the best parts of Sharepoint, minus all the crap, and inside of a modern UI. I'm a diehard Apple user, but I also subscribe to the "right tool for the job" principl…
I'd feel extremely violated.
I use google docs, very sparingly. One of the spreadsheets there contains a URL that is not linked from anywhere else and impossible to guess. If that URL ever gets tripped it will send me an email and the day that happens is the day I'll stop using google services (so far so good, and of course I should say 'google drive' now instead of 'google docs').
Re: Hello Facebook Crawler
#23Earlier quoted context omitted.
You assume they were indexing skydrive documents. It could well be that one of the people who visited the link had a Bing toolbar installed. Either way, all publicly accesible documents will get indexed sooner or later.
This was before the document had been sent to anyone. It was still being edited, only my friend and I were working on it. Also, the documents were not public.
I would be surprised if Microsoft is intentionally indexing links in private documents, but my point stands: Google et al are remarkably good at indexing the web. If you don't want an otherwise public URL indexed you must use robots.txt or equivalent.
Re: Hello Facebook Crawler
#2412 lines of code instead of:
tail -f /var/log/apache2/access.logRe: Hello Facebook Crawler
#25Why is this even news? Facebook has been crawling links for ages every time you post on the site. The crawler is how the link you paste gets a title, description, and sometimes a thumbnail.
"What to Submit On-Topic: Anything that good hackers would find interesting. That includes more than hacking and startups. If you had to reduce it to a sentence, the answer might be: anything that gratifies one's intellectual curiosity."
Wow I'm sick of people posting claims of off topic. And wow it's funny that someone has the patience to reply with the posting rules.
Re: Hello Facebook Crawler
#26This reminds me of a recent experience I had with the Bing bot. This most recent YC round, my co-founder and I used Skydrive to edit our application. Skydrive integrates pretty nicely with Word, even on a Mac, to allow for collaborative editing. It's like the best parts of Sharepoint, minus all the crap, and inside of a modern UI. I'm a diehard Apple user, but I also subscribe to the "right tool for the job" principl…
I'd feel extremely violated. I use google docs, very sparingly. One of the spreadsheets there contains a URL that is not linked from anywhere else and impossible to guess. If that URL ever gets tripped it will send me an email and the day that happens is the day I'll stop using google services (so far so good, and of course I should say 'google drive' now instead of 'google docs').
How would you know it was google indexing your document vs., say, your browser prefetching the link?
Re: Hello Facebook Crawler
#27Earlier quoted context omitted.
I'd feel extremely violated. I use google docs, very sparingly. One of the spreadsheets there contains a URL that is not linked from anywhere else and impossible to guess. If that URL ever gets tripped it will send me an email and the day that happens is the day I'll stop using google services (so far so good, and of course I should say 'google drive' now instead of 'google docs').
How would you know it was google indexing your document vs., say, your browser prefetching the link?
Because my browser has never looked at the document with the link in it. Obviously that would defeat the purpose.
Re: Hello Facebook Crawler
#28Earlier quoted context omitted.
There was a period where Hacker News consisted primarily of people on the right-hand side of the spectrum. People who were working inside of startups or had lots of experience with the web and our industry. Pretty much everyone knew what sharding was, and MongoDB wan't very popular. These days we've got a lot more people and they show up all across the board. Clearly if this is on the homepage, it was voted there by…
Even so, for those who are up to that point, that headline could give the implication that Facebook is getting into the search business.
That's why you should read the articles and not just the headlines. Headlines more often than not give a wrong impression.
Re: Hello Facebook Crawler
#29By looking at the headers you now have a great way of writing some analytics tools to see how much your website is shared on Facebook...
If you register your site with Facebook, you can get that information on Facebook's end:
https://developers.facebook.com/docs/opengraphprotocol/
under "Domain Insights."
Re: Hello Facebook Crawler
#30I would imagine they're checking the URL for malware as well.
Probably, I've seen then ban whole domains (droplr.com) previously for distributing malware.