Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wanna have zillions of google profiles?

google.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Wanna have zillions of google profiles?

#21

Post by pierrefar »

blueberry wrote: post #7
pierrefar wrote: post #2

The odd thing, is that getting a list of all profiles correctly doesn't wor. If you go to Google's robots.txt and scroll to the bottom to find the profiles sitemap is located at: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml Which 404s. Luckily, it's quite easy to reverse engineer. Start with: http://www.gstatic.com/s2/sitemaps/sitemap-000.txt and increment 000 till you 404. From quick testing, it's less th…

As of this comment it's exactly 3278. I wonder why you didn't bother doing 3-4 more trials before writing the comment :) http://www.gstatic.com/s2/sitemaps/sitemap-3278.txt

Because I spent too much time trying to figure out why the index sitemap XML file was returning a 404. I'm really surprised that Google, one of the inventors of sitemaps and big proponent of using them, can have such a high-profile error. I dug around to see if I could find what's going on.

By the time I got to figuring out how many text files there are, I was getting bored.

Re: Wanna have zillions of google profiles?

#24

Post by chronomex »

ulrich wrote: post #18
retube wrote: post #14

Earlier quoted context omitted.

Re facebook: really? I'd be interested in a hint here....

http://www.commandlinefu.com/commands/view/4726/view-faceboo...

I have my Facebook profile closed down as tightly as Zuckerberg will allow, so this doesn't return anything for me.

Re: Wanna have zillions of google profiles?

#25

Post by chronomex »

ulrich wrote: post #18
retube wrote: post #14

Earlier quoted context omitted.

Re facebook: really? I'd be interested in a hint here....

http://www.commandlinefu.com/commands/view/4726/view-faceboo...

I have my Facebook profile closed down as tightly as Zuckerberg will allow, so this doesn't return anything for me.

Re: Wanna have zillions of google profiles?

#27

Post by thamer »

ulrich wrote: post #18
retube wrote: post #14

Earlier quoted context omitted.

Re facebook: really? I'd be interested in a hint here....

http://www.commandlinefu.com/commands/view/4726/view-faceboo...

Have you tried it? This was posted in January, but doesn't seem to work anymore.

    curl -A Opera 'http://www.facebook.com/ajax/typeahead_friends.php?u=4&__a=1' 
    for (;;);{"error":0,"errorSummary":"","errorDescription":"","errorIsWarning":false,"silentError":0,"payload":{"friends":[]}}

Re: Wanna have zillions of google profiles?

#28

Post by joshfraser »

blueberry wrote: post #10
underdown wrote: post #6

I may be a little daft but how is this a big deal other than these files showing up in G's index?

I don't think it's a big deal but I guess the author wanted to point out an easy way to crawl this public info. There are similar ways to crawl public Facebook profiles too, so it's really not a big deal.

pete warden tried that and got sued by facebook. they have since changed their robots.txt to forbid public crawling of their site although they still want the google juice so they make exceptions for major search engines.
Post reply on HN