Wanna have zillions of google profiles? • Hacker News Board

Re: Wanna have zillions of google profiles?

#11

Post by datasink » Thu, Jul 22, 2010, 2:36 PM UTC

underdown wrote: post #6

I may be a little daft but how is this a big deal other than these files showing up in G's index?

A google search caps out at 2,000 results.

Re: Wanna have zillions of google profiles?

#12

Post by acangiano » Thu, Jul 22, 2010, 2:38 PM UTC

    curl -O http://www.gstatic.com/s2/sitemaps/sitemap-[000-3278].txt

Re: Wanna have zillions of google profiles?

#13

Post by avar » Thu, Jul 22, 2010, 2:44 PM UTC

Here's a Torrent with a tar.xz that contains all of them: http://v.nix.is/~leech/google-profiles.tar.xz.torrent

That's files sitemap-000 through sitemap-3278. They contain a total of 16,256,271 profile URLs.

Re: Wanna have zillions of google profiles?

#14

Post by retube » Thu, Jul 22, 2010, 2:45 PM UTC

blueberry wrote: post #10

underdown wrote: post #6

I may be a little daft but how is this a big deal other than these files showing up in G's index?

I don't think it's a big deal but I guess the author wanted to point out an easy way to crawl this public info. There are similar ways to crawl public Facebook profiles too, so it's really not a big deal.

Re facebook: really? I'd be interested in a hint here....

Re: Wanna have zillions of google profiles?

#15

Post by [deleted] » Thu, Jul 22, 2010, 2:49 PM UTC

[deleted]

Re: Wanna have zillions of google profiles?

#16

Post by Mark_B » Thu, Jul 22, 2010, 2:54 PM UTC

I wonder - what could be some non-evil things to do with easy access to all of this information?

Re: Wanna have zillions of google profiles?

#17

Post by cj » Thu, Jul 22, 2010, 3:00 PM UTC

NewSoftzzz wrote: post #5

Or easier.. http://www.google.com/#q=site%3Ahttp%3A%2F%2Fgoogle.com%2Fpr...

No, harder. Your link would require you to crawl thousands of Google results pagea, while the OP's link gives you sitemap URLs of many profiles at once.

Re: Wanna have zillions of google profiles?

#18

Post by ulrich » Thu, Jul 22, 2010, 3:01 PM UTC

retube wrote: post #14

blueberry wrote: post #10

Earlier quoted context omitted.

I don't think it's a big deal but I guess the author wanted to point out an easy way to crawl this public info. There are similar ways to crawl public Facebook profiles too, so it's really not a big deal.

Re facebook: really? I'd be interested in a hint here....

http://www.commandlinefu.com/commands/view/4726/view-faceboo...

Re: Wanna have zillions of google profiles?

#19

Post by fdb » Thu, Jul 22, 2010, 3:05 PM UTC

It's interesting that the profiles list the public followers and following stats. With some deep crawling, you could build a huge social graph from this data.

Re: Wanna have zillions of google profiles?

#20

Post by blueberry » Thu, Jul 22, 2010, 3:09 PM UTC

retube wrote: post #14

blueberry wrote: post #10

Earlier quoted context omitted.

I don't think it's a big deal but I guess the author wanted to point out an easy way to crawl this public info. There are similar ways to crawl public Facebook profiles too, so it's really not a big deal.

Re facebook: really? I'd be interested in a hint here....

browse through the family names at your convenience: http://facebook.com/family Search the public statuses of Facebook users (this is done via the API): http://youropenbook.org/