Ask HN: Best way to distribute multi-GB dataset?

#1

Post by ggoodale » Mon, Dec 09, 2013, 9:27 PM UTC

I'm in the process of shutting down Word2[1], an MMO scrabble-like word game we built for Node Knockout back in 2010. The game's world is amazing - ~265MM tiles, ~123MM played words, over 1MM unique players. Once all personally identifiable information has been removed, I'd like to provide it to interested parties to play with. The clean dataset will be ~15GB compressed - what's the preferred way to share multi-GB files these days? Bittorrent, S3 (probably Requester Pays unless there's another option with manageable costs), or something else? Suggestions welcome.

[1] http://massivelyfun.com/saying-goodbye/

Re: Ask HN: Best way to distribute multi-GB dataset?

#2

Post by 0x006A » Mon, Dec 09, 2013, 9:36 PM UTC

Bittorrent

Re: Ask HN: Best way to distribute multi-GB dataset?

#3

Post by logn » Mon, Dec 09, 2013, 9:39 PM UTC

Sourceforge

Re: Ask HN: Best way to distribute multi-GB dataset?

#4

Post by ggoodale » Mon, Dec 09, 2013, 9:40 PM UTC

0x006A wrote: post #2

Bittorrent

Seems reasonable, though given the size of the file I'm probably going to have to host it somewhere for seeding purposes rather than just parking it on my home fileserver (yay Comcast monopoly in my area).

Re: Ask HN: Best way to distribute multi-GB dataset?

#5

Post by ggoodale » Mon, Dec 09, 2013, 9:40 PM UTC

logn wrote: post #3

Sourceforge

Interesting - hadn't considered that possibility. Thanks! I'll look into it.

Re: Ask HN: Best way to distribute multi-GB dataset?

#6

Post by markcarlson » Mon, Dec 09, 2013, 10:09 PM UTC

http://archiveteam.org

Re: Ask HN: Best way to distribute multi-GB dataset?

#7

Post by ggoodale » Mon, Dec 09, 2013, 10:14 PM UTC

markcarlson wrote: post #6

http://archiveteam.org

Rad! Haven't run across Archive Team before. Looking into them now.

Re: Ask HN: Best way to distribute multi-GB dataset?

#8

Post by davidbanham » Mon, Dec 09, 2013, 10:30 PM UTC

ggoodale wrote: post #4

0x006A wrote: post #2

Bittorrent

Seems reasonable, though given the size of the file I'm probably going to have to host it somewhere for seeding purposes rather than just parking it on my home fileserver (yay Comcast monopoly in my area).

How about a torrent hosted on s3?

http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.htm...

Re: Ask HN: Best way to distribute multi-GB dataset?

#9

Post by willmacdonald » Mon, Dec 09, 2013, 10:33 PM UTC

Bittorrent Sync

Re: Ask HN: Best way to distribute multi-GB dataset?

#10

Post by ggoodale » Tue, Dec 10, 2013, 10:17 PM UTC

davidbanham wrote: post #8

ggoodale wrote: post #4

Earlier quoted context omitted.

Seems reasonable, though given the size of the file I'm probably going to have to host it somewhere for seeding purposes rather than just parking it on my home fileserver (yay Comcast monopoly in my area).

How about a torrent hosted on s3? http://docs.aws.amazon.com/AmazonS3/latest/dev/S3Torrent.htm...

Looks like that only works for 5GB files or less, which is a shame.