Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Academic Torrents

academictorrents.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: Academic Torrents

#51

Post by lojack »

rodolphoarruda wrote: post #47
pixelcort wrote: post #16

Earlier quoted context omitted.

BitTorrent Sync might be useful for that: http://www.bittorrent.com/sync

I'd used BT Sync for a couple of weeks to sync data between my own machines. It works neatly. One question here. When you modify some part of a big file, does the program send out only the difference to the other authorized machines, or entire file? Let's say a researcher exports her data to a 1GB CSV file of my interest. I download it. In the following week the same researcher updates her CVS with more data, now it…

Seems as though it supports patching so only the parts that are changed would be synced. Of course, the download size is completely dependent on what parts of the file were changed.

http://forum.bittorrent.com/topic/19228-latest-sync-build-11...

Re: Academic Torrents

#53

Post by nl »

_delirium wrote: post #27
cing wrote: post #7

The team should learn from the ghost-town that is BioTorrents[1] and offer more than just a tracker. [1] http://www.biotorrents.net/browse.php?incldead=1

That's one reason I'd prefer that academics just put data into some kind of local university archive, where possible. Many universities provide resources to host scientific data (and have done so for decades, since the days of ftp.dept.university.edu servers), and putting it there makes it more likely that it'll still be there in 10 years. Torrents by comparison tend to be: 1) slow, as you rely on random seeders rath…

I agree about university data.

But there is a need for a way to distributed large datasets that come out of nonacademic projects.

For example, the DBPedia data dumps are very slow to download at the moment.

Re: Academic Torrents

#56

Post by axman6 »

sillysaurus2 wrote: post #11

One problem with offering a dataset as a torrent is that it's impossible to edit it after it's released. However, it seems like that doesn't matter at all in this case, because any scenario I can think of which could be solved by editing the dataset (like redacting private info that was accidentally included) wouldn't avoid the original problem: that they accidentally released private info in the first place. Perhaps…

If it's stored on ZFS, Copy on Write will let you edit a copy that only stores the changed files, and deduplication could give back even more space (if necessary and RAM permits).

Re: Academic Torrents

#57

Post by lancemjoseph »

Many of the datasets that I've seen in academia are stored in static SQL databases that tend to be about 10-20 terabytes. Where does this leave individuals with limited resources who would like to query large databases without having to juggle the data management side of research? Are there softwares that make database querying P2P accessible?

Re: Academic Torrents

#59

Post by dnautics »

sillysaurus2 wrote: post #11

One problem with offering a dataset as a torrent is that it's impossible to edit it after it's released. However, it seems like that doesn't matter at all in this case, because any scenario I can think of which could be solved by editing the dataset (like redacting private info that was accidentally included) wouldn't avoid the original problem: that they accidentally released private info in the first place. Perhaps…

One problem with offering a dataset as a torrent is that it's impossible to edit it after it's released.

for scientific endeavours, this should be considered a feature, not a bug.

Post reply on HN