Academic Torrents
academictorrents.com
Academic Torrents
1–10 of 33 posts
Re: Academic Torrents
#22016 HN discussion: https://news.ycombinator.com/item?id=12381791
2014 HN discussion: https://news.ycombinator.com/item?id=7149006
Re: Academic Torrents
#3Cool idea, it is impressive that it is still around - alas it is flawed the same way all scientific data is flawed.
There is no metadata - all you have is an awkward imprecise textual search of the abstract that comes with the data. Good luck hosting the world's data that way.
Re: Academic Torrents
#4Cool idea, it is impressive that it is still around - alas it is flawed the same way all scientific data is flawed. There is no metadata - all you have is an awkward imprecise textual search of the abstract that comes with the data. Good luck hosting the world's data that way.
One nice thing about digital data, as opposed to physical artefacts, is that you don’t need to keep digital data’s metadata attached to the data “at the hip.”
Through the magic of cryptographic hash algorithms, you can just keep your data sets floating around “raw” (like in these torrents), and then, elsewhere, ascribe metadata to the hash of the content it is meant to annotate.
Then, later, you can reassemble them in either order—either by first finding a data set, hashing it, and then looking up metadata in some metadata-hosting service; or by first browsing a catalogue of indexed metadata, finding out about a dataset that meets your needs, and then retrieving the data set by its hash.
Which is to say: with digital data, library science (creating metadata and chains-of-custody and indexing them for search) and archiving (ensuring access to pristine artifacts over time) don’t need to happen at the same time, in the same place. There can be separate “artifact hosting” and “metadata library” services. (Which is especially helpful in contexts where private IP is involved—you can still keep in your metadata library, the metadata for a data-set you don’t have the rights to; and those with the rights can go get the data-set themselves.)
Re: Academic Torrents
#5This is amazing, really a great source of data.
Re: Academic Torrents
#6Cool idea, it is impressive that it is still around - alas it is flawed the same way all scientific data is flawed. There is no metadata - all you have is an awkward imprecise textual search of the abstract that comes with the data. Good luck hosting the world's data that way.
One nice thing about digital data, as opposed to physical artefacts, is that you don’t need to keep digital data’s metadata attached to the data “at the hip.” Through the magic of cryptographic hash algorithms, you can just keep your data sets floating around “raw” (like in these torrents), and then, elsewhere , ascribe metadata to the hash of the content it is meant to annotate. Then, later, you can reassemble them…
Aaaand someone has to do the work for computing the index and annotating the hashes.
Re: Academic Torrents
#7Cool idea, it is impressive that it is still around - alas it is flawed the same way all scientific data is flawed. There is no metadata - all you have is an awkward imprecise textual search of the abstract that comes with the data. Good luck hosting the world's data that way.
There is metadata. It is stored in bibtex along with every torrent. This format allows it to be a freeform database where the user can add fields as they want. We (Academic Torrents) can then build new ways to display this metadata. Also the "abstract" part of the metadata is rendered as markdown on the details page of a torrent. Here is a good example: https://academictorrents.com/details/d52ccc21455c7a82fd6e589...
Re: Academic Torrents
#8https://academictorrents.com/about.php#mirroring
Using RSS to allow mirrors to host different subjects is really clever, although some of the categories seem quite large (>5TB). It may be worth breaking up each category (sharding) to keep each to 100GB or less so a volunteer can pick a couple and not worry about running out of disk when a category grows.
Then it would be good to track how many seeds each category-shard has so volunteers can help where it's most needed.
Re: Academic Torrents
#9Cool idea, it is impressive that it is still around - alas it is flawed the same way all scientific data is flawed. There is no metadata - all you have is an awkward imprecise textual search of the abstract that comes with the data. Good luck hosting the world's data that way.
One nice thing about digital data, as opposed to physical artefacts, is that you don’t need to keep digital data’s metadata attached to the data “at the hip.” Through the magic of cryptographic hash algorithms, you can just keep your data sets floating around “raw” (like in these torrents), and then, elsewhere , ascribe metadata to the hash of the content it is meant to annotate. Then, later, you can reassemble them…
This flexibility in time, specialization, and order of operation is surely one of the joys of modern digital collections.
Library scientists might say archiving and structuring and curation are all facets of that science. And you'll also want a hash search engine that finds related hashes, as there can be many revisions + versions, only some of which have some metadata.
Re: Academic Torrents
#10Earlier quoted context omitted.
One nice thing about digital data, as opposed to physical artefacts, is that you don’t need to keep digital data’s metadata attached to the data “at the hip.” Through the magic of cryptographic hash algorithms, you can just keep your data sets floating around “raw” (like in these torrents), and then, elsewhere , ascribe metadata to the hash of the content it is meant to annotate. Then, later, you can reassemble them…
Aaaand someone has to do the work for computing the index and annotating the hashes.
I think it's worth recognizing that this is a good first step in a hard problem. Hosting many TB of data for free isn't easy. Building an index on top of that data isn't easy either, and it looks like no such index exists today, but if someone decided to build that index they wouldn't need to worry about the hosting portion of the problem. That's a great starting point.