Live data from Hacker News

Academic Torrents

academictorrents.com

11–20 of 33 posts

Re: Academic Torrents

#11
post #7
post #3

Cool idea, it is impressive that it is still around - alas it is flawed the same way all scientific data is flawed. There is no metadata - all you have is an awkward imprecise textual search of the abstract that comes with the data. Good luck hosting the world's data that way.

There is metadata. It is stored in bibtex along with every torrent. This format allows it to be a freeform database where the user can add fields as they want. We (Academic Torrents) can then build new ways to display this metadata. Also the "abstract" part of the metadata is rendered as markdown on the details page of a torrent. Here is a good example: https://academictorrents.com/details/d52ccc21455c7a82fd6e589...

Ok, I see that there is code provided there. Better than nothing but geez, it is not really what metadata should be like

  def get_labels(rightside):
    met = {}
    met['brain'] = (
        1. * (rightside != 0).sum() / (rightside == 0).sum())
    met['tumor'] = (
        1. * (rightside > 2).sum() / ((rightside != 0).sum() + 1e-10))
    met['has_enough_brain'] = met['brain'] > 0.30
    met['has_tumor'] = met['tumor'] > 0.01
    return met
I will say that it is very handy to know exactly how the labels were computed.

What I really meant is a way to search and select data based on metadata. For example has_tumor.

Also note how everything is still one single blob, to get one line of any of the files, one would need to download everything.

Re: Academic Torrents

#12
Yeah, it could really benefit from some organizational work, like on more mature music torrent trackers or such. Categories, mandatory tags, unified names, reviewed by community-chosen category-wise moderators. In it's current state in's basically a file dump, either you have the direct link, or you can only hope to find something interesting. Not that much better than sharing magnet links via public pastebin records...

Re: Academic Torrents

#13
post #11
post #7

Earlier quoted context omitted.

There is metadata. It is stored in bibtex along with every torrent. This format allows it to be a freeform database where the user can add fields as they want. We (Academic Torrents) can then build new ways to display this metadata. Also the "abstract" part of the metadata is rendered as markdown on the details page of a torrent. Here is a good example: https://academictorrents.com/details/d52ccc21455c7a82fd6e589...

Ok, I see that there is code provided there. Better than nothing but geez, it is not really what metadata should be like def get_labels(rightside): met = {} met['brain'] = ( 1. * (rightside != 0).sum() / (rightside == 0).sum()) met['tumor'] = ( 1. * (rightside > 2).sum() / ((rightside != 0).sum() + 1e-10)) met['has_enough_brain'] = met['brain'] > 0.30 met['has_tumor'] = met['tumor'] > 0.01 return met I will say that…

Bittorrent does support partial downloads that request only some files or byte ranges out of a torrent. Some of the torrents are just compressed zip's but for the others you could look at the code / documentation to see which files were relevant before downloading 10GB of data.

I think the abstract is sufficient for searching data; expecting some kind of smart database that can handle all the weird formats science uses is a bit much.

Re: Academic Torrents

#15
post #11
post #7

Earlier quoted context omitted.

There is metadata. It is stored in bibtex along with every torrent. This format allows it to be a freeform database where the user can add fields as they want. We (Academic Torrents) can then build new ways to display this metadata. Also the "abstract" part of the metadata is rendered as markdown on the details page of a torrent. Here is a good example: https://academictorrents.com/details/d52ccc21455c7a82fd6e589...

Ok, I see that there is code provided there. Better than nothing but geez, it is not really what metadata should be like def get_labels(rightside): met = {} met['brain'] = ( 1. * (rightside != 0).sum() / (rightside == 0).sum()) met['tumor'] = ( 1. * (rightside > 2).sum() / ((rightside != 0).sum() + 1e-10)) met['has_enough_brain'] = met['brain'] > 0.30 met['has_tumor'] = met['tumor'] > 0.01 return met I will say that…

| one would need to download everything

Just download it then. We got mp3 albums off Napster on modems back in the day, surely getting that torrent is easier and faster today.

Re: Academic Torrents

#16
post #12

Yeah, it could really benefit from some organizational work, like on more mature music torrent trackers or such. Categories, mandatory tags, unified names, reviewed by community-chosen category-wise moderators. In it's current state in's basically a file dump, either you have the direct link, or you can only hope to find something interesting. Not that much better than sharing magnet links via public pastebin records…

One very interesting thing I wish would be studied in depth are the virtual economies of mature trackers. Limiting access to resources and granting increasing access for contributing and correcting quality has in places been extremely successful. It is interesting to see the varying quality and associated economic mechanics.

Some environments, based just on prestige, have big problems with toxicity (StackOverflow, Wikipedia) which I didn't see at all in some music trackers.

Re: Academic Torrents

#17
post #12

Yeah, it could really benefit from some organizational work, like on more mature music torrent trackers or such. Categories, mandatory tags, unified names, reviewed by community-chosen category-wise moderators. In it's current state in's basically a file dump, either you have the direct link, or you can only hope to find something interesting. Not that much better than sharing magnet links via public pastebin records…

One very interesting thing I wish would be studied in depth are the virtual economies of mature trackers. Limiting access to resources and granting increasing access for contributing and correcting quality has in places been extremely successful. It is interesting to see the varying quality and associated economic mechanics. Some environments, based just on prestige, have big problems with toxicity (StackOverflow, Wi…

Wikipedia does cover that issue. Competing views are difficult to reconcile.

https://en.wikipedia.org/w/index.php?title=Wikipedia:Systemi...

(using a version of the article from ten years ago because everything is unnecessarily verbose on wikipedia now)

Re: Academic Torrents

#19
I wish I could add Gwern's Danbooru dataset [0] here - 2.7TB of labeled anime images. But they only support torrent files up to 10MB, and that's over 20MB for the full dataset or 12MB for the SFW low-rez set...

Incidentally, when the torrent file for your anime image collection passes 20MB, something has obviously gone very w̵r̵o̵n̵g̵ right.

0: https://www.gwern.net/Danbooru2019

Re: Academic Torrents

#20
post #8

https://academictorrents.com/about.php#mirroring Using RSS to allow mirrors to host different subjects is really clever, although some of the categories seem quite large (>5TB). It may be worth breaking up each category (sharding) to keep each to 100GB or less so a volunteer can pick a couple and not worry about running out of disk when a category grows. Then it would be good to track how many seeds each category-sha…

Some individual items are multiple TB, which would make 100GB shards a little difficult.
Post reply on HN