Live data from Hacker News

1.5 TB of Dark Net Market scrapes

gwern.net

41–50 of 78 posts

Re: 1.5 TB of Dark Net Market scrapes

#41
post #25

This is why I love the internet. This article has given me a fascinating glimpse into a world I have no idea about. Author: thank you so much for taking the time to document this.

Unrelated to the darknet, but this Twitter account evoked a very similar feeling in me (random snippets from userboards in the 80's and early 90's): https://twitter.com/wwwtxt

what a great twitter account. thanks for bringing this to my attention.

Re: 1.5 TB of Dark Net Market scrapes

#43
post #39
post #24

Earlier quoted context omitted.

Just FYI, MDMA is ecstasy.

What people think of as ecstasy (the pill that makes you dance all night) is usually mdma mixed with some kind of speed. That's why this statistic was amusing.

"Usually" is a real stretch. ecstacydata.org's lab testing report analysis indicates amphetamines were present in under 5% of tests for the last 15 years. http://www.ecstasydata.org/stats_substance_by_year.php?style...

Re: 1.5 TB of Dark Net Market scrapes

#44

What is the legality with respect to downloading this file? Could it contain material that would put us at legal risk?

I really don't know the answer. But I just want to remark that it seems to be a terrible situation that more than one of us have to wonder about the legality of downloading the dataset (there was a comment thread below on the same topic)

Re: 1.5 TB of Dark Net Market scrapes

#46
post #39

Earlier quoted context omitted.

What people think of as ecstasy (the pill that makes you dance all night) is usually mdma mixed with some kind of speed. That's why this statistic was amusing.

"Usually" is a real stretch. ecstacydata.org's lab testing report analysis indicates amphetamines were present in under 5% of tests for the last 15 years. http://www.ecstasydata.org/stats_substance_by_year.php?style...

If you also include the methamphetamine row, that number is a lot higher, in 2007 in particular it was present in 38.6% of the pills tested. For contrast, last year only 33.1% percent of the pills tested actually contained MDMA.

Re: 1.5 TB of Dark Net Market scrapes

#47
post #36
post #3

What amazing work! I am very interested in doing research with Tor and a dataset like this could make my job a heck of a lot easier. I have a legal question though: Are your scrapes text only? Before I work with this dataset, I want to make sure that there's no possibility it contains illegal images (child porn).

They are generally not text only. I feel that images are useful to allow browsing the markets as they were and may be highly valuable in their own right as research material, so I tried to collects images where applicable. (The forums usually did not support any kind of image upload other than avatars, so this is more relevant to the markets than forum scrapes.) As far as CP goes, there should be essentially zero CP…

DNM users almost universally loathe CP, and no market has ever dared to permit sales.

These users are willing to do so many other illegal things, but the thought of being known as a pedophile or supporting pedophilia in any way, is completely abhorrent to them? Interesting datapoint.

Re: 1.5 TB of Dark Net Market scrapes

#48
post #24
post #20

I've had a very slow-moving hobby project to parse and analyze a subset of this data: https://github.com/rcompton/black-market-recommender-systems So far I've had some ok results along the lines of "91.7% of vendors who sold speed and MDMA also sold ecstasy" http://ryancompton.net/2015/03/24/darknet-market-basket-anal... I am working on extending this to markets besides evolution now.

Just FYI, MDMA is ecstasy.

I don't do MDMA personally but I have some friends who like going to raves with some chemical supplies. On the streets, some make it a point of honor to sell/buy only "MDMA" because "ecstasy" is "crap".

The only precise name is MDMA. In my opinion the name users give to what they're ingesting is inconsequential. Most users have no idea what MDMA means anyway, nor would they know how to chemically identify if what they bought might contain MDMA.

Re: 1.5 TB of Dark Net Market scrapes

#49
post #36

Earlier quoted context omitted.

They are generally not text only. I feel that images are useful to allow browsing the markets as they were and may be highly valuable in their own right as research material, so I tried to collects images where applicable. (The forums usually did not support any kind of image upload other than avatars, so this is more relevant to the markets than forum scrapes.) As far as CP goes, there should be essentially zero CP…

DNM users almost universally loathe CP, and no market has ever dared to permit sales. These users are willing to do so many other illegal things, but the thought of being known as a pedophile or supporting pedophilia in any way, is completely abhorrent to them? Interesting datapoint.

> These users are willing to do so many other illegal things, but the thought of being known as a pedophile or supporting pedophilia in any way, is completely abhorrent to them?

I'm pretty sure the distinction is that voluntary transactions have no victims, and DNM folks care more about morality and ethics than legalities.

Post reply on HN