1.5 TB of Dark Net Market scrapes • Hacker News Board

Re: 1.5 TB of Dark Net Market scrapes

#31

Post by amenghra » Thu, Jul 16, 2015, 2:07 AM UTC

setuptools wrote: post #10

curiousg wrote: post #8

Earlier quoted context omitted.

Possession laws are pretty strict and hard to decode. I wouldn't want to be the test case in court. The idea of "poisoning" a dataset is an interesting theoretical. But in practice, I just want to judge the likelihood that the dataset is poisoned by the presence of images. If it is then there's not much I can do with it.

Yes, this absolutely needs to be clarified by Gwern. This is a very dangerous thing to link researchers to if it contains any illegal content.

Guess what, laws aren't universal! Unless gwern has a complete understanding of your jurisdiction and can somehow guess how you plan to use the data, he cannot know what is legal and wasn't isn't. The burden lies on you.

Re: 1.5 TB of Dark Net Market scrapes

#32

Post by amenghra » Thu, Jul 16, 2015, 2:10 AM UTC

ryanlol wrote: post #18

smeyer wrote: post #11

Earlier quoted context omitted.

The whole point of CAPTCHAs is to be difficult to automate. Or are you suggesting automating farming out the CAPTCHA solutions to cheap workers?

There's a plenty of service providers that sell APIs to captcha solving services at reasonable prices.

Or if you don't want to spend money, you can always re-host things on your own site. Let your visitors do the work for you.

Re: 1.5 TB of Dark Net Market scrapes

#33

Post by mikeash » Thu, Jul 16, 2015, 2:12 AM UTC

tomcam wrote: post #28

setuptools wrote: post #10

Earlier quoted context omitted.

Yes, this absolutely needs to be clarified by Gwern. This is a very dangerous thing to link researchers to if it contains any illegal content.

Um.. It says Black Market right on the tin.

Indeed, a warning that it may contain illegal content would be about as sensible as the standard "Warning: may contain nuts" label on a tin of nuts.

Re: 1.5 TB of Dark Net Market scrapes

#34

Post by branchless » Thu, Jul 16, 2015, 2:18 AM UTC

ethanbond wrote: post #23

branchless wrote: post #17

This is why I love the internet. This article has given me a fascinating glimpse into a world I have no idea about. Author: thank you so much for taking the time to document this.

You should take the time to read his other posts. Gwern is one of the most fascinating people/websites I know of.

Thanks, I will read at least another by him.

Re: 1.5 TB of Dark Net Market scrapes

#35

Post by gwern » Thu, Jul 16, 2015, 2:23 AM UTC

ez123 wrote: post #24

rcpt wrote: post #20

I've had a very slow-moving hobby project to parse and analyze a subset of this data: https://github.com/rcompton/black-market-recommender-systems So far I've had some ok results along the lines of "91.7% of vendors who sold speed and MDMA also sold ecstasy" http://ryancompton.net/2015/03/24/darknet-market-basket-anal... I am working on extending this to markets besides evolution now.

Just FYI, MDMA is ecstasy.

The system works!

Re: 1.5 TB of Dark Net Market scrapes

#36

Post by gwern » Thu, Jul 16, 2015, 2:26 AM UTC

curiousg wrote: post #3

What amazing work! I am very interested in doing research with Tor and a dataset like this could make my job a heck of a lot easier. I have a legal question though: Are your scrapes text only? Before I work with this dataset, I want to make sure that there's no possibility it contains illegal images (child porn).

They are generally not text only. I feel that images are useful to allow browsing the markets as they were and may be highly valuable in their own right as research material, so I tried to collects images where applicable. (The forums usually did not support any kind of image upload other than avatars, so this is more relevant to the markets than forum scrapes.)

As far as CP goes, there should be essentially zero CP anywhere in the archive. DNM users almost universally loathe CP, and no market has ever dared to permit sales. (You may find this funny: CP is so taboo, on the DNMs like elsewhere, that it's been used in at least one attack - SR2's DoctorClu/Brian Farrell infamously attacked a rival market's forum by posting CP to it.)

Re: 1.5 TB of Dark Net Market scrapes

#37

Post by gwern » Thu, Jul 16, 2015, 2:28 AM UTC

fractalcat wrote: post #16

joshmn wrote: post #4

This is so cool. Thanks gwern. If someone's feeling bored, you're welcome to put the entire archive on a web server for us to look at...... Or maybe I'll just do it.

You mean like this? https://archive.org/download/dnmarchives

I don't think that works. It's not remotely browsable or searchable. It would be quite challenging to put these scrapes up, anyway. They're regular wget crawls with a regular directory/file structure, the problem is that there's so much material and so many files that it can be almost impossible to find what you are looking for... (Plus you need to rewrite links into relative links to make everything render properly.)

Re: 1.5 TB of Dark Net Market scrapes

#38

Post by gwern » Thu, Jul 16, 2015, 2:29 AM UTC

ryanlol wrote: post #9

>collating and creating these scrapes has absorbed an enormous amount of my time & energy due to the need to solve CAPTCHAs,... Have you considered automating this?

I did, but the problem is that I never expected to be scraping for so long and it was always easier to just solve by hand than do a complete rewrite and allow for using CAPTCHA libraries. If I had known I would be scraping for ~3 years, I would have done many things differently: http://www.gwern.net/Black-market%20archives#how-to-crawl-ma...

Re: 1.5 TB of Dark Net Market scrapes

#39

Post by woah » Thu, Jul 16, 2015, 2:45 AM UTC

ez123 wrote: post #24

rcpt wrote: post #20

I've had a very slow-moving hobby project to parse and analyze a subset of this data: https://github.com/rcompton/black-market-recommender-systems So far I've had some ok results along the lines of "91.7% of vendors who sold speed and MDMA also sold ecstasy" http://ryancompton.net/2015/03/24/darknet-market-basket-anal... I am working on extending this to markets besides evolution now.

Just FYI, MDMA is ecstasy.

What people think of as ecstasy (the pill that makes you dance all night) is usually mdma mixed with some kind of speed. That's why this statistic was amusing.

Re: 1.5 TB of Dark Net Market scrapes

#40

Post by escape_goat » Thu, Jul 16, 2015, 2:58 AM UTC

rcpt wrote: post #30

ez123 wrote: post #24

Earlier quoted context omitted.

Just FYI, MDMA is ecstasy.

I chose that statistic to be a bit tongue-in-cheek, cf > While 'ecstasy' is the popular name for MDMA, the functional definition of ecstasy is any pill represented as MDMA on the street. Ecstasy pills are notoriously unreliable in content, more so than most other street drugs, and commonly contain either caffeine, ephedrine, amphetamines, MDA, MDE, DXM, or--in rare cases--DOB, and don't necessarily contain MDMA or an…

To follow up on your response and clarify some: in your dataset, not only do 91.7% of the vendors of 'speed and MDMA' also sell 'Ecstasy'; 76.8% of the vendors of 'Stimulants and Ecstasy' also sell 'MDMA'.

If I understand correctly, this means that vendors were significantly more likely to mention 'Ecstasy' in a product name (or description) if they claimed to be selling 'MDMA' than they were to mention 'MDMA' in a product name/description if they claimed to be selling 'Ecstasy', reinforcing the point that you have just made.