1.5 TB of Dark Net Market scrapes
gwern.net
1.5 TB of Dark Net Market scrapes
1–10 of 78 posts
Re: 1.5 TB of Dark Net Market scrapes
#2gwern, you are an absolute force of nature when it comes to generating and collecting and presenting information in a useful way. Thank you.
Re: 1.5 TB of Dark Net Market scrapes
#3What amazing work! I am very interested in doing research with Tor and a dataset like this could make my job a heck of a lot easier. I have a legal question though: Are your scrapes text only? Before I work with this dataset, I want to make sure that there's no possibility it contains illegal images (child porn).
Re: 1.5 TB of Dark Net Market scrapes
#4This is so cool. Thanks gwern.
If someone's feeling bored, you're welcome to put the entire archive on a web server for us to look at......
Or maybe I'll just do it.
Re: 1.5 TB of Dark Net Market scrapes
#5What amazing work! I am very interested in doing research with Tor and a dataset like this could make my job a heck of a lot easier. I have a legal question though: Are your scrapes text only? Before I work with this dataset, I want to make sure that there's no possibility it contains illegal images (child porn).
Lower down on the page, he says he did scrape at least one site with such images, although he specifically only took text. Can't verify that this was the case for all scraped sites.
Re: 1.5 TB of Dark Net Market scrapes
#6gwern, you are an absolute force of nature when it comes to generating and collecting and presenting information in a useful way. Thank you.
I hope a he gets some good donations from HN, Ill send him a few dollars in BTC.
Re: 1.5 TB of Dark Net Market scrapes
#7What amazing work! I am very interested in doing research with Tor and a dataset like this could make my job a heck of a lot easier. I have a legal question though: Are your scrapes text only? Before I work with this dataset, I want to make sure that there's no possibility it contains illegal images (child porn).
How about ascii art?
Actually, this is an interesting topic. Poisoning a dataset. CP would work for private security investigators, and to poison against government investigators you could use leaked classified secrets.
Could you work around this by operating on the files on VPS you don't own, streaming a very low-res ('Basilisk'-proof - https://en.wikipedia.org/wiki/BLIT_(short_story) ) remote desktop image.
Re: 1.5 TB of Dark Net Market scrapes
#8What amazing work! I am very interested in doing research with Tor and a dataset like this could make my job a heck of a lot easier. I have a legal question though: Are your scrapes text only? Before I work with this dataset, I want to make sure that there's no possibility it contains illegal images (child porn).
How about ascii art? Actually, this is an interesting topic. Poisoning a dataset. CP would work for private security investigators, and to poison against government investigators you could use leaked classified secrets. Could you work around this by operating on the files on VPS you don't own, streaming a very low-res ('Basilisk'-proof - https://en.wikipedia.org/wiki/BLIT_(short_story) ) remote desktop image.
Possession laws are pretty strict and hard to decode. I wouldn't want to be the test case in court. The idea of "poisoning" a dataset is an interesting theoretical. But in practice, I just want to judge the likelihood that the dataset is poisoned by the presence of images. If it is then there's not much I can do with it.
Re: 1.5 TB of Dark Net Market scrapes
#9>collating and creating these scrapes has absorbed an enormous amount of my time & energy due to the need to solve CAPTCHAs,...
Have you considered automating this?
Re: 1.5 TB of Dark Net Market scrapes
#10Earlier quoted context omitted.
How about ascii art? Actually, this is an interesting topic. Poisoning a dataset. CP would work for private security investigators, and to poison against government investigators you could use leaked classified secrets. Could you work around this by operating on the files on VPS you don't own, streaming a very low-res ('Basilisk'-proof - https://en.wikipedia.org/wiki/BLIT_(short_story) ) remote desktop image.
Possession laws are pretty strict and hard to decode. I wouldn't want to be the test case in court. The idea of "poisoning" a dataset is an interesting theoretical. But in practice, I just want to judge the likelihood that the dataset is poisoned by the presence of images. If it is then there's not much I can do with it.
Yes, this absolutely needs to be clarified by Gwern. This is a very dangerous thing to link researchers to if it contains any illegal content.