Live data from Hacker News

Building a Dark Web Crawler in Go

creekorful.me

21–30 of 113 posts

Re: Building a Dark Web Crawler in Go

#21

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

[deleted]

Re: Building a Dark Web Crawler in Go

#22

I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.

You are right. That's why it's an educational project and not a public search engine

IANAL but "educational project" won't fly in court, and nor should it.

Re: Building a Dark Web Crawler in Go

#23
To anyone experimenting with such stuff, take care and don't make your services publically available. Especially the dark web is full with highly illegal content such as child pornography and in some jurisdictions even "involuntary possession" such as in browser caches may be enough to convict you.

Re: Building a Dark Web Crawler in Go

#24

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

Must one rely on appeals such as this (i.e. a cultural solution) or does tor have a technological solution to the problem you're describing?

Re: Building a Dark Web Crawler in Go

#25
post #20

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

I am no TOR expert, but how does decreasing the amount of hops or running your own relay decrease the privacy of other people?

Running large requests over many hops takes up limited Tor resources so it makes using Tor slower/harder for other people. Decreasing the amount of hops means that you use less resources, running your own relay means that you provide resources to others.

Re: Building a Dark Web Crawler in Go

#26

Earlier quoted context omitted.

You are right. That's why it's an educational project and not a public search engine

IANAL but "educational project" won't fly in court, and nor should it.

I don't host a Trandoshan instance neither give access to a database of results. I Only provide access to the source code.

Why should I face legal problems?

Re: Building a Dark Web Crawler in Go

#27

Earlier quoted context omitted.

You are right. That's why it's an educational project and not a public search engine

IANAL but "educational project" won't fly in court, and nor should it.

Programs like https://www.hacksplaining.com/ exist purely as educational programs that teach you to exploit known flaws in web security and have no issue with the law.

Re: Building a Dark Web Crawler in Go

#28

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

Must one rely on appeals such as this (i.e. a cultural solution) or does tor have a technological solution to the problem you're describing?

They tried to enforce users to become a relay when using as a client but relays should be stable servers so the plan was dismissed.

Re: Building a Dark Web Crawler in Go

#29

Earlier quoted context omitted.

IANAL but "educational project" won't fly in court, and nor should it.

I don't host a Trandoshan instance neither give access to a database of results. I Only provide access to the source code. Why should I face legal problems?

Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example.

I am guessing you have some personal instance you use at least for testing/"education", right?

Re: Building a Dark Web Crawler in Go

#30

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

Must one rely on appeals such as this (i.e. a cultural solution) or does tor have a technological solution to the problem you're describing?

Well if we made a BLOCKCHAIN where you had to give a certain amount of bandwidth for torcoins then cash them in for bandwidth you could use...

I mean, it's a dumb idea, but until you force people to contribute, most won't. I've looked at doing it, but didn't want to deal with legal headaches b/c creepy pedos use tor, too.

Post reply on HN