Live data from Hacker News

Building a Dark Web Crawler in Go

creekorful.me

31–40 of 113 posts

Re: Building a Dark Web Crawler in Go

#31

Earlier quoted context omitted.

IANAL but "educational project" won't fly in court, and nor should it.

I don't host a Trandoshan instance neither give access to a database of results. I Only provide access to the source code. Why should I face legal problems?

Probably okay to have the source code to the engine.

However, if you have used the the system which creates a database of questionable dark web links on your machine, that could be tricky to explain... and easy to implicate

Re: Building a Dark Web Crawler in Go

#32
post #15

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just…

Assume 100 pages on each onion address (it’s probably power-law but let’s just assume that’s the mean). Latency with Tor is super high. Assume average of 5s to load a single page. This is generous because tail latency will probably dominate mean latency in this setting.

These things can happen in parallel but let’s also assume no more than 32 simultaneous TCP connections per host through a Tor proxy.

So we’re looking at ~75k1005/32 seconds = 14 days to run through all of them. You may not need to distribute this but there are situations (e.g. I want a fresh index daily) where it is warranted.

Re: Building a Dark Web Crawler in Go

#33

Earlier quoted context omitted.

I don't host a Trandoshan instance neither give access to a database of results. I Only provide access to the source code. Why should I face legal problems?

Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example. I am guessing you have some personal instance you use at least for testing/"education", right?

Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children

Re: Building a Dark Web Crawler in Go

#34

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

i just like how 'dark web' turned into 'Tor' at some point :'). there's tons of others... :s guess ppl forgot

Re: Building a Dark Web Crawler in Go

#35

Earlier quoted context omitted.

Must one rely on appeals such as this (i.e. a cultural solution) or does tor have a technological solution to the problem you're describing?

They tried to enforce users to become a relay when using as a client but relays should be stable servers so the plan was dismissed.

Could you explain why relays should be stable servers?

Re: Building a Dark Web Crawler in Go

#36

Earlier quoted context omitted.

Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example. I am guessing you have some personal instance you use at least for testing/"education", right?

Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children

[deleted]

Re: Building a Dark Web Crawler in Go

#37

Earlier quoted context omitted.

Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example. I am guessing you have some personal instance you use at least for testing/"education", right?

Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children

I didn't talk about the dev getting contacted about third parties abusing the software, but about the dev keeping a DB of indexed content for development/testing/"education" that would most likely include illegal-to-possess content.

And that some eager police people like to "inconvenience" people connected to TOR somehow isn't exactly new, either. E.g. there have been multiple raids against TOR exit node operators in different countries around the world in the past, even when the police was fully aware it was a TOR exit node that did not store information.

Maybe I'm just too paranoid - then again I used to run a TOR exit node myself and had a bunch of less than pleasant run-ins with the police, tho thankfully no raids.

Re: Building a Dark Web Crawler in Go

#38

Earlier quoted context omitted.

Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example. I am guessing you have some personal instance you use at least for testing/"education", right?

Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children

As others have pointed out, that's not what he said, but since you asked:

https://www.npr.org/sections/alltechconsidered/2016/04/04/47...

Re: Building a Dark Web Crawler in Go

#39

Earlier quoted context omitted.

Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children

As others have pointed out, that's not what he said, but since you asked: https://www.npr.org/sections/alltechconsidered/2016/04/04/47...

There is a bunch of stories like that.

E.g. https://www.ccc.de/en/updates/2018/hausdurchsuchungen-bei-ve...

> On June 20th, board members of the „Zwiebelfreunde“ association in multiple German cities had their homes searched under the dubious pretence that they were „witnesses“ while their computers and storage media were confiscated.

Post reply on HN