Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Dark Web Crawler in Go

creekorful.me

Post reply on HN
Print view View original

41–50 of 113 posts

Re: Building a Dark Web Crawler in Go

#41

Post by buildbuildbuild »

bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

A polite suggestion, but this is not currently possible.

The Tor Project recently added a consensus flag which can globally disable single hop client connections as a DDoS mitigation approach. It is currently enabled. (DoSRefuseSingleHopClientRendezvous)

Re: Building a Dark Web Crawler in Go

#42

Post by bureaucrat »

kerkeslager wrote: post #35
bureaucrat wrote: post #28

Earlier quoted context omitted.

They tried to enforce users to become a relay when using as a client but relays should be stable servers so the plan was dismissed.

Could you explain why relays should be stable servers?

Simple. Imagine you are using tor and one of the relay has problems(unstable, high latency or packet loss). You don’t know which of the 3 hops failed. We have no option but to build a new circuit and we don’t want that. That’s why tor needs stable, trustworthy relays.

From tor blog:

A new relay, assuming it is reliable and has plenty of bandwidth, goes through four phases: the unmeasured phase (days 0-3) where it gets roughly no use, the remote-measurement phase (days 3-8) where load starts to increase, the ramp-up guard phase (days 8-68) where load counterintuitively drops and then rises higher, and the steady-state guard phase (days 68+).

https://blog.torproject.org/lifecycle-new-relay

Re: Building a Dark Web Crawler in Go

#43

Post by bluesign »

creekorful wrote: post #16
Hitton wrote: post #15

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just…

Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and ha…

I think 75000 comment is coming from stats[1].

[1] https://metrics.torproject.org/hidserv-dir-onions-seen.html

Re: Building a Dark Web Crawler in Go

#44

Post by [deleted] »

creekorful wrote: post #16
Hitton wrote: post #15

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just…

Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and ha…

[deleted]

Re: Building a Dark Web Crawler in Go

#45

Post by luckylion »

rndgermandude wrote: post #39
kerkeslager wrote: post #38

Earlier quoted context omitted.

As others have pointed out, that's not what he said, but since you asked: https://www.npr.org/sections/alltechconsidered/2016/04/04/47...

There is a bunch of stories like that. E.g. https://www.ccc.de/en/updates/2018/hausdurchsuchungen-bei-ve... > On June 20th, board members of the „Zwiebelfreunde“ association in multiple German cities had their homes searched under the dubious pretence that they were „witnesses“ while their computers and storage media were confiscated.

That's something very different though. Exit nodes are providing a service and are, for all intents and purposes, the only visible client on the clearnet (and might not even be involved: there's nothing stopping you from running a private proxy on the same machine you run your exit node on). TOR-developers that do not run exit nodes but contribute to TOR typically don't get searched, at least to my knowledge.

Content that's illegal to possess is a different issue, though I'm sure they'd make for an interesting case because a crawler downloading, saving and parsing an HTML page isn't as clear cut as a human evaluating and deciding what to download and store. "The suspect has the hard- and software necessary to download this content" shouldn't be enough to convince a judge to issue a search warrant, but then again, judges probably have very little technical knowledge.

Re: Building a Dark Web Crawler in Go

#46

Post by F147H34D »

andrewjrhill wrote: post #27
mellosouls wrote: post #22

Earlier quoted context omitted.

IANAL but "educational project" won't fly in court, and nor should it.

Programs like https://www.hacksplaining.com/ exist purely as educational programs that teach you to exploit known flaws in web security and have no issue with the law.

Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning.

I would tread lightly crawling the dark web. There are cases where the FBI has admitted to running services on TOR, to collect IP addresses:

https://www.wired.com/2013/09/freedom-hosting-fbi/

Re: Building a Dark Web Crawler in Go

#47

Post by sandworm101 »

jstanley wrote: post #9
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

Actually, the opposite is true. People who actually need anonymity need to hide among traffic that is boring. If you reduce the number of hops your crawler is using, you're reducing the amount of boring traffic and making it easier to find the interesting people. Running a relay in addition to using Tor in the normal way is a good idea, however, as it increases the bandwidth of the network.

There is plenty of traffic in which to hide already. Another bot making and breaking random connections 24/7 is of no additional help.

Re: Building a Dark Web Crawler in Go

#48

Post by sandworm101 »

creekorful wrote: post #7
malux85 wrote: post #6

Earlier quoted context omitted.

> First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?

Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety

It's now just "Tor" and when accessing hidden services it isnt really a proxy. The Onion Router acronym went away back with the vidalia proxy. But i do miss the oldschool torbutton. It was fun.

Re: Building a Dark Web Crawler in Go

#49

Post by creekorful »

sandworm101 wrote: post #48
creekorful wrote: post #7

Earlier quoted context omitted.

Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety

It's now just "Tor" and when accessing hidden services it isnt really a proxy. The Onion Router acronym went away back with the vidalia proxy. But i do miss the oldschool torbutton. It was fun.

the proxy which I refer is the local SOCKS proxy used by tor browser / your applications to be routed through the network.

Re: Building a Dark Web Crawler in Go

#50

Post by apta »

F147H34D wrote: post #46
andrewjrhill wrote: post #27

Earlier quoted context omitted.

Programs like https://www.hacksplaining.com/ exist purely as educational programs that teach you to exploit known flaws in web security and have no issue with the law.

Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. I would tread lightly crawling the dark web. There are cases where the FBI has admitted to running services on TOR, to collect IP addresses: https://www.wired.com/2013/09/freedom-hosting-fbi/

> Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning.

What about when the FBI/CIA does it? Genuine question.

Post reply on HN