Live data from Hacker News

Building a Dark Web Crawler in Go

creekorful.me

71–80 of 113 posts

Re: Building a Dark Web Crawler in Go

#71

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Anonymity allows you to sow an action without reaping the societal karma of the action.

In a good, free society, maybe anonymity isn’t important.

But in a bad society, one in which collaboration on a cause is punished, but each individual desperately wants to collaborate and change something fundamental...

Anonymity allows the planning of synchronized action.

——

Mass or targeted misinformation also threatens the planning of synchronized action.

Re: Building a Dark Web Crawler in Go

#72

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Aside from situations like China where state censorship of history and news are an actual thing, or in situations where whistleblowers need to protect their identity... Some people just want privacy.

Give me a solid reason for why you want corporations and governments to have access to detailed records of everything you do online.

There's value in that data to certain groups of people and we may not like what the future looks like once that value is tapped to its potential.

Re: Building a Dark Web Crawler in Go

#73

Earlier quoted context omitted.

Could you explain why relays should be stable servers?

Simple. Imagine you are using tor and one of the relay has problems(unstable, high latency or packet loss). You don’t know which of the 3 hops failed. We have no option but to build a new circuit and we don’t want that. That’s why tor needs stable, trustworthy relays. From tor blog: A new relay, assuming it is reliable and has plenty of bandwidth, goes through four phases: the unmeasured phase (days 0-3) where it get…

> We have no option but to build a new circuit and we don’t want that.

That's true, you never want to rebuild the circuit. But it strikes me that the idea that this is avoidable falls into at least two of the Eight Fallacies of Distributed Computing[1], namely "The Network Is Reliable" and "Topology Doesn't Change".

If we instead assume that the network isn't reliable, and topology does change, then instead of eliminating unreliable nodes and being conservative with changes to the topology, we would focus on reducing the costs of rebuilding a circuit so that network unreliability and topology changes aren't disastrous.

But it sounds like the Tor team has instead decided to bolster these assumptions, to make them less of assumptions; trying to make the network as reliable as possible and trying to make the topology change as little as possible.

I don't mean this to be a harsh criticism of the Tor team. I'm an outsider, and beyond an uncompromising privacy constraint, I don't know all the constraints Tor was built under. I'm sure the tradeoffs made by the Tor team make sense within the context of their constraints. Obviously, the Tor network works well enough to have a large user base, so they have provided a good-enough solution.

But I wonder if changes could be made to Tor's design in the future which would allow quicker adding and removing nodes, and handle network reliability issues better, so that Tor would be faster.

One possibility which stands out to me is to pool circuits and load-balance between them, so that if a circuit begins to have issues, you still are connected along other circuits while you build a new circuit to replace the unreliable one. This possibly would run into issues where correlate could correlate traffic from different circuits to unmask clients, so you'd have to be careful, but I'm not sure these problems would be insurmountable.

[1] https://en.wikipedia.org/wiki/Fallacies_of_distributed_compu...

Re: Building a Dark Web Crawler in Go

#76
post #50

Earlier quoted context omitted.

Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. I would tread lightly crawling the dark web. There are cases where the FBI has admitted to running services on TOR, to collect IP addresses: https://www.wired.com/2013/09/freedom-hosting-fbi/

> Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. What about when the FBI/CIA does it? Genuine question.

There is a legal exception for legitimate law enforcement activities.

Re: Building a Dark Web Crawler in Go

#77

Earlier quoted context omitted.

Actually, the official term is Onion Services ( https://2019.www.torproject.org/docs/onion-services.html.en check out the first paragraph)

It seems that they changed the name from hidden service to onion service.

That is correct. Tor Project has really struggled with negative connotations in the media with the so called "dark web" so I believe the terminology does play a part.

Re: Building a Dark Web Crawler in Go

#78
post #9

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

Actually, the opposite is true. People who actually need anonymity need to hide among traffic that is boring. If you reduce the number of hops your crawler is using, you're reducing the amount of boring traffic and making it easier to find the interesting people. Running a relay in addition to using Tor in the normal way is a good idea, however, as it increases the bandwidth of the network.

In fact it is a bit more severe than that as you are effectively deanonymizing yourself. If everyone else is using a 3-hop circuit but your crawler is using just 2 hops, it wouldn't take much effort to isolate your activity in the network since you're effectively standing out.

Re: Building a Dark Web Crawler in Go

#79

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Circumventing censored websites is a legitimate use. VPNs are more popular but the thing is that the VPN obviously knows who you are and what you're browsing. Tor makes you anonymous to the middleman as well. Tor wouldn't know if you're watching porn but a VPN would. By the way, that's not a recommended use of tor since you consume a decent amount of bandwidth (:P)

Re: Building a Dark Web Crawler in Go

#80
post #68

Earlier quoted context omitted.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Journalism, whistleblowing, accessing censored information, preventing stalkers from tracking you, etc.

How exactly can a stalker track me online if I simply stop logging in to services? Honest question, because I don't know why Tor would be any better than simply browsing in incognito mode
Post reply on HN