Live data from Hacker News

Building a Dark Web Crawler in Go

creekorful.me

101–110 of 113 posts

Re: Building a Dark Web Crawler in Go

#101

Earlier quoted context omitted.

i just like how 'dark web' turned into 'Tor' at some point :'). there's tons of others... :s guess ppl forgot

A short list of those tons would be wonderfully helpful to share. :)

I think by a technical definition, every company intranet would count (minus some poor security configurations). But that doesn't really seem to be a fair companion to the more commonly discussed dark web.

Re: Building a Dark Web Crawler in Go

#102

Earlier quoted context omitted.

IANAL but "educational project" won't fly in court, and nor should it.

Programs like https://www.hacksplaining.com/ exist purely as educational programs that teach you to exploit known flaws in web security and have no issue with the law.

But it is totally possible to host your own server with flawed security that you are able to legally hack. It might be rare for that to be the actual use case, but it is totally a possibility. With the illegal material being discussed here, there is no such equivalent no matter how out of the box a justification one is willing to aim for.

Re: Building a Dark Web Crawler in Go

#103
post #96

Earlier quoted context omitted.

I think this is an important cultural reframing that needs to occur, sooner rather than later. Most people, when they hear of things like "the dark web" and cryptocurrency think about the massively publicized instances of drug trafficking and ordering a hit on someone. It's going to take a lot of work to reframe the utility and purpose of them to a more universal, humanitarian angle. People in this world live in oppr…

Under the idea of legitimate and illegitimate use, why wouldn't drug trafficking be legitimate? It gets drugs off the streets, decreases violence compare to street level drug dealing, increases safety (while the reputation of online sellers isn't a great metric, we are talking relative to the person on the street corner), and generally involves only adults. If one is willing to argue that the US government throwing s…

I suspect that since illicit drug trafficking has a strong social stigma, it may not be the best thing to lead with. It can, however, be discussed with nuance in a way that could change minds. I definitely think there's a lot of legitimacy to what you're saying.

Reminds me a bit of what you see with how some societies approach drug addiction. Providing a safe space with clean needles vs throwing in a prison. There's a lot to think about.

And I think we've seen some of that with the marijuana legalization across the US. The state adoption had strong initial resistance, but public opinion began to shift once it got out of the shroud of stigma and moral enforcement.

Re: Building a Dark Web Crawler in Go

#104
post #6

Earlier quoted context omitted.

> First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?

Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety

> The dark web is a vast groups of services that cannot be accessed without using special software

This takes me to the 1990's.... "groups of services that cannot be accessed without using special software" definitely matches NNTP, FTP, SMTP, SSH, HTTP, etc.

Re: Building a Dark Web Crawler in Go

#105

Earlier quoted context omitted.

i just like how 'dark web' turned into 'Tor' at some point :'). there's tons of others... :s guess ppl forgot

A short list of those tons would be wonderfully helpful to share. :)

From the top of my head:

I2P Freenet Tor Zeronet

Re: Building a Dark Web Crawler in Go

#106

Earlier quoted context omitted.

Simple. Imagine you are using tor and one of the relay has problems(unstable, high latency or packet loss). You don’t know which of the 3 hops failed. We have no option but to build a new circuit and we don’t want that. That’s why tor needs stable, trustworthy relays. From tor blog: A new relay, assuming it is reliable and has plenty of bandwidth, goes through four phases: the unmeasured phase (days 0-3) where it get…

> We have no option but to build a new circuit and we don’t want that. That's true, you never want to rebuild the circuit. But it strikes me that the idea that this is avoidable falls into at least two of the Eight Fallacies of Distributed Computing[1], namely "The Network Is Reliable" and "Topology Doesn't Change". If we instead assume that the network isn't reliable, and topology does change, then instead of elimin…

Yeah, your suggestion sounds good.

But remember: What tor is doing is hard. They are doing complex crypto, networking, security.... The hard stuff. The real stuff. Torproject is a nonprofit organization with limited capabilities. They are doing their best. It took 3 years to design and implement dos mitigation techniques, for example.

Your proposed plan could take over 10 years, even for a well funded corporation. It might take time and fail. It might create huge vulnerability due to code complexity. Afaik, tor can’t risk that.

Re: Building a Dark Web Crawler in Go

#107

I’ve been pretty surprised at how big hidden services have become Dread, the dark net reddit, is surprisingly vibrant I think its weird that people almost don't want to hear positive stories about dark net. It’ll be funny when news articles and romcoms just start “forgetting” to qualify their plot piece with the “its scary” trope

I thought dread was dead?

Its not, hit up dark fail for the onion link to dark fail and browse the latest onion links

Re: Building a Dark Web Crawler in Go

#108

Earlier quoted context omitted.

> We have no option but to build a new circuit and we don’t want that. That's true, you never want to rebuild the circuit. But it strikes me that the idea that this is avoidable falls into at least two of the Eight Fallacies of Distributed Computing[1], namely "The Network Is Reliable" and "Topology Doesn't Change". If we instead assume that the network isn't reliable, and topology does change, then instead of elimin…

Yeah, your suggestion sounds good. But remember: What tor is doing is hard. They are doing complex crypto, networking, security.... The hard stuff. The real stuff. Torproject is a nonprofit organization with limited capabilities. They are doing their best. It took 3 years to design and implement dos mitigation techniques, for example. Your proposed plan could take over 10 years, even for a well funded corporation. It…

Agreed, as I said, "I'm sure the tradeoffs made by the Tor team make sense within the context of their constraints."

Re: Building a Dark Web Crawler in Go

#109
post #74

I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.

If you avoid storing images, is there any other items you could be liable for?

The URLs to those items perhaps? I do recall relatively old-ish debate about the legality of Google et al having "links" to illegal content, despite not hosting/storing them.

Re: Building a Dark Web Crawler in Go

#110

Earlier quoted context omitted.

Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety

> The dark web is a vast groups of services that cannot be accessed without using special software This takes me to the 1990's.... "groups of services that cannot be accessed without using special software" definitely matches NNTP, FTP, SMTP, SSH, HTTP, etc.

hell I still have servers that won't do anything for me without https://
Post reply on HN