Earlier quoted context omitted.
i just like how 'dark web' turned into 'Tor' at some point :'). there's tons of others... :s guess ppl forgot
A short list of those tons would be wonderfully helpful to share. :)
Building a Dark Web Crawler in Go
101–110 of 113 posts
Re: Building a Dark Web Crawler in Go
#102Earlier quoted context omitted.
IANAL but "educational project" won't fly in court, and nor should it.
Programs like https://www.hacksplaining.com/ exist purely as educational programs that teach you to exploit known flaws in web security and have no issue with the law.
Re: Building a Dark Web Crawler in Go
#103Earlier quoted context omitted.
I think this is an important cultural reframing that needs to occur, sooner rather than later. Most people, when they hear of things like "the dark web" and cryptocurrency think about the massively publicized instances of drug trafficking and ordering a hit on someone. It's going to take a lot of work to reframe the utility and purpose of them to a more universal, humanitarian angle. People in this world live in oppr…
Under the idea of legitimate and illegitimate use, why wouldn't drug trafficking be legitimate? It gets drugs off the streets, decreases violence compare to street level drug dealing, increases safety (while the reputation of online sellers isn't a great metric, we are talking relative to the person on the street corner), and generally involves only adults. If one is willing to argue that the US government throwing s…
Reminds me a bit of what you see with how some societies approach drug addiction. Providing a safe space with clean needles vs throwing in a prison. There's a lot to think about.
And I think we've seen some of that with the marijuana legalization across the US. The state adoption had strong initial resistance, but public opinion began to shift once it got out of the shroud of stigma and moral enforcement.
Re: Building a Dark Web Crawler in Go
#104Earlier quoted context omitted.
> First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?
Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety
This takes me to the 1990's.... "groups of services that cannot be accessed without using special software" definitely matches NNTP, FTP, SMTP, SSH, HTTP, etc.
Re: Building a Dark Web Crawler in Go
#105Re: Building a Dark Web Crawler in Go
#106Earlier quoted context omitted.
Simple. Imagine you are using tor and one of the relay has problems(unstable, high latency or packet loss). You don’t know which of the 3 hops failed. We have no option but to build a new circuit and we don’t want that. That’s why tor needs stable, trustworthy relays. From tor blog: A new relay, assuming it is reliable and has plenty of bandwidth, goes through four phases: the unmeasured phase (days 0-3) where it get…
> We have no option but to build a new circuit and we don’t want that. That's true, you never want to rebuild the circuit. But it strikes me that the idea that this is avoidable falls into at least two of the Eight Fallacies of Distributed Computing[1], namely "The Network Is Reliable" and "Topology Doesn't Change". If we instead assume that the network isn't reliable, and topology does change, then instead of elimin…
But remember: What tor is doing is hard. They are doing complex crypto, networking, security.... The hard stuff. The real stuff. Torproject is a nonprofit organization with limited capabilities. They are doing their best. It took 3 years to design and implement dos mitigation techniques, for example.
Your proposed plan could take over 10 years, even for a well funded corporation. It might take time and fail. It might create huge vulnerability due to code complexity. Afaik, tor can’t risk that.
Re: Building a Dark Web Crawler in Go
#107I’ve been pretty surprised at how big hidden services have become Dread, the dark net reddit, is surprisingly vibrant I think its weird that people almost don't want to hear positive stories about dark net. It’ll be funny when news articles and romcoms just start “forgetting” to qualify their plot piece with the “its scary” trope
I thought dread was dead?
Re: Building a Dark Web Crawler in Go
#108Earlier quoted context omitted.
> We have no option but to build a new circuit and we don’t want that. That's true, you never want to rebuild the circuit. But it strikes me that the idea that this is avoidable falls into at least two of the Eight Fallacies of Distributed Computing[1], namely "The Network Is Reliable" and "Topology Doesn't Change". If we instead assume that the network isn't reliable, and topology does change, then instead of elimin…
Yeah, your suggestion sounds good. But remember: What tor is doing is hard. They are doing complex crypto, networking, security.... The hard stuff. The real stuff. Torproject is a nonprofit organization with limited capabilities. They are doing their best. It took 3 years to design and implement dos mitigation techniques, for example. Your proposed plan could take over 10 years, even for a well funded corporation. It…
Re: Building a Dark Web Crawler in Go
#109I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.
If you avoid storing images, is there any other items you could be liable for?
Re: Building a Dark Web Crawler in Go
#110Earlier quoted context omitted.
Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety
> The dark web is a vast groups of services that cannot be accessed without using special software This takes me to the 1990's.... "groups of services that cannot be accessed without using special software" definitely matches NNTP, FTP, SMTP, SSH, HTTP, etc.