Building a Dark Web Crawler in Go
creekorful.me
Building a Dark Web Crawler in Go
1–10 of 113 posts
Re: Building a Dark Web Crawler in Go
#2Crawlers are fun!
If you're new to the field and want something that's easy to set up & polite, I strongly recommend Apache Storm Crawler (https://github.com/DigitalPebble/storm-crawler).
Re: Building a Dark Web Crawler in Go
#3A well written article with lot of technical details. Well done.
However, I'm wondering what would be a good practical purpose of crawling dark web.
Re: Building a Dark Web Crawler in Go
#4First of all, it’s hidden sevices, not dark web.
Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
Re: Building a Dark Web Crawler in Go
#5I’ve been pretty surprised at how big hidden services have become
Dread, the dark net reddit, is surprisingly vibrant
I think its weird that people almost don't want to hear positive stories about dark net.
It’ll be funny when news articles and romcoms just start “forgetting” to qualify their plot piece with the “its scary” trope
Re: Building a Dark Web Crawler in Go
#6First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
> First of all, it’s hidden sevices, not dark web
For the uninitiated, can you please explain the differences in what they are and how they're accessed?
Re: Building a Dark Web Crawler in Go
#7First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
> First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?
Author here.
The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy.
The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy.
They are a type of dark web services, but not the entirety
Re: Building a Dark Web Crawler in Go
#8A well written article with lot of technical details. Well done. However, I'm wondering what would be a good practical purpose of crawling dark web.
Thank you!
There's no practical purpose for the crawler. It's more an educational project than anything.
Re: Building a Dark Web Crawler in Go
#9First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
Actually, the opposite is true.
People who actually need anonymity need to hide among traffic that is boring. If you reduce the number of hops your crawler is using, you're reducing the amount of boring traffic and making it easier to find the interesting people.
Running a relay in addition to using Tor in the normal way is a good idea, however, as it increases the bandwidth of the network.
Re: Building a Dark Web Crawler in Go
#10First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
> First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?