Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Dark Web Crawler in Go

creekorful.me

Post reply on HN
Print view View original

81–90 of 113 posts

Re: Building a Dark Web Crawler in Go

#81

Post by malux85 »

creekorful wrote: post #7
malux85 wrote: post #6

Earlier quoted context omitted.

> First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?

Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety

Thank you.

> The dark web is a vast groups of services that cannot be accessed without using special software / proxy.

Can you name a few for examples?

Re: Building a Dark Web Crawler in Go

#82

Post by jerf »

creekorful wrote: post #16
Hitton wrote: post #15

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just…

Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and ha…

"Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available."

As a defense against the parent comment, though, this proves way too much. It doesn't matter how much k8s you throw at that, you're never going to so much as find your first site, if you're looking at the problem that way.

That's not really a relevant number here.

Re: Building a Dark Web Crawler in Go

#83

Post by blotter_paper »

airstrike wrote: post #80
nickles wrote: post #68

Earlier quoted context omitted.

Journalism, whistleblowing, accessing censored information, preventing stalkers from tracking you, etc.

How exactly can a stalker track me online if I simply stop logging in to services? Honest question, because I don't know why Tor would be any better than simply browsing in incognito mode

IP addresd is the big one, but there are other things that let you narrow down users on the same IP or a user switching between IPs, like tracking cookies, identification of which subset of hardware your GPU falls into based on how it renders some WebGL stuff (which can sometimes allow identification of a specific model of phone, especially when combined with other fingerprinting methods), specifics of screen size, what plugins/extensions you have installed at specific version numbers, etc. Tor only directly addresses the IP point, but the Tor browser should be disabling that other leaky browser stuff as well. I think they were accidentally leaking IPs through WebRTC a while back, or something like that, and I'm sure there will be more issues going forward.

Re: Building a Dark Web Crawler in Go

#84

Post by fishtacos »

penagwin wrote: post #52
Havoc wrote: post #40

Sounds like a recipe to score yourself a free FBI visit

Generally the FBI doesn't give a hoot until you start distributing illegal stuff....

What does suck is being put on IP blacklists by various providers for merely running a Tor relay, not an exit node. There are several websites I can only access through VPN because of my IP is associated with running a relay.

Re: Building a Dark Web Crawler in Go

#85

Post by weatherlight »

creekorful wrote: post #63
mschuster91 wrote: post #23

To anyone experimenting with such stuff, take care and don't make your services publically available. Especially the dark web is full with highly illegal content such as child pornography and in some jurisdictions even "involuntary possession" such as in browser caches may be enough to convict you.

Do you think I should add a license in Github to mention that? To protect me and the users who will use the crawler?

yes.

Re: Building a Dark Web Crawler in Go

#87

Post by marcrosoft »

getpolarized wrote: post #86

Go is a horrible language in which to write a crawler. The main problem is that NLP and machine learning code simply isn't as prevalent and robust as it is in Java and Python.

Go is great for a crawler. What does NLP and ML have to do with crawling?

Re: Building a Dark Web Crawler in Go

#88

Post by [deleted] »

clashmeifyoucan wrote: post #78
jstanley wrote: post #9

Earlier quoted context omitted.

Actually, the opposite is true. People who actually need anonymity need to hide among traffic that is boring. If you reduce the number of hops your crawler is using, you're reducing the amount of boring traffic and making it easier to find the interesting people. Running a relay in addition to using Tor in the normal way is a good idea, however, as it increases the bandwidth of the network.

In fact it is a bit more severe than that as you are effectively deanonymizing yourself. If everyone else is using a 3-hop circuit but your crawler is using just 2 hops, it wouldn't take much effort to isolate your activity in the network since you're effectively standing out.

[deleted]

Re: Building a Dark Web Crawler in Go

#89

Post by fluffything »

mhluongo wrote: post #33
rndgermandude wrote: post #29

Earlier quoted context omitted.

Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example. I am guessing you have some personal instance you use at least for testing/"education", right?

Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children

What are you talking about?

The OP wrote a crawler and used it to crawl Tor. Depending on where they live, accessing the content might be illegal, and storing some of the content in your computer might be illegal as well.

Law enforcement might be monitoring some domains, or have set up some honeypots that the OP might crawl automatically.

You don't want to end up in court having to argue about why your computed accessed some child pornography and downloaded it, and trying to explain to a jury that you did not did those things, but that the crawler that you programmed to do those things did.

Sure, nobody might end up raiding the OPs home, and even if they do, the OP might be able to successfully survive a jury. But just having to go through that might suck.

If the OP only wrote the software and never used it, then they are fine. But from the article, they did use it, so who knows where the crawler landed. Chances are nowhere good.

Re: Building a Dark Web Crawler in Go

#90

Post by tgragnato »

buildbuildbuild wrote: post #41
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

A polite suggestion, but this is not currently possible. The Tor Project recently added a consensus flag which can globally disable single hop client connections as a DDoS mitigation approach. It is currently enabled. (DoSRefuseSingleHopClientRendezvous)

If I were to judge by the statistics of the heartbeat messages I have access to.. The number of people trying to create one-hop circuits is huge.
Post reply on HN