Building a Dark Web Crawler in Go • Hacker News Board

Re: Building a Dark Web Crawler in Go

#11

Post by warent » Mon, Sep 23, 2019, 9:12 AM UTC

creekorful wrote: post #8

sbmthakur wrote: post #3

A well written article with lot of technical details. Well done. However, I'm wondering what would be a good practical purpose of crawling dark web.

Thank you! There's no practical purpose for the crawler. It's more an educational project than anything.

Weird, for some reason your comments are being instantly marked as "dead." I think there's some kind of filter that's tripping out for your account since it's new. I vouched for your two comments so hopefully everyone can see them now, but an admin (i.e. dang?) will need to look into this for a longer term solution.

Re: Building a Dark Web Crawler in Go

#12

Post by fs111 » Mon, Sep 23, 2019, 9:13 AM UTC

Any http-aware software that supports socks proxies can access information on hidden services, so any crawler can do it. I fail to see what is novel about that, except that it uses k8s and mongo and a catchy blog title.

Re: Building a Dark Web Crawler in Go

#13

Post by jmnicolas » Mon, Sep 23, 2019, 9:34 AM UTC

I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.

Re: Building a Dark Web Crawler in Go

#14

Post by creekorful » Mon, Sep 23, 2019, 9:37 AM UTC

jmnicolas wrote: post #13

I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.

You are right. That's why it's an educational project and not a public search engine

Re: Building a Dark Web Crawler in Go

#15

Post by Hitton » Mon, Sep 23, 2019, 9:43 AM UTC

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code.

From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just absolute ones.

Re: Building a Dark Web Crawler in Go

#16

Post by creekorful » Mon, Sep 23, 2019, 9:50 AM UTC

Hitton wrote: post #15

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just…

Author here. I'm fairly new to Golang too and it's my first project.

Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available.

Not taken but available.

I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and has to evolve. You can make a PR If you want to help me to improve it!

Re: Building a Dark Web Crawler in Go

#17

Post by robjan » Mon, Sep 23, 2019, 9:54 AM UTC

jmnicolas wrote: post #13

I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.

A crawler of the surface web will have this problem too.

Re: Building a Dark Web Crawler in Go

#18

Post by creekorful » Mon, Sep 23, 2019, 9:57 AM UTC

warent wrote: post #11

creekorful wrote: post #8

Earlier quoted context omitted.

Thank you! There's no practical purpose for the crawler. It's more an educational project than anything.

Weird, for some reason your comments are being instantly marked as "dead." I think there's some kind of filter that's tripping out for your account since it's new. I vouched for your two comments so hopefully everyone can see them now, but an admin (i.e. dang?) will need to look into this for a longer term solution.

Thank you sir. Actually my other comments are invisible too. That's weird.

Re: Building a Dark Web Crawler in Go

#19

Post by Phenomenit » Mon, Sep 23, 2019, 10:12 AM UTC

rolltiide wrote: post #5

I’ve been pretty surprised at how big hidden services have become Dread, the dark net reddit, is surprisingly vibrant I think its weird that people almost don't want to hear positive stories about dark net. It’ll be funny when news articles and romcoms just start “forgetting” to qualify their plot piece with the “its scary” trope

I thought dread was dead?

Re: Building a Dark Web Crawler in Go

#20

Post by alufers » Mon, Sep 23, 2019, 10:14 AM UTC

bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

I am no TOR expert, but how does decreasing the amount of hops or running your own relay decrease the privacy of other people?