Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Dark Web Crawler in Go

creekorful.me

Post reply on HN
Print view View original

61–70 of 113 posts

Re: Building a Dark Web Crawler in Go

#61

Post by bureaucrat »

clashmeifyoucan wrote: post #59
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

Actually, the official term is Onion Services ( https://2019.www.torproject.org/docs/onion-services.html.en check out the first paragraph)

It seems that they changed the name from hidden service to onion service.

Re: Building a Dark Web Crawler in Go

#62

Post by luckylion »

rndgermandude wrote: post #57
luckylion wrote: post #45

Earlier quoted context omitted.

That's something very different though. Exit nodes are providing a service and are, for all intents and purposes, the only visible client on the clearnet (and might not even be involved: there's nothing stopping you from running a private proxy on the same machine you run your exit node on). TOR-developers that do not run exit nodes but contribute to TOR typically don't get searched, at least to my knowledge. Content…

The Zwiebelfreunde raids were not because of their TOR (hi dewey) activities, but rather they collected donations for the riseup email service. If the police can convince a judge to raid the board members and their families of registered club just because they, among many other things, collected some donations for an US org, then some overzealous police detective or DA going after some dev who made a webcrawler for t…

They can, but I'm not so sure that "may or may not possess illegal content" is enough for a search warrant - it's true for most of the population after all (running an exit node or collecting funds on behalf of a third party on the other hand is true only for a tiny fraction of the population). Granted, the chances are somewhat higher for IT people and higher still for people that write crawlers, but "we think he might, it's not impossible that he doesn't" is a bit thin, and unless they're trying to go after you for unrelated reasons, DAs don't love to have their asses handed to them by judges.

Re: Building a Dark Web Crawler in Go

#63

Post by creekorful »

mschuster91 wrote: post #23

To anyone experimenting with such stuff, take care and don't make your services publically available. Especially the dark web is full with highly illegal content such as child pornography and in some jurisdictions even "involuntary possession" such as in browser caches may be enough to convict you.

Do you think I should add a license in Github to mention that? To protect me and the users who will use the crawler?

Re: Building a Dark Web Crawler in Go

#64

Post by ValleZ »

creekorful wrote: post #16
Hitton wrote: post #15

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just…

Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and ha…

I assure you that there are less than 10k unique onion addresses. This is a huge overkill to have a distributed system to crawl something this small.

edit: onion services, not addresses

Re: Building a Dark Web Crawler in Go

#65

Post by rndgermandude »

luckylion wrote: post #62
rndgermandude wrote: post #57

Earlier quoted context omitted.

The Zwiebelfreunde raids were not because of their TOR (hi dewey) activities, but rather they collected donations for the riseup email service. If the police can convince a judge to raid the board members and their families of registered club just because they, among many other things, collected some donations for an US org, then some overzealous police detective or DA going after some dev who made a webcrawler for t…

They can , but I'm not so sure that "may or may not possess illegal content" is enough for a search warrant - it's true for most of the population after all (running an exit node or collecting funds on behalf of a third party on the other hand is true only for a tiny fraction of the population). Granted, the chances are somewhat higher for IT people and higher still for people that write crawlers, but "we think he mi…

In many jurisdictions around the world it is enough. In Germany you need a "begründeter Anfangsverdacht" (reasonable initial suspicion) and what's reasonable is essentially up to the judge signing the warrant.

Hell, they used to raid people accused by third parties of copyright infringement (for private personal use), about a decade back or one and a half, but thankfully that stopped now. They would come early in the morning, present you with a warrant that said "based on evidence provided by ..." (i.e. somebody somehow collected an IP address you might have used off of some file sharing swarm), take all your shit and scare your neighbors and quite often your parents because they raided a lot of minors too.

I know two people who this happened to personally. One guy wanted his stuff back, to which the DA replied that if they got to keep his stuff they would drop the case (I kid you not), and the other guy had his stuff returned about 2 years later, except for his HDDs. And his stuff not only included computers, CDs, DVDs, a printer, but they had actually seized books... paper ones... wat. Neither was convicted of any crimes in the end (IIRC they both had the thing dropped because "minor offense" not worth pursuing).

Turns out that a little googling of the German internet around that time turned up a lot of similar cases and some people claiming the police and DAs did that to get new computers "cheap"...

Re: Building a Dark Web Crawler in Go

#66

Post by MuffinFlavored »

bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity

Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Re: Building a Dark Web Crawler in Go

#67

Post by goda90 »

MuffinFlavored wrote: post #66
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

The "not illegal" part is the catch here. Something can be illegal but still legitimate if the laws are illegitimate. Someone trying to exercise freedom of speech or the press under an oppressive regime would need anonymity to avoid being jailed or killed.

Re: Building a Dark Web Crawler in Go

#68

Post by nickles »

MuffinFlavored wrote: post #66
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Journalism, whistleblowing, accessing censored information, preventing stalkers from tracking you, etc.

Re: Building a Dark Web Crawler in Go

#69

Post by bavell »

MuffinFlavored wrote: post #66
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Whistleblowing for one, though legality is questionable I suppose. To me this falls under the more general premise of fighting back against a (perceived or otherwise) tyrannical govt or organization, a very grey legal area.

Re: Building a Dark Web Crawler in Go

#70

Post by zrth »

MuffinFlavored wrote: post #66
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Would you mind posting your full name and personal details here please now? If not, then is seems you are also (desperate) in need of being anonymous right now...
Post reply on HN