Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Dark Web Crawler in Go

creekorful.me

Post reply on HN
Print view View original

91–100 of 113 posts

Re: Building a Dark Web Crawler in Go

#91

Post by kodablah »

akklesed wrote: post #54
creekorful wrote: post #16

Earlier quoted context omitted.

Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and ha…

Offtopic nitpick: >Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. I sorta understand what you mean, technically it's 32 characters per position (5 bits), and 16 positions. In v2 .onion addresses, that is. v3 ones [1] are 56 positions, but not all the bits are used for addressing, so the same formula wouldn't quite…

> IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course)

V3 addresses are just ed25519 pub keys and a couple byte changes. You can use Go libraries like Bine [0] to generate as many V3 (or V2) addresses as you want from keys.

0 - https://godoc.org/github.com/cretz/bine/torutil#OnionService...

Re: Building a Dark Web Crawler in Go

#92

Post by SquishyPanda23 »

rytill wrote: post #71
MuffinFlavored wrote: post #66

Earlier quoted context omitted.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

Anonymity allows you to sow an action without reaping the societal karma of the action. In a good, free society, maybe anonymity isn’t important. But in a bad society, one in which collaboration on a cause is punished, but each individual desperately wants to collaborate and change something fundamental... Anonymity allows the planning of synchronized action. —— Mass or targeted misinformation also threatens the plan…

> In a good, free society, maybe anonymity isn’t important

There is something of a Catch 22 here I think. A society in which it's difficult or costly to be do something anonymously is essentially a society with total surveillance.

And it seems intuitive that a surveillance state is not good or free.

So there is an argument you can make that good and free societies should allow anonymity even if they are the sort of society where it is least needed.

Re: Building a Dark Web Crawler in Go

#94

Post by rolltiide »

clashmeifyoucan wrote: post #59
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

Actually, the official term is Onion Services ( https://2019.www.torproject.org/docs/onion-services.html.en check out the first paragraph)

wow news to me, wonder how long that will take to catch on

onion like "layered" services

Re: Building a Dark Web Crawler in Go

#95

Post by marc_abonce »

creekorful wrote: post #14
jmnicolas wrote: post #13

I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.

You are right. That's why it's an educational project and not a public search engine

By the way, Ahmia publishes a blacklist with the hashed addresses for every onion URL that they have discovered to host abusive content. You could use that blacklist to filter out those sites so you don't even crawl them and also to periodically purge any matching URL that may have already made it into your index.

Re: Building a Dark Web Crawler in Go

#96

Post by metamet »

goda90 wrote: post #67
MuffinFlavored wrote: post #66

Earlier quoted context omitted.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

The "not illegal" part is the catch here. Something can be illegal but still legitimate if the laws are illegitimate. Someone trying to exercise freedom of speech or the press under an oppressive regime would need anonymity to avoid being jailed or killed.

I think this is an important cultural reframing that needs to occur, sooner rather than later.

Most people, when they hear of things like "the dark web" and cryptocurrency think about the massively publicized instances of drug trafficking and ordering a hit on someone.

It's going to take a lot of work to reframe the utility and purpose of them to a more universal, humanitarian angle.

People in this world live in oppressive circumstances. This should be viewed as a step toward helping them not be systemically silenced.

Re: Building a Dark Web Crawler in Go

#97

Post by pests »

alufers wrote: post #20
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

I am no TOR expert, but how does decreasing the amount of hops or running your own relay decrease the privacy of other people?

I don't think the other reply you got answered your question.

The thought is that other people are hiding in your noise. Make less noise, other people stand out more.

Re: Building a Dark Web Crawler in Go

#98

Post by pests »

airstrike wrote: post #80
nickles wrote: post #68

Earlier quoted context omitted.

Journalism, whistleblowing, accessing censored information, preventing stalkers from tracking you, etc.

How exactly can a stalker track me online if I simply stop logging in to services? Honest question, because I don't know why Tor would be any better than simply browsing in incognito mode

You still are sending packets over your router, to your ISP, out into the internet, and to the destination server. You leave fingerprints everywhere (browser, os, resolution, fonts, enabled features, cookies, etc.) Forever cookies, DNS cookies. The list goes on.

You are being tracked at the very least as an abstract person. If any of the above fingerprints are linked to a real identity (logging in just once even, or posting your email on a forum) then you are now being tracked even logged out.

If you use Tor and log into services it has no benefits. Tor, the browser and other distributions, will still leave fingerprints but they will no longer be unique and match only you but everyone using tor.

Tor, the protocol, will hide that you are the one receiving or sending packets.

"why Tor would be any better than simply browsing in incognito mode"

Incognito mode does nothing to hide packets or source/destination you are communicating to. Your ISP could literally pull up all non-https sites you visited along with their content, assignable to you, airstrike, as a person. Tor would block this.

Re: Building a Dark Web Crawler in Go

#99

Post by SkyBelow »

metamet wrote: post #96
goda90 wrote: post #67

Earlier quoted context omitted.

The "not illegal" part is the catch here. Something can be illegal but still legitimate if the laws are illegitimate. Someone trying to exercise freedom of speech or the press under an oppressive regime would need anonymity to avoid being jailed or killed.

I think this is an important cultural reframing that needs to occur, sooner rather than later. Most people, when they hear of things like "the dark web" and cryptocurrency think about the massively publicized instances of drug trafficking and ordering a hit on someone. It's going to take a lot of work to reframe the utility and purpose of them to a more universal, humanitarian angle. People in this world live in oppr…

Under the idea of legitimate and illegitimate use, why wouldn't drug trafficking be legitimate? It gets drugs off the streets, decreases violence compare to street level drug dealing, increases safety (while the reputation of online sellers isn't a great metric, we are talking relative to the person on the street corner), and generally involves only adults.

If one is willing to argue that the US government throwing someone in a cage because the grew or bought the wrong plant is legitimate, then I don't see how they have any standing to complain about China doing something for someone who held up the wrong sign at a protest.

Re: Building a Dark Web Crawler in Go

#100

Post by SkyBelow »

MuffinFlavored wrote: post #66
bureaucrat wrote: post #4

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

> other’s desperate need for anonymity Can somebody list some positive, legitimate, not illegal uses to desperately be anonymous?

The whole idea is to enable it for positive legitimate uses that are illegal. If it isn't illegal, then you really don't need that level of anonymity and there are simpler technical solutions that don't give up as much quality.
Post reply on HN