Earlier quoted context omitted.
You are right. That's why it's an educational project and not a public search engine
IANAL but "educational project" won't fly in court, and nor should it.
Unless you’re Pete Townshend.
Building a Dark Web Crawler in Go
51–60 of 113 posts
Re: Building a Dark Web Crawler in Go
#52Sounds like a recipe to score yourself a free FBI visit
Generally the FBI doesn't give a hoot until you start distributing illegal stuff....
Re: Building a Dark Web Crawler in Go
#53Earlier quoted context omitted.
Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children
I didn't talk about the dev getting contacted about third parties abusing the software, but about the dev keeping a DB of indexed content for development/testing/"education" that would most likely include illegal-to-possess content. And that some eager police people like to "inconvenience" people connected to TOR somehow isn't exactly new, either. E.g. there have been multiple raids against TOR exit node operators in…
It’s called Tor not TOR.
Re: Building a Dark Web Crawler in Go
#54Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just…
Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and ha…
Offtopic nitpick:
>Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available.
I sorta understand what you mean, technically it's 32 characters per position (5 bits), and 16 positions. In v2 .onion addresses, that is.
v3 ones [1] are 56 positions, but not all the bits are used for addressing, so the same formula wouldn't quite work to calculate real theoretical capacity. IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course).
[1] https://trac.torproject.org/projects/tor/wiki/doc/NextGenOni...
Re: Building a Dark Web Crawler in Go
#55First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
i just like how 'dark web' turned into 'Tor' at some point :'). there's tons of others... :s guess ppl forgot
A short list of those tons would be wonderfully helpful to share. :)
Re: Building a Dark Web Crawler in Go
#56Earlier quoted context omitted.
I didn't talk about the dev getting contacted about third parties abusing the software, but about the dev keeping a DB of indexed content for development/testing/"education" that would most likely include illegal-to-possess content. And that some eager police people like to "inconvenience" people connected to TOR somehow isn't exactly new, either. E.g. there have been multiple raids against TOR exit node operators in…
It’s called Tor not TOR. https://support.torproject.org/about/why-is-it-called-tor/
Who gives a fuck?
Re: Building a Dark Web Crawler in Go
#57Earlier quoted context omitted.
There is a bunch of stories like that. E.g. https://www.ccc.de/en/updates/2018/hausdurchsuchungen-bei-ve... > On June 20th, board members of the „Zwiebelfreunde“ association in multiple German cities had their homes searched under the dubious pretence that they were „witnesses“ while their computers and storage media were confiscated.
That's something very different though. Exit nodes are providing a service and are, for all intents and purposes, the only visible client on the clearnet (and might not even be involved: there's nothing stopping you from running a private proxy on the same machine you run your exit node on). TOR-developers that do not run exit nodes but contribute to TOR typically don't get searched, at least to my knowledge. Content…
The Zwiebelfreunde raids were not because of their TOR (hi dewey) activities, but rather they collected donations for the riseup email service.
If the police can convince a judge to raid the board members and their families of registered club just because they, among many other things, collected some donations for an US org, then some overzealous police detective or DA going after some dev who made a webcrawler for the "dark web" and is probably in possession (knowingly or not) or illegal content isn't much of a stretch either.
Re: Building a Dark Web Crawler in Go
#58Earlier quoted context omitted.
Actually, the opposite is true. People who actually need anonymity need to hide among traffic that is boring. If you reduce the number of hops your crawler is using, you're reducing the amount of boring traffic and making it easier to find the interesting people. Running a relay in addition to using Tor in the normal way is a good idea, however, as it increases the bandwidth of the network.
There is plenty of traffic in which to hide already. Another bot making and breaking random connections 24/7 is of no additional help.
For one project, I agree, it makes no difference.
But if the received wisdom becomes "if you're not rebelling against an oppressive regime, you should only be using 1 hop" then the advice has real harmful effects.
Re: Building a Dark Web Crawler in Go
#59First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
Actually, the official term is Onion Services (https://2019.www.torproject.org/docs/onion-services.html.en check out the first paragraph)
Re: Building a Dark Web Crawler in Go
#60Earlier quoted context omitted.
Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. I would tread lightly crawling the dark web. There are cases where the FBI has admitted to running services on TOR, to collect IP addresses: https://www.wired.com/2013/09/freedom-hosting-fbi/
> Right, but possession of those items do not constitute a violation of law. Whereas, the possession of child exploitation material does. No matter the reasoning. What about when the FBI/CIA does it? Genuine question.
No one watches the watchmen.