Show HN: I have written a cloud native dark web crawler in Go
1–8 of 8 posts
Re: Show HN: I have written a cloud native dark web crawler in Go
#2Release 1.0.0 will be out very soon. Check it out guys!
Re: Show HN: I have written a cloud native dark web crawler in Go
#3Can you share some examples of what it finds? As long as it is safe to post that.
Re: Show HN: I have written a cloud native dark web crawler in Go
#4Can you share some examples of what it finds? As long as it is safe to post that.
I have an instance running for about 5 days+ with 1,300,000 docs. It's mostly huge amount of porns, some weird forums & lot of marketplaces.
There's still some good finds like a 'sysadmin' forum with huge amount of good posts & tips.
Re: Show HN: I have written a cloud native dark web crawler in Go
#5Can you share some examples of what it finds? As long as it is safe to post that.
I'll certainly publish an index of interesting (legal) hostnames on my GitHub account.
Re: Show HN: I have written a cloud native dark web crawler in Go
#6nice, what is actually needed to crawl onion sites ?
i am not a go dev, would try it with scala/java
Re: Show HN: I have written a cloud native dark web crawler in Go
#7nice, what is actually needed to crawl onion sites ? i am not a go dev, would try it with scala/java
Just a tor proxy (socks) instance, and a HTTP client configured to use the proxy. Other than that nothing more than what a web crawler should have.