Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A free, privacy preserving, archive of public Discord servers

searchcord.io

Post reply on HN
Print view View original

1–10 of 91 posts

Show HN: A free, privacy preserving, archive of public Discord servers

#1

Post by searchcord »

Hey HN!

I have been working on this project for a while, and I think this solves a problem that a lot of people here have: not being able to easily search Discord servers.

Currently, I only scrape servers that are marked as "discoverable" on Discord. However, if there's enough interest in the project, I'm open to adding specific servers by request. I'm primarily focused on informational servers rather than casual hangout spaces, such as open source projects, Minecraft mods, and support communities for tools, services, or platforms (for example, hosting providers).

I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived.

This is my first large scale project, so I'd love to hear your feedback!

Show HN: A free, privacy preserving, archive of public Discord servers
searchcord.io

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#2

Post by hofrogs »

This is really cool and actually useful for peeking behind those annoying login walls. What software do you use to store/index/search in so much data? How did you get the data in the first place? Discord isn't exactly known for letting its data be available easily. Have the administrators of the guilds asked you for this? Have you contacted them and made them aware after the fact?

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#3

Post by searchcord »

hofrogs wrote: post #2

This is really cool and actually useful for peeking behind those annoying login walls. What software do you use to store/index/search in so much data? How did you get the data in the first place? Discord isn't exactly known for letting its data be available easily. Have the administrators of the guilds asked you for this? Have you contacted them and made them aware after the fact?

Hey,

Thanks for your feedback.

For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable servers. I plan to write a blog post about the technical aspect of how this was done soon.

Admins of these servers weren't contacted, as the content indexed is already publicly accessible, comparable to a forum like this or public subreddit. That said, I understand the sensitivity around data visibility, and I've made it very simple for any user to opt out of indexing at any time. Private or invite-only servers are, of course, completely excluded.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#4

Post by hofrogs »

searchcord wrote: post #3
hofrogs wrote: post #2

This is really cool and actually useful for peeking behind those annoying login walls. What software do you use to store/index/search in so much data? How did you get the data in the first place? Discord isn't exactly known for letting its data be available easily. Have the administrators of the guilds asked you for this? Have you contacted them and made them aware after the fact?

Hey, Thanks for your feedback. For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable servers. I plan to write a blog post about the technical aspect of how this was done soon. Admins of these servers weren't contacted, as the content indexed is already publicly accessible, c…

That's a lot of compute, how much does it cost to keep it running? I don't see how that project would generate any income on its own

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#6

Post by searchcord »

hofrogs wrote: post #4
searchcord wrote: post #3

Earlier quoted context omitted.

Hey, Thanks for your feedback. For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable servers. I plan to write a blog post about the technical aspect of how this was done soon. Admins of these servers weren't contacted, as the content indexed is already publicly accessible, c…

That's a lot of compute, how much does it cost to keep it running? I don't see how that project would generate any income on its own

I already own the hardware, so I only pay for colocation and transit. It's probably a lot less than you think. I hope to find some way to monetize it, but it is cheap enough that I can keep it running for quite a long time without any income.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#7

Post by searchcord »

legionof7 wrote: post #5

I've been looking for something like this for so long, thanks for making! There's so much stuff locked in Discord now that forums have fallen in popularity, think this sort of thing really helps unlock that knowledge again.

Thanks for your feedback! <3

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#8

Post by hofrogs »

searchcord wrote: post #6
hofrogs wrote: post #4

Earlier quoted context omitted.

That's a lot of compute, how much does it cost to keep it running? I don't see how that project would generate any income on its own

I already own the hardware, so I only pay for colocation and transit. It's probably a lot less than you think. I hope to find some way to monetize it, but it is cheap enough that I can keep it running for quite a long time without any income.

Thanks for this. Well good luck with keeping it up, it's a really useful service.
Post reply on HN