Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A free, privacy preserving, archive of public Discord servers

searchcord.io

Post reply on HN
Print view View original

11–20 of 91 posts

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#11

Post by klntsky »

searchcord wrote: post #3
hofrogs wrote: post #2

This is really cool and actually useful for peeking behind those annoying login walls. What software do you use to store/index/search in so much data? How did you get the data in the first place? Discord isn't exactly known for letting its data be available easily. Have the administrators of the guilds asked you for this? Have you contacted them and made them aware after the fact?

Hey, Thanks for your feedback. For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable servers. I plan to write a blog post about the technical aspect of how this was done soon. Admins of these servers weren't contacted, as the content indexed is already publicly accessible, c…

I suggest you to remove the opt-out functionality and let it scrape private servers that it discovers via publicly posted invite links. You don't owe anyone posting on a public forum any privacy. Moreover, the most valuable data to search for is probably somewhat obscured.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#12

Post by uniqueuid »

searchcord wrote: post #6
hofrogs wrote: post #4

Earlier quoted context omitted.

That's a lot of compute, how much does it cost to keep it running? I don't see how that project would generate any income on its own

I already own the hardware, so I only pay for colocation and transit. It's probably a lot less than you think. I hope to find some way to monetize it, but it is cheap enough that I can keep it running for quite a long time without any income.

Wow, that must be quite expensive! You said the files alone are a few PB. So at least 2PB / 8 servers ~= 250TB per server, which would probably put each server at > 20k $ (unless you’re putting it together with duct tape and scraps, but even then the disks will cost a ton).

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#13

Post by searchcord »

klntsky wrote: post #11
searchcord wrote: post #3

Earlier quoted context omitted.

Hey, Thanks for your feedback. For software, I use ScyllaDB and Elasticsearch. It's split across 6 physical nodes (8 including the CDN). Data collection is handled using standard user accounts, accessing only public, discoverable servers. I plan to write a blog post about the technical aspect of how this was done soon. Admins of these servers weren't contacted, as the content indexed is already publicly accessible, c…

I suggest you to remove the opt-out functionality and let it scrape private servers that it discovers via publicly posted invite links. You don't owe anyone posting on a public forum any privacy. Moreover, the most valuable data to search for is probably somewhat obscured.

Hey,

Thanks for your suggestions. However, this does not work for a few reasons:

1. Joining servers is protected by increasingly difficult to solve captchas that have no commercially available solver. This is not a battle I want to fight.

2. There are a LOT of CSAM rings that spam invite links in public servers. This is also not something I want to go anywhere near.

Moreover, after the fallout of spy.pet, I think it is very important that users are able to opt out.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#14

Post by searchcord »

uniqueuid wrote: post #12
searchcord wrote: post #6

Earlier quoted context omitted.

I already own the hardware, so I only pay for colocation and transit. It's probably a lot less than you think. I hope to find some way to monetize it, but it is cheap enough that I can keep it running for quite a long time without any income.

Wow, that must be quite expensive! You said the files alone are a few PB. So at least 2PB / 8 servers ~= 250TB per server, which would probably put each server at > 20k $ (unless you’re putting it together with duct tape and scraps, but even then the disks will cost a ton).

Hey,

Not exactly. Attachments are only fetched from Discord as the user requests them. This means that the vast majority of attachments are never stored on my server. Right now, I only have about 280TB of attachments locally on my own infrastructure. You can see more stats here: https://searchcord.io/about

Thanks for your question!

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#15

Post by treyd »

Would you consider making regular dumps of the database available in sharded torrents like Anna's Archive does so that users can back up the data themselves for preservation purposes? This would complicate retroactively removing users' activity, but that data could already be scraped.

And related, I'd like to be able to run this locally for exports of guilds that I'm on myself. Is that even possible with the architect you've built?

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#17

Post by searchcord »

treyd wrote: post #15

Would you consider making regular dumps of the database available in sharded torrents like Anna's Archive does so that users can back up the data themselves for preservation purposes? This would complicate retroactively removing users' activity, but that data could already be scraped. And related, I'd like to be able to run this locally for exports of guilds that I'm on myself. Is that even possible with the architec…

Hey,

This is absolutely something I want to do, but at the guild level. The database itself is over 13TB which is much to large to create regular exports of. I will probably provide a SQLite export of each guild, regenerated each week/month. Anyone is free to download whatever they want in real time from the API.

Thanks for your question!

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#20

Post by searchcord »

pabs3 wrote: post #19

There are some Discord archives on archive.org too btw. https://archive.org/search?query=subject%3A%22DiscordChatExp... https://archive.org/search?query=subject%3A%22archiveteam_di... https://wiki.archiveteam.org/index.php/Discord

Hey,

This is interesting, I somehow missed this. Unfortunately, those are not full text searchable. Maybe I will download them and import them into Searchcord, with proper credit of course.

Thanks for this!

Post reply on HN