Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: A free, privacy preserving, archive of public Discord servers

searchcord.io

Post reply on HN
Print view View original

41–50 of 91 posts

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#41

Post by squigz »

searchcord wrote: post #13
klntsky wrote: post #11

Earlier quoted context omitted.

I suggest you to remove the opt-out functionality and let it scrape private servers that it discovers via publicly posted invite links. You don't owe anyone posting on a public forum any privacy. Moreover, the most valuable data to search for is probably somewhat obscured.

Hey, Thanks for your suggestions. However, this does not work for a few reasons: 1. Joining servers is protected by increasingly difficult to solve captchas that have no commercially available solver. This is not a battle I want to fight. 2. There are a LOT of CSAM rings that spam invite links in public servers. This is also not something I want to go anywhere near. Moreover, after the fallout of spy.pet, I think it…

It is arguably more important that the world doesn't lose as much information as Discord now contains, or leave it gated behind logins and captchas and whatever other nonsense each server implements.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#42

Post by squigz »

searchcord wrote: post #17
treyd wrote: post #15

Would you consider making regular dumps of the database available in sharded torrents like Anna's Archive does so that users can back up the data themselves for preservation purposes? This would complicate retroactively removing users' activity, but that data could already be scraped. And related, I'd like to be able to run this locally for exports of guilds that I'm on myself. Is that even possible with the architec…

Hey, This is absolutely something I want to do, but at the guild level. The database itself is over 13TB which is much to large to create regular exports of. I will probably provide a SQLite export of each guild, regenerated each week/month. Anyone is free to download whatever they want in real time from the API. Thanks for your question!

Big +1 for dumps

You might try reaching out to Anna's Archive and see if this would be a dataset they'd be interested in helping host/distribute. I think they'd agree that such data is important and should be archived.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#43

Post by id44037319 »

This is great, I'll be returning to this tool often. Thanks.

A few suggestions and ideas for futher projects.

-allow for "keyword", -negate operators and "mult word string" searches, [Pubmed](https://pubmed.ncbi.nlm.nih.gov/advanced) is what I'd consider an Ideal search interface

-allow for regex, or direct sql lookups with limited query time ratelimited by POW. for example, if the server is under load, require a token from something like [anubis](https://anubis.techaro.lol/) and lower the maximum DB query time

-Index the title of all discussion/forum type posts with a VectorDB for semantic search. And add an option to sort by replies. (Like [answer overflow](https://github.com/AnswerOverflow/AnswerOverflow))This would make it possible to find relevant discussions among ~60B messages. ScyllaDB doesn't support vector search, so I'd suggest something like [usearch](https://github.com/unum-cloud/usearch) for a detached index. Embedding models are faster and smaller than most people realize, pick whatever's on top of the [mteb leaderboard](https://huggingface.co/spaces/mteb/leaderboard) after deciding on size.

-calculate the jaccard similarity (user overlap) between discord server members, this would allow for searching in "similar" severs, and potentially, mapping discord. [github](https://anvaka.github.io/map-of-github) [reddit](https://anvaka.github.io/map-of-reddit)

-fix doxing. Searching by is currently possible.

-expect the alternative to the cloudflare captcha to be abused, it's too simple for modern solvers.

-open source the stack? I'm interested in the scraper.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#44

Post by sReinwald »

searchcord wrote: post #26
johnQdeveloper wrote: post #21

> This is my first large scale project, so I'd love to hear your feedback! > I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived. 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique i…

Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(use…

The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance.

And "privacy preserving"? With a one-click opt-out, that 99.999% of the affected users will never even know exists because they have no idea their conversations are now part of your archive, and you want it indexed by search engines? That's not "privacy preserving" - that's a bad joke. If privacy was a genuine concern, this project wouldn't exist in its current form. What you're offering is an opt-out fig leaf for a mass data harvesting operation.

Most people using Discord, even on "public, discoverable" servers, aren't posting with the expectation that their words will be systematically scraped, archived indefinitely, and made globally searchable outside the platform's context. It's a fundamental misunderstanding (or willful dismissal) of user expectations on what is essentially a semi-public, yet distinctly siloed, platform. This isn't an open-web forum where content is implicitly intended for broad public consumption and indexing.

Look, I get the frustration that (likely) motivated this. Discord has become an information black hole for many communities, and the shift away from open, searchable forums for project support is a genuine problem I've been incredibly frustrated with myself. But this "solution" - creating a massive, non-consensual archive that tramples over user privacy (and platform terms) - creates far graver ethical and practical issues than the one it purports to solve.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#45

Post by deakam »

sReinwald wrote: post #44
searchcord wrote: post #26

Earlier quoted context omitted.

Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(use…

The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. And "privacy preserving"? With a one-click opt…

Ridiculous take. If you're posting in a server that's intentionally open to the public and accessible to anyone with a link or even indexed by server discovery you shouldn't expect privacy. That's just the basic reality of the internet.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#46

Post by searchcord »

sReinwald wrote: post #44
searchcord wrote: post #26

Earlier quoted context omitted.

Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(use…

The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. And "privacy preserving"? With a one-click opt…

> The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance.

Not really, it is not free to host and serve this data. If they want to get the data for free, they can get it directly from Discord. I did that work for them.

> And "privacy preserving"? With a one-click opt-out, that 99.999% of the affected users will never even know exists because they have no idea their conversations are now part of your archive, and you want it indexed by search engines? That's not "privacy preserving" - that's a bad joke. If privacy was a genuine concern, this project wouldn't exist in its current form. What you're offering is an opt-out fig leaf for a mass data harvesting operation.

Again, not really. It's impossible to search for users without already knowing what server they are in. This is functionally identical to Discord's in-built search feature.

> Most people using Discord, even on "public, discoverable" servers, aren't posting with the expectation that their words will be systematically scraped, archived indefinitely, and made globally searchable outside the platform's context. It's a fundamental misunderstanding (or willful dismissal) of user expectations on what is essentially a semi-public, yet distinctly siloed, platform. This isn't an open-web forum where content is implicitly intended for broad public consumption and indexing.

I believe that people need to realize that their messages were already being logged by many different moderation bots, just not publicized. This also happens on platforms like Telegram, look at the SangMata_BOT for example. Unless the messages are end to end encrypted, it was just a matter of time before they were scooped up and archived.

Thanks for your input, though, I really do want to build a platform that balances privacy and usability.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#47

Post by belst »

This clearly breaks discord TOS.

I did consent for discord to have my data, I did NOT consent to you having my data.

The discord TOS clearly state: > Our services might also provide you with access to other people’s content. You may not use this content without that person’s consent, or as allowed by law.

As I was not informed of the usage BEFORE it was taken, I could neither opt in nor opt out.

GDPR clearly states, even in the case of "legitimate interest" I have to be informed.

I only found this randomly, but if I hadn't, I had no idea of the data validation happening, so I couldn't opt out.

Technically, cool project. Legally, not so cool.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#48

Post by belst »

searchcord wrote: post #46
sReinwald wrote: post #44

Earlier quoted context omitted.

The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. And "privacy preserving"? With a one-click opt…

> The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. Not really, it is not free to host and serve…

> I believe that people need to realize that their messages were already being logged by many different moderation bots, just not publicized. Unless the messages are end to end encrypted, it was just a matter of time before they were scooped up and archived.

and that makes it ok for you to do aswell? Bots storing all the messages is also not ok, but they also don't publish it, so it is way less problematic

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#50

Post by Abishek_Muthian »

Congratulations and all the best.

Could Searchcord API be useful for discord servers which want to archive their chats to their own website?

e.g. I have discord server for my product and I want to copy the Q&A threads to FAQ section of my product website will Searchcord be useful for that or are there better solutions?

Post reply on HN