Live data from Hacker News

Show HN: A free, privacy preserving, archive of public Discord servers

searchcord.io

51–60 of 91 posts

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#51

Earlier quoted context omitted.

The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. And "privacy preserving"? With a one-click opt…

> The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. Not really, it is not free to host and serve…

Okay, the "not really" and "I'll solve that problem if and when" responses are... something else. It feels like you're speedrunning how to get into a world of trouble while hand-waving away every legitimate concern. Let's try to unpack this again, because your justifications are frankly baffling.

> Again, not really. It's impossible to search for users without already knowing what server they are in. This is functionally identical to Discord's in-built search feature.

That's not quite correct, and frankly it borders on willful obfuscation. In your own words elsewhere in this thread, you're eager for search engines to index this archive. That "privacy preserving" barrier of needing to know both a user ID and a server/channel id evaporates the moment Google or any other search engine hoovers up your pages. At that point, any combination of keywords, usernames, aliases, or snippets could reveal someone's posting history, across contexts and years. How is that "functionally identical" to Discord's walled-garden search or "privacy preserving"?

> I believe that people need to realize that their messages were already being logged by many different moderation bots, just not publicized.

This is a disingenuous deflection.

  - Moderation bots operate within a specific server, for a specific purpose (moderation, utility) defined by the server admins. Their logs are typically for admin/moderator use, not for creating a global, publicly searchable archive.
  - Users joining a server often see these bots, understand their function, and server admins explicitly add these bots. It's a known quantity. What you're doing is orders of magnitude different - an external, uninvited entity scraping everything discoverable and making it universally public.
  - Just a matter of time" is a lazy, fatalistic excuse for unethical data harvesting. Just because something can be technically scraped doesn't mean it should be, or that you doing so is fine.
    
Your "I really do want to build a platform that balances privacy and usability" line sounds utterly hollow when the entire foundation of the project demonstrates a profound misunderstanding, or disregard, for basic privacy, consent, and intellectual property.

Speaking of which... have you actually thought about the legal Pandora's Box you're prying open? Your casual "I'll deal with Discord's ToS issues if they arise" attitude is quaint, because Discord's ToS is likely the tip of a colossal iceberg of legal trouble.

You're not just 'breaking ToS', you're potentially looking at:

  - Data Protection Law Violations (GDPR, CCPA, etc.) because you're scraping personal data of EU/California (and other) residents without any lawful basis. The fines can be astronomical. "Opt-out" after the fact for data you had no right to take in the first place isn't how this works.
  - COPPA Violations if you scraped any messages from a 12-year-old on a "public, discoverable" server before their account was deleted by Discord. Guess who's holding that data now without parental consent? You.
  - Every original, creative message is copyrighted by its author. Roleplay, detailed discussions, code snippets, even well-crafted tirades – you're republishing millions of these. While not every "lol" is copyrightable, a massive volume of content on Discord absolutely is. "Fair use" for wholesale, non-transformative republication on this scale? Unlikely.
  - And last but not least, CSAM (Child Sexual Abuse Material): This is the nightmare scenario. You are scraping public Discord. Some public, poorly moderated Discords inevitably contain links to or text-based CSAM. Even if you don't intend to host it, if your scraper picks it up and it becomes accessible via your archive (even just a link), you are in profoundly serious trouble. "But I don't re-publish attachments" is irrelevant if you're archiving and re-publishing the links. This isn't just fines; this is potential prison time.
Good luck with all of this.

I hope you have a good lawyer, ideally multiple. You might need them.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#52

I'm in more than a hundred Discord servers. I've been wanting to scrape the members of each of them to discover the people with whom I share the most servers but we're not yet friends. Someone with 10+ would highly likely be a new friend since we'd have a lot of shared niche interests

This is something I have been trying to make as a way to learn about graph theory. If I can find a way to make it work efficiently, I will definitely add this.

Yeah that would be awesome! You could build a whole new social network on top of Discord. Whether in this way or others, I believe we'll all be finding an increasing number of hypercompatible people as technology advances.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#54

> This is my first large scale project, so I'd love to hear your feedback! > I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived. 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique i…

Hey, Thanks for your suggestions. > 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later) In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(use…

> In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(userid+guildid)`, truncated to the first 8 characters. Unfortunately, it was pretty hard to follow chats. I will try your idea and see how it works, though.

I suggest you do since tbh you are likely (as others have said) to be violating privacy laws with your current implementation + the discord ToS. If its anonymized better, you are less likely to be a target of someone who gets angry about not knowing you exist.

Up to you, your life your circus y'know?

> I have absolutely zero idea what industry would be interested in this, in what form, and if anyone would even pay.

LLM data collection if its not being bought via discord already directly.

Same reason I'd want to use highly anonymized and curated data from the roleplay / writing discords as training data. It is just I'd have to go through and anonymize your data and curate it / clean it up before I would dare to send it to an LLM for legal reasons.

If I send/share PII, I'd be screwed just like you will be if someone gets upset.

> I really don't care what people do with the data, as long as they are not spamming requests or using the data for commercial purposes without permission.

Fair, for me, this is for hobby implementations of solo roleplaying content similar to AI Dungeon and other implementations so its not commercial but my use case (for your purposes) would be better served by just being able to download a database dump (properly anoynmized or me doing it) for specific servers since most data is useless to me that you collect since I've got a specific goal in mind and want to minimize data collection for legal liability reasons. (i.e. non-commercial roleplaying with no PII or other privacy risky info is likely to be a safe use case)

EDIT:

I'd consider dropping attachments + links and only recording text as well for CSAM and other abusive material reasons. I doubt you have the moderation in place to protect yourself.

Pictures and videos and what not are a lot more dangerous to you than text would be. (i.e. despite what people say about it, realistically, most text in a public forum on the internet w/o PII is not going to get you hit with fines)

That said, personally, I would not publish this as you have because I don't have that kind of risk tolerance but I can see it being "safe enough" for some people. But the images/attachements are in "are you really sure you want to do that? You could go bankrupt" territory.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#55
post #15

Would you consider making regular dumps of the database available in sharded torrents like Anna's Archive does so that users can back up the data themselves for preservation purposes? This would complicate retroactively removing users' activity, but that data could already be scraped. And related, I'd like to be able to run this locally for exports of guilds that I'm on myself. Is that even possible with the architec…

Hey, This is absolutely something I want to do, but at the guild level. The database itself is over 13TB which is much to large to create regular exports of. I will probably provide a SQLite export of each guild, regenerated each week/month. Anyone is free to download whatever they want in real time from the API. Thanks for your question!

I was imagining something like a month-by-month historical export so only recent data would need to be regularly exported.

SQLite dumps would be awesome to work with though.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#56
post #53

How does this tool handle it when a user deletes or edits their message? Is the message still logged on your site?

This all depends on when the message is scraped. The scraper does not go back and check for edits/deletions. This is something that while could be added, would be incredibly inefficient.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#57
post #45

Earlier quoted context omitted.

The sheer audacity here is quite something. You're stating people can't use your scraped data for commercial purposes "without permission," while your entire project is built on vacuuming up content from countless users without their permission, and in direct violation of Discord's ToS. That's not just a double standard; it's bordering on next-level cognitive dissonance. And "privacy preserving"? With a one-click opt…

Ridiculous take. If you're posting in a server that's intentionally open to the public and accessible to anyone with a link or even indexed by server discovery you shouldn't expect privacy. That's just the basic reality of the internet.

No, what's "ridiculous" is this simplistic, black-and-white framing that deliberately ignores any nuance, the concept of contextual integrity or reasonable user expectations.

Of course, no one expects absolute secrecy in a public-facing Discord server. That's a straw man. The issue isn't about some naive belief that messages are invisible. It's about the scope, permanence, and method of access and archiving.

People participating in public Discord spaces have reasonable contextual expectations about how their words will be accessed and by whom. They expect their messages to be seen by current and maybe future server members - not extracted, permanently archived, and made globally searchable by entirely unrelated third parties.

This is similar to how conversations in a public park are technically "public," but most people would be rightfully disturbed if someone recorded everything, transcribed it, published it online with their names attached, and made it all searchable forever. Just because something isn't strictly private doesn't mean any and all forms of collection, republication, and indexing are ethically justified.

If you can't see the distinction between "not perfectly private within this specific semi-public space" and "archived indefinitely, and globally searchable forever by anyone, anywhere, for any reason," then you're either arguing in bad faith or your understanding of these issues is so superficial that further engagement is pointless.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#59
post #45

Earlier quoted context omitted.

Ridiculous take. If you're posting in a server that's intentionally open to the public and accessible to anyone with a link or even indexed by server discovery you shouldn't expect privacy. That's just the basic reality of the internet.

No, what's "ridiculous" is this simplistic, black-and-white framing that deliberately ignores any nuance, the concept of contextual integrity or reasonable user expectations. Of course, no one expects absolute secrecy in a public-facing Discord server. That's a straw man. The issue isn't about some naive belief that messages are invisible. It's about the scope, permanence, and method of access and archiving. People p…

[deleted]

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#60
post #45

Earlier quoted context omitted.

Ridiculous take. If you're posting in a server that's intentionally open to the public and accessible to anyone with a link or even indexed by server discovery you shouldn't expect privacy. That's just the basic reality of the internet.

No, what's "ridiculous" is this simplistic, black-and-white framing that deliberately ignores any nuance, the concept of contextual integrity or reasonable user expectations. Of course, no one expects absolute secrecy in a public-facing Discord server. That's a straw man. The issue isn't about some naive belief that messages are invisible. It's about the scope, permanence, and method of access and archiving. People p…

[flagged]
Post reply on HN