Live data from Hacker News

Show HN: A free, privacy preserving, archive of public Discord servers

searchcord.io

21–30 of 91 posts

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#21
> This is my first large scale project, so I'd love to hear your feedback!

> I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived.

1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later)

2) You not seem to have a clear rate limit documentation. If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer.

3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#24
Do you plan to handle servers where you need to do some action (like send a message) to join all channels ?

I was scrolling through the home page and came across afew where the only channels you're allowed to access are the verify-yourself or welcome channels.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#25

All discord servers require an invitation link as far as I know, do you consider a link you find online as a public server?

Some very large servers are eligible for what Discord calls "discovery". This makes their data visible without joining the server. You can find a list of those on Discord's site here: https://discord.com/servers

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#26

> This is my first large scale project, so I'd love to hear your feedback! > I have placed restrictions on searching directly by user ID to prevent doxing. I also made the opt out process one click, for those who do not want to be archived. 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique i…

Hey,

Thanks for your suggestions.

> 1) I'd suggest anonymizing the usernames / author ids to something more privacy friendly such as how some image sites were generating 3-4 random words as a human readable unique id. This removes a lot of the reason people would opt out (i.e. posts being tracked down years later)

In the original iteration of Searchcord, it used to work similarly to that. The username was `sha256(userid+guildid)`, truncated to the first 8 characters. Unfortunately, it was pretty hard to follow chats. I will try your idea and see how it works, though.

> 2) You not seem to have a clear rate limit documentation.

This is a good idea. The rate limit varies by endpoint, and I haven't gotten around to documenting each one.

> If you are asking people to pay for commercial use, I'd suggest making it clear what the rough original limits are as well as the rough price range of what you'd offer.

I have absolutely zero idea what industry would be interested in this, in what form, and if anyone would even pay.

> 3) Tbh, the only real thing I want from this project is basically narrative / roleplay / writing content for LLM reasons as I'm trying to build a rules-oriented system that narrates via LLM. If you don't want people using this data for this purpose, I'd suggest making that clear.

I really don't care what people do with the data, as long as they are not spamming requests or using the data for commercial purposes without permission.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#27

Do you plan to handle servers where you need to do some action (like send a message) to join all channels ? I was scrolling through the home page and came across afew where the only channels you're allowed to access are the verify-yourself or welcome channels.

Probably not. Discord will aggressively captcha you and every server has a different implementation of verification. It might be possible with a captcha solver and then some LLM to figure out the next steps.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#28

All discord servers require an invitation link as far as I know, do you consider a link you find online as a public server?

Some very large servers are eligible for what Discord calls "discovery". This makes their data visible without joining the server. You can find a list of those on Discord's site here: https://discord.com/servers

Suggestion: a bot for smaller servers that do want to be archived like a public forum. Their admins could install the bot themselves and perhaps specify what channels they want archived.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#29
post #28

Earlier quoted context omitted.

Some very large servers are eligible for what Discord calls "discovery". This makes their data visible without joining the server. You can find a list of those on Discord's site here: https://discord.com/servers

Suggestion: a bot for smaller servers that do want to be archived like a public forum. Their admins could install the bot themselves and perhaps specify what channels they want archived.

This is something I have already completed but have not finished bug testing. The bot also includes functionality to recover any server in case it was nuked/wiped and Searchcord has a backup of it. It uses webhooks to resend the messages so you have an approximation of what the channels used to be.

Re: Show HN: A free, privacy preserving, archive of public Discord servers

#30
nice project. how are you going to handle the issues involved with breaking Discord's TOS?

> "scraping our services without our written consent"

additionally, are these pages indexable? i know of other projects (opt-in) that create pages made from user discussion.

Post reply on HN