131–140 of 204 posts
To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 It's 118 gigabytes of JSON.
Is there a faster way to download than the link there? It's steady, but a roughly 11 hour download.
Fantastic. I wonder how many random technical info is buried in these servers. I hate what it's done for game modding.
I wonder if LLM companies don't have ways to scrape private Discord servers already. Creating accounts and pulling all the historic data doesn't sound impossible.
It's like back in the days of IRC. People just logged all of it.
Earlier quoted context omitted.
> public spaces on the internet But discord servers aren't considered "public spaces", hence the concept of an "invite". This is akin to someone revealing they've been going to private parties and secretly recording everything. It might not be illegal, but it's definitely not polite.
These servers do seem to be public, in the sense that anyone can join without explicitly being invited. It sounds more like they went to the mall, picked 10% of the stores, and recorded conversations taking place in those stores.
The biggest problem that sucks about discord is that it isn't normally publicly searchable. And it seems to be a modern replacement for internet forums which historically were publicly searchable and often had a lot of great information about various hobbies and things.
There is a special place in hell for software (including game) developers who exclusively use Discord to release patch notes, documentation, technical support, etc.
Now those of us who've been around the block know that Discord is merely the latest iteration on chat servers such as IRC. I'm interested to know, from anyone here who's an IRC operator or server/network admin, how the IRC community deals with scraping and bots, because in the early 90s, it was never an issue of corporate Terms of Service or legalese, but typically handled by community standards, and probably, people…
A big difference is that on Discord anybody who joins a server gets access to full history of chat logs whereas with IRC you don't get access to any past logs. So compared to IRC, Discord users should have an even lower expectation of privacy.
Now imagine the data mining that Discord can do on the complete DM history of every user. It’s not e2ee, remember.
So many users expect their entire decade+ history of DM contents, attachments included, to be available wherever they are and on any device, gated only by having their login/2fa or passkey. Switching to E2EE would be a major overhaul of that expectation, and it would be a huge task to train users to now keep their encryption key safe, backed up, and available across multiple devices.
Although, mostly unrelated, is that they absolutely are going to have to cull old attachments eventually. There are attachments sitting in their GCP buckets that haven't been accessed since 2015. I'm sure their storage bill is in at least a few million a month at this point, even if most is marked coldline.
Earlier quoted context omitted.
There is a special place in hell for software (including game) developers who exclusively use Discord to release patch notes, documentation, technical support, etc.
Why? I follow a lot of solo game developers who only use Discord and I completely understand them not wanting do deal with multiple platforms. They should focus on on the game.
Discord is walled and hard to search. If a channel or server closes then all that information is lost.
Tons of data will be lost to discord when it goes down.
Idk if you've ever tried to use discord for mods or other software but it sucks. It's confusing. Information isn't cataloged well. It's search sucks. It just isn't good for this kind of thing.
I don't know if Discord fixed it as I haven't checked in a few years, but I tinkered with scraping some public Discords and I found that I could see hidden channels, not the data, but the channel names, which could do things like reveal to me if the same Discord was used for in-house development if it was a product Discord. Not great.
Earlier quoted context omitted.
The algebraic topology server probably contains a huge number of treasures in modern research algebraic topology. I really really hope it's archived in full
Its not difficult to archive yourself, if you really care[0] I use a dedicated alt account to archive tons of various servers I'm in, and auto-download all attachments. It's nice having regex search capabilities on my local copy of the data too. [0] https://github.com/Tyrrrz/DiscordChatExporter
Now those of us who've been around the block know that Discord is merely the latest iteration on chat servers such as IRC. I'm interested to know, from anyone here who's an IRC operator or server/network admin, how the IRC community deals with scraping and bots, because in the early 90s, it was never an issue of corporate Terms of Service or legalese, but typically handled by community standards, and probably, people…
A big difference is that on Discord anybody who joins a server gets access to full history of chat logs whereas with IRC you don't get access to any past logs. So compared to IRC, Discord users should have an even lower expectation of privacy.