To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676
It's 118 gigabytes of JSON.
Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
21–30 of 204 posts
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#22insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed
I'm not sure this is awful. These are public discords, no more private than newsgroups, or StackOverflow. Why is it awful to scrape this, but not that?
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#23Fantastic. I wonder how many random technical info is buried in these servers. I hate what it's done for game modding.
The algebraic topology server probably contains a huge number of treasures in modern research algebraic topology. I really really hope it's archived in full
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#24> Usernames are replaced with consistent pseudonyms generated by the mimesis library, ensuring that identifiers remain unique and contextually meaningful across records. Similarly, user IDs and message IDs are hashed using the SHA-256 algorithm and truncated to 12 characters. This deterministic hashing approach maintains linkage between related records while effectively masking the original identifiers. The global na…
> public spaces on the internet But discord servers aren't considered "public spaces", hence the concept of an "invite". This is akin to someone revealing they've been going to private parties and secretly recording everything. It might not be illegal, but it's definitely not polite.
The researchers clarify that it's only those servers that are listed in the discovery tab - you don't need an invite link to join those.
> In this regard, this paper introduces the most extensive Discord dataset available to date, comprising 2,052,206,308 messages from 4,735,057 unique users across 3,167 servers – approximately 10% of the servers listed in Discord’s Discovery tab, a feature designed to highlight public servers that users can join.
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#25insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed
You'll probably catch a lot of flak for that on HN, but I 100% agree with you. Just because something is public and can be saved for later doesn't mean it should be done so en-masse.
These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for everyone to see?
Similarly, we have a huge problem right now with massive partisanship in the west. Changing one's opinion should be viewed in a positive light, but unfortunately our society doesn't seem to see it that way. Someone with an odious opinion when they were young, who changes that opinion to something more moderate when they grow up, should be viewed as a positive change. But increasingly what we're seeing is people going through data sets to find those old odious opinions of somebody when they were 14 years old and using that as proof that they must still be a terrible person at 24. It's a paranoid, completely self-defeating worldview, but unfortunately it's all too common right now, and I think it's honestly a huge reason for the massive political polarization we're seeing in the moment.
So yes, shame on these researchers. I know they claim to have anonymized the data set, but let's be honest, that never works. It's always easy to find common threads and identify someone, and it's particularly easy now when we have access to all sorts of machine learning models that can really do very effective denonymization.
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#26> Usernames are replaced with consistent pseudonyms generated by the mimesis library, ensuring that identifiers remain unique and contextually meaningful across records. Similarly, user IDs and message IDs are hashed using the SHA-256 algorithm and truncated to 12 characters. This deterministic hashing approach maintains linkage between related records while effectively masking the original identifiers. The global na…
So if I can identify a chat, that I have direct access to, in the dataset, I can get the hashed user ID of my contacts, and the search for any other messages from them?
If they didn't salt it, you might not even need to identify the chat - just hash the username you want to look up. I'd check, but it's a >100GB download.
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#27> Usernames are replaced with consistent pseudonyms generated by the mimesis library, ensuring that identifiers remain unique and contextually meaningful across records. Similarly, user IDs and message IDs are hashed using the SHA-256 algorithm and truncated to 12 characters. This deterministic hashing approach maintains linkage between related records while effectively masking the original identifiers. The global na…
> public spaces on the internet But discord servers aren't considered "public spaces", hence the concept of an "invite". This is akin to someone revealing they've been going to private parties and secretly recording everything. It might not be illegal, but it's definitely not polite.
These servers do seem to be public, in the sense that anyone can join without explicitly being invited.
It sounds more like they went to the mall, picked 10% of the stores, and recorded conversations taking place in those stores.
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#28insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed
I'm not sure this is awful. These are public discords, no more private than newsgroups, or StackOverflow. Why is it awful to scrape this , but not that ?
they are "open invite" in the sense that its nice to have serendipitous encounters with people on the internet youd otherwise never meet. its for chatting, and the pace of things ensures whatever you say will be buried.
the point isn't to make an artifact, like stack overflow. and certainly not be be experimented on.
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#29insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed
People should realize when what they write is public though. The world retreating into private group chats is not a bad thing.
Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)
#30Earlier quoted context omitted.
> public spaces on the internet But discord servers aren't considered "public spaces", hence the concept of an "invite". This is akin to someone revealing they've been going to private parties and secretly recording everything. It might not be illegal, but it's definitely not polite.
These servers do seem to be public, in the sense that anyone can join without explicitly being invited. It sounds more like they went to the mall, picked 10% of the stores, and recorded conversations taking place in those stores.
right, which would be awful.