Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

arxiv.org

Post reply on HN
Print view View original

51–60 of 204 posts

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#51

Post by jazzyjackson »

stronglikedan wrote: post #38
Sanzig wrote: post #25

Earlier quoted context omitted.

You'll probably catch a lot of flak for that on HN, but I 100% agree with you. Just because something is public and can be saved for later doesn't mean it should be done so en-masse. These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for every…

There is no expectation that anything said on social media will remain private - zero - and anyone that thinks differently is just lying to themselves and everyone else.

Discord "servers" are groups you click on an "invite" to join, there is some framing where you are lead to believe the conversation is not public.

Bluesky had a similar freakout when HuggingFace packaged up a snapshot of the firehose and this discussion was had ad infinitum - in Bluesky's face every post is explicitly public -- but a very vocal minority of users still felt they had been wronged because no one asked if they were OK with being a part of a research project. There's definitely a gap between what users /should/ be aware of and what impression they actually get using a service, and you have to keep in mind most people are not techies that have spent years of their life wrestling with issues of privacy and data ownership, so announcements like this can come as a surprise, the first time they considered that what they said in one context can be moved to another.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#52

Post by jowea »

kd5bjo wrote: post #13

A quick read through of their anonymization process seems to indicate that they didn’t scan the message contents for PII (other than usernames). If true, that seems like a huge oversight. I also wonder what would happen if someone finds their information in the dataset and requests it to be removed per GDPR or other privacy legislation.

I understand wanting to be careful, but didn't they only grab messages from servers that are already very public? Are Twitter message datasets anonymized?

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#53

Post by squigz »

diggan wrote: post #36
pavel_lishin wrote: post #21

To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 It's 118 gigabytes of JSON.

> It's 118 gigabytes of JSON. 118.0 GB of ZST compressed JSON ( https://zenodo.org/records/15170676 ). The actual uncompressed JSON would most likely be much, much larger.

I imagine it can be reduced a fair bit simply by stripping out unneeded fields from the messages; I imagine it can be reduced even further by removing unneeded messages entirely (i.e., bot welcome messages), deduping messages, etc

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#54

Post by AStonesThrow »

jazzyjackson wrote: post #51
stronglikedan wrote: post #38

Earlier quoted context omitted.

There is no expectation that anything said on social media will remain private - zero - and anyone that thinks differently is just lying to themselves and everyone else.

Discord "servers" are groups you click on an "invite" to join, there is some framing where you are lead to believe the conversation is not public. Bluesky had a similar freakout when HuggingFace packaged up a snapshot of the firehose and this discussion was had ad infinitum - in Bluesky's face every post is explicitly public -- but a very vocal minority of users still felt they had been wronged because no one asked i…

“Three men may keep a secret, if two of them are dead.”

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#57

Post by jowea »

mjr00 wrote: post #31
SirMaster wrote: post #5

The biggest problem that sucks about discord is that it isn't normally publicly searchable. And it seems to be a modern replacement for internet forums which historically were publicly searchable and often had a lot of great information about various hobbies and things.

> The biggest problem that sucks about discord is that it isn't normally publicly searchable. This is a feature of the platform, not a bug. Because of the lack of discoverability people act more genuine, for better or for worse, than public places like Twitter, Bsky, Facebook, Instagram, etc where you have to maintain your public image and/or act like HR is watching over your shoulder. That being said, this feature a…

You think so? I believe that's properly a consequence of culture, Discord being originally a gaming platform, and of pseudoanonymity, the same thing you have on reddit. Anyone who cares even a tiny bit can join any of those public servers and see what you posted. The big difference is that you don't have as many lurkers who just got there by googling and are going to leave immediately. By the 1-9-90 rule of thumb, that's a lot of people.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#58

Post by squigz »

Sanzig wrote: post #25
spencerflem wrote: post #15

insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed

You'll probably catch a lot of flak for that on HN, but I 100% agree with you. Just because something is public and can be saved for later doesn't mean it should be done so en-masse. These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for every…

> But increasingly what we're seeing is people going through data sets to find those old odious opinions of somebody when they were 14 years old and using that as proof that they must still be a terrible person at 24. It's a paranoid, completely self-defeating worldview, but unfortunately it's all too common right now, and I think it's honestly a huge reason for the massive political polarization we're seeing in the moment.

I have to seriously question the actual prevalence of this. If someone posts and says "Look at what this person posted 12 years ago!" I'm not going to take them very seriously. I don't know anyone who would. This sounds like the usual "cancel culture" stuff that mostly boils down to "people face consequences for their present shitty behavior"

Anyway, as an avid Discord user, manager of a Discord community, and privacy advocate, I sympathize with this position, but as a software user and developer, and advocate for accessible FOSS (and accompanying information), I unfortunately have to side with the release of this and similar datasets. I would much rather this than the inevitable loss of so much valuable information.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#59

Post by paulirish »

pavel_lishin wrote: post #21

To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 It's 118 gigabytes of JSON.

Here's the top 500 servers in the dataset: https://pastebin.com/pANc3WkT

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#60

Post by strogonoff »

msp26 wrote: post #18

Fantastic. I wonder how many random technical info is buried in these servers. I hate what it's done for game modding.

Game modding is profitable and people doing it professionally (which they increasingly do) are quite attuned to the fact that making it too accessible would decimate their revenue. As a result, you either pay for the mod (early access, extra content, etc.), or you pay to join some Discord, but ideally you pay for something. Discord, which I generally dislike, is not necessarily the cause of it; if there was no Discord, people would probably use some other closed community platform instead.

I expect this would become more widespread as more traditional jobs are subsumed by unregulated ML tech (which, incidentally, the encumbent job-holders are helping train) and more people turn to what used to be generally a hobby as their means of making a living (not that that would last for too long either).

Post reply on HN