Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

arxiv.org

Post reply on HN
Print view View original

31–40 of 204 posts

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#31

Post by mjr00 »

SirMaster wrote: post #5

The biggest problem that sucks about discord is that it isn't normally publicly searchable. And it seems to be a modern replacement for internet forums which historically were publicly searchable and often had a lot of great information about various hobbies and things.

> The biggest problem that sucks about discord is that it isn't normally publicly searchable.

This is a feature of the platform, not a bug. Because of the lack of discoverability people act more genuine, for better or for worse, than public places like Twitter, Bsky, Facebook, Instagram, etc where you have to maintain your public image and/or act like HR is watching over your shoulder.

That being said, this feature also makes Discord inappropriate for things like release announcements, patch notes, etc. which should be publicly accessible.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#32

Post by spencerflem »

d--b wrote: post #29
spencerflem wrote: post #15

insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed

People should realize when what they write is public though. The world retreating into private group chats is not a bad thing.

its a shame that we can't get the benefits of both. it sucks having to individually vet everyone one at a time. you miss a lot.

but people like these researchers who only want to exploit are making it necessary

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#33

Post by trevor-e »

Sanzig wrote: post #25
spencerflem wrote: post #15

insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed

You'll probably catch a lot of flak for that on HN, but I 100% agree with you. Just because something is public and can be saved for later doesn't mean it should be done so en-masse. These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for every…

I'm not sure I understand the outrage. Before Discord, everyone used forums. And those were archived by Google for the entire world to see. What's the difference?

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#34

Post by philipkglass »

fwip wrote: post #26
Y_Y wrote: post #11

Earlier quoted context omitted.

So if I can identify a chat, that I have direct access to, in the dataset, I can get the hashed user ID of my contacts, and the search for any other messages from them?

If they didn't salt it, you might not even need to identify the chat - just hash the username you want to look up. I'd check, but it's a >100GB download.

I'd check, but it's a >100GB download.

The page given by pavel_lishin above includes a sample data set that's only 6.2 GB:

https://zenodo.org/records/15170676/files/dataset_sample.zst...

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#35

Post by sillysaurusx »

Sanzig wrote: post #25
spencerflem wrote: post #15

insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed

You'll probably catch a lot of flak for that on HN, but I 100% agree with you. Just because something is public and can be saved for later doesn't mean it should be done so en-masse. These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for every…

Wrong or not, it’s the world we live in. It falls to the parents to teach their kids what not to say from a young age.

Besides, most of the salacious talk happens in DMs and private channels, which weren’t scraped.

If you can find evidence of some way to use this data harmfully, I’ll agree with you. Till then you’re making a fuss possibly without merit.

In general I’m skeptical of the power of data alone to meaningfully harm someone, except in obvious cases like private health info, exposing affairs, financial documents, and so on.

Public information is public, and it’s arguably wrong to keep it locked up rather than the other way around. Datasets like this are the only way open source ML has any chance against the big players.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#36

Post by diggan »

pavel_lishin wrote: post #21

To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 It's 118 gigabytes of JSON.

> It's 118 gigabytes of JSON.

118.0 GB of ZST compressed JSON (https://zenodo.org/records/15170676). The actual uncompressed JSON would most likely be much, much larger.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#37

Post by GaggiX »

mjr00 wrote: post #31
SirMaster wrote: post #5

The biggest problem that sucks about discord is that it isn't normally publicly searchable. And it seems to be a modern replacement for internet forums which historically were publicly searchable and often had a lot of great information about various hobbies and things.

> The biggest problem that sucks about discord is that it isn't normally publicly searchable. This is a feature of the platform, not a bug. Because of the lack of discoverability people act more genuine, for better or for worse, than public places like Twitter, Bsky, Facebook, Instagram, etc where you have to maintain your public image and/or act like HR is watching over your shoulder. That being said, this feature a…

>than public places like Twitter

That seems to be a counterpoint to your argument. Users on Twitter usually do not hold back.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#38

Post by stronglikedan »

Sanzig wrote: post #25
spencerflem wrote: post #15

insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed

You'll probably catch a lot of flak for that on HN, but I 100% agree with you. Just because something is public and can be saved for later doesn't mean it should be done so en-masse. These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for every…

There is no expectation that anything said on social media will remain private - zero - and anyone that thinks differently is just lying to themselves and everyone else.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#39

Post by d--b »

Sanzig wrote: post #25
spencerflem wrote: post #15

insane. people doing awful stuff like this is why the world is retreating into private group chats. these researchers should be ashamed

You'll probably catch a lot of flak for that on HN, but I 100% agree with you. Just because something is public and can be saved for later doesn't mean it should be done so en-masse. These are social spaces where a lot of young people essentially grew up. An important part of social development is making mistakes and learning from them. How can you make mistakes when those mistakes are archived for all time for every…

> Just because something is public and can be saved for later doesn't mean it should be done so en-masse.

Just because something is public means that someone is actually going to save it for later en-masse.

Educating 14-year-old kids so that they don't post public chat with their real name is more important than shaming the researchers.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#40

Post by ldoughty »

msp26 wrote: post #18

Fantastic. I wonder how many random technical info is buried in these servers. I hate what it's done for game modding.

I think the average server size here is in the ballpark of 1200 people.

These are servers that asked to be advertised by Discord ("Discovery"). These are unlikely to be any kind of servers used for private or even semi-private discussions. You likely don't know most of the people on the server.

Most likely, the 'hottest' kind of data you might find is someone accidentally leaking info akin to the World of Tanks forum post 'corrections'.

Post reply on HN