Live data from Hacker News

Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

arxiv.org

101–110 of 204 posts

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#101

Earlier quoted context omitted.

Would the TOS even prevent something like joining a guild, downloading all messages, then leaving?

User bots (including hacked clients) are officially banned by the TOS, which addresses that concern. The only acceptable API usage is via bots that server owners choose to invite. And while it might be legally OK (if the bot's own TOS says it), I promise no server owner is expecting an invited bot to slurp up every message for use in a data set, whether that be for academic purposes or a potential stalking/"dirt" dat…

IIRC data slurping (for exporting) is also not allowed bot usage.

> B. API Data Sharing & Retention

> You will not share API Data with any third party, except in the following circumstances, subject to compliance with the Terms and applicable laws and regulations: (i) with a Service Provider; (ii) to the extent required under applicable laws or regulations; and (iii) when a user of your Application expressly directs you to share their API Data with the third party (and you will provide us proof thereof upon request).

https://support-dev.discord.com/hc/en-us/articles/8562894815...

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#102

More info here: https://www.404media.co/researchers-scrape-2-billion-discord...

As usual, 404 nails it:

----

It should be noted, however, that almost no one reads end-user license agreements and many of Discord’s users are children and teenagers. Discord is, first and foremost, a platform for gamers to organize communities and it’s not plausible that a 15 year old looking for a Fortnite meme server ever thought their dumb jokes about Tomato Town would end up in a public database five years later.

----

Same as other commenters here: I think this is shameful action under the guise of research and I cannot fathom why any IRB board would approve this (and perhaps it did not in this case, I do not know if Brazil has such a thing).

Back in the day (15ish years ago), I wrote a paper where I scraped the World of Warcraft API. It wasn't hard to do, I started on a realm, looked for arena teams, then went to guilds and got character sheets from there. I took the opinion that if Blizzard doesn't throttle me it's fair game.

Looking back now, I think that to have been pretty naive. I wouldn't say reckless, but definitely naive. In my mind, I had not made a delineation between "I can access this thing manually one at a time" and "I can access all of it automatically". As far as I was concerned, it was just the computer pressing the buttons. It was the same thing.

I think in the fullness of time we have collectively come to realize it is 100% not the same thing. The _availability_ of a thing and the _collection_ of a thing are two different issues with their own thorny problems. The researchers here have made the same mistake I did, but instead of it just being what gear your character was wearing, they took actual communications instead.

I hope this paper gets retracted, all data deleted and a sincere apology offered.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#103

To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 It's 118 gigabytes of JSON.

Is there a faster way to download than the link there? It's steady, but a roughly 11 hour download.

hmm they should have uploaded a torrent link to download the data, I wonder why they didn't do that?

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#104
post #19

The biggest problem that sucks about discord is that it isn't normally publicly searchable. And it seems to be a modern replacement for internet forums which historically were publicly searchable and often had a lot of great information about various hobbies and things.

There is a special place in hell for software (including game) developers who exclusively use Discord to release patch notes, documentation, technical support, etc.

they dont work for you?

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#106

More info here: https://www.404media.co/researchers-scrape-2-billion-discord...

As usual, 404 nails it: ---- It should be noted, however, that almost no one reads end-user license agreements and many of Discord’s users are children and teenagers. Discord is, first and foremost, a platform for gamers to organize communities and it’s not plausible that a 15 year old looking for a Fortnite meme server ever thought their dumb jokes about Tomato Town would end up in a public database five years later…

On the contrary, I think that what these researchers did was the only ethical thing to do once they discovered that this was possible.

There's no way that this hasn't been done dozens of times before by intelligence agencies, hacker groups, and whoever else you care to worry about. Most of us here were well aware that public Discord channels have always been public and durable. It's hardly a secret from the technically savvy, it's just that Discord doesn't make it clear enough to regular users.

All this paper changes is that it draws mainstream attention to what was already happening illicitly for as long as Discord has been around. This can only be a good thing: the children and teenagers 404 is so worried about have always been vulnerable to their data getting leaked just like this, it's just that up until now that's been happening in the dark so as not to kill the golden goose.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#107
post #31

The biggest problem that sucks about discord is that it isn't normally publicly searchable. And it seems to be a modern replacement for internet forums which historically were publicly searchable and often had a lot of great information about various hobbies and things.

> The biggest problem that sucks about discord is that it isn't normally publicly searchable. This is a feature of the platform, not a bug. Because of the lack of discoverability people act more genuine, for better or for worse, than public places like Twitter, Bsky, Facebook, Instagram, etc where you have to maintain your public image and/or act like HR is watching over your shoulder. That being said, this feature a…

While being public plays a role in the kind of conversations that happen on those platforms, I think engagement hacking feeds play a larger role. Discord has none of that. It's sorted by time.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#108

I don't know if Discord fixed it as I haven't checked in a few years, but I tinkered with scraping some public Discords and I found that I could see hidden channels, not the data, but the channel names, which could do things like reveal to me if the same Discord was used for in-house development if it was a product Discord. Not great.

You can still see them. Using alternate clients you will see them, and bots also see them.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#109

I don't know if Discord fixed it as I haven't checked in a few years, but I tinkered with scraping some public Discords and I found that I could see hidden channels, not the data, but the channel names, which could do things like reveal to me if the same Discord was used for in-house development if it was a product Discord. Not great.

This is still the case. There are even some client mods that let you view hidden channel names and know what roles/permissions are required to participate in them.
Post reply on HN