Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

arxiv.org

Post reply on HN
Print view View original

151–160 of 204 posts

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#151

Post by DaSHacka »

__loam wrote: post #147
DaSHacka wrote: post #121

Earlier quoted context omitted.

He said _ethical_, not _legal_. Would you agree abusive ToS's by massive corpos are unethical? What about the Disney+ ToS hiding a binding arbitration agreement preventing you from suing them? [0]. Or are you one of those "my personal ethics are whatever the law says" folk? [0] https://www.nbcnews.com/news/us-news/disney-says-man-cant-su...

The last guy who scraped discord like this was a freak who also got sued to hell by them.

The difference is they ran a private service to profit off the scraped data, and explicitly marketed it as a "dox-for-hire" service, so ethically I think the situations are quite different (the researchers explitly took steps to censor usernames in this dataset)

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#152

Post by DaSHacka »

judge2020 wrote: post #139
DaSHacka wrote: post #99

Earlier quoted context omitted.

Its not difficult to archive yourself, if you really care[0] I use a dedicated alt account to archive tons of various servers I'm in, and auto-download all attachments. It's nice having regex search capabilities on my local copy of the data too. [0] https://github.com/Tyrrrz/DiscordChatExporter

Using a user account to do this is still considered risky since any automated API usage by a non-bot user is against TOS, and they have heuristics (maybe now ML-based heuristics) for banning accounts for 'things that "don't look like what our official client does"'[0]. 0: https://news.ycombinator.com/item?id=25215415

This is why I use a dedicated account to scrape servers, since I unfortunately need my main to interact with(/run) communities unavailable elsewhere.

FWIW, I haven't exactly been careful with it (oftentimes scraping 2 servers at once, and downloading all attachments) and have never had an account get banned.

The only time I got 'banned' in any capacity was when I hammered the internal JSON API to get information about server's invite links, and even then it was only an automated IP ban from Cloudflare for a couple days. Although, it was an unauthenticated API.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#153

Post by mvieira38 »

AStonesThrow wrote: post #98

Now those of us who've been around the block know that Discord is merely the latest iteration on chat servers such as IRC. I'm interested to know, from anyone here who's an IRC operator or server/network admin, how the IRC community deals with scraping and bots, because in the early 90s, it was never an issue of corporate Terms of Service or legalese, but typically handled by community standards, and probably, people…

It's not. At least in the RPG scene, which I experience, it's almost fully replaced forums, and lots of great fanmade content and insightful discussion goes into that low discoverability cesspool which may go offline any day and scrapes all of your data

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#154

Post by sneak »

judge2020 wrote: post #136
sneak wrote: post #105

Now imagine the data mining that Discord can do on the complete DM history of every user. It’s not e2ee, remember.

E2EE is definitely only possible in DMs (there's no chance for servers/guilds), but the cat is out of the bag in terms of user expectations on how DMs work. So many users expect their entire decade+ history of DM contents, attachments included, to be available wherever they are and on any device, gated only by having their login/2fa or passkey. Switching to E2EE would be a major overhaul of that expectation, and it w…

e2ee works fine for Signal group chats; there is no reason it couldn’t be implemented on Discord group chats.

That’s not the issue. The issue is that Discord believes they deliver value through aggressively censoring their platform. e2ee prevents that.

e2ee also doesn’t prevent a user from storing their long term keys on the server to be retrieved on new devices and decrypted locally so they can access message history. e2ee does not require PFS.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#155

Post by 01HNNWZ0MV43FF »

prmph wrote: post #74

Why is it so hard to export your own messages out of Discord, Slack, etc? We have regressed from the open email standard and gone back to these opaque islands of data that do not adhere to any standard. Slack refused to show me my own messages past a certain age unless I paid up, and eventually deleted them.

It's hard because they want you to keep paying. Same reason AWS has free ingress and paid egress. The walled gardens are all built like carnivorous plants, the thorns face inwards.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#156

Post by lolinder »

ysavir wrote: post #149
lolinder wrote: post #126

Earlier quoted context omitted.

> it is actually the spiritual successor to IRC, AIM, and similar chat services, where the data is typically ephemeral I can't speak for AIM, but I've never assumed IRC was ephemeral. Most IRC servers had and have numerous people idling in the channel with their client just archiving everything that happened. Many of those archives ended up published on the internet and that usually surprised no one. Discord is even…

Sure, but there's a massive difference between an actor joining a chat and maintaining indefinitely, constant access to it vs an actor joining a chat and having access to the full history automatically. > All these researchers have done is expose the kind of archiving was always possible and always happening. What they've done is brought forward another instance of "this is why we can't have nice things". A useful fe…

You're not getting it: this is already happening everywhere. There is no private data on public Discord servers. Never has been.

All these researchers have done is admit to doing what many less well-meaning people and organizations are already doing on a regular basis. If that admission leads to more people realizing that public Discord servers are public, then the researchers have done everyone a service.

Blaming the researchers in this case is like blaming LockPickingLawyer for locks being pickable. The bad guys were already picking the locks before he started his channel, all he did was shine a light on just how bad most locks are.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#157

Post by 01HNNWZ0MV43FF »

pavel_lishin wrote: post #21

To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 It's 118 gigabytes of JSON.

Anyone who has it, please post the SHA384 or BLAKE3 (BLAKE3 will be way faster on 100+ GB) so I can verify it if I get a torrent later. Zenodo requires a sign-in, and I won't auth with GitHub as they want some WILD permissions on my GitHub.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#158

Post by hampus »

diggan wrote: post #36
pavel_lishin wrote: post #21

To save folks a click, the dataset itself has been made available here: https://zenodo.org/records/15170676 It's 118 gigabytes of JSON.

> It's 118 gigabytes of JSON. 118.0 GB of ZST compressed JSON ( https://zenodo.org/records/15170676 ). The actual uncompressed JSON would most likely be much, much larger.

I downloaded it and decompressed it — it's approximately 2.1 terabytes in size.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#159

Post by wswope »

chneu wrote: post #137
Kiro wrote: post #134

Earlier quoted context omitted.

Why? I follow a lot of solo game developers who only use Discord and I completely understand them not wanting do deal with multiple platforms. They should focus on on the game.

Discord is horrible for this kind of stuff. There's a reason that GitHub and other type of sites exist Discord is walled and hard to search. If a channel or server closes then all that information is lost. Tons of data will be lost to discord when it goes down. Idk if you've ever tried to use discord for mods or other software but it sucks. It's confusing. Information isn't cataloged well. It's search sucks. It just…

Github is walled and hard to search now too. Not as bad as discord yet, but headed there.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#160

Post by jowea »

Macha wrote: post #123
jowea wrote: post #57

Earlier quoted context omitted.

You think so? I believe that's properly a consequence of culture, Discord being originally a gaming platform, and of pseudoanonymity, the same thing you have on reddit. Anyone who cares even a tiny bit can join any of those public servers and see what you posted. The big difference is that you don't have as many lurkers who just got there by googling and are going to leave immediately. By the 1-9-90 rule of thumb, th…

I think the YouTube real names and the nymwars era in general showed that requiring people to use their legal name doesn't actually change community standards.

Good point, but I think there is a difference between turning what was once pseudoanonymous social media and where you mostly talk to people you don't know in real life into a real names social media, and having social media where you use your real name from the start and anything you post will likely be seen by people you know in real life.
Post reply on HN