Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

arxiv.org

Post reply on HN
Print view View original

191–200 of 204 posts

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#191

Post by gynvael »

philipkglass wrote: post #170
polarix wrote: post #169

Earlier quoted context omitted.

It is not just a login wall, they have restricted access even for logged in users, presumably to only the uploaders. A magnet would be nice.

About the decompressed data: zstdcat dataset.zst | sha384sum 0812f3876a7e319081f596a5545321e5c8e8def501add3a4f5ff039568fe59aa5d4ac5d2c3e549532f529bd09b887596 zstdcat dataset.zst | wc 2059116741 22128178392 2099550453760 If you can post your email and you have a sftp server or other accessible means to receive this large file, I'll contact you and then maybe you can help distribute it more widely. (Offer also applies…

Hi Philip. Would you be able to share with me just a few anonymized user IDs (random ones are fine)? I wanted to double-check their format, since I'm pretty sure their method is broken. My email address is there → https://gynvael.coldwind.pl/?id=50 (contact section)

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#192

Post by chneu »

Kiro wrote: post #142
chneu wrote: post #137

Earlier quoted context omitted.

Discord is horrible for this kind of stuff. There's a reason that GitHub and other type of sites exist Discord is walled and hard to search. If a channel or server closes then all that information is lost. Tons of data will be lost to discord when it goes down. Idk if you've ever tried to use discord for mods or other software but it sucks. It's confusing. Information isn't cataloged well. It's search sucks. It just…

What kind of stuff? Gamers want to talk about the game and engage with the developers. GitHub must be the absolute worst alternative.

I'm talking software and modifications to games. Stuff you have to download.

Discord becomes a maze of random posts going back years with links to expired third party sites. Searching them is difficult and time consuming.

Then it all changes based on which server you're on and how those people decided to do it, so none of your experience from the previous server transfers.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#193

Post by noman-land »

fwip wrote: post #26
Y_Y wrote: post #11

Earlier quoted context omitted.

So if I can identify a chat, that I have direct access to, in the dataset, I can get the hashed user ID of my contacts, and the search for any other messages from them?

If they didn't salt it, you might not even need to identify the chat - just hash the username you want to look up. I'd check, but it's a >100GB download.

Got a link to the full dataset?

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#194

Post by strogonoff »

squigz wrote: post #187
strogonoff wrote: post #186

Earlier quoted context omitted.

Which tracks the experience of an average person how? If you think everybody these days runs a million mods and admins a rented server, you should really go out more.

Do you think the average person is hosting the Minecraft server(s) they play on on their home computer?

Average person doesn’t host servers, full stop.

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#195

Post by gynvael »

gynvael wrote: post #191
philipkglass wrote: post #170

Earlier quoted context omitted.

About the decompressed data: zstdcat dataset.zst | sha384sum 0812f3876a7e319081f596a5545321e5c8e8def501add3a4f5ff039568fe59aa5d4ac5d2c3e549532f529bd09b887596 zstdcat dataset.zst | wc 2059116741 22128178392 2099550453760 If you can post your email and you have a sftp server or other accessible means to receive this large file, I'll contact you and then maybe you can help distribute it more widely. (Offer also applies…

Hi Philip. Would you be able to share with me just a few anonymized user IDs (random ones are fine)? I wanted to double-check their format, since I'm pretty sure their method is broken. My email address is there → https://gynvael.coldwind.pl/?id=50 (contact section)

Received, thanks!

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#197

Post by topcatto »

philipkglass wrote: post #170
polarix wrote: post #169

Earlier quoted context omitted.

It is not just a login wall, they have restricted access even for logged in users, presumably to only the uploaders. A magnet would be nice.

About the decompressed data: zstdcat dataset.zst | sha384sum 0812f3876a7e319081f596a5545321e5c8e8def501add3a4f5ff039568fe59aa5d4ac5d2c3e549532f529bd09b887596 zstdcat dataset.zst | wc 2059116741 22128178392 2099550453760 If you can post your email and you have a sftp server or other accessible means to receive this large file, I'll contact you and then maybe you can help distribute it more widely. (Offer also applies…

[dead]

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#198

Post by topcatto »

hampus wrote: post #158
diggan wrote: post #36

Earlier quoted context omitted.

> It's 118 gigabytes of JSON. 118.0 GB of ZST compressed JSON ( https://zenodo.org/records/15170676 ). The actual uncompressed JSON would most likely be much, much larger.

I downloaded it and decompressed it — it's approximately 2.1 terabytes in size.

[dead]

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#199

Post by philipkglass »

diggan wrote: post #183
hampus wrote: post #158

Earlier quoted context omitted.

I downloaded it and decompressed it — it's approximately 2.1 terabytes in size.

I managed to download the file, but since they restricted the downloads, it seems they hid the checksums too. Could you possibly share the md5/sha256 of the .zst file?

  sha256sum dataset.zst 
  0196416253fab4bce08504737bc81215927d9afdc6ccc81f75345518109266a4  dataset.zst

Re: Discord Unveiled: A Comprehensive Dataset of Public Communication (2015-2024)

#200

Post by datasetdotzst »

philipkglass wrote: post #170
polarix wrote: post #169

Earlier quoted context omitted.

It is not just a login wall, they have restricted access even for logged in users, presumably to only the uploaders. A magnet would be nice.

About the decompressed data: zstdcat dataset.zst | sha384sum 0812f3876a7e319081f596a5545321e5c8e8def501add3a4f5ff039568fe59aa5d4ac5d2c3e549532f529bd09b887596 zstdcat dataset.zst | wc 2059116741 22128178392 2099550453760 If you can post your email and you have a sftp server or other accessible means to receive this large file, I'll contact you and then maybe you can help distribute it more widely. (Offer also applies…

[dead]
Post reply on HN