Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Helldivers 2 on-disk size 85% reduction

store.steampowered.com

Post reply on HN
Print view View original

271–280 of 317 posts

Re: Helldivers 2 on-disk size 85% reduction

#271

Post by unusualmonkey »

kappaking wrote: post #195
mywittyname wrote: post #174

Earlier quoted context omitted.

Optimization takes up time, and often it takes up the time of an expert. Given that, people need to accept higher costs, longer development times, or reduced scope if they want better optimized games. But what is worse, is just trying to optimize software is not the same as successfully optimizing it. So time and money spent on optimization might yield no results because there might not be anymore efficiency to be ga…

> Given that, people need to accept higher costs, longer development times, or reduced scope if they want better optimized games. God why can’t it just be longer development time. I’m sick of the premature fetuses of games.

Just wait until after launch. You get a refined experience and often much lower prices.

Re: Helldivers 2 on-disk size 85% reduction

#272

Post by simplyinfinity »

debugnik wrote: post #48
NullPrefix wrote: post #44

Earlier quoted context omitted.

such games can only get private servers

Yes, but those are rarely a thing for most live service games. Unless someone is working on a reimplementation of the entire server side, there's no point in offering or downloading pirate copies.

There is - albeit a dwindling - community that does reimplement entire backends for mmo games. Look up the ragezone forums. I grew up around Mu Online private servers. And I'm sure in time a private server for HD 2 will appear if arrowhead don't release one themselves :)

Re: Helldivers 2 on-disk size 85% reduction

#273

Post by lynnharry »

maccard wrote: post #34
habbekrats wrote: post #2

it seems wild the state of games and development today... imagine 131GB out of 154GB of data was not needed....

This isn't unique to games, and it's not just "today". Go back a decade [0] find people making similar observations about one of the largest tech companies on the planet. [0] https://news.ycombinator.com/item?id=10066338

> FB App is 114MB in size, but loading this page in Chrome will use a good 450MB, idk how they managed that.

This reminds me of the old days when I check who's using my PC memory every now and then.

Re: Helldivers 2 on-disk size 85% reduction

#274

Post by nearbuy »

justsomehnguy wrote: post #269
nearbuy wrote: post #265

Earlier quoted context omitted.

> there's no delay inherent to playing back a recorded sound. There absolutely is. You can decompress compressed audio files when loading so they play immediately, but if you want to keep your mp3 compressed, you get a delay. Games keep the sound effects in memory uncompressed. > Not all sound effects are short Long ambient background noises often aren't latency sensitive and can be streamed. For most games textures…

> but if you want to keep your mp3 compressed, you get a delay If that really bothers you then write your own on-disk compression format. > why we use uncompressed audio > ADPCM ... which is a compressed and lossy format.

> If that really bothers you then write your own on-disk compression format.

Why? What are you trying to solve here? You're going to have a hard time making a new format that serves you better than any of the existing formats.

The most common solution for instant playback is just to store the sound uncompressed in memory. It's not a problem that needs solving for most games.

ADPCM and PCM are both pretty common. ADPCM for audio is kinda like DXT compression for textures: a very simple compression that produces files many times larger than mp3, and doesn't have good sound quality, but has the advantage that playback and seek costs virtually nothing over regular PCM. The file sizes of ADPCM are closer to PCM than mp3. I should have been clearer in my first comment that the delay is only for mp3/Vorbis and not for PCM/ADPCM.

There isn't a clean distinction between compressed and uncompressed and lossy/lossless in an absolute sense. Compression is implicitly (or explicitly) against some arbitrary choice of baseline. We normally call 16-bit PCM uncompressed and lossless but if your baseline is 32-bit floats, then it's lossy and compressed from that baseline.

Re: Helldivers 2 on-disk size 85% reduction

#275

Post by Dylan16807 »

wtallis wrote: post #270
Dylan16807 wrote: post #211

Earlier quoted context omitted.

If what they were familiar with was a good SSD, then they didn't need to do anything. I don't see how anything Sony said about their SSD would have affected things. Maybe you're saying the hearsay was Sony exaggerating how bad hard drives are? But they didn't really do that, and the devs would already have experience with hard drives.

What Sony said about their SSD was that it enabled game developers to not duplicate assets like they did for rotating storage. One specific example I recall in Sony's presentation was the assets for a mailbox used in a Spider Man game, with hundreds of copies of that mailbox duplicated on disk because the game divided Manhattan into chunks and tried to have all the assets for each chunk stored more or less contiguous…

Weird, since that's a benefit of any kind of SSD at all. The stuff their fancy implementation made possible was per-frame loading, not just convenient asset streaming.

But uh if the devs didn't realize that, I blame them. It's their job to know basics like that.

Re: Helldivers 2 on-disk size 85% reduction

#276

Post by zamadatix »

mort96 wrote: post #234
zamadatix wrote: post #188

Earlier quoted context omitted.

I think what makes this a bit different from the usual "time/value tradeoff" discussion is bloating the size by 6x-7x was the result of unnecessary work in the name of optimization instead of lack of cycles to spend on optimization.

Eh probably not, it's probably handled by some automated system when making release builds of the game. Sure, implementing that initially was probably some work (or maybe it was just checking a checkbox in some tool), but there's probably not much manual work involved anymore to keep it going. Reverting it now though, when the game is out there on a million systems, requires significant investigation to ensure they'r…

Reverting it now was certainly a pile of work, but that's neither here nor there for the portion of the story bothering people. It's like they threw rocks threw the windows years ago to make them slightly clearer to see through and now put a ton of work in to undo that because they discovered that made no sense in reality.

It's great they did all the work to fix it after the fact, but that doesn't justify why it was worth throwing rocks through the window in the first place (which is different than not doing optimizations).

Re: Helldivers 2 on-disk size 85% reduction

#277

Post by nomel »

myself248 wrote: post #257
braiamp wrote: post #217

Earlier quoted context omitted.

It didn't. They downloaded 43 GB instead of 152 GB, according to SteamDB: https://steamdb.info/app/553850/depots/ Now it is 20 GB => 21 GB. Steam is pretty good at deduplicating data in transit from their servers. They are not idiots that will let developers/publishers eat their downstream connection with duplicated data. https://partner.steamgames.com/doc/sdk/uploading#AppStructur...

Furthermore, this raises the possibility of a "de-debloater" that HDD users could run, which would duplicate the data into its loading-optimized form, if they decided they wanted to spend the space on it. (And a "de-de-debloater" to recover the space when they're not actively playing the game...) The whole industry could benefit from this.

> to recover the space when they're not actively playing the game

This would defeat the purpose. The goal of the duplication is to place the related data physically close, on the disk. Hard links, removing then replacing, etc, wouldn't preserve the physical spacing of the data, meaning the terrible slow read head has to physically sweep around more.

I think the sane approach would be to have a HDD/SDD switch for the file lookups, with all the references pointing to the same file, for SDD.

Re: Helldivers 2 on-disk size 85% reduction

#278

Post by justsomehnguy »

nearbuy wrote: post #274
justsomehnguy wrote: post #269

Earlier quoted context omitted.

> but if you want to keep your mp3 compressed, you get a delay If that really bothers you then write your own on-disk compression format. > why we use uncompressed audio > ADPCM ... which is a compressed and lossy format.

> If that really bothers you then write your own on-disk compression format. Why? What are you trying to solve here? You're going to have a hard time making a new format that serves you better than any of the existing formats. The most common solution for instant playback is just to store the sound uncompressed in memory. It's not a problem that needs solving for most games. ADPCM and PCM are both pretty common. ADPC…

> Why? What are you trying to solve here? You're going to have a hard time making a new format that serves you better than any of the existing formats.

Storage space. But this is the way for the same guys who duplicate 20Gb seven times 'to serve better by the industry standard'.

More sane people would just pack that AD/PCM in a .pk3^W sorry in a .zip file (or any other packaging format with LZ/7z/whatever compatible compression method) with the fastest profile and would have the best of the both worlds: sane storage requirements, uncompressed in memory. As a bonus it would be loaded faster from HDD because a data chunk which is 10 times smaller than uncompressed one would be loaded surprise 10 times faster.

Re: Helldivers 2 on-disk size 85% reduction

#279

Post by brokenmachine »

teamonkey wrote: post #228
mort96 wrote: post #194

Earlier quoted context omitted.

You make a million decisions in the beginning of every project. I'm certain they made the choice to do this "optimization" at an early point (or even incidentally copied the choice over from an earlier project) at a stage where the disk footprint was small (a game being 7GB when it could've been 1GB doesn't exactly set off alarm bells). Then they just didn't reconsider the choice until, well, now.

Even at the end of development it’s a sensible choice. It’s the default strategy for catering to machines with slow disk access. The risk of some players experiencing slow load times is catastrophic at launch. In absence of solid user data, it’s a fine assumption to make.

Call me a dinosaur, but I don't consider a 154Gb download before I can start playing a good first impression.

In fact, I would seriously consider even buying a game that big if I knew beforehand. When a 500Gb SSD is $120 Aussie bucks, that's $37 of storage.

Re: Helldivers 2 on-disk size 85% reduction

#280

Post by hbn »

hbn wrote: post #69
wtallis wrote: post #68

Earlier quoted context omitted.

> the Switch cartridges use NAND memory which means if you aren't popping them into a system to refresh the charge every once in a while, your physical cartridge might not last as long You've been paying attention to the wrong sources for information about NAND flash. A new Switch cartridge will have many years of reliable data retention, even just sitting on a shelf. Data retention only starts to become a concern fo…

What's "many years"? I've read about people's 3DS cartridges already failing just sitting on a shelf.

Speak of the devil, I just got this tweet in my feed today

https://x.com/marcdwyz/status/1999226723322261520

Post reply on HN