Live data from Hacker News

Faster LZ is not the answer to 150-250 GB video game downloads

richg42.blogspot.com

131–140 of 267 posts

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#131

Earlier quoted context omitted.

This is mostly true. The packages of assets for games are compressed but the individual assets are usually not. It’s mostly fancy zip of files with maybe a different header.

So they're compressed? What's wrong with compressing packages? That usually produces a better compression ratio by exploiting redundancy cross-file.

When optimizing load times, it’s usually console that’s the benchmark. PC is assumed to have faster or equal hardware so they optimize for the lowest denominator. Consoles aren’t PC’s but they are very close. They lack some instructions while gaining others. I can’t speak on current gen consoles but older ones like the Xbox 360, only could handle zip format and at a slow pace. You could show a loading screen while you unpack the data in memory or you can unpack on disk on installation saving some loading time, or it’s all uncompressed on consoles leaving the console to load the data as fast as console do.

The issue at hand around install size comes down to the engines, their ability to decompress asset bundles, the fidelity of the assets (8k vs 8k,4k,2k,1024) if no runtime optimization is done. WAV sounds vs OGG vs MP3 vs flac, there’s a lot of shit in those asset packs. You could compress it all like they do, into a folder structure you can index and load from, or you can write that game code to make boss fights cause camera shake, the decisions you make making a game are really all about making the game and little to do with how best to deliver your content.

Some texture formats like DXT are already compressed. Raw PNG’s could have compression. All these things are possible if the engine supports it.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#132
post #85
post #75

Earlier quoted context omitted.

Because PNG isn't a good format for GPU textures. Oodle texture+Kraken+something like BC7 is an optimal solution and is used in many games.

I think in today's machine (be it PC, Mac or even Smartphones) the reading-and-transform of a png asset for the GPU is hard coded in the processor : so VERY fast. It works well.

There is no hardware acceleration for PNG in any consumer device that I know of. And there is such thing as VRAM usage, if you decompress PNG to bitmap it would use 3x-4x more VRAM than BC7. If you decompress it from PNG and then compress to BC7 it would increase loading times and basically destroys all point of using PNG instead of lossy compression.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#133
post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

that IS his point. the problem is not the compression, the problem is the size of the assets. compression can not solve this. alleviate it a bit, yes, but not solve.

he's saying that game asset paradigms are headed in the wrong direction rapidly.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#134
post #71

Compression optimizes for wall clock time which is (roughly speaking) sum of time it takes to get the compressed data and time it takes to decompress it. Downloads over the network are almost always bottlenecked on the network speed, so the heavier compression you have the better. However, when loading data from the local disk, the I/O can be so fast that the decompression time becomes the bottleneck and slows down l…

Heavy compression for the first install is a common tactic. Steam and itch do it, and brotli transfer compression is widely used on the web.

Lossless transfer compression is good to have, but it's not enough. Lossy compression can give you over 10 times smaller files, but it's format-specific and needs to be carefully applied by the developer.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#135
post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

that IS his point. the problem is not the compression, the problem is the size of the assets. compression can not solve this. alleviate it a bit, yes, but not solve. he's saying that game asset paradigms are headed in the wrong direction rapidly.

The problem is that it wouldn't help. We have good enough solutions right now, but not many are using them. Better compression will be nice, but it will yield less results than adoption of something that is already here.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#136
post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

In terms of attracting engineering talent, how does the video games industry fare? In a crunch situation, I can think of how easy it is for some to be able to circumnavigate various pitfalls while others fall into them and have no time to address it. Very much a skill/experience thing. There’s countless examples of games that run horribly given their level of presentation, or have no business being so impressively pe…

game developers work so much harder than enterprise developers, and enterprise developers look down on game developers as is they are sub-human or something. it's completely inverted to reality.

I'm not saying that you are doing this, I am saying that this is done.

watch some GDC developer videos on YouTube. these people work HARD for performance.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#137

Earlier quoted context omitted.

There's no excuse for Jedi Survivor not to compress it's assets. Respawn is skipping last-gen consoles with this game. The modern consoles have significantly more CPU power at their disposal, and dedicated hardware for decompressing texture and sound data. There is no excuse for them to be shipping a 150GB game on the PS5 when so many cross-gen games are 40-60% smaller on PS5 than PS4

It's easy to say something like that, but as mentioned above, games are generally duct taped together at the literal last minute and it's easy to miss things that, to outsiders, seem obvious. The difference is that we aren't looking at literally every piece of the game and rushing to fix as many bugs as possible, like the devs, we're just looking at the file size.

Of course, the other possibility is that they were aware of it, but it was somewhere in a long list of things that they would have liked to fix, but did not have time for.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#138
post #126

Earlier quoted context omitted.

Saying "They take hundreds of gigs because they don't use ANY compression." is absurd when you immediately have to caveat that codecs like BC7 are used for texture data, which makes up a large portion of the shipped data. Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it. I don't know why y…

>immediately have to caveat Yeah, I could've indicated better that I'm talking about on-disk compression of files. >Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it. Not really if we look at 3rd party games. Jedi Survivor is likely not using that since it takes 147 gigs on PS5 and only 139…

> Not really if we look at 3rd party games.

I find this extremely unlikely. The packaging tools by default will compress files when building a package. Decompression is transparent and done by the hardware; the game doesn't have to implement anything or link any libraries. You would have to explicitly select "no compression" during packaging. Warzone definitely uses it - I worked on Cold War, Vanguard, and MW2 specifically on streaming and decompression across our supported consoles.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#139
post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

Saying "They take hundreds of gigs because they don't use ANY compression." is absurd when you immediately have to caveat that codecs like BC7 are used for texture data, which makes up a large portion of the shipped data. Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it. I don't know why y…

>(the most important thing, since raw assets often load faster from an SSD as long as you have available storage.)

>You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.

Is this really true? I feel like you'd need an absolutely blisteringly fast SSD to match something like an modern, optimized lz decompressor's speed.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#140

Earlier quoted context omitted.

Switching from PNG to basis for my game (currently with a pretty minimal set of textures) was like a 50% reduction in load times with no other changes. Note that I'm not storing assets at rest in a ZIP file because there's no reason to. If I was doing that it might close the gap some.

Yes I agree but the PNG compresion size is so great ! The Textures in total reduce enormously my game file size. By 15 times : so good. And because it is small the web download is VERY quick. Very.... So in a way I loose a bit on decompresing textures time, BUT a gain a lot od seconds in dowloading reduced textures files.

How many times do you decompress the textures, and how many times do you download them?
Post reply on HN