Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster LZ is not the answer to 150-250 GB video game downloads

richg42.blogspot.com

Post reply on HN
Print view View original

71–80 of 267 posts

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#71

Post by pornel »

Compression optimizes for wall clock time which is (roughly speaking) sum of time it takes to get the compressed data and time it takes to decompress it. Downloads over the network are almost always bottlenecked on the network speed, so the heavier compression you have the better. However, when loading data from the local disk, the I/O can be so fast that the decompression time becomes the bottleneck and slows down loading.

Because of that optimization for download time and startup time end up optimizing for different things.

With loading speeds so different in these two cases, I wonder if the best solution would be to recompress the data with a different compression level during download and installation. For example, download textures compressed with a good but slow and non-random-access image compression format, and then convert them to a lighter GPU-friendly compression format locally.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#72

Post by terafo »

forrestthewoods wrote: post #57
marcellus23 wrote: post #29

Earlier quoted context omitted.

Maybe not for that particular game, but a lot of games have been shown to use uncompressed assets. It's not just speculation.

> It's not just speculation. So it’s exactly speculation. A lot of games have done a lot of things for a lot of reasons over the decades. Why are new games so large? Good question. This entire thread is speculative.

Jedi Survivor having uncompressed textures is a speculation, but I'm extremely confident that it is this way. The fact that games nowadays ship without LOD systems, without texture compression, without adequate streaming is not just speculation. It is a fact.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#73

Post by ehaliewicz2 »

babypuncher wrote: post #28
terafo wrote: post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

There's no excuse for Jedi Survivor not to compress it's assets. Respawn is skipping last-gen consoles with this game. The modern consoles have significantly more CPU power at their disposal, and dedicated hardware for decompressing texture and sound data. There is no excuse for them to be shipping a 150GB game on the PS5 when so many cross-gen games are 40-60% smaller on PS5 than PS4

It's easy to say something like that, but as mentioned above, games are generally duct taped together at the literal last minute and it's easy to miss things that, to outsiders, seem obvious. The difference is that we aren't looking at literally every piece of the game and rushing to fix as many bugs as possible, like the devs, we're just looking at the file size.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#74

Post by jjoonathan »

Waterluvian wrote: post #67
terafo wrote: post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

In terms of attracting engineering talent, how does the video games industry fare? In a crunch situation, I can think of how easy it is for some to be able to circumnavigate various pitfalls while others fall into them and have no time to address it. Very much a skill/experience thing. There’s countless examples of games that run horribly given their level of presentation, or have no business being so impressively pe…

It attracts truly extraordinary talent. One of many examples: https://www.youtube.com/watch?v=eviSykqSUUw . There is more technical ambition in a single slide of that project than an entire department at places I've worked.

However, it also tends to demand more and pay less, so there is churn and burnout and when it comes time to triage, the triage can cut very deep.

If I were to dive into the JIRA for any one of these "missing compression" incidents, I wouldn't expect to find an easy win, I would expect to find "missing compression" beneath a pile of much worse bugs like "level 2 crashes on AMD cards." So it goes.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#75

Post by terafo »

tmilard wrote: post #60

Chocked at FAT size video games file ? Me too. Giga bytes Damned ! To reduce the size, I decided : 1) To use the standard PNG compression. 2) To limit mesh size at less than 300 points for a scene. - Result : a) File size Check it here : https://free-visit.net/fr/demo01 (pleaseload url twice : I still have a bug) Question : - why video games STUDIOS do not use this strategy ?

Because PNG isn't a good format for GPU textures. Oodle texture+Kraken+something like BC7 is an optimal solution and is used in many games.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#76

Post by jsheard »

codedokode wrote: post #38

Why not simply make several versions? Like 1 Gb, 10 Gb and 100 Gb versions?

Some games do this by splitting off the highest resolution textures into a free "DLC" that the user can opt not to install if they want to save space. It's sometimes done for audio as well, e.g. optional Japanese voices for the English release of a JRPG.

Remedy's Quantum Break had an unusual spin on letting users choose the install size, it made heavy use of live action FMVs which could either be downloaded ahead of time or streamed on the fly as they played. Downloading the FMVs more than doubled the games footprint.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#77

Post by gabereiser »

terafo wrote: post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

This is mostly true. The packages of assets for games are compressed but the individual assets are usually not. It’s mostly fancy zip of files with maybe a different header.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#78

Post by circuit10 »

terafo wrote: post #68
circuit10 wrote: post #48

Earlier quoted context omitted.

That would be far, far slower than just downloading the image

8k texture with mipmaps takes around 100 megs uncompressed. Downloading it would be 8-10 seconds on 100 mbit connection. It is definitely possible to generate that image in 8 to 10 seconds on consumer GPU if you are using some kind of consistency models that are trained to generate specific game style and inferences in 8 bit.

I doubt you can do it in 8K at that speed and also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#79

Post by thequux »

RajT88 wrote: post #47
hellotomyrars wrote: post #8

Game size is a perennial talking point and some games earn it and some games don’t. There was a phase of games shipping with uncompressed audio in every language they support for no good reason that ballooned file size (Titanfall) Some games have gotten around this problem by having higher resolution textures or additional languages being optional installs within whatever ecosystems DLC implementation is. I think thi…

The .kkreiger devs really showed how far you can go to reduce download size: https://en.wikipedia.org/wiki/.kkrieger https://www.youtube.com/watch?v=ya_MUKc343U It is dated, but looks pretty good by 2004 standards.

While I'm all for reducing download sizes, perhaps lekktor[1] is a bridge too far.

[1]: https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for...

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#80

Post by uoaei »

taeric wrote: post #22
terafo wrote: post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

This feels wrong. In that, I'd be surprised if they are not using compression. I think folks are just drastically underestimating how many resources are under discussion. And how big they are. Heck, just the geometry is probably far larger than folks anticipate.

The geometry would be hard to compress, but I suspect there's some clever tricks with some kind of fractal compression[1] that could drastically reduce texture sizes for camo, foliage, etc. without any effect on aesthetics.

Encoding is the hard part, but only need be done once when building the final release.

[1] https://en.wikipedia.org/wiki/Fractal_compression

Post reply on HN