Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster LZ is not the answer to 150-250 GB video game downloads

richg42.blogspot.com

Post reply on HN
Print view View original

81–90 of 267 posts

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#81

Post by syntheweave »

Waterluvian wrote: post #67
terafo wrote: post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

In terms of attracting engineering talent, how does the video games industry fare? In a crunch situation, I can think of how easy it is for some to be able to circumnavigate various pitfalls while others fall into them and have no time to address it. Very much a skill/experience thing. There’s countless examples of games that run horribly given their level of presentation, or have no business being so impressively pe…

Game programmers can do some amazing things, but they generally aren't given the schedules to do them "properly".

If most commercial software is a feature factory, games are "megafactories" because they are not really competing on the robustness of the implementation, just on delivering a certain kind of experience that delivers some interactivity and shows off some assets, and that tends to lead towards approaches that hack one feature on top of another and then lean on manual testing to catch "enough" errors. Many games also take characteristic shortcuts that eliminate some options for game design in order to simplify the tech.

The larger companies will often keep their tech teams separated from their game teams and treat the tech coders more generously. This comes with all the pluses and minuses of siloing the work - sometimes they solve the wrong problem, but do it well.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#82

Post by kevingadd »

terafo wrote: post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

Saying "They take hundreds of gigs because they don't use ANY compression." is absurd when you immediately have to caveat that codecs like BC7 are used for texture data, which makes up a large portion of the shipped data.

Sony included dedicated compression support in the PS5, licensed from a leading middleware company, specifically to make it easier to use compression in games. Games are using it.

I don't know why you're trotting out this misinformed BS.

You can also simply watch the statistics when Steam is installing a game - the amount written to disk is larger than the amount downloaded over the network, because they also compress assets in flight to reduce bandwidth usage (the most important thing, since raw assets often load faster from an SSD as long as you have available storage.)

You can argue every single file should be compressed at rest on persistent storage after install, but that's a losing argument. People really hate longer load times, so compression at rest is only appropriate if it improves them.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#83

Post by klodolph »

terafo wrote: post #42
ricardobayes wrote: post #36

Earlier quoted context omitted.

I remember Fallout New Vegas was almost 50 GB because it was a parallel install of all language/localization files. Once you manually deleted the languages you didn't want it went back to like 10 GB.

And there was a hilarious situation with audio in Titanfall, which included 35 gigs of uncompressed audio(whole game took 48 gigs).

As I recall, this was a purposeful choice. Titanfall came out in 2014. The minimum requirements were a Core 2 Duo at 2.4 GHz or Athlon X2 at 2.8 GHz. I think that puts it on processors dating back to 2006 or so.

Games sometimes have a lot of sound effects. It's easier to stream 35 GB of data from disk than it is to decompress, like, 10 streams of audio at the same time with whatever CPU you can dedicate to audio.

I do think they could have figured something out, like compressing the right subset of audio assets, but I just want to give the context for why they made that choice.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#85

Post by tmilard »

terafo wrote: post #75
tmilard wrote: post #60

Chocked at FAT size video games file ? Me too. Giga bytes Damned ! To reduce the size, I decided : 1) To use the standard PNG compression. 2) To limit mesh size at less than 300 points for a scene. - Result : a) File size Check it here : https://free-visit.net/fr/demo01 (pleaseload url twice : I still have a bug) Question : - why video games STUDIOS do not use this strategy ?

Because PNG isn't a good format for GPU textures. Oodle texture+Kraken+something like BC7 is an optimal solution and is used in many games.

I think in today's machine (be it PC, Mac or even Smartphones) the reading-and-transform of a png asset for the GPU is hard coded in the processor : so VERY fast. It works well.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#86

Post by kevingadd »

overthrow wrote: post #20
terafo wrote: post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

If you compress data, it will be smaller and require less i/o to load, so it will load faster. That's ESPECIALLY true on HDDs, but even on SSDs there are decompression algos that run faster than SSD bandwidth. The uncompressed asset cargo cult is hurting everyone who plays their games, no matter what kind of disk they use.

Compression does not guarantee faster loads anymore. Uncompressed assets can be DMA'd directly to their destination. This is part of why GPU decompression (gdeflate like in directstorage, or the special decompression tech in the PS5) is on the rise because it allows you to DMA resources to their destination on the GPU and then decompress them there to avoid the loading time hit involved in decompressing them on-CPU after copying them disk->page cache->userspace app memory.

For HDDs, yes, compression is generally a win. HDDs are dead as far as game development is concerned, all modern targets have flash storage.

Some compression codecs that produce smaller files are also MUCH slower to decompress, which swamps any benefits from the reduced i/o. This is part of why sometimes people pick less efficient formats for use in games.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#87

Post by klodolph »

Waterluvian wrote: post #67
terafo wrote: post #5

He completely misses the point. Games don't take hundreds of gigs because they have sub-optimal compression. They take hundreds of gigs because they don't use ANY compression. Jedi Survivor takes 139 gigs on Series X and mere 44 gigs on Series S. It didn't come out yet, but I'm 99.99% certain that this difference is just from inclusion of 8k textures into Series X version. That kind of scaling means that they don't u…

In terms of attracting engineering talent, how does the video games industry fare? In a crunch situation, I can think of how easy it is for some to be able to circumnavigate various pitfalls while others fall into them and have no time to address it. Very much a skill/experience thing. There’s countless examples of games that run horribly given their level of presentation, or have no business being so impressively pe…

> In terms of attracting engineering talent, how does the video games industry fare?

When I talk to people in the game industry, it sounds a lot like my experiences outside it. Plenty of programming skill to go around, but the skill is often allocated inefficiently, and everyone's under pressure to just ship stuff and move on to the next deliverable.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#88

Post by dalmo3 »

terafo wrote: post #68
circuit10 wrote: post #48

Earlier quoted context omitted.

That would be far, far slower than just downloading the image

8k texture with mipmaps takes around 100 megs uncompressed. Downloading it would be 8-10 seconds on 100 mbit connection. It is definitely possible to generate that image in 8 to 10 seconds on consumer GPU if you are using some kind of consistency models that are trained to generate specific game style and inferences in 8 bit.

> on consumer GPU

You make it sound like my electricity bill would be lower by just downloading the bigger files...

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#89

Post by egypturnash »

tmilard wrote: post #60

Chocked at FAT size video games file ? Me too. Giga bytes Damned ! To reduce the size, I decided : 1) To use the standard PNG compression. 2) To limit mesh size at less than 300 points for a scene. - Result : a) File size Check it here : https://free-visit.net/fr/demo01 (pleaseload url twice : I still have a bug) Question : - why video games STUDIOS do not use this strategy ?

To limit mesh size at less than 300 points for a scene.

I am pretty sure most modern video games use more mesh points than that for, like, a hat your character can wear.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#90

Post by kevingadd »

terafo wrote: post #42
ricardobayes wrote: post #36

Earlier quoted context omitted.

I remember Fallout New Vegas was almost 50 GB because it was a parallel install of all language/localization files. Once you manually deleted the languages you didn't want it went back to like 10 GB.

And there was a hilarious situation with audio in Titanfall, which included 35 gigs of uncompressed audio(whole game took 48 gigs).

The uncompressed audio was on purpose to manage CPU usage, you can find quotes explaining this in interviews.

https://www.escapistmagazine.com/titanfall-dev-explains-the-...

I can attest that MP3 and OGG decompression (the two audio codecs that would have likely been used at the time) are both pretty slow. It adds up.

Post reply on HN