Earlier quoted context omitted.
This feels wrong. In that, I'd be surprised if they are not using compression. I think folks are just drastically underestimating how many resources are under discussion. And how big they are. Heck, just the geometry is probably far larger than folks anticipate.
The geometry would be hard to compress, but I suspect there's some clever tricks with some kind of fractal compression[1] that could drastically reduce texture sizes for camo, foliage, etc. without any effect on aesthetics. Encoding is the hard part, but only need be done once when building the final release. [1] https://en.wikipedia.org/wiki/Fractal_compression
Faster LZ is not the answer to 150-250 GB video game downloads
101–110 of 267 posts
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#102Earlier quoted context omitted.
To limit mesh size at less than 300 points for a scene. I am pretty sure most modern video games use more mesh points than that for, like, a hat your character can wear.
I it is true : I was very agressive on this one. But still, thousands and tousands of triangle feels like so much overweight for me. Specialy for indoor scenes. Quality for the eye is a lot to do with texture, NOT the number of triangle.
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#103Earlier quoted context omitted.
> This feels wrong. In that, I'd be surprised if they are not using compression. Check this out, you can sort by the number of submissions to filter out indie games https://docs.google.com/spreadsheets/d/14CVXd6PTIYE9XlNpRsxJ... from https://github.com/IridiumIO/CompactGUI
This list appears to be largely indie games, many with sizes under a gigabyte. I'm not sure why at-rest compression would be a high priority for a small team when their game fits on a $5 thumbdrive and will be compressed during the install process by Steam or Itch.
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#104We recently had blog post closely related to the topic: https://ph3at.github.io/posts/Asset-Compression/
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#105Earlier quoted context omitted.
8k texture with mipmaps takes around 100 megs uncompressed. Downloading it would be 8-10 seconds on 100 mbit connection. It is definitely possible to generate that image in 8 to 10 seconds on consumer GPU if you are using some kind of consistency models that are trained to generate specific game style and inferences in 8 bit.
I doubt you can do it in 8K at that speed and also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator
All inference of Stable Diffusion is done in FP16 at best, with multi-step models, and models themselves are highly inefficient. I would be surprised if in May 2024 4090 isn't capable of generating 8k(8192x8192) Stable Diffusion quality image in less than 1 second. And also you wouldn't realistically need to
> also why not just compress it? Any quality degradation from compression will be less than the artifacts produced by an AI image generator
Because it allows for much more variance in content. I'm a bit sick of seeing tiled textures everywhere TBH. Also virtually all textures would be AI generated in less than 5 years.
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#106Earlier quoted context omitted.
And there was a hilarious situation with audio in Titanfall, which included 35 gigs of uncompressed audio(whole game took 48 gigs).
The uncompressed audio was on purpose to manage CPU usage, you can find quotes explaining this in interviews. https://www.escapistmagazine.com/titanfall-dev-explains-the-... I can attest that MP3 and OGG decompression (the two audio codecs that would have likely been used at the time) are both pretty slow. It adds up.
MP3 isn't slow. I was playing MP3s on a 100 Mhz 486 DX4 in 1996. Certainly a system in 2014 can handle decompressing MP3s on the fly.
Even if you didn't want to do it on the fly during game play, they easily could have been decompressed at load time and it would have taken only a couple seconds. It's not like ENTIRE 35 GB of audio needed to be loaded all at once, nobody had that RAM back then, and very few people (especially gamers) have that much now.
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#107Earlier quoted context omitted.
Jedi Survivor having uncompressed textures is a speculation, but I'm extremely confident that it is this way. The fact that games nowadays ship without LOD systems, without texture compression, without adequate streaming is not just speculation. It is a fact.
You're saying Jedi Survivor doesn't use block texture compression? On which target? All targets? Can you capture a trace in RenderDoc to prove this? Or show raw bmp/uncompressed dds files on disk?
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#108Chocked at FAT size video games file ? Me too. Giga bytes Damned ! To reduce the size, I decided : 1) To use the standard PNG compression. 2) To limit mesh size at less than 300 points for a scene. - Result : a) File size Check it here : https://free-visit.net/fr/demo01 (pleaseload url twice : I still have a bug) Question : - why video games STUDIOS do not use this strategy ?
PNG decompression is way too slow. Some games use it but it's on the way out, either being replaced by codecs like Basis or QOI in most cases
The time to : download the globel zip file + open and reading the points and all textures => 5 to 7 seconds. This is super quick !
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#109* Steam, Itch, etc all compress game files before sending them to you over the network for an install. The amount of stuff you download is usually much less than what gets stored on disk. There are still games with unusually massive updates though, especially ones where the files change enough that delta compression fails. This is difficult to avoid when using some middleware due to the design of its file formats. Compressed archives actually make this worse because you can't delta compress big archives as well as individual files.
* Smaller on disk doesn't mean faster to load anymore. In the era of spinning rust HDDs, smaller usually loaded faster, but there were extremes - some compression formats that produced massive size reductions were also 10x slower to decompress or worse, which could produce slower overall game load times. Now that most targets have SSDs, this means that less compression often produces faster loads. See https://ph3at.github.io/posts/Asset-Compression/ (from elsewhere in this thread) for one example.
* For image and geometry data that is headed for the GPU, it's best to pick compression formats that can either be natively consumed by the hardware (block texture formats like BC7, etc) or that can be efficiently decompressed on the GPU once you copy it there, like gdeflate. This means the file on disk will probably be bigger, unless you supercompress it using something like Basis. Basis adds measurable CPU time to loading, though, and as a result I'm considering doing an uncompressed texture cache in my game to improve load times even though I plan to continue shipping textures as basis KTX2.
* For streaming asset loads, the CPU usage to decompress the assets during gameplay could lead to frame drops, inconsistent frame pacing, and pop-in. My understanding is that Titanfall opted for uncompressed audio in order to keep CPU usage low when streaming in all the sound effects and voices that could play during a hectic match - at that point in time CPU usage for audio mixing was a serious concern; modern CPUs more headroom so it's less of a problem now. It made sense for them to choose this as a trade-off because high-framerate twitch online shooters prioritize low consistent latency.
* In some cases the install you get depends on the hardware you have. It's not uncommon for older platforms to see 'high res texture packs' that could be installed optionally. The Xbox platform has something called "smart delivery" where it will deliver a smaller game bundle depending on whether your particular console needs high-def textures. This takes real engineering work to do, so many games opt not to support it because that time is better spent elsewhere.
* Ultimately, unless your game is too big to fit on people's hard drives at all, there is usually something more important for you to work on vs implementing some complex compression algorithm for every one of your files at rest. You are probably also not as smart as the compression experts who wrote the compression codecs that Steam/Itch/Xbox/Playstation offer, either for install-time compression (to reduce network bandwidth) or texture compression, etc. So in practice you probably just pick an off-the-shelf compression solution and apply it where it improves load times and you leave everything else alone. It doesn't make sense to spend weeks shrinking your on-disk size by 5% when you could spend that time fixing crash bugs, improving accessibility with better control customization, or improving the game's framerate instead.
* Lots of people decide a game "isn't compressed" because you can shove it into an lzma archive and make it smaller. This misunderstands the issue. Many compressed or optimized-for-size formats can still be made smaller by using a really heavyweight compression codec like lzma or brotli, but critically, there are tradeoffs involved - slower compression means slower build times means longer development times, and slower decompression means worse load times. The best compression typically also requires you to bundle all your files together, which loses the ability to load only the data you want when you want it.
An anecdote: Back when I was working on the original WebAssembly spec, I spent multiple weeks doing lots of research into custom compression and how we could design the format to make it compress more efficiently. I even had some chats with the Brotli team. In the end, we barely did anything, because even months of hard work barely produced better results than just Brotli-compressing our files at maximum settings. We were able to improve gzipped sizes tremendously by filtering and rearranging the data, but there's no point doing it if you can just use the browser's built in brotli decoder instead.
Re: Faster LZ is not the answer to 150-250 GB video game downloads
#110Earlier quoted context omitted.
And there was a hilarious situation with audio in Titanfall, which included 35 gigs of uncompressed audio(whole game took 48 gigs).
As I recall, this was a purposeful choice. Titanfall came out in 2014. The minimum requirements were a Core 2 Duo at 2.4 GHz or Athlon X2 at 2.8 GHz. I think that puts it on processors dating back to 2006 or so. Games sometimes have a lot of sound effects. It's easier to stream 35 GB of data from disk than it is to decompress, like, 10 streams of audio at the same time with whatever CPU you can dedicate to audio. I d…