Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster LZ is not the answer to 150-250 GB video game downloads

richg42.blogspot.com

Post reply on HN
Print view View original

161–170 of 267 posts

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#161

Post by izacus »

kevingadd wrote: post #122
Sohcahtoa82 wrote: post #106

Earlier quoted context omitted.

What a bunch of absolute horse shit. MP3 isn't slow. I was playing MP3s on a 100 Mhz 486 DX4 in 1996. Certainly a system in 2014 can handle decompressing MP3s on the fly. Even if you didn't want to do it on the fly during game play, they easily could have been decompressed at load time and it would have taken only a couple seconds. It's not like ENTIRE 35 GB of audio needed to be loaded all at once, nobody had that R…

I've benchmarked it myself in my game. It adds a measurable amount of time to loads if you're decompressing all the audio up front to reduce latency, and during streaming it increases CPU usage. Obviously a modern machine can handle a few ogg/mp3 streams in the background during gameplay, but some games are playing dozens or hundreds of sounds at once. "It's not like ENTIRE 35 GB of audio needed to be loaded all at o…

Why are you assuming MP3/OGG formats are the ones to use for game audio compression ?

It's like trying to use PNG for texture compression instead of better suited formats.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#162

Post by Animats »

Meanwhile, over in metaverse land, there are petabytes of content in the big worlds.

I've been working on a faster Second Life / Open Simulator client. Those have a reputation for being sluggish. That's rapidly changing.

All content is loaded on demand. Textures are stored in JPEG 2000 format. JPEG 2000 decoders come in several flavors - slow and flaky, fast but need the GPU, and midrange but expensive. If you want to rewrite OpenJPEG in Rust, go for it. You can download part of a JPEG 2000 file and get a lower resolution texture, and that's what's done for distant objects. The general idea is to download textures at a size that gets one texel per screen pixel. (Yes, subpixel textures would be better.) Textures are locally cached to SSD. Background threads are frantically loading and unloading textures in the GPU at different resolutions as the viewpoint moves. The highest resolution is compressed losslessly; the others are lossy.

In metaverse systems with user generated content, there's not much instancing. So the texture load is much higher. Wander around Cyberpunk 2077 and you'll see the same railing show up in completely different contexts. This doesn't happen in worlds with user-created content. There's no producer or art director to insist on reusing content.

If your game downloads content on the fly, you need substantial content delivery capacity. Second Life front-ends Amazon AWS with Akamai. Open Simulator, on the other hand, often just hosts the content on a single game server, along with the the simulator itself. My own viewer pulls about 200Mb/s from the content servers for about 15 seconds when you first connect. Then it's done until the viewpoint moves.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#163

Post by bhouston »

dendriti wrote: post #155
bhouston wrote: post #154

250gB? No problem with 1gb/S to 3gb/s fiber to the home. At least here in Ottawa, Canada Fiber to the home is prevalent and affordable. But I understand where fiber isn’t available these are large downloads.

Well shit. Does your "Canada Fiber to the home" come with unlimited fast storage for free?

The article mentioned primarily download sizes and times. Honestly having +1Gb/S download speeds is quite nice. I also run a local 10Gb/S local network and again it makes large files seem small.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#164

Post by Animats »

terafo wrote: post #26
KMnO4 wrote: post #10

Games want to be more photorealistic and the hardware to fake it is getting faster and cheaper. I remember in 2008 when Intel demoed ray tracing[0] for Quake Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS, and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene. Considering between then and now, we have much prettier looking games that do NOT require r…

> and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things. Not really. A lot of this research included making rudimentary versions of RTGI that could run on hardware of that age(see SVOGI). Of course you can bake everything and make amazing looking levels, that are completely static. But that doesn't w…

I'd rather have more RAM in the GPU and give up the ray tracing hardware.

The real problem in the industry is that GPUs are not getting cheaper. It cost about $250 for an 6GB NVidia 1060 in 2016, and it costs about $250 for a basic NVidia card today that can do slightly better. Meanwhile, people are buying $100 WalMart laptops and trying to play games on them.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#165

Post by comex »

landr0id wrote: post #143
MrFoof wrote: post #12

The Moore's Law is Dead podcast recently did a podcast with an Infinity Ward award-winning effects artist: https://www.youtube.com/watch?v=tmfHxJT1I3I . The crazy thing he mentioned is that if there was zero compression, the latest COD games genuinely had over 1 terabyte of uncompressed assets.

>The crazy thing he mentioned is that if there was zero compression, the latest COD games genuinely had over 1 terabyte of uncompressed assets. I think what this shows is there should be smart delivery of assets. Before installing perhaps you should be able to choose your graphical fidelity and only download assets related to that fidelity level and below. This makes manifest building and delivery more complicated, a…

CoD itself can download some high-resolution textures during gameplay:

https://www.charlieintel.com/warzone-on-demand-texture-strea...

I've never tested how much bandwidth it uses. It seems a little crazy but also neat.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#166

Post by midnightclubbed »

recursive wrote: post #110
klodolph wrote: post #83

Earlier quoted context omitted.

As I recall, this was a purposeful choice. Titanfall came out in 2014. The minimum requirements were a Core 2 Duo at 2.4 GHz or Athlon X2 at 2.8 GHz. I think that puts it on processors dating back to 2006 or so. Games sometimes have a lot of sound effects. It's easier to stream 35 GB of data from disk than it is to decompress, like, 10 streams of audio at the same time with whatever CPU you can dedicate to audio. I d…

That's like 50 hours of uncompressed stereo audio at CD quality. (2×16bit@44.1kHz) Do games really have that much stuff in them these days? I mean, I wouldn't doubt it, but it's surprising.

For story based games (with voice acting) absolutely yes. Dozens of story characters, hundreds of supporting npc and all of them talking as either part of your missions or just to fill in the world.

Maybe 50 hours is a lot for a non story based game but even then once you factor in all the sounds and all their variations to reduce repetition you might be up there.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#167

Post by midnightclubbed »

izacus wrote: post #161
kevingadd wrote: post #122

Earlier quoted context omitted.

I've benchmarked it myself in my game. It adds a measurable amount of time to loads if you're decompressing all the audio up front to reduce latency, and during streaming it increases CPU usage. Obviously a modern machine can handle a few ogg/mp3 streams in the background during gameplay, but some games are playing dozens or hundreds of sounds at once. "It's not like ENTIRE 35 GB of audio needed to be loaded all at o…

Why are you assuming MP3/OGG formats are the ones to use for game audio compression ? It's like trying to use PNG for texture compression instead of better suited formats.

What formats are better suited? To my knowledge most games use mp3-like compression for all their audio. Console hardware often has decompression support, it’s reasonably compact, quality/bitrate can be tuned by the sound engineers and supports a wide range of content.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#168

Post by bullen »

I use zip (in flight) and 2 different lz (on disk) for my 3D action voxel MMO.

But the characters are flat shaded so those textures are tiny.

You need to look at the data, animation weights compresses 95% with a simple lz.

One big dissappointment was half-float support having low performance on the some drivers, that would have saved 50% off the models.

One advantage of not compressing is that your disk shows the (V)RAM usage you'll have.

I'm aiming for 1GB total (to support 2GB Raspberry 4) and my biggest concern is uncompressed 120Hz animations! (to save CPU)

But so many of those can be hot-loaded off disk.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#169

Post by midnightclubbed »

terafo wrote: post #56
sroussey wrote: post #32

Earlier quoted context omitted.

Textures are not compressed for size on disk, they are compressed to the format that can be directly loaded into the GPU.

You can add lossless compression, like Kraken, on top of that. PS5 has hardware accelerated Kraken decompression. And you can encode textures for GPU in a way that is better compressable with lossless algorithms.

Hardware accelerated is still not zero cost. Plus if you are publishing for the other console and/or PC there is a question as to whether you want to do the extra work for one platform.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#170

Post by kevingadd »

usefulcat wrote: post #152
kevingadd wrote: post #94

Earlier quoted context omitted.

Titanfall shipped uncompressed audio on purpose to cut out audio decompression CPU usage. You can disagree with the decision, but it wasn't an accident or negligence.

Are spare cores really such a scarce resource? I thought most games didn't come close to using all available cores. And audio decompression is not known for being that cpu intensive.

It was a long time ago, the CPU usage was an actual problem.
Post reply on HN