Faster LZ is not the answer to 150-250 GB video game downloads • Hacker News Board

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#251

Post by starkparker » Sat, Apr 22, 2023, 3:26 PM UTC

cubefox wrote: post #248

KMnO4 wrote: post #10

Games want to be more photorealistic and the hardware to fake it is getting faster and cheaper. I remember in 2008 when Intel demoed ray tracing[0] for Quake Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS, and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene. Considering between then and now, we have much prettier looking games that do NOT require r…

> Considering between then and now, we have much prettier looking games that do NOT require ray tracing for their visual effects (and can run on much fewer CPU cycles), thanks to much more "clever" techniques and algorithms (DoF, bloom, subsurface scattering, PBR, etc), and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having…

There's a good breakdown from one of the Lumen devs for more detail on how it came about: https://knarkowicz.wordpress.com/2022/08/18/journey-to-lumen...

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#252

Post by phire » Sat, Apr 22, 2023, 3:45 PM UTC

KMnO4 wrote: post #10

Games want to be more photorealistic and the hardware to fake it is getting faster and cheaper. I remember in 2008 when Intel demoed ray tracing[0] for Quake Wars. It ran on a quad-socket (4 CPUs x 2.7GHz) at around 25 FPS, and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene. Considering between then and now, we have much prettier looking games that do NOT require r…

> and even Intel admitted that ray tracing is just "brute forcing" the complex optical physics of a scene

Modern ray traced games are much less "brute force" than what they were doing back in 2008. The use clever techniques and algorithms like temporal denoising and radiance caching to achieve high quality ray tracing effects with a tiny number of secondary rays per pixel (The budget is often under one ray per pixel).

It's not really fair to call it "brute force" at all, just a different set of clever algorithms.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#253

Post by yboris » Sat, Apr 22, 2023, 4:20 PM UTC

Animats wrote: post #162

Meanwhile, over in metaverse land, there are petabytes of content in the big worlds. I've been working on a faster Second Life / Open Simulator client. Those have a reputation for being sluggish. That's rapidly changing. All content is loaded on demand. Textures are stored in JPEG 2000 format. JPEG 2000 decoders come in several flavors - slow and flaky, fast but need the GPU, and midrange but expensive. If you want t…

What about using JPEG XL - .jxl format instead? https://jpegxl.info/

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#254

Post by maccard » Sat, Apr 22, 2023, 4:39 PM UTC

terafo wrote: post #249

maccard wrote: post #207

Earlier quoted context omitted.

> Fortnite doesn't use it(despite Epic's acquisition of RAD), You're talking nonsense. I worked on Fortnite and have direct experience with it and the HW decompression on PS5. I can even point you to the patch [0] where Fortnite enabled oodle compression and shrunk the game by 60GB. [0] https://twitter.com/FortniteStatus/status/131867784125345382...

I apologize, it appears that my way of estimating whether game uses HW compression is garbage.

Hey no worries, it can be hard to parse intent from text. My response was a bit harsh too!

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#255

Post by Animats » Sat, Apr 22, 2023, 5:07 PM UTC

bick_nyers wrote: post #232

Animats wrote: post #162

Meanwhile, over in metaverse land, there are petabytes of content in the big worlds. I've been working on a faster Second Life / Open Simulator client. Those have a reputation for being sluggish. That's rapidly changing. All content is loaded on demand. Textures are stored in JPEG 2000 format. JPEG 2000 decoders come in several flavors - slow and flaky, fast but need the GPU, and midrange but expensive. If you want t…

Check out HTJ2K if you haven't already, it's J2K with SIMD instructions, it will have solid support because the medical industry is (slowly) moving towards adopting it. Sounds like you already saw NVIDIA's J2K decoder. There are FPGA solutions as well that are insanely fast (CAST and some others). HTJ2K has support in OpenJPEG (master branch at least), might need to compile with a specific flag or something I can't r…

The content already exists. I'm just decoding it.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#256

Post by aseipp » Sat, Apr 22, 2023, 5:28 PM UTC

Animats wrote: post #164

terafo wrote: post #26

Earlier quoted context omitted.

> and if it weren't for the RTX graphics cards setting the precedent, there would probably be more research into making things prettier without having to "brute force" things. Not really. A lot of this research included making rudimentary versions of RTGI that could run on hardware of that age(see SVOGI). Of course you can bake everything and make amazing looking levels, that are completely static. But that doesn't w…

I'd rather have more RAM in the GPU and give up the ray tracing hardware. The real problem in the industry is that GPUs are not getting cheaper. It cost about $250 for an 6GB NVidia 1060 in 2016, and it costs about $250 for a basic NVidia card today that can do slightly better. Meanwhile, people are buying $100 WalMart laptops and trying to play games on them.

I mean, you could have a bunch of other things in the GPU instead of those cores, but the reality is that ray tracing serves multiple target markets for GPUs very well, and actually meets a demand people have: better graphical quality. In contrast, while I'm jealous of my friend with a 16GB 4080, versus my paltry 10GB 3080, it does not realistically impact anything I do outside of occasional ML retraining tasks. Neither did my old 8GB card. Gaming workloads just generally don't need it, because people there are mostly bound by compute, not texture or RAM limits -- and solutions like super-resolution/upscaling instead improve performance and memory usage all at once.

> and it costs about $250 for a basic NVidia card today that can do slightly better

Eh, the 3050 is like 30% more efficient in every single benchmark versus a 1060, with 30% more VRAM, and costs like $280. I don't think 30% is "slightly better". The Arc A750 also is around the same price point and is nearly 80% better, but I admit Nvidia has the best software stack on the market at the moment so the 3050 is a more fair comparison.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#257

Post by Animats » Sat, Apr 22, 2023, 5:38 PM UTC

ghilston wrote: post #242

Animats wrote: post #162

Meanwhile, over in metaverse land, there are petabytes of content in the big worlds. I've been working on a faster Second Life / Open Simulator client. Those have a reputation for being sluggish. That's rapidly changing. All content is loaded on demand. Textures are stored in JPEG 2000 format. JPEG 2000 decoders come in several flavors - slow and flaky, fast but need the GPU, and midrange but expensive. If you want t…

I used to love Second Life back in the day. Can you tell us more about your "game" , and less about the technical details? I know this is HN, but what would you tell a prospective player about your game?

I'm interested in metaverses that work. The NFT industry has crashed and burned [1], and Facebook/Meta has bombed in that space [2]. With the clown car out of the way, it's now possible to make progress. The lesson of these failures is that there's not much of a role for ads or brands in the metaverse. They're distracting and don't fit in. The successes, from Roblox to Fortnite to Second Life, charge users a modest fee each month.

A metaverse is not a game. Games can be built within it by users. This has been done in Second Life, but the games are 1) sluggish, and 2) space-constrained, because land is expensive. Those are scaling problems which can be solved.

This requires solving the scaling problems that led to the original article here. You can't just download everything in advance. There's too much stuff in the larger games.

Open Simulator is an open source re-implementation of Second Life servers, written in C#. It's been around for a while, and now it's getting a bit more developer attention. There are multiple federated grids of Open Simulator servers, and content stores where you can buy items, all under different management. Land is much cheaper than in Second Life, but servers tend to be under-resourced and slow. There are some people working quietly on trying to improve the Open Simulator technology to work better.

Other attempts to solve this problem include Improbable's system. Improbable managed to blow through $400 million on the scaling problem, producing a system that's too expensive to run.[3] (They're funded by SoftBank.) Some good indy games tried to use their system, but the server bill was too high. Their approach is a general-purpose distributed object manager, which seems to be the wrong tool for the job. Otherside uses Improbable, and they only turn on their world maybe twice a year for a few hours for special events.

[1] https://web3isgoinggreat.com/

[2] https://www.wsj.com/articles/meta-metaverse-horizon-worlds-z...

[3] https://www.improbable.io/

[4] https://www.ft.com/content/3508bec7-a2f8-414e-8059-7b96b2700...

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#258

Post by dontlaugh » Sat, Apr 22, 2023, 7:50 PM UTC

gabereiser wrote: post #131

kevingadd wrote: post #93

Earlier quoted context omitted.

So they're compressed? What's wrong with compressing packages? That usually produces a better compression ratio by exploiting redundancy cross-file.

When optimizing load times, it’s usually console that’s the benchmark. PC is assumed to have faster or equal hardware so they optimize for the lowest denominator. Consoles aren’t PC’s but they are very close. They lack some instructions while gaining others. I can’t speak on current gen consoles but older ones like the Xbox 360, only could handle zip format and at a slow pace. You could show a loading screen while yo…

It does vary a bit. One exception is that current consoles guarantee fast SSDs and DMA to GPU, which can't be said of PCs.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#259

Post by kevingadd » Sat, Apr 22, 2023, 9:32 PM UTC

ksec wrote: post #186

kevingadd wrote: post #109

Rich is a compression expert who wrote a good post, and there are some good comments in this thread already, but a few key summary points to help people understand the tradeoffs here: * Steam, Itch, etc all compress game files before sending them to you over the network for an install. The amount of stuff you download is usually much less than what gets stored on disk. There are still games with unusually massive upd…

Not sure why this isn't top post. Also not sure why the author of the post wasn't clear in a lot of these trade offs. It is either the author is missing the point, or am I missing something?

I think the post just assumes context that most people don't have.

Re: Faster LZ is not the answer to 150-250 GB video game downloads

#260

Post by xboxnolifes » Sat, Apr 22, 2023, 9:51 PM UTC

jon-wood wrote: post #227

xboxnolifes wrote: post #123

Earlier quoted context omitted.

With the size of games nowadays, that's likely a net time save for many people from the discounted download time.

Gamers being gamers though they’re not going to say “well, we on reflection I’ve not had to wait so long”, they’re going to slam you with one star reviews for the fact there’s an hour long loading screen on first start.

The recompression/decompression logic could (and I would argue, should) be packaged separately for distribution platform instead of being embedded as part of the game's logic. In the case of using Steam, it would be no different than how it shows separation of time spent downloading vs time spent installing.

This allows it to be ran automatically after download without having the user need to launch the game.