Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A RAM Edition of Dirty Coding Tricks

gamasutra.com

Post reply on HN
Print view View original

21–30 of 76 posts

Re: A RAM Edition of Dirty Coding Tricks

#21

Post by mattnewport »

chollida1 wrote: post #18
mattnewport wrote: post #11

A pretty common trick that was part of game programmer lore back in the PS2 / Xbox era was to have a large static array hidden away in some code file somewhere. When days before shipping you couldn't quite fit the release build in to memory this allowed a heroic programmer to miraculously 'find' a few hundred extra kilobytes of memory by reducing the size of the array by just enough to fit. There was a less common va…

I used to be a huge gamasutra reader so I've read the postmortem that described this. I was under the assumption that it was more of an apocryphal story that didn't actually happen. I mean how can someone hide a static array or no op loop from a team of 20+ programmers in a game code base. As far as I'm aware, it was never confirmed and seems a bit to far fetched to be real, but then again.... :)

I've seen it done. In those days programming teams were usually smaller and this would be something put in place by a lead (usually a grizzled veteran of shipping multiple titles) and not advertised to the entire team. It might be hidden away somewhere like the core memory manager which would be owned by that lead and not generally touched by anyone else without consulting them. If someone happened to find it when running a memory profile they'd probably go and ask the lead about it and be let in on the 'secret'.

I actually found one of these when pulled in to help a title ship that had been put in for a previous title in the franchise and forgotten about when the lead moved on. I found it when memory profiling and after talking with a few people we figured out what it was there for and I got to be the 'hero' who found some extra memory to ship.

Re: A RAM Edition of Dirty Coding Tricks

#22

Post by FRex »

chollida1 wrote: post #18
mattnewport wrote: post #11

A pretty common trick that was part of game programmer lore back in the PS2 / Xbox era was to have a large static array hidden away in some code file somewhere. When days before shipping you couldn't quite fit the release build in to memory this allowed a heroic programmer to miraculously 'find' a few hundred extra kilobytes of memory by reducing the size of the array by just enough to fit. There was a less common va…

I used to be a huge gamasutra reader so I've read the postmortem that described this. I was under the assumption that it was more of an apocryphal story that didn't actually happen. I mean how can someone hide a static array or no op loop from a team of 20+ programmers in a game code base. As far as I'm aware, it was never confirmed and seems a bit to far fetched to be real, but then again.... :)

Noel Llopis is quoted here as saying it did happen: https://www.gamasutra.com/view/feature/132500/dirty_coding_t...

I'd assume that in the era where lots of stuff was global and code was messy and convoluted to fit into a console no one would take issue with yet another global variable if it was named a clever enough name as some buffer, temporary, cache, etc. and compilers didn't yet warn about unused variables or remove them (which is an easy fix in C anyway with a single totally portable line or a special compiler specific directive).

Re: A RAM Edition of Dirty Coding Tricks

#23

Post by MaulingMonkey »

fnl wrote: post #5

Makes me wonder why Rust isn't popular with game devs. Wouldn't Rust's borrow checker and resource management at compilation time make most of the described memory reclaiming issues obsolete?

> Makes me wonder why Rust isn't popular with game devs.

Too new to have a mature gamedev ecosystem. I only finally started tooling around with rust when someone tweeted they'd gotten it working on the PS4, but that still puts me in the early adopter boat.

> Wouldn't Rust's borrow checker and resource management at compilation time make most of the described memory reclaiming issues obsolete?

No. It might help you verify that the systems you're building to tackle these problems are correctly implemented, but it's not going to help you tackle the fundamental problem of e.g. "this game has more 'active' data than we actually have the RAM for".

In theory, this is the problem that virtual memory and page files solve. In practice, occasional small 50-100ms stalls which might be acceptable for a text editor are really nasty for a real time twitch and flow based game. 60fps gives you a frame budget of 16ms - if you want your animation to remain at a fluid 60fps, your absolute max frame time is 16ms.

So, games will end up writing their own memory management systems to e.g. partially load and unload textures at runtime. They'll build in knowledge of your level layout to try and prefetch textures before they're actually necessary based on where the player is moving, so they're available by the time the game needs them - and they might chose to render using a lower resolution version of a texture that was kept in memory if the high resolution version wasn't loaded in time instead of stalling the game.

Re: A RAM Edition of Dirty Coding Tricks

#24

Post by twic »

meheleventyone wrote: post #17
fnl wrote: post #5

Makes me wonder why Rust isn't popular with game devs. Wouldn't Rust's borrow checker and resource management at compilation time make most of the described memory reclaiming issues obsolete?

Rust is definitely being looked at. The reason there are no big projects is that it’s still so new. For example the ability to write custom allocator is still going through the RFC process last I looked. Further it takes many man years of effort to write a AAA level game engine so there is quite a lot of inertia preventing moving to something new. We’re at the phase where people are experimenting within a relatively…

Chucklefish are working on some kind of medium-sized RPG:

http://www.pcgamer.com/new-details-and-screens-from-stardew-...

In Rust:

https://www.reddit.com/r/rust/comments/78bowa/hey_this_is_ky...

Re: A RAM Edition of Dirty Coding Tricks

#25

Post by pjmlp »

mattnewport wrote: post #11

A pretty common trick that was part of game programmer lore back in the PS2 / Xbox era was to have a large static array hidden away in some code file somewhere. When days before shipping you couldn't quite fit the release build in to memory this allowed a heroic programmer to miraculously 'find' a few hundred extra kilobytes of memory by reducing the size of the array by just enough to fit. There was a less common va…

Not only in game programming.

Chet Haase & Romain Guy also told on their Android history keynote that the kernel team did the same. They have hidden 20 MB.

https://www.youtube.com/watch?v=rimXGaUdaLg&t=773s

Re: A RAM Edition of Dirty Coding Tricks

#26

Post by MaulingMonkey »

fnl wrote: post #20
royjacobs wrote: post #9

Earlier quoted context omitted.

You can't do resource management at compile-time for games that have 50gb of assets. Also, Rust may help with accidental memory leaks, but you still have to deal with things like heap fragmentation.

Interesting point. Normally, your programming language can do little about fragmentation other than requesting a better allocator, if available (Windows?). Makes me think, maybe Rust's model would allow it to plan for better allocation requests? (Not saying Rust does that, just came as an idea.)

The main advantage I see Rust having here is making it a little easier to (ab)use the stack for temporary stuff in a way that can be verified to be safe. Nothing you can't already do in C++ at the expense of the occasional heisenbug.

Games often create their own allocators for a variety of reasons (specialized allocation strategies for speed or fragmentation reasons, adding debug statistics, enforcing memory budgets for (sub)systems, etc.) - although that's not terribly OS or language specific.

Re: A RAM Edition of Dirty Coding Tricks

#27

Post by meheleventyone »

twic wrote: post #24
meheleventyone wrote: post #17

Earlier quoted context omitted.

Rust is definitely being looked at. The reason there are no big projects is that it’s still so new. For example the ability to write custom allocator is still going through the RFC process last I looked. Further it takes many man years of effort to write a AAA level game engine so there is quite a lot of inertia preventing moving to something new. We’re at the phase where people are experimenting within a relatively…

Chucklefish are working on some kind of medium-sized RPG: http://www.pcgamer.com/new-details-and-screens-from-stardew-... In Rust: https://www.reddit.com/r/rust/comments/78bowa/hey_this_is_ky...

Awesome, it’ll be a good for there to be a commercial game project out there written in Rust when this gets released.

Re: A RAM Edition of Dirty Coding Tricks

#28

Post by wging »

warent wrote: post #3

When finishing a level, the game would reboot the console and restart itself with a command-line argument (the name of the level to start) ... into the next level. Voila, the perfect(?) way to clear all memory between levels. OH MY At 22 years old, this is one of those moments where I'm in awe of the strange issues and workarounds that existed merely ~5-10 years ago which I'll probably never have to deal with. Very f…

You'll run into today's equivalent soon enough :)

Re: A RAM Edition of Dirty Coding Tricks

#29

Post by Bartweiss »

mattnewport wrote: post #21
chollida1 wrote: post #18

Earlier quoted context omitted.

I used to be a huge gamasutra reader so I've read the postmortem that described this. I was under the assumption that it was more of an apocryphal story that didn't actually happen. I mean how can someone hide a static array or no op loop from a team of 20+ programmers in a game code base. As far as I'm aware, it was never confirmed and seems a bit to far fetched to be real, but then again.... :)

I've seen it done. In those days programming teams were usually smaller and this would be something put in place by a lead (usually a grizzled veteran of shipping multiple titles) and not advertised to the entire team. It might be hidden away somewhere like the core memory manager which would be owned by that lead and not generally touched by anyone else without consulting them. If someone happened to find it when ru…

> If someone happened to find it when running a memory profile they'd probably go and ask the lead about it and be let in on the 'secret'.

Perhaps the obvious question here is "hidden from who?" Blinding the entire dev team to the trick might be possible for the lead, but sounds like a pain.

But squirreling away a bit of memory that won't be announced to PMs/artists/designers/producers? I can certainly picture a couple of programmers realizing that the person calling the shots intended to push them to the absolute limits of "what will fit", and deciding to fudge those limits. Hell, hedging is common practice for programmers and freelancers today when they anticipate bad requirements, and everything I know of the game development's history says the problem used to be much worse.

Re: A RAM Edition of Dirty Coding Tricks

#30

Post by Tobba_ »

russellbeattie wrote: post #10

Whenever I think, "Thank goodness the limited memory days are behind us", it pops up again and again. Sure, you can buy a new iMac Pro with 128GB of RAM(!!) and smartphones regularly have 8GB available, but the increasingly popular IOT devices and smart consumer hardware (like streaming media boxes, etc.) try to limit the BOM cost and thus limit memory as much as possible. Tiny memory leaks become an issue, or random…

You'll still get an increase in the number of things you can have loaded into RAM at once that's proportional to the reduction. That'll give you far more leeway in when trying to load resources on the fly, which most games still struggle with. So reducing memory usage is definitely something you want to do regardless of how much you have.

Also, if it's a PC game, you're looking at memory limits far below what the HW may offer - at least if you want it to run without annoyances. Windows limits the amount of memory it'll allow to be committed to the DRAM size + (current) pagefile size (may only be 4-8GB on SSDs); note how that doesn't include GPU memory - and allocating that counts as commit due to how residency works. Then you're looking at the OS eating 2-3GB of that, plus 1+GB if the user has a browser opened, plus anything slowly leaking memory. And a LOT of recent Windows applications leak memory or actually use insane amounts (including, amazingly, a built-in service on Win10) - if they never access it again it'll eventually get evicted from the working set, which means it only counts towards committed memory; task manager doesn't display per-process committed memory by default. That's only a fraction of all the problems.

Post reply on HN