> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…
This might be an optimisation to avoid disc seeks on wildly far apart distances, which would introduce more latency.
A file format uncracked for 20 years
41–50 of 63 posts
Re: A file format uncracked for 20 years
#42Earlier quoted context omitted.
Could you explain a bit more about that code path optimisation? Why wouldn’t the compiler eliminate dead code? It seems like a very haphazard blunt force optimisation method.
The compiler can’t determine which code paths are never used in practice at runtime.
Re: A file format uncracked for 20 years
#43> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…
There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino. Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on. So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed…
https://old.reddit.com/r/Doom/comments/bnsy4o/psa_deactivate...
Re: A file format uncracked for 20 years
#44Re: A file format uncracked for 20 years
#45Earlier quoted context omitted.
The compiler can’t determine which code paths are never used in practice at runtime.
I see. It sounds like it would be a source of countless headaches, and I don’t think I’d ever want to do something that risks breaking the program like that, but I guess that’s why I’m not a game programmer.
Re: A file format uncracked for 20 years
#46So if I understand this right: * common.lin contains filenames, so that filename-expansion code in the game can work. But the offsets and sizes associated with the files are garbage * .lin contains a stream of every byte read from every file, while loading the level . The stream is then compressed in 16k chunks by zlib. * There is no indication in that stream of which real file was being read, nor the length of each…
Re: A file format uncracked for 20 years
#47Re: A file format uncracked for 20 years
#48I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?
You can basically divide the world into read/write/write-only formats and read-only formats.
For read/write/write-only formats, usually the in-memory data structures were written first, and then the serialization/deserialization code. So it almost always more useful to see how the code works, than try to just figure out what random bytes in the file mean. A not insignificant percent of the time, the serialization/deserialization code is fairly straightforward - read some bytes, maybe decompress them, maybe checksum them and compare to a checksum field, shove in right place in memory structure/create a class using it, move on.
Different parts of the program may read different parts of a file, but again, usually a given part of the deserialization/serialization code is fairly understandable.
Read-only formats are scattershot. Lots of reasons. I'll just cover a few. First, because the code doesn't usually have both the writing and reading, you have less of a point of reference for what reading code is doing. Second, they are not uncommonly memory mapped serializations of in-memory structures. But not necessarily even for the current platform. So it may even make perfect sense and be easy to undersatnd on some platform, but on your platform, the code is doing weird conversions and such. This is essentially a variant of "the format was designed before the code". Lots and lots more issues.
I still would start by trying to understand deserialization code rather the format directly in this case, but it often is significantly harder to get a handle on.
There are commonalities in some types of programs (IE you will find commonalities between games using the same engine, etc), but if you are talking "in general", the above is the best "in general" i can give you.
One other tip - it is common to expect things to be logical and make sense - you can even see an example in this very article. Don't expect this.
For example, data fields that don't make sense or are broken, but the program doesn't use it so it doesn't matter, checksums that don't actually check anything, signed/verified files where the signing key is changeable easily, encryption where the key is hardcoded or stored in the file, you name it.
Most folks verify that their program works, they don't usually go look and verify that everything written/read makes any sense.
Re: A file format uncracked for 20 years
#49Earlier quoted context omitted.
The compiler can’t determine which code paths are never used in practice at runtime.
I see. It sounds like it would be a source of countless headaches, and I don’t think I’d ever want to do something that risks breaking the program like that, but I guess that’s why I’m not a game programmer.
https://fgiesen.wordpress.com/2012/04/08/metaprogramming-for...
Now it makes sense.
Re: A file format uncracked for 20 years
#50> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…
Then game dev was always full of fresh junior devs with tons of energy, ideas and dreams, but who are coming from home brew where things like reliable, beautiful, readable code are unnecessary.
And tons of things get missed. I keep hoping that the one published game I have was accidentally built with debug symbols in it so it can be easily traced. Two of us on the project were heavily into performance optimization, and I absolutely remember us going through compiler options, but things were crazy. I remember one major milestone build for Eidos I was hallucinating badly when I compiled and burned the CD because I'd been working for three days straight with no sleep.