Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A file format uncracked for 20 years

landaire.net

Post reply on HN
Print view View original

31–40 of 63 posts

Re: A file format uncracked for 20 years

#31

Post by oarsinsync »

landr0id wrote: post #17
richardfey wrote: post #15

Earlier quoted context omitted.

This might be an optimisation to avoid disc seeks on wildly far apart distances, which would introduce more latency.

For this file in particular I'm unsure. common.lin is a separate file which I believe is supposed to contain data common to all levels _before_ the level is loaded. There's a single exported object that all levels of the game have called `MyLevel`. The game attempts to load this and it triggers a load of the level data and all its unique dependencies. The common.lin file is a snapshot of everything read before this e…

ISO9660 has support for something that resembles hard links - IE, a file can exist in multiple places in the directory structure, but always point to the same underlying data blocks on disc.

I think XISO is derived from ISO9660, so may have the same properties?

Re: A file format uncracked for 20 years

#32

Post by tomaytotomato »

Loved Splinter Cell

I wonder if any of the original devs will stumble upon the author's article and then remember why they did those weird file offsets.

There was a difference in the PC and Xbox versions, so it will be interesting to find out if the author sees any snippets or missing game assets in the Xbox version.

Re: A file format uncracked for 20 years

#34

Post by LunicLynx »

tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

The way I do it is looking for markers. Most files have some kind of magic number in the beginning. So these can valuable to recognize.

The next part is always looking into the values of 32 bit or 64 bit integers, if their value is higher than 0 but less then the files size they often are offsets into the file, meaning they address specific parts.

Another recommendation is to understand what you are looking for. For games, you are most likely looking for meshes and textures. For meshes in 3D every vertex of a mesh is most likely represented by 3 floats / doubles. If you see clusters of 3 floats with sensical values (e.g. without an +/-E component) its likely that your looking at real floats.

When looking for textures it can help to adjust the view on the data to the same resolution of the data your looking for. For example, if you are looking for a 8bit alpha map with a resolution of 64 x 64 then try to get 64 bytes in a row in your hex editor, you might be lucky to see the pattern show up.

For save games I can only reiterated what has been mentioned before. Look for unique specific values in the file as integers. For example how much gold you have.

I used these technics to reverse engineer: * Diablo 2 save games * World of Worcraft adt chunks * .NET Assembly files (I would recommend reading the ECMA specification though) * jade format of Beyond good and evil

Ah yes, invest in a good hex editor of course. For me Hex Workshop has been part of this journey.

Re: A file format uncracked for 20 years

#35

Post by rswail »

ralferoo wrote: post #28
richardfey wrote: post #25

Earlier quoted context omitted.

They were doing this kind of optical media seek times tests/optimisations for PS1 games, like Crash Bandicoot. You certainly have more and better context than me on this console/game, I just mentioned it in case it wasn't considered. By the way, could the nonsensical offsets be checksums instead? Nice reverse engineering work and analysis there!

IIRC the average seek time across optical media is around 120ms, so ideally you want all reads to be linear. I remember one game I worked on, I spent months optimising loading, especially boot flow, to ensure that every file the game was going to load was the very next file on the disk, or else the next file was an optionally loaded file that could be skipped (as reading and ignoring was quicker than seeking). For th…

This reminds me of Mel:

    Mel's job was to re-write
    the blackjack program for the RPC-4000.
    (Port?  What does that mean?)
    The new computer had a one-plus-one
    addressing scheme,
    in which each machine instruction,
    in addition to the operation code
    and the address of the needed operand,
    had a second address that indicated where, on the revolving drum,
    the next instruction was located.
https://users.cs.utah.edu/~elb/folklore/mel.html

Re: A file format uncracked for 20 years

#36

Post by bargainbin »

mcdeltat wrote: post #12

> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…

Exactly that - once it’s shipped it’s shipped. Doesn’t matter if the code is “clean” or “maintainable” or whatever.

The longer it’s not released for sale, the more debt you’re incurring paying the staff.

I’ve worked with a few ex-game devs and they’re always great devs, specifically at optimising. They’re not great at the “forward maintainability” aspect though because they’ve largely never had experience having to do it.

Re: A file format uncracked for 20 years

#37

Post by amiga386 »

So if I understand this right:

* common.lin contains filenames, so that filename-expansion code in the game can work. But the offsets and sizes associated with the files are garbage

* .lin contains a stream of every byte read from every file, while loading the level . The stream is then compressed in 16k chunks by zlib.

* There is no indication in that stream of which real file was being read, nor the length of each read, nor what seeking was done (if any). All that metadata is gone.

* The only way to recover this metadata is to run the game code and log the exact sequence of file opens, seeks, reads.

* Alternatively, extract all that Unreal object loader code from the game and reimplement it yourself, so that you can let the contents of the stream drive the correct reading of the stream. The code should be deterministic.

This sounds pretty hellish for the game developers, and I bet the debug versions of their game _ignored_ .lin and used the real source files, but _wrote_ .lin immediately after every load... any change to the Unreal objects could alter how they were read, and if the data streamed didn't perfectly match up with what was in the real files, you'd be toast.

It reminds me of the extreme optimisation that Farbrausch did for .kkrieger -- they built a single binary, then ran and played it under instrumentation, and _any_ code path that wasn't taken was deleted from the binary to make it smaller. They forgot to take any damage in that playthrough, so all the code that applies damage to the player was deleted. Oops!

Re: A file format uncracked for 20 years

#38

Post by debugnik »

amiga386 wrote: post #37

So if I understand this right: * common.lin contains filenames, so that filename-expansion code in the game can work. But the offsets and sizes associated with the files are garbage * .lin contains a stream of every byte read from every file, while loading the level . The stream is then compressed in 16k chunks by zlib. * There is no indication in that stream of which real file was being read, nor the length of each…

About .kkrieger's trimming, I had only heard that they forgot to press up on the main menu so it doesn't work, not about the damage thing.

Re: A file format uncracked for 20 years

#39

Post by cedws »

amiga386 wrote: post #37

So if I understand this right: * common.lin contains filenames, so that filename-expansion code in the game can work. But the offsets and sizes associated with the files are garbage * .lin contains a stream of every byte read from every file, while loading the level . The stream is then compressed in 16k chunks by zlib. * There is no indication in that stream of which real file was being read, nor the length of each…

Could you explain a bit more about that code path optimisation? Why wouldn’t the compiler eliminate dead code? It seems like a very haphazard blunt force optimisation method.

Re: A file format uncracked for 20 years

#40

Post by anamexis »

cedws wrote: post #39
amiga386 wrote: post #37

So if I understand this right: * common.lin contains filenames, so that filename-expansion code in the game can work. But the offsets and sizes associated with the files are garbage * .lin contains a stream of every byte read from every file, while loading the level . The stream is then compressed in 16k chunks by zlib. * There is no indication in that stream of which real file was being read, nor the length of each…

Could you explain a bit more about that code path optimisation? Why wouldn’t the compiler eliminate dead code? It seems like a very haphazard blunt force optimisation method.

The compiler can’t determine which code paths are never used in practice at runtime.
Post reply on HN