A file format uncracked for 20 years

A file format uncracked for 20 years
landaire.net

So in this file seek doesn’t do anything because seek kills the requirement of 45 sec per loading screen.

Instead the logic is as follows: check if a .lin file exists. Yes: open a handle to it and only read from it with fread, what ever currently is at the current file position. No: while reading any file write the read bytes to a .lin file in the order they are read.

This gives a highlyy optimized .lin file which can be read from the disk into memory, without creating a better dedicated loading mechanism.

So if your really would like to unpack this. The first file being read is most likely the key, as it dictates what comes next. If it is a level model, then the position of the player in it might affect which other files to load etc.

In short it’s not a file format in the classical sense, it’s a linear stream of game data.

Yes, it's buried deep in the details but it's basically just every byte read being written in a linear stream to an output file.

I don't know which stage of grief this is, but since I wrote this blog post I've now ported my IDA debugger scripts to a dedicated QEMU plugin which logs all I/O operations and some other metadata. I tried using this technique to statically rewrite files by basically following DataLoad (with unique identifier) -> Seek -> Read patterns.

There's some annoying nuance to deal with (like seeking backwards implying that data was read, tested, then discarded) but I got this working. Unfortunately some object types encode absolute offsets in them that need to be touched up, so a couple of object types fail to load correctly in external tooling and the PC build of the game.

Now I'm just using this data to completely reimplement the game engine's loading logic from scratch using a custom IO stream which checks the incoming IO operation (seek/read) against what I logged from the game engine to ensure a 1:1 match with how the game loads data.

You stated in the blog post, that your goal is to try and find unused content, however if as described, the file is just a record of how the game loads the data, then it won't contain any hidden unused assets, since unused assets would never have been read from the original unoptimised file, and thus never written to this optimized file.