Live data from Hacker News

A file format uncracked for 20 years

landaire.net

11–20 of 63 posts

Re: A file format uncracked for 20 years

#11
This is an interesting post. I've been spending time on a hobby project[1] that requires reading some old archives and game asset files. I didn't have to do any reverse engineering since it's already done by others and documented on on the moddingwiki. However, I did implement the algorithms myself to work with the assets.

It's an interesting rabbit hole to go down into and this post makes me appreciate the way in which this kind of forensic analysis is done.

1: https://eye-of-the-gopher.github.io/

Re: A file format uncracked for 20 years

#12
> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this.

This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as audio to save space, but then forget to compile in release mode or something. Really odd juxtaposition of near-genius-level optimisation with naive inefficiency. I'm assuming it's because, while there may be strict performance requirements, the devs are under the pump and there's so much going on that silly stuff ends up happening?

Re: A file format uncracked for 20 years

#13

> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…

And passion to deliver. Engineers will kill themselves for a game release for no extra money and far less salary than their abilities would demand at a bigcorp. But they love it so they do it, and hack as best they can, to get their art into the world.

Re: A file format uncracked for 20 years

#14

> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…

There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino.

Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on.

So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed to generate revenue at any cost……

[0] https://news.ycombinator.com/item?id=26296339

Re: A file format uncracked for 20 years

#15

> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…

This might be an optimisation to avoid disc seeks on wildly far apart distances, which would introduce more latency.

Re: A file format uncracked for 20 years

#16
post #14

> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…

There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino. Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on. So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed…

There's also relative pain scales

Loading happen once per session and is less painful than frame stuttering all game, for example, so given a tight deadline one would get prioritized over the other

Re: A file format uncracked for 20 years

#17

> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…

This might be an optimisation to avoid disc seeks on wildly far apart distances, which would introduce more latency.

For this file in particular I'm unsure.

common.lin is a separate file which I believe is supposed to contain data common to all levels _before_ the level is loaded.

There's a single exported object that all levels of the game have called `MyLevel`. The game attempts to load this and it triggers a load of the level data and all its unique dependencies. The common.lin file is a snapshot of everything read before this export. AFAIK this is deterministic so it should be the exact same across all maps but I've not tested all levels.

When loading a level, the training level for instance contains two distinct parts. Part 1 of the map loads 0_0_2_Training.lin, and the second part loads 0_0_3_Training.lin. These parts are completely independent -- loading the second part does not require loading the first. It does a complete re-launch of the game using the Xbox's XLaunchNewImage API, so all prior memory I think should be evicted but maybe there's some flag I'm unaware of. That is to say, I'm fairly confident they are mutually exclusive.

So basically the game launches, looks In the "Training" map folder for common.lin, opens a HANDLE, then looks for whichever section it's loading, grabs a HANDLE, then starts reading common.lin and .lin.

There's multiple parts, but only one common.lin in each map folder. So no matter what it's not going to be laid out in a contiguous disc region for common.lin leading into .lin. Part 1 may be right after common.lin, but if you're loading any other part you'll have to make a seek.

I don't know enough about optical media seek times to say if semi-near locality is noticeably better for the worst case than the files being on complete opposite sector ranges of the disc.

Re: A file format uncracked for 20 years

#18
I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

Re: A file format uncracked for 20 years

#19
post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

> country format

Country ?! What's the meaning

Re: A file format uncracked for 20 years

#20
post #14

Earlier quoted context omitted.

There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino. Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on. So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed…

There's also relative pain scales Loading happen once per session and is less painful than frame stuttering all game, for example, so given a tight deadline one would get prioritized over the other

[deleted]
Post reply on HN