Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A file format uncracked for 20 years

landaire.net

Post reply on HN
Print view View original

21–30 of 63 posts

Re: A file format uncracked for 20 years

#21

Post by GranPC »

avereveard wrote: post #16
rusk wrote: post #14

Earlier quoted context omitted.

There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino. Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on. So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed…

There's also relative pain scales Loading happen once per session and is less painful than frame stuttering all game, for example, so given a tight deadline one would get prioritized over the other

Loading in GTA Online absolutely does not happen once per session. It happens before and after every mission and activity. I am not sure whether it's a full load/was also affected by that bug, but I can certainly tell you that around 20% of my GTAO "playtime" consisted of staring at a load screen.

Re: A file format uncracked for 20 years

#22

Post by fainpul »

iberator wrote: post #19
tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

> country format Country ?! What's the meaning

Might be "binary format", autocorrected.

Re: A file format uncracked for 20 years

#23

Post by Orygin »

avereveard wrote: post #16
rusk wrote: post #14

Earlier quoted context omitted.

There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino. Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on. So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed…

There's also relative pain scales Loading happen once per session and is less painful than frame stuttering all game, for example, so given a tight deadline one would get prioritized over the other

I tried playing GTAO when it was free, and oh boy. Loading for 10 minutes, arrive into the game and see you're not with your friends. So 10 more minutes to load into their server. Then you start a mission and 10 more minutes of loading. The server disconnected? 10 minutes load to go back without your friend. Join your friend? guessed it: 10 more minutes of loading. For a billion dollar game, it's insane I spent more time loading than playing the game. Imagine how many more $$ they could have gotten if players could double their play time.

Re: A file format uncracked for 20 years

#24

Post by tralarpa »

tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

There are two approaches (sometimes mixed):

(a) you reverse engineer the application writing or reading the file. Even without fully understanding the application it can give you valuable information about the format (e.g. "The application calls fwrite in a for loop ten times, maybe those are related to the ten elements that I see on the screen").

(b) you reverse engineer only the file. For example, you change one value in the application and compare the resulting output file. Or the opposite way: you change one value in the file and see what happens in the application when you load it.

Re: A file format uncracked for 20 years

#25

Post by richardfey »

landr0id wrote: post #17
richardfey wrote: post #15

Earlier quoted context omitted.

This might be an optimisation to avoid disc seeks on wildly far apart distances, which would introduce more latency.

For this file in particular I'm unsure. common.lin is a separate file which I believe is supposed to contain data common to all levels _before_ the level is loaded. There's a single exported object that all levels of the game have called `MyLevel`. The game attempts to load this and it triggers a load of the level data and all its unique dependencies. The common.lin file is a snapshot of everything read before this e…

They were doing this kind of optical media seek times tests/optimisations for PS1 games, like Crash Bandicoot. You certainly have more and better context than me on this console/game, I just mentioned it in case it wasn't considered.

By the way, could the nonsensical offsets be checksums instead?

Nice reverse engineering work and analysis there!

Re: A file format uncracked for 20 years

#26

Post by rusk »

Orygin wrote: post #23
avereveard wrote: post #16

Earlier quoted context omitted.

There's also relative pain scales Loading happen once per session and is less painful than frame stuttering all game, for example, so given a tight deadline one would get prioritized over the other

I tried playing GTAO when it was free, and oh boy. Loading for 10 minutes, arrive into the game and see you're not with your friends. So 10 more minutes to load into their server. Then you start a mission and 10 more minutes of loading. The server disconnected? 10 minutes load to go back without your friend. Join your friend? guessed it: 10 more minutes of loading. For a billion dollar game, it's insane I spent more…

Put me right off the game.

Re: A file format uncracked for 20 years

#27

Post by ashdnazg »

tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

The bare basics are working with a hex editor and understanding data types - ints, floats, null-terminated strings, length-prefixed strings etc.

I'd recommend taking a well documented binary file format (Doom WAD file?), go over the documentation, and see that you manage to see the individual values in the hex editor.

Now, after you have a feel for how things might look in hex, look at your own file. Start by saving an empty project from your program and identifying the header, maybe it's compressed?

If it's not, change a tiny thing in the program and save again, compare the files to see what changed. Or alternatively change the file a tiny bit and load it.

Write a parser and add things as you learn more. If the file isn't intentionally obfuscated, it should probably be just a matter of persevering until you can parse the entire file.

Re: A file format uncracked for 20 years

#28

Post by ralferoo »

richardfey wrote: post #25
landr0id wrote: post #17

Earlier quoted context omitted.

For this file in particular I'm unsure. common.lin is a separate file which I believe is supposed to contain data common to all levels _before_ the level is loaded. There's a single exported object that all levels of the game have called `MyLevel`. The game attempts to load this and it triggers a load of the level data and all its unique dependencies. The common.lin file is a snapshot of everything read before this e…

They were doing this kind of optical media seek times tests/optimisations for PS1 games, like Crash Bandicoot. You certainly have more and better context than me on this console/game, I just mentioned it in case it wasn't considered. By the way, could the nonsensical offsets be checksums instead? Nice reverse engineering work and analysis there!

IIRC the average seek time across optical media is around 120ms, so ideally you want all reads to be linear.

I remember one game I worked on, I spent months optimising loading, especially boot flow, to ensure that every file the game was going to load was the very next file on the disk, or else the next file was an optionally loaded file that could be skipped (as reading and ignoring was quicker than seeking). For the few non-deterministic cases where order couldn't be predicted (e.g. music loaded from a different thread), I preloaded a bunch of assets up front so that the rest of the assets were deterministic.

One fun thing we often did around this era is eschew filenames and instead hash the name. If we were loading a file directly from C code, we'd use the preprocessor the hash the code via some complicated macros, so the final call would be compiled like LoadAsset(0x184e49da) but still retain a run-time hasher for cases where the filename was generated dynamically. This seems like a weird optimisation, but actually avoiding the directory scan and filename comparisons can save a lot of unnecessary seeking / CPU operations, especially for multi-level directories. The "file table" then just became a list of disk offset and lengths, with a few gaps because the hash table size was a little bigger than the number of files to avoid hash conflicts. Ironically, on one title I worked on we had the same modulo for about 2 years in development, and just before launch we needed to change it twice in a week due to conflicts!

Re: A file format uncracked for 20 years

#29

Post by blixt »

The quirks of field values not matching expectations reminds me of a rabbit hole when I was reverse engineering the Starbound engine[1] and eventually figured out the game was using a flawed implementation of SHA-256 hashing and had to create a replica of it [2]. Originally I used Python [3] which is a really nice language for reverse engineering data formats thanks to its flexibility.

[1] Starbounded was supposed to become an editor: https://github.com/blixt/starbounded

[2] https://github.com/blixt/starbound-sha256

[3] https://github.com/blixt/py-starbound

Re: A file format uncracked for 20 years

#30

Post by mrgaro »

tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

It helps tremendously if you have a programming background as usually the developers behind the original format didn't have any need to make things harder than they need to be. Because of this, you can often guess how the format works, aka. "If I was the original developer, how would I do this?"
Post reply on HN