Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A file format uncracked for 20 years

landaire.net

Post reply on HN
Print view View original

51–60 of 63 posts

Re: A file format uncracked for 20 years

#51

Post by qingcharles »

monero-xmr wrote: post #13
mcdeltat wrote: post #12

> Compressing data means you save space on the disc... If you conveniently ignore the fact that common.lin is duplicated in each map's directory and is the same for every map I tested, which kinda negates part of this. This is an interesting thing I've noticed about game dev, it seems to sometimes live in a weird space of optimisation requirements vs hackiness. Where you'll have stuff like using instruction data as a…

And passion to deliver. Engineers will kill themselves for a game release for no extra money and far less salary than their abilities would demand at a bigcorp. But they love it so they do it, and hack as best they can, to get their art into the world.

I bought into that scam. My dream job was video game dev. I spent my whole childhood writing video games. It was how I ended up doing it professionally for near zero money and 80 hour weeks and burning out completely in two years.

Re: A file format uncracked for 20 years

#52

Post by RandomBacon »

ramses0 wrote: post #43
rusk wrote: post #14

Earlier quoted context omitted.

There was a running theme in mythic quest about the engineers sweating over the system while monetisation just went bolted on a casino. Also happened in GTA5 [0] there was a ridiculous loading glitch that was quite well documented on here a while ago. Also a monetisation bolt on. So you have competing departments one of whom must justify itself by producing a heavily after my system. And another one which is licensed…

There was one similar issue with DOOM framerate, I'm assuming an intern got tasked with adding the "blink the LED on the fancy mouse" code (due to a marketing partnership) and it absolutely _trashes_ the framerate! https://old.reddit.com/r/Doom/comments/bnsy4o/psa_deactivate...

Edit: Thanks!

-----

Original comment:

Please use old.* when posting reddit links:

https://old.reddit.com/r/Doom/comments/bnsy4o/psa_deactivate...

Re: A file format uncracked for 20 years

#53

Post by tapia »

iberator wrote: post #19
tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

> country format Country ?! What's the meaning

Sorry, that was a typo (autocorrected). I meant binary format.

Re: A file format uncracked for 20 years

#54

Post by jcranmer »

tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

The most important resource you'll need is a hex editor that can let you drop at a cursor and see what the value is at the cursor for all the basic datatypes (u8/u16/u32/u64, float, double, at minimum). Something like 010 Editor or ImHex.

If it's a really simple format, since you appear to have the ability to generate arbitrary file contents using the program, you can get some mileage by generating a suite of small contents with few changes between them. I reverse engineered the DSP sphere blueprint format by generating a blueprint with one node, then the same node located elsewhere, then two nodes, then two nodes and one frame between them, etc. But this process is really only possible for the simplest formats; I'd gander that most reverse-engineered file formats are heavily based on decompilation of the deserialization code.

A lot of binary file formats end up being some form of "container" format--essentially, a file contains some form of directory mapping an item ID to a location in the file, and the contents of that is in some other binary format. It's worth first checking if this is the case, and matching against known formats like ZIP or HDF5.

Re: A file format uncracked for 20 years

#56

Post by tapia »

ashdnazg wrote: post #27
tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

The bare basics are working with a hex editor and understanding data types - ints, floats, null-terminated strings, length-prefixed strings etc. I'd recommend taking a well documented binary file format (Doom WAD file?), go over the documentation, and see that you manage to see the individual values in the hex editor. Now, after you have a feel for how things might look in hex, look at your own file. Start by saving…

Thanks. That is kind of what I imagined. But I am not good at understanding the information from the hex editor. Reading the article I was a bit lost with the terms like little-endian and thought that that might be someone important concept to know for the task . I guess that that is what I should learn first.

Re: A file format uncracked for 20 years

#57

Post by tapia »

mrgaro wrote: post #30
tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

It helps tremendously if you have a programming background as usually the developers behind the original format didn't have any need to make things harder than they need to be. Because of this, you can often guess how the format works, aka. "If I was the original developer, how would I do this?"

I am an engineer, but not a computer scientist or developer. I have been using Linux for 20 years and program a lot at work and at home. I think it should be possible, but find it difficult to interpret the hex code. I do have a general idea of how the format should be organized, as it contains mostly geometric data and associated results.

Re: A file format uncracked for 20 years

#58

Post by tapia »

jcranmer wrote: post #54
tapia wrote: post #18

I'm always amazed by people doing reverse engineering of some country formats. There's a binary format that I've been wanting to reverse engineer, but I don't know exactly how to start. It's for the result file of a proprietary finite element program. Could anyone point me to some resources and also what are the basics that I need to learn to achieve this?

The most important resource you'll need is a hex editor that can let you drop at a cursor and see what the value is at the cursor for all the basic datatypes (u8/u16/u32/u64, float, double, at minimum). Something like 010 Editor or ImHex. If it's a really simple format, since you appear to have the ability to generate arbitrary file contents using the program, you can get some mileage by generating a suite of small c…

That sounds interesting. But how can you test these internal binary formats? Do I need to extract that somehow?

Re: A file format uncracked for 20 years

#59

Post by landr0id »

richardfey wrote: post #25
landr0id wrote: post #17

Earlier quoted context omitted.

For this file in particular I'm unsure. common.lin is a separate file which I believe is supposed to contain data common to all levels _before_ the level is loaded. There's a single exported object that all levels of the game have called `MyLevel`. The game attempts to load this and it triggers a load of the level data and all its unique dependencies. The common.lin file is a snapshot of everything read before this e…

They were doing this kind of optical media seek times tests/optimisations for PS1 games, like Crash Bandicoot. You certainly have more and better context than me on this console/game, I just mentioned it in case it wasn't considered. By the way, could the nonsensical offsets be checksums instead? Nice reverse engineering work and analysis there!

Thank you.

>By the way, could the nonsensical offsets be checksums instead?

If you're referring to those weird "addresses" that quickly became irrelevant, there's a CRC32 somewhere in the header immediately after them. The address value is the same across files with different contents too.

I was talking to a friend of mine about it and he suggested that maybe whatever process generated the files included the file's load address in case it could be mapped to the same address for some other optimization?

Re: A file format uncracked for 20 years

#60

Post by traverseda »

tapia wrote: post #58
jcranmer wrote: post #54

Earlier quoted context omitted.

The most important resource you'll need is a hex editor that can let you drop at a cursor and see what the value is at the cursor for all the basic datatypes (u8/u16/u32/u64, float, double, at minimum). Something like 010 Editor or ImHex. If it's a really simple format, since you appear to have the ability to generate arbitrary file contents using the program, you can get some mileage by generating a suite of small c…

That sounds interesting. But how can you test these internal binary formats? Do I need to extract that somehow?

ImHex will tell you if it's compressed. Do you understand data structures? Floats, all those data types?

I'd suggest looking at a format like msgpack to see what a binary data format could look like: https://msgpack.org/

Then be aware that proprietary formats are going to be a lot more complicated. Or maybe it's just zipped up json data, only way to tell is to start poking around at it.

Post reply on HN