Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Reverse Engineering Unknown File Formats with ImHex

werwolv.net

Post reply on HN
Print view View original

31–40 of 55 posts

Re: Reverse Engineering Unknown File Formats with ImHex

#31

Post by userbinator »

hmartin wrote: post #11

Spoiler: half the time, that unknown file format is a zip containing a sqlite db

The older the system/the more changes it's had, especially if it's "enterprise software", the more insane things tend to be; in one memorable case, it was a .zip containing JSON, which then contained base64-encoded XML, wrapping another base64-encoded .zip containing more XML, and in that XML was base16-encoded ASN.1 PER with the actual payload (encoded in a proprietary format). I won't disclose specifics, but perhaps someone may recognise that from its description; suffice to say the industry that was related to started using computers in the days of mainframes and punched cards.

Re: Reverse Engineering Unknown File Formats with ImHex

#32

Post by kstrauser »

inventor7777 wrote: post #15
kstrauser wrote: post #6

Story time: I worked the night shift at a motel during college. "Bored to tears" was an understatement. I was poking around the front desk computer system one night, as one does, and found its data file. Lacking anything else to do, I wrote a little hex dumper in BASIC so I could explore the file. The first thing I noted was that customer names were spaced exactly N bytes apart. Oooh, fixed records! Then I spent the…

How did your program suggest auto-fixed names? A manually entered list of common names, or did it go by previous records? BASIC is over 40 years older than me, and I have never played with it, thus my question :-)

BASIC, while not a language I'd use for any reason today, is Turing complete. You can write any program in BASIC that you could write in Rust. Of course, you'd mainly do that for the same reason you'd write something in, say, Brainfuck: mainly to see if you can. But when that's all we had, we could get pretty creative with it.

It went by previous records. I'd read something about Levenshtein distance somewhere, and implemented my own half-assed version of it that was surely slower, more complicated, and less accurate. It still got the job done, though. I showed it to a coworker, who described it as "magic", and I floated on that compliment for a long time after.

I was pretty proud of it, although the resulting QBasic code was pretty awful in retrospect. It was probably my first commercial-scale project, built with the software that happened to ship with the computer.

Re: Reverse Engineering Unknown File Formats with ImHex

#33

Post by inventor7777 »

kstrauser wrote: post #32
inventor7777 wrote: post #15

Earlier quoted context omitted.

How did your program suggest auto-fixed names? A manually entered list of common names, or did it go by previous records? BASIC is over 40 years older than me, and I have never played with it, thus my question :-)

BASIC, while not a language I'd use for any reason today, is Turing complete. You can write any program in BASIC that you could write in Rust. Of course, you'd mainly do that for the same reason you'd write something in, say, Brainfuck: mainly to see if you can. But when that's all we had, we could get pretty creative with it. It went by previous records. I'd read something about Levenshtein distance somewhere, and i…

That's neat, thanks for explaining!

Re: Reverse Engineering Unknown File Formats with ImHex

#34

Post by vivzkestrel »

NegativeLatency wrote: post #27
vivzkestrel wrote: post #26

- stupid question: do you have any ideas how to go about doing this on a .unr file - it comes from a game called splinter cell conviction made in custom unreal 2.5 - I have neither been able to get UE-Explorer or UEViewer to work with it - I can send you the file, I just want to know how to change enemy AI spawn types on it

Try an LLM, either to do it directly or to guide your own explorations

as a guy who doesnt have experience fiddling with this kinda stuff, what is the step by step process if any to go about figuring out how an unknown file format is actually made of and modifying it?

- the post itself quotes "I usually couldn’t really give them a good answer except, “Look at the decompiled code of whatever program reads/writes these files and work backwards from there.”

- i dont have the program and that is the real challenge

Re: Reverse Engineering Unknown File Formats with ImHex

#35

Post by isaacbot »

truekonrads wrote: post #24

My few rules are: 1) What's the entropy? This helps with encrypted/compressed. You probably need to overcome this? 2) What's the context? Apps are developed in context and context suggests formats. Is it C? Then expect structs. Does it need to go over the wire? Expect run-length encodings. Python/JS - JSON/pickle. 3) There are broadly speaking only a few ways you can read data back into memory: fixed layout, run-leng…

[flagged]

Re: Reverse Engineering Unknown File Formats with ImHex

#37

Post by noduerme »

kstrauser wrote: post #6

Story time: I worked the night shift at a motel during college. "Bored to tears" was an understatement. I was poking around the front desk computer system one night, as one does, and found its data file. Lacking anything else to do, I wrote a little hex dumper in BASIC so I could explore the file. The first thing I noted was that customer names were spaced exactly N bytes apart. Oooh, fixed records! Then I spent the…

All was fun... sorry (about two decades younger) when I was a junior web designer in the late 90s, keyword designer, not coder, I got approached with a freelance job to build an online store, which I knew nothing about doing. I knew some PHP, mainly how to write stuff in and out of files and inline variables into html. Naturally, I said yes to the job even though I was totally unqualified. I mean, who was qualified? I had to build a whole login portal from scratch for them to upload products, and images of the products, and sub-sub-products, and add-ons and prices and descriptions, and then have all of that reflect on the public website where people would buy them by adding them to a shopping cart which was basically a very long GET query that kept getting added to.

So but words like "database" and "sql" scared the shit out of me at the time. I was doing most of my coding out of a starbucks in Eagle Rock and there was one other guy there on a laptop, John P, who kept looking at what I was writing and telling me I had to learn databases.

What I ended up coding was basically a database based on text files and pages and a custom encoding methods and a system for locking. Imagine a year later in 1999 when I realized this had all been solved and I'd been too stubborn to try it or understand what it did...

Fast forward a few years and someone put two sequential asterisks in in the name of a product and blew up the site and my encoding system, since they'd accidentally stumbled on the row break symbol. lol. Good times.

Re: Reverse Engineering Unknown File Formats with ImHex

#38

Post by tyffanypastecf »

truekonrads wrote: post #24

My few rules are: 1) What's the entropy? This helps with encrypted/compressed. You probably need to overcome this? 2) What's the context? Apps are developed in context and context suggests formats. Is it C? Then expect structs. Does it need to go over the wire? Expect run-length encodings. Python/JS - JSON/pickle. 3) There are broadly speaking only a few ways you can read data back into memory: fixed layout, run-leng…

[dead]

Re: Reverse Engineering Unknown File Formats with ImHex

#39

Post by subscribed »

vivzkestrel wrote: post #34
NegativeLatency wrote: post #27

Earlier quoted context omitted.

Try an LLM, either to do it directly or to guide your own explorations

as a guy who doesnt have experience fiddling with this kinda stuff, what is the step by step process if any to go about figuring out how an unknown file format is actually made of and modifying it? - the post itself quotes "I usually couldn’t really give them a good answer except, “Look at the decompiled code of whatever program reads/writes these files and work backwards from there.” - i dont have the program and th…

I would start with something like this: "Today's session will be a bit different and very exciting to me.

I have this game, Game Name exact version, which I love and I logged hundreds of hours in it, however there's this little detail that bothers me like a pebble in a shoe: (details of the problem here). I think that it's all encoded in the file (file name), but it's binary and I have no idea how to access it :(

Could you perhaps guide me how to analyse and "decode" this file so I could later change this behaviour? I don't have any experience with reverse engineering but I'm excited to try.

All game files are in /full/path/to/the/game

Thanks!"

Works for me.

Be excited, frame it as a learning opportunity, make sure it's obvious its for your own enjoyment (not for distribution).

Re: Reverse Engineering Unknown File Formats with ImHex

#40

Post by subscribed »

monster_truck wrote: post #3
someperson wrote: post #2

Damn a video game spoiler warning, guess I can't read the rest of the article. I may feed it into an LLM to strip out the spoilers actually

spoilers aren't real, it's like declaring you're immune to propaganda only it actually works

Come again?
Post reply on HN