Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kaitai: Describe the structure of data, not how you read or write it

kaitai.io

Post reply on HN
Print view View original

21–30 of 66 posts

Re: Kaitai: Describe the structure of data, not how you read or write it

#21

Post by convolvatron »

jacb wrote: post #17
imtringued wrote: post #13

Earlier quoted context omitted.

It would be a CS breakthrough if it could... You're asking a lot...

I might be missing something - why would this be a breakthrough? It sounds complicated to generate the interfaces, sure, but is there a theoretical problem blocking this, or just practical?

I've looked at this problem quite a bit over the years...I agree with you completely. there isn't anything fundamental here, just the normal cultural adoption issues, usability, etc. There may be some compilation/complexity issues around formats with variable length fields and self-description, but certainly less problematic than general purpose programming.

I really wish though that there were more traction here, as I really believe that we should be quite prepared do deal with bits and not protobufs by default. nothing wrong with protobufs for quite a number of uses. I just don't know why people are so afraid of and/or biased against bit strings.

Re: Kaitai: Describe the structure of data, not how you read or write it

#22

Post by comex »

kevinherron wrote: post #18
imtringued wrote: post #13

Earlier quoted context omitted.

It would be a CS breakthrough if it could... You're asking a lot...

Are we talking about different things? ASN.1 does more or less what I was hoping Kaitai can do. Where's the breakthrough? I just want to be able to describe network protocols in some "language" and generate code that can serialize/deserialize it.

Kaitai is designed to describe arbitrary, preexisting binary formats. You can’t do that with ASN.1.

Re: Kaitai: Describe the structure of data, not how you read or write it

#23

Post by wpietri »

comex wrote: post #15
floppy123 wrote: post #11

Earlier quoted context omitted.

Any other lib that also generates Encoder/Writer code?

The Construct library for Python can do it: https://construct.readthedocs.io/en/latest/intro.html#exampl... I’ve long searched for something better than Construct, but so far I have yet to find even an equal.

Ooh, exciting! I built a parser [1] for AIS messages [2], a quirky ship-to-ship protocol. My lower-level stuff always felt clumsy to me. I'll have to see if this cleans it up.

[1] https://github.com/wpietri/simpleais [2] https://gpsd.gitlab.io/gpsd/AIVDM.html

Re: Kaitai: Describe the structure of data, not how you read or write it

#24

Post by attheicearcade »

comex wrote: post #15
floppy123 wrote: post #11

Earlier quoted context omitted.

Any other lib that also generates Encoder/Writer code?

The Construct library for Python can do it: https://construct.readthedocs.io/en/latest/intro.html#exampl... I’ve long searched for something better than Construct, but so far I have yet to find even an equal.

It can, but it can get incredibly slow for large formats. I was using it to reverse engineer some binary game formats but the parser would take a couple of minutes to complete. I rewrote it using struct and that time dropped to a few seconds. Useful for probing an unknown format, but I prefer the 010 editor since it’s more interactive.

Re: Kaitai: Describe the structure of data, not how you read or write it

#25

Post by joshspankit »

convolvatron wrote: post #21
jacb wrote: post #17

Earlier quoted context omitted.

I might be missing something - why would this be a breakthrough? It sounds complicated to generate the interfaces, sure, but is there a theoretical problem blocking this, or just practical?

I've looked at this problem quite a bit over the years...I agree with you completely. there isn't anything fundamental here, just the normal cultural adoption issues, usability, etc. There may be some compilation/complexity issues around formats with variable length fields and self-description, but certainly less problematic than general purpose programming. I really wish though that there were more traction here, as…

Through this thread I really feel like I’m missing something.

Are we not talking about writing binary files that conform to the spec?

Like, in the case of a GIF: simply writing valid garbage data should produce a file that presents as a valid GIF with noise for the image. Similarly: reading the file through the parser and writing it out unmodified should create an identical file (assuming no stenography).

Right?

Re: Kaitai: Describe the structure of data, not how you read or write it

#26

Post by jackjeff »

vander_elst wrote: post #4
nullspace wrote: post #2

This is very interesting, but just curious, when would you use something like this versus, say, protobufs?

If I understand it correctly, katai generates for you the reader as well, while protobuf generates only the data container. Katai is more a sort of protobuf + grpc for reading data.

Big difference is Katai is designed to parse existing binary formats like PNG, JPEG, MIDI, WAV files.

Things like Protobuf can serialize/deserialize any data but it’s very opiniated about how to do this. You won’t convince it to work with existing file formats.

Re: Kaitai: Describe the structure of data, not how you read or write it

#28

Post by kevinherron »

comex wrote: post #22
kevinherron wrote: post #18

Earlier quoted context omitted.

Are we talking about different things? ASN.1 does more or less what I was hoping Kaitai can do. Where's the breakthrough? I just want to be able to describe network protocols in some "language" and generate code that can serialize/deserialize it.

Kaitai is designed to describe arbitrary, preexisting binary formats. You can’t do that with ASN.1.

I know, I was just using it as kind of an example of what I want and of a similar problem. Maybe not the right example.

Re: Kaitai: Describe the structure of data, not how you read or write it

#29

Post by armsforsharks »

floppy123 wrote: post #11
carom wrote: post #5

I do love kaitai and recently contributed a grammar, but both the title and the copy talk about writing. > Reading and writing binary formats is hard... Kaitai Struct tries to make this job easier... Kaitai cannot write data back out. [1] This is a major limitation for me. It would be nice to use it as a mutation engine for fuzzing, but without being able to write it back out, it is mostly just beneficial for analysi…

Any other lib that also generates Encoder/Writer code?

I have been working on Deku: a declarative binary reading and writing: bit-level, symmetric, serialization/deserialization library. https://github.com/sharksforarms/deku

Re: Kaitai: Describe the structure of data, not how you read or write it

#30

Post by spullara »

joshspankit wrote: post #25
convolvatron wrote: post #21

Earlier quoted context omitted.

I've looked at this problem quite a bit over the years...I agree with you completely. there isn't anything fundamental here, just the normal cultural adoption issues, usability, etc. There may be some compilation/complexity issues around formats with variable length fields and self-description, but certainly less problematic than general purpose programming. I really wish though that there were more traction here, as…

Through this thread I really feel like I’m missing something. Are we not talking about writing binary files that conform to the spec? Like, in the case of a GIF: simply writing valid garbage data should produce a file that presents as a valid GIF with noise for the image. Similarly: reading the file through the parser and writing it out unmodified should create an identical file (assuming no stenography). Right?

I honestly don't get it either. The inverse of the read spec is the write spec. My guess having not dug deeply is that they don't distinguish between required and optional fields, that said, they should still be able to write what they have based on the read spec but could potentially be still an invalid file.
Post reply on HN