Kaitai: Describe the structure of data, not how you read or write it
1–10 of 66 posts
Re: Kaitai: Describe the structure of data, not how you read or write it
#2This is very interesting, but just curious, when would you use something like this versus, say, protobufs?
Re: Kaitai: Describe the structure of data, not how you read or write it
#3i found the Kaitai toolset (IDE, compiler) to be useful in parsing or "deserialization" for proprietary financial protocols. if you are just reading data (especially in an environment with clients in different languages) it's a strong recommend, however lack of support for "serialization" means that you will still need to roll your own encoders. ultimately we created our own tooling/DSL for encoding as well as generating KSY files to generate clients as part of our builds
edit: i forgot to add - there is an issue for serialization on kaitai github repo for some time [1], with some interesting discussion around the implementation challenges
Re: Kaitai: Describe the structure of data, not how you read or write it
#4This is very interesting, but just curious, when would you use something like this versus, say, protobufs?
If I understand it correctly, katai generates for you the reader as well, while protobuf generates only the data container. Katai is more a sort of protobuf + grpc for reading data.
Re: Kaitai: Describe the structure of data, not how you read or write it
#5I do love kaitai and recently contributed a grammar, but both the title and the copy talk about writing.
> Reading and writing binary formats is hard... Kaitai Struct tries to make this job easier...
Kaitai cannot write data back out. [1] This is a major limitation for me. It would be nice to use it as a mutation engine for fuzzing, but without being able to write it back out, it is mostly just beneficial for analysis.
Re: Kaitai: Describe the structure of data, not how you read or write it
#6This is very interesting, but just curious, when would you use something like this versus, say, protobufs?
If you try to speak a protocol which is not yours.
Re: Kaitai: Describe the structure of data, not how you read or write it
#7I do love kaitai and recently contributed a grammar, but both the title and the copy talk about writing. > Reading and writing binary formats is hard... Kaitai Struct tries to make this job easier... Kaitai cannot write data back out. [1] This is a major limitation for me. It would be nice to use it as a mutation engine for fuzzing, but without being able to write it back out, it is mostly just beneficial for analysi…
Yikes, huge limitation. Guess I won't be looking into this any further.
Re: Kaitai: Describe the structure of data, not how you read or write it
#8This is very interesting, but just curious, when would you use something like this versus, say, protobufs?
Kaitai Struct can read any 'ol random data format while protobufs (I assume) are only able to read/write a specific protocol.
I was playing with it a while back along with wasm and got it to decode all the individual opcodes (along with the rest of the file) but it turned out to be really, really slow in the generated python version. C++ probably has much better performance but I haven't actually tested the difference.
Re: Kaitai: Describe the structure of data, not how you read or write it
#9I do love kaitai and recently contributed a grammar, but both the title and the copy talk about writing. > Reading and writing binary formats is hard... Kaitai Struct tries to make this job easier... Kaitai cannot write data back out. [1] This is a major limitation for me. It would be nice to use it as a mutation engine for fuzzing, but without being able to write it back out, it is mostly just beneficial for analysi…
Wow - thanks for the heads-up
Re: Kaitai: Describe the structure of data, not how you read or write it
#10This is very interesting, but just curious, when would you use something like this versus, say, protobufs?
Protocol Buffers and Kaitai structs solve different problems, although they both deal with serialized data. With a protobuf, you don't really care about how the data that you're serializing into the binary buffer is represented in the binary format. All you care about is that your data can be serialized and deserialized. Conversely, Kaitai allows you to control and specify the representation of the serialized data. This allows you to specify arbitrary formats (like image formats, for example).
Essentially you start with the serialization format instead of starting with the deserialized data (or like how the title of this submission says, "describe the structure of data"). As such, you can somewhat describe protobufs [0] using Kaitai structs, but the converse is not necessarily true.
There's actually a section under their FAQ [1] with a more in-depth response to this.