Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kaitai: Describe the structure of data, not how you read or write it

kaitai.io

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Kaitai: Describe the structure of data, not how you read or write it

#51

Post by touisteur »

comex wrote: post #22
kevinherron wrote: post #18

Earlier quoted context omitted.

Are we talking about different things? ASN.1 does more or less what I was hoping Kaitai can do. Where's the breakthrough? I just want to be able to describe network protocols in some "language" and generate code that can serialize/deserialize it.

Kaitai is designed to describe arbitrary, preexisting binary formats. You can’t do that with ASN.1.

Err. I'm not sure what asn.1 is missing for this? I've seen lots of people use asn.1 exactly for this (i.e. writing a grammar for an arbitrary pre-existing binary format not readily described in asn.1).

Re: Kaitai: Describe the structure of data, not how you read or write it

#52

Post by ithkuil »

touisteur wrote: post #51
comex wrote: post #22

Earlier quoted context omitted.

Kaitai is designed to describe arbitrary, preexisting binary formats. You can’t do that with ASN.1.

Err. I'm not sure what asn.1 is missing for this? I've seen lots of people use asn.1 exactly for this (i.e. writing a grammar for an arbitrary pre-existing binary format not readily described in asn.1).

asn.1 distinguishes between schema and encoding; there are many binary encodings and you can technically devise a custom one that would let you describe the high level structure with an asn.1 grammar and then lay out the actual bits with a custom encoding format so that it matches the pre-existing format you're writing the new serde for). This may work as many formats have this leveled approach. E.g. the lowest level of the spec may tell something about how to encode integers (all integers are 32-bit big endian, or varlen encoded ...), sequences (ength prefixed, or terminated by a sentinel.

Any chances you have some reference to what you saw?

Re: Kaitai: Describe the structure of data, not how you read or write it

#54

Post by floppy123 »

floppy123 wrote: post #11
carom wrote: post #5

I do love kaitai and recently contributed a grammar, but both the title and the copy talk about writing. > Reading and writing binary formats is hard... Kaitai Struct tries to make this job easier... Kaitai cannot write data back out. [1] This is a major limitation for me. It would be nice to use it as a mutation engine for fuzzing, but without being able to write it back out, it is mostly just beneficial for analysi…

Any other lib that also generates Encoder/Writer code?

and still there is no generator that creates efficient and partialy streamed readers/writers for high performance protocols or resource constraint environments (as less pre allocations as possible, zero copy concepts,..., streamed reading/writing, good inline possibility, ...)

100% fix formats with no self reference or for example checksums that sits in front of the checksumd data (no streaming possible)...

Re: Kaitai: Describe the structure of data, not how you read or write it

#55

Post by Mikhail_Edoshin »

In ID3 tags there's such a thing as unsynchronization: the MP3 syncword is 11 bits set to one so for the tag data not to be mistaken for an MPR3 frame it must not have a similar sequence of bits. The solution was to replace every 0xFF in tag data with 0xFF 0x00. Or not to replace, as this mistake may only be made by old players that do not understand ID3. So there's a special setting for this that may occur in two places: in the whole tag or in an individual frame within a tag.

The logic itself is simple, as you read data byte-by-byte you need to check check if the previous byte was 0xFF and the frame or tag is marked as unsychronized. Yet it's not that simple to describe this declaratively. I wonder if Kaitai can actually do this. From what I see Kaitai does have at least part of ID3 described, but it doesn't seem to actually do unsynchronization, as far as I can tell from the code.

Re: Kaitai: Describe the structure of data, not how you read or write it

#56

Post by pie_flavor »

purpleidea wrote: post #49
ddevault wrote: post #33

This is a use-case for which a DSL is well-suited. They would be wise to abandon YAML.

I think that I agree of you, and I'm obviously fond of clever DSL's cough mgmt config cough , but it's not clear what specifics you had in mind here. Share away if you're interested!

If it was something that looked like YAML, but wasn't, that'd be fine. YAML in its simplest form is a neat idea. Indentation is simple and obvious. Same as it doesn't matter that TOML looks a heck of a lot like INI. But using YAML itself is terrible, because YAML is a colossally bloated language that does crazy things at the wrong points and has a million syntaxes that step on each others' toes.

Re: Kaitai: Describe the structure of data, not how you read or write it

#57

Post by touisteur »

ithkuil wrote: post #52
touisteur wrote: post #51

Earlier quoted context omitted.

Err. I'm not sure what asn.1 is missing for this? I've seen lots of people use asn.1 exactly for this (i.e. writing a grammar for an arbitrary pre-existing binary format not readily described in asn.1).

asn.1 distinguishes between schema and encoding; there are many binary encodings and you can technically devise a custom one that would let you describe the high level structure with an asn.1 grammar and then lay out the actual bits with a custom encoding format so that it matches the pre-existing format you're writing the new serde for). This may work as many formats have this leveled approach. E.g. the lowest level…

Right I think we were working with PER (tagless) to match a complex multi-layer protocol. Can't share the exemple but let's say asn.1 helped generate saner code than the handrolled one AND allowed other languages to decode...

I understand what you're saying though. Memories of using this were... unpleasant. I think the 'best' alternative for me would be RecordFlux. Ada-like syntax, generation of AoRTE-provable SPARK code, and recently the expressivity of the tool has increased. And the whole thing is in python, easily extensible to build things from the type description: generators, fuzzers, advanced specific parsers (need only one field and want the control fields' positions and sizes), wireshark plugins, Postgres extensions...

I really like what they're doing there. Might be the one of the low-effort (for the user!) lead-bullets for safer software.

Re: Kaitai: Describe the structure of data, not how you read or write it

#58

Post by ddevault »

purpleidea wrote: post #49
ddevault wrote: post #33

This is a use-case for which a DSL is well-suited. They would be wise to abandon YAML.

I think that I agree of you, and I'm obviously fond of clever DSL's cough mgmt config cough , but it's not clear what specifics you had in mind here. Share away if you're interested!

YAML has far too many features and doesn't let you structure data well for this use-case. There's no concise way to provide typings for its fields, for instance - note how every field requires two lines in the examples, far too verbose.

I don't really know what to tell you other than that YAML is not well-suited to this domain, and designing a domain-specific language which meets the particular needs of the system, and only just, would be better. You could base it off of C structs or Rust types or s-expressions or whatever else, the exact syntax isn't important so long as it allows you to concisely and precisely specify the semantics of the tool.

Re: Kaitai: Describe the structure of data, not how you read or write it

#59

Post by HelloNurse »

afranchuk wrote: post #42
jacb wrote: post #17

Earlier quoted context omitted.

I might be missing something - why would this be a breakthrough? It sounds complicated to generate the interfaces, sure, but is there a theoretical problem blocking this, or just practical?

Yeah I was also confused. I wrote a bidirectional parser/writer layer for yaml in a haskell program I had at work. The yaml structure mappings were all declarative in the code and even allowed documentation for the structures to be printed out. It's not that hard once you define the primitive bidirectional (higher-order) mapping function to go from a `Configurable a` to a `Configurable b`, the rest kind of unfolds fr…

Binary file formats can be vastly more convoluted than YAML. For example, consider roundtripping ZIP archives or PDF documents, or both at the same time (see https://www.alchemistowl.org/pocorgtfo/).
Post reply on HN