Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

github.com

Post reply on HN
Print view View original

21–30 of 43 posts

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#21

Post by delaguardo »

sevensor wrote: post #20

I don’t wish to pick on this post, it looks quite well done. However, in general, I have some doubts about data formats with typed primitives. JSON, TOML, ASN.1, what have you. There’s very little you can do with the data unless you apply a schema, so why decode before then? The schema tells you what type you need anyway, so why add syntax complexity if you have to double check the result of parsing?

I can do a lot without applying schema at all. For that I only need handful of types defined in EDN specification and Clojure programming language.

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#23

Post by sevensor »

delaguardo wrote: post #21
sevensor wrote: post #20

I don’t wish to pick on this post, it looks quite well done. However, in general, I have some doubts about data formats with typed primitives. JSON, TOML, ASN.1, what have you. There’s very little you can do with the data unless you apply a schema, so why decode before then? The schema tells you what type you need anyway, so why add syntax complexity if you have to double check the result of parsing?

I can do a lot without applying schema at all. For that I only need handful of types defined in EDN specification and Clojure programming language.

Suppose you have the EDN text

    (
      {
        :name "Fred"
        :age 35
      }
      {
        :name 37
        :age "Wilma"
      }
    )
There's a semantic error here; the name and age fields have been swapped in the second element of the list. At some point, somebody has to check whether :name is a string and :age is a number. If your application is going to do that anyway, why do syntax typing? You might as well just try to construct a number from "Wilma" at the point where you know you need a number.

Obviously I have an opinion here, but I'm putting it out there in the hope of being contradicted. The whole world seems to run on JSON, and I'm struggling to understand how syntax typing helps with JSON document validation rather than needlessly complicating the syntax.

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#24

Post by fulafel »

exceptione wrote: post #14

Interesting, I had to look up what EDN is. Important to note that EDN doesn't have a concept of a schema like JSON Schema. This is a `map`, which bears semblence with a Json object. The following might look like an incorrect paylood, but will actually parse as valid EDN: {:a 1, "foo" :bar, [1 2 3] four} // Note that keys and values can be elements of any type. // The use of commas above is optional, as they are parse…

JSON doesn't have schemas either, JSON Schema is just a separate schema spec that happens to build on JSON, but you might be using for example Zod instead of that. Similarly systems that consume EDN can have various schema systems. For example spec or malli in the Clojure world. (Or you could be using Zod with EDN, etc).

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#25

Post by Jeaye »

delaguardo wrote: post #15
Jeaye wrote: post #3

This is superb. Thank you for making it and licensing it MIT. I think this is a contender to replace the lexer within jank. I'll do some benchmarking next year and we'll see!

Wow, that is a greate news!) Thanks for looking at it from this perspective! There are some benchmarks already available in the project - https://github.com/DotFox/edn.c/blob/main/bench/bench_integr... you can run it locally with `make bench bench-clj bench-wasm` Let me know if I can do anything to help you with support in jank.

It looks like the key missing part which would be needed for a lexer is source information (bare minimum: byte offset and size). I don't think edn.c can be used as a lexer without that, since error reporting requires accurate source information.

As a side note, I'm curious how much AI was used in the creation of edn.c. These days, I like to get a measure of that for every library I use.

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#26

Post by zzo38computer »

delaguardo wrote: post #17
zzo38computer wrote: post #9

I think it would be better to not use Unicode (so that you can use any character set), and to use "0o" instead of "0" prefix for octal numbers. Also, EDN seems to lack a proper format for binary data. I think ASN.1 (and ASN.1X which is I added a few additional types such as key/value list and TRON string) is better. (I also made up a text-based ASN.1 format called TER which is intended to be converted to the binary D…

> EDN seems to lack a proper format for binary data The best part of EDN that it is extendable :) #binary/base64 "SGVsbG8sIHp6bzM4Y29tcHV0ZXIhIEhvdyBhcmUgeW91IGRvaW5nPw==" This is a tagged literal that can be read by provided (if provided) custom reader during reading of the document. The result could be any type you want.

OK, this is possible, but it seems the type that ought to be a built-in type.

Also, if there is not a binary file format for the data then you will need to always convert to/from base64 when working with this file whether or not you should need to.

Furthermore, this does not work very well when you want to deal with character sets rather than binary data, since (as far as I can tell from the specification) the input will still need to be UTF-8 and follow the EDN syntax of an existing type.

From what I can understand from the specification, the EDN decoder will still need to run and cannot be streamed if the official specification is used (which can make it inefficient), although it would probably be possible to make an implementation that can do this with streaming instead (but I don't know if the existing one does).

So, the extensibility is still restricted. (In my opinion, ASN.1 (and ASN.1X) does it better.)

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#27

Post by zzo38computer »

sevensor wrote: post #20

I don’t wish to pick on this post, it looks quite well done. However, in general, I have some doubts about data formats with typed primitives. JSON, TOML, ASN.1, what have you. There’s very little you can do with the data unless you apply a schema, so why decode before then? The schema tells you what type you need anyway, so why add syntax complexity if you have to double check the result of parsing?

I think it depends what you will intend to do with the data (which is true for all of the formats that you mentioned); not everyone will do the same thing with it even if it is the same file. It might be helpful to know from other programs that do not know this schema to be able to parse the data (not always the case when using IMPLICIT types in ASN.1, which is one reason to use EXPLICIT instead, although it has avantages and disadvantages compared with IMPLICIT; however, in DER all types will use the same framing allowing the framing to be parsed even if the specific type cannot be understood by the reader), and can also be used in case the schema is later extended to use types other than the ones that were originally expected. (I prefer to use ASN.1 DER in my stuff, although JSON and other formats are also used by other formats that were made by someone else)

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#28

Post by zzo38computer »

eliasdejong wrote: post #18

[dead]

> EDN also has no builtin 'raw bytes' type.

That was my complaint too.

> I am working on a format consisting of serialized B-tree. It is essentially a dictionary, but serialized

I had wanted something a bit similar; a serialized B-tree (or a similar structure) but with only a 'raw bytes' type, for keys and values (I will use DER for the values; I have my own library to work with DER already), and the ability to easily find all records whose key matches a specified prefix.

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#29

Post by delaguardo »

Jeaye wrote: post #25
delaguardo wrote: post #15

Earlier quoted context omitted.

Wow, that is a greate news!) Thanks for looking at it from this perspective! There are some benchmarks already available in the project - https://github.com/DotFox/edn.c/blob/main/bench/bench_integr... you can run it locally with `make bench bench-clj bench-wasm` Let me know if I can do anything to help you with support in jank.

It looks like the key missing part which would be needed for a lexer is source information (bare minimum: byte offset and size). I don't think edn.c can be used as a lexer without that, since error reporting requires accurate source information. As a side note, I'm curious how much AI was used in the creation of edn.c. These days, I like to get a measure of that for every library I use.

It should be easy to add source info for every token, some of them already keep both (size and offset) I can create a branch for that.

> I'm curious how much AI was used in the creation of edn.c

A fair amount. This is my first big public project written in pure C. I did consult LLM about best practices for code organisation, memory management, difference in SIMD instructions between platforms, etc. All the things Clojure developer typically don't think about (luxury of a hosted language). Ultimately, the goal was to learn some part of C programming, working reader is a side effect of that.

> These days, I like to get a measure of that for every library I use.

Btw, I'm curious, what kind of measuring you are looking for?

Re: A fast EDN (Extensible Data Notation) reader written in C11 with SIMD boost

#30

Post by delaguardo »

sevensor wrote: post #23
delaguardo wrote: post #21

Earlier quoted context omitted.

I can do a lot without applying schema at all. For that I only need handful of types defined in EDN specification and Clojure programming language.

Suppose you have the EDN text ( { :name "Fred" :age 35 } { :name 37 :age "Wilma" } ) There's a semantic error here; the name and age fields have been swapped in the second element of the list. At some point, somebody has to check whether :name is a string and :age is a number. If your application is going to do that anyway, why do syntax typing? You might as well just try to construct a number from "Wilma" at the poi…

What do you mean under "syntax typing" and complications in the syntax?

> The whole world seems to run on JSON

That is true, and I don't like that :)

From my perspective JSON syntax is too "light" and that translates to many complications typically in the form of convention: {"id": {"__MW__type": "LONG NUMBER", "value": "9999999999999999999999999"}}.

Post reply on HN