Viewing profile — creationix

I see. We have different definitions of serialized. The way I typically see it used is to describe how something is transmitted and stored, not how it is read. Yes, by your definit…

So you're saying that random access formats that are encoded to disk as a stream of bytes are not "serialized" because you don't alway read them in order? Yes, many formats are rea…

> Just copy and paste files If all your workflows allow copying as binary files, more power to you! But there are a lot of workflows where that is not possible. This was inspired b…

> what sort of storage device does not allow your computers to use all 256 byte values - clipboards - logs - terminal output - alerts - yaml configs - JSON configs - hacker news co…

And don't worry about railroad diagrams. I already intended to create them, I've just been extra busy this week with other things.

Tha main reason for the reverse encoding is it makes it easier on the writer. You simply do a depth-first traversal of the data graph and emit data on the way back up the stack. Ze…

Also good luck parsing 10 MiB of JSON in a loop that can't tolerate blocking the CPU for more than 10ms. What's expensive is very relative to the use case.

Very true. I had forgotten about bencode, I should read up on that again. It makes sense they need a canonical form because they want same values to have same content hashes.

I meant computers can read it without any preprocessing. It's random access. You don't need to parse it, you don't need to decompress it. You just start at the end and follow point…

yeah, LuaJIT is one of the use cases I had in mind working on this. JSON is pretty fast in modern JS engines, but in Lua land, JSON kinda sucks and doesn't really match the languag…

> it only has a text encoding as long as you can guarantee you don't have any unicode? The format is technically a binary format in that length prefixes are counts of bytes. But in…

How does CBOR retain JSON compatibility more than RX? RX can represent any value JSON can represent. It doesn't even lose key order like some random-access formats do. In fact, RX …

I've rewritten the framing in the README to first explain when you should use RX and when you should not. Most uses of JSON should probably stay JSON. Let me know what you think ht…

Initial format docs are now here: https://github.com/creationix/rx/blob/main/docs/rx-format.md Railroad diagrams will come later when I have more time.

Thanks for the feedback. I've improved the framing to make the purpose/value more clear. What do you think about "RX is a read-only embedded store for JSON-shaped data"? https://ww…

That benchmark is a fair comparison for a real-world production workload and use case. Sadly I can't share the details. But suffice it to say that the dataset is a huge object with…

the project framing needs some help perhaps. JSON is really good at a lot of use cases that this will never replace. But there are cases where JSON is currently used where this is …

I'm happy to hear suggestions. This format was actually the internal .rexc bytecode for Rex (routing expressions), but when I realized it was actually a pretty good standalone form…

You're right. Some important differences: sick is binary, rx is textual (this matters for tooling) sick has size limits (65534 max keys for example. I have real-world rx datasets r…

> Does this duplicate the name of keys? Yes, the format allows for objects to be stored with a pointer to a shared schema (either an array of keys or another object that has the de…

The current format version is the exact same feature set as JSON. I even encode numbers as arbitrary precision decimals (which JSON also does). This is quite different from CBOR wh…

it's not really possible to stay human readable and get the compression levels and random access properties I was going for. But it is as human tooling friendly as possible given t…

yes, this would work very well for any case where you have embedded databases of unstructured data that you want to query in a website or edge server

yep. I built custom JSON parsers as a first solution. The problem is you can't get away from scanning at least half the document bytes on average. With RX and other truly random-ac…

I did seriously consider SQLite, but my existing datasets don't map easily to relational database tables. This is essentially no-sql for sqlite.