Shame there's no PHP lib :(
your chance
Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
141–150 of 240 posts
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#142Shame there's no PHP lib :(
Disclosure: I manage the Ion and PartiQL teams at Amazon.
If you want to create an issue for it (the best repo is probably the ion-docs one: https://github.com/amzn/ion-docs/issues) that will help to show us there is demand for it. Providing information on your use case helps us prioritize.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#143Earlier quoted context omitted.
Ouch that’s rough. One nice bit of IPv6 is that it doesn’t allow fragmentation. It often much nicer to get no message or an error than subtly missing data.
IPv6 does allow fragmentation.
Ah yah that’s right. I’m just learning more of ipv6 and get it mixed up. It appears what I had in my mind was about intermediate routers: “Unlike in IPv4, IPv6 routers (intermediate nodes) never fragment IPv6 packets.” (Wikipedia). To the previous point, it looks like ipv6 does require networks to send 1280 byte or smaller packets unfragmented.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#144Earlier quoted context omitted.
Does birthday paradox apply here? It’s about any pair of people having the same birthday, whereas in this case you need someone else with a specific birthday. For example, if you generate 2 numbers and they are the same, but are different to the capnproto number, that’s a collision but doesn’t actually matter. EDIT: It does apply, I misunderstood what the number was being used for.
It does apply, according to https://www.johndcook.com/blog/2017/01/10/probability-of-sec...
But if my application only uses 100 schemas, I only care about a collision if it's with one of those 100.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#145This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
What you're describing is exactly what still takes place in trading platforms, although a few i've seen now use SBE for consistency sake (it's very common on the market data side)
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#146Earlier quoted context omitted.
I don't understand why serialization formats that separate structure and content aren't more popular. Imagine a system every message is a UID or DID ( https://www.w3.org/TR/did-core/ ) followed by raw binary data. The UID completely describes the shape of the rest of the message. You can also transmit messages to define new UIDs: these messages' UID is a shared global UID that everyone knows about. Once a client lear…
You just described protobufs and all its successors. See the “@0xdbb9ad1f14bf0b36” at the top of this capnproto file for example: https://capnproto.org/language.html It’s a 64bit random number so it’ll never have unintentional collisions. Also note that a capnp schema is natively represented as a capnp message. Pretty convenient for the “You can also transmit messages to define new UIDs” part of your scheme :)
Protobufs is a boring old tag-length-value format. It's kind of the worst of both worlds because it has no type information encoded in to it, meaning it's useless without the schema, while still having quite a bit of overhead.
Capn'Proto is more like a formalization of C structs in that new fields are only added to the end. If memory serves, on the wire there is no tag, type or length info (for fixed size field types), and everything is rooted at fixed offsets
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#147This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
Well that's just like using C structs. The best serialization protocol :).
Some finance software systems do that too. It tends to be a nightmare because people end up adding new message types just to add a single field
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#148This reminded me of a tight-packed binary format we used in the trading systems domain almost 20 years ago. Instead of including metadata/field names in each message, it had a central message dictionary that every client/server would first download a copy from. Messages had only type IDs, followed by binary packed data in the correct field order. Because of microsecond latency requirements, we even avoided the serial…
Is it FIX messages? https://en.wikipedia.org/wiki/Financial_Information_eXchange It's a good idea, extensible (ranges available for banks to implement their own codes), and fast.
Old school texty FIX is incredibly slow. FAST FIX is faster but not fun to use. Largely SBE has won adoption on the market data side, with huge platforms like Euronext (biggest on Europe) using it.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#149It's staggering to me that people keep making these "rich" data formats without sum types. At least to me, the "ors" are just as important as the "ands" in domain modeling. Apart from that, while you can always sort of fake it with a bunch of optional fields I believe that you kind of need a native encoding to a tagged union if you want to avoid bloating your messages.
data interchange formats try to encode as little backwards incompatible information as possible. in this case, it would be the restriction that something is a sum type when it could have multiple fields set in the future. another example is protobuf moving to all fields being optional by default. as for the wire format, a variant struct where you've only instantiated a single field will encode down to just about the…
Avro went the opposite way to most and just makes the concept of an optional field implementable via a union with null
Non union fields can even be upgraded to unions later
Personally I find the protobufs "everything is optional!" Behaviour fucking insane and awful to deal with, but it is true to the semantics of its underlying wire format.
Re: Amazon Ion – A richly-typed, self-describing, hierarchical serialization format
#150Earlier quoted context omitted.
No, that is not a sumtype, that's an array. In the same sense "1e-12" is not a number, it's a string. Yes, it's a string that encodes a number in a certain notion, but for alle the tooling, the IDE, the libraries, etc. it will stay a string.
What I mean is, it is an array of a sumtype `number | string | object`. So precisely, you could call it a `list `
That's a list union[number, string, object] or list[Any], not a sum type, no? This
`data X = A | B
[A, B, ...]`
Is a list containing a sum type: list[X]