Super-Structured Data: Rethinking the Schema
brimdata.io
Super-Structured Data: Rethinking the Schema
1–10 of 48 posts
Re: Super-Structured Data: Rethinking the Schema
#2It looks like the use case is specifying types for dataflow operators (aka endpoints for dataflow pipes) [0] and I surmise composition should be super easy. I was surprised not to see any mention of XML or XML Schema as prior art, especially with their discussion of schema registries. Edit: Oh, the point of reference is Kafka [1]
[0] https://zed.brimdata.io/docs/language/overview/ [1] https://docs.confluent.io/platform/current/schema-registry/i...
Re: Super-Structured Data: Rethinking the Schema
#3tldr; Don't use relational tables or unstructured document databases. Instead use structured types. The "schema" here is ultimately a collection of independent objects / classes with well-defined fields.
Ok, fine. But I'm not sure how this helps if you have six different systems with six different definitions of a customer, and more importantly, different relationships between customers and other objects like orders or transactions or locations or communications.
I don't see their approach as ground-breaking, but it is definitely worthy of discussion.
Re: Super-Structured Data: Rethinking the Schema
#4It looks like the use case is specifying types for dataflow operators (aka endpoints for dataflow pipes) [0] and I surmise composition should be super easy. I was surprised not to see any mention of XML or XML Schema as prior art, especially with their discussion of schema registries. Edit: Oh, the point of reference is Kafka [1] [0] https://zed.brimdata.io/docs/language/overview/ [1] https://docs.confluent.io/platfo…
I also thought about XML. It has the Document Object Model (DOM), the structure which describes the data.
Re: Super-Structured Data: Rethinking the Schema
#5tldr; Don't use relational tables or unstructured document databases. Instead use structured types. The "schema" here is ultimately a collection of independent objects / classes with well-defined fields. Ok, fine. But I'm not sure how this helps if you have six different systems with six different definitions of a customer, and more importantly, different relationships between customers and other objects like orders…
It helps if this machinery can reject data and thus perform validation. Since recursive construction of union types (valid records can look like this, or also like that...) is trivial, a programmer somewhere has to draw the line between "loosen the schema to allow this record" and "reject this record to enforce the schema".
Re: Super-Structured Data: Rethinking the Schema
#6Perhaps I don't understand their use case fully, but it seems to me that every schema can be defined as a child protobuf message, and each child can then be added to a oneof field of a parent protobuf message. This way, you get the strict/optional type checks that are required, and the efficiency and ecosystem around protobufs.
Re: Super-Structured Data: Rethinking the Schema
#7Perhaps I don't understand their use case fully, but it seems to me that every schema can be defined as a child protobuf message, and each child can then be added to a oneof field of a parent protobuf message. This way, you get the strict/optional type checks that are required, and the efficiency and ecosystem around protobufs.
Author here. This totally makes sense. The challenge here is you need to store the type definitions somewhere (e.g., in the .proto files) and any system that processes protocol buffers needs to know which proto to apply to which messages. The theme of super-structured data is that this type structure should be native to the serialized data and our premise is this leads to better DX (though Zed is early and the jury is out). Perhaps flexbuffers is a closer analogy, which I should have mentioned in the article.
Re: Super-Structured Data: Rethinking the Schema
#8tldr; Don't use relational tables or unstructured document databases. Instead use structured types. The "schema" here is ultimately a collection of independent objects / classes with well-defined fields. Ok, fine. But I'm not sure how this helps if you have six different systems with six different definitions of a customer, and more importantly, different relationships between customers and other objects like orders…
It helps if this machinery can reject data and thus perform validation. Since recursive construction of union types (valid records can look like this, or also like that...) is trivial, a programmer somewhere has to draw the line between "loosen the schema to allow this record" and "reject this record to enforce the schema".
Author here. Agreed! Validation is important. While I didn't make this point in the article, our thinking is schema validation does not require that the serialization format utilize schemas as the building block and you can always implementation schema (or type) validation (and versioning) on top of super-structured data (as can also be done with document databases).
Re: Super-Structured Data: Rethinking the Schema
#9Note: The relational model (even SQL) is THIS.
Despite the claims, SQL is NOT "schema-fixed".
You can 100% create new schemas, alter them and modify them.
What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts).
--
SQL have limitations in lack of composability, the biggest reason "NoSQL" work is this: A JSON is composable. A "stringy" SQL is not. If SQL were really around "relations, tupes" like (stealing from my project, TablaM):
[Customer id:i32, name:Str; 1, "Jhon"]
Re: Super-Structured Data: Rethinking the Schema
#10Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…
I’m not sure what you mean by “composable” here — could you elaborate?