I got interested when the different spectrum points of json and relational were contrasted. So I read the whole thing. I got lost and disheartened when the new terminology, starting with the super-structured name was introduced and completely went downhill with the other z names. Maybe it's just me and maybe it is like quantum mechanics and any other innovation where new names don't make sense and feel ugly.
Super-Structured Data: Rethinking the Schema
11–20 of 48 posts
Re: Super-Structured Data: Rethinking the Schema
#12I didn't follow this part:
> EdgeDB is essentially a new data silo whose type system cannot be used to serialize data external to the system.
I think this implies that serializing external data to zson is easier than writing an INSERT into edgedb, but not sure why that would be.
Re: Super-Structured Data: Rethinking the Schema
#13This is a data serialization format, not a replacement for storing your business data. Your business data needs to have the same schema enforced everywhere, otherwise how are you going to reconcile your user data now and your user data 5 months ago if their schemas are radically different?
Re: Super-Structured Data: Rethinking the Schema
#14Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…
I’m not sure what you mean by “composable” here — could you elaborate?
Composable is the ability to define things in the small and combine with confidence.
SQL not allow this:
by_id := WHERE id = $1
SELECT * | by_idRe: Super-Structured Data: Rethinking the Schema
#15The words "anarchy" and "authoritarianism" seem unnecessarily emotional and pejorative, and because of their semantic baggage I personally wouldn't use them in a professional situation. The author counts on the emotional color of those words to attempt an argument that both are somehow bad.
Instead of those words I'd suggest something like "schema on write" vs. "schema on read", or "persisted structured" vs. "persisted unstructured". "Document" vs. "relational" doesn't quite capture it, since unstructured data can have late-binding relations applied at read time, and structured data doesn't have to be relational.
And of course, modern relational databases can store unstructured data as easily as structured data.
Re: Super-Structured Data: Rethinking the Schema
#16Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…
Author here. All good points. Yes, you can build a super-structured type system on top of tables. EdgeDB does this well. And you can put JSON into relational columns. Then you might ask what the "type" of that column is? Well, if you want deep types, the row type varies from column to column as the JSON values vary and you have to walk the JSON to determine the type. SQL implementation are beginning to try to do deal with this mess by adding layers on top of tables. We're saying, maybe we should think differently about the problem and build tables on top of types as a special case of a type system. This also gives a very nice way to get data into and out of systems without having to go through the messiness of ODBC and special casing tables vs tuples vs scalars etc.
Re: Super-Structured Data: Rethinking the Schema
#17Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…
Author here. All good points. Yes, you can build a super-structured type system on top of tables. EdgeDB does this well. And you can put JSON into relational columns. Then you might ask what the "type" of that column is? Well, if you want deep types, the row type varies from column to column as the JSON values vary and you have to walk the JSON to determine the type. SQL implementation are beginning to try to do deal…
This is true and is a limitation of SQL (not of the relational model per-se), and also is part of the problem that SQL is not composable (so you don't have a way to nested table definitions)
Re: Super-Structured Data: Rethinking the Schema
#18Earlier quoted context omitted.
It helps if this machinery can reject data and thus perform validation. Since recursive construction of union types (valid records can look like this, or also like that...) is trivial, a programmer somewhere has to draw the line between "loosen the schema to allow this record" and "reject this record to enforce the schema".
Author here. Agreed! Validation is important. While I didn't make this point in the article, our thinking is schema validation does not require that the serialization format utilize schemas as the building block and you can always implementation schema (or type) validation (and versioning) on top of super-structured data (as can also be done with document databases).
this is a major hassle when converting from avro (from kafka which uses a schema registry, so schemas are not shipped with the avro data) and storing in parquet which requires a schema in the file but you can 'upgrade' it with another schema when reading it. It would be great to have a binary protocol-like format (schema-less avro), and a schema-less columnar storage format.. which is I guess is what these guys are doing.
Re: Super-Structured Data: Rethinking the Schema
#19The first few sections of this post nearly lost me, waffling on about NoSQL vs whatever.
Eventually we get to the meat:
> For example, the JSON value
{"s":"foo","a":[1,"bar"]}
> type record with field s of type string and field a of type array of type union of types integer and string
> We call the former style of typing a “shallow” type system and the latter style of typing a “deep” type system. The hierarchy of a shallow-typed value must be traversed to determine its structure whereas the structure of a deeply-typed value is determined directly from its type.
This is a bit confusing, since JSON data commonly has an implicit schema, or "deep type system" as this post calls it, and if you consume data in any statically-typed language you will materialise the implicit "deep" types in your host language.
So it seems that ZSON is sort of like a TypeScript-ified version of JSON, where the implicit types are made explicit.
It seems the point is not to have an external schema that documents must comply to, so I guess at the end of the day has similar aim to other "self-describing" message formats like https://amzn.github.io/ion-docs/ ? i.e. each message has its own schema
So the interesting part is perhaps the new data tools to work with large collections of self-describing messages?
Re: Super-Structured Data: Rethinking the Schema
#20Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…
Author here. All good points. Yes, you can build a super-structured type system on top of tables. EdgeDB does this well. And you can put JSON into relational columns. Then you might ask what the "type" of that column is? Well, if you want deep types, the row type varies from column to column as the JSON values vary and you have to walk the JSON to determine the type. SQL implementation are beginning to try to do deal…
Normalize to the max then denormalize till you achieve the performance trade-offs you want. That's the rule in relational schema design.
Adding JSON traversal operators and functions helps a lot when you end up denormalizing bits of the schema. It's not hard.