Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Super-Structured Data: Rethinking the Schema

brimdata.io

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Super-Structured Data: Rethinking the Schema

#21

Post by vosper »

mccanne wrote: post #16
mamcx wrote: post #9

Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…

Author here. All good points. Yes, you can build a super-structured type system on top of tables. EdgeDB does this well. And you can put JSON into relational columns. Then you might ask what the "type" of that column is? Well, if you want deep types, the row type varies from column to column as the JSON values vary and you have to walk the JSON to determine the type. SQL implementation are beginning to try to do deal…

You mentioned EdgeDB in the blog post, too, but I just think you and them are dealing with different problems.

My understanding of EdgeDB is they're mostly trying to make correct data-modeling simpler and more intuitive; to let people model relations in the same way they speak and think about it, rather than having to map to SQL concepts like join tables. I rather like what they're going for, though I haven't used it.

EdgeDB seems to be mostly for business logic and OLTP. They're not trying to deal with arbitrary incoming data that might be outside of the control of the ingestion system. You wouldn't even have an ingestion system with EdgeDB.

Re: Super-Structured Data: Rethinking the Schema

#22

Post by vosper »

anentropic wrote: post #19

The first few sections of this post nearly lost me, waffling on about NoSQL vs whatever. Eventually we get to the meat: > For example, the JSON value {"s":"foo","a":[1,"bar"]} > would traditionally be called “schema-less” and in fact is said have the vague type “object” in the world of JavaScript or “dict” in the world of Python. However, the super-structured interpretation of this value’s type is instead: > type rec…

> The first few sections of this post nearly lost me, waffling on about NoSQL vs whatever.

Since the author of the blog post is here, I'll just jump in to agree with this part: there is a lot of unecessary background text before we get to the meat of it. I don't think people need a history lesson on NoSQL and SQL, and IMO the "authoritarianism" metaphor is a stretch, and that word has pretty negative connotations.

I think there's some value in setting the scene, but I think you will lose readers before they get to the much more interesting content further down. I recommend revising it to be a lot shorter.

Re: Super-Structured Data: Rethinking the Schema

#23

Post by flappyeagle »

mamcx wrote: post #9

Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…

why hasn't someone built a composable flavor of SQL? it seems like a burning need

Re: Super-Structured Data: Rethinking the Schema

#24

Post by pjungwir »

flappyeagle wrote: post #23
mamcx wrote: post #9

Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…

why hasn't someone built a composable flavor of SQL? it seems like a burning need

This is what Tutorial D is, but it's never been widely adopted.

Re: Super-Structured Data: Rethinking the Schema

#25

Post by natemcintosh »

So it sounds like one of the advantages of the Zed ecosystem is that its data can go into three file formats (zson, zng, zst), each designed for a specific use case, and convert between them easily and without loss.

And it seems like the newer "zed lake" format is like a large blob managed by a server. Can you also convert data to and from and the file formats to the lake format? What is the lake's main use case?

Re: Super-Structured Data: Rethinking the Schema

#26

Post by thinkharderdev »

Arrow has union types (as well as structs and dictionary types). Parquet doesn't but I think it has an intentionally shallow types system to allow flexibility in encoding. Basically everything is either a numeric or binary and the logical type for binary columns is defined in metadata. So you can use, for instance, Arrow as the encoding.

Re: Super-Structured Data: Rethinking the Schema

#27

Post by munro »

I love the idea of getting rid of tables, when developing application code I'm often thinking in terms of Maps/Sets/Lists--I wish I could just take that code and make it persistent. PRIMARY KEY is really like a map. Also I wish I had transactional memory in my application. Not sure what the future looks like, but I am loving all this development in the database space.

Re: Super-Structured Data: Rethinking the Schema

#28

Post by kmerroll »

Interesting discussion, but buried in a lot of legacy thinking about schemas and personally, I don't find Yet-Another-Schema-Abstraction (YASA)™ layer very compelling when better solutions in functional programming and semantic ontologies are far ahead in this area.

Suggest looking into JSON-LD which was intended to solve many of the type and validation use-cases related to type and schema.

Re: Super-Structured Data: Rethinking the Schema

#29

Post by abraxaz »

kmerroll wrote: post #28

Interesting discussion, but buried in a lot of legacy thinking about schemas and personally, I don't find Yet-Another-Schema-Abstraction (YASA)™ layer very compelling when better solutions in functional programming and semantic ontologies are far ahead in this area. Suggest looking into JSON-LD which was intended to solve many of the type and validation use-cases related to type and schema.

To pile on a bit here, JSON-LD is based on RDF, which is an abstract syntax for data as semantic triples (i.e. RDF statements), there is also RDF* which is in development which extends this basic data model to make statements about statements.

RDF has concrete syntaxes, one of them being JSON-LD, and it can be used to model relational databases fairly well with R2RML (https://www.w3.org/TR/r2rml/) which essentially turns relation databases into a concrete syntax for RDF.

schema.org is also based on RDF, and is essentially an ontology (one of many) that can be used for RDF and non RDF data, but mainly because almost all data can be represented as RDF - so non RDF data is just data that does not have a formal mapping to RDF yet.

Ontologies is a concept used frequently in RDF but rarely outside of it, it is quite important for federated or distributed knowledge, or descriptions of entities. It focuses heavily on modelling properties instead of modelling objects, and then whenever a property occurs that property can be understood within the context of an ontology.

An example is the age of a person (https://schema.org/birthDate)

When I get a semantic triple:

https://schema.org/birthDate> "2000-01-01"^^https://schema.org/Date>

This tells me that the entity identified by the IRI is a person - and their birth date is 2000-01-01. I however don't expect that i will get all other descriptions of this person at the same time, I won't necessarily get their https://schema.org/nationality> for example, even though this is a property of a https://schema.org/Person> defined by schema.org

I can also combine https://schema.org/ based descriptions with other descriptions, and these descriptions can be merged from multiple sources and then queried together using SPARQL.

Re: Super-Structured Data: Rethinking the Schema

#30

Post by ducharmdev »

kmerroll wrote: post #28

Interesting discussion, but buried in a lot of legacy thinking about schemas and personally, I don't find Yet-Another-Schema-Abstraction (YASA)™ layer very compelling when better solutions in functional programming and semantic ontologies are far ahead in this area. Suggest looking into JSON-LD which was intended to solve many of the type and validation use-cases related to type and schema.

You're right, I think Yet-Another-Schema-Solution (YASS)™ would be much more compelling!

(Please forgive me)

Post reply on HN