Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Super-Structured Data: Rethinking the Schema

brimdata.io

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Super-Structured Data: Rethinking the Schema

#31

Post by abraxaz »

ccleve wrote: post #3

tldr; Don't use relational tables or unstructured document databases. Instead use structured types. The "schema" here is ultimately a collection of independent objects / classes with well-defined fields. Ok, fine. But I'm not sure how this helps if you have six different systems with six different definitions of a customer, and more importantly, different relationships between customers and other objects like orders…

> Ok, fine. But I'm not sure how this helps if you have six different systems with six different definitions of a customer, and more importantly, different relationships between customers and other objects like orders or transactions or locations or communications.

If you have this problem, consider giving RDF a look - you can fairly easily use RDF based technologies to map the data in these systems onto a common model, some examples of tools that may be useful here is https://www.w3.org/TR/r2rml/ and https://github.com/ontop/ontop - you can also use JSON-LD to convert most JSON data to RDF. For more info ask in https://gitter.im/linkeddata/chat

Re: Super-Structured Data: Rethinking the Schema

#32

Post by vlmutolo »

kmerroll wrote: post #28

Interesting discussion, but buried in a lot of legacy thinking about schemas and personally, I don't find Yet-Another-Schema-Abstraction (YASA)™ layer very compelling when better solutions in functional programming and semantic ontologies are far ahead in this area. Suggest looking into JSON-LD which was intended to solve many of the type and validation use-cases related to type and schema.

What are the "better solutions in functional programming and semantic ontologies"? What would I Google?

Re: Super-Structured Data: Rethinking the Schema

#33

Post by simonw »

> The idea here is that instead of manually creating schemas, what if the schemas were automatically created for you? When something doesn’t fit in a table, how about automatically adding columns for the missing fields?

I've been experimenting with this approach against SQLite for a few years now, and I really like it.

My sqlite-utils package does exactly this. Try running this on the command line:

    brew install sqlite-utils
    echo '[
      {"id": 1, "name": "Cleo"},
      {"id": 2, "name": "Azy", "age": 1.5}
    ]' | sqlite-utils insert /tmp/demo.db creatures - --pk id
    sqlite-utils schema /tmp/demo.db
It outputs the generated schema:

    CREATE TABLE [creatures] (
       [id] INTEGER PRIMARY KEY,
       [name] TEXT,
       [age] FLOAT
    );
When you insert more data you can use the --alter flag to have it automatically create any missing columns.

Full documentation here: https://sqlite-utils.datasette.io/en/stable/cli.html#inserti...

It's also available as a Python library: https://sqlite-utils.datasette.io/en/stable/python-api.html

Re: Super-Structured Data: Rethinking the Schema

#35

Post by feoren »

Wow, what a waste of time. I've been doing it correctly for so long that I forget that virtually everyone else on the planet has no idea how to build a good data model. What pisses me off is that I actually have the right answer on how to avoid all of this pain, but if I typed it out here I'd either waste my time and get ignored or (much, much less likely) get my idea poached. It takes hours to fully communicate anyway. What do you do when you know you're sitting on an approach & tech that could revolutionize the X-hundred-billion-dollar data management industry but you can barely even get your own fucking employer to take you seriously?

Anyway this article is crap and gets everything wrong, just like all of you do. Whatever, nothing to see here I guess.

Re: Super-Structured Data: Rethinking the Schema

#36

Post by Kinrany »

thinkharderdev wrote: post #26

Arrow has union types (as well as structs and dictionary types). Parquet doesn't but I think it has an intentionally shallow types system to allow flexibility in encoding. Basically everything is either a numeric or binary and the logical type for binary columns is defined in metadata. So you can use, for instance, Arrow as the encoding.

Yes, the comparison with Arrow ecosystem should really be more in depth since that's the closest thing that exists.

Re: Super-Structured Data: Rethinking the Schema

#37

Post by mccanne »

cmollis wrote: post #18
mccanne wrote: post #8

Earlier quoted context omitted.

Author here. Agreed! Validation is important. While I didn't make this point in the article, our thinking is schema validation does not require that the serialization format utilize schemas as the building block and you can always implementation schema (or type) validation (and versioning) on top of super-structured data (as can also be done with document databases).

this is a major hassle when converting from avro (from kafka which uses a schema registry, so schemas are not shipped with the avro data) and storing in parquet which requires a schema in the file but you can 'upgrade' it with another schema when reading it. It would be great to have a binary protocol-like format (schema-less avro), and a schema-less columnar storage format.. which is I guess is what these guys are d…

Hear, hear!

Re: Super-Structured Data: Rethinking the Schema

#38

Post by mccanne »

simonw wrote: post #33

> The idea here is that instead of manually creating schemas, what if the schemas were automatically created for you? When something doesn’t fit in a table, how about automatically adding columns for the missing fields? I've been experimenting with this approach against SQLite for a few years now, and I really like it. My sqlite-utils package does exactly this. Try running this on the command line: brew install sqlit…

Simon, brilliant observations here and kudos on sqlite-utils.

I'm all for layers, a fundamental approach in our field to tame complexity. And the SQL model and SQLite have stood the test of time and are solid foundations.

I'm just wondering could we be stuck in a local maximum where the presumed answer is always the relational model? Maybe if we built the relational model on top of a different set of lower-level primitives (a type system instead of schemas and tables) we could escape local maximum we're stuck in? Just a thought.

There are a few somewhat ad hoc perf measurements here regarding the sqlite-utils and sqlite... https://zed.brimdata.io/docs/commands/zq/#73-performance-com...

I'm not a SQLite expert so if I did something wrong, please holler and let me know :)

Re: Super-Structured Data: Rethinking the Schema

#39

Post by abraxaz »

vlmutolo wrote: post #32
kmerroll wrote: post #28

Interesting discussion, but buried in a lot of legacy thinking about schemas and personally, I don't find Yet-Another-Schema-Abstraction (YASA)™ layer very compelling when better solutions in functional programming and semantic ontologies are far ahead in this area. Suggest looking into JSON-LD which was intended to solve many of the type and validation use-cases related to type and schema.

What are the "better solutions in functional programming and semantic ontologies"? What would I Google?

A place to start looking may be the OWL primer (https://www.w3.org/TR/owl2-primer/) and the RDF primer (https://www.w3.org/TR/rdf11-primer/)

Other resources: https://github.com/semantalytics/awesome-semantic-web

Re: Super-Structured Data: Rethinking the Schema

#40

Post by rco8786 »

vosper wrote: post #22
anentropic wrote: post #19

The first few sections of this post nearly lost me, waffling on about NoSQL vs whatever. Eventually we get to the meat: > For example, the JSON value {"s":"foo","a":[1,"bar"]} > would traditionally be called “schema-less” and in fact is said have the vague type “object” in the world of JavaScript or “dict” in the world of Python. However, the super-structured interpretation of this value’s type is instead: > type rec…

> The first few sections of this post nearly lost me, waffling on about NoSQL vs whatever. Since the author of the blog post is here, I'll just jump in to agree with this part: there is a lot of unecessary background text before we get to the meat of it. I don't think people need a history lesson on NoSQL and SQL, and IMO the "authoritarianism" metaphor is a stretch, and that word has pretty negative connotations. I…

+1 the second 60% of the article is incredibly interesting, but I almost gave up before I got to the meat.
Post reply on HN