Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Super-Structured Data: Rethinking the Schema

brimdata.io

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Super-Structured Data: Rethinking the Schema

#41

Post by rco8786 »

anentropic wrote: post #19

The first few sections of this post nearly lost me, waffling on about NoSQL vs whatever. Eventually we get to the meat: > For example, the JSON value {"s":"foo","a":[1,"bar"]} > would traditionally be called “schema-less” and in fact is said have the vague type “object” in the world of JavaScript or “dict” in the world of Python. However, the super-structured interpretation of this value’s type is instead: > type rec…

> This is a bit confusing, since JSON data commonly has an implicit schema, or "deep type system" as this post calls it, and if you consume data in any statically-typed language you will materialise the implicit "deep" types in your host language

That is an incredibly expensive operation to perform. Being able to look at two binary blobs of data and quickly determining whether or not they are the same type of data unlocks a whole host of functionality over large amounts of data that is otherwise prohibitively expensive and slow.

Re: Super-Structured Data: Rethinking the Schema

#42

Post by simonw »

mccanne wrote: post #38
simonw wrote: post #33

> The idea here is that instead of manually creating schemas, what if the schemas were automatically created for you? When something doesn’t fit in a table, how about automatically adding columns for the missing fields? I've been experimenting with this approach against SQLite for a few years now, and I really like it. My sqlite-utils package does exactly this. Try running this on the command line: brew install sqlit…

Simon, brilliant observations here and kudos on sqlite-utils. I'm all for layers, a fundamental approach in our field to tame complexity. And the SQL model and SQLite have stood the test of time and are solid foundations. I'm just wondering could we be stuck in a local maximum where the presumed answer is always the relational model? Maybe if we built the relational model on top of a different set of lower-level prim…

Cool, I didn't realize you used sqlite-utils for your performance demo!

It's not particularly designed for speed - it should be fast as far as Python code goes (I use some generator tricks to stream data and avoid having to load everything into memory at once) but I wouldn't expect "sqlite-utils insert" to win any performance competitions with tools written in other languages.

Those benchmarks against sqlite itself are definitely interesting. I'm looking forward to playing with the "native ZNG support for Python" mentioned on https://github.com/brimdata/zed/blob/main/docs/libraries/pyt... when that becomes available.

Re: Super-Structured Data: Rethinking the Schema

#43

Post by simonw »

simonw wrote: post #42
mccanne wrote: post #38

Earlier quoted context omitted.

Simon, brilliant observations here and kudos on sqlite-utils. I'm all for layers, a fundamental approach in our field to tame complexity. And the SQL model and SQLite have stood the test of time and are solid foundations. I'm just wondering could we be stuck in a local maximum where the presumed answer is always the relational model? Maybe if we built the relational model on top of a different set of lower-level prim…

Cool, I didn't realize you used sqlite-utils for your performance demo! It's not particularly designed for speed - it should be fast as far as Python code goes (I use some generator tricks to stream data and avoid having to load everything into memory at once) but I wouldn't expect "sqlite-utils insert" to win any performance competitions with tools written in other languages. Those benchmarks against sqlite itself a…

One trick I sometimes use is to pipe it through pv to get a progress bar:

    % pv conn.json | sqlite-utils insert conn.db conn - --nl
    36.9MiB 0:00:06 [5.81MiB/s] [==>             ]  9% ETA 0:01:00

Re: Super-Structured Data: Rethinking the Schema

#44

Post by simonw »

mamcx wrote: post #14
zmgsabst wrote: post #10

Earlier quoted context omitted.

I’m not sure what you mean by “composable” here — could you elaborate?

Composable is the ability to define things in the small and combine with confidence. SQL not allow this: by_id := WHERE id = $1 SELECT * | by_id

CTEs provide some pretty useful composability for SQL queries. I find myself using them all the time.

Re: Super-Structured Data: Rethinking the Schema

#45

Post by hobofan »

flappyeagle wrote: post #23
mamcx wrote: post #9

Note: The relational model (even SQL) is THIS. Despite the claims, SQL is NOT "schema-fixed". You can 100% create new schemas, alter them and modify them. What actual happens is that if you have a CENTRAL repository of data (aka "source of truth"), then you bet you wanna "freeze" your schemas (because is like a API, where you need to fulfill contracts). -- SQL have limitations in lack of composability, the biggest re…

why hasn't someone built a composable flavor of SQL? it seems like a burning need

I would say that in a way, that's what substrait[0] is trying to achieve.

[0]: https://substrait.io

Re: Super-Structured Data: Rethinking the Schema

#46

Post by miguelrochefort »

abraxaz wrote: post #29
kmerroll wrote: post #28

Interesting discussion, but buried in a lot of legacy thinking about schemas and personally, I don't find Yet-Another-Schema-Abstraction (YASA)™ layer very compelling when better solutions in functional programming and semantic ontologies are far ahead in this area. Suggest looking into JSON-LD which was intended to solve many of the type and validation use-cases related to type and schema.

To pile on a bit here, JSON-LD is based on RDF, which is an abstract syntax for data as semantic triples (i.e. RDF statements), there is also RDF* which is in development which extends this basic data model to make statements about statements. RDF has concrete syntaxes, one of them being JSON-LD, and it can be used to model relational databases fairly well with R2RML ( https://www.w3.org/TR/r2rml/ ) which essentially…

CBOR-based Serialization for Linked Data:

https://digitalbazaar.github.io/cbor-ld-spec/

RDF Binary Encoding using Thrift:

https://afs.github.io/rdf-thrift/rdf-binary-thrift.html

Re: Super-Structured Data: Rethinking the Schema

#47

Post by tabtab »

"Dynamic Relational" needs to be implemented. Columns and (optionally) tables are "create on write". If you issue "SELECT nonExistingColumn FROM myTable" you get nulls (if rows exist), not an error. One can incrementally "lock down" the schema as a project matures by adding constraints. Unlike the "NoSql" movement, it does not throw out most of RDBMS concepts, just tweaks them only enough to be dynamic-friendly. This reduces the learning curve.

Re: Super-Structured Data: Rethinking the Schema

#48

Post by zedlover »

natemcintosh wrote: post #25

So it sounds like one of the advantages of the Zed ecosystem is that its data can go into three file formats (zson, zng, zst), each designed for a specific use case, and convert between them easily and without loss. And it seems like the newer "zed lake" format is like a large blob managed by a server. Can you also convert data to and from and the file formats to the lake format? What is the lake's main use case?

It seems the zed lake is a data lake implementation rather than a format. It uses ZNG for both metadata and data storage internally I think.
Post reply on HN