Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

41–50 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#41

I never understood why we don’t have SQL databases that track all changes in a “third dimension” (column being one dimension, row being the second dimension). It might be a bit slower to write, but hook the logic in to write/delete, and suddenly you can see exactly when a field was changed to break everything. The right middleware and you could see the user, IP, and query that changed it (along with any other queries…

This has existed for a very long time as a data modeling strategy (most commonly, a "type 2 dimension") and is the way that all MVCC databases work under the covers. You don't need a special database to do this, just add another column to your database and populate it with a trigger or on update.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#43
post #36

You can also use Git for data! It’s a bit slower, but smart use of partial/shallow clones can address performance degradation on large repositories over time. You just need to take care of the transformation between “physical” trees/blobs and “logical” objects in your dataset (which may not have 1:1 mapping, as having physical layer more granular reduces likelihood of merge conflicts). I’m also following Pijul, which…

Git is too complicated. It's barely usable for daily tasks. Look at how many people have to Google for basic things like uncommitting a commit, or cleaning your local repo to mirror a remote one. Complexity is a liability. Mercurial has a nicer interface. And now I see the real simplicity of non-distributed source control systems. I have never actually needed to work in a distributed manner, just client-server. I hav…

Git is used prolifically in the tech industry. What on earth are you talking about?

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#45

What people usually miss about these things is normal version control benefits hugely from content addressing and normal forms. The salient aspect of relational data is that it's cyclic, this makes content addressing unable to provide normal forms on it's own (unless someone figures out how to Merkle cylic graphs!), but the normal form can still made other ways. The first part is easier enough, store rows in some ord…

A lot of your comment went over my head, but I have modeled relational data in a way conducive to being stored in a Merkle tree. The trick being that every entity in the system ended up having two IDs. A hash ID, identifying this specific version of the object, and an entity ID (probably a UUID or OID), which remained constant as new versions were added. In a situation where people can have friends that are also people, they are friends with the person long-term, not just a specific version of the friend, so in that case you'd use the entity IDs. Though you might also include a reference to the specific version of the person at the point in time at which they became friends, in which case they would necessarily reference an older version of that person. If you friend yourself, you're actually friending that person a moment ago.

A list of all current entities, by hash, is stored higher up the tree. Whether it's better that objects themselves store their entity ID or if that's a separate data structure mapping entity to hash IDs depends on the situation.

On second reading I guess your comment was actually about how to come up with content-based IDs for objects. I guess my point was that in the real world you don't usually need to do that, because if object identity besides its content is important you can just give it an arbitrary ID. How often does the problem of differentiating between a graph containing identical Alices vs one with a single self-friending Alice actually come up? Is there any way around it other than numbering the Alices?

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#46
post #43
post #36

Earlier quoted context omitted.

Git is too complicated. It's barely usable for daily tasks. Look at how many people have to Google for basic things like uncommitting a commit, or cleaning your local repo to mirror a remote one. Complexity is a liability. Mercurial has a nicer interface. And now I see the real simplicity of non-distributed source control systems. I have never actually needed to work in a distributed manner, just client-server. I hav…

Git is used prolifically in the tech industry. What on earth are you talking about?

Being needlessly complicated seldomly stops the tech industry from using something as long as the complexity is slightly out of the way.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#47

I never understood why we don’t have SQL databases that track all changes in a “third dimension” (column being one dimension, row being the second dimension). It might be a bit slower to write, but hook the logic in to write/delete, and suddenly you can see exactly when a field was changed to break everything. The right middleware and you could see the user, IP, and query that changed it (along with any other queries…

MS SQL server 2016 onwards has temporal tables that support this (point in time data)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#48

I never understood why we don’t have SQL databases that track all changes in a “third dimension” (column being one dimension, row being the second dimension). It might be a bit slower to write, but hook the logic in to write/delete, and suddenly you can see exactly when a field was changed to break everything. The right middleware and you could see the user, IP, and query that changed it (along with any other queries…

This does exist, though support for it is pretty sparse; it's called "Temporal Tables" in the SQL:2011 standard - https://sigmodrecord.org/publications/sigmodRecord/1209/pdfs...

Last time I checked, it was supported in SQL server and MariaDB, and Postgres via an extension.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#49
post #15

It's amazing this isn't a standard feature. The database world seems to have focused on large, high volume, globally distributed databases. Presumably you would't version clickstream or IoT sensor data. Features like this that are only feasible below a certain scale are underdeveloped and I think there's opportunity there.

Every DB engine used at scale has a concept of snapshots and backups. This just looks like someone making a git-like porcelain for the same kind of DB management constructs.

It's not just snapshots though.

Dolt actually stores the tables, rows, and commits in a Merkle DAG, like Git. So you get branch and merge. You can't do branch and merge with snapshots.

(You also get the full git / github toolbox: push and pull, fork and clone, rebase, and most other git commands)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#50
post #40

I would rather have a "GitHub for data" - an SQL database I could have hosted for free, give everybody read-only access to and give some people I choose R/W access to. That's a thing I miss really.

Dolt also have a paid product called DoltHub, check it out.

Free for public repositories, and for private repos under a gig.
Post reply on HN