Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

31–40 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#31

You can also use Git for data! It’s a bit slower, but smart use of partial/shallow clones can address performance degradation on large repositories over time. You just need to take care of the transformation between “physical” trees/blobs and “logical” objects in your dataset (which may not have 1:1 mapping, as having physical layer more granular reduces likelihood of merge conflicts). I’m also following Pijul, which…

The fact that I can use git for data if I carefully avoid all the footguns is exactly why I don’t use git for data.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#32
I never understood why we don’t have SQL databases that track all changes in a “third dimension” (column being one dimension, row being the second dimension).

It might be a bit slower to write, but hook the logic in to write/delete, and suddenly you can see exactly when a field was changed to break everything. The right middleware and you could see the user, IP, and query that changed it (along with any other queries before or after).

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#33
post #16

I collected all the git for data open source projects I could find a few months back, there have been a bunch of interesting approaches https://docs.google.com/spreadsheets/d/1jGQY_wjj7dYVne6toyzm...

I've had a fairly long-term side project working on git for chronological data (data is a cause and effect DAG), know of anybody doing that?

It might not be exactly what you are looking for, but git-bug[1] is encoding data into regular git objects, with merges and conflict resolution. I'm mentioning this because the hard part is providing an ordering of events. Once you have that you can store and recreate whatever state you want.

This branch[2] I'm almost done with remove the purely linear branch constraint and allow to use full DAGs (that is, concurrent edition) and still provide a good ordering.

[1]: https://github.com/MichaelMure/git-bug [2]: https://github.com/MichaelMure/git-bug/pull/532

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#34
post #5

Problematic name, could become a millstone on the neck of the developer far into the future.

Already I would not use this project because of its name. I’m not offended by it, but I know others will be, and it will only be a matter of time before we have to replace it with something else. So why bother in the first place?

I know the name is not DOLT but it is close enough to cause offense. Imagine the N-word with one typo. Would it still be offensive? Probably to some.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#35

I never understood why we don’t have SQL databases that track all changes in a “third dimension” (column being one dimension, row being the second dimension). It might be a bit slower to write, but hook the logic in to write/delete, and suddenly you can see exactly when a field was changed to break everything. The right middleware and you could see the user, IP, and query that changed it (along with any other queries…

Because you can do that with after update triggers or server-side in software.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#36

You can also use Git for data! It’s a bit slower, but smart use of partial/shallow clones can address performance degradation on large repositories over time. You just need to take care of the transformation between “physical” trees/blobs and “logical” objects in your dataset (which may not have 1:1 mapping, as having physical layer more granular reduces likelihood of merge conflicts). I’m also following Pijul, which…

Git is too complicated. It's barely usable for daily tasks. Look at how many people have to Google for basic things like uncommitting a commit, or cleaning your local repo to mirror a remote one. Complexity is a liability. Mercurial has a nicer interface. And now I see the real simplicity of non-distributed source control systems. I have never actually needed to work in a distributed manner, just client-server. I have never sent a patch to another dev to patch into their local repo or whatnot. All this complexity seems like a solution chasing after a problem - at least for most developers. What works for Linux isn't necessary for most teams.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#37
post #15

It's amazing this isn't a standard feature. The database world seems to have focused on large, high volume, globally distributed databases. Presumably you would't version clickstream or IoT sensor data. Features like this that are only feasible below a certain scale are underdeveloped and I think there's opportunity there.

Isnt the mysql log journal* what you are looking for?

* I dont remember the exact name but I refer the feature that is used to replicate actions if there was an error.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#38
What people usually miss about these things is normal version control benefits hugely from content addressing and normal forms.

The salient aspect of relational data is that it's cyclic, this makes content addressing unable to provide normal forms on it's own (unless someone figures out how to Merkle cylic graphs!), but the normal form can still made other ways.

The first part is easier enough, store rows in some order.

The second part is more interesting: making the choice of surrogate keys not matter (quotienting it away). Sorting table rows containing surrogate keys depending on the sorting of table rows makes for some interesting bags of constraints, for which there may be more than one fixed point.

Example:

  CREATE TABLE Foo (
    a uuid PRIMARY KEY,
    b text,
    best_friend uuid REFERENCES Foo(b)
  );
DB 0:

  0 Alice 0
1 reclusive Alice, best friends with herself. Just fine.

  0 Alice 1
  1 Alice 1
2 reclusive Alices, both best friends with the second one. The alices are the same up to primary keys, but while primary keys are to be quotiented out, primary key equality isn't, so this is valid. And we have an asymmetry by which to sort.

  0 Alice 1
  1 Alice 0
2 reclusive Alices, each best friends with the other. The Alices are completely isomorphic, and one notion of normal forms would say this is exactly the same as DB 0: as if this is reclusive Alice in a fun house of mirrors.

All this is resolvable, but it's subtle. And there's no avoiding complexity. E.g. if one wants to cross reference two human data entries who each assigned their own surrogate IDs, this type of analysis must be done. Likewise when merging forks of a database.

I'd love be wrong, but I don't think any of the folks doing "git for data" are planning their technology with this level of mathematical rigor.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#40

I would rather have a "GitHub for data" - an SQL database I could have hosted for free, give everybody read-only access to and give some people I choose R/W access to. That's a thing I miss really.

Dolt also have a paid product called DoltHub, check it out.
Post reply on HN