Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

Post reply on HN
Print view View original

141–150 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#141

Post by stjohnswarts »

1f60c wrote: post #108

I'm not sure if this is supposed to be Dolt or DoIt, but using a swear word for a name (even a relatively mild one) is pretty distracting, IMHO.

DoIt isn't really a swear word. It's a euphemism for coitus. It's also a very common phrase in general so it really isn't strongly associated with "doing it" to cause any issues with English speakers. Aka it's fine as a project name. See Nike's "Just do it." advertisement campaign, they would have never gone with the phrase if it had strong negative connotations.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#142

Post by Aeolun »

zachmu wrote: post #61
Aeolun wrote: post #42

What does this look this look like when merging? Does it need a specialized tool? I’m a bit sad it doesn’t seem to have git style syncing though.

Merge is built in, same as with git. Same syntax too: dolt checkout -b dolt merge What do you mean git-style syncing? It has `push`, `pull`, and `fetch`.

In regards to merging, I understand that part, but say I have a conflict, how does it get resolved. Does it open some sort of text editor where I delete the unwanted lines?

I mean, if I have a repository on one PC, I can clone from there on a different PC.

As far as I can see this only allows S3/GC or Dolthub remotes.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#143

Post by px43 »

Is there a nice way to distribute a database across many systems?

Would this be useful for a data-set like Wikipedia?

It would be really nice if anyone could easily follow Wikipedia in such a way that anyone could fork it, and merge some changes, but not others, etc. This is something that really needs to happen for large, distributed data-sets.

I really want to see large corpuses of data that can have a ton of contributors sending pull requests to build out some great collaborative bastion of knowledge.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#145

Post by zachmu »

stjohnswarts wrote: post #141
1f60c wrote: post #108

I'm not sure if this is supposed to be Dolt or DoIt, but using a swear word for a name (even a relatively mild one) is pretty distracting, IMHO.

DoIt isn't really a swear word. It's a euphemism for coitus. It's also a very common phrase in general so it really isn't strongly associated with "doing it" to cause any issues with English speakers. Aka it's fine as a project name. See Nike's "Just do it." advertisement campaign, they would have never gone with the phrase if it had strong negative connotations.

It's DOLT. San serif fonts.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#146

Post by zachmu »

Aeolun wrote: post #142
zachmu wrote: post #61

Earlier quoted context omitted.

Merge is built in, same as with git. Same syntax too: dolt checkout -b dolt merge What do you mean git-style syncing? It has `push`, `pull`, and `fetch`.

In regards to merging, I understand that part, but say I have a conflict, how does it get resolved. Does it open some sort of text editor where I delete the unwanted lines? I mean, if I have a repository on one PC, I can clone from there on a different PC. As far as I can see this only allows S3/GC or Dolthub remotes.

Merge conflicts get put into a special conflicts table. You have to resolve them one by one just like text conflicts from a git merge. You can do this with the command line or with SQL.

It's true we don't support SSH remotes yet. Wouldn't be too hard to add though. File an issue if this is preventing you from using the product. Or use sshfs, which is a pretty simple workaround.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#147

Post by graderjs »

strogonoff wrote: post #4

You can also use Git for data! It’s a bit slower, but smart use of partial/shallow clones can address performance degradation on large repositories over time. You just need to take care of the transformation between “physical” trees/blobs and “logical” objects in your dataset (which may not have 1:1 mapping, as having physical layer more granular reduces likelihood of merge conflicts). I’m also following Pijul, which…

I find a balance between this using git on JSON files. And I build the JSON files into a database (1 file per record, 1 directory per table, subdirectories for indexes). The whole thing is pretty beautiful, and it's functioning well for a user-account, access management database I'm running in production. I like that I can go back and do:

`git diff -p` to see the users who have signed up recently, for example.

You can get the code, over at: https://github.com/i5ik/sirdb

The advantages of this approach are using existing unix tooling for text files, solid versioning, easy inspect-ability, and leveraging the filesystem B-Tree indexing as a fast index structure (rather than having to write my b-trees). Another advantage is hardware-linked scaling. For example, if I use regular hard disks, it's slower. But if I use SSDs it's faster. And i should also be possible to mount the DB as a RAM disk and make it super fast.

The disadvantages are that the database side still only supports a couple of operations (like exact, multikey searches, lookup by ID, and so on) rather than a rich query language. I'm OK with that for now, and I'm also thinking of using skiplists in future to get nice ordering property for the keys in an index so I can easily iterate and page over those.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#148

Post by 0xbadcafebee »

If this could somehow work with existing production MariaDB/PostgreSQL databases, this would be the next Docker. Sad that it requires its own proprietary DB (SQL compatible though it may be, it's not my existing database).

I wish something completely free like this existed just to manage database migrations. I don't think anything is quite this powerful/user-friendly (if you can call Git that) and free.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#149

Post by metaodi »

px43 wrote: post #143

Is there a nice way to distribute a database across many systems? Would this be useful for a data-set like Wikipedia? It would be really nice if anyone could easily follow Wikipedia in such a way that anyone could fork it, and merge some changes, but not others, etc. This is something that really needs to happen for large, distributed data-sets. I really want to see large corpuses of data that can have a ton of contr…

This is basically Wikidata [1], where the distribution is linked data/RDF with a triple store and federation to query different triple stores at once using SPARQL.

[1] https://wikidata.org

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#150

Post by 2malaq »

jimsmart wrote: post #125
kenniskrag wrote: post #58

Earlier quoted context omitted.

they have a theme preview now. :)

The parent post here is speaking to the content stored in the database, not the templates on the filesystem. Dolt enables one to merge, push and pull data , just as one would with files and git.

I read it as sarcasm.
Post reply on HN