Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

Post reply on HN
Print view View original

101–110 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#101

Post by skybrian »

scottmcdot wrote: post #55

Dolt might be good but never underestimate the power of Type 2 Slowly Changing Dimension tables [1]. For example, if you had an SSIS package that took CSV and imported them into a database, and one day you noticed it accidently rounded the value incorrectly, you could fix the data and retain traceability of the data which was there originally. E.g., SSIS package writes row of data: https://imgur.com/DClXAi5 Then a fe…

The commit log in Dolt is edit history. (When did someone import or edit the data? Who made the change?) It's not about when things happened.

To keep track of when things happened, you would still need date columns to handle it. But at least you don't need to handle two-dimensional history for auditing purposes. So, in your example, I think the "effective" date columns wouldn't be needed.

They have ways to query how the dataset appeared at some time in the past. However, with both data changes and schema changes being mixed together in a single commit log, I could see this being troublesome.

I suppose writing code to convert old edit history to the new schema would still be possible, similar to how git allows you to create a new branch by rewriting an existing one.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#103

Post by justinclift »

bargle0 wrote: post #96
jonnycomputer wrote: post #95

Naive question here. Aside from it being mysql, what is different here than just using git + sqlite. Update: When I posted, I'd forgotten that SQLite db file is a binary. Not sure what I was thinking.

Merging a SQLite database is challenging.

We do it on DBHub.io. :)

If you're into Go, these are the commits where the pieces were hooked together.

* https://github.com/sqlitebrowser/dbhub.io/commit/6c40e051ff7...

* https://github.com/sqlitebrowser/dbhub.io/commit/989ee0d08e6...

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#104

Post by mzs »

zachmu wrote: post #25
rapnie wrote: post #13

Earlier quoted context omitted.

I was going back and forth between the two until seeing doLt in terminal font.

This ambiguity in sans serif fonts has actually been pretty annoying. Especially since GitHub doesn't let you choose your font on readmes and stuff.

just lowercase it

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#105

Post by qbasic_forever »

zachmu wrote: post #49
qbasic_forever wrote: post #28

Earlier quoted context omitted.

Every DB engine used at scale has a concept of snapshots and backups. This just looks like someone making a git-like porcelain for the same kind of DB management constructs.

It's not just snapshots though. Dolt actually stores the tables, rows, and commits in a Merkle DAG, like Git. So you get branch and merge. You can't do branch and merge with snapshots. (You also get the full git / github toolbox: push and pull, fork and clone, rebase, and most other git commands)

Yeah it's a neat idea but I struggle to think of good use-cases for merge, other than toy datasets. If I'm working on a service that's sharding millions of users across dozens of DB instances a merge is going to be incomprehensible to understand and reason about conflicts.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#106

Post by unnouinceput »

bargle0 wrote: post #96
jonnycomputer wrote: post #95

Naive question here. Aside from it being mysql, what is different here than just using git + sqlite. Update: When I posted, I'd forgotten that SQLite db file is a binary. Not sure what I was thinking.

Merging a SQLite database is challenging.

I'd venture more and say merging any DB is challenging, SQLite or not.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#107

Post by mixedCase »

TedDoesntTalk wrote: post #34
Klwohu wrote: post #5

Problematic name, could become a millstone on the neck of the developer far into the future.

Already I would not use this project because of its name. I’m not offended by it, but I know others will be, and it will only be a matter of time before we have to replace it with something else. So why bother in the first place? I know the name is not DOLT but it is close enough to cause offense. Imagine the N-word with one typo. Would it still be offensive? Probably to some.

> but I know others will be, and it will only be a matter of time before we have to replace it with something else

Or we can just not give in to such insanity. That's always an option, and would help prevent things from getting increasingly worse as we cede ground to claims that increasingly get further and further away from the realm of what's reasonable.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#109

Post by drewwwwww »

1f60c wrote: post #108

I'm not sure if this is supposed to be Dolt or DoIt, but using a swear word for a name (even a relatively mild one) is pretty distracting, IMHO.

presumably a riff on git, the well known famously unsuccessful version control system

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#110

Post by rapjr9 »

strogonoff wrote: post #4

You can also use Git for data! It’s a bit slower, but smart use of partial/shallow clones can address performance degradation on large repositories over time. You just need to take care of the transformation between “physical” trees/blobs and “logical” objects in your dataset (which may not have 1:1 mapping, as having physical layer more granular reduces likelihood of merge conflicts). I’m also following Pijul, which…

We used git to store and manage data sets for a machine learning project involving chewing detection with audio data used in training. It was cumbersome and the huge datasets caused some problems with git (e.g., searches of our code base got really slow because the data was being searched also until we moved the data to a different repo). Something easier to use that could manage large datasets would be useful.

I wonder if DoIt could be used to create a clone of Apple's Time Machine. Seems like the basics are there.

Post reply on HN