Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

Post reply on HN
Print view View original

131–140 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#131

Post by Klwohu »

TedDoesntTalk wrote: post #34
Klwohu wrote: post #5

Problematic name, could become a millstone on the neck of the developer far into the future.

Already I would not use this project because of its name. I’m not offended by it, but I know others will be, and it will only be a matter of time before we have to replace it with something else. So why bother in the first place? I know the name is not DOLT but it is close enough to cause offense. Imagine the N-word with one typo. Would it still be offensive? Probably to some.

This is the issue with names. Even though the project is called doit the DoIt stylizing makes it look problematic. It's a non-starter, hopefully the author makes a big change. Just choosing lower case for the project would be enough.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#132

Post by [deleted] »

101008 wrote: post #37
jrumbut wrote: post #15

It's amazing this isn't a standard feature. The database world seems to have focused on large, high volume, globally distributed databases. Presumably you would't version clickstream or IoT sensor data. Features like this that are only feasible below a certain scale are underdeveloped and I think there's opportunity there.

Isnt the mysql log journal* what you are looking for? * I dont remember the exact name but I refer the feature that is used to replicate actions if there was an error.

[deleted]

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#134

Post by martincolorado »

curryst wrote: post #87
crazygringo wrote: post #60

This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…

I might look at it for work. Compliance requires us to keep track of who made what change when, and who it was approved by in case regulators need it. Right now, this often means making an MR on Git with your snippet of SQL, getting it approved, and then manually executing it. This would let us bring the apply stage in, as well as avoid "that SQL query didn't do exactly what I expected" issues. It's possible to do it…

I see a use-case for public resource data sets. For example, right of way, land use variance, permits, and land records. These are fundamentally public data sets that are often maintained by entities with limit budgets and the potential (even incentive) for fraud is substantial. Also, there is significant value in being able to access the data set for analytical purposes such as real estate analysis.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#135

Post by justinclift »

zachmu wrote: post #115
justinclift wrote: post #97

Earlier quoted context omitted.

Yeah, that sized database is likely to be a challenge unless the computer system it's running on has scads of memory. One of my projects (DBHub.io) is putting effort towards working through the problem with larger sized SQLite databases (~10GB), and that's mainly through using bare metal hosts with lots of memory. eg 64GB, 128GB, etc. Putting the same data into PostgreSQL, or even MySQL, would likely be much more eff…

We have 200 GB databases in dolt format that are totally queryable. They don't work well querying on the web though - you need a local copy to query it effectively. Making web query as fast as local is an ongoing project.

Yeah the "on the web" piece is the thing we're talking about. :)

200GB databases for PostgreSQL (etc) isn't any kind of amazing.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#137

Post by MyNameIsFred »

qbasic_forever wrote: post #105
zachmu wrote: post #49

Earlier quoted context omitted.

It's not just snapshots though. Dolt actually stores the tables, rows, and commits in a Merkle DAG, like Git. So you get branch and merge. You can't do branch and merge with snapshots. (You also get the full git / github toolbox: push and pull, fork and clone, rebase, and most other git commands)

Yeah it's a neat idea but I struggle to think of good use-cases for merge, other than toy datasets. If I'm working on a service that's sharding millions of users across dozens of DB instances a merge is going to be incomprehensible to understand and reason about conflicts.

> Yeah it's a neat idea but I struggle to think of good use-cases [...] If I'm working on a service [...]

I suspect that's simply not the use-case they're targeting. You're thinking of a database as simply the persistence component for your service, a means to an end. For you, the service/app/software is the thing you're trying to deliver. Where this looks useful is the cases where the data itself is the thing you're trying to deliver.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#138

Post by acidbaseextract »

justinclift wrote: post #136
zachmu wrote: post #119

Earlier quoted context omitted.

We think dolt can :)

How do you send someone a dolt database as a file?

How do you send someone a git repository as a file? Why would a tarball not work?

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#139

Post by 411111111111111 »

justinclift wrote: post #136
zachmu wrote: post #119

Earlier quoted context omitted.

We think dolt can :)

How do you send someone a dolt database as a file?

Probably by first pushing it into a file. This command is in the readme.

  dolt remote add  file:///Users/xyz/abs/path/

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#140

Post by vvanders »

strogonoff wrote: post #122
vvanders wrote: post #116

Earlier quoted context omitted.

Sure keep in mind that my data is a little old but last time I peeked into the git LFS space it seemed like there were still a few gaps. First, most of my background in this area comes from gamedev so YMMV if the same applies in your use cases. For our usage we'd usually have a repo history size that crossed the 1TB mark and even upwards of 2-3TB in some cases. The developer sync was 150-200GB, the art sync was close…

This is super informative! In my case it’s different since Git isn’t accessed by users directly, rather I’m working on some tools that work on top of Git (on user’s machine). Data is primarily text-based, though sometimes binary assets come up (options for offloading them out of Git are being investigated). So far there were no major issues. I predict degradation over time as repos grow in size and history (Git is no…

Yeah, my experience has been that you start seeing issues with long delta decompression times around the 1-2gb mark. That climbs quicker of you have binary formats that push the delta compression algorithm into cases where it does poorly(which makes sense since it was optimized for source code).

If you have binary assets and they don't support merging or regeneration from source artifacts that mandates locking(ideally built into SCM but I've seen wiki pages in a pinch at small scale).

Post reply on HN