Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

111–120 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#111

This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…

This won't work for usual database usecases. This is meant for interactive work with data same way you work with code. Who needs that?

Data scientists working with large datasets. You want to be able to update data without redownloading everything. Also make your local changes (some data cleaning) and propose your updates upstream same way you would with git. Having many people working interactively with data is common here.

One of the companies I work with provided set of data distributed to their partners on a daily basis. Once it grew larger, downloading everything daily became an issue. So that would be desirable,

I have large data model that I need to deploy to production and update once in a while. For code, network usage is kept to minimum because we have git. For data, options are limited.

As with git, it is something that once you have, you will find a lot of usecases that make life easier and open many new doors.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#112
post #37
post #15

It's amazing this isn't a standard feature. The database world seems to have focused on large, high volume, globally distributed databases. Presumably you would't version clickstream or IoT sensor data. Features like this that are only feasible below a certain scale are underdeveloped and I think there's opportunity there.

Isnt the mysql log journal* what you are looking for? * I dont remember the exact name but I refer the feature that is used to replicate actions if there was an error.

No. Relational DB logs are kept for short time, and do not allow for branching/merging. And even if you would store full log since day 1, the only way to check state of database at some point would be to start from scratch and reply everything to desired point. For any sizeable db, that's not practical.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#113

Is it for versions of the database design or versions of the data?

Both. Schema changes are versioned like everything else. But depending on what the change is, it might make merges difficult. (I haven’t used it; I just read the blog.)

BTW. I wish all databases versioned their schema and kept full history. This should be a standard feature.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#114

Earlier quoted context omitted.

This is the dataset on Kaggle - https://www.kaggle.com/simiotic/github-code-snippets

Yeah, that sized database is likely to be a challenge unless the computer system it's running on has scads of memory. One of my projects (DBHub.io) is putting effort towards working through the problem with larger sized SQLite databases (~10GB), and that's mainly through using bare metal hosts with lots of memory. eg 64GB, 128GB, etc. Putting the same data into PostgreSQL, or even MySQL, would likely be much more eff…

Can't beat SQLite for distribution as a public dataset, though.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#115

Earlier quoted context omitted.

This is the dataset on Kaggle - https://www.kaggle.com/simiotic/github-code-snippets

Yeah, that sized database is likely to be a challenge unless the computer system it's running on has scads of memory. One of my projects (DBHub.io) is putting effort towards working through the problem with larger sized SQLite databases (~10GB), and that's mainly through using bare metal hosts with lots of memory. eg 64GB, 128GB, etc. Putting the same data into PostgreSQL, or even MySQL, would likely be much more eff…

We have 200 GB databases in dolt format that are totally queryable. They don't work well querying on the web though - you need a local copy to query it effectively. Making web query as fast as local is an ongoing project.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#116

Earlier quoted context omitted.

Nope, been there done that, no thanks. Lack of locking for binary files, overhead > 1gb and all the shenanigans you need to do for proxy servers. There's better solutions out there but they aren't free.

Would be very curious to hear more about issues with proxy servers (where were they required?), overheads (do you mean RAM usage?) and locking.

Sure keep in mind that my data is a little old but last time I peeked into the git LFS space it seemed like there were still a few gaps.

First, most of my background in this area comes from gamedev so YMMV if the same applies in your use cases.

For our usage we'd usually have a repo history size that crossed the 1TB mark and even upwards of 2-3TB in some cases. The developer sync was 150-200GB, the art sync was closer to 500-600GB and the teams were regularly churning through 50-100GB/week depending on where we were in production.

You need discipline specific views into the repo. It just speeds everything up and means that only the teams that need to take the pain have to. From a performance perspective Perforce blows the pants off anything else I've seen, SVN tries, but P4 was easily an order of magnitude faster to sync or do a clean fetch.

I've seen proxy servers done with git but it's usually some really hack thing scripted together with a ton of ductape and client-specific host overrides. When you have a team split across East Coast/West Coast(or other country) you need that proxy so that history is cached in a way that it only gets pulled in locally once. Having a split push/pull model is asking for trouble and last I checked it wasn't clear to me if stuff like git LFS actually handles locking cleanly across it.

From an overhead perspective git just falls over at ~1gb(hence git LFS, which I've seen teams use to varying degrees of success based on project size). The need to do shallow history and sidestep resolving deltas is a ton of complexity that isn't adding anything.

With a lot of assets, merging just doesn't exist and a DVCS totally falls over here. I've seen fights nearly break out in the hallway multiple times when two artist/animators both forgot to checkout a file(usually because someone missed the metadata to say it's an exclusive access file). With unmergeable binary files that don't get locked your choice is who gets to drop 1-3 days of work on the floor when the other person blows away their changes to commit. If those changes span multiple interconnected packages/formats/etc you have a hard fork that you can never bring back together.

There's a couple other details but that's the large ones, Perforce worked incredibly well in this space but it is not cheap and so I've seen teams try to go their own way to mixed success. I'll admit that you can't do a monorepo in P4(and even tools like repo in the Android world have their problems too) but if you segregate your large business/product lines across P4 repos it scales surprisingly well.

Anyway, you may or may not hit any or all of this but I've yet to see git tackle a 1TB+ repo history well(and things like repo that uses many mini-repos doesn't count in my book due to the lack of atomicity across submissions that span multiple repos).

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#117
post #108

I'm not sure if this is supposed to be Dolt or DoIt, but using a swear word for a name (even a relatively mild one) is pretty distracting, IMHO.

presumably a riff on git, the well known famously unsuccessful version control system

Huh, I had no idea. (I'm not a native speaker.)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#118
post #99

I love the idea of this project so much. Being able to more easily share comprehensible data is an interest of mine. It is not the first time I have seen immutable B-trees being used as a method for being able to query a dataset on a different point in time. Spanner (and its derivatives) uses a similar technique to ensure backup consistency. Solutions such as CockroachDB also allows you to query data in the past [1],…

Albeit, now it makes me wonder how much of the functionality of dolt is possible to be replicated with CockroachDB. The internal data structures of both databases are mostly similar.

You can have infinite point-time-querying by setting the TTL of data to forever.

You have the ability to do distributed BACKUP/IMPORT of data (Which is mostly a file copy, and also includes historical data)

A transaction would be the equivalent of a commit, but I do not think there is a way to list out all transactions on CRDB, so that would have to be done separately.

And gain other benefits, such as distributed querying, and high availability.

I just find interesting that both databases (CockroachDB and Dolt) share the same principal of immutable B-Trees.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#119

Earlier quoted context omitted.

Yeah, that sized database is likely to be a challenge unless the computer system it's running on has scads of memory. One of my projects (DBHub.io) is putting effort towards working through the problem with larger sized SQLite databases (~10GB), and that's mainly through using bare metal hosts with lots of memory. eg 64GB, 128GB, etc. Putting the same data into PostgreSQL, or even MySQL, would likely be much more eff…

Can't beat SQLite for distribution as a public dataset, though.

We think dolt can :)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#120

Earlier quoted context omitted.

Already I would not use this project because of its name. I’m not offended by it, but I know others will be, and it will only be a matter of time before we have to replace it with something else. So why bother in the first place? I know the name is not DOLT but it is close enough to cause offense. Imagine the N-word with one typo. Would it still be offensive? Probably to some.

> but I know others will be, and it will only be a matter of time before we have to replace it with something else Or we can just not give in to such insanity. That's always an option, and would help prevent things from getting increasingly worse as we cede ground to claims that increasingly get further and further away from the realm of what's reasonable.

It’s not your choice when you’re an employee of a woke company (unless you want to quit) Don’t you know that by now?
Post reply on HN