Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

Post reply on HN
Print view View original

71–80 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#71

Post by ghusbands »

Ericson2314 wrote: post #38

What people usually miss about these things is normal version control benefits hugely from content addressing and normal forms. The salient aspect of relational data is that it's cyclic, this makes content addressing unable to provide normal forms on it's own (unless someone figures out how to Merkle cylic graphs!), but the normal form can still made other ways. The first part is easier enough, store rows in some ord…

> The salient aspect of relational data is that it's cyclic

This is an odd claim. Most relational data is not cyclic, and it's easy enough to come up with a scheme to handle cyclic data in a consistent fashion.

Conflicting changes (two changes to the same 'cell' of a database table) are a much more likely issue to hit and will need handling in much the same way merge conflicts are currently handled, so there are already situations in which manual effort will be needed.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#72

Post by Ericson2314 »

barrkel wrote: post #65
Ericson2314 wrote: post #38

What people usually miss about these things is normal version control benefits hugely from content addressing and normal forms. The salient aspect of relational data is that it's cyclic, this makes content addressing unable to provide normal forms on it's own (unless someone figures out how to Merkle cylic graphs!), but the normal form can still made other ways. The first part is easier enough, store rows in some ord…

Consider the whole database - the whole set of facts across all relations - as the state in the tree. Each transaction a serialized delta that produces a new node in the tree, a new HEAD. That's closer to what's being gotten at, as I see it. Transaction logs are already not that different to patch sets, and merge conflicts have an isomporphism with replication inconsistencies.

> Consider the whole database - the whole set of facts across all relations - as the state in the tree.

I tried to demonstrate that this is easier said than done. Deciding the equality/redundancy of facts is very subtle. At some it might even be guess whether your two clerks each met the same Alice when entering in their DB forks or not.

Transactions are just patches, I completely agree. And patches are just partial functions. But deciding what the action is of a partial function, or whether the input is in the domain requires a notion of equality. (Partial functions are encoded very nicely with incomplete pattern matching; in this case the question is whether Alice matches the pattern.)

Basically I know where you are coming from, and I want it to work too, but you cannot just wave away the math and issues it points out.)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#73

Post by jedberg »

sixdimensional wrote: post #70
jorgemf wrote: post #66

Earlier quoted context omitted.

Machine Learning. I don't think it has many more use cases

Or more simply put, how about table-driven logic in general? It doesn't have to be as complex as machine learning. There are more use cases than just machine learning, IMHO.

Such as? I'm having difficulty coming up with any myself.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#74

Post by simonw »

TedDoesntTalk wrote: post #34
Klwohu wrote: post #5

Problematic name, could become a millstone on the neck of the developer far into the future.

Already I would not use this project because of its name. I’m not offended by it, but I know others will be, and it will only be a matter of time before we have to replace it with something else. So why bother in the first place? I know the name is not DOLT but it is close enough to cause offense. Imagine the N-word with one typo. Would it still be offensive? Probably to some.

The name is DOLT with an L

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#75

Post by zachmu »

crazygringo wrote: post #60

This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…

The application backing use case is best suited for when you have parts of your database that get updated periodically and need human review. So you have a production database that you serve to your customers. Then you have a branch / fork of that (dev) that your development team adds batches of products to. Once a week you do a data release: submit a PR from dev -> prod, have somebody review all the new copy, and merge it once you're happy. If there's a big mistake, just back it out again. We have several paying customers building products around this workflow.

As for lots of people collaborating on data together, we have started a data bounties program where we pay volunteers to assemble large datasets. Two have completed so far, and a third is in progress. For the first one, we paid $25k to assemble precinct-level voting data for the 2016 and 2020 presidential elections. For the second, we paid $10k to get procedure prices for US hospitals. You can read about them here:

https://www.dolthub.com/blog/2021-02-15-election-bounty-revi...

https://www.dolthub.com/blog/2021-03-03-hpt-bounty-review/

What's cool is that novices can make a really good income from data entry as a side gig, and it's two orders of magnitude cheaper than hiring a firm to build data sets for you.

You're right that the site is kind of vague about what dolt is "for." It's a really general, very multi-purpose tool that we think will get used a lot of places. Here's a blog we wrote a while back about some of the use cases we envision.

https://www.dolthub.com/blog/2020-03-30-dolt-use-cases/

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#76

Post by zomglings »

If anyone from the dolt team is reading this, I'd like to make an enquiry:

At bugout.dev, we have an ongoing crawl of public GitHub. We just created a dataset of code snippets crawled from popular GitHub repositories, listed by language, license, github repo, and commit hash and are looking to release it publicly and keep it up-to-date with our GitHub crawl.

The dataset for a single crawl comes in at about 60GB. We uploaded the data to Kaggle because we thought it would be a good place for people to work with the data. Unfortunately, the Kaggle notebook experience is not tailored to such large datasets. Our dataset is in a SQLite database. It takes a long time for the dataset to load into Kaggle notebooks, and I don't think they are provisioned with SSDs as queries take a long time. Our best workaround to this is to partition into 3 datasets on Kaggle - train, eval, and development, but it will be a pain to manage this for every update, especially as we enrich the dataset with results of static analysis, etc.

I'd like to explore hosting the public dataset on Dolthub. If this sounds interesting to you please, reach out to me - email is in my HN profile.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#77

Post by joshspankit »

tthun wrote: post #47
joshspankit wrote: post #32

I never understood why we don’t have SQL databases that track all changes in a “third dimension” (column being one dimension, row being the second dimension). It might be a bit slower to write, but hook the logic in to write/delete, and suddenly you can see exactly when a field was changed to break everything. The right middleware and you could see the user, IP, and query that changed it (along with any other queries…

MS SQL server 2016 onwards has temporal tables that support this (point in time data)

Huh. I just read the spec. Not quite three 'dimension', but looks like exactly what I was asking for: a (reasonably) automatic and transparent record of previous values, as well as timestamps for when they changed.

I'll call this a "you learn something every day" and a "hey thanks @tthun (and @predakanga)"

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#78

Post by zomglings »

zomglings wrote: post #76

If anyone from the dolt team is reading this, I'd like to make an enquiry: At bugout.dev, we have an ongoing crawl of public GitHub. We just created a dataset of code snippets crawled from popular GitHub repositories, listed by language, license, github repo, and commit hash and are looking to release it publicly and keep it up-to-date with our GitHub crawl. The dataset for a single crawl comes in at about 60GB. We u…

This is the dataset on Kaggle - https://www.kaggle.com/simiotic/github-code-snippets

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#79

Post by resonantjacket5 »

einpoklum wrote: post #30

But do you really need this functionality, if you already have an SQL database? That is, you can: 1. Create a table with an extra changeset id column and a branch id column, so that you can keep historical values. 2. Have a view on that table with the latest version of each record on the master branch. 3. Express branching-related actions as actions on the main table with different record versions and branch names 4.…

That doesn't let you merge in someone else's changes easily. Aka two team members make changes to the (local copy of) database and now you want to merge it.

I mean sure you can have another database tracking their changes and a merging algorithm but that's what dolt is doing for you

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#80

Post by zachmu »

zomglings wrote: post #76

If anyone from the dolt team is reading this, I'd like to make an enquiry: At bugout.dev, we have an ongoing crawl of public GitHub. We just created a dataset of code snippets crawled from popular GitHub repositories, listed by language, license, github repo, and commit hash and are looking to release it publicly and keep it up-to-date with our GitHub crawl. The dataset for a single crawl comes in at about 60GB. We u…

We'll be in touch :)
Post reply on HN