Live data from Hacker News

Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

github.com

81–90 of 185 posts

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#81

This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…

I am not associated to Dolt, but I really like the idea of Dolt personally. I do see use cases, but not without challenges.

One of the main use cases you can see them targeting, and that I think makes a ton of sense, is providing tools for collecting, maintaining and publishing reference data sets using crowd sourcing.

For example, they are doing this with hospital charge codes (a.k.a. chargemaster data). Hospitals in the US are required to publish this data for transparency.. however, I have never seen a single aggregated national (or international) data set of all these charges. In fact, such a data set could be worth a lot of money to a lot of organizations for so many reasons. I used to work in health insurance, gathering data from all kinds of sources (government rules/regs, etc.) and it was a lot of hard work, scraping, structuring, maintaining, etc.

This reference data can be used for analytics, to power table-driven business logic, machine learning - to help identify cost inequalities, efficiencies, maybe even illicit price gouging, etc. There are so many reference data sets that have similar characteristics... and "data marketplaces" in a way are targeted at making "private" reference data sets available for sale - so then where is the "open" data marketplace? Well, here you go.. Dolt.

I have often realized that the more ways we can make things collaborative, the better off we will be.

Data is one of those things where, coming up with common, public reference datasets is difficult and there are lots of different perspectives ("branches"), sometimes your data set is missing something and it would be cool if someone could propose it ("pull request"), sometimes you want to compare the old and new version of a dataset ("diff") to see what is different.

One difficult thing about Dolt is, it will only be successful if people are actually willing to work together to cook up and maintain common data sets collaboratively, or if those doing so have an incentive to manage an "open data" project on Dolt as benevolent maintainers, for example. But, I could say then it has the same challenges as "open source" in general, so therefore it is not really that different.

Dolt could even be used as a foundation for a master data management registry - in the sense of you could pop it in as a "communal data fountain" if you will where anybody in your org, or on the Web, etc. could contribute - and you can have benevolent maintainers look over the general quality of the data. Dolt would be missing the data quality/fuzzy matching aspect that master data tools offer, but this is a start for sure.

For example, I work in a giant corporation right now. Each department prefers to maintain its own domain data and in some cases duplicates common data in that domain. Imagine using Dolt to make it possible for all these different domains to collaborate on a single copy of common data in a "Dolt" data set - now people can share data on a single copy and use pull requests, etc. to have an orderly debate on what that common data schema and data set should look like.

I think it's an idea that is very timely.

P.S. Dolt maintainers, if you read this and want to talk, I'm game! Awesome work :)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#82
post #73

Earlier quoted context omitted.

Or more simply put, how about table-driven logic in general? It doesn't have to be as complex as machine learning. There are more use cases than just machine learning, IMHO.

Such as? I'm having difficulty coming up with any myself.

Say, network configuration.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#83
Some related past threads:

Dolt is Git for data - https://news.ycombinator.com/item?id=22731928 - March 2020 (191 comments)

Git for Data – A TerminusDB Technical Paper [pdf] - https://news.ycombinator.com/item?id=22045801 - Jan 2020 (5 comments)

Ask HN: Would you use a “git for data”? - https://news.ycombinator.com/item?id=11537934 - April 2016 (10 comments)

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#85
post #73

Earlier quoted context omitted.

Or more simply put, how about table-driven logic in general? It doesn't have to be as complex as machine learning. There are more use cases than just machine learning, IMHO.

Such as? I'm having difficulty coming up with any myself.

See my post earlier in this thread [1].

Yes you need reference data for machine learning, but the world isn't only about machine learning. You might want reference data for human-interpreted analytics, table-driven logic (business rule engines, for example), etc.

[1] https://news.ycombinator.com/item?id=26371748.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#86

This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…

I am not associated to Dolt, but I really like the idea of Dolt personally. I do see use cases, but not without challenges. One of the main use cases you can see them targeting, and that I think makes a ton of sense, is providing tools for collecting, maintaining and publishing reference data sets using crowd sourcing. For example, they are doing this with hospital charge codes (a.k.a. chargemaster data). Hospitals i…

We totally agree on all points! Come chat with us on our discord about how we can help your org solve its problems:

https://discord.com/invite/RFwfYpu

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#87

This is absolutely fascinating, conceptually. However, I'm struggling to figure out a real-world use case for this. I'd love if anyone here can enlighten me. I don't see how it can be for production databases involving lots of users, because while it seems appealing as a way to upgrade and then roll back, you'd lose all the new data inserted in the meantime. When you roll back, you generally want to roll back changes…

I might look at it for work. Compliance requires us to keep track of who made what change when, and who it was approved by in case regulators need it.

Right now, this often means making an MR on Git with your snippet of SQL, getting it approved, and then manually executing it. This would let us bring the apply stage in, as well as avoid "that SQL query didn't do exactly what I expected" issues.

It's possible to do it within the SQL engine, but then I have to maintain that, which I would prefer not to do. As well as dealing with performance implications from that.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#88

Dolt might be good but never underestimate the power of Type 2 Slowly Changing Dimension tables [1]. For example, if you had an SSIS package that took CSV and imported them into a database, and one day you noticed it accidently rounded the value incorrectly, you could fix the data and retain traceability of the data which was there originally. E.g., SSIS package writes row of data: https://imgur.com/DClXAi5 Then a fe…

If all you want is AS OF semantics, then SCD2 is a great match. Used it a ton in application development myself.

Dolt actually makes branch and merge possible, totally different beast.

Re: Dolt is Git for Data: a SQL database that you can fork, clone, branch, merge

#89
post #84

Scanned for this comment before making it. This optically reads as "Dolt", not "DoIt"

Just optically? It's meant to be "dolt", not "do it", optically and semantically.

It's a pun on git, which also means "stupid/unpleasant/etc person".

Post reply on HN