As far as I can tell the only way to use this is to push everything into a mysql instance. def some pros and cons there.
Dolt is Git for data
31–40 of 196 posts
Re: Dolt is Git for data
#32> Dolt is the only database with branches. datomic has branching too afaik.
Amazon has an internal k/v store with branches (as well as schema versioning and hierarchical keys). It was primarily used for config management, which it excelled at.
Re: Dolt is Git for data
#33As far as I can tell the only way to use this is to push everything into a mysql instance. def some pros and cons there.
You can export to whatever you like. In our imagination, data versioning would sit upstream of production just like source code versioning. You take the data out and do what you need with it in a "compile" step.
Re: Dolt is Git for data
#34Earlier quoted context omitted.
What does git mean?
honestly, maybe this reflects my americanness but I presumed it was derived from the (western film/culture) word, a corruption of 'get'. Today I learned that it means something else in british.
You can hardly watch more than a few hours of british comedy without someone being called a 'stupid git' at some point.
Re: Dolt is Git for data
#35Earlier quoted context omitted.
"I'm an egotistical bastard, and I name all my projects after myself." --Linus
In case it isn't obvious, this is a joke, Linus actually didn't name Linux after himself, the person who hosted the original source tree did: https://en.wikipedia.org/wiki/Linux#Naming
However, Linux was a name that Linus had considered before dismissing it as too self-centered. It wasn't invented by the later host.
Re: Dolt is Git for data
#36Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774
I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data contests with The Economist - but at the end of the day, we couldn't find anyone with an urgent problem that they were willing to pay to solve.
I wish these folks luck! Perhaps things have changed; we were part of a flock of 5 or 10 similar projects and I'm pretty sure the only one still around today is Kaggle.
Re: Dolt is Git for data
#37Pardon my ignorance, but is data copy writable? Or can it be owned? Obviously someone can get into trouble upload propriety code to git. Is there proprietary data?
Re: Dolt is Git for data
#38Can you give some more information about what you're doing with your cloud infrastructure? Would be intrigued to hear about what you're running.
Re: Dolt is Git for data
#39Very cool! The world needs better version control for data. How does this compare to something like Pachyderm? How does it work under the covers? What is a splice and what does it mean when it overlaps? https://github.com/liquidata-inc/dolt/blob/84d9eded517167eb2... Is it feasible to use Conflict-free Replicated Data Types (CRDT) for this?
Here is an earlier blog we published on comparison's to Pachyderm: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-... We got a blog on the storage system coming on Wednesday. It's a mashup of a Merkle DAG and a B-tree called a Prolly Tree. It comes from an open source package called Noms ( https://github.com/attic-labs/noms ). I'm not familiar with CRDT. Will read up on that.
Weighing in as Pachyderm founder.
The post Tim links here is a very apt description of what Pachyderm does. We're designed for version controlling data pipelines, as well as the data they input and output. Pachyderm's filesystem, pfs, is the component that's most similar to dolt. Pfs is a filesystem, rather than a database, so it tends to be used for bigger data formats like videos, genomics files, sometimes databases dumps. And the main reason people do that is so they can run pipelines on top of those data files.
Under the hood the datastructures are actually very similar though, we use a Merkle Tree, rather than a DAG. But the overall algorithm is very similar. Dolt, I think, is a great approach to version controlling SQL style data and access. Noms was a really cool idea that didn't seem to quite find its groove. Whereas dolt seems to have taken the algorithm and made it into more of a tool with practical uses.
Re: Dolt is Git for data
#40Very cool! The world needs better version control for data. How does this compare to something like Pachyderm? How does it work under the covers? What is a splice and what does it mean when it overlaps? https://github.com/liquidata-inc/dolt/blob/84d9eded517167eb2... Is it feasible to use Conflict-free Replicated Data Types (CRDT) for this?
Here is an earlier blog we published on comparison's to Pachyderm: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-... We got a blog on the storage system coming on Wednesday. It's a mashup of a Merkle DAG and a B-tree called a Prolly Tree. It comes from an open source package called Noms ( https://github.com/attic-labs/noms ). I'm not familiar with CRDT. Will read up on that.
What is your take on the need for time-travel queries for versioned, mutable data? Versioning immutable data items is not enough if you have structured data that is updated. Every time you update a data item, you store a full copy - not a diff of the actual data. You are not able to make "time-travel queries" - give me the data that was generated in this time-range, for example.
For example, if you have a Feature Store for ML, and you want to say "Give me train/test data for these features for the years 2012-2020". This isn't possible with versioned immutable data items. Also, if you don't store the diffs in data - if you store immutable copies, you get explosive growth in data volumes. There are 2 (maybe 3) frameworks that allow such time-travel queries i am aware of: Apache Hudi (Uber) and Databricks Delta. (Apache Iceberg by Netflix will have support soon.)
Reference:
https://www.logicalclocks.com/blog/mlops-with-a-feature-stor...