Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolt is Git for data

dolthub.com

Post reply on HN
Print view View original

181–190 of 196 posts

Re: Dolt is Git for data

#181

Post by zachmu »

tgb wrote: post #150

An example use case that "git for data" seems to break: storing data for medical research where the participants are allowed to withdraw from the study after the fact. Then their data must be deleted retroactively, not just in the head node. I don't know of a good methodology for dealing with this at all as it breaks backups, for example. The problem extends beyond medical research due to privacy laws like the GDPR.…

You can rebase to change the history. As with git, if you do this, everyone with a clone will need to clone a fresh copy, as they can no longer merge with the remote HEAD.

Re: Dolt is Git for data

#182

Post by BiteCode_dev »

tobr wrote: post #109
BiteCode_dev wrote: post #106

Dolt is not Git for data. Git take existing files, and allow you to version them. Git for data would take existing tables or rows, and allow you to version them. A uniform, drop in, open source way to have an history or row, merge them, restore them, etc. that works for Postgres, Mysql or Oracle in the same way. And is compatible with migrations. You can have an history if you use big table or couchdb, not need for D…

You can nitpick any comparison like that by pointing out the different ways the metaphor breaks.

It's not a metaphor. It's a sell pitch.

Re: Dolt is Git for data

#184

Post by jnbiche »

zachmu wrote: post #180
jnbiche wrote: post #164

Is there a way to page sql results? Also, it would be awesome if I could use rlwrap with `dolt sql`, so I can use the shortcuts I'm used to in an REPL environment.

Yeah, the SQL shell needs some work, including its readline implementation which is kind of broken. Filed an issue for this: https://github.com/liquidata-inc/dolt/issues/505

Awesome. Yeah, my question wasn't clear, but I meant paging in the shell, as you correctly assumed. In a pinch, I can run page an inline `dolt sql -q` query in the OS shell. But it would be idea to be able to page results in the dolt shell, as we can in most SQL database shells.

BTW, I should have written it above, but dolthub/dolt is quite impressive. I hope you all make it, because it's a great product that I would love to use at work if I eventually shift back over to a data science position (right now, working as a software dev).

Re: Dolt is Git for data

#185

Post by jdoliner »

visarga wrote: post #81
jdoliner wrote: post #39

Earlier quoted context omitted.

Weighing in as Pachyderm founder. The post Tim links here is a very apt description of what Pachyderm does. We're designed for version controlling data pipelines, as well as the data they input and output. Pachyderm's filesystem, pfs, is the component that's most similar to dolt. Pfs is a filesystem, rather than a database, so it tends to be used for bigger data formats like videos, genomics files, sometimes database…

How does Pachyderm deal with GDPR requests. Is it possible to remove a file not just from the present but also from the history? It would be no use to delete a file on GPDR request from the current version while still keeping it around in past commits.

Request to purge data are one aspect of the GDPR that Pachyderm makes trickier. It makes it easier to remove a piece of data and recompute all of your models without it, because it can deduplicate the computation. But to truly purge a piece of data deduplication becomes a hinderance, because the data can be reference by previous commits, and even by other user's data. You can delete a piece of data and have it not be truly purged.

The best recommendation we have for that is that user's data should be encrypted with a key that's unique to the user, and when that user asks you to purge their data you should throw away the key. That means that even if two users have the same data it will be stored encrypted by different keys, so if one asks for the data to be purged the other can still keep their data.

Re: Dolt is Git for data

#187

Post by quickthrower2 »

jon_richards wrote: post #22
quickthrower2 wrote: post #16

I think data (as in raw, collected / measured / surveyed data) doesn't really change, but you get more of it. Some data may occasionally supersede old data. Maybe the schema of the data changes, so your first set of data is in one form, and subsequent data might have more information, or recorded in a different way.

One really important feature of time series data is the preservation of what the dataset looked like at each point in time. Financial data providers will make a mistake (off by order of magnitude, missed a stock split, etc) and then go back and correct it. This means you end up training models entirely on corrected data, but trade based on uncorrected data.

Thanks, I didn't consider training of models, this is a great use case for a tool.

Re: Dolt is Git for data

#188

Post by TylerE »

wainstead wrote: post #135
tinco wrote: post #129

Earlier quoted context omitted.

99.99999% of projects are not the Linux kernel, so how could Git have succeeded because of Linus, other than Linus originating the genius design of it? The Ruby community jumped onto Git even though there was no Github, and Ruby itself didn't use Git. In my opinion it was because Git was the first tool that was superior to SVN in every way. The first time I used Git I swore I would never use SVN again. It was even po…

> so how could Git have succeeded because of Linus, other than Linus originating the genius design of it? Perhaps that is exactly the point. There was a fair amount of hype and press coverage over Git when it was first unveiled. And it was because Linus wrote it, and wrote it in an unexpectedly short time. And it was on the coattails of the whole Bitkeeper saga.

Similar to how, say, Go and Rust became popular while nim and D have largely remained niche products

PS: Hi, Steve!

Re: Dolt is Git for data

#189

Post by sytse »

aboodman wrote: post #103
sytse wrote: post #13

Very cool! The world needs better version control for data. How does this compare to something like Pachyderm? How does it work under the covers? What is a splice and what does it mean when it overlaps? https://github.com/liquidata-inc/dolt/blob/84d9eded517167eb2... Is it feasible to use Conflict-free Replicated Data Types (CRDT) for this?

Hi Sid, if you are curious about how it works internally, you can read some of the old docs from Noms here (Dolt uses a fork of Noms as its internal storage layer). https://github.com/attic-labs/noms/blob/master/doc/intro.md To answer your question, it is pretty easy to make Noms (or Dolt) into a CRDT by defining a merge function that is deterministic. We experimented with this in Noms but the result wasn't that sati…

Thanks for your answer!
Post reply on HN