Dolt is Git for data • Hacker News Board

Re: Dolt is Git for data

#21

Post by StavrosK » Tue, Mar 31, 2020, 1:56 AM UTC

dnautics wrote: post #19

StavrosK wrote: post #10

Earlier quoted context omitted.

What does git mean?

honestly, maybe this reflects my americanness but I presumed it was derived from the (western film/culture) word, a corruption of 'get'. Today I learned that it means something else in british.

Oh huh, I didn't realize it didn't mean the same in American English. I guess I knew it, I just never realized it.

Re: Dolt is Git for data

#22

Post by jon_richards » Tue, Mar 31, 2020, 1:59 AM UTC

quickthrower2 wrote: post #16

I think data (as in raw, collected / measured / surveyed data) doesn't really change, but you get more of it. Some data may occasionally supersede old data. Maybe the schema of the data changes, so your first set of data is in one form, and subsequent data might have more information, or recorded in a different way.

One really important feature of time series data is the preservation of what the dataset looked like at each point in time. Financial data providers will make a mistake (off by order of magnitude, missed a stock split, etc) and then go back and correct it. This means you end up training models entirely on corrected data, but trade based on uncorrected data.

Re: Dolt is Git for data

#23

Post by nmstoker » Tue, Mar 31, 2020, 2:07 AM UTC

hinkley wrote: post #9

timdorr wrote: post #7

Any reason or history behind the name? It means "a stupid person", which seems like a bad choice IMHO: https://www.merriam-webster.com/dictionary/dolt

it's of a piece with git: https://www.merriam-webster.com/dictionary/git

That definition misses a significant portion of the commonly used meaning: someone who is unpleasant (as in mean, annoying)

For instance: stop being such a git - this is not imploring someone not to be stupid, it's saying don't be annoying, awkward etc.

https://www.collinsdictionary.com/amp/english/git

Re: Dolt is Git for data

#24

Post by benatkin » Tue, Mar 31, 2020, 2:18 AM UTC

dominotw wrote: post #6

> Dolt is the only database with branches. datomic has branching too afaik.

So does git. I know, I know, git doesn't provide the features you expect from a full fledged database, but neither does Dolt.

Dolt even has some content in a GitHub wiki, which uses git as a database backend for a web app: https://github.com/liquidata-inc/dolt/wiki

Re: Dolt is Git for data

#25

Post by timsehn » Tue, Mar 31, 2020, 2:25 AM UTC

sytse wrote: post #13

Very cool! The world needs better version control for data. How does this compare to something like Pachyderm? How does it work under the covers? What is a splice and what does it mean when it overlaps? https://github.com/liquidata-inc/dolt/blob/84d9eded517167eb2... Is it feasible to use Conflict-free Replicated Data Types (CRDT) for this?

Here is an earlier blog we published on comparison's to Pachyderm: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...

We got a blog on the storage system coming on Wednesday. It's a mashup of a Merkle DAG and a B-tree called a Prolly Tree. It comes from an open source package called Noms (https://github.com/attic-labs/noms).

I'm not familiar with CRDT. Will read up on that.

Re: Dolt is Git for data

#26

Post by timsehn » Tue, Mar 31, 2020, 2:26 AM UTC

cjbprime wrote: post #17

Comparison to Dat? https://docs.dat.foundation/docs/intro

Dat is more of a distributed data sharing protocol. It does not do branch/merge which Dolt does.

Re: Dolt is Git for data

#27

Post by timsehn » Tue, Mar 31, 2020, 2:28 AM UTC

timdorr wrote: post #7

Any reason or history behind the name? It means "a stupid person", which seems like a bad choice IMHO: https://www.merriam-webster.com/dictionary/dolt

You are correct. It is an homage to Git. We needed a word that meant "idiot" or similar, that started with D for data that was short enough to type on the command line.

Re: Dolt is Git for data

#28

Post by timsehn » Tue, Mar 31, 2020, 2:29 AM UTC

quickthrower2 wrote: post #16

I think data (as in raw, collected / measured / surveyed data) doesn't really change, but you get more of it. Some data may occasionally supersede old data. Maybe the schema of the data changes, so your first set of data is in one form, and subsequent data might have more information, or recorded in a different way.

We think this is the assumption (that data does not change) but we don't think it holds and data diffs help show you when something you did not expect did change.

Re: Dolt is Git for data

#29

Post by timsehn » Tue, Mar 31, 2020, 2:30 AM UTC

oldgregg wrote: post #5

Really interesting. Would be nice to see documentation. All their examples show modifying the database by running command line sql queries, does it turn up a normal mysql instance or just emulate? Are hooks available in Go? Surprised they don't market it as a blockchain database. I'm building a Dapp right now and this could be really useful.

You can spin up a MySQL compatible server using `dolt sql-server`. It does not implement everything right now but we'll get there.

Re: Dolt is Git for data

#30

Post by timsehn » Tue, Mar 31, 2020, 2:31 AM UTC

kdamica wrote: post #4

It's a cool idea. There's also https://quiltdata.com/ but I haven't heard anything about them in a long time.

This blog compares us to Quilt and others: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...