Dolt is Git for data • Hacker News Board

Re: Dolt is Git for data

#51

Post by philipov » Tue, Mar 31, 2020, 4:02 AM UTC

peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

Git succeeded because it was free, and then business models were able to be built up around the open-source ecosystem after a market evolved naturally. There is a need, but if you go into it trying to build a business from scratch, you're going to have a bad time.

Re: Dolt is Git for data

#52

Post by brachi » Tue, Mar 31, 2020, 4:05 AM UTC

> With Dolt, you can view a human-readable diff of the data you received last time versus the data you received this time.

How is this accomplished if the data is binary?

Also, how does this compare to git lfs?

Re: Dolt is Git for data

#53

Post by rburhum » Tue, Mar 31, 2020, 4:12 AM UTC

Eh, I worked on database with branches in 2002 for 3 years while I was at ESRI. It is called a versioned system... Here is how it works from an answer I gave several years back on gis.stackexchange https://gis.stackexchange.com/questions/15203/when-versionin...

Re: Dolt is Git for data

#54

Post by AlexCoventry » Tue, Mar 31, 2020, 4:17 AM UTC

matthewbauer wrote: post #50

I thought Git was Git for data.

Spotted the lisp fan.

Re: Dolt is Git for data

#55

Post by timsehn » Tue, Mar 31, 2020, 4:19 AM UTC

brachi wrote: post #52

> With Dolt, you can view a human-readable diff of the data you received last time versus the data you received this time. How is this accomplished if the data is binary? Also, how does this compare to git lfs?

No data diffs is the data is binary. But diffs are cell-wise like so: https://www.dolthub.com/repositories/Liquidata/corona-virus/...

git-lfs lets you put store large files on GitHub. With Dolt, we offer a similar a similar utility called git-dolt. Both these allow you to store large things on GitHub as a reference to another storage system, not the object itself.

Re: Dolt is Git for data

#56

Post by hypewatch » Tue, Mar 31, 2020, 4:21 AM UTC

> As far as we can tell, Dolt is the only database with branches

What about pachyderm?

Re: Dolt is Git for data

#57

Post by timsehn » Tue, Mar 31, 2020, 4:21 AM UTC

honksillet wrote: post #37

Pardon my ignorance, but is data copy writable? Or can it be owned? Obviously someone can get into trouble upload propriety code to git. Is there proprietary data?

We wrote a couple blog post on data licensing. We're not lawyers but we did a bit of research:

https://www.dolthub.com/blog/2020-02-24-data-licenses/ https://www.dolthub.com/blog/2020-02-26-copyrightable-materi...

Re: Dolt is Git for data

#58

Post by timsehn » Tue, Mar 31, 2020, 4:22 AM UTC

hypewatch wrote: post #56

> As far as we can tell, Dolt is the only database with branches What about pachyderm?

See the comment down below from a cofounder of Pachyderm :-)

Re: Dolt is Git for data

#59

Post by timsehn » Tue, Mar 31, 2020, 4:22 AM UTC

zyang wrote: post #48

peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

The need is definitely there. My day job involves such need. But we simply cannot trust a drive-by startup to fill the gap. It's safer just to roll your own.

Dolt is and always will be open source. Feel free to fork a copy and make it your own.

Re: Dolt is Git for data

#60

Post by timsehn » Tue, Mar 31, 2020, 4:26 AM UTC

matthewbauer wrote: post #50

I thought Git was Git for data.

You can make a solid case and I tried: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-...

But there are some limitations for large tables. Diffs only work if you order the file in the same order. Plus, you have to import the data into some other tool to do anything with it.