Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dolt is Git for data

dolthub.com

Post reply on HN
Print view View original

41–50 of 196 posts

Re: Dolt is Git for data

#41

Post by timsehn »

jamesblonde wrote: post #40
timsehn wrote: post #25

Earlier quoted context omitted.

Here is an earlier blog we published on comparison's to Pachyderm: https://www.dolthub.com/blog/2020-03-06-so-you-want-git-for-... We got a blog on the storage system coming on Wednesday. It's a mashup of a Merkle DAG and a B-tree called a Prolly Tree. It comes from an open source package called Noms ( https://github.com/attic-labs/noms ). I'm not familiar with CRDT. Will read up on that.

What is your take on the need for time-travel queries for versioned, mutable data? Versioning immutable data items is not enough if you have structured data that is updated. Every time you update a data item, you store a full copy - not a diff of the actual data. You are not able to make "time-travel queries" - give me the data that was generated in this time-range, for example. For example, if you have a Feature Sto…

The storage system we use only stores the rows that change. We have a blog post we're publishing on Wednesday explaining how.

Re: Dolt is Git for data

#42

Post by flashman »

So, we ingest a third-party dataset that changes daily. One of our problems is that we need to retrospectively measure arbitrary metrics (how many X had condition Y on days 1 through 180 of the current year?). Imagine the external data like this:

UUID,CategoryA,CategoryACount,CategoryB,CategoryBCount,BooleanC,BooleanD...etc

When we ingest a new UUID, we add a column "START_DATE" which is the first date the UUID's metrics were valid. When any of the metric counts changes, we add "END_DATE" to the row and add a new row for that UUID with an updated START_DATE.

It works, but it sucks to analyse because you have to partition the database by the days each row was valid and do your aggregations on those partitions. And it sucks to get a snapshot of how a dataset looked on a particular day. It would be much easier if we could just access the daily diffs, which seems like a task Dolt would accomplish.

I mean it has a better chance of working than getting the third party to implement versioning on their data feed.

Re: Dolt is Git for data

#43

Post by sjtindell »

peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

Just want to say, really appreciate this food for thought. Where do you go and see someone say "my company tried it and...". This site is a godsend.

Re: Dolt is Git for data

#45

Post by timsehn »

perfect_wave wrote: post #38

Can you give some more information about what you're doing with your cloud infrastructure? Would be intrigued to hear about what you're running.

We use AWS but will switch over to Google or multi-cloud when we exhaust our credits.

The system is still pretty simple. The main cost is the storage for the blobs in the Dolt repos pushed to DoltHub. We use S3 for that. There is an API that receives pushes and writes any other metadata (user, permissions, etc) into an RDS instance that stores metadata for DoltHub. That instance is also used to cache some critical things. Then it's just a set of web servers and a GraphQL sitting on top serving our React app.

Re: Dolt is Git for data

#46

Post by buu700 »

rockmeamedee wrote: post #18
timdorr wrote: post #7

Any reason or history behind the name? It means "a stupid person", which seems like a bad choice IMHO: https://www.merriam-webster.com/dictionary/dolt

Yeah, I too am tired of projects named mean-ly. Git, dolt, LAME, Gimp... DWARF is borderline, even if it has historical reasons (being based on ELF) it's not nice to hear out of context. Even if it's a joke on yourself, just like, why would you give anyone who hasn't heard of your project the idea that it might be mean? You wouldn't name your pet Dumbass. Why your pet project.

A bit off topic, but at summer camp many years ago, a counselor looked over my shoulder while I was using GIMP, and said something like "that's a sick joke". And that's how I learned about BDSM.

Re: Dolt is Git for data

#47

Post by timsehn »

peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

I'll check it out. We think the world is a little more ready for this now, given how widely Git is adopted and the advances in other data tooling (like ML). But, as we're all aware, starting a business is hard :-)

Re: Dolt is Git for data

#48

Post by zyang »

peteforde wrote: post #36

Only 39 days since the last "GitHub for data" was announced: https://news.ycombinator.com/item?id=22375774 I'll say what I said in February: I started a company with the same premise 9 years ago, during the prime "big data" hype cycle. We burned through a lot of investor money only to realize that there was not a market opportunity to capture. That is, many people thought it was cool - we even did co-sponsored data c…

The need is definitely there. My day job involves such need. But we simply cannot trust a drive-by startup to fill the gap. It's safer just to roll your own.

Re: Dolt is Git for data

#49

Post by jamesblonde »

flashman wrote: post #42

So, we ingest a third-party dataset that changes daily. One of our problems is that we need to retrospectively measure arbitrary metrics (how many X had condition Y on days 1 through 180 of the current year?). Imagine the external data like this: UUID,CategoryA,CategoryACount,CategoryB,CategoryBCount,BooleanC,BooleanD...etc When we ingest a new UUID, we add a column "START_DATE" which is the first date the UUID's met…

You can accomplish this using time-travel queries in frameworks like Apache Hudi and Databricks Delta that i mentioned in more detail in an earlier comment. They only work for Spark-based data pipelines.
Post reply on HN