Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

41–50 of 183 posts

Re: Show HN: Quilt – manage data like code

#41

Post by akarve »

fiatjaf wrote: post #39

What does this do that cannot be done with git or similar software + data stored in some standard format?

Four things: serialization, virtualization, querying, big data (even Git LFS isn't super performant for large files). Quilt actually transforms data into Parquet and wraps it in a virtualization layer so that the data can be injected directly into code. Efficient querying is a function of the serialization.

By contrast, GitHub is a blob store, it doesn't transform the data either for serialization or for virtualization.

Re: Show HN: Quilt – manage data like code

#42

Post by akarve »

koolba wrote: post #7
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

This looks pretty cool. Am I right to understand that this is providing a virtualized filesystem interface that dynamically loads the slices of data actually being accessed (but through a direct API rather than something generic like FUSE)? Unrelated I found a typo on your blog. Search for "seriailize" on your " Manage data like source code " post.

As to your slicing question, yes. Data is lazily loaded. With Parquet as our data store we can do even more (but haven't yet): load only the columns referenced.

Re: Show HN: Quilt – manage data like code

#44

Post by akarve »

masklinn wrote: post #22
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Hmm. We were able to get the pip handle so didn't see major conflicts in our target space. Are there places in code/cli where we could name conflict with the patch sets manager?

Re: Show HN: Quilt – manage data like code

#45

Post by akarve »

ah- wrote: post #43

How much of it is open source? Can I run my own?

The client is fully open source. You can indeed run your own and we are just starting to roll that out. I can get you started: feedback at quiltdata dot io. We are deliberating open sourcing the registry as well (making everything open source). What do you think?

Re: Show HN: Quilt – manage data like code

#46

Post by reggieband »

I think you're missing a trick with the pricing. My guess is the real money will come once data is treated like a commodity. So the big, big money will be in brokerage's and exchanges.

Paying flat fees for access to repos is fundamentally thinking about the problem incorrectly.

Re: Show HN: Quilt – manage data like code

#49

Post by akarve »

torchous wrote: post #47

Any thoughts on adding DOIs? It's a complex subject wrt versioning, in particular (new DOI per version? How to keep track?). It would help tremendously with the academic community; for the bean counting.

The package name + hash is an implicit DOI. What if we added web support for it so that users could https://quiltdata.com/packages/USER/PKG?doi=SOME_HASH ?

Re: Show HN: Quilt – manage data like code

#50

Post by xyzzy_plugh »

akarve wrote: post #44
masklinn wrote: post #22

Earlier quoted context omitted.

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Hmm. We were able to get the pip handle so didn't see major conflicts in our target space. Are there places in code/cli where we could name conflict with the patch sets manager?

I think it conflicts a bit:

  $ brew install quilt >/dev/null 2>&1 && which quilt
  /usr/local/bin/quilt
I have used it extensively in automation in the past. I don't know if it's a good idea to call it "quilt" on the $PATH. Maybe "dataquilt" or "data-quilt"? "quilted"?
Post reply on HN