Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

71–80 of 183 posts

Re: Show HN: Quilt – manage data like code

#71

Post by lobster_johnson »

gouggoug wrote: post #65

This makes me think of http://www.pachyderm.io/ . Although Quilt seems to be more like github for data, whereas Pachyderm is more like git for data.

Pachyderm was the first thing I thought of. In fact there should be opportunities for collaboration here.

Pachyderm looks brilliant for high-scalability parallel data processing, and the versioned data part is a way to not just maintain the history of the data, but also avoid reprocessing of data that hasn't changed since the previous run.

Re: Show HN: Quilt – manage data like code

#72

Post by akarve »

heinrichhartman wrote: post #66
akarve wrote: post #45

Earlier quoted context omitted.

The client is fully open source. You can indeed run your own and we are just starting to roll that out. I can get you started: feedback at quiltdata dot io. We are deliberating open sourcing the registry as well (making everything open source). What do you think?

Being able to host my own repository is a must for me. We have many TB of data and don't want to stream that over the internet. Having a repo on-site is a must. I'd appreciate the code being open source. I can afford paying for a (perpetual) license.

Understood. We can do something very close to that. Right now we give the registry source to our on prem users as part of the license. Email sales at quiltdata dot io and we can discuss.

Re: Show HN: Quilt – manage data like code

#74

Post by kevinemoore »

ivotron wrote: post #53
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

are you aware of datapackages [1]? do you plan on opensourcing the backend components? [1]: https://github.com/frictionlessdata

Yes and yes. Do you use frictionless data packages? If so, what do you like and not like? We've looked at their specs and have thought about ways we could integrate. We'd love to hear your suggestions.

Re: Show HN: Quilt – manage data like code

#77

Post by smilliken »

xyzzy_plugh wrote: post #68
smilliken wrote: post #64

It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support distributed workflows, don't support commit ids as first class objects, or most other basic features of VCSs. Databases just aren't a solution to version control. I can't imagine a future where we don't treat data version control like a necessity in th…

I've often thought about this problem space. Imagine you have a database with all the bells and whistles you suggest. At first, it's great. But at some point, when you start experiencing growth (compute/storage) pressure (many petabytes) all of this metadata adds up. Does it remain cost effective at scale? All the source code in the world is a drop in the bucket compared to the raw data collected by a large business.

Well, there's human-generated data, and there's computer-generated data. The former is tiny but more dynamic, and where we want all of these nice creature comforts. The latter can be large, but workflows tend to be static so there's less need for these features.

Re: Show HN: Quilt – manage data like code

#78

Post by Lightbody »

smilliken wrote: post #64

It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support distributed workflows, don't support commit ids as first class objects, or most other basic features of VCSs. Databases just aren't a solution to version control. I can't imagine a future where we don't treat data version control like a necessity in th…

I just joined a company that solves this very problem: https://www.delphix.com/

What you described is literally what we offer -- the ability to go backwards and forwards in time and create branches, bookmarks, and refreshes with very little overhead. [Edit: This is all for databases that by default, as you noted, don't do any of this stuff.]

We aren't (yet) super well known, but we've got a number of things in flight that should change that :)

And if you're curious how this magic works, the hint is: we have a number of really smart ZFS contributors on the engineering team.

Re: Show HN: Quilt – manage data like code

#79

Post by kevinemoore »

RangerScience wrote: post #60
akarve wrote: post #29

Earlier quoted context omitted.

Which languages are most interesting to you? We wrote the client with an eye towards supporting R and Scala. A PR or FR on GitHub would be ideal. We also have a Slack channel where we can support you if you want to tackle adding language bindings to Quilt. feedback at quiltdata dot io.

Ruby and Go, personally.

The package metadata is stored in JSON so that should be pretty easy to access in either Ruby or Go. Tabular data is stored in Parquet by default. Do you know of any good libraries for reading and writing Parquet in Ruby and Go? Do either of those languages have a DataFrame-like class/struct?

Re: Show HN: Quilt – manage data like code

#80

Post by edraferi »

This looks like a cool project -- always glad to see new tools for statistical collaboration and reproducible research.

How does this compare to what data.world [1] is doing? They recently released a Python SDK [2] as well.

[1] https://data.world/ [2] https://github.com/datadotworld/data.world-py

Post reply on HN