Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

131–140 of 183 posts

Re: Show HN: Quilt – manage data like code

#131

Post by akarve »

philipov wrote: post #128
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

Is this only cloud hosted? I have been looking for this exact solution, and I'm very excited to try yours, but banks and other institutions will not tolerate storing their data in my butt. It must be totally local without ever needing an outside network. Can you provide that?

We are rolling out on-prem where the customer runs the package registry on their own infrastructure. The set up process takes a bit of customization depending on your environment. If you have Docker containers it's easier. If you email me I can get you rolling: aneesh at quiltdata dot io.

Re: Show HN: Quilt – manage data like code

#132

Post by akarve »

eternauta3k wrote: post #104
stewbrew wrote: post #89

It looks like a plain html page but requires JS to view anything except: "Please enable JavaScript to use this site." What a wonderful time to live in. Anyway, do I get this right: They expect users to be experts in data analysis but not being able to load the data into whatever software they use? They want me to share data and to offload my data into their walled garden that can be accessed only via their service? I…

> If I wanted to share my data, wouldn't I rather use something more accessible? Still, it's good inspiration. Maybe I'll make a github repo with my city's open datasets loaded into python.

There are some important differences from git: https://news.ycombinator.com/item?id=14772036

Re: Show HN: Quilt – manage data like code

#133

Post by akarve »

stewbrew wrote: post #89

It looks like a plain html page but requires JS to view anything except: "Please enable JavaScript to use this site." What a wonderful time to live in. Anyway, do I get this right: They expect users to be experts in data analysis but not being able to load the data into whatever software they use? They want me to share data and to offload my data into their walled garden that can be accessed only via their service? I…

It's not about people not being able to load their data, but about accelerating the loading with serialization and about whether or not people want to focus on data cleaning, or have the cleaning done once and then available for posterity.

Quilt is, in my view, as open as git or GitHub. The de/serialization code is all open source, and uses an open format (Parquet). Parquet is accessible (and more optimal than text files) for things like Presto DB, Hive, etc.

Re: Show HN: Quilt – manage data like code

#134

Post by akarve »

amygdyl wrote: post #103
masklinn wrote: post #22

Earlier quoted context omitted.

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

I used to like this style of naming software with a allusive dictionary noun, but the sheer volume of new code is driving me to wish there was a canonical clearing house for naming, with agreed conventions for the "given name" even. e.g. Quilt, which in documentation and prose reference is the most natural choice, and a real use handle with a context qualifier, especially e.g. Quilt_Data and the use of a clearing hou…

Java has the right idea. Namespace by [reverse] domain name. It's sadly a tad verbose though. There's high contention for short names which is at odds with uniqueuness.

Re: Show HN: Quilt – manage data like code

#135

Post by reggieband »

akarve wrote: post #117
reggieband wrote: post #46

I think you're missing a trick with the pricing. My guess is the real money will come once data is treated like a commodity. So the big, big money will be in brokerage's and exchanges. Paying flat fees for access to repos is fundamentally thinking about the problem incorrectly.

We charge business and on-prem users in TB-sized blocks. So that part is variable cost, not flat. And we sell user seats in blocks of 10. What else should we be thinking about? We want to be fair and also price in a way that encourages sharing behind the firewall (e.g. shouldn't require manager approval to add every new user).

> What else should we be thinking about?

My feeling is brokering. Consider the market for wheat where there is pricing based on supply and demand. There are futures, options, etc.

Consider a NYSE for data. Why host the data? Be a discovery service both for the price and for brokering of access.

Data should not be priced on it's size to store/transfer. That is leaving huge money on the table. It should be priced based on what people are willing to pay for it.

Why not allow someone to pay for the option for exclusive resale rights of some weather data service? Then allow someone to make a profit off their ability to re-sale that data. etc.

You may not be well placed to do that now but someone is going to. And why would I go to you hoping to find specific data when I can go to a market full of data, full of data re-sellers, etc.

Re: Show HN: Quilt – manage data like code

#137

Post by coredog64 »

akarve wrote: post #134
amygdyl wrote: post #103

Earlier quoted context omitted.

I used to like this style of naming software with a allusive dictionary noun, but the sheer volume of new code is driving me to wish there was a canonical clearing house for naming, with agreed conventions for the "given name" even. e.g. Quilt, which in documentation and prose reference is the most natural choice, and a real use handle with a context qualifier, especially e.g. Quilt_Data and the use of a clearing hou…

Java has the right idea. Namespace by [reverse] domain name. It's sadly a tad verbose though. There's high contention for short names which is at odds with uniqueuness.

Namespacing by reverse domain name is no panacea. Your company might change it's name, might be bought, or might buy another company.

On the open source side, how much stuff was org.sourceforge that then had to move when SourceForge turned into a scummy operation?

Microsoft's CLR guidelines are a little less verbose but still run into the same problem with corporate name being permanently unique.

Re: Show HN: Quilt – manage data like code

#138

Post by akarve »

nosefouratyou wrote: post #136

What's the difference between Apache Parquet and Apache Arrow? They are both columnar formats right?

Parquet is stationary data on-disk. Arrow is focused on in-memory analytics and serializes out to a variety of formats (including Feather and Parquet).

Re: Show HN: Quilt – manage data like code

#139

Post by akarve »

jinjin2 wrote: post #118

I would love if someone did this with http://realm.io , so that the data could be "live" and multiple users could collaborate on it in realtime.

We've talked about this. Though streams (e.g. Apache beam) are closer to where we think realtime data is going. It would be possible to wire Quilt to something like firebase to get realtime behavior... Happy to brainstorm other solutions: aneesh at quiltdata dot io.

Re: Show HN: Quilt – manage data like code

#140

Post by akarve »

servilio wrote: post #75

What's the difference with CVMFS[1] and Nix[2]? [1] http://cernvm.cern.ch/portal/filesystem [2] http://nixos.org/nix/

Those are both interesting projects. Quilt has a rather different emphasis, though. The difference from CVMFS is that Quilt is a full set of services around data (build, push, install) whereas CVMFS is just the file system. In the big data community Paruqet, which we use as a virtualization format, has far more traction than CVMFS. Nixos is specialized for software package management so the distinctions between Quilt and git apply https://news.ycombinator.com/item?id=14772036.
Post reply on HN