Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

111–120 of 183 posts

Re: Show HN: Quilt – manage data like code

#111

Post by akarve »

mbeigel wrote: post #98
kevinemoore wrote: post #28

Earlier quoted context omitted.

Yes! We'd like to get to R and Scala next and hopefully C++ soon. We'd love help from open-source collaborators. Our team is definitely strongest in Python.

I'd definitely be interested in looking into this but I'm on an R team. Didn't see anything on how to contribute in another language or an API to hit. Will be there be information soon?

Hi. R is really important to us and we want to add support (probably through SparklyR). If you email me I can add you to our Slack channel and we can talk through extending Quilt (aneesh at quiltdata dot io). There is a sliver in our docs but it's not complete: https://docs.quiltdata.com/basics.html

The hooks for extending build targets are here: https://github.com/quiltdata/quilt/blob/4aa6897f9e33349b7778...

Again it's a bit raw but we're here to help make R support easy.

Re: Show HN: Quilt – manage data like code

#112

Post by noobermin »

stewbrew wrote: post #89

It looks like a plain html page but requires JS to view anything except: "Please enable JavaScript to use this site." What a wonderful time to live in. Anyway, do I get this right: They expect users to be experts in data analysis but not being able to load the data into whatever software they use? They want me to share data and to offload my data into their walled garden that can be accessed only via their service? I…

I know git is not great with binaries, but wouldn't just a git repo be a better start for what you want?

Re: Show HN: Quilt – manage data like code

#113

Post by raimue »

akarve wrote: post #44
masklinn wrote: post #22

Earlier quoted context omitted.

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Hmm. We were able to get the pip handle so didn't see major conflicts in our target space. Are there places in code/cli where we could name conflict with the patch sets manager?

When choosing a name, you should check the package manager index sites of large distros for potential conflicts. For example in Debian:

https://packages.debian.org/search?suite=default&section=all...

Re: Show HN: Quilt – manage data like code

#114

Post by noobermin »

mdevere wrote: post #84

Hey, this sounds really interesting and I'd like to play around with it. However, I'm a novice and run into the following issue: >>> examples.sales No idea what a DataNode is so am struggling to actually see the data! Any tips?

For the future, learn to use dir().

Re: Show HN: Quilt – manage data like code

#115

Post by akarve »

gouggoug wrote: post #65

This makes me think of http://www.pachyderm.io/ . Although Quilt seems to be more like github for data, whereas Pachyderm is more like git for data.

We really like pachyderm and know the founders. Quilt is zero-config focused on storage and versioning, pachyderm is more focused on on-prem and compute (as a Hadoop replacement).

Re: Show HN: Quilt – manage data like code

#116

Post by mgalka »

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

Love the concept! Clear value proposition for anyone who knows both worlds (code + data). Look forward to giving it a try.

Re: Show HN: Quilt – manage data like code

#117

Post by akarve »

reggieband wrote: post #46

I think you're missing a trick with the pricing. My guess is the real money will come once data is treated like a commodity. So the big, big money will be in brokerage's and exchanges. Paying flat fees for access to repos is fundamentally thinking about the problem incorrectly.

We charge business and on-prem users in TB-sized blocks. So that part is variable cost, not flat. And we sell user seats in blocks of 10. What else should we be thinking about? We want to be fair and also price in a way that encourages sharing behind the firewall (e.g. shouldn't require manager approval to add every new user).

Re: Show HN: Quilt – manage data like code

#119

Post by rspeer »

I'm very excited. I want to use this to version ConceptNet's raw input and its built data, all of which is public.

So I can assume this isn't going to be afraid of gigabytes, right? I've seen services before that want to be a repository of data, and I try to upload a mere 20 GB of data and they're like "oh shit nevermind". Even S3 requires it to be broken into files of less than 5 GB for some inscrutable reason.

Re: Show HN: Quilt – manage data like code

#120

Post by akarve »

wodenokoto wrote: post #93

The workflow I would have imagined for versioning data is: 1) Load original data from source into quilt 2) Do transformation 3) Commit transformations to quilt, with commit message 4) Run experiment 5) Do new transformations 6) Commit to quilt 7) Run experiment Rinse and repeat. Looking at the video and documentations, this is not emphasised at all, suggesting that edits to data should be saved as a new package.

You can absolutely edit in place and that will go into `quilt log` for the package--as long as you are the package owner. Our docs were a bit confusing on this point. I just updated them: https://docs.quiltdata.com/edit-a-package.html
Post reply on HN