Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

161–170 of 183 posts

Re: Show HN: Quilt – manage data like code

#161

Post by kchr »

kchr wrote: post #160
kevinemoore wrote: post #55

Earlier quoted context omitted.

Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.

No worries, just make the name switch sooner rather than later. Since your target audience is obviously developers in general - to whom the patch tool is well known - it will most likely affect your business negatively if you don't... How did this not come up during your market research?

Also, fantastic product! My only criticism is the poor name choice and lack of market research...

Re: Show HN: Quilt – manage data like code

#162

Post by zitterbewegung »

akarve wrote: post #157
zitterbewegung wrote: post #102

Earlier quoted context omitted.

How about deduplication. Also table / dictionary lookup . And string replacement and regex replacement .

Got it. If you'd like me to ping you once we have custom build hooks: aneesh at quiltdata dot io. Have you tried Luigi or Bonobo for data cleaning?

No I haven't I think I just did it myself I'll look into that .

Re: Show HN: Quilt – manage data like code

#163

Post by lmeyerov »

akarve wrote: post #34
fredcash25 wrote: post #25

How is this different/the same as the DAT project ( https://datproject.org/ and https://github.com/datproject )?

Dat is a distributed transport layer for raw data. Quilt is a centralized (your infrastructure or ours) transport and consumption layer for virtualized data. As such we'll be able to, for example, run efficient queries across all of Quilt, allow users to import data the same way (no data prep scripts) across a variety of platforms, etc.

I recently encountered https://data.world/datanerd/inc-5000-2016-the-full-list and was impressed by the combo of a pandas-friendly client library & centralized online tier with bells & whistles. Feels closer to that direction.

There's obviously friction in this space -- this weekend I'm playing with databricks spark notebooks and public data on S3 and the data prep will still be annoying -- so I'm looking forward to design innovation in it.

Re: Show HN: Quilt – manage data like code

#164

Post by nicodjimenez »

This seems like a great way of publishing public datasets. However, as someone who works on a computer vision startup I don't think I could really use this. In my work data annotation, visualization, and versioning cannot be easily separated. The effort we would need to put in to use quilt might be better spent building a simple versioning system on top of our current data infrastructure.

Re: Show HN: Quilt – manage data like code

#165

Post by falsedan »

styfle wrote: post #144
falsedan wrote: post #86

Earlier quoted context omitted.

I think the whole project is doomed, the technology is trivially clonable and the backend is a thin API over S3. Their best bet is an acquisition by Amazon etc. to become e.g. Elastic Data Packs

I don't be think that trivial tech means they are doomed. GitHub, for example, is basically a GUI around git with a social aspect. Many others have implemented the same features: GitLab, Gogs, BitBucket, etc. Yet GitHub has not failed. It's all about the execution.

GitHub solves a lot of performance bottlenecks, and codified forking. They didn’t write an app or client bindings, which it seems that quilt is giving away. It sounds to me like they have a great interface to a closed hosting platform, and they hope enough people won’t mind maybe paying to access their data in the future. Hence why I think a cloud provider is the most probable acquirer.

edit by ‘doomed’ I mean “hefty discount on employee stock options/wouldn’t put my own money into it”

Re: Show HN: Quilt – manage data like code

#166

Post by Galanwe »

akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

> We built Quilt to bring package management to data.

No, you built Quilt to own the data. Package managers don't make users pay if they want their own package repositories.

> so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data.

You should spend more time with data analysts then. You would understand that cleaning, filtering and preparing the data is actually part of data analysis. I would refuse to do any work on data that was pre processed without having the exact list Of what was done to the raw data.

Honestly all I see technically speaking in this project is a python program to download data frames... Woo-hoo. And for the business side, the now usual attempt to own a community of users and their data. Boring.

Re: Show HN: Quilt – manage data like code

#167

Post by akarve »

Galanwe wrote: post #166
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

> We built Quilt to bring package management to data. No, you built Quilt to own the data. Package managers don't make users pay if they want their own package repositories. > so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. You should spend more time with data analysts then. You would understand that cleaning, filtering and preparing the data is actually part of da…

Namespaces are free. Public repos are free for unlimited data. Should we run a charity and also make private date free? :)

We understand that cleaning/filtering/preparation are key to the analysis, and we plan to support those operations as part of package construction. The question is whether or not that work should be done repeatedly, or once for the benefit of your collaborators.

You can store any kind of data in Quilt. Not just data frames. What's on Quilt today is just the beginning of what is possible.

Do you also dislike GitHub? We've opened up the client source and the community for as much free data as people can publish. It seems naive to expect that we wouldn't charge for anything ever. The users always and forever control their own data, by the way.

Re: Show HN: Quilt – manage data like code

#168

Post by akarve »

lmeyerov wrote: post #163
akarve wrote: post #34

Earlier quoted context omitted.

Dat is a distributed transport layer for raw data. Quilt is a centralized (your infrastructure or ours) transport and consumption layer for virtualized data. As such we'll be able to, for example, run efficient queries across all of Quilt, allow users to import data the same way (no data prep scripts) across a variety of platforms, etc.

I recently encountered https://data.world/datanerd/inc-5000-2016-the-full-list and was impressed by the combo of a pandas-friendly client library & centralized online tier with bells & whistles. Feels closer to that direction. There's obviously friction in this space -- this weekend I'm playing with databricks spark notebooks and public data on S3 and the data prep will still be annoying -- so I'm looking forward to…

I'd like to chat more about this. Quilt uses S3 and I think we could make the process of getting data into Databricks much simpler. Drop me a line if you'd like to discuss aneesh at quiltdata dot io.

Re: Show HN: Quilt – manage data like code

#169

Post by Galanwe »

akarve wrote: post #167
Galanwe wrote: post #166

Earlier quoted context omitted.

> We built Quilt to bring package management to data. No, you built Quilt to own the data. Package managers don't make users pay if they want their own package repositories. > so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. You should spend more time with data analysts then. You would understand that cleaning, filtering and preparing the data is actually part of da…

Namespaces are free. Public repos are free for unlimited data. Should we run a charity and also make private date free? :) We understand that cleaning/filtering/preparation are key to the analysis, and we plan to support those operations as part of package construction. The question is whether or not that work should be done repeatedly, or once for the benefit of your collaborators. You can store any kind of data in…

I'm not criticizing the price here. It is obviously totally OK to pay for data hosting.

> "Contact us to start Business or On-premise service." I DO reject the whole concept of having a "hosted-only" software. To me it makes the whole project useless. It means I cannot have my own private or confidential data. I cannot use it in my company, etc. More importantly, the data is not mine anymore. It's yours.

I cannot find any place on your website that explains what happens to the data once it is uploaded! As far as I'm concerned, it means I am giving you my data, for free, without any restriction on your side. You can resell it, modify it, rebrand it, prevent me from accessing it. Since there is no mention of how the data is stored, I also cannot know if the data is encrypted on your side, or have the capability to read everything that is uploaded.

Re: Show HN: Quilt – manage data like code

#170

Post by akarve »

Galanwe wrote: post #169
akarve wrote: post #167

Earlier quoted context omitted.

Namespaces are free. Public repos are free for unlimited data. Should we run a charity and also make private date free? :) We understand that cleaning/filtering/preparation are key to the analysis, and we plan to support those operations as part of package construction. The question is whether or not that work should be done repeatedly, or once for the benefit of your collaborators. You can store any kind of data in…

I'm not criticizing the price here. It is obviously totally OK to pay for data hosting. > "Contact us to start Business or On-premise service." I DO reject the whole concept of having a "hosted-only" software. To me it makes the whole project useless. It means I cannot have my own private or confidential data. I cannot use it in my company, etc. More importantly, the data is not mine anymore. It's yours. I cannot fin…

We need to clarify the EULA and terms on the website. The data belongs to the users and we want to keep it that way. Quilt is not "hosted-only". The whole point of the on-prem install is that customers can run Quilt on their own infrastructure (it's Dockerized, etc.). We'll roll out a more formal on-prem solution as the project evolves. We're doing one thing at a time right now :)
Post reply on HN