Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

151–160 of 183 posts

Re: Show HN: Quilt – manage data like code

#151

Post by geraldbauer »

FYI: I've built a datapackage manager called datapak in ruby [1][2]. datapak supports the tabular datapackages (.csv with .json schema) from the frictionless data initiative (by the open knowledge foundation). All open source and public domain. See some examples such as the Standard&Poors 500. By default the datapackage gets auto-added from .csv to an in-memory SQLite database for easy querying etc. Thanks to ActiveRecord you can use PostgreSQL, MySQL, etc.

[1] https://github.com/textkit/datapak [2] http://okfnlabs.org/blog/2015/04/26/datapak.html

Re: Show HN: Quilt – manage data like code

#152

Post by masklinn »

akarve wrote: post #44
masklinn wrote: post #22

Earlier quoted context omitted.

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Hmm. We were able to get the pip handle so didn't see major conflicts in our target space. Are there places in code/cli where we could name conflict with the patch sets manager?

Quilt is a patchset manager and used a lot in linux distro spaces (it was created by one of Linux's maintainers for his own use). Any global install of your tool will conflict with its existence since the utility names are the same, which you'll get will depend on your PATH setup.

Re: Show HN: Quilt – manage data like code

#153

Post by hn_throwaway_99 »

smilliken wrote: post #64

It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support distributed workflows, don't support commit ids as first class objects, or most other basic features of VCSs. Databases just aren't a solution to version control. I can't imagine a future where we don't treat data version control like a necessity in th…

Have you seen data.world? "GitHub for data" is basically their entire model.

Re: Show HN: Quilt – manage data like code

#157

Post by akarve »

zitterbewegung wrote: post #102
akarve wrote: post #52

Earlier quoted context omitted.

Data cleaning is so necessary. `build.yml` already supports a limited set of feature (through pandas). In addition to custom data transformations, any "out of the box" cleaning functions you'd like? In the spirit of dplyr? We've looked at e.g. scikit feature for normalization, 1-of-n encoding, etc.

How about deduplication. Also table / dictionary lookup . And string replacement and regex replacement .

Got it. If you'd like me to ping you once we have custom build hooks: aneesh at quiltdata dot io. Have you tried Luigi or Bonobo for data cleaning?

Re: Show HN: Quilt – manage data like code

#158

Post by ma2rten »

masklinn wrote: post #22
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

There is also a relatively well-known startup called Quid. They are doing something with data visualization. I first though this was them.

Re: Show HN: Quilt – manage data like code

#159

Post by akarve »

ivotron wrote: post #53
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

are you aware of datapackages [1]? do you plan on opensourcing the backend components? [1]: https://github.com/frictionlessdata

Frictonless is interesting but 1) doesn't handle serialization (which is essential for performance); 2) requires users to hand annotate schemas (we think schemas should be auto-generated whenever possible).

Re: Show HN: Quilt – manage data like code

#160

Post by kchr »

kevinemoore wrote: post #55
masklinn wrote: post #22

Earlier quoted context omitted.

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.

No worries, just make the name switch sooner rather than later. Since your target audience is obviously developers in general - to whom the patch tool is well known - it will most likely affect your business negatively if you don't...

How did this not come up during your market research?

Post reply on HN