Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

171–180 of 183 posts

Re: Show HN: Quilt – manage data like code

#171

Post by justinclift »

smilliken wrote: post #64

It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support distributed workflows, don't support commit ids as first class objects, or most other basic features of VCSs. Databases just aren't a solution to version control. I can't imagine a future where we don't treat data version control like a necessity in th…

As a thought, keep an eye on DBHub.io:

https://github.com/sqlitebrowser/dbhub.io

Still in early dev stages, and I'm on holidays for a few weeks atm, but we'll have something useful online in a few months. Supporting forking, branching, merging, etc.

Re: Show HN: Quilt – manage data like code

#172

Post by IanCal »

azag0 wrote: post #54

I am a scientist who sometimes publishes data sets with academic papers and this looks super useful both as a tool and as a potential publishing best practice. Currently doing away with HDF5 and figshare. One necessary future for academia would be to be able to assign a DOI to a given version of data. Is it feasible for quilt to have such a feature?

[disclaimer, work for digital science but not figshare]

Figshare currently has versioned dois, so you can refer to a specific release, and the API will let you download each one. I might put some small tooling around this as I release data there myself.

To add on the feature requests, for academia having trust that the data will survive the company going away is something important to me.

Re: Show HN: Quilt – manage data like code

#173

Post by Galanwe »

akarve wrote: post #170
Galanwe wrote: post #169

Earlier quoted context omitted.

I'm not criticizing the price here. It is obviously totally OK to pay for data hosting. > "Contact us to start Business or On-premise service." I DO reject the whole concept of having a "hosted-only" software. To me it makes the whole project useless. It means I cannot have my own private or confidential data. I cannot use it in my company, etc. More importantly, the data is not mine anymore. It's yours. I cannot fin…

We need to clarify the EULA and terms on the website. The data belongs to the users and we want to keep it that way. Quilt is not "hosted-only". The whole point of the on-prem install is that customers can run Quilt on their own infrastructure (it's Dockerized, etc.). We'll roll out a more formal on-prem solution as the project evolves. We're doing one thing at a time right now :)

> "The data belongs to the users and we want to keep it that way." Does it mean you plan to include in the EULA that Quilt will not parse, read, sell or otherwise use the data that users upload?

> "The whole point of the on-prem install is that customers can run Quilt on their own infrastructure (it's Dockerized, etc.)."

Yes, and on premise is not possible at the moment, and on the website it is marketed the same a business use.

Anyway, I lost my cool a bit on previous comments, thanks for keeping yours, and good luck :)

Re: Show HN: Quilt – manage data like code

#174

Post by forkLding »

akarve wrote: post #19
forkLding wrote: post #9

Is there an option to be able to sign up using Github? That would make life much easier for me.

GH sign up would be useful. If you email us I will ping you when we add it: feedback at quiltdata dot io.

I've downloaded Quilt using pip, is the login more for cloud storage for your own data and projects similar to github? Maybe clarify that point on the website? Because not too sure what I would be signing up for, thanks again and love the service.

Re: Show HN: Quilt – manage data like code

#175

Post by rattray »

Galanwe wrote: post #166
akarve wrote: post #2

Hi, I'm one of the founders of Quilt Data (YCW16). We built Quilt to bring package management to data. The goal is to create a community of versioned, reusable building blocks of data, so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. Our general inspiration is to create a new kind of data warehouse based on code management practices that haven't yet reached the data…

> We built Quilt to bring package management to data. No, you built Quilt to own the data. Package managers don't make users pay if they want their own package repositories. > so that analysts can spend more time analyzing and less time finding, cleaning, and organizing data. You should spend more time with data analysts then. You would understand that cleaning, filtering and preparing the data is actually part of da…

> Package managers don't make users pay if they want their own package repositories.

Isn't that npm's business model?

Re: Show HN: Quilt – manage data like code

#176

Post by akarve »

forkLding wrote: post #174
akarve wrote: post #19

Earlier quoted context omitted.

GH sign up would be useful. If you email us I will ping you when we add it: feedback at quiltdata dot io.

I've downloaded Quilt using pip, is the login more for cloud storage for your own data and projects similar to github? Maybe clarify that point on the website? Because not too sure what I would be signing up for, thanks again and love the service.

The login allows you to push packages. If you only want to consume public packages: no login required.

Re: Show HN: Quilt – manage data like code

#177

Post by akarve »

nicodjimenez wrote: post #164

This seems like a great way of publishing public datasets. However, as someone who works on a computer vision startup I don't think I could really use this. In my work data annotation, visualization, and versioning cannot be easily separated. The effort we would need to put in to use quilt might be better spent building a simple versioning system on top of our current data infrastructure.

Where does it break down? Quilt can package and version directly from in-memory objects so if you are working in Python (more languages planned) you can package as you go and include any dependencies?

Re: Show HN: Quilt – manage data like code

#178

Post by akarve »

dawiddutoit wrote: post #154

Awesome idea. Have you thought about a marketplace perhaps? Say for instance I create a package with all the cities in the world or all species of canine, host it in a marketplace and others can buy that data package to use in their own projects.

Stay tuned :)

Re: Show HN: Quilt – manage data like code

#179

Post by rspeer »

IanCal wrote: post #148
rspeer wrote: post #119

I'm very excited. I want to use this to version ConceptNet's raw input and its built data, all of which is public. So I can assume this isn't going to be afraid of gigabytes, right? I've seen services before that want to be a repository of data, and I try to upload a mere 20 GB of data and they're like "oh shit nevermind". Even S3 requires it to be broken into files of less than 5 GB for some inscrutable reason.

I don't think you have had to break files up yourself for a long time on S3. You can treat files up to 5TB as a single object. I think you have to do a multipart upload but that's probably not a bad idea anyway.

Ah, right. I'm remembering from when I was trying to use git-annex to version the data, which was a problem for multiple reasons, including that their S3 driver didn't use multipart uploads.

Re: Show HN: Quilt – manage data like code

#180

Post by edraferi »

geraldbauer wrote: post #151

FYI: I've built a datapackage manager called datapak in ruby [1][2]. datapak supports the tabular datapackages (.csv with .json schema) from the frictionless data initiative (by the open knowledge foundation). All open source and public domain. See some examples such as the Standard&Poors 500. By default the datapackage gets auto-added from .csv to an in-memory SQLite database for easy querying etc. Thanks to ActiveR…

Very interesting, I've heard of the Open Knowledge Foundation [1] but wasn't aware of the Frictionless Data Initiative [2]. Looks like it's complementary to Common Workflow Language [3]

[1] https://okfn.org/ [2] http://frictionlessdata.io/ [3] http://www.commonwl.org/

Post reply on HN