Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Quilt – manage data like code

quiltdata.com

Post reply on HN
Print view View original

141–150 of 183 posts

Re: Show HN: Quilt – manage data like code

#141

Post by akarve »

huac wrote: post #90
kevinemoore wrote: post #81

Earlier quoted context omitted.

No, there's no charge for bandwidth. The most common uses so far are users installing datasets locally, which caches the data at the destination or running batch jobs in ECS/EC2, which doesn't accrue charges on AWS.

Huh, that's a pretty good setup for you then! I might mess around and see about writing an R package/interface, because this looks very useful.

We welcome your contributions to the R interface. If you email me, aneesh at quiltdata dot io, I can add you to our Slach channel where our engineers can support your efforts. Several users have asked about R and if we combine them together I think we have the horsepower to build an R layer for Quilt.

Re: Show HN: Quilt – manage data like code

#142

Post by klodolph »

pinhead wrote: post #48

Not to be confused with http://quilt.io/

Huh, I was thinking of https://linux.die.net/man/1/quilt

Basically the sanest way of maintaining patch sets (multiple patches -> quilt, get it?). The logic of using "quilt" as the name for anything that doesn't have to do with patches baffles me.

Re: Show HN: Quilt – manage data like code

#143

Post by axk »

kevinemoore wrote: post #55
masklinn wrote: post #22

Earlier quoted context omitted.

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.

It's quite surprising. Quilt is a very common tool, e.g. for Debian developers: https://wiki.debian.org/UsingQuilt

You should probably rename the pip package and the CLI tool to quiltdata to avoid conflicts.

Re: Show HN: Quilt – manage data like code

#144

Post by styfle »

falsedan wrote: post #86
reggieband wrote: post #46

I think you're missing a trick with the pricing. My guess is the real money will come once data is treated like a commodity. So the big, big money will be in brokerage's and exchanges. Paying flat fees for access to repos is fundamentally thinking about the problem incorrectly.

I think the whole project is doomed, the technology is trivially clonable and the backend is a thin API over S3. Their best bet is an acquisition by Amazon etc. to become e.g. Elastic Data Packs

I don't be think that trivial tech means they are doomed. GitHub, for example, is basically a GUI around git with a social aspect. Many others have implemented the same features: GitLab, Gogs, BitBucket, etc. Yet GitHub has not failed. It's all about the execution.

Re: Show HN: Quilt – manage data like code

#145

Post by mncharity »

amygdyl wrote: post #103
masklinn wrote: post #22

Earlier quoted context omitted.

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

I used to like this style of naming software with a allusive dictionary noun, but the sheer volume of new code is driving me to wish there was a canonical clearing house for naming, with agreed conventions for the "given name" even. e.g. Quilt, which in documentation and prose reference is the most natural choice, and a real use handle with a context qualifier, especially e.g. Quilt_Data and the use of a clearing hou…

> naming software with a allusive dictionary noun

Perhaps , , or . Or . And so on.

Frog (Design, 200M ggl hits); Happy Frog (soil, copy center, 500k hits); Happy Frog Swimming (pool, 4 hits); Happy Frog Swimming Sideways (0 hits).

Quilted Data (250 hits).

But my impression is common nouns have trademark advantages.

Re: Show HN: Quilt – manage data like code

#146

Post by akarve »

reggieband wrote: post #135
akarve wrote: post #117

Earlier quoted context omitted.

We charge business and on-prem users in TB-sized blocks. So that part is variable cost, not flat. And we sell user seats in blocks of 10. What else should we be thinking about? We want to be fair and also price in a way that encourages sharing behind the firewall (e.g. shouldn't require manager approval to add every new user).

> What else should we be thinking about? My feeling is brokering. Consider the market for wheat where there is pricing based on supply and demand. There are futures, options, etc. Consider a NYSE for data. Why host the data? Be a discovery service both for the price and for brokering of access. Data should not be priced on it's size to store/transfer. That is leaving huge money on the table. It should be priced based…

Spot on. I get where you're coming from. The value of data is whatever the buyer and seller agree upon. I was mostly taking about the value/price of the service.

Re: Show HN: Quilt – manage data like code

#147

Post by dtech81 »

akarve wrote: post #127
mbreese wrote: post #97

Earlier quoted context omitted.

Could you tell us how this compares with Synapse ( https://www.synapse.org/ )?. They've been doing this for a long time and have a large presence in computational biology.

I know only a bit about Synapse. It seems like they have some valuable data. I think the biggest differences are in the culture and user experience. Quilt gets its DNA from the open/community-driven cultures of GitHub and npm. We want to make it extremely light-weight for anyone to push and share data packages. Synapse frankly seems to have a lot more features. We want to compete on simplicity. Does Synapse offer ser…

> We want to compete on simplicity.

Bravo, this is an excellent strategy. I've been using NPM packages to share small datasets for a little while due to the simplicity of distribution.

Re: Show HN: Quilt – manage data like code

#148

Post by IanCal »

rspeer wrote: post #119

I'm very excited. I want to use this to version ConceptNet's raw input and its built data, all of which is public. So I can assume this isn't going to be afraid of gigabytes, right? I've seen services before that want to be a repository of data, and I try to upload a mere 20 GB of data and they're like "oh shit nevermind". Even S3 requires it to be broken into files of less than 5 GB for some inscrutable reason.

I don't think you have had to break files up yourself for a long time on S3. You can treat files up to 5TB as a single object. I think you have to do a multipart upload but that's probably not a bad idea anyway.

Re: Show HN: Quilt – manage data like code

#149

Post by IanCal »

akarve wrote: post #133
stewbrew wrote: post #89

It looks like a plain html page but requires JS to view anything except: "Please enable JavaScript to use this site." What a wonderful time to live in. Anyway, do I get this right: They expect users to be experts in data analysis but not being able to load the data into whatever software they use? They want me to share data and to offload my data into their walled garden that can be accessed only via their service? I…

It's not about people not being able to load their data, but about accelerating the loading with serialization and about whether or not people want to focus on data cleaning, or have the cleaning done once and then available for posterity. Quilt is, in my view, as open as git or GitHub. The de/serialization code is all open source, and uses an open format (Parquet). Parquet is accessible (and more optimal than text f…

> Quilt is, in my view, as open as git or GitHub

Can I run something that hosts a quilt repo on my own server?

Re: Show HN: Quilt – manage data like code

#150

Post by cyphar »

kevinemoore wrote: post #55
masklinn wrote: post #22

Earlier quoted context omitted.

The naming is a bit sad in that it conflicts with the patch sets manager: https://en.wikipedia.org/wiki/Quilt_(software)

Sorry to hear about the name conflict. We weren't familiar with the patch sets manager. If it helps to keep the quilt (data) command line tools out of your path, you can still run all the quilt commands from inside Python.

Quilt is an incredibly widely used tool. Many kernel developers use it as an alternative to git for patch management, and it's used by most distributions to manage their patches on top of upstream projects.
Post reply on HN