Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Quilt (YC W16) – A versioned data portal for S3

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 65 posts

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#21

Post by timsehn »

Congratulations to the Quilt team on the launch!

Quilt reached out to me and suggested I chime in suggesting that people interested in versioning data also check out Dolt (https://github.com/liquidata-inc/dolt) and DoltHub (https://www.dolthub.com).

We've taken the Git and GitHub for data analogy a lot more literally than Quilt has :-) We are a SQL database with native Git semantics. Instead of versioning files like Git, we version table rows. This allows for diff and conflict detection down to the cell level. We are built on top of another open source project called Noms (https://github.com/attic-labs/noms).

We think there is a ton of room in this space for a bunch of tools: Quilt, Noms, QRI (https://qri.io/), Pachyderm (https://www.pachyderm.io/), and even Git. We're excited to see so many bright minds trying to solve this problem.

We're going to be populating DoltHub with a bunch of datasets we harvest from the open data community to show off the capabilities of Dolt. The coolest one so far is the Google open images dataset: https://www.dolthub.com/repositories/Liquidata/open-images.

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#22

Post by akarve »

breck wrote: post #19
akarve wrote: post #17

Earlier quoted context omitted.

Oh I would love to get the UH Cancer Center data into Quilt! Do you happen to have an S3 bucket with that data live? If the bucket is publicly permissioned it should "just work." We can talk about indexing the data for search. We are comfortable in the TB-PB range :) I will look more closely at Ohayo.

> Do you happen to have an S3 bucket with that data live? No. However, I'm helping start the Data Curation Core at the AIPHI here ( https://aiphi.shepherdresearchlab.org/ ). Our intent is to be a one stop shop for all medical data in Hawaii. We don't yet have a plan on where we will actually store the public datasets (have solutions for private data), but it sounds like from what you folks are saying S3 is the place,…

We have a data curators program on Quilt and I encourage you to apply (page bottom on open.quiltdata.com). For high-value public data sets, AWS's registry of open data will, if accepted, cover the costs of storage and egress. We went through this process with Allen Cell and I'm happy to help.

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#23

Post by kevinemoore »

breck wrote: post #19
akarve wrote: post #17

Earlier quoted context omitted.

Oh I would love to get the UH Cancer Center data into Quilt! Do you happen to have an S3 bucket with that data live? If the bucket is publicly permissioned it should "just work." We can talk about indexing the data for search. We are comfortable in the TB-PB range :) I will look more closely at Ohayo.

> Do you happen to have an S3 bucket with that data live? No. However, I'm helping start the Data Curation Core at the AIPHI here ( https://aiphi.shepherdresearchlab.org/ ). Our intent is to be a one stop shop for all medical data in Hawaii. We don't yet have a plan on where we will actually store the public datasets (have solutions for private data), but it sounds like from what you folks are saying S3 is the place,…

I just want to give a plug for sharing data in the public cloud and S3 in particular. Jed Sundwall (AWS Global Open Data Lead) sums it up really well: "The cloud completely changes the dynamic for sharing data. When data is shared in the cloud, researchers no longer have to worry about downloading or copying data before getting to work. Instead, they can deploy compute resources on-demand in the cloud, where a single copy of the data is made available. It is much more efficient to move algorithms to where the data is, than to move the data to where the algorithms are, and this makes it cheaper for researchers to ask more questions and experiment often." See the full whitepaper here: https://s3-us-west-2.amazonaws.com/opendata.aws/AWS_Sharing_...

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#24

Post by kevinemoore »

timsehn wrote: post #21

Congratulations to the Quilt team on the launch! Quilt reached out to me and suggested I chime in suggesting that people interested in versioning data also check out Dolt ( https://github.com/liquidata-inc/dolt ) and DoltHub ( https://www.dolthub.com ). We've taken the Git and GitHub for data analogy a lot more literally than Quilt has :-) We are a SQL database with native Git semantics. Instead of versioning files l…

Thanks Tim! I definitely second your observation that there's room and reason for plenty of tools in this space. DVC probably belongs in your list too: https://github.com/iterative/dvc. Looking forward to checking out Open Images.

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#25

Post by thegagne »

akarve wrote: post #13
foxhop wrote: post #11

Does it work with digital ocean spaces?

Not yet. But it's closer than one might think. Spaces has an S3-compatible API, and we have plans to use something like min.io to make Quilt work "all the blobs": GCP, Azure, Digital Ocean, etc. OSS contributions welcome :) What would you use Quilt for in Spaces?

Is it possible to implement this using Backblaze B2 instead of S3?

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#27

Post by lichtenberger »

kevinemoore wrote: post #20
lichtenberger wrote: post #12

So you basically store S3 Buckets in Elastic Search and you're using Git for versioning a hierarchy of buckets, right? It's interesting that versioning now finally seems to be getting some traction in mainstream database systems (even though they are not really optimal in these systems my opinion) and for instance also in your data store. You position this as a Dropbox or Google Drive replacement, right? :-) I'm aski…

Sirix sounds like a very interesting system! Is it similar in its internal structure to noms ( https://github.com/attic-labs/noms ) or Dolt ( https://github.com/liquidata-inc/dolt )? I think S3 is a good match for storing database pages as long as they are immutable. The Vectorized query processing model seems to fit this approach very well (e.g., http://oai.cwi.nl/oai/asset/14075/14075B.pdf ) Anyone out there from S…

I'm not really sure how Noms versions stuff. It seems they use a variant of a B-tree index. Or maybe only for secondary indexes?

I think if you can use a simple monotonically increasing sequence number SirixDB has indexing advantages as for instance when storing XML and JSON documents or graph data.

The cool thing also is that SirixDB not only copies changed database pages but it implements a sliding window algorithm for versioning the database pages itself along with well known backup versioning strategies. Furthermore user-specified, typed secondary index structures are also naturally versioned.

One downside is that SirixDB doesn't support branching, even though it would be relatively easily possible to implement I guess, but I'm not convinced that it's needed. I don't want that anyone has to merge merge conflicts. I think automatic algorithms to do this are also not the right thing. But of course it's really interesting and I also thought about it :-) maybe someone has a really good use case? :-)

BTW: Everything in SirixDB is immutable regarding updates of resources in databases. Of course you can revert to an old revision and change stuff, but the revisions in-between will still be accessible.

Keep up the great work on Quilt :-)

Post reply on HN