Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Quilt (YC W16) – A versioned data portal for S3

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 65 posts

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#31

Post by timsehn »

kevinemoore wrote: post #24
timsehn wrote: post #21

Congratulations to the Quilt team on the launch! Quilt reached out to me and suggested I chime in suggesting that people interested in versioning data also check out Dolt ( https://github.com/liquidata-inc/dolt ) and DoltHub ( https://www.dolthub.com ). We've taken the Git and GitHub for data analogy a lot more literally than Quilt has :-) We are a SQL database with native Git semantics. Instead of versioning files l…

Thanks Tim! I definitely second your observation that there's room and reason for plenty of tools in this space. DVC probably belongs in your list too: https://github.com/iterative/dvc . Looking forward to checking out Open Images.

The coolest thing to do is diff between the V2 and V3 branch for a label_descriptions table.

https://www.dolthub.com/repositories/Liquidata/open-images/c...

You can start to see the power of column-wise diffs. You can start to imagine what it would be like to change this table and then merge Google's changes in V3 onto your modified copy. Very powerful. We need a query interface on top of diffs. Lots to build...

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#32

Post by kevinemoore »

trailerfins wrote: post #29

I also really appreciated your lessons learned — pretty compelling. The showcase buckets on the site are awesome. What's the mechanism by which the public data ends up in S3, just out of curiosity?

In general, a data publisher simply creates an S3 bucket in their account and sets the access control to allow public read. In the specific case of the AWS Registry of Open Data, data providers create a clean AWS account to hold the data (the account will have only S3 and won't run any compute services). Once the dataset is accepted into RODA, AWS will cover the costs (S3 storage and egress bandwidth) for the clean account.

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#33

Post by akarve »

thegagne wrote: post #25
akarve wrote: post #13

Earlier quoted context omitted.

Not yet. But it's closer than one might think. Spaces has an S3-compatible API, and we have plans to use something like min.io to make Quilt work "all the blobs": GCP, Azure, Digital Ocean, etc. OSS contributions welcome :) What would you use Quilt for in Spaces?

Is it possible to implement this using Backblaze B2 instead of S3?

Contingent upon support in something like ceph or minio, yes.

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#34

Post by akarve »

Felz wrote: post #10
akarve wrote: post #9

Earlier quoted context omitted.

Can do. A lot of the magic happens in the es/indexer and search lambdas here: https://github.com/quiltdata/quilt/tree/master/lambdas . The short of what we do: we listen for bucket notifications in Lambda, open the object metadata and send it, along with a snippet of the file contents, to ElasticSearch for indexing. ElasticSearch mappings are a bit of a bear and we had to lock those down to get them to behave well. W…

Seems like it'd be more elegant (and probably cost effective) if you stored the Lucene indexes inside the buckets themselves.

That is an interesting idea. What kind of performance could we expect, especially in the federated case of searching multiple buckets? Elastic has sub-second latency (at the cost of running dedicated containers).

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#35

Post by rsync »

akarve wrote: post #17
breck wrote: post #16

This is great. Thank you for so openly sharing your strategic thinking and lessons learned. I follow about 100 projects in this space "github for data" and haven't yet seen a breakout hit. Yours looks like it has potential. I like the simplicity and the "objects by file extension". Lots of these sites I think get too complex too quick. At the UH Cancer Center we routinely deal with datasets in the TB - PB range, and…

Oh I would love to get the UH Cancer Center data into Quilt! Do you happen to have an S3 bucket with that data live? If the bucket is publicly permissioned it should "just work." We can talk about indexing the data for search. We are comfortable in the TB-PB range :) I will look more closely at Ohayo.

"At the UH Cancer Center we routinely deal with datasets in the TB - PB range ..."

...

"Do you happen to have an S3 bucket with that data live?"

As someone not working in academia (or in this field at all) can you help me understand the question you have just asked ?

Specifically, wouldn't it be tremendously profligate for them to have that PB range dataset living in S3 ?

Given the resources that a university has (in both Internet2 connectivity, hardware budget and (relatively) cheap manpower), why would they ever store that data outside of their own UH datacenter ?

If the answer is "offsite backup" wouldn't it be glacier or nearline or ... anything but S3 ?

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#37

Post by kevinemoore »

rabidrat wrote: post #36

Is there a way to get a "dataset of datasets"? That is, all datasets you have, in downloadable tabular form with metadata for each dataset?

Hmm, yes and no. It's certainly possible to nest a package (dataset) inside a package. But, there's no (current) API that returns the list of datasets as a table/DataFrame. How would you imagine using that feature?

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#38

Post by akarve »

rabidrat wrote: post #36

Is there a way to get a "dataset of datasets"? That is, all datasets you have, in downloadable tabular form with metadata for each dataset?

The package landing page is essentially that: a list of all of your datasets. This is constructed from the special s3://.quilt/* directory. Since a Quilt manifest is a list of keys, you can compose N manifests into a single manifest using the API, see e.g. https://docs.quiltdata.com/advanced-usage/working-with-manif...

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#39

Post by rabidrat »

kevinemoore wrote: post #37
rabidrat wrote: post #36

Is there a way to get a "dataset of datasets"? That is, all datasets you have, in downloadable tabular form with metadata for each dataset?

Hmm, yes and no. It's certainly possible to nest a package (dataset) inside a package. But, there's no (current) API that returns the list of datasets as a table/DataFrame. How would you imagine using that feature?

I use local tools for all my data processing, and if I want to search for a relevant dataset, I'd like to do it locally instead of having to contact an API and do all the paging etc. The kinds of queries that might be important to me are outside the scope of any fixed API; I might want all datasets updated in December (of any year), or only those with between 100k and 1m rows, or to get a frequency table of datasets by license terms. It's very easy to do any of these queries if I can download a TSV (or equivalent) with all the metadata, and usually too frustrating to even attempt if I can't.

Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3

#40

Post by kevinemoore »

rabidrat wrote: post #39
kevinemoore wrote: post #37

Earlier quoted context omitted.

Hmm, yes and no. It's certainly possible to nest a package (dataset) inside a package. But, there's no (current) API that returns the list of datasets as a table/DataFrame. How would you imagine using that feature?

I use local tools for all my data processing, and if I want to search for a relevant dataset, I'd like to do it locally instead of having to contact an API and do all the paging etc. The kinds of queries that might be important to me are outside the scope of any fixed API; I might want all datasets updated in December (of any year), or only those with between 100k and 1m rows, or to get a frequency table of datasets…

Maybe the Elasticsearch cluster would help you here. Because all the datasets (and files within datasets) are indexed along with their metadata, you could write elastic queries to find the datasets you want--as long as the dataset creators are including the relevant metadata in the dataset annotations.
Post reply on HN