Earlier quoted context omitted.
The list of datasets is pretty dynamic. Would an API call work as a URL? (assuming it could return a CSV/TSV/etc.)?
Yes! That is ideal.
We'll look into that for sure!
Launch HN: Quilt (YC W16) – A versioned data portal for S3
51–60 of 65 posts
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#52> Try searching for anything on https://open.quiltdata.com/ and let us know how search works for you.
I suggest adding the possibility of searching for exact matches with quotation marks, and also to ensure that it works with the quotation marks that the default keyboard on iOS has.
For example, I want to search for “Irish Setter” and only see results that include those two words next to each other like that.
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#53> Try searching for anything on https://open.quiltdata.com/ and let us know how search works for you. I suggest adding the possibility of searching for exact matches with quotation marks, and also to ensure that it works with the quotation marks that the default keyboard on iOS has. For example, I want to search for “Irish Setter” and only see results that include those two words next to each other like that.
The search is powered by Elasticsearch. We want to support the full power of elastic’s query language, but so far haven’t found a great way to prevent overly intense (or malicious) queries and still allow arbitrary queries—love to hear from Elastic experts out there. Exact match on multiple words seems possible though.
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#54> Try searching for anything on https://open.quiltdata.com/ and let us know how search works for you. I suggest adding the possibility of searching for exact matches with quotation marks, and also to ensure that it works with the quotation marks that the default keyboard on iOS has. For example, I want to search for “Irish Setter” and only see results that include those two words next to each other like that.
You can do that :) See here for search syntax https://www.elastic.co/guide/en/elasticsearch/reference/6.8/... (there is a faint link under the big search bar). Let us know how that works for you.
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#55Earlier quoted context omitted.
Seems like it'd be more elegant (and probably cost effective) if you stored the Lucene indexes inside the buckets themselves.
That is an interesting idea. What kind of performance could we expect, especially in the federated case of searching multiple buckets? Elastic has sub-second latency (at the cost of running dedicated containers).
That's a bit of an open question right now, unfortunately. Using S3 to store Lucene indexes is a roll-your-own thing since last I checked, and the implementation I wrote currently deals with smaller indexes where files can be pulled to fully disk as needed. S3 does support range requests, which I'd think would mimic random access well enough.
Assuming whatever ElasticSearch implementation you're using is backed by SSDs there'd likely be more latency with S3, but I'd expect it to scale pretty well. Internally, a Lucene index is an array of immutable self-contained segment files that store all indices for particular documents. Searching in multiple indices is pretty much just searching through all their segments- which can be as parallel as you want it to be.
To be honest, I'm actually surprised the Elasticsearch company doesn't offer this as an option. Maybe because they sell hardware at markup?
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#56Congratulations to the Quilt team on the launch! Quilt reached out to me and suggested I chime in suggesting that people interested in versioning data also check out Dolt ( https://github.com/liquidata-inc/dolt ) and DoltHub ( https://www.dolthub.com ). We've taken the Git and GitHub for data analogy a lot more literally than Quilt has :-) We are a SQL database with native Git semantics. Instead of versioning files l…
Is what you're describing something like transaction time / "AS OF SYSTEM TIME"? From a different starting point, so to speak.
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#57Is there any tool in this space that also handles permissions e.g. per column or table?
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#58Earlier quoted context omitted.
Thanks Tim! I definitely second your observation that there's room and reason for plenty of tools in this space. DVC probably belongs in your list too: https://github.com/iterative/dvc . Looking forward to checking out Open Images.
The coolest thing to do is diff between the V2 and V3 branch for a label_descriptions table. https://www.dolthub.com/repositories/Liquidata/open-images/c... You can start to see the power of column-wise diffs. You can start to imagine what it would be like to change this table and then merge Google's changes in V3 onto your modified copy. Very powerful. We need a query interface on top of diffs. Lots to build...
Great work on Dolt and seeing temporal data stores emerging :-)
Regarding XML and JSON SirixDB[1] already provides full blown time-travel queries using a fork of Brackit[2], that is basically XQuery to process and query both the XML as well JSON documents. That said SirixDB in principal could also store relational data or graph data. The storage engine has been built from scratch to offer the best possible versioning capabilities. However I've never implemented branching/merging as I didn't come up with good use cases. It seems it's then always more of a versioning system like Git, but more fine granular.
I always struggled to implement this as SirixDB currently only allows a single read-write transaction on a resource. Thus, if it would support branching and merging users would have to manually handling conflicts when merging (or automatically -- using a merge-strategy which is often case not good).
There's however plently of optimization potential, as SirixDB optionally stores a lot of metadata for each node (number of descendants, a rolling hash, Dewey-IDs, number of children... as well as user-defined, typed secondary index-structures). I'll have to look how to build AST rewrite rules and implement a lot of optimizations into my Brackit binding in the future, so it's just the starting point (but everything should at least work already) :-)
[1] https://sirix.io and https://github.com/sirixdb/sirix
[2] http://wwwlgis.informatik.uni-kl.de/cms/fileadmin/publicatio...
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#59Excited to see this being re-launched. "git for data" ranks pretty high on my all time list of tech I want to see succeed. I find the business model very interesting: A kind of "middle layer" SAAS, where you provide a new front-end for an existing service. Not seen that very often. Certainly helps with the data privacy issues. Rapid on-boarding is another immediate benefit.
Just determining the shape of "git for data" has been a nontrivial exercise. We found that, if you do the naive translation, you get a "one size fits none," because data and code are fundamentally different. What would be your main use cases with said "git for data"?
One use-case is this one: https://github.com/HeinrichHartmann/arxiv_meta
Re: Launch HN: Quilt (YC W16) – A versioned data portal for S3
#60Really excited to see this relaunched. Every DS team has issues around dataset management. We previously shared a tutorial on how to get a fully reproducible pipeline with Quilt + Comet.ml https://blog.quiltdata.com/building-a-fully-reproducible-mac...