No charge for bandwidth?
No, there's no charge for bandwidth. The most common uses so far are users installing datasets locally, which caches the data at the destination or running batch jobs in ECS/EC2, which doesn't accrue charges on AWS.
Show HN: Quilt – manage data like code
81–90 of 183 posts
Re: Show HN: Quilt – manage data like code
#82Earlier quoted context omitted.
Good suggestion! From what we've heard from academic users, they'll want a DOI for a specific version, e.g., data from a particular paper or journal article. Any thing else we should watch out for?
It's problematic when data publisher != data user/paper writer. I'm not familiar enough with DOI minting and therefore don't know what issues DOI generation on large scales for miniscule changes in the data might bring. Ultimately, if I make data openly available the worst case is that every change to the data requires a new DOI as I don't know how many people have downloaded earlier versions and not published on tho…
Thanks for describing the problem. That's really interesting. We can certainly aggregate counts of downloads and installs across versions in Quilt. I'll definitely look into providing DOIs within Quilt and see if it's something we can do.
Re: Show HN: Quilt – manage data like code
#83I think it was a really interesting (and smart) choice to convert to Parquet format. Columnar storage is so much more efficient, and working with data in Parquet is pretty fast using the engines they mention (Apache Spark, Impala, Hive, etc.).
I actually had been thinking about Parquet as a component of ETL, and if it might be possible to make ETL many times faster by compressing to Parquet format on the source and then transmitting to a destination - especially when you're talking about limited bandwidth situations where you need entire data sets moved around in bulk.
This looks really nice for sharing public data sets, but I wish that there was a better public non-profit org running indexes of public data sets.... I guess if something like the semantic web had ever taken off, then the Internet itself would be the index of public data sets, but it seems like that dream is still yet to materialize.
Re: Show HN: Quilt – manage data like code
#84Hey, this sounds really interesting and I'd like to play around with it. However, I'm a novice and run into the following issue:
>>> examples.sales
No idea what a DataNode is so am struggling to actually see the data! Any tips?
Re: Show HN: Quilt – manage data like code
#85It's outrageous how little tooling support there is for version control in data compared to code. Every mainstream database forgets history with updates, don't support distributed workflows, don't support commit ids as first class objects, or most other basic features of VCSs. Databases just aren't a solution to version control. I can't imagine a future where we don't treat data version control like a necessity in th…
A lot of the time, people get confused by source control & think, “I need to refer to this data as it was at a specific point in time, I’ll use git”. Then they get all the overhead of a DVCS without getting most of the benefits (blame, bisect, branches, merge resolution) when what they wanted was versioned releases of data packages…
Re: Show HN: Quilt – manage data like code
#86I think you're missing a trick with the pricing. My guess is the real money will come once data is treated like a commodity. So the big, big money will be in brokerage's and exchanges. Paying flat fees for access to repos is fundamentally thinking about the problem incorrectly.
I think the whole project is doomed, the technology is trivially clonable and the backend is a thin API over S3. Their best bet is an acquisition by Amazon etc. to become e.g. Elastic Data Packs
Re: Show HN: Quilt – manage data like code
#87I think it was a really interesting (and smart) choice to convert to Parquet format. Columnar storage is so much more efficient, and working with data in Parquet is pretty fast using the engines they mention (Apache Spark, Impala, Hive, etc.). I actually had been thinking about Parquet as a component of ETL, and if it might be possible to make ETL many times faster by compressing to Parquet format on the source and t…
Once the interfaces are mostly settled, we plan to open-source the server so that other organizations can run Quilt registries. If you know of non-profit data-indexes that Quilt should work with or organizations who might be interested in running a Quilt registry, please let us know.
Re: Show HN: Quilt – manage data like code
#88Hey, this sounds really interesting and I'd like to play around with it. However, I'm a novice and run into the following issue: >>> examples.sales No idea what a DataNode is so am struggling to actually see the data! Any tips?
try: example.sales()
it's shorthand for: example.sales._data()
Re: Show HN: Quilt – manage data like code
#89It looks like a plain html page but requires JS to view anything except: "Please enable JavaScript to use this site." What a wonderful time to live in.
Anyway, do I get this right: They expect users to be experts in data analysis but not being able to load the data into whatever software they use? They want me to share data and to offload my data into their walled garden that can be accessed only via their service? If I wanted to share my data, wouldn't I rather use something more accessible?
Re: Show HN: Quilt – manage data like code
#90No charge for bandwidth?
No, there's no charge for bandwidth. The most common uses so far are users installing datasets locally, which caches the data at the destination or running batch jobs in ECS/EC2, which doesn't accrue charges on AWS.
Huh, that's a pretty good setup for you then! I might mess around and see about writing an R package/interface, because this looks very useful.