Viewing profile — tomnicholas1

> Data from physics, chemistry, biology experiments There are PetaBytes of important scientific data locked in archival file formats. The first step is to make this efficiently rea…

Yes! But if you want to share it with anyone else that would be great, since we're advocating for fairly radical changes within a big bureaucracy here, as I'm sure you will appreci…

Anyone interested in accessing GOES data at scale will find this interesting - I created a Zarr index over the 7 billion chunks of data in the GOES-16 archive. https://www.earthmov…

The last 2 years of my professional life have in some sense all been working towards this blog post. It shows how you can make an enormous archive of public scientific data[1] much…

Awesome to see this project here - it was partly inspired by my blog post (original is linked from the OP, but there's a slightly newer version on my personal site here[0]). [0]: h…

What you describe is very similar to how Icechunk[1] works. It works beautifully for transactional writes to "repos" containing PBs of scientific array data in object storage. [1]:…

People have literally used Zarr for this - at one point Gemini used Zarr for checkpointing model weights. Not sure what the current fashion in that space is though. It's definitely…

IMO Zarr is that newer format. It abstracts over the features of all these other formats so neatly that it can literally subsume them. I feel that we no longer really need TIFF etc…

The generalized form of this range-request-based streaming approach looks something like my project VirtualiZarr [0]. Many of these scientific file formats (HDF5, netCDF, TIFF/COG,…

God this article is 10000% better than the posted one. This is great: > Names should not describe what you currently think the thing you’re naming is for. Imagine naming your newbo…

Thank you for the explanation! But what a mess. I would love to bring these benefits to the multidimensional array world, via integration with the Zarr/Icechunk formats somehow (wh…

The pitch for this sounds very similar to the pitch for Vortex (i.e. obviating the need to create a new format every time a shift occurs in data processing and computing by providi…

The really depressing part is if you plot rate of new delays against real time elapsed, the projected finishing date is even further. This is why much of the fusion research commun…

Presumably because everyone in MCF has been waiting for ITER for decades, and JET is being decommissioned after a last gasp. Every other tokamak is considerably smaller (or similar…

I wrote the article I wish I could have read back when I first heard of Zarr and cloud-native science back in 2018. This explains how object storage and conventional filesystems ar…

Isn't that more about enumerating the microstates? The Pauli exclusion principle just ends up forbidding some of the microstates (forbidding a significant fraction of them if you'r…

Yes, that assumption is called the Ergodic Hypothesis, and generally justified in undergraduate statistical mechanics courses by proving and appealing to Liouville's theorem. [1] h…

The scientific community works primarily with array (or "tensor") data, using tools like numpy, xarray, and zarr. People familiar with modern relational database tools such as Duck…

I think the posted article was generated from this one - the structure of the content is so similar.

This entire stack also now exists for arrays as well as for tabular data. It's still S3 for storage, but Zarr instead of parquet, Icechunk instead of Iceberg, and Xarray for querie…

Icechunk can handle growing dimensions with ACID transactions! For irregular shapes in some cases using multiple groups + xarray.DataTree can help you, but in general yeah ragged d…