21–30 of 51 posts
Please don’t use AWS S3 there’s vast numbers of much cheaper compatible choices.
There are choices that speak the S3 data plane API (GetObject, ListBucket, etc).
There are no alternatives that support most of the AWS S3 functionality such as replication, event notifications.
Please don’t use AWS S3 there’s vast numbers of much cheaper compatible choices.
Edit: Looks like this is a slight discrepancy between the HN title and the GitHub description.
Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contiguous multidimensional array(/"tensor") chunks embedded alongside metadata about what's in the chunks. Efficiently fetching these from object storage is just about efficiently fetching the metadata up front so you know where the chunks you want are [1].
The data model of Zarr [2] generalizes this pattern pretty well, so that when backed by Icechunk [3], you can store a "datacube" of "virtual chunk references" that point at chunks anywhere inside the original files on S3.
This allows you to stream data out as fast as the S3 network connection allows [4], and then you're free to pull that directly, or build tile servers on top of it [5].
In the Pangeo project and at Earthmover we do all this for Weather and Climate science data. But the underlying OSS stack is domain-agnostic, so works for all sorts of multidimensional array data, and VirtualiZarr has a plugin system for parsing different scientific file formats.
I would love to see if someone could create a virtual Zarr store pointing at this WSI data!
[0]: https://virtualizarr.readthedocs.io/en/stable/
[1]: https://earthmover.io/blog/fundamentals-what-is-cloud-optimi...
[2]: https://earthmover.io/blog/what-is-zarr
[3]: https://earthmover.io/blog/icechunk-1-0-production-grade-clo...
[4]: https://earthmover.io/blog/i-o-maxing-tensors-in-the-cloud
Maybe a bit pedantic, but if you're streaming it, then you're still downloading portions of it, yah? Just not persisting the whole thing locally before viewing it. Edit: Looks like this is a slight discrepancy between the HN title and the GitHub description.
https://dicom.nema.org/dicom/dicomwsi/ Interesting guide to the Whole Slide Images (WSI) format. The surprising thing for me is that compression is used, and they note does not affect use in diagnostics. Back in the day we used TIFF for a similar application (X-ray detector images).
Digital pathology are just a lot bigger than radiology, we regularly see slides 500k x 500k pixels.
Earlier quoted context omitted.
Digital pathology are just a lot bigger than radiology, we regularly see slides 500k x 500k pixels.
Yes, they can be huge, and for modalities like multiplex immunofluorescence with up to 20 channels, you're often dealing with very faint proteomic signals. Preserving that signal is critical, and compression can destroy it quickly.