Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Streaming gigabyte medical images from S3 without downloading them

github.com

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#31

Post by nszceta »

iberator wrote: post #19
tokyovigilante wrote: post #7

This is really a job for JPEG-XL, which supports decode of portions of larger images and has recently been added to the DICOM standard.

No. Jpg conpression sucks. Medical data should not be compressed loosely. PNG and TIFF for the win

The original JPEG supports a lossless mode.

JPEG-LL refers to the lossless mode of the original JPEG standard (ISO/IEC 10918-1 or ITU-T T.81), also known as JPEG Lossless, and not to be confused with JPEG-LS (ISO/IEC 14495-1, Transfer Syntax 1.2.840.10008.1.2.4.80), which offers better ratios and speed via LOCO-I algorithm. JPEG-LL is older and less efficient yet more widely implemented in legacy systems.

The lossless mode in JPEG-XL is superior to all of those.

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#32

Post by el_pa_b »

tomnicholas1 wrote: post #25

The generalized form of this range-request-based streaming approach looks something like my project VirtualiZarr [0]. Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contiguous multidimensional array(/"tensor") chunks embedded alongside metadata about what's in the chunks. Efficiently fetching these from object storage is just about efficiently fetching t…

Thanks for sharing! I agree that newer scientific formats will need to deeply think about how they are deciphered directly from cloud storage.

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#33

Post by el_pa_b »

mlhpdx wrote: post #30

A while back I worked on a project where s3 held giant zip files containing zip files (turtles all the way down) and also made good use of range requests. I came up with seekable-s3-stream[1] to generalize working with them via an idiomatic C# stream. [1] https://github.com/mlhpdx/seekable-s3-stream

Nice!

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#34

Post by [deleted] »

tomnicholas1 wrote: post #25

The generalized form of this range-request-based streaming approach looks something like my project VirtualiZarr [0]. Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contiguous multidimensional array(/"tensor") chunks embedded alongside metadata about what's in the chunks. Efficiently fetching these from object storage is just about efficiently fetching t…

[deleted]

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#35

Post by tomnicholas1 »

el_pa_b wrote: post #32
tomnicholas1 wrote: post #25

The generalized form of this range-request-based streaming approach looks something like my project VirtualiZarr [0]. Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contiguous multidimensional array(/"tensor") chunks embedded alongside metadata about what's in the chunks. Efficiently fetching these from object storage is just about efficiently fetching t…

Thanks for sharing! I agree that newer scientific formats will need to deeply think about how they are deciphered directly from cloud storage.

IMO Zarr is that newer format. It abstracts over the features of all these other formats so neatly that it can literally subsume them.

I feel that we no longer really need TIFF etc. - for scientific use cases in the cloud Zarr is all that's needed going forwards. The other file formats become just archival blobs that either are converted to Zarr or pointed at by virtual Zarr stores.

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#37

Post by bwfan123 »

tomnicholas1 wrote: post #35
el_pa_b wrote: post #32

Earlier quoted context omitted.

Thanks for sharing! I agree that newer scientific formats will need to deeply think about how they are deciphered directly from cloud storage.

IMO Zarr is that newer format. It abstracts over the features of all these other formats so neatly that it can literally subsume them. I feel that we no longer really need TIFF etc. - for scientific use cases in the cloud Zarr is all that's needed going forwards. The other file formats become just archival blobs that either are converted to Zarr or pointed at by virtual Zarr stores.

thanks for sharing !

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#38

Post by derefr »

tomnicholas1 wrote: post #25

The generalized form of this range-request-based streaming approach looks something like my project VirtualiZarr [0]. Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contiguous multidimensional array(/"tensor") chunks embedded alongside metadata about what's in the chunks. Efficiently fetching these from object storage is just about efficiently fetching t…

Sounds like an approach that would also work for ML model weights files — just another kind of multidimensional array with metadata.

I wonder what exactly the big multi-model AI companies are doing to optimize model cold-start latency, and how much it just looks like Zarr on top of on-prem object storage.

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#39

Post by PunchyHamster »

thenaturalist wrote: post #13
andrewstuart wrote: post #12

Please don’t use AWS S3 there’s vast numbers of much cheaper compatible choices.

Pretty bold half claim while not backing it up with a single data point. :D

It's trivial to find and there are many alternatives.

Main problem is most support subset of the more advanced S3 features and often not all that big one. But if you just want to dump some backups in the cloud backblaze and other alternatives is cheaper

Re: Show HN: Streaming gigabyte medical images from S3 without downloading them

#40

Post by tomnicholas1 »

derefr wrote: post #38
tomnicholas1 wrote: post #25

The generalized form of this range-request-based streaming approach looks something like my project VirtualiZarr [0]. Many of these scientific file formats (HDF5, netCDF, TIFF/COG, FITS, GRIB, JPEG and more) are essentially just contiguous multidimensional array(/"tensor") chunks embedded alongside metadata about what's in the chunks. Efficiently fetching these from object storage is just about efficiently fetching t…

Sounds like an approach that would also work for ML model weights files — just another kind of multidimensional array with metadata. I wonder what exactly the big multi-model AI companies are doing to optimize model cold-start latency, and how much it just looks like Zarr on top of on-prem object storage.

People have literally used Zarr for this - at one point Gemini used Zarr for checkpointing model weights. Not sure what the current fashion in that space is though.

It's definitely one of many fields that see convergent evolution towards something that just looks like Zarr. In fact you can use VirtualiZarr to parse HuggingFace's "SafeTensors" format [0].

[0]: https://github.com/zarr-developers/VirtualiZarr/pull/555

Post reply on HN