Show HN: DemandMap – Memory Mapping S3 into Polars on macOS Without FUSE

Basically this exists to solve one of my biggest pain-points with dataframe libraries and big data. It's basically impossible to lazily download data on S3 with polars unless you're processing it strictly sequentially, and even then, it won't work with arrow and numpy data.

This allows you to "download" a multi-gigabyte file on S3 into a Polars DataFrame in # Demo

    alloc = demandmap.S3Alloc(
        "./cache.bin",
        # number of blocks
        capacity=512,
        # one megabyte block (per request chunk size)
        block_size=1048576
    )

    buf1 = alloc.get(S3_PATH)
    # Big file
    assert buf1.nbytes > 400000000
    # But this takes ~100ms
    df = pl.DataFrame([buf1])

It's one of those problems that manages to be both simple and difficult. I'd wager I'd really like to build a cross platform user faulting library covering Linux and Windows too, because nearly everyone who's touched a dataframe has had this exact problem.

Show HN: DemandMap – Memory Mapping S3 into Polars on macOS Without FUSE
github.com