Fast drilldown dashboards from a single Parquet file
hamiltonulmer.com
Fast drilldown dashboards from a single Parquet file
1–10 of 25 posts
Re: Fast drilldown dashboards from a single Parquet file
#2Great read!
Re: Fast drilldown dashboards from a single Parquet file
#3A clever repurposing of technologies but realistically only worthwhile for static datasets with range payloads small enough to fit into a web response.
> your pipeline has to rebuild each customer’s file fast enough to meet the update cadence. ... data that updates on a coarse schedule rather than in realtime
Re: Fast drilldown dashboards from a single Parquet file
#4Interesting how noise complaints dwarf any other type of complaint in NYC.
Re: Fast drilldown dashboards from a single Parquet file
#5A clever repurposing of technologies but realistically only worthwhile for static datasets with range payloads small enough to fit into a web response. > your pipeline has to rebuild each customer’s file fast enough to meet the update cadence. ... data that updates on a coarse schedule rather than in realtime
"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads.
I expect that if your overall data is less than a GB this trick will work really well for you.
Re: Fast drilldown dashboards from a single Parquet file
#6A clever repurposing of technologies but realistically only worthwhile for static datasets with range payloads small enough to fit into a web response. > your pipeline has to rebuild each customer’s file fast enough to meet the update cadence. ... data that updates on a coarse schedule rather than in realtime
It doesn't have to all live in the same Parquet file. you can have a Parquet file for all your historical data, plus one for the current week which is updated often cheaply, and then when the week is over you merge that into your big parquet file.
You're making it seem like there's hard limits to what can be done but while there definitely is, you can do incredible stuff.
Re: Fast drilldown dashboards from a single Parquet file
#7> The bytes pass through a small Cloudflare Worker on the way, because the free r2.dev URL is rate-limited.
For a 40MB file I suggest hosting it directly on GitHub Pages - that's effectively a free CORS-enabled CDN and supports HTTP range requests, so you should be able to get that demo working without needing to involve Cloudflare Workers at all.
Re: Fast drilldown dashboards from a single Parquet file
#8> A dashboard like this one is designed to answer a bounded set of analytical questions ~ requests per day, requests per day for one agency, all-time totals by borough. Each question can be answered by GROUP BY queries, so we can precompute them all ahead of time and save each result as its own small table, called a grouping set. Stack all of the grouping sets in one Parquet file, one section per set, and you have a data cube. A grouping set is only useful if it either enables a question to be answered, or reduces the latency of pulling the data.
What's the benefit of "data cubes" over caching?
Re: Fast drilldown dashboards from a single Parquet file
#9> The bytes pass through a small Cloudflare Worker on the way, because the free r2.dev URL is rate-limited. For a 40MB file I suggest hosting it directly on GitHub Pages - that's effectively a free CORS-enabled CDN and supports HTTP range requests, so you should be able to get that demo working without needing to involve Cloudflare Workers at all.
Agreed, for a public demo like this one, GitHub Pages would work great (or any host that speaks HTTP range requests with CORS). I used R2 partly because I wanted to see how it behaved, and partly because the real use-case doesn't fit Pages. The source data already lives on R2 as Iceberg, the files are per-customer and would probably need auth (signed URLs or a session-checking Worker), and obviously 10k customer cubes on a schedule works better with object PUTs rather than git deploys
Re: Fast drilldown dashboards from a single Parquet file
#10> A dashboard like this one is designed to answer a bounded set of analytical questions ~ requests per day, requests per day for one agency, all-time totals by borough. Each question can be answered by GROUP BY queries, so we can precompute them all ahead of time and save each result as its own small table, called a grouping set. Stack all of the grouping sets in one Parquet file, one section per set, and you have a…
It's a good question. In a sense, the cube is caching, just materialized ahead of time instead of memoized on demand. A result cache still needs a live database behind it for misses; the cube has no misses, since every question the dashboard is designed to answer has data in the cube already. And for this experiment, the goal was to forgo a database to serve the data in the first place.
I provide caveats for when this would work vs. when it doesn't in the post. For a lot of customer-facing dashboards, I think it's probably pretty good.