Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast drilldown dashboards from a single Parquet file

hamiltonulmer.com

Post reply on HN
Print view View original

11–20 of 25 posts

Re: Fast drilldown dashboards from a single Parquet file

#11

Post by nicoritschel »

simonw wrote: post #5
mrbluecoat wrote: post #3

A clever repurposing of technologies but realistically only worthwhile for static datasets with range payloads small enough to fit into a web response. > your pipeline has to rebuild each customer’s file fast enough to meet the update cadence. ... data that updates on a coarse schedule rather than in realtime

"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads. I expect that if your overall data is less than a GB this trick will work really well for you.

Most reporting I've ever worked on is based on live data and users expect updates. Range requests over parquet cubes are a cool party trick, but you outgrow it quickly once you need to start regularly updating the dataset such as to avoid full recompuation.

The next step in this journey is Iceberg (and a proper incremental pipeline), which can also be read directly in the browser via WASM either via DuckDB or without. This is from the same author as the parquet library mentioned in the OP https://github.com/hyparam/icebird

Re: Fast drilldown dashboards from a single Parquet file

#12

Post by nicoritschel »

simlevesque wrote: post #6
mrbluecoat wrote: post #3

A clever repurposing of technologies but realistically only worthwhile for static datasets with range payloads small enough to fit into a web response. > your pipeline has to rebuild each customer’s file fast enough to meet the update cadence. ... data that updates on a coarse schedule rather than in realtime

It doesn't have to all live in the same Parquet file. you can have a Parquet file for all your historical data, plus one for the current week which is updated often cheaply, and then when the week is over you merge that into your big parquet file. You're making it seem like there's hard limits to what can be done but while there definitely is, you can do incredible stuff.

And then you eventually just take the weekly files and combine them and you've reinvented data lakes with worse (no) metadata management

Re: Fast drilldown dashboards from a single Parquet file

#13

Post by hamilton »

nicoritschel wrote: post #11
simonw wrote: post #5

Earlier quoted context omitted.

"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads. I expect that if your overall data is less than a GB this trick will work really well for you.

Most reporting I've ever worked on is based on live data and users expect updates. Range requests over parquet cubes are a cool party trick, but you outgrow it quickly once you need to start regularly updating the dataset such as to avoid full recompuation. The next step in this journey is Iceberg (and a proper incremental pipeline), which can also be read directly in the browser via WASM either via DuckDB or without…

I think it depends on (1) customer expectations for freshness and (2) scale (both for the cubes and for the customer data in toto). There are many types of customer facing dashboards where giving "live data" is a bad idea for them and for you. And recomputation is indeed a problem, but if the volume of data isn't that high to begin with, it's probably easier than setting up an incremental pipeline architecture vs. a grouping set query in DuckDB. But I am not really a data engineer, so perhaps this is naive.

Re: Fast drilldown dashboards from a single Parquet file

#14

Post by wyck »

On the surface (I haven't tested it) it looks like a great cost and runtime saver, but only for data sets that need a cadence above 5 or so minutes. You wouldn't be able to have a refresh button to get the "latest" data outside this window, depending on size and build time? Wondering if you can apply a hybrid approach, combining the historical parquet file with a live query.

Re: Fast drilldown dashboards from a single Parquet file

#15

Post by masterj »

wyck wrote: post #14

On the surface (I haven't tested it) it looks like a great cost and runtime saver, but only for data sets that need a cadence above 5 or so minutes. You wouldn't be able to have a refresh button to get the "latest" data outside this window, depending on size and build time? Wondering if you can apply a hybrid approach, combining the historical parquet file with a live query.

Cloudflare Pipelines has a configurable interval to write to parquet files that's as low as 10s: https://developers.cloudflare.com/pipelines/sinks/available-... so you could have a pretty fast refresh

It will cost (slightly) more to write this frequently to R2 since you are charged per-write, but this is something you can tune.

Re: Fast drilldown dashboards from a single Parquet file

#17

Post by wlessard_ca »

How is this any different from the old OLAP cubes like IRL / Express? The cube is basically an array in memory and super fast, the dashboards based on Group By as well. I was an Express consultant at Oracle, great to see old concepts come back to life if it’s what I think it is.

Re: Fast drilldown dashboards from a single Parquet file

#18

Post by closeparen »

nicoritschel wrote: post #11
simonw wrote: post #5

Earlier quoted context omitted.

"static datasets with range payloads small enough to fit into a web response" fits a lot of workloads. I expect that if your overall data is less than a GB this trick will work really well for you.

Most reporting I've ever worked on is based on live data and users expect updates. Range requests over parquet cubes are a cool party trick, but you outgrow it quickly once you need to start regularly updating the dataset such as to avoid full recompuation. The next step in this journey is Iceberg (and a proper incremental pipeline), which can also be read directly in the browser via WASM either via DuckDB or without…

I work at a household name tech company and pretty much all our "data" dashboards are running SQL queries against a Hive/Presto environment that's at least 24 hours behind. Real-time dashboards are limited to the service metrics time series database (so combinations of categorical variables, no individual records) or rare expensive Pinot tables.

Re: Fast drilldown dashboards from a single Parquet file

#19

Post by nicoritschel »

closeparen wrote: post #18
nicoritschel wrote: post #11

Earlier quoted context omitted.

Most reporting I've ever worked on is based on live data and users expect updates. Range requests over parquet cubes are a cool party trick, but you outgrow it quickly once you need to start regularly updating the dataset such as to avoid full recompuation. The next step in this journey is Iceberg (and a proper incremental pipeline), which can also be read directly in the browser via WASM either via DuckDB or without…

I work at a household name tech company and pretty much all our "data" dashboards are running SQL queries against a Hive/Presto environment that's at least 24 hours behind. Real-time dashboards are limited to the service metrics time series database (so combinations of categorical variables, no individual records) or rare expensive Pinot tables.

Perhaps changing is a better word.

Yes this was also my experience working at a large tech co. I work in fintech now and data volumes are low enough to maintain 2-3 minute up to a few hour data freshness.

Re: Fast drilldown dashboards from a single Parquet file

#20

Post by hamilton »

wlessard_ca wrote: post #17

How is this any different from the old OLAP cubes like IRL / Express? The cube is basically an array in memory and super fast, the dashboards based on Group By as well. I was an Express consultant at Oracle, great to see old concepts come back to life if it’s what I think it is.

Indeed, OLAP cubes keep finding new life. The critical difference here is that there isn't a heavy query engine (just a Parquet reader) and a few http requests to pull from the cube. So it's light on the backend and light on the frontend.
Post reply on HN