Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fast drilldown dashboards from a single Parquet file

hamiltonulmer.com

Post reply on HN
Print view View original

21–25 of 25 posts

Re: Fast drilldown dashboards from a single Parquet file

#21

Post by plaingrid »

closeparen wrote: post #18
nicoritschel wrote: post #11

Earlier quoted context omitted.

Most reporting I've ever worked on is based on live data and users expect updates. Range requests over parquet cubes are a cool party trick, but you outgrow it quickly once you need to start regularly updating the dataset such as to avoid full recompuation. The next step in this journey is Iceberg (and a proper incremental pipeline), which can also be read directly in the browser via WASM either via DuckDB or without…

I work at a household name tech company and pretty much all our "data" dashboards are running SQL queries against a Hive/Presto environment that's at least 24 hours behind. Real-time dashboards are limited to the service metrics time series database (so combinations of categorical variables, no individual records) or rare expensive Pinot tables.

I have seen this on the business side too. A lot of we need live data requests don’t really need live data. If the decision is made once a day, hourly data is more than enough. Yet we still build dashboards that refresh every few minutes just because people asked for it.

Re: Fast drilldown dashboards from a single Parquet file

#22

Post by nicoritschel »

plaingrid wrote: post #21
closeparen wrote: post #18

Earlier quoted context omitted.

I work at a household name tech company and pretty much all our "data" dashboards are running SQL queries against a Hive/Presto environment that's at least 24 hours behind. Real-time dashboards are limited to the service metrics time series database (so combinations of categorical variables, no individual records) or rare expensive Pinot tables.

I have seen this on the business side too. A lot of we need live data requests don’t really need live data. If the decision is made once a day, hourly data is more than enough. Yet we still build dashboards that refresh every few minutes just because people asked for it.

In this case, we do actually need live data! We use wide events for monitoring business events and it lets us understand our platform with utmost flexibility, which is important for us as a fintech company.

Traditional observability is ill-suited for observability around business events. What if you forget to instrument a counter or gauge for something? In my experience it's far easier to log wide events with as much context as possible instead of agonizing over anticipating the dimensionality of metrics upfront (you're going to miss something).

Re: Fast drilldown dashboards from a single Parquet file

#23

Post by MattBlissett »

youngtaff wrote: post #16

I’d loved to have seen more detail on how the Parquet file was actually built — are there any good resources that cover this?

With the GROUPING SETS or CUBE features in DuckDB, I think.

https://duckdb.org/docs/current/sql/query_syntax/grouping_se...

Re: Fast drilldown dashboards from a single Parquet file

#24

Post by owebmaster »

hamilton wrote: post #9
simonw wrote: post #7

> The bytes pass through a small Cloudflare Worker on the way, because the free r2.dev URL is rate-limited. For a 40MB file I suggest hosting it directly on GitHub Pages - that's effectively a free CORS-enabled CDN and supports HTTP range requests, so you should be able to get that demo working without needing to involve Cloudflare Workers at all.

Agreed, for a public demo like this one, GitHub Pages would work great (or any host that speaks HTTP range requests with CORS). I used R2 partly because I wanted to see how it behaved, and partly because the real use-case doesn't fit Pages. The source data already lives on R2 as Iceberg, the files are per-customer and would probably need auth (signed URLs or a session-checking Worker), and obviously 10k customer cube…

> Agreed, for a public demo like this one, GitHub Pages would work great

If you are okay with it being down regularly

Edit: Ironically, that would be the case now: https://www.githubstatus.com/incidents/hcbtzksccj2f

Re: Fast drilldown dashboards from a single Parquet file

#25

Post by plaingrid »

nicoritschel wrote: post #22
plaingrid wrote: post #21

Earlier quoted context omitted.

I have seen this on the business side too. A lot of we need live data requests don’t really need live data. If the decision is made once a day, hourly data is more than enough. Yet we still build dashboards that refresh every few minutes just because people asked for it.

In this case, we do actually need live data! We use wide events for monitoring business events and it lets us understand our platform with utmost flexibility, which is important for us as a fintech company. Traditional observability is ill-suited for observability around business events. What if you forget to instrument a counter or gauge for something? In my experience it's far easier to log wide events with as much…

[dead]
Post reply on HN