Use DuckDB-WASM to query TB of data in browser • Hacker News Board

Re: Use DuckDB-WASM to query TB of data in browser

#61

Post by alex-korr » Sat, Nov 01, 2025, 5:51 PM UTC

wewewedxfgdf wrote: post #27

thenaturalist wrote: post #25

Earlier quoted context omitted.

How long ago was this, or can you share more context about data and mem size you experienced this with? DuckDB has introduced spilling to disk and some other tweaks since a good year now: https://duckdb.org/2024/07/09/memory-management

3 days ago. The final straw was an index which generated fine on MacOS and failed on Linux - exact same code. Machine had plenty of RAM. The thing is, it is really the responsibility of the application to regulate its behavior based on available memory. Crashing out just should not be an option but that's the way DuckDB is built.

I had the same experience - everything runs great on an AWS Linux EC2 with 32GB of memory, same workload in a docker on ECS with 32GB allocated gets an OOM. But for smaller workloads, DuckDB is fantastic... however, there's a certain point when Spark or Snowflake start to make more sense.

Re: Use DuckDB-WASM to query TB of data in browser

#62

Post by bzmrgonz » Sat, Nov 01, 2025, 6:53 PM UTC

This is brilliant guys, omg this is brilliant. If you think about it, freely available data always suffer with this burden... "But but we don't make money, all this stuff is public data by law, and government doesn't give us a budget". This solves that, the "can't afford it" spirit of public agencies.

Re: Use DuckDB-WASM to query TB of data in browser

#63

Post by tuhgdetzhh » Sat, Nov 01, 2025, 7:24 PM UTC

thenaturalist wrote: post #26

tuhgdetzhh wrote: post #11

Earlier quoted context omitted.

I can recommend earlyoom ( https://github.com/rfjakob/earlyoom ). Instead of freezing or crashing your system this tool kills the memory eating process just in time (in this case duckdb). This allows you repeat with smaller chunks of the dataset, until it fits into your mem.

This looks amazing! Have you used this in conjunction with DuckDB?

Yes, it works just fine.

Re: Use DuckDB-WASM to query TB of data in browser

#64

Post by victor106 » Sun, Nov 02, 2025, 3:22 PM UTC

leetrout wrote: post #60

victor106 wrote: post #58

Earlier quoted context omitted.

> data ponds instead of data lakes What are data ponds? Never heard the term before

Haha, my term. Somewhere between a data lake and warehouse - still unstructured but not _everything_ in one place. For instance, if I have a multi-tenant app I might choose to have a duckdb setup for each customer with pre-filtered data living alongside some global unstructured data. Maybe there's already a term that covers this but I like the imagery of the metaphor... "smaller, multiple data but same idea as the bi…

Got it, Thanks for the explanation.