Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use DuckDB-WASM to query TB of data in browser

lil.law.harvard.edu

Post reply on HN
Print view View original

11–20 of 64 posts

Re: Use DuckDB-WASM to query TB of data in browser

#11

Post by tuhgdetzhh »

wewewedxfgdf wrote: post #7

I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…

I can recommend earlyoom (https://github.com/rfjakob/earlyoom). Instead of freezing or crashing your system this tool kills the memory eating process just in time (in this case duckdb). This allows you repeat with smaller chunks of the dataset, until it fits into your mem.

Re: Use DuckDB-WASM to query TB of data in browser

#13

Post by wewewedxfgdf »

tuhgdetzhh wrote: post #11
wewewedxfgdf wrote: post #7

I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…

I can recommend earlyoom ( https://github.com/rfjakob/earlyoom ). Instead of freezing or crashing your system this tool kills the memory eating process just in time (in this case duckdb). This allows you repeat with smaller chunks of the dataset, until it fits into your mem.

Yeah memory and thread management is the job of the application, not me.

Re: Use DuckDB-WASM to query TB of data in browser

#14

Post by 85392_school »

jdnier wrote: post #12

Yesterday there was a somewhat similar DuckDB post, "Frozen DuckLakes for Multi-User, Serverless Data Access". https://news.ycombinator.com/item?id=45702831

This also reminded me of an approach using SQLite: https://news.ycombinator.com/item?id=45748186

Re: Use DuckDB-WASM to query TB of data in browser

#15

Post by QuantumNomad_ »

tuhgdetzhh wrote: post #11
wewewedxfgdf wrote: post #7

I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…

I can recommend earlyoom ( https://github.com/rfjakob/earlyoom ). Instead of freezing or crashing your system this tool kills the memory eating process just in time (in this case duckdb). This allows you repeat with smaller chunks of the dataset, until it fits into your mem.

When I there is a specific program I want to run with a limit on how much memory it is allowed to allocate, I have found systemd-run to work well.

It uses cgroups to enforce resource limits.

For example, there’s a program I wrote myself which I run on one of my Raspberry Pi. I had a problem where my program would on rare occasions use up too much memory and I wouldn’t even be able to ssh into the Raspberry Pi.

I run it like this:

  systemd-run --scope -p MemoryMax=5G --user env FOOBAR=baz ./target/release/myprog
The only difficulty I had was that I struggled to find the right name to use in the MemoryMax=… part because they’ve changed the name of it around between versions so different Linux systems may or may not use the same name for the limit.

In order to figure out if I had the right name for it, I tested different names for it with a super small limit that I knew was less than the program needs even in normal conditions. And when I found the right name, the program would as expected be killed right off the bat and so then I could set the limit to 5G (five gigabytes) and be confident that if it exceeds that then it will be killed instead of making my Raspberry Pi impossible to ssh into again.

Re: Use DuckDB-WASM to query TB of data in browser

#17

Post by cyanydeez »

SteveMoody73 wrote: post #16

My initial thought is why query 1TB of data in a browser, maybe I'm the wrong target audience for this but it seems that it's pushing that everything has to be in a browser rather than using appropriate tools

Browsers are now the write-once works everywhere target. Where java failed, many hope browsers succeed. WASM is definitely a key to that, particularly because it can be output by tools like rust, so they can also be the appropriate tools.

Re: Use DuckDB-WASM to query TB of data in browser

#18

Post by greatNespresso »

theultdev wrote: post #9
timeflex wrote: post #3

Earlier quoted context omitted.

S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.

R2 is S3 compatible with no egress fees. Cloudflare actually has built in iceberg support for R2 buckets. It's quite nice. Combine that with their pipelines it's a simple http request to ingest, then just point duckdb to the iceberg enabled R2 bucket to analyze.

Was about to jump in to say the same thing. R2 is a much cheaper alternative to S3 that just works and I have used it with DuckDB, works smoothly

Re: Use DuckDB-WASM to query TB of data in browser

#20

Post by majormajor »

SteveMoody73 wrote: post #16

My initial thought is why query 1TB of data in a browser, maybe I'm the wrong target audience for this but it seems that it's pushing that everything has to be in a browser rather than using appropriate tools

Why pay for RAM for servers when you can let your users deal with it? ;)

(Does not seem like a realistic scenario to me for many uses, for RAM among other resource reasons.)

Post reply on HN