21–30 of 64 posts
My initial thought is why query 1TB of data in a browser, maybe I'm the wrong target audience for this but it seems that it's pushing that everything has to be in a browser rather than using appropriate tools
But, if you'd like to instead read the article, you'll see that they qualify the reasoning in the first section of the article, titled, "Rethinking the Old Trade-Off: Cost, Complexity, and Access".
I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…
I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…
what did you use instead? if you hit OOM with the dataset in duckdb, I'd think you'd hit the OOM with most other things on the same machine.
OK, this is really neat: - S3 is really cheap static storage for files. - DuckDB is a database that uses S3 for its storage. - WASM lets you run binary (non-JS) code in your browser. - DuckDB-Wasm allows you to run a database in your browser. Put all of that together, and you get a website that queries S3 with no backend at all. Amazing.
S3 might be relatively cheap for storing files, but with bandwidth you could easily be paying $230/mo. If you make it public facing & want to try to use their cloud reporting, metrics, etc. to prevent people for running up your bandwidth, your "really cheap" static hosting could easily cost you more than $500/mo.
You have to store the data somehow anyway, and you have to retrieve some of it to service a query. If egress costs too much you could always change later to put the browser code on a server. Also it would presumably be possible to quantify the trade-off between processing the data client side and on the server.
I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…
DuckDB has introduced spilling to disk and some other tweaks since a good year now: https://duckdb.org/2024/07/09/memory-management
I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…
I can recommend earlyoom ( https://github.com/rfjakob/earlyoom ). Instead of freezing or crashing your system this tool kills the memory eating process just in time (in this case duckdb). This allows you repeat with smaller chunks of the dataset, until it fits into your mem.
Have you used this in conjunction with DuckDB?
I tried DuckDB - liked it a lot - was ready to go further. But found it to be a real hassle to help it understand the right number of threads and the amount of memory to use. This led to lots of crashes. If you look at the projects github issues you will see many OOM out of memory errors. And then there was some indexed bug that crashed seemingly unrelated to memory. Life is too short for crashy database software so…
How long ago was this, or can you share more context about data and mem size you experienced this with? DuckDB has introduced spilling to disk and some other tweaks since a good year now: https://duckdb.org/2024/07/09/memory-management
The final straw was an index which generated fine on MacOS and failed on Linux - exact same code.
Machine had plenty of RAM.
The thing is, it is really the responsibility of the application to regulate its behavior based on available memory. Crashing out just should not be an option but that's the way DuckDB is built.
We have been doing it for quite some time in our product to bring real time system observability with eBPF to the browser and have even found other techniques to really max-it-out beyond what you get off the shelf.
I didn't use the in browser WASM but I did expose an api endpoint that passed data exploration queries directly to the backend like a knock off of what new relic does. I also use that same endpoint for all the graphs and metrics in the UI.
DuckDB is phenomenal tech and I love to use it with data ponds instead of data lakes although it is very capable of large sets as well.
Yesterday there was a somewhat similar DuckDB post, "Frozen DuckLakes for Multi-User, Serverless Data Access". https://news.ycombinator.com/item?id=45702831
We didn't find the frozen DuckLake setup useful for our use case. Mostly because the frozen catalog kind of doesn't make sense with the DuckLake philosophy and the cost-benefit wasn't there over a regular duckdb catalog. It also made making updates cumbersome because you need to pull the DuckLake catalog, commit the changes, and re-upload the catalog (instead of just directly updating the Parquet files). I get that we are missing the time travel part of the DuckLake, but that's not critical for us and if it becomes important, we would just roll out a PostgreSQL database to manage the catalog.