11–20 of 104 posts
There are so many infrastructure products, especially database products for some reason, where the marketing team takes control of the messaging away from engineers, and push outlandish claims on how their new DB is faster than all the competition, can support any workload, can scale infinitely, etc.
I have successfully used DuckDB like above for preparing an ML dataset from about 100GB of input.
DuckDB is undergoing rapid development these days. There have been format-breaking changes and bugs that could lose data. I would not yet trust DuckDB for long-term storage or archival purposes. Parquet is a better choice for that.
DuckDB is terrific. I'm bullish on its potential for simplifying many big data pipelines. Particularly, it's plausible that DuckDB + Parquet could be used on a large SMP machine (32+ cores and 128GB+ memory) to deal with data munging for 100s of gigabytes to several terabytes, all from SQL, without dealing with Hadoop, Spark, Ray, etc. I have successfully used DuckDB like above for preparing an ML dataset from about…
How big can you go, and how does speed compare to Spark? (I'm guessing significantly faster from my experience using Duckdb on smaller machines)
We at MotherDuck at working very closely with the DuckDB folks to build a DuckDB-based cloud service. I'm talking to various folks in the industry about the details in 1:1. Feel free to reach out to tino at motherduck.com.
(co-founder and head of Produck at MotherDuck)
I'm duck-curious. Looking at how it's deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB. EG if you deployed new code 1. Do you have a stateful machine you're doing an old school "Kill the old process, start the new process" deploy, and there's some duckdb file on disk t…
I'm duck-curious. Looking at how it's deployed, as an in process database, how do people actually use this in production? Trying to figure out where I might actually want to think about replacing current databases or analyses with DuckDB. EG if you deployed new code 1. Do you have a stateful machine you're doing an old school "Kill the old process, start the new process" deploy, and there's some duckdb file on disk t…
Podcast/interview link for anyone interested: https://www.dataengineeringpodcast.com/duckdb-in-process-ola...
DuckDB is terrific. I'm bullish on its potential for simplifying many big data pipelines. Particularly, it's plausible that DuckDB + Parquet could be used on a large SMP machine (32+ cores and 128GB+ memory) to deal with data munging for 100s of gigabytes to several terabytes, all from SQL, without dealing with Hadoop, Spark, Ray, etc. I have successfully used DuckDB like above for preparing an ML dataset from about…
I'd love to hear any real world experiences of anyone who's tried to run jobs that would usually require a spark cluster on a single machine with loads of cores and memory. How big can you go, and how does speed compare to Spark? (I'm guessing significantly faster from my experience using Duckdb on smaller machines)
Mixed experience, would definitely not put it in a system that isn’t ok crashing frequently.
It segfaults on Alpine (argh, C++!), and force exits the whole NodeJS process when it gets unexpected HTTP responses from S3.
In an archive run of 2M pushes it’ll crash the process 4-5 times.
Overall still really, really like it, but learned to not trust it
Great to see this posted here! DuckDB is an integral part of an in-browser data analytics tool that I've been working on. It compiles to WASM and runs in a web worker. Queries against WASM DuckDB regularly run 10x faster than the original JavaScript implementation!