21–30 of 54 posts
Earlier quoted context omitted.
Can confirm: together with `dbt` and `rill` I'm able do to [this]( https://github.com/idesis-gmbh/GitHubExperiments/blob/master... ) on my laptop.
is rill open source?
duckdb is a generational technology innovation. insanely good ergonomics, great performance, it's awesome.
I got introduced to it by Claude the other day as I was interrogating several GB of public csv files. Seemed magical as it out them all in parquet files and transformed what I needed into the normalized sqllite for my server. Coding agents seen quite comfortable with it!
Data engineer here: I use this all the time. It's amazing. For most of the data the sizes we often deal with it's perfect.
> For most of the data the sizes we often deal with it's perfect. Interested here: for me it works for out of core work. Where is the limit? On a related note: do you need to handle concurrency restrictions?
Any opinions on DuckLake?
It needs manual tuning to avoid those errors and I couldn’t find the right incantation, nor should I need to - memory management is the job of the db, not me. Far too flakey for any production usage.
Earlier quoted context omitted.
> For most of the data the sizes we often deal with it's perfect. Interested here: for me it works for out of core work. Where is the limit? On a related note: do you need to handle concurrency restrictions?
i must be doing something wrong but if i try a huge join on a table bigger than my ram no matter the flags or the spill-to-disk modes enabled i get crashes. im sure im doing something wrong.
I found it unusable due to out of memory errors with a billion row 8 column dataset. It needs manual tuning to avoid those errors and I couldn’t find the right incantation, nor should I need to - memory management is the job of the db, not me. Far too flakey for any production usage.
I found it unusable due to out of memory errors with a billion row 8 column dataset. It needs manual tuning to avoid those errors and I couldn’t find the right incantation, nor should I need to - memory management is the job of the db, not me. Far too flakey for any production usage.
That sounds like a rather serious application. Did you file an issue?
Search the issues of the duckdb GitHub there’s at least 110 open and closed oom (out of memory) and maybe 400 to 500 that reference “memory”.
Earlier quoted context omitted.
That sounds like a rather serious application. Did you file an issue?
No, I tried Clickhouse instead, which worked without crashing or manual memory tuning. Search the issues of the duckdb GitHub there’s at least 110 open and closed oom (out of memory) and maybe 400 to 500 that reference “memory”.