Can you share some use cases with details where I can learn more how DuckDB is doing better?
DuckDB 0.8
41–50 of 101 posts
Re: DuckDB 0.8
#42I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is th…
> results lose their original casing and returned all lowercased.
That's a shame. Have you reported this bug on their repo?
Re: DuckDB 0.8
#43I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is th…
I know this is irrelevant, and I'm really no a fan of SQL Server either, but I'm curious - did you try partitioning those tables on the batch ID, and then truncating partitions instead of deleting old rows?
Re: DuckDB 0.8
#44Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…
I thought like this until I discovered Prisma and EdgeDB. They provide all the niceties of a NoSQL / graph databases with the robustness of battle tested SQL databases
Re: DuckDB 0.8
#45I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
This would be a good blog post to read. And I’m sure the team behind the project would like to see it, too.
Re: DuckDB 0.8
#46I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is th…
I'm curious, what kind of problem needs 5 billion rows every 3 minutes to replace the previous 5 billion rows?
By "micro batch" I understand batch processing of some kind?
Re: DuckDB 0.8
#47I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is th…
I know this is irrelevant, and I'm really no a fan of SQL Server either, but I'm curious - did you try partitioning those tables on the batch ID, and then truncating partitions instead of deleting old rows?
It's a bit more complicated than I put it above. The schema is quite normalised in SQL Server and represents financial risk, with several views on it. The schema makes sense for EOD data but not for live data. The whole performance of EOD was impacted by these live insertions. Moving to another DB (on another server) would incur licence fees and costs. DuckDB simple files are cost free. Many of these files are deleted even before anyone can be bothered to read them.
Re: DuckDB 0.8
#48I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is th…
> results lose their original casing and returned all lowercased. That's a shame. Have you reported this bug on their repo?
Someone else beat me to it https://github.com/duckdb/duckdb/issues/3821
Re: DuckDB 0.8
#49I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
I remember chasing memory bugs in C# and Java projects at work. These are usually considered memory-safe languages yet their sophisticated garbage collectors are not panacea. There is a reason things like this https://docs.oracle.com/javase/8/docs/api/java/lang/ref/Weak... exist. Or why you might re-use existing objects in certain situations https://www.oreilly.com/library/view/java-performance-tuning... .
On the other hand, there are garbage collectors available for C and C++ programs (they are not part of their standard libraries so you have to choose whether you use them or not). C++ standard library has had smart pointers for some time, they existed in Boost library beforehand and RAII pattern is even older.
Don't put all the blame for memory bugs to languages. C and C++ programs are more prone to memory leaks than programs written in "memory-safe" languages but these are not safe from memory bugs either.
Disclaimer: I like C (plain C, not C++, though that's not that as bad as many people claim) and I hate soydevs.
Re: DuckDB 0.8
#50Earlier quoted context omitted.
What feature was removed? I'm pretty fine with a project deciding to remove a feature with known footguns, but curious what it was.
Blob to bitstring type casting for Parquet. They were doing a straight reinterpret cast on it which was causing an allocation of 18446744073709551503 bytes. I was wanting to take a blob from Parquet and bitwise-and it against a bitstring in memory.
Is blob an efficient type?