Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 0.8

duckdb.org

Post reply on HN
Print view View original

31–40 of 101 posts

Re: DuckDB 0.8

#31

Post by rendaw »

1egg0myegg0 wrote: post #22
mkhattab wrote: post #21

Minor annoyance: Yesterday I was reading the docs and one of the first things I tried is the following, copy and pasted, which didn't work, using version 0.71: COPY (SELECT 42 AS a, 'hello' AS b) TO 'query.json' (FORMAT JSON, ARRAY TRUE); >>> Error: Binder Error: Unrecognized option CSV writer "array" Just tried the above again with this release and it's fixed now. Ideally, all commands listed in the documentation sh…

Howdy! The docs reflect the latest dev branch. For a specific release, you can pick it from the version dropdown in the top right!

Why not show the latest stable release docs by default? I assume that's what most people would use.

Re: DuckDB 0.8

#33

Post by theLiminator »

emmanueloga_ wrote: post #11

Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…

Imo lazy dataframe libraries (like polars or datafusion) give you most of what you need in OLAP SQL but without the pain.

Re: DuckDB 0.8

#34

Post by AbuAssar »

> DuckDB is an in-process SQL OLAP databasemanagement system

> In-process, serverless C++11, no dependencies, single file build APIs for Python/R/Java/…

> Transactions, persistence Extensive SQL support Direct Parquet & CSV querying

> Vectorized engine Optimized for analytics, Parallel query processing

> Free & Open Source Permissive MIT License

Re: DuckDB 0.8

#35

Post by paulddraper »

de6u99er wrote: post #13
TACIXAT wrote: post #6

I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…

- DuckDB is written in C++ - SQLite is written in C I wouldn't consider any of those written in a memory safe language. Although SQLite has been battle hardened over many years, while DuckDB is a relatively new project. That being said, has been efforts of reimplementing SQLite in a more memory safe language like Rust. e.g. https://github.com/epilys/rsqlite3

to be clear, OP said

> I just don't understand why someone would start something in a memory unsafe language these days.

It takes a lot of time and testing to iron out all the bugs. Not impossible. Just takes a lot of time and testing.

Re: DuckDB 0.8

#36

Post by TACIXAT »

vamega wrote: post #18
TACIXAT wrote: post #6

I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…

What feature was removed? I'm pretty fine with a project deciding to remove a feature with known footguns, but curious what it was.

Blob to bitstring type casting for Parquet. They were doing a straight reinterpret cast on it which was causing an allocation of 18446744073709551503 bytes.

I was wanting to take a blob from Parquet and bitwise-and it against a bitstring in memory.

Re: DuckDB 0.8

#37

Post by paulddraper »

emmanueloga_ wrote: post #11

Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…

> Setting up SQL

You'll have to set up something.

> creating a well-designed schema

You'll have to design your schema well. (Or else.)

> optimizing queries can be a complex, multi-step process.

You'll have to optimize your queries. (Or else.)

---

I don't see why SQL is more owrk than the alternative.

Re: DuckDB 0.8

#38

Post by TACIXAT »

1egg0myegg0 wrote: post #10
TACIXAT wrote: post #6

I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…

Sorry to hear that! (I work on docs for the DuckDB Foundation) Starting in this release, the DuckDB team invested significantly in adding memory safety throughout catalog operations. There is more on the roadmap, but I would expect this release and all following to have improved stability! That said, at my primary company, we have used it in production for years now with great success!

Yea, legit a cool project and if it wasn't on customer machines and being passed user-defined SQL it would have just been worked around.

Re: DuckDB 0.8

#39

Post by noobcoder »

The feature to register arbitrary Python functions as scalar functions within SQL queries is pretty insane. Writing custom functions to manipulate data in ways that aren't possible with SQL, basically without fetching all of it at once in memory and then applying. I hope thats the case

Re: DuckDB 0.8

#40

Post by meitham »

I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is the case sensitivity, where in DuckDB if you ask for your queries to be case insensitive, your results lose their original casing and returned all lowercased.
Post reply on HN