Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 0.8

duckdb.org

Post reply on HN
Print view View original

11–20 of 101 posts

Re: DuckDB 0.8

#11

Post by emmanueloga_ »

Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results.

So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think of many things I can do with SQL that i CANNOT do easier in any programming language with a good library for indexing and retrieving data.

Any good industrial strength key-value database like rocksdb, lmdb and others would provide these APIs with lots of bindings for multiple programming languages.

Maybe we need a second coming of NoSQL (relational no-sql?) focused in providing good cross-language APIs so instead of compiling SQL into efficient programs that use the right indexes, we could just write our queries manually in code. Bad queries would be a lot easier to debug and fix this way, and it seems like this would require a lot less machinery.

Re: DuckDB 0.8

#12

Post by 1egg0myegg0 »

ChrisCinelli wrote: post #7

This is the first time I heard about DuckDB. Why someone would use that instead of a mature engine like SQLite?

Great question! For analytical queries, DuckDB is ~100x faster than SQLite. It also has great connectivity with Parquet, Pandas, Arrow, Postgres, and more! Plus, the syntax follows Postgres closely, so you have strong data typing and true timestamps, etc.

All that said - SQLite is still faster for transactions. Now DuckDB can actually read and write from/to SQLite, so you can use SQLite for OLTP and DuckDB for OLAP.

https://duckdb.org/docs/sql/statements/attach

Re: DuckDB 0.8

#13

Post by de6u99er »

TACIXAT wrote: post #6

I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…

- DuckDB is written in C++

  - SQLite is written in C
I wouldn't consider any of those written in a memory safe language. Although SQLite has been battle hardened over many years, while DuckDB is a relatively new project.

That being said, has been efforts of reimplementing SQLite in a more memory safe language like Rust.

e.g. https://github.com/epilys/rsqlite3

Re: DuckDB 0.8

#14

Post by beoberha »

emmanueloga_ wrote: post #11

Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…

The simple answer to this is that not only engineers write database queries.

Personally, I’d be interested to see a system like the one you describe. Part of me thinks that SQL would still be simpler to reason about (though probably not to maintain).

Re: DuckDB 0.8

#15

Post by Icathian »

emmanueloga_ wrote: post #11

Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…

I think you are drastically overestimating the ability of most devs to write performant code. Do you really think you're going to get better results asking devs who, by your measure cannot write good declarative SQL schemas and queries, to roll their own data storage and retrieval raw?

RDBMS software is complicated in large part due to it needing to serve a wide range of use cases, so a custom-built version might end up simpler to use, but you can't sell me that Joe Backend-Dev is going to outperform however many million dev-hours have gone into Postgres.

Re: DuckDB 0.8

#16

Post by fbdab103 »

1egg0myegg0 wrote: post #12
ChrisCinelli wrote: post #7

This is the first time I heard about DuckDB. Why someone would use that instead of a mature engine like SQLite?

Great question! For analytical queries, DuckDB is ~100x faster than SQLite. It also has great connectivity with Parquet, Pandas, Arrow, Postgres, and more! Plus, the syntax follows Postgres closely, so you have strong data typing and true timestamps, etc. All that said - SQLite is still faster for transactions. Now DuckDB can actually read and write from/to SQLite, so you can use SQLite for OLTP and DuckDB for OLAP.…

>Now DuckDB can actually read and write from/to SQLite

This feels like it opens up some possibilities, but I am struggling to figure out where. Are you aware of any interesting use cases this has enabled? Does all of the Duck syntax then work against the SQLite database? For example, could I now run a pivot without a bunch of hoops?

Re: DuckDB 0.8

#17

Post by Vt71fcAqt7 »

emmanueloga_ wrote: post #11

Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…

>relational no-sql

Do you mean something like edgeDB?[0]

Or do you mean some non-declarative language completely? I don't see the latter making much sense. The issue with SQL for me is the "natural language" which quickly loses all intended readabilty when you have SELECT col1, col2 FROM (SELECT * FROM ... WHERE 1=0 AND ... which is what edgeDB is trying to solve.

[0]https://edgedb.com/

Re: DuckDB 0.8

#18

Post by vamega »

TACIXAT wrote: post #6

I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…

What feature was removed? I'm pretty fine with a project deciding to remove a feature with known footguns, but curious what it was.

Re: DuckDB 0.8

#19

Post by 1egg0myegg0 »

fbdab103 wrote: post #16
1egg0myegg0 wrote: post #12

Earlier quoted context omitted.

Great question! For analytical queries, DuckDB is ~100x faster than SQLite. It also has great connectivity with Parquet, Pandas, Arrow, Postgres, and more! Plus, the syntax follows Postgres closely, so you have strong data typing and true timestamps, etc. All that said - SQLite is still faster for transactions. Now DuckDB can actually read and write from/to SQLite, so you can use SQLite for OLTP and DuckDB for OLAP.…

>Now DuckDB can actually read and write from/to SQLite This feels like it opens up some possibilities, but I am struggling to figure out where. Are you aware of any interesting use cases this has enabled? Does all of the Duck syntax then work against the SQLite database? For example, could I now run a pivot without a bunch of hoops?

You could definitely run a pivot with the SQLite attach! All of the DuckDB syntax works!

Re: DuckDB 0.8

#20

Post by nine_k »

emmanueloga_ wrote: post #11

Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…

OK, you got a good KV store, and a good indexing library in your favorite language. You now write special-case procedural code for each retrieval and each update. Eventually you get tired of copy-pasting such code with slight modifications; you factor things out. You make them generic. You end up with a poorly specified "language" for making queried, joins, projections. You notice how sometimes performance requires different approaches, and bake the logic tp choose them into your library. Then you realise that there are cases when you need two related updates either succeed or fail together, not halfway. Transactions.

This all has been tried many times in 1990s when RDBMSes were pricey or otherwise unattainable. Paradox engine, various stuff on top of Berkeley DB, etc. MySQL is) or at least was) an SQL engine on top of a KV storage engine.

Now we have Postgres that fills most small- and median-scale needs out of the box.

A modern RDBMS is one of the most powerful and versatile tools in developer's hands, and usually works amazingly well even with stock settings. Usually it by far outperforms makeshift replacements to it. Learn to operate an RDBMS, learn to wield this power. It's very much humanly possible, and is much nicer experience than learning, say, C++. Then maybe you will have less of a desire to replace it with manually written imperative code.

Post reply on HN