Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB Community Extensions

duckdb.org

Post reply on HN
Print view View original

11–20 of 21 posts

Re: DuckDB Community Extensions

#11

Post by shubhamjain »

Honest question, how feasible it would be for DuckDB to release a non-columnar version of their DB (or at least make DuckDB a decent choice for a typical web app)? I don't know any other DB that makes installing extensions this easy. The rate at which they're shipping awesome features makes me wonder if they could eventually become a great generic database.

I know, I know, this could just as easily be a double-edged sword. A database should prioritize stability above everything else, but there is no reason why we shouldn't expect them to reach there.

Re: DuckDB Community Extensions

#14

Post by wild_egg »

shubhamjain wrote: post #11

Honest question, how feasible it would be for DuckDB to release a non-columnar version of their DB (or at least make DuckDB a decent choice for a typical web app)? I don't know any other DB that makes installing extensions this easy. The rate at which they're shipping awesome features makes me wonder if they could eventually become a great generic database. I know, I know, this could just as easily be a double-edged…

Are we certain that it's _not_ a decent choice for a typical web app? I'm tempted to swap it into one of mine and see how it behaves. Even if some operations are internally slower, that might be offset by having zero network latency to deal with

It would be nice though if other DBs made extensions this easy. There are a handful of package managers for Postgres but they're not generally supported on managed platforms like RDS.

Anyone know if there are comparable options for SQLite? Seems like an obvious thing that should exist but a quick search isn't showing me any

Re: DuckDB Community Extensions

#15

Post by mgaunard »

shubhamjain wrote: post #11

Honest question, how feasible it would be for DuckDB to release a non-columnar version of their DB (or at least make DuckDB a decent choice for a typical web app)? I don't know any other DB that makes installing extensions this easy. The rate at which they're shipping awesome features makes me wonder if they could eventually become a great generic database. I know, I know, this could just as easily be a double-edged…

Most of my web apps are built around tabular data.

Re: DuckDB Community Extensions

#16

Post by immibis »

1egg0myegg0 wrote: post #7
9cb14c1ec0 wrote: post #5

> What happens behind the scenes is that DuckDB downloads an extension binary The baser part of me wonders how hard it would be to compromise that supply chain.

Extension downloads are validated using a signature check to prevent tampering! (I work for DuckDB Labs and MotherDuck)

The backdoored version of xz was also signed.

Re: DuckDB Community Extensions

#18

Post by snidane »

shubhamjain wrote: post #11

Honest question, how feasible it would be for DuckDB to release a non-columnar version of their DB (or at least make DuckDB a decent choice for a typical web app)? I don't know any other DB that makes installing extensions this easy. The rate at which they're shipping awesome features makes me wonder if they could eventually become a great generic database. I know, I know, this could just as easily be a double-edged…

What do you need non-columnar layout for? Do you expect thousands of concurrent single row writes at a time?

If you use embedded duckdb on the client, unless the person goes crazy clicking their mouse at 60 clicks/s, duckdb should handle it fine.

If you run it on the backend and expect concurrent writes, you can buffer the writes in concatenated arrow tables, one per minibatch, and merge to duckdb every say 10 seconds. You'd just need to query both the historical duckdb and realtime arrow tables separately and combine results later.

I agree that having a native support for this so called Lambda architecture would be cool to have natively in duckdb. Especially when drinking fast moving data from a firehose.

Re: DuckDB Community Extensions

#19

Post by rustyconover »

xnx wrote: post #2

Very cool. The shellfs extension ( https://github.com/rustyconover/duckdb-shellfs-extension ) that allows shell commands to be used for input and output will make DuckDB even more useful as a command line analysis tool. I'm not sure how I'll use it yet, but I'm betting I can streamline some multi-step data processes.

As the author I'm happy to answer any questions. I'm glad you like the idea of the extension.

Re: DuckDB Community Extensions

#20

Post by 1egg0myegg0 »

shubhamjain wrote: post #11

Honest question, how feasible it would be for DuckDB to release a non-columnar version of their DB (or at least make DuckDB a decent choice for a typical web app)? I don't know any other DB that makes installing extensions this easy. The rate at which they're shipping awesome features makes me wonder if they could eventually become a great generic database. I know, I know, this could just as easily be a double-edged…

Hello! I would recommend trying out DuckDB's SQLite attach feature! You can read or write data, and even make schema changes, all with DuckDB's engine and syntax. The storage then uses SQLite, which is row oriented!

https://duckdb.org/docs/extensions/sqlite

(I work at MotherDuck and DuckDB Labs)

Post reply on HN