Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Quack: The DuckDB Client-Server Protocol

duckdb.org

Post reply on HN
Print view View original

31–40 of 91 posts

Re: Quack: The DuckDB Client-Server Protocol

#31

Post by appplication »

hermitcrab wrote: post #30

I have a C++ application. Everything is in memory during execution. Saved to disk between session as XML. Works great, except that that it is strictly single user and some of my customers would love me to generalize it for multiple concurrent users reading and writing. Performance requirements are quite low - a few thousand records being updated by 2 or 3 people at a time. Would DuckDb + Quack be a good choice for th…

DuckDB is more for analytics. I don’t think you’re going to find good options for a DB that can handle concurrent users without hosting it in some way server side. It’s certainly possible (think how some games create their own client servers for direct multiplayer) but honestly hosting Postgres or SQLite is ridiculously cheap, easy, and more importantly the standard approach to this issue.

Re: Quack: The DuckDB Client-Server Protocol

#32

Post by hermitcrab »

appplication wrote: post #31
hermitcrab wrote: post #30

I have a C++ application. Everything is in memory during execution. Saved to disk between session as XML. Works great, except that that it is strictly single user and some of my customers would love me to generalize it for multiple concurrent users reading and writing. Performance requirements are quite low - a few thousand records being updated by 2 or 3 people at a time. Would DuckDb + Quack be a good choice for th…

DuckDB is more for analytics. I don’t think you’re going to find good options for a DB that can handle concurrent users without hosting it in some way server side. It’s certainly possible (think how some games create their own client servers for direct multiplayer) but honestly hosting Postgres or SQLite is ridiculously cheap, easy, and more importantly the standard approach to this issue.

IIRC SQLite is in-process and says in it's documentation that it is not a client-server database.

Re: Quack: The DuckDB Client-Server Protocol

#34

Post by geysersam »

ozgrakkurt wrote: post #9

> It would be rather misguided not to build a database protocol on top of HTTP in 2026 This is wrong, HTTP is bad for transferring large amount of data and it is also bad for doing streaming. It is bad for large amount of data because you have timeout issues on some clients, you hit request/response size limits etc. It is obviously bad for streaming as there is no concept of streaming in it. It is comical to go the p…

They mention in the benchmarks section that the network they're on is a "up to" 15 Gbps connection. So to max out 50GB/s is not realistic.

I agree they should have also listed the compressed size of the table instead of only mentioning the CSV size. But the compressed dataset is probably not smaller than 1/10 of the CSV size. If that's the case they're transferring ~8GB in 4.6 s on a 2GB/s (15Gbps) connection. Seems pretty close to max.

Re: Quack: The DuckDB Client-Server Protocol

#35

Post by apitman »

hermitcrab wrote: post #30

I have a C++ application. Everything is in memory during execution. Saved to disk between session as XML. Works great, except that that it is strictly single user and some of my customers would love me to generalize it for multiple concurrent users reading and writing. Performance requirements are quite low - a few thousand records being updated by 2 or 3 people at a time. Would DuckDb + Quack be a good choice for th…

I think the term you want to search for is local-first.

Re: Quack: The DuckDB Client-Server Protocol

#36

Post by esafak »

whalesalad wrote: post #6
simlevesque wrote: post #2

I like DuckDB but I'm not sure what it wants to be. There's always new ways to use it and it's not easy to see what's the right one.

Our data pipeline produces .duckdb files that our app downloads (it watches the asset in S3 and pulls when etag changes). Makes it easy to get BQ/Clickhouse like performance without running or paying for that infrastructure. Not perfect for all cases, but it handles a lot more than you would expect.

Roughly how big are the datasets?

Re: Quack: The DuckDB Client-Server Protocol

#37

Post by dangoodmanUT »

pdet wrote: post #21
mritchie712 wrote: post #15

> Can I use DuckDB with Quack as the catalog database for DuckLake? > Not yet, but we are working on it! Seems like a niche use case, but it's the one I'm most interested in. Our lakehouse uses ducklake with postgres as the catalog. Seems like a DuckDB / Quack catalog would be an excellent alternative.

I think that Quack will become the primary option for a DuckLake catalog in the future, for several reasons. To list a few: 1. No type mismatches for inlining. If you use a non-DuckDB catalog, many types do not have a 1:1 mapping, which introduces additional overhead when operating on those data types. 2. You get the raw performance of DuckDB analytics (and now transactions) over the catalog. DuckDB reading DuckDB is…

This. Type casting is an insidious problem (both correctness, and perf)

Re: Quack: The DuckDB Client-Server Protocol

#38

Post by ozgrakkurt »

geysersam wrote: post #34
ozgrakkurt wrote: post #9

> It would be rather misguided not to build a database protocol on top of HTTP in 2026 This is wrong, HTTP is bad for transferring large amount of data and it is also bad for doing streaming. It is bad for large amount of data because you have timeout issues on some clients, you hit request/response size limits etc. It is obviously bad for streaming as there is no concept of streaming in it. It is comical to go the p…

They mention in the benchmarks section that the network they're on is a "up to" 15 Gbps connection. So to max out 50GB/s is not realistic. I agree they should have also listed the compressed size of the table instead of only mentioning the CSV size. But the compressed dataset is probably not smaller than 1/10 of the CSV size. If that's the case they're transferring ~8GB in 4.6 s on a 2GB/s (15Gbps) connection. Seems…

That makes sense. I meant to write 50gbps, I don’t mean they should reach that, I mean you could use any protocol that is fairly efficient and it would reach that.

The size of the dataset should be under 3GB in parquet from what I understand. [0]

So it did 3*8/4.94 = 4.85 Gbps which is underwhelming in terms of network performance.

It is still not possible to make any conclusions since we don’t know how specifically they encode it or how they are running the query.

I just mean this writing is useless in terms of engineering perspective, also what it says about http doesn’t make sense

[0] - https://clickhouse.com/docs/getting-started/example-datasets...

Re: Quack: The DuckDB Client-Server Protocol

#40

Post by whalesalad »

esafak wrote: post #36
whalesalad wrote: post #6

Earlier quoted context omitted.

Our data pipeline produces .duckdb files that our app downloads (it watches the asset in S3 and pulls when etag changes). Makes it easy to get BQ/Clickhouse like performance without running or paying for that infrastructure. Not perfect for all cases, but it handles a lot more than you would expect.

Roughly how big are the datasets?

~30GB .duckdb file
Post reply on HN