I have been wanting to try DuckDB forever and finally took it for a spin recently. It is an amazing technology that I hope to utilize in production one day. Hat's off to the UI/UX team behind the website and docs. Love the aesthetic!
DuckDB 0.8
81–90 of 101 posts
Re: DuckDB 0.8
#82The main use case for me for DuckDB is in Data Lake-related tasks where cloud providers often do not provide great/cost-effective/otherwise limited services. In the case of AWS, repartitioning Parquet files in S3 via Athena CTAS statements in limited to 100 active partitions, which is a bummer to work around. Therefore, I’m using DuckDB with repartitioning queries, because it doesn’t have the 100 partition limit. I w…
We extensively explored this approach for a use-case and dropped it in favour of BigQuery external data query for one core reason: It can't do streaming reads if you have to do any kind of operation on the underlying data, for almost everything it will have to load all the relevant parquet files locally to do the operations. You might have been able to solve this using row partitions on the underlying files beforehan…
Very interesting. Would you mind mentioning which external table format you were using? Iceberg by any chance?
I wonder if there wasn't a middle-ground that could still utilize DuckDB or similar. You still rely on BQ to deliver the subset of data needed, and then rely on DuckDB to do the "last-mile" analytics: grouping, filtering, ordering, etc...?
Re: DuckDB 0.8
#83I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is th…
How do you insert into DuckDB fast and what settings ("Indices") do you use? As far as I understand DuckDB builds up statistics for each "block" of data (number of different values, ... ). So I assume inserting is slow. There is a paper [0] and a comment [1] that mentions that DuckDB is 10-500 times slower in a write-heavy workload. [0] https://simonwillison.net/2022/Sep/1/sqlite-duckdb-paper/ [1] https://vldb.org/pv…
I have a large number of small and frequent batches, think of it like discrete ETL, where each process operates on a pandas DataFrame. This frame ends up being written to disc as parquet and immediately followed by creating a DuckDB that imports the parquet. The duckdb file from then on will only be opened for read, no further writes.
I use a python odata library to convert user queries in rest to a SQL similar to Postgres and run it on these duckdb for applying any filters where needed.
Re: DuckDB 0.8
#84I've been thinking about rebuilding the website analytics service I have, to be around sharded duckdb and just giving people raw SQL access. ClickHouse is the alternative, but this approach has some interesting advantages, like simplicity.
I find clickhouse equally simple. One command to install and then you can either read from local/remote files, create tables, etc
From the brief time I used it running signoz it used a surprising amount of resources, which requires further investigation, but yes it's a nice stable option.
Re: DuckDB 0.8
#85I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
> I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite instead. I am not sure that you realize that SQLite is written entirely in C -- a quintessential memory unsafe language. I guess quality of software depends on many things b…
I’m quite sure that the OP DOES realize this.
Re: DuckDB 0.8
#86I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
> The other thing that rubbed me the wrong way was that rather than fix the issue, they just removed functionality. Yeah, DuckDB has some very cool features, but I with the community were less abrasive. I remember someone asking for ORC columnar format support, and DuckDB replied "that is not as popular as Parquet so we're not doing it, issue closed". Same story with Delta vs Iceberg. Meanwhile Clickhouse supports bo…
> DuckDB has some very cool features, but I with the community were less abrasive.
I’m on the Discord and the community in my experience has been anything but “abrasive”. I’m just a random guy and yet I’ve received stellar and patient help for many of my naive questions. Saying they are abrasive because they’re not willing to build something seems so entitled to me.
Focused engineering teams have to be willing to say no in order to achieve excellence with limited bandwidth. I’m glad they said no when they did so they could deliver quality on DuckDB.
I certainly think ORC is a good thing to say no too - in my years of working in this space I’ve only rarely encountered ORC files (technically ORC is superior to Parquet in some ways but adoption has never been high)
Also realize that the team is not being paid by the people who ask for new features. If you’re willing to pay them on a retainer through DuckDB labs then you can expect priority, otherwise the sentiment expressed in your comment just seems so uncalled for.
Re: DuckDB 0.8
#87Earlier quoted context omitted.
Is anyone having luck with this? It seems too "creative".
I haven't tried Claude but I have been tinkering with a lot of this in my home lab and there are various theories I have: - GPT4 is not a model, it's a platform. I believe the platform picks the best model for your query in the background and this is part of the magic behind it. - The platform will also query multiple data sources depending on your prompt if necessary. OpenAI is just now opening up this plugin archit…
Interesting speculation but I don’t think GPT-4 chooses any model, I’m pretty sure it’s just how good that one model is. I played with a lot of local models but the reality is, even with wizard vicuna, were at least an order of magnitude away from the size of GPT-4.
Re: DuckDB 0.8
#88Such an amazing project. Once the geospatial extension lands I'm going to seriously consider switching our geoparquet workflows over.
You mean this one?
DuckDB Spatial Extension https://duckdb.org/2023/04/28/spatial.html
Re: DuckDB 0.8
#89I've been thinking about rebuilding the website analytics service I have, to be around sharded duckdb and just giving people raw SQL access. ClickHouse is the alternative, but this approach has some interesting advantages, like simplicity.
I find clickhouse equally simple. One command to install and then you can either read from local/remote files, create tables, etc
Clickhouse looks more complicated because instead of
create table bla (
id number,
xyz varchar
)
create table bla (
id uint64 CODEC(DoubleDelta(8)),
xyz Nullable(LowCardinality(Varchar)) CODEC(ZSTD(1))
)
engine=MergeTree
order by id
Re: DuckDB 0.8
#90Earlier quoted context omitted.
>The other thing that rubbed me the wrong way was that rather than fix the issue, they just removed functionality. It is a limited team size. If they feel a feature is causing too much grief, I would rather they drop it than post a, "Here be dragons" sign and let users pick up the pieces. Edit: missed an obvious opportunity to take a shot at MySQL
I think the critique is that not that they should have left the thing broken, but that a limited team should limit the work to match the team size so that they do not release broken things in the first place.
It is often difficult to ascertain how much iceberg is beneath the water until you attempt to solve the problem.