What a great database. This and sqlite-utils are my go to tools for ad hoc data wrangling.
Excited to see how it will progress.
DuckDB 0.8
71–80 of 101 posts
Re: DuckDB 0.8
#72Earlier quoted context omitted.
I don’t even have access to regular Claude so can’t confirm this but the 100K token model they released should in theory be able to handle this to a certain degree.
Is anyone having luck with this? It seems too "creative".
I haven't tried Claude but I have been tinkering with a lot of this in my home lab and there are various theories I have:
- GPT4 is not a model, it's a platform. I believe the platform picks the best model for your query in the background and this is part of the magic behind it.
- The platform will also query multiple data sources depending on your prompt if necessary. OpenAI is just now opening up this plugin architecture to the masses but I would think they have been running versions of this internally since last year.
- There is also some sort of feedback loop that occurs before the platform gives you a response.
This is why we can have two different entities use the same open source model yet the quality of the experience can vary significantly. Better models will produce better outputs "by default", but the tooling and process built around it is what will matter more in the future when we may or may not hit some sort of plateau. At some point we're going to have a model trained on all human knowledge current as of Now. It's inevitable right? After that, platform architecture is what will determine who competes.
Re: DuckDB 0.8
#73Setting up SQL, creating a well-designed schema, and optimizing queries can be a complex, multi-step process. Databases often feel like black boxes, requiring significant training and engineering to get right. One needs to tweak both queries, schema and database configuration to get best results. So why do we need SQL? I know NoSQL came and went (was it nosql or no-relations? not sure...) but honestly I can't think o…
>relational no-sql Do you mean something like edgeDB?[0] Or do you mean some non-declarative language completely? I don't see the latter making much sense. The issue with SQL for me is the "natural language" which quickly loses all intended readabilty when you have SELECT col1, col2 FROM (SELECT * FROM ... WHERE 1=0 AND ... which is what edgeDB is trying to solve. [0] https://edgedb.com/
"It's like a relational database with an object-oriented data model"
Dear god. Send these people back to a decent computer science program to study what the relational model actually is. They're embarrassing themselves. Or I'm embarrassed for them.
Talk about abusing terms and throwing the baby out with the bath water.
SQL is awful. It's an aberration (but a successful one, so). BUT the relational model is beautiful. AND these people clearly don't know what relational means. They appear to have made what in the old days would have been called a "network" database, where the relationships are hard coded in the data itself, aka pointers (they're calling them "links"). Which is anathema to the relational model which is based on sets of sets where the relationships emerge from the query (with hints from schema).
The relational model does not have "links" or pointers. Its "keys" are really a suggestion (or constraint) but not a hard link. Relations are sets of sets (tuples) where those tuples can be joined arbitrarily and so it represents a fundamentally more flexible model where the relationships are not fixed, but emergent from the query.
It was the existence of databases like this back in the 60s and 70s, and the intrinsic problems with them, that led to Codd's development of the relational model in the first place, as a way to free data from hardcoded relationships.
"Graph relational" as a term could mean something I guess (see RelationalAI's product for example, though they don't use that term). But this is not this. "Object identity" is fundamentally philosophically opposed to the relational data model. Identity in the relational algebra is available through the comparison of arbitrary tuples emerging from operations, but is not intrinsic to any "row" or "tuple" or "object".
ARGH. Buzzword marketing, incoherent conceptually.
The world needs a successful non-SQL relational database, but this is not what this is.
Re: DuckDB 0.8
#74I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
> The other thing that rubbed me the wrong way was that rather than fix the issue, they just removed functionality.
Yeah, DuckDB has some very cool features, but I with the community were less abrasive. I remember someone asking for ORC columnar format support, and DuckDB replied "that is not as popular as Parquet so we're not doing it, issue closed". Same story with Delta vs Iceberg.
Meanwhile Clickhouse supports both and if you ask for things they might say "tha tis low priority but we'll take a look". Clickhouse-local can work as CLI (though not in-process) DuckDB too.
Re: DuckDB 0.8
#75I used duckdb successfully in prod to replace SQL Server. We have a micro batch that generates around 5 billion rows of very wide tables every 3 minutes. These data used to go into SQL server, only to be replaced by the new batch of 5 billions and gets marked for deletion. SQL Server was struggling with all these purge activities. Replacing with DuckDB made things much lighter and faster. The only issue I faced is th…
How do you insert into DuckDB fast and what settings ("Indices") do you use? As far as I understand DuckDB builds up statistics for each "block" of data (number of different values, ... ). So I assume inserting is slow. There is a paper [0] and a comment [1] that mentions that DuckDB is 10-500 times slower in a write-heavy workload.
[0] https://simonwillison.net/2022/Sep/1/sqlite-duckdb-paper/ [1] https://vldb.org/pvldb/volumes/15/paper/SQLite%3A%20Past%2C%...
Re: DuckDB 0.8
#76I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
>The other thing that rubbed me the wrong way was that rather than fix the issue, they just removed functionality. It is a limited team size. If they feel a feature is causing too much grief, I would rather they drop it than post a, "Here be dragons" sign and let users pick up the pieces. Edit: missed an obvious opportunity to take a shot at MySQL
I think the critique is that not that they should have left the thing broken, but that a limited team should limit the work to match the team size so that they do not release broken things in the first place.
Re: DuckDB 0.8
#77Earlier quoted context omitted.
> why someone would start something in a memory unsafe language these days You might like what we (Splitgraph) are building with Seafowl [0], a new database which is written in Rust and based on Datafusion and delta-rs [1]. It's optimized for running at the edge and responding to queries via HTTP with cache-friendly semantics. [0] https://seafowl.io [1] https://www.splitgraph.com/blog/seafowl-delta-storage-layer
Getting a 5xx error for your site. Using Firefox mobile if that helps
Hmm... thanks. Maybe it was a hiccup? Does it happen when you click these links in my comment? We haven't been able to replicate on Firefox mobile, but we do have an issue with 500 errors in Firefox when double clicking links in the sidebar of the docs (I know, I know...)
Re: DuckDB 0.8
#78Nice to see Parquet support in DuckDB improving with each new release! In earlier versions, I had trouble with enums and nested schema, these issues have been resolved since.
The only missing feature for me now is full table, column, &c. metadata support in the DuckDB JDBC driver.
Thanks!
Re: DuckDB 0.8
#79Earlier quoted context omitted.
Sorry to hear that! (I work on docs for the DuckDB Foundation) Starting in this release, the DuckDB team invested significantly in adding memory safety throughout catalog operations. There is more on the roadmap, but I would expect this release and all following to have improved stability! That said, at my primary company, we have used it in production for years now with great success!
Yea, legit a cool project and if it wasn't on customer machines and being passed user-defined SQL it would have just been worked around.
Would you mind explaining your issues in a little more detail?
> being passed user-defined SQL
What does this mean exactly? Customers were writing their own SQL on their own machines? Maybe expensive operations such as:
`SELECT ROW_NUMBER() OVER (PARTITION BY THING ORDER BY TS), * FROM EVENTS`
And since it was on a customer's machine, it became a problem you had no control over?
Re: DuckDB 0.8
#80I recently pulled DuckDB out of a project after hitting a memory corruption issue in regular usage. Upon investigating, they had an extremely long list of fuzzer-found issues. I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine. We ended up rewriting a component to drop support for Parquet and to just use SQLite…
> I just don't understand why someone would start something in a memory unsafe language these days. I cannot in good conscience put that on a customer's machine.
We ended up rewriting a component to drop support for Parquet and to just use SQLite instead.
I am not sure that you realize that SQLite is written entirely in C -- a quintessential memory unsafe language. I guess quality of software depends on many things besides a choice of language.