Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DuckDB 0.7.0

duckdb.org

Post reply on HN
Print view View original

61–66 of 66 posts

Re: DuckDB 0.7.0

#61

Post by isoprophlex »

smt88 wrote: post #59
isoprophlex wrote: post #13

Earlier quoted context omitted.

Quick, in-core data transformation. If you want to transform some data right now, one option is writing pyspark and running that on a spark cluster. But noone really has big big data, there are relatively few cases where you have multi TB datasets, warranting the complexities of running the analyics in a distributed way. DuckDB lets you process all that locally. It's the OLAP equivalent to SQLite's OLTP. If I wasn't…

Interesting. I need to think about this one a little bit. Thank you. Is there a reason "in-core" is a specific requirement here?

Not really, and duckdb doesn't need to hold everything in RAM as i recalll. But it's fast, far faster than several read-process-write steps can be, especially when coordinated over multiple machines

(By the way, maybe I was vague, using overloaded terminology. To be precise with 'in-core' i meant that the solution to an analytic query is held completely in memory, not that it's restricted to using one cpu thread.)

Re: DuckDB 0.7.0

#62

Post by talolard »

stinos wrote: post #52
talolard wrote: post #40

I use Duckdb as a data scientist / analyst. It’s amazing for working with large data locally, because it is very fast and there is almost 0 overhead for use. For example, I helped an Israeli ngo analyze retailer pricing data (supermarkets must publish prices every day by law). Pandas chokes on data that large, Postgres can handle it but aggregations are very slow. Duckdb is lightning fast. The traditional alternative…

So I'm not super familiar with different databases, but do understand the basics and do know how to work wit data with e.g. pandas, and do think I understand what Duckdb is useful for, but what I'm still completely missing is: how do I get data in Duckdb? I.e. how did you get that data into Duckdb? Or: suppose I have a device producing sensor data, normally I'd connect to some MySQL endpoint somehow and tell it to in…

You can do both ways but the latter is the more useful one. Duckdb is designed to read the data very fast and to operate on it fast. So you load a csv/json/parquet and then “create table” and Duckdb lays out the data in a way that makes it fast to read.

But you(I) wouldn’t use it like a standard db where stuff gets constantly written in, rather like a tool to effectively analyze data that’s already somewhere

Re: DuckDB 0.7.0

#63

Post by 1egg0myegg0 »

smt88 wrote: post #57
shanipribadi wrote: post #18

Earlier quoted context omitted.

think BI tools, analytics dashboards for exploratory analysis, or even just exploratory analysis on the terminal with it's rich query capabilities. you can keep analytics data in SQLite, but DuckDB will process it faster/easier for the analytics use cases.

> think BI tools, analytics dashboards for exploratory analysis, or even just exploratory analysis on the terminal with it's rich query capabilities I thought about that, but I'd never use DuckDB for it because DuckDB is locked into a single process. I can't figure out a benefit of being suck with one core when I always have between 2 and 32 available to me.

DuckDB uses all of your cores! It just uses threads, not processes!

Re: DuckDB 0.7.0

#64

Post by nattaylor »

smt88 wrote: post #58
adlpz wrote: post #12

Earlier quoted context omitted.

I'm interested in this, too. I can totally see how not having to manage a standalone RDBMS makes sense. But, what's the real-world advantage over something like SQLite? I mean, the idea of an in-memory relational engine for things like games or embedded totally makes sense, but this seems to target large datasets and deep analysis. As far as I understand with this model you pretty much re-ingest data from the "raw" s…

> But, what's the real-world advantage over something like SQLite? This very specific question is what I'm trying to understand. SQLite can be run in memory and as a temporary store.

Duckdb excels at OLAP workloads that SQLite would choke on

Re: DuckDB 0.7.0

#65

Post by Tarq0n »

isoprophlex wrote: post #7

> After this release DuckDB will also be able to write hive-partitioned data using the PARTITION_BY clause. These files can be exported locally or remotely to S3 compatible storage. Kudos to the team for their consistently useful, interesting work. They really seem to know their audience well, to have a well-thought-out feature roadmap. Makes you wonder if a single, well-specced box running DuckDB is going to be 2024…

Is Duckdb designed for multiple users? I always got the impression the default use-case is single-user.

Re: DuckDB 0.7.0

#66

Post by RyEgswuCsn »

stinos wrote: post #52
talolard wrote: post #40

I use Duckdb as a data scientist / analyst. It’s amazing for working with large data locally, because it is very fast and there is almost 0 overhead for use. For example, I helped an Israeli ngo analyze retailer pricing data (supermarkets must publish prices every day by law). Pandas chokes on data that large, Postgres can handle it but aggregations are very slow. Duckdb is lightning fast. The traditional alternative…

So I'm not super familiar with different databases, but do understand the basics and do know how to work wit data with e.g. pandas, and do think I understand what Duckdb is useful for, but what I'm still completely missing is: how do I get data in Duckdb? I.e. how did you get that data into Duckdb? Or: suppose I have a device producing sensor data, normally I'd connect to some MySQL endpoint somehow and tell it to in…

My experience has been that most of the time you don’t tell DuckDB to insert data. One is expected to point DuckDB to an existing data file (parquet, csv, json with this new release, etc.) and either import/copy the data into DuckDB tables, or issue SQL queries directly against the file.

Think of it as a SQL engine for ad-hoc querying larger-than-memory datasets.

Post reply on HN