Is duckdb multi region active active?
DuckDB Internals Part 1
101–110 of 165 posts
Re: DuckDB Internals Part 1
#102Holy LLM slop article…
> You open a connection, send SQL over TCP (a protocol to send data over a network),
Thanks for the clarification, LLM!!
Re: DuckDB Internals Part 1
#103DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…
>> DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop
I agree, and the dirty (not so) secret big data providers like Snowflake try to hide: the majority of your work is not big data and WILL fit on your local machine. My last company was spending $2M/yr on contract with Snowflake, and another million between Fivetran and Matillion. Of the 1200 clients using analytics maybe 2 had enough data to warrant "infinite scalability" and a dozen wanted Snowflake because they already had corporate warehouses in Snowflake (they probably didn't need it either). Turns out the Extract and Load could be handled by bog-standard C# code and a bunch of SQL, while almost everyone was better off with a DuckDB database running locally, often in the browser. You've probably heard YAGNI before (You Ain't Gonna Need It) but it's even more likely with "Big Data". #SmallDataConvert
Re: DuckDB Internals Part 1
#104Earlier quoted context omitted.
Like sqlite, duckdb is underappreciated as a production database. You can totally run it on servers or even "serverless" and do some heavy data transformations or with the right server size work with large scale datasets (up to a TB compressed seems fine).
This. I've recently used both duckdb and sqlite to power a dashboard for a small restaurant of a family member. It converts all their sales to a very tiny parquet files, daily. The file fits in memory and can do all sort of computation in the browser itself. The backend is extremely simple, it just loads the JS and serves the parquet files. It was also trivial to let the owner do their own queries, just give the sche…
neat
Re: DuckDB Internals Part 1
#105Earlier quoted context omitted.
Like sqlite, duckdb is underappreciated as a production database. You can totally run it on servers or even "serverless" and do some heavy data transformations or with the right server size work with large scale datasets (up to a TB compressed seems fine).
This. I've recently used both duckdb and sqlite to power a dashboard for a small restaurant of a family member. It converts all their sales to a very tiny parquet files, daily. The file fits in memory and can do all sort of computation in the browser itself. The backend is extremely simple, it just loads the JS and serves the parquet files. It was also trivial to let the owner do their own queries, just give the sche…
Similar experience here. The best thing I've built in a long time is replacing a complex (and scary) permissions system built on top of Snowflake with single role duckdb databases that - aside from no longer worrying about bugs leaking data across roles - are more performant, timely and flexible. Combined with the use of AI this is the way forward IMO.
At the other end of the spectrum, working with random data on "what if?" and exploration tasks with DuckDB is fun again. it's so straightforward and fast, with tools and functions for pretty much everything.
Re: DuckDB Internals Part 1
#106Earlier quoted context omitted.
That's going to be a difficult business in this age unless you have some uniquely strong ideas and products.
I'm not convinced that is true. The JCB digger didn't put groundworkers out of business. A consultant that can get more done in a day is worth a lot more than one who can't. There is still skill required in wielding the tool of the day and that skill is marketable.
Sorry, I don't think your non-software analogy that existed before 2025 helps at all in this specific topic about software consultancy.
Re: DuckDB Internals Part 1
#107DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop Recently at work I've been using it to analyse the Claude code sessions of every engineer at our company (that we upload to S3) and it's been extremely helpful to help us find gaps in devex and have clear metrics to back up the impact of fixing them Another thing it's been really useful for has been getti…
>> DuckDB is amazing for any sort of fast data analysis when the data is small enough that it can fit on your laptop I agree, and the dirty (not so) secret big data providers like Snowflake try to hide: the majority of your work is not big data and WILL fit on your local machine. My last company was spending $2M/yr on contract with Snowflake, and another million between Fivetran and Matillion. Of the 1200 clients usi…
Folks have been beating this drum for as long as I've worked in software, dating to the Hadoop era, and it remains true today. So much of "big data" only appears big because it's poorly stored, or is represented wastefully (in persistent storage or in memory).
Re: DuckDB Internals Part 1
#108It's an interesting project, but the discussion on HN looks weird. It gets brought up every few weeks[1] and everyone just spams comments with messages about how "fast" it is. DuckDB is fast for some specific workloads . If you use it for most other things, it is at least an order of magnitude slower than SQLite. It also has some limitations in terms of what SQL it will currently run (e.g. I immediately ran into an i…
I'm going to sound like a broken record but... different use cases. They're analogous in the comparison "sqlite for analytics" but completely different architectures and implementations. Part of this is the fault of the developers, but I feel they were trying to highlight the similar focus on in-process, zero dependencies, simplicity and test coverage - not a direct "vs" comparison. IME recursive queries in analytical workflows are not very common; they typically work against the fundamental data layout on disk.
SQLite is awesome and I would love to see more posts about it, but the reality is one of the major reasons it's awesome is the no-drama/stability/it just works. DuckDB is seeing a lot of development on many fronts so there's a lot more to learn and talk about right now.
Re: DuckDB Internals Part 1
#109Earlier quoted context omitted.
Like sqlite, duckdb is underappreciated as a production database. You can totally run it on servers or even "serverless" and do some heavy data transformations or with the right server size work with large scale datasets (up to a TB compressed seems fine).
This. I've recently used both duckdb and sqlite to power a dashboard for a small restaurant of a family member. It converts all their sales to a very tiny parquet files, daily. The file fits in memory and can do all sort of computation in the browser itself. The backend is extremely simple, it just loads the JS and serves the parquet files. It was also trivial to let the owner do their own queries, just give the sche…
I have a a theory that LLMs are going to be the death knell of big SaaS. It's so much harder to build and maintain an massive SaaS that does 80% of what 80% of your customers want, than it is to build something small and simple that does 100% of what one customer wants.
Re: DuckDB Internals Part 1
#110Why is DuckDB so popular when one can use Python + Pandas? Better perf + SQL is that mostly it?
The better question is, why is DuckDB so popular when one can use Polars which has a sane, lintable, typesafe API compared to the mess that is SQL: WITH lagged AS ( SELECT *, LAG(event_time) OVER (PARTITION BY user_id ORDER BY event_time) AS prev_time FROM events ), sessions AS ( SELECT *, SUM(COALESCE((date_diff('minute', prev_time, event_time) > 30)::INT, 1)) OVER (PARTITION BY user_id ORDER BY event_time) AS sessi…
How can you tell if someone is a polars fan? Don’t worry, they’ll tell you. :)