Databases in 2025: A Year in Review • Hacker News Board

Re: Databases in 2025: A Year in Review

#101

Post by dietr1ch » Mon, Jan 05, 2026, 3:56 PM UTC

sanderjd wrote: post #92

theanonymousone wrote: post #90

Earlier quoted context omitted.

The way I understood it, you can do your inserts with SQLite "proper", and simultaneously use DuckDB for analytics (aka read-only).

Aha! That makes so much sense. Thank you for this. Edit: Ah, right, the downside is that this is not going to have good olap query performance when interacting directly with the sqlite tables. So still necessary to copy out to duckdb tables (probably in batches) if this matters. Still seems very useful to me though.

Analytics is done in "batches" (daily, weekly) anyways, right?

We know you can't get both, row and column orders at the same time, and that continuously maintaining both means duplication and ensuring you get the worst case from both worlds.

Local, row-wise writing is the way to go for write performance. Column-oriented reads are the way to do analytics at scale. It seems alright to have a sync process that does the order re-arrangement (maybe with extra precomputed statistics, and sharding to allow many workers if necessary) to let queries of now historical data run fast.

Re: Databases in 2025: A Year in Review

#102

Post by throw0101d » Mon, Jan 05, 2026, 3:59 PM UTC

Regarding distributed(-ish) Postgres, does anyone know if something like My/MariaSQL's multi-master Galera† is around for Pg:

> MariaDB Galera Cluster provides a synchronous replication system that uses an approach often called eager replication. In this model, nodes in a cluster synchronize with all other nodes by applying replicated updates as a single transaction. This means that when a transaction COMMITs, all nodes in the cluster have the same value. This process is accomplished using write-set replication through a group communication framework.

* https://mariadb.com/docs/galera-cluster/galera-architecture/...

This isn't necessarily about being "web scale", but having a first-party, fairly-automated replication solution would make HA easier for a number internal-only stuff much simpler.

† Yes, I am aware: https://aphyr.com/posts/327-jepsen-mariadb-galera-cluster

Re: Databases in 2025: A Year in Review

#103

Post by chasd00 » Mon, Jan 05, 2026, 4:04 PM UTC

sandos wrote: post #36

pjmlp wrote: post #20

Over here, it is DB2, SQL Server or Oracle if using a plain RDMS, or whatever DB abstraction layer is provided on top of a SaaS product, where we get to query with some kind of ORM abstraction preventing raw SQL, or GraphQL, without knowing the implementation details.

This sounds like a flashback to J2EE. Which I know is still alive and well. Banks, insurance companies and the tax agency do not much care for fancy new stuff, but that it works.

I describe these techs like garbage trucks. No one likes to see them but they’re there every day doing a decent part of what it takes to hold society together hah.

Re: Databases in 2025: A Year in Review

#104

Post by pjmlp » Mon, Jan 05, 2026, 4:13 PM UTC

chasd00 wrote: post #103

sandos wrote: post #36

Earlier quoted context omitted.

This sounds like a flashback to J2EE. Which I know is still alive and well. Banks, insurance companies and the tax agency do not much care for fancy new stuff, but that it works.

I describe these techs like garbage trucks. No one likes to see them but they’re there every day doing a decent part of what it takes to hold society together hah.

Scott Hanselman has a good term for all these kind of jobs, the dark matter developers.

https://www.hanselman.com/blog/dark-matter-developers-the-un...

Re: Databases in 2025: A Year in Review

#105

Post by cfors » Mon, Jan 05, 2026, 4:13 PM UTC

swyx wrote: post #77

delaminator wrote: post #10

Earlier quoted context omitted.

From my perspective, everything's DuckDB. Single file per database, Multiple ingestion formats, full text search, S3 support, Parquet file support, columnar storage. fully typed. WASM version for full SQL in JavaScript.

very interesting. whats the vector indexing story like in duckdb these days? also are there sqlite-duckdb sync engines or is that an oxymoron

https://duckdb.org/docs/stable/core_extensions/vss

It's not bad if you need something quick. I haven't had a large need of ANN in duckdb since it's doing more analytical/exploratory needs, but it's definitely there if you need it.

Re: Databases in 2025: A Year in Review

#106

Post by pjmlp » Mon, Jan 05, 2026, 4:14 PM UTC

sanderjd wrote: post #85

pjmlp wrote: post #57

Earlier quoted context omitted.

Yep, Fortune 500 enterprise consulting, boring technology that pays the bills. Java, .NET, C++, nodejs, Sitecore, Adobe Experience Manager, Optimizely, SAP, Dynamics, headless CMSes,...

Never felt so old, seeing nodejs in a list of old boring stuff.

Yeah, it is on the edge, but unavoidable in many Web projects.

Re: Databases in 2025: A Year in Review

#107

Post by znpy » Mon, Jan 05, 2026, 4:24 PM UTC

danielfalbo wrote: post #4

Maybe off-topic but, If you're not familiar with the CMU DB Group you might want to check out their eccentric teaching style [1]. I absolutely love their gangsta intros like [2] and pre-lecture dj sets like [3]. I also remember a video where he was lecturing with someone sleeping on the floor in the background for some reason. I can't find that video right now. Not too sure about the context or Andy's biography, I'll…

I can't understand if their "intro to database systems" is an introductory (undergrad) level course or some advanced course (as in, introduction to database (internals)).

Anyone willing to clarify this? I'm quite weak at database stuff, i'd love to find some undergrad-level proper course to learn and catch up.

Re: Databases in 2025: A Year in Review

#108

Post by qcnguy » Mon, Jan 05, 2026, 4:24 PM UTC

sanderjd wrote: post #81

qcnguy wrote: post #61

Earlier quoted context omitted.

Oracle is mentioned at the start, where he proclaims the "dominance" of Postgres and then admits its newest features have been in Oracle for nearly a quarter of a century already. The dominance he's talking about is only about how many startups raise how many millions from investors, not anything technical. And then of course at the end he has a whole section about Larry Ellison, like always.

Isn't it because it's about news , as in what's changing, rather than being about what's staying the same? He's a researcher, so his interests are always going to be more oriented toward new systems and new companies more than the big dominant systems.

There's nothing technically new that he's covering here though? It's all just startups adding stuff to Postgres that Oracle had for decades already.

Re: Databases in 2025: A Year in Review

#109

Post by sanderjd » Mon, Jan 05, 2026, 4:35 PM UTC

qcnguy wrote: post #108

sanderjd wrote: post #81

Earlier quoted context omitted.

Isn't it because it's about news , as in what's changing, rather than being about what's staying the same? He's a researcher, so his interests are always going to be more oriented toward new systems and new companies more than the big dominant systems.

There's nothing technically new that he's covering here though? It's all just startups adding stuff to Postgres that Oracle had for decades already.

The startups are new.

Re: Databases in 2025: A Year in Review

#110

Post by sanderjd » Mon, Jan 05, 2026, 4:36 PM UTC

dietr1ch wrote: post #101

sanderjd wrote: post #92

Earlier quoted context omitted.

Aha! That makes so much sense. Thank you for this. Edit: Ah, right, the downside is that this is not going to have good olap query performance when interacting directly with the sqlite tables. So still necessary to copy out to duckdb tables (probably in batches) if this matters. Still seems very useful to me though.

Analytics is done in "batches" (daily, weekly) anyways, right? We know you can't get both, row and column orders at the same time, and that continuously maintaining both means duplication and ensuring you get the worst case from both worlds. Local, row-wise writing is the way to go for write performance. Column-oriented reads are the way to do analytics at scale. It seems alright to have a sync process that does the…

Not all olap-like queries are for daily reporting.

I agree that the basic architecture should be row order -> delay -> column order, but the question (in my mind) is balancing the length of that delay with the usefulness of column order queries for a given workload. I seem to keep running into workloads that do inserts very quickly and then batch reads on a slower cadence (either in lockstep with the writes, or concurrently) but not on the extremely slow cadence seen in the typical olap reporting type flow. Essentially, building up state and then querying the results.

I'm not so sure about "continuously maintaining both means duplication and ensuring you get the worst case from both worlds". Maybe you're right, I'm just not so sure. I agree that it's duplicating storage requirements, but is that such a big deal? And I think if fast writes and lookups and fast batch reads are both possible at the cost of storage duplication, that would actually be the best case from both worlds?

I mean, this isn't that different conceptually from the architecture of log-structured merge trees, which have this same kind of "duplication" but for good purpose. (Indeed, rocksdb has been the closest thing to what I want for this workload that I've found; I just think it would be neat if I could use sqlite+duckdb instead, accepting some tradeoffs.)