Postgres data stored in Parquet on S3: LTAP architecture explained

[dead]

Surprisingly, I'm already encountering a second solution that involves storing data chunks on S3 — and this is all within the same week.

This is becoming popular. At Playcode, we built what we believe is a revolutionary file system for our Playcode Cloud (https://playcode.io/cloud), which enables the creation of full-stack web software. The FS built completely from scratch using Rust. We thought we were the smartest ones around and that nobody else had figured this out. But it turns out Databricks, Neon, and several others have as well.

The idea behind a *Bottomless File System* is really cool, and it works very well for us. Essentially, as described here:

- There is a *page server* - A *Linux file system* split into chunks (let's call them chunks instead of pages) - A *cache on NVMe* - And of course, *object storage*, where everything is asynchronously synchronized

It works quite well, though it has its downsides.

One clear advantage is that NVMe drives have become expensive lately, while object storage remains cheap — so the benefits are undeniable. That said, latency is also a factor.

On top of that, uplink costs are rising. To run an object storage-backed file system, you need a very strong uplink with consistent speed — 1 Gbps is simply not enough. Ideally, you want *5 to 10 Gbps*, depending on the load.

We spend a lot of time optimizing and experimenting with different hosting providers — specifically bare metal hardware. The main challenges are:

- *Slow disks* - *Slow uplink* - And as it turns out, *object storage can be unreliable* — unless you're using S3

But AWS hardware is expensive, so nothing in life is ever that simple.

I have to wonder: What's the open protocol - some sort of LTAP read/write path - for compute engines to target LTAP / lakebases? Sort of like how iceberg > deltalake, except now one layer up (OLTP+OLAP).

LTAP / lakebase poses an interesting gap we have been tracking ever since iceberg as we build gfql (the first oss fully-vectorized property graph compute engine with CPU+GPU targets): There's a natural desire to support both OLAP + OLTP modes for new engines, but also a gap for the cloud-native storage tier we don't want to own.

Iceberg etc were exciting as introducing more write-friendly standards-based OLAP backends better than just S3, eg, gives atomicity, and via iceberg, with open governance for code+protocol. LTAP introduces the pattern we can target in theory for OLTP+OLAP backends... but in practice, via a proprietary manner. A potential 'open' form of writers is to support postgres for oltp path writes + spark writers (presumably arrow-flight-friendly) for olap path writes, but that is so hacky.

because they are making an extra step: instead of the snapshot being actually an object manifest, they store the snapshot metadata in a relation database.

I have a use case where we do 100k changes a second. I bet you "their unlimited snapshots because we are cool and using dbs for metadata" will fail in such a use case.

Every commit in Delta is time travel-able. Defaults in Delta clean up logs in 30 days (and data in 7 days but requires a VACUUM be issued.) You can set those retention periods to any arbitrary time period, but it's typically advised against due to the cost to store all the historical data.