I found myself thinking about Cloudflare Durable objects new SQLite offering.
Nicely detailed here https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... And https://developers.cloudflare.com/durable-objects/best-pract...
Transactional Object Storage?
21–30 of 37 posts
Re: Transactional Object Storage?
#22If I had time, I'd like to implement an Iceberg catalog this way.
Re: Transactional Object Storage?
#23There is also SlateDB, another work in progress take on this. HN link: https://news.ycombinator.com/item?id=41714858
Yeah I think it's very interesting to compare the two. SlateDB expects a single writer and fences writes. This means you can make some serious savings on S3 costs because you're using S3 for consistency but you're batching writes.
GlassDB is much more accessible for smaller volume workloads, but gets very costly for high volume because of requests to S3 per-transaction. In-turn the consistency model is easier to reason about because the system is entirely stateless.
Re: Transactional Object Storage?
#24Was it considered to separate each table into its own S3 object?
Re: Transactional Object Storage?
#25Congrats on reinventing the data lake? This is actually how most of the newer generations of "cloud native" databases work, where they separate compute and storage. The key is that they have a more sophisticated caching layer so that the latency cost of a query can be amortized across requests.
It's my understanding that the newer generation of data lakes still make use of a tiny, strongly consistent metadata database to keep track of what is where. This is orders of magnitudes smaller than what you'd have by putting everything in the same database, but it's still there. This is also the case in newer data streaming platforms (e.g. https://www.warpstream.com/blog/kafka-is-dead-long-live-kafk... ). I'm curio…
> if you have examples of any database using only object storage as a backend
I think DuckDB is very close to this. It's a bit different, because it's mostly for read-heavy workloads.
https://duckdb.org/docs/extensions/httpfs/s3api
(BTW great article, excellent read!)
Re: Transactional Object Storage?
#26TiDB Serverless is built on S3, it's in production for more than 2 years, Blog link: https://me.0xffff.me/dbaas2.html
Re: Transactional Object Storage?
#27If I had time, I'd like to implement an Iceberg catalog this way.
100% this! S3’s lack of write preconditions spawned the whole Iceberg catalog ecosystem anyways.
Re: Transactional Object Storage?
#28Whenever I saw the claim that “S3 is cheap “, I just cannot take it too seriously.
Re: Transactional Object Storage?
#29Was it considered to separate each table into its own S3 object?
Only if you don’t want transactions across tables?
Re: Transactional Object Storage?
#30I appreciate the effective use of diagrams. The boundaries are separated really nicely.