31–37 of 37 posts
Pretty cool! Do you have any ideas already about how to make it work with S3, considering it doesn't support If- headers?
However it will be much simpler with the new conditional writes
Was it considered to separate each table into its own S3 object?
But this is entirely possible. You can wrap GlassDB transactions and encode multiple keys into the same object at a higher level. Transactions across different objects will still preserve the same isolation.
The current version is meant to be a base from which to build higer level APIs, somewhat like FoundationDB.
Whenever I saw the claim that “S3 is cheap “, I just cannot take it too seriously.
Is it the cheapest possible storage in existence? No, if you take raw disks and put them in a rack, but I also feel it wouldn't be an entirely fair comparison.
I found myself thinking about Cloudflare Durable objects new SQLite offering. Nicely detailed here https://simonwillison.net/2024/Oct/13/zero-latency-sqlite-st... And https://developers.cloudflare.com/durable-objects/best-pract...
The flipside is that Cloudflare DO will be a lot faster.
Interesting that all these similar solutions are popping out now.
I think it would be interesting to combine a SQLite per-object approach with transactions on top of different objects.
Whenever I saw the claim that “S3 is cheap “, I just cannot take it too seriously.
You're right indeed:) but it depends on what you are comparing it with. In this case the comparison is against other managed cloud storage and databases, and in that context I think the claim holds. Is it the cheapest possible storage in existence? No, if you take raw disks and put them in a rack, but I also feel it wouldn't be an entirely fair comparison.
When i moved from S3 to DO, my bill went from hundreds to $20/mo. The only thing that changed was the hosting provider.
Earlier quoted context omitted.
You're right indeed:) but it depends on what you are comparing it with. In this case the comparison is against other managed cloud storage and databases, and in that context I think the claim holds. Is it the cheapest possible storage in existence? No, if you take raw disks and put them in a rack, but I also feel it wouldn't be an entirely fair comparison.
S3 is one of the most expensive platforms out there, however. Look at backblaze B2 for an example of just HOW expensive S3 is. When i moved from S3 to DO, my bill went from hundreds to $20/mo. The only thing that changed was the hosting provider.
Earlier quoted context omitted.
It's my understanding that the newer generation of data lakes still make use of a tiny, strongly consistent metadata database to keep track of what is where. This is orders of magnitudes smaller than what you'd have by putting everything in the same database, but it's still there. This is also the case in newer data streaming platforms (e.g. https://www.warpstream.com/blog/kafka-is-dead-long-live-kafk... ). I'm curio…
> I'm curious to hear if you have examples of any database using only object storage as a backend, because back when I started, I couldn't fin any. Take a look at Delta Lake https://notes.eatonphil.com/2024-09-29-build-a-serverless-ac...
> In Databricks service deployments, we use a separate lightweight coordination service to ensure that only one client can add a record with each log ID.
The key difference is that Delta Lake implements MVCC and relies on total ordering of transaction IDs. Something I didn't want to do to avoid forced synchronization points (multiple clients need to fight for IDs). This is certainly a trade-off, because in my case you are forced to read the latest version or retry (but then you get strict serializability), while in Delta Lake you can rely on snapshot isolation, which might give you slightly stale, but consistent data and minimize retries on reads.
It also seems that you can't get transactions across different tables? Another interesting tradeoff.