Have you considered using techniques like conditional PUT to enable multiple writers? https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3... https://docs.aws.amazon.com/AmazonS3/latest/userguide/condit...
Short answer: conditional PUTs for distributed scare me for multiwriter. The issue isn't doing the writes, it's ensuring that the writer is writing against the most current data. For OLTP workloads with upserts, this is very hard! If you have immutable data without any upserts and your writes don't depend on reads, that actually works really well. But in any other scenario it's dangerous. The writer needs to ensure that the other writers have checkpointed first, and there's not a great way to do that.
One thing that could make this work in turbolite is a fast distributed lock system with transaction and lock timeouts. E.g. use Redis as the lock lease holder, and distributed writers acquire it, fetch the latest manifest from s3, sync any data they need, do the write, checkpoint, update the lock to be released and note the new manifest version, and return success to the user. then before any reads, they simple check Redis for the new manifest (or S3 for guarantee that the manifest update/lock release didn't fail). All writes have an N second timeout, and the write lock has N second + T timeout, so it's guaranteed that successful checkpoints are used in the next read, as long as readers check the manifest first.
This could work, but it's single writer lol. But you'd only want it with infrequent writes. And reads cost an S3 GET. So I guess it would work best with Wasabi, which doesn't charge for operations, or self-hosted MinIO.