Viewing profile — thomas_fa

[dead]

We would wait for Bss data and journal DirectIO and the acking (sending response back to api_server) in the callback function. What you are implying is what s3 actually doing and y…

Well said and there are some bitter lessons in the storage industry. In my last company we need to disable the disk write cache during each reboot, and we also heard a lot industry…

Thanks for the feedback, since I have relied in other thread related to O_DSYNC which a lot of folks have already suggested, and I will not repeat it here. For the benchmark result…

That's a lot of valuable information and thanks for the input. Yes the original blog post is mainly focusing on reducing the metadata overhead due to fsync(), and I got a lot of go…

Yes, especially for our object storage each putObject would need to create new entry for in the (data)name space which would need fsync for dir fd.

Thanks for the kind words! You check more of our work in https://github.com/fractalbits-labs/fractalbits .

Thanks for the encouragement! Another author here. Yes, if you are interested you can check our another blog [1] for the internal storage engine. Yes, we are limiting the delimeter…

Yes, that's right. We could go even further, to use the raw devices without relying on any filesystem. We then need to allocate/format raw disk spaces and we can not just open file…

Yes, that has also been pointed out in other threads. Yes this could be very important settings, and even some of common Linux file systems actually don't do that every time and we…

Yes, as we mentioned in the post, it is targeted for the virtualized NVME disk and we don't have control for actually issing FUA command. We are also changing to open data files wi…

Thanks for pointing it out the mistakes. We should make it clearer, when fsync an opened file descriptor, it would only sync its own metadata. To make it truly persistent, we need …

Congrats to the TigerBeetle team's new feature! and it looks like TB has already moved from shared nothing to partially shared disk (object storage) architecture. We are always a b…

Indeed this is not an easy problem. And our s3-compatible system do support the atomic rename with extended protocol in a graceful way, see the demo with our tool [1]. [1] https://…

Thanks for the analysis. Interestingly when we first released our low latency s3-compatible storage (1M IOPS, p99 ~5ms)[1], a lot of people asking the same questions why we tried t…

A lot of good insights here. I am also wandering if they can just simply put different jobs (unclaimed, in-progress, deleted/done) into different directory/prefix, and rely on atom…

well I checked the code in their repo，but could not find any snippets as mentioned in the post，which made the claimed perf numbers very suspicious. There was also no cluster config…