love this. one person, 119GB, two drives, rsync. no kubernetes, no distributed cluster, no nonsense. just works.
this is the kind of setup that lets you actually go to bed without checking your phone every 20 minutes.
Healthchecks.io now uses self-hosted object storage
61–70 of 82 posts
Re: Healthchecks.io now uses self-hosted object storage
#62As someone who has dealt with wacky storage issues/designs, a lot of this "felt" strange to me. Btrfs? Rsync? Then I got to the bottom and saw that they were only handling about 100 GB of data! At that scale, nearly anything will work great and TFA was right to just pick the thing with the fewest knobs. At a previous job years ago, we had a service that was essentially a file server for something like 50TB of tiny fi…
what happens if you suffer catastrophic failure, you'd lose up to one hour of data?
The system had RAID, so disks could be replaced without taking anything down.
We never had a catastrophic failure, but yeah, and hour of lost data wouldn't be the end of the world. For regular maintenance we would just coordinate with the QA team to pause testing temporarily. (Testing was only around-the-clock near a product release.)
Re: Healthchecks.io now uses self-hosted object storage
#63Earlier quoted context omitted.
The S3 API doesn't work like normal filesystem APIs. Part of it is that it follows the object storage model, and part of it is just to lock people into AWS once they start working with it.
Apart from all these other products that implement s3? MinIO, Ceph (RGW), Garage, SeaweedFS, Zenko CloudServer, OpenIO, LakeFS, Versity, Storj, Riak CS, JuiceFS, Rustfs, s3proxy.
Add Tigris to the list as well please.
We maintain a page that shows our compatibility with S3 API. It's at https://www.tigrisdata.com/docs/api/s3/. The test runner is open source at https://github.com/tigrisdata-community/s3-api-compat-tests
Re: Healthchecks.io now uses self-hosted object storage
#64I don't get it, if it's running on the same (mentioning "local") machine, why does it even need the S3 API? Could just be plain IO on the local drive(s)
The S3 API doesn't work like normal filesystem APIs. Part of it is that it follows the object storage model, and part of it is just to lock people into AWS once they start working with it.
The API has sort of become a standard. There are many providers providing S3 API-compatible storage.
Re: Healthchecks.io now uses self-hosted object storage
#65how do you provide highly available fault tolerant storage?
versity does not include any erasure coding or replication...
Re: Healthchecks.io now uses self-hosted object storage
#66> In March 2026, I migrated to self-hosted object storage powered by Versity S3 Gateway. Thanks for sharing this, I wasn't even aware of Versity S3 from my searches and discussions here. I recently migrated my projects from MinIO to Garage, but this seems like another viable option to consider.
First time hearing about Versity for me too. I thought "S3 Gateways" were an Amazon-only service rather than something mere mortals could set up. I've been trying to give some containers (LXC/D and OCI) unprivileged access to a network-accessible ZFS filesystem and this might be what I need. Managing UID/GID through bind-mounts from the host to the container (ie NFS on host) has been trickier than I was expecting.
Not really s3, and I haven't touched LXC in a long but this may help on the OCI side. I apologize if this is redundant to you.
Remember that UID mapping on namespaces is just a facade with an offset and a range typically based on subuid[0] and subgid[1] today.
In the container `cat /proc/self/uid_map` or by looking at the pid from the host `cat /proc/$PID/uid_map` you can tell what those offsets are.
$ cat /proc/self/uid_map
0 1000 1
1 100000 65536
The Container PID offset of 1 maps to the host offset of 100000 for a length of 65536
With subuid/subgid you can assign ranges to the user that is instantiating the container, in the flowing I have two users that launch containers.
$ cat /etc/subuid
debian:100000:65536
runner:165536:65536
$ cat /etc/subgid
debian:100000:65536
runner:165536:65536
But note that only highly trusted containers should ever really use host bind mounts, it is often much safer to use mount NFS internally.
Host bind mounts of network filesystems, if that is what you are doing, is also fragile as far as dataloss goes. I am an object store fan, but just wanted to give you the above info as it seems hard for people to find.
I would highly encourage you to look into the history of security problems with host bind mounts to see the wack-a-mole that is required with them to see if it fits in with your risk appitite. But if you choose to use them, setting up dedicated uid/gid mappings and setting the external host to the expected effective ID of the container users is a better way than using Idmapped mounts etc...
[0] https://man7.org/linux/man-pages/man5/subuid.5.html [1] https://man7.org/linux/man-pages/man5/subgid.5.html
Re: Healthchecks.io now uses self-hosted object storage
#67> In March 2026, I migrated to self-hosted object storage powered by Versity S3 Gateway. Thanks for sharing this, I wasn't even aware of Versity S3 from my searches and discussions here. I recently migrated my projects from MinIO to Garage, but this seems like another viable option to consider.
First time hearing about Versity for me too. I thought "S3 Gateways" were an Amazon-only service rather than something mere mortals could set up. I've been trying to give some containers (LXC/D and OCI) unprivileged access to a network-accessible ZFS filesystem and this might be what I need. Managing UID/GID through bind-mounts from the host to the container (ie NFS on host) has been trickier than I was expecting.
It sounds like you're not using Incus, but for those who are, it looks like there is a S3 compatible object storage API built in. I haven't used it for anything yet but it seems relevant:
https://linuxcontainers.org/incus/docs/main/howto/storage_bu...
Re: Healthchecks.io now uses self-hosted object storage
#68Earlier quoted context omitted.
Riak CS been dead for over a decade which makes me question the rest. Some of these also do not have the same behaviors when it comes to paths (MinIO is one of those IIRC). Also, none of them implement full S3 API and features.
There's a difference between S3 API spec and what Amazon does with S3 - for isntance, the new CAS capabilities with Amazon are not part of the spec. Ceph certainly implements the full API spec, though it may lag behind some changes. It's mostly a question of engineering time available to the projects to keep up with changes.
> There's a difference between S3 API spec and what Amazon does with S3 - for isntance, the new CAS capabilities with Amazon are not part of the spec.
Sure, but those are S3 APIs and features that provided by AWS. We not talking about S3 spec, we're talking about s3 product.
Re: Healthchecks.io now uses self-hosted object storage
#69You can wire lsyncd for near real-time syncs and btrfs snapshots for offline storage
Re: Healthchecks.io now uses self-hosted object storage
#70For S3 self-hosters check out Garage, who are developing with NLnet funding: https://garagehq.deuxfleurs.fr
Covered by the article:
>I ran local experiments with Minio, SeaweedFS, and Garage. My primary objection to all of them was the operational complexity