Live data from Hacker News

Healthchecks.io now uses self-hosted object storage

blog.healthchecks.io

71–80 of 82 posts

Re: Healthchecks.io now uses self-hosted object storage

#71
post #66

Earlier quoted context omitted.

First time hearing about Versity for me too. I thought "S3 Gateways" were an Amazon-only service rather than something mere mortals could set up. I've been trying to give some containers (LXC/D and OCI) unprivileged access to a network-accessible ZFS filesystem and this might be what I need. Managing UID/GID through bind-mounts from the host to the container (ie NFS on host) has been trickier than I was expecting.

Not really s3, and I haven't touched LXC in a long but this may help on the OCI side. I apologize if this is redundant to you. Remember that UID mapping on namespaces is just a facade with an offset and a range typically based on subuid[0] and subgid[1] today. In the container `cat /proc/self/uid_map` or by looking at the pid from the host `cat /proc/$PID/uid_map` you can tell what those offsets are. $ cat /proc/self…

I appreciate the detailed explanation of ID/GID mapping.

> it is often much safer to use mount NFS internally

This is the config I'm trying to move away from! I don't see how an unprivileged LXC with a bind mount is worse than a privileged container with NFS, FUSE, and nesting enabled (I need all of that if I can't aggregate on the host).

NFS and CIFS within the container requires kernel-level access and therefore the LXC must be privileged. I'd rather have a single defined path.

I tried to get around this using FUSE but it creates its own issues with snapshots/backups (fsfreeze).

If my solutiom works for a regular LXC it will probably work for Podman.

Re: Healthchecks.io now uses self-hosted object storage

#73
post #24
post #12

Earlier quoted context omitted.

> part of it is just to lock people into AWS once they start working with it. This is some next-level conspiracy theory stuff. What exactly would the alternative have been in 2006? S3 is one of the most commonly implemented object storage APIs around, so if the goal is lock-in, they're really bad at it.

> What exactly would the alternative have been in 2006? Well, WebDAV (Document Authoring and Versioning) had been around for 8 years when AWS decided they needed a custom API. And what service provider wasn't trying to lock you into a service by providing a custom API (especially pre-GPT) when one existed already? Assuming they made the choice for a business benefit doesn't require anything close to a conspiracy theo…

When S3 launched the core API could be described with 4 requests. It was (and still mostly is) super simple.

Saying they should have used WebDAV instead shows a lack of knowledge on your end rather than theirs.

Re: Healthchecks.io now uses self-hosted object storage

#74
post #66

Earlier quoted context omitted.

Not really s3, and I haven't touched LXC in a long but this may help on the OCI side. I apologize if this is redundant to you. Remember that UID mapping on namespaces is just a facade with an offset and a range typically based on subuid[0] and subgid[1] today. In the container `cat /proc/self/uid_map` or by looking at the pid from the host `cat /proc/$PID/uid_map` you can tell what those offsets are. $ cat /proc/self…

I appreciate the detailed explanation of ID/GID mapping. > it is often much safer to use mount NFS internally This is the config I'm trying to move away from! I don't see how an unprivileged LXC with a bind mount is worse than a privileged container with NFS, FUSE, and nesting enabled (I need all of that if I can't aggregate on the host). NFS and CIFS within the container requires kernel-level access and therefore th…

I can't comment directly on LXC but LXC is very different from runc/crun/your-CRI here, not better or worse, just different.

With podman, unfortunately we don't k8s Container Storage Interface (CSI), so you have to work with what you have.

When I said:

> it is often much safer to use mount NFS internally

What is more correct, is having the container runtime or container manager mount them, not the user inside the container.

But as you are trying to run unprivileged or at least with minimal privileges, which is all we can do with namespaces, you are cutting across the grain.

I do use podman pods and containers, mostly for the ease of development, but on more traditional long lived hosts.

I have a very real need to separate UIDs between co-hosted products, but don't need to actually run a VM for these specific use cases.

So I have particular rootful tasks that have to be done as the user root root in ansible:

1) Install OS packages 2) Create service admin and daemon user 3) Assign subuid/subgids ranges to those user security domains as needed 4) For specific services add NFS data directories to /etc/fstab with the 'user' and 'noauto' flags

In Podman I would then create

     podman volume create --driver local --opt type=nfs --opt device=192.168.1.84:/path/to/share --opt o=addr=192.168.1.84,....

     podman run -d  --name nfs_test -v nfs-shared:/opt docker.io/library/debian:latest
Which if you don't have the fstab entry will give you:

     Error: mounting volume nfs-shared for container ...: mount.nfs: Operation not permitted for 192.168.1.84:/path/to/share on /home/user/.local/share/containers/storage/volumes/nfs-shared/_data
That `_data` is one of the hints of the risk of host bind mounts, the risk is either having an inode that the host cares about or issues across containers etc...

While imperfect, this is following the named volume pattern, which really just uses tells about it being in a container and doesn't expose the mount inode to the container.

What does happen inside the container entry point is validating that the expected UID is reachable, adding a user with the right UID offset and switching to that user.

A misconfigured host bind mount or leaking because you can't view who has access are the most common problems, and as containers run with elevated privileges until you drop them they can get around those protections, even if they aren't elevating to root in a rootless situation, they can still access the data of any running container with just a few trivial mistakes or new discovered vulnerabilities.

    $ capsh --decode=00000000800405fb 0x00000000800405fb=cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_sys_chroot,cap_setfcap

While NFS is absolutely a whole new ball of wax with other issues, one nice thing is that (at least the servers I know of) don't even support the concept of user namespaces and UID mapping, which makes it fragile and dangerous if you start mapping uid/gid's in, but can be an advantage if you can simply isolate uid/gid ranges.

IMHO it will be horses for courses and depend on your risk appetite as all options are least worst and there simply will be no best option, especially with OCI.

Re: Healthchecks.io now uses self-hosted object storage

#75
post #70
post #58

For S3 self-hosters check out Garage, who are developing with NLnet funding: https://garagehq.deuxfleurs.fr

Covered by the article: >I ran local experiments with Minio, SeaweedFS, and Garage. My primary objection to all of them was the operational complexity

Out of the listed options, Garage is actually really simple for something that runs on multiple servers. It was also the only easy option I found that can replicate over WAN and isn't super latency sensitive.

Re: Healthchecks.io now uses self-hosted object storage

#76
post #74

Earlier quoted context omitted.

I appreciate the detailed explanation of ID/GID mapping. > it is often much safer to use mount NFS internally This is the config I'm trying to move away from! I don't see how an unprivileged LXC with a bind mount is worse than a privileged container with NFS, FUSE, and nesting enabled (I need all of that if I can't aggregate on the host). NFS and CIFS within the container requires kernel-level access and therefore th…

I can't comment directly on LXC but LXC is very different from runc/crun/your-CRI here, not better or worse, just different. With podman, unfortunately we don't k8s Container Storage Interface (CSI), so you have to work with what you have. When I said: > it is often much safer to use mount NFS internally What is more correct, is having the container runtime or container manager mount them, not the user inside the con…

Wow, I really appreciate you coming back for the follow-up! It's too late for me to read through it in detail at this moment but:

In the end I discovered that I can combine a "mapall/squash" on the NFS server, a regular NFS mount on Proxmox, and then an `lxc.mount.entry` for the LXC config and the combined effect is an unprivileged container with read-write permissions for the UID/GID specified on the NFS server. If I need more UID/GID combinations I can just create bind mounts and then export those with the appropriate mapall/squash settings.

Thanks again :)

Re: Healthchecks.io now uses self-hosted object storage

#77
post #68

Earlier quoted context omitted.

There's a difference between S3 API spec and what Amazon does with S3 - for isntance, the new CAS capabilities with Amazon are not part of the spec. Ceph certainly implements the full API spec, though it may lag behind some changes. It's mostly a question of engineering time available to the projects to keep up with changes.

> There's a difference between S3 API spec and what Amazon does with S3 - for isntance, the new CAS capabilities with Amazon are not part of the spec. Sure, but those are S3 APIs and features that provided by AWS. We not talking about S3 spec, we're talking about s3 product.

You’re wrong. “Implementing the S3 API” means the spec, not an Amazon product.

Re: Healthchecks.io now uses self-hosted object storage

#78
post #68

Earlier quoted context omitted.

> There's a difference between S3 API spec and what Amazon does with S3 - for isntance, the new CAS capabilities with Amazon are not part of the spec. Sure, but those are S3 APIs and features that provided by AWS. We not talking about S3 spec, we're talking about s3 product.

You’re wrong. “Implementing the S3 API” means the spec, not an Amazon product.

We're talking about using AWS S3 and using something that implements S3 spec. Really not that hard to understand.

When customer wants to switch away from AWS S3, customer cares about AWS S3 feature coverage and not the spec.

Re: Healthchecks.io now uses self-hosted object storage

#79
post #78

Earlier quoted context omitted.

You’re wrong. “Implementing the S3 API” means the spec, not an Amazon product.

We're talking about using AWS S3 and using something that implements S3 spec. Really not that hard to understand. When customer wants to switch away from AWS S3, customer cares about AWS S3 feature coverage and not the spec.

Spec and features are intertwined. Customers who switch away from AWS S3 still want to use the same SDKs, libraries, etc that support S3 API. They don't want to rewrite their applications to use a new API. So then is it feature coverage or spec coverage?

Re: Healthchecks.io now uses self-hosted object storage

#80
post #78

Earlier quoted context omitted.

We're talking about using AWS S3 and using something that implements S3 spec. Really not that hard to understand. When customer wants to switch away from AWS S3, customer cares about AWS S3 feature coverage and not the spec.

Spec and features are intertwined. Customers who switch away from AWS S3 still want to use the same SDKs, libraries, etc that support S3 API. They don't want to rewrite their applications to use a new API. So then is it feature coverage or spec coverage?

It's both? Customer doesn't care if spec is 100% covered if feature that they are using in AWS S3 isn't supported.

Also, who is rewriting their application to change interaction with an object storage? People that directly use some S3 sdk all around the app should read a book on software engineering or at least a blog post.

Post reply on HN