Live data from Hacker News

I am building a cloud

crawshaw.io

311–320 of 589 posts

Re: I am building a cloud

#311

> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…

A single VM is indeed the most pragmatic setup that most apps really need. However I still prefer to have at least two for little redundancy and peace of mind. It’s just less stressful to do any upgrades or changes knowing there is another replica in case of a failure. And I’m building and happily using Uncloud ( https://github.com/psviderski/uncloud ) for this (inspired by Kamal). It makes multi-machine setups as si…

People have it backwards.

If you have an app and you want to run a single app yeah silly to look for K8s.

If you have a beefy server or two you want to utilize fully and put as many apps on it without clashing dependencies you want to use K8s or docker or other containers. Where K8s enables you to go further.

Re: I am building a cloud

#312

> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…

So... if you're at the point where you're using a single VM, I have to ask why bother with docker at all? You're paying a context switch overhead, memory overhead, and disk overhead that you do not need to. Just make an image of the VM in case you need to drop it behind an LB.

If you've ever had the displeasure of seeing the sorry state of VM tooling you would have known that building custom VM images is a very complicated endeavour compared to podman build or docker build.

I once tried to build a simple setup using VM images and the complexity exploded to the point where I'm not sure why anyone should bother.

When building a container you can just throw everything into it and keep the mess isolated from other containers. If you use a VM, you can't use the OCI format, you need to build custom packages for the OS in question. The easiest way to build a custom package is to use docker. After that you need to build the VM images which requires a convoluted QEMU and libvirt setup and a distro specific script and a way to integrate your custom packages. Then after all of this is done you still need to test it, which means you need to have a VM and you need to make it set itself up upon booting, meaning you need to learn how to use cloud-init.

Just because something is "mature" doesn't mean it is usable.

The overhead of docker is basically insignificant and imperceptible (especially if you use host networking) compared to the day to day annoyances you've invited into your life by using VM images. Starting a a VM for testing purposes is much slower than starting a container.

Re: I am building a cloud

#313

Earlier quoted context omitted.

Kubernetes is not bad, it's just low level. Most applications share the exact same needs (proof: you could run any web app on a simple platform like Heroku). That's why some years ago I built an open source tool (with 0 dependencies) that simplify Kubernetes deployments with a compact syntax which works well for 99% of web apps (instead of allowing any configuration, it makes many "opinionated" choices): https://gith…

It's not just low level, in most cases, it's also overkill. Most companies aren't "web scale" ™ and don't need an orchestrator built for google level elasticity, they need a vm autoscaling group if anything. Most apps don't need such granular control over fs access, network policies, root access, etc, they need `ufw allow 80 && ufw enable` Most apps don't need a 15 stage, docker layer caching optimized, archive promo…

If this works `git clone me@github.com:me/mine.git release_01 && ln -s release_01 /var/www/me/mine/current` then your Docker builds should also be extremely quick. Where I have seen extremely slow docker builds is with Python services using ML libraries. But those I reallly don't want to be building on the production servers.

"ECS would have worked for 99% of these apps, if they even needed that."

I used to agree with that but is EKS really that much more complicated? Yes you pay for the k8s control plane but you gain tooling that is imho much easier to work with than IaC.

Re: I am building a cloud

#314
post #290

Earlier quoted context omitted.

Last I read the docs while troubleshooting this very problem, you cannot specify node names as the source or destination of a grant. You can specify direct IP address ranges, node groups (including autogenerated ones) or tags, but not names. Tags permanently erase the user identity from a device, and disable things like Taildrop. When I tried to assign a tag for ACLs, I found that I then could not remove it and had t…

You can ack based on groups, and you can out users into groups. So if you auth a node, it’s now your node and the ACL for your user / group will apply. But yes I don’t think you can ACL based o the hostname

Hi there, I work at Tailscale.

Part of the reason that we don't (currently) let you do this is that a hostname is a user-reported field, and can change over time; it's not a durable form of identity that you can write ACLs on. One could imagine, for example:

1. Creating an ACL rule that allows hostname "webserver" to hostname "db".

2. (time passes)

3. Hostname "webserver" is deleted/changed to "web"/etc.

4. Someone can now register a user device with the system hostname set to "webserver"

Should they be allowed to inherit the pre-existing ACL rule?

However, you can accomplish something very close to what you're asking for, I think, by defining a "host" in the policy file (https://tailscale.com/docs/reference/syntax/policy-file#host...) that points to a single Tailscale IP. Since we don't allow non-admins to change their Tailscale IP, this uniquely identifies a single device even if the hostname changes, and thus you can write a policy similar to:

  "hosts": {
    "myhost": "100.64.1.2",
  },
  "grants": [
    {
      "src": ["myhost"],
      "dst": ["tag:db"],
    },
  ]

Re: I am building a cloud

#315

> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…

A single VM is indeed the most pragmatic setup that most apps really need. However I still prefer to have at least two for little redundancy and peace of mind. It’s just less stressful to do any upgrades or changes knowing there is another replica in case of a failure. And I’m building and happily using Uncloud ( https://github.com/psviderski/uncloud ) for this (inspired by Kamal). It makes multi-machine setups as si…

this is dope work.

Re: I am building a cloud

#316
post #85
post #49

> Agents, by making it easiest to write code, means there will be a lot more software. Economists would call this an instance of Jevons paradox. Each of us will write more programs, for fun and for work. There is already so much software out there, which isn't used by anyone. Just take a look at any appstore. I don't understand why we are so obsessed with cranking out even more, whereas the obvious usecase for LLMs s…

There will be only 1 Microsoft® Excel, 1 Google Sheets and 1 LibreOffice and the rest are billions of dead vibe-coded "Excel killers" that no-one uses.

Democratization of software through SaaS & new engineers brought Airtable, Smartsheet, Baserow, Monday, and many more that I can't remember though.

Re: I am building a cloud

#317
finally a cloud 'vendor' that understands that modern computers are fast.

if we go back to the principle that modern computers are really fast, SSDs are crazy fast

and we remove the extra cruft of abstractions - software will be easier to develop - and we wouldn't have people shilling 'agents' as a way for faster development.

ultimately the bottleneck is our own thinking.

simple primitives, simpler thinking.

Re: I am building a cloud

#318

Earlier quoted context omitted.

There's one extra process that takes up a tiny bit of CPU and memory. For that, you get an immutable host, simple configuration, a minimal SBOM, a distributable set of your dependencies, x-platform for dev, etc.

Yes but NixOS does all of these things already, without the process overhead

Even the minimal SBOM part? It's hard to be more minimal than a busybox binary.

Re: I am building a cloud

#319

Earlier quoted context omitted.

> My agents work inside sandboxes Out of interest, what sandboxing solution do you use?

Not sure what you mean. I use the above linked personal project, bhatti, which internally uses Firecracker microVMs.

Ah, for some reason I didn't make the connection between your VM setup and your agent sandbox setup and thought those were two separate things. Sorry about that!

Re: I am building a cloud

#320

Earlier quoted context omitted.

If you spin up Kubernetes for "a couple of containers to run your web app", I think you're doing something wrong in the first place, also coupled with your comment about adding SDN to Kubernetes. People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.

It depends what you're doing it. My app is fairly simple node process with some side car worker processes. k8s enables me to deploy it 30 times for 30 PRs, trivially, in a standard way, with standard cleanup. Can I do that without k8s? Yes. To the same standard with the same amount of effort? Probably not. Here, I'd argue the k8s APIs and interfaces are better than trying to do this on AWS ( or your preferred cloud p…

Call me old fashion but I prefer tools like Dokploy that make deployment across different VPS extremely easy. Dokploy allows me to utilize my home media server, using local instances of forgejo to deploy code, to great effect.

k8s appears to be a corporate welfare jobs program where trillion dollar multinational monopolistic companies are the only ones who can collectively spend 100s of millions sustaining. Since most companies aren't trillion dollar monopolies, adopting such measures seems extremely poor.

All it signals to me is that we have to stop letting SV + VC dictate the direction of tech in our industry, because their solutions are unsustainable and borderline useless for the vast majority of use cases.

I'll never forget the insurance companies I worked at that orchestrated every single repo with a k8s deployment whose cloud spend was easily in the high six figures a month to handle a work load of 100k/MAU where the concurrent peak never went more than 5,000 users, something the company did know with 40 years of records. Literally had a 20 person team whose entire existence was managing the companies k8s setup. Only reason the company could sustain this was that it's an insurance company (insurance companies are highly profitable, don't let them convince you otherwise; so profitable that the government has to regulate how much profit they're legally allowed to make).

Absolute insanity, unsustainable, and a tremendous waste of limited human resources.

Glad you like it for your node app tho, happy for you.

Post reply on HN