Lots of negativity towards k8s in here. It's always funny to me when $WILDLY_POPULAR_TECH gets ripped apart like this, as though no one has ever had a positive experience with it. I've seen similar pile-ons for React, microservices, git, PHP, JavaScript, cloud services, really anything that's been adopted at scale.
I am building a cloud
571–580 of 589 posts
Re: I am building a cloud
#572Congrats. Just checked your homepage. I love the fact you also show this comment "That must be worst website ever made" Made me love the site and style even more
Re: I am building a cloud
#573Earlier quoted context omitted.
Are you using a service like GKE, or running your own cluster? I've set up clusters "from scratch" in dev environments to get familiar with it.
We do both: managed Kubernetes when it's available (AWS, Nebius, others), but for some hardware vendors they just give us raw machines and we self-host K3s on their nodes. We're an open-source LLM inference company so we're basically always scrambling for GPUs wherever we can get them, which means we need to be fairly scrappy with what we support while still having a semi-sane interface for ourselves internally. Kube…
I run it at home and at work, and while I do hate installing it, once that part is done I've never run into these problems that people claim requires a 20 person(!) team to baby sit it. Maybe my scale is too small or whatever, but its hard not to think that maybe they are just "holding it wrong"...
Re: I am building a cloud
#574Earlier quoted context omitted.
I've run an Openstack cloud. Local to the host NVME's directly attached to VMs is unbeatable. All clouds offer this. But that storage is ephemeral and it was when I implemented it in Openstack too. There's not enough redundancy. You could raid1 those NVME's when before they get attached to a VM and that helps with hardware failures, but you get less of them to attach. Even if you RAID them, there's not a good way to…
> There's not enough redundancy So build resiliency into your application layer.
When you're an OpenStack cloud provider, your customers choose.
When you're a customer using Open Source software, your vendors choose.
Using a mixture of directly attached NVMe and network-attached volumes with backup is the sweet spot for me.
I don't need to maintain my own network filesystem (Ceph), and I can put applications that mirrors its database natively on NVMe and everything I don't have much control over on network-attached volumes.
I feel like there's something better not yet made.
Re: I am building a cloud
#575Earlier quoted context omitted.
> Docker is great development tooling (still some rough edges, of course). Show me a Docker in use where build caching was solved optimally for development builds (like eg. make did for C 40 or 50 years ago)? Perhaps you consider Docker layers one of the "rough edges", but I believe instant, iterative development builds are a minimum required for "great development tooling". I did have great fun optimizing Docker bui…
A multi-stage Docker build where you separate pulling in dependencies from building the thing you want is as close as you're going to get. Something like the following works well in practice: 1) pinned base image (e.g. Ubuntu LTS) 2) your own custom base image in a registry rebuilt whenever you want (e.g. with tools you need for debugging or available across all of your images) 3) your own runtime-specific base image…
- I changed one source file, I need to rebuild the whole app
- I upped one dependency by a minor version
...
How long is the rebuild time? As I said, compare it to simple, standard make-based setup.
The problem is that layers are really — in the end — sequential, and even a simple change in an earlier one invalidates the latter cached layers.
I do not consider up-to-few-seconds for cases like above "very high standards", but it is a lot of work and care with Docker — especially as you need it not to mess up development or production environments as people skip versions and similar (if you use tricks like reading some config from a file).
Re: I am building a cloud
#576Earlier quoted context omitted.
You can ack based on groups, and you can out users into groups. So if you auth a node, it’s now your node and the ACL for your user / group will apply. But yes I don’t think you can ACL based o the hostname
Hi there, I work at Tailscale. Part of the reason that we don't (currently) let you do this is that a hostname is a user-reported field, and can change over time; it's not a durable form of identity that you can write ACLs on. One could imagine, for example: 1. Creating an ACL rule that allows hostname "webserver" to hostname "db". 2. (time passes) 3. Hostname "webserver" is deleted/changed to "web"/etc. 4. Someone c…
Re: I am building a cloud
#577> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…
I think this comment and replies capture the problem with Kubernetes. Nobody gets fired for choosing Kubernetes now. It's obvious to you, me and the other 2 presumably techie people who've responded within 15 mins that you shouldn't have been using Kubernetes. But you probably work in a company of full of techie people, who ended up using Kubernetes. We have HN, an environment full of techie people here who immediate…
Re: I am building a cloud
#578Earlier quoted context omitted.
How did you eventually get funding after those initial rejections? What changed?
One VC (well, two) understood it. Despite what you hear, there is a lot of variation. Speak to a lot of people.
Re: I am building a cloud
#579Earlier quoted context omitted.
I took over tech for a POS company some years ago. They were a .net shop with about 80 developers, less than 200 concurrent connections, 6 figures spend cloud, and 0 nines uptime with a super traditional setup. Point being, it's not the tools the causes the probem.
Read this as a Piece of sh... company. Then I saw response of someone saying they're a POS developer and was like oh I think he means point of sale. Or that guy is just a really bad programmer.