Live data from Hacker News

I am building a cloud

crawshaw.io

571–580 of 589 posts

Re: I am building a cloud

#571
post #399

Lots of negativity towards k8s in here. It's always funny to me when $WILDLY_POPULAR_TECH gets ripped apart like this, as though no one has ever had a positive experience with it. I've seen similar pile-ons for React, microservices, git, PHP, JavaScript, cloud services, really anything that's been adopted at scale.

people love to make posts like "I used k8s to host my personal website from my homelab and it was too complicated!"

Re: I am building a cloud

#573

Earlier quoted context omitted.

Are you using a service like GKE, or running your own cluster? I've set up clusters "from scratch" in dev environments to get familiar with it.

We do both: managed Kubernetes when it's available (AWS, Nebius, others), but for some hardware vendors they just give us raw machines and we self-host K3s on their nodes. We're an open-source LLM inference company so we're basically always scrambling for GPUs wherever we can get them, which means we need to be fairly scrappy with what we support while still having a semi-sane interface for ourselves internally. Kube…

This makes a lot of sense and is like the perfect use case for programming towards a "standard api".

I run it at home and at work, and while I do hate installing it, once that part is done I've never run into these problems that people claim requires a 20 person(!) team to baby sit it. Maybe my scale is too small or whatever, but its hard not to think that maybe they are just "holding it wrong"...

Re: I am building a cloud

#574

Earlier quoted context omitted.

I've run an Openstack cloud. Local to the host NVME's directly attached to VMs is unbeatable. All clouds offer this. But that storage is ephemeral and it was when I implemented it in Openstack too. There's not enough redundancy. You could raid1 those NVME's when before they get attached to a VM and that helps with hardware failures, but you get less of them to attach. Even if you RAID them, there's not a good way to…

> There's not enough redundancy So build resiliency into your application layer.

You don't always choose the application layer.

When you're an OpenStack cloud provider, your customers choose.

When you're a customer using Open Source software, your vendors choose.

Using a mixture of directly attached NVMe and network-attached volumes with backup is the sweet spot for me.

I don't need to maintain my own network filesystem (Ceph), and I can put applications that mirrors its database natively on NVMe and everything I don't have much control over on network-attached volumes.

I feel like there's something better not yet made.

Re: I am building a cloud

#575

Earlier quoted context omitted.

> Docker is great development tooling (still some rough edges, of course). Show me a Docker in use where build caching was solved optimally for development builds (like eg. make did for C 40 or 50 years ago)? Perhaps you consider Docker layers one of the "rough edges", but I believe instant, iterative development builds are a minimum required for "great development tooling". I did have great fun optimizing Docker bui…

A multi-stage Docker build where you separate pulling in dependencies from building the thing you want is as close as you're going to get. Something like the following works well in practice: 1) pinned base image (e.g. Ubuntu LTS) 2) your own custom base image in a registry rebuilt whenever you want (e.g. with tools you need for debugging or available across all of your images) 3) your own runtime-specific base image…

Start thinking about things like:

- I changed one source file, I need to rebuild the whole app

- I upped one dependency by a minor version

...

How long is the rebuild time? As I said, compare it to simple, standard make-based setup.

The problem is that layers are really — in the end — sequential, and even a simple change in an earlier one invalidates the latter cached layers.

I do not consider up-to-few-seconds for cases like above "very high standards", but it is a lot of work and care with Docker — especially as you need it not to mess up development or production environments as people skip versions and similar (if you use tricks like reading some config from a file).

Re: I am building a cloud

#576
post #290

Earlier quoted context omitted.

You can ack based on groups, and you can out users into groups. So if you auth a node, it’s now your node and the ACL for your user / group will apply. But yes I don’t think you can ACL based o the hostname

Hi there, I work at Tailscale. Part of the reason that we don't (currently) let you do this is that a hostname is a user-reported field, and can change over time; it's not a durable form of identity that you can write ACLs on. One could imagine, for example: 1. Creating an ACL rule that allows hostname "webserver" to hostname "db". 2. (time passes) 3. Hostname "webserver" is deleted/changed to "web"/etc. 4. Someone c…

That's why my next instinct was to try specifying a node key, which does not change unless the device is re-registered, but that does not work either.

Re: I am building a cloud

#577
post #105

> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…

I think this comment and replies capture the problem with Kubernetes. Nobody gets fired for choosing Kubernetes now. It's obvious to you, me and the other 2 presumably techie people who've responded within 15 mins that you shouldn't have been using Kubernetes. But you probably work in a company of full of techie people, who ended up using Kubernetes. We have HN, an environment full of techie people here who immediate…

You hit the nail on the head here. Saying and doing are two very different things. It's also especially tempting to find an excuse to use some shiny new thing that everyone is always talking about. Both for personal learning and curiosity but also for future job prospects. The reality is that it's easier to get a k8s job if you have k8s experience.

Re: I am building a cloud

#578
post #395

Earlier quoted context omitted.

How did you eventually get funding after those initial rejections? What changed?

One VC (well, two) understood it. Despite what you hear, there is a lot of variation. Speak to a lot of people.

Surprising that a founding team as strong as Tailscale's had to go door-to-door to get seed funding in. Glad that Tailscale did & changed the industry like it ought to, though I'm sure, y'all would have managed to self-fund it either way.

Re: I am building a cloud

#579

Earlier quoted context omitted.

I took over tech for a POS company some years ago. They were a .net shop with about 80 developers, less than 200 concurrent connections, 6 figures spend cloud, and 0 nines uptime with a super traditional setup. Point being, it's not the tools the causes the probem.

Read this as a Piece of sh... company. Then I saw response of someone saying they're a POS developer and was like oh I think he means point of sale. Or that guy is just a really bad programmer.

I worked at a company that developed a niche POS as part of a larger system. It was, by far, the worst part of the code base. Just imagine a bunch of late 90's era Java 1.2 code, complete with a Swing UI, tons of concurrency issues, singleton objects and synchronized blocks all over the place, custom binary protocols...

Re: I am building a cloud

#580
exe.dev は大好きなプロジェクトです。単一の仮想マシン/ローカルマシンに全てを詰め込むアイデアが好きです。私は単一マシン上でのワークフローオーケストレーションの複雑性に関わる問題を解決するため Dagu.sh を開発しています。
Post reply on HN