Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I am building a cloud

crawshaw.io

Post reply on HN
Print view View original

451–460 of 589 posts

Re: I am building a cloud

#451

Post by germandiago »

dajonker wrote: post #74

> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…

I have designed a backend with exactly the same underlying philosophy as you ended up: load balancer? Oh, a problem. So better client-side hashing and get rid of a discovery service via a couple dns tricks already handled elsewhere robustly.

I took it to its maximum: every service is a piece that can break ---> fewer pieces, fewer potential breakages.

When I can (which is 95% of the time, I add certain other services inside the processed themselves inside the own server exes and make them activatable at startup (though I want all my infra not to drift so I use the same set of subservices in each).

But the idea is -- the fewer services, the fewer problems. I just think, even with the trade-offs, it is operationally much more manageable and robust in the end.

Re: I am building a cloud

#452

Post by t_sawyer »

stingraycharles wrote: post #2

Potentially useful context: OP is one of the cofounders of Tailscale. > Traditional Cloud 1.0 companies sell you a VM with a default of 3000 IOPS, while your laptop has 500k. Getting the defaults right (and the cost of those defaults right) requires careful thinking through the stack. I wish them a lot of luck! I admire the vision and am definitely a target customer, I'm just afraid this goes the way things always go…

I've run an Openstack cloud. Local to the host NVME's directly attached to VMs is unbeatable. All clouds offer this. But that storage is ephemeral and it was when I implemented it in Openstack too.

There's not enough redundancy. You could raid1 those NVME's when before they get attached to a VM and that helps with hardware failures, but you get less of them to attach. Even if you RAID them, there's not a good way to move that VM to another host if there's a RAM or CPU or other hardware issue on that host.

These VM's with NVME's directly attached have to basically be treated as bare metal servers and you have to do redundancy at the application layer (like database replication).

But again, all of the major cloud services offer these types of machines if you NEED NVME IO speed. There are quirks though. For example, in Azure it seems like you have to be able to expect the VM to be moved whenever Azure feels like it and expect that ephemeral data to be wiped. Whereas in Openstack, we would do local block level migrations if we HAD to move the VM to another host. That block level migration required the VM to be turned off but it did copy the local NVME data to another host. If this happened it was all planned and the particular application had app level redundancy built in so it was not a problem. If the host crashed, that particular VM would just be down till the host was fixed and came back online.

Re: I am building a cloud

#453

Post by Gareth321 »

skybrian wrote: post #387
Gareth321 wrote: post #269

Earlier quoted context omitted.

If we take it a step further, in a few years, why would anyone purchase generic software anymore? If we can perfectly customise software for our needs and preferences for almost free, why would anyone purchase generic software from an App Store? I genuinely think Apple's business model is in jeopardy.

Most apps aren’t standalone and the services they depend on are nontrivial to build. For example, maybe you could vibe code a guitar tuner app, but not a ride share app.

I agree. The services which will be left standing will be those with a competitive moat: critical mass (Tinder, Facebook), content (YouTube, AppleTV), and scale (frontier AI models requiring expensive hardware), etc.

That said, if you look at the apps on your phone, I wager a large proportion don't have these moats. Translation, passwords, budget, reminders, email, to do, project management, messaging, browser, calendar, fitness, games, game tracking, etc.

Re: I am building a cloud

#454

Post by Hasz »

ipsento606 wrote: post #419
Hasz wrote: post #412

Earlier quoted context omitted.

Took a while to find this. K8s is great, IMO most of the people with alternative setups are just rebuilding (usually worse) or compressing (specific to their use case) k8s features that have been GA for a long time. Spend some time learning it, using it to deploy simple apps, and you won't go back to deploying in a VM again imo. This only gets better with ai-assisted development, any model is going to produce much be…

I deploy prod by running a shell script I wrote that rsyncs the latest version of the codebase to my server, then sshs into the server and restarts the relevant services how could k8s improve my deployment process?

You know your app better than me, but here are some practical reasons for the typical B2C app:

split deployments -- perhaps you want to see how an update impacts something: if error rates change, if conversion rates change, w/e. K8s makes this pretty easy to do via something like a canary or blue green deployment. Likewise, if you need to rollback, you can do this easily as well from a known good image.

Perhaps you need multiple servers -- not for scale -- but to be closer to your users geographically. 1 server in each of -5-10 AZs makes the updates a bit more complicated, especially if you need to do something like a db schema update.

Perhaps your traffic is lumpy and peaks during specific times of the year. Instead of provisioning a bigger VM during these times, your would prefer to scale horizontally automatically. Likewise, depending on the predictable-ness of the distribution of traffic, running a larger machine all the time might be very expensive for only the occasional burst of traffic.

To be very clear, you can do all of this without k8s. The question is, is it easier to do it with or without? IMO, it is a personal decision, and k8s makes a lot of sense to me. If it doesn't make a ton of sense for your app, don't use it.

Re: I am building a cloud

#455

Post by mbesto »

adamtulinius wrote: post #89
dajonker wrote: post #74

> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…

If you spin up Kubernetes for "a couple of containers to run your web app", I think you're doing something wrong in the first place, also coupled with your comment about adding SDN to Kubernetes. People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.

> it sounds like you don't have the scale for actually running Kubernetes.

You don't set up k8s because your current load can't be handled, you do for future growth. Sometimes that growth doesn't pan out and now you're left with a complex infrastructure that is expensive to maintain and not getting any of the benefit.

Re: I am building a cloud

#456

Post by KaiserPro »

dajonker wrote: post #74

> Making Kubernetes good is inherently impossible, a project in putting (admittedly high quality) lipstick on a pig. So well put, my good sir, this describes exactly my feelings with k8s. It always starts off all good with just managing a couple of containers to run your web app. Then before you know it, the devops folks have decided that they need to put a gazillion other services and an entire software-defined netw…

> and an entire software-defined networking layer on top of it.

This is one of the main fuckups of k8s, the networking is batshit.

The other problems is that secrets management is still an afterthought.

The thing that really winds me up is that it doesn't even scale up that much. 2k nodes and it starts to really fall apart.

Re: I am building a cloud

#457

Post by KaiserPro »

KronisLV wrote: post #163
bfivyvysj wrote: post #99

Earlier quoted context omitted.

I thought we collectively learned this with stack overflows engineering blog years ago. Scale vertically until you can't because you're unlikely to hit a limit and if you do you'll have enough money to pay someone else to solve it. Docker is amazing development tooling but it makes for horrible production infrastructure.

Docker is great development tooling (still some rough edges, of course). Docker Compose is good for running things on a single server as well. Docker Swarm and Hashicorp Nomad are good for multi-server setups. Kubernetes is... enterprise and I guess there's a scale where it makes sense. K3s and similar sort of fill the gap, but I guess it's a matter of what you know and prefer at that point. Throw on Portainer on a s…

> Docker Swarm

IS that a thing still?

> Kubernetes is... enterprise

I would contest that. Its complex, but not enterprise.

Nomad is a great tool for running processes on things. The problem is attaching loadbalancers/reverse proxies to those processes requires engineering. It comes for "free" with k8s with ingress controllers.

Re: I am building a cloud

#458

Post by d_burfoot »

I'm not sure if this is the direction the OP is going, but I would love to see a world where local small-time investors can get a bank loan, rent a facility, set up a bunch of computers, and run open-source cloud software on them that provides 95% of the features that most businesses need.

Running a cloud data center could be a business like operating a self-storage facility or a car wash. Small investors love this kind of operation.

Re: I am building a cloud

#459

Post by sgt »

jasonjayr wrote: post #391
marcosdumay wrote: post #294

Earlier quoted context omitted.

Systemd is on the wrong layer here. You need something that can set your machine up, like docker.

Systemd seems to be moving in that direction, the features are coming together to actually enable this. Though imagining the unholy existence of an init system who's only job is to spin up containers, that can contain other inits, OS images, or whatever ..... turtles all the way down.

I don't see why not. Maybe it should be on the same layer going forward - for true cloud compute (including on-premise cloud)

Re: I am building a cloud

#460

Post by giobox »

firesteelrain wrote: post #427
adamtulinius wrote: post #89

Earlier quoted context omitted.

If you spin up Kubernetes for "a couple of containers to run your web app", I think you're doing something wrong in the first place, also coupled with your comment about adding SDN to Kubernetes. People use Kubernetes for way too small things, and it sounds like you don't have the scale for actually running Kubernetes.

We have a hobby web based app that consists of multiple containers. It runs in docker compose. Serves 1000 users right now (runs 24/7). Single VM. No Kubernetes whatsoever. I agree with you.

Docker compose is brilliant while your stack remains on a single box, and will scale quite nicely for some time this way for most applications with minimum maintenance overhead.

My personal strategy has always been to start off in docker compose, and break out to a k8s configuration later if I have to start scaling beyond single box.

Post reply on HN