Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Highly available Kubernetes with batteries for small business

cinaq.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: Highly available Kubernetes with batteries for small business

#61

Post by kavalg »

justinclift wrote: post #60
alephu5 wrote: post #59

Earlier quoted context omitted.

Yes exactly, I have an API and a couple of static sites exposed via nginx. None of the other services have exposed ports and all the networking is done through docker.

Just to double check (as a safety measure), are you using an external firewall for the docker container to ensure nothing can access them, or are you using customised firewall rules on the docker hosts? Asking because from what I've seen (so far) it's only safe to use an external firewall - eg Digital Ocean (or whoever) firewall applied to hosts - rather than iptables on the hosts themselves. Saying that because when…

Had the same issue with swarm and iptables/ufw. The solution I found was to replace nginx with traefik and never expose service ports out of the swarm. All HTTP/TCP/UDP termination is done by Traefik (itself running in Swarm too). So far the experience has been good. However, you need to bind the traefik container to the host network. Otherwise you will get not be able to get the remote host IP properly (x-forwarded-for). I am mostly doing scale up (Hetzner can go up to 256GB RAM and dedicated CPUs). This saves me the hassle of managing distributed storage (I just use local docker volumes and schedule a snapshot of the whole drive as a local backup at Hetzner). Should you need to scale traefik horizontally (e.g. > ~30K req/s), then you would need something like e.g. Consul to manage the distributed configuration. Dealing with docker volumes on a distributed storage may be harder though. Would be glad if someone can share their experience doing that.

Re: Highly available Kubernetes with batteries for small business

#63

Post by battery423 »

ThinkBeat wrote: post #41

The small business buys AS/400. IBM configures it. Drop it in a closet, forget about it. If something goes wrong it will call for help, before you even know something was wrong. It depends on what you define a small business as though. Back in the days these were extremely reliable. I have seen them in closets, covered with everything else you would find in a closet. Used by everyone day in and day out, nobody knew w…

The last ~10 years, i have had rarely any real hardware issues.

My current server is hardware from 2012.

Someone said that just a few weeks ago 'Cars you buy today are all good enough. You will not buy a car today and it will rust tomorrow.'

The only reason why i migrated my desktop pc to become my server, was the new DSLR which made Lightroom unusable due to the raw image size.

I might even think, that Computer Hardware in general is very very reliable. Besides the capacitor plague issue https://en.wikipedia.org/wiki/Capacitor_plague that gave systems a bad lifetime.

And i have plenty of systems here which are running for years and have seen plenty of ubuntu upgrades as well.

Re: Highly available Kubernetes with batteries for small business

#64

Post by llama052 »

cyberpunk wrote: post #27
solotronics wrote: post #11

Earlier quoted context omitted.

For someone who is interested and capable the costs are an order of magnitude lower for self hosting. Your right for the most part small business would be better off using an existing cloud, they dont have the skillset to properly design and manage something like this themselves.

Does that cost estimation include the 4-6 devops freelancers you're going to need for like 6 months to get it going? The bigger cost is in staffing than cloud bills unless you're running really a lot of infra..

If you need 4-6 devops people to run infra from the ground up you're doing it wrong. I've seen 4 man devops/ops teams run thousands of machines at scale.

Unless you're talking about a huge magnitude of infra (which you'd have a budget for) then I think your 4-6 number is a little off.

Re: Highly available Kubernetes with batteries for small business

#65

Post by llama052 »

ahnick wrote: post #53
Greybeard92 wrote: post #49

Earlier quoted context omitted.

1. IaC / no servers isn't a plus. It's vastly easier to SSH into a server and do anything you already can on a desktop 2. So do CoLocos, but better "HA" with no effort at all 3. /var/log + bash grep 4. Reverse proxy, just add another IP address and a load balancer 5. This is true anywhere 6. Nginx and apache both do this, dead simply 7. SSH + cli editor of your choice Kubernetes does nothing new, but pad more resumes…

1. You can't replace everything that Terraform does with "SSH into a server". For example, in Terraform you are specifying exactly what the compute resources are going to be used on the node and even SSH itself might be part of the software that gets provisioned. 2. "do CoLocos" definitely has effort and does not in anyway guarantee HA. (HA is really more of a software application layer concern not hardware) Also, ar…

Piggybacking on your comment, not agreeing with the previous poster.

Currently using all of the tooling you've mentioned on-premise and it's great! Using DNS Anycast (cloudflare/akamai/insert provider) and LB technology we can get highly available without much work. We've got more 9's than our Azure production setups, sadly though I feel like Azure isn't a good standard to compare to.

Bridging the gap from the cloud provider managed services to on-premise (colo) is actually relatively simple. By all means 95% of the tooling these providers use is Open source or has a major open source alternative.

SMB world does change that to your point as I doubt you'll have a devops rockstar who can actually grok these things, more than likely you'll have a dinosaur MSP or IT guy. (for really small companies)

Re: Highly available Kubernetes with batteries for small business

#66

Post by ClumsyPilot »

NikolaeVarius wrote: post #2

> This setup is not trully highly available. The whole cluster depends on the Synology as data storage. You could improve this further by replacing the centralized NAS with a distributed solution. But besides that the cluster is very solid and scalable I understand that my local harddrive with excel spreadsheets detailing my entire business along side critical software is not the best place to store the lifeblood of…

As long as you have enabled backups, this is a fairly reasonable solution. And on a 'Product' NAS like Synology backup is fist-class, 1-click solution.

Re: Highly available Kubernetes with batteries for small business

#67

Post by solotronics »

manquer wrote: post #19
solotronics wrote: post #11

Earlier quoted context omitted.

For someone who is interested and capable the costs are an order of magnitude lower for self hosting. Your right for the most part small business would be better off using an existing cloud, they dont have the skillset to properly design and manage something like this themselves.

Even with the skill sets, i am not sure the costs are lower in most cases, depending on where you are your uplink costs can be substantial, you might have a asymmetric link or your ToS with the ISP may forbid any hosting services or just poor internet speeds. Without a leased line ISPs are not going to give a uptime SLA or a ring topology for redundancy etc, Reliability is expensive, there are enough cheap k8s/VPS pr…

You are right about rate limited internet. Something else to consider is renting some rack space is super cheap and can come with free internet and electricity. Last time I checked a few years ago in DFW it was $60 a month with 1Gb/s and some IP addresses for 1RU.

Many small and medium businesses have a "business" ISP circuit. This typically doesn't have caps and has the option of a dedicated IP. Speaking from experience I was an ISP network engineer for a big telco years ago.

Re: Highly available Kubernetes with batteries for small business

#68

Post by p_l »

legostormtroopr wrote: post #56
p_l wrote: post #55

Earlier quoted context omitted.

It's not about scaling the applications you run on it. It's about providing leverage to your team so you need less effort to do more. And SMB can be anywhere up to 1000 employees if using semi-legal definitions, 500 employees and under 1 billion euro of turnover in case of EU's SME definition. Depending on the area of work, there can be a lot of software to run. And that's where k8s comes in. It might seem "hard" whe…

> SMB can be anywhere up to 1000 employees, 500 employees and under 1 billion euro of turnover in case of EU's SME definition By no-ones metric is 1000 employees a small business. The EU defines a small business as under 50 staff, and a medium business as under 250 staff. https://ec.europa.eu/growth/smes/business-friendly-environme...

SMB is not "small business". It's "Small and MEDIUM Business". And even larger business is often supported by smaller subcontractors.

In this area, k8s had been personal saviour where we could not take on a project otherwise. Just thinking of dealing with ~70 applications that had subtly different requirements leaves me shuddering. And that was pretty small deal.

Re: Highly available Kubernetes with batteries for small business

#69

Post by justinclift »

kavalg wrote: post #61
justinclift wrote: post #60

Earlier quoted context omitted.

Just to double check (as a safety measure), are you using an external firewall for the docker container to ensure nothing can access them, or are you using customised firewall rules on the docker hosts? Asking because from what I've seen (so far) it's only safe to use an external firewall - eg Digital Ocean (or whoever) firewall applied to hosts - rather than iptables on the hosts themselves. Saying that because when…

Had the same issue with swarm and iptables/ufw. The solution I found was to replace nginx with traefik and never expose service ports out of the swarm. All HTTP/TCP/UDP termination is done by Traefik (itself running in Swarm too). So far the experience has been good. However, you need to bind the traefik container to the host network. Otherwise you will get not be able to get the remote host IP properly (x-forwarded-…

Interesting. Sounds like a reasonable path go with when initially growing.

eg go bigger (hosts) first, then figure out horizontal scaling later if/when needed :)

Post reply on HN