Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rolling your own servers with Kubernetes

gravitational.com

Post reply on HN
Print view View original

161–170 of 209 posts

Re: Rolling your own servers with Kubernetes

#161

Post by sroussey »

jasode wrote: post #149
pathseeker wrote: post #148

Earlier quoted context omitted.

That's still in datacenters that aren't Amazon's, Google's, etc. It's the most critical component of netflix (the actual video delivery) and it's not "in the cloud".

>That's still in datacenters The point isn't that they are still _in_ datacenters. Yes, of course, they are. Even the "cloud" ultimately resolves down to somebody's datacenter somewhere. The point is that Reed Hastings & Netflix wanted to get out of managing their own datacenters. Putting their Netflix appliances inside of ISP owned datacenters still lets them avoid managing their own datacenters. Their critical user…

Netflix employees still walk inside those datacenters.

I’ve avoided my own datacenter by using Equinix, for example. I wouldn’t call that “cloud” though.

Re: Rolling your own servers with Kubernetes

#162

Post by sroussey »

hn_throwaway_99 wrote: post #107
gambler wrote: post #91

Earlier quoted context omitted.

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

I worked at a couple of those self-hosted companies in the 200Xs. For me, as a developer, it was painful and way, way, way worse than AWS or GCP. Planning for big expected spikes of traffic was extremely painful, and once the cloud vendors started adding new services we felt continually behind the times. I can only think it was "interesting" if you've forgotten how difficult and expensive managing your own infrastruc…

Spiky usage patterns are great for cloud.

Otherwise, I’ve did some tests years ago and what I spent over 5 years would have gotten me only a few months on AWS. And I think I only walked into the data center once in those five years.

Re: Rolling your own servers with Kubernetes

#163

Post by yowlingcat »

mfatica wrote: post #15
xrisk wrote: post #13

The article seems to focus on K8s with reference to micro-services. How well does K8s do if you're running a monolith?

Using kubernetes without microservices is like using hadoop on a 1000 row csv file

This is a poor analogy. To quote a sibling comment:

"If you consider what it takes to manage the end-to-end lifecycle of a single application, monolith or micro-service, you need a solution for the following items: deployments, application configuration, high availability, log and metrics aggregation, autoscaling, and load balancing across multiple application instances.

Kubernetes provides an opinionated way of doing all of those things. For example, Kubernetes leverages container images and declarative configs for packaging and deploying applications. For many people this approach is much simpler than what Puppet, Chef, and Ansible bring to the table in terms managing applications."

Using Kubernetes even with a monolith can be pretty nice, if (big if here) Kubernetes behaves.

Re: Rolling your own servers with Kubernetes

#164

Post by yowlingcat »

q3k wrote: post #12
NickBusey wrote: post #6

I am curious to see how they recommend handling storage on bare metal k8s.

I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself. [1] - https://rook.io/

Really curious about Rook. The stakes of running something like that on K8S successfully seem quite a bit higher than stateless apps, and commensurately the rewards as well.

Re: Rolling your own servers with Kubernetes

#165

Post by marcinzm »

gambler wrote: post #91
jasode wrote: post #80

Earlier quoted context omitted.

>The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to afford their own infrastructure. The decision framework the executives use isn't just the "hardware+software" -- it's the whole "IT organization" . In other words, it's not "in-house cpu" vs "Amazon's cpu". It's in-house I…

>Most companies are not like Facebook or Dropbox that can maintain an internal IT organization at a high level equivalent to AWS. Let's try this with different phrasing. In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time. Forward 10-13 years. We have cheaper hardware. We have extra 10+ years of development in open-source software. And yet that list…

>In 200X a lot of companies were able to maintain their own infrastructure, just like Facebook and Amazon did at the time.

I worked at a decently large tech-ish company at that time. It took 3 months to provision a handful of dev servers in the data center. After filling out forms in triplicate. We asked for extras since we had almost zero ability to manage them (reboot, etc.) or get them re-built if something broke. No backups of any kind as far as I knew. We once had one break at night and apparently someone had to psychically drive there at 3am to reboot the thing after we filed a ticket. Used some sort of in-house distribution of linux that had it's own quirks.

I'll take the cloud any day of the week.

Re: Rolling your own servers with Kubernetes

#166

Post by momofarm »

I once heard a story that the reason some people would build their own server is they don't trust those service provider on the internet. They feared that the data store there are quite easily breached.

Thanks to these guys, they just lower the unemployment rate a little more!

Re: Rolling your own servers with Kubernetes

#167

Post by devonkim »

ecnahc515 wrote: post #133
devonkim wrote: post #126

Earlier quoted context omitted.

What used to be behind firewalls oftentimes became accidentally public through the many, many blunders of publicly available S3 buckets that people started to use because they were frustrated with AWS IAM. This is ironic because IAM was supposed to help you secure your data and such, not make things worse for your security profile. Even with all the tooling that exists to help people write better IAM policies and fig…

It's extremely easy to prevent publically available S3 buckets, and there's even built in policies for it. It's actually fairly difficult to make a bucket public without going through a few hoops even without that. The only reason this is happening is sure laziness and lack of any true change control processes. It's not because it's hard.

Having witnessed what people do, it’s easy to prevent it now for the most part but badly written / enforced access controls and laziness (in the form of overcommit of engineers to projects) are the norm for most large companies. Most of the compromised buckets were launched years ago before a lot of safeguards were put in, and object level permissions can override bucket policies anyway. Getting sharing of objects across a Byzantine bureaucracy in internal IT is a great way to increase the chance some engineer desperate to get their work done will mark something public and forget about it.

S3 based URLs to get cheap web hosting for low traffic sites is exactly what leads to bad permissions as well. I’ve seen plenty of S3 objects that are made public so that they can be viewed from a web browser and are just a badly targeted script run away from being on the latest tech blog about how some other institution leaked PII.

Re: Rolling your own servers with Kubernetes

#168

Post by nijave »

gambler wrote: post #53

>Should you roll your own servers? If you are not certain, the answer is most likely “no”. The staggering growth of AWS happened for a reason. Funny how for many decades companies and people were running their own servers. The hardware was getting cheaper each year. More software became available via open source. Then several ubecorporations entered the hosting/cloud business, and suddenly no one seems to be able to…

Yeah and how long did it take to update software back then? Weeks, months, years? Now changes can reliably get pushed to production in hours or even minutes which has incredible business value

Re: Rolling your own servers with Kubernetes

#169

Post by tech_tuna »

isatty wrote: post #43
falcolas wrote: post #32

Earlier quoted context omitted.

> spend weeks arguing with AWS support Yup. We've run into that repeatedly. The "we didn't notice anything on our side, please send more screenshots and logs" gets really old when working with "managed services". Network packet losses/truncations, EKS control plane failures, cloudformation stacks getting stuck in really wierd states, inconsistent cloudformation implementations for new and existing services, ENI weird…

Cloudformation is a tragedy in itself.

I am a Terraform convert, I agree.

Re: Rolling your own servers with Kubernetes

#170

Post by the-rc »

vallard wrote: post #153
q3k wrote: post #12

Earlier quoted context omitted.

I highly recommend Rook [1], which is based on Ceph, provides PersistentVolumes to k8s workloads and is also running on k8s itself. [1] - https://rook.io/

Big fan of rook myself. We took several nodes loaded with 16 disk drives and used Rook to make a Ceph and Minio cluster out of it. Great storage solution.

Have you done any disaster recovery exercises? We've started to play with it, but haven't gone that far yet.
Post reply on HN