The sad state of sysadmin in the age of containers (2015) • Hacker News Board

Re: The sad state of sysadmin in the age of containers (2015)

#321

Post by geggam » Wed, May 16, 2018, 11:34 PM UTC

tapoxi wrote: post #279

spiderjerusalem wrote: post #249

Earlier quoted context omitted.

I'm not saying there are no young people doing sysadmin. What I'm trying to say is that if the new 'infrastructure' that all sysadmins learn is not an open UNIXy system where you can grok all the internals if you want to, but closed systems owned by 2-3 major cloud players, then we kinda maybe have a problem in 20 years? Of course, one can argue that that will just cause a new wave of openness and the cycle continues…

As a "young" (30) sysadmin/devops dude I think that open, Unixy system is Kubernetes. I can take an application, dockerize it, write a helm chart and run it anywhere. The risk is in treating anything as a black box, whether its a managed service or a container you pull from Dockerhub. It's something you'll get burned by eventually and need to learn from experience.

You setup and manage K8s and all the kernel tunings for the host systems so the NAT layers are optimized ?

Re: The sad state of sysadmin in the age of containers (2015)

#322

Post by zzzeek » Wed, May 16, 2018, 11:38 PM UTC

spookthesunset wrote: post #307

zzzeek wrote: post #234

Earlier quoted context omitted.

This comment makes way more sense to me that the original blog post. Yes, nobody should be relying upon docker as their distribution platform. That's pretty terrible. Ubiquity I've observed seems pretty uncomfortable just supporting the major distros, I actually wrote some docker stuff to pull down their .debs, crack them open and install the binaries inside on a fedora/centos system. That's closed source for ya.

> I actually wrote some docker stuff to pull down their .debs, crack them open and install the binaries inside on a fedora/centos system. Why would you want to do that though? Treat the whole thing as a black box running inside docker and be done with it. The second you crack it open, you get to support it. Let Ubiquiti support it, after all that's what you are paying them the big bucks for.

because....they only offered .deb files and I wasnt running debian or ubuntu (nor do I like to bother with it in containers I'm building myself b.c. i have no clue about debian)

the package in question has since finally offered .rpms but i haven't had time / interest in updating it. this is wifi software I'm running personally, ubiquiti only supports the windows/mac versions of it in any case.

Re: The sad state of sysadmin in the age of containers (2015)

#323

Post by nineteen999 » Wed, May 16, 2018, 11:41 PM UTC

Thank heavens people are starting to point this out.

The last devops team I worked on had an obsession with shiny. Never mind that they couldn't bootstrap a new base database for their application anymore or automate the entire application deployment (even with a phased approach) on either VMware or AWS. They wanted to keep piling new tools (often with sub 1.0 version numbers) of new tools on top of an unstable foundation, and would just shrug when it fell apart in production (which it commonly did).

I tried pointing out to them that by giving commit access to their internal puppet git repository to every developer in the building, they had effectively given root access to them as well. All I received were shrugs and blank looks all around.

One thing from the article that doesn't seem to be discussed enough here in the comments is trend of pulling random Docker images from the internet and deploying on your infrastructure simply because it's easier to integrate random versions with feature X than maintain your own builds, or work with the vendor's provided packages to achieve the same result. The security implications of this in particular has been bugging me for years.

Re: The sad state of sysadmin in the age of containers (2015)

#324

Post by mmt » Wed, May 16, 2018, 11:50 PM UTC

cookiecaper wrote: post #235

BurritoAlPastor wrote: post #170

Earlier quoted context omitted.

> But your core team can't tolerate being segmented out by component/implementation detail. And yet tolerate it will, because it is somewhat impossible to hire a team composed entirely of people who are each experienced and competent in writing and designing frontends, writing and architecting backends, deploying and maintaining whatever backing services you've using, build and release engineering, Linux, networking,…

> And yet tolerate it will, because it is somewhat impossible to hire a team composed entirely of people who are each experienced and competent in writing and designing frontends, writing and architecting backends, deploying and maintaining whatever backing services you've using, build and release engineering, Linux, networking, etc. You're right that everyone is not going to start out knowing everything. No matter h…

>The only real question is whether there are enough people capable of this out there. I think there would be if we set it up as a general expectation.

That strikes me as merely wishful thinking. It's not as if there isn't already research on human cognitive abilities in general.

Do you have any scientific basis for thinking engineers are merely being held back by our acceptance of specialization, rather than by inherent cognitive limitations?

Re: The sad state of sysadmin in the age of containers (2015)

#325

Post by mmt » Thu, May 17, 2018, 12:02 AM UTC

cookiecaper wrote: post #257

cookiecaper wrote: post #123

Earlier quoted context omitted.

The thing is that any separation in the roles in ineffective. Things shift around some if you embed an ops guy into the dev team directly, but it doesn't resolve the core problem. This applies to DBAs as well as ops or any other software-side segmentation as well. The core problem is that there are "ops guys" and "dev guys". That creates conflicting incentives, even within the same team. It creates tension and a dyna…

Phew, this has been a good exercise. Let me clarify the thesis. The thesis is NOT that a crew of superhumans can supersede all DBAs, security engineers, and infra people in the world. It is rather that you can be a great software-side engineer, and that you can skew/focus on a few primary concerns, and develop and maintain a working knowledge in the others, sufficient to service your core project's needs. Specialists…

> In software, everything goes down to the same place: the system hardware. And these days at least, this is pretty much homogeneous between software segments. If you know how this functions, the differences are in the modes of expression and the conventions, not really the principles.

Interesting that you mention this, since I think it's become something of a self-fulfilling prophecy, especially with giant cloud IAAS providers making one-size-fits-all choices of hardware to sell.

I certainly agree with you that that the basic principles are certainly the same, but that ignores the performance (and, arguably, reliability) possibilities that open up when not limited by the hardware (including network) choices of others.

Re: The sad state of sysadmin in the age of containers (2015)

#326

Post by mmt » Thu, May 17, 2018, 12:19 AM UTC

commandlinefan wrote: post #203

loteck wrote: post #23

The clearest explanation of why this happens is at the end: Before, admins would try hard to prevent security holes, now they call themselves “devops” and happily introduce them to the network themselves! 1) The merging of devs into the sysadmin role was a product of: the work of sysadmins (particularly systems change control and security compliance) not being valued in our culture. 2) Devs delighted to be free of th…

> shackles placed upon them by sysadmins Well, potentially unpopular opinion here, but an awful lot of sysadmins brought their looming obsolesence on themselves. I'm an app (as in "a program that runs on a computer", not an iOS add-on) developer, always have been. I get requirements from the business types, code it up in vi or Eclipse or whatever, get it working, and then they (the business) want to deploy the workin…

Actually, as a sysadmin, I sympathize with you, since I consider that kind of situation to be a sign of, essentially, bad system administration. It also sounds like it might be at a larger company.

Personally, I've always considered it a significant part of my job to make developers' jobs easier, especially with something like deployments and dependencies.

As such, I disagree that we've brought our own "obsolesence" on ourselves, but I do agree that those of use who have perhaps forgotten that ours is a service profession have hastened its demise.

Re: The sad state of sysadmin in the age of containers (2015)

#327

Post by mmt » Thu, May 17, 2018, 12:32 AM UTC

nineteen999 wrote: post #323

Thank heavens people are starting to point this out. The last devops team I worked on had an obsession with shiny. Never mind that they couldn't bootstrap a new base database for their application anymore or automate the entire application deployment (even with a phased approach) on either VMware or AWS. They wanted to keep piling new tools (often with sub 1.0 version numbers) of new tools on top of an unstable found…

That helps to explain why there are so many DevOps job postings with nearly identical lists of tools, especially for startups here in the SFBA.

Re: The sad state of sysadmin in the age of containers (2015)

#328

Post by arca_vorago » Thu, May 17, 2018, 12:38 AM UTC

I was all ready to rant and then read the first part: "I’m not complaining about old-school sysadmins. They know how to keep systems running, manage update and upgrade paths.

This rant is about containers, prebuilt VMs, and the incredible mess they cause because their concept lacks notions of “trust” and “upgrades”."

Fair enough, rant preemptively avoided.

Re: The sad state of sysadmin in the age of containers (2015)

#329

Post by dredmorbius » Thu, May 17, 2018, 12:44 AM UTC

zapita wrote: post #301

dredmorbius wrote: post #291

Earlier quoted context omitted.

Risk arbitrage. Present capability vs. future catastrophic risk. Technology is debt.

I honestly can't tell whether you're making a sincere but unintelligible argument, or just trolling. Do you disagree with what I said?

Yes, re: last.

Tech-as-debt is a notion I'm playing with.

https://mastodon.social/@natecull/99318348047974414

https://plus.google.com/104092656004159577193/posts/jS1K9Mto...

Obvious antecedent: technical debt, Ward Cunningham.

Re: The sad state of sysadmin in the age of containers (2015)

#330

Post by hodgesrm » Thu, May 17, 2018, 12:55 AM UTC

scurvy wrote: post #188

hodgesrm wrote: post #11

I see this as more of an opportunity than a problem. The fact that Hadoop, Kubernetes, and other platform-like systems are complex to manage properly with good attention to security implies they should be delivered as cloud services rather than having everyone run their own. This enables K8s users to focus on apps while offloading management to specialists who can focus on running the services well. If you operating…

One of the gating factors here is the both the speed of Kubernetes development (move fast and break all the things), and the terrible state of accompanying documentation. If "they should be delivered as cloud services" is some sort of k8s apologist stance for its sorry state of maturity, then we got issues. OTOH, if it's "You shouldn't run it in house unless you have an army of people to read every new commit", that'…

First of all let me be clear I'm not an expert in K8s and certainly not an apologist for bad software.

On the other hand there's a level of complexity in distributed systems that is impossible to avoid even in stable infrastructure. You have a design choice of trying to make the system as easy as possible to operate (at the cost of other features) vs. finding operating models that make it less of an issue.

Personally I would rather spend time futzing around with my applications that run on kubernetes vs. trying to run kubernetes itself. It would be sufficient if Kubernetes services were portable across a marketplace of providers so I could pick a place to run my applications.