The sad state of sysadmin in the age of containers (2015) • Hacker News Board

Re: The sad state of sysadmin in the age of containers (2015)

#241

Post by colanderman » Wed, May 16, 2018, 7:03 PM UTC

kazinator wrote: post #120

jiveturkey wrote: post #87

Earlier quoted context omitted.

careful, your dinosaur scales are showing. make is so bad you need automake to manage it. there are much better tools. sadly, nothing LCD (least common denominator) so as to gain wide traction. That said, for anyone distributing software, shame on them for not packaging their custom build so as to be runnable via ‘make all’ (just using make to drive everything else).

Make is only bad under the Autotools mess. Every build-related struggle in an open-source project that uses Autotools can be traced to Autotools, not to make. Autotools wasn't invented to overcome deficiencies in make, but deficiencies in C portability across Unix flavors. Those deficiencies are greatly diminished today, both by POSIX standardization, and there being fewer viable surviving Unix variants that anyone c…

Exactly. Don't forget `pkg-config` [1], which I've found eliminates most cross-POSIXish-platform linking issues.

[1] https://en.wikipedia.org/wiki/Pkg-config

Re: The sad state of sysadmin in the age of containers (2015)

#242

Post by dsr_ » Wed, May 16, 2018, 7:04 PM UTC

spiderjerusalem wrote: post #149

ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

What fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?

That's part of the reason why my last two hires have been at the beginning of their career. For both of them, it was their first major sysadmin responsibility after having jobs involving tech support and occasional Linux experience.

The key is to pick smart people who are good at learning and find complex systems interesting. Then, of course, you need to have interesting projects for them to work on.

Re: The sad state of sysadmin in the age of containers (2015)

#243

Post by gm-conspiracy » Wed, May 16, 2018, 7:04 PM UTC

seorphates wrote: post #205

ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

Grumpykins here. I think the term "Modern devops" sort of nails it but not quite how you used it. Most departmental/enterprise sys admins/engineers of lore that had even the slightest necessity for life outside the box scaled anything resembling automation to its breaking points. Combined with knowing and serving the reasons for their existence - developers, users etc., and devops is nothing new - it is now simply th…

A good sysadmin would not look like they are doing much work (everything is humming along and can self-heal minus physical problems), but a good devops person is constantly busy.

Re: The sad state of sysadmin in the age of containers (2015)

#244

Post by colanderman » Wed, May 16, 2018, 7:05 PM UTC

avar wrote: post #239

colanderman wrote: post #232

Earlier quoted context omitted.

> The GP does this via a neat hack, but you can also do this in a much more understandable fashion by simply having the body of every Makefile rule start out by shelling out to some script in your favorite language. I'm not sure what you mean? How would this allow you to use, say, Python as the language for recipes? Just having Make drop straight into Python kind of defeats the purpose of Make.

You'd use Python as the language for the recipe that turns (in this example) a given .c file into a .o file, while leaving the Makefile to do what it's good at, declaring the DAG dependency tree needed to incrementally build it. The point is that people conflate these two things. They open some random Makefile and see that it's mostly doing stuff with shellscripts, and think "oh I should do this all in Python", and t…

Nevermind, I misread you. I missed "rule" where you said "beginning of every Makefile rule". (I thought you were suggesting just having the default rule run some enormous Python script, which I've unfortunately seen before.)

Re: The sad state of sysadmin in the age of containers (2015)

#245

Post by robohoe » Wed, May 16, 2018, 7:08 PM UTC

kazen44 wrote: post #183

user5994461 wrote: post #176

Earlier quoted context omitted.

CFEngine is basic text manipulation, it's not comparable to the rest. Puppet and Chef was the first generation. I wouldn't recommend. All the companies and people I know using Chef migrated away from it after many disasters. Nowadays, it's only mentioned in interviews to find out if candidates have real world fire fighting experiences. Ansible is good. Used that for managing hundreds of machines at multiple jobs (som…

> Not sure about SaltStack. Never had the opportunity to try. I'd be a bit worried though on the long term prospect because I don't think they have much backing or user base. saltstack is a well thoughout solution in my opinion. It makes more logical sense and is less of a mumbled mess then either chef or puppet and has miles better performance then ansible. I know quite a few shops who use it. Its definitly smaller…

SaltStack is around. Lots of big orgs take the time to understand. Ansible is more popular because you can use it with just one playbook. Saltstack requires you to think about your environment and design your configuration management properly.

Re: The sad state of sysadmin in the age of containers (2015)

#246

Post by phaedrus » Wed, May 16, 2018, 7:08 PM UTC

agentultra wrote: post #78

loteck wrote: post #23

The clearest explanation of why this happens is at the end: Before, admins would try hard to prevent security holes, now they call themselves “devops” and happily introduce them to the network themselves! 1) The merging of devs into the sysadmin role was a product of: the work of sysadmins (particularly systems change control and security compliance) not being valued in our culture. 2) Devs delighted to be free of th…

I recall the idea of "devops" from this book: https://landing.google.com/sre/book.html The stated goal of putting both systems administrators and software engineers on the same team is to reduce friction and increase communication. One of the worst, productivity-killing situations you can find yourself in when developing network software and services is caused by the traditional "old school" mentality of separating t…

I once worked at a company which separated IT into 3 teams: developers, DB-sysadmin (ops), and QA (who also managed deployments). Releases were supposed to go in a waterfall model from the Dev group -> QA group -> Ops. QA wanted Dev to submit Word documents for each release with blanks to be filled in with server names. However Ops was so distrustful of Dev that it was not enough for them to lock us out of Prod using regular security tools, we were also not allowed to know the NAMES of servers in Prod or how currently deployed systems were grouped.

Every release was an Abbott-and-Costello "Who's on first?" routine. Do you have any idea how hard it is (especially in computing) to ask for something without being able to utter its name?

QA: "You left servername blank on this deployment document."

Dev: "I know; Ops won't tell me. Just ask them for where the service is currently."

QA: "Ops says there's 5 unrelated legacy services with that same project name, on different servers."

Dev: "5? I only knew about 3. You know, if I could query the schemas of the Prod DB, I could tell you in a jiffy which one it is."

Ops: "Pound sand. If you want look at databases that's what the dev DB server is for."

Dev: "Erm, OK well can I give you a listing of the Dev DB schema and you tell me if it looks like the one the Prod service is talking to?"

Ops: "Oh I see you want us to do your job for you now? You can compare the schemas."

Dev: "OK..."

Ops: "Just tell us which DB server you want the schema pulled for."

Dev: "But you won't tell me the server names."

Ops: "No."

My point is this is how bad communication can be when ops and dev are not on the same team.

Re: The sad state of sysadmin in the age of containers (2015)

#247

Post by Max_aaa » Wed, May 16, 2018, 7:08 PM UTC

kazen44 wrote: post #202

Animats wrote: post #179

If you thought having to deal with old COBOL programs was a problem, it gets much worse. When there are 10 year old containers in production, and the parts to rebuild them are long gone, then you have a real problem. There's a reason that Google has internal systems which can and do rebuild everything from source.

This is actually a very important point that gets ignored a lot. What will happen if public container images are no longer available for building your application? Your in a for a world of hurt if a major dependency of your application doesn't work anymore because someone pulled the image.

This is why we have a local registry and for any container that is goes to testing or production or is needed for builds should be build-able from source.

We still need to figure out how to best tackle the issue of online repositories being taken down or vanishing. For now we run apt-mirror.

Re: The sad state of sysadmin in the age of containers (2015)

#248

Post by ams6110 » Wed, May 16, 2018, 7:10 PM UTC

Twirrim wrote: post #189

spiderjerusalem wrote: post #149

Earlier quoted context omitted.

What fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?

There's rarely anything old school sysadmins have learned that hasn't come from experience. Been there, done that, fought that shit the first time. And the second. And the third. (it's amazing how often I find myself solving what are essentially the same problems over and over.) It's one reason why you'll find we'll push back on the "ohh shiny". There are many wonderful and fascinating things coming out. Tech is an a…

Ansible is one "ohh shiny" thing that has greatly increased my productivity as a sysadmin. Before that I would automate what I could with ssh and pdsh and scripts, but it was never as well polished as Ansible.

I'm even using ansible for ad-hoc stuff (tweaking a config, restarting a service) because it's easier to do that from a management server than log in to some remote host, get oriented as to the OS distribution and version, and run commends in the shell there.

Re: The sad state of sysadmin in the age of containers (2015)

#249

Post by spiderjerusalem » Wed, May 16, 2018, 7:12 PM UTC

karlkatzke wrote: post #222

spiderjerusalem wrote: post #149

Earlier quoted context omitted.

What fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?

What makes you think there aren’t young people doing systems administration? Our last two hires in my most recent job were 23 and 27 respectively. Sure, they’re getting trained in the new hot cloud stuff... as the grumpy seniors figure it out first and set patterns... but they are still doing daily work with some rather ancient stuff.

I'm not saying there are no young people doing sysadmin. What I'm trying to say is that if the new 'infrastructure' that all sysadmins learn is not an open UNIXy system where you can grok all the internals if you want to, but closed systems owned by 2-3 major cloud players, then we kinda maybe have a problem in 20 years?

Of course, one can argue that that will just cause a new wave of openness and the cycle continues.

Re: The sad state of sysadmin in the age of containers (2015)

#250

Post by kokey » Wed, May 16, 2018, 7:14 PM UTC

I think part of the problem is that Docker is the now popular answer to the problem of 'it works on my machine'. Unfortunately 'works on my machine' also involved going to the web site of a particular tool or library and following the steps recommended for quickly trying the tool out, which gives things like curl http://somewebsite.com | sudo bash situation or following the steps in a blog post where someone quickly compiled a bleeding edge version of it, with all the dependencies to build it, on their Ubuntu laptop.