Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sad state of sysadmin in the age of containers (2015)

vitavonni.de

Post reply on HN
Print view View original

381–390 of 435 posts

Re: The sad state of sysadmin in the age of containers (2015)

#381

Post by auslander »

ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

> Modern devops ... million tools ... running joke

So fucking true, bro

Re: The sad state of sysadmin in the age of containers (2015)

#382

Post by twic »

virgilp wrote: post #259
user5994461 wrote: post #221

Earlier quoted context omitted.

Yes. The host will run 100% CPU to handle the hundreds of SSH connections. I've been re configuring 300 to 800 hosts many times a day, never had a problem. I think it would take a few thousands hosts for the performance to be noticeably slow and I am really not sure that other tools or systems could take it much better.

I know our SREs once screwed the config for sshd, and considered themselves very lucky that they had puppet on the machines and could push a fixed configuration (if they had used exclusively ansible, that'd be the end of it - no way to connect or to deploy new configuration) [edit] To clarify - ansible is great, and we use it. Just saying that, as everything, it still has (sometimes subtle) downsides in various scena…

When i worked with physical machines, they had embedded management systems, which were on a physically separate network to the machines' main interfaces, ran a little embedded SSH server, and would (amongst other things) give you a console on the machine.

Simpler machines should still have serial consoles, and you can get those on the network via a terminal concentrator or a serial-to-ethernet adaptor.

I would love it if Ansible could control machines over an interface like that, rather than via SSH. Then you wouldn't even need to run SSH on machines which don't need it, which is most of them.

Re: The sad state of sysadmin in the age of containers (2015)

#383

Post by switch007 »

dozzie wrote: post #302
noir_lord wrote: post #260

Earlier quoted context omitted.

I like ansible with vagrant as well - it makes for a nice clean way of deploying to development environments while also been nicely 'self' documenting and not limiting (you can drop back to shell), it's a lovely tool for the most part. Edit: The thing I really like about Ansible is how unsexy it is, it's just a nice sane way of doing largely what you could do yourself with ssh and bash but in a language that doesn't…

It would be even nicer if people weren't advocating it as a configuration management system, only as what it is: a deployment system.

It calls itself one, I imagine many people also consider it one. What is a true configuration management system?

Re: The sad state of sysadmin in the age of containers (2015)

#384

Post by shakna »

Intermernet wrote: post #359
segmondy wrote: post #265

Earlier quoted context omitted.

Great sysadmins get fired unless they learn to pretend to look busy. They automate everything till they have nothing to do.

Logging is your friend here. You can spend days scrolling through logs, doing an occasional grep and making disapproving noises occasionally. Bonus points for developing some graphs for the next meeting.

My introduction to statistics was learned as a sysadmin, trying to show that I was doing 'stuff'.

Re: The sad state of sysadmin in the age of containers (2015)

#385

Post by mmt »

spookthesunset wrote: post #314
w4tson wrote: post #299

Earlier quoted context omitted.

I’ve wasted enough time tinkering with gulp, grunt and webpack to sympathize

> I’ve wasted enough time tinkering with gulp, grunt and webpack to sympathize And yet, those tools fill a need that would be very hard to replicate with the toolchains that came before it. Good luck doing half of what gulp / webpack do from, say, a Makefile.

I'm not familiar with either of those particular two tools, and what you say may well be absolutely true for both of them.

It's just that this argument keeps getting used for every single new "reinvented wheel" (to borrow from the GP). Sometimes the argument is as strong as "it couldn't be done any other way" and sometimes it's as weak as "this one is just incrementally better," but it feels a little like crying wolf.

Was it really "very hard" to make the old wheel do what you needed, or perhaps somehow extend it or add a library, or was it just far more fun and exciting to build something from scratch?

I generally don't mind a proliferation of tools, except when they start to break or conflict with each other, which is, I believe the GP's main concern, and at least tangentially related to the article.

Re: The sad state of sysadmin in the age of containers (2015)

#386

Post by Jedd »

switch007 wrote: post #383
dozzie wrote: post #302

Earlier quoted context omitted.

It would be even nicer if people weren't advocating it as a configuration management system, only as what it is: a deployment system.

It calls itself one, I imagine many people also consider it one. What is a true configuration management system?

I used puppet -- unhappily -- for a while before discovering SaltStack.

Salt, like Ansible, is a response to the Puppet/Chef hegemony -- ruby and DSL's and tacked-together bits that are a nightmare to install and upgrade in themselves).

I'd suggest Salt did some things much cleaner than Ansible, and while it can be an orchestrator, and deployment system, it also excels as a configuration management system ... but I think typically people are talking about configuration management systems that are tightly coupled with more formal change management systems (of which I've found pretty much none that work well).

Re: The sad state of sysadmin in the age of containers (2015)

#387

Post by auslander »

nunez wrote: post #355

I think a lot of stuff that I used to need to know —- OS kernel internals, hardware specifics, relatively deep network knowledge —— is less useful in the age of public cloud, containers and immutable infrastructure. There isn’t really a need to tune a kernel for performance or do deep troubleshooting to root cause OS issues and ensure uptime; if something seems off, kill the machine or container and let auto scaling…

> There isn’t really a need to tune a kernel for performance

Maybe, but knowing SElinux, kernel namespaces and syscalls for Docker seccomp profiles are crucial for security, and security is the Next Big Thing.

Sad you are getting into management. Lets push to undo that notion that manager is worth more salary than lead engineer. I'm pushing :D

Re: The sad state of sysadmin in the age of containers (2015)

#388

Post by auslander »

nwmcsween wrote: post #106

This has been my complaint from day one. Instead of docker have something like ports or pkgsrc and simply create tools that simplify sandboxing, like cgexec, a google kafel -> ebpf filter then all the package manager has to do is well package. Docker IMO is a mudball of concerns that need to be separated.

SElinux, kernel namespaces and syscalls for Docker seccomp profiles.

Docker is not production ready yet.

Re: The sad state of sysadmin in the age of containers (2015)

#389

Post by auslander »

gm-conspiracy wrote: post #133
nwmcsween wrote: post #106

This has been my complaint from day one. Instead of docker have something like ports or pkgsrc and simply create tools that simplify sandboxing, like cgexec, a google kafel -> ebpf filter then all the package manager has to do is well package. Docker IMO is a mudball of concerns that need to be separated.

BSD jails?

OpenBSD pledge

Re: The sad state of sysadmin in the age of containers (2015)

#390

Post by mmt »

nunez wrote: post #365
JohnStudio wrote: post #195

I think the entire statosphere of DevOps is just about dead on the whole in 2018 .. in retro, after working with things like Docker .. and more specific industry variations beyond the Amazon tech, it makes no sense to dwell on the security / control of a dedicated systems admin professional since the tools are all outside the local domain anyway. The rest from VoIP to IoT to container services are managed whole-sale…

Nah, you’re right The entire purpose of DevOps IMO was to close the gap between sysadmins and devs through code. Devs doing everything, including infrastructure, was and is the entire plan! Public cloud made this super duper easy. The problem is devs don’t want to manage core infrastructure (VPCs, networking, modules for deploying lambdas and database clusters and container orchestration clusters, etc) and somebody h…

> Ideally, those would just be features like any other software team, as it’s all API calls at the end of the day. But lots of companies have issues with structuring their platform teams like software teams because its “not software” even though it is

Just because it's is implemented as API calls at the end of the day doesn't necessarily make it not "not software" (if you'll pardon the double negative), at least in the sense that I believe you mean.

To whit, I believe you're suggesting that if something can be expressed as code, it's all "software" and can therefore be designed, written, and maintained by the same kinds of experts, software developers.

I disagree, because the nature of the infrastructure-as-code code is too different from the application software code.

One could, similarly, express an FPGA configuration in code, but a software developer would not automatically be good at programming one. This is even likely to be true for less extreme examples, such as programming expertise not automatically transferring from general software (for lack of a betterm term) to code that works well on, say, GPUs.

In the case of IAC "software", a more mature design is more likely to resemble traditional sysadmin/network/security best practices than application software features. It could also have significant financial side effects if there's an error, assuming public cloud, which could require more stringent standards of control, review, and quality, especially if a company ends up in SOX territory.

>Then there’s the whole protectionist “You’re taking my job” and “devs can’t possibly know this much about $infra” that isn’t dying off anytime soon

I'm sure some of this exists, but my own experience is an attitude not that devs can't know a certain amount about infrastructure but that they simply don't, often because they actually don't want to.

Perhaps they fear that if they do end up knowing that much, they'll end up being the ones to manage that core infrastructure, which you indentified that they don't want to do!

Post reply on HN