Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sad state of sysadmin in the age of containers (2015)

vitavonni.de

Post reply on HN
Print view View original

331–340 of 435 posts

Re: The sad state of sysadmin in the age of containers (2015)

#331

Post by random_throw »

user5994461 wrote: post #176
brightball wrote: post #160

Earlier quoted context omitted.

Would be interested to get your opinion on Puppet/Ansible/Chef/CFEngine/SaltStack

CFEngine is basic text manipulation, it's not comparable to the rest. Puppet and Chef was the first generation. I wouldn't recommend. All the companies and people I know using Chef migrated away from it after many disasters. Nowadays, it's only mentioned in interviews to find out if candidates have real world fire fighting experiences. Ansible is good. Used that for managing hundreds of machines at multiple jobs (som…

I'd venture to say you're wrong about Salt. It's being used at some large enterprises. I use it (in one of the large tech companies) on thousands of servers, with plans to up that an order of magnitude or more. Of all of the solutions mentioned, it has been the most powerful, while also being the most scalable.

Other than that, my experiences line up with yours almost exactly.

Re: The sad state of sysadmin in the age of containers (2015)

#332

Post by telchar »

twic wrote: post #256
spiderjerusalem wrote: post #149

Earlier quoted context omitted.

What fascinates me about this is, and sorry for being morbid, but what happens when y'all die? Does knowledge of the lower levels of the stack go away with your generation, or will there be enough of us young ones picking the important stuff up?

Same problem as making sure your system doesn't lose data when a server dies. Make sure you have enough copies of the knowledge by propagating it between people. Try to have some kind of offline recording (books?) for recovery from a disaster where you lose everyone. Have an idea of how to recover at a business level if you do lose the data forever. The trouble is making sure these plans actually work. This is why Ne…

"This is why Netflix randomly execute some of their employees every month"

Would that be their King Kong application?

Re: The sad state of sysadmin in the age of containers (2015)

#333

Post by hodgesrm »

hodgesrm wrote: post #330
scurvy wrote: post #188

Earlier quoted context omitted.

One of the gating factors here is the both the speed of Kubernetes development (move fast and break all the things), and the terrible state of accompanying documentation. If "they should be delivered as cloud services" is some sort of k8s apologist stance for its sorry state of maturity, then we got issues. OTOH, if it's "You shouldn't run it in house unless you have an army of people to read every new commit", that'…

First of all let me be clear I'm not an expert in K8s and certainly not an apologist for bad software. On the other hand there's a level of complexity in distributed systems that is impossible to avoid even in stable infrastructure. You have a design choice of trying to make the system as easy as possible to operate (at the cost of other features) vs. finding operating models that make it less of an issue. Personally…

Also as far as security is concerned it appears to me that a lot of people are deploying technology that they simply don't understand. This is not just a problem with Docker but with apps from ecosystems running on npm and pip.

You can build images securely with Docker but it requires building them yourself, using private registries, checking carefully for vulnerabilities, and testing. If you don't want to do this, pay somebody else to do it right. There's no free lunch.

Re: The sad state of sysadmin in the age of containers (2015)

#334

Post by geggam »

user5994461 wrote: post #176
brightball wrote: post #160

Earlier quoted context omitted.

Would be interested to get your opinion on Puppet/Ansible/Chef/CFEngine/SaltStack

CFEngine is basic text manipulation, it's not comparable to the rest. Puppet and Chef was the first generation. I wouldn't recommend. All the companies and people I know using Chef migrated away from it after many disasters. Nowadays, it's only mentioned in interviews to find out if candidates have real world fire fighting experiences. Ansible is good. Used that for managing hundreds of machines at multiple jobs (som…

Ansible starts getting painful around 1500 nodes.

Re: The sad state of sysadmin in the age of containers (2015)

#335

Post by elcritch »

w4tson wrote: post #47
cwyers wrote: post #13

Earlier quoted context omitted.

There are over 2.9 million lines of code in Apache Hadoop alone, not counting dependencies. If you can't trust Apache, you can't trust Hadoop, regardless of whether or not you can compile it yourself.

EXACTLY. It’s just software. There’s no easy answers here. There’s vulns in everything from hypervisors to node modules. Building from scratch isn’t going help. Pragmatic solutions where possible. Like scanning containers, using OWASP tools on your repos etc

I wish I could upvote this twice. Auto scanning containers should become the default process for everyone.

Re: The sad state of sysadmin in the age of containers (2015)

#336

Post by nwmcsween »

gm-conspiracy wrote: post #133
nwmcsween wrote: post #106

This has been my complaint from day one. Instead of docker have something like ports or pkgsrc and simply create tools that simplify sandboxing, like cgexec, a google kafel -> ebpf filter then all the package manager has to do is well package. Docker IMO is a mudball of concerns that need to be separated.

BSD jails?

I don't know enough about BSD to really comment but do any of them have resource and syscall limiting (xgroips, seccomp) like functionality? IMO it should be mandatory for distributed application bundles.

Re: The sad state of sysadmin in the age of containers (2015)

#337

Post by mmt »

mancerayder wrote: post #113
Angostura wrote: post #38

Earlier quoted context omitted.

I think that may be hehind some of these tales of 'luddite sysadmins'. Sysadmins need to keep things running and complexity and dependencies, even if they being convenience is something that makes them nervous. It's not about being a luddite, its about being able to hold a mental map of how it all works, so that when it stops working you can dive in.

And not to mention getting called at 2 am because something didn't build or the release bombed and having to examine, for the first time, some over-complicated mechanism to build things that goes through some pipeline where you're eyeballing large log files full of long exception chains. Devs see the world as one where velocity and progress is among the most important, while 'sysadmins' (a term no longer used by comp…

> It feels more that devs broke down the wall sometimes.

That's because only programmers get Peter Principled into the manager roles[1] that decide on whom to hire and how to run "Devops".

[1] or become startup founders

Re: The sad state of sysadmin in the age of containers (2015)

#338

Post by w8rbt »

setquk wrote: post #213
scurvy wrote: post #194

Earlier quoted context omitted.

It's a legit concern. There was a NANOG panel about this exact thing. I believe the quote was, "Take a look around. We're all old and greying. We have a severe pipeline problem." And then much to AWS' dude's dismay, the topic shifted towards blaming cloud services because no one takes the time to learn how any of this works any more. Want to guarantee your child's future employment? Don't just teach them to code (the…

I’m going to teach my children how to navigate the world of insane Harry-Potter-esque rules which all IaaS/PaaS platforms enforce upon you. They will become software language lawyers and be masters of the electric Disney dollar. You know like “ahh don’t call the messaging endpoint more than 800 mega-milli-times per mega-nano-second or it will cost you three bazillion CPU credits, but only on three and a half cores wh…

DDOS is now a billing issue.

Re: The sad state of sysadmin in the age of containers (2015)

#339

Post by dsr_ »

mathnode wrote: post #275
ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

I am genuinely curious, how much ops is hand written in just plain old bash or simple scripts?

Bash is everywhere but in small quantities. If it's more than a page of bash, it's probably time to rewrite in something with stricter rules and fewer surprises, or better libraries, or both. Perl or Python is quite common at that level.

Re: The sad state of sysadmin in the age of containers (2015)

#340

Post by spc476 »

rootlocus wrote: post #266
gaius wrote: post #262

Earlier quoted context omitted.

It can reshape itself, and it most definitely understand network protocols It really, really can't! That quote about the Internet interpreting censorship as damage and routing around it? Or the one about information "wanting" to be free? Taken wildly out of context.

Then what about routing protocols such as RIP, OSPF, etc? I'm sorry, but I don't know what quotes you're referring to.

RIP and OSPF are interior routing protocols---that is, they're used for routing within an organization (or autonomous system in Internet lingo) and deal with technical routing issues (fastest link, most bandwidth, etc). BGP is for routing between organizations and deals with political issues than technical issues (we need to send all traffic here due to contracts, unless it goes down, then shift traffic over there; and refuse routing information from such-n-such organization because they don't have their act together).
Post reply on HN