Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The sad state of sysadmin in the age of containers (2015)

vitavonni.de

Post reply on HN
Print view View original

271–280 of 435 posts

Re: The sad state of sysadmin in the age of containers (2015)

#271

Post by rootlocus »

gaius wrote: post #269
rootlocus wrote: post #266

Earlier quoted context omitted.

Then what about routing protocols such as RIP, OSPF, etc? I'm sorry, but I don't know what quotes you're referring to.

Then what about routing protocols such as RIP, OSPF, etc? Don't forget BGP. I don't think they do what you think they do, at least, not to the extent you think they do them. There is a hell of a lot of manual work in running any sizeable network even within a single organisation.

And what exactly do they do that I don't understand?

> There is a hell of a lot of manual work in running any sizeable network even within a single organisation.

I'm not trying to say they do everything, and no manual work is required. I'm trying to say machines are already doing part of it. OP believes that give a world where machines can code, they can't design or maintain networks, which I find truly ridiculous, since machines are pretty far from doing any kind of "coding" today, but they do networking and network protocols pretty well.

Re: The sad state of sysadmin in the age of containers (2015)

#272

Post by rconti »

toomuchtodo wrote: post #219
sudosteph wrote: post #166

Earlier quoted context omitted.

I feel like there has always been a contingent of sysadmin / ops folks who preferred the "Better to ask for forgiveness than permission" model. They still hate when things break, (so not quite fans of developers with a "move fast and break things" philosophy) but they care more about big picture improvements and ease of upkeep than enforcing any particular process. Detecting problems and being able to roll back is ty…

> It makes sense that these types naturally gravitated towards the devops models. I'm really not sure where this leaves the more compliance-minded systems folks though. Working for profitable businesses where stability is valued over velocity.

It's rough because, like many backend type jobs, the best thing that can happen is nothing breaks. Incremental improvements in stability or scalability will not be noticed, but every single change you make is a massive risk of a page at 2am, all-nighters trying to fix things, outage reports, incident reports, root cause analysis reports, etc. You're stuck between process and outcome.

You have to constantly fight the urge to just never touch anything.

Re: The sad state of sysadmin in the age of containers (2015)

#273

Post by ixxie »

davexunit wrote: post #76
myWindoonn wrote: post #9

Reproducible build tools are a thing. Try Nix sometime. nixpkgs has, say, Hadoop. nixpkgs can make Docker containers too.

nixpkgs isn't rigorous about reproducible builds. Hadoop is actually a great example of this. They do not build it from source, a prerequisite for calling a build reproducible. Instead, they download the binaries that the Apache project has already built and run patchelf on them to make them work. https://github.com/NixOS/nixpkgs/blob/master/pkgs/applicatio...

That is true, but in my experience in small scale personal desktop and cloud computing, NixOS is in practice reasonably reproducible on the system level.

For me in terms of interface, the strength of the Nix ecosystem is declarative system, ops and service configs in the same language used for package and build specification. The technical strength is striving towards reproducibe builds by hashing the dependency tree to build an immutable store. Yes sometimes this means getting binaries but you can always pin the package version or even have multiple versions in tandem. The practical upshot of this immutability is system level rollbacks, which are generally reliable although there are ways to break it. Yes there is garbage collection.

Nixpkgs has is quite an achievement, and yes it has its warts but we are working hard to make it better. If we manage to shape up the data science I will try it out at work too. Im very curious how it might scale.

Re: The sad state of sysadmin in the age of containers (2015)

#274

Post by WiseWeasel »

AlexCoventry wrote: post #255
phaedrus wrote: post #246

Earlier quoted context omitted.

I once worked at a company which separated IT into 3 teams: developers, DB-sysadmin (ops), and QA (who also managed deployments). Releases were supposed to go in a waterfall model from the Dev group -> QA group -> Ops. QA wanted Dev to submit Word documents for each release with blanks to be filled in with server names. However Ops was so distrustful of Dev that it was not enough for them to lock us out of Prod using…

I have trouble imagining the incentives driving Ops in that conversation.

Ideally the guarding of valuable company (and incidentally customer) data.

Re: The sad state of sysadmin in the age of containers (2015)

#275

Post by mathnode »

ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

I am genuinely curious, how much ops is hand written in just plain old bash or simple scripts?

Re: The sad state of sysadmin in the age of containers (2015)

#276

Post by zapita »

ex_amazon_sde wrote: post #63

Ex Amazon here. Most grumpy system engineers did not disappear: we got hired by Google/Amazon/etc to build large-scale infrastructure... and sometimes sell it back to you as a service. Believe me or not, most of the underlying infra does not run on the popular technology of the year. Far, far from it. That's why it works. Modern devops, with its million tools that break backward compatibility every month sometimes be…

> Modern devops, with its million tools that break backward compatibility every month sometimes becomes the running joke at lunch.

Ironically, modern devops and its million broken tools are a primary source of revenue for cloud providers, helping pay for your lunch in the first place.

Re: The sad state of sysadmin in the age of containers (2015)

#277

Post by gaius »

rootlocus wrote: post #271
gaius wrote: post #269

Earlier quoted context omitted.

Then what about routing protocols such as RIP, OSPF, etc? Don't forget BGP. I don't think they do what you think they do, at least, not to the extent you think they do them. There is a hell of a lot of manual work in running any sizeable network even within a single organisation.

And what exactly do they do that I don't understand? > There is a hell of a lot of manual work in running any sizeable network even within a single organisation. I'm not trying to say they do everything, and no manual work is required. I'm trying to say machines are already doing part of it. OP believes that give a world where machines can code, they can't design or maintain networks, which I find truly ridiculous, s…

since machines are pretty far from doing any kind of "coding" today

The first attempt at a system to turn plain English that even managers could write into executable code was 1959 - COBOL. So you're right in a sense, even 59 years later - but also wrong if you think networks are any more advanced than this. The Internet really cannot "reshape itself" and probably never will be able to.

Re: The sad state of sysadmin in the age of containers (2015)

#278

Post by seorphates »

gm-conspiracy wrote: post #243
seorphates wrote: post #205

Earlier quoted context omitted.

Grumpykins here. I think the term "Modern devops" sort of nails it but not quite how you used it. Most departmental/enterprise sys admins/engineers of lore that had even the slightest necessity for life outside the box scaled anything resembling automation to its breaking points. Combined with knowing and serving the reasons for their existence - developers, users etc., and devops is nothing new - it is now simply th…

A good sysadmin would not look like they are doing much work (everything is humming along and can self-heal minus physical problems), but a good devops person is constantly busy.

I sort of agree but a good sysadmin was never idle on the inside. I'm seeing good devops people getting worked well beyond what I'd consider reasonable expectations i.e. "Oh, look, you can do everything! Here's everything!".

They're being perverted into a role having a full load of pure operations with shit for processes (and, often, systems) and an expectation that you have time to automate and shore up all the shit and technical debt accumulated since.

Can most good or even extraordinary developers simultaneously be elbow deep on a dozen unrelated products and actually get reasonable traction? I can barely keep one glass castle together, myself.

Re: The sad state of sysadmin in the age of containers (2015)

#279

Post by tapoxi »

spiderjerusalem wrote: post #249
karlkatzke wrote: post #222

Earlier quoted context omitted.

What makes you think there aren’t young people doing systems administration? Our last two hires in my most recent job were 23 and 27 respectively. Sure, they’re getting trained in the new hot cloud stuff... as the grumpy seniors figure it out first and set patterns... but they are still doing daily work with some rather ancient stuff.

I'm not saying there are no young people doing sysadmin. What I'm trying to say is that if the new 'infrastructure' that all sysadmins learn is not an open UNIXy system where you can grok all the internals if you want to, but closed systems owned by 2-3 major cloud players, then we kinda maybe have a problem in 20 years? Of course, one can argue that that will just cause a new wave of openness and the cycle continues…

As a "young" (30) sysadmin/devops dude I think that open, Unixy system is Kubernetes. I can take an application, dockerize it, write a helm chart and run it anywhere.

The risk is in treating anything as a black box, whether its a managed service or a container you pull from Dockerhub. It's something you'll get burned by eventually and need to learn from experience.

Re: The sad state of sysadmin in the age of containers (2015)

#280

Post by user5994461 »

virgilp wrote: post #259
user5994461 wrote: post #221

Earlier quoted context omitted.

Yes. The host will run 100% CPU to handle the hundreds of SSH connections. I've been re configuring 300 to 800 hosts many times a day, never had a problem. I think it would take a few thousands hosts for the performance to be noticeably slow and I am really not sure that other tools or systems could take it much better.

I know our SREs once screwed the config for sshd, and considered themselves very lucky that they had puppet on the machines and could push a fixed configuration (if they had used exclusively ansible, that'd be the end of it - no way to connect or to deploy new configuration) [edit] To clarify - ansible is great, and we use it. Just saying that, as everything, it still has (sometimes subtle) downsides in various scena…

Well, teach your sysadmin to use the system configuration tester when they edit a system configuration file.

Nothing to do with ansible really, except that ansible allows to prevent that easily.

Post reply on HN