Live data from Hacker News

The Big DevOps Misunderstanding

wolfoliver.medium.com

121–130 of 321 posts

Re: The Big DevOps Misunderstanding

#121
post #61

Earlier quoted context omitted.

A sysadmin wouldn’t write payment code. A sysadmin would write the tooling to ensure the payment code could work (failover databases, redundancies) and put safeguards in place so that it does not get traffic that is not intended for it. Sysadmins used to write the code that touched thousands of machines at once, developers used to write the more complex code which were the bread and butter of the company. But where I…

> sysadmin would write the tooling to ensure the payment code could work Uhh if by tooling you mean a bash script to deploy ssh keys to prevent team A from accessing a machine owned by team B. You really have no idea what you're saying.

Dijit is completely correct.

A junior sysadmin executes checklists, writes simple scripts, reads logs, and solves common problems, all while learning the complex supersystem of their workplace. They learn configuration of systems in use and how infrastructure is arranged and commonly changed.

A sysadmin helps to create checklists, writes more complex code to work in an automation system, and generally solves problems with non-obvious causes and/or remedies. They know all the routine events in their workplace, and add to documentation. They build, configure, test and recommend new tools and solutions to users, devs and the rest of the ops staff.

A senior sysadmin works on policy, debugs automation, solves complex interactions, and has an excellent understanding of complex systems and networks, not just at their workplace but in interactions and hypothetical interactions with the Internet at large.

Re: The Big DevOps Misunderstanding

#122
post #50

Earlier quoted context omitted.

So this, but instead of starting with Sysadmins with tens of years of unix experience, start with junior developers with 1 year of experience using Docker as a developer and have them read a blog post on how-to set up Kubernetes.

Now I feel this is how it is exactly happening at work even though I am not part of "devops" team. Any working solution at my place is now legacy ready to be dumped and replaced with next generation stack. Other day "devops" engineer couldn't figure out wrong directory permissions and whole deployment failed.

we once had this issue at an old job.

People wanted to replace a monitoring solution for something new, because snmp was "bad and outdated".

SNMP is definitely not the holy grail of solutions (to be honest, its kind of a mess). But atleast SNMP is widly supported, works reasonably well, and gets you the data in a dashboard you need. Even for those old Cisco 7200's with an uptime of over a decade.

Being new and shiny in operations is usually a sign of inmature technology. (with the bugs and issues associated with it).

Re: The Big DevOps Misunderstanding

#123

I had to basically give up my full time role as a Data Engineer who is supposed to solve "big data" problems (Storage, Streaming, Machine Learning pipelines) etc, and had to solely focus on DevOps because there was no other expert in the team. We (a colleague and I) basically learned Terraform, then Kubernetes, and then Helm in an extremely stressful environment. I wouldn't recommend that to most people. (Also, I can…

> regular Java/Go/Rust SDK style

I thought that until I started working with Gradle (which is a build script tool who's syntax is nearly-fully-functioning Groovy). I miss Makefiles - they were strictly declarative and, while I've seen some hairy Makefiles, you'll never need a debugger to figure out what they're actually doing wrong.

Re: The Big DevOps Misunderstanding

#124

I am so tired of devops as a term. It's a stupid bandaid solution. Infrastructure, regardless if you use a paas or not, is a tough cookie to crack and deserves a normal sysadmin role. Monitoring, resource utilization, network management, library updates, costs among a few are things you need to master, observe and proactively manage, not keep it as an afterthought since it magically just works.

troubleshooting infrastructure is hard.

You cannot usually bring down or change something in production without extensive research because well, its production. Issue usually happen either at seemingly random times, or at moments when doing something directly about it is nearly impossible because of the impact it will have on the production network/system.

Also, in production (in the greater sense of the word, for instance, a datacenter with a virtualisation system running multiple workloads/applications). that most issues between systems start to creep up. Usually, systems themselves have been tested enough, its the unintended interaction between them that throws things off balance.

Re: The Big DevOps Misunderstanding

#125

As someone who has lead a devops team, and still has a devops team reporting to me, I have also struggled with the idea of having a team called devops, since it does go against the devops philosophy of devs owning their software in production. However, in our case (and probably many others), our devops team is responsible for creating and maintaining the tools and processes that allow the dev teams to manage their sy…

Disclosure: this is my role.

I work as a sort of "dev tooling engineer" or "platform engineer" for several companies who are our customers.

After years of consulting in the area of DevOps, the pretty smart people I work with figured out that while the developers' user story in the process of devops transformation is a "revelation", the one thing that usually gets forgotten is the maintenance and upkeep of those tools. So we set up a pilot project where we started hosting, maintaining and upkeeping a set of tools consisting roughly of Jira+Confluence, Jenkins as the CI/CD server, Artifactory for binary hosting and Bitbucket/Gitlab for source code.

It has worked damn well for the first pilot company (signed on 5+ years ago), the second one, and by the third the platform started selling itself (getting a public reference from the first couple companies helped a lot). The second reason for the easy buy-in is to use known software that our customers could host themselves if they had to. This practically means that we aren’t vendor-locking them with our own proprietary tools. We just take care of off-the-shelf tools.

If there is a competent tooling team that takes care of the intricacies of the tools and maintains their uptimes, the dream of "you build it you run it" can still live. There is a giant gap between "you build it you run it" and reality -- much how there is a gap between a pilot being able to fly a plane and a pilot knowing how to maintain an Airbus, the airport and the runways.

It says something about the job market that losing a customer pretty much never happens. It’s so much easier to find sysops/devs than it is to find people who specialize in tooling.

Re: The Big DevOps Misunderstanding

#126
post #32

Personally I think it's pretty great that I can write Dockerfiles, run them locally, and call it the day because this is also what runs in production. So basically I have no clue what the dude is talking about - we have a great setup which is x10 better than what we had before. No?

> Personally I think it's pretty great that I can write Dockerfiles, run them locally, and call it the day because this is also what runs in production. It goes far, far beyond Dockerfiles. Are you on-call for production? Can you debug a problem in production? If not, then that's the root of the problem that the dude is talking about. If these are true for you (not specifically you, any reader of this): - You don't h…

Not to mention the layer that lives behind this.

Can you troubleshoot network level issues that seem to occur between your system and someone else's?

What about storage? Can you make sure your data is still their if your environment gets royally screwed.

I know these are usually done by seperate companies nowadays, but getting an an actual computing environment up and running with redundancy in place (down to the cooling and electrical level) is no easy feat.

Re: The Big DevOps Misunderstanding

#127
post #37

Really great post. When I was at AWS, we did everything with IaC but that luxury of time and investment is not avail to many. We're working on this problem but ensuring that devs has a) full transparency what's going on, b) ability to evolve their arch and c) building right on your AWS account. We just moved someone from Heroku to AWS handling 1B requests/day at 1/4 of the cost. have a bunch of AWS execs backing us t…

cloud is a marketing scam. It is sold to execs as a magic black box that scales and you don't have to be concerned with how it works internally. In reality it's a set of leaky abstractions and you absolutely have to understand how black box works internally plus be aware of thousands of bizarre hard/soft limits and other gotchas.

Re: The Big DevOps Misunderstanding

#128
There seems to be a lot of people here, and presumably a lot of companies, where no one has ever actually seen a successfully run devops team and what they provide.

For one, people are arguing past each other about deployment environments vs developer tools vs being a sysadmin.

Re: The Big DevOps Misunderstanding

#129

Earlier quoted context omitted.

could you give me a hint why you didn't like jsonnet? I dreaded k8s yaml when needing to (write) and use helm packages and think of indenting levels, but jsonnet via grafanas tanka gave me the option to be concise, writing functions and creating reusable bits. Of course then you're not far away from using a real language "sdk style". As you so aptly put, k8s learning curve is beginner hostile and needs system knowled…

Yeah, I think the difference is that I was a) Learning K8s and Helm Concepts (both different things) b) Reading/Debugging K8s and Helm YAML (both different things) c) Ending up at Helm Chart Configs (another config) d) Figuring out how to set all this to helm me deploy the solution I needed and customize it acc to my needs ...all within the span of a few weeks. So by the time I got to some GitHub repos which were usi…

yes, both different things. I guess helm is still the most common entry point for people to handle the contents of a cluster. I'm checking out other suggestions in this thread (dhall), but from my personal experience I can already tell, grafanas k8s jsonnet-libs was a second chance to kubernetes I'm glad I got to know (thanks coworkers).
Post reply on HN