My hope is that declarative distros, who are a practical implementation in software of the "Datacenter as a Computer" by "ancient" Google, like NixOS or Guix System became widespread and the NixOps/Morth/Disnix model can evolve in a more structured and stable solution pushing classic distros from the late 80s to the graveyard alongside with "the product of devops" witch are not CI/CD but paravirtualization for anything pushing docker/k*s and so on as a less absurd full-stack-virtualisation just to keep ignorant able to deploy proprietary stuff knowing the outcome is Serverless http://evrl.com/devops/cloud/2020/12/18/serverless.html or the modern mainframe named cloud.
A Eulogy for DevOps
131–140 of 175 posts
Re: A Eulogy for DevOps
#132> The cause of its death was a critical misunderstanding over what was causing software to be hard to write. The belief was by removing barriers to deployment, more software would get deployed and things would be easier and better. Effectively that the issue was that developers and operations teams were being held back by ridiculous process and coordination.
So many arguments are based on strawmen...
I like devops / daily deploys, because they're part of the puzzle leading to higher quality code being deployed on production, and associated less stress.
The point is (for any individual developer) not to actually deploy their progress every day on prod, but to have the option to do so. This leads to code going on prod when it's ready, but no sooner. If the problem is more difficult than anticipated, code still sucks and needs refactoring, well, you're just going to work on it as long as it needs it and deploy it only then.
Meanwhile if you have let's say monthly releases, you will get the death marches, because delay of one day can mean delay of one month / quarter / whatever. Everyone feels the pressure to deliver, leading to suboptimal choices, bad code being approved etc.
Re: A Eulogy for DevOps
#133Am I correct in understanding that microservices and DevOps are closely related - in that microservices trade code base complexity for operations complexity?
DevOps predated microservices by a good bit. And DevOps was meant to reduce operational complexity, because the people who best knew how to run the software were the ones who made it. (DevOps also predated commoditization of the infra layer, we didn't know about Docker or k8s yet.) And they could use the direct experiential feedback loop of driving the car they had built to plow understanding right back into the next…
Yes but in the end there was corporate capture and so that resulted in engineering teams like: dev teams create the SW but has no right, DevOps team that has no idea about what is doing the software or how to do dev btw but they have all the production rights...
Re: A Eulogy for DevOps
#134Earlier quoted context omitted.
I bumped into a DevOps job at Teradata, did it for two years, then left for where I belong, in gaming as a backend developer. It was a corporate with good pays and perks including international travel etc. Some friends are still there doing DevOps very happily. Talking to them it never feels a like a dying field. I hated my DevOps role so pardon me for only reading the headings of the article, but talking about it in…
The deep deep irony of hiring someone specifically to do "devops". That's not devops. Devops is when you have developers who are empowered. When you hire someone specifically to do devops you are hiring an ops team and calling it devops. So, yeah. That still happens, sure. However, I do think it's changing these days: linkedin > connections -> control-F -> 'devops' => 0 hits "Site Reliability Engineer" -> 10 hits I t…
> Devops is when you have developers who are empowered.
That’s just half true. If something doesn’t work in production, guess who can get mysql production logs? Not the developers. Can you access production dbs (not that that’s the way to fix things, but I have seen that in many orgs)? Nop. And if there’s anything you can do yourself (like changing things via TF files), you’ll need to wait until the gatekeepers approve your request.
The gatekeeper are usually called platform engineers.
But in any case, I don’t want all that power (and responsibility) either if it does not come with an increased compensation.
Re: A Eulogy for DevOps
#135> Now servers were effectively dumb boxes running containers, either on their own with Docker compose or as part of a fleet with Kubernetes/ECS/App Engine/Nomad/whatever new thing that has been invented in the last two weeks.
No joke, containers are amazing, regardless of how quickly you try to move or how often you need to deploy.
I remember a project where the performance turned out to be horrible because someone was running Oracle JDK 8 instead of OpenJDK 8 and that was enough to result in a huge discrepancy, here's an example of the request processing times during load tests: https://blog.kronis.dev/images/j/d/k/-/t/jdk-testing-compari...
That would have been solved by Ansible or something like it, of course, but containers get rid of that risk altogether, since you need to package the JDK your app needs (and that it will be tested on).
With a bit of work, using containers can be quite consistent and manageable - have Ansible or something similar set up your nodes that will run the containers, run a Docker Swarm, Hashicorp Nomad or Kubernetes cluster (K3s is great) that's more or less vanilla, something like Portainer or Rancher for easier management, Skywalking or one of those OpenTelemetry solutions for tracing and observability, throw in some uptime monitoring tools like Uptime Kuma, maybe even something like Zabbix or a more modern alternative for node monitoring and alerting and you're set. Anything that's self-hostable and doesn't tie you up with restrictive licenses (this also applies to using PostgreSQL or MariaDB instead of something like Oracle, if you can).
You don't need to have every team branch out into completely different tools because those are the new hotness, you don't need to run everything on PaaS/SaaS platforms when IaaS is enough, realistically most of what you need can be stored in a Git repo that will contain a pretty clear history of why things have been changed and even some Wiki pages and/or ADRs that explain how you've gotten here.
The situations in the article feel very much like corporate not caring and teams not talking to one another and having no coordination, or growing to a scale where direct communication no longer works yet not having anything in place to address that. If you're at that point, you should be able to throw money and human-years of work at the problem until it disappears, provided that people who hold the bag actually care.
For what it's worth, regardless of the tech you use or the scale you're at, you can still have someone in charge of the platform (or a team, where applicable), you can still have a DBA or a sysadmin, if you recognize their skills as important and needed.
Re: A Eulogy for DevOps
#136Earlier quoted context omitted.
This is the elephant in the room: all "new" technology isn't fundamentally new. Terraform is just curl with state management for lots of different websites, Andible is just a YAML to Bash converter, and so on. If you have good fundamentals, not only are these things easy, they're also incredibly frustrating, because you can easily see their limitations. You can always tell how experienced someone is by how well they…
Ansible is a lot more than just "YAML to Bash". For instance, Bash scripts are not idempotent. Ansible playbooks are. Sure, you can hack together a way to ssh and run commands very easily but it will break at some point
Ansible playbooks are as idempotent as bash scripts.
You need to include checks when writing them so they actually are idempotent, just like with bash script, and they happily leak idempotency-breaking details like "restarting a service" works only if a) service currently runs b) because "restart" is a command, not expected state of "the service was restarted and is now running
Re: A Eulogy for DevOps
#137Earlier quoted context omitted.
>> "discovered" that troubleshooting Kubernetes was a bit like Warhammer 40k Adeptus Mechanicus waving incense in front of machines they didn't understand in the hopes that it would make the problem go away I'm not familiar with the reference but logs, events and metrics seem pretty useful. Port forward, shell into containers, view logs, etc. I don't see what is so bad about it.
> shell into containers The problem here is that once your in and need tooling what do you do? Your likely going to have to re-build the whole container with that tooling on board (if its even possible)... Containers aren't problematic in themselves. There are reasons to use them. Ruby, php, JS python (fucking venvs) have this habit of contaminating a system. Containers are a great way of... containing that shit. But…
That's why debug container facility was added... Few years ago by now, I believe.
Ability to inject a container with debug tooling into specific pod.
Re: A Eulogy for DevOps
#138Earlier quoted context omitted.
that doesn't work at scale though. the difference between dev, staging, and prod when there's a handful of services is fine. when there's 300 of them, and 200 of them are broken in the dev environment at any given time, that means you can't actually use the dev environment to do development in because the 300 other teams are also trying to do development in that same environment so their stuff is just as broken as yo…
We have environments with thousands of services and it scales fine. Why would dev (or your lowest integration environment) be broken most of the time? > except that means the Kafka team That (a Kafka team, or a DB2 team) is a bit of a red flag for me. Many (but not all) “tech” teams like that are part of the problem. Cross functional delivery teams work much better, because running tech X in isolation is often not va…
Where are you that has thousands or services and it scales fine? Twitter and Facebook famously both don't have a separate staging environment because they have thousands or services and it doesn't scale fine. They do canary releases and feature flags so as to do gradual deployment and testing in prod. If they can't solve the problem but you're somewhere that has, my next question is are you hiring?
Dev is broken because devs are doing dev on it. I mean, it generally works, but it's the bleeding edge of development so there's no real guarantee that someone didn't push something that doesn't work in a way that the rest of the company is relying on.
What is the DBaaS or pub/sub team's commitment to uptime in the staging environment? It's staging. if they have to commit to a reasonable uptime, they can't actually use it as staging for themselves. Saying they need to improve is trying to handwave out the fact that they need a staging environment where they get to run experimental DBaaS or pub/sub things.
Re: A Eulogy for DevOps
#139Earlier quoted context omitted.
that doesn't work at scale though. the difference between dev, staging, and prod when there's a handful of services is fine. when there's 300 of them, and 200 of them are broken in the dev environment at any given time, that means you can't actually use the dev environment to do development in because the 300 other teams are also trying to do development in that same environment so their stuff is just as broken as yo…
Development should never be broken, ever. Ready to ship and broken are two different things. By ready to ship it more like v2 has 8 total new endpoints and only 2 are ready, then 4, then 6, then 8. When the tests fail, or the code quality goes down, the deployment fails. I would rather have DEV broken than PROD. Not sure how going directly to PROD would make anything better in this scenario. That is what Integration…
> Development should never be broken, ever.
And devs should just not write bugs in their code, ever. Development mostly works, but is going to be broken in subtle ways that other teams are going to pull their hair out because their thing doesn't work, but their thing doesn't work because your team has broken development in a very subtle way that only that one team tickles. Your tests didn't catch it and the rest of your team didn't catch it during code review.
Re: A Eulogy for DevOps
#140The funny thing is, when ZIRP ended, people realized that the original dev / ops separation was actually better & regressed back to it with an armada of new tools and acronyms.