321–330 of 363 posts
Co-opted, branded and misinterpreted by the enterprise - "#DevOps" is not DevOps as it was intended. Pretty spot on.
Earlier quoted context omitted.
My theory is we need to add "software history" to computer science education programs. How many developers whose careers began after 2000 have ever hand written or know the power of a Makefile? I've worked at research labs and major animation studios that back in the 90's the entire infrastructure was fully automated thru Makefiles, and it purred like a well fed cat. There are hundreds of thousands of forgotten perfe…
As projects grow the Makefile becomes pretty much a big bash script, but with a specific syntax that I always struggle with.
Run: Task runner that helps you easily manage and invoke small scripts and wrappers
https://github.com/TekWizely/run
Defining commands feels like make, but comes with a bunch of extras targeted at the needs of a task-runner.
I hope you'll check it out!
Earlier quoted context omitted.
>>the only alternative is to start making CPUs by hand Agreed. For some applications the cloud difference is significant; for many (most?) others though, "Cloud" is just rebrand of "Hosted". And even for more cloudy offerings, while I'm in a very specific and different part of IBM, some of the old timers/architects/powers-that-be keep trying to explain "We had that in 1969 !!!" :-D Agreed also at rewriting of history…
> My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :) As someone with 20+ years in IT, I agree - a lot of these "new and exciting paradigms" are not new at all. My personal favourite is how many large multi-nationals are now building in-house clouds? WTF is the difference between an "in-house…
I have bewilderingly tried to discern why software development continues to grow more and more complex. It wasn’t always like this. There was a time when we talked about languages and OSes and libraries as if they made a difference on how much you could get done with as little people and cognitive load as possible (the claims were very much overrated, but the point was we acted like it mattered). And then it started…
There is this moronic obsession with "scaling" in this industry, and it just blows my mind. You can have a wildly successful and profitable company while serving just a few hundred customers that requires nothing more than a basic LAMP stack. Meanwhile people are now spending millions of dollars and years of person-hours building MVPs in the cloud that won't ever go anywhere because the business model sucks. Focus on…
This is us. We have a couple thousand customers in a B2B space with large revenue per customer. Growing rapidly and our node.js stack is running on a tiny EC2 instance and RDS server.
I spent a lot of time in the last couple years trying to fight back against attempts to overcomplicate this by using new services & new tooling. I just see it all as stuff that's going to slow down the single most important thing to us from a business perspective - which is writing new features to optimise our internal workflows - and give us more headaches from an operational perspective.
Even the stack we're using feels ridiculously over the top and complicated compared to a basic LAMP application. Everything in node.js feels like a huge pain in the ass. If there's not a node module you can install to immediately solve your problem (which of course just adds a different set of problems) even writing basic things feels exceptionally arcane and time consuming compared to a basic PHP implementation of the same thing. Maybe it's just a side effect of our dev team size (currently very small, only five full time devs) but the overhead of it all just feels like it's not worth it.
I have bewilderingly tried to discern why software development continues to grow more and more complex. It wasn’t always like this. There was a time when we talked about languages and OSes and libraries as if they made a difference on how much you could get done with as little people and cognitive load as possible (the claims were very much overrated, but the point was we acted like it mattered). And then it started…
I like the Uncle Bob explanation: the number of developers has a doubling rate of ~5 years due to the constant entry of new developers [1]. Over time, the number of inexperienced developers far outweighs the experienced ones. Inexperienced developers naturally gravitate toward complexity because they don't know any better. Couple that with a social drift toward hyper-specialization. That sort of hierarchy naturally c…
There are thresholds of effort for certain things at largely constant levels. E.g.: designing a new language, build tool, web application framework, or database all have some minimum effort that needs to be invested. In the past, there just weren't enough developers in the entire world to "overdo" these things, so there was a relatively small pool of languages, tools, and frameworks to choose from.
Now, individual corporations have armies of junior developers spitting out frameworks and query languages like a machine gun. There's so many now that you or I haven't even heard of 99% of them!
As the number of available developers grows exponentially, so does their capacity to "reinvent wheels". Their ignorance of existing wheels to they could be reusing grows exponentially also. The result is an exponentially exploding set of ad-hoc, incompatible systems.
In my recent semi-DevOps, semi-Cloud-Engineer role I've come across an absolutely bewildering array of tools even when I've generally restricted things to one cloud and one language's ecosystem. Heaven help you if your project has multiple languages!
Several years ago, I did "DevOps" work for a year or two. My gig essentially consisted of automating the repetitive Ops tasks and giving engineers self-service tools to get their stuff into prod efficiently.
At the time, things like Puppet & Chef were commonly-used tools, Vagrant was a widely-used development tool and Ansible was the new kid on the block. I don't remember there being much YAML yet, and I'd only heard a few things about Docker. I wrote several custom tools in Python, and we used Jenkins as our CI/CD server.
"DevOps" in those days was a cultural thing. The DevOps engineers owned the infrastructure and the software engineers owned the software. It was the DevOps engineers' responsibility to give the software engineers the tools needed to deploy, and also to ensure that the SWEs weren't causing outages (where we were the first line of defense).
It started to get bollocksed up when hiring managers started defining DevOps roles in terms of tools used, and the tools themselves supplanted communication & culture as the definition of DevOps.
With YAML, k8s and the rest of the nonsense, I don't think DevOps is very possible these days, because when your config is tens to hundreds of thousands of lines of YAML, the tooling doesn't even allow for a culture of self-service & communication. SWEs more or less have no choice but to chuck stuff over the wall, and DevOps or DevSecOps engineers (or whatever buzzword nonsense the industry has adopted) have to perform augury to construct the configuration. Because of cloud vendor lock-in or just sheer complexity, whatever runs in prod is quite different than the dev environment, and everything just limps along.
EDIT: Now I read things about "MLOps" and so on. When will the madness stop?! Apparently we keep allowing recruiters and dimwit HR bureaucrats to define software processes and culture.
Earlier quoted context omitted.
I don’t think you’re disagreeing with me. Developers should know what their code runs on. They shouldn’t have to add managing that to an already full schedule of work. That’s the difference.
Back when I was in the real world [1] working for a startup, I would do your typical serviceless solution with Lambdas, S3,SQS, etc. I couldn’t just use ClickOps and create everything on the console and expect someone else to recreate everything with IAC. I had to know how to do it. I think to push back is rightfully coming from the “ops” part. I consider “creating the CloudFormation/CDK/Terraform” code as part of “d…
> If you use Docker, wouldn’t you consider creating the Dockerfile as part of development?
Sure you could argue a developer could, or even should create these things in theory. The problem is then when it goes down I have made two problems out of one. Now I have to manage the infrastructure of a system and what is running on it. Realistically, and even in my current job, it's actually several systems. Now when something breaks I have to pray I can fix it. Instead of allowing a team of infrastructure professionals to at least insure the hardware is working my 8 hour day turns into 14 or 16 very quickly the second one thing goes wrong.
Earlier quoted context omitted.
> My dad has been IT director and he chuckles when I talk to him about "new and exciting paradigms" which he of course sees as turning a circle to what they had in 70's and 80's :) As someone with 20+ years in IT, I agree - a lot of these "new and exciting paradigms" are not new at all. My personal favourite is how many large multi-nationals are now building in-house clouds? WTF is the difference between an "in-house…
I think the difference is the interface. The interface to the shared-use datacenter, if you're lucky, is a spreadsheet that declares the static resources you own and a remote hands guy that can tackle things beyond the capabilities of your remote KVM. If you need more capacity you need to work with the datacenter folks to order physical machines that might show up in a few months. The interface to the in-house cloud…
Not to say your scenario isn't valid and real, but I live that scenario every day today with in house cloud and virtualization too - it takes months of approvals and solutioning and security assessment and network engineering and procurement and costing and whatnot... To deploy a windows vm.
Earlier quoted context omitted.
Any general purpose architecture allows spaghetti, there's no way around it.
Has there been an architecture movement that formally embraces the spaghetti, aiming for peaceful coexistence? Detractors will certainly point at SOA and shout "that one!", but I mean one that openly admits..
- Systems get messy unless you have configuration management that converges on top of ephemeral instances
- Build packages that work the same in all environments if possible
- Dev-prod parity: if that means running Docker locally or on some throw-away dev servers, do it
- 12factor principles
- Every business department has an API and shares the same or similar messaging and storage tech
- Change control: have high-available (HA) to where you're never upgrading individual machines and always backup/restore/replacing with fresh nodes
- Repeatable, precise, cached, incremental, distributed builds. Even if it means throwing out timestamps in the binaries
- Cache build artifacts forever (almost)
- Cryptographically sign commits and build artifacts (don't sign hashes because those are weaker)
- Canary, sharded deployments
- Rarely/never allow changes directly to boxes
- Require PRs made through configuration management
- Require PRs have to have another developer code review them
- Automate the heck out of linting and testing before it's generally allowed to drop in prod
- Store secrets either in conf mgmt or in a separate system like Vault
- Have SREs who know how things will end up in prod to troubleshoot the complexity and provide CI/CD and {I,P,Db,S}aaS
- Reduce the number of duplicated systems to a minimum but not to where it is too awkward or difficult to maintain
- Remember that prod, corp, and endpoints (phones and laptops) aren't the same but try to share bits as much as possible while isolating differences
- Corp tends to run more heterogeneous than prod. It's okay but don't let it get out-of-hand without having recommended (as opposed to mandated) standards except for security and third-party component review
- Eliminate technical debt: don't allow layers of crap or be precious about gross code just because it works
- Consider the risks and impact before making changes
- Have a backup/DR/BCP plan