Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Eulogy for DevOps

matduggan.com

Post reply on HN
Print view View original

81–90 of 175 posts

Re: A Eulogy for DevOps

#81

Post by physicles »

osigurdson wrote: post #37

>> abandon technology like Kubernetes I think a lot of Kubernetes hate is misplaced. It is a great piece of software engineering, well supported and runs everywhere. You certainly don't always need it but don't create a bunch of random bash scripts running all of the place instead of learning how to use it.

Agreed. If we didn’t use kubernetes, we’d have to reimplement a bunch of its features. Back when we were getting started I tried docker swarm because it was supposed to be simpler, but I had weird issues with its networking.

The best part is multiple environments. I can run our full stack on my laptop with k3s and one call to make. We use the same yaml (with kustomize) for all three cloud environments.

It feels pretty boring, which is fantastic.

Re: A Eulogy for DevOps

#82

Post by doctor_eval »

I’m compelled to make a couple of comments:

1. I feel that one big and important aspect of devops that isn’t mentioned is that smaller releases are less likely to have killer bugs. If you can release one change a day rather than 100 changes a quarter then overall I think there’s a strong argument, not to be had here, that you’ll have faster releases and less bugs overall, assuming my next point. This doesn’t take away from the article, but it’s just something I don’t see discussed much.

2. I think a huge part of the problem is that business management keeps trying to abstract away engineering management. The most productive team I’ve ever been part of was when I was able to spend most of my time planning and coordinating the work, as part of an overall vision, while my peers did the implementation and gave me feedback. One side effect of this was that productivity was actually measurable. But the value of productivity is lost on business management who saw me as just engineer - one who had the authority, furthermore, to push back against stupidity and was therefore a pain in the ass. Technical management is not valued, because it’s not understood, and this is seen in the endless cycle of fads designed to make all engineers fungible.

Re: A Eulogy for DevOps

#83

Post by Aeolun »

milkglass wrote: post #48
zer00eyz wrote: post #43

Earlier quoted context omitted.

Its not. The whole ecosystem around it is an example of Conways law and and a Google product. None of the people using it are google. Google, also, runs its own hardware. Shockingly it is a great product if you rent hardware, autoscaling is autospending. No one knows what a feature costs any more because its all just a big bucket your pouring money into for amazon to have 30 percent margin on. We need operations peop…

Where's all the good operations people at nowadays? Have worked with numerous cloud native engineers that do not have good foundational knowledge.

> Where's all the good operations people at nowadays?

Still in the same place they were 20 years ago. They’ve just been lost in the swamp of their peers that know nothing.

Re: A Eulogy for DevOps

#84

Post by [deleted] »

sadops wrote: post #68
betaby wrote: post #61

Earlier quoted context omitted.

> cloud native engineers That's the problem. Yes, I'm the old man yelling at cloud, but indeed 'kids this days' don't know Linux, they know how to provision things from Terraform. In fact it feels like logging in via ssh and checking process with with `strace` is a lost art. Checking PCAP? That's a black magic!

This is the elephant in the room: all "new" technology isn't fundamentally new. Terraform is just curl with state management for lots of different websites, Andible is just a YAML to Bash converter, and so on. If you have good fundamentals, not only are these things easy, they're also incredibly frustrating, because you can easily see their limitations. You can always tell how experienced someone is by how well they…

[deleted]

Re: A Eulogy for DevOps

#85

Post by fragmede »

Sn0wCoder wrote: post #56
lolinder wrote: post #51

Earlier quoted context omitted.

The big advantage to deploying to prod constantly is trunk-based development. Any system where you maintain a separate development branch is one where you're invariably going to be asked to cherry pick feature B to production but not feature A. This is a problem because you tested everything on a version of the code where B follows A, but now B needs to stand on its own. Can it? Maybe. But you didn't test that. With…

Congratulations on moving fast! If possible ‘trunk’ is best. Long lived branches are a problem if you are doing hot fixes to prod. The goal is the process avoids hot fixes and critical bugs. Some industries would not allow software to be released more than every few months. SQA needs to sign off. Users need to UAT. 2-week notice, 15-minute notice to log off. Downtime needs to be communicated to multiple time zones. S…

that doesn't work at scale though. the difference between dev, staging, and prod when there's a handful of services is fine. when there's 300 of them, and 200 of them are broken in the dev environment at any given time, that means you can't actually use the dev environment to do development in because the 300 other teams are also trying to do development in that same environment so their stuff is just as broken as your stuff.

so then you either have to accept that sometimes it's broken and just wait on it, or otherwise agree that staging should always have a working copy of Kafka, except that means the Kafka team can now no longer use staging to stage Kafka changes, so then they have to setup their own separate staging environment and then plumb sufficiently representative test data into that new staging, and then and then and then.

Re: A Eulogy for DevOps

#86

Post by eddythompson80 »

sadops wrote: post #76
TranquilMarmot wrote: post #17

Earlier quoted context omitted.

I _really_ miss having a dedicated QA team / process. I noticed that the role essentially disappeared a few years back, and now developers and users have to be the ones asking "is this working the way it should be?"

Pouring out for QA over here. My first gig ever, we had lovely QA people who spent all day just... using the application and trying to make it break. When it did, they'd fail tickets back to us, and we'd fix them. It worked so amazingly well. Now? Hurried, bored devs do it to get the next release out the door, and it shows.

I worked in QA for the first 4 years of my career. There was a lot of creativity in “how to make an application/feature break”. Sure, most of the time I didn’t realize that 10 scenarios I would run through were moot because of how a feature was implemented. But often, that “creativity” would expose major issues that I had no idea about their root cause, but would drive significant (or important) changes to the product.

One of my favorites in my first couple of months in my career was testing a service that accepted user uploads and returned some value based on the upload. I thought “oh, I should just test a butt-load of invalid files”. Where am I gonna get a butt-load of invalid files for our service? I’ll just write a script that iterates and uploads everything under my `C:\Windows` folder. Discovered 2 bugs. One was that some random binary files there caused the parser the service was using to segfault and another about how the service stored those temp files before validating them. The first required involvement from a totally different team in the company because of “how serious it was”. As a fresh college hire at the time I got a ton of praise for “exposing such a critical buffer overflow” that went undetected for years (lol, I had no idea what I was doing but I took the praise). The latter was because the developer of the service never cleared their /tmp directory where they wrote user uploads before validating them. I ended up filling the VM disk with junk that took down the entire service.

Re: A Eulogy for DevOps

#87

Post by physicles »

clvx wrote: post #44

One item the DevOps mindset missed was reproducibility. Fast feedback loops in spirit tell to have a way to know what’s wrong but it doesn’t tell how to reproduce it as you have layers and layers where your code is run. So, you are in a spot of I kinda know what’s going on but I have no way to reproduce it because: - the application has hardcoded paths. - the service discovery isn’t dynamic - the branching strategy d…

Reproducibility is one of the things we got with containers and k8s (and before that, 12-factor apps). I run k3s on my laptop with a minified stack, which is good enough to mimic nearly any behavior on prod.

Cloud-specific saas (like proprietary databases and AWS lambda) are awful for this though, and we avoid them where possible. Our S3 code was a bug farm until I discovered minio. Now we’re stuck with Snowflake because scale and speed, but the fact that I can’t iterate fast locally has caused me to lose days of my life that I’ll never get back. Hoping I can hack something together with duckdb when I get time.

Re: A Eulogy for DevOps

#88

Post by weitendorf »

sadops wrote: post #70
weitendorf wrote: post #52

Earlier quoted context omitted.

It's basically a prototype that industry ran away with. It leaks implementation details everywhere and pushes way too many config options up to the developer. Because industry ran away with it before it could be good, it takes projects like Cilium to push it in the right direction, but those take forever to get adopted and are really hard because they don't live in the product itself. You'd need like 10 more Ciliums…

Kubernetes is just a poor Linux clone with extra steps. Seriously, it has all the basic parts of an OS, just half-assed: the scheduler, the networking, the state management. We already had way better operating systems that can, you know, schedule workloads and talk to the network, and it didn't require gigabytes of YAML and string templating to make it happen.

Yes. I actually wrote another top line comment about this which I deleted. Kubernetes is like someone decided to create a distributed Linux where you had to configure and learn about every part of the operating system. Linux proves it’s possible to provide simple “deep interfaces” to OS-class, hard problems.

Most people in industry have never used Borg or something like it, and so think Kubernetes is like apex technology rather than a step backwards. And to most people working directly on tech like Linux/containers/low-level infra is something they don’t even consider doing because they subconsciously think it’s off limits. So to them Kubernetes is amazing rather than riddled with fixable problems.

Re: A Eulogy for DevOps

#89

Post by weitendorf »

llama052 wrote: post #62
weitendorf wrote: post #52

Earlier quoted context omitted.

It's basically a prototype that industry ran away with. It leaks implementation details everywhere and pushes way too many config options up to the developer. Because industry ran away with it before it could be good, it takes projects like Cilium to push it in the right direction, but those take forever to get adopted and are really hard because they don't live in the product itself. You'd need like 10 more Ciliums…

Can you clarify what you mean by "leaks implementation details everywhere"? I like to think of kubernetes as a big orchestration platform that you can choose to use what you need. If an ingress and pods work then use that, otherwise extend an throw an operator up for what you need (it likely already exists). Cilium for instance is great for that, so is Istio and the like. They aren't hard you just have to understand…

You shouldn’t have to think about all the implementation details of your deployment target. There shouldn’t be platform engineers or Kubernetes experts. Nobody should be writing YAML, or getting paid to set up Istio. Nobody should have to learn the Kubernetes architecture or know about the kubelet or EBPF. The tools should be simple enough with good defaults that application developers just click a button and have their code run somewhere. Right now platform engineers fill that gap, because the underlying tech doesn’t.

IMO you are thinking too much like an engineer if you are saying you “just have to understand networking”. Why? It’s always better when the problem gets solved in a way that allows you to not think of it too much. Right now SREs and the platform team do that for application developers because Kubernetes only does it halfway.

Infrastructure/devops/SRE is a pure means to an end of getting actual applications (the ultimate source of all the value in software) to run. It’s an obstacle. Right now the obstacle is bigger than it needs to be

Re: A Eulogy for DevOps

#90

Post by Sn0wCoder »

fragmede wrote: post #85
Sn0wCoder wrote: post #56

Earlier quoted context omitted.

Congratulations on moving fast! If possible ‘trunk’ is best. Long lived branches are a problem if you are doing hot fixes to prod. The goal is the process avoids hot fixes and critical bugs. Some industries would not allow software to be released more than every few months. SQA needs to sign off. Users need to UAT. 2-week notice, 15-minute notice to log off. Downtime needs to be communicated to multiple time zones. S…

that doesn't work at scale though. the difference between dev, staging, and prod when there's a handful of services is fine. when there's 300 of them, and 200 of them are broken in the dev environment at any given time, that means you can't actually use the dev environment to do development in because the 300 other teams are also trying to do development in that same environment so their stuff is just as broken as yo…

Development should never be broken, ever. Ready to ship and broken are two different things. By ready to ship it more like v2 has 8 total new endpoints and only 2 are ready, then 4, then 6, then 8.

When the tests fail, or the code quality goes down, the deployment fails. I would rather have DEV broken than PROD. Not sure how going directly to PROD would make anything better in this scenario. That is what Integration is for never broken always ready for promotion to prod. Plus a few broken dependencies means ‘yellow’ === impairment not unusable. If the app is unusable for any one dependency that is another issue.

Mocking data when you are ahead of another team is always the reality…

Post reply on HN