51–60 of 84 posts
Earlier quoted context omitted.
Actually devs focusing on Ops lead to development of the automation we use today for most Ops activities, so I would say that is a success.
We already had such automation tools under system administration umbrella. What do we think we were using Perl for?
Job Security (joking)
It doesn’t mean anything for two reasons: companies have treated it as a catch—all, and there is a glut of people calling themselves SREs who have never operated a server that wasn’t in a cloud. You can learn enough about Linux to be decent at your job on only VMs if you’re dedicated, but I’d argue that until you’ve also dealt with hypervisors, bare metal, and hardware issues, you’re missing some of the picture. “Tha…
> people calling themselves SREs who have never operated a server that wasn’t in a cloud Wait. Since when this is a requirement?
In my career so far, the best Ops-adjacent people all had homelabs. I view it as a huge green flag if you have one. Not necessarily a red flag if you don’t, but it warrants some more questions. That’s not to say “I have a server” means you’re great; you could just be running Plex with nothing in IaC, no config management, no monitoring, etc.
Earlier quoted context omitted.
Moreover, they must be cargo cults. An SRE role was my first job at Google in 2006. It was a product of its time and place. Google could hire the sort of full stack programmer/sysadmin types that the article talks about because they had enough clout and desirability at that time that they could get very skilled programmers to give up programming and do sucky on-call work instead. Even then, just two years after IPO a…
Aren't there things we can do to simplify the software stack, so we're more likely to find candidates with all the required skills in one package? There are certainly fashions we can push back against, e.g. premature use of microservices, Kubernetes, and distributed systems in general.
DHH isn’t always right, and his “you only need a server” mantra isn’t true for everyone, but IMO it is DEFINITELY true for nearly any startup that’s shown sustainable growth. By all means, use cheap cloud resources when you have minimal revenue (unless you already have someone who can administer Linux). But after that? Run a couple of boring servers and front then with a load balancer. You don’t need multi-regional HA at that stage, so don’t worry about it.
Earlier quoted context omitted.
As long as that very description is spelled out clearly near the top of the job ad, it doesn't matter terribly much (within reason) what you call it - job searchers will try various different strings to find it. Personally, I'd call it an ops role, however. In general, most of the issues I've seen with these sorts of roles aren't the naming of them but rather the third bullet in your list. Having an ops role that's r…
I haven’t started trying to hire for this position yet, it’s been something I’ve been thinking of for a while though. Right a small number of developers, myself included, are on call and I’d like to reduce that a bit or just share the burden. My goal isn’t to throw crap over the fence and say “make it work” but rather empower someone to make maintaining and growing our platform their main goal. The developers (again,…
HARD disagree on this. I've done both. Most of the really excellent programmers I've worked with have, at least a little. You can't write highly reliable networking software without a deep understanding of how networking actually works. You can't write highly performing software without a deep understanding of the infrastructure and hardware it's running on. And so on.
I'm not trying to bersmirch yours or your teams abilities here - if you're writing in a high level language and most of your challenges are implementing biz logic then not knowing very much about the underlying infrastructure and hardware is fine, you aren't trying to write a distributed RDBMS, you probably don't need to know this stuff.
But do remember that there are lots of people for whom the hardware, the infrastructure and the application they're writing are inextricably linked. It's not a different mindset, it's just people with additional skills you haven't needed to learn yet.
you're going to be the "ops bitch" for the "real" programmers Rachel is spot on about what is often wrong with IT culture; "typecasting" people for someone's convenience or to get a fancy title leads to learned helplessness and dissmissing other people's expertise and interests. I rather we all try to keep things simple and encourage people to be well-rounded engineers.
Yet she keeps calling ops people “monkeys.”
Just last week, we had some mails escalating, client had "issues" with their on-prem install of our software, which ran on a dedicated VM.
I read the mail thread and turned out the database service "used a lot of memory", and they'd tried rebooting the server several times but it just kept using a lot of memory. So now they had escalated because they couldn't figure out how to "fix" this issue.
Of course, this was not an issue. The database is designed to use all available memory by default, and this was a dedicated VM so it didn't affect other services.
I've seen many such and other instances over the years. While there are certainly awesome ops people out there, which is always a pleasure to interact with, a significant amount are at a much more basic level.
Earlier quoted context omitted.
That's because it's difficult to really define a job in IT with clear description along with set roles and responsibilities. In some orgs, it's the title that dictates what you can/can't do. In some orgs, eventually your title is decided based on your role/responsibilities. In many orgs, your title is just an HR/accounting construct and has no relation to what you do and you do whatever your boss asks you to regardle…
I was in a company where we had separate external and internal titles. The external ones were all fancy like "Principal Architect" and all that. The internal ones told people inside the company what you _actually_ did.
Companies want SRE people but aren't willing to give SRE empowerment and authority.
So companies do what companies to: take a regular team of operations people and slap the SRE term on it, and call it a day.
And it doesn't work, of course.
---
Regarding the empowerment & authority: according to "the book", SREs often play the role of "launch coordination engineers" as in vetting (read: roasting) a service before it goes live and have authority to say "this won't go live, fix this first" and to do so no matter what deadline is going to be missed.
Also SRE team have the extreme prerogative to "give back the pager" as in take a service back to the development team and say: it's not stable enough, YOU will be on-call for it until you fix the shit you wrote.
These are two emblematic examples, but there are many more in the book.
Can you imagine any of the non-google (and non-faang) companies actually doing something like that?
Earlier quoted context omitted.
Moreover, they must be cargo cults. An SRE role was my first job at Google in 2006. It was a product of its time and place. Google could hire the sort of full stack programmer/sysadmin types that the article talks about because they had enough clout and desirability at that time that they could get very skilled programmers to give up programming and do sucky on-call work instead. Even then, just two years after IPO a…
Aren't there things we can do to simplify the software stack, so we're more likely to find candidates with all the required skills in one package? There are certainly fashions we can push back against, e.g. premature use of microservices, Kubernetes, and distributed systems in general.
"Can't we simplify the human bodies, so that we can crank out doctors after a two years bootcamp rather than 10+ years of med school and specialization?"
The answer is no.
To me, a SRE is *both* a sysadmin AND a programmer, developer, whatever you want to call it. It's a logical-and, not an XOR.
By sysadmin, I mean "runs a mean Unix box, including fixing things and diving deeply when they break", and by the programmer/whatever part of it, I mean "makes stuff come into existence that wasn't there before".
The skillset described in the above quotes are essentially the skills of an SWE and of a Sysadmin. So essentially you're doing two jobs for one salary.
There are people capable of doing two jobs, but you won't find them until you start advertising and paying actual-SRE salaries.
Earlier quoted context omitted.
I haven’t started trying to hire for this position yet, it’s been something I’ve been thinking of for a while though. Right a small number of developers, myself included, are on call and I’d like to reduce that a bit or just share the burden. My goal isn’t to throw crap over the fence and say “make it work” but rather empower someone to make maintaining and growing our platform their main goal. The developers (again,…
> Infra/Ops and programming are two different mindsets [...] HARD disagree on this. I've done both. Most of the really excellent programmers I've worked with have, at least a little. You can't write highly reliable networking software without a deep understanding of how networking actually works. You can't write highly performing software without a deep understanding of the infrastructure and hardware it's running on…
I have zero doubt that I could do an Ops job well, what I can’t do is switch between writing business logic and maintaining servers at the drop of a hat. Similar to how I can’t go from QA to engineer without a context switch/penalty.
As I said elsewhere in this thread if I could clone myself and do both roles I would in an instant. But if I have to pick one I’d pick writing code (as my primary thing), not saying that Ops doesn’t write code, just not the same type of code.