Live data from Hacker News

Etsy CTO: We Need Software Engineers, Not Developers

thenewstack.io

101–110 of 160 posts

Re: Etsy CTO: We Need Software Engineers, Not Developers

#101
One of the best pieces of sleight of hand that modern companies are pulling on modern engineering job seekers is the complete demise of the job description.

Job descriptions act like a constraint on the human complexity of a team or organization. They help codify the way people conceive of modularizing known workflows and discretizing for the purposes of having employees solve them.

Crucially, a job description defines the boundaries of what an employee can say no to -- "that's not in my job description."

In the modern hiring landscape, this would be immediately met with vapid criticisms, like you are not a team player or you won't "wear many hats" or whatever.

But the benefit of job description boundaries is not related to entitling programmers to be whiny and complacent about tasks they don't like. That's naive. The reason it's good to empower your workers to tell you no, especially knowledge workers, is because it preserves organization structure and planning. It highlights bottlenecks for you and points out the discrepancy between what you need and what you have. Otherwise, while your machine learning expert is repressing her fury over being assigned to clean up some legacy Rails codebase, you might be off playing golf thinking you've got your Rails maintenance needs correctly covered, when really you don't at all (and you're leaving money on the table by not extract the full value from your machine learning engineer that you otherwise could).

This, by the way, is perhaps the biggest hallmark of a good manager. Good managers will act like double-sided adapters who process arbitrary inputs from the business-problem-stream on one side, and turn them into collaborations with subordinates that fully respect those subordinates' specialities, aptitudes, and goals. That is a damn hard job, and when done properly it is big justification for the increased compensation and status usually given to management-level employees. Unfortunately, many modern firms do not expect managers to do this, and instead they seek to build teams of so-called "full-stack" developers, for everything, so that managers are only receiving business problem inputs and never doing the actually difficult part of turning them into workflows that show respect for subordinates (... you know, managing them).

In fact, I would argue that this is exactly what it means to even have a business model at all. If your "business model" involves hiring people and then screaming at them to do whatever they are told, instead of what they are good at or motivated to do, because "that's just how jobs work" then in effect you actually don't have a business model. You have not yet done the hard part -- figuring out that double-sided adapter layer that translates real world business concerns into things that disparate specialists want to do and won't refuse to do.

When I read this by the Etsy CTO, it strikes me as a big, whiny excuse to try to hire more people who are "arbitrary work receptacles." That seems to be what he is trying to describe as "engineering" -- He uses words like multi-disciplinary and messy, when really he's trying to convey "I want you to do whatever arbitrary work I say, regardless of how poorly it matches your aptitudes, interests, or goals."

I feel developers should more strongly demand legitimate job descriptions, and ask during the hiring process for specifics on the ways that managers will be held accountable for protecting the job description, instead of forcing employees to bottomlessly compromise on their goals in order to be arbitrary work receptacles.

It's not about mere displeasure over having to do arbitrary tasks. It's about the fact that a lack of planning or modularity codified through job descriptions is bad for business. Properly respecting employee aptitudes is a sign of a healthy business. Demanding arbitrary cross-functionality all the time is a sign of needless chaos. We can see this clearly in software itself: if all of your classes make everything public, silently mutate each other's state, and everything grows purely by attrition with no attention paid to e.g. Single Responsibility Principle, it's a huge red flag of bad code and impending system failure. But we bury our heads in the sand when the same ideas become apparent when managing people complexity instead of software complexity.

Re: Etsy CTO: We Need Software Engineers, Not Developers

#102
post #35

Earlier quoted context omitted.

> I was looking for someone who could build something as good or better than HDFS from nothing if they had to. Well, for 90 percent of job offers, what a company claims to need is not what it actually needs (i.e. the typical "10 years experience with 5 year old technology" bullshit). If your company is not a very unique snowflake or in an academic setting, believing that it makes zero economic sense to completely rei…

Your characterization of the typical job marketplace in tech jives pretty closely with my experience. I've often said, in public speaking forums no less, that to me, "The overwhelming majority of the 'Big Data' marketplace seems predicated on selling to the hubris that software engineers want to believe they have problems bigger than they actually are." Despite being a denizen of the whole "NoSQL" thing personally. T…

I upvoted you for the quality of your prose. Congrats!

Re: Etsy CTO: We Need Software Engineers, Not Developers

#103
post #77

Joel Spolsky wrote an essay on "leaky abstractions," and from reading the interview with the Etsy CTO, the two seem related. Allspaw is making the distinction between engineers and developers this way. To him, an engineer is willing and able to get under the hood, if necessary; whereas a developer is limited to using libraries and frameworks as-is, and filing bug reports on them when things don't go as planned, or fe…

As someone who likes to "get under the hood", I'm used to grumbling about developers these days who just glue together existing frameworks. But the real cause of my dissatisfaction is that I find it harder to glue together existing frameworks -- I don't know where to begin, until I've dug in a bit to find out how something works underneath. Meanwhile other people whose approaches I think of as simplistic are able to…

I think that's part of it.

I'm the kind of person you'd find frustrating at first. I don't want to know all the internals of what I'm doing before I dive in. I want a nice API and good docs, and I want to build some stuff right away. And if I can't do that, I give up and move on to something else.

If I like it, and it gives me quick wins, then I'll dive into the guts of how it works. Because I'm a responsible developer.

But I have learned that with all the things out there to learn and absorb, I don't have time to deep-dive a framework before choosing to use it. There's something else coming that I have to do next week that uses something I don't know.

So the more I build stuff, the faster I get at picking it up.

Re: Etsy CTO: We Need Software Engineers, Not Developers

#104

I am sympathetic to the distinction. I had a bunch of applicants for a position claiming to be "Distributed Systems Engineers" mostly because they'd stood-up, maintained, and or used a Hadoop or Spark cluster in their current or prior job. But to me that was more of an advanced (one hopes) end-user. Someone who could take a bunch of large, mostly-complete logical components that somebody else engineered and then use…

What I was looking for wasn't somebody who knew how to use something like HDFS. I was looking for someone who could build something as good or better than HDFS from nothing if they had to.

Did you try articulating that distinction in your job ads?

Or if not, can you really blame people for applying when your ads read like 99% of help wanted ads in the fin-de-boom era; like, you know:

We've got the coolest office in the Mission, with a climbing wall and jamming room, we do beer bongs every Thursday and play lawn bowling together on weekends! And of course you can bring your dog in everyday, too! Keywords: Node, Python, D3, Spark, Hadoop, HDFS

Re: Etsy CTO: We Need Software Engineers, Not Developers

#105

High-assurance software and security field have been doing real engineering for decades. Here's a nice presentation with a tie-in to another high-assurance discipline: http://web.cecs.pdx.edu/~hook/cs491sp08/AssuranceSp08.ppt Here's an old method, Cleanroom, that was cost-effective for business with low, defect rate even on first use: http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=E10... . Altran[-Praxis] i…

About Cleanroom, the prohibition on unit testing is insane; CPU and compiler bug exists, so do subtleties in languages, and even good quality models in formalized forms have some defects that probably prevent to get a mathematical proof of correctness even if the CPU/Compiler would be defect free (ex: C++11 memory model - and I think the formal Java memory model also has some issues). They should do as if they are forbidden to do unit testing, but then once they are "sure" all their code is correct do it anyway. Having a separate team do some level of testing is fine, but there is value in also having some done by authors, because conveying all the intents and caveats and things taken into account when writing it but also things NOT done and the reason for that, about a piece of source code, even very small, is incredibly hard.

Re: Etsy CTO: We Need Software Engineers, Not Developers

#106
post #99
post #92

Earlier quoted context omitted.

> I think a lot of engineers are motivated by other things. Sure, but it doesn't mean good engineers don't know what they are worth. If Etsy has hard time recruiting "engineers", Etsy should ask themselves what kind of deal they are offering in order to attract the profile they seek and not write a blog that dismiss "developers" as inferior. Because it reads like "if you're a developer stop applying", well if Etsy wa…

The "best of the best" have frequently made enough money to stop working but they keep working, that's because they're not working for money.

> The "best of the best" have frequently made enough money to stop working but they keep working, that's because they're not working for money.

Who's going to work at Etsy for a pittance when other companies offer them a better deal? If Etsy has hard time recruiting "engineers" then problem is at Etsy. They should ask themselves why they have hard time recruiting the profile they seek instead of blaming "developers".

Re: Etsy CTO: We Need Software Engineers, Not Developers

#107
post #72

Earlier quoted context omitted.

I've literally seen the opposite interviewing as distributed computing person, most people ask for a distributed engineer and then want someone to stitch together Hadoop and Spark or how to effectively leverage HDFS. When you start talking about concurrency issues, Lamport clocks, consensus algorithms, etc. their eyes glaze over and they ask how to efficiently rotate an array. Clearly something is broken but I'm not…

This a thousand times. For many firms, the roles of "data scientist" "data engineer" "distributed system engineer" and "platform engineer" are all fully synonymous, and all of them really mean "Hadoop* babysitter with a dash of full-stack whenever we arbitrarily feel like asking you to do other stuff too." It's depressing. * or substitute whatever other enterprise framework you want

all of them really mean "Hadoop' babysitter with a dash of full-stack

...because that's all that most business require, 99% of the time. To you know, get things done and make money and stuff.

Which may not fit your needs, but why be "depressed" about it? It's just the way things are in the commercial world.

If you want someone with more fine-grained stills, try articulating that in your job postings. What we see, all the time, are ads mentioning platform X, with no articulation whatsoever as to where, even on some approximate logarithmic skill, they'd like the skill level and comfort with platform X to be.

Re: Etsy CTO: We Need Software Engineers, Not Developers

#108
post #72

Earlier quoted context omitted.

I've literally seen the opposite interviewing as distributed computing person, most people ask for a distributed engineer and then want someone to stitch together Hadoop and Spark or how to effectively leverage HDFS. When you start talking about concurrency issues, Lamport clocks, consensus algorithms, etc. their eyes glaze over and they ask how to efficiently rotate an array. Clearly something is broken but I'm not…

This a thousand times. For many firms, the roles of "data scientist" "data engineer" "distributed system engineer" and "platform engineer" are all fully synonymous, and all of them really mean "Hadoop* babysitter with a dash of full-stack whenever we arbitrarily feel like asking you to do other stuff too." It's depressing. * or substitute whatever other enterprise framework you want

Do you think a decent full stack engineer would have a problem picking up "proper" data science on the job? Both are fairly intelligent jobs with a fair bit of overlap.

Re: Etsy CTO: We Need Software Engineers, Not Developers

#109

Joel Spolsky wrote an essay on "leaky abstractions," and from reading the interview with the Etsy CTO, the two seem related. Allspaw is making the distinction between engineers and developers this way. To him, an engineer is willing and able to get under the hood, if necessary; whereas a developer is limited to using libraries and frameworks as-is, and filing bug reports on them when things don't go as planned, or fe…

I've always felt that in the various jobs I've held the degree to which I could "get under the hood" was determined by people higher up the chain of command than me. Run into a complex bug in library X? Well, we don't want you spending a week to debug it, write a patch and try to upstream it. Work around it for now instead. Or we'll defer that feature since we don't want you to spend that much time on "non-company" p…

I can relate to your last paragraph. I used to (and still) source dive to find good workarounds to library issues. But I never asked for permission, I just did it. I also delivered on time.

Nowadays, I'm a team lead and I am guilty of telling people "not to go there" (about half the time). It's funny because it actually conflicts with my opinion that I want people to dig in! The choice to "dig in" is a personal risk/reward. It's a risk that an engineer must take while practicing good time management. Asking your manager is akin to making them take that risk for you (the risk of wasted time, passing deadlines, etc).

My crappy advice is... ask for forgiveness, not for permission. If you're a good engineer, you'll come out on top!

Re: Etsy CTO: We Need Software Engineers, Not Developers

#110
post #72

Earlier quoted context omitted.

This a thousand times. For many firms, the roles of "data scientist" "data engineer" "distributed system engineer" and "platform engineer" are all fully synonymous, and all of them really mean "Hadoop* babysitter with a dash of full-stack whenever we arbitrarily feel like asking you to do other stuff too." It's depressing. * or substitute whatever other enterprise framework you want

all of them really mean "Hadoop' babysitter with a dash of full-stack ...because that's all that most business require, 99% of the time. To you know, get things done and make money and stuff. Which may not fit your needs, but why be "depressed" about it? It's just the way things are in the commercial world. If you want someone with more fine-grained stills, try articulating that in your job postings. What we see, all…

That's rarely why these enterprise frameworks are bought or operated. More often it's for various permutations of the "no one got fired for buying IBM" excuse. Big showy re-orgs around Hadoop are mostly for status effects, hardly ever related to engineering realities.

And in the few firms that actually do have real engineering trade-offs that favor the use of those types of frameworks, they tend to hire people who are well-suited for the role, and then create job functions surrounding them that are respectful of aptitudes and skills of the people they hire.

In most firms that adopt these frameworks (for status effects), they are just desperate to fill seats and increase engineering headcount. They don't respect your skill set or even care if it matches the business need. They just need to get you in the door, and then find a way to deal with inevitable dissatisfaction later.

Post reply on HN