Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

41–50 of 178 posts

Re: Engineers Shouldn’t Write ETL

#41

Post by fizixer »

Data-driven decision-making to change the course of a business, is so internally disruptive it's unlikely to happen in an org-chart culture full of management layer.

Because that's what it is:

- It is attempting to question, critique, override, everyday decisions made by the management (including the CEO) based on available data.

- It is doing that with maximal knowledge of the whole organization. That means all the records, finances, secrets, what not, have to be divulged to the data-science team. (which in itself is an unsurmountable challenge, i.e., to convince the management to allow full data access; think emails, chat logs, meetings minutes of CEO's, VP's, etc, etc).

This will make the management go, "so let me get this straight, I authorize you access to data of the whole organization, and you come up with a conclusion (some of the times at least) that I'm full of it?"

I highly doubt any organization would be up for this kind of internal disruption, even if that means more success for the company.

Re: Engineers Shouldn’t Write ETL

#42

Post by yxhuvud »

throwawy31816 wrote: post #2

This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.

TLA is the common abbreviation for what you call TLI, that is Three Letter Abbreviation.

The common abbreviation for a four letter abbreviation is ETLA. That is, Extended Three Letter Abbreviation.

Re: Engineers Shouldn’t Write ETL

#43

Post by TeMPOraL »

throwawy31816 wrote: post #2

This author seriously needs to expand all of his TLIs (three-letter initialisms) the first time he uses them, as any writer worth his or her salt would do. There are those who may be interested in what he has to say, but can't follow because of assuming abbreviations.

100% agreed. I was clueless about ETL too. While the acronym itself may be old, people forget that this field is hip and sexy for maybe like last 3 years or so.

Re: Engineers Shouldn’t Write ETL

#44

Post by TeMPOraL »

Someone with big enough clout should utter a proper rule of thumb at some prestigious software conference. Something like "if your data can fit on a single commercially-available hard drive, it's not big data". Maybe then it has a chance to filter down to university education over the next decade or so.

(Corollary to that rule of thumb: if your data fits on a hard drive, all "big data" tools you need are shell scrips and SQLite.)

Re: Engineers Shouldn’t Write ETL

#45

Post by ajnin »

This is not very convincing. He starts off by saying that the "traditional" model where the data scientists do the thinking while the engineers do the doing is unsuccessful because the engineers need to get invested in other people's ideas, need to maintain them and get blamed if they fail while the data scientists get all the praise. So he suggests replacing this with a new model where the engineers work horizontally aka in the shadows, have to be "Tony Stark tailors" and get out of the way while the data scientists get to be Tony Stark. Which is basically the same thing.

Re: Engineers Shouldn’t Write ETL

#46

Post by iblaine »

I think the conclusion is this. Data Scientists, Data Engineers, and Infrastructure Engineers exist in their respective roles. Data Engineers should enable Data Scientists to be better engineers by creating frameworks for Data Scientists. By doing so, Data Scientists will be less likely to put stress on everyone else.

Another point I'd like to make is that not everyone hates ETL and pipeline management. I happen to like it. It's rewarding to stand up reliable self-healing data pipelines and ETLs.

Re: Engineers Shouldn’t Write ETL

#47

Post by gtrubetskoy »

I agree with the beginning of the article, which describes the present state pretty well, the part about "better engineers than statisticians and better statisticians than engineers", etc. But then I disagree with the rest.

The distinction between "Data Scientists" and "Engineers" is bogus, and the point about whether your data is "Big" is a red herring.

In reality, there should not be any distinctions between "scientists" and "engineers", you must strive to be both a "doer" and a "thinker". You can't think without doing, and can't do without thinking.

If you're in this field, and consider yourself an "engineer" but your math sucks, go read up on all you can about mathematics and statistics, just like you did back when you were learning about programming, operating systems and networking.

If you consider yourself a "data scientist" but don't know anything other than R and basic Python, go study programming and operating systems and networking, like you studied math at some point.

Somewhere on youtube I remember Dr. Donald Knuth (who is definitely an excellent programmer/engineer/computer scientist, arguably one of the best the world has known) saying that he considers himself primarily a mathematician.

Or, if you've read (or at least heard of) "the dragon book", you might find it interesting and inspiring that one of its main authors Dr. Jeffrey Ullman (whom I'd place in the same league as Knuth) went on to write another excellent (and available freely online, BTW) book "Mining of Massive Datasets", which IMHO is the one fundamental "big data" book out there.

So Data Scientists - go learn some programming languages like C and study UNIX and may be read "The Art of Computer Programming" and Engineers go read http://www.mmds.org/.

Then you'll all get along.

Re: Engineers Shouldn’t Write ETL

#48

Post by brown9-2 »

If you manage to hire them, they will be bored. If they are bored, they will leave you for Google, Facebook, LinkedIn, Twitter, … – places where their expertise is actually needed. If they are not bored, chances are they are pretty mediocre.

Granted that yes, lots of solutions don't exactly require a Hadoop cluster with thousands of nodes, this is a pretty gross and mean-spirited dig at "mediocre engineers" a number of times. It would be nice if we didn't treat people that don't work at Amazon/Google/Twitter/LinkedIn as lesser beings because they find their jobs at a probably-doesn't-have-Big-Data company.

(Does StitchFix have Big Data? If the answer is no, are their "Data platform engineers" mediocre?

Re: Engineers Shouldn’t Write ETL

#49

Post by v64 »

> The fundamental flaw that prevents the Thinker and Doer model from living up to its recruiting hype is the assumption that there exists an army of soulless non-mediocre Doer engineers who eagerly implement the ideas and vision of data scientists.

There's a large, active community of engineers who specialize in data, whose job is to technologically enable data scientists the means to perform their analyses. I know these people exist because I'm one of them, and I work with them, and I've met them at meetups and conferences. I don't know why the author doesn't think these types of engineers exist. Not all of us who code want to work with the web.

> If you read the recruiting propaganda of data science and algorithm development departments in the valley, you might be convinced that the relationship between data scientists and engineers is highly collaborative, organic, and creative. Just like peas and carrots.

Almost every data team I've worked with is structured this way. I work daily with data scientists. I have a data scientist sitting to my right, two data scientists sitting across from me. Our teams are highly integrated and I can't imagine it working any other way. If the teams the author is familiar with don't operate in this manner, then I can see why he'd think the endeavor is hopeless.

I also disagree with the author's conclusion. The data scientist's job is to analyze and interpret data. They should not be spending any time thinking about how to get that data. They should not be concerned about where the data is coming from. The more time scientists have to spend thinking about ETL, the less time they have to do what their training is in, statistical analysis.

Re: Engineers Shouldn’t Write ETL

#50

Post by nissimk »

Title should have been "Engineers Shouldn't only write ETL." I agree with the author's statement of the problem, but not with the proposed solution. Succinctly, I think the problem is compartmentalization and specialization. These are qualities that are sometimes promoted by management so that it is easier to maintain control over the organization and to hire people who won't require much training to do their jobs. Unfortunately, compartmentalization and specialization both lead to unhappiness in the workers, and are net negative for production. I believe the solution is fostering a wholistic approach among the specialists. Data scientists (who should be statisticians or machine learning experts) should interact regularly with software development engineers that have to productionize their research and they should also both interact regularly with systems and database administrators who make it all work in production. Rather than being separate teams working on parts of the same goal, they should all be one team. By working together through the poroblems faced in each area, they can learn more about each other's areas of expertise and will create a better solution faster. This isn't true just for data science, but throughout technology, where operational software developers should work together with product development, marketing, testing and operations to break down the divide and get all team members working towards the same goal.
Post reply on HN