Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

71–80 of 120 posts

Re: Engineers Shouldn’t Write ETL

#71

Post by pishpash »

closeparen wrote: post #51
humbleMouse wrote: post #38

Earlier quoted context omitted.

This quote is ridiculous. Some people enjoy plumbing high speed reliable/transparent data pipes. I could care less what goes thru the pipes I make.

The unglamorous ETL work is the config and query writing to apply infrastructure building blocks to particular pairs of tables, not the creation of the generic infrastructure.

Exactly. They are talking about manual, custom, one-use ETL that needs to be maintained forever. Don't nobody got time for that.

On the other hand, sometimes you can't get away from that because different orgs/humans generate trash data in idiosyncratic forms. Things will get much better once we pry all the human hands off of data and let engineers redesign all of them across the world. Not going to happen soon.

Re: Engineers Shouldn’t Write ETL

#72

Post by Annatar »

I see this every day in my job. He so nailed the problems. Data scientists must be made responsible and accountable end-to-end for their solutions. And they must be grilled on operational deployability and maintanability before, during and after deployment. They have to become accountable.

Re: Engineers Shouldn’t Write ETL

#73

Post by Annatar »

yeukhon wrote: post #55
matwood wrote: post #26

Earlier quoted context omitted.

Talking is easy, executing is hard. Executing requires discipline which so many people seem to lack. One of the reasons I love making people write down their idea (myself included) before talking about it is that writing forces a small initial execution step. Even a step this small can often filter the useless ideas away.

This is where design doc should be required. At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week. At first I thought this was a step back because I felt it was a water fall but after writing my own design doc I quickly realized “talking is really fucking cheap.” Sitting down and write a doc that considers as m…

"At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week."

Wow, true system engineering! Your company sounds like a good place to do some professional work.

Re: Engineers Shouldn’t Write ETL

#74

Post by Annatar »

RhodesianHunter wrote: post #21
whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

Completely agree with your take, and as an engineer in a similar role, the post rubbed me the wrong way. I don't find ETL work soul-sucking, and I certainly don't think that my colleagues or I are mediocre.

I was once accused by a Windows C++ programmer who just couldn't grok UNIX to be a "string cutter", but honestly, there is nothing so viscerally information technology as extract-transform-load. The irony of his accusation was that it was just one of many facets of things that I do and can/know how to do. ETL is really the core of IT by definition.

Re: Engineers Shouldn’t Write ETL

#75

Post by Annatar »

sagichmal wrote: post #68
jerglingu wrote: post #20

Earlier quoted context omitted.

I'm wondering if anybody lives in the hypothetical perfect world scenario the author writes about. I'm at one of the larger tech companies and it's inconceivable that something like this could exist (though the churn here is extremely high - a mature shop with longstanding membership could implement the hypothetical in some form). Everything sounds nice when dreaming it up in one's head, but discounts the reality of…

I think you may be experiencing a local minimum, so to speak. Data scientist programmers who "simply refuse" to think about basic productionization criteria like SLAs shouldn't remain employed.

They shouldn't, but they do because understanding a little bit of mathematics is at a premium these days and it's very much in vogue to crap all over computer science majors even though we had the same kind of mathematics curriculum as they at the university.

Re: Engineers Shouldn’t Write ETL

#76

Post by carlmr »

whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

Completely agree, especially as people that stop "doing" also often forget how to "think" about the actual problems properly.

Re: Engineers Shouldn’t Write ETL

#77

Post by fraserharris »

NovaX wrote: post #44
fraserharris wrote: post #17

Earlier quoted context omitted.

I would suggest checking out my employer, Fivetran (fivetran.com). Many tech firms use us to centralize their data for analysis. We have startup pricing for sub-50 employees.

Do you know when DB/2 support will be added (Windows, AS/400)? I've been wanting to switch over from AWS DMS and our in-house support.

We are planning on adding support for DB/2 in Q1/Q2 2019. We are getting SAP HANA out first

Re: Engineers Shouldn’t Write ETL

#78

Post by Roritharr »

My father in law is such a "thinker". He has been since the 70s and worked on all kinds of projects from IBM Mainframes for up to Hadoop and Kafka for Insurance Companies and Telkos.

It's ridiculous to me how hard it is for him to find a new job at 60. He financially doesn't have to, but he wants to train younger guys on how to deal with all the weirdness one encounters in ETL Jobs.

Re: Engineers Shouldn’t Write ETL

#79

Post by dagw »

I often really enjoy it when I get a chance to do ETL work. The 'T' in ETL can many times involve some pretty fun and creative challenges. And even in the general case there is something really satisfying about putting together a clever and well constructed ETL pipeline.

Re: Engineers Shouldn’t Write ETL

#80

Post by lordnacho »

internet555 wrote: post #16
whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

> What’s funny to me is how many incompetent “thinkers” appear in meetings.

My thought as well. I also worked in hedge funds for a long time, and I kept getting resistance from the self-proclaimed "thinkers" to do even the most basic project management things like keeping a shared list of bugs, using version control, etc. It became clear that they simply didn't know how to use these tools, while claiming to specialize in financial modelling.

This also turned out to be false, as moving on to other funds I discovered their way of seeing things was quite limited. Which I had a good suspicion of, but other people actually showed me how one could approach things.

Part of that reckoning was that to be good at building financial strategies -something virtually nobody will tell you about- you need to be fairly good with writing code. Not just your Frankenstein of VBA, Excel, and Matlab, but including a fairly deep understanding of algorithms as well as common DevOps tools.

Post reply on HN