Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

51–60 of 120 posts

Re: Engineers Shouldn’t Write ETL

#51

Post by closeparen »

humbleMouse wrote: post #38
amarshall wrote: post #9

> There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. Instead, give people end-to-end ownership of the work they produce (autonomy). I think this is more the point than “engineers shouldn’t write ETL”: the engineering-related department consuming the ETL’s output should likely be the ones writing/maintaining it. Or,…

This quote is ridiculous. Some people enjoy plumbing high speed reliable/transparent data pipes. I could care less what goes thru the pipes I make.

The unglamorous ETL work is the config and query writing to apply infrastructure building blocks to particular pairs of tables, not the creation of the generic infrastructure.

Re: Engineers Shouldn’t Write ETL

#52

Post by closeparen »

haney wrote: post #35
meritt wrote: post #10

Earlier quoted context omitted.

I'm curious what innovative tools have emerged in the past two years that changed the dynamic?

Not sure what the author was referring to but, Airflow has gotten better / gained wider adoption during that time and my team started using DBT which saved a bunch time and was new during that period.

Airflow makes the distinction wider, as creating data pipelines requires even less software engineering.

Re: Engineers Shouldn’t Write ETL

#54

Post by jimbokun »

I think this diagnoses the problem well, but ignores an obvious solution.

A team of one data scientist and one engineer, completely responsible for building a model, and seeing it through into production, meeting all applicable SLAs and performance metrics.

Or maybe it's two data scientists and one engineer, or one scientist and two engineers, whatever is required.

The point is to have a small team you can hold completely accountable for their output. They sink or swim together, so there is no debating whether the scientists or engineers get the credit or take the blame. They are assessed by the effectiveness of the end product they produce.

Re: Engineers Shouldn’t Write ETL

#55

Post by yeukhon »

matwood wrote: post #26
internet555 wrote: post #16

Earlier quoted context omitted.

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

Talking is easy, executing is hard. Executing requires discipline which so many people seem to lack. One of the reasons I love making people write down their idea (myself included) before talking about it is that writing forces a small initial execution step. Even a step this small can often filter the useless ideas away.

This is where design doc should be required. At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week. At first I thought this was a step back because I felt it was a water fall but after writing my own design doc I quickly realized “talking is really fucking cheap.” Sitting down and write a doc that considers as many corner cases and implementation really produce high quality work. It’s all about discipline indeed.

Re: Engineers Shouldn’t Write ETL

#56

Post by technofiend »

It's not in my experience performant but Pentaho is definitely ETL-for-dummies easy to use. Similar to your average user pivoting data in Excel rather than learning Python or R, sometimes having a tool with suboptimal performance is better than optimizing an adhoc or short term process.

Re: Engineers Shouldn’t Write ETL

#57

Post by raihansaputra »

technofiend wrote: post #56

It's not in my experience performant but Pentaho is definitely ETL-for-dummies easy to use. Similar to your average user pivoting data in Excel rather than learning Python or R, sometimes having a tool with suboptimal performance is better than optimizing an adhoc or short term process.

YES. I just knew about Pentaho recently and it’s amazing. Sad that they just scrubbed off info about the free community version on their webpage, and to automate the jobs on the community version you need to do cron/Task Scheduler stuff outside of the app. I know it’s a limitation to make people jump ship to the paid version, but I just hope it’s integrated so I don’t have to think about setting up cron jobs to have automated ETLs and just have people responsible to create the jobs do the scheduling too.

Re: Engineers Shouldn’t Write ETL

#58

Post by dgudkov »

technofiend wrote: post #56

It's not in my experience performant but Pentaho is definitely ETL-for-dummies easy to use. Similar to your average user pivoting data in Excel rather than learning Python or R, sometimes having a tool with suboptimal performance is better than optimizing an adhoc or short term process.

If you're looking for a real ETL-for-dummies, take a look at my EasyMorph (https://easymorph.com). We've made a number of simplifications that specifically target "dummy" users, e.g. columns may mix values of different types (text, numbers, etc.).

Re: Engineers Shouldn’t Write ETL

#59

Post by spc476 »

kartan wrote: post #43
bacon_waffle wrote: post #12

ETL means "Extract, Transform, Load" https://en.wikipedia.org/wiki/Extract,_transform,_load

Thank you. I think that is good practice to introduce abbreviations correctly. Even that it is easy to forget when you work with them all the time. "How do I introduce an abbreviation in the text? The first time you use an abbreviation in the text, present both the spelled-out version and the short form." https://blog.apastyle.org/apastyle/abbreviations/

Or, you know, actually use HTML with the tag ...
Post reply on HN