Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

81–90 of 120 posts

Re: Engineers Shouldn’t Write ETL

#81

Post by v4n4d1s »

dgudkov wrote: post #58
technofiend wrote: post #56

It's not in my experience performant but Pentaho is definitely ETL-for-dummies easy to use. Similar to your average user pivoting data in Excel rather than learning Python or R, sometimes having a tool with suboptimal performance is better than optimizing an adhoc or short term process.

If you're looking for a real ETL-for-dummies, take a look at my EasyMorph ( https://easymorph.com ). We've made a number of simplifications that specifically target "dummy" users, e.g. columns may mix values of different types (text, numbers, etc.).

Thanks for developing easymorph! Free version helped me through my bachelors degree. It's my go-to tool to introduce people to ETL and similar concepts.

Re: Engineers Shouldn’t Write ETL

#82

Post by badpun »

Annatar wrote: post #74
RhodesianHunter wrote: post #21

Earlier quoted context omitted.

Completely agree with your take, and as an engineer in a similar role, the post rubbed me the wrong way. I don't find ETL work soul-sucking, and I certainly don't think that my colleagues or I are mediocre.

I was once accused by a Windows C++ programmer who just couldn't grok UNIX to be a "string cutter", but honestly, there is nothing so viscerally information technology as extract-transform-load. The irony of his accusation was that it was just one of many facets of things that I do and can/know how to do. ETL is really the core of IT by definition.

A lot of the hardcore C++ programmers look down on the whole IT area of programming (as opposed to systems, graphics, embedded etc.), maybe that was the reason.

Re: Engineers Shouldn’t Write ETL

#83

Post by erikb »

kartan wrote: post #43
bacon_waffle wrote: post #12

ETL means "Extract, Transform, Load" https://en.wikipedia.org/wiki/Extract,_transform,_load

Thank you. I think that is good practice to introduce abbreviations correctly. Even that it is easy to forget when you work with them all the time. "How do I introduce an abbreviation in the text? The first time you use an abbreviation in the text, present both the spelled-out version and the short form." https://blog.apastyle.org/apastyle/abbreviations/

Yep, especially when they are not general purpose like "PC" but from a very specific knowledge area. In this case it's even an old one. New engineers won't be educated to do ETL anymore.

Re: Engineers Shouldn’t Write ETL

#84

Post by craig_asp »

I've worked in BI (end-to-end - data modelling, reporting, ETL, etc.) for more than 10 years now across various organisations and since "data science" became all the rage, I had the pleasure to work with a few data scientists. From what I've seen so far, they are very good as statisticians (some of them university lecturers) but when it comes to building ETL pipelines, I don't think any of them could actually do it properly. Properly as in an ETL process which connects to various data sources, writes to logs, is repeatable, restartable and so on. It is not easy to get to know how to build a proper ETL process and it is not easy to learn how to "do data science" correctly as well. I see it as more productive (from my personal experience) to let the "data engineers" do the "data engineering" work - build data models, ETLs, etc. and let the "data scientists" do the "data science" work - build and fiddle with statistical models. Just like with a "full stack" developer, and the separation of work between "back end" and "front end" developers, it might be better to let each do what they do best unless you have people who can do both properly (but often it's hard to find them and they would actually be better in one area or the other). The frustration between the two camps - data "engineers" and "scientists" is usually due to mismanagement (distinct teams doing each bit separately, coordinated by one to many management layers) rather than suboptimal division and allocation of labour. Small teams of two to four people which contain the correct mix of experts would benefit from the strengths of both data professional types, and would avoid the problems around syncing the effort.

Re: Engineers Shouldn’t Write ETL

#85

Post by dalbasal »

internet555 wrote: post #16
whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

You mean the type of people that rule the world?

I'm joking (sort of) but this is half the job description for leadership jobs, politicians, executives and such.

The differences are of quality, not type. Jobs' thinkering was good, for the most part, for example.

Re: Engineers Shouldn’t Write ETL

#87

Post by didibus »

> We strive to lead the business with our output rather than to inform it

I think the business hires data scientist to be informed. Not to make business decisions on their behalf.

> Data scientists love working on problems that are vertically aligned with the business and make a big impact on the success of projects/organization through their efforts. They set out to optimize a certain thing or process or create something from scratch. These are point-oriented problems and their solutions tend to be as well. They usually involve a heavy mix of business logic, reimagining of how things are done, and a healthy dose of creativity

Again, I'm confused? That sounds like the data scientists should have majored in business then. If data scientists start doing that, what will all the other business folk do then?

Data scientists should just build out reports that provide valuable insights and potential patterns that can help make business decisions. The difference with prior reports engineer or data analysts or wtv, is that a data scientist is assumed to be able to generate statistical analysis or/and pattern analysis over the data. While prior, a data analyst only needed to perform basic versions of that which did not go beyond what SQL could do.

The data engineer should enable the data scientist to perform this analysis by both working with the software engineers to acquire it safely, securely, reliably and at scale. And working witj the data scientist in order to apply his statistical analysis efficiently and at scale to a possibly very large data set. Finally, he might need to work with both software engineer and data scientist to setup real time or close to real time versions of the analysis.

All result from the analysis should be presented (aka reported) to the business. The data scientist can suggest interpretations or ideas to address findings, but it's the business role to make tactical and strategic decisions about business processes and products.

And if you're doing ML as part of a process, then you need a ML scientists. Say you need to build out voice recognition, or the likes. Basically comp sci or math majors with ML masters or PHDs.

Re: Engineers Shouldn’t Write ETL

#90

Post by ghc »

It’s a bad situation in your typical enterprise, but it’s even worse where I’ve spent my career: working with realtime industrial data. I became convinced that building time series data pipipelines was a bad idea after many late nights in the office fixing fragile systems that couldn’t handle real-world complexity.

As fun as it is to build with and learn new technologies, it’s a bad idea to build data pipelines unless you have a lot of resources and good leadership that can make peace between all the different people who touch the data.

Unfortunately in the world of sensors and equipment there aren’t many solutions, so I started a company (at https://sentenai.com ) to save others from my years of struggle. It turns out it’s even harder to build a general time series data pipeline solution, but we’re making progress.

Post reply on HN