Engineers Shouldn’t Write ETL • Hacker News Board

Re: Engineers Shouldn’t Write ETL

#41

Post by emeraldd » Mon, Nov 05, 2018, 2:18 AM UTC

DoreenMichele wrote: post #32

internet555 wrote: post #16

Earlier quoted context omitted.

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

You see that a lot with 2e people -- bright people with some weaknesses or a disability. That could account for how negatively you have experienced this. Many 2e people have never really been taught good ways to handle the combination of big strengths and big weaknesses. I serve as a sounding board a lot for my oldest son and that works well, but it's not uncommon for such people to just be trying to meet their own n…

What does "2e people" mean in this context ... That's not a term I've run across before?

Re: Engineers Shouldn’t Write ETL

#42

Post by DoreenMichele » Mon, Nov 05, 2018, 2:19 AM UTC

emeraldd wrote: post #41

DoreenMichele wrote: post #32

Earlier quoted context omitted.

You see that a lot with 2e people -- bright people with some weaknesses or a disability. That could account for how negatively you have experienced this. Many 2e people have never really been taught good ways to handle the combination of big strengths and big weaknesses. I serve as a sounding board a lot for my oldest son and that works well, but it's not uncommon for such people to just be trying to meet their own n…

What does "2e people" mean in this context ... That's not a term I've run across before?

Twice exceptional.

https://en.m.wikipedia.org/wiki/Twice_exceptional

Re: Engineers Shouldn’t Write ETL

#43

Post by kartan » Mon, Nov 05, 2018, 2:29 AM UTC

bacon_waffle wrote: post #12

ETL means "Extract, Transform, Load" https://en.wikipedia.org/wiki/Extract,_transform,_load

Thank you. I think that is good practice to introduce abbreviations correctly. Even that it is easy to forget when you work with them all the time.

"How do I introduce an abbreviation in the text? The first time you use an abbreviation in the text, present both the spelled-out version and the short form." https://blog.apastyle.org/apastyle/abbreviations/

Re: Engineers Shouldn’t Write ETL

#44

Post by NovaX » Mon, Nov 05, 2018, 2:32 AM UTC

fraserharris wrote: post #17

msencenb wrote: post #11

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

I would suggest checking out my employer, Fivetran (fivetran.com). Many tech firms use us to centralize their data for analysis. We have startup pricing for sub-50 employees.

Do you know when DB/2 support will be added (Windows, AS/400)? I've been wanting to switch over from AWS DMS and our in-house support.

Re: Engineers Shouldn’t Write ETL

#45

Post by m_ke » Mon, Nov 05, 2018, 2:40 AM UTC

humbleMouse wrote: post #38

amarshall wrote: post #9

> There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. Instead, give people end-to-end ownership of the work they produce (autonomy). I think this is more the point than “engineers shouldn’t write ETL”: the engineering-related department consuming the ETL’s output should likely be the ones writing/maintaining it. Or,…

This quote is ridiculous. Some people enjoy plumbing high speed reliable/transparent data pipes. I could care less what goes thru the pipes I make.

The "T" in ETL is important here. You might not care but the end users of that data should. It's very easy to take raw data and strip it of a lot of useful information by transforming and normalizing it.

The person doing modeling or data analysis should ideally be dealing with the raw data, know how it was collected and understand what each field really means.

Re: Engineers Shouldn’t Write ETL

#46

Post by closeparen » Mon, Nov 05, 2018, 2:42 AM UTC

The author completely lost me. Analysts produce reports. Data scientists produce models. We don’t ask a data scientist to produce a model unless we have a serious intention to put it in production. There are significant engineering challenges in taking a model from the data scientist’s batch-mode workbooks and Hadoop queries to a reliable near-real-time online service, and the relationship can get dysfunctional, but it has nothing to do with data scientists being BI in disguise.

Re: Engineers Shouldn’t Write ETL

#47

Post by polm23 » Mon, Nov 05, 2018, 2:46 AM UTC

Previously.

https://news.ycombinator.com/item?id=11312243

Re: Engineers Shouldn’t Write ETL

#48

Post by drunkpotato » Mon, Nov 05, 2018, 2:46 AM UTC

msencenb wrote: post #11

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

At the company I work for, we have integrations with Stitch Data and Fivetran. Both are good and have been responsive to my needs. Neither have been perfect, so when I've noticed a problem I've had to keep on top of them to fix it. I also maintain a few of our own ETL jobs for data sources that aren't supported. I will say that I recommend using an ETL vendor without reservation. The nominal cost is more than made up for in the headaches you'll save yourself in creating and maintaining a homegrown ETL.

Re: Engineers Shouldn’t Write ETL

#49

Post by protomyth » Mon, Nov 05, 2018, 2:47 AM UTC

bigger_cheese wrote: post #34

protomyth wrote: post #24

Earlier quoted context omitted.

Ok, I think I'm getting the specifics of this situation. So, we are talking about internal reports, not something that could actually get in the external customer's hands.

Yes this is internal stuff. I work at a large industrial manufacturing plant. Reports that go externally are done by certified people. (Laboratory technicians for product specifications and finance analysts for stock market stuff).

I’ve done external reports for clinical trials and agriculture, and I guess they weren’t as up on getting certifications. Thanks for the very detailed replies.

Re: Engineers Shouldn’t Write ETL

#50

Post by jpulec » Mon, Nov 05, 2018, 2:52 AM UTC

msencenb wrote: post #11

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

I've used StitchData at a startup with AWS Redshift. Pair it with something like dbt for transforming your data, and you have a great match. A little pricey, but totally worth it, IMO.