Article from 2016
Engineers Shouldn’t Write ETL
61–70 of 120 posts
Re: Engineers Shouldn’t Write ETL
#62I think this diagnoses the problem well, but ignores an obvious solution. A team of one data scientist and one engineer, completely responsible for building a model, and seeing it through into production, meeting all applicable SLAs and performance metrics. Or maybe it's two data scientists and one engineer, or one scientist and two engineers, whatever is required. The point is to have a small team you can hold compl…
Small teams are awesome in so many scenarios. I recently wrapped up a 4-week proof of concept for a client on knowledge management and discovery using NLP.
I was able to work with someone apt at machine learning while I focused on building out the UI and backend. We delivered a first release about 3 days after we started, giving ample time to seek feedback and let the users shape the direction.
Re: Engineers Shouldn’t Write ETL
#63I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…
Completely agree with your take, and as an engineer in a similar role, the post rubbed me the wrong way. I don't find ETL work soul-sucking, and I certainly don't think that my colleagues or I are mediocre.
“ETL” also involves data modeling. This is a thinking role beyond many people’s understanding. Quite often the “Thinkers” demand data but have very little understanding how to structure it when you get temporal aspects (slow changing dimensions etc), new types of data (rules for types) etc. These and more are quite often left/relegated to ETL guys as housekeeping. Except it isn’t. It’s architecting the house.
Re: Engineers Shouldn’t Write ETL
#64As an undergraduate who is about to graduate with a degree in "Data Science" this post encapsulates a lot of my worries as I move into the work world. Should I focus on being a "thinker" a "doer" or a "plumber"? For the first three years I was planning on being a CS major until I was denied from the department: now the data science major is my only hope to graduate. I feel as though my programming skills are solid: but not good enough to be on any sort of fast paced infrastructure/devops team. On the flipside: I feel as though I am so far behind on stats/math knowledge that it's pointless to try and become a data scientist/analyst. I've thought about data engineering (the 'doer') as a happy compromise between the two. However there are barely any intern or entry level data engineering positions that I can find. The ones I do find require knowledge of so many frameworks that I don't know where to start. Additionally, I'm not even sure if data engineering even is a happy compromise, especially after reading the post. Time is ticking, and sooner or later I'm going to have to figure out what route to take, and how I want to specialize. I go to a hyper competitive university in a hyper competitive region of the country and I'm starting to feel like I'm falling behind and getting lost.
If any of you older/more experienced engineers and scientist have advice or wisdom for me, I would very much appreciate it.
Re: Engineers Shouldn’t Write ETL
#65As an undergraduate who is about to graduate with a degree in "Data Science" this post encapsulates a lot of my worries as I move into the work world. Should I focus on being a "thinker" a "doer" or a "plumber"? For the first three years I was planning on being a CS major until I was denied from the department: now the data science major is my only hope to graduate. I feel as though my programming skills are solid: b…
A bit OT, but as a more experienced engineer who dropped out of school to start a company, I'm curious: why weren't you able to get into your school's CS program?
Don't worry too much about "falling behind". There will always be time to learn more math or a new framework. Worry more about finding that first job, any job, then you can branch out once inside the industry. Networking beats recruiters beats sending a resume, so try to find a friend who already works where you want to be.
Re: Engineers Shouldn’t Write ETL
#66Can't this whole thing be boiled down to "DevOps for Data Science/Engineering"?
Different parts of the org with different skillsets and cultures practicing empathy for each other by communicating interests in version-controlled code, allowing for guard-railed autonomy, which leads to business agility.
Yep. Sounds about right.
> Optimize for autonomy not efficiency
Optimizing for efficiency without considering the cost of work in progress (WIP) (irrelevant ETL models), rework (unscalable models), or unplanned work (unscalable models that make it to production) results in company silos (data engineering, infrastructure engineering) cheering local maxima while covering their ass in the face of a business that's suffering from a long lead time. Two teams with two backlogs will accomplish work exponentially faster compared to three teams with three backlogs.
It boggles my mind how books like The Phoenix Project are not required reading.
Re: Engineers Shouldn’t Write ETL
#67As an undergraduate who is about to graduate with a degree in "Data Science" this post encapsulates a lot of my worries as I move into the work world. Should I focus on being a "thinker" a "doer" or a "plumber"? For the first three years I was planning on being a CS major until I was denied from the department: now the data science major is my only hope to graduate. I feel as though my programming skills are solid: b…
Focus on the one you enjoy most, graduate, then go get some work experience.
Re: Engineers Shouldn’t Write ETL
#68I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…
I'm wondering if anybody lives in the hypothetical perfect world scenario the author writes about. I'm at one of the larger tech companies and it's inconceivable that something like this could exist (though the churn here is extremely high - a mature shop with longstanding membership could implement the hypothetical in some form). Everything sounds nice when dreaming it up in one's head, but discounts the reality of…
I think you may be experiencing a local minimum, so to speak. Data scientist programmers who "simply refuse" to think about basic productionization criteria like SLAs shouldn't remain employed.
Re: Engineers Shouldn’t Write ETL
#69Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.
I would highly highly recommend ETL as service, after adopting it recently. It substantially changes your relationship with your data sources in a really positive way. And frankly, ETL for common data sources is code that you just don't need to write. I would say that you should pilot with a few ETL vendors. We currently use Fivetran, they're fine but we've had enough burps that I cannot cold recommend them over othe…
I'm Fivetran's CEO and I just want you to know, whatever "burps" you experienced, these things keep me up at night and the whole team is always striving to make the pipeline "just work". The whole vision of our product is that you should be able to plug in and get a perfect mirror image of all your data sources in your data warehouse. Anytime we fall short of that it drives us crazy.
Re: Engineers Shouldn’t Write ETL
#70As an undergraduate who is about to graduate with a degree in "Data Science" this post encapsulates a lot of my worries as I move into the work world. Should I focus on being a "thinker" a "doer" or a "plumber"? For the first three years I was planning on being a CS major until I was denied from the department: now the data science major is my only hope to graduate. I feel as though my programming skills are solid: b…
A bit OT, but as a more experienced engineer who dropped out of school to start a company, I'm curious: why weren't you able to get into your school's CS program? Don't worry too much about "falling behind". There will always be time to learn more math or a new framework. Worry more about finding that first job, any job, then you can branch out once inside the industry. Networking beats recruiters beats sending a res…
I did poorly on a math class that was required to declare the major. It's ironic since now that I'm in the data science major, I have to do even more math classes and less programming classes.
I would love to do my own startup. I have a few ideas floating around. But I feel like I lack the discipline to sit down every day and force myself to work on them without external deadlines/pressure.
In terms of jumping into the tech industry: I understand the advice about looking for any job when starting out. It just seems that even a lot of the entry level jobs are very specialized.