Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

21–30 of 120 posts

Re: Engineers Shouldn’t Write ETL

#21

Post by RhodesianHunter »

whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

Completely agree with your take, and as an engineer in a similar role, the post rubbed me the wrong way. I don't find ETL work soul-sucking, and I certainly don't think that my colleagues or I are mediocre.

Re: Engineers Shouldn’t Write ETL

#22

Post by mycelium »

msencenb wrote: post #11

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

I would highly highly recommend ETL as service, after adopting it recently. It substantially changes your relationship with your data sources in a really positive way. And frankly, ETL for common data sources is code that you just don't need to write.

I would say that you should pilot with a few ETL vendors. We currently use Fivetran, they're fine but we've had enough burps that I cannot cold recommend them over other vendors. I cannot for the life of me remember the details, but I think we went with them over Stitch for pricing reasons.

Re: Engineers Shouldn’t Write ETL

#23

Post by veritas3241 »

msencenb wrote: post #11

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

We used them for a single data source at GitLab (Zendesk). Worked pretty well! But we quickly hit a road block where what you could get via the UI wasn't all the tables available. We wound up forking it and adding to the tap. Basically all of our extractors and loaders are going to follow the Singer spec for taps and targets - it's a pretty nice model.

Internally, we're using a tool called Meltano which is aimed at solving just your problem. Most of our data warehouse is coming from external business ops tools (Salesforce, Zuora, Zendesk, Marketo, etc.) and we're using dbt for transformations w/ Looker as the BI layer. Definitely check our primary analytics repo [0] as all of our code is out in the open. Feel free to ping me if you have more questions - tmurphy at gitlab.

[0] https://gitlab.com/meltano/analytics/

Re: Engineers Shouldn’t Write ETL

#24

Post by protomyth »

bigger_cheese wrote: post #18
protomyth wrote: post #4

Report Developers, on the other hand, are folks who have made a career around designing reports in a specific tool (e.g. Microstrategy, et al). They are specialists. Is this the common perception, because it really doesn't line up with my experience?

At least in my Org Reports are pretty much an after thought left to the data engineers (like me) to "take this metric I've developed" and display it on the morning report. Writing/updating a report is easiest part of my job it's the data that goes into building it that is hard translating the "simple metric I've developed" and getting it to run in a robust automated and sane fashion is the difficult part. The complex…

Ok, I think I'm getting the specifics of this situation. So, we are talking about internal reports, not something that could actually get in the external customer's hands.

Re: Engineers Shouldn’t Write ETL

#25

Post by zimablue »

whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

I do this job and I'd say it's only enjoyable for now because I have complete freedom and I'm still learning. Splitting up the parts: Design architecture Wire up pipelines (once architecture is decided, this tends to declarative, connected is choosing schemas). Data science

I'd definitely not want to be stuck doing stage 2 forever, would prefer 3. I think you're saying that you enjoyed a job which was some 1 and some 2. I'm sure there's someone out there who wants to wire up pipelines with no engineering and no analysis all day but I'd imagine it's a rare breed.

Edit: I think the important distinction is team/company size. Doing a bit of everything as a 1 man team is challenging, if you have a team where devops/engineering/reports/tools have been chosen/built/standardized by specialist and you really are just wiring pipelines up, I think that would be tough. On the other hand being in a small team condemns you to always be doing the same fractions of work because there's noone to hand off to.

Re: Engineers Shouldn’t Write ETL

#26

Post by matwood »

internet555 wrote: post #16
whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

Talking is easy, executing is hard. Executing requires discipline which so many people seem to lack.

One of the reasons I love making people write down their idea (myself included) before talking about it is that writing forces a small initial execution step. Even a step this small can often filter the useless ideas away.

Re: Engineers Shouldn’t Write ETL

#27

Post by itronitron »

internet555 wrote: post #16
whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

Yes, those are talkers masquerading as thinkers. Not all talkers are bad, in fact some are great to work with, but the bad ones seem unable to not burn bridges.

Re: Engineers Shouldn’t Write ETL

#28

Post by internet555 »

itronitron wrote: post #27
internet555 wrote: post #16

Earlier quoted context omitted.

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

Yes, those are talkers masquerading as thinkers. Not all talkers are bad, in fact some are great to work with, but the bad ones seem unable to not burn bridges.

That is also very true. I worked with someone like that who was really wonderful (and a wonderful person)

Re: Engineers Shouldn’t Write ETL

#29

Post by CoolGuySteve »

internet555 wrote: post #16
whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

I once worked with a "Data Scientist" at a hedge fund that was clearly pattern matching whatever problem you had to some random Apache/Google tool without actually listening to the problem.

  {
    "Unit test" : "google mock",
    "CSV" : "Apache Parquet",
    "message" : "google protobuf",
    "queue" : "Apache Kafka",
  }
etc

His data science recommendations looked like a markov chain of various analysis algorithms.

One time I started digging into his recommendation trying to figure out why it was even on topic and he starts going on about how he's a "big picture" guy and not to bother him with implementation details. The thing was, his 'big picture' ETL was breaking our trading system every other week due to some inane dependency strictness that wasn't necessary.

There's nothing more 'big picture' than not fucking trading!

I guess because portfolio managers aren't specialized in engineering, we see a lot more of these fakers in finance than tech.

Re: Engineers Shouldn’t Write ETL

#30

Post by itronitron »

jerglingu wrote: post #20
whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

I'm wondering if anybody lives in the hypothetical perfect world scenario the author writes about. I'm at one of the larger tech companies and it's inconceivable that something like this could exist (though the churn here is extremely high - a mature shop with longstanding membership could implement the hypothetical in some form). Everything sounds nice when dreaming it up in one's head, but discounts the reality of…

I've experienced that hypothetical perfect world, it comprised teams of data scientists, developers, and system engineers working on making progress towards multiple distinct 'business insights' and operating within a big data ecosystem. People's job roles were tightly defined but their participation in tasks was largely determined through self-selection and reputation.
Post reply on HN