Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

101–110 of 120 posts

Re: Engineers Shouldn’t Write ETL

#101

Post by NovaX »

jakestein wrote: post #100
NovaX wrote: post #44

Earlier quoted context omitted.

Do you know when DB/2 support will be added (Windows, AS/400)? I've been wanting to switch over from AWS DMS and our in-house support.

We (Stitch) have DB2 in production today https://www.stitchdata.com/integrations/db2/

Is it log-based cdc? The documentation says only 2 dbs are supported that way. We want to replicate customer databases which have schemas outside of our control, with the goal of migrating them to our products. The other vendor schemas might not have your required columns. There are also many flavors of DB2. So very interested but the details are sparse at best.

Re: Engineers Shouldn’t Write ETL

#102

Post by WorldMaker »

liamconnell wrote: post #31
amarshall wrote: post #9

> There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. Instead, give people end-to-end ownership of the work they produce (autonomy). I think this is more the point than “engineers shouldn’t write ETL”: the engineering-related department consuming the ETL’s output should likely be the ones writing/maintaining it. Or,…

> the engineering-related department consuming the ETL’s output should likely be the ones writing/maintaining it. This is exactly the author's point. The Data Scientists are consuming the ETL's output, so they should learn how to write and maintain ETL since it isnt very hard or time consuming with modern tools.

Though this backfires sometimes for the engineering department in that then the engineers get forced into an "inner-platform effect" problem that they instead have to build an ETL platform abstracting enough ETL abilities for their company's data scientists' skill level, yet generic enough for their company's data scientists' arbitrary questions/needs.

That is its own soul-sucking experience. "Can't we just hire people that can learn Power BI better? Why are we still writing data tools for people that think they know Access but barely know Excel?"

Re: Engineers Shouldn’t Write ETL

#103

Post by dsclough »

whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

I'm working for a small hedge fund in basically the same role, do you have any resources you could recommend for learning how to do these things the "right" way?

Re: Engineers Shouldn’t Write ETL

#104

Post by dv_dt »

whack wrote: post #8

I've worked in a hedge fund in the past, where my role sounded a lot like what the author describes as a "Data Engineer". I would have thinkers, ie people with a lot of financial experience, come up with ideas on which datasets we want to import from which vendors, and how we should handle the 80 different types of corporate actions that are contained within this dataset. I sometimes gave my own suggestions on how to…

> I was getting paid hundreds of thousands of dollars, because the "thinkers" recognized the value that I brought to the table. They appreciated...

I think this is the key idea there. It's good that you found a situation where you're both appreciated, and compensated for it. It's too easy for engineers to be devalued as replaceable cogs in the pipeline of things that need to happen to bring in revenue.

Re: Engineers Shouldn’t Write ETL

#105

Post by yeukhon »

Annatar wrote: post #73
yeukhon wrote: post #55

Earlier quoted context omitted.

This is where design doc should be required. At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week. At first I thought this was a step back because I felt it was a water fall but after writing my own design doc I quickly realized “talking is really fucking cheap.” Sitting down and write a doc that considers as m…

"At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week." Wow, true system engineering! Your company sounds like a good place to do some professional work.

Feel free to visit oscar career page.

Re: Engineers Shouldn’t Write ETL

#106

Post by yeukhon »

matwood wrote: post #91
yeukhon wrote: post #55

Earlier quoted context omitted.

This is where design doc should be required. At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week. At first I thought this was a step back because I felt it was a water fall but after writing my own design doc I quickly realized “talking is really fucking cheap.” Sitting down and write a doc that considers as m…

That's great. A design doc is even a step beyond just writing down the idea. Bezos is famous for making people write one pagers to pass out prior to talking about any idea. I see one pagers as way to quickly weed through a bunch of ideas, and a design doc as the next step to determine feasibility.

It really is a great tool for validate ideas.

Sorry for my grammar above as I was typing on my mobile phone halfway on the bus.

Re: Engineers Shouldn’t Write ETL

#108

Post by Petefine »

georgewfraser wrote: post #69
mycelium wrote: post #22

Earlier quoted context omitted.

I would highly highly recommend ETL as service, after adopting it recently. It substantially changes your relationship with your data sources in a really positive way. And frankly, ETL for common data sources is code that you just don't need to write. I would say that you should pilot with a few ETL vendors. We currently use Fivetran, they're fine but we've had enough burps that I cannot cold recommend them over othe…

I'm Fivetran's CEO and I just want you to know, whatever "burps" you experienced, these things keep me up at night and the whole team is always striving to make the pipeline "just work". The whole vision of our product is that you should be able to plug in and get a perfect mirror image of all your data sources in your data warehouse. Anytime we fall short of that it drives us crazy.

Do you have a forum or suggestions tool at all? Fivetran has been amazing for our new datawarehouse and we're very pleased with the service, but there are a few little (non-bug) things that would have made it even easier.

Re: Engineers Shouldn’t Write ETL

#109

Post by Annatar »

badpun wrote: post #82
Annatar wrote: post #74

Earlier quoted context omitted.

I was once accused by a Windows C++ programmer who just couldn't grok UNIX to be a "string cutter", but honestly, there is nothing so viscerally information technology as extract-transform-load. The irony of his accusation was that it was just one of many facets of things that I do and can/know how to do. ETL is really the core of IT by definition.

A lot of the hardcore C++ programmers look down on the whole IT area of programming (as opposed to systems, graphics, embedded etc.), maybe that was the reason.

No, this guy doesn't know anything but Windows.

Re: Engineers Shouldn’t Write ETL

#110

Post by jakestein »

NovaX wrote: post #101
jakestein wrote: post #100

Earlier quoted context omitted.

We (Stitch) have DB2 in production today https://www.stitchdata.com/integrations/db2/

Is it log-based cdc? The documentation says only 2 dbs are supported that way. We want to replicate customer databases which have schemas outside of our control, with the goal of migrating them to our products. The other vendor schemas might not have your required columns. There are also many flavors of DB2. So very interested but the details are sparse at best.

Our DB2 integration is SELECT replication. Our MySQL, Posgtgres, and Oracle integrations are log base. MySQL and Postgres actually have the option of either logs or SELECT.

You're totally right about all of the flavors DB2. Our support team would be the ones to figure out for sure whether or not we can work with your setup, and you reach them at support@stitchdata.com

Post reply on HN