Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

91–100 of 120 posts

Re: Engineers Shouldn’t Write ETL

#91

Post by matwood »

yeukhon wrote: post #55
matwood wrote: post #26

Earlier quoted context omitted.

Talking is easy, executing is hard. Executing requires discipline which so many people seem to lack. One of the reasons I love making people write down their idea (myself included) before talking about it is that writing forces a small initial execution step. Even a step this small can often filter the useless ideas away.

This is where design doc should be required. At my company we engineers are required to come up with a design doc and share with the whole org for feedback then a design meeting that takes place every week. At first I thought this was a step back because I felt it was a water fall but after writing my own design doc I quickly realized “talking is really fucking cheap.” Sitting down and write a doc that considers as m…

That's great. A design doc is even a step beyond just writing down the idea. Bezos is famous for making people write one pagers to pass out prior to talking about any idea. I see one pagers as way to quickly weed through a bunch of ideas, and a design doc as the next step to determine feasibility.

Re: Engineers Shouldn’t Write ETL

#92

Post by leoc »

DoreenMichele wrote: post #32
internet555 wrote: post #16

Earlier quoted context omitted.

What’s funny to me is how many incompetent “thinkers” appear in meetings. Obviously, thought (even removed from implementation entirely) often has immense value. Eg, many people spent a lot of time thinking about arithmetic, linear algebra, floating point, compilers, and now I can go run whatever cool algorithm on my computer. But I continually seem to run into these people who seem borderline incompetent at anything…

You see that a lot with 2e people -- bright people with some weaknesses or a disability. That could account for how negatively you have experienced this. Many 2e people have never really been taught good ways to handle the combination of big strengths and big weaknesses. I serve as a sounding board a lot for my oldest son and that works well, but it's not uncommon for such people to just be trying to meet their own n…

To be a little blunt about it,

> they have a lot of baggage that makes them openly hostile to meaningful feedback and they crave validation. Anything other than praising their half-baked ideas is met with toxic reactions. In such cases, the best you may be able to do is basically make a few polite noises and then disengage as quickly as possible.

would serve pretty well as a fair description of normal people, ime.

Re: Engineers Shouldn’t Write ETL

#95

Post by mdisc »

datademon wrote: post #64

As an undergraduate who is about to graduate with a degree in "Data Science" this post encapsulates a lot of my worries as I move into the work world. Should I focus on being a "thinker" a "doer" or a "plumber"? For the first three years I was planning on being a CS major until I was denied from the department: now the data science major is my only hope to graduate. I feel as though my programming skills are solid: b…

1. I think it’s better to focus on doing - especially if you’re interested in working with an earier stage company. You’re much more versatile and if you choose the right company with an upward trajectory then you have the chance to specialize more into data science and learn model building if you want to. Also, data science seems sexy, but I find it most rewarding when you can put your own models into prod and also I think it’s useful for people to have the context around what it involves before they specialize. I’m 28, and had a lot of peers go into data science and quickly realize that it was the hype that lead them there and that they enjoy engineering more.

2. Look for work in a different part of the country... or maybe just the right organization that’s willing to take a chance on you. We’re in Austin and we’ve hired smart, hardworking kids who’ve never touched the languages we use and get them contributing meaningfully in 3) I used to work for a startup where our CTO would not hire a data scientist unless they could write production code (in backbone and rails which I didn’t know at the time) and after I started, I spend 4-5 months just learning to be a full stack dev- I think that was so useful for my career as a data scientist. It meant that data scientists at this company could put whatever models they were running into the product- it drastically simplified org structure—- much more autonomy and fewer project management dependencies.

Sure not a lot of data scientists wanted to also be or make them selves into full stack developed, but you’d end up with the really gritty ones who and they’d end up being more loyal and much more on the same page as the rest of the engineering team it was way better for the whole org.

We’re hiring interns + junior full time people by the way

https://angel.co/schoolinks/jobs

Re: Engineers Shouldn’t Write ETL

#96

Post by technofiend »

dgudkov wrote: post #58
technofiend wrote: post #56

It's not in my experience performant but Pentaho is definitely ETL-for-dummies easy to use. Similar to your average user pivoting data in Excel rather than learning Python or R, sometimes having a tool with suboptimal performance is better than optimizing an adhoc or short term process.

If you're looking for a real ETL-for-dummies, take a look at my EasyMorph ( https://easymorph.com ). We've made a number of simplifications that specifically target "dummy" users, e.g. columns may mix values of different types (text, numbers, etc.).

And it's a partner with Tableau? Yeah that sounds pretty good! Thanks for the tip.

Re: Engineers Shouldn’t Write ETL

#97

Post by tkyjonathan »

disposedtrolley wrote: post #62
jimbokun wrote: post #54

I think this diagnoses the problem well, but ignores an obvious solution. A team of one data scientist and one engineer, completely responsible for building a model, and seeing it through into production, meeting all applicable SLAs and performance metrics. Or maybe it's two data scientists and one engineer, or one scientist and two engineers, whatever is required. The point is to have a small team you can hold compl…

Small teams are awesome in so many scenarios. I recently wrapped up a 4-week proof of concept for a client on knowledge management and discovery using NLP. I was able to work with someone apt at machine learning while I focused on building out the UI and backend. We delivered a first release about 3 days after we started, giving ample time to seek feedback and let the users shape the direction.

In 4 weeks, I was able to create a data mart that had self-healing (we had issues with Python/events missing data which should have reached the existing data warehouse) and the physical data models in it sped up an existing 6 hour ETL task down to 0.15 seconds AND speed up a production query that took 5 seconds per click down to 0.07 seconds.

No team needed or proof of concepts. Actual working data models, up to date tables + ETL code in production.

Background is DBA.

Re: Engineers Shouldn’t Write ETL

#98

Post by dgudkov »

technofiend wrote: post #96
dgudkov wrote: post #58

Earlier quoted context omitted.

If you're looking for a real ETL-for-dummies, take a look at my EasyMorph ( https://easymorph.com ). We've made a number of simplifications that specifically target "dummy" users, e.g. columns may mix values of different types (text, numbers, etc.).

And it's a partner with Tableau? Yeah that sounds pretty good! Thanks for the tip.

Yes, we're partners with Tableau. Just returned from Tableau Conference 2018 in New Orleans where we had a booth.

Re: Engineers Shouldn’t Write ETL

#99

Post by dgudkov »

v4n4d1s wrote: post #81
dgudkov wrote: post #58

Earlier quoted context omitted.

If you're looking for a real ETL-for-dummies, take a look at my EasyMorph ( https://easymorph.com ). We've made a number of simplifications that specifically target "dummy" users, e.g. columns may mix values of different types (text, numbers, etc.).

Thanks for developing easymorph! Free version helped me through my bachelors degree. It's my go-to tool to introduce people to ETL and similar concepts.

You're welcome! Great to hear it happened to be of help :)

Re: Engineers Shouldn’t Write ETL

#100

Post by jakestein »

NovaX wrote: post #44
fraserharris wrote: post #17

Earlier quoted context omitted.

I would suggest checking out my employer, Fivetran (fivetran.com). Many tech firms use us to centralize their data for analysis. We have startup pricing for sub-50 employees.

Do you know when DB/2 support will be added (Windows, AS/400)? I've been wanting to switch over from AWS DMS and our in-house support.

We (Stitch) have DB2 in production today https://www.stitchdata.com/integrations/db2/
Post reply on HN