Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Engineers Shouldn’t Write ETL

multithreaded.stitchfix.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: Engineers Shouldn’t Write ETL

#111

Post by ficklepickle »

leoc wrote: post #92
DoreenMichele wrote: post #32

Earlier quoted context omitted.

You see that a lot with 2e people -- bright people with some weaknesses or a disability. That could account for how negatively you have experienced this. Many 2e people have never really been taught good ways to handle the combination of big strengths and big weaknesses. I serve as a sounding board a lot for my oldest son and that works well, but it's not uncommon for such people to just be trying to meet their own n…

To be a little blunt about it, > they have a lot of baggage that makes them openly hostile to meaningful feedback and they crave validation. Anything other than praising their half-baked ideas is met with toxic reactions. In such cases, the best you may be able to do is basically make a few polite noises and then disengage as quickly as possible. would serve pretty well as a fair description of normal people, ime.

True, almost everyone has baggage. I think the difference is one of scale.

For example, in grade 5 I got a C+ in fine arts. It devastated me to the point where I questioned my abilities and disengaged from schooling. Permanently. It's only been in the last few years that I've actually been able to apply myself to anything.

Now, I can see that was an unreasonable reaction. At the time though, I didn't even understand what was happening.

If I hadn't got a lot of help, I believe I would still be driven by my insecurities to this day and would be impossible to work with.

Edit: typo

Re: Engineers Shouldn’t Write ETL

#112

Post by nitrogen »

datademon wrote: post #70
nitrogen wrote: post #65

Earlier quoted context omitted.

A bit OT, but as a more experienced engineer who dropped out of school to start a company, I'm curious: why weren't you able to get into your school's CS program? Don't worry too much about "falling behind". There will always be time to learn more math or a new framework. Worry more about finding that first job, any job, then you can branch out once inside the industry. Networking beats recruiters beats sending a res…

I did poorly on a math class that was required to declare the major. It's ironic since now that I'm in the data science major, I have to do even more math classes and less programming classes. I would love to do my own startup. I have a few ideas floating around. But I feel like I lack the discipline to sit down every day and force myself to work on them without external deadlines/pressure. In terms of jumping into t…

I'd recommend against doing a startup straight out of school unless you get accepted into a notable accelerator with a solid cofounder. Apply for the seemingly specialized jobs anyway, the worst they can do is say no.

Re: Engineers Shouldn’t Write ETL

#114

Post by motymichaely »

Different parts of engineering require different skill set.. Someone has to do the data engineering part (be it the data scientist, data engineer, ops, whatever..). This requirement hasn't changed since 2016: 50 to 90 percent of time is spent "Cleaning" Data for Analytics. You just need engineers with the right skills and tools to help reducing this time and get things done.

https://s3.amazonaws.com/xplenty-assets/infographics/raw_dat...

Re: Engineers Shouldn’t Write ETL

#115

Post by [deleted] »

msencenb wrote: post #11

Does anyone have experience with ETL as a service like StitchData (not related to stitchfix)? The startup I'm employed at needs some data analysis, but it is not big data, simply a way to unify analytics into a queryable database. I'm not looking forward to writing any ETL code, and was hoping someone here had a tool to help.

[deleted]

Re: Engineers Shouldn’t Write ETL

#116

Post by ArchTypical »

I wrote plenty of ETL. Maintained high throughput using whatever I could find. Then I got another job and had to write ETL for AdTech, where the volume is unlimited. Nothing about it is surprising or hard. Engineers are great at handling known data and transforms, then adapting to unknown data.

Re: Engineers Shouldn’t Write ETL

#117

Post by internet555 »

Annatar wrote: post #75
sagichmal wrote: post #68

Earlier quoted context omitted.

I think you may be experiencing a local minimum, so to speak. Data scientist programmers who "simply refuse" to think about basic productionization criteria like SLAs shouldn't remain employed.

They shouldn't, but they do because understanding a little bit of mathematics is at a premium these days and it's very much in vogue to crap all over computer science majors even though we had the same kind of mathematics curriculum as they at the university.

Really? I had one sort of similar because I took as many cross listed math electives as possible and avoided any software type courses, but even then I learned a lot less analysis, algebra, geometry than a good math student would learn

Re: Engineers Shouldn’t Write ETL

#118

Post by just_myles »

tkyjonathan wrote: post #88

ETLs, physical data modelling and data marts/warehouses used to be handled within database admin's task in small to medium sized companies and largely with ETL tools or just SQL.

Yup. That's been my experience. The DBA used to handle all these tasks and as of I don't know a 5 years ago, it's been segmented into data engineering. I think in this case it's a good thing. I always considered that a non-administrative tasks.

Re: Engineers Shouldn’t Write ETL

#119

Post by fraserharris »

Petefine wrote: post #108
georgewfraser wrote: post #69

Earlier quoted context omitted.

I'm Fivetran's CEO and I just want you to know, whatever "burps" you experienced, these things keep me up at night and the whole team is always striving to make the pipeline "just work". The whole vision of our product is that you should be able to plug in and get a perfect mirror image of all your data sources in your data warehouse. Anytime we fall short of that it drives us crazy.

Do you have a forum or suggestions tool at all? Fivetran has been amazing for our new datawarehouse and we're very pleased with the service, but there are a few little (non-bug) things that would have made it even easier.

You can email me: fraser@fivetran.com

Re: Engineers Shouldn’t Write ETL

#120

Post by iblaine »

meritt wrote: post #10
iblaine wrote: post #5

> March 16, 2016 It's 2018. A lot has changed since 2016. The line between sw engineering & data engineering is much thinner.

I'm curious what innovative tools have emerged in the past two years that changed the dynamic?

Airflow and tools like it are probably the biggest reason for the shift. Another issue is the need to integrate different technologies requires having the skills of a software engineer.

When the landscape for tech in DE was oracle, mysql and Cognos, DE's didn't need to know about OOP or consensus algorithms. Because the landscape now includes hadoop, redshift, kafka, spark, airflow, notebooks, TiDB and lord knows what else, DE's need to have most of the skills of a software engineer to be successful.

Post reply on HN