Cheers for the beautiful website that doesn't require javascript.
Engineers Shouldn’t Write ETL
31–40 of 178 posts
Re: Engineers Shouldn’t Write ETL
#32This article is so one sided it is painful. it is almost like Sheldon Cooper wrote this article. As an engineer I am offended and hurt that we are referred to as “Tony Stark’s tailor”.
Btw didn't he build the suits himself and just share it with his less able cop friend?
I'd agree, it feels like it invalidates the claim about getting credit for 'being a thinker'.
Re: Engineers Shouldn’t Write ETL
#33Earlier quoted context omitted.
I thought it nailed a lot of dynamics at my last gig as well. Of course, I joined for the challenges of scaling and wrote an ETL framework in Rails (that was a mixed bag but very instructive) and then got bored after I realized how small our data really was. Then I left for Google and all my data went up by two prefixes. I do love reading an article that supports my contention that ETL is the Charlie Work [0] of soft…
It's all Charlie Work for the VCs, "founders", and (occasional) early hires who take in the lion's share of profits and prestige for the long hours most of us pour into our jobs -- day after day, year after year. BTW, that may sound polemic, but it's not -- that's really how a lot of business types, academic researchers, and others think of nearly all programming-related work.
Not that polemic indeed, but only if we're willing to stop gazing our own navels. Most of us here aren't hired with nice paycheks because we're smart and creative, but because we're smart, creative, and yet without the sense to find better work than Charlie Work.
Anyway, this is a shame. I thought "data science" still had nice R&D flavored jobs. Colour me disillusioned.
Re: Engineers Shouldn’t Write ETL
#34Is ETL really even necessary anymore? Why not just run fast ad hoc queries over the raw data with something like Google BigQuery?
Yeah - just Extract it from your MySQL / Mongo / Postgres / logfiles / whatever system it's in right now, Transform it into a CSV or whatever the input needs to be and Load it into BigQuery. Once it's there, you can do whatever you need!
Re: Engineers Shouldn’t Write ETL
#35... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…
This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…
What technologies / architecture does Amazon use for business intelligence? I've just done a business intelligence course so I'm interested how do more "technology-centered" companies approach BI, whether it's the same thing I learned in the course (put everything into an integrated relational database if I simplify a lot).
Re: Engineers Shouldn’t Write ETL
#36> We are not optimizing the organization for efficiency, we are optimizing for autonomy.
This is one of the toughest parts of building a scalable organization (with or without big data). Getting past the idea of efficiency and being OK with redundancy.
This means allowing two teams to both build a common feature they might need, rather than establishing a dependency. It means making one teams job broader even if it overlaps with another team.
I find it interesting that we are perfectly willing to have redundancy on the software side (load balancing, slaves, etc) but not on the development side.
Re: Engineers Shouldn’t Write ETL
#37Nobody enjoys writing and maintaining data pipelines or ETL. It’s the industry’s ultimate hot potato. It really shouldn’t come as a surprise then that ETL engineering roles are the archetypal breeding ground of mediocrity.
There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume.
This is like... your opinion. Some people find pushing around HTML / JS / CSS absolutely soul-crushing. Considering the lion's share of websites are ugly, unusable, and slow, does this mean that front-end engineering is a breeding ground of mediocrity, so server-side devs and CFOs should all be sharing in the pain?
Some people actually enjoy working with data, and don't find ETL and pipelining horrible to do at all. It is a different set of challenges, but calling people people mediocre because of ETL is a non-sequitur.
Re: Engineers Shouldn’t Write ETL
#38Earlier quoted context omitted.
This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…
What technologies / architecture does Amazon use for business intelligence? I've just done a business intelligence course so I'm interested how do more "technology-centered" companies approach BI, whether it's the same thing I learned in the course (put everything into an integrated relational database if I simplify a lot).
Lots and lots of SQL (a very good thing IMO). This can come in the form of Oracle, Redshift, or any of the commonly available RDS databases (probably not SQL Server though). This is augmented with a lot of Big Data stuff, which used to be pretty diverse (Pig, Hive, raw Hadoop, etc.) but is moving very quickly towards a Spark-centric platform. There is occasionally some commercial software like Tableau/Microstrategy. Apart from that, it is a whole lot of homegrown technology.
As far as architecture is concerned, every team is different. At least on the retail side, we tend to center around a large data warehouse that is currently transitioning from oracle to redshift, and with daily log parsing ETL jobs from most services to populate tables in that data warehouse. Downstream of the data warehouse is fair game for pretty much anything.
Re: Engineers Shouldn’t Write ETL
#39... a highly specialized team of dedicated engineers...If they are not bored, chances are they are pretty mediocre. Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. OMG, the author just described the last place I was at. Processed a few Tb of data and suddenly there's this R. Goldbergesque system of MongoDb getting transformed into PostGres...oh…
This is so true. I do business intelligence at Amazon, and I've seen this play out millions of times over. The fetishization of big data ends up meaning that everybody thinks their problem needs big data. After 4 years in a role where I am expected to use big data clusters regularly, I've really only needed it twice. To be fair, in a complex environment with multiple data sources (databases, flat files, excel docs, s…
Fellow amazonian here. We switched from a massively distributed datastore (not to be named) to rodb for storage and found 10x improvement, not to mention eliminating cost and other head-aches; kind of expected since rodb is an embedded db...
Re: Engineers Shouldn’t Write ETL
#40There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. I'm not sure I get why writing ETL code for data you'll never consume is any more soul-sucking than, say, refactoring JS code for a website you couldn't begin to care about (and which will never be properly re-designed anyway); or even doing "thinker"-level work bu…
And I also fundamentally disagree with the notion that moving a large amount of realtime data reliably and with accuracy, monitored and consistent with relatively little failure is not an interesting engineering challenge in itself. I find that for all the talk about data-driven organizations, most don't use a tenth of what is available but that when the tenth is needed, it's hugely satisfying to be able to provide it.