Using an ETL framework vs. writing yet another ETL script
1–10 of 77 posts
Re: Using an ETL framework vs. writing yet another ETL script
#2I had this same kind of impression writing product/business reports as an engineering manager at Oracle and elsewhere. But my way of solving it was to build a smarter IDE that helps you do this stuff without ETL.
You shouldn't need to waste time every time you build a report figuring out again how to query Postgres/MySQL/Elastic in Python and how to generate graphs and where to host it or how to set up recurring alerts on changes. The only part you should care about is the actual queries to write, the joins to do between datasets, and the fields to graph. The actual integration code (connecting to the database, copying a file from a remote server, etc.) should be handled for you.
The tool I'm building to solve this is open source if you're curious!
Re: Using an ETL framework vs. writing yet another ETL script
#3I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do.
Before you know it you have super long and super inefficient code just to fit the framework. It takes about the same time to read and understand an ETL framework as it is to write your own python/bash script, and at least with your own code it's easier to see bottlenecks.
Re: Using an ETL framework vs. writing yet another ETL script
#4Of course, the flip side is that sometimes that initial step of:
> "We are doing a prototype on our new mapping product, we need to write a one-off script to pull in some map data."
... is just that - a one off script. And it can prove to be a lot quicker to write a one-off script than getting involved with an ETL framework. I am not arguing against ETL Frameworks (Airbyte etc). Just that over-engineering carries its own costs, just like under-engineering does.
Re: Using an ETL framework vs. writing yet another ETL script
#5Hiya, I'm the original author. tl;dr for those deciding whether or not to read it:
If you are thinking about build versus buy for your ETL solution, in this day and age, there are enough great tools out there where buy is almost always the right option. You may think you can write a "simple" little ETL script to solve your problem, but invariably it grows into a monster that will be a reliability liability and engineering time suck. The post goes into more depth on why that is. Enjoy!
Re: Using an ETL framework vs. writing yet another ETL script
#6I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do. Before you know it you have super long and supe…
I agree. I've used AWS Data Pipelines for some jobs but there is a steep learning curve. It is good for launching servers on demand to run your ETL jobs if you need that.
The best solution I have found is writing ETL scripts in Laravel which I use for most projects anyway. The framework has built-in scheduling and error reporting.
Re: Using an ETL framework vs. writing yet another ETL script
#7I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do. Before you know it you have super long and supe…
Prophecy.io let’s you create visual components from any Spark function. Same with Airflow. So you can use standard components (built-in or your new ones) without being restricted.
Founder here - we’re working to solve this exact problem.
Re: Using an ETL framework vs. writing yet another ETL script
#8Their standard scenario to avoid is actually a perfectly acceptable process to grow though prior to wedging another but of infrastructure into your org that needs it's own provisioning, maintenance and support, redundancy planning etc.
In that scenario the situation is so nebulous that the original implementers had no way to know at what point the business/use case would end up so why would they immediately jump to a belt and braces solution.
It's the infrastructure version of my go to phrase: don't code for every future.
Re: Using an ETL framework vs. writing yet another ETL script
#9I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do. Before you know it you have super long and supe…
Indeed. A classic one is dealing with oauth2…
Airbyte docs:
> Note that the OAuth2Authenticator currently only supports refresh tokens and not the full OAuth2.0 loop.
Re: Using an ETL framework vs. writing yet another ETL script
#10I gave up on all these frameworks and just use Cron and Make and Python