Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using an ETL framework vs. writing yet another ETL script

airbyte.io

Post reply on HN
Print view View original

31–40 of 77 posts

Re: Using an ETL framework vs. writing yet another ETL script

#31

Post by sherifnada »

switch007 wrote: post #9
jumpman500 wrote: post #3

I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do. Before you know it you have super long and supe…

Indeed. A classic one is dealing with oauth2… Airbyte docs: > Note that the OAuth2Authenticator currently only supports refresh tokens and not the full OAuth2.0 loop.

I think this is saying that particular class expects to receive a refresh token as input. The "full oauth loop" means the UI needs to produce a refresh token via user consent in the browser.

Re: Using an ETL framework vs. writing yet another ETL script

#33

Post by mcguire »

jumpman500 wrote: post #3

I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do. Before you know it you have super long and supe…

Couple that with the way ETL frameworks quickly become undocumented, featuriferous that are opaque to anyone who isn't deeply embedded into the framework, yeah.

Re: Using an ETL framework vs. writing yet another ETL script

#35

Post by wobblykiwi »

ibains wrote: post #7
jumpman500 wrote: post #3

I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do. Before you know it you have super long and supe…

Prophecy.io let’s you create visual components from any Spark function. Same with Airflow. So you can use standard components (built-in or your new ones) without being restricted. Founder here - we’re working to solve this exact problem.

What's the difference between Prophecy and the multitude of other ETL tools out there, like StreamSets, Talend, Ab Initio, and plenty more?

Re: Using an ETL framework vs. writing yet another ETL script

#37

Post by huetius »

jumpman500 wrote: post #3

I've always found ETL frameworks to have their own problems. They seem great on paper but usually they don't account for a specific source system, APIs, applications, data size, data distribution or scheduling situations. If your project is using it then developers end up hacking the frameworks instead of writing simple code that does the specific thing they need to do. Before you know it you have super long and supe…

I started writing and never completed a dead simple ETL framework that left most of the work up to the programmer. It was basically just an http server that you could hit with cron jobs, a DAG data structure for mapping the process, and some annotations that you could use to specify that a node/program step was concurrency safe, blocking, etc. You’re entirely right that there’s way more to ETL than meets the eye, but this still makes me want to dig it back up.

Re: Using an ETL framework vs. writing yet another ETL script

#38

Post by ibains »

wobblykiwi wrote: post #35
ibains wrote: post #7

Earlier quoted context omitted.

Prophecy.io let’s you create visual components from any Spark function. Same with Airflow. So you can use standard components (built-in or your new ones) without being restricted. Founder here - we’re working to solve this exact problem.

What's the difference between Prophecy and the multitude of other ETL tools out there, like StreamSets, Talend, Ab Initio, and plenty more?

We’re very different from the ETL tools in that we’re bring software development best practices to data.

When you do visual drag and drop - prophecy is generating high quality code on git that is 100% open source (spark, airflow), you have tests and CI/CD - so you’re visually doing solid data engineering.

You can toggle between code and visual - so if you change the code (some), the visual graph updates - so small edits directly to git don’t break the visual layer.

All visual components are generated from a spec - think a Spark function with a some more info. So the data platform teams will create their own library/framework and roll it out to the wider teams. How it works us that in the visual editor, you start with standard Spark library, but can load visual components for delta, or encryption or data quality.

Our customers are typically fed up of these ETL tools and moving to us. We can also import the ETL formats (AbInitio, Informatica, …) in an automated way (we reverse engineered their formats and created source to source compilers)

Re: Using an ETL framework vs. writing yet another ETL script

#39

Post by at_a_remove »

I am just an amateur, but gosh I love doing ETL. There's something about the specification of the data you're supposed to be getting, and then the harsh reality of the filthy trash you will actually be fed, and making something that can carefully examine each of the assumptions in the spec to test for how it won't go right, making tests for things that will, you are told, "never happen" (only to get an email from your program three years later that this has, in fact, happened), interpolating data where you can, and in general making a "simple load process" into one of those grinders people can feed cows and tungsten carbide rods into.

I feel like Data mentioning that he just ... loves scanning for life forms.

Re: Using an ETL framework vs. writing yet another ETL script

#40

Post by lixtra »

davinchia wrote: post #21
lloydatkinson wrote: post #14

Earlier quoted context omitted.

That's crazy, why would it have the approach of "select all columns" when you might never need all of them? Could you create a view for it to select from instead? Either way if someone told me a ETL supports only literally all the columns or nothing then I'd assume it was a MVP product to demonstrate the idea... not a production system.

We are working on it. This should be out next quarter!

It’s strange to advertise a framework that doesn’t support more than “select *”. A framework should make easy things easy and complicated things possible. Ideally there is a natural way to do stuff, so different people understand each other’s approach.

The whole article sounds like you target total amateurs in the ETL domain. Every decent ETL engineer knows all of that what is revealed after “a few months”.

That said, I always welcome new competition in the field. Tools still suck.

Post reply on HN