Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Source Python ETL

amphi.ai

Post reply on HN
Print view View original

31–40 of 127 posts

Re: Open Source Python ETL

#31

Post by stoperaticless »

hbarka wrote: post #24
iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

>> Code as ETL was a huge industry shift No it’s not. Try and see what banks, retail, manufacturing, various large enterprises still use. They need scale, observability, modularity, and maintainability.

Gui Etl makes a promise that you dont need a programmer to wield it. But it is a false promise.

> They need scale, observability, modularity, and maintainability.

Seems orthogonal to code-vs-gui dimension.

Re: Open Source Python ETL

#32

Post by bigger_cheese »

hbarka wrote: post #24
iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

>> Code as ETL was a huge industry shift No it’s not. Try and see what banks, retail, manufacturing, various large enterprises still use. They need scale, observability, modularity, and maintainability.

I work in manufacturing (large industrial plant) and the data processes we have are honestly not great - mostly it is because there are a heap of legacy system and not a lot of commonality between our data sources we have a hideous mashup of Oracle, DB2, Microsoft SQL Server etc and different versions of the different databases. There's also more bespoke industry stuff like time series historians and SCADA systems/PLCs (ABB, Citect etc) to complicate the process.

From my experience SQL is basically the lowest common denominator everything speaks and even then the Oracle SQL dialect is subtly different to Microsoft SQL for example - things are subtly different enough it introduces frustrations.

There has been movement in last couple of years to hoist everything into a common "datalake" but my understanding has been that ingestion into this lake is not a simple process by any means and requires batch processes that need demanding compute resources and is slow (i.e. takes many hours and runs over night).

Re: Open Source Python ETL

#33

Post by stoperaticless »

banku_brougham wrote: post #26
thibautdr wrote: post #12

Earlier quoted context omitted.

Thanks for your comment! I do believe it depends on who you ask and ultimately both will co-exist. I also think low-code solutions democratize access to ETL development offering a significant productivity advantage for smaller teams. With Amphi, I'm trying to avoid the common pitfalls of other low-code ETL tools, such as scalability issues, inflexibility, and vendor lock-in, while embracing the advantages of modern E…

Im very leery of low code, but I like the idea of ETL defined as configuration.

Etl as text is good, because you can save it in version control. (Is it “code” or “json” is irrelevant for the vcs)

Edit: save in vcs stringly implies usability of ‘diff’ and ‘grep’

Re: Open Source Python ETL

#34

Post by happymellon »

hbarka wrote: post #24
iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

>> Code as ETL was a huge industry shift No it’s not. Try and see what banks, retail, manufacturing, various large enterprises still use. They need scale, observability, modularity, and maintainability.

Completely agree.

I would also add that in my 25 years in the industry I have never actually come across an ETL workflow that was complex enough that it required people working it in code. Those opinions seem to occur before proper analysis happens.

Re: Open Source Python ETL

#35

Post by pm90 »

ic_fly2 wrote: post #13

With all the data issues strong quality and normalisation I often get the impression that enabling more people with non CS backgrounds to do this work is not necessarily a good thing. In other words, if writing python and sql is the skill requirement that stops you from making an etl pipeline, maybe do something else.

With this argument, Computer Science wouldn’t have progressed beyond assemblers.

Re: Open Source Python ETL

#36

Post by js8 »

iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

It's fascinating that somebody sees these things as "original". I used to maintain a 40-year old ETL codebase in SAS. IMHO the original low code movement was the 4GL movement from the 70s (and, according to Wikipedia, rapid application development movement from the 90s, things like Visual Basic).

I think fundamentally, the problem is that any DSL lives on a design tradeoff spectrum between domain-specificity and programming-generality. There is only so much savings that a generic ETL tool will give you, because it doesn't want to be domain specific. On the other hand, the more domain specific a tool is, the more limited it will be in it's capacity to handle other domains.

No amount of graphic or textual interface is gonna save you from this fundamental design tradeoff (it's kind of complexity tradeoff - things inside domain are easy but outside the domain are hard). But then, you can as well handle this tradeoff in a decent library/framework for a general programming language, which is, effectively, a "return to code".

Other way to look at it is that we want limited programming languages, because the implicit assumptions make things easier for the reader of the code (the person who wants to understand). However, limitations make things harder for the writer of the code (the person who wants to solve some problem).

Re: Open Source Python ETL

#37

Post by teej »

banku_brougham wrote: post #25
tiew9Vii wrote: post #19

Earlier quoted context omitted.

You see this a lot. A new generation comes in to relearn all the old lessons. It’s also happening on the frontend “server side rendering is bad, it’s slow” to the latest frontend frameworks and thought leaders “you should be using SSR” (server side rendering) ETL is hot again with machine learning. Companies have massive amounts of data they need to get in shape for models. The promise of a GUI anyone can use with li…

What is your estimate of the tine required to begin building on the new(old) foundation, discover all the problems after a large investment of effort, and then declare to all that ita the wrong path? My personal pain is Tableau trying to move uo the stack into data orchestration and compute. Code is completely inaccessible.

Don’t build ETL on Tableau. They haven’t made meaningful product progress in 10 years and completely missed the changes in data transformation. They are playing catch-up, they don’t understand where the world is moving.

Re: Open Source Python ETL

#38

Post by vekker »

Does this also manage the infrastructure side of ETL? Usually some parts in a complex ETL process take a lot more processing power, so are run on different machines. From a quick glance at this, it seems like a WYSIWYG ETL tool for running ETL jobs on one machine?

Re: Open Source Python ETL

#39

Post by pydry »

anakaine wrote: post #15
ic_fly2 wrote: post #13

With all the data issues strong quality and normalisation I often get the impression that enabling more people with non CS backgrounds to do this work is not necessarily a good thing. In other words, if writing python and sql is the skill requirement that stops you from making an etl pipeline, maybe do something else.

This is elitist and frankly, unhelpful. The answer to a skills shortage is not a practitioner lockdown, but policy, training, guidance and mentoring. If you're stuck in start up land and you have this issue, you have hired the wrong skills. If you're encountering this in enterprise land, your organisation, and potentially you depending on your position of influence, should be angling to improve compliance and literac…

Skills shortage?

Re: Open Source Python ETL

#40

Post by js8 »

banku_brougham wrote: post #25
tiew9Vii wrote: post #19

Earlier quoted context omitted.

You see this a lot. A new generation comes in to relearn all the old lessons. It’s also happening on the frontend “server side rendering is bad, it’s slow” to the latest frontend frameworks and thought leaders “you should be using SSR” (server side rendering) ETL is hot again with machine learning. Companies have massive amounts of data they need to get in shape for models. The promise of a GUI anyone can use with li…

What is your estimate of the tine required to begin building on the new(old) foundation, discover all the problems after a large investment of effort, and then declare to all that ita the wrong path? My personal pain is Tableau trying to move uo the stack into data orchestration and compute. Code is completely inaccessible.

> My personal pain is Tableau trying to move uo the stack into data orchestration and compute. Code is completely inaccessible.

Not really a Tableau user, but I bet they are doing it to increase vendor lock-in. Which has always been an industry wide problem, proliferation of APIs and languages for that reason. I am not even sure how this could be any different, if the remuneration in a capitalist society depends on scarcity of the traded product or service (and the commons are privatized as a means to increase it). Here, the commons are the common understanding, standard APIs, and interoperability.

Post reply on HN