Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Source Python ETL

amphi.ai

Post reply on HN
Print view View original

21–30 of 127 posts

Re: Open Source Python ETL

#21

Post by thibautdr »

slt2021 wrote: post #20
thibautdr wrote: post #3

Hi everyone, thanks for posting Amphi :) To give some context, Amphi is a low-code ETL tool for both structured and unstructured data. The key use cases include file integration, data preparation, data migration, and creating data pipelines for AI tasks like data extraction and RAG. What sets it apart from traditional ETL tools is that it generates Python code that you own and can deploy anywhere. Amphi is available…

i liked the idea of leveraging jupyterlab as server. data engineers/scientists already use jupyter, so this is neat idea. custom extension for jupyterlab is a great way to leverage existing jupyterlab install base: not everyone will be willing to install and jump through hoops to install software X, but installing extension is one pip install away and no need to run separate process, since you are running inside jupy…

Thanks! Being based on JupyterLab also allows Amphi to benefit from the vast ecosystem of extensions already available, such as the Git extension or using different file systems (S3).

Some users pointed out they were Alteryx users but liked the Python code generation from Amphi :)

Re: Open Source Python ETL

#22

Post by slt2021 »

thibautdr wrote: post #21
slt2021 wrote: post #20

Earlier quoted context omitted.

i liked the idea of leveraging jupyterlab as server. data engineers/scientists already use jupyter, so this is neat idea. custom extension for jupyterlab is a great way to leverage existing jupyterlab install base: not everyone will be willing to install and jump through hoops to install software X, but installing extension is one pip install away and no need to run separate process, since you are running inside jupy…

Thanks! Being based on JupyterLab also allows Amphi to benefit from the vast ecosystem of extensions already available, such as the Git extension or using different file systems (S3). Some users pointed out they were Alteryx users but liked the Python code generation from Amphi :)

just an idea: is it possible to code generate Airflow code? since a lot of companies use airflow as ETL orchestrator

Re: Open Source Python ETL

#23

Post by maphew »

cvalka wrote: post #14

THIS IS NOT OPEN SOURCE!

It's published on GitHub under license ELv2 - Elastic License v2. This does not meet the open source definition, so indeed it's not Open Source. ELv2 is an open source sibling though, closer than many other openish licenses: https://www.elastic.co/pricing/faq/licensing

Still, Amphi should not claim to be 'Open Source'.

Re: Open Source Python ETL

#24

Post by hbarka »

iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

>> Code as ETL was a huge industry shift

No it’s not. Try and see what banks, retail, manufacturing, various large enterprises still use. They need scale, observability, modularity, and maintainability.

Re: Open Source Python ETL

#25

Post by banku_brougham »

tiew9Vii wrote: post #19
iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

You see this a lot. A new generation comes in to relearn all the old lessons. It’s also happening on the frontend “server side rendering is bad, it’s slow” to the latest frontend frameworks and thought leaders “you should be using SSR” (server side rendering) ETL is hot again with machine learning. Companies have massive amounts of data they need to get in shape for models. The promise of a GUI anyone can use with li…

What is your estimate of the tine required to begin building on the new(old) foundation, discover all the problems after a large investment of effort, and then declare to all that ita the wrong path?

My personal pain is Tableau trying to move uo the stack into data orchestration and compute. Code is completely inaccessible.

Re: Open Source Python ETL

#26

Post by banku_brougham »

thibautdr wrote: post #12
iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

Thanks for your comment! I do believe it depends on who you ask and ultimately both will co-exist. I also think low-code solutions democratize access to ETL development offering a significant productivity advantage for smaller teams. With Amphi, I'm trying to avoid the common pitfalls of other low-code ETL tools, such as scalability issues, inflexibility, and vendor lock-in, while embracing the advantages of modern E…

Im very leery of low code, but I like the idea of ETL defined as configuration.

Re: Open Source Python ETL

#28

Post by necovek »

anakaine wrote: post #15
ic_fly2 wrote: post #13

With all the data issues strong quality and normalisation I often get the impression that enabling more people with non CS backgrounds to do this work is not necessarily a good thing. In other words, if writing python and sql is the skill requirement that stops you from making an etl pipeline, maybe do something else.

This is elitist and frankly, unhelpful. The answer to a skills shortage is not a practitioner lockdown, but policy, training, guidance and mentoring. If you're stuck in start up land and you have this issue, you have hired the wrong skills. If you're encountering this in enterprise land, your organisation, and potentially you depending on your position of influence, should be angling to improve compliance and literac…

FWIW, while I disagree with the parent comment, I don't see you arguing against it.

They actually implied that you should try upskilling first — but if that fails, you shouldn't be doing ETL yourself.

I mostly disagree with the parent comment because there's so many things one can easily do up to a level, and then when the going gets tough, you need to call in an expert. Eg. most people can operate a screwdriver or impact driver to fix things, but to fix some problems, you really need a trained technician (or well, an experienced DIY person, but that's not everybody).

The fact that you are not strong enough to screw in an M14 bolt does not mean you should be forbidden from using an impact driver: tools are there to help you. The logic of the parent comment was seemingly that if you are not strong enough to tighten an M14 bolt, you probably don't know what you are doing regardless of the type of the bolt you are tightening, so you should simply not do it.

The point I agree with in a parent comment is that not everybody can achieve a similar level of proficiency: while upskilling and improving/simplifying tools can get you most of the way there, there's always going to be that extra bit that requires a sudden, sharp jump in knowledge, smartness or experience to be able to deal with it.

Post reply on HN