Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Source Python ETL

amphi.ai

Post reply on HN
Print view View original

101–110 of 127 posts

Re: Open Source Python ETL

#101

Post by hermitcrab »

iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

The debate about GUI vs code solutions has raged for decades, generating more heat than light. I've tried to give a summary of the pros and cons of the 2 approaches here:

https://successfulsoftware.net/2024/01/16/visual-vs-text-bas...

TLDR: Best depends on the situation and the people involved. Given the wide range of both situations and people, I think there are is plenty of room for both approaches.

Re: Open Source Python ETL

#102

Post by [deleted] »

iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

[deleted]

Re: Open Source Python ETL

#103

Post by thibautdr »

johhns4 wrote: post #59
thibautdr wrote: post #3

Hi everyone, thanks for posting Amphi :) To give some context, Amphi is a low-code ETL tool for both structured and unstructured data. The key use cases include file integration, data preparation, data migration, and creating data pipelines for AI tasks like data extraction and RAG. What sets it apart from traditional ETL tools is that it generates Python code that you own and can deploy anywhere. Amphi is available…

Wow amazing work! How does the inputs work, are they created for you or does it support custom as well?

Thank you! Inputs components are pre-built for now but the ability to add custom inputs is coming soon!

Re: Open Source Python ETL

#104

Post by m463 »

awesomebytes wrote: post #46

I was not familiar with the acronym ETL and it is not explained anywhere in the website! My feedback would be to at least write it once, on the first instance so others like me will know what they are reading :)

In computing, extract, transform, load (ETL) is a three-phase process where data is extracted from an input source, transformed (including cleaning), and loaded into an output data container. The data can be collated from one or more sources and it can also be output to one or more destinations.

https://en.wikipedia.org/wiki/Extract,_transform,_load

Re: Open Source Python ETL

#106

Post by thibautdr »

paulvnickerson wrote: post #105

Very cool, thanks for sharing. Does it support the pandas-like rapidsai dask_cudf framework? ( https://docs.rapids.ai/api/dask-cudf/stable/ )

Great, thanks for sharing. I was familiar with Dask and cudf separately but not this one.I was planning to implement dask support through Modin but I'll definitely take a look at dask_cudf.

Re: Open Source Python ETL

#107

Post by mrtranscendence »

lma21 wrote: post #83
jamesblonde wrote: post #50

#dang The title needs changing - it's not open-source, it is license ELv2 - Elastic License v2.

isn't the code available here? https://github.com/amphi-ai/amphi-etl what makes it not OSS?

It's open source if you're using language like a normal human being. If you're a bit of a pedant and wish everyone to adhere to definitions imposed from on high regardless of real-world usage, it's absolutely not open source.

Re: Open Source Python ETL

#108

Post by mrtranscendence »

iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

I used to be an Informatica ETL dev, briefly. I laugh at the description of it as a low-code solution, as we used it as a simple frontend for staggering amounts of hand-coded SQL. I'm not sure I ever used Informatica as it was intended.

Re: Open Source Python ETL

#109

Post by OutOfHere »

thibautdr wrote: post #3

Hi everyone, thanks for posting Amphi :) To give some context, Amphi is a low-code ETL tool for both structured and unstructured data. The key use cases include file integration, data preparation, data migration, and creating data pipelines for AI tasks like data extraction and RAG. What sets it apart from traditional ETL tools is that it generates Python code that you own and can deploy anywhere. Amphi is available…

You know what does not set it apart? AI-washing. Also, lying about being open source when it isn't.

Re: Open Source Python ETL

#110

Post by cvalka »

mrtranscendence wrote: post #107
lma21 wrote: post #83

Earlier quoted context omitted.

isn't the code available here? https://github.com/amphi-ai/amphi-etl what makes it not OSS?

It's open source if you're using language like a normal human being. If you're a bit of a pedant and wish everyone to adhere to definitions imposed from on high regardless of real-world usage, it's absolutely not open source.

It's source available. It's definitely not open source. It's deception plain and simple.
Post reply on HN