Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Source Python ETL

amphi.ai

Post reply on HN
Print view View original

81–90 of 127 posts

Re: Open Source Python ETL

#81

Post by tayloramurphy »

thibautdr wrote: post #8
tayloramurphy wrote: post #6

I'm curious as to the story of how things like this come to be. It seems like there are already a ton of "open source python ETL" tools on the market. Was this a passion project by the author? Was this born out of academia? Was there a specific problem they were trying to solve that others didn't? It's not necessary to answer these questions in the docs but it is useful for folks who may be familiar with the other op…

Thanks for your comment, those are valid points. I come from the industry, having worked for an ETL vendor for 6 years. I've personally witnessed a need for a low-code (graphical) ETL for Python environments. In short, traditional ETLs are GUI ETLs for Java environments while modern data tools are either focusing on the EL part or are code-oriented (dbt). With Amphi, I want to offer a low-code graphical alternative t…

Appreciate the reply! Thanks for the context.

Re: Open Source Python ETL

#82

Post by ic_fly2 »

necovek wrote: post #28
anakaine wrote: post #15

Earlier quoted context omitted.

This is elitist and frankly, unhelpful. The answer to a skills shortage is not a practitioner lockdown, but policy, training, guidance and mentoring. If you're stuck in start up land and you have this issue, you have hired the wrong skills. If you're encountering this in enterprise land, your organisation, and potentially you depending on your position of influence, should be angling to improve compliance and literac…

FWIW, while I disagree with the parent comment, I don't see you arguing against it. They actually implied that you should try upskilling first — but if that fails, you shouldn't be doing ETL yourself. I mostly disagree with the parent comment because there's so many things one can easily do up to a level, and then when the going gets tough, you need to call in an expert. Eg. most people can operate a screwdriver or i…

I’m all for upskilling.

On your example, when I was an intern in a factory, I was banned from the pneumatic tools with a counter grip. Because if you use them incorrectly your finger/ hand / arm is a flesh pancake.

My solution suggestion here is definitely to empower more people to work on this by teaching them basics of data base design and enough python to write a dag in airflow.

Re: Open Source Python ETL

#84

Post by hipadev23 »

kaon_ wrote: post #57
iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

I would love to have your advice. What tool would you recommend to do straightforward ETL's as a single developer? Think of tasks like ETL-ing data from Production to Test or Local. Or quickly combining data from 2 databases to answer some business question. Six years ago I used Pentaho to do it. And it worked really well. It was easy and quick. Though maintenance was hard sometimes and it felt very dated: The javasc…

clickhouse can enable all the things you mentioned

Re: Open Source Python ETL

#86

Post by thibautdr »

C4stor wrote: post #75

It's a good idea, but from the docs it looks like the high level abstractions are wrong. If my data pipeline is "take this table, filter it, output it", I really don't want to use a "csv file input" or a "excel file output". I want to say "anything here in the pipeline that I will define that behaves like a table, apply it this transformation", so that I can swap my storage later without touching the pipeline. Same t…

Hey, not sure I get your point here. I believe the abstraction provides what you're describing. You can swap a file input with a table input without touching the rest of the components (provided you don't have major structural changes). Let me know what you meant :)

Re: Open Source Python ETL

#87

Post by jdnier »

mritchie712 wrote: post #85

If you're looking for "open source Python ETL", two things that are better options: https://dlthub.com/ https://hub.meltano.com/ we[0] use meltano in production and I'm happy with it. I've played around with dlt and it's great, just not a ton of sources yet. 0 - https://www.definite.app/

Are you able to describe what makes them better? (Honest question, I'm not familiar with either or with Amphi.)

It seems Definite's use case is focused on connecting to lots of data sources. For much smaller scale, how does Amphi compare?

Re: Open Source Python ETL

#88

Post by thibautdr »

mritchie712 wrote: post #85

If you're looking for "open source Python ETL", two things that are better options: https://dlthub.com/ https://hub.meltano.com/ we[0] use meltano in production and I'm happy with it. I've played around with dlt and it's great, just not a ton of sources yet. 0 - https://www.definite.app/

Hey, Amphi's developer here. Those two tools are great, big fan of dlt myself :)

However, Amphi is a low-code solution while those two are code-based. Also, those two focus on the ingestion part (EL) while Amphi is focusing on different ETL use-cases (file integration, data preparation, AI pipelines).

Re: Open Source Python ETL

#89

Post by mritchie712 »

jdnier wrote: post #87
mritchie712 wrote: post #85

If you're looking for "open source Python ETL", two things that are better options: https://dlthub.com/ https://hub.meltano.com/ we[0] use meltano in production and I'm happy with it. I've played around with dlt and it's great, just not a ton of sources yet. 0 - https://www.definite.app/

Are you able to describe what makes them better? (Honest question, I'm not familiar with either or with Amphi.) It seems Definite's use case is focused on connecting to lots of data sources. For much smaller scale, how does Amphi compare?

most data engineers would think of something like Fivetran when you say "ETL" (look at the ETL section here[0]).

It looks like Amphi could handle some low code transformations (the "T" in ETL), but calling it ETL feels like a stretch.

So to rephrase a bit, if you're looking for an open source, python based Fivetran alternatives, dlt and meltano would be my picks.

0 - https://mattturck.com/landscape/mad2024.pdf)

Re: Open Source Python ETL

#90

Post by mritchie712 »

thibautdr wrote: post #88
mritchie712 wrote: post #85

If you're looking for "open source Python ETL", two things that are better options: https://dlthub.com/ https://hub.meltano.com/ we[0] use meltano in production and I'm happy with it. I've played around with dlt and it's great, just not a ton of sources yet. 0 - https://www.definite.app/

Hey, Amphi's developer here. Those two tools are great, big fan of dlt myself :) However, Amphi is a low-code solution while those two are code-based. Also, those two focus on the ingestion part (EL) while Amphi is focusing on different ETL use-cases (file integration, data preparation, AI pipelines).

I understand that. I'd change the title / H1 though, "Open Source Python ETL" doesn't describe what you're building very well.

Good luck! Looks cool.

Post reply on HN