Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open Source Python ETL

amphi.ai

Post reply on HN
Print view View original

91–100 of 127 posts

Re: Open Source Python ETL

#91

Post by hbarka »

teej wrote: post #37
banku_brougham wrote: post #25

Earlier quoted context omitted.

What is your estimate of the tine required to begin building on the new(old) foundation, discover all the problems after a large investment of effort, and then declare to all that ita the wrong path? My personal pain is Tableau trying to move uo the stack into data orchestration and compute. Code is completely inaccessible.

Don’t build ETL on Tableau. They haven’t made meaningful product progress in 10 years and completely missed the changes in data transformation. They are playing catch-up, they don’t understand where the world is moving.

Tableau, once full of creatives, has been gutted by Salesforce since the acquisition.

Re: Open Source Python ETL

#95

Post by hermitcrab »

ic_fly2 wrote: post #13

With all the data issues strong quality and normalisation I often get the impression that enabling more people with non CS backgrounds to do this work is not necessarily a good thing. In other words, if writing python and sql is the skill requirement that stops you from making an etl pipeline, maybe do something else.

So if Bob from accounts needs a new report generating, he has to wait for 6 month for an IT guy to do it? Who probably won't do a very good job, because he doesn't understand what Bob needs as well as Bob does? Bob is going to hack something horrific together in Excel instead. Better surely to let Bob have a GUI point and click tool more appropriate to the job?

Re: Open Source Python ETL

#96

Post by mulmen »

bigger_cheese wrote: post #32
hbarka wrote: post #24

Earlier quoted context omitted.

>> Code as ETL was a huge industry shift No it’s not. Try and see what banks, retail, manufacturing, various large enterprises still use. They need scale, observability, modularity, and maintainability.

I work in manufacturing (large industrial plant) and the data processes we have are honestly not great - mostly it is because there are a heap of legacy system and not a lot of commonality between our data sources we have a hideous mashup of Oracle, DB2, Microsoft SQL Server etc and different versions of the different databases. There's also more bespoke industry stuff like time series historians and SCADA systems/PL…

I have no experience in the manufacturing domain but it fascinates me as a data engineer. I do have experience building data lakes at scale with sub-day (microbatch/“realtime”) latency and with disparate sources. I don’t think this needs to be as complicated or painful as you expect but I don’t know enough about your data or needs to be sure. If you want to discuss specifics send me an email at the domain in my profile, I’d love to know more.

Re: Open Source Python ETL

#97

Post by NortySpock »

bigger_cheese wrote: post #32
hbarka wrote: post #24

Earlier quoted context omitted.

>> Code as ETL was a huge industry shift No it’s not. Try and see what banks, retail, manufacturing, various large enterprises still use. They need scale, observability, modularity, and maintainability.

I work in manufacturing (large industrial plant) and the data processes we have are honestly not great - mostly it is because there are a heap of legacy system and not a lot of commonality between our data sources we have a hideous mashup of Oracle, DB2, Microsoft SQL Server etc and different versions of the different databases. There's also more bespoke industry stuff like time series historians and SCADA systems/PL…

I just started using sqlglot to convert Microsoft SQL Server code to Databricks SQL, and it has been able to automate 80% of the translation (assuming it's just a select statement). You might take a look.

https://github.com/tobymao/sqlglot

Re: Open Source Python ETL

#98

Post by uneekname »

lma21 wrote: post #83
jamesblonde wrote: post #50

#dang The title needs changing - it's not open-source, it is license ELv2 - Elastic License v2.

isn't the code available here? https://github.com/amphi-ai/amphi-etl what makes it not OSS?

That is source available, not open source. The term "open-source" is widely used to describe software that is licensed using a specific set of software licenses that grant certain freedoms to users. You can read more here[0]

[0] https://opensource.org/licenses

Re: Open Source Python ETL

#99

Post by NortySpock »

kaon_ wrote: post #57
iblaine wrote: post #10

Low code ETL tools (informatica, Appworx, talend, pentaho, ssis) were the original services for ELT/ETL. A lot of progress was made to go towards ETL-as-code starting with Airflow/Luigi. Going back to low code seems backwards as this point. (I have used all of the above tools in my 15+ yr career. Code as ETL was a huge industry shift)

I would love to have your advice. What tool would you recommend to do straightforward ETL's as a single developer? Think of tasks like ETL-ing data from Production to Test or Local. Or quickly combining data from 2 databases to answer some business question. Six years ago I used Pentaho to do it. And it worked really well. It was easy and quick. Though maintenance was hard sometimes and it felt very dated: The javasc…

I've gotten some quick wins with Benthos (now RedPanda Connect) but I agree it's an unsolved problem as there are typically gotchas.

If you can get a true CDC stream from the database to analytics, that would be ideal, but when that isn't available you spend 100x more time trying to bodge together an equivalent batch/retry system.

Re: Open Source Python ETL

#100

Post by CactusOnFire »

thegiogi wrote: post #44
tiew9Vii wrote: post #19

Earlier quoted context omitted.

You see this a lot. A new generation comes in to relearn all the old lessons. It’s also happening on the frontend “server side rendering is bad, it’s slow” to the latest frontend frameworks and thought leaders “you should be using SSR” (server side rendering) ETL is hot again with machine learning. Companies have massive amounts of data they need to get in shape for models. The promise of a GUI anyone can use with li…

But that’s the point - it’s just a promise. I have a similar career as the top comment, and turns out that Low code approaches only work when people who understand code use them. Be that Python or sql, real use cases are not the toy examples shown in all typical introductions to these tools. That said, if this thing is customizable enough, a good data engineer can prepare canned steps that fit the general structure o…

I imagine the use-case for low-code tools is when your ratio of "business experts" heavily outweighs programmers, and the cost of inefficient dev processes/tech debt is less than the cost of waiting to onboard people with a coding background.
Post reply on HN