Ask HN: What tools do you use for data munging and merging? • Hacker News Board

Re: Ask HN: What tools do you use for data munging and merging?

#71

Post by drewbanin » Tue, Jan 14, 2020, 6:00 PM UTC

eyeball wrote: post #68

westonsankey wrote: post #28

I try to avoid GUI-based tools as they generally don't lend themselves well to version control, code reuse, etc. There are several orchestration tools that allow you to create arbitrarily complex ETL pipelines using SQL, Python, Spark, etc. Here are links to a few: http://airflow.apache.org/ https://www.prefect.io/ https://dagster.readthedocs.io/en/0.6.7/ https://www.getdbt.com/

Dbt looks awesome. Too bad no oracle support.

Hi there - I run product for dbt - happy to chat more about Oracle support if you're interested. There are a handful of community-supported dbt plugins out in the wild. Oracle could very well be one of the next ones. Check some of these out to see what's involved in building a database plugin:

- https://github.com/fishtown-analytics/dbt-spark

- https://github.com/fishtown-analytics/dbt-presto

- https://github.com/mikaelene/dbt-sqlserver

- https://github.com/jacobm001/dbt-mssql

Re: Ask HN: What tools do you use for data munging and merging?

#72

Post by eyeball » Wed, Jan 15, 2020, 1:08 AM UTC

drewbanin wrote: post #71

eyeball wrote: post #68

Earlier quoted context omitted.

Dbt looks awesome. Too bad no oracle support.

Hi there - I run product for dbt - happy to chat more about Oracle support if you're interested. There are a handful of community-supported dbt plugins out in the wild. Oracle could very well be one of the next ones. Check some of these out to see what's involved in building a database plugin: - https://github.com/fishtown-analytics/dbt-spark - https://github.com/fishtown-analytics/dbt-presto - https://github.com/mik…

cool. would be an interested user, but building a plug in ... beyond my team's ability.

Re: Ask HN: What tools do you use for data munging and merging?

#73

Post by cube2222 » Wed, Jan 15, 2020, 9:12 PM UTC

rhombocombus wrote: post #65

jhoechtl wrote: post #42

Octosql https://github.com/cube2222/octosql

The installation from the docs on the readme is broken!

Just to put it here on HN, the installation command is corrected now. You need to have go modules turned on to use go get for installation.

Re: Ask HN: What tools do you use for data munging and merging?

#74

Post by hermitcrab » Sun, Jan 19, 2020, 10:22 PM UTC

I have recently released a Windows/Mac tool for data munging: https://www.easydatatransform.com

It is a GUI based solution with an emphasis on ease of use, aimed at people who aren't coders or data science professionals. For example a marketer who has several different Excel and CSV files they need to merge, clean, filter and restructure to create a report. Any feedback would be very welcome.