Ask HN: What tools do you use for data munging and merging?

#1

Post by hellectronic » Mon, Jan 13, 2020, 1:10 PM UTC

Hello,

what tools do you use or know about, e.g. something like Talend DataPreparation (https://www.talend.com/products/data-preparation) ?

I found OpenRefine (http://openrefine.org), which is browser based like Talend DataPreparation.

It would be nice to have a "true" desktop application :)

Re: Ask HN: What tools do you use for data munging and merging?

#2

Post by tixocloud » Mon, Jan 13, 2020, 2:26 PM UTC

There is an incredible ecosystem of tools used for "data preparation" that it almost always necessitates defining what the usecase and workflow would be first. There are tools from UI-based to command-line-based.

Tools: Avora, FiveTran, Informatica, Excel, Alteryx, Datameer, Tableau Prep, etc. ... list could go on and on.

Re: Ask HN: What tools do you use for data munging and merging?

#3

Post by mgaitan09 » Mon, Jan 13, 2020, 2:27 PM UTC

alteryx

Re: Ask HN: What tools do you use for data munging and merging?

#4

Post by Xophmeister » Mon, Jan 13, 2020, 2:27 PM UTC

Structured data rarely lends itself to a GUI paradigm because its structure is arbitrary. I would argue that it would only fit with a GUI if it represents something specific that said GUI explicitly models (e.g., a very specific lump of JSON could be interpreted as a tree, etc.). As such, command line and scripting tools are vastly superior.

Re: Ask HN: What tools do you use for data munging and merging?

#5

Post by mgaitan09 » Mon, Jan 13, 2020, 2:29 PM UTC

mgaitan09 wrote: post #3

alteryx

to expand on my reply, I use to be a full on python / pandas advocate, but in comparison to alteryx it's slow and the best is that you can get business users to understand alteryx enough to provide value on the data cleansing / merging part

Re: Ask HN: What tools do you use for data munging and merging?

#6

Post by 2data222 » Mon, Jan 13, 2020, 2:32 PM UTC

We use Talend. It does have a gigantic Java-based Windows desktop application. It's pretty powerful and I don't hate it. We looked at MuleSoft as well but it's not ready for Enterprise prime-time like we need it to be.

The data preparation Wikipedia page mentions Talend by name along with Paxata, Trifacta, Alteryx, and Ataccama.

Re: Ask HN: What tools do you use for data munging and merging?

#7

Post by cmollis » Mon, Jan 13, 2020, 2:32 PM UTC

Usually spark, but it depends on what the source data is. Whether it needs more preprocessing, etc. for json, there are some good command- line tools available for stripping, flattening, if u need to do that. Spark can usually handle most of the data sources.

Re: Ask HN: What tools do you use for data munging and merging?

#8

Post by apohn » Mon, Jan 13, 2020, 2:34 PM UTC

I used KNIME at a previous job. Open source desktop application is free. Server is paid. Competes with other workflow based GUI tools such as Alteryx. I liked it a lot.

Re: Ask HN: What tools do you use for data munging and merging?

#9

Post by hkchad » Mon, Jan 13, 2020, 2:40 PM UTC

It's been a few years but I use to use Altova suite specifically MapForce (https://www.altova.com/mapforce) DiffDog, etc. Like I said it's been a few years but it was super powerful, this was back in my XML days when building stylesheets was a big part of my job.

Re: Ask HN: What tools do you use for data munging and merging?

#10

Post by psv1 » Mon, Jan 13, 2020, 2:41 PM UTC

Xophmeister wrote: post #4

Structured data rarely lends itself to a GUI paradigm because its structure is arbitrary. I would argue that it would only fit with a GUI if it represents something specific that said GUI explicitly models (e.g., a very specific lump of JSON could be interpreted as a tree, etc.). As such, command line and scripting tools are vastly superior.

The model works absolutely fine with structured data. In these kinds of applications, you usually connect different nodes on a canvas, each node representing a data source, join, transformation etc. and each connection represents the flow of data.