Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What tools do you use for data munging and merging?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 74 posts

Re: Ask HN: What tools do you use for data munging and merging?

#51

Post by j88439h84 »

billfruit wrote: post #34
hprotagonist wrote: post #15

The python ecosystem is really good here. I do a lot of exploratory coding in ipython, though the threshold for "switch to a real editor and run git init and poetry new" is pretty low. Want to munge CSV? stdlib, or pandas( https://pandas.pydata.org/pandas-docs/stable/ ) Want to munge JSON? stdlib. Want to munge a database? pandas, stdlib, anything that speaks sqlalchemy Want to validate your json/sql/CSV/whatever and…

I think it is not that rich when dealing with binary data.

Sure it is. https://construct.readthedocs.io/

Re: Ask HN: What tools do you use for data munging and merging?

#52

Post by psv1 »

JPKab wrote: post #49
psv1 wrote: post #43

Earlier quoted context omitted.

> There are examples of this online. Can you link to any of them? I wasn't able to find anything that actually works when I needed to do this a while ago. (I needed to repeatedly hit a REST API until it returned no results.)

Absolutely. https://community.alteryx.com/t5/Alteryx-Designer-Discussion... Let me know if that doesn't cover your use case. BTW, if you're hitting a REST API, you might want to try using the Python tool (it's an embedded Jupyter interface that will default to just running the pure python code when the workflow is executed without a user configuring it), just because I prefer using requests library over using the Alt…

Yep, I've seen the link. Never managed to get it to run "until the iteration output stream does not receive any more records" though, nor did I see that documented anywhere.

> you might want to try using the Python tool

Yep, I just end up using Python outside of Alteryx altogether.

Re: Ask HN: What tools do you use for data munging and merging?

#53

Post by j88439h84 »

Xophmeister wrote: post #4

Structured data rarely lends itself to a GUI paradigm because its structure is arbitrary. I would argue that it would only fit with a GUI if it represents something specific that said GUI explicitly models (e.g., a very specific lump of JSON could be interpreted as a tree, etc.). As such, command line and scripting tools are vastly superior.

> Structured data rarely lends itself to a GUI paradigm because its structure is arbitrary.

I really can't tell what this means.

Re: Ask HN: What tools do you use for data munging and merging?

#56

Post by hellectronic »

hn12345 wrote: post #21

Looks like you're looking for ETL solutions. It's funny reading some of the replies here, you can tell who's coming from a more BI background compared to a software engineering background. Anyways, I think Alteryx does this job really well. It's great for ETL that the average Joe can pick up and learn very quickly. Sure, the UX could be improved, but it's miles better than AWS' Data Pipeline and other tools. It is a…

I do not search for an ETL solution, but thanks for your reply because I am using Pentaho Data Integration for ETL and reading about other tools can help :)

Re: Ask HN: What tools do you use for data munging and merging?

#57

Post by hellectronic »

cube2222 wrote: post #17

A project I'm using, which I'm also one of the authors of, is OctoSQL[1]. It allows you to transform and join data from different data sources (including databases like mysql, postgres, redis, and CSV or JSON files, more to come) using a familiar interface - plain SQL. Other than that, I like Exploratory[2]. And for nested JSON data I use another project of mine, jql[3] - an easier to use (in my opinion) lispy altern…

OctoSQL sounds interesting thanks.

Re: Ask HN: What tools do you use for data munging and merging?

#59

Post by hellectronic »

hprotagonist wrote: post #15

The python ecosystem is really good here. I do a lot of exploratory coding in ipython, though the threshold for "switch to a real editor and run git init and poetry new" is pretty low. Want to munge CSV? stdlib, or pandas( https://pandas.pydata.org/pandas-docs/stable/ ) Want to munge JSON? stdlib. Want to munge a database? pandas, stdlib, anything that speaks sqlalchemy Want to validate your json/sql/CSV/whatever and…

Thank you for the list.
Post reply on HN