Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What tools do you use for data munging and merging?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 74 posts

Re: Ask HN: What tools do you use for data munging and merging?

#41

Post by hprotagonist »

billfruit wrote: post #34
hprotagonist wrote: post #15

The python ecosystem is really good here. I do a lot of exploratory coding in ipython, though the threshold for "switch to a real editor and run git init and poetry new" is pretty low. Want to munge CSV? stdlib, or pandas( https://pandas.pydata.org/pandas-docs/stable/ ) Want to munge JSON? stdlib. Want to munge a database? pandas, stdlib, anything that speaks sqlalchemy Want to validate your json/sql/CSV/whatever and…

I think it is not that rich when dealing with binary data.

fsvo 'binary', i think i agree. I've written my fair share of struct.unpack-heavy code, and it's just tedious.

I also think that __str__ and __repr__ behavior on bytes is misleading at best. I do not like getting ascii values for some bytes and not others... and iterating to get base-10 ints!?

Pillow[simd], soundfile, and imageio can do nice things for image and sound data.

Re: Ask HN: What tools do you use for data munging and merging?

#43

Post by psv1 »

JPKab wrote: post #36
psv1 wrote: post #25

Earlier quoted context omitted.

The connection would prevent it from starting up for me because it verifies the license each time you start the program. Neither vectorisation, nor batch macros (nor iterative macros) allow you to run the same set of commands until a condition is met in Alteryx.

It verifies the license against a local cache. The remote licensing check only occurs upon activating a license. Once activated, there is a local cached copy it uses that I happen to know about because I accidentally deleted it once. I start up Alteryx in an air-gapped environment on a routine basis. I wouldn't use the tool if it had to check a license server every time it started. You're incorrect on your last state…

> There are examples of this online.

Can you link to any of them? I wasn't able to find anything that actually works when I needed to do this a while ago. (I needed to repeatedly hit a REST API until it returned no results.)

Re: Ask HN: What tools do you use for data munging and merging?

#44

Post by psv1 »

Xophmeister wrote: post #40
psv1 wrote: post #10

Earlier quoted context omitted.

The model works absolutely fine with structured data. In these kinds of applications, you usually connect different nodes on a canvas, each node representing a data source, join, transformation etc. and each connection represents the flow of data.

But those nodes are going to be either relatively limited in their ability, highly specific, or just expose scripting tools; the linkage between nodes is trivial. Why waste time connecting all that up visually, when a good shell will do the job?

I agree with this and I also prefer a programming language over a GUI. My point was that there isn't a limitation of GUI ETL tools that's specific to structured data.

Re: Ask HN: What tools do you use for data munging and merging?

#45

Post by Zhyl »

If I need to combine a bunch of files in a 'union' or 'concat' type fashion I will usually use UNIX tools or perl. Ditto for dropping or re-arranging columns or filtering out entries. The syntax is usually very terse and for simple tasks isn't too bad in terms of readability/

Re: Ask HN: What tools do you use for data munging and merging?

#49

Post by JPKab »

psv1 wrote: post #43
JPKab wrote: post #36

Earlier quoted context omitted.

It verifies the license against a local cache. The remote licensing check only occurs upon activating a license. Once activated, there is a local cached copy it uses that I happen to know about because I accidentally deleted it once. I start up Alteryx in an air-gapped environment on a routine basis. I wouldn't use the tool if it had to check a license server every time it started. You're incorrect on your last state…

> There are examples of this online. Can you link to any of them? I wasn't able to find anything that actually works when I needed to do this a while ago. (I needed to repeatedly hit a REST API until it returned no results.)

Absolutely.

https://community.alteryx.com/t5/Alteryx-Designer-Discussion...

Let me know if that doesn't cover your use case.

BTW, if you're hitting a REST API, you might want to try using the Python tool (it's an embedded Jupyter interface that will default to just running the pure python code when the workflow is executed without a user configuring it), just because I prefer using requests library over using the Alteryx download/curl tool myself.

Re: Ask HN: What tools do you use for data munging and merging?

#50

Post by pjot »

westonsankey wrote: post #28

I try to avoid GUI-based tools as they generally don't lend themselves well to version control, code reuse, etc. There are several orchestration tools that allow you to create arbitrarily complex ETL pipelines using SQL, Python, Spark, etc. Here are links to a few: http://airflow.apache.org/ https://www.prefect.io/ https://dagster.readthedocs.io/en/0.6.7/ https://www.getdbt.com/

+1 for dbt.

Great tool with a really active community along with it

Post reply on HN