Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What tools do you use for data munging and merging?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 74 posts

Re: Ask HN: What tools do you use for data munging and merging?

#61

Post by JPKab »

psv1 wrote: post #52
JPKab wrote: post #49

Earlier quoted context omitted.

Absolutely. https://community.alteryx.com/t5/Alteryx-Designer-Discussion... Let me know if that doesn't cover your use case. BTW, if you're hitting a REST API, you might want to try using the Python tool (it's an embedded Jupyter interface that will default to just running the pure python code when the workflow is executed without a user configuring it), just because I prefer using requests library over using the Alt…

Yep, I've seen the link. Never managed to get it to run "until the iteration output stream does not receive any more records" though, nor did I see that documented anywhere. > you might want to try using the Python tool Yep, I just end up using Python outside of Alteryx altogether.

Yeah, it all comes down to your needs and the use case. If there is a lot of value in allowing non-coders to leverage your work, and more importantly, modify pieces of it, then a tool like Alteryx is worth it. However, if you can just write a Python script, and aren't constantly being hassled to update it for whatever data source or new need, then just go that route. Otherwise, if you don't empower the non-tech users, eventually you'll produce enough stuff that you'll end up spending an increasing percentage of your time maintaining and dealing with your own products. For me, that's been the biggest benefit. Nothing better than telling somebody to just switch the input on a workflow to a db now that the file isn't being sent directly, instead of having to do it for them.

Re: Ask HN: What tools do you use for data munging and merging?

#62

Post by kmerrol »

mipmap04 wrote: post #30
hn12345 wrote: post #21

Looks like you're looking for ETL solutions. It's funny reading some of the replies here, you can tell who's coming from a more BI background compared to a software engineering background. Anyways, I think Alteryx does this job really well. It's great for ETL that the average Joe can pick up and learn very quickly. Sure, the UX could be improved, but it's miles better than AWS' Data Pipeline and other tools. It is a…

I totally agree with your first sentence. As a former enterprise data warehouse architect, it's interesting seeing some of the responses. Surprised I'm not seeing more Informatica and SSIS. I'll also say SSIS is a good solution for nearly any data storage platform and is included in a SQL Server license so might be a good, cheap enterprise solution if you already have SQL Server.

From my experience SSIS suffers mightily from the Write-Once Read Only problem as well as huge manual click-fests when package metadata changes. That and a lack of development from MSFT with obvious direction towards ADF means SSIS is not really great for modern data architecture.

If you're a MSFT shop then sure and if you're willing to drop the box-line GUI and move to .NET assembly-based packages even more so, but I've spent enough of my life keeping SSIS running and not going back.

Re: Ask HN: What tools do you use for data munging and merging?

#64

Post by zzo38computer »

I use SQLite myself, although I would like that those who provide the data should make the data available in a format such as CSV, or TSV, or a SQLite database, or to have some sort of protocol which a SQLite virtual table module can be made to access any data using such a protocol, so that it is possible to do so without having to deal with complicated stuff in a web browser or whatever else it may be, sometimes which is not even compatible with my computer. SQLite is common on all computers, I think, and CSV or TSV will be simple enough to work with many programs, so it should do.

Re: Ask HN: What tools do you use for data munging and merging?

#66

Post by livingmargot »

stuxnet79 wrote: post #31
wodenokoto wrote: post #18

I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as recommended by tidyverse) and it also requires you to code, instead of using a UI

Is there really no way of processing dataframes by chunks in the R ecosystem? I'm firmly in the Python camp but I wonder how much I'm missing out on in the R world.

Sorry to butt in, but could you point me to a resource on processing data in chunks using Python?

Re: Ask HN: What tools do you use for data munging and merging?

#68

Post by eyeball »

westonsankey wrote: post #28

I try to avoid GUI-based tools as they generally don't lend themselves well to version control, code reuse, etc. There are several orchestration tools that allow you to create arbitrarily complex ETL pipelines using SQL, Python, Spark, etc. Here are links to a few: http://airflow.apache.org/ https://www.prefect.io/ https://dagster.readthedocs.io/en/0.6.7/ https://www.getdbt.com/

Dbt looks awesome. Too bad no oracle support.

Re: Ask HN: What tools do you use for data munging and merging?

#69

Post by dspillett »

131012 wrote: post #13
mgaitan09 wrote: post #5

Earlier quoted context omitted.

to expand on my reply, I use to be a full on python / pandas advocate, but in comparison to alteryx it's slow and the best is that you can get business users to understand alteryx enough to provide value on the data cleansing / merging part

Would you say that the main reason to use a software instead of code is to onboard non-coding business partners? edit: After looking at the website, I can't help but feel that they are surfing on the Asteryx clothing brand to overprice their products...

> Would you say that the main reason to use a software instead of code is to onboard non-coding business partners

That is usually a key selling point of GUI based ETL solutions. Relatively non-technical users, that know little of coding but do understand their data and what they want out of it, can put together basic (and sometimes not so basic) data flows and debug them.

Re: Ask HN: What tools do you use for data munging and merging?

#70

Post by santa_boy »

data.table has both an R and Python version. The python is version getting pretty mature now.

Munging and Merging pretty large datasets using data.table is very efficient and intuitive after a bit of practice.

The vignette in R is very good. Datacamp has a course (I think little dated). Also a search for videos by Matt Dowle and Arun Srinivasan can give a very good quick idea of data.table's capabilities. I think both are involved in some capacity with h2o.ai which seems to be pretty good and on my to-do list.

Desktop coding using Rstudio is a wonderful experience with the R version.

Post reply on HN