31–40 of 74 posts
I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as recommended by tidyverse) and it also requires you to code, instead of using a UI
It’s exciting to see new tooling coming out, depending on your need, watching a video or two about how Monarch runs could be helpful in assessing what tool is for you.
I try to avoid GUI-based tools as they generally don't lend themselves well to version control, code reuse, etc. There are several orchestration tools that allow you to create arbitrarily complex ETL pipelines using SQL, Python, Spark, etc. Here are links to a few: http://airflow.apache.org/ https://www.prefect.io/ https://dagster.readthedocs.io/en/0.6.7/ https://www.getdbt.com/
The python ecosystem is really good here. I do a lot of exploratory coding in ipython, though the threshold for "switch to a real editor and run git init and poetry new" is pretty low. Want to munge CSV? stdlib, or pandas( https://pandas.pydata.org/pandas-docs/stable/ ) Want to munge JSON? stdlib. Want to munge a database? pandas, stdlib, anything that speaks sqlalchemy Want to validate your json/sql/CSV/whatever and…
I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as recommended by tidyverse) and it also requires you to code, instead of using a UI
Is there really no way of processing dataframes by chunks in the R ecosystem? I'm firmly in the Python camp but I wonder how much I'm missing out on in the R world.
The dbplyr package makes it possible to use a local db table as if they are in-memory data frames (https://dbplyr.tidyverse.org/).
https://diskframe.com/ is also a pretty great solution that processes data by chunks.
Earlier quoted context omitted.
The internet being down doesn't block Alteryx at all. I routinely use it without internet. Sharing of workflows hasn't been an issue either. Understood on the loop part. You have to think on terms of vectorization for that, or use what they call batch macros.
The connection would prevent it from starting up for me because it verifies the license each time you start the program. Neither vectorisation, nor batch macros (nor iterative macros) allow you to run the same set of commands until a condition is met in Alteryx.
You're incorrect on your last statement. I'm a coder, and don't enjoy using Alteryx for more iterative/recursive operations, but I've built several macros that leverage loops and DO run the same set of commands until a condition is met. There are examples of this online.
Typically, when I run into more loop oriented operations, I use the python or R tool to write a snippet of code to perform the operation within the workflow.
I really like R / Tidyverse, but that requires that you have more memory than your datasize (especially if you are using narrow and long dataframes, such as recommended by tidyverse) and it also requires you to code, instead of using a UI
Is there really no way of processing dataframes by chunks in the R ecosystem? I'm firmly in the Python camp but I wonder how much I'm missing out on in the R world.
Earlier quoted context omitted.
to expand on my reply, I use to be a full on python / pandas advocate, but in comparison to alteryx it's slow and the best is that you can get business users to understand alteryx enough to provide value on the data cleansing / merging part
Would you say that the main reason to use a software instead of code is to onboard non-coding business partners? edit: After looking at the website, I can't help but feel that they are surfing on the Asteryx clothing brand to overprice their products...
Alteryx is definitely high-priced. It's enterprise software and priced that way. Which sucks if you're on a budget.
Looks like you're looking for ETL solutions. It's funny reading some of the replies here, you can tell who's coming from a more BI background compared to a software engineering background. Anyways, I think Alteryx does this job really well. It's great for ETL that the average Joe can pick up and learn very quickly. Sure, the UX could be improved, but it's miles better than AWS' Data Pipeline and other tools. It is a…
I totally agree with your first sentence. As a former enterprise data warehouse architect, it's interesting seeing some of the responses. Surprised I'm not seeing more Informatica and SSIS. I'll also say SSIS is a good solution for nearly any data storage platform and is included in a SQL Server license so might be a good, cheap enterprise solution if you already have SQL Server.
Structured data rarely lends itself to a GUI paradigm because its structure is arbitrary. I would argue that it would only fit with a GUI if it represents something specific that said GUI explicitly models (e.g., a very specific lump of JSON could be interpreted as a tree, etc.). As such, command line and scripting tools are vastly superior.
The model works absolutely fine with structured data. In these kinds of applications, you usually connect different nodes on a canvas, each node representing a data source, join, transformation etc. and each connection represents the flow of data.