The modern data workflow orchestration platform – Apache DolphinScheduler

Apache DolphinScheduler is a cloud-native platform with powerful visual interfaces. Compared with other open source orchestration platform, it has lower latency, higher concurrency and more stability, which has been produced and used by more than 1000 companies, including IBM, Accenture, Walmart, VMware, SheIn, Cisco.

The key features for DolphinScheduler are as follows:

- Easy to deploy, we provide 4 ways to deploy, such as Standalone deployment,Cluster deployment,Docker / Kubernetes deployment and Rainbond deployment

- Easy to use, there are 3 ways to create workflows:

  - Visually, create tasks by dragging and dropping tasks

  - Creating workflows by PyDolphinScheduler(Python way)

  - Creating workflows through Open API

- Highly Reliable, DolphinScheduler uses a decentralized multi-master and multi-worker architecture, which naturally supports horizontal scaling and high availability

- High performance, its performance is N times faster than other orchestration platform and it can support tens of millions of tasks per day

- Supports multi-tenancy

- Supports various task types: Shell, MR, Spark, SQL (MySQL, PostgreSQL, Hive, Spark SQL), Python, Procedure, Sub_Workflow, Http, K8s, Jupyter, MLflow, SageMaker, DVC, Pytorch, Amazon EMR, etc

Apache DolphinScheduler provides a distributed and easy to expand visual workflow task scheduling open-source platform. It is suitable for enterprise-level scenarios. It provides a solution to visualize operation tasks, workflows, and the entire data processing procedures.

Apache DolphinScheduler aims to solve complex big data task dependencies and to trigger relationships in data OPS orchestration for various big data applications. Solves the intricate dependencies of data R&D ETL and the inability to monitor the health status of tasks. DolphinScheduler assembles tasks in the Directed Acyclic Graph (DAG) streaming mode, which can monitor the execution status of tasks in time, and supports operations like retry, recovery failure from specified nodes, pause, resume, and kill tasks, etc.

A good view on big data modern stack, very interesting.

Just migrated from Airflow to Dolphinscheduler because of airflow's bad performance. Latency is high while CPU usage is only 10%. I have to create many instances of airflow while 1 instance of Dolphin cluster works well. It is time to try some new orchestration tools beside airflow, oozie.

My task dolph is the best scheduling system I have ever used