>nobody wants to run multiple orchestrators
Straw man argument.
>I would expect someone who's seriously writing a new orchestrator in 2024 to cite the alternatives, their shortcomings, and how you intend to address them. Bonus points if you make a neat little table.
Did you even try to read the docs before you launched this critical diatribe?
From the docs (https://engineering.nike.com/koheesio/latest/tutorials/onboa...):
Advantages of Koheesio
Using Koheesio instead of raw Spark has several advantages:
Modularity: Each step in the pipeline (reading, transformation, writing) is encapsulated in its own class, making the code easier to understand and maintain.
Reusability: Steps can be reused across different tasks, reducing code duplication.
Testability: Each step can be tested independently, making it easier to write unit tests.
Flexibility: The behavior of a task can be customized using a Context class.
Consistency: Koheesio enforces a consistent structure for data processing tasks, making it easier for new developers to understand the codebase.
Error Handling: Koheesio provides a consistent way to handle errors and exceptions in data processing tasks.
Logging: Koheesio provides a consistent way to log information and errors in data processing tasks.
In contrast, using the plain PySpark API for transformations can lead to more verbose and less structured code, which can be harder to understand, maintain, and test. It also doesn't provide the same level of error handling, logging, and flexibility as the Koheesio Transform class.
It took me less than 15 seconds to find the solution to the problem you propose. How long did it take you to formulate your critique? Do you perhaps just have a prejudice against Nike (corporate haze), or is it an investment in a 'competing orchestrator' that is clouding your judgement?