Earlier quoted context omitted.
Puppet and the like are state enforcement tools, not task schedulers. As someone who used to run a 6000 core farm in 2007 (its not 25,000) I can tell you that puppet isn't going to help task placement. It can create machine that will run a certain app, but without some heavy programming it'll never balance or detect need and respond sensibly.
DevOps tools are rapidly evolving. Now that they've basically finished with machine provisioning, dependency management and orchestration, all major DevOps automation frameworks are going into managing reactive infrastructure. Enforcing a task schedule is just another form of state enforcement. These are all pretty similar: - configuration management: ensure package oracle-java-8 is installed on machines A,B,C with t…
For distributed cron, we use jenkins. Which has the advantage of keeping "build history"
For task placement we use alfred (https://renderman.pixar.com/resources/RPS_13.5/rps_manuals/a...) yup its old. However it works like a champ, and its fast. (as in it'll dispatch thousands of tasks a second.)
You do hit a limit when you go over 6000 "slots" (each slot accepts one task, and the main dispatcher is single threaded). Dispatching is simple and task building has simple syntax that easily grows to thousands of tasks in one job. monitoring is also simple, as each task ships logs and exit status back to the dispatcher. It also has mechanisms to cope with bad/slow/unhappy machines.