Kestra

An open-source, infinitely-scalable orchestration platform that enables declarative workflow definition using YAML. Designed for event-driven and scheduled data workflows with rich plugin ecosystem and multi-language support for MLOps automation.

Use it when

• •Declarative workflow definition requiring minimal coding and YAML-based configuration
• •Event-driven MLOps pipelines triggered by data changes, model updates, or external events
• •Multi-language ML workflows integrating Python, R, Julia, and other data science languages
• •Scalable data orchestration requiring millions of workflow executions
• •Teams preferring visual pipeline editor with infrastructure-as-code practices
• •Dynamic resource provisioning for compute-heavy ML tasks using cloud services
• •MLOps workflows requiring integration with diverse data sources and APIs
• •Organizations needing both scheduled and real-time data processing capabilities

Watch out

• ⚠Relatively new platform with smaller community compared to established alternatives
• ⚠Learning curve for teams transitioning from code-first to declarative approaches
• ⚠Limited third-party integrations compared to mature orchestration platforms
• ⚠Documentation gaps for complex enterprise deployment scenarios
• ⚠Plugin ecosystem still developing compared to Airflow's extensive library
• ⚠Performance characteristics not yet proven at massive enterprise scale
• ⚠Requires minimum 4GiB RAM and 2vCPU resources for proper operation
• ⚠Docker-in-Docker limitations in certain cloud environments like AWS Fargate

Available in stages

Installation

Example stacks