Dagster

Dagster is an orchestration platform for developing, producing, and observing data assets.

Use it when

• •You want to create workflows as DAGs (Direct Acyclic Graphs).
• •You need flexibility (manual, scheduled, modify individual jobs according to schedule) when running DAGs.
• •You want to define inputs and output for individual jobs more explicitly to streamline data movement between tasks.
• •You want an easy way to create dynamic workflows.
• •You want to generate pipelines as Python code.
• •You want to automate ML workflows.
• •You want to separately abstract compute and storage.

Available in stages

Installation

Example stacks