Airflow control the parallelism and concurrency (draw)

Airflow control the parallelism and concurrency

Airflow configuration to allow for a larger scheduling capacity and frequency:

DAGs have configurations that improve efficiency:

Operators or tasks also have configurations that improves efficiency and scheduling priority:

  • max_active_tis_per_dag: This parameter controls the number of concurrent running task instances across dag_runs per task.
  • pool: See Pools.
  • priority_weight: See Priority Weights.
  • queue: See Queues for CeleryExecutor deployments only.

Credits

DataData, Data Engineering, Airflow