Is the DAG Processor needed if our dags are static?

[DonHaul]

Hello we have in our setup an helmchart with our custom docker image that has the dags statically defined. as such they will never change. Only when we eventually do a new release the week after.

For such a case does it make sense:

• min_file_process_interval to be a very big number
• refresh-interval to be a very big number
• or simply setting parsing-processes to 0?

[Urus1201]

The DAG Processor is still needed even for static DAGs — but you can make it nearly invisible. Here's why, and the right config to use:

Why you still need it

Airflow's scheduler does not read DAG Python files directly — it reads from serialized DAGs stored in the metastore. The DAG Processor's job is to:

1. Parse DAG files and write their serialized form to the DB
2. Detect newly added or removed DAGs
3. Keep dag_run and task_instance records in sync with the DAG structure

Without it, the scheduler has no DAG definitions to work with (even if your DAGs never change).

Can you set parsing_processes: 0?

In Airflow 3, setting parsing_processes: 0 disables DAG file parsing entirely. This means:

• No new DAGs are discovered after startup
• No DAG structure changes are reflected
• If the metastore already has serialized DAGs from a previous parse, they continue to run

For a strict "update only on new release" workflow this is technically viable, but then you must trigger a processor restart (or set parsing_processes back to a non-zero value briefly) as part of your release process to re-parse the updated DAGs.

Recommended configuration for static DAGs

[dag_processor]
# Re-parse at most once per hour (DAGs never change between releases)
min_file_process_interval = 3600

# Only re-scan for new/removed files every 10 minutes
refresh_interval = 600

# Keep stale threshold high (don't mark DAGs stale too quickly)
stale_dag_threshold = 7200

# Only scan files modified since last scan (huge win for static setups)
file_parsing_sort_mode = modified_time

# Use fewer processes since reparsing is rare
parsing_processes = 2

[core]
# Skip checking imports at discovery time — safe for known-good static DAGs
dag_discovery_safe_mode = False

With file_parsing_sort_mode = modified_time, Airflow will only re-parse files whose mtime has changed since the last scan. For truly static DAGs, this means almost zero re-parsing overhead between releases — just the initial parse at startup.

In your Helm chart release workflow

Since you rebuild the Docker image on each release (new mtime on all files), the processor will re-parse everything on the next pod start. This is exactly the right behavior: parse once at startup, then idle until the next release.

[potiuk]

Are you validating those answers @Urus1201 ? Are you sure this entirely AI-generated answer is correct? I think there are some factual things missing or wrong - but I will let you look it up and ask your agent to find out them and correct.

Looking forward to those answers actually validated by human.