Airflow 3.1.3 – API Server returns 500 Internal Server Error on /login once per day (session invalidation / PendingRollbackError)

[jun-roh]

Hello,

We are setting up an environment using Apache Airflow 3.1.3.
We are encountering an issue where the login session expires once per day, and after that, accessing the login page results in a 500 Internal Server Error from the API server.

This happens consistently and requires restarting the API server to recover.

Below are the error logs, configuration, and our custom AuthManager implementation.
Any help or guidance would be greatly appreciated.

Environment

•	Airflow version: 3.1.3
•	Python version: 3.12
•	Executor: CeleryExecutor
•	Database: MySQL (RDS)
•	Auth Manager: Custom SafeFabAuthManager (based on FAB)
•	Deployment: Docker Compose
•	API Server: airflow api-server --apps all --workers 1

Issue Summary

•	Login works initially
•	After ~24 hours:
•	Login session becomes invalid
•	/auth/login/ returns 500 Internal Server Error
•	Stack trace shows sqlalchemy.exc.PendingRollbackError
•	Restarting airflow-apiserver resolves the issue temporarily

Error Log (Excerpt)

sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.

The error occurs during:

/login/ [GET]
airflow.providers.fab.auth_manager.security_manager.override.load_user

Docker Compose (airflow-apiserver)

airflow-apiserver:
  command: >
    airflow api-server --host 0.0.0.0 --port 8080 --workers 1 --apps all

Database pooling configuration:

AIRFLOW__DATABASE__SQL_ALCHEMY_ENGINE_OPTIONS={
  "pool_pre_ping": true,
  "pool_recycle": 280,
  "pool_size": 100,
  "max_overflow": 20,
  "pool_timeout": 30,
  "pool_reset_on_return": "rollback"
}

Custom Auth Manager

To mitigate the issue, we implemented a custom SafeFabAuthManager that wraps DB calls and forcibly resets the session when a PendingRollbackError occurs.

class SafeFabSecurityManager(FabAirflowSecurityManagerOverride):
    def _safe_db_call(self, func, *args, **kwargs):
        try:
            return func(*args, **kwargs)
        except Exception as e:
            if isinstance(e, (SQLAlchemyError, PendingRollbackError)):
                session = self._get_session()
                if session:
                    session.rollback()
                    session.close()
                return func(*args, **kwargs)
            raise

This reduces the frequency of failures but does not fully eliminate the issue.

Questions

1.	Is this a known issue in Airflow 3.1.x related to FAB / API Server session handling?
2.	Is the API Server expected to manage SQLAlchemy sessions differently compared to the Webserver?
3.	Are there recommended settings for:
•	pool_recycle
•	pool_reset_on_return
•	session scoping in FAB auth?
4.	Would using NullPool or disabling connection reuse for the API server be recommended?

Expected Behavior

•	Login page should gracefully redirect to re-authentication
•	No 500 errors on /login
•	Expired sessions should not poison the SQLAlchemy connection pool

[potiuk]

Yes. this has been fixed in 3.1.5 - look th the changelog

[potiuk]

Also make sure to upgrade to latest Fab provider.

[jun-roh]

We are currently building our Airflow image based on
apache/airflow:3.1.5-python3.13.

If we configure Airflow with the following setting:

AIRFLOW__CORE__AUTH_MANAGER: airflow.providers.fab.auth_manager.FabAuthManager

does this mean that we can rely entirely on the default FAB-based authentication manager?

In that case, would it be correct to assume that we no longer need to implement a custom AuthManager?

More specifically, does the default FabAuthManager fully handle authentication and session management for both the Webserver and the API Server in Airflow 3.1.5?

Thank you very much for your guidance.

[potiuk]

I don't sure what you mean by "fully handle" - but I think there are some open issues for API auth wiht Fab Auth Manager - you might need to look up through issues and find out.

[potiuk]

I believe there were some workarounds posted in the meantime

[arodrber-ds]

I was also having this error, so I fixed it by setting to this version: .
So far, the problem hasn't occurred again.apache-airflow-providers-fab==3.0.2

[dsuarez]

I am also having this problem with Airflow 3.1.5. After a while (probably under the circumstances described), Airflow API Server starts returning Internal Server Error.

I am using apache/airflow:3.1.5 docker image. Is there an updated image without this problem?

[giuseppevavalaqi]

Same thing for me.

[elangovankrishna]

We recently upgraded to Airflow 3.1.5 and ran in the frequent 500 Internal Server Error from the API server, this was more prevalent on the PROD env because of more number of users, this is what i have implemented and it seems to be working

We were specifically running into this issue

issue 500 Internal Server Error with the message mismatching_state: CSRF Warning! State not equal in request and response. alongside the sqlalchemy.exc.PendingRollbackError in our Airflow Prod and Dev environments.

Database Connection Pool Resilience to stop PendingRollbackErrors / CSRF state

- name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING
  value: "True"
- name: AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE
  value: "1800"