QPU auth management - #79
Conversation
SBOM vulnerability checkTotal vulnerabilities observed: 1
|
| Vulnerability | Severity | Component(s) | Reference |
|---|---|---|---|
CVE-2025-65896 |
CRITICAL | asyncmy@0.2.11 |
link |
warden-sbom-pg.cdx.json
- No vulnerabilities listed in this SBOM.
warden-sbom.cdx.json
- No vulnerabilities listed in this SBOM.
428aca8 to
514b4e4
Compare
Cache for half the lifespan (with a WARNING) instead of disabling the cache entirely when expires_in <= leeway_s. Also catch AttributeError alongside ValueError when parsing a non-object token error body, narrow the unlocked-cache ponytail comment to the async path where the race is actually reachable, tighten config.py's auth_flow typing, note that QPUClientRequestError subclasses can propagate through retry(), and add coverage for the no-auth-header and POST-401-replay paths.
514b4e4 to
a84c49e
Compare
badtst
left a comment
There was a problem hiding this comment.
LGTM from my (limited) understanding of Keycloak
| # auth: | ||
| # # Keycloak base URL (not the token endpoint; the OIDC path is appended). | ||
| # url: http://keycloak:8080 | ||
| # realm: pasqos |
There was a problem hiding this comment.
Should we leave the pasqos example name visible in user-facing documentation ? At the same time it is the correct realm name
| request.headers["Authorization"] = f"Bearer {self._sync_token()}" | ||
| response = yield request | ||
| if response.status_code == httpx.codes.UNAUTHORIZED: | ||
| logger.info("QPU API returned 401, refreshing token and retrying once") |
There was a problem hiding this comment.
Should those logs be INFO or should they be hidden in DEBUG level, token failures are documented by raising an exception
| return self.tls_verify | ||
|
|
||
| @property | ||
| def auth_flow(self) -> httpx.Auth | None: |
There was a problem hiding this comment.
nit: maybe rename to something like qpu_auth to not avoid confusion with the auth_flow method of httpx.Auth
| url: str | ||
| realm: str = "pasqos" | ||
| # OIDC client_id | ||
| id: str | ||
| # OIDC client_secret. Provide via WARDEN_QPU_AUTH_SECRET, never in YAML. | ||
| secret: str | ||
| # Refresh this many seconds before the token actually expires. | ||
| leeway_s: float = 30 |
There was a problem hiding this comment.
nit: Avoid in code config default values ?
| token_requests = [r for r in httpx_mock.get_requests() if str(r.url) == TOKEN_URL] | ||
| assert len(token_requests) == 1 |
There was a problem hiding this comment.
nit: IIRC if a mock response does not have is_reusable set to True it will raise an error if it is called more than once (httpx.TimeoutException because httpx_mock will not re-send the response to an already matched request)
If a response is not requestes, an error is raised even if the test passes
| token_requests = [r for r in httpx_mock.get_requests() if str(r.url) == TOKEN_URL] | |
| assert len(token_requests) == 1 |
| qpu_requests = [r for r in httpx_mock.get_requests() if str(r.url) == QPU_URL] | ||
| assert len(qpu_requests) == 2 |
There was a problem hiding this comment.
nit: IIRC not needed
| qpu_requests = [r for r in httpx_mock.get_requests() if str(r.url) == QPU_URL] | |
| assert len(qpu_requests) == 2 |
| assert second.request.headers["Authorization"] == "Bearer tok-2" | ||
|
|
||
|
|
||
| def test_401_triggers_one_refresh_and_one_retry(httpx_mock: HTTPXMock, auth_conf): |
There was a problem hiding this comment.
Isn't there a way to know that the 401 is specifically triggered by the token being stale ?
| # expires_in 30 with the default leeway_s 30 would clamp to 0 without the | ||
| # half-lifespan fallback, disabling the cache entirely and forcing a | ||
| # Keycloak round-trip on every request. |
There was a problem hiding this comment.
nit: same comment on leeway as above
| token_requests = [r for r in httpx_mock.get_requests() if str(r.url) == TOKEN_URL] | ||
| assert len(token_requests) == 1 |
| assert response.status_code == 200 | ||
| assert response.request.headers["Authorization"] == "Bearer fresh" | ||
| qpu_requests = [r for r in httpx_mock.get_requests() if str(r.url) == QPU_URL] | ||
| assert len(qpu_requests) == 2 |
| # Deliberately generous: this budget is only ever spent by a test that is | ||
| # already failing, so it costs nothing on the happy path. Tight per-test budgets | ||
| # turned a slow CI runner into a flake instead of catching anything. | ||
| JOB_WAIT_TIMEOUT_S = 30 |
There was a problem hiding this comment.
Maybe it is a bit long for a test fail in a local environment. Maybe we can set it to something lower like 10 in a local dev environment and switch it to 30 in CI ?
| JOB_WAIT_TIMEOUT_S = 30 | |
| JOB_WAIT_TIMEOUT_S = 30 if "CI" in os.environ else 10 |
| assert self._token is not None | ||
| return self._token | ||
| url, data = self._token_request() | ||
| with httpx.Client(verify=self.verify) as client: |
There was a problem hiding this comment.
We could reuse the same http client to reuse http pool connection across token requests
| assert self._token is not None | ||
| return self._token | ||
| url, data = self._token_request() | ||
| async with httpx.AsyncClient(verify=self.verify) as client: |
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| TERMINAL_STATUSES: tuple[JobStatus, ...] = ("ERROR", "DONE", "CANCELED") |
There was a problem hiding this comment.
nit: why not use the JobStatus enum values instead of strings?
| if self._status in TERMINAL_STATUSES and not was_terminal: | ||
| # Logged here rather than by the caller so that the closing line is | ||
| # part of the same JobUpdate - hence the same transaction - as the | ||
| # terminal status. Flushed separately, the DB would briefly hold a | ||
| # finished job whose logs are truncated, and anything that stops | ||
| # polling once the status is terminal reads incomplete logs. | ||
| logger.info("Job execution ended with status '%s'", self._status) |
There was a problem hiding this comment.
nit: This final log line was moved out of line 169, so it won't appear for job who time out and the cancelation fail because the same logic has not been added to to_error method. Is that intentional?
| expires_in = float(payload.get("expires_in", 0)) | ||
| self._token = token | ||
| ttl = max(expires_in - self.conf.leeway_s, expires_in / 2) | ||
| if expires_in <= self.conf.leeway_s: |
There was a problem hiding this comment.
The warning log is not emitted for all cases where we use ttl=expires_in/2.
e.g. if expires_in=40, leeway=30 then ttl=20 and yet expires_in>leeway
| if expires_in <= self.conf.leeway_s: | |
| if ttl > expires_in - self.conf.leeway_s: |
| NotRetriedHTTPStatus: If the HTTP request returns with a non-retryable error code. | ||
| MaxRetryError: If the maximum number of retries without success has been reached. | ||
| QPUClientRequestError: Any subclass already classified as non-retryable | ||
| by the wrapped function (e.g. TokenRequestError) propagates unchanged. |
There was a problem hiding this comment.
That's not true in the case of no_retry=True, see line 82
|
LGTM, I haven't tested locally tbh but I believe Thomas has already confirmed the proper integration with local pasqos using keycloak |
|
I finally tested locally and confirmed this works as expected 👍 |
No description provided.