Prevent ack timeout from dropping callback ids

[EvanDietzMorris]

Keep callback ids for lookups that time out waiting for the ACK

Problem

I noticed this running helms deep on the dev ARS on Sterling. The dev instance of
Retriever was responding very slowly for some reason and it surfaced this issue.

When submitting an /asyncquery to Retriever, bte and aragorn lookup workers
use httpx.AsyncClient(timeout=20), so submission times over that boundary
raise ReadTimeout, but it's possible for Retriever to take over 20s to respond
to the initial async query without actually failing.

The workers treat every exception during that call as "submission failed" and delete the
callback id. But in these cases, Retriever has the query and is running it and it doesn't know
Shepherd already gave up. When it POSTs the results, /aragorn/callback/<id>
finds no callback id mapping and returns 500. Then the lookup still waits out its full deadline
for results it had already prevented being accepted.

Fix

• A new AsyncResponse.in_flight flag distinguishes "sent but not ACKed" from
  "never reached the service."
• The fan-out loop keeps the callback id for in_flight submissions and logs a
  warning instead of calling remove_callback_id. The lookup's existing
  deadline still bounds the wait.
• The hard-coded timeout=20 (and the non-infer branch's timeout=100) are
  replaced by a new setting, kg_retrieval_submit_timeout (default 100s).
• Error strings for ack timeouts and connection errors are distinguished
• BTE's run_async_lookup now records exceptions on its span, matching
  Aragorn.

Applied identically to workers/aragorn_lookup and workers/bte_lookup.

Deployment

KG_RETRIEVAL_SUBMIT_TIMEOUT is a new, defaulted env var but was not added to deployment settings etc.
Nothing needs to change in compose or Helm unless a different value is wanted.

Future Considerations

success of an /asyncquery ACK remains status_code == 200, per TRAPI.
Technically, Retriever should probably be returning 202 instead of 200.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.09%. Comparing base (c7e0954) to head (ff968dc).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Coverage Δ
shepherd_utils/config.py	95.57% <100.00%> (+0.03%)	⬆️
workers/aragorn_lookup/worker.py	80.09% <100.00%> (+3.65%)	⬆️
workers/bte_lookup/worker.py	86.02% <100.00%> (+0.57%)	⬆️

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9f5cd1e...ff968dc. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.