Add shared Decision model support for Foundry evaluators

[Salma El-Shafey (salma-elshafey)]

Description

Adds shared Microsoft Foundry Decision-model support for registry evaluator assets alongside the existing Prompty path. Bundled SDK evaluators are not newly enabled for Decision models.

• Provides shared runners, strategies, native answer parsing, composite scoring, and common exports; recognizes Output Quality and Tool Use Quality aggregate/member metrics.
• Requires an explicit API key or caller-owned Entra credential for Decision. API keys take precedence; callers can reuse a synchronous credential across sync/batch calls and close it afterward. Prompty authentication is unchanged.
• Enforces HTTPS except for loopback, plus timeout, retry, cancellation, and request/response byte controls. Decision defaults to 120 seconds. Native async HTTP stops response reads at deadline/cancellation; TLS uses Python/OpenSSL trust settings such as SSL_CERT_FILE, not REQUESTS_CA_BUNDLE.
• Validates score probability sums with precision-aware tolerances: hundredth-precision values allow 0.005 rounding error per level plus 1e-12; other values use an absolute 0.001 tolerance. Invalid distributions raise a system-classified EvaluationException; probabilities are not normalized and native mean consistency remains unchanged.
• Preserves known usage/model/provenance on completed, skipped, and error paths; counts shared requests once, sums split-turn usage, and omits repeated legends and empty chat-only metadata. Documents account-root endpoints and scoring thresholds.
• Keeps numeric overflow on the typed, metadata-preserving error path. If a later split-conversation turn fails, preserves the independently known turn counts with an explicit reported_turns_only usage scope, avoids mixed-model attribution, and does not present the failed turn's execution summary as an aggregate summary.

Companion assets: Azure/azureml-assets#5370.

Validation: 514 affected SDK tests passed on Windows/Python 3.12, including 22 focused regressions for numeric overflow, public/batch split-turn failure accounting, partial usage, first-turn errors, mixed models and unchanged Prompty behavior. Formatting and diff checks passed. A fresh live bundled Fluency public-call smoke passed on 2026-10-19 at SDK 3fdd873c013578a9c44dbab97475016915392faa against the existing authorized gpt-4.1 deployment: HTTP 200, completed score 3, and 918/129/1047 prompt/completion/total tokens. All 321 installed evaluation-package files matched the qualified 1.18.10 wheel before and after the call. This is not a native Decision or deployed worker-image test.

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[bonanza89]

The SDK change also needs a matching ACA integration. We should replace the private factory hook with the shared runner while preserving trusted deployment binding, evidence handling, execution limits and accounting.

For the service follow-up, we need to verify version pinning before admission, QA child evaluation levels, OpenInference conversation ownership, and cancellation cleanup when initialization fails or is cancelled.

Before enabling this, I want the exact final wheel and asset versions exercised through the public .NET API, ACA, the real Decision deployment and the persisted result APIs. Include cancellation, failures and usage accounting for shared requests. Run full CI against that exact pair, then publish the packages, pin the ACA image and roll out behind the flag with rollback verified.

[Salma El-Shafey (salma-elshafey)]

The SDK change also needs a matching ACA integration. We should replace the private factory hook with the shared runner while preserving trusted deployment binding, evidence handling, execution limits and accounting.

For the service follow-up, we need to verify version pinning before admission, QA child evaluation levels, OpenInference conversation ownership, and cancellation cleanup when initialization fails or is cancelled.

Updates to the ACA are done

Before enabling this, I want the exact final wheel and asset versions exercised through the public .NET API, ACA, the real Decision deployment and the persisted result APIs. Include cancellation, failures and usage accounting for shared requests. Run full CI against that exact pair, then publish the packages, pin the ACA image and roll out behind the flag with rollback verified.

This would need deploying the service to INT, correct? We would need to publish the assets to the staging registry so that ACA can access it. Will someone from engineering team be able to review before we deploy to INT to test there?

bonanza89

[bonanza89]

Reviewed at 0d1a2ee with the evaluation service and worker running locally on this SDK source against the real Decision deployment, through the four companion assets, rubrics and the chat route. The shape is right: one base class that picks the runner, strategies that own the parsing, and failed rows that keep their usage and provenance. Four things I would change before the release (the endpoint path check, the per-request default credential, the echoed legend in persisted answers, and the chat-only fields on decision results), plus a few smaller notes inline.

[bonanza89]

Three things the hosted worker depends on that this branch does not have at a69217c, in case you want them here rather than in a follow-up.

1. An execution logger hook on DecisionModelRunner. The worker binds its run logger to the runner so decision observations land in the run log and the metrics, and it refuses to construct native decision evaluators when the hook is missing. A worker built against the current head therefore rejects every native run before inference.

2. The failed-row metadata filter in _run_submitter_client.py keeps only the deployment and the probabilities, so persisted failed rows from the independent route lose usage_available, execution, backend and protocol.

3. The new probability sum check at 1e-6 rejects what the deployment actually returns: distributions rounded to two decimals that add up to 0.99 or 1.01. The worker's independent mapper allows 0.005 per rounded value and keeps the raw values; the shared validator needs the same tolerance, otherwise native Likert rows become row errors once 1 is fixed.

I have one change on top of a69217c that does all three, keeping your native transport and the explicit credential requirement (no default credential): the hook with bounded per-attempt and per-operation observations and execution metadata on results and errors, the wider failed-row filter, the rounding tolerance with its boundary tests, plus the operation-owned transport with a per-attempt read deadline, the bounded Application Insights export and integer token sums for split conversations. The decision unit test modules pass with it (492 cases), and the worker's required native gate passes against this head plus the change and the assets head plus the task completion fix (378 cases). I can open it as a PR against your branch, or send the patch, whichever you prefer.

The assets CI pins a69217c as the candidate; it will have to move to the commit that carries these.

[Salma El-Shafey (salma-elshafey)]

Three things the hosted worker depends on that this branch does not have at a69217c, in case you want them here rather than in a follow-up.

1. An execution logger hook on DecisionModelRunner. The worker binds its run logger to the runner so decision observations land in the run log and the metrics, and it refuses to construct native decision evaluators when the hook is missing. A worker built against the current head therefore rejects every native run before inference.
2. The failed-row metadata filter in _run_submitter_client.py keeps only the deployment and the probabilities, so persisted failed rows from the independent route lose usage_available, execution, backend and protocol.
3. The new probability sum check at 1e-6 rejects what the deployment actually returns: distributions rounded to two decimals that add up to 0.99 or 1.01. The worker's independent mapper allows 0.005 per rounded value and keeps the raw values; the shared validator needs the same tolerance, otherwise native Likert rows become row errors once 1 is fixed.

I have one change on top of a69217c that does all three, keeping your native transport and the explicit credential requirement (no default credential): the hook with bounded per-attempt and per-operation observations and execution metadata on results and errors, the wider failed-row filter, the rounding tolerance with its boundary tests, plus the operation-owned transport with a per-attempt read deadline, the bounded Application Insights export and integer token sums for split conversations. The decision unit test modules pass with it (492 cases), and the worker's required native gate passes against this head plus the change and the assets head plus the task completion fix (378 cases). I can open it as a PR against your branch, or send the patch, whichever you prefer.

The assets CI pins a69217c as the candidate; it will have to move to the commit that carries these.

Yes, please open a PR against this branch with these changes. bonanza89

[github-actions]

[Pilot] PR Pipeline Failure Analysis

What failed

Azure Pipeline build 6971063 — check "python - pullrequest (Build Test macoslatest_311)" failed. Both the whl and sdist checks for sdk/evaluation/azure-ai-evaluation ran the full pytest suite (2169 collected tests); the sdist run finished cleanly (OK in 720.59s), but the whl run was marked FAIL(1) even though every collected test reported PASSED/SKIPPED — there is no FAILED test in the entire log.

The failure is a crash during pytest's session teardown, not a test failure:

Traceback (most recent call last):
  ...
  File ".../_pytest/warnings.py", line 141, in pytest_sessionfinish
    return (yield)
  File ".../sdk/evaluation/azure-ai-evaluation/tests/conftest.py", line 762, in pytest_sessionfinish
    stop_promptflow_service()
  File ".../sdk/evaluation/azure-ai-evaluation/tests/conftest.py", line 760, in stop_promptflow_service
    stop_service()
  File ".../promptflow/_cli/_pf/_service.py", line 325, in stop_service
    kill_exist_service(port)
  File ".../promptflow/_sdk/_service/utils/utils.py", line 176, in kill_exist_service
    proc = _get_process_by_port(port) if port else None
  File ".../promptflow/_sdk/_service/utils/utils.py", line 172, in _get_process_by_port
    return psutil.Process(int(pid))
  File ".../psutil/__init__.py", line 360, in _init
    raise NoSuchProcess(pid, msg=msg) from None
psutil.NoSuchProcess: process PID not found (pid=44288)
2026-10-17 23:01:29,054 [ERROR] azure-sdk-tools: pytest failed for azure-ai-evaluation with exit code 1.

Relevant pipeline output

=== SUMMARY ===
PACKAGE                                                    CHECK  STATUS  DURATION(s)
-------------------------------------------------------------------------------------
/Users/runner/work/1/s/sdk/evaluation/azure-ai-evaluation  sdist  OK            720.59
/Users/runner/work/1/s/sdk/evaluation/azure-ai-evaluation  whl    FAIL(1)       720.21

Total checks: 2 | Failed: 1 | Worst exit code: 1
##[error]PowerShell exited with code '1'.

Root cause

tests/conftest.py's pytest_sessionfinish hook calls stop_promptflow_service(), which looks up the promptflow service's PID by port and calls psutil.Process(pid) to kill it. By the time teardown ran (whl environment, macOS 3.11), the process behind that port had already exited, so psutil.Process() raised NoSuchProcess, which was not caught, crashing the pytest session-finish hook and causing pytest to exit non‐zero despite every actual test passing. This is a race condition in test-fixture cleanup (existing shared infra in azure-ai-evaluation's conftest.py), not something introduced by this PR's Decision-model changes — no Decision-model or PR-authored code path appears in the traceback, and the sdist run of the identical suite completed without the race.

Recommended next steps

• Retry the pipeline; this is a flaky teardown race, not a reproducible logic bug tied to the PR's diff.
• Consider hardening stop_promptflow_service() in tests/conftest.py to catch psutil.NoSuchProcess around the kill_exist_service(port) call so a service that already exited doesn't fail the whole pytest session.
• See https://aka.ms/ci-fix

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a
fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis Next Steps · copilot · sonnet50 · 67.5 AIC · ⌖ 6.57 AIC · ⊞ 8.9K · ◷

[bonanza89]

Completed the review follow-up at 3fdd873c013578a9c44dbab97475016915392faa. All 14 previously open review threads are now resolved; the two already resolved threads were left unchanged. The execution logger, independent failed-row metadata, probability validation and paid-skip accounting requests are addressed in this head.

I corrected the stale probability-tolerance statement in the description and recorded a fresh bundled Fluency public-call smoke: HTTP 200, completed score 3, and 918/129/1047 prompt/completion/total tokens with the qualified 1.18.10 wheel. No SDK product code changed during this follow-up.

The PR is ready for review and the evaluation SDK code-owner team has been requested. New CI is running. Release, published asset rollout and worker-image qualification remain separate from resolving these implemented code requests.

[Copilot]

🟡 Changes recommended

Numeric overflow handling and failed conversation-turn accounting can discard known usage and provenance.

2 open issues

• Preserve accumulated usage when later conversation turns fail · New
• Handle OverflowError for out-of-range numeric metadata · New

What changed in this PR

Adds shared Foundry Decision-model support for registry evaluators while keeping bundled evaluators on Prompty.

Changes:

• Adds model runners, scoring strategies, authentication, and execution controls.
• Preserves usage and provenance through result processing and batch persistence.
• Adds regression coverage and release notes.

Paths below are relative to sdk/evaluation/.

File	Description
cspell.yaml	Updates spelling configuration.
azure-ai-evaluation/​tests/​unittests/​test_prompty_strategy.py	Tests parsing and skipped-result metadata.
azure-ai-evaluation/​tests/​unittests/​test_decision_model_support.py	Tests Decision configuration, scoring, and execution.
azure-ai-evaluation/​tests/​unittests/​test_decision_default_transport.py	Tests transport deadlines and cancellation.
azure-ai-evaluation/​tests/​unittests/​test_decision_batch_persistence.py	Tests failure metadata and shared usage accounting.
azure-ai-evaluation/​tests/​unittests/​test_async_token_provider.py	Tests credential ownership and cleanup.
azure-ai-evaluation/​CHANGELOG.md	Documents Decision support and behavior.
azure-ai-evaluation/​azure/​ai/​evaluation/​_legacy/​_common/​_async_token_provider.py	Closes owned credentials and environment clients.
azure-ai-evaluation/​azure/​ai/​evaluation/​_exceptions.py	Adds failure metadata properties.
azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​_prompty_strategy.py	Centralizes Prompty result parsing.
azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​_model_runners.py	Adds backend execution and Decision validation.
azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​_decision_strategy.py	Maps native answers to evaluator results.
azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​_base_prompty_eval.py	Delegates to shared model infrastructure.
azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​_base_model_eval.py	Selects backends and aggregates Decision usage.
azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​__init__.py	Exports shared runners and strategies.
azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluate/​_evaluate.py	Persists Decision failure metadata.
azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluate/​_batch_run/​_run_submitter_client.py	Collects validated per-row failure metadata.
azure-ai-evaluation/​azure/​ai/​evaluation/​_constants.py	Registers aggregate and member metrics.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Unresolved metadata aggregation and integer-overflow errors can misrepresent usage or interrupt batch summaries.

No open issues 🎉

3 previously missed issues

In code that hasn't changed since last review

Safely reject oversized probability metadata values

sdk/​evaluation/​azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluate/​_batch_run/​_run_submitter_client.py:58

An oversized integer in an asset evaluator's status_completion_probability or status_probability_threshold (for example, 10**309) makes math.isfinite() raise OverflowError before the range check. get_run_summary() calls this helper without catching that error, so malformed failure metadata aborts the batch summary instead of preserving the original failure and known usage. Check the [0, 1] bounds first so these values are discarded safely.

Reconcile split-conversation aggregate decision properties

sdk/​evaluation/​azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​_base_model_eval.py:236

Successful split-conversation calls inherit the last turn's *_properties from super(), but this override only corrects the sibling token and model fields. With a 15-token/model-A turn followed by a 39-token/model-B turn, the aggregate reports 54 tokens and omits the aggregate model, while its persisted properties still report 39 tokens, model-B, and a one-turn execution summary. Reconcile aggregate Decision properties with the summed usage, remove mixed-model attribution and the last-turn execution summary, and retain turn-specific provenance under evaluation_per_turn. Cover successful aggregation as well as the existing failure cases.

Classify oversized probability thresholds as configuration errors

sdk/​evaluation/​azure-ai-evaluation/​azure/​ai/​evaluation/​_evaluators/​_common/​_model_runners.py:338

Configuring a binary Decision runner with probability_threshold=10**309 raises OverflowError in math.isfinite(). The constructor catches only ValueError, so this bypasses the intended user-classified EvaluationException. Check the numeric type and range before converting to float; the range check also rejects NaN and infinity.

2 resolved since last review

• Handle OverflowError for out-of-range numeric metadata
• Preserve accumulated usage when later conversation turns fail

_{Review effort: Balanced}

[bonanza89]

Final code follow-up is published at e3b1653e6034b210920ae0de2504ed791193ef5e. The two additional review findings were independently reproduced and fixed: numeric overflow now keeps the typed error metadata, and failed split conversations retain known prior-turn usage without misleading model or execution attribution. All 514 affected SDK tests passed.

A fresh 1.18.10 wheel from this commit (SHA-256 74c3ef3ba143d59423a001df892dc5a317189d57ce25f4b8c2c690c6f466d5f8) passed 331 native asset cases, 65 compatibility cases and the committed 487-case worker gate. The assets companion now pins this exact commit. All current SDK review threads are resolved and the SDK CI rerun has 22 successful checks. Required maintainer approval and the documented release workflow remain pending; no package has been published.