Repository navigation
test: Azure MCP 3.0.0 end-to-end integration - #764
Draft
jonathan-vella wants to merge 29 commits into
Draft
jonathan-vella wants to merge 29 commits into
jonathan-vella wants to merge 29 commits into
Conversation
The Azure Updates feed URL now redirects to HTML, so the fetcher silently found nothing. Point it at the releasecommunications RSS endpoint, filter on the Retirements category, dedupe by link, add retired/upcoming status, and add the Redis and Linux Consumption retirements.
3.0.0 rejects unknown tool parameters, so align the three call sites that used names absent from the 3.0.0 schema (policy, applens, monitor log query), then move the pin in both configure-mcp.mjs and .vscode/mcp.json so the release check reports CURRENT. The 3.0.0 schema was checked against the published command reference; a live tools list was not possible in this environment.
Point the planner, Terraform CodeGen and the shared references at the Azure MCP azureterraform commands (avm versions, avm get, azurerm get, azapi get) for module versions, READMEs and provider docs, with the Registry API as fallback. The AVM module index and cache stay authoritative for module existence, lifecycle and resource-type mapping, and the validators are unchanged. Only a documented module-not-found for the exact module name counts as unpublished; any other lookup error is a verification failure. The tool output shapes were taken from the published 3.0.0 command reference and not yet probed at runtime.
Replace the committed real-tenant baseline with an offline-generated ALZ Corp default: the root, landing_zones and corp archetypes of the pinned platform/alz/2026.10.0 library, resolved with built-in definitions from a pinned azure-policy commit. The generator reuses discover.discover through an offline REST adapter so classification matches a live run. discover.py now evaluates the initiative member pattern if(contains(list, type), effect, 'Disabled') and treats enforceSetting like Append, so these definitions no longer leave discovery PARTIAL. Assignments shipped as DoNotEnforce are kept as informational. The renderer labels the result as assumed governance and stamps the render time, and the loader gains --alz-default.
Phase 0.45 offers the user a choice between live discovery and the shipped ALZ Corp default when the target subscription is absent from the baseline. Choosing the default is recorded as assumed, labelled in the artifacts, and keeps the Step 3.5 review mandatory. The discovery instructions carry the single exception to the no-assumptions rule.
The live 3.0.0 tools list no longer includes extension azqr unless azqr 3.0.0 or later is installed locally, while 2.0.5 listed it unconditionally. Document the prerequisite and the direct CLI fallback.
Probed against @azure/mcp 3.0.0: avm versions returns status 400 with an ArgumentException "not found in available modules" for an unpublished, nonexistent or malformed name; avm get returns 422 for an unknown version; and azurerm get returns 200 with empty arguments for an unknown resource type. Document these so agents only treat the first as "unpublished".
Azure MCP 3.0.0 lists extension_azqr only when azqr 3.0.0 or later is on PATH. Pin azqr 4.2.0 for amd64 and arm64 in download-checksums.json and install it from post-create.sh with SHA-256 verification. The install is optional so a failed download does not break container creation.
Neither Azure MCP 2.0.5 nor 3.0.0 exposes an azd tool. Point schema validation at the published azure.yaml schema and drop the tool-table row.
The 218-line AVM certification checklist targets authors of AVM modules, which APEX never writes; it only consumes them. Nothing but the skill's own table and index referenced it, and the public AVM specification remains the source. Keep the provider compatibility matrix, which the azureterraform tools cannot supply.
…vice Review feedback on #759: two call sites still used legacy command names. The live server in namespace mode lists resourcehealth_availability-status_get and appservice_webapp_diagnostic_diagnose (the latter requires app, detector-id and resource-group).
…fetcher Review feedback on #761. The extractor read the first date in the description, which is often an earlier milestone, so two feed entries got a wrong sunset date and status. It now prefers a date in the title, skips milestone dates such as "Beginning on ..." unless their own sentence announces the retirement, and parses dates in UTC. Export parseFeed, extractSunsetDate and mergeDeprecations and cover the comma-separated Retirements category, link-based dedupe, status stamping and title versus description conflicts with an RSS fixture test. Regenerate the data.
Phase 0.45 offers the user a choice between live discovery and the shipped ALZ Corp default when the target subscription is absent from the baseline. Choosing the default is recorded as assumed, labelled in the artifacts, and keeps the Step 3.5 review mandatory. The discovery instructions carry the single exception to the no-assumptions rule.
chore(mcp): Bump @azure/mcp to 3.0.0
…okup feat(agents): Use Azure MCP azureterraform for Terraform lookup
…tions-data chore(scripts): Refresh AVM module index and Azure deprecations
…baseline feat(scripts): Ship ALZ Corp default governance baseline
….0.0 probes End-to-end testing against a live subscription showed two wrong claims. - Azure MCP does expose an azd tool in the default namespace mode (2.0.5 and 3.0.0): validate_azure_yaml, error_troubleshooting and provision_common_error. It validates azure.yaml and returns troubleshooting guidance but does not run azd. Restore the validate_azure_yaml step and list the tool accurately. - extension_azqr is exposed only with --mode all and azqr on PATH, not in the default namespace mode used by .vscode/mcp.json. Say so in both quick-review references; the azqr CLI remains the default path. Also make the azqr installer tests independent of a host azqr. A real 4.2.0 on PATH made the installer exit early and failed two tests.
The SK-14 fixtures test reads its own source and matches String.raw fixtures with LF-only regexes. A working tree checked out with CRLF line endings (the repo requires LF, but 170 tracked files here were CRLF) fails with "Missing Node fixture", so the whole test:tool-contracts step fails. Normalize the source to LF before parsing. Passes with both CRLF and LF copies.
Azure MCP does not expose the extension namespace in its default namespace mode, so extension_azqr, extension_cli_generate and extension_cli_install calls in the skills could never run in the editor. - azqr: document the CLI with its real flags (--subscription-id, --resource-group, --json, --output-name), verified against azqr 4.2.0. The old examples used --subscription and --output json, which it rejects. - Resource Graph: use vetted az graph query patterns; extension_cli_generate is mentioned only as optional. - Installs: link the official az and azd install guides. - Add an opt-in azure-mcp-extension server to the devcontainer README. It exposes exactly the three tools with --tool and --read-only, verified live. - validate-mcp-config checks that server when present, with tests.
…alls" This reverts commit e2cbd98.
…lt mode Re-probing showed the earlier claim was wrong: in the default namespace mode Azure MCP 2.0.5 and 3.0.0 do expose extension_cli_generate and extension_cli_install as root tools, and extension_azqr whenever azqr 3.0.0 or later is on PATH. All three worked live. The preceding commit that moved the skills to CLI-first and added an optional server is reverted. Fix the azqr notes accordingly, and correct the CLI fallback flags to the ones azqr 4.2.0 accepts (--subscription-id, --json, --output-name).
…pshot Add tools/registry/azure-mcp-tools.json, a deterministic snapshot of the default-mode tools, commands and parameters of the pinned @azure/mcp server. refresh:mcp-tool-snapshot regenerates it from a live server (needs network, the npm allow-remote opt-in and azqr on PATH) and replaces it atomically only on success. Two consecutive refreshes produce identical bytes. validate:mcp-tool-refs runs offline in both validate suites. It fails on an unknown tool, command or parameter in .github and tools/apex-prompts markdown, treats parameter names as case sensitive, honours a mcp-ref:ignore marker, and fails when the snapshot version differs from .vscode/mcp.json or .devcontainer/configure-mcp.mjs. First run found two wrong best-practices command names (get_bestpractices should be get_azure_bestpractices_get); both are fixed.
Add azure-sla-section.mjs to print one service section from the official Online Services SLA document, with unit tests and npm scripts. Document the SLA lookup rule and the one-URL-per-fetch rule in research-workflow.md. Fetching azure.microsoft.com SLA pages returned a very large, repeated licensing index that stayed in the chat.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| export function docxXmlToParagraphs(xml) { | ||
| const paragraphs = xml.match(/<w:p[ >].*?<\/w:p>/gs) ?? []; | ||
| return paragraphs.map((p) => decodeEntities(p.replace(/<[^>]+>/g, "")).trim()).filter((t) => t.length > 0); |
… fallback A run log showed the model calling mcp_azure_mcp_ser_<tool> while the skills write mcp_azure-mcp_<tool>. The server reports its name as "Azure MCP Server" on both 2.0.5 and 3.0.0 and VS Code derives the prefix from it, so this is not an MCP 3 change and nothing broke: the tools were found with tool_search. - apex-azure-defaults: state the convention once and say to match on <tool>. - validate:mcp-tool-refs: accept both prefixes, skip a prefix followed by a <placeholder>, and cover both with tests. - agent operating frame: when the skill tool says a skill is not found, read .github/skills/<name>/SKILL.md by path. The log showed the tool's list stopping at apex-iac-common, so later skills can be missing.
…ites
validate-session-state required id, step (number), agent, title, choice and
rationale on every decision_log entry. apex-recall decide, complete-step and
the risk gate write { decision, timestamp, rationale?, step? } with step as a
string key such as "1" or "3_5"; only the risk gate adds the structured
fields. The session-state JSON schema already allows this. The validator had
never seen a real project, so a normal run (nordic-foods) failed on every
entry.
Require a decision string, or both title and choice, and validate id, step
(any step key, as a number or string) and alternatives when present. Add tests
that fail against the old validator.
…ases A dotfiles alias (rm='rm -iv') made a plain `rm path` call in an agent terminal hang waiting for a confirmation prompt it never got, then dumped ~55KB of terminal history into chat when the tool polled the backgrounded process. The existing rule only forbade an explicit -i flag. - no-interactive-shell.instructions.md: add sub-rule 1a — prefer -f (which overrides an aliased -i) or `command rm` to bypass the alias, and treat a stalled rm/mv/cp with no -f as a likely prompt rather than polling it. - post-start.sh: warn once per container start if rm, mv or cp resolve to an interactive alias, since post-start.sh itself is non-interactive and must probe .bashrc's aliases via `bash -i -c`.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Integration branch for end-to-end testing of Azure MCP 3.0.0. Draft: do not merge until the testing below is signed off.
What changed
Merged from earlier PRs
@azure/mcpto 3.0.0, fix strict-parameter call sites, add the pinned azqr installerazureterraformAVM lookup tools in the Terraform planner and codegen agents--subscriptionrequired)Added on this branch after live testing against 3.0.0
azdMCP tool (validate_azure_yaml,error_troubleshooting,provision_common_error); it validates andtroubleshoots but does not run azd.
extension_azqrappears in default mode only while azqr 3.0.0 or later is onPATH;extension_cli_generateandextension_cli_installare always listed. All three were called live. azqr CLI fallback flags corrected to the ones azqr 4.2.0accepts (
--subscription-id,--json,--output-name).get_azure_bestpractices_get).tools/registry/azure-mcp-tools.json: deterministic snapshot of the pinned server's default-mode tools, commands andparameters.
npm run refresh:mcp-tool-snapshotregenerates it (network, npm allow-remote opt-in, azqr onPATH; atomic write,unchanged on failure).
npm run validate:mcp-tool-refs(in both validate suites): offline check of everymcp_azure-mcp_*reference in.githuband
tools/apex-promptsagainst the snapshot, plus a pin-consistency check across.vscode/mcp.json,.devcontainer/configure-mcp.mjsand the snapshot. Add<!-- mcp-ref:ignore -->to skip a pseudo-reference.Verification
npm run validate:allexits 0 locally (54 Node validators, 242 Python tests).azureterraformversions anddocs,
azd validate_azure_yaml,extension_azqr,extension_cli_installandextension_cli_generate, live governance discovery(
COMPLETE), ALZ default render with--subscription.quota,documentation,wellarchitectedframework,applens,redis,aks,kusto,compute,functionapp,eventhubs,servicebus,get_azure_bestpractices), theazdtool (validate_azure_yaml,error_troubleshooting,provision_common_error) and theazureterraformtools (avm list/versions/get,azapi get,azurerm get): all 23 calls accepted the documented parameters.servicebus_queue_detailsneeds the full<name>.servicebus.windows.nethost asnamespace.azure-mcpon this branch reachesRunningand discovers 70 tools, matching the snapshot.75a9cd4d: all 19 checks pass. The four devcontainer build jobs (baseline and candidate, amd64 and arm64) each runpostCreateCommand, and their logs showazqr.zip: OKandInstalled azqr 4.2.0, so the amd64 azqr install is confirmed.Still open
eol=lf; only the SK-14 test was hardened.discover.py