Separate observability documentation from the Front Door infrastructure PR

Remove the Application Insights and monitoring documentation addition from this PR so it can be reviewed independently, with single-region and multi-region Azure Functions coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b5102217-1e24-4fd7-abc5-13fecb0dabaf

Author: Hou Chi Chan

docs/FRONTDOOR.md

@@ -13,8 +13,6 @@ and leaves the source endpoint and authentication policy unchanged. Failover can
 
 Run from a reviewed repository checkout. [frontdoor-regions.bicep](../setup/infra/frontdoor-regions.bicep)
 reuses the existing [Function module](../setup/infra/resources.bicep); single-region defaults stay unchanged.
-For operators, see [Application Insights queries](#application-insights-and-sample-queries) and
-[monitoring setup](#monitoring-setup-for-operators).
 
 ### Supported source and prerequisites
 
@@ -361,277 +359,10 @@ or successful delivery to a phone.
   plans, and telemetry remain billable until appropriately decommissioned; respect Key Vault purge
   protection and retention.
 
-### Application Insights and sample queries
-
-Setup provisions **workspace-based Application Insights and Log Analytics per region**. The Function
-receives `APPLICATIONINSIGHTS_CONNECTION_STRING` and `APPLICATIONINSIGHTS_AUTHENTICATION_STRING`;
-its identity receives the Monitoring Metrics Publisher role. Local/key-based ingestion is disabled.
-The JavaScript handler emits the [structured service events](CONTRACT.md#application-logs), and
-readiness emits `request_completed` and, on failure, `request_failed` with `functionName=FrontDoorHealth`.
-No extra application logging code is needed for these events.
-
-Open the Application Insights component's **linked Log Analytics workspace > Logs** to run the
-queries below. They target JavaScript JSON messages in `AppTraces`, not Python/.NET's provider-specific
-formatting. In an Application Insights resource-scoped query, the equivalent source is
-`traces | project TimeGenerated=timestamp, Message=message, AppRoleName=cloud_RoleName`.
-For both regions, replace the first `AppTraces` line with:
-
-```kusto
-union workspace("<first-workspace-id>").AppTraces,
-      workspace("<second-workspace-id>").AppTraces
-```
-
-Both workspaces require query access. Filter by `AppRoleName` if the workspaces contain other apps.
-The default host configuration samples traces, so counts below are **observed records**, not
-guaranteed traffic totals. Keep Information-level application logs enabled; excluding `Request`
-from sampling does not exclude these trace events. Health probes can dominate ingestion volume:
-keep their charts separate from delivery traffic and review retention/sampling costs.
-
-**Delivery/evaluation volume, failures and handler latency**
-
-```kusto
-AppTraces
-| where TimeGenerated > ago(24h)
-| extend e = parse_json(Message)
-| where e.logType == "service" and e.functionName == "SendOtp"
-| where e.eventName == "request_completed"
-| summarize ObservedCompletions=count(),
-            ObservedFailures=countif(toint(e.httpStatus) >= 400),
-            P95HandlerMs=percentile(todouble(e.elapsedMs), 95)
-    by bin(TimeGenerated, 15m), AppRoleName, Evaluation=tostring(e.evaluation)
-| order by TimeGenerated desc
-```
-
-`Evaluation=true` is a non-delivering test; `false` is live mode. A blank value means the mode was
-not established, for example because envelope validation failed. Handler latency is not end-to-end
-Front Door latency or proof of delivery to a handset.
-
-**Failure stages and reasons**
-
-```kusto
-AppTraces
-| where TimeGenerated > ago(24h)
-| extend e = parse_json(Message)
-| where e.logType == "service" and e.functionName == "SendOtp"
-| where e.eventName == "request_failed"
-| summarize ObservedFailures=count(), LastSeen=max(TimeGenerated)
-    by AppRoleName, Stage=tostring(e.failureStage),
-       Reason=tostring(e.failureReason), HttpStatus=toint(e.httpStatus)
-| order by ObservedFailures desc
-```
-
-**Follow one request across its service events**
-
-```kusto
-let TraceId = "<function-request-id-or-microsoft-correlation-id>";
-AppTraces
-| where TimeGenerated > ago(24h)
-| extend e = parse_json(Message)
-| where e.logType == "service"
-| where tostring(e.functionRequestId) == TraceId
-    or tostring(e["x-ms-correlation-id"]) == TraceId
-| project TimeGenerated, AppRoleName, Event=tostring(e.eventName),
-          FunctionRequestId=tostring(e.functionRequestId),
-          InvocationId=tostring(e.functionInvocationId),
-          HttpStatus=toint(e.httpStatus), Stage=tostring(e.failureStage),
-          Reason=tostring(e.failureReason), ElapsedMs=todouble(e.elapsedMs)
-| order by TimeGenerated asc
-```
-
-Use `functionRequestId` to isolate one invocation; a Microsoft correlation ID can cover multiple
-attempts. Missing sampled events do not prove a stage never ran.
-
-**Readiness by origin**
-
-```kusto
-AppTraces
-| where TimeGenerated > ago(1h)
-| extend e = parse_json(Message)
-| where e.logType == "service" and e.functionName == "FrontDoorHealth"
-| where e.eventName == "request_completed"
-| summarize ObservedProbes=count(),
-            NotReady=countif(toint(e.httpStatus) != 200), LastSeen=max(TimeGenerated)
-    by AppRoleName, bin(TimeGenerated, 5m)
-| order by TimeGenerated desc
-```
-
-No rows is **not** a healthy result: an unavailable Function cannot emit a readiness event.
-
-**Provider outcomes for live sends**
-
-```kusto
-AppTraces
-| where TimeGenerated > ago(24h)
-| extend e = parse_json(Message)
-| where e.logType == "service" and e.eventName == "provider_response_processed"
-| summarize ObservedResponses=count(),
-            P95ProviderMs=percentile(todouble(e.providerElapsedMs), 95)
-    by AppRoleName, Provider=tostring(e.providerName),
-       Outcome=tostring(e.providerOutcome), ProviderHttpStatus=toint(e.providerHttpStatus),
-       ProviderStatus=tostring(e.providerStatus), Reason=tostring(e.failureReason)
-| order by ObservedResponses desc
-```
-
-Evaluation intentionally emits no provider events. A provider HTTP 200 is not necessarily a
-successful outcome; inspect the normalized outcome and reason.
-
-If expected events are absent, check the Function's instrumentation settings, identity role,
-Information-level log filters, sampling, workspace scope/time range and ingestion delay.
-Front Door access/probe diagnostics are **not configured by this script**: enable the required
-categories under the Front Door profile's **Diagnostic settings**, targeting a Log Analytics
-workspace, when edge-level diagnostics are needed. Easy Auth and ingress rejections can happen
-before the handler; investigate platform/authentication and Front Door diagnostics rather than
-expecting a `request_failed` service event. Review diagnostic fields, access and retention before
-enabling collection; never add OTP bodies, plaintext delivery data, keys or tokens to application logs.
-
-### Monitoring setup for operators
-
-The following is a starting configuration, **not an SLA or a production threshold recommendation**.
-Tune it against normal traffic, ingestion delay, sampling and the on-call team's response policy.
-Creating diagnostics, alerts, availability tests and notification actions can incur charges.
-
-1. In **Azure Monitor > Action groups**, create a group for the service owners and test its
-   notification delivery. Give operators appropriate read/query access to the regional workspaces;
-   alert creators also need permission on the monitored resources.
-2. In each regional Application Insights component, open **Logs** and run the delivery and readiness
-   queries above. Save them to a shared **Workbook**; use an explicit cross-workspace union for both
-   regions. Include failures by stage, latency, provider outcomes and last telemetry seen per origin.
-3. On the **Front Door profile > Diagnostic settings**, enable `FrontDoorAccessLog` and
-   `FrontDoorHealthProbeLog` to a chosen workspace. The script does not create this setting.
-   Front Door metrics are available separately without exporting `AllMetrics` to logs.
-   Review workspace retention and access: platform access logs may contain client IPs and URLs
-   even though the application service events exclude delivery bodies and credentials.
-4. In **Front Door > Alerts > Create alert rule**, select the metrics below, use the suggested
-   dimensions to separate origins/endpoints, and attach the action group. Configure auto-resolution
-   and a maintenance suppression policy appropriate for the service.
-5. In **Log Analytics workspace > Alerts > Create alert rule**, select **Custom log search** and
-   paste one of the alert queries below. Evaluate every 5 minutes. Each query includes its own
-   lookback and returns only candidate alert rows: measure **Table rows > 0**. Split by `AppRoleName`
-   for per-origin signals. Ensure the rule's query time range covers the stated lookback.
-   Test the notification path and recovery in nonproduction before enabling paging.
-
-| Signal | Example condition | Scope / caveat |
-|---|---|---|
-| `OriginHealthPercentage` | Average below 100% over 5 minutes | Split by `Origin` and `OriginGroup`. Expect transitional noise; use this with readiness events and absence detection. |
-| `Percentage5XX` | Average above 1% over 5 minutes | Split by `Endpoint`; inspect request volume because small denominators exaggerate percentages. |
-| `OriginLatency` | Average above 1000 ms over 5 minutes | Split by `Origin` and `Endpoint`; replace with a workload-derived latency budget. |
-| `Percentage4XX` | Investigate a sustained rise above baseline | Includes deliberate negative tests and client/authentication failures; do not page on every 401/403. |
-| Application failures / readiness | Custom log queries below | Trace sampling means these are observed failures, not complete request counts. |
-
-An Application Insights **Availability > Standard test** can send HTTPS GET to the public
-`/api/health/ready` route and expect 200, using multiple locations. For example, alert on failures
-from two locations over five minutes. Do not put an EPP bearer token in the test, and do not probe
-the protected SendOtp route anonymously as an availability test. A shared health URL proves neither
-every origin's health nor provider delivery; retain the per-origin metric alerts.
-
-**Observed live-delivery server failures: example log alert**
-
-```kusto
-AppTraces
-| where TimeGenerated > ago(15m)
-| extend e = parse_json(Message)
-| where e.logType == "service" and e.functionName == "SendOtp"
-| where e.eventName == "request_completed" and tobool(e.evaluation) == false
-| summarize ObservedRequests=count(),
-            ObservedServerFailures=countif(toint(e.httpStatus) >= 500) by AppRoleName
-| extend ObservedFailurePct=100.0 * ObservedServerFailures / ObservedRequests
-| where ObservedServerFailures >= 3 and ObservedFailurePct >= 5.0
-```
-
-This deliberately excludes evaluation traffic and failures before a request's mode is known.
-Use the failure-stage query and Front Door diagnostics for those cases.
-
-**Readiness failures: example log alert**
-
-```kusto
-AppTraces
-| where TimeGenerated > ago(5m)
-| extend e = parse_json(Message)
-| where e.logType == "service" and e.functionName == "FrontDoorHealth"
-| where e.eventName == "request_completed" and toint(e.httpStatus) != 200
-| summarize ObservedFailures=count() by AppRoleName
-| where ObservedFailures >= 3
-```
-
-**Missing per-origin telemetry: example log alert**
-
-Replace both names below with the deployed Function names, and replace the `AppTraces` source with
-the cross-workspace union when each region has its own workspace. Keep the full expected-origin
-list: building it from recent events would silently omit an origin that stopped reporting.
-
-```kusto
-let Expected = datatable(AppRoleName:string)
-[
-    "<prefix>-0-func",
-    "<prefix>-1-func"
-];
-let Seen =
-    AppTraces
-    | where TimeGenerated > ago(15m)
-    | extend e = parse_json(Message)
-    | where e.logType == "service" and e.functionName == "FrontDoorHealth"
-    | where e.eventName == "request_completed"
-    | summarize LastSeen=max(TimeGenerated) by AppRoleName;
-Expected
-| join kind=leftouter Seen on AppRoleName
-| where isnull(LastSeen)
-| project AppRoleName, LastSeen, Signal="No readiness telemetry in the last 15 minutes"
-```
-
-An absence alert can mean an outage, sampling/filtering, an ingestion problem, or a wrong query
-scope/name. It is a telemetry-gap signal, not proof of an application outage. Suppress during
-planned maintenance and correlate with edge health and deployment status.
-
-**Front Door SendOtp status codes: monitoring query**
-
-Run this in the workspace receiving the profile's diagnostics. Replace the profile resource ID.
-Front Door Standard/Premium exports these logs to `AzureDiagnostics`. The status conversion handles the string
-and numeric columns found in AzureDiagnostics. It does not display client IPs or full request URLs.
-
-```kusto
-let ProfileId = "/subscriptions/<subscription-id>/resourceGroups/<global-rg>/providers/Microsoft.Cdn/profiles/<profile-name>";
-AzureDiagnostics
-| where TimeGenerated > ago(24h)
-| where _ResourceId =~ ProfileId and Category == "FrontDoorAccessLog"
-| where tostring(parse_url(requestUri_s).Path) =~ "/api/SendOtp"
-| extend HttpStatus=toint(coalesce(
-    tostring(column_ifexists("httpStatusCode_s", "")),
-    tostring(column_ifexists("httpStatusCode_d", real(null))),
-    tostring(column_ifexists("httpStatus_d", real(null)))))
-| summarize ObservedRequests=count() by bin(TimeGenerated, 15m), HttpStatus
-| order by TimeGenerated desc
-```
-
-Use this to spot 401/403 and 5xx responses that never reached the Function handler.
-A 403 alone does not identify whether Easy Auth, origin restrictions or an edge policy rejected it.
-`FrontDoorHealthProbeLog` is useful for failed probe diagnostics, **not a stream of all successful
-probes**; absence of those log rows does not establish health. Use `OriginHealthPercentage` for
-the per-origin health trend.
-
-**Front Door failed probes by origin**
-
-```kusto
-let ProfileId = "/subscriptions/<subscription-id>/resourceGroups/<global-rg>/providers/Microsoft.Cdn/profiles/<profile-name>";
-AzureDiagnostics
-| where TimeGenerated > ago(24h)
-| where _ResourceId =~ ProfileId and Category == "FrontDoorHealthProbeLog"
-| summarize FailedProbes=count(), LastSeen=max(TimeGenerated)
-    by Origin=tostring(originName_s), Result=tostring(result_s),
-       HttpStatus=toint(httpStatusCode_s)
-| order by FailedProbes desc
-```
-
-Also monitor Key Vault certificate expiry (for example, a 30-day renewal warning), resolved
-Function key references, subscription budgets and telemetry ingestion volume. Never test a paging
-rule by disabling production authentication or sending unsolicited OTP messages.
-
 ## References
 
 - [Front Door routing methods](https://learn.microsoft.com/azure/frontdoor/routing-methods)
 - [Health probes and all-origins-unhealthy behavior](https://learn.microsoft.com/azure/frontdoor/health-probes)
 - [Origin security and caller-header considerations](https://learn.microsoft.com/azure/frontdoor/origin-security)
 - [Front Door caching](https://learn.microsoft.com/azure/frontdoor/front-door-caching)
 - [Key Vault backup/restore constraints](https://learn.microsoft.com/azure/key-vault/general/backup)
-- [Azure Functions monitoring and sampling](https://learn.microsoft.com/azure/azure-functions/configure-monitoring)
-- [Front Door monitoring data reference](https://learn.microsoft.com/azure/frontdoor/monitor-front-door-reference)
-- [Log search alert rules](https://learn.microsoft.com/azure/azure-monitor/alerts/alerts-create-log-alert-rule)