Document Application Insights queries and operator monitoring setup

Add validated JavaScript service-event and Front Door KQL examples, cross-region workbooks, metric/log alert starting points, absence detection and diagnostic setup instructions. Explain sampling, privacy, evaluation-versus-live traffic and separate edge diagnostic configuration.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: b5102217-1e24-4fd7-abc5-13fecb0dabaf

Author: Hou Chi Chan

docs/FRONTDOOR.md

@@ -13,6 +13,8 @@ and leaves the source endpoint and authentication policy unchanged. Failover can
 
 Run from a reviewed repository checkout. [frontdoor-regions.bicep](../setup/infra/frontdoor-regions.bicep)
 reuses the existing [Function module](../setup/infra/resources.bicep); single-region defaults stay unchanged.
+For operators, see [Application Insights queries](#application-insights-and-sample-queries) and
+[monitoring setup](#monitoring-setup-for-operators).
 
 ### Supported source and prerequisites
 
@@ -359,10 +361,277 @@ or successful delivery to a phone.
   plans, and telemetry remain billable until appropriately decommissioned; respect Key Vault purge
   protection and retention.
 
+### Application Insights and sample queries
+
+Setup provisions **workspace-based Application Insights and Log Analytics per region**. The Function
+receives `APPLICATIONINSIGHTS_CONNECTION_STRING` and `APPLICATIONINSIGHTS_AUTHENTICATION_STRING`;
+its identity receives the Monitoring Metrics Publisher role. Local/key-based ingestion is disabled.
+The JavaScript handler emits the [structured service events](CONTRACT.md#application-logs), and
+readiness emits `request_completed` and, on failure, `request_failed` with `functionName=FrontDoorHealth`.
+No extra application logging code is needed for these events.
+
+Open the Application Insights component's **linked Log Analytics workspace > Logs** to run the
+queries below. They target JavaScript JSON messages in `AppTraces`, not Python/.NET's provider-specific
+formatting. In an Application Insights resource-scoped query, the equivalent source is
+`traces | project TimeGenerated=timestamp, Message=message, AppRoleName=cloud_RoleName`.
+For both regions, replace the first `AppTraces` line with:
+
+```kusto
+union workspace("<first-workspace-id>").AppTraces,
+      workspace("<second-workspace-id>").AppTraces
+```
+
+Both workspaces require query access. Filter by `AppRoleName` if the workspaces contain other apps.
+The default host configuration samples traces, so counts below are **observed records**, not
+guaranteed traffic totals. Keep Information-level application logs enabled; excluding `Request`
+from sampling does not exclude these trace events. Health probes can dominate ingestion volume:
+keep their charts separate from delivery traffic and review retention/sampling costs.
+
+**Delivery/evaluation volume, failures and handler latency**
+
+```kusto
+AppTraces
+| where TimeGenerated > ago(24h)
+| extend e = parse_json(Message)
+| where e.logType == "service" and e.functionName == "SendOtp"
+| where e.eventName == "request_completed"
+| summarize ObservedCompletions=count(),
+            ObservedFailures=countif(toint(e.httpStatus) >= 400),
+            P95HandlerMs=percentile(todouble(e.elapsedMs), 95)
+    by bin(TimeGenerated, 15m), AppRoleName, Evaluation=tostring(e.evaluation)
+| order by TimeGenerated desc
+```
+
+`Evaluation=true` is a non-delivering test; `false` is live mode. A blank value means the mode was
+not established, for example because envelope validation failed. Handler latency is not end-to-end
+Front Door latency or proof of delivery to a handset.
+
+**Failure stages and reasons**
+
+```kusto
+AppTraces
+| where TimeGenerated > ago(24h)
+| extend e = parse_json(Message)
+| where e.logType == "service" and e.functionName == "SendOtp"
+| where e.eventName == "request_failed"
+| summarize ObservedFailures=count(), LastSeen=max(TimeGenerated)
+    by AppRoleName, Stage=tostring(e.failureStage),
+       Reason=tostring(e.failureReason), HttpStatus=toint(e.httpStatus)
+| order by ObservedFailures desc
+```
+
+**Follow one request across its service events**
+
+```kusto
+let TraceId = "<function-request-id-or-microsoft-correlation-id>";
+AppTraces
+| where TimeGenerated > ago(24h)
+| extend e = parse_json(Message)
+| where e.logType == "service"
+| where tostring(e.functionRequestId) == TraceId
+    or tostring(e["x-ms-correlation-id"]) == TraceId
+| project TimeGenerated, AppRoleName, Event=tostring(e.eventName),
+          FunctionRequestId=tostring(e.functionRequestId),
+          InvocationId=tostring(e.functionInvocationId),
+          HttpStatus=toint(e.httpStatus), Stage=tostring(e.failureStage),
+          Reason=tostring(e.failureReason), ElapsedMs=todouble(e.elapsedMs)
+| order by TimeGenerated asc
+```
+
+Use `functionRequestId` to isolate one invocation; a Microsoft correlation ID can cover multiple
+attempts. Missing sampled events do not prove a stage never ran.
+
+**Readiness by origin**
+
+```kusto
+AppTraces
+| where TimeGenerated > ago(1h)
+| extend e = parse_json(Message)
+| where e.logType == "service" and e.functionName == "FrontDoorHealth"
+| where e.eventName == "request_completed"
+| summarize ObservedProbes=count(),
+            NotReady=countif(toint(e.httpStatus) != 200), LastSeen=max(TimeGenerated)
+    by AppRoleName, bin(TimeGenerated, 5m)
+| order by TimeGenerated desc
+```
+
+No rows is **not** a healthy result: an unavailable Function cannot emit a readiness event.
+
+**Provider outcomes for live sends**
+
+```kusto
+AppTraces
+| where TimeGenerated > ago(24h)
+| extend e = parse_json(Message)
+| where e.logType == "service" and e.eventName == "provider_response_processed"
+| summarize ObservedResponses=count(),
+            P95ProviderMs=percentile(todouble(e.providerElapsedMs), 95)
+    by AppRoleName, Provider=tostring(e.providerName),
+       Outcome=tostring(e.providerOutcome), ProviderHttpStatus=toint(e.providerHttpStatus),
+       ProviderStatus=tostring(e.providerStatus), Reason=tostring(e.failureReason)
+| order by ObservedResponses desc
+```
+
+Evaluation intentionally emits no provider events. A provider HTTP 200 is not necessarily a
+successful outcome; inspect the normalized outcome and reason.
+
+If expected events are absent, check the Function's instrumentation settings, identity role,
+Information-level log filters, sampling, workspace scope/time range and ingestion delay.
+Front Door access/probe diagnostics are **not configured by this script**: enable the required
+categories under the Front Door profile's **Diagnostic settings**, targeting a Log Analytics
+workspace, when edge-level diagnostics are needed. Easy Auth and ingress rejections can happen
+before the handler; investigate platform/authentication and Front Door diagnostics rather than
+expecting a `request_failed` service event. Review diagnostic fields, access and retention before
+enabling collection; never add OTP bodies, plaintext delivery data, keys or tokens to application logs.
+
+### Monitoring setup for operators
+
+The following is a starting configuration, **not an SLA or a production threshold recommendation**.
+Tune it against normal traffic, ingestion delay, sampling and the on-call team's response policy.
+Creating diagnostics, alerts, availability tests and notification actions can incur charges.
+
+1. In **Azure Monitor > Action groups**, create a group for the service owners and test its
+   notification delivery. Give operators appropriate read/query access to the regional workspaces;
+   alert creators also need permission on the monitored resources.
+2. In each regional Application Insights component, open **Logs** and run the delivery and readiness
+   queries above. Save them to a shared **Workbook**; use an explicit cross-workspace union for both
+   regions. Include failures by stage, latency, provider outcomes and last telemetry seen per origin.
+3. On the **Front Door profile > Diagnostic settings**, enable `FrontDoorAccessLog` and
+   `FrontDoorHealthProbeLog` to a chosen workspace. The script does not create this setting.
+   Front Door metrics are available separately without exporting `AllMetrics` to logs.
+   Review workspace retention and access: platform access logs may contain client IPs and URLs
+   even though the application service events exclude delivery bodies and credentials.
+4. In **Front Door > Alerts > Create alert rule**, select the metrics below, use the suggested
+   dimensions to separate origins/endpoints, and attach the action group. Configure auto-resolution
+   and a maintenance suppression policy appropriate for the service.
+5. In **Log Analytics workspace > Alerts > Create alert rule**, select **Custom log search** and
+   paste one of the alert queries below. Evaluate every 5 minutes. Each query includes its own
+   lookback and returns only candidate alert rows: measure **Table rows > 0**. Split by `AppRoleName`
+   for per-origin signals. Ensure the rule's query time range covers the stated lookback.
+   Test the notification path and recovery in nonproduction before enabling paging.
+
+| Signal | Example condition | Scope / caveat |
+|---|---|---|
+| `OriginHealthPercentage` | Average below 100% over 5 minutes | Split by `Origin` and `OriginGroup`. Expect transitional noise; use this with readiness events and absence detection. |
+| `Percentage5XX` | Average above 1% over 5 minutes | Split by `Endpoint`; inspect request volume because small denominators exaggerate percentages. |
+| `OriginLatency` | Average above 1000 ms over 5 minutes | Split by `Origin` and `Endpoint`; replace with a workload-derived latency budget. |
+| `Percentage4XX` | Investigate a sustained rise above baseline | Includes deliberate negative tests and client/authentication failures; do not page on every 401/403. |
+| Application failures / readiness | Custom log queries below | Trace sampling means these are observed failures, not complete request counts. |
+
+An Application Insights **Availability > Standard test** can send HTTPS GET to the public
+`/api/health/ready` route and expect 200, using multiple locations. For example, alert on failures
+from two locations over five minutes. Do not put an EPP bearer token in the test, and do not probe
+the protected SendOtp route anonymously as an availability test. A shared health URL proves neither
+every origin's health nor provider delivery; retain the per-origin metric alerts.
+
+**Observed live-delivery server failures: example log alert**
+
+```kusto
+AppTraces
+| where TimeGenerated > ago(15m)
+| extend e = parse_json(Message)
+| where e.logType == "service" and e.functionName == "SendOtp"
+| where e.eventName == "request_completed" and tobool(e.evaluation) == false
+| summarize ObservedRequests=count(),
+            ObservedServerFailures=countif(toint(e.httpStatus) >= 500) by AppRoleName
+| extend ObservedFailurePct=100.0 * ObservedServerFailures / ObservedRequests
+| where ObservedServerFailures >= 3 and ObservedFailurePct >= 5.0
+```
+
+This deliberately excludes evaluation traffic and failures before a request's mode is known.
+Use the failure-stage query and Front Door diagnostics for those cases.
+
+**Readiness failures: example log alert**
+
+```kusto
+AppTraces
+| where TimeGenerated > ago(5m)
+| extend e = parse_json(Message)
+| where e.logType == "service" and e.functionName == "FrontDoorHealth"
+| where e.eventName == "request_completed" and toint(e.httpStatus) != 200
+| summarize ObservedFailures=count() by AppRoleName
+| where ObservedFailures >= 3
+```
+
+**Missing per-origin telemetry: example log alert**
+
+Replace both names below with the deployed Function names, and replace the `AppTraces` source with
+the cross-workspace union when each region has its own workspace. Keep the full expected-origin
+list: building it from recent events would silently omit an origin that stopped reporting.
+
+```kusto
+let Expected = datatable(AppRoleName:string)
+[
+    "<prefix>-0-func",
+    "<prefix>-1-func"
+];
+let Seen =
+    AppTraces
+    | where TimeGenerated > ago(15m)
+    | extend e = parse_json(Message)
+    | where e.logType == "service" and e.functionName == "FrontDoorHealth"
+    | where e.eventName == "request_completed"
+    | summarize LastSeen=max(TimeGenerated) by AppRoleName;
+Expected
+| join kind=leftouter Seen on AppRoleName
+| where isnull(LastSeen)
+| project AppRoleName, LastSeen, Signal="No readiness telemetry in the last 15 minutes"
+```
+
+An absence alert can mean an outage, sampling/filtering, an ingestion problem, or a wrong query
+scope/name. It is a telemetry-gap signal, not proof of an application outage. Suppress during
+planned maintenance and correlate with edge health and deployment status.
+
+**Front Door SendOtp status codes: monitoring query**
+
+Run this in the workspace receiving the profile's diagnostics. Replace the profile resource ID.
+Front Door Standard/Premium exports these logs to `AzureDiagnostics`. The status conversion handles the string
+and numeric columns found in AzureDiagnostics. It does not display client IPs or full request URLs.
+
+```kusto
+let ProfileId = "/subscriptions/<subscription-id>/resourceGroups/<global-rg>/providers/Microsoft.Cdn/profiles/<profile-name>";
+AzureDiagnostics
+| where TimeGenerated > ago(24h)
+| where _ResourceId =~ ProfileId and Category == "FrontDoorAccessLog"
+| where tostring(parse_url(requestUri_s).Path) =~ "/api/SendOtp"
+| extend HttpStatus=toint(coalesce(
+    tostring(column_ifexists("httpStatusCode_s", "")),
+    tostring(column_ifexists("httpStatusCode_d", real(null))),
+    tostring(column_ifexists("httpStatus_d", real(null)))))
+| summarize ObservedRequests=count() by bin(TimeGenerated, 15m), HttpStatus
+| order by TimeGenerated desc
+```
+
+Use this to spot 401/403 and 5xx responses that never reached the Function handler.
+A 403 alone does not identify whether Easy Auth, origin restrictions or an edge policy rejected it.
+`FrontDoorHealthProbeLog` is useful for failed probe diagnostics, **not a stream of all successful
+probes**; absence of those log rows does not establish health. Use `OriginHealthPercentage` for
+the per-origin health trend.
+
+**Front Door failed probes by origin**
+
+```kusto
+let ProfileId = "/subscriptions/<subscription-id>/resourceGroups/<global-rg>/providers/Microsoft.Cdn/profiles/<profile-name>";
+AzureDiagnostics
+| where TimeGenerated > ago(24h)
+| where _ResourceId =~ ProfileId and Category == "FrontDoorHealthProbeLog"
+| summarize FailedProbes=count(), LastSeen=max(TimeGenerated)
+    by Origin=tostring(originName_s), Result=tostring(result_s),
+       HttpStatus=toint(httpStatusCode_s)
+| order by FailedProbes desc
+```
+
+Also monitor Key Vault certificate expiry (for example, a 30-day renewal warning), resolved
+Function key references, subscription budgets and telemetry ingestion volume. Never test a paging
+rule by disabling production authentication or sending unsolicited OTP messages.
+
 ## References
 
 - [Front Door routing methods](https://learn.microsoft.com/azure/frontdoor/routing-methods)
 - [Health probes and all-origins-unhealthy behavior](https://learn.microsoft.com/azure/frontdoor/health-probes)
 - [Origin security and caller-header considerations](https://learn.microsoft.com/azure/frontdoor/origin-security)
 - [Front Door caching](https://learn.microsoft.com/azure/frontdoor/front-door-caching)
 - [Key Vault backup/restore constraints](https://learn.microsoft.com/azure/key-vault/general/backup)
+- [Azure Functions monitoring and sampling](https://learn.microsoft.com/azure/azure-functions/configure-monitoring)
+- [Front Door monitoring data reference](https://learn.microsoft.com/azure/frontdoor/monitor-front-door-reference)
+- [Log search alert rules](https://learn.microsoft.com/azure/azure-monitor/alerts/alerts-create-log-alert-rule)