docs(nginx-one): add local troubleshooting and verification guide for security monitoring

content/nginx-one-console/waf-integration/waf-security-dashboard/_index.md

@@ -12,6 +12,7 @@ This section covers:
 - [Security monitoring overview]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/security-monitoring-overview.md" >}}) — what the security dashboard is, the data pipeline behind it, and what you can do with it.
 - [secops_dashboard log profile]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/default-log-profile.md" >}}) — the immutable, pre-compiled log profile the dashboard depends on.
 - [Set up security monitoring]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/set-up-security-monitoring.md" >}}) — install F5 WAF for NGINX, configure the log profile, and forward events through NGINX Agent.
+- [Troubleshoot security monitoring on the local data plane]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/local-dataplane-troubleshooting.md" >}}) — verify the embedded OpenTelemetry Collector log, the generated collector pipeline, and debug forwarding on the data plane.
 - [Security dashboard reference]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/dashboard-metrics-reference.md" >}}) — dashboard tabs, global controls, and how each widget maps to an underlying dimension.
 - [Find a security event by Support ID]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/find-event-by-support-id.md" >}}) — look up a single security event by its Support ID for quick triage.
 - [Query security events through the API]({{< ref "/nginx-one-console/api/query-events-api.md" >}}) — list events and run analytics queries programmatically.

content/nginx-one-console/waf-integration/waf-security-dashboard/local-dataplane-troubleshooting.md

@@ -0,0 +1,180 @@
+---
+nd-content-type: how-to
+nd-docs: DOCS-000
+nd-product: NONECO
+title: Troubleshoot security monitoring on the local data plane
+description: "Check the local NGINX Agent and OpenTelemetry Collector configuration when F5 WAF for NGINX security events do not appear in NGINX One Console."
+weight: 450
+toc: true
+nd-keywords: "security monitoring, troubleshooting, local data plane, nginx-agent, opentelemetry collector, secops_dashboard, WAF events"
+nd-summary: >
+  Use this guide when F5 WAF for NGINX security events do not appear in the NGINX One Console security dashboard even after you complete the setup flow.
+  It walks through the local data plane checks for invalid log profiles, missing OpenTelemetry log pipelines, and debug logging.
+  These checks help confirm whether NGINX Agent is receiving, parsing, and forwarding security events correctly.
+nd-audience: operator
+---
+
+## Overview
+
+Use this guide when you completed [Set up security monitoring]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/set-up-security-monitoring.md" >}}), sent test traffic, and still do not see F5 WAF for NGINX security events in the NGINX One Console dashboard.
+
+This guide focuses on the **local data plane**. It helps you verify four things:
+
+1. Whether the embedded OpenTelemetry Collector is dropping security logs because the deployed log profile format is wrong.
+2. Whether the NGINX Agent embedded OpenTelemetry Collector is the only process listening on port `1514`.
+3. Whether the NGINX agent generated OpenTelemetry Collector config has the expected security log pipeline.
+4. Whether debug logging shows the collector forwarding security logs to NGINX One Console.
+
+---
+
+## Before you begin
+
+Before you begin, ensure you have:
+
+- Access to the data plane host where NGINX Plus, F5 WAF for NGINX, and NGINX Agent are running.
+- Permission to read `/var/log/nginx-agent/` and `/etc/nginx-agent/`.
+- Security monitoring already configured by following [Set up security monitoring]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/set-up-security-monitoring.md" >}}).
+
+---
+
+## 1. Check for invalid log profile errors
+
+Check whether NGINX Agent's embedded OpenTelemetry Collector is rejecting incoming F5 WAF for NGINX security logs. This happens when the logs don't match the format the security monitoring pipeline expects.
+
+Open the collector log on the data plane:
+
+```shell
+sudo tail -f /var/log/nginx-agent/opentelemetry-collector-agent.log
+```
+
+Look for either of the following errors:
+
+```text
+Security violation log body is not a string. All security violation logs will be dropped until the collector is restarted.
+```
+
+```text
+Security violation log does not appear to be CSV format. Ensure the NAP logging profile uses the secops-dashboard-log format. All security violation logs will be dropped until the collector is restarted.
+```
+
+If you see either message, the data plane probably isn't using the NGINX One Console default `secops_dashboard` log profile for `app_protect_security_log`. Make sure every http, server, or location block that should log violations uses this profile.
+
+### Fix
+
+1. In NGINX One Console, redeploy the default [`secops_dashboard` log profile]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/default-log-profile.md" >}}) to the affected instance.
+2. Update the NGINX configuration so `app_protect_security_log` points to that deployed bundle.
+3. Restart NGINX Agent to reset the embedded OpenTelemetry Collector after you correct the log profile.
+
+After the restart, send a new test request and check the dashboard again.
+
+---
+
+## 2. Check that port `1514` is reserved for the collector
+
+F5 WAF for NGINX sends security logs to `syslog:server=127.0.0.1:1514`. If another process is listening on port `1514`, the NGINX Agent embedded OpenTelemetry Collector may never receive the security logs.
+
+Run the following command on the data plane:
+
+```shell
+sudo ss -ltnp | grep 1514
+```
+
+Confirm that no unexpected process is listening on port `1514`. If another service is bound to that port, stop or reconfigure it. The embedded collector needs port `1514` free to receive F5 WAF for NGINX security logs.
+
+---
+
+## 3. Verify the generated OpenTelemetry log pipeline
+
+If the collector log does **not** show either invalid-log-profile error, verify that the generated OpenTelemetry Collector config still contains the security log pipeline.
+
+{{< call-out "note" >}}NGINX Agent generates this security log pipeline only when at least one `http`, `server`, or `location` block is set up with `app_protect_security_log` pointing to `syslog:server=127.0.0.1:1514`. If no protected context uses that syslog destination, the pipeline isn't generated. Without it, no WAF security logs are forwarded to NGINX One Console.{{< /call-out >}}
+
+Open the generated collector config:
+
+```shell
+sudo grep -A 12 "logs/default:" /etc/nginx-agent/opentelemetry-collector-agent.yaml
+```
+
+Confirm it includes the following pipeline:
+
+```yaml
+logs/default:
+  receivers:
+    - tcplog/nginx_app_protect
+  processors:
+    - securityviolationsfilter/default
+    - batch/default_logs
+    - resource/default
+  exporters:
+    - otlp/default
+```
+
+This pipeline accepts F5 WAF for NGINX security logs from `tcplog/nginx_app_protect`. It filters and batches the logs, then exports them to NGINX One Console through `otlp/default`.
+
+If this pipeline is missing or materially different, the collector isn't set up as expected for security monitoring. In that case, review any custom collector configuration merged through `nginx-agent.conf`, then restart NGINX Agent so it regenerates the collector config.
+
+---
+
+## 4. Enable debug logging for the collector pipeline
+
+If the collector log doesn't show the invalid-log-profile errors and the generated pipeline looks correct, turn on debug logging. This lets you confirm the embedded collector is processing and forwarding security logs.
+
+Add the following configuration to the end of `/etc/nginx-agent/nginx-agent.conf`:
+
+```yaml
+collector:
+  exporters:
+    debug: {}
+  pipelines:
+    logs:
+      default:
+        receivers:
+          - tcplog/nginx_app_protect
+        processors:
+          - securityviolationsfilter/default
+          - batch/default_logs
+        exporters:
+          - otlp/default
+          - debug
+```
+
+Restart NGINX Agent so the updated collector configuration is applied.
+
+The `debug` exporter causes the embedded OpenTelemetry Collector to write its processed log output to:
+
+```text
+/var/log/nginx-agent/opentelemetry-collector-agent.log
+```
+
+Use this to confirm the collector is handling F5 WAF for NGINX security events locally. It continues forwarding them to NGINX One Console through `otlp/default`.
+
+{{< call-out "note" >}}The debug exporter increases log volume. Remove it after troubleshooting so the collector log returns to its normal verbosity.{{< /call-out >}}
+
+---
+
+## What to do next
+
+After each fix:
+
+1. Restart NGINX Agent.
+2. Send a new test request through the protected application path. For example requests, see [Example test requests for a default blocking policy]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/set-up-security-monitoring.md#example-test-requests-for-a-default-blocking-policy" >}}).
+3. Check `/var/log/nginx-agent/opentelemetry-collector-agent.log`.
+4. Recheck the **WAF** > **Security Dashboard** and **Event Logs** tab in NGINX One Console.
+
+If the collector is processing events locally but the dashboard remains empty, gather the following and provide them to F5 support:
+
+- NGINX Agent configuration: `/etc/nginx-agent/nginx-agent.conf`
+- Generated OpenTelemetry Collector configuration: `/etc/nginx-agent/opentelemetry-collector-agent.yaml`
+- NGINX Agent log: `/var/log/nginx-agent/agent.log`
+- Embedded OpenTelemetry Collector log: `/var/log/nginx-agent/opentelemetry-collector-agent.log`
+
+---
+
+## References
+
+For more information, see:
+
+- [Set up security monitoring]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/set-up-security-monitoring.md" >}})
+- [secops_dashboard log profile]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/default-log-profile.md" >}})
+- [Security dashboard reference]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/dashboard-metrics-reference.md" >}})
+- [Export NGINX instance metrics]({{< ref "/nginx-one-console/agent/configure-otel-metrics.md" >}})

content/nginx-one-console/waf-integration/waf-security-dashboard/set-up-security-monitoring.md

@@ -18,7 +18,7 @@ nd-audience: operator
 
 Use this guide to enable F5 WAF for NGINX security monitoring on an NGINX Plus instance that is already connected to NGINX One Console. After completing the steps, security events appear in the [security monitoring dashboard]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/_index.md" >}}), where you can review attacks, violations, and triggered signatures.
 
-You deploy the [`secops_dashboard` log profile]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/default-log-profile.md" >}}) to the instance through NGINX One Console. You then add the F5 WAF for NGINX directives to the NGINX configuration using the console's config editor, and verify the pipeline by triggering test violations. NGINX Agent automatically configures its OpenTelemetry collector to forward security events to NGINX One Console when it detects the correct directives in the NGINX configuration. You do not need to edit the NGINX Agent configuration by hand.
+You deploy the [`secops_dashboard` log profile]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/default-log-profile.md" >}}) to the instance through NGINX One Console. You then add the F5 WAF for NGINX directives to the NGINX configuration using the console's config editor, generate test traffic, and confirm the resulting events appear in the dashboard. NGINX Agent automatically configures its OpenTelemetry collector to forward security events to NGINX One Console when it detects the correct directives in the NGINX configuration. You do not need to edit the NGINX Agent configuration by hand.
 
 ---
 
@@ -50,7 +50,7 @@ The security dashboard relies on the `secops_dashboard` log profile to capture s
 
 4. Select **Next**. The wizard displays the F5 WAF for NGINX directive snippet to paste into your NGINX configuration. The wizard also opens the config editor for the target instance.
 
-5. Open the server block where you want to enable F5 WAF for NGINX (for example, `/etc/nginx/conf.d/default.conf`) and paste the snippet into the `server`, `http`, or `location` context. The snippet looks like this:
+5. Open the NGINX configuration file that handles the traffic you want to monitor (for example, `/etc/nginx/conf.d/default.conf`) and paste the snippet into the `http`, `server`, or `location` context where F5 WAF for NGINX is already enabled. The snippet looks like this:
 
     ```nginx
     app_protect_security_log_enable on;
@@ -69,10 +69,32 @@ For more on the deployment wizard and the alternative **Add File** > **Existing
 
 When you select **Publish** in the previous step, NGINX One Console pushes the configuration change to the instance and displays a confirmation message. At that point, the F5 WAF for NGINX policy and the `secops_dashboard` log profile are in place on the data plane, and the security log directive is wired to NGINX Agent.
 
-Any request that F5 WAF for NGINX inspects on the instance produces a security event that flows to NGINX One Console. To see security events in the dashboard:
+Any request that F5 WAF for NGINX inspects on the instance produces a security event that flows to NGINX One Console. Use the following checks to confirm the pipeline end to end:
 
-1. In NGINX One Console, go to **WAF** > **Security Dashboard**.
-2. As your instance handles traffic, attacks, violations, and triggered signatures appear on the dashboard within about a minute of the request being processed.
+1. Send one or more requests through the protected application path on the instance you just configured. If you have a staging policy or a known test case that triggers a violation, use it so the event is easy to identify. Otherwise, normal inspected traffic is enough to confirm the pipeline.
+2. In NGINX One Console, go to **WAF** > **Security Dashboard**.
+3. Set the time window to **Last 5 minutes**, then add a global filter for the target **Instance**, **Hostname**, or **Policy** so you only see events from the instance you just configured.
+4. Open the **Event Logs** tab and confirm at least one event appears for the request you just sent. Check that the row shows the expected URI, policy, and request status (`blocked`, `alerted`, or `passed`).
+5. Open the event row to confirm the detail panel shows the request context, triggered violations or signatures, and the generated **Support ID**. If you need to verify a single event later, copy the Support ID and use [Find a security event by Support ID]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/find-event-by-support-id.md" >}}).
+
+Events typically appear within about a minute of the request being processed.
+
+### Example test requests for a default blocking policy
+
+If the protected path uses the default F5 WAF for NGINX policy in blocking mode, the following requests commonly produce a `blocked` event because they match high-confidence attack signatures or raise the violation rating to a blocked threshold. Replace `https://app.example.com/` with a protected URL in your environment.
+
+```shell
+# Cross-site scripting (XSS) test
+curl -G "https://app.example.com/" --data-urlencode "a=<script>alert(1)</script>"
+
+# Path traversal test
+curl -G "https://app.example.com/" --data-urlencode "file=../../../../etc/passwd"
+
+# SQL injection test
+curl -G "https://app.example.com/" --data-urlencode "id=1' UNION SELECT 1,2,3--"
+```
+
+If your policy is in transparent mode, if signatures are staged, or if you heavily customized the default policy, these requests may appear as `alerted` instead of `blocked`. The dashboard still confirms that the security event pipeline is working.
 
 For details on how the dashboard is organized and how to read each widget, see the [security dashboard reference]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/dashboard-metrics-reference.md" >}}).
 
@@ -110,6 +132,8 @@ Re-run the deployment wizard after fixing the configuration.
 
 If events still do not appear after a request is processed, contact F5 support with the instance hostname and the time window you tested.
 
+For local data plane checks of the embedded OpenTelemetry Collector, generated collector pipeline, and debug forwarding, see [Troubleshoot security monitoring on the local data plane]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/local-dataplane-troubleshooting.md" >}}).
+
 ---
 
 ## References
@@ -122,6 +146,7 @@ If events still do not appear after a request is processed, contact F5 support w
 
 - [secops_dashboard log profile]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/default-log-profile.md" >}})
 - [Dashboard metrics reference]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/dashboard-metrics-reference.md" >}})
+- [Troubleshoot security monitoring on the local data plane]({{< ref "/nginx-one-console/waf-integration/waf-security-dashboard/local-dataplane-troubleshooting.md" >}})
 
 **Related how-to guides**