PMM-15228 Improve NGINX and Auth server performance

[maxkondr]

Ticket number: PMM-15228

Percona-Lab/pmm-submodules#4481

This pull request introduces significant improvements and optimizations to the NGINX configuration for PMM, focusing on authentication, caching, static asset delivery, and proxying for metrics and UI components. It also includes minor configuration updates for Go linters, mockery, and a dashboard plugin version.

Key highlights:

• The NGINX config is refactored for better maintainability, performance, and scalability, with new upstreams, enhanced keepalive settings, improved authentication flow, and optimized handling of static and dynamic content.
• Caching is now more granular and efficient for both authentication and static assets.
• Proxying for metrics, alerts, and UI endpoints has been streamlined and grouped by theme for clarity.

---

NGINX Configuration Improvements

Upstreams and Keepalive Optimization:

• Increased keepalive_requests for several upstreams (e.g., managed-json, qan-api-json, vmproxy, nomad-server-json) to 10,000 and added new upstreams for victoriametrics and vmalert with tuned keepalive pools to handle higher throughput and reduce socket churn. [1] [2]

Authentication and Caching Enhancements:

• Get rid of an additional auth sub-request in case the initial auth request fails.
• Refactored authentication flow to use separate cached and non-cached subrequests, extracting custom headers for error handling, and constructing JSON error responses. Added a dedicated cache for authentication responses to reduce backend load. [1] [2]
• Moved the proxy_set_header X-Forwarded-For directive earlier to ensure all proxied services receive the correct client IP.

Static Content and Asset Delivery:

• Added optimized locations for serving PMM UI and asset files with sendfile, open file cache, and aggressive browser/proxy caching headers to reduce latency and server load.
• Introduced caching for Grafana static assets using a dedicated memory zone.

Metrics, Alerts, and Proxy Routing:

• Grouped and clarified proxy locations for Prometheus, VictoriaMetrics, and VMAlert, including use of new upstreams, body size tuning, and buffering controls. Added a dedicated cached auth flow for /victoriametrics/api/v1/write.
• Updated and grouped proxy locations for gRPC APIs, JSON APIs, and static dump files, improving maintainability and adding performance optimizations.

General Performance and Maintenance:

• Enabled tcp_nodelay and multi_accept for reduced latency and improved throughput under load. [1] [2]

---

Other Minor Changes

• Go Linter and Mockery Config:*

  • Allowed generic types in the Go linter config and added new interfaces for Grafana services in mockery. [1] [2]

• Dashboard Update:*

  • Rolled back the pluginVersion in the PMM Health dashboard JSON.

[codecov]

Codecov Report

❌ Patch coverage is 88.71595% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.45%. Comparing base (ea5f54c) to head (00d98c9).

Files with missing lines	Patch %	Lines
managed/services/grafana/auth_server.go	80.17%	19 Missing and 4 partials ⚠️
managed/services/grafana/helpers.go	97.82%	2 Missing and 1 partial ⚠️
managed/services/grafana/access_control_cache.go	0.00%	2 Missing ⚠️
managed/cmd/pmm-managed/main.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@                             Coverage Diff                              @@
##           PMM-15228-pmm-server-performance-metrics    #5658      +/-   ##
============================================================================
+ Coverage                                     43.47%   45.45%   +1.98%     
============================================================================
  Files                                           433      549     +116     
  Lines                                         35128    45816   +10688     
  Branches                                        592      585       -7     
============================================================================
+ Hits                                          15271    20825    +5554     
- Misses                                        18310    23006    +4696     
- Partials                                       1547     1985     +438     

Flag	Coverage Δ
admin	34.96% <ø> (ø)
agent	51.12% <ø> (?)
managed	45.38% <88.71%> (+0.25%)	⬆️
unittests	41.29% <ø> (ø)
vmproxy	72.22% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR adds Prometheus observability to the Grafana AuthServer in pmm-managed, exposing request/cache/latency metrics and wiring the server into the Prometheus registry, plus updating the PMM Health Grafana dashboard to visualize the new signals.

Changes:

• Implemented a custom Prometheus collector in AuthServer with counters/gauges/histograms for auth requests, Grafana calls, cache behavior, in-flight requests, and latencies.
• Registered the AuthServer collector during pmm-managed startup so metrics are exposed automatically.
• Updated the PMM Health dashboard to include panels for the new auth metrics and additional runtime/health visualizations.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 9 comments.

File	Description
managed/services/grafana/auth_server.go	Adds Prometheus metric descriptors/state, implements prometheus.Collector, and instruments key auth/cache/DB/Grafana code paths.
managed/cmd/pmm-managed/main.go	Registers the AuthServer as a Prometheus collector at startup.
dashboards/dashboards/PMM Health/PMM_Health.json	Adds/adjusts dashboard panels and queries to surface new auth metrics and runtime health info.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 5 comments.

Comments suppressed due to low confidence (2)

managed/services/grafana/auth_server.go:406

• The metrics label uses the raw X-Original-Uri header when extractOriginalRequest fails. That header can include query strings and high-cardinality / potentially sensitive values, which is risky to expose in Prometheus labels. Use a stable, safe route label in this error path (e.g., the current req.URL.Path which will be /auth_request).

		s.incAuthRequests(req.Method, req.Header.Get("X-Original-Uri"), http.StatusBadRequest)

managed/services/grafana/auth_server.go:451

• route is recorded as the full cleaned request path (e.g. /graph/api/datasources/proxy/8/ in tests). That can create unbounded label cardinality and an ever-growing sync.Map (memory leak over time) when paths contain IDs or other variable segments. Consider using the matched rule prefix from resolveRule (or another normalized route name) as the route label instead of the raw path.

	s.incAuthRequests(req.Method, req.URL.Path, http.StatusOK)

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

managed/services/grafana/auth_server.go:258

• mAuthRequests uses the raw cleaned request path as a label and as part of the sync.Map key. For paths with variable segments (e.g. Grafana proxy routes like /graph/api/datasources/proxy//), this can create unbounded time series and unbounded in-process memory growth because entries are never evicted from the sync.Map. Consider normalizing the label (e.g., use the matched rule prefix from resolveRule / nextPrefix chain, or otherwise bucket variable segments) to keep cardinality bounded.

		mAuthRequestsDesc: prom.NewDesc(
			prom.BuildFQName(prometheusNamespace, prometheusSubsystem, "requests_total"),
			"Total number of authentication requests.",
			[]string{"method", "route", "status_code"},
			nil,

[Copilot]

Pull request overview

Copilot reviewed 3 out of 4 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

managed/services/grafana/auth_server.go:444

• route label is currently set to the full request path (req.URL.Path / X-Original-Uri). That can create unbounded label cardinality (IDs, arbitrary paths) and also grows s.mAuthRequests without bound (one sync.Map entry per unique path/method/status), which can become a memory/DoS risk over time. Consider using the matched rule prefix (from rules/methodRules) or another bounded route identifier for the metric label instead of the raw path.

		status := httpStatusForAuthError(authErr.code)
		s.incAuthRequests(req.Method, req.URL.Path, status)
		s.returnError(rw, status, m, l)

[Copilot]

Pull request overview

Copilot reviewed 18 out of 20 changed files in this pull request and generated 7 comments.

Files not reviewed (2)

• managed/services/grafana/mock_access_control_test.go: Generated file
• managed/services/grafana/mock_grafana_auth_user_getter_test.go: Generated file

Suppressed comments (5)

managed/services/grafana/auth_server.go:1

• errors.AsType is not part of the Go standard library errors package (and this file imports errors from stdlib), so this will not compile unless there's an intentional build-time replacement. Replace with var cErr *clientError; if errors.As(err, &cErr) { ... }.

// Copyright (C) 2023 Percona LLC

managed/services/grafana/auth_server.go:347

• On extractOriginalRequest failure, the metric label values will be GET /auth_request (the subrequest), not the original client method/path. The previous fallback logic (using X-Original-* headers and trimming query, optionally cleaning when possible) produced more accurate observability for these failures; consider restoring that fallback before calling incAuthRequests.

	err := extractOriginalRequest(req)
	if err != nil {
		s.l.WithError(err).Warn("Failed to parse original request headers.")
		rw.WriteHeader(http.StatusBadRequest)

		s.incAuthRequests(req.Method, req.URL.Path, http.StatusBadRequest)
		return
	}

build/ansible/roles/nginx/files/conf.d/pmm.conf:239

• The 403 response body hardcodes Access denied (no trailing period), while the Go AuthServer uses Access denied. (with a period) in errStaticAuthErrorPermissionDenied.message. This creates inconsistent client-visible error messaging (and differs from headers set on the auth subrequest). Consider aligning the NGINX static body with the Go-side message exactly, or standardizing both to the same string.

    location @access_denied {
      auth_request off;
      default_type application/json;
      return 403 '{"code":7,"error":"Access denied","message":"Access denied"}';
    }

build/ansible/roles/nginx/files/conf.d/pmm.conf:232

• The JSON body is built via string interpolation without JSON-escaping $auth_error / $auth_message. If these variables ever contain quotes/backslashes, the response can become invalid JSON (or allow JSON injection). Prefer returning a static JSON body keyed only by $status/$auth_code, or ensure values are properly escaped before embedding into JSON.

    location @auth_failed {
      auth_request off;
      default_type application/json;

      # Construct JSON payload using the extracted variables
      return 401 '{"code": $auth_code, "error": "$auth_error", "message": "$auth_message"}';
    }

build/ansible/roles/nginx/files/conf.d/pmm.conf:192

• Correct spelling of 'immidiatly' to 'immediately' in comment.

    # Cache entry key - cached response will be used immidiatly (if it exists), otherwise

[Copilot]

Pull request overview

Copilot reviewed 20 out of 23 changed files in this pull request and generated 1 comment.

Files not reviewed (2)

• managed/services/grafana/mock_access_control_test.go: Generated file
• managed/services/grafana/mock_grafana_auth_user_getter_test.go: Generated file

Suppressed comments (4)

managed/services/grafana/auth_server.go:183

• Spelling/grammar: “details that is a result …” and “encoded filers” should be corrected to improve readability.

// authResult contains authentication response details that is a result of all
// authentication and authorization (including LBAC) checks.
type authResult struct {
	// encoded filers to be added as proxy headers.
	vmProxyFilters string

build/ansible/roles/nginx/files/conf.d/pmm.conf:193

• Typos in this block’s comments (e.g., “It incoming”, “immidiatly”, “it's response”) make the config harder to read/maintain.

    # Internal location for authentication via pmm-managed/Grafana.
    # It uses cache for request authentication results. It incoming request matches the
    # Cache entry key - cached response will be used immidiatly (if it exists), otherwise
    # a subrequest to auth backend is sent and it's response is cached.

managed/services/grafana/deps.go:25

• Spelling/grammar: “grafanaAuthUserGetter exist only …” → “exists only …”.

// grafanaAuthUserGetter exist only to make fuzzing simpler.

managed/services/grafana/auth_server.go:176

• Spelling/wording in this comment is off (“Ttl”, “validiness”). This is user-facing for maintainers and shows up in generated docs.

This issue also appears on line 179 of the same file.

	// Ttl for auth response validiness in auth cache.
	cacheItemTTL = 60 * time.Second
	// Auth response cache cleanup interval.
	cacheInvalidationInterval = 2 * cacheItemTTL