fix: set UseLocationHost=true to fix kubectl logs/exec on Istio-fronted API servers#3827
fix: set UseLocationHost=true to fix kubectl logs/exec on Istio-fronted API servers#3827udayakp wants to merge 1 commit into
Conversation
… on Istio-fronted API servers When WithRedirect proxies pods/log, pods/exec, pods/portforward, and pods/attach to the host cluster API, the outgoing HTTP request was forwarding the original client's Host header (the vcluster LoadBalancer hostname) instead of the actual backend API server hostname. On clusters where the kube-apiserver is fronted by an Istio proxy (e.g. Gardener shoots), Istio enforces virtual host routing and returns HTTP 404 with an empty body for any Host header it does not recognise. This caused all kubectl logs, exec, port-forward, and attach commands to fail with "Error from server (NotFound): the server could not find the requested resource". The fix is one line: setting UseLocationHost = true on the UpgradeAwareHandler tells the reverse proxy to overwrite req.Host with the backend URL's hostname before forwarding the request. This is already the correct behaviour for a transparent reverse proxy. Root cause: the UpgradeAwareHandler default for UseLocationHost is false, which preserves the original req.Host. UseRequestLocation = true (already set) only controls the URL path rewriting, not the Host header. Affected: all vcluster versions using pkg/server/filters/redirect.go (WithRedirect path, i.e. when privateNodes is disabled, the default). Tested on: - Gardener shoot with Istio-fronted API server: fixed (was 404) - Kyma cluster (Istio not in API server path): unaffected (worked before) - Kind cluster (no Istio): unaffected (worked before)
|
@codex review |
|
Codex Review: Didn't find any major issues. Swish! ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
|
Hi @FabianKramm /Team— could you take a look at this when you get a chance? This is a one-line fix to The fix has been verified end-to-end on a real Gardener shoot. Automated Codex review found no issues. Happy to add a unit test or make any changes you'd like. |
|
Hi @FabianKramm / Team, Please let me know if you need additional details or changes. |
/kind bugfix
What does this pull request do? Which issues does it resolve?
kubectl logs,kubectl exec,kubectl port-forward, andkubectl attachfail with:on any vcluster whose host cluster API server is fronted by an Istio proxy (e.g. Gardener shoots).
Root cause
When
WithRedirect(pkg/server/filters/redirect.go) proxiespods/log|exec|portforward|attachto the host cluster, it callsHandlerWithErrorResponder()which creates anUpgradeAwareHandler. The handler'sUseLocationHostfield defaults tofalse, so the outbound HTTP request carries the original client'sHostheader (the vcluster LoadBalancer hostname, e.g.abc123.us-west-2.elb.amazonaws.com) instead of the actual backend API server hostname.On clusters with an Istio-fronted API server, Istio enforces virtual host routing and returns
404 Not Found(Content-Length: 0) for anyHostvalue it doesn't recognise — before the request ever reaches the kube-apiserver.Note:
UseRequestLocation = true(already set) only rewrites the URL path. It does not fix theHostheader.The fix
UseLocationHost = truetells theUpgradeAwareHandlerto overwritereq.Hostwithh.Location.Host(the actual backend hostname) before forwarding. This is the correct behaviour for a transparent reverse proxy.Verification — wget from inside the syncer pod
Patched image tested end-to-end on a Gardener shoot —
kubectl logs,kubectl exec, andkubectl port-forwardall work after the fix.Why this only surfaces on certain clusters
server: istio-envoyin every response)404 Not Found— Istio rejects unknown virtual hostThe bug exists in all environments but is only observable where Istio sits in front of the API server.
Please provide a short message that should be published in the vcluster release notes
Fixed
kubectl logs,kubectl exec,kubectl port-forward, andkubectl attachreturningError from server (NotFound)on clusters with an Istio-fronted API server (e.g. Gardener shoots). TheUpgradeAwareHandlerwas forwarding the client's originalHostheader to the backend instead of the backend's own hostname, causing Istio virtual host routing to reject the request with HTTP 404.