Skip to content

fix: set UseLocationHost=true to fix kubectl logs/exec on Istio-fronted API servers#3827

Open
udayakp wants to merge 1 commit into
loft-sh:mainfrom
udayakp:fix/use-location-host-for-istio-api-proxy
Open

fix: set UseLocationHost=true to fix kubectl logs/exec on Istio-fronted API servers#3827
udayakp wants to merge 1 commit into
loft-sh:mainfrom
udayakp:fix/use-location-host-for-istio-api-proxy

Conversation

@udayakp
Copy link
Copy Markdown

@udayakp udayakp commented Apr 10, 2026

/kind bugfix

What does this pull request do? Which issues does it resolve?

kubectl logs, kubectl exec, kubectl port-forward, and kubectl attach fail with:

Error from server (NotFound): the server could not find the requested resource

on any vcluster whose host cluster API server is fronted by an Istio proxy (e.g. Gardener shoots).

Root cause

When WithRedirect (pkg/server/filters/redirect.go) proxies pods/log|exec|portforward|attach to the host cluster, it calls HandlerWithErrorResponder() which creates an UpgradeAwareHandler. The handler's UseLocationHost field defaults to false, so the outbound HTTP request carries the original client's Host header (the vcluster LoadBalancer hostname, e.g. abc123.us-west-2.elb.amazonaws.com) instead of the actual backend API server hostname.

On clusters with an Istio-fronted API server, Istio enforces virtual host routing and returns 404 Not Found (Content-Length: 0) for any Host value it doesn't recognise — before the request ever reaches the kube-apiserver.

Note: UseRequestLocation = true (already set) only rewrites the URL path. It does not fix the Host header.

The fix

// pkg/server/handler/handler.go — HandlerWithErrorResponder()
proxy.UseRequestLocation = true
proxy.UseLocationHost = true  // ← add this line

UseLocationHost = true tells the UpgradeAwareHandler to overwrite req.Host with h.Location.Host (the actual backend hostname) before forwarding. This is the correct behaviour for a transparent reverse proxy.

Verification — wget from inside the syncer pod

# Correct Host → 200 OK
wget --header="Host: api.vclster-hst.isbnco-poc.internal.canary.k8s.ondemand.com" \
  "https://api.vclster-hst.isbnco-poc.internal.canary.k8s.ondemand.com/api/v1/.../log"
# → HTTP/1.1 200 OK

# Wrong Host (what vcluster currently sends) → 404
wget --header="Host: abc123.us-west-2.elb.amazonaws.com" \
  "https://api.vclster-hst.isbnco-poc.internal.canary.k8s.ondemand.com/api/v1/.../log"
# → HTTP/1.1 404 Not Found, server: istio-envoy, Content-Length: 0

Patched image tested end-to-end on a Gardener shoot — kubectl logs, kubectl exec, and kubectl port-forward all work after the fix.

Why this only surfaces on certain clusters

Environment API server behind Istio? Result with wrong Host header
Gardener shoot Yes (server: istio-envoy in every response) 404 Not Found — Istio rejects unknown virtual host
Kyma cluster No (Istio only for workload traffic, not API server) Ignored — plain kube-apiserver accepts any Host
Kind / vanilla k8s No Ignored — plain kube-apiserver accepts any Host

The bug exists in all environments but is only observable where Istio sits in front of the API server.

Please provide a short message that should be published in the vcluster release notes

Fixed kubectl logs, kubectl exec, kubectl port-forward, and kubectl attach returning Error from server (NotFound) on clusters with an Istio-fronted API server (e.g. Gardener shoots). The UpgradeAwareHandler was forwarding the client's original Host header to the backend instead of the backend's own hostname, causing Istio virtual host routing to reject the request with HTTP 404.

… on Istio-fronted API servers

When WithRedirect proxies pods/log, pods/exec, pods/portforward, and
pods/attach to the host cluster API, the outgoing HTTP request was
forwarding the original client's Host header (the vcluster LoadBalancer
hostname) instead of the actual backend API server hostname.

On clusters where the kube-apiserver is fronted by an Istio proxy
(e.g. Gardener shoots), Istio enforces virtual host routing and returns
HTTP 404 with an empty body for any Host header it does not recognise.
This caused all kubectl logs, exec, port-forward, and attach commands
to fail with "Error from server (NotFound): the server could not find
the requested resource".

The fix is one line: setting UseLocationHost = true on the
UpgradeAwareHandler tells the reverse proxy to overwrite req.Host with
the backend URL's hostname before forwarding the request. This is
already the correct behaviour for a transparent reverse proxy.

Root cause: the UpgradeAwareHandler default for UseLocationHost is
false, which preserves the original req.Host. UseRequestLocation = true
(already set) only controls the URL path rewriting, not the Host header.

Affected: all vcluster versions using pkg/server/filters/redirect.go
(WithRedirect path, i.e. when privateNodes is disabled, the default).

Tested on:
- Gardener shoot with Istio-fronted API server: fixed (was 404)
- Kyma cluster (Istio not in API server path): unaffected (worked before)
- Kind cluster (no Istio): unaffected (worked before)
@udayakp udayakp requested a review from a team as a code owner April 10, 2026 08:59
@udayakp
Copy link
Copy Markdown
Author

udayakp commented Apr 11, 2026

@codex review

@chatgpt-codex-connector
Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Swish!

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@udayakp
Copy link
Copy Markdown
Author

udayakp commented Apr 15, 2026

Hi @FabianKramm /Team— could you take a look at this when you get a chance?

This is a one-line fix to pkg/server/handler/handler.go that corrects kubectl logs, exec, and port-forward on Gardener shoots (and any cluster where the API server is Istio-fronted). The root cause is that UseLocationHost defaults to false, causing the UpgradeAwareHandler to forward the client's original Host header to the backend instead of the actual API server hostname — which Istio's virtual host router rejects with 404.

The fix has been verified end-to-end on a real Gardener shoot. Automated Codex review found no issues. Happy to add a unit test or make any changes you'd like.

@udayakp
Copy link
Copy Markdown
Author

udayakp commented Apr 29, 2026

Hi @FabianKramm / Team,

Please let me know if you need additional details or changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant