Skip to content

feat(terminal): interactive terminal in cluster-admin over SSH, login inside the pane - #1266

Draft
stephdl wants to merge 8 commits into
mainfrom
terminal-cluster-admin
Draft

feat(terminal): interactive terminal in cluster-admin over SSH, login inside the pane#1266
stephdl wants to merge 8 commits into
mainfrom
terminal-cluster-admin

Conversation

@stephdl

@stephdl stephdl commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Adds an interactive terminal to cluster-admin, towards any cluster node. sshd
on the target node creates the pseudo-terminal; api-server opens the SSH
connection over the WireGuard VPN and relays the byte stream to the browser.

Disabled on every node by default. docs/core/terminal.md carries the full
rationale.

The login prompt runs inside the terminal

There is no credentials form. The pane opens as soon as the websocket does,
api-server draws a login: prompt, and the password questions come from sshd
over keyboard-interactive.

SSH carries the user name in the authentication request, so there is no remote
login prompt to relay: api-server draws that one itself, which is why it does
its own line editing.

This does not narrow who can read the password. It still crosses api-server in
clear on its way to the node, exactly as a credentials frame would. What it
removes is the password field, and the browser vault entry that field invited.

What it is, and what it is not

A jump host. open-terminal does not grant root. It grants the right to
reach port 22 of a node through the cluster VPN, which is usually not routable
from an administrator's workstation. sshd still authenticates a system
account with a password supplied by the user.

The design assumes two administrator populations: "NS8 cluster admin" and
"operating system admin" may be different people. open-terminal can be
granted on a single node to an operator holding no other privilege there, and
an NS8 owner cannot get a shell without system credentials. This is why the
authentication factor always comes from the user, and why nothing here writes
to authorized_keys.

A key enrolment action was considered and dropped: owner holds * on every
node, so it would hold any enrolment action by construction, enrol itself and
obtain root with no system factor.

Enabling a node changes its sshd configuration

enable-node-terminal sets the flag and the node writes
/etc/ssh/sshd_config.d/90-ns8-terminal.conf, which allows password
authentication from the cluster VPN only, on an opted-in node only.
prohibit-password has been OpenSSH's compiled-in default since 7.0, so
without that drop-in root cannot log in with a password on either supported
family.

sshd -t runs before any reload and the drop-in is reverted if validation
fails, so a syntax error cannot take sshd down. The action fails loudly when
sshd_config has no Include /etc/ssh/sshd_config.d/*.conf, because the file
would be inert and reporting success would be a lie. set-terminal-sshd takes
no parameter: it reads the flag and converges the drop-in to it, so a caller
can only realign the node with the cluster policy, never open password
authentication on its own.

Notable decisions

  • The REST route only authorizes and mints a one-shot ticket, bound to the
    client address and to a single node reservation.
  • The browser sends only a node id. Address and port come from Redis; taking a
    host from the client would make api-server an arbitrary SSH proxy.
  • Host keys are published by the node into node/<id>/ssh and accepted from
    there. This is not pinning: a regenerated key is accepted as soon as it
    is republished. The property is that the key arrives over the cluster's
    authenticated channel.
  • A dedicated melody instance, because the shared /ws keeps melody's 512 byte
    read limit that a paste would exceed, and raising it there would raise it for
    an endpoint reachable without authentication.
  • The disabled flag is re-read on every keepalive tick, and by a watchdog
    during login, so disabling a node closes live sessions and also interrupts
    one parked at the password prompt.
  • Only enabled is written to node/<id>/terminal. Who flipped it belongs in
    audit.db, which modules cannot read, whereas node/* is readable by every
    installed module through its %R~node/* grant.

Security

api-server sees everything typed, the system password included. Inherent to
any browser terminal, and end-to-end encryption would not help, since
api-server serves the JavaScript that would perform it.

This is not an escalation the feature opens. api-server's Redis ACL is ~* on
keys and &* on channels with lpush, and NS8 dispatches work by pushing onto
task/<agent>/…, so that access already queues arbitrary actions on any node
agent, which runs as root. Whoever controls api-server holds the cluster
without needing anyone's password.

What the terminal adds is persistence rather than access. Cluster secrets are
rotated by rebuilding: JWT secret, Redis ACL passwords, WireGuard keys. A node
root password is not. It stays valid after a rebuild and carries outside NS8
wherever it is reused. The cluster-admin password already travels the same path,
reaching Redis as an AUTH command, but it is a cluster secret and rotates with
the cluster.

/ws/terminal carries no JWT middleware. Access rests on the one-shot
ticket: minted only by an authenticated POST holding open-terminal, bound to
the client address, valid 30 seconds. A caller without one gets a socket that
closes with nothing. Cross-origin hijacking does not apply either, as
authentication is not cookie-based and a third-party page cannot read the
ticket. What is missing is a cap: nothing limits concurrent connections to this
endpoint, and each unauthenticated one holds a session and a 30 second timer.
The pre-existing /ws has the same exposure.

open-terminal is a password oracle. It allows testing system passwords
against sshd from an address in firewalld's trusted zone, bypassing network
restrictions set for external access. Resistance rests entirely on the
api-server throttle, since pam_faillock has even_deny_root disabled by
default and is absent from Debian's default stack. Failed handshakes are
audited with the cluster-admin identity, which lastb cannot show.

CrowdSec can ban the leader and cut a node out of the cluster. All terminal
traffic reaches a node from the leader's VPN address, so sshd attributes
failed logins to it and never to the real client. Reproduced on the test
cluster: four failed attempts produced eight crowdsecurity/ssh-bf events and a
ban of 10.5.4.1 on the node. The firewall bouncer drops the source address on
all ports, not just SSH, so Redis replication and the node agent went down
with the terminal — the replication link had to re-establish afterwards. The ban
was one minute here; with CrowdSec's default duration the node would leave the
cluster for four hours, and it cannot be repaired from the UI, because the UI
drives the node through the very connection that is blocked.

The trigger is an administrator mistyping a root password.

CrowdSec should be patched to never ban the cluster VPN network. That is the
only mitigation covering every source of failures coming from the leader, not
just this feature. Spacing the attempts on the api-server side reduces the rate
but does not close the hole.

Verified

Built here: go build, go vet, gofmt clean, yarn lint, yarn build.

On a two-node test cluster, Rocky 9 leader and Debian 13 node: sessions opened
against both nodes, Include /etc/ssh/sshd_config.d/*.conf present and the
drop-in effective on Debian 13, sshd -T -C returning the expected fields after
activation, host keys published and used by the handshake, audit rows and the
ns8-terminal correlation line landing on the node.

@xterm/xterm 5.5 does not build: webpack 4, pinned by vue-cli 4, cannot parse
its bundle. The 5.3 series is used instead.

Not verified

  • Debian 12.
  • The audit join on the local TCP port, which depends on the default LogLevel.
  • No automated integration test covers a session.
Capture.video.du.2026-08-03.13-18-25.mp4

stephdl added 4 commits July 31, 2026 17:39
…rminal

Enabling the terminal on a node modifies that node's sshd configuration:
prohibit-password is OpenSSH's compiled-in default since 7.0, so without a
drop-in root cannot authenticate with a password on either supported
distribution family. The drop-in restricts that access to the cluster VPN
and to nodes whose flag is set.

set-terminal-sshd takes no parameter and converges the drop-in to the flag,
so a caller can only realign the node with the cluster policy. owner holds
the * pattern on every node and would hold this action anyway.

Read cluster/network directly: api-server's getClusterNetworks() prepends
the loopback addresses, and the support tunnel DNATs port 22 to 127.0.0.1.
api-server opens the SSH connection to the node VPN address and relays the
byte stream. The handshake runs on the websocket channel rather than in the
REST handler, so a browser-held signer can replace the password later without
touching the relay or the interface.

The REST route only authorizes and mints a one-shot ticket. Its body carries
the open-terminal action so the existing Authorizator matches it against the
grants of the node taken from the URL, and the browser never sends a host or
a port: that would turn api-server into an arbitrary SSH proxy.

Host keys come from node/<id>/ssh, published by the node over the cluster's
authenticated channel. This is not pinning, and HostKeyAlgorithms is set from
the published types to avoid a refusal that looks like a mismatch.

A dedicated melody instance is required: the shared one keeps melody's 512
byte read limit, which a paste would exceed, and raising it there would raise
it for an endpoint reachable without authentication. HandleError closes the
session because melody drops outbound frames silently when its buffer fills,
which would truncate a command line before Enter.

Resistance to password guessing rests entirely on the throttle: pam_faillock
has even_deny_root disabled by default and is absent from Debian's default
stack. Counted per node and SSH account and per cluster-admin user, kept out
of the session so reconnecting does not reset it. Failed handshakes are
audited with the cluster-admin identity, which lastb cannot show.
The page probes the node before asking for anything, so an administrator is
not made to type a root password for a handshake that cannot succeed: sshd
refuses root with a password by default on both supported families, and the
probe says which of the three ways in is open.

Pin the xterm 5.3 series rather than the renamed @xterm/xterm 5.5: webpack 4,
which vue-cli 4 pins, cannot parse the newer bundle. Use the DOM renderer and
FitAddon rather than WebGL and renderer internals, so the page works in a VM
without acceleration. Terminal bytes are written as a Uint8Array, never
through a JS string, which would break a multi-byte character split across
two frames. AttachAddon is not used: it is bidirectional by default and would
send every keystroke twice next to our own handler.

The per-node switch sits next to the terminal rather than in the settings
section, because the person able to fix sshd is the one looking at this page.
It states that enabling the terminal changes the node's sshd configuration.

list-nodes reports terminal_enabled so the page can show the state without a
new read path.
The probe publishes the sshd host keys into node/<id>/ssh, but the default
Redis user is read-only, so hset raised NoPermissionError and the key was
never written. The SSH handshake then refused the node for lack of a
published host key, and no terminal could be opened.
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
@gsanchietti
gsanchietti marked this pull request as draft August 3, 2026 08:10
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
Drop the credentials modal. The pane is mounted as soon as the WebSocket
opens, api-server draws the user name prompt, and sshd asks for the
password over keyboard-interactive.

SSH carries the user name in the authentication request, so there is no
remote login prompt to relay: api-server draws that one itself, which is
why it now does its own line editing. The password questions come from
the node, so they are stripped of control characters before reaching the
terminal.

Move the flow to its own goroutine: the prompts need keystrokes, and
melody runs its handlers one at a time, so waiting for input from inside
a handler would stop that input from ever being read.

Resolve the SSH target once instead of per attempt, and record the
throttle and audit entries in the caller: a Ctrl-C at the prompt reaches
the dialer as a failed handshake and must not count as a wrong password.

Resynchronise the enable toggle when the selected node changes. It was
only refreshed after the node list was reloaded, so switching nodes left
the previous node's state on screen.

Confirm before closing a live session, since the shell and everything
running in it are killed with it. NsModal rather than NsDangerDeleteModal:
the latter forces the operator to retype a resource name, which suits an
irreversible deletion, not a terminal that can be reopened.

This does not narrow who can read the password. It still crosses
api-server in clear on its way to the node, exactly as the credentials
frame did. What it removes is the password field, and with it the browser
vault entry that field invited.
@stephdl
stephdl force-pushed the terminal-cluster-admin branch from e9b9798 to aa215b7 Compare August 3, 2026 08:53
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
@stephdl
stephdl force-pushed the terminal-cluster-admin branch from d171ab0 to 9b8a2b3 Compare August 3, 2026 09:41
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
@stephdl
stephdl force-pushed the terminal-cluster-admin branch from 9b8a2b3 to c914e06 Compare August 3, 2026 09:55
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
Identify a node reservation by an id instead of the holder's name. A late
release could otherwise free a reservation already handed to someone else,
leaving two sessions on one node. Drop the duplicate release in attachShell
for the same reason, since closeTerminal already releases.

Make ticket adoption atomic and refuse it on a closed session. closeTerminal
runs on melody's write pump, so it can land between consuming the ticket and
installing it; installing it anyway revived a closed session whose later
closes all returned at the closed guard, holding the node for good.

Bound the two SSH round trips that had no watchdog. The keepalive opens the
loop that enforces expiry, idle, duration and the disabled flag, so a
black-holed node kept a shell none of those checks could reach. The
correlation record runs through the account's login shell and now happens
after the shell is up, being best effort.

Watch the disabled flag during login too: the keepalive loop only runs once
a shell exists, so disable-node-terminal could not interrupt a session
parked at the password prompt.

Keep what follows the newline in readLine. A paste carrying the user name
and the password in one frame lost the second line and hung the prompt.
Discard that remainder after a failed attempt, so a mistyped paste cannot
spill a password into the user name prompt, which echoes.

Roll the enable flag back when convergence fails. The UI advertised a
working terminal on a node whose sshd refuses every login, and three
attempts are enough to throttle the operator across the cluster.

Keep the pane mounted after the session ends. Nulling the socket unmounted
it and threw away the closing notice, which the 50 ms sleep before the close
frame exists to deliver, along with the whole transcript. Take the pane down
when the selected node changes instead: the transcript belongs to the node
it was opened on.
@stephdl
stephdl force-pushed the terminal-cluster-admin branch from c914e06 to cf7a5a7 Compare August 3, 2026 10:17
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
@stephdl stephdl changed the title feat(terminal): interactive terminal in cluster-admin over SSH feat(terminal): interactive terminal in cluster-admin over SSH, login inside the pane Aug 3, 2026
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
github-actions Bot pushed a commit that referenced this pull request Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant