Merge origin/main into wip/clip-audit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PFPK14wBcLD1s4dYUowpP4

Author: andy

CMakeLists.txt

@@ -1994,6 +1994,16 @@ if(PYTHON3_EXECUTABLE)
   set_tests_properties(apps_summary PROPERTIES TIMEOUT 900
                        SKIP_RETURN_CODE 77)
 
+  # The Quantum ESPRESSO tutorial (help/src/qe-first-calculation.md) replayed
+  # in the real apps: silicon scf and a water relax, run by pw.x through the
+  # Launcher and the job monitor, checked against pw.x on a hand-written deck.
+  add_test(NAME apps_qe_walkthrough
+           COMMAND ${PYTHON3_EXECUTABLE}
+                   ${CMAKE_CURRENT_SOURCE_DIR}/tests/apps/qe_walkthrough_test.py
+           WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/apps)
+  set_tests_properties(apps_qe_walkthrough PROPERTIES TIMEOUT 3000
+                       SKIP_RETURN_CODE 77)
+
   # The Builder with a calculation's MOs panel open in every panel layout,
   # on a 1366x768 screen: viewer and panel on screen and inside the window.
   add_test(NAME apps_panel_layouts
@@ -2417,6 +2427,13 @@ if(PYTHON3_EXECUTABLE)
                    --build ${CMAKE_BINARY_DIR}
            WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/queues)
   set_tests_properties(queues_config PROPERTIES TIMEOUT 300 SKIP_RETURN_CODE 77)
+  # Which layer a machine comes from (user, server, install) and localhost.
+  add_test(NAME queues_layers
+           COMMAND ${PYTHON3_EXECUTABLE}
+                   ${CMAKE_CURRENT_SOURCE_DIR}/tests/queues/layers_test.py
+                   --build ${CMAKE_BINARY_DIR}
+           WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/queues)
+  set_tests_properties(queues_layers PROPERTIES TIMEOUT 300 SKIP_RETURN_CODE 77)
   add_test(NAME queues_configedit
            COMMAND ${PYTHON3_EXECUTABLE}
                    ${CMAKE_CURRENT_SOURCE_DIR}/tests/queues/configedit_test.py
@@ -2457,8 +2474,9 @@ if(PYTHON3_EXECUTABLE)
   add_test(NAME queues_publish
            COMMAND ${PYTHON3_EXECUTABLE}
                    ${CMAKE_CURRENT_SOURCE_DIR}/tests/queues/publish_test.py
+                   --build ${CMAKE_BINARY_DIR}
            WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/tests/queues)
-  set_tests_properties(queues_publish PROPERTIES TIMEOUT 120 SKIP_RETURN_CODE 77)
+  set_tests_properties(queues_publish PROPERTIES TIMEOUT 300 SKIP_RETURN_CODE 77)
   # ecce -admin on a -remote client: ecce-site-admin on a simulated central
   # server, publishing, and the client's ecce-remote-setup --refresh (#234).
   add_test(NAME queues_siteadmin

GETTING_STARTED.md

@@ -518,10 +518,13 @@ sudo ecce-remote-setup <server-host>
 ```
 
 `ecce-remote-setup` also copies the server's registered site machine list
-(`sudo ecce -admin` on the server) onto the client, so students don't
-register machines by hand; run `sudo ecce-remote-setup --refresh` on the
-client after the admin changes that list (#188). It writes only the data server's address; the
-client finds the broker on the same host, port 8088 (set `ECCE_BROKER_PORT`
+onto the client, so students don't register machines by hand. The published
+files are readable without a login; `--login NAME` (password asked for, or in
+`ECCE_SETUP_PASSWORD`) is used if given. Run `sudo ecce-remote-setup --refresh`
+after the admin changes the list (`sudo ecce -admin` on the server publishes
+it, #188, #192). This copy stays until clients fetch the files themselves
+(#192 stage 3), which is also when the folder starts to need a login.
+The client finds the broker on the same host, port 8088 (set `ECCE_BROKER_PORT`
 in the client's environment if the server uses another).
 
 ##### TLS for the central server (optional)
@@ -567,8 +570,8 @@ An administrator can edit the server's site machine list from a client
 with `ecce -admin -remote`. Register Machines then saves over ssh, as the
 administrator's own login on the server (`-l LOGIN` for another one, or a
 `User` line in `~/.ssh/config`): `ecce-site-admin` on the server writes
-`siteconfig` with the same writers as `ecce -admin` there, publishes the
-files to clients, and the client fetches its copy again. The data server
+`siteconfig` with the same writers as `ecce -admin` there and publishes the
+files to clients. The data server
 password plays no part in this; the ssh login and the right to write the
 server's `siteconfig` decide.
 
@@ -600,9 +603,7 @@ echo "$d" > /opt/ecce/siteconfig/PublishDir # as an administrator
 Every directory above `$d` must be searchable (`x`) by the administrators,
 e.g. `chmod 711 ~ecce` if the home directory is private.
 
-The administrator's own client refreshes its copy when that user can write
-its `siteconfig` (the same group setup on the client); otherwise, and on
-every other client, `sudo ecce-remote-setup --refresh` fetches the new list.
+On the administrator's own client, and on every other, `sudo ecce-remote-setup --refresh` fetches the new list.
 
 Users then run `ecce -remote`. A client quitting never stops the server's
 services, and neither does the server account's own plain quit; its
@@ -709,8 +710,9 @@ the data server's contents are untouched.
   OpenWire).
 - **Clients must be 9.x as well.** An 8.x client cannot talk to a 9.x
   broker, nor a 9.x client to an 8.x server. Upgrade the server and every
-  client together, then re-run `sudo ecce-remote-setup <server-host>` on
-  each client only if the server's machine list changed.
+  client together. The data server's `httpd.conf` is regenerated at each
+  start, so restart it once after upgrading to pick up the read-only rule on
+  the published site files.
 
 **Shared system broker (mode 3).** Add an account per user with
 `sudo ecce-broker-setup --user NAME` (the 8.x broker had no accounts),

docs/claude/codereg/ai-qe-resolver-returning-empty-leaves-its-tag.md

@@ -0,0 +1,23 @@
+---
+type: pitfall
+title: "A resolver that returns \"\" without setting `$_` leaves its `##tag##` in the deck"
+area: codereg
+section: "Pitfalls found more than once (check for siblings)"
+paths: ["scripts/parsers/ai.qe", "scripts/parsers/qe.tpl", "tests/apps/qe_walkthrough_test.py"]
+issues: []
+---
+**A `##tag##` resolver that wants to emit nothing must set `$_ = ""` itself.**
+`modifyInputFile` only overwrites the line when the resolver returns a
+non-empty string; on `""` the line still holds `##QEIons##` and is written
+to the deck. `ai.qe`'s `QEIons` and `QECell` returned `""` for an scf run
+without clearing `$_`, so every Quantum ESPRESSO energy deck had two literal
+`##QEIons##`/`##QECell##` lines (pw.x ignores unknown cards with a warning,
+which is why nobody noticed). Found by reading the deck the Calculation
+Editor stored in `tests/apps/qe_walkthrough_test.py`; set `ECCE_AI_DEBUG=1`
+for the "returned EMPTY and nothing replaced the tag" line.
+
+Same walkthrough: pw.x's own default `ecutrho = 4 * ecutwfc` is too low for the
+ultrasoft/PAW files of the SSSP set (they want 8x), and the energy depends on
+it (0.5 mRy for the 8-atom silicon cell: -91.35634 Ry at 120 Ry, -91.35686 Ry at 240 Ry). `ai.qe` writes `ecutrho = 8 * ecutwfc` when the
+"Charge Density Cutoff" box is unticked and one of the chosen files is
+ultrasoft or PAW.

docs/claude/codereg/config-machine-is-site-then-user-per-key.md

@@ -32,10 +32,17 @@ form only when the first word is followed by `:`; otherwise `key value:with:colo
 is space form. No trailing `# comment` stripping any more (the C++ reader
 used to strip it, gensub never did). Blocks are replaced whole, never appended.
 
-**`-remote`:** the site files are the copies `ecce-remote-setup` fetched into
-the client's `siteconfig/`; the user file merges over them exactly as in local
-mode. `ecce-dataserver-start` publishes `submit.site` and `QueueManagers` too;
-a client of an older server simply keeps its own.
+**Layers (#192):** the readers do not build `$ECCE_HOME/siteconfig/<name>`
+themselves but ask `Ecce::siteConfigLayers(machine)` (`siteConfigFile`,
+`userRegistrationDir`, `siteConfigDirs`): `[user, server site, install]` when
+`-remote` has a cache under `~/.ECCE/server/<host>_<port>/` (`site/`,
+`user-<login>/`), else `[~/.ECCE, install]`. A machine comes whole from one
+site layer (the highest whose `Machines` lists it; its `CONFIG.<m>` is never
+merged with a lower layer's), the user's `CONFIG.<m>` merges per key on top.
+`localhost` is always the client's: server layers are dropped for it.
+gensub gets the same list as `ECCE_SITECONFIG_DIRS` (set by `Launch.C`).
+Tests: `queues_config` (three layers, C++ against gensub), `queues_layers`.
+Before a cache exists (stage 3 of #192 writes it) nothing differs from local mode.
 
 **The invariant is tested**: `tests/queues/config_test.py` runs
 `build/configdump` (C++) and `GENSUB_DUMP_CONFIG=1 gensub` on one site+user

docs/claude/codereg/site-admin-from-a-client.md

@@ -23,8 +23,9 @@ text it was edited from) and `SiteAdminClient::send` runs it on the server:
    write bit and membership are the authorisation, the data-server password is
    not involved - runs processmachine, writes `CONFIG.<m>` through
    `ConfigFile`, and runs `$ECCE_HOME/bin/ecce-site-publish`.
-4. The client runs `ecce-remote-setup --refresh` when it can write its
-   `siteconfig`; otherwise it says to run it with sudo. Then `redo()` reloads.
+4. The client's `SiteAdminClient::refresh` still runs `ecce-remote-setup
+   --refresh`, which since #192 only prints a note and exits 0 (the machine
+   list is no longer copied to the client). Then `redo()` reloads.
 
 Things that are easy to get wrong:
 
@@ -40,9 +41,12 @@ Things that are easy to get wrong:
 - A raw edit is refused when the server's file no longer reads as the
   client's copy (someone else changed it); per-key edits merge like
   processmachine does.
-- `--refresh` removes a `CONFIG.*`/`*.Q` that its previous fetch brought and
-  the server no longer publishes (it keeps the last MANIFEST in
-  `siteconfig/RemoteServer/MANIFEST`), so a deleted machine disappears.
+- `ecce-site-publish` also writes `INDEX` (sha256sum format, last, and
+  unchanged when no file changed) and publishes `StartupMessage` and
+  `NewUserMessage`. `httpd.conf.ecce` serves the folder read-only
+  without a login (`Require all granted` + `Require method GET HEAD OPTIONS
+  PROPFIND` under `AuthMerging Off`; a login requirement waits for #192 stage 3; the `<Limit>` sections of
+  `/Ecce` do not carry over, hence the method list).
 - Tests run the "server" on this machine with no ssh: host `127.0.0.1` and no
   login make `RCommand` use `DirectTransport`, and an `ecce-site-admin`
   wrapper on PATH switches to the server's `ECCE_HOME`/`ECCE_REALUSERHOME`.

docs/claude/services/remote-views-use-ecces-own-login.md

@@ -3,7 +3,7 @@ type: pitfall
 title: "A remote view must ride ECCE's own login, not start an ssh of its own"
 area: services
 section: "Pitfalls"
-paths: ["src/comm/commtools/TailSource.C", "src/wxgui/comm/WxTailWindow.C", "src/comm/commtools/EcceShell.C", "src/comm/rcommand/SshTransport.C", "src/apps/organizer/CalcMgr.C"]
+paths: ["src/comm/commtools/TailSource.C", "src/comm/commxt/JobStore.C", "src/comm/commxt/Launch.C", "src/tdat/resources/AuthCache.C", "src/wxgui/comm/WxTailWindow.C", "src/comm/commtools/EcceShell.C", "src/comm/rcommand/SshTransport.C", "src/apps/organizer/CalcMgr.C"]
 issues: [204]
 ---
 **Anything that shows a remote file runs over the RCommand connection that
@@ -16,8 +16,8 @@ and a two-factor one wanted a second code.
 Tail now opens `WxTailWindow`, fed by `TailSource`: one `RCommand`, a file
 check, then `startStream(script, true)`, which on libssh is
 `SshTransport::openStreamOnLogin` -- an exec channel on the session that
-just authenticated (the ordinary `openStream`, used by the job monitor,
-opens a session of its own and logs in again). Until `closeStream()` that
+just authenticated (the ordinary `openStream` opens a session of its own and
+logs in again; nothing in ECCE uses it any more). Until `closeStream()` that
 session belongs to the stream's pump thread, and `run()`/SFTP on it fail
 with "busy". On OpenSSH (a shared connection) and locally (DirectTransport)
 the plain stream already shares the login. The script ends `tail` when its
@@ -30,3 +30,18 @@ The terminal route remains behind the `TailInTerminal` preference
 ECCE shares is reused, but over libssh it still logs in again.
 Tests: `transport_tail`, `transport_tail_window` (ctest),
 `tests/transport/sshd/tail_test.sh` (one login per Tail, counted).
+
+The job monitor follows the same rule. eccejobstore (one process per job,
+restarted by eccejobmaster) streams eccejobmonitor with
+`startStream(cmd, true)` on the login `initConn()` made, and copies the
+output files at the end with `RCommand::copyOnLogin` over that login too;
+only if that copy fails does it log in again (`RCommand::get`). So one run
+of eccejobstore is one login; a dropped connection ends the run, and the
+restarted one logs in once. While the stream runs, `remoteconn` can only be
+interrupted (`exec("\003")`); `cleanup()` stops the stream before its
+`rm`. Launch copies its files with `copyOnLogin` too (`putOnLogin`), so a
+job costs two logins: Launch's and eccejobstore's, each its own process.
+eccejobstore sets `AuthCache::LAST_URL`; that fallback takes the newest
+credential *of the same kind*, because `initConn()` caches the machine's
+password (`ssh://`) before `initDAV()` asks for the data server's. Test: `tests/transport/sshd/monitor_login_test.sh` (sshd's
+log counts the password logins per job).

docs/claude/wx-viewer/geometry-trace-panel-needs-a-plottable-series.md

@@ -0,0 +1,19 @@
+---
+type: pitfall
+title: "The Geometry Trace panel asked for a property named \"\" when nothing can be plotted"
+area: wx-viewer
+paths: ["src/apps/builder/GeomTracePropertyPanel.C", "scripts/parsers/qe.desc"]
+issues: []
+---
+**`GeomTracePropertyPanel::fillPlot()` fetched the property `""` when the
+calculation stores a trace but no per-step series.** Quantum ESPRESSO stores
+`GEOMTRACE` and no `TEVEC`/gradient series, so `p_currentProp` stayed empty;
+`getProperty("")` fetches the whole `Props/` collection, the data server
+answered `301 Moved Permanently` with an HTML page, and `PropertyDoc::parse`
+crashed (SIGSEGV when the panel was first shown, found with gdb on a
+relax calculation of the QE tutorial). `fillPlot` now returns for an empty
+name; the panel then has the frames to play and an empty plot. A per-step
+energy series for QE (TEVEC) is not stored: pw.x prints the energy of the
+starting geometry first and `qe.geomtrace` starts at the first move, so the
+two are offset by one and a stateless `.desc` entry cannot pair them (see
+`qe.desc`).