tests/apps: bound the output drain, which hung CI for 26 minutes

This is what wedged the runners all day.  The GUI apps job ran for
exactly its wall-clock limit on every push; before that limit existed it
held a runner slot for GitHub's 6-hour default, which is why nothing
completed between 10:10 and the evening.

It is not a slow app.  _drain() ended with

    return proc.stdout.read().decode(...)

which blocks until EVERY writer closes the pipe.  _terminate() kills the
app's process group, but the per-user services its wrapper starts --
apache2 for the data server, Xvfb, the JMS dispatcher -- outlive it and
still hold the inherited write end, so there is no EOF to wait for.  The
runner's cleanup listed exactly those as orphans when it killed the job:
python3, Xvfb, and six apache2 processes.

The log pins it precisely: machregister finished at 17:16:11 and the
next app never reported, for 26 minutes, until cancellation at 17:42.

Never reproduced locally, and the reason matters -- on a developer box
the data server is usually already running, so the wrapper starts
nothing that could inherit the pipe.  It needs a machine where the
services come up fresh, which is every CI run and no interactive one.

_drain now reads with a deadline: non-blocking, select-driven, stopping
on real EOF or when the app itself is gone and only services still hold
the pipe.  A stuck app now costs its own timeout and gets reported
instead of taking the run with it.

All 21 apps still pass locally.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EFMUNbw2UpRPmgsqviAowb

Author: andy

tests/apps/apps.py

@@ -25,7 +25,9 @@
 services.  Running the binary directly is explicitly not supported.
 """
 
+import fcntl
 import os
+import select
 import signal
 import subprocess
 import time
@@ -205,10 +207,53 @@ def _terminate(proc):
             pass
 
 
-def _drain(proc):
+def _drain(proc, timeout=15):
+    """Read whatever the app wrote, WITHOUT waiting forever for EOF.
+
+    This used to be a bare proc.stdout.read(), which blocks until every
+    writer closes the pipe.  _terminate() kills the app's process group,
+    but the per-user services the wrapper starts -- apache2 for the data
+    server, Xvfb, the JMS dispatcher -- can outlive it and still hold the
+    inherited write end, so there is no EOF to wait for.
+
+    That hung the whole suite for 26 minutes on one app in CI, until the
+    job's wall-clock limit killed it (orphaned apache2 processes were
+    listed in the runner's cleanup).  It never reproduced on a developer
+    box, where the data server is usually already running and the wrapper
+    therefore starts nothing that could inherit the pipe.
+
+    A bounded read means a stuck app costs its own timeout and is
+    reported, instead of taking the run with it.
+    """
+    if proc.stdout is None:
+        return ""
+    fd = proc.stdout.fileno()
     try:
-        if proc.stdout is None:
-            return ""
-        return proc.stdout.read().decode("utf-8", "replace")
-    except Exception:
+        flags = fcntl.fcntl(fd, fcntl.F_GETFL)
+        fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)
+    except OSError:
         return ""
+
+    chunks = []
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        try:
+            ready, _, _ = select.select([fd], [], [], 0.5)
+        except (OSError, ValueError):
+            break
+        if not ready:
+            # Nothing pending.  If the app itself is gone, anything still
+            # holding the pipe is a service we do not want to wait on.
+            if proc.poll() is not None:
+                break
+            continue
+        try:
+            data = proc.stdout.read()
+        except (OSError, ValueError):
+            break
+        if data is None:          # non-blocking read with nothing ready
+            continue
+        if data == b"":           # real EOF
+            break
+        chunks.append(data)
+    return b"".join(chunks).decode("utf-8", "replace")