Skip to content

Commit 1fcc3fc

Browse files
andyclaude
andcommitted
tests/e2e: run on Debian, and lead with what the code actually said
Two fixes to the same failure. WHAT THE DIAGNOSTIC SHOWED. The previous commit made a failed code report itself, and it worked -- but it printed the last twelve lines of the output, which for a Fortran crash are all "#14 0x... in ???". A code that aborts prints its message FIRST and the backtrace after, so the one line naming the cause had scrolled off and a buffer overflow was reported as a list of hex addresses. Now the lines that look like a diagnosis lead, with the tail kept for context. WHAT IT SAID, once it could. Quantum ESPRESSO aborts under glibc's FORTIFY buffer-overflow check inside pw.x -- exit -6, SIGABRT, in __snprintf_chk. Not an ECCE bug, and not a parsing bug, which is what the three missing properties had made it look like for several days. Debian's build of the same upstream QE 6.7 runs the identical deck without complaint, on the machine this is developed on. The Debian changelog shows no fix for it, so the overflow is latent in QE either way and Ubuntu's stricter hardening is what catches it. So the e2e job moves to a debian:trixie container. This suite exists to notice when a CODE's behaviour changes; running it against a different distribution's build of that code reports differences that have nothing to do with ECCE, and ECCE targets Debian anyway. Only this job moves -- the others test our own Python, Perl and C++, where the distribution is not the variable. The apt step now runs BEFORE the checkout, which is not the usual order: a bare Debian image has neither git nor ca-certificates, and actions/checkout needs both. Nothing in that step needs the repository. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFMUNbw2UpRPmgsqviAowb
1 parent fbaa60b commit 1fcc3fc

3 files changed

Lines changed: 81 additions & 6 deletions

File tree

‎.github/workflows/test.yml‎

Lines changed: 27 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -96,25 +96,48 @@ jobs:
9696
e2e:
9797
name: End-to-end (real code, real monitor, real parsers)
9898
runs-on: ubuntu-latest
99+
# Debian, not the runner's Ubuntu, and deliberately.
100+
#
101+
# This suite exists to notice when a CODE's behaviour changes, so
102+
# running it against a different distribution's build of that code
103+
# reports differences that have nothing to do with ECCE. It did:
104+
# Ubuntu's quantum-espresso 6.7 aborts on this deck with a glibc
105+
# FORTIFY buffer-overflow check inside pw.x, while Debian's build of
106+
# the same upstream 6.7 runs it fine. The overflow is latent in QE
107+
# either way -- Ubuntu's hardening is simply stricter and catches it
108+
# -- and neither the bug nor the difference is ECCE's.
109+
#
110+
# ECCE targets Debian, so this is also the platform whose answers
111+
# matter. Only this job moves: the others test our own Python,
112+
# Perl and C++, where the distribution is not the variable.
113+
container: debian:trixie
99114
# Bounded so a hung job cannot hold a runner slot for GitHub's
100115
# 6-hour default; see the concurrency note above. Roomier than the
101116
# script suites because nwchem-openmpi is a large apt install; the
102117
# jobs themselves are seconds on these molecules.
103118
timeout-minutes: 30
104119
steps:
105-
- uses: actions/checkout@v4
106-
120+
# BEFORE the checkout, which is not the usual order: a bare Debian
121+
# image has neither git nor ca-certificates, and actions/checkout
122+
# needs both. Nothing here needs the repository.
123+
#
107124
# MOPAC is Debian-packaged and runs a small molecule in seconds,
108125
# which is what makes this tier possible at all. ORCA and
109126
# Gaussian cannot be installed here -- one is registration-walled,
110127
# the other licensed -- so they stay a developer-box tier and the
111128
# suite skips them by name rather than passing silently.
129+
#
130+
# No sudo: a container runs as root, and Debian's image has none.
112131
- name: Install dependencies
113132
run: |
114-
sudo apt-get update
115-
sudo apt-get install -y perl python3 mopac nwchem nwchem-openmpi \
133+
apt-get update
134+
apt-get install -y --no-install-recommends \
135+
ca-certificates git perl python3 \
136+
mopac nwchem nwchem-openmpi \
116137
quantum-espresso quantum-espresso-data-sssp gromacs
117138
139+
- uses: actions/checkout@v4
140+
118141
# Declare what this job installed. Without this a code whose
119142
# binary is not where the suite looks simply skips, and the job
120143
# goes green with that coverage quietly gone -- which is exactly

‎tests/e2e/README.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,26 @@ conversion were dropped, and `ENTHALPY − ETHERM` must equal RT exactly.
124124

125125
## Codes
126126

127+
## CI runs this on Debian, not the runner's Ubuntu
128+
129+
The suite exists to notice when a **code's** behaviour changes, so
130+
running it against a different distribution's build of that code reports
131+
differences that have nothing to do with ECCE.
132+
133+
It did. Ubuntu's `quantum-espresso` 6.7 aborts on the Si deck with a
134+
glibc FORTIFY buffer-overflow check inside `pw.x`; Debian's build of the
135+
same upstream 6.7 runs it. The overflow is latent in QE either way —
136+
Ubuntu's hardening is simply stricter and catches it — and neither the
137+
bug nor the difference belongs to ECCE.
138+
139+
ECCE targets Debian, which is also the platform whose answers matter.
140+
Only this job moves; the others test our own Python, Perl and C++, where
141+
the distribution is not the variable.
142+
143+
The apt step runs **before** the checkout, which is not the usual order:
144+
a bare Debian image has neither git nor ca-certificates, and
145+
`actions/checkout` needs both. Nothing in that step needs the repository.
146+
127147
| Code | Tier | Notes |
128148
|---|---|---|
129149
| MOPAC | CI | Debian `mopac`; seconds per job |

‎tests/e2e/run_tests.py‎

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424

2525
import argparse
2626
import os
27+
import re
2728
import shutil
2829
import sys
2930
import tempfile
@@ -65,6 +66,38 @@ def required_codes():
6566
return set(c.strip() for c in raw.split(',') if c.strip())
6667

6768

69+
# Lines that are the code SAYING something went wrong, as opposed to
70+
# the frames it printed afterwards.
71+
_ERROR_LINE = re.compile(
72+
r'error|fatal|abort|overflow|stack smashing|not found|no such|'
73+
r'cannot|unable|%%%%|\*\*\*', re.IGNORECASE)
74+
75+
76+
def _diagnosis(text):
77+
"""The lines worth showing from a failed run's output.
78+
79+
The tail alone is the wrong choice for a crash: a Fortran code that
80+
aborts prints its message FIRST and then a backtrace, so the last
81+
dozen lines are all "#14 0x... in ???" and the one line naming the
82+
cause has scrolled off. That is what the first version of this did,
83+
and it reported a buffer overflow as a list of hex addresses.
84+
85+
So: the lines that look like a diagnosis, then the tail for context.
86+
"""
87+
lines = [l.rstrip() for l in text.splitlines() if l.strip()]
88+
if not lines:
89+
return ['(the code produced no output)']
90+
91+
said = [l for l in lines if _ERROR_LINE.search(l)][:8]
92+
tail = lines[-6:]
93+
94+
out = list(said)
95+
for l in tail:
96+
if l not in out:
97+
out.append(l)
98+
return out
99+
100+
68101
def run_case(case, verbose, keep, required=()):
69102
name = case['name']
70103
codedef = CASEDEFS.CODES[case['code']]
@@ -135,12 +168,11 @@ def run_case(case, verbose, keep, required=()):
135168
detail = open(job_out, errors='replace').read()
136169
except IOError:
137170
detail = ''
138-
tail = [l for l in detail.splitlines() if l.strip()][-12:]
139171
report.check(False,
140172
'the code itself failed (exit %d) -- every property '
141173
'below is missing because of that, not because of '
142174
'the parsers:\n %s'
143-
% (rc, '\n '.join(tail)))
175+
% (rc, '\n '.join(_diagnosis(detail))))
144176
return report
145177

146178
report.check(True, 'code ran and produced output')

0 commit comments

Comments
 (0)