Skip to content

Commit 9ed6d44

Browse files
andyclaude
andcommitted
tests/e2e: say when the CODE failed, not just which properties are missing
CI has been red on the Quantum ESPRESSO case, reporting ok code ran and produced output FAIL TE extracted FAIL STRESS extracted FAIL PRESSURE extracted which reads exactly like a parsing bug and is not one. The suite only checked that the output FILE existed, and for a code whose output is its stdout the harness writes that file itself -- so it exists however badly the run went, and the first check passes no matter what. Now a non-zero exit fails the case immediately, quoting the code's own last words, and says in as many words that the properties below are missing because of that rather than because of the parsers. For a stdout code the diagnostic is in the output file rather than stderr, so it falls back to reading it. This does not fix the QE failure, which does not reproduce here: niobium and CI run the same QE 6.7 and the same sssp pseudopotential package to within a Debian revision, so whatever differs is not visible from the current log. The next CI run will state its own reason, which is the point. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFMUNbw2UpRPmgsqviAowb
1 parent 9300ee1 commit 9ed6d44

1 file changed

Lines changed: 26 additions & 0 deletions

File tree

‎tests/e2e/run_tests.py‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -117,6 +117,32 @@ def run_case(case, verbose, keep, required=()):
117117
report.check(False, 'code produced %s (exit %d) %s'
118118
% (case['output'], rc, err.strip()[:300]))
119119
return report
120+
121+
# A code that exits non-zero has failed, and every property
122+
# after this point will be missing for that reason and not
123+
# because anything is wrong with the parsers. Reported here,
124+
# with the code's own last words, because the alternative --
125+
# which this suite did for a while -- is a list of absent
126+
# properties that reads exactly like a parsing bug.
127+
#
128+
# The message matters more than the check: for a code whose
129+
# output IS its stdout, the file exists no matter how badly the
130+
# run went, so the diagnostic is in the file rather than stderr.
131+
if rc != 0:
132+
detail = err.strip()
133+
if not detail:
134+
try:
135+
detail = open(job_out, errors='replace').read()
136+
except IOError:
137+
detail = ''
138+
tail = [l for l in detail.splitlines() if l.strip()][-12:]
139+
report.check(False,
140+
'the code itself failed (exit %d) -- every property '
141+
'below is missing because of that, not because of '
142+
'the parsers:\n %s'
143+
% (rc, '\n '.join(tail)))
144+
return report
145+
120146
report.check(True, 'code ran and produced output')
121147

122148
# --- stage 2: the real monitor --------------------------------

0 commit comments

Comments
 (0)