Skip to content

Commit d95da94

Browse files
andyclaude
andcommitted
tests/install: check that the installed ECCE matches the working tree (#124)
ECCE runs a mix of compiled binaries and interpreted files -- Perl parsers, wxPython dialogs, XML config, basis data. The interpreted ones are read from $ECCE_HOME at run time, so they drift from the repository without any build failing and without the application saying anything. That has cost real time twice, both times presenting as something else: A Gaussian dispersion dropdown was reported as reverted. Nothing was reverted; the installed ged16theory.py predated the commit and had no Dispersion field at all. A job-monitor fix was reported as not working. The installed eccejobstore was eighteen minutes older than the build. Both were one diff away from obvious. This runs that diff and reports four kinds of drift: files missing from the install (usually the scripts/ install() rule CLAUDE.md records hitting five times), files differing, leftovers from an earlier version, and binaries older than the build. Binaries are compared by mtime rather than content, because the packaged one is stripped and legitimately differs byte for byte. siteconfig/ is excluded: CMake rewrites parts of it at configure time, so comparing it would cry wolf every run. Not for CI, where build and install happen together and cannot drift. It is a developer-box tool for the moment a fix "didn't take", before re-reading the code and certainly before concluding something was reverted. Run against the current state it correctly names tonight's drift: the new mopac.mgf missing from the install, ai.mopac and mopac.desc differing, the five .edml files changed for #96, and 24 stale binaries. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EFMUNbw2UpRPmgsqviAowb
1 parent dfb2cf3 commit d95da94

2 files changed

Lines changed: 240 additions & 0 deletions

File tree

‎tests/install/README.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
# tests/install — is the ECCE you are running the ECCE you are reading?
2+
3+
```
4+
tests/install/run_tests.py check the default install (/opt/ecce)
5+
tests/install/run_tests.py -v list matching files too
6+
ECCE_TEST_HOME=... run_tests.py check an install elsewhere
7+
ECCE_TEST_BUILD=... run_tests.py compare binaries against another build
8+
```
9+
10+
Exit 0 when the install matches, 1 on drift, 2 when there is no install.
11+
12+
## Why
13+
14+
ECCE runs a mix of compiled binaries and interpreted files — Perl
15+
parsers, wxPython dialogs, XML config, basis data. The interpreted ones
16+
are read from `$ECCE_HOME` at run time, so they can drift from the
17+
repository without any build failing and without the application saying
18+
anything.
19+
20+
Guessing wrong about that has cost real time twice:
21+
22+
- A Gaussian dispersion dropdown was reported as reverted. Nothing was
23+
reverted — `/opt/ecce/scripts/codereg/ged16theory.py` predated the
24+
commit and had no Dispersion field at all.
25+
- A job-monitor fix was reported as not working. The installed
26+
`eccejobstore` was eighteen minutes older than the build.
27+
28+
Both were one `diff` away from being obvious. This runs that diff.
29+
30+
## What it checks
31+
32+
| | |
33+
|---|---|
34+
| **MISSING** | in the repo, absent from the install — usually a file under `scripts/` with no `install()` rule in CMakeLists.txt, the trap CLAUDE.md records hitting independently five times |
35+
| **DIFFERS** | present in both, different bytes |
36+
| **LEFTOVER** | in the install, absent from the repo — residue from an earlier version, like the bundled-NWChem files |
37+
| **STALE** | an installed binary older than the one in `build-cmake` |
38+
39+
Binaries are compared by **mtime, not content**: the packaged binary is
40+
stripped, so it legitimately differs byte for byte from the build-tree
41+
one. What matters is whether it predates the last build.
42+
43+
`siteconfig/` is deliberately **not** compared. CMake rewrites several of
44+
those files at configure time (paths, ports), so the installed copies are
45+
expected to differ and comparing them would cry wolf on every run.
46+
47+
## Where it belongs
48+
49+
Not CI. There the build and the install happen in one go and cannot
50+
drift, so the check would always pass and prove nothing.
51+
52+
This is a developer-box tool, and the right moment to reach for it is
53+
when a fix "didn't take" — before re-reading the code, and certainly
54+
before concluding that something was reverted.

‎tests/install/run_tests.py‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
#!/usr/bin/env python3
2+
"""Is the ECCE you are running the ECCE you are reading?
3+
4+
tests/install/run_tests.py check the default install
5+
tests/install/run_tests.py -v list every file, not just drift
6+
ECCE_TEST_HOME=/some/where ... check a different install
7+
8+
ECCE runs a mix of compiled binaries and interpreted files -- Perl
9+
parsers, wxPython dialogs, XML config, basis data. The interpreted ones
10+
are read from ``$ECCE_HOME`` at run time, so they can drift from the
11+
repository without any build failing and without the application saying
12+
anything. What you are running stops being what you are reading, and
13+
nothing tells you.
14+
15+
This is not a correctness test and does not belong in CI, where the
16+
build and the install happen in one go and cannot drift. It is the
17+
first thing to run on a developer box when a fix "didn't take" -- which
18+
is worth automating because guessing wrong at it has cost real time
19+
twice:
20+
21+
* A Gaussian dispersion dropdown was reported as reverted. Nothing
22+
was reverted; /opt/ecce/scripts/codereg/ged16theory.py predated the
23+
commit and had no Dispersion field at all.
24+
* A job-monitor fix was reported as not working. The installed
25+
eccejobstore was eighteen minutes older than the build.
26+
27+
Exit status is 0 when the install matches, 1 when it has drifted, and 2
28+
when there is no install to compare against.
29+
"""
30+
31+
import argparse
32+
import filecmp
33+
import os
34+
import subprocess
35+
import sys
36+
37+
HERE = os.path.dirname(os.path.abspath(__file__))
38+
ROOT = os.path.dirname(os.path.dirname(HERE))
39+
40+
INSTALL = os.environ.get("ECCE_TEST_HOME", "/opt/ecce")
41+
BUILD = os.environ.get("ECCE_TEST_BUILD", os.path.join(ROOT, "build-cmake"))
42+
43+
# Trees installed verbatim from the source tree, so a byte difference is
44+
# real drift. Deliberately NOT siteconfig: CMake rewrites several of
45+
# those files at configure time (paths, ports), so the installed copy is
46+
# expected to differ and comparing them would cry wolf every run.
47+
VERBATIM = [
48+
"scripts/parsers",
49+
"scripts/codereg",
50+
"data/client/cap",
51+
"data/client/config",
52+
]
53+
54+
# Noise: caches, editor droppings, and files a running ECCE writes back.
55+
IGNORE_NAMES = {"__pycache__", ".gitignore"}
56+
IGNORE_SUFFIX = (".pyc", "~", ".orig", ".rej", ".bak")
57+
58+
59+
def interesting(name):
60+
return (name not in IGNORE_NAMES
61+
and not name.endswith(IGNORE_SUFFIX))
62+
63+
64+
def walk(root):
65+
"""Relative paths of the interesting files under root."""
66+
found = set()
67+
for dirpath, dirnames, filenames in os.walk(root):
68+
dirnames[:] = [d for d in dirnames if interesting(d)]
69+
for name in filenames:
70+
if interesting(name):
71+
full = os.path.join(dirpath, name)
72+
found.add(os.path.relpath(full, root))
73+
return found
74+
75+
76+
class Result(object):
77+
def __init__(self):
78+
self.missing = [] # in the repo, absent from the install
79+
self.differs = [] # present in both, different content
80+
self.extra = [] # in the install, absent from the repo
81+
self.stale = [] # binaries older than the build
82+
83+
84+
def compareTrees(res, verbose):
85+
for rel in VERBATIM:
86+
src = os.path.join(ROOT, rel)
87+
dst = os.path.join(INSTALL, rel)
88+
if not os.path.isdir(src):
89+
continue
90+
if not os.path.isdir(dst):
91+
res.missing.append("%s/ (whole directory)" % rel)
92+
continue
93+
94+
srcFiles = walk(src)
95+
dstFiles = walk(dst)
96+
97+
for name in sorted(srcFiles - dstFiles):
98+
# The trap CLAUDE.md records: a file under scripts/ with no
99+
# install() rule is simply absent from the package, and
100+
# presents as "command not found" only once installed.
101+
res.missing.append(os.path.join(rel, name))
102+
for name in sorted(dstFiles - srcFiles):
103+
res.extra.append(os.path.join(rel, name))
104+
for name in sorted(srcFiles & dstFiles):
105+
a = os.path.join(src, name)
106+
b = os.path.join(dst, name)
107+
if not filecmp.cmp(a, b, shallow=False):
108+
res.differs.append(os.path.join(rel, name))
109+
elif verbose:
110+
print(" same %s" % os.path.join(rel, name))
111+
112+
113+
def compareBinaries(res, verbose):
114+
"""Installed binaries against the build tree.
115+
116+
Compared by MTIME rather than content: the packaged binary is
117+
stripped, so it legitimately differs byte for byte from the one in
118+
build-cmake. What matters is whether it predates the last build.
119+
"""
120+
binDir = os.path.join(INSTALL, "bin")
121+
if not os.path.isdir(binDir) or not os.path.isdir(BUILD):
122+
return
123+
for name in sorted(os.listdir(binDir)):
124+
built = os.path.join(BUILD, name)
125+
installed = os.path.join(binDir, name)
126+
if not os.path.isfile(built) or not os.path.isfile(installed):
127+
continue
128+
builtTime = os.path.getmtime(built)
129+
instTime = os.path.getmtime(installed)
130+
# A second of slack: the packaging step copies these, so exact
131+
# equality is not expected even when they correspond.
132+
if builtTime > instTime + 1:
133+
res.stale.append((name, instTime, builtTime))
134+
elif verbose:
135+
print(" current bin/%s" % name)
136+
137+
138+
def main():
139+
ap = argparse.ArgumentParser()
140+
ap.add_argument("-v", "--verbose", action="store_true")
141+
args = ap.parse_args()
142+
143+
print("Install parity: %s\n against: %s\n" % (INSTALL, ROOT))
144+
if not os.path.isdir(INSTALL):
145+
print("No install found at %s." % INSTALL)
146+
print("Set ECCE_TEST_HOME if it lives elsewhere.")
147+
return 2
148+
149+
res = Result()
150+
compareTrees(res, args.verbose)
151+
compareBinaries(res, args.verbose)
152+
153+
for path in res.missing:
154+
print(" MISSING %s" % path)
155+
print(" in the repo, not in the install -- check for an")
156+
print(" install() rule in CMakeLists.txt")
157+
for path in res.differs:
158+
print(" DIFFERS %s" % path)
159+
for path in res.extra:
160+
print(" LEFTOVER %s" % path)
161+
print(" in the install, not in the repo -- residue from")
162+
print(" an earlier version")
163+
for name, instTime, builtTime in res.stale:
164+
import time
165+
print(" STALE bin/%s" % name)
166+
print(" installed %s, built %s"
167+
% (time.strftime("%Y-%m-%d %H:%M", time.localtime(instTime)),
168+
time.strftime("%Y-%m-%d %H:%M", time.localtime(builtTime))))
169+
170+
total = (len(res.missing) + len(res.differs)
171+
+ len(res.extra) + len(res.stale))
172+
print("")
173+
if total == 0:
174+
print("PASS the installed ECCE matches this working tree")
175+
return 0
176+
print("DRIFT %d file(s): %d missing, %d differing, %d leftover, "
177+
"%d stale binaries" % (total, len(res.missing), len(res.differs),
178+
len(res.extra), len(res.stale)))
179+
print("")
180+
print("Anything tested against this install is testing the older copy.")
181+
print("Rebuild and reinstall before drawing conclusions from a GUI test.")
182+
return 1
183+
184+
185+
if __name__ == "__main__":
186+
sys.exit(main())

0 commit comments

Comments
 (0)