agent-loop-fenced

An autonomous research loop, and the fence that keeps its objective from being gamed by the agent optimising against it.

A language model proposes a code change. A harness compiles it, checks it for escapes, runs it under timeout and crash containment, scores it, and either commits it to a ledger or reverts it. Then it does that again, forever. That loop is easy to write and hard to trust, because the moment you hand an optimiser a number and ask it to raise the number, every split that number is computed on becomes training data.

This repository contains the loop, the containment, and a fence that closes that gap: three data splits with different rights, a typed channel the agent's prompt is built from, a process boundary around the holdout, a veto with a counted look budget, and a Deflated Sharpe Ratio computed against the loop's own trial count. Every piece is tested. The whole thing runs on synthetic data with no API key.

---

Quickstart

pip install -r requirements.txt

python3 demo/run_demo.py          # the loop, twice: unfenced and fenced, then the reveal
python3 demo/replay_lineage.py    # a real search lineage, replayed through the gates
python3 -m pytest tests/ -q       # 82 tests

No API key, no network, no vendor data. The proposer in the demo is an offline mutation search that is greedy in the same way a model steered by a scalar is greedy. To drive the same loop with a real model:

python3 demo/run_with_claude.py --iterations 40 --model sonnet

---

The mechanism

The outer loop (fenced_loop/loop.py) is about two hundred lines and does six things per iteration.

Step	Where	What it does
Build the agent's view	prompt.py	Assemble an AgentView — a frozen dataclass with a fixed field set — and render the prompt from it and nothing else
Get a proposal	proposers.py	Headless claude -p --model sonnet --no-session-persistence --allowedTools "", marker-delimited output, tolerant of markdown fences and truncation
Gate the source	program.py	compile() syntax gate, then an AST walk that refuses imports outside a numeric allowlist, open/eval/exec/__import__, dunder attribute access, and missing entry points
Evaluate	backtest.py, metrics.py	Bar-by-bar simulation with entry slippage, stop slippage and commission; the candidate's own exceptions are caught and become a revert
Decide	fence.py, gates.py	Safety, then the train-split gate cascade, then the steering score, then the holdout veto — in that order, so the expensive check runs last
Record	ledger.py	One TSV row per iteration carrying the split every number came from, plus an optional git commit per kept change

Two containment details worth naming. Candidate code executes with a minimal __builtins__ mapping and no __file__, so a module cannot locate itself on disk and walk to a sibling data file. And a candidate that raises does not crash the loop — the exception is contained, the iteration is recorded as a revert, and the search continues.

The git ledger is the original design's best idea and it is preserved: one commit per accepted change, hypothesis and score delta in the message, revert by checkout. It is opt-in here, so importing the package cannot mutate a repository.

---

The fence

The problem in one sentence: chronological ordering does not make a split out-of-sample; not having been selected on does.

Feed an agent a scalar computed on segment X, keep the proposal when it rises and revert it when it falls, and after N iterations you have run N steps of selection against X. Whatever the calendar says, X is now in-sample.

Five mechanisms, each separately testable.

1. Three splits with different rights

TRAIN       the agent may see everything: bars, trades, metrics, per-quarter breakdown
VALIDATION  exactly one scalar per iteration. This channel is *expected* to leak --
            steering an optimiser is what it is for
HOLDOUT     nothing. Not the value, not the sign, not the trade count, not whether it moved

DataSpec refuses to construct if any session appears in two splits.

2. A typed channel

The prompt is rendered from an AgentView whose field set contains no holdout quantity, and render_prompt accepts nothing else. Scores are ScoreCard objects that refuse float() and refuse comparison across splits:

>>> validation = ScoreCard(Split.VALIDATION, 1.0)
>>> holdout = ScoreCard(Split.HOLDOUT, 2.0)
>>> holdout > validation
FenceViolation: refusing to compare a holdout score to a validation score:
they are not the same quantity

Behind the types, every rendered prompt is scanned for every holdout-derived number the run has produced, in every format this codebase prints. Types stop the honest mistake — someone adding a helpful "holdout so far" line. The scan stops the one that gets past types.

The scan has a stated limit: renderings with fewer than five significant digits are not enforced, because 1.54 is a plausible profit factor and a plausible holdout Sharpe at once, and a scanner that cries wolf gets switched off. That limitation is pinned by a test rather than left in a comment.

3. A process boundary

Holdout scoring runs in a child interpreter with a scrubbed environment, an empty scratch working directory rather than the directory the split files live in, and a single JSON document as the only return channel. The static safety check is a static check and can be wrong; the process boundary makes a missed AST case cost a wrong score instead of a silent read. test_the_child_process_is_not_run_inside_the_split_directory pins that.

4. A veto, not a maximand — with a counted budget

The holdout is never maximised and never ranked on. It can only refuse a promotion that has already cleared safety, cleared the train gates, and improved the steering score. Each refusal costs one look from a fixed budget, and every look is recorded with its iteration and its reason:

holdout looks recorded : 10 of 10
  iter 1    promotion_check_iter_1
  iter 4    promotion_check_iter_4
  ...
  iter 31   promotion_check_iter_31

Ordering the stages this way bounds holdout looks by promotions attempted, not by iterations. A veto with a budget of K leaks at most K bits. A maximand leaks without bound. When the budget is spent the run keeps exploring but stops certifying, and the honest next move is new data rather than a bigger budget.

5. Deflation by the loop's own trial count

The veto threshold is a Deflated Sharpe Ratio evaluated against the number of trials the search has actually run — a number the loop knows exactly, because it evaluated every one of them. The more the search looks, the higher the bar:

trials evaluated       : 32
trial Sharpe variance  : 0.033784   (measured, not assumed)

---

The worked example

The loop this repository is built on finished a run reporting an out-of-sample Sharpe ratio of 54.03.

The harness did exactly what it was written to do. Its evaluator held two of five chronological periods back, computed a Sharpe on those and printed it as SCORE. The loop fed that score to the agent every iteration as CURRENT BEST SCORE, kept the proposal when it rose, reverted it when it fell, and recorded each keep as a commit. Sixty iterations later the reported out-of-sample statistic was 54.03, and the twelve accepted programs — preserved verbatim in lineage/ — read as a clean, legible search:

recorded objective    accepted change
        -0.1615       (starting point)
         0.6515       long side only
         0.8831       take-profit multiplier 1.5 -> 1.0
         1.7575       later session close
         1.8188       stop buffer to zero
         2.8097       volume lookback 20 -> 10 bars
        11.1336       maximum opening range 200 -> 100 points
        19.4169       require the whole breakout bar above the boundary
        31.5133       opening range 15 -> 30 bars
        36.3469       take-profit multiplier 1.0 -> 0.75
        53.8278       take-profit multiplier 0.75 -> 0.5
        54.0271       volume filter 1.5x -> 1.2x

Every step is a plausible hypothesis. Every step raised the number. The number was the wrong number, because it was the one the agent was steered by, and by the end that "out-of-sample" split had absorbed sixty steps of selection.

The harness worked. The objective was reachable. A loop that ships without a fence is a loop whose headline figure cannot be interpreted at any magnitude — and it is worth saying that the failure is legible only because the ledger recorded the hypothesis, the score delta and the commit for every step. A harness that records less would have produced 54.03 and no way to see where it came from.

Reproducing it

demo/run_demo.py runs the same loop twice on identical synthetic data with an identical proposer — once in the arrangement above, once fenced — and then scores both winners on a fourth block of sessions that neither run has ever touched.

==============================================================================
  RUN A -- UNFENCED: the agent is steered by the split the run reports
==============================================================================
[0] baseline sharpe (holdout split) = 2.325400 (train n=314, gt=0.3519)
[4] KEEP sharpe=    3.5013 best=    3.5013 n=279
[5] KEEP sharpe=    4.4418 best=    4.4418 n=237
[6] KEEP sharpe=    7.0243 best=    7.0243 n=73
[7] KEEP sharpe=  102.3898 best=  102.3898 n=6

Four promotions. Read the trade count in the last column: 314, 279, 237, 73, 6.

==============================================================================
  RUN B -- FENCED: steered by validation, gated on train, holdout vetoes only
==============================================================================
[12] KEEP GT-score=    1.0081 best=    1.0081 n=239
[14]   .  GT-score=    0.0000 best=    1.0081 n=124  train_gates: volume_floor (124 < 157, 50% of initial 314)
[20] KEEP GT-score=    1.9108 best=    1.9108 n=206
[22]   .  GT-score=    0.0000 best=    1.9108 n=102  train_gates: volume_floor (102 < 157, 50% of initial 314)
[31] KEEP GT-score=    2.2727 best=    2.2727 n=206
     holdout budget exhausted after 10 looks; no further promotions will be certified

Ten promotions, and the trade count holds at 206 of an original 314.

==============================================================================
  THE REVEAL -- both winners scored once on the untouched audit sessions
==============================================================================
  run             claimed   computed on   train n   audit Sharpe   audit n    audit $
  UNFENCED       102.3898       holdout         6       175.4768         2         82
  FENCED           2.2727    validation       206         6.2685       102       3532

The unfenced winner's headline figure is not merely optimistic. It is a ratio computed on six trades, from a program that takes two trades across a hundred and sixty untouched sessions. It is not an estimate of anything, in either direction — and that is the point: without the fence you cannot tell which direction it is wrong in.

==============================================================================
  AUDIT
==============================================================================
  UNFENCED
    [LEAK] steering split = holdout
      holdout looks / budget : 0 / 0
      The number this run reports as out-of-sample was the number it was steered by.
      It is an in-sample fit statistic and cannot be read as evidence of
      generalisation at any magnitude.
  FENCED
    [CLEAN] steering split = validation
      holdout looks / budget : 10 / 10
      The reported holdout figure was consulted only as a veto, within budget, and
      never entered the agent channel.

The leak test is provenance, not statistics. It asks which split the accept decision was conditioned on. A statistical leak detector has to out-argue a search that is optimising against it; a provenance tag does not have to argue.

---

The gates, verified

Three gates from the original scoring module survive here because they cost a program something for shrinking, which is the direction every unconstrained ratio search travels:

• min_trades — a ratio on four trades is not a measurement.
• a volume floor at 50% of the original baseline's trade count — anchored to the first baseline, never the current one. Without a permanent anchor the floor ratchets: accept a candidate at 60% of the trades and next iteration 60% of that passes. tests/test_gates.py pins both behaviours.
• the ln(N) term in the GT-score mu * ln(N) * R^2 / sigma, so a result is worth more when it rests on more observations.

Run by hand against the unfenced winner:

  baseline train trades : 314
  winner   train trades : 6
  winner   train GT     : 0.0
  gate verdict          : REJECT  -- insufficient_trades (6 < 15)

And applied to the historical lineage, replayed on the shipped synthetic data by demo/replay_lineage.py:

  gen09  REJECT  volume_floor (125 < 157, 50% of initial 314)
  gen10  REJECT  volume_floor (125 < 157, 50% of initial 314)
  gen11  REJECT  volume_floor (125 < 157, 50% of initial 314)
  gen12  REJECT  volume_floor (155 < 157, 50% of initial 314)

  Recorded objective, first accepted program to last: -0.1615 -> 54.0271
  Train trade count over the same twelve steps: 314 -> 155 (-51%)

The recorded scores come from bars that are not in this repository and are quoted only as that run's own ledger; the gate verdicts above are computed on the synthetic data shipped here, which is why they are reproducible.

---

The statistics module

fenced_loop/statistics.py implements the Probabilistic and Deflated Sharpe Ratios (Bailey and Lopez de Prado, 2012 and 2014). It is standard-library only — the normal CDF and its inverse are implemented in the file — so it can be vendored on its own.

It replaces a predecessor that looked like this:

def compute_dsr(sharpe, n, skew, kurtosis):
    se = np.sqrt((1 + 0.5*sharpe**2 - skew*sharpe
                  + (kurtosis - 3)/4 * sharpe**2) / max(n - 1, 1))
    return round(norm.cdf(sharpe / se), 4)

Across 1,313 recorded evaluations that function returned 1,129 exact 0.0 values and 184 exact 1.0 values and nothing in between. It was a binary indicator of n >= 10 and sharpe > 0 wearing the name of the one statistic in the file meant to detect overfitting.

The variance term is algebraically correct. Three things around it were not, and a fourth was a naming error:

1. Units. Callers passed an annualised Sharpe (mu/sigma*sqrt(252)) into a formula whose sharpe is per-observation — inflating the argument 15.87x. In the large-Sharpe limit the sharpe**2 term dominates the variance, sharpe cancels out of sharpe/se, and the statistic collapses to roughly sqrt(2*(n-1)), which is 5.3 at n = 15. norm.cdf(5.3) rounds to 1.0 at four decimals.
2. Sample size. n was the trade count while the Sharpe was estimated from daily aggregates — two different sample sizes.
3. Kurtosis convention. The (kurtosis - 3)/4 term wants Pearson kurtosis; callers passed pandas.Series.kurtosis(), which is Fisher. A normal sample entered as -3 instead of 0.
4. Name. The statistic computed was the Probabilistic Sharpe Ratio against a benchmark of zero, not a Deflated one.

Point 4 is the interesting one. The deflated version differs by setting the benchmark to the Sharpe you would expect the best of N trials to show by luck alone. Everywhere else that N is a nuisance parameter you have to guess. In a search loop it is an observable, because the loop counts its own trials and measures their spread — which is why the deflated form is the statistic this repository actually wants, and why deflated_sharpe_ratio(sample, n_trials, sharpe_variance_across_trials) has no default for that last argument.

The corrected statistic on the same input domain that made its predecessor constant:

                trials=1     trials=5    trials=13    trials=60   trials=500
DSR              0.98087      0.86563      0.75559      0.56832      0.34429

tests/test_statistics.py pins the defect (test_legacy_step_function_is_degenerate), pins the fix at exact interior values, and pins monotonicity in Sharpe, in sample size, in skew, in kurtosis and in trial count. The predecessor is kept in the module as legacy_step_function for exactly one reason: so that reinstating its arithmetic fails a test by name.

One limit, stated plainly. The sqrt(n - 1) term punishes a short sample only as sqrt(n), while a Sharpe estimated from a handful of sessions can be arbitrarily large. A program that took two trades can therefore carry a high deflated Sharpe. No single statistic closes that door — the trade-count gate and the volume floor do. Fence.holdout_dsr returns 0.0 below ten return observations for the same reason, and the demo prints the refusal rather than a number.

---

Repository layout

fenced_loop/
  splits.py           Split, ScoreCard, FenceViolation -- provenance types
  fence.py            Fence, HoldoutBudget, AcceptanceRule, audit
  prompt.py           AgentView, render_prompt, the channel scanner
  program.py          syntax gate, static safety check, isolated exec
  holdout_runner.py   out-of-process holdout scoring
  gates.py            the gate cascade
  statistics.py       PSR and DSR, standard library only
  metrics.py          trade-ledger metrics, carrying units explicitly
  backtest.py         bar-by-bar simulator
  data.py             the supplied-data seam and the synthetic generator
  proposers.py        Claude CLI, offline mutation search, scripted replay
  loop.py             the outer loop, fenced and unfenced
  ledger.py           TSV telemetry and the optional git ledger
candidate/            the program the agent rewrites, and its instructions
lineage/              twelve accepted programs from the original run, verbatim
demo/                 run_demo.py, replay_lineage.py, run_with_claude.py
tests/                82 tests

---

Using your own data

The evaluator no longer opens a path. It is handed a DataSpec:

from fenced_loop.data import DataSpec
data = DataSpec.from_dir("path/to/your/bars")

with a manifest that assigns each file to a split:

{
  "timestamp_column": "minute",
  "segments": {
    "train":      ["2024_h1.csv", "2024_h2.csv"],
    "validation": ["2025_h1.csv"],
    "holdout":    ["2025_h2.csv"]
  }
}

Each CSV needs open, high, low, close, volume and the timestamp column. Construction fails if a session appears in more than one split.

The synthetic generator contains a known, planted edge — roughly a third to a half of sessions continue in the direction price left the opening range — and a slow regime drift across the series so that a threshold tuned on one block is mistuned on the next. Both are stated because they are what make the demonstration legible rather than flattering. Nothing here claims the generated series resembles any real market.

---

What this does not do

• It does not make a holdout reusable. Spend the budget and it is spent.
• The static safety check is sized for an optimiser taking shortcuts, not for a hostile author with a Python exploit. The process boundary is what makes any single missed case non-catastrophic.
• It does not detect a leak that arrives through the data itself. Overlapping sessions are refused; a feature computed across a split boundary is the author's problem.
• The channel scanner is entropy-limited by construction, as described above.
• The audit reports selection pressure as log2(iterations), which is an accounting convenience, not a tight bound.

---

What is not in this repository

This is one component of a larger system. The rest of it is inseparable from private data and live credentials, and it is not extractable in a form anyone could usefully read — so what is here was rebuilt to stand on its own: the loop, the fence, the statistics, synthetic data, and a demonstration a stranger can run in one command.

What is here is complete on its own terms. The fence is the engineering, the tests are the evidence, and the worked example is the reason the fence is shaped the way it is.

Read access to more can be arranged for a serious conversation.

---

License

MIT. See LICENSE.