Contributions are welcome. agentstats keeps its dependency list short (numpy, scipy, pandas, tabulate) and its claims honest.
- Loaders for other agent benchmarks
- Additional variance, significance or minimum-runs methods
- Worked examples on public run data
- Bug fixes, clearer docs, faster code
- No new dependencies without an issue first. The current set is declared in pyproject.toml.
- Tests for behavior changes.
python3 -m pytest tests/ -qmust stay green. - Deterministic. Seed everything.
- Honest numbers. If a method does not change the conclusion on the bundled example, say so in the PR.
- Real data needs a license. New bundled examples must come from public datasets with a stated license (the current one is ASSERT-KTH, CC-BY-4.0).
- Fork the repo and create a branch from
main. - Make your change with tests.
- Run the full suite:
python3 -m pytest tests/ -q. - Open a pull request describing what changed and the before/after numbers.