Add AGENTS.md / CLAUDE.md by ndellosa95 · Pull Request #23275 · pantsbuild/pants

ndellosa95 · 2026-04-20T15:40:33Z

I think we can all agree that AI coding is here to stay at this point. Even as someone who was resistant for a long time, I've now been using Claude more and more and admittedly it can be extremely effective with surgical prompts and some light editing. I used it fairly extensively for this PR and generating the AGENTS.md / CLAUDE.md, which I had it generate from the Pants docsite, made it much more effective at navigating the repo and using pants correctly.

cburroughs · 2026-04-20T18:52:39Z

Are these all things things you have had to tell a LLM? That is things it could not figure out from the existing context?

To be concrete, I use Claude Code with Opus and don't recall having to tell it to pants test instead of pytest or have it get stuck thinking it could run mypy in a venv. Did you have an issue with non-frozen dataclasses?

ndellosa95 · 2026-04-20T19:44:57Z

Are these all things things you have had to tell a LLM? That is things it could not figure out from the existing context?

To be concrete, I use Claude Code with Opus and don't recall having to tell it to pants test instead of pytest or have it get stuck thinking it could run mypy in a venv. Did you have an issue with non-frozen dataclasses?

Also using claude with opus and yeah, without the CLAUDE.md it would try and globally pip install stuff that it should be running with pants. Idk if my org has something setup with a bunch of Python context baked in already or what, but it was hugely frustrating.

sureshjoshi · 2026-04-21T03:05:05Z

Interesting, I've been experimenting with Composer2 and haven't had to tell it any of this stuff. I just told it to first look at the actions to figure out what to do and sent it off to make a bunch of experimental, mid, changes.

sureshjoshi · 2026-04-21T03:06:21Z

Also, you should strip out the CLAUDE.md from your other PR so it's not conflated.

ndellosa95 · 2026-04-21T15:59:56Z

I supposed it's not as extensive as what I've got here but the bazel repo also has an AGENTS.md with some details on how to use bazel so I swear I'm not crazy. The additional context could also be potentially helpful for poor souls who try to make a go at making changes with less powerful models. Happy to amend or remove anything from the doc though.

cburroughs · 2026-04-21T18:20:00Z

To be clear, I don't think you are crazy! I just have no idea how to test this sort of thing and 'best practices' are in wild flux. At DAYJOB we have a wide spread between "LLMs can't work with pants unless we repeat --no-dynamic-ui at least 150 times in instructions" and "eh, it mostly just works".

benjyw · 2026-04-21T20:19:08Z

+)
+```
+
+All BUILD files must have the copyright header:


I have seen LLMs use the literal year (2024 in this case) from these kinds of instructions instead of updating it to the current year. Maybe a nudge to update the year will teach them to be sensible.

benjyw · 2026-04-21T20:20:00Z

+| Run a single test function | `pants test path/to/file_test.py -- -k test_function_name` |
+| Debug a test (interactive) | `pants test --debug path/to/file_test.py` |
+| Debug with specific test | `pants test --debug path/to/file_test.py -- -k test_name` |
+| Lint changed files | `pants --changed-since=HEAD lint` |


You might want to --changed-since=main ? If you're stacking commits and you don't run this on every commit.

benjyw · 2026-04-21T20:20:41Z

+
+```bash
+# Run tests matching a pattern
+pants test path/to/file_test.py -- -k "test_foo or test_bar"


You can point out that everything after -- is passed through to pytest, and tell the LLM to go learn the pytest cli?

I think it's worth mentioning the passthrough behavior, so that the LLM doesn't go off on a tangent trying to learn those options from Pants when they are actually pytest options.

benjyw · 2026-04-21T20:21:11Z

+pants test --output=all path/to/file_test.py
+
+# Force re-run (skip cache)
+pants --no-local-cache test path/to/file_test.py


Specifically for tests you can test --force which forces the test to rerun but allows everything up to that point to use the cache.

benjyw · 2026-04-21T20:23:00Z

+
+### File headers
+
+All source files must have the copyright header:


Did we not already say this above?

benjyw · 2026-04-21T20:23:34Z

+
+### Test function naming
+
+- Test files: `*_test.py` (NOT `test_*.py`)


Again, why repeat above?

benjyw · 2026-04-21T20:24:10Z

+### Integration vs unit tests
+
+- Unit tests: `*_test.py` - run in normal test target
+- Integration tests: `*_integration_test.py` - get their own BUILD target with longer timeouts


integration tests typically run an instance of Pants, whereas unit tests run individual functions or rules.

benjyw · 2026-04-21T20:25:16Z

+
+PEX is special - update both:
+1. The `pex-cli` subsystem in `src/python/pants/backend/python/util_rules/pex_cli.py`
+2. The requirement in `3rdparty/python/requirements.txt` and regenerate lockfile


This is not true any more. Pex is no longer consumed as a requirement, only as a CLI tool.

If this is still mentioned in the docs somewhere then that needs fixing.

ndellosa95 · 2026-04-22T13:34:00Z

To be clear, I don't think you are crazy! I just have no idea how to test this sort of thing and 'best practices' are in wild flux. At DAYJOB we have a wide spread between "LLMs can't work with pants unless we repeat --no-dynamic-ui at least 150 times in instructions" and "eh, it mostly just works".

I've had a lot of success getting LLMs to do what I want by threatening them with things that would probably get me flagged by HR 😆

sureshjoshi · 2026-04-22T20:23:46Z

Just a heads up, I have no decent opinions on any of this nor any good feedback.

@tdyas @cburroughs added you two as reviewers, as I think you both have more experience (definitely more than me, anyways) re: approving this kind of PR

I don't even know what a metric would be for this kinda thing 🤷🏽 My (EXTREMELY limited) experience with AGENTS.md and stuff is copied from some random Github repo that seemed to align with my mentality of "write as little code as possible, don't try to be too clever, ask when you don't understand" - seems to work 🤷🏽

sureshjoshi · 2026-04-22T20:26:39Z

+pants fmt src/python/pants/backend/python/goals/pytest_runner.py
+pants check src/python/pants/backend/python/goals/pytest_runner.py
+
+# WRONG - never do this


It hurts my soul that it's not enough to show just the positive approach, but you need to show the negative as well 🤦🏽

sureshjoshi · 2026-04-22T20:27:33Z

+
+### Running Pants from source
+
+In this repo, `./pants` is a special bootstrap script that runs Pants from the local source tree. Use `pants` (which resolves to the `./pants` script) for all operations. The first run compiles the Rust engine and may take several minutes.


For anything non-performance, until a certain upcoming PR gets done, it may be smarter to run:
MODE=debug pants to save on compilation time, and to re-use the same compiled objects for tests.

sureshjoshi · 2026-04-22T20:27:58Z

@@ -0,0 +1,555 @@
+# Pants Build System Contributor Guide
+
+This is the Pants build system repo -- a self-hosting build system where `./pants` runs Pants from source. Follow the conventions, architecture, and workflows specific to this codebase.


Later on, the script says to use pants not ./pants

sureshjoshi · 2026-04-22T20:28:39Z

+
+Every directory with source code needs a `BUILD` file. Common patterns:
+
+```python


I don't use tailor - but I think running a tailor check would ensure this rule.

sureshjoshi · 2026-04-22T20:29:53Z

+| Show target info | `pants peek path/to/target` |
+| Generate BUILD files | `pants tailor` |
+| Validate BUILD files | `pants lint --only=ruff-format '**BUILD'` |
+| Run pre-push checks | `build-support/githooks/pre-push` |


Is there value in also showing the merged forms? e.g. fix fmt lint check test?

sureshjoshi · 2026-04-22T20:33:01Z

+def rules():
+    return [
+        *collect_rules(),
+        *FooTestRequest.rules(),
+    ]


Suggested change

def rules():

return [

*collect_rules(),

*FooTestRequest.rules(),

]

def rules() -> Iterable[Rule | UnionRule]:

return (

*collect_rules(),

*FooTestRequest.rules(),

)

sureshjoshi · 2026-04-22T20:34:16Z

+- Must be complete sentences ending with a period
+- Max 100 characters per line
+- Explain **why**, not **what**
+- TODOs must reference a GitHub issue: `# TODO(#1234): Description.`


I don't understand this. Is the expectation that we'd get new issues for a TODO that the machine creates?

tdyas · 2026-04-22T20:34:22Z

What are people's thoughts on the fact that AGENT.md / CLAUDE.md are loaded unconditionally into the context window? There is a cost (in tokens) to always having the instructions loaded into the context window.

A trend I have been seeing is using SKILLs.md and let the agent choose how much to load into the context. A set of "Pants maintainer skills" would also let Pants developers choose whether they want the pollute their context window with the instructions. Moreover, we could build a more comprehensive set of "Pants maintainer skills" (for actions like adding new third-party dependencies) which we would not want in a general AGENT.md.

sureshjoshi · 2026-04-22T20:35:59Z

+async def my_rule(request: MyRequest) -> MyResult:
+    ...
+
+def rules():


Suggested change

def rules():

def rules() -> Iterable[Rule | UnionRule]:

sureshjoshi · 2026-04-22T20:36:39Z

+python_tests(
+    name="integration",
+    sources=["*_integration_test.py"],
+    timeout=240,


This is only necessary if it times out

sureshjoshi · 2026-04-22T20:40:00Z

+- Be concise but descriptive
+- Reference GitHub issues where applicable
+
+## Maintenance Tasks


I'd rather not have any PRs for lockfiles/tooling updates. I've got some PRs I'm working on to deterministically automate that stuff, so we don't need to burn down a rainforest to update a sha256.

Also, they're basically impossible to review for supply chain concerns.

sureshjoshi · 2026-04-22T20:40:55Z

+    result.assert_success()
+```
+
+## PR and Contribution Workflow


I don't see a section for the PR message itself, but full disclosure about LLM generation is important.

sureshjoshi · 2026-04-22T20:45:08Z

What are people's thoughts on the fact that AGENT.md / CLAUDE.md are loaded unconditionally into the context window? There is a cost (in tokens) to always having the instructions loaded into the context window.

A trend I have been seeing is using SKILLs.md and let the agent choose how much to load into the context. A set of "Pants maintainer skills" would also let Pants developers choose whether they want the pollute their context window with the instructions. Moreover, we could build a more comprehensive set of "Pants maintainer skills" (for actions like adding new third-party dependencies) which we would not want in a general AGENT.md.

I was literally just joking about this with a friend after seeing some "Claude Code not available on $20 plan anymore" comments on the interwebs. Can only burn so much cash, and having to maintain some long Agents files may not be ideal.

I don't know what the state-of-the-art is here, but if we had a slimmed down Agents, I would like to see some of the "never do this" cases that we've discussed over the past few weeks/months. As always, ghostty seems to have some good ideas: https://github.com/ghostty-org/ghostty/blob/main/AGENTS.md

sureshjoshi · 2026-05-08T19:10:23Z

Question on this - what are the tangible next steps? There is a lot of (really good) content in the file, but there is also the question of what kind of need this has, token budgets, skills, etc.

For a tangible next step, personally as a maintainer (again - LLMs are not really my thing), I would prefer what I've seen in some other repos I use, where AGENTS is used a bit more for HOW a clanker should interact with the repo/issues/PRs, rather than enumerating everything about the repo.

Short and sweet, and with the intention of making our lives a bit easier with PRs we don't want to see, what we expect/don't, etc.

Something more like:

https://github.com/bootc-dev/bootc/blob/main/AGENTS.md (though this calls out to other files anyways, so who knows)
https://github.com/neovim/neovim/blob/master/AGENTS.md

As I say this, I guess that could also just be a CONTRIBUTING.md that this calls out to...

tdyas · 2026-05-09T01:31:59Z

For a tangible next step, personally as a maintainer (again - LLMs are not really my thing), I would prefer what I've seen in some other repos I use, where AGENTS is used a bit more for HOW a clanker should interact with the repo/issues/PRs, rather than enumerating everything about the repo.

Yes this. Models are getting better all the time and can just read code to discover information about the code itself. AGENTS.md files should encode conventions that the AI cannot discover.

Moreover, I still have an objection to imposing large AGENTS.md files on other developers where our AI usage is going to have to include those files in our context windows and either eat up our rate limits faster and/or cost us money in tokens.

enragedginger · 2026-05-09T12:44:27Z

I don't think we should have a single large AGENTS.md or CLAUDE.md file in the root of this repo. A small one? Yes. But not a large one like this. Most of the guidelines in here could be broken down into smaller "skills" (.claude/skills) that the agent would load when it needs them.

Guidelines for BUILD files? Make a build files skill that loads when working with BUILD files.

Guidelines for running tests? Make a tests skill that loads when working with tests.

etc

add claude md

e8e24b6

ndellosa95 added the release-notes:not-required [CI] PR doesn't require mention in release notes label Apr 20, 2026

use generic AGENTS.md

e9ed545

ndellosa95 changed the title ~~Add CLAUDE.md~~ Add AGENTS.md / CLAUDE.md Apr 20, 2026

benjyw reviewed Apr 21, 2026

View reviewed changes

address comments

a2090a5

ndellosa95 requested a review from benjyw April 22, 2026 17:16

sureshjoshi requested review from cburroughs and tdyas April 22, 2026 20:17

sureshjoshi reviewed Apr 22, 2026

View reviewed changes

enragedginger reviewed May 9, 2026

View reviewed changes


		### File headers

		All source files must have the copyright header:


		### Test function naming

		- Test files: `_test.py` (NOT `test_.py`)


		### Running Pants from source

		In this repo, `./pants` is a special bootstrap script that runs Pants from the local source tree. Use `pants` (which resolves to the `./pants` script) for all operations. The first run compiles the Rust engine and may take several minutes.

		@@ -0,0 +1,555 @@
		# Pants Build System Contributor Guide

		This is the Pants build system repo -- a self-hosting build system where `./pants` runs Pants from source. Follow the conventions, architecture, and workflows specific to this codebase.


		Every directory with source code needs a `BUILD` file. Common patterns:

		```python

Uh oh!

Conversation

ndellosa95 commented Apr 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cburroughs commented Apr 20, 2026

Uh oh!

ndellosa95 commented Apr 20, 2026

Uh oh!

sureshjoshi commented Apr 21, 2026

Uh oh!

sureshjoshi commented Apr 21, 2026

Uh oh!

ndellosa95 commented Apr 21, 2026

Uh oh!

cburroughs commented Apr 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ndellosa95 commented Apr 22, 2026

Uh oh!

sureshjoshi commented Apr 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdyas commented Apr 22, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sureshjoshi commented Apr 22, 2026

Uh oh!

sureshjoshi commented May 8, 2026

Uh oh!

tdyas commented May 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

ndellosa95 commented Apr 20, 2026 •

edited

Loading