feat(policy): intent/scope, decision trace, and stable reason codes

CHANGELOG.md

@@ -7,6 +7,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- Structured intent and scope metadata on `CapabilityRequest`: new optional
+  `intent: str | None` and `scope: dict[str, Any]` fields let policy engines
+  authorize based on machine-readable intent and scope alongside the existing
+  free-text `goal`. `DeclarativePolicyEngine` rules can match on these via new
+  `intent: [...]` and `scope: {key: value}` clauses in YAML/TOML policy files.
+  Intent-aware allow rules fail closed for legacy callers that don't set an
+  intent. (#72)
+- Structured policy decision trace (`PolicyDecisionTrace` + `PolicyTraceStep`):
+  both built-in policy engines now attach a step-by-step trace to every
+  `PolicyDecision` (allow and deny paths). Each step records the rule
+  considered, the outcome (`matched`/`skipped`/`denied`/`allowed`/
+  `constraint_applied`), a human-readable detail, and — for terminal
+  steps — the stable reason code. Traces echo `intent` and `scope_keys`
+  (scope dimension names only — values redacted) from the request and contain
+  no raw argument values. `DryRunResult.policy_decision`
+  also carries a synthesized single-step trace. (#73)
+- Stable machine-readable denial reason codes: new `DenialReason` and
+  `AllowReason` enums in `agent_kernel.policy_reasons` (also exported as
+  `from agent_kernel import DenialReason, AllowReason`). Every built-in
+  denial path on `DefaultPolicyEngine` and `DeclarativePolicyEngine` populates
+  `PolicyDecision.reason_code`, `DenialExplanation.reason_code`,
+  `FailedCondition.reason_code`, and `PolicyDenied.reason_code`. Tests should
+  assert on these codes instead of matching the human-readable `reason` /
+  `narrative` strings, which remain part of the API but may evolve for
+  clarity. Codes: `missing_role`, `missing_tenant_attribute`,
+  `missing_attribute`, `insufficient_justification`, `invalid_constraint`,
+  `rate_limited`, `no_matching_rule`, `explicit_deny_rule`,
+  `intent_not_allowed`, `scope_not_allowed`; allow-side: `default_policy_allow`,
+  `rule_allow`, `default_fallthrough_allow`. (#77)
+- New public exports: `AllowReason`, `DenialReason`, `PolicyDecisionTrace`,
+  `PolicyTraceStep`.
+
+### Changed
+- `PolicyDecision` gained optional `reason_code: str | None` and
+  `trace: PolicyDecisionTrace | None` fields (both default `None` so
+  third-party engines that don't populate them keep working).
+- `DenialExplanation` and `FailedCondition` gained optional `reason_code`
+  fields populated by both built-in engines on every denial path.
+- `PolicyDenied(reason_code=...)` keyword argument: the exception now carries
+  a `reason_code` attribute so callers can branch on a stable code without
+  matching the human-readable message.
+
 ## [0.6.0] - 2026-05-19
 
 ### Added

docs/architecture.md

@@ -54,11 +54,54 @@ Both built-in engines satisfy `ExplainingPolicyEngine`:
   5. **SECRETS** — requires role `admin|secrets_reader` + `justification ≥ 15 chars`
   6. **max_rows** — 50 (user), 500 (service)
   7. **Rate limiting** — sliding-window per `(principal_id, capability_id)` (60 READ / 10 WRITE / 2 DESTRUCTIVE per 60s; service role gets 10×)
-- **`DeclarativePolicyEngine`** — loads rules from a YAML or TOML file (or a plain dict). Supports `safety_class`, `sensitivity`, `roles`, `attributes`, and `min_justification` match conditions; `allow`/`deny` actions; per-rule `constraints` merged into the resulting `PolicyDecision`; configurable `default` action. Rules are evaluated top-down with first-match-wins. `pyyaml` and `tomli` are optional dependencies — `import agent_kernel` works without them; calling `from_yaml`/`from_toml` without the parser raises `PolicyConfigError` with an install hint.
+- **`DeclarativePolicyEngine`** — loads rules from a YAML or TOML file (or a plain dict). Supports `safety_class`, `sensitivity`, `roles`, `attributes`, `min_justification`, `intent`, and `scope` match conditions; `allow`/`deny` actions; per-rule `constraints` merged into the resulting `PolicyDecision`; configurable `default` action. Rules are evaluated top-down with first-match-wins. `pyyaml` and `tomli` are optional dependencies — `import agent_kernel` works without them; calling `from_yaml`/`from_toml` without the parser raises `PolicyConfigError` with an install hint.
+
+#### Intent and scope on requests
+
+`CapabilityRequest` carries optional structured metadata alongside its free-text `goal`:
+
+- `intent: str | None` — a machine-readable label (e.g. `"customer_support_lookup"`).
+- `scope: dict[str, Any]` — a small structured map (e.g. `{"region": "eu-west", "customer_id": "C-42"}`).
+
+`DeclarativePolicyEngine` rules can match on these via top-level keys in `match`:
+
+```yaml
+- name: support_eu_lookup
+  match:
+    safety_class: [READ]
+    intent: [customer_support_lookup]
+    scope: { region: "eu-west" }
+  action: allow
+```
+
+Intent-aware rules fail closed: a request with `intent=None` never matches a rule that requires a specific intent. `scope: { key: "*" }` means "the key must be present with any value".
 
 #### Denial explanations
 
-`PolicyEngine.explain()` (when available) returns a structured `DenialExplanation` with `denied`, `rule_name`, a `failed_conditions: list[FailedCondition]` describing each missing condition with `required`/`actual`/`suggestion`, a `remediation` list, and a human-readable `narrative`. Engines collect all failing conditions (no short-circuit) so callers get the full picture. For `DeclarativePolicyEngine`, an explicit deny rule that fully matches is reported as the cause; partial-match deny rules are skipped during explanation so the surfaced advice is actionable rather than self-defeating.
+`PolicyEngine.explain()` (when available) returns a structured `DenialExplanation` with `denied`, `rule_name`, a `failed_conditions: list[FailedCondition]` describing each missing condition with `required`/`actual`/`suggestion`/`reason_code`, a `remediation` list, a human-readable `narrative`, and a top-level `reason_code` (the code of the first failed condition). Engines collect all failing conditions (no short-circuit) so callers get the full picture. For `DeclarativePolicyEngine`, an explicit deny rule that fully matches is reported as the cause; partial-match deny rules are skipped during explanation so the surfaced advice is actionable rather than self-defeating.
+
+#### Reason codes
+
+Every `PolicyDecision`, `DenialExplanation`, `FailedCondition`, and `PolicyDenied` from the built-in engines carries a stable `reason_code`. Assert on these codes — not on the human-readable `reason` / `narrative` strings:
+
+| Code (`DenialReason.*`) | When |
+|---|---|
+| `missing_role` | Principal lacks a required role |
+| `missing_tenant_attribute` | PII/PCI capability needs `tenant` attribute |
+| `missing_attribute` | Declarative rule's required attribute absent or mismatched |
+| `insufficient_justification` | Justification shorter than the minimum |
+| `invalid_constraint` | Constraint value (e.g. `max_rows`) not parseable |
+| `rate_limited` | Sliding-window rate limit exceeded |
+| `no_matching_rule` | DSL: no rule matched + default `deny` |
+| `explicit_deny_rule` | DSL: a `deny` rule matched fully |
+| `intent_not_allowed` | DSL: `match.intent` rejected the request's intent |
+| `scope_not_allowed` | DSL: `match.scope` rejected the request's scope |
+
+Allow-side codes (`AllowReason.*`): `default_policy_allow`, `rule_allow`, `default_fallthrough_allow`, `token_verified`.
+
+#### Decision trace
+
+Every `PolicyDecision` from a built-in engine carries a `PolicyDecisionTrace` describing how the decision was reached: the engine name, the capability and principal IDs, the request's `intent` (echoed) and `scope_keys` (scope dimension names only — values are redacted), and an ordered list of `PolicyTraceStep` entries. Each step records the rule name, the outcome (`matched`/`skipped`/`denied`/`allowed`/`constraint_applied`), a human-readable detail, and — for terminal steps — the same stable `reason_code` carried on the decision. Traces are safe to log and serialize: they contain rule names, condition names, and codes only — never raw argument values.
 
 #### Dry-run mode
 

src/agent_kernel/__init__.py

@@ -16,6 +16,8 @@
 Policy::
 
     from agent_kernel import DefaultPolicyEngine, DeclarativePolicyEngine
+    from agent_kernel import PolicyDecisionTrace, PolicyTraceStep
+    from agent_kernel import DenialReason, AllowReason
 
 Firewall::
 
@@ -82,6 +84,8 @@
     Handle,
     ImplementationRef,
     PolicyDecision,
+    PolicyDecisionTrace,
+    PolicyTraceStep,
     Principal,
     Provenance,
     RawResult,
@@ -91,6 +95,7 @@
 )
 from .policy import DefaultPolicyEngine, ExplainingPolicyEngine, PolicyEngine
 from .policy_dsl import DeclarativePolicyEngine, PolicyMatch, PolicyRule
+from .policy_reasons import AllowReason, DenialReason
 from .registry import CapabilityRegistry
 from .router import StaticRouter
 from .tokens import CapabilityToken, HMACTokenProvider
@@ -117,6 +122,8 @@
     "Handle",
     "ImplementationRef",
     "PolicyDecision",
+    "PolicyDecisionTrace",
+    "PolicyTraceStep",
     "Principal",
     "Provenance",
     "RawResult",
@@ -145,8 +152,10 @@
     "TokenRevoked",
     "TokenScopeError",
     # policy
+    "AllowReason",
     "DefaultPolicyEngine",
     "DeclarativePolicyEngine",
+    "DenialReason",
     "ExplainingPolicyEngine",
     "PolicyEngine",
     "PolicyMatch",

src/agent_kernel/errors.py

@@ -28,7 +28,24 @@ class TokenRevoked(AgentKernelError):
 
 
 class PolicyDenied(AgentKernelError):
-    """Raised when the policy engine rejects a capability request."""
+    """Raised when the policy engine rejects a capability request.
+
+    Carries an optional ``reason_code`` attribute holding a stable
+    :class:`~agent_kernel.policy_reasons.DenialReason` value so callers can
+    branch on it without matching the human-readable message:
+
+    .. code-block:: python
+
+        try:
+            kernel.grant_capability(request, principal, justification="...")
+        except PolicyDenied as exc:
+            if exc.reason_code == DenialReason.MISSING_ROLE:
+                ...
+    """
+
+    def __init__(self, message: str, *, reason_code: str | None = None) -> None:
+        super().__init__(message)
+        self.reason_code: str | None = reason_code
 
 
 class PolicyConfigError(AgentKernelError):

src/agent_kernel/kernel.py

@@ -23,11 +23,14 @@
     Frame,
     Handle,
     PolicyDecision,
+    PolicyDecisionTrace,
+    PolicyTraceStep,
     Principal,
     ResponseMode,
     RoutePlan,
 )
 from .policy import DefaultPolicyEngine, PolicyEngine
+from .policy_reasons import AllowReason
 from .registry import CapabilityRegistry
 from .router import Router, StaticRouter
 from .tokens import CapabilityToken, HMACTokenProvider, TokenProvider
@@ -303,13 +306,32 @@ async def invoke(
                 SafetyClass.WRITE: "medium",
                 SafetyClass.DESTRUCTIVE: "high",
             }
+            dry_run_trace = PolicyDecisionTrace(
+                engine="Kernel.invoke[dry_run]",
+                capability_id=token.capability_id,
+                principal_id=principal.principal_id,
+                intent=None,
+                scope_keys=[],
+                steps=[
+                    PolicyTraceStep(
+                        name="token_verified",
+                        outcome="allowed",
+                        detail="Token verified; original policy decision was at grant time.",
+                        reason_code=str(AllowReason.TOKEN_VERIFIED),
+                    )
+                ],
+                final_outcome="allowed",
+                final_reason_code=str(AllowReason.TOKEN_VERIFIED),
+            )
             return DryRunResult(
                 capability_id=token.capability_id,
                 principal_id=principal.principal_id,
                 policy_decision=PolicyDecision(
                     allowed=True,
                     reason="Token verified. Policy was evaluated at grant time.",
                     constraints=dict(token.constraints),
+                    reason_code=str(AllowReason.TOKEN_VERIFIED),
+                    trace=dry_run_trace,
                 ),
                 driver_id=driver_id,
                 operation=operation,

src/agent_kernel/models.py

@@ -134,6 +134,22 @@ class CapabilityRequest:
     constraints: dict[str, Any] = field(default_factory=dict)
     """Optional execution constraints (e.g. ``{"max_rows": 10}``)."""
 
+    intent: str | None = None
+    """Structured intent label (e.g. ``"customer_support_lookup"``).
+
+    Free-text :attr:`goal` is still required for human-readable audit; ``intent``
+    is the machine-readable counterpart that declarative policies can match
+    on directly without parsing the goal. See :class:`PolicyMatch.intent`.
+    """
+
+    scope: dict[str, Any] = field(default_factory=dict)
+    """Structured scope metadata describing what the request narrows to.
+
+    Examples: ``{"region": "eu-west"}``, ``{"customer_id": "C-42"}``. Policies
+    can deny a capability invocation that is technically allowed but unsafe
+    for a particular scope. See :class:`PolicyMatch.scope`.
+    """
+
 
 @dataclass(slots=True)
 class Principal:
@@ -149,6 +165,77 @@ class Principal:
     """Arbitrary attributes, e.g. ``{"tenant": "acme"}``."""
 
 
+@dataclass(slots=True)
+class PolicyTraceStep:
+    """A single step recorded while a policy engine evaluated a request.
+
+    Steps describe what the engine considered, in order — which rule it
+    examined, whether it matched, what condition (if any) failed, and what
+    constraint (if any) was applied. Steps never contain raw argument values
+    from the caller; they reference fields and IDs only.
+    """
+
+    name: str
+    """Short label for the step (e.g. ``"safety_class:WRITE"`` or rule name)."""
+
+    outcome: Literal["matched", "skipped", "denied", "allowed", "constraint_applied"]
+    """What happened at this step.
+
+    - ``"matched"``: a rule's match clause matched and evaluation continues.
+    - ``"skipped"``: the step did not apply (e.g. wildcard, wrong safety class).
+    - ``"denied"``: this step produced the final denial.
+    - ``"allowed"``: this step produced the final allow.
+    - ``"constraint_applied"``: the step merged a constraint into the decision.
+    """
+
+    detail: str = ""
+    """Human-readable detail, e.g. ``"role 'writer' required, principal had ['reader']"``."""
+
+    reason_code: str | None = None
+    """For ``"denied"`` steps, the :class:`~agent_kernel.policy_reasons.DenialReason`.
+    For ``"allowed"`` steps, the :class:`~agent_kernel.policy_reasons.AllowReason`.
+    ``None`` for ``"matched"``, ``"skipped"``, and ``"constraint_applied"`` steps.
+    """
+
+
+@dataclass(slots=True)
+class PolicyDecisionTrace:
+    """Structured trace of how a :class:`PolicyDecision` was reached.
+
+    The trace lists every step the policy engine took, in order, so callers
+    can audit which rule matched, which conditions failed, and which
+    constraints were applied. The trace must not contain raw argument
+    values — only field names, role names, attribute names, rule names, and
+    safe IDs — so it is safe to serialize and log.
+    """
+
+    engine: str
+    """Engine identifier (e.g. ``"DefaultPolicyEngine"``)."""
+
+    capability_id: str
+    """The capability that was being evaluated."""
+
+    principal_id: str
+    """The principal the decision was made for."""
+
+    intent: str | None
+    """Echoed :attr:`CapabilityRequest.intent` (may be ``None``)."""
+
+    scope_keys: list[str] = field(default_factory=list)
+    """Scope dimension names present on the request (values redacted for safety)."""
+
+    steps: list[PolicyTraceStep] = field(default_factory=list)
+    """Ordered list of evaluation steps."""
+
+    final_outcome: Literal["allowed", "denied"] = "denied"
+    """The decision the engine reached."""
+
+    final_reason_code: str | None = None
+    """The :class:`~agent_kernel.policy_reasons.AllowReason` or
+    :class:`~agent_kernel.policy_reasons.DenialReason` for the final outcome.
+    """
+
+
 @dataclass(slots=True)
 class PolicyDecision:
     """Result of a policy engine evaluation."""
@@ -157,11 +244,27 @@ class PolicyDecision:
     """``True`` if the request is permitted."""
 
     reason: str
-    """Human-readable explanation."""
+    """Human-readable explanation. Wording may evolve; assert on
+    :attr:`reason_code` for stable behavior."""
 
     constraints: dict[str, Any] = field(default_factory=dict)
     """Any additional constraints imposed by the policy (e.g. ``max_rows``)."""
 
+    reason_code: str | None = None
+    """Stable machine-readable code (typically a :class:`~agent_kernel.policy_reasons.AllowReason`
+    or :class:`~agent_kernel.policy_reasons.DenialReason` value).
+
+    Use this for assertions, metrics, and UI mapping. ``None`` only when an
+    out-of-tree policy engine has not populated it.
+    """
+
+    trace: PolicyDecisionTrace | None = None
+    """Structured trace of how this decision was reached.
+
+    Populated by both built-in engines on allow and deny paths. ``None`` for
+    third-party engines that don't produce a trace.
+    """
+
 
 @dataclass(slots=True)
 class CapabilityGrant:
@@ -306,6 +409,12 @@ class FailedCondition:
     suggestion: str
     """Actionable remediation hint."""
 
+    reason_code: str | None = None
+    """Stable machine-readable code (a :class:`~agent_kernel.policy_reasons.DenialReason` value).
+    Use this for assertions instead of matching the human-readable
+    :attr:`suggestion` string.
+    """
+
 
 @dataclass(slots=True)
 class DenialExplanation:
@@ -326,6 +435,12 @@ class DenialExplanation:
     narrative: str
     """Human-readable single-sentence summary."""
 
+    reason_code: str | None = None
+    """Primary :class:`~agent_kernel.policy_reasons.DenialReason` for the denial
+    (typically the code of the first :class:`FailedCondition`). ``None`` on the
+    allow path (``denied=False``).
+    """
+
 
 # ── Dry-run ───────────────────────────────────────────────────────────────────