Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Changed
- Agent adapters now consume one protocol-aware model connection selected from
provider configuration. OpenAI- and Anthropic-compatible endpoints under the
same provider no longer pass through the legacy flattened credential view,
while absent credentials and endpoints continue to delegate to agent-local
login and routing.

### Fixed
- Agent judges selecting a different provider no longer inherit runner credentials
or endpoints, preventing judge keys from being sent to a runner endpoint.

## [0.10.0] - 2026-09-01

### Added
Expand Down
29 changes: 16 additions & 13 deletions docs/design/agent-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,20 +44,23 @@ The runner path resolves values in these stages:
3. Build one role-aware `ResolvedAgentConfig`. Legacy slash disambiguation is
performed once at this point instead of mutating and later repairing the
loaded eval config.
4. Resolve provider-scoped `MODEL`, `API_KEY`, and `BASE_URL` values. A
provider-scoped model environment variable currently overrides the YAML
model; provider environment credentials override the credential file.
4. Resolve the provider-scoped `MODEL` value. A provider-scoped model
environment variable currently overrides the YAML model.
5. Preserve explicit CLI `--model` and `--api-key` precedence.
6. Apply the selected adapter's declared protocol and capability contract.
6. Apply the selected adapter's declared protocol and capability contract,
then resolve credentials and endpoints for that `(provider, protocol)`.
Keep the requested model intact, compute the applied model once, remove
unsupported kwargs, and emit warnings for ignored or invalid explicit
settings before case execution.
7. Pass the applied value to the selected adapter, which constructs its
command and environment without repeating model normalization.
7. Pass the applied connection to the selected adapter, which constructs its
command and environment without repeating credential or model resolution.

Runner and judge roles use the same resolution flow. Until an explicit judge
engine schema is introduced, the judge inherits the runner engine lifecycle and
kwargs, while resolving its provider/model and credentials as a separate role.
Connection inheritance is limited to the same provider namespace. A judge that
selects a different provider resolves its own key and endpoint by protocol;
missing values delegate to agent-local state, never to the runner connection.
Reports use the resolved runner identity rather than reconstructing it from a
CLI-mutated eval config. `result.json` retains the legacy `engine_name` and
`model_name` fields while also recording credential-free
Expand Down Expand Up @@ -86,12 +89,11 @@ configuration are related but distinct layers:
local login databases remain owned by the agent and may be opaque to
skill-up.

`ResolvedAgentConfig` currently spans the first two steps and the capability
pass computes what skill-up can apply. A later protocol-aware resolver may
introduce an internal `ResolvedModelConnection`, but it should not be confused
with observed runtime state: it describes the connection selected for the
adapter, not proof of the provider, model, or credential ultimately used by the
agent.
`ResolvedAgentConfig` retains requested values and its `AppliedConnection`
records the protocol-aware `ResolvedModelConnection` selected by the capability
pass. The connection must not be confused with observed runtime state: it
describes what skill-up materializes for the adapter, not proof of the provider,
model, credential, or local login ultimately used by the agent.

This layering is informed by Harbor's
[`ProviderAccess`, `ModelConnectionSpec`, and `ResolvedModelConnection`](https://github.com/harbor-framework/harbor/blob/71180a2e6fb40626b661c13f261b1d44517ad91a/src/harbor/agents/model_connection.py),
Expand Down Expand Up @@ -145,7 +147,8 @@ Tagged Actions and historical commands therefore remain valid.

## Known gaps for later phases

- Nested provider endpoints are flattened before the adapter protocol is known.
- The legacy flattened credential lookup remains available to older internal
callers, but adapter construction uses the protocol-aware connection.
- Provider-scoped `MODEL` currently overrides an explicit YAML model.
- `engine.version`, `engine.entry`, and `engine.model.params` now produce
warnings when ineffective, but their final implementation/removal is deferred
Expand Down
8 changes: 5 additions & 3 deletions internal/agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,9 +124,11 @@ type SessionResult struct {
## Adapter capability resolution

`ResolveAdapterConfig` runs after YAML/CLI/credential resolution and before
agent construction. It keeps the requested model intact, records the model
that skill-up will forward as `AppliedModel`, and emits warnings for
explicit settings that the selected adapter does not consume.
agent construction. It selects credentials and endpoints for the adapter's
protocol, keeps the requested model intact, records the model that skill-up will
forward as `AppliedModel`, and emits warnings for explicit settings that the
selected adapter does not consume. Factory construction consumes the resulting
`AppliedConnection` rather than resolving provider configuration again.

Applied values describe the invocation, not the CLI's final runtime choice.
Local configuration may override them. `SessionResult.Model` remains empty
Expand Down
56 changes: 47 additions & 9 deletions internal/agent/agent_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -546,10 +546,48 @@ func TestProbeAndMergePATH_SkipsMergeOnEmptyStdout(t *testing.T) {
}
}

func detectMaterializedAgentForTest(params credential.ResolvedAgentConfig) (Agent, error) {
return DetectAgentWithResolvedConfig(ResolveAdapterConfig(params, nil))
}

func TestDetectAgentWithResolvedConfig_RequiresMaterialization(t *testing.T) {
t.Parallel()

_, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{Engine: "codex"})
if err == nil || !strings.Contains(err.Error(), "was not materialized") {
t.Fatalf("error = %v, want materialization error", err)
}
}

func TestDetectAgentWithResolvedConfig_ConsumesAppliedConnection(t *testing.T) {
t.Parallel()

params := ResolveAdapterConfig(credential.ResolvedAgentConfig{
Engine: "claude_code",
Provider: "anthropic",
APIKey: "connection-key",
BaseURL: "https://connection.example.test",
}, nil)
params.AppliedAPIKey = "stale-key"
params.AppliedBaseURL = "https://stale.example.test"

ag, err := DetectAgentWithResolvedConfig(params)
if err != nil {
t.Fatalf("DetectAgentWithResolvedConfig failed: %v", err)
}
claudeAgent, ok := ag.(*ClaudeCodeAgent)
if !ok {
t.Fatalf("agent = %T, want *ClaudeCodeAgent", ag)
}
if claudeAgent.Cfg.APIKey != "connection-key" || claudeAgent.Cfg.BaseURL != "https://connection.example.test" {
t.Fatalf("adapter config = key %q base URL %q, want applied connection values", claudeAgent.Cfg.APIKey, claudeAgent.Cfg.BaseURL)
}
}

func TestDetectAgentWithResolvedConfig_SetsTypedCredentialFields(t *testing.T) {
t.Parallel()

ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{
Engine: "codex",
Provider: "openai",
Model: "gpt-5.4",
Expand Down Expand Up @@ -590,7 +628,7 @@ func TestDetectAgentWithResolvedConfig_SetsTypedCredentialFields(t *testing.T) {
func TestDetectAgentWithResolvedConfig_CodexFallbackOmitsRejectedProviderCredential(t *testing.T) {
t.Parallel()

ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{ //nolint:gosec // dummy test credential
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{ //nolint:gosec // dummy test credential
Engine: "codex",
Provider: testDashscopeProvider,
Model: "qwen3.6-plus",
Expand Down Expand Up @@ -618,7 +656,7 @@ func TestDetectAgentWithResolvedConfig_QoderMapsAPIKeyToRuntimeEnv(t *testing.T)
token := "qoder-runtime-token" //nolint:gosec // test credential, not real
t.Setenv(credential.EnvQoderPersonalAccessToken, token)

ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{
Engine: "qoder-cli",
Provider: "qoder",
Model: "auto",
Expand All @@ -641,7 +679,7 @@ func TestDetectAgentWithResolvedConfig_QoderCNMapsKeychainAliasToOfficialEnv(t *
t.Setenv(credential.EnvQoderCNAccessToken, token)
t.Setenv(credential.EnvQoderCNPersonalAccessToken, "")

ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{
Engine: "qoder-cli",
Provider: "qoder",
Model: "auto",
Expand All @@ -667,7 +705,7 @@ func TestDetectAgentWithResolvedConfig_QoderCNPrefersOfficialEnv(t *testing.T) {
t.Setenv(credential.EnvQoderCNPersonalAccessToken, "official-token")
t.Setenv(credential.EnvQoderCNAccessToken, "alias-token")

ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{
Engine: "qoder-cli",
Kwargs: map[string]string{KwargEdition: qoderEditionCN},
})
Expand All @@ -686,7 +724,7 @@ func TestDetectAgentWithResolvedConfig_QoderCNPrefersOfficialEnv(t *testing.T) {
func TestDetectAgentWithResolvedConfig_QoderIgnoresParamsAPIKey(t *testing.T) {
t.Parallel()

ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{ //nolint:gosec // test dummy key
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{ //nolint:gosec // test dummy key
Engine: "qoder-cli",
Provider: "anthropic",
Model: "auto",
Expand Down Expand Up @@ -731,7 +769,7 @@ func TestUnsupportedAgentError(t *testing.T) {
func TestDetectAgentWithResolvedConfig_UsesResolvedModelWithoutNormalization(t *testing.T) {
t.Parallel()

ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{
Engine: "claude-code",
Model: "",
})
Expand All @@ -751,7 +789,7 @@ func TestDetectAgentWithResolvedConfig_UsesResolvedModelWithoutNormalization(t *
func TestDetectAgentWithResolvedConfig_PreservesAutoForQoderCLI(t *testing.T) {
t.Parallel()

ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{
Engine: "qoder-cli",
Model: "auto",
})
Expand All @@ -772,7 +810,7 @@ func TestDetectAgentWithResolvedConfig_ForwardsKwargs(t *testing.T) {
t.Parallel()

kwargs := map[string]string{KwargBypassSandbox: "true", "future_key": "x"}
ag, err := DetectAgentWithResolvedConfig(credential.ResolvedAgentConfig{
ag, err := detectMaterializedAgentForTest(credential.ResolvedAgentConfig{
Engine: "codex",
Provider: "openai",
Model: "gpt-5.4",
Expand Down
101 changes: 94 additions & 7 deletions internal/agent/capabilities.go
Original file line number Diff line number Diff line change
Expand Up @@ -95,26 +95,113 @@ func CapabilitiesForEngine(engineName string) Capabilities {
// records the model that skill-up will forward to the CLI. It does not claim
// that the CLI ultimately selected that model: local CLI configuration may
// override it, and most adapters do not report their final runtime choice.
func ResolveAdapterConfig(params credential.ResolvedAgentConfig) credential.ResolvedAgentConfig {
func ResolveAdapterConfig(params credential.ResolvedAgentConfig, resolver *credential.Resolver) credential.ResolvedAgentConfig {
if params.Role == "" {
params.Role = credential.AgentRoleRunner
}
params.Kwargs = maps.Clone(params.Kwargs)
params.ModelParams = maps.Clone(params.ModelParams)
params.Warnings = slices.Clone(params.Warnings)
params.AppliedAPIKey = params.APIKey
params.AppliedBaseURL = params.BaseURL

capabilities := CapabilitiesForEngine(params.Engine)
params.Protocol = string(capabilities.Protocol)
connection := resolveModelConnection(params, resolver, capabilities.Protocol)
params.AppliedAPIKey = connection.APIKey
params.AppliedBaseURL = connection.BaseURL
params.AppliedProvider = resolveAppliedProvider(&params, capabilities)
params.AppliedModel = resolveAppliedModel(&params, capabilities)
validateBaseURL(&params, capabilities)
validateDeferredFields(&params, capabilities)
validateKwargs(&params, capabilities)
params.AppliedConnection = appliedModelConnection(params, connection)
return params
}

func resolveModelConnection(
params credential.ResolvedAgentConfig,
resolver *credential.Resolver,
protocol credential.Protocol,
) credential.ResolvedModelConnection {
if params.AppliedConnection.Protocol == protocol {
return params.AppliedConnection
}
if resolver == nil {
resolved := credential.ResolvedModelConnection{
Provider: params.Provider,
Protocol: protocol,
APIKey: params.APIKey,
BaseURL: params.BaseURL,
APIKeySet: params.APIKeySource != "" || params.APIKey != "",
BaseURLSet: params.BaseURLSource != "" || params.BaseURL != "",
APIKeySource: params.APIKeySource,
BaseURLSource: params.BaseURLSource,
AuthMode: credential.AuthModeAgentLocal,
RoutingMode: credential.RoutingModeAgentLocal,
}
if resolved.APIKey != "" {
resolved.AuthMode = credential.AuthModeInjected
}
if resolved.BaseURL != "" {
resolved.RoutingMode = credential.RoutingModeExplicit
}
return resolved
}

spec := credential.ModelConnectionSpec{
Provider: params.Provider,
Protocol: protocol,
}
if value, ok := explicitConnectionValue(params.APIKey, params.APIKeySource); ok {
spec.APIKey = value
}
if value, ok := explicitConnectionValue(params.BaseURL, params.BaseURLSource); ok {
spec.BaseURL = value
}
return resolver.ResolveModelConnection(spec)
}

func explicitConnectionValue(value string, source credential.ValueSource) (credential.ConnectionValue, bool) {
if source == "" && value == "" {
return credential.ConnectionValue{}, false
}
if source == credential.ValueSourceEnv || source == credential.ValueSourceResolver {
return credential.ConnectionValue{}, false
}
return credential.ExplicitConnectionValue(value, source), true
}

func appliedModelConnection(
params credential.ResolvedAgentConfig,
resolved credential.ResolvedModelConnection,
) credential.ResolvedModelConnection {
apiKeyChanged := resolved.APIKey != params.AppliedAPIKey
baseURLChanged := resolved.BaseURL != params.AppliedBaseURL
resolved.Provider = params.AppliedProvider
resolved.APIKey = params.AppliedAPIKey
resolved.BaseURL = params.AppliedBaseURL
if apiKeyChanged {
resolved.APIKeySet = params.AppliedAPIKey != ""
if !resolved.APIKeySet {
resolved.APIKeySource = ""
}
}
if baseURLChanged {
resolved.BaseURLSet = params.AppliedBaseURL != ""
if !resolved.BaseURLSet {
resolved.BaseURLSource = ""
}
}
resolved.AuthMode = credential.AuthModeAgentLocal
if resolved.APIKey != "" {
resolved.AuthMode = credential.AuthModeInjected
}
resolved.RoutingMode = credential.RoutingModeAgentLocal
if resolved.BaseURL != "" {
resolved.RoutingMode = credential.RoutingModeExplicit
}
return resolved
}

func resolveAppliedProvider(params *credential.ResolvedAgentConfig, capabilities Capabilities) string {
switch capabilities.ModelPolicy {
case ModelPolicyQoderTier:
Expand All @@ -125,7 +212,7 @@ func resolveAppliedProvider(params *credential.ResolvedAgentConfig, capabilities
params.AppliedBaseURL = ""
return ""
case ModelPolicyCodexProvider:
if reason := codexCustomProviderUnavailableReason(params.Provider, params.BaseURL); reason != "" {
if reason := codexCustomProviderUnavailableReason(params.Provider, params.AppliedBaseURL); reason != "" {
params.Warnings = appendUniqueWarning(params.Warnings, fmt.Sprintf(
"engine %q cannot apply provider %q: provider %s; the provider override, endpoint, and provider-scoped credential are omitted and local Codex settings will be used",
params.Engine, params.Provider, reason,
Expand All @@ -134,7 +221,7 @@ func resolveAppliedProvider(params *credential.ResolvedAgentConfig, capabilities
params.AppliedBaseURL = ""
return ""
}
if params.BaseURL == "" {
if params.AppliedBaseURL == "" {
// Codex receives no model_provider override in this case. Even an
// explicit provider namespace cannot prove which local provider Codex
// ultimately selects.
Expand Down Expand Up @@ -164,7 +251,7 @@ func resolveAppliedModel(params *credential.ResolvedAgentConfig, capabilities Ca
))
return ""
case ModelPolicyCodexProvider:
if reason := codexCustomProviderUnavailableReason(params.Provider, params.BaseURL); reason != "" {
if reason := codexCustomProviderUnavailableReason(params.Provider, params.AppliedBaseURL); reason != "" {
return ""
}
}
Expand All @@ -173,7 +260,7 @@ func resolveAppliedModel(params *credential.ResolvedAgentConfig, capabilities Ca
}

func validateBaseURL(params *credential.ResolvedAgentConfig, capabilities Capabilities) {
if params.BaseURL == "" || capabilities.SupportsBaseURL {
if params.AppliedBaseURL == "" || capabilities.SupportsBaseURL {
return
}
params.Warnings = appendUniqueWarning(params.Warnings, fmt.Sprintf(
Expand Down
Loading
Loading