Repository navigation
Summary evaluators with a defaulted extra parameter receive raw Run/Example lists instead of mapped outputs #22
Description
Tracing Method
- SDK/Client
- REST API
- With LangChain
- Other
Language
- Python
- JavaScript/TypeScript
- Other
Host
- Hosted (https://api.smith.langchain.com)
- Self-hosted
- Other
Other System Info
langsmith at f6b73d8374b30b6dd69b703d4c1e3617ed2fb6e6 (main). Reproduced offline.
Reproduction
In _normalize_summary_evaluator (evaluator.py), the validity check accepts parameters that have defaults, but the later elif not all(pname in supported_args for pname in all_args) branch does not. A summary evaluator with an extra defaulted parameter therefore takes the "backwards compatibility" branch and is returned unwrapped, so outputs is a list[Run] rather than a list of output dicts. The row-level normalizer accepts defaulted params in the equivalent condition.
Public API:
import uuid, datetime
from langsmith import evaluate, schemas
examples = [schemas.Example(id=uuid.uuid4(), dataset_id=uuid.uuid4(), inputs={"q": 1},
outputs={"a": 1}, created_at=datetime.datetime.now())]
def summ(outputs, reference_outputs, threshold=0.5):
return {"key": "s", "score": float(isinstance(outputs[0], dict))}
r = evaluate(lambda i: {"a": 1}, data=examples, summary_evaluators=[summ], upload_results=False)
print(r._summary_results)
# score=0.0Removing threshold=0.5 gives score=1.0.
Direct repro (this one imports the private _normalize_summary_evaluator):
import uuid, datetime
from langsmith import schemas
from langsmith.evaluation.evaluator import _normalize_summary_evaluator
r = schemas.Run(id=uuid.uuid4(), name="n", start_time=datetime.datetime.now(), run_type="chain",
trace_id=uuid.uuid4(), dotted_order="x", outputs={"a": 1})
e = schemas.Example(id=uuid.uuid4(), dataset_id=uuid.uuid4(), inputs={}, outputs={"a": 1},
created_at=datetime.datetime.now())
def summ(outputs, reference_outputs, threshold=0.5):
return {"key": "s", "score": float(isinstance(outputs[0], dict))}
def summ2(outputs, reference_outputs):
return {"key": "s", "score": float(isinstance(outputs[0], dict))}
print(_normalize_summary_evaluator(summ)([r], [e]))
print(_normalize_summary_evaluator(summ2)([r], [e]))Output:
{'key': 's', 'score': 0.0}
{'key': 's', 'score': 1.0}
Expected behavior
Both evaluators give 1.0: the defaulted parameter should not change how outputs and reference_outputs are mapped. There is no error, so the metric is silently wrong.
A possible fix is to mirror the row-level condition in the elif: not all(p in supported_args or p in args_with_defaults for p in all_args), plus a test.
Activity
thibautlehmann commentedon Oct 16, 2026
Hey Muneeb Ullah (@moneebullah25), langsmith-sdk is archived, so I moved this issue to langchain-ai/langsmith-python, where the SDK now lives 🙏