Skip to content

fix: guard against empty/filtered LLM responses in make_me_say utils#1666

Open
qizwiz wants to merge 1 commit into
openai:mainfrom
qizwiz:fix/llm-response-unguarded
Open

fix: guard against empty/filtered LLM responses in make_me_say utils#1666
qizwiz wants to merge 1 commit into
openai:mainfrom
qizwiz:fix/llm-response-unguarded

Conversation

@qizwiz
Copy link
Copy Markdown

@qizwiz qizwiz commented May 18, 2026

What

Adds a null check in evals/elsuite/make_me_say/utils.py:get_content() before accessing response.choices[0].message.

Why

Two silent failure modes crash at this line:

  1. IndexErrorchoices is an empty list ([]) when the provider returns no completions
  2. AttributeErrorchoices[0].message is None when content is filtered (some providers return HTTP 200 with PROHIBITED_CONTENT finish reason)

Fix

if not response.choices or response.choices[0].message is None:
    raise ValueError("LLM returned empty or filtered response")
return response.choices[0].message.content

Detected by pact static analysis.

An empty choices list raises IndexError; a None message raises
AttributeError. Add a null check before accessing choices[0].message.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant