Mint.TransportError not caught by retry logic

[ospaarmann]

Hey, thank you for providing this package!

I noticed that I quite often get a Mint.TransportError exception that is not being caught by the retry logic. So I have to wrap the Instructor call with another retry, which is not ideal.

Call:

case Instructor.chat_completion(
        model: "gpt-4o",
        temperature: 0.5,
        max_retries: 5,
        response_model: MyResponseModel,
        messages: [
          %{role: "system", content: system_message},
          %{role: "user", content: user_message}
        ]
      ) do
  {:ok, %MyResponseModel{} = result} ->
    {:ok, result}

  {:error, error} ->
    Logger.error("Failed to generate model: #{inspect(error)}")
    {:error, error}
end

Exception

[error] Task #PID<0.2901.0> started from #PID<0.2900.0> terminating
** (Mint.TransportError) socket closed
    (req 0.4.14) lib/req.ex:996: Req.request!/2
    (instructor 0.0.5) lib/instructor/adapters/openai.ex:80: Instructor.Adapters.OpenAI.do_chat_completion/2
    (instructor 0.0.5) lib/instructor.ex:430: Instructor.do_chat_completion/3
    (my_app 0.1.0) lib/my_app/reports/generate_report.ex:94: MyApp.Reports.generate_dimension_scores/1
    (my_app 0.1.0) lib/my_app/reports/generate_report.ex:24: MyApp.Reports.generate_report/1
    (elixir 1.16.0) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
Function: #Function<0.44892832/0 in MyAppWeb.PersonalityProfileChannel.handle_in/3>
    Args: []
iex(3)>

My expectation is, that the exception is caught and Instructor tries again (4 more times). This is not happening.

Thanks!

[mxgrn]

I suppose, max_retries is meant to re-ask the LLM when it returns something that doesn't validate. The error you're getting seems to happen earlier, when the LLM API isn't reachable due to some reason. I'm facing the same problem, though, although Req is supposed to retry connecting in this case, too, at least with this option:

config :instructor, :openai, http_options: [retry: :safe_transient, max_retries: 6]

UPD: Maybe I'm wrong, and it does handle API errors, too.

UPD2: I figured it out: you need retry: transient because Instructor POSTs to the LLMs. Then retries work OOB.