common : only load backends when required

[angt]

Overview

Only load backends when required.

Additional information

This fixe the following issues: #20186, #21708 and maybe others.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES (kimi): I've asked the list of functions to modify for a minimal patch

[ggerganov]

The test-thread-safety is failing

[ggerganov]

Hm, I don't see where we would call ggml_backend_load_all() for llama-cli, llama-server, etc. Building with -DGGML_BACKEND_DL=ON makes these binaries not load any backends.

[angt]

Hm, I don't see where we would call ggml_backend_load_all() for llama-cli, llama-server, etc. Building with -DGGML_BACKEND_DL=ON makes these binaries not load any backends.

Damn, checking. I don’t use AI often (for llama.cpp!). Kimi is this bad ? 🤣

[angt]

The last call was missing...
Edit: So Kimi is fine, it was my fault 🙄

[angt]

@ggerganov maybe I can take this PR as an opportunity to remove ggml_backend_load_all from common_params_parser_init and call it explicitly where it’s needed, instead of hiding it there?

[ggerganov]

@ggerganov maybe I can take this PR as an opportunity to remove ggml_backend_load_all from common_params_parser_init and call it explicitly where it’s needed, instead of hiding it there?

Yes, let's do that.

[angt]

I finally have updated llama_backend_init() to call ggml_backend_load_all(). The name was a perfect fit, so I decided to try this, but I’m not yet 100% convinced too, and maybe we should just call ggml_backend_load_all() everywhere...

[rgerganov]

This patch broke RPC (issue #22721) and introduced obscure regressions (issue #22748). Having multiple calls to ggml_backend_load_all() is not how loading dynamic backends is supposed to work and breaks many implicit assumptions.

I suggest to revert this patch.

[angt]

I think it’s just because of some missing calls it’s hard to track all of them

[angt]

#22752 should fix both issues (and others)