Implement per-site configuration for FedAvg and FedEval recipes

docs/user_guide/data_scientist_guide/federated_xgboost/secure_xgboost_user_guide.rst

@@ -211,6 +211,8 @@ Protection
 
   .. code-block:: python
 
+     from nvflare.recipe import set_per_site_config
+
      recipe = XGBHorizontalRecipe(
          name="xgb_higgs_horizontal",
          min_clients=2,
@@ -222,8 +224,8 @@ Protection
              "eval_metric": "auc",
              "min_child_weight": 100,  # increase for coarser trees and reduced privacy exposure
          },
-         per_site_config=per_site_config,
      )
+     set_per_site_config(recipe, per_site_config)
 
 Closest Reconstructed Samples (CreditCard)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

docs/user_guide/data_scientist_guide/job_recipe.rst

@@ -175,9 +175,9 @@ Per-Site Configuration
 ----------------------
 
 Some recipes accept site-keyed configuration so that each site can use different
-arguments, scripts, data loaders, or validators. For recipes that support this
-helper, use ``set_per_site_config`` to attach the site-keyed input after
-creating the recipe:
+arguments, scripts, or data loaders. Call ``set_per_site_config`` immediately
+after constructing the recipe, before adding client configuration, files,
+filters, components, or tracking:
 
 .. code-block:: python
 
@@ -193,35 +193,34 @@ creating the recipe:
 
    env = SimEnv(clients=recipe.configured_sites())
 
-``configured_sites()`` returns the top-level site names from
-``set_per_site_config`` when helper-provided configuration exists. Otherwise, for
-backward compatibility, it returns site names from a recipe's constructor
-``per_site_config`` when available. It does not infer sites from recipe metadata
-such as resource specs, launcher specs, or mandatory clients. It also does not
-mean those sites are currently connected, validate production enrollment, or
-replace the execution environment.
-
-``set_per_site_config`` stores the mapping and calls the recipe's per-site
-configuration hook. It does not by itself create client targets or rebuild an
-already generated job. A recipe must implement the hook for helper-provided
-per-site fields to affect generated app configuration, command-line arguments,
-data loaders, validators, or other recipe behavior.
+The helper validates and stores the mapping; it does not build and then replace
+client apps. The recipe materializes its client topology once, before the first
+client-targeted customization or before export or execution. Built-in FedAvg
+recipes and ``FedEvalRecipe`` create one app per configured site directly. If
+per-site configuration is omitted, they create the default ``@ALL`` app at that
+same preparation point. XGBoost bagging, horizontal, and vertical recipes add
+the required data loader and executor components to each configured site and
+must be configured before client customization, export, or execution. The
+mapping must be non-empty and define at least ``min_clients`` sites. Reserved
+targets such as ``server`` and ``@ALL`` are not site names.
+
+``configured_sites()`` returns the configured top-level site names. It does not
+infer sites from recipe metadata, indicate which clients are connected, validate
+production enrollment, or replace the execution environment.
 
 .. important::
 
-   The second argument to ``set_per_site_config`` is recipe-specific. The helper
-   validates only that it is a dictionary whose top-level keys are site names and
-   whose values are dictionaries. Users must ensure each per-site dictionary uses
-   fields that the selected recipe understands and can convert into generated app
-   configuration, command-line arguments, data loaders, validators, or other
-   recipe-specific settings.
-
-   For example, FedAvg's current per-site support is through the legacy
-   ``FedAvgRecipe(per_site_config=...)`` constructor argument. That constructor
-   path understands per-site values such as ``train_args``, ``train_script``,
-   and ``command``. It does not automatically interpret arbitrary keys such as
-   ``data_path`` or ``batch_size``. For FedAvg, pass those values through
-   ``train_args`` unless your recipe explicitly documents another shape.
+   Each site's dictionary is recipe-specific. FedAvg recipes support
+   ``train_script``, ``train_args``, ``launch_external_process``, ``command``,
+   ``framework``, ``server_expected_format``, ``params_transfer_type``,
+   ``launch_once``, and ``shutdown_timeout``. ``FedEvalRecipe`` supports the
+   corresponding ``eval_script`` and ``eval_args`` fields plus its launch,
+   command, and exchange-format overrides. XGBoost recipes require a
+   ``data_loader`` for every site; bagging also accepts ``lr_scale``.
+
+   The older ``per_site_config=...`` constructor argument remains temporarily
+   available for compatibility, emits ``FutureWarning``, and delegates to this
+   helper behavior. New code should use ``set_per_site_config``.
 
 No Secrets In Recipe Parameters
 -------------------------------

docs/user_guide/data_scientist_guide/recipe_api.rst

@@ -104,16 +104,13 @@ File packaging helpers:
    Bundle a file or directory into the generated server app package.
 
 Helpers that accept ``clients`` target specific generated client apps. This
-requires per-site client apps: construct the recipe with the
-``per_site_config`` constructor argument on recipes that support it, and each
-name in ``clients`` must match an existing per-site client app. With the
-default all-clients topology, targeted calls raise an error rather than
-silently dropping the change from the generated job, and unknown site names
-raise an error rather than deploying a bare app to that site. Calling
-``set_per_site_config`` after construction records the configuration for
-``configured_sites()`` but does not yet rebuild an existing all-clients app
-into per-site apps; recipes will interpret helper-provided per-site config as
-follow-up work.
+requires per-site client apps: call ``set_per_site_config`` immediately after
+constructing a recipe that supports it. The recipe prepares those apps before
+applying the first client-targeted helper, and each name in ``clients`` must
+match a configured per-site client app. With the default all-clients topology,
+targeted calls raise an error rather than silently dropping the change from the
+generated job, and unknown site names raise an error rather than deploying a
+bare app to that site.
 
 Filter helpers:
 
@@ -170,15 +167,18 @@ Per-Site And Metadata Helpers
 For NVFlare 2.9, the public Recipe configuration surface also includes:
 
 ``set_per_site_config(recipe, config)``
-   Provide site-keyed, recipe-specific configuration. Each concrete recipe
-   interprets the site dictionaries for its own workflow. Nested values become
-   part of the job definition and must not contain secret values.
+   Provide non-empty, site-keyed, recipe-specific configuration. Call it once,
+   immediately after recipe construction and before client customizations. Each
+   concrete recipe interprets the site dictionaries for its own workflow.
+   Built-in FedAvg, FedEval, and XGBoost recipes require at least ``min_clients``
+   entries. The helper validates and stores the mapping; the recipe creates its
+   client apps once, immediately before the first client customization or before
+   export or execution. Nested values become part of the job definition and must
+   not contain secret values.
 
 ``recipe.configured_sites()``
-   Return top-level site names from helper-provided per-site config when
-   present. For backward compatibility, recipes may return site names from
-   legacy constructor ``per_site_config`` when available. This method does not
-   indicate that sites are connected or replace the execution environment.
+   Return top-level site names from applied per-site config. This method does
+   not indicate that sites are connected or replace the execution environment.
 
 ``set_recipe_meta(recipe, key, value)``
    Set selected generated job metadata by ``JobMetaKey``. The accepted keys are

examples/advanced/experiment-tracking/README.md

@@ -227,9 +227,9 @@ add_experiment_tracking(
 )
 ```
 
-If sites need different tracking URIs, credentials, or experiment names, construct
-the recipe with per-site client apps (for example,
-`per_site_config={"site-1": {}, "site-2": {}}`) and call
+If sites need different tracking URIs, credentials, or experiment names, call
+`set_per_site_config(recipe, {"site-1": {}, "site-2": {}})` immediately
+after recipe construction to create per-site client apps, then call
 `add_experiment_tracking()` once per configuration with `clients=[site_name]`.
 The `clients` argument requires that per-site topology; it intentionally rejects a
 recipe whose client app is deployed through the default `@ALL` target.

examples/advanced/experiment-tracking/mlflow/hello-pt-mlflow-client/README.md

@@ -193,15 +193,15 @@ each site. In simulation, all clients share the host filesystem, so use the
 default `tracking_uri=None` behavior or configure distinct paths if the stores
 must remain physically separate.
 
-For different configuration at each site, construct the recipe with per-site
-client apps and then target each site explicitly:
+For different configuration at each site, create per-site client apps before
+targeting each site explicitly:
 
 ```python
+from nvflare.recipe import set_per_site_config
+
 sites = ["site-1", "site-2"]
-recipe = FedAvgRecipe(
-    ...,
-    per_site_config={site: {} for site in sites},
-)
+recipe = FedAvgRecipe(...)
+set_per_site_config(recipe, {site: {} for site in sites})
 
 for site in sites:
     add_experiment_tracking(
@@ -217,7 +217,7 @@ for site in sites:
     )
 ```
 
-Targeted `clients=[...]` placement requires `per_site_config` when the recipe is
+Targeted `clients=[...]` placement requires per-site configuration when the recipe is
 constructed; it cannot split an existing `@ALL` client app after the fact.
 
 ### Add Experiment Tags

examples/advanced/gnn/README.md

@@ -111,12 +111,15 @@ The recipe is configured in `job.py` with parameters such as:
 
 The recipe uses **per-site configuration** (`per_site_config`) to provide site-specific training arguments:
 ```python
+from nvflare.recipe import set_per_site_config
+
 per_site_config = {}
 for i in range(1, num_clients + 1):
     site_name = f"site-{i}"
     per_site_config[site_name] = {
         "train_args": f"--data_path {data_path} --epochs {epochs_per_round} ..."
     }
+set_per_site_config(recipe, per_site_config)
 ```
 
 This pattern allows each site to receive customized arguments, making it easy to:

examples/advanced/gnn/finance/job.py

@@ -17,7 +17,7 @@
 from model import SAGE
 
 from nvflare.app_opt.pt.recipes.fedavg import FedAvgRecipe
-from nvflare.recipe import ProdEnv, SimEnv
+from nvflare.recipe import ProdEnv, SimEnv, set_per_site_config
 
 
 def main():
@@ -124,9 +124,9 @@ def main():
         min_clients=args.num_clients,
         num_rounds=args.num_rounds,
         train_script="client.py",
-        per_site_config=per_site_config,
         key_metric="validation_auc",
     )
+    set_per_site_config(recipe, per_site_config)
 
     # Export job
     print(f"Exporting job to {args.job_dir}")

examples/advanced/gnn/protein/job.py

@@ -17,7 +17,7 @@
 from torch_geometric.nn import GraphSAGE
 
 from nvflare.app_opt.pt.recipes.fedavg import FedAvgRecipe
-from nvflare.recipe import ProdEnv, SimEnv
+from nvflare.recipe import ProdEnv, SimEnv, set_per_site_config
 
 
 def main():
@@ -122,9 +122,9 @@ def main():
         min_clients=args.num_clients,
         num_rounds=args.num_rounds,
         train_script="client.py",
-        per_site_config=per_site_config,
         key_metric="validation_f1",
     )
+    set_per_site_config(recipe, per_site_config)
 
     # Export job
     print(f"Exporting job to {args.job_dir}")

examples/advanced/llm_hf/README.md

@@ -99,6 +99,8 @@ Key features:
 
 **Recipe-Based Approach:**
 ```python
+from nvflare.recipe import set_per_site_config
+
 # Create recipe with FedAvgRecipe
 # Model can be class instance or dict config
 # For pre-trained weights: initial_ckpt="/server/path/to/pretrained.pt"
@@ -110,7 +112,6 @@ recipe = FedAvgRecipe(
     train_script="client.py",
     server_expected_format=server_expected_format,  # "pytorch" or "numpy"
     launch_external_process=True,
-    per_site_config=per_site_config,  # Site-specific configurations
     key_metric="neg_eval_loss",
 )
 ```
@@ -133,6 +134,7 @@ per_site_config = {
                    "--nproc_per_node=2 --master_port=8888"
     }
 }
+set_per_site_config(recipe, per_site_config)
 ```
 
 **Optional Features:**

examples/advanced/llm_hf/job.py

@@ -24,7 +24,7 @@
 from nvflare.app_opt.pt.quantization.quantizer import ModelQuantizer
 from nvflare.app_opt.pt.recipes.fedavg import FedAvgRecipe
 from nvflare.private.fed.utils.fed_utils import split_gpus
-from nvflare.recipe import ProdEnv, SimEnv, add_experiment_tracking
+from nvflare.recipe import ProdEnv, SimEnv, add_experiment_tracking, set_per_site_config
 
 
 def define_parser():
@@ -176,9 +176,9 @@ def main():
         train_script="client.py",
         server_expected_format=server_expected_format,
         launch_external_process=True,  # Always use external process for LLM training
-        per_site_config=per_site_config,
         key_metric="neg_eval_loss",
     )
+    set_per_site_config(recipe, per_site_config)
 
     # Add client params to reduce timeout failures for longer LLM runs
     recipe.add_client_config({"get_task_timeout": 300, "submit_task_result_timeout": 300})

examples/advanced/medgemma/job.py

@@ -26,7 +26,7 @@
 from data_utils import DEFAULT_MODEL_NAME_OR_PATH
 
 from nvflare.app_opt.pt.recipes.fedavg import FedAvgRecipe
-from nvflare.recipe import SimEnv, add_experiment_tracking
+from nvflare.recipe import SimEnv, add_experiment_tracking, set_per_site_config
 
 
 def define_parser():
@@ -271,7 +271,6 @@ def main():
         num_rounds=args.num_rounds,
         model=model,
         train_script="client.py",
-        per_site_config=per_site_config,
         aggregator=(
             HLoRAMaxRankAggregator(global_lora_rank=args.global_lora_rank)
             if args.lora_aggregation == "hlora"
@@ -281,6 +280,7 @@ def main():
         server_expected_format="pytorch",
         key_metric="",
     )
+    set_per_site_config(recipe, per_site_config)
     _configure_timeouts(recipe, client_names)
 
     if args.wandb:

examples/advanced/multi-gpu/pt/README.md

@@ -103,14 +103,16 @@ distributed job, while `LOCAL_RANK` is only unique on the current node.
 ### Multiple Clients on Same Machine
 When running multiple clients on the same machine, use different master ports:
 ```python
-per_site_config={
+from nvflare.recipe import set_per_site_config
+
+set_per_site_config(recipe, {
     "site-1": {
         "command": "... --master_port=7777",
     },
     "site-2": {
         "command": "... --master_port=8888",
     },
-}
+})
 ```
 
 ## Troubleshooting

examples/advanced/multi-gpu/pt/job.py

@@ -21,7 +21,7 @@
 from model import Net
 
 from nvflare.app_opt.pt.recipes.fedavg import FedAvgRecipe
-from nvflare.recipe import SimEnv, add_experiment_tracking
+from nvflare.recipe import SimEnv, add_experiment_tracking, set_per_site_config
 
 
 def define_parser():
@@ -52,7 +52,10 @@ def main():
         train_script=train_script,
         train_args=train_args,
         launch_external_process=True,  # DDP modes require external process launch
-        per_site_config={
+    )
+    set_per_site_config(
+        recipe,
+        {
             "site-1": {
                 "command": "python3 -m torch.distributed.run --nnodes=1 --nproc_per_node=2 --master_port=7777",
             },

examples/advanced/qwen3-vl/job.py

@@ -22,7 +22,7 @@
 import re
 
 from nvflare.app_opt.pt.recipes.fedavg import FedAvgRecipe
-from nvflare.recipe import SimEnv, add_experiment_tracking
+from nvflare.recipe import SimEnv, add_experiment_tracking, set_per_site_config
 
 
 def define_parser():
@@ -187,12 +187,12 @@ def main():
         num_rounds=args.num_rounds,
         model=model,
         train_script="client.py",
-        per_site_config=per_site_config,
         launch_external_process=True,
         server_expected_format="pytorch",
         # train_qwen.train() does not emit structured eval metrics for model selection in this example.
         key_metric="",
     )
+    set_per_site_config(recipe, per_site_config)
 
     # Configure timeouts for large model / tensor streaming.
     _configure_timeouts(recipe, client_names)