Implement per-site configuration for FedAvg and FedEval recipes

docs/user_guide/data_scientist_guide/job_recipe.rst

@@ -183,6 +183,7 @@ creating the recipe:
 
    from nvflare.recipe import SimEnv, set_per_site_config
 
+   # Apply this to the FedAvg recipe created in the previous example.
    set_per_site_config(
        recipe,
        {
@@ -202,10 +203,20 @@ mean those sites are currently connected, validate production enrollment, or
 replace the execution environment.
 
 ``set_per_site_config`` stores the mapping and calls the recipe's per-site
-configuration hook. It does not by itself create client targets or rebuild an
-already generated job. A recipe must implement the hook for helper-provided
-per-site fields to affect generated app configuration, command-line arguments,
-data loaders, validators, or other recipe behavior.
+configuration hook. A recipe must implement that hook for helper-provided
+fields to affect generated app configuration, command-line arguments, data
+loaders, validators, or other recipe behavior.
+
+The unified FedAvg recipe family and PyTorch ``FedEvalRecipe`` implement this
+hook. Calling ``set_per_site_config`` before export or run replaces their
+all-clients app with one app per configured site and applies each site's
+supported training or evaluation runner overrides. Files, filters, components,
+and top-level client configuration already added through Recipe APIs are
+preserved. The helper can be reapplied for the same sites; create a new recipe
+to change the site set. Passing ``per_site_config`` to these recipes' constructors
+remains supported. XGBoost recipes require
+site-specific data loaders during construction and therefore continue to use
+their ``per_site_config`` constructor argument.
 
 .. important::
 
@@ -216,12 +227,11 @@ data loaders, validators, or other recipe behavior.
    configuration, command-line arguments, data loaders, validators, or other
    recipe-specific settings.
 
-   For example, FedAvg's current per-site support is through the legacy
-   ``FedAvgRecipe(per_site_config=...)`` constructor argument. That constructor
-   path understands per-site values such as ``train_args``, ``train_script``,
-   and ``command``. It does not automatically interpret arbitrary keys such as
-   ``data_path`` or ``batch_size``. For FedAvg, pass those values through
-   ``train_args`` unless your recipe explicitly documents another shape.
+   For example, FedAvg understands per-site values such as ``train_args``,
+   ``train_script``, and ``command`` through either this helper or its
+   ``per_site_config`` constructor argument. It does not automatically interpret
+   arbitrary keys such as ``data_path`` or ``batch_size``. Pass those values
+   through ``train_args`` unless your recipe explicitly documents another shape.
 
 No Secrets In Recipe Parameters
 -------------------------------

docs/user_guide/data_scientist_guide/recipe_api.rst

@@ -104,16 +104,19 @@ File packaging helpers:
    Bundle a file or directory into the generated server app package.
 
 Helpers that accept ``clients`` target specific generated client apps. This
-requires per-site client apps: construct the recipe with the
-``per_site_config`` constructor argument on recipes that support it, and each
-name in ``clients`` must match an existing per-site client app. With the
+requires per-site client apps: use ``set_per_site_config`` before targeting
+clients, or pass the ``per_site_config`` constructor argument on recipes that
+support it. Each name in ``clients`` must match an existing per-site client app. With the
 default all-clients topology, targeted calls raise an error rather than
 silently dropping the change from the generated job, and unknown site names
-raise an error rather than deploying a bare app to that site. Calling
-``set_per_site_config`` after construction records the configuration for
-``configured_sites()`` but does not yet rebuild an existing all-clients app
-into per-site apps; recipes will interpret helper-provided per-site config as
-follow-up work.
+raise an error rather than deploying a bare app to that site. For the unified
+FedAvg recipe family and PyTorch ``FedEvalRecipe``, calling
+``set_per_site_config`` after construction rebuilds the existing all-clients
+app into per-site apps while preserving client configuration, files, filters,
+and components added through Recipe APIs. This must be done before exporting
+or running the recipe. Other recipe families must implement their
+recipe-specific per-site configuration hook before the helper can change their
+generated jobs.
 
 Filter helpers:
 
@@ -172,7 +175,35 @@ For NVFlare 2.9, the public Recipe configuration surface also includes:
 ``set_per_site_config(recipe, config)``
    Provide site-keyed, recipe-specific configuration. Each concrete recipe
    interprets the site dictionaries for its own workflow. Nested values become
-   part of the job definition and must not contain secret values.
+   part of the job definition and must not contain secret values. Call the
+   helper before exporting or executing the recipe.
+
+   .. list-table:: Recipe support
+      :header-rows: 1
+      :widths: 24 46 30
+
+      * - Recipe family
+        - Per-site fields
+        - How to configure
+      * - Unified FedAvg and its PyTorch, TensorFlow, NumPy, and scikit-learn variants
+        - ``train_script``, ``train_args``, ``launch_external_process``,
+          ``command``, ``framework``, ``server_expected_format``,
+          ``params_transfer_type``, ``launch_once``, and ``shutdown_timeout``
+        - Helper or constructor
+      * - PyTorch ``FedEvalRecipe``
+        - ``eval_script``, ``eval_args``, ``launch_external_process``,
+          ``command``, and ``server_expected_format``
+        - Helper or constructor
+      * - XGBoost bagging, horizontal, and vertical recipes
+        - Required site data loaders, plus recipe-specific values such as
+          bagging ``lr_scale``
+        - Constructor only; these values are needed while building the job
+
+   On FedAvg and FedEval, the first helper call replaces the all-clients app
+   with one app per configured site while preserving files, filters,
+   components, and top-level client configuration already added through Recipe
+   APIs. The helper may be reapplied for the same site names. To change the
+   site set, create a new recipe.
 
 ``recipe.configured_sites()``
    Return top-level site names from helper-provided per-site config when

examples/advanced/experiment-tracking/mlflow/hello-pt-mlflow-client/README.md

@@ -193,15 +193,17 @@ each site. In simulation, all clients share the host filesystem, so use the
 default `tracking_uri=None` behavior or configure distinct paths if the stores
 must remain physically separate.
 
-For different configuration at each site, construct the recipe with per-site
-client apps and then target each site explicitly:
+For different configuration at each site, use `set_per_site_config` before
+targeting each site explicitly:
 
 ```python
+from nvflare.recipe import set_per_site_config
+
 sites = ["site-1", "site-2"]
 recipe = FedAvgRecipe(
     ...,
-    per_site_config={site: {} for site in sites},
 )
+set_per_site_config(recipe, {site: {} for site in sites})
 
 for site in sites:
     add_experiment_tracking(
@@ -217,8 +219,10 @@ for site in sites:
     )
 ```
 
-Targeted `clients=[...]` placement requires `per_site_config` when the recipe is
-constructed; it cannot split an existing `@ALL` client app after the fact.
+Targeted `clients=[...]` placement requires per-site client apps. For FedAvg,
+`set_per_site_config` converts the initial `@ALL` client app before the recipe
+is exported or run; passing `per_site_config` to the constructor remains
+supported as well.
 
 ### Add Experiment Tags
 

examples/advanced/sklearn-linear/README.md

@@ -89,13 +89,16 @@ Modify `calculate_data_splits()` in `job.py` to implement different split strate
 - **Unbalanced splits**: Give clients different amounts of data (e.g., linear, exponential ratios)
 - **Overlapping validation**: Use different validation sets per client
 
-The key is passing a dict to `train_args`:
+The key is building site-specific `train_args` entries and applying them with
+`set_per_site_config`:
 ```python
-train_args = {
-    "site-1": "--data_path /data/HIGGS.csv --train_start 1100000 --train_end 3080000 ...",
-    "site-2": "--data_path /data/HIGGS.csv --train_start 3080000 --train_end 5060000 ...",
-    # ... more sites
+from nvflare.recipe import set_per_site_config
+
+per_site_config = {
+    "site-1": {"train_args": "--data_path /data/HIGGS.csv --train_start 1100000 --train_end 3080000 ..."},
+    "site-2": {"train_args": "--data_path /data/HIGGS.csv --train_start 3080000 --train_end 5060000 ..."},
 }
+set_per_site_config(recipe, per_site_config)
 ```
 
 **Alternative: Using Separate Data Files**
@@ -108,11 +111,12 @@ Instead of using data ranges, you can split your data into separate files for ea
 # - /data/site2_HIGGS.csv
 # - /data/site3_HIGGS.csv
 
-train_args = {
-    "site-1": "--data_path /data/site1_HIGGS.csv",
-    "site-2": "--data_path /data/site2_HIGGS.csv",
-    "site-3": "--data_path /data/site3_HIGGS.csv",
+per_site_config = {
+    "site-1": {"train_args": "--data_path /data/site1_HIGGS.csv"},
+    "site-2": {"train_args": "--data_path /data/site2_HIGGS.csv"},
+    "site-3": {"train_args": "--data_path /data/site3_HIGGS.csv"},
 }
+set_per_site_config(recipe, per_site_config)
 ```
 
 This replaces the old `prepare_job_config.sh` approach with a more flexible Python-based solution.

examples/advanced/sklearn-linear/job.py

@@ -24,7 +24,7 @@
 import argparse
 
 from nvflare.app_opt.sklearn import SklearnFedAvgRecipe
-from nvflare.recipe import SimEnv
+from nvflare.recipe import SimEnv, set_per_site_config
 
 
 def define_parser():
@@ -85,7 +85,6 @@ def main():
 
     # Calculate per-client data splits (non-overlapping ranges)
     splits = calculate_data_splits(n_clients)
-    clients = [site_name for site_name in splits.keys()]
     per_site_config = {
         site_name: {
             "train_args": f"--data_path {data_path} --train_start {split['train_start']} "
@@ -108,12 +107,12 @@ def main():
             "fit_intercept": 1,
         },
         train_script="client.py",
-        per_site_config=per_site_config,
         key_metric="AUC",
     )
+    set_per_site_config(recipe, per_site_config)
 
     print("Executing recipe in simulation environment...")
-    env = SimEnv(clients=clients, num_threads=n_clients)
+    env = SimEnv(clients=recipe.configured_sites(), num_threads=n_clients)
     run = recipe.execute(env)
 
     print()

nvflare/app_common/np/recipes/fedavg.py

@@ -67,6 +67,8 @@ class NumpyFedAvgRecipe(UnifiedFedAvgRecipe):
             Defaults to TransferType.FULL.
         per_site_config: Per-site configuration for the federated learning job. Nested
             values become part of the generated job definition and must not contain secrets.
+            It may be supplied here or with ``set_per_site_config`` after construction
+            and before export or execution.
         launch_once: Whether external process is launched once or per task. Defaults to True.
         shutdown_timeout: Seconds to wait before shutdown. Defaults to 0.0.
         key_metric: Metric used to determine if the model is globally best. Defaults to "accuracy".

nvflare/app_opt/pt/recipes/fedavg.py

@@ -69,6 +69,8 @@ class FedAvgRecipe(UnifiedFedAvgRecipe):
         model_locator: Custom model locator. If None, PTFileModelLocator will be used.
         per_site_config: Per-site configuration for the federated learning job. Nested
             values become part of the generated job definition and must not contain secrets.
+            It may be supplied here or with ``set_per_site_config`` after construction
+            and before export or execution.
         launch_once: Whether external process is launched once or per task. Defaults to True.
         shutdown_timeout: Seconds to wait before shutdown. Defaults to 0.0.
         key_metric: Metric used to determine if the model is globally best. Defaults to "accuracy".

nvflare/app_opt/pt/recipes/fedeval.py

@@ -16,6 +16,7 @@
 
 from pydantic import BaseModel, field_validator
 
+from nvflare.apis.job_def import ALL_SITES, SERVER_SITE_NAME
 from nvflare.app_common.workflows.model_controller import ModelController
 from nvflare.client.config import ExchangeFormat
 from nvflare.job_config.base_fed_job import BaseFedJob
@@ -88,6 +89,8 @@ class FedEvalRecipe(Recipe):
             eval_script, eval_args, launch_external_process, command, server_expected_format.
             Values are stored in the job definition and must not contain actual secret values.
             If not provided, the same configuration will be used for all clients. Defaults to None.
+            The same mapping may instead be applied with ``set_per_site_config`` after construction
+            and before export or execution.
 
     Example:
         Basic usage with model instance:
@@ -154,6 +157,7 @@ def __init__(
         self.per_site_config = per_site_config
         self.client_memory_gc_rounds = client_memory_gc_rounds
         self.cuda_empty_cache = cuda_empty_cache
+        self._validate_per_site_config(self.per_site_config)
 
         # Create BaseFedJob
         job = BaseFedJob(
@@ -193,34 +197,49 @@ def __init__(
         # Add client executors
         if self.per_site_config is not None:
             for site_name, site_config in self.per_site_config.items():
-                script = site_config.get("eval_script", self.eval_script)
-                script_args = site_config.get("eval_args", self.eval_args)
-                launch_external = site_config.get("launch_external_process", self.launch_external_process)
-                cmd = site_config.get("command", self.command)
-                expected_format = site_config.get("server_expected_format", self.server_expected_format)
-
-                executor = ScriptRunner(
-                    script=script,
-                    script_args=script_args,
-                    launch_external_process=launch_external,
-                    command=cmd,
-                    framework=FrameworkType.PYTORCH,
-                    server_expected_format=expected_format,
-                    memory_gc_rounds=self.client_memory_gc_rounds,
-                    cuda_empty_cache=self.cuda_empty_cache,
-                )
-                job.to(executor, site_name)
+                self._add_client_runner(job, site_name, site_config)
         else:
-            executor = ScriptRunner(
-                script=self.eval_script,
-                script_args=self.eval_args,
-                launch_external_process=self.launch_external_process,
-                command=self.command,
-                framework=FrameworkType.PYTORCH,
-                server_expected_format=self.server_expected_format,
-                memory_gc_rounds=self.client_memory_gc_rounds,
-                cuda_empty_cache=self.cuda_empty_cache,
-            )
-            job.to_clients(executor)
+            self._add_client_runner(job, ALL_SITES, {})
 
         Recipe.__init__(self, job)
+
+    def _create_client_runner(self, site_config: Dict) -> tuple[ScriptRunner, str]:
+        script = site_config.get("eval_script", self.eval_script)
+        runner = ScriptRunner(
+            script=script,
+            script_args=site_config.get("eval_args", self.eval_args),
+            launch_external_process=site_config.get("launch_external_process", self.launch_external_process),
+            command=site_config.get("command", self.command),
+            framework=FrameworkType.PYTORCH,
+            server_expected_format=site_config.get("server_expected_format", self.server_expected_format),
+            memory_gc_rounds=self.client_memory_gc_rounds,
+            cuda_empty_cache=self.cuda_empty_cache,
+        )
+        return runner, script
+
+    def _add_client_runner(self, job: BaseFedJob, target: str, site_config: Dict) -> None:
+        runner, script = self._create_client_runner(site_config)
+        component_ids = job.to(runner, target)
+        self._record_client_runner(target, component_ids, script)
+
+    def _apply_per_site_config(self, config: Dict[str, Dict]) -> None:
+        """Replace the all-clients evaluator with site-specific client apps."""
+        self._validate_per_site_config(config)
+        self._replace_client_runners_for_sites(config, self._add_client_runner)
+
+    @staticmethod
+    def _validate_per_site_config(per_site_config: Optional[Dict[str, Dict]]) -> None:
+        if per_site_config is None:
+            return
+
+        reserved_targets = {SERVER_SITE_NAME, ALL_SITES}
+        for site_name, site_config in per_site_config.items():
+            if not isinstance(site_name, str):
+                raise ValueError(f"per_site_config key must be str, got {type(site_name).__name__}")
+            if site_name in reserved_targets:
+                raise ValueError(
+                    f"'{site_name}' is a reserved target name and cannot be used in per_site_config. "
+                    f"Reserved names: {sorted(reserved_targets)}"
+                )
+            if not isinstance(site_config, dict):
+                raise ValueError(f"per_site_config['{site_name}'] must be a dict, got {type(site_config).__name__}")

nvflare/app_opt/sklearn/recipes/fedavg.py

@@ -59,7 +59,8 @@ class SklearnFedAvgRecipe(UnifiedFedAvgRecipe):
         per_site_config: Per-site configuration for the federated learning job. Dictionary mapping
             site names to configuration dicts. If not provided, the same configuration will be used
             for all clients. Nested values become part of the generated job definition and must not
-            contain secrets.
+            contain secrets. It may also be applied with ``set_per_site_config`` after construction
+            and before export or execution.
         key_metric: Metric used to determine if the model is globally best. If validation metrics are
             a dict, key_metric selects the metric used for global model selection. Defaults to "accuracy".
         launch_once: Whether the external process will be launched only once at the beginning
@@ -97,14 +98,18 @@ class SklearnFedAvgRecipe(UnifiedFedAvgRecipe):
 
         ```python
         from nvflare.app_opt.sklearn import SklearnFedAvgRecipe
+        from nvflare.recipe import set_per_site_config
 
         recipe = SklearnFedAvgRecipe(
             name="sklearn_linear",
             min_clients=3,
             num_rounds=50,
             model_params={"n_classes": 2, "learning_rate": "constant", "eta0": 1e-4},
             train_script="client.py",
-            per_site_config={
+        )
+        set_per_site_config(
+            recipe,
+            {
                 "site-1": {"train_args": "--data_path /tmp/data/site1.csv"},
                 "site-2": {"train_args": "--data_path /tmp/data/site2.csv"},
                 "site-3": {"train_args": "--data_path /tmp/data/site3.csv"},

nvflare/app_opt/sklearn/recipes/kmeans.py

@@ -82,7 +82,8 @@ class KMeansFedAvgRecipe(FedAvgRecipe):
         per_site_config: Per-site configuration for the federated learning job. Dictionary mapping
             site names to configuration dicts. If not provided, the same configuration will be used
             for all clients. Nested values become part of the generated job definition and must not
-            contain secrets.
+            contain secrets. It may also be applied with ``set_per_site_config`` after construction
+            and before export or execution.
         key_metric: Metric used to determine if the model is globally best. If validation metrics are
             a dict, key_metric selects the metric used for global model selection. Defaults to "metrics"
             (which corresponds to the homogeneity score sent by the K-Means client).

nvflare/app_opt/sklearn/recipes/svm.py

@@ -81,7 +81,8 @@ class SVMFedAvgRecipe(FedAvgRecipe):
         per_site_config: Per-site configuration for the federated learning job. Dictionary mapping
             site names to configuration dicts. If not provided, the same configuration will be used
             for all clients. Nested values become part of the generated job definition and must not
-            contain secrets.
+            contain secrets. It may also be applied with ``set_per_site_config`` after construction
+            and before export or execution.
         key_metric: Metric used to determine if the model is globally best. If validation metrics are
             a dict, key_metric selects the metric used for global model selection. Defaults to "AUC"
             (which corresponds to the ROC AUC score sent by the SVM client in round 1).