diff --git a/.cursor/rules/mod-000a-reusable-layers-belong-in-nn.mdc b/.cursor/rules/mod-000a-reusable-layers-belong-in-nn.mdc new file mode 100644 index 0000000000..d1f5d544bb --- /dev/null +++ b/.cursor/rules/mod-000a-reusable-layers-belong-in-nn.mdc @@ -0,0 +1,58 @@ +--- +description: Reusable layers and building blocks should be placed in physicsnemo/nn, not physicsnemo/models. Examples include FullyConnected, attention layers, and UNetBlock. +alwaysApply: false +--- + +When creating or refactoring reusable layer code, rule MOD-000a must be followed. Explicitly reference "Following rule MOD-000a, which states that reusable layers should go in physicsnemo/nn..." when explaining placement decisions. + +## MOD-000a: Reusable layers/blocks belong in physicsnemo.nn + +**Description:** + +Reusable layers that are the building blocks of more complex architectures +should go into `physicsnemo/nn`. Those include for instance `FullyConnected`, +various variants of attention layers, `UNetBlock` (a block of a U-Net), etc. + +All layers that are directly exposed to the user should be imported in +`physicsnemo/nn/__init__.py`, such that they can be used as follows: + +```python +from physicsnemo.nn import MyLayer +``` + +The only exception to this rule is for layers that are highly specific to a +single example. In this case, it may be acceptable to place them in a module +specific to the example code, such as `examples//utils/nn.py`. + +**Rationale:** + +Ensures consistency in the organization of reusable layers in the repository. +Keeping all reusable components in a single location makes them easy to find +and promotes code reuse across different models. + +**Example:** + +```python +# Good: Reusable layer in physicsnemo/nn/attention.py +class MultiHeadAttention(Module): + """A reusable attention layer that can be used in various architectures.""" + pass + +# Good: Import in physicsnemo/nn/__init__.py +from physicsnemo.nn.attention import MultiHeadAttention + +# Good: Example-specific layer in examples/weather/utils/nn.py +class WeatherSpecificLayer(Module): + """Layer highly specific to the weather forecasting example.""" + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Reusable layer placed in physicsnemo/models/ +# File: physicsnemo/models/attention.py +class MultiHeadAttention(Module): + """Should be in physicsnemo/nn/ not physicsnemo/models/""" + pass +``` diff --git a/.cursor/rules/mod-000b-complete-models-belong-in-models.mdc b/.cursor/rules/mod-000b-complete-models-belong-in-models.mdc new file mode 100644 index 0000000000..889bb4aae3 --- /dev/null +++ b/.cursor/rules/mod-000b-complete-models-belong-in-models.mdc @@ -0,0 +1,54 @@ +--- +description: Complete models composed of multiple layers should be placed in physicsnemo/models, not physicsnemo/nn. These are domain-specific or modality-specific models. +alwaysApply: false +--- + +When creating or refactoring complete model code, rule MOD-000b must be followed. Explicitly reference "Following rule MOD-000b, which states that complete models should go in physicsnemo/models..." when explaining placement decisions. + +## MOD-000b: Complete models belong in physicsnemo.models + +**Description:** + +More complete models, composed of multiple layers and/or other sub-models, +should go into `physicsnemo/models`. All models that are directly exposed to +the user should be imported in `physicsnemo/models/__init__.py`, such that they +can be used as follows: + +```python +from physicsnemo.models import MyModel +``` + +The only exception to this rule is for models that are highly specific to a +single example. In this case, it may be acceptable to place them in a module +specific to the example code, such as `examples//utils/nn.py`. + +**Rationale:** + +Ensures consistency and clarity in the organization of models in the repository, +in particular a clear separation between reusable layers and more complete +models that are applicable to a specific domain or specific data modality. + +**Example:** + +```python +# Good: Complete model in physicsnemo/models/transformer.py +class TransformerModel(Module): + """A complete transformer model composed of attention and feedforward layers.""" + def __init__(self): + super().__init__() + self.attention = MultiHeadAttention(...) + self.ffn = FeedForward(...) + +# Good: Import in physicsnemo/models/__init__.py +from physicsnemo.models.transformer import TransformerModel +``` + +**Anti-pattern:** + +```python +# WRONG: Complete model placed in physicsnemo/nn/ +# File: physicsnemo/nn/transformer.py +class TransformerModel(Module): + """Should be in physicsnemo/models/ not physicsnemo/nn/""" + pass +``` diff --git a/.cursor/rules/mod-001-use-physicsnemo-module-as-base-class.mdc b/.cursor/rules/mod-001-use-physicsnemo-module-as-base-class.mdc new file mode 100644 index 0000000000..38e8d36b53 --- /dev/null +++ b/.cursor/rules/mod-001-use-physicsnemo-module-as-base-class.mdc @@ -0,0 +1,47 @@ +--- +description: All model and layer classes must inherit from physicsnemo.Module (not torch.nn.Module directly) to ensure proper serialization, versioning, and registry functionality. +alwaysApply: false +--- + +When creating or modifying model classes, rule MOD-001 must be strictly followed. Explicitly reference "Following rule MOD-001, which states that all model classes must inherit from physicsnemo.Module..." when explaining inheritance decisions. + +## MOD-001: Use physicsnemo.Module as model base classes + +**Description:** + +All model classes must inherit from `physicsnemo.Module`. Direct subclasses of +`torch.nn.Module` are not allowed. Direct subclasses of `physicsnemo.Module` +are allowed (note that `physicsnemo.Module` is a subclass of `torch.nn.Module`). +Ensure proper initialization of parent classes using `super().__init__()`. Pass +the `meta` argument to the `super().__init__()` call if appropriate, otherwise +set it manually with `self.meta = meta`. + +**Rationale:** + +Ensures invariants and functionality of the `physicsnemo.Module` class for all +models. In particular, instances of `physicsnemo.Module` benefit from features +that are not available in `torch.nn.Module` instances. Those include serialization +for checkpointing and loading modules and submodules, versioning system to +handle backward compatibility, as well as ability to be registered in the +`physicsnemo.registry` for easy instantiation and use in any codebase. + +**Example:** + +```python +from physicsnemo import Module + +class MyModel(Module): + def __init__(self, input_dim: int, output_dim: int): + super().__init__(meta=MyModelMetaData()) + self.linear = nn.Linear(input_dim, output_dim) +``` + +**Anti-pattern:** + +```python +from torch import nn + +class MyModel(nn.Module): + def __init__(self, input_dim: int, output_dim: int): + self.linear = nn.Linear(input_dim, output_dim) +``` diff --git a/.cursor/rules/mod-002a-experimental-models-belong-in-experimental.mdc b/.cursor/rules/mod-002a-experimental-models-belong-in-experimental.mdc new file mode 100644 index 0000000000..44076619e5 --- /dev/null +++ b/.cursor/rules/mod-002a-experimental-models-belong-in-experimental.mdc @@ -0,0 +1,65 @@ +--- +description: New model classes should start in physicsnemo/experimental/nn or physicsnemo/experimental/models during development, where backward compatibility is not guaranteed. +alwaysApply: false +--- + +When creating new model or layer classes, rule MOD-002a must be followed. Explicitly reference "Following rule MOD-002a, which states that new models should start in physicsnemo/experimental/..." when explaining where to place new code. + +## MOD-002a: New models and layers belong in physicsnemo.experimental + +**Description:** + +For the vast majority of models, new classes are created either in +`physicsnemo/experimental/nn` for reusable layers, or in +`physicsnemo/experimental/models` for more complete models. The `experimental` +folder is used to store models that are still under development (beta or alpha +releases) during this stage, backward compatibility is not guaranteed. + +One exception is when the developer is highly confident that the model is +sufficiently mature and applicable to many domains or use cases. In this case +the model class can be created in the `physicsnemo/nn` or `physicsnemo/models` +folders directly, and backward compatibility is guaranteed. + +Another exception is when the model class is highly specific to a single +example. In this case, it may be acceptable to place it in a module specific to +the example code, such as `examples//utils/nn.py`. + +After staying in experimental for a sufficient amount of time (typically at +least 1 release cycle), the model class can be promoted to production. It is +then moved to the `physicsnemo/nn` or `physicsnemo/models` folders, based on +whether it's a reusable layer or complete model (see MOD-000a and MOD-000b). + +**Note:** Per MOD-008a, MOD-008b, and MOD-008c, it is forbidden to move a model +out of the experimental stage/directory without the required CI tests. + +**Rationale:** + +The experimental stage allows rapid iteration without backward compatibility +constraints, enabling developers to refine APIs based on user feedback. This +protects users from unstable APIs while allowing innovation. + +**Example:** + +```python +# Good: Stage 1 - New experimental model +# File: physicsnemo/experimental/models/new_diffusion.py +class DiffusionModel(Module): + """New diffusion model under active development. API may change.""" + pass + +# Good: After 1+ release cycles, promoted to production +# File: physicsnemo/models/diffusion.py (moved from experimental/) +class DiffusionModel(Module): + """Stable diffusion model with backward compatibility guarantees.""" + pass +``` + +**Anti-pattern:** + +```python +# WRONG: New model directly in production folder +# File: physicsnemo/models/brand_new_model.py (should be in experimental/ first) +class BrandNewModel(Module): + """Skipped experimental stage - risky for stability""" + pass +``` diff --git a/.cursor/rules/mod-002b-add-deprecation-warnings-to-model.mdc b/.cursor/rules/mod-002b-add-deprecation-warnings-to-model.mdc new file mode 100644 index 0000000000..45e7f66883 --- /dev/null +++ b/.cursor/rules/mod-002b-add-deprecation-warnings-to-model.mdc @@ -0,0 +1,69 @@ +--- +description: Model classes being deprecated must include deprecation warnings in both docstring and runtime, explaining why and what users should use instead, for at least 1 release cycle. +alwaysApply: false +--- + +When deprecating a model class, rule MOD-002b must be followed. Explicitly reference "Following rule MOD-002b, which requires adding deprecation warnings to both docstring and runtime..." when implementing deprecation. + +## MOD-002b: Add deprecation warnings to deprecating model class + +**Description:** + +For a model class in the pre-deprecation stage in `physicsnemo/nn` or +`physicsnemo/models`, the developer should start planning its deprecation. This +is done by adding a warning message to the model class, indicating that the +model class is deprecated and will be removed in a future release. + +The warning message should be a clear and concise message that explains why the +model class is being deprecated and what the user should do instead. The +deprecation message should be added to both the docstring and should be raised +at runtime. The developer is free to choose the mechanism to raise the +deprecation warning. + +A model class cannot be deprecated without staying in the pre-deprecation stage +for at least 1 release cycle before it can be deleted from the codebase. + +**Rationale:** + +Ensures users have sufficient time to migrate to newer alternatives, preventing +breaking changes that could disrupt their workflows. This graduated approach +balances innovation with stability, a critical requirement for a scientific +computing framework. + +**Example:** + +```python +# Good: Pre-deprecation with warning +# File: physicsnemo/models/old_diffusion.py +class DiffusionModel(Module): + """ + Legacy diffusion model. + + .. deprecated:: 0.5.0 + ``OldDiffusionModel`` is deprecated and will be removed in version 0.7.0. + Use :class:`~physicsnemo.models.NewDiffusionModel` instead. + """ + def __init__(self): + import warnings + warnings.warn( + "OldDiffusionModel is deprecated. Use NewDiffusionModel instead.", + DeprecationWarning, + stacklevel=2 + ) + super().__init__() +``` + +**Anti-pattern:** + +```python +# WRONG: No deprecation warning in code +# File: physicsnemo/models/old_model.py +class OldModel(Module): + """Will be removed next release.""" # Docstring mentions it but no runtime warning + def __init__(self): + # Missing: warnings.warn(..., DeprecationWarning) + super().__init__() + +# WRONG: Deprecation without sufficient warning period +# (Model deprecated and removed in same release) +``` diff --git a/.cursor/rules/mod-002c-remove-deprecated-model-from-codebase.mdc b/.cursor/rules/mod-002c-remove-deprecated-model-from-codebase.mdc new file mode 100644 index 0000000000..735ae3ce9c --- /dev/null +++ b/.cursor/rules/mod-002c-remove-deprecated-model-from-codebase.mdc @@ -0,0 +1,50 @@ +--- +description: After at least 1 release cycle in pre-deprecation stage with warnings, deprecated model classes can be deleted from the codebase. +alwaysApply: false +--- + +When removing deprecated models, rule MOD-002c must be followed. Explicitly reference "Following rule MOD-002c, which states that a model can only be deleted after at least 1 release cycle in pre-deprecation..." when removing code. + +## MOD-002c: Remove deprecated model from codebase + +**Description:** + +After staying in the pre-deprecation stage (Stage 3) for at least 1 release +cycle, the model class is considered deprecated (Stage 4). It can then be +deleted from the codebase. + +A model class cannot be deleted without first spending at least 1 release cycle +in the pre-deprecation stage with proper deprecation warnings (see MOD-002b). + +**Rationale:** + +This ensures users have sufficient warning and time to migrate their code to +newer alternatives. Premature deletion of models would break user code without +adequate notice, violating the framework's commitment to stability. + +**Example:** + +```python +# Good: Model spent 1 release cycle in pre-deprecation (v0.5.0 with warnings) +# Now in v0.6.0, can be deleted +# File: physicsnemo/models/old_diffusion.py - DELETED + +# Release timeline: +# v0.5.0: Added deprecation warnings (Stage 3) +# v0.6.0: Model can be safely removed (Stage 4) +``` + +**Anti-pattern:** + +```python +# WRONG: Deleting model without deprecation period +# v0.5.0: Model exists without warnings +# v0.6.0: Model deleted - BREAKS USER CODE! + +# WRONG: Breaking changes in production without deprecation cycle +# File: physicsnemo/models/diffusion.py +class DiffusionModel(Module): + def __init__(self, new_required_param): # Breaking change! + # Changed API without deprecation warning - breaks user code + pass +``` diff --git a/.cursor/rules/mod-003a-missing-or-incomplete-docstring.mdc b/.cursor/rules/mod-003a-missing-or-incomplete-docstring.mdc new file mode 100644 index 0000000000..77268c581a --- /dev/null +++ b/.cursor/rules/mod-003a-missing-or-incomplete-docstring.mdc @@ -0,0 +1,65 @@ +--- +description: Every model/layer requires comprehensive docstrings following NumPy style with Sphinx RST formatting, including all sub-rules MOD-003b through MOD-003k. +alwaysApply: false +--- + +When writing model or layer documentation, rule MOD-003a must be followed. Explicitly reference "Following rule MOD-003a, which requires comprehensive docstrings following all MOD-003 sub-rules..." when creating documentation. + +## MOD-003a: Missing or incomplete docstring for model/layer code + +**Description:** + +Every new model or modification of any model code should be documented with a +comprehensive docstring following all the sub-rules MOD-003b through MOD-003k. +All docstrings should be written in the NumPy style and adopt formatting to be +compatible with our Sphinx restructured text (RST) documentation. + +**Rationale:** + +Comprehensive and well-formatted documentation is essential for scientific +software. It enables users to understand model capabilities, expected inputs, +and outputs without inspecting source code. + +**Example:** + +```python +class MyEncoder(Module): + r""" + A simple encoder network. + + Parameters + ---------- + input_dim : int + Dimension of input features. + output_dim : int + Dimension of output features. + + Forward + ------- + x : torch.Tensor + Input tensor of shape :math:`(B, D_{in})`. + + Outputs + ------- + torch.Tensor + Output tensor of shape :math:`(B, D_{out})`. + + Examples + -------- + >>> model = MyEncoder(input_dim=784, output_dim=128) + >>> x = torch.randn(32, 784) + >>> output = model(x) + >>> output.shape + torch.Size([32, 128]) + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Missing all required sections +class BadEncoder(Module): + '''A simple encoder.''' # Wrong quotes, no sections + pass +``` diff --git a/.cursor/rules/mod-003b-docstring-must-use-raw-string-prefix.mdc b/.cursor/rules/mod-003b-docstring-must-use-raw-string-prefix.mdc new file mode 100644 index 0000000000..98ff124e15 --- /dev/null +++ b/.cursor/rules/mod-003b-docstring-must-use-raw-string-prefix.mdc @@ -0,0 +1,54 @@ +--- +description: Model and method docstrings must be prefixed with r""" (raw string with triple double quotes) for proper LaTeX rendering in Sphinx documentation. +alwaysApply: false +--- + +When writing docstrings, rule MOD-003b must be followed. Explicitly reference "Following rule MOD-003b, which requires docstrings to be prefixed with r"""..." when explaining docstring format. + +## MOD-003b: Docstring must use raw string prefix r""" + +**Description:** + +Each docstring should be prefixed with `r"""` (not `"""` or `'''`). The `r` +prefix creates a raw string that prevents Python from interpreting backslashes, +which is essential for LaTeX math notation to render correctly in Sphinx +documentation. + +**Rationale:** + +LaTeX commands in docstrings use backslashes (e.g., `\math`, `\text`). Without +the raw string prefix, Python interprets these as escape sequences, breaking the +documentation rendering. + +**Example:** + +```python +class MyModel(Module): + r""" + A model with LaTeX notation. + + Parameters + ---------- + dim : int + Dimension :math:`D` of input features. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Using ''' instead of r""" +class MyModel(Module): + ''' + A model with LaTeX notation. + ''' + pass + +# WRONG: Missing 'r' prefix +class MyModel(Module): + """ + Parameters with shape :math:`(B, D)` # Won't render correctly + """ + pass +``` diff --git a/.cursor/rules/mod-003c-missing-required-class-docstring-sections.mdc b/.cursor/rules/mod-003c-missing-required-class-docstring-sections.mdc new file mode 100644 index 0000000000..927ed4ac62 --- /dev/null +++ b/.cursor/rules/mod-003c-missing-required-class-docstring-sections.mdc @@ -0,0 +1,80 @@ +--- +description: Class docstrings must contain three mandatory sections - Parameters, Forward, and Outputs. Optional sections include Notes, Examples, ..important::, and ..code-block::. +alwaysApply: false +--- + +When writing class docstrings, rule MOD-003c must be followed. Explicitly reference "Following rule MOD-003c, which requires class docstrings to have Parameters, Forward, and Outputs sections..." when structuring documentation. + +## MOD-003c: Missing required class docstring sections + +**Description:** + +The class docstring should at least contain three sections: `Parameters`, +`Forward`, and `Outputs`. The forward method should be documented in the +docstring of the model class, instead of being in the docstring of the forward +method itself. A docstring for the forward method is still possible but it +should be concise and to the point. + +Other sections such as `Notes`, `Examples`, or `..important::` or `..code-block:: +python` are possible. Other sections are not recognized by our Sphinx +documentation and are prohibited. + +**Rationale:** + +Standardized sections ensure documentation is consistent and complete across all +models. The Forward and Outputs sections in the class docstring provide a +centralized place to document the model's primary behavior, making it easier for +users to understand the model's API. + +**Example:** + +```python +class MyModel(Module): + r""" + A simple encoder model. + + Parameters + ---------- + input_dim : int + Dimension of input features. + output_dim : int + Dimension of output features. + + Forward + ------- + x : torch.Tensor + Input tensor of shape :math:`(B, D_{in})`. + + Outputs + ------- + torch.Tensor + Output tensor of shape :math:`(B, D_{out})`. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Missing Parameters, Forward, or Outputs sections +class BadModel(Module): + r""" + A simple encoder model. + + No proper sections defined. + """ + pass + +# WRONG: Using unrecognized section names +class BadModel(Module): + r""" + Description + ----------- + A simple encoder model. + + Args + ---- + input_dim: dimension + """ + pass +``` diff --git a/.cursor/rules/mod-003d-missing-required-method-docstring-sections.mdc b/.cursor/rules/mod-003d-missing-required-method-docstring-sections.mdc new file mode 100644 index 0000000000..8684b063c3 --- /dev/null +++ b/.cursor/rules/mod-003d-missing-required-method-docstring-sections.mdc @@ -0,0 +1,73 @@ +--- +description: All methods must have docstrings with at least Parameters and Returns sections. Optional sections include Notes, Examples, ..important::, and ..code-block::. +alwaysApply: false +--- + +When writing method docstrings, rule MOD-003d must be followed. Explicitly reference "Following rule MOD-003d, which requires method docstrings to have Parameters and Returns sections..." when documenting methods. + +## MOD-003d: Missing required method docstring sections + +**Description:** + +All methods should be documented with a docstring, with at least a `Parameters` +section and a `Returns` section. Other sections such as `Notes`, `Examples`, or +`..important::` or `..code-block:: python` are possible. Other sections are not +recognized by our Sphinx documentation and are prohibited. + +Note: The forward method is a special case - its full documentation should be in +the class docstring (see MOD-003c), though a concise forward method docstring is +permitted. + +**Rationale:** + +Complete method documentation ensures users understand how to call methods and +what to expect in return. Standardized sections make documentation consistent +and easier to parse for both humans and AI agents. + +**Example:** + +```python +def compute_loss( + self, + pred: torch.Tensor, + target: torch.Tensor, +) -> torch.Tensor: + r""" + Compute mean squared error loss. + + Parameters + ---------- + pred : torch.Tensor + Predicted values of shape :math:`(B, D)`. + target : torch.Tensor + Target values of shape :math:`(B, D)`. + + Returns + ------- + torch.Tensor + Scalar loss value. + """ + return torch.nn.functional.mse_loss(pred, target) +``` + +**Anti-pattern:** + +```python +# WRONG: No docstring at all +def helper_method(self, x): + return x * 2 + +# WRONG: Using unrecognized section names +def compute_loss(self, pred, target): + """ + Compute loss. + + Args: + pred: predicted values + target: target values + + Returns: + loss value + """ + pass +``` diff --git a/.cursor/rules/mod-003e-tensor-shapes-must-use-latex-math-notation.mdc b/.cursor/rules/mod-003e-tensor-shapes-must-use-latex-math-notation.mdc new file mode 100644 index 0000000000..10697d7a4a --- /dev/null +++ b/.cursor/rules/mod-003e-tensor-shapes-must-use-latex-math-notation.mdc @@ -0,0 +1,67 @@ +--- +description: All tensor shapes in docstrings must use LaTeX math notation like :math:`(B, C, H, W)` for proper rendering in Sphinx documentation. +alwaysApply: false +--- + +When documenting tensor shapes, rule MOD-003e must be followed. Explicitly reference "Following rule MOD-003e, which requires tensor shapes to use LaTeX math notation :math:`...`..." when documenting tensors. + +## MOD-003e: Tensor shapes must use LaTeX math notation + +**Description:** + +All tensors should be documented with their shape, using LaTeX math notation +such as `:math:`(N, C, H_{in}, W_{in})``. There is flexibility for naming the +dimensions, but the math format should be enforced. + +Our documentation is rendered using LaTeX, and supports a rich set of LaTeX +commands, so it is recommended to use LaTeX commands whenever possible for +mathematical variables in the docstrings. The mathematical notations should be +to some degree consistent with the actual variable names in the code (even +though that is not always possible, to avoid too complex formatting). + +**Rationale:** + +LaTeX math notation ensures tensor shapes render correctly and consistently in +Sphinx documentation. This is critical for scientific software where precise +mathematical notation is expected. Plain text shapes don't render properly and +can be ambiguous. + +**Example:** + +```python +def forward(self, x: torch.Tensor) -> torch.Tensor: + r""" + Process input tensor. + + Parameters + ---------- + x : torch.Tensor + Input of shape :math:`(B, C, H_{in}, W_{in})` where :math:`B` is batch + size, :math:`C` is channels, and :math:`H_{in}, W_{in}` are spatial dims. + + Returns + ------- + torch.Tensor + Output of shape :math:`(B, C_{out}, H_{out}, W_{out})`. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Not using :math: notation +def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Parameters + ---------- + x : torch.Tensor + Input of shape (B, C, H, W) # Missing :math:`...` + + Returns + ------- + torch.Tensor + Output shape: (B, C_out, H_out, W_out) # Missing :math:`...` + """ + pass +``` diff --git a/.cursor/rules/mod-003f-callback-functions-must-have-code-block-specification.mdc b/.cursor/rules/mod-003f-callback-functions-must-have-code-block-specification.mdc new file mode 100644 index 0000000000..f4df348a82 --- /dev/null +++ b/.cursor/rules/mod-003f-callback-functions-must-have-code-block-specification.mdc @@ -0,0 +1,66 @@ +--- +description: Callback function parameters must include a ..code-block:: specification showing the required signature and return type, placed outside Parameters/Forward/Outputs sections. +alwaysApply: false +--- + +When documenting callback functions or complex API parameters, rule MOD-003f must be followed. Explicitly reference "Following rule MOD-003f, which requires callback functions to have a ..code-block:: specification..." when adding these specifications. + +## MOD-003f: Callback functions must have code-block specification + +**Description:** + +For arguments or variables that are callback functions (e.g. Callable), the +docstring should include a clear separated `..code-block::` that specifies the +required signature and return type of the callback function. This is not only +true for callback functions, but for any type of parameters or arguments that +has some complex type specification or API requirements. + +The explanation code block should be placed in the top or bottom section of the +docstrings, but not in the `Parameters` or `Forward` or `Outputs` sections, for +readability and clarity. + +**Rationale:** + +Callback functions have complex type signatures that are difficult to express +clearly in the Parameters section alone. A dedicated code-block provides a clear +visual reference for the expected signature, making it much easier for users to +implement compatible callbacks. + +**Example:** + +```python +class MyModel(Module): + r""" + Model with callback function. + + .. code-block:: python + + def preprocess_fn(x: torch.Tensor) -> torch.Tensor: + '''Preprocessing function signature.''' + ... + return y + + where ``x`` is input of shape :math:`(B, D_{in})` and ``y`` is output + of shape :math:`(B, D_{out})`. + + Parameters + ---------- + preprocess_fn : Callable[[torch.Tensor], torch.Tensor], optional + Optional preprocessing function. See code block above for signature. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: No code-block specification for callback +class MyModel(Module): + r""" + Parameters + ---------- + preprocess_fn : Callable[[torch.Tensor], torch.Tensor], optional + Preprocessing function. # No specification of signature! + """ + pass +``` diff --git a/.cursor/rules/mod-003g-inline-code-must-use-double-backticks.mdc b/.cursor/rules/mod-003g-inline-code-must-use-double-backticks.mdc new file mode 100644 index 0000000000..5c28eb71f4 --- /dev/null +++ b/.cursor/rules/mod-003g-inline-code-must-use-double-backticks.mdc @@ -0,0 +1,51 @@ +--- +description: Inline code in docstrings must be formatted with double backticks ``code``, not single backticks, as single backticks don't render properly in Sphinx. +alwaysApply: false +--- + +When writing inline code in docstrings, rule MOD-003g must be followed. Explicitly reference "Following rule MOD-003g, which requires inline code to use double backticks..." when formatting code references. + +## MOD-003g: Inline code must use double backticks + +**Description:** + +Inline code should be formatted with double backticks, such as ``my_variable``. +Single backticks are not allowed as they don't render properly in our Sphinx +documentation. + +**Rationale:** + +Sphinx uses reStructuredText, which requires double backticks for inline code +literals. Single backticks are interpreted differently and don't produce the +expected code formatting in the rendered documentation. + +**Example:** + +```python +class MyModel(Module): + r""" + Model with inline code references. + + If ``True``, enables dropout. Set ``model.training`` to control behavior. + The parameter ``hidden_dim`` controls layer size. + + Parameters + ---------- + hidden_dim : int + Size of hidden layer. Access via ``self.hidden_dim``. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Using single backticks +class MyModel(Module): + r""" + If `True`, enables dropout. # WRONG: single backticks + Set `model.training` to control behavior. # WRONG + The parameter `hidden_dim` controls size. # WRONG + """ + pass +``` diff --git a/.cursor/rules/mod-003h-parameters-must-be-documented-on-single-line.mdc b/.cursor/rules/mod-003h-parameters-must-be-documented-on-single-line.mdc new file mode 100644 index 0000000000..89206e8285 --- /dev/null +++ b/.cursor/rules/mod-003h-parameters-must-be-documented-on-single-line.mdc @@ -0,0 +1,61 @@ +--- +description: All parameters must be documented with their type and default values on a single line following NumPy docstring style format. +alwaysApply: false +--- + +When documenting parameters, rule MOD-003h must be followed. Explicitly reference "Following rule MOD-003h, which requires parameters to be documented with type and default on a single line..." when formatting parameter documentation. + +## MOD-003h: Parameters must be documented on single line + +**Description:** + +All parameters should be documented with their type and default values on a +single line, following the NumPy docstring style format: + +``` +parameter_name : type, optional, default=value +``` + +The description then follows on the next line(s), indented. + +**Rationale:** + +This standardized format makes parameter documentation consistent and easy to +parse. It provides all key information (name, type, optionality, default) at a +glance, improving readability. + +**Example:** + +```python +class MyModel(Module): + r""" + Model with properly documented parameters. + + Parameters + ---------- + input_dim : int + Dimension of input features. + hidden_dim : int, optional, default=128 + Dimension of hidden layer. + dropout : float, optional, default=0.1 + Dropout probability. + activation : str, optional, default="relu" + Activation function to use. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Type and default not on same line +class MyModel(Module): + r""" + Parameters + ---------- + hidden_dim : int + optional, default=128 # WRONG: should be on line above + Dimension of hidden layer. + """ + pass +``` diff --git a/.cursor/rules/mod-003i-docstrings-should-include-cross-references.mdc b/.cursor/rules/mod-003i-docstrings-should-include-cross-references.mdc new file mode 100644 index 0000000000..9dcecabb39 --- /dev/null +++ b/.cursor/rules/mod-003i-docstrings-should-include-cross-references.mdc @@ -0,0 +1,64 @@ +--- +description: Docstrings should use Sphinx cross-references (:class:, :func:, :meth:) to link to other code elements and external resources for better documentation connectivity. +alwaysApply: false +--- + +When writing docstrings with references to other code, rule MOD-003i should be followed. Explicitly reference "Following rule MOD-003i, which encourages cross-references using :class:, :func:, :meth:..." when adding links. + +## MOD-003i: Docstrings should include cross-references + +**Description:** + +When possible, docstrings should use links to other docstrings using Sphinx +cross-reference syntax: +- Classes: `:class:`~physicsnemo.models.some_model.SomeModel`` +- Functions: `:func:`~physicsnemo.utils.common_function`` +- Methods: `:meth:`~physicsnemo.models.some_model.SomeModel.some_method`` + +When referencing external resources, such as papers, websites, or other +documentation, docstrings should use links to the external resource in the +format `some link text `_. + +**Rationale:** + +Cross-references create a navigable documentation structure where users can +easily jump between related classes, methods, and functions. External links +provide context and attribution for algorithms and techniques. This +improves the overall quality and usability of the documentation. + +**Example:** + +```python +class MyEncoder(Module): + r""" + Encoder network using attention mechanism. + + This implementation is based on `Transformer Architecture `_. + See :class:`~physicsnemo.nn.MultiHeadAttention` for attention details. + + Parameters + ---------- + activation : str + Activation function. See :func:`~torch.nn.functional.relu` for details. + + Notes + ----- + Can be paired with :class:`~physicsnemo.models.decoder.MyDecoder` for + autoencoding tasks. + """ + pass +``` + +**Anti-pattern:** + +```python +# Not necessarily wrong, but missing opportunities for useful links +class MyEncoder(Module): + r""" + Encoder network using attention mechanism. + + Based on the Transformer paper. # Could link to paper + Uses MultiHeadAttention. # Could link to class + """ + pass +``` diff --git a/.cursor/rules/mod-003j-docstrings-should-include-examples-section.mdc b/.cursor/rules/mod-003j-docstrings-should-include-examples-section.mdc new file mode 100644 index 0000000000..6c075250e0 --- /dev/null +++ b/.cursor/rules/mod-003j-docstrings-should-include-examples-section.mdc @@ -0,0 +1,84 @@ +--- +description: Docstrings should include an Examples section with executable code demonstrating usage, as these are automatically tested by CI for correctness. +alwaysApply: false +--- + +When writing model docstrings, rule MOD-003j should be followed. Explicitly reference "Following rule MOD-003j, which encourages an Examples section that CI will automatically test..." when adding examples. + +## MOD-003j: Docstrings should include Examples section + +**Description:** + +Docstrings are strongly encouraged to have an `Examples` section that +demonstrates basic construction and usage of the model. These example sections +serve as both documentation and tests, as our CI system automatically tests +these code sections for correctness when present. + +Examples should be executable Python code showing typical use cases, including +model instantiation, input preparation, and forward pass execution. The examples +should use realistic tensor shapes and demonstrate key features of the model. + +**Rationale:** + +Example sections provide immediate value to users by showing concrete usage +patterns. By automatically testing these examples in CI, we ensure that +documentation stays synchronized with code and that examples remain correct as +the codebase evolves. This catches API changes that would otherwise break user +code without warning. + +**Example:** + +```python +class MyEncoder(Module): + r""" + A simple encoder network. + + Parameters + ---------- + input_dim : int + Dimension of input features. + output_dim : int + Dimension of output features. + + Forward + ------- + x : torch.Tensor + Input tensor of shape :math:`(B, D_{in})`. + + Outputs + ------- + torch.Tensor + Output tensor of shape :math:`(B, D_{out})`. + + Examples + -------- + >>> import torch + >>> from physicsnemo.models import MyEncoder + >>> + >>> # Create model + >>> model = MyEncoder(input_dim=784, output_dim=128) + >>> + >>> # Process a batch + >>> x = torch.randn(32, 784) + >>> output = model(x) + >>> output.shape + torch.Size([32, 128]) + """ + pass +``` + +**Anti-pattern:** + +```python +# Not wrong, but strongly discouraged - no Examples section +class MyEncoder(Module): + r""" + A simple encoder network. + + Parameters + ---------- + input_dim : int + Dimension of input features. + """ + pass +``` diff --git a/.cursor/rules/mod-003k-add-high-level-comments-for-complex-tensor-operations.mdc b/.cursor/rules/mod-003k-add-high-level-comments-for-complex-tensor-operations.mdc new file mode 100644 index 0000000000..5e6254db58 --- /dev/null +++ b/.cursor/rules/mod-003k-add-high-level-comments-for-complex-tensor-operations.mdc @@ -0,0 +1,89 @@ +--- +description: Complex tensor operations should include high-level semantic comments explaining what blocks of code do, plus inline shape comments for chained operations using symbols consistent with docstrings. +alwaysApply: false +--- + +When writing model code with complex tensor operations, rule MOD-003k should be followed. Explicitly reference "Following rule MOD-003k, which recommends high-level comments for complex tensor operations..." when adding explanatory comments. + +## MOD-003k: Add high-level comments for complex tensor operations + +**Description:** + +Model code that involves complex tensor operations should include high-level +comments that explain what blocks of code accomplish semantically. One-line +comments every few lines of tensor operations is sufficient. + +Comments should focus on high-level semantic explanations rather than +low-level syntactic details. For example, use "Compute the encodings" instead of +"Doing a concatenation followed by a linear projection, followed by a nonlinear +activation". The goal is to give a high-level overview of what a block of tensor +operations accomplishes. + +When multiple tensor operations are chained, it is welcomed to add short inline +comments with the tensor shapes of computed tensors, e.g.: + +```python +x = torch.cat([y, z], dim=1) # (B, 2*C_in, H, W) +``` + +The symbols chosen in the comments should be consistent with the docstring +(possibly shortened versions of dimension names for explicitness). + +**Rationale:** + +High-level comments make complex tensor manipulation code more understandable +without cluttering it with excessive detail. Shape annotations help developers +track tensor dimensions through complex operations, catching shape mismatches +early. Consistency with docstring notation creates a unified mental model. + +**Example:** + +```python +def forward(self, x: torch.Tensor, context: torch.Tensor) -> torch.Tensor: + """Process input with context conditioning.""" + # Encode input features + h = self.encoder(x) # (B, C_enc, H, W) + + # Combine with context information + c = self.context_proj(context) # (B, C_enc) + c = c[:, :, None, None].expand(-1, -1, h.shape[2], h.shape[3]) # (B, C_enc, H, W) + h = torch.cat([h, c], dim=1) # (B, 2*C_enc, H, W) + + # Apply attention mechanism + h = self.attention(h) # (B, 2*C_enc, H, W) + + # Decode to output + out = self.decoder(h) # (B, C_out, H, W) + + return out +``` + +**Anti-pattern:** + +```python +# WRONG: No comments for complex operations +def forward(self, x: torch.Tensor, context: torch.Tensor) -> torch.Tensor: + h = self.encoder(x) + c = self.context_proj(context) + c = c[:, :, None, None].expand(-1, -1, h.shape[2], h.shape[3]) + h = torch.cat([h, c], dim=1) + h = self.attention(h) + out = self.decoder(h) + return out + +# WRONG: Too low-level, syntactic comments +def forward(self, x: torch.Tensor, context: torch.Tensor) -> torch.Tensor: + # Pass x through encoder layer + h = self.encoder(x) + # Project context using linear layer + c = self.context_proj(context) + # Add two None dimensions and expand + c = c[:, :, None, None].expand(-1, -1, h.shape[2], h.shape[3]) + # Concatenate h and c along dimension 1 + h = torch.cat([h, c], dim=1) + # Apply attention + h = self.attention(h) + # Pass through decoder + out = self.decoder(h) + return out +``` diff --git a/.cursor/rules/mod-004-model-code-is-not-self-contained.mdc b/.cursor/rules/mod-004-model-code-is-not-self-contained.mdc new file mode 100644 index 0000000000..5eca5175bd --- /dev/null +++ b/.cursor/rules/mod-004-model-code-is-not-self-contained.mdc @@ -0,0 +1,80 @@ +--- +description: All utility functions specific to a model must be in the same module file as the model itself, not in separate utility files, to maintain self-contained modules. +alwaysApply: false +--- + +When organizing model code, rule MOD-004 must be followed. Explicitly reference "Following rule MOD-004, which states that all utility functions for a model class should be contained in the same module file as the model class itself..." when deciding where to place utility functions. + +## MOD-004: Model code is not self-contained + +**Description:** + +All utility functions for a model class should be organized together with the +model class in a clear and logical structure. Acceptable patterns include: + +1. A single self-contained file: `physicsnemo//model_name.py` +2. A subdirectory: `physicsnemo//model_name/` containing: + - `model_name.py` with the main model class + - Additional modules for utility functions specific to this model + +What should be avoided is a flat organization where model files and their +utility files are all mixed together in `physicsnemo//`, making it +unclear which utilities belong to which models. + +The only exception is when a utility function is used across multiple models. In +that case, the shared utility should be placed in an appropriate shared module. + +**Rationale:** + +Self-contained modules are easier to understand, maintain, and navigate. Having +all model-specific code in one place reduces cognitive load and makes it clear +which utilities are model-specific versus shared. This also simplifies code +reviews and reduces the likelihood of orphaned utility files when models are +refactored or removed. + +**Example:** + +```python +# Good Pattern 1: Single self-contained file +# File: physicsnemo/models/my_simple_model.py + +def _compute_attention_mask(seq_length: int) -> torch.Tensor: + """Helper function specific to MySimpleModel.""" + mask = torch.triu(torch.ones(seq_length, seq_length), diagonal=1) + return mask.masked_fill(mask == 1, float('-inf')) + +class MySimpleModel(Module): + """A simple model with utilities in same file.""" + def forward(self, x: torch.Tensor) -> torch.Tensor: + mask = _compute_attention_mask(x.shape[1]) + return self._apply_attention(x, mask) + +# Good Pattern 2: Subdirectory organization +# File: physicsnemo/models/my_complex_model/my_complex_model.py +from physicsnemo.models.my_complex_model.utils import helper_function + +class MyComplexModel(Module): + """A complex model with utilities in subdirectory.""" + pass + +# File: physicsnemo/models/my_complex_model/utils.py +def helper_function(x): + """Utility specific to MyComplexModel.""" + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Flat organization with utilities mixed in main directory +# File: physicsnemo/models/my_transformer.py +from physicsnemo.models.my_transformer_utils import _compute_mask # WRONG + +class MyTransformer(Module): + pass + +# File: physicsnemo/models/my_transformer_utils.py (WRONG: mixed with other models) +# File: physicsnemo/models/other_model.py +# File: physicsnemo/models/other_model_utils.py (WRONG: utilities scattered) +# All mixed together in flat structure - unclear organization! +``` diff --git a/.cursor/rules/mod-005-invalid-or-missing-tensor-shape-validation.mdc b/.cursor/rules/mod-005-invalid-or-missing-tensor-shape-validation.mdc new file mode 100644 index 0000000000..a679efcec1 --- /dev/null +++ b/.cursor/rules/mod-005-invalid-or-missing-tensor-shape-validation.mdc @@ -0,0 +1,86 @@ +--- +description: All forward and public methods must validate tensor shapes at the beginning, wrapped in torch.compiler.is_compiling() guard, with standardized error messages. +alwaysApply: false +--- + +When implementing forward or public methods, rule MOD-005 must be followed. Explicitly reference "Following rule MOD-005, which requires tensor shape validation at the beginning of methods with torch.compiler.is_compiling() guard..." when adding validation code. + +## MOD-005: Invalid or missing tensor shape validation logic + +**Description:** + +All forward methods and other public methods that accept tensor arguments must +validate tensor shapes at the beginning of the method. This rule applies to: +- Individual tensor arguments +- Containers of tensors (lists, tuples, dictionaries) + +For containers, validate their length, required keys, and the shapes of +contained tensors. Validation statements should be concise (ideally one check +per argument). Error messages must follow the standardized format: +`"Expected tensor of shape (B, D) but got tensor of shape {actual_shape}"`. + +To avoid interactions with `torch.compile`, all validation must be wrapped in a +conditional check using `torch.compiler.is_compiling()`. Follow the "fail-fast" +approach by validating inputs before any computation. + +**Rationale:** + +Early shape validation catches errors at the API boundary with clear, actionable +error messages, making debugging significantly easier. Without validation, shape +mismatches result in cryptic errors deep in the computation graph. The +`torch.compile` guard ensures that validation overhead is eliminated in +production compiled code while preserving debug-time safety. + +**Example:** + +```python +def forward(self, x: torch.Tensor, mask: Optional[torch.Tensor] = None) -> torch.Tensor: + """Forward pass with shape validation.""" + ### Input validation + # Skip validation when running under torch.compile for performance + if not torch.compiler.is_compiling(): + # Extract expected dimensions + B, C, H, W = x.shape if x.ndim == 4 else (None, None, None, None) + + # Validate x shape + if x.ndim != 4: + raise ValueError( + f"Expected 4D input tensor (B, C, H, W), got {x.ndim}D tensor with shape {tuple(x.shape)}" + ) + + if C != self.in_channels: + raise ValueError( + f"Expected {self.in_channels} input channels, got {C} channels" + ) + + # Validate optional mask + if mask is not None: + if mask.shape != (B, H, W): + raise ValueError( + f"Expected mask shape ({B}, {H}, {W}), got {tuple(mask.shape)}" + ) + + # Actual computation happens after validation + return self._process(x, mask) +``` + +**Anti-pattern:** + +```python +# WRONG: No validation at all +def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.layer(x) # Will fail with cryptic error if shape is wrong + +# WRONG: Validation not guarded by torch.compiler.is_compiling() +def forward(self, x: torch.Tensor) -> torch.Tensor: + if x.ndim != 4: # Breaks torch.compile + raise ValueError(f"Expected 4D tensor, got {x.ndim}D") + return self.layer(x) + +# WRONG: Validation after computation has started +def forward(self, x: torch.Tensor) -> torch.Tensor: + h = self.layer1(x) # Computation started + if x.shape[1] != self.in_channels: # Too late! + raise ValueError(f"Wrong number of channels") + return self.layer2(h) +``` diff --git a/.cursor/rules/mod-006-invalid-or-missing-jaxtyping-tensor-annotations.mdc b/.cursor/rules/mod-006-invalid-or-missing-jaxtyping-tensor-annotations.mdc new file mode 100644 index 0000000000..a5a233db77 --- /dev/null +++ b/.cursor/rules/mod-006-invalid-or-missing-jaxtyping-tensor-annotations.mdc @@ -0,0 +1,66 @@ +--- +description: All tensor arguments in model methods must have jaxtyping type annotations with shape specifications (e.g. Float[torch.Tensor, "b c h w"]) for runtime-checkable shape information. +alwaysApply: false +--- + +When adding type hints to model methods, rule MOD-006 must be followed. Explicitly reference "Following rule MOD-006, which requires all tensor arguments to use jaxtyping annotations..." when adding type hints. + +## MOD-006: Invalid or missing jaxtyping tensor annotations in public function signature + +**Description:** + +All tensor arguments and variables in model `__init__`, `forward`, and other +public methods must have type annotations using `jaxtyping`. This provides +runtime-checkable shape information in type hints. + +Use the format `Float[torch.Tensor, "shape_spec"]` where shape_spec describes +tensor dimensions using space-separated dimension names (e.g., `"batch channels height width"` +or `"b c h w"`). + +**Rationale:** + +Jaxtyping annotations provide explicit, machine-readable documentation of +expected tensor shapes. This enables better IDE support, catches shape errors +earlier, and makes code more self-documenting. The annotations serve as both +documentation and optional runtime checks when jaxtyping's validation is +enabled. + +**Example:** + +```python +from jaxtyping import Float +import torch + +class MyConvNet(Module): + def __init__(self, in_channels: int, out_channels: int): + super().__init__() + self.conv = torch.nn.Conv2d(in_channels, out_channels, kernel_size=3) + + def forward( + self, + x: Float[torch.Tensor, "batch in_channels height width"] + ) -> Float[torch.Tensor, "batch out_channels height width"]: + """Process input with convolution.""" + return self.conv(x) +``` + +**Anti-pattern:** + +```python +# WRONG: No jaxtyping annotations +def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.layer(x) + +# WRONG: Using plain comments instead of jaxtyping +def forward(self, x: torch.Tensor) -> torch.Tensor: + # x: (batch, channels, height, width) # Use jaxtyping instead + return self.layer(x) + +# WRONG: Incomplete annotations +def forward( + self, + x: Float[torch.Tensor, "b c h w"], + mask: torch.Tensor # Missing jaxtyping annotation +) -> Float[torch.Tensor, "b c h w"]: + return self.layer(x, mask) +``` diff --git a/.cursor/rules/mod-007a-cannot-add-required-parameters-without-defaults.mdc b/.cursor/rules/mod-007a-cannot-add-required-parameters-without-defaults.mdc new file mode 100644 index 0000000000..21e0cd8789 --- /dev/null +++ b/.cursor/rules/mod-007a-cannot-add-required-parameters-without-defaults.mdc @@ -0,0 +1,58 @@ +--- +description: Cannot add new required parameters to production model __init__ or public methods without default values, as this breaks backward compatibility with existing code and checkpoints. +alwaysApply: false +--- + +When adding parameters to production models, rule MOD-007a must be strictly followed. Explicitly reference "Following rule MOD-007a, which forbids adding required parameters without defaults to maintain backward compatibility..." when modifying signatures. + +## MOD-007a: Cannot add required parameters without defaults + +**Description:** + +For any model in `physicsnemo/nn` or `physicsnemo/models`, adding new required +parameters (parameters without default values) to `__init__` or any public +method is strictly forbidden. This breaks backward compatibility. + +New parameters must have default values to ensure existing code and checkpoints +continue to work. If a new parameter is truly required, increment the model +version number using `__model_checkpoint_version__` and add appropriate +versioning support. + +**Rationale:** + +Adding required parameters breaks all existing code that instantiates the model, +and breaks loading of old checkpoints. This violates PhysicsNeMo's commitment to +backward compatibility and would disrupt user workflows. + +**Example:** + +```python +# Good: Adding parameter with default value +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + + def __init__( + self, + input_dim: int, + output_dim: int, + dropout: float = 0.0, # New parameter with default - backward compatible + new_feature: bool = False # New parameter with default - backward compatible + ): + super().__init__(meta=MyModelMetaData()) +``` + +**Anti-pattern:** + +```python +# WRONG: Adding required parameter without default +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + + def __init__( + self, + input_dim: int, + output_dim: int, + new_param: int # WRONG: No default! Breaks old checkpoints + ): + super().__init__(meta=MyModelMetaData()) +``` diff --git a/.cursor/rules/mod-007b-cannot-remove-or-rename-parameters-without-compat-mapper.mdc b/.cursor/rules/mod-007b-cannot-remove-or-rename-parameters-without-compat-mapper.mdc new file mode 100644 index 0000000000..9862e0e6ee --- /dev/null +++ b/.cursor/rules/mod-007b-cannot-remove-or-rename-parameters-without-compat-mapper.mdc @@ -0,0 +1,86 @@ +--- +description: Cannot remove or rename parameters in production models without implementing _backward_compat_arg_mapper and incrementing __model_checkpoint_version__ to maintain compatibility. +alwaysApply: false +--- + +When removing or renaming parameters in production models, rule MOD-007b must be strictly followed. Explicitly reference "Following rule MOD-007b, which requires _backward_compat_arg_mapper for parameter changes..." when modifying model signatures. + +## MOD-007b: Cannot remove or rename parameters without compat mapper + +**Description:** + +For any model in `physicsnemo/nn` or `physicsnemo/models`, removing or renaming +parameters is strictly forbidden without proper backward compatibility support. + +If a parameter must be renamed or removed, the developer must: +1. Increment `__model_checkpoint_version__` +2. Add the old version to `__supported_model_checkpoint_version__` dict +3. Implement `_backward_compat_arg_mapper` classmethod to handle the mapping +4. Maintain support for the old API for at least 2 release cycles + +**Rationale:** + +Removing or renaming parameters breaks existing checkpoints and user code. +Proper version management and argument mapping ensures old checkpoints can still +be loaded and users have time to migrate to the new API. + +**Example:** + +```python +# Good: Proper backward compatibility for parameter rename +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + __supported_model_checkpoint_version__ = { + "1.0": ( + "Loading checkpoint from version 1.0 (current is 2.0). " + "Parameter 'hidden_dim' renamed to 'hidden_size'." + ) + } + + @classmethod + def _backward_compat_arg_mapper( + cls, version: str, args: Dict[str, Any] + ) -> Dict[str, Any]: + """Map arguments from older versions.""" + args = super()._backward_compat_arg_mapper(version, args) + + if version == "1.0": + # Map old parameter name to new name + if "hidden_dim" in args: + args["hidden_size"] = args.pop("hidden_dim") + + # Remove deprecated parameters + if "legacy_param" in args: + _ = args.pop("legacy_param") + + return args + + def __init__( + self, + input_dim: int, + hidden_size: int = 128, # Renamed from 'hidden_dim' + ): + super().__init__(meta=MyModelMetaData()) +``` + +**Anti-pattern:** + +```python +# WRONG: Renaming without backward compat +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + # Missing: __supported_model_checkpoint_version__ and _backward_compat_arg_mapper + + def __init__(self, input_dim: int, hidden_size: int): # Renamed! + super().__init__(meta=MyModelMetaData()) + # WRONG: Old checkpoints with 'hidden_dim' will fail! + +# WRONG: Not calling super() in mapper +class MyModel(Module): + @classmethod + def _backward_compat_arg_mapper(cls, version: str, args: Dict[str, Any]) -> Dict[str, Any]: + # WRONG: Missing super()._backward_compat_arg_mapper(version, args) + if "hidden_dim" in args: + args["hidden_size"] = args.pop("hidden_dim") + return args +``` diff --git a/.cursor/rules/mod-007c-cannot-change-return-types-of-public-methods.mdc b/.cursor/rules/mod-007c-cannot-change-return-types-of-public-methods.mdc new file mode 100644 index 0000000000..dd347c0ab6 --- /dev/null +++ b/.cursor/rules/mod-007c-cannot-change-return-types-of-public-methods.mdc @@ -0,0 +1,64 @@ +--- +description: Cannot change return types of public methods in production models, as this breaks user code that depends on the existing return type structure. +alwaysApply: false +--- + +When modifying public method return types, rule MOD-007c must be strictly followed. Explicitly reference "Following rule MOD-007c, which forbids changing return types of public methods..." when considering API changes. + +## MOD-007c: Cannot change return types of public methods + +**Description:** + +For any model in `physicsnemo/nn` or `physicsnemo/models`, changing the return +type of any public method (including `forward`) is strictly forbidden. This +includes: +- Changing from returning a single value to returning a tuple +- Changing from a tuple to a single value +- Changing the number of elements in a returned tuple +- Changing the type of returned values + +If a return type change is absolutely necessary, create a new method with a +different name and deprecate the old method following the deprecation lifecycle +(MOD-002b). + +**Rationale:** + +Changing return types is a breaking change that silently breaks user code. Users +who unpack return values or depend on specific return structures will experience +runtime errors. Unlike parameter changes (which can be managed with versioning), +return type changes affect runtime behavior and are harder to detect. + +**Example:** + +```python +# Good: Keeping consistent return type +class MyModel(Module): + def forward(self, x: torch.Tensor) -> torch.Tensor: + """Always returns single tensor.""" + return self.process(x) + +# Good: If new return is needed, add new method +class MyModel(Module): + def forward(self, x: torch.Tensor) -> torch.Tensor: + """Returns output tensor.""" + output, loss = self._forward_with_loss(x) + return output + + def forward_with_loss(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + """New method for returning both output and loss.""" + return self._forward_with_loss(x) +``` + +**Anti-pattern:** + +```python +# WRONG: Changing return type +class MyModel(Module): + # v1.0 + def forward(self, x: torch.Tensor) -> torch.Tensor: + return output + + # v2.0 - BREAKS USER CODE! + def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + return output, loss # Users expecting single tensor will break! +``` diff --git a/.cursor/rules/mod-008a-model-missing-constructor-attributes-tests.mdc b/.cursor/rules/mod-008a-model-missing-constructor-attributes-tests.mdc new file mode 100644 index 0000000000..b188c133ae --- /dev/null +++ b/.cursor/rules/mod-008a-model-missing-constructor-attributes-tests.mdc @@ -0,0 +1,69 @@ +--- +description: Every model must have CI tests verifying constructor instantiation and all public attributes (excluding buffers/parameters) using pytest parameterization. +alwaysApply: false +--- + +When creating tests for models, rule MOD-008a must be followed. Explicitly reference "Following rule MOD-008a, which requires constructor and attribute tests..." when implementing test cases. + +## MOD-008a: Model missing constructor/attributes tests + +**Description:** + +Every model in `physicsnemo/nn` or `physicsnemo/models` must have tests that +verify model instantiation and all public attributes (excluding buffers and +parameters). + +These tests should: +- Use `pytest` parameterization to test at least 2 configurations +- Test one configuration with all default arguments +- Test another configuration with non-default arguments +- Verify all public attributes have expected values + +**Rationale:** + +Constructor tests ensure the model can be instantiated correctly with various +configurations and that all attributes are properly initialized. This catches +issues early in the development cycle. + +**Example:** + +```python +@pytest.mark.parametrize( + "config", + ["default", "custom"], + ids=["with_defaults", "with_custom_args"] +) +def test_my_model_constructor(config): + """Test model constructor and attributes.""" + if config == "default": + model = MyModel(input_dim=64, output_dim=32) + assert model.hidden_dim == 128 # Default value + assert model.dropout == 0.0 # Default value + else: + model = MyModel( + input_dim=64, + output_dim=32, + hidden_dim=256, + dropout=0.1 + ) + assert model.hidden_dim == 256 + assert model.dropout == 0.1 + + # Test common attributes + assert model.input_dim == 64 + assert model.output_dim == 32 +``` + +**Anti-pattern:** + +```python +# WRONG: Only testing default configuration +def test_my_model_bad(): + model = MyModel(input_dim=64, output_dim=32) + # Only tests defaults, not custom configurations + +# WRONG: Not verifying attributes +def test_my_model_bad(): + model = MyModel(input_dim=64, output_dim=32) + # Model created but attributes not tested +``` diff --git a/.cursor/rules/mod-008b-model-missing-non-regression-test-with-reference-data.mdc b/.cursor/rules/mod-008b-model-missing-non-regression-test-with-reference-data.mdc new file mode 100644 index 0000000000..cf6d86f8ce --- /dev/null +++ b/.cursor/rules/mod-008b-model-missing-non-regression-test-with-reference-data.mdc @@ -0,0 +1,81 @@ +--- +description: Every model must have non-regression tests comparing outputs against reference data saved in .pth files, using realistic tensor shapes and pytest parameterization. +alwaysApply: false +--- + +When creating tests for models, rule MOD-008b must be followed. Explicitly reference "Following rule MOD-008b, which requires non-regression tests with reference data..." when implementing test cases. + +## MOD-008b: Model missing non-regression test with reference data + +**Description:** + +Every model must have non-regression tests that: +1. Instantiate the model with reproducible random parameters +2. Run forward pass with test data +3. Compare outputs against reference data saved in a `.pth` file + +Requirements: +- Use `pytest` parameterization to test multiple configurations +- Test tensors must have realistic shapes (no singleton dimensions except batch) +- Test data should be meaningful and representative of actual use cases +- Compare actual tensor values, not just shapes +- All public methods (not just forward) need similar non-regression tests + +**Critical:** Per MOD-002a, models cannot move out of experimental without these +tests. + +**Rationale:** + +Non-regression tests with reference data catch subtle numerical changes that +could break reproducibility. Simply checking output shapes is insufficient to +detect algorithmic changes or numerical instabilities. Comparing against +saved reference values ensures the model produces consistent results across code +changes. + +**Example:** + +```python +@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +@pytest.mark.parametrize("config", ["default", "custom"]) +def test_my_model_non_regression(device, config): + """Test model forward pass against reference output.""" + if config == "default": + model = _instantiate_model(MyModel, input_dim=64, output_dim=32) + else: + model = _instantiate_model( + MyModel, + input_dim=64, + output_dim=32, + hidden_dim=256 + ) + + model = model.to(device) + + # Load reference data (meaningful shapes, no singletons) + data = torch.load(f"test/models/data/my_model_{config}_v1.0.pth") + x = data["x"].to(device) # Shape: (4, 64), not (1, 64) + out_ref = data["out"].to(device) + + # Run forward and compare values + out = model(x) + assert torch.allclose(out, out_ref, atol=1e-5, rtol=1e-5) +``` + +**Anti-pattern:** + +```python +# WRONG: Only testing output shapes +def test_my_model_bad(device): + model = MyModel(input_dim=64, output_dim=32).to(device) + x = torch.randn(4, 64).to(device) + out = model(x) + assert out.shape == (4, 32) # NOT SUFFICIENT! + +# WRONG: Using singleton dimensions +def test_my_model_bad(device): + x = torch.randn(1, 1, 64) # WRONG: Trivial shapes + +# WRONG: No parameterization +def test_my_model_bad(): + model = MyModel(input_dim=64, output_dim=32) # Only tests defaults +``` diff --git a/.cursor/rules/mod-008c-model-missing-checkpoint-loading-test.mdc b/.cursor/rules/mod-008c-model-missing-checkpoint-loading-test.mdc new file mode 100644 index 0000000000..39cde7809c --- /dev/null +++ b/.cursor/rules/mod-008c-model-missing-checkpoint-loading-test.mdc @@ -0,0 +1,62 @@ +--- +description: Every model must have tests that load from checkpoint files (.mdlus), verify attributes, and compare outputs against reference data to ensure serialization works correctly. +alwaysApply: false +--- + +When creating tests for models, rule MOD-008c must be followed. Explicitly reference "Following rule MOD-008c, which requires checkpoint loading tests..." when implementing test cases. + +## MOD-008c: Model missing checkpoint loading test + +**Description:** + +Every model must have tests that load the model from a checkpoint file +(`.mdlus`) using `physicsnemo.Module.from_checkpoint()` and verify that: +1. The model loads successfully +2. All public attributes have expected values +3. Forward pass outputs match reference data + +This ensures the model's serialization and deserialization work correctly. + +**Critical:** Per MOD-002a, models cannot move out of experimental without these +tests. + +**Rationale:** + +Checkpoint tests verify that the model's custom serialization logic works +correctly and that saved models can be loaded in different environments. This is +critical for reproducibility and for users who need to save and load trained +models. These tests also validate the backward compatibility system. + +**Example:** + +```python +@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +def test_my_model_from_checkpoint(device): + """Test loading model from checkpoint and verify outputs.""" + model = physicsnemo.Module.from_checkpoint( + "test/models/data/my_model_default_v1.0.mdlus" + ).to(device) + + # Verify attributes after loading + assert model.input_dim == 64 + assert model.output_dim == 32 + + # Load reference data and verify outputs + data = torch.load("test/models/data/my_model_default_v1.0.pth") + x = data["x"].to(device) + out_ref = data["out"].to(device) + out = model(x) + assert torch.allclose(out, out_ref, atol=1e-5, rtol=1e-5) +``` + +**Anti-pattern:** + +```python +# WRONG: No checkpoint loading test +# (Missing test_my_model_from_checkpoint entirely) + +# WRONG: Only loading checkpoint without verifying outputs +def test_my_model_bad(): + model = physicsnemo.Module.from_checkpoint("checkpoint.mdlus") + # Should verify attributes and outputs! +``` diff --git a/.cursor/rules/mod-009-avoid-string-based-class-selection.mdc b/.cursor/rules/mod-009-avoid-string-based-class-selection.mdc new file mode 100644 index 0000000000..ce6cc658aa --- /dev/null +++ b/.cursor/rules/mod-009-avoid-string-based-class-selection.mdc @@ -0,0 +1,75 @@ +--- +description: Avoid string-based class selection with many options (>3 choices) in model constructors; prefer dependency injection with instances for better type safety and clearer APIs. +alwaysApply: false +--- + +When designing model constructor APIs, rule MOD-009 should be followed. Explicitly reference "Following rule MOD-009, which discourages string-based class selection when there are many choices..." when deciding constructor parameter design. + +## MOD-009: Avoid string-based class selection in model constructors + +**Description:** + +Passing a string that represents a class name, which is then used to instantiate +an internal submodule, should be avoided unless there are only a few choices (2 +or 3 maximum) for the class name. + +When there are more than 2-3 choices, the recommended practice is to pass an +already instantiated instance of a submodule instead of a string primitive for +dependency injection. This promotes better type safety, clearer APIs, and easier +testing. + +**Rationale:** + +String-based class selection makes code harder to type-check, debug, and test. +It obscures dependencies and makes it difficult for static analysis tools to +understand the code. Direct instance injection provides better IDE support, +type safety, and makes testing easier by allowing mock object injection. + +**Example:** + +```python +# Good: Limited choices (2-3 max) - string selection acceptable +class MyModel(Module): + def __init__( + self, + activation: Literal["relu", "gelu"] = "relu" + ): + if activation == "relu": + self.act = nn.ReLU() + elif activation == "gelu": + self.act = nn.GELU() + +# Good: Many choices - use instance injection +class MyModel(Module): + def __init__( + self, + encoder: Module, # Pass instance, not string + decoder: Module # Pass instance, not string + ): + self.encoder = encoder + self.decoder = decoder + +# Usage: +model = MyModel( + encoder=MyCustomEncoder(dim=128), + decoder=MyCustomDecoder(dim=128) +) +``` + +**Anti-pattern:** + +```python +# WRONG: String selection with many choices +class MyModel(Module): + def __init__( + self, + encoder_type: str = "transformer" # Many possible values + ): + # String-based factory pattern with 10+ choices + if encoder_type == "transformer": + self.encoder = TransformerEncoder() + elif encoder_type == "cnn": + self.encoder = CNNEncoder() + # ... many more options + # WRONG: Should accept encoder instance instead +``` diff --git a/.cursor/rules/mod-010-avoid-splatted-kwargs-in-constructors.mdc b/.cursor/rules/mod-010-avoid-splatted-kwargs-in-constructors.mdc new file mode 100644 index 0000000000..94134b2341 --- /dev/null +++ b/.cursor/rules/mod-010-avoid-splatted-kwargs-in-constructors.mdc @@ -0,0 +1,66 @@ +--- +description: Avoid splatted kwargs (**kwargs) in model constructors; use explicit Dict parameters instead to prevent naming conflicts and make APIs clearer. +alwaysApply: false +--- + +When designing model constructor APIs, rule MOD-010 should be followed. Explicitly reference "Following rule MOD-010, which recommends explicit Dict parameters instead of splatted kwargs..." when deciding constructor parameter design. + +## MOD-010: Avoid splatted kwargs in model constructors + +**Description:** + +Passing splatted arguments like `**kwargs_for_submodules` should be avoided in +model constructors as it might create conflicts in the names of these kwargs and +makes the API unclear. + +Instead, it is recommended to pass non-splatted arguments in the form of a +`Dict` when configuration for submodules needs to be passed through. This makes +parameter passing explicit and avoids naming conflicts. + +**Rationale:** + +Splatted kwargs obscure the actual parameters being passed, make type checking +impossible, and can lead to subtle bugs from name conflicts. Explicit dictionary +parameters make the API clearer and enable better IDE support and error +detection. + +**Example:** + +```python +# Good: Explicit dict parameter +class MyModel(Module): + def __init__( + self, + input_dim: int, + output_dim: int, + encoder_config: Optional[Dict[str, Any]] = None + ): + encoder_config = encoder_config or {} + self.encoder = Encoder(input_dim=input_dim, **encoder_config) + +# Usage: +model = MyModel( + input_dim=64, + output_dim=32, + encoder_config={"hidden_dim": 128, "num_layers": 3} +) +``` + +**Anti-pattern:** + +```python +# WRONG: Splatted kwargs +class MyModel(Module): + def __init__( + self, + input_dim: int, + output_dim: int, + **encoder_kwargs # WRONG: Unclear what's accepted + ): + self.encoder = Encoder(input_dim=input_dim, **encoder_kwargs) + # Risk of name conflicts, unclear API + +# Usage - unclear what parameters are valid: +model = MyModel(input_dim=64, output_dim=32, hidden_dim=128, num_layers=3) +# Are hidden_dim and num_layers for MyModel or Encoder? Unclear! +``` diff --git a/.cursor/rules/mod-011-use-proper-optional-dependency-handling.mdc b/.cursor/rules/mod-011-use-proper-optional-dependency-handling.mdc new file mode 100644 index 0000000000..4b20ecec8c --- /dev/null +++ b/.cursor/rules/mod-011-use-proper-optional-dependency-handling.mdc @@ -0,0 +1,100 @@ +--- +description: Use check_min_version() to check optional dependencies without importing, and @require_version decorator to protect version-specific features; pyproject.toml is the single source of truth for dependencies. +alwaysApply: false +--- + +When handling optional dependencies in model code, rule MOD-011 must be followed. Explicitly reference "Following rule MOD-011, which requires using check_min_version() for optional dependencies..." when implementing dependency checks. + +## MOD-011: Use proper optional dependency handling + +**Description:** + +When a model requires optional dependencies (packages not installed by default), +use the PhysicsNeMo APIs for dependency handling: + +1. **`check_min_version(package, version, hard_fail=False)`**: Use this function + to check if a package is installed and available without actually importing + it. Set `hard_fail=True` for hard requirements, `hard_fail=False` for soft + requirements. This is the primary method for handling optional dependencies. + +2. **`@require_version(package, version)`**: Use this decorator when core code + must always be available but certain features need to be protected against + older versions. This is rare and should only be used when you need to protect + specific methods or classes. + +3. **`pyproject.toml`**: This file is the one, only, and universal source of + truth for all dependencies in PhysicsNeMo. All optional dependencies must be + declared there. + +**Rationale:** + +Centralized dependency handling ensures consistent error messages and version +checking across the codebase. Checking availability without importing prevents +import errors and allows graceful degradation. Using `pyproject.toml` as the +single source of truth prevents dependency specification from becoming scattered +and inconsistent. + +**Example:** + +```python +import torch +from physicsnemo.core import Module +from physicsnemo.core.version_check import check_min_version, require_version + +# Check optional dependency availability without importing +APEX_AVAILABLE = check_min_version("apex", "0.1.0", hard_fail=False) + +class MyModel(Module): + def __init__( + self, + input_dim: int, + use_apex: bool = False + ): + super().__init__() + self.use_apex = use_apex + + if use_apex and not APEX_AVAILABLE: + raise RuntimeError( + "apex is required for use_apex=True but is not installed. " + "Install with: pip install apex>=0.1.0" + ) + + if use_apex: + import apex # Only import when actually needed + self.fused_layer = apex.FusedLayer() + else: + self.fused_layer = None + +# Using @require_version for protecting version-specific features +class AdvancedModel(Module): + @require_version("torch", "2.4.0") + def use_device_mesh(self): + """This feature requires torch>=2.4.0.""" + from torch.distributed.device_mesh import DeviceMesh + # Protected code +``` + +**Anti-pattern:** + +```python +# WRONG: Direct import without checking availability +import apex # Will fail if apex not installed! + +class MyModel(Module): + def __init__(self, use_apex: bool = False): + if use_apex: + self.layer = apex.FusedLayer() # Already failed at import! + +# WRONG: Try/except for dependency checking +try: + import apex + APEX_AVAILABLE = True +except ImportError: + APEX_AVAILABLE = False +# Use check_min_version instead! + +# WRONG: Hardcoded version strings in multiple places +if version.parse(apex.__version__) < version.parse("0.1.0"): + raise ImportError("apex>=0.1.0 required") +# Should use check_min_version or require_version! +``` diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 0000000000..ef9a613668 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,8 @@ +.git +.github +.gitlab +.coverage* +.*cache +examples +docs +test \ No newline at end of file diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS new file mode 100644 index 0000000000..63a1ddf000 --- /dev/null +++ b/.github/CODEOWNERS @@ -0,0 +1,160 @@ +# PHYSICSNEMO CODEOWNERS + +# The owners file is organized as follows: +# First, customer-facing locations with high priority (docs, readme.md) + +# Next, core utilities (not attached to specific models) + +# Next, most active models (most likely to get a PR) +# Model-specific utilities are included here, and relevant examples + +# Currently, the code owners is using specific people: however, we could +# and probably should make teams in the NVIDIA org, assign people to those +# teams, and use org teams instead. Let's do that in an update. + + +# Top level markdown files: +# Keep CHANGELOG separate - don't require specific people to approve updates to it. +./CHANGELOG.md +./*.md @ram-cherukuri @megnvidia + +.github/workflows @ktangsali @coreyjadams @nickgeneva +.github/workflows/merge-queue-blossom-passthrough.yml @coreyjadams + +# All changes to documentation, except images: +docs/ @megnvidia @ktangsali +docs/img/ + + +# Core release files +Dockerfile @ktangsali +pyproject.toml @ktangsali + +# CORE UTILITIES not attached to specific models: +physicsnemo/utils/profiling @coreyjadams +physicsnemo/utils/neighbors @coreyjadams @peterdsharpe +physicsnemo/utils/version_check.py @coreyjadams +physicsnemo/utils/capture.py @nickgeneva @ktangsali + +physicsnemo/registry/ @loliverhennigh + +physicsnemo/models/module.py @CharlelieLrt + +# METRICS that are general: +physicsnemo/metrics/cae @RishikeshRanade @mnabian @peterdsharpe +physicsnemo/metrics/general @dallasfoster @nickgeneva +physicsnemo/metrics/climate @dallasfoster @nickgeneva + +# Active learning components +physicsnemo/active_learning @laserkelvin @dallasfoster + +# LAUNCH and DEPLOY TOOLS +# NEEDSOWNER +physicsnemo/launch +physicsnemo/deploy + +# Mesh +physicsnemo/mesh @peterdsharpe + +# DATAPIPES +physicsnemo/datapipes/core @coreyjadams +physicsnemo/datapipes/climate @dallasfoster @nickgeneva @pzharrington +physicsnemo/datapipes/cae/domino_datapipe.py @RishikeshRanade @coreyjadams +physicsnemo/datapipes/cae/mesh_datapipe.py @mnabian +physicsnemo/datapipes/cae/readers.py @mnabian + +physicsnemo/datapipes/gnn @mnabian +physicsnemo/datapipes/healpix @pzharrington +examples/minimal/datapipes @coreyjadams +test/datapipes/core @coreyjadams + +# Distributed tools +physicsnemo/distributed @coreyjadams + +# MODEL SPECIFIC OWNERSHIP. These are in roughly the same order as the code repo. +# They are grouped "logically" together, though. +# For example, gnn_layers, graphcast, and meshgraphnet are together. + +# AFNO +# NEEDSOWNER + +# CorrDiff and Diffusion +physicsnemo/models/diffusion @CharlelieLrt +physicsnemo/utils/corrdiff @CharlelieLrt +physicsnemo/utils/diffusion @CharlelieLrt +physicsnemo/utils/generative @CharlelieLrt +physicsnemo/utils/patching.py @CharlelieLrt +physicsnemo/metrics/diffusion @CharlelieLrt + + +# DLWP +# NEEDSOWNER +physicsnemo/models/dlwp @pzharrington +physicsnemo/models/dlwp_healpix @pzharrington +physicsnemo/models/dlwp_healpix_layers @pzharrington +physicsnemo/utils/insolation.py @pzharrington + +# DoMINO +physicsnemo/models/domino @RishikeshRanade +physicsnemo/utils/domino @RishikeshRanade +physicsnemo/utils/sdf.py @RishikeshRanade @peterdsharpe +examples/cfd/external_aerodynamics/domino @RishikeshRanade + + +# fengwu, pangu, swinrnn +# NEEDSOWNER +physicsnemo/models/fengu @dallasfoster +physicsnemo/models/pangu @dallasfoster +physicsnemo/models/swinvrnn @dallasfoster + +# figconvnet +physicsnemo/models/figconvnet @coreyjadams + +# FNO +# NEEDSOWNER + +# GNN layers, mesh graphnet, graphcast +physicsnemo/models/gnn_layers @mnabian +physicsnemo/models/meshgraphnet @mnabian +physicsnemo/models/graphcast @mnabian +physicsnemo/models/mesh_reduced @mnabian +physicsnemo/utils/mesh @mnabian + +# pix2pix +# NEEDSOWNER +physicsnemo/models/pix2pix + +# rnn, srnn +physicsnemo/models/rnn @ktangsali +physicsnemo/models/srnn @ktangsali + +# topodiff +# NEEDSOWNER +physicsnemo/models/topodiff + +# transolver +physicsnemo/models/transolver @coreyjadams + +# unet +physicsnemo/models/unet @mnabian @peterdsharpe + +# vfgn +physicsnemo/models/vfgn @mnabian + +# EXAMPLE SPECIFIC OWNERSHIP + +# corrdiff +examples/weather/corrdiff @CharlelieLrt +examples/active_learning @laserkelvin @dallasfoster + +# TEST SPECIFIC OWNERSHIP + +# CorrDiff and diffusion +test/models/diffusion @CharlelieLrt +test/metrics/diffusion @CharlelieLrt +test/utils/generative @CharlelieLrt +test/utils/corrdiff @CharlelieLrt +test/active_learning @laserkelvin @dallasfoster + +# Mesh +test/mesh @peterdsharpe diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index ca39893dd7..43f68ecbf3 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -14,7 +14,7 @@ # limitations under the License. name: Bug Report -description: File a bug report for Modulus (Core) +description: File a bug report for PhysicsNeMo (Core) title: "🐛[BUG]: " labels: ["bug", "? - Needs Triage"] @@ -22,16 +22,16 @@ body: - type: markdown attributes: value: | - Thanks for taking the time to help Modulus and fill out this bug report! - - By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/NVIDIA/modulus/blob/main/CONTRIBUTING.md) - - You also confirm that you have searched the [open bugs](https://github.com/NVIDIA/modulus/issues) and have found no duplicates for this request + Thanks for taking the time to help PhysicsNeMo and fill out this bug report! + - By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/NVIDIA/physicsnemo/blob/main/CONTRIBUTING.md) + - You also confirm that you have searched the [open bugs](https://github.com/NVIDIA/physicsnemo/issues) and have found no duplicates for this request - type: input id: version attributes: label: Version - description: What version of Modulus are you running? - placeholder: "example: 0.4.0" + description: What version of PhysicsNeMo are you running? + placeholder: "example: 1.3.0" validations: required: true diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index fbbc16b106..9ca26a0803 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -1,5 +1,5 @@ blank_issues_enabled: true contact_links: - name: Ask a Question - url: https://forums.developer.nvidia.com/c/physics-simulation/modulus-physics-ml-model-framework + url: https://forums.developer.nvidia.com/t/welcome-to-the-physicsnemo-ml-model-framework-forum/178556 about: Thanks for taking the time to ask us a question! \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/documentation_request.yml b/.github/ISSUE_TEMPLATE/documentation_request.yml index 4b2b5a3676..abe0b62943 100644 --- a/.github/ISSUE_TEMPLATE/documentation_request.yml +++ b/.github/ISSUE_TEMPLATE/documentation_request.yml @@ -14,7 +14,7 @@ # limitations under the License. name: Documentation Request -description: Request updates or additions to Modulus (Core) documentation +description: Request updates or additions to PhysicsNeMo (Core) documentation title: "📚[DOC]: " labels: ["documentation", "? - Needs Triage"] @@ -22,9 +22,9 @@ body: - type: markdown attributes: value: | - Thanks for taking the time to help Modulus and improve our documentation! - - By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/NVIDIA/modulus/blob/main/CONTRIBUTING.md) - - You also confirm that you have searched the [open documentation issues](https://github.com/NVIDIA/modulus/issues) and have found no duplicates for this request + Thanks for taking the time to help PhysicsNeMo and improve our documentation! + - By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/NVIDIA/physicsnemo/blob/main/CONTRIBUTING.md) + - You also confirm that you have searched the [open documentation issues](https://github.com/NVIDIA/physicsnemo/issues) and have found no duplicates for this request - type: dropdown id: criticality diff --git a/.github/ISSUE_TEMPLATE/feature_request.yml b/.github/ISSUE_TEMPLATE/feature_request.yml index 5a2094e573..cb6e5cb8cb 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.yml +++ b/.github/ISSUE_TEMPLATE/feature_request.yml @@ -14,7 +14,7 @@ # limitations under the License. name: Feature Request -description: Request new or improved functionality or changes to existing Modulus (Core) functionality +description: Request new or improved functionality or changes to existing PhysicsNeMo (Core) functionality title: "🚀[FEA]: " labels: ["enhancement", "? - Needs Triage"] @@ -22,9 +22,9 @@ body: - type: markdown attributes: value: | - Thanks for taking the time to help Modulus and fill out this feature request! - - By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/NVIDIA/modulus/blob/main/CONTRIBUTING.md) - - You also confirm that you have searched the [open documentation issues](https://github.com/NVIDIA/modulus/issues) and have found no duplicates for this request + Thanks for taking the time to help PhysicsNeMo and fill out this feature request! + - By submitting this issue, you agree to follow our [Code of Conduct](https://github.com/NVIDIA/physicsnemo/blob/main/CONTRIBUTING.md) + - You also confirm that you have searched the [open documentation issues](https://github.com/NVIDIA/physicsnemo/issues) and have found no duplicates for this request - type: dropdown id: new_or_improvement diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index b8f3b8d08d..63b2542f0a 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,5 +1,5 @@ -# Modulus Pull Request +# PhysicsNeMo Pull Request ## Description @@ -8,12 +8,27 @@ ## Checklist -- [ ] I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/modulus/blob/main/CONTRIBUTING.md). +- [ ] I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/physicsnemo/blob/main/CONTRIBUTING.md). - [ ] New or existing tests cover these changes. - [ ] The documentation is up to date with these changes. -- [ ] The [CHANGELOG.md](https://github.com/NVIDIA/modulus/blob/main/CHANGELOG.md) is up to date with these changes. -- [ ] An [issue](https://github.com/NVIDIA/modulus/issues) is linked to this pull request. +- [ ] The [CHANGELOG.md](https://github.com/NVIDIA/physicsnemo/blob/main/CHANGELOG.md) is up to date with these changes. +- [ ] An [issue](https://github.com/NVIDIA/physicsnemo/issues) is linked to this pull request. +- [ ] If I am implementing a new model or modifying any existing model, I have followed the [Models Implementation Coding Standards](https://github.com/NVIDIA/physicsnemo/wiki/Coding-standards-for-models-implementation). ## Dependencies - \ No newline at end of file + + +## Review Process + +**All PRs are reviewed by the PhysicsNeMo team before merging.** + +Depending on which files are changed, GitHub may automatically assign a maintainer for review. + +We are also testing AI-based code review tools (e.g., Greptile), which may add automated comments with a confidence score. +This score reflects the AI’s assessment of merge readiness and is **not** a qualitative judgment of your work, nor is +it an indication that the PR will be accepted / rejected. + +AI-generated feedback should be reviewed critically for usefulness. +You are not required to respond to every AI comment, but they are intended to help both authors and reviewers. +Please react to Greptile comments with 👍 or 👎 to provide feedback on their accuracy. diff --git a/.github/workflows/blossom-ci.yml b/.github/workflows/blossom-ci.yml index 2090bac8d2..4e236b6ac6 100644 --- a/.github/workflows/blossom-ci.yml +++ b/.github/workflows/blossom-ci.yml @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -18,27 +20,52 @@ on: # CI job triggers here issue_comment: types: [created] - + workflow_dispatch: inputs: platform: - description: 'runs-on argument' + description: 'runs-on argument' required: false args: - description: 'argument' + description: 'argument' required: false jobs: Authorization: name: Authorization - runs-on: blossom + runs-on: blossom outputs: args: ${{ env.args }} - + # This job only runs for pull request comments # Comment must be `/blossom-ci` - if: | - contains( 'nickgeneva,ktangsali,dallasfoster,akshaysubr,loliverhennigh,mnabian,stadlmax,daviddpruitt,', format('{0},', github.actor) ) && - github.event.comment.body == '/blossom-ci' + if: > + (github.event.comment.body == '/blossom-ci' || github.event.comment.body == '/multi-gpu-ci') && + ( + github.actor == 'nickgeneva' || + github.actor == 'ktangsali' || + github.actor == 'dallasfoster' || + github.actor == 'akshaysubr' || + github.actor == 'loliverhennigh' || + github.actor == 'mnabian' || + github.actor == 'stadlmax' || + github.actor == 'daviddpruitt' || + github.actor == 'Alexey-Kamenev' || + github.actor == 'jleinonen' || + github.actor == 'MortezaMardani' || + github.actor == 'tge25' || + github.actor == 'nbren12' || + github.actor == 'yairchn' || + github.actor == 'pzharrington' || + github.actor == 'hasethinvd' || + github.actor == 'RishikeshRanade' || + github.actor == 'hakhondzadeh' || + github.actor == 'peterdsharpe' || + github.actor == 'coreyjadams' || + github.actor == 'CharlelieLrt' || + github.actor == 'abokov-nv' || + github.actor == 'saikrishnanc-nv' || + github.actor == 'laserkelvin' + ) steps: - name: Check if comment is issued by authorized person run: blossom-ci @@ -46,11 +73,11 @@ jobs: OPERATION: 'AUTH' REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }} REPO_KEY_DATA: ${{ secrets.BLOSSOM_KEY }} - + Vulnerability-scan: name: Vulnerability scan needs: [Authorization] - runs-on: ubuntu-latest + runs-on: vulnerability-scan steps: - name: Checkout code uses: actions/checkout@v2 @@ -58,20 +85,20 @@ jobs: repository: ${{ fromJson(needs.Authorization.outputs.args).repo }} ref: ${{ fromJson(needs.Authorization.outputs.args).ref }} lfs: 'true' - - # repo specific steps + + # repo specific steps #- name: Setup java # uses: actions/setup-java@v1 # with: # java-version: 1.8 - + # add blackduck properties https://synopsys.atlassian.net/wiki/spaces/INTDOCS/pages/631308372/Methods+for+Configuring+Analysis#Using-a-configuration-file #- name: Setup blackduck properties # run: | # PROJECTS=$(mvn -am dependency:tree | grep maven-dependency-plugin | awk '{ out="com.nvidia:"$(NF-1);print out }' | grep rapids | xargs | sed -e 's/ /,/g') # echo detect.maven.build.command="-pl=$PROJECTS -am" >> application.properties # echo detect.maven.included.scopes=compile >> application.properties - + - name: Run blossom action uses: NVIDIA/blossom-action@main env: @@ -81,7 +108,7 @@ jobs: args1: ${{ fromJson(needs.Authorization.outputs.args).args1 }} args2: ${{ fromJson(needs.Authorization.outputs.args).args2 }} args3: ${{ fromJson(needs.Authorization.outputs.args).args3 }} - + Job-trigger: name: Start ci job needs: [Vulnerability-scan] @@ -93,7 +120,7 @@ jobs: OPERATION: 'START-CI-JOB' CI_SERVER: ${{ secrets.CI_SERVER }} REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }} - + Upload-Log: name: Upload log runs-on: blossom diff --git a/.github/workflows/github-nightly-container.yml b/.github/workflows/github-nightly-container.yml new file mode 100644 index 0000000000..8f0ff383ff --- /dev/null +++ b/.github/workflows/github-nightly-container.yml @@ -0,0 +1,206 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This CI runs nightly to generate the coverage report and testmon database. +# It runs ALL tests and caches the testmon database for use by PR workflows. +# The tests run here will only use UV. This is meant to be nightly functionality +# testing AND a baseline dependency graph for PRs. + + +# TO DO: THE COVERAGE LIMIT IS VERY LOW, BECAUSE THIS IS NOT USING GPU TESTS OR +# THE DATA-DRIVEN TESTS. RAISE THIS UP AGAIN EVENTUALLY. + + +name: Nightly Github Workflow +on: + schedule: + # Run nightly at 2 AM UTC + - cron: '0 2 * * *' + workflow_dispatch: + # Allow manual triggering + +# Container image used across all jobs - update this single value to change everywhere +# Note: env context not available in container.image, so we hardcode the value +jobs: + # Stage 1: Build and cache the environment + build-environment: + name: Build Environment + runs-on: linux-amd64-cpu8 + container: + image: nvcr.io/nvidia/pytorch:25.12-py3 + + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore uv cache + id: cache-uv-restore + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-nightly-latest + + - name: Install package with uv + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + # Install core dependencies and development group + uv sync --group dev --preview-features extra-build-dependencies + + - name: Free disk space before caching + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + rm -rf ~/.cache/uv + df -h + + - name: Delete old environment cache + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + gh cache delete "uv-env-nightly-latest" || true + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Save environment to cache + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: .venv + key: uv-env-nightly-latest + + # Stage 2: Run testmon tests and cache the database + testmon: + name: Testmon + needs: build-environment + runs-on: linux-amd64-gpu-h100-latest-1 + container: + image: nvcr.io/nvidia/pytorch:25.12-py3 + + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore environment from cache + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-nightly-latest + fail-on-cache-miss: true + + - name: Run core tests (collect all for testmon) + run: | + # This populates the testmon database for PR workflows + uv run python -m pytest --testmon --ignore-glob="*docs*" --ignore-glob="*examples*" + + - name: Delete old testmon cache + run: | + gh cache delete "testmon-nightly-latest" || true + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Save testmon database to cache + uses: actions/cache/save@v4 + with: + path: | + .testmondata + .testmondata-shm + .testmondata-wal + key: testmon-nightly-latest + + # Stage 3: Run coverage tests and upload artifacts + coverage: + name: Coverage + needs: build-environment + runs-on: linux-amd64-gpu-h100-latest-1 + container: + image: nvcr.io/nvidia/pytorch:25.12-py3 + + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore environment from cache + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-nightly-latest + fail-on-cache-miss: true + + - name: Run core tests for coverage report + run: | + uv run coverage run --rcfile='test/coverage.pytest.rc' -m pytest --ignore-glob="*docs*" --ignore-glob="*examples*" + + - name: Run doc tests (testmon not supported for doctests) + run: | + uv run coverage run --rcfile='test/coverage.docstring.rc' -m pytest --doctest-modules physicsnemo/ --ignore-glob="*internal*" --ignore-glob="*experimental*" + + - name: Delete old coverage cache + run: | + gh cache delete "coverage-nightly-latest" || true + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Save coverage files to cache + uses: actions/cache/save@v4 + with: + path: .coverage* + key: coverage-nightly-latest + + - name: Merge coverage reports + run: | + uv run coverage combine + uv run coverage report --show-missing --omit="*test*" --omit="*internal*" --omit="*experimental*" --fail-under=45 + uv run coverage html + # Also create an XML report for potential CI integrations + uv run coverage xml -o coverage.xml + + - name: Upload coverage HTML report + uses: actions/upload-artifact@v4 + with: + name: coverage-report-nightly + path: htmlcov/ + retention-days: 7 + + - name: Upload combined coverage data + uses: actions/upload-artifact@v4 + with: + name: coverage-data-nightly + path: | + .coverage + coverage.xml + retention-days: 30 diff --git a/.github/workflows/github-nightly-uv.yml b/.github/workflows/github-nightly-uv.yml new file mode 100644 index 0000000000..391c65aa5e --- /dev/null +++ b/.github/workflows/github-nightly-uv.yml @@ -0,0 +1,213 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This CI runs nightly to generate the coverage report and testmon database. +# It runs ALL tests and caches the testmon database for use by PR workflows. +# The tests run here will only use UV. This is meant to be nightly functionality +# testing AND a baseline dependency graph for PRs. + + +# TO DO: THE COVERAGE LIMIT IS VERY LOW, BECAUSE THIS IS NOT USING GPU TESTS OR +# THE DATA-DRIVEN TESTS. RAISE THIS UP AGAIN EVENTUALLY. + + +name: Nightly Github Workflow +on: + schedule: + # Run nightly at 2 AM UTC + - cron: '0 2 * * *' + workflow_dispatch: + # Allow manual triggering + +jobs: + # Stage 1: Build and cache the environment + build-environment: + name: Build Environment + runs-on: linux-amd64-cpu8 + + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore uv cache + id: cache-uv-restore + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-nightly-latest + + - name: Install package with uv + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + # Install core dependencies and development group + uv sync --group dev --preview-features extra-build-dependencies + + - name: Free disk space before caching + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + rm -rf ~/.cache/uv + df -h + + - name: Delete old environment cache + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + gh cache delete "uv-env-nightly-latest" || true + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Save environment to cache + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: .venv + key: uv-env-nightly-latest + + # Stage 2: Run testmon tests and cache the database + testmon: + name: Testmon + needs: build-environment + runs-on: linux-amd64-gpu-h100-latest-1 + + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore environment from cache + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-nightly-latest + fail-on-cache-miss: true + + - name: Run core tests (collect all for testmon) + run: | + # This populates the testmon database for PR workflows + uv run python -m pytest --testmon --ignore-glob="*docs*" --ignore-glob="*examples*" + + - name: Delete old testmon cache + run: | + gh cache delete "testmon-nightly-latest" || true + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Save testmon database to cache + uses: actions/cache/save@v4 + with: + path: | + .testmondata + .testmondata-shm + .testmondata-wal + key: testmon-nightly-latest + + # Stage 3: Run coverage tests and upload artifacts + coverage: + name: Coverage + needs: build-environment + runs-on: linux-amd64-gpu-h100-latest-1 + + steps: + - uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore environment from cache + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-nightly-latest + fail-on-cache-miss: true + + - name: Run core tests for coverage report + run: | + uv run coverage run --rcfile='test/coverage.pytest.rc' -m pytest --ignore-glob="*docs*" --ignore-glob="*examples*" + + - name: Run doc tests (testmon not supported for doctests) + run: | + uv run coverage run --rcfile='test/coverage.docstring.rc' -m pytest --doctest-modules physicsnemo/ --ignore-glob="*internal*" --ignore-glob="*experimental*" + + - name: Delete old coverage cache + run: | + gh cache delete "coverage-nightly-latest" || true + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Save coverage files to cache + uses: actions/cache/save@v4 + with: + path: .coverage* + key: coverage-nightly-latest + + - name: Merge coverage reports + run: | + uv run coverage combine + uv run coverage report --show-missing --omit="*test*" --omit="*internal*" --omit="*experimental*" --fail-under=45 + uv run coverage html + # Also create an XML report for potential CI integrations + uv run coverage xml -o coverage.xml + + - name: Upload coverage HTML report + uses: actions/upload-artifact@v4 + with: + name: coverage-report-nightly + path: htmlcov/ + retention-days: 7 + + - name: Upload combined coverage data + uses: actions/upload-artifact@v4 + with: + name: coverage-data-nightly + path: | + .coverage + coverage.xml + retention-days: 30 diff --git a/.github/workflows/github-pr.yml b/.github/workflows/github-pr.yml new file mode 100644 index 0000000000..4712b205ad --- /dev/null +++ b/.github/workflows/github-pr.yml @@ -0,0 +1,197 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This CI runs on pull requests and uses testmon to skip tests +# that don't have changed dependencies based on the nightly cache. +# The tests run here will only use UV. + +# TO DO: THE COVERAGE LIMIT IS VERY LOW, BECAUSE THIS IS NOT USING GPU TESTS OR +# THE DATA-DRIVEN TESTS. RAISE THIS UP AGAIN EVENTUALLY. + +name: Pull Request Github CI +on: + push: + branches: + - "pull-request/[0-9]+" + workflow_dispatch: + # Allow manual triggering for debugging the CI. + +# Container image used across all jobs - update this single value to change everywhere +# Note: env context not available in container.image, so we hardcode the value +jobs: + # Stage 1: Build and cache the environment + build-environment: + name: Build Environment + runs-on: linux-amd64-cpu8 + container: + image: nvcr.io/nvidia/pytorch:25.12-py3 + + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore uv cache + id: cache-uv-restore + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-pr-latest + + - name: Install package with uv + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + # Install core dependencies and development group + uv sync --group dev --preview-features extra-build-dependencies + + - name: Free disk space before caching + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + rm -rf ~/.cache/uv + df -h + + - name: Delete old environment cache + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + run: | + gh cache delete "uv-env-pr-latest" || true + env: + GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} + + - name: Save environment to cache + if: steps.cache-uv-restore.outputs.cache-hit != 'true' + uses: actions/cache/save@v4 + with: + path: .venv + key: uv-env-pr-latest + + # Stage 2: Run testmon tests + testmon: + name: Testmon + needs: build-environment + runs-on: linux-amd64-gpu-h100-latest-1 + container: + image: nvcr.io/nvidia/pytorch:25.12-py3 + + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore environment from cache + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-pr-latest + fail-on-cache-miss: true + + - name: Restore testmon database from cache + uses: actions/cache/restore@v4 + with: + path: | + .testmondata + .testmondata-shm + .testmondata-wal + key: testmon-nightly-latest + + - name: Run core tests (with testmon) + run: | + uv run python -m pytest --testmon --ignore-glob="*docs*" --ignore-glob="*examples*" + + # Stage 3: Run coverage tests and upload artifacts + coverage: + name: Coverage + needs: build-environment + runs-on: linux-amd64-gpu-h100-latest-1 + container: + image: nvcr.io/nvidia/pytorch:25.12-py3 + + steps: + - uses: actions/checkout@v4 + + - name: Install uv + uses: nick-fields/retry@v3 + with: + timeout_minutes: 5 + max_attempts: 3 + command: | + curl -LsSf https://astral.sh/uv/install.sh | sh + echo "$HOME/.cargo/bin" >> $GITHUB_PATH + + - name: Restore environment from cache + uses: actions/cache/restore@v4 + with: + path: .venv + key: uv-env-pr-latest + fail-on-cache-miss: true + + - name: Restore testmon database from cache + uses: actions/cache/restore@v4 + with: + path: | + .testmondata + .testmondata-shm + .testmondata-wal + key: testmon-nightly-latest + + - name: Restore nightly coverage baseline from cache + id: cache-coverage-restore + uses: actions/cache/restore@v4 + with: + path: .coverage* + key: coverage-nightly-latest + + - name: Run core tests for coverage report (testmon-selected) + run: | + # Use testmon to only run tests affected by changes, with coverage + uv run coverage run --rcfile='test/coverage.pytest.rc' -m pytest --testmon --ignore-glob="*docs*" --ignore-glob="*examples*" + + - name: Run doc tests (testmon not supported for doctests) + run: | + uv run coverage run --rcfile='test/coverage.docstring.rc' -m pytest --doctest-modules physicsnemo/ --ignore-glob="*internal*" --ignore-glob="*experimental*" + + - name: Merge coverage reports + run: | + # List all coverage files being combined + echo "Coverage files to combine:" + ls -la .coverage* 2>/dev/null || echo "No coverage files found" + + # Combine all .coverage* files + uv run coverage combine + + uv run coverage report --show-missing --omit="*test*" --omit="*internal*" --omit="*experimental*" --fail-under=45 + uv run coverage html + uv run coverage xml -o coverage.xml + + - name: Upload coverage HTML report + uses: actions/upload-artifact@v4 + with: + name: coverage-report-pr + path: htmlcov/ + retention-days: 7 diff --git a/.github/workflows/install-ci.yml b/.github/workflows/install-ci.yml new file mode 100644 index 0000000000..30ef177dd4 --- /dev/null +++ b/.github/workflows/install-ci.yml @@ -0,0 +1,103 @@ +name: Install CI + +on: + push: + branches: [main] + pull_request: + branches: [main, v2.0-refactor] + schedule: + - cron: '0 6 * * *' # 6 AM UTC daily + workflow_dispatch: # Allows manual triggering from GitHub UI + merge_group: + +jobs: + test-pip: + name: Test with pip (Python ${{ matrix.python-version }}, OS ${{ matrix.os }}) + runs-on: ${{ matrix.os }} + # Only run pip tests for scheduled and manual dispatch jobs, Linux only + if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' + strategy: + fail-fast: false + matrix: + os: [ubuntu-latest] + python-version: ["3.13"] + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + + - name: Install package with pip + run: | + python -m pip install --upgrade pip + # pip 25.1+ supports dependency groups natively via --group + pip install -e . --group dev + + - name: Run tests + run: pytest test/ + + test-uv: + name: Test with uv (Python ${{ matrix.python-version }}, OS ${{ matrix.os }}) + runs-on: ${{ matrix.os }} + strategy: + fail-fast: false + matrix: + # Matrix varies by event type: + # PR/merge_group: ubuntu × 3.12 only (1 job) + # push: ubuntu × all Python versions (3 jobs) + # schedule/manual: ubuntu × all versions + ubuntu-arm/mac/win × 3.13 (6 jobs) + include: >- + ${{ + (github.event_name == 'pull_request' || github.event_name == 'merge_group') + && fromJSON('[ + {"os": "ubuntu-latest", "python-version": "3.12"} + ]') + || github.event_name == 'push' + && fromJSON('[ + {"os": "ubuntu-latest", "python-version": "3.11"}, + {"os": "ubuntu-latest", "python-version": "3.12"}, + {"os": "ubuntu-latest", "python-version": "3.13"} + ]') + || fromJSON('[ + {"os": "ubuntu-latest", "python-version": "3.11"}, + {"os": "ubuntu-latest", "python-version": "3.12"}, + {"os": "ubuntu-latest", "python-version": "3.13"}, + {"os": "ubuntu-24.04-arm", "python-version": "3.13"}, + {"os": "macos-latest", "python-version": "3.13"}, + {"os": "windows-latest", "python-version": "3.13"} + ]') + }} + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + + # Set up MSVC compiler on Windows (does nothing on Linux/macOS) + - uses: ilammy/msvc-dev-cmd@v1 + + - name: Install uv + uses: nick-fields/retry@v3 + # uv download can be flaky. Wrapping in a retry. + with: + timeout_minutes: 5 + max_attempts: 3 + shell: ${{ runner.os == 'Windows' && 'pwsh' || 'bash' }} + command: >- + ${{ + runner.os == 'Windows' + && 'irm https://astral.sh/uv/install.ps1 | iex; + echo "$env:USERPROFILE\.local\bin" | Out-File -FilePath $env:GITHUB_PATH -Append' + || 'curl -LsSf https://astral.sh/uv/install.sh | sh && + echo "$HOME/.cargo/bin" >> $GITHUB_PATH' + }} + + - name: Install package with uv + run: uv sync --group dev + + - name: Run tests + run: uv run pytest test/ diff --git a/.github/workflows/merge-queue-blossom-passthrough.yml b/.github/workflows/merge-queue-blossom-passthrough.yml new file mode 100644 index 0000000000..61424fe3cf --- /dev/null +++ b/.github/workflows/merge-queue-blossom-passthrough.yml @@ -0,0 +1,50 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2024 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This script is for merge queue purposes only. +# It's role is to post the blossom-ci status to a commit on a temp branch +# in order to clear the branch protection rule. + +# It must only ever be run on the `merge_group`. +# Blossom-ci must be run as normal on the PR before a +# merge can even get to the merge queue. + +name: blossom-ci-merge-queue-shim + +on: + merge_group: + +permissions: + statuses: write + contents: read + +jobs: + post-blossom-ci-status: + name: Post blossom-ci status + runs-on: ubuntu-latest + steps: + - name: Post blossom-ci status + uses: actions/github-script@v7 + with: + script: | + await github.rest.repos.createCommitStatus({ + owner: context.repo.owner, + repo: context.repo.repo, + sha: context.sha, + state: "success", + context: "blossom-ci", + description: "Auto-pass for merge queue" + }) diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml new file mode 100644 index 0000000000..40724cbbc8 --- /dev/null +++ b/.github/workflows/pre-commit.yml @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Pre-commit + +on: + pull_request: + branches: [main, v2.0-refactor] + merge_group: + +jobs: + pre-commit: + name: Run pre-commit hooks + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 + + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + + - name: Install project with dev dependencies + run: | + python -m pip install --upgrade pip + pip install -e . --group dev + + - uses: pre-commit/action@v3.0.1 diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml new file mode 100644 index 0000000000..32d566f693 --- /dev/null +++ b/.github/workflows/stale.yml @@ -0,0 +1,38 @@ +name: 'Close stale issues and PRs' + +on: + schedule: + - cron: '30 1 * * 0' # Runs weekly on Sunday at 1:30 AM UTC + workflow_dispatch: # Allows manual triggering + +jobs: + stale: + runs-on: ubuntu-latest + permissions: + issues: write + pull-requests: write + actions: write + steps: + - uses: actions/stale@v10 + with: + # Process lots of issues per run: + operations-per-run: 250 + + # Issue-specific configuration + stale-issue-message: 'This issue is stale because it has been open for 60 days with no activity. Remove the stale label or comment to keep it open, otherwise this will be closed in 14 days.' + days-before-issue-stale: 60 + days-before-issue-close: 14 + + # PR-specific configuration + stale-pr-message: 'This pull request is stale because it has been open for 90 days with no activity. Remove the stale label or comment to keep it open, otherwise this will be closed in 14 days.' + days-before-pr-stale: 90 + days-before-pr-close: 14 + + # Shared configuration + close-issue-message: 'This issue was closed because it has been inactive for 14 days since being marked as stale.' + close-pr-message: 'This pull request was closed because it has been inactive for 14 days since being marked as stale.' + stale-issue-label: 'stale' + stale-pr-label: 'stale' + exempt-issue-labels: 'pinned,security,roadmap' + exempt-pr-labels: 'pinned,work-in-progress' + diff --git a/.gitignore b/.gitignore index 244f9797fa..f2e7e12518 100644 --- a/.gitignore +++ b/.gitignore @@ -49,6 +49,7 @@ coverage.xml *.py,cover .hypothesis/ .pytest_cache/ +.testmon* cover/ # Translations @@ -70,6 +71,7 @@ instance/ # Sphinx documentation docs/_build/ +docs/examples/ # PyBuilder .pybuilder/ @@ -153,6 +155,7 @@ cython_debug/ # VsCode .vscode/ +.cursor/ # VIM *.swp @@ -162,7 +165,16 @@ cython_debug/ nsight-systems* build/ mlruns/ +checkpoints/ # Hydra outputs/ multirun/ +.hydra/ + +# SLURM +slurm-*.out +sbatch_logs/ + +# ASV +.asv/ diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 0000000000..5dc7de28a1 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,11 @@ +stages: + - test + +include: + + - project: 'modulus/modulus-ci' + ref: main + file: '.gitlab-ci/physicsnemo/security.gitlab-ci.yml' + - project: 'modulus/modulus-ci' + ref: main + file: '.gitlab-ci/physicsnemo/multigpu.gitlab-ci.yml' \ No newline at end of file diff --git a/.gitlab/issue_templates/bug.md b/.gitlab/issue_templates/bug.md index 5d01e7b9a0..a009433f06 100644 --- a/.gitlab/issue_templates/bug.md +++ b/.gitlab/issue_templates/bug.md @@ -1,6 +1,6 @@ # Bug -For technical questions please refer to our [forum](https://forums.developer.nvidia.com/c/physics-simulation/modulus-physics-ml-model-framework/443). +For technical questions please refer to our [forum](https://forums.developer.nvidia.com/t/welcome-to-the-physicsnemo-ml-model-framework-forum/178556). Before submitting an issue, please review the existing issues to avoid duplicates. @@ -35,12 +35,12 @@ python collect_env.py - GPU models and configuration: - Versions of any other relevant libraries: -## Modulus Info +## PhysicsNeMo Info Please fill out the list below: -- Modulus version: -- How Modulus is used (Docker image/ bare-metal installation): +- PhysicsNeMo version: +- How PhysicsNeMo is used (Docker image/ bare-metal installation): - (If using OptiX) OptiX version: - Exact command to reproduce: -- (If using Modulus Docker image): Exact docker-run command: +- (If using PhysicsNeMo Docker image): Exact docker-run command: diff --git a/.gitlab/issue_templates/documentation.md b/.gitlab/issue_templates/documentation.md index 3974c2c905..499e652b44 100644 --- a/.gitlab/issue_templates/documentation.md +++ b/.gitlab/issue_templates/documentation.md @@ -1,6 +1,6 @@ # Documentation -For technical questions please refer to our [forum](https://forums.developer.nvidia.com/c/physics-simulation/modulus-physics-ml-model-framework/443). +For technical questions please refer to our [forum](https://forums.developer.nvidia.com/t/welcome-to-the-physicsnemo-ml-model-framework-forum/178556). Before submitting an issue, please review the existing issues to avoid duplicates. @@ -8,7 +8,7 @@ For documenation issues, please preface the title with [documentation]. ## The documentation issue -A clear and concise description of what content in Modulus documentation is an issue. +A clear and concise description of what content in PhysicsNeMo documentation is an issue. ## Suggest a potential alternative/fix diff --git a/.gitlab/issue_templates/feature_request.md b/.gitlab/issue_templates/feature_request.md index 1f92017500..ce29f28667 100644 --- a/.gitlab/issue_templates/feature_request.md +++ b/.gitlab/issue_templates/feature_request.md @@ -1,6 +1,6 @@ # Feature Request -For technical questions please refer to our [forum](https://forums.developer.nvidia.com/c/physics-simulation/modulus-physics-ml-model-framework/443). +For technical questions please refer to our [forum](https://forums.developer.nvidia.com/t/welcome-to-the-physicsnemo-ml-model-framework-forum/178556). Before submitting an issue, please review the existing issues to avoid duplicates. diff --git a/.gitlab/issue_templates/security.md b/.gitlab/issue_templates/security.md index bfe3148412..d652521a9a 100644 --- a/.gitlab/issue_templates/security.md +++ b/.gitlab/issue_templates/security.md @@ -1,6 +1,6 @@ # Security -For technical questions please refer to our [forum](https://forums.developer.nvidia.com/c/physics-simulation/modulus-physics-ml-model-framework/443). +For technical questions please refer to our [forum](https://forums.developer.nvidia.com/t/welcome-to-the-physicsnemo-ml-model-framework-forum/178556). Before submitting an issue, please review the existing issues to avoid duplicates. @@ -8,6 +8,6 @@ For security issues, please preface the title with [security]. ## The Security Issue -If you believe you have found a security vulnerability in Modulus, we encourage you to +If you believe you have found a security vulnerability in PhysicsNeMo, we encourage you to let us know right away. We will investigate all legitimate reports and do our best to quickly fix the problem. diff --git a/.gitlab/merge_request_templates/Default.md b/.gitlab/merge_request_templates/Default.md index ecff496894..c3630974d1 100644 --- a/.gitlab/merge_request_templates/Default.md +++ b/.gitlab/merge_request_templates/Default.md @@ -1,4 +1,5 @@ -# Modulus Pull Request + +# PhysicsNeMo Pull Request ## Description @@ -7,11 +8,12 @@ ## Checklist -- [ ] I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/modulus/blob/main/CONTRIBUTING.md). +- [ ] I am familiar with the [Contributing Guidelines](https://github.com/NVIDIA/physicsnemo/blob/main/CONTRIBUTING.md). - [ ] New or existing tests cover these changes. - [ ] The documentation is up to date with these changes. -- [ ] The [CHANGELOG.md](https://github.com/NVIDIA/modulus/blob/main/CHANGELOG.md) is -up to date with these changes. +- [ ] The [CHANGELOG.md](https://github.com/NVIDIA/physicsnemo/blob/main/CHANGELOG.md) is up to date with these changes. +- [ ] An [issue](https://github.com/NVIDIA/physicsnemo/issues) is linked to this pull request. +- [ ] If I am implementing a new model or modifying any existing model, I have followed the [Models Implementation Coding Standards](https://github.com/NVIDIA/physicsnemo/wiki/Coding-standards-for-models-implementation). ## Dependencies diff --git a/.importlinter b/.importlinter new file mode 100644 index 0000000000..f679830504 --- /dev/null +++ b/.importlinter @@ -0,0 +1,108 @@ +[importlinter] +root_package = physicsnemo +include_external_packages = True +exclude_type_checking_imports = True +contract_types = + forbidden_import: test.ci_tests.prevent_untracked_imports.ForbiddenImportContract + +[importlinter:contract:physicsnemo-modules] +name = Prevent Upward Imports in the PhysicsNemo Structure +type = layers +containers= + physicsnemo +layers = + experimental + active_learning : diffusion + models : datapipes : metrics : domain_parallel + nn + utils + distributed + core + +[importlinter:contract:physicsnemo-core] +name = Control Dependencies in PhysicsNeMo core +type = layers +containers= + physicsnemo.core +layers = + module : registry + meta + warnings | version_check | filesystem + + +[importlinter:contract:physicsnemo-distributed] +name = Control Dependencies in PhysicsNeMo distributed +type = layers +containers= + physicsnemo.distributed +layers = + fft | autograd + mappings + utils + manager + config + +[importlinter:contract:physicsnemo-utils] +name = Control Dependencies in PhysicsNeMo utils +type = layers +containers= + physicsnemo.utils +layers = + mesh | insolation | zenith_angle + profiling + checkpoint + capture + logging | memory + +[importlinter:contract:physicsnemo-nn] +name = Control Dependencies in PhysicsNeMo nn +type = layers +containers= + physicsnemo.nn +layers = + module.fourier_layers | module.transformer_layers | module.unet_layers + module.dgm_layers | module.mlp_layers | module.fully_connected_layers | module.gnn_layers | module.attention_layers + module.activations | module.ball_query | module.conv_layers | module.drop | module.fft | module.fused_silu | module.kan_layers | module.resample_layers | module.siren_layers | module.spectral_layers | module.transformer_decoder | module.weight_fact | module.weight_norm | module.embedding_layers | module.group_norm | module.hpx + functional + module.utils + +[importlinter:contract:physicsnemo-nn-gnn-layers] +name = Control Internal Dependencies in PhysicsNeMo nn GNN Layers +type = layers +containers= + physicsnemo.nn.module.gnn_layers +layers = + bsms + mesh_graph_decoder | mesh_graph_encoder + mesh_node_block | mesh_edge_block + mesh_graph_mlp + utils + graph + distributed_graph + graph_types + +[importlinter:contract:physicsnemo-models] +name = Prevent Imports between physicsnemo models +type = layers +containers= + physicsnemo.models +layers = + mesh_reduced + afno | dlwp | dlwp_healpix | dit |domino | dpot | fengwu | figconvnet | fno | graphcast | meshgraphnet | pangu | pix2pix | rnn | srrn | swinvrnn | topodiff | transolver | vfgn | diffusion_unets + unet + +[importlinter:contract:physicsnemo-diffusion] +name = Control Internal Dependencies in PhysicsNeMo diffusion +type = layers +containers= + physicsnemo.diffusion +layers = + generate + samplers : metrics + noise_schedulers | multi_diffusion | preconditioners | guidance + utils + +[importlinter:contract:physicsnemo-external-imports] +name = Prevent Non-listed external imports in physicsnemo +type = forbidden_import +container = physicsnemo diff --git a/.markdownlint.yaml b/.markdownlint.yaml index 393f21422b..29ed832dfa 100644 --- a/.markdownlint.yaml +++ b/.markdownlint.yaml @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/.markdownlintignore b/.markdownlintignore new file mode 100644 index 0000000000..fb9dc39e47 --- /dev/null +++ b/.markdownlintignore @@ -0,0 +1 @@ +CODE_OF_CONDUCT.md diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 488f564906..62f46b8fd0 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,14 +15,16 @@ # limitations under the License. repos: -- repo: https://github.com/psf/black - rev: 22.10.0 +- repo: https://github.com/astral-sh/ruff-pre-commit + rev: v0.12.5 hooks: - - id: black - exclude: "(docs|modulus/experimental)" + - id: ruff-check + args: [--fix] + exclude: ^examples/ + - id: ruff-format - repo: https://github.com/econchick/interrogate - rev: 1.5.0 + rev: 1.7.0 hooks: - id: interrogate args: [ @@ -29,7 +33,7 @@ repos: "--ignore-magic", "--fail-under=99", "--exclude=['setup.py', 'test', 'build', 'docs']", "--ignore-regex=['forward', 'backward', 'reset_parameters', 'extra_repr', 'MetaData', 'apply_activation','exec_activation']", "--color", "--"] - exclude: ^modulus/internal/|^modulus/experimental/|^docs/ + exclude: ^docs/|^physicsnemo/experimental/ - repo: https://github.com/igorshubovych/markdownlint-cli rev: v0.35.0 @@ -42,17 +46,17 @@ repos: name: license entry: python test/ci_tests/header_check.py language: python - pass_filenames: false - -- repo: https://github.com/astral-sh/ruff-pre-commit - rev: v0.0.290 - hooks: - - id: ruff - args: [--fix] - exclude: ^docs/ + files: \.(py|yaml|ci|release)$|Dockerfile$ + exclude: ^(physicsnemo/internal/|docs/) - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.4.0 hooks: - id: check-added-large-files args: [--maxkb=5000] + +- repo: https://github.com/seddonym/import-linter + rev: v2.5.2 + hooks: + - id: import-linter + args: [--verbose] \ No newline at end of file diff --git a/CHANGELOG.md b/CHANGELOG.md index ef0b467857..5f0dee4192 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,456 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.5.0a0] - 2024-01-XX +## [2.0.0a0] - 2026-XX-YY + +### Added + +- Refactored diffusion preconditioners in + `physicsnemo.diffusion.preconditioners` relying on a new abstract base class + `BaseAffinePreconditioner` for preconditioning schemes using affine + transformations. Existing preconditioners (`VPPrecond`, `VEPrecond`, + `iDDPMPrecond`, `EDMPrecond`) reimplemented based on this new interface. +- New `physicsnemo.experimental.nn.symmetry` module that implements building + blocks that preserve 2D and 3D rotational equivariance using a + grid-based layout for efficient GPU parallelization, and an emphasis on + compact `einsum` operations. + +### Changed + +- PhysicsNemo v2.0 contains significant reorganization of tools. Please see + the v2.0-MIGRATION-GUIDE.md to understand what has changed and why. +- DiT (Diffusion Transformer) has been moved from `physicsnemo.experimental.models.dit` + to `physicsnemo.models.dit`. + +### Deprecated + +### Removed + +### Fixed + +- Shape mistmatch bug in the Lennard Jones example + +### Security + +### Dependencies + +- CUDA backend is now selected via orthogonal `cu12` / `cu13` extras rather + than being hardcoded to CUDA 13. Feature extras (`nn-extras`, `utils-extras`, + etc.) are now CUDA-agnostic and can be combined with either backend, e.g. + `pip install "nvidia-physicsnemo[cu13,nn-extras]"`. When neither `cu12` nor + `cu13` is specified, PyTorch is installed from PyPI using its default build + (currently CUDA 12.8 on Linux). For development with `uv`, use + `uv sync --extra cu13` (or `--extra cu12`) to select the backend. + +## [1.3.0] - 2025-11-17 + +### Added + +- Added mixture_of_experts for weather example in physicsnemo.examples.weather. + **⚠️Warning:** - It uses experimental DiT model subject to future API changes. + Added some modifications to DiT architecture in physicsnemo.experimental.models.dit. + Added learnable option to PositionalEmbedding in physicsnemo.models.diffusion.layers. +- Added lead-time aware training support to the StormCast example. +- Add a device aware kNN method to physicsnemo.utils.neighbors. Works with CPU or GPU + by dispatching to the proper optimized library, and torch.compile compatible. +- Added additional testing of the DoMINO datapipe. +- Examples: added a new example for full-waveform inversion using diffusion + models. Accessible in `examples/geophysics/diffusion_fwi`. +- Domain Parallelism: Domain Parallelism is now available for kNN, radius_search, + and torch.nn.functional.pad. +- Unified recipe for crash modeling, supporting Transolver and MeshGraphNet, + and three transient schemes. +- Added a check to `stochastic_sampler` that helps handle the `EDMPrecond` model, + which has a specific `.forward()` signature +- Examples: added a new example for reservoir simulation using X-MeshGraphNet. + Accessible in `examples/reservoir_simulation` +- Added abstract interfaces for constructing active learning workflows, contained + under the `physicsnemo.active_learning` namespace. A preliminary example of how + to compose and define an active learning workflow is provided in `examples/active_learning`. + The `moons` example provides a minimal (pedagogical) composition that is meant to + illustrate how to define the necessary parts of the workflow. +- Added a new example for temporal interpolation of weather forecasts using ModAFNO. + Accessible in `examples/weather/temporal_interpolation`. + +### Changed + +- Migrated Stokes MGN example to PyTorch Geometric. +- Migrated Lennard Jones example to PyTorch Geometric. +- Migrated physicsnemo.utils.sdf.signed_distance_field to a static return, + torch-only interface. It also now works on distributed meshes and input fields. +- Refactored DiTBlock to be more modular +- Added NATTEN 2D neighborhood attention backend for DiTBlock +- Migrated blood flow example to PyTorch Geometric. +- Refactored DoMINO model code and examples for performance optimizations and improved readability. +- Migrated HydroGraphNet example to PyTorch Geometric. +- Support for saving and loading nested `physicsnemo.Module`s. It is now + possible to create nested modules with `m = Module(submodule, ...)`, and save + and load them with `Module.save` and `Module.from_checkpoint`. + **⚠️Warning:** - The modules have to be `physicsnemo.Module`s, and not + `torch.nn.Module`s. +- Support passing custom tokenizer, detokenizer, and attention `Module`s in + experimental DiT architecture +- Improved Transolver training recipe's configuration for checkpointing and normalization. +- Bumped `multi-storage-client` version to 0.33.0 with rust client. +- Improved configuration for DLWP Healpix (checkpoint directory) and GraphCast (W&B settings). + +### Fixed + +- Set `skip_scale` to Python float in U-Net to ensure compilation works. +- Ensure stream dependencies are handled correctly in physicsnemo.utils.neighbors +- Fixed the issue with incorrect handling of files with consecutive runs of + `combine_stl_solids.py` in the X-MGN recipe. +- Fixed the `RuntimeError: Worker data receiving interrupted` error in the datacenter example. + +## [1.2.0] - 2025-08-26 + +### Added + +- Diffusion Transformer (DiT) model. The DiT model can be accessed in + `physicsnemo.experimental.models.dit.DiT`. **⚠️Warning:** - Experimental feature + subject to future API changes. +- Improved documentation for diffusion models and diffusion utils. +- Safe API to override `__init__`'s arguments saved in checkpoint file with + `Module.from_checkpoint("chkpt.mdlus", override_args=set(...))`. +- PyTorch Geometric MeshGraphNet backend. +- Functionality in DoMINO to take arbitrary number of `scalar` or `vector` + global parameters and encode them using `class ParameterModel` +- TopoDiff model and example. +- Added ability for DoMINO model to return volume neighbors. +- Added functionality in DoMINO recipe to introduce physics residual losses. +- Diffusion models, metrics, and utils: implementation of Student-t + distribution for EDM-based diffusion models (t-EDM). This feature is adapted + from the paper [Heavy-Tailed Diffusion Models, Pandey et al.](https://arxiv.org/abs/2410.14171>). + This includes a new EDM preconditioner (`tEDMPrecondSuperRes`), a loss + function (`tEDMResidualLoss`), and a new option in corrdiff `diffusion_step`. + ⚠️ This is an experimental feature that can be accessed through the + `physicsnemo.experimental` module; it might also be subjected to API changes + without notice. +- Bumped Ruff version from 0.0.290 to 0.12.5. Replaced Black with `ruff-format`. +- Domino improvements with Unet attention module and user configs +- Hybrid MeshGraphNet for modeling structural deformation +- Enabled TransformerEngine backend in the `transolver` model. +- Inference code for x-meshgraphnet example for external aerodynamics. +- Added a new example for external_aerodynamics: training `transolver` on + irregular mesh data for DrivaerML surface data. +- Added a new example for external aerodynamics for finetuning pretrained models. + +### Changed + +- Diffusion utils: `physicsnemo.utils.generative` renamed into `physicsnemo.utils.diffusion` +- Diffusion models: in CorrDiff model wrappers (`EDMPrecondSuperResolution` and + `UNet`), the arguments `profile_mode` and `amp_mode` cannot be overriden by + `from_checkpoint`. They are now properties that can be dynamically changed + *after* the model instantiation with, for example, `model.amp_mode = True` + and `model.profile_mode = False`. +- Updated healpix data module to use correct `DistributedSampler` target for + test data loader +- Existing DGL-based vortex shedding example has been renamed to `vortex_shedding_mgn_dgl`. + Added new `vortex_shedding_mgn` example that uses PyTorch Geometric instead. +- HEALPixLayer can now use earth2grid HEALPix padding ops, if desired +- Migrated Vortex Shedding Reduced Mesh example to PyTorch Geometric. +- CorrDiff example: fixed bugs when training regression `UNet`. +- Diffusion models: fixed bugs related to gradient checkpointing on non-square + images. +- Diffusion models: created a separate class `Attention` for clarity and + modularity. Updated `UNetBlock` accordingly to use the `Attention` class + instead of custom attention logic. This will update the model architecture + for `SongUNet`-based diffusion models. Changes are not BC-breaking and are + transparent to the user. +- ⚠️ **BC-breaking:** refactored the automatic mixed precision + (AMP) API in layers and models defined in `physicsnemo/models/diffusion/` for + improved usability. Note: it is now, not only possible, but *required* to + explicitly set `model.amp_mode = True` in order to use the model in a + `torch.autocast` clause. This applies to all `SongUNet`-based models. +- Diffusion models: fixed and improved API to enable fp16 forward pass in + `UNet` and `EDMPrecondSuperResolution` model wrappers; fp16 forward pass can + now be toggled/untoggled by setting `model.use_fp16 = True`. +- Diffusion models: improved API for Apex group norm. `SongUNet`-based models + will automatically perform conversion of the input tensors to + `torch.channels_last` memory format when `model.use_apex_gn` is `True`. New + warnings are raised when attempting to use Apex group norm on CPU. +- Diffusion utils: systematic compilation of patching operations in `stochastic_sampler` + for improved performance. +- CorrDiff example: added option for Student-t EDM (t-EDM) in `train.py` and + `generate.py`. When training a CorrDiff diffusion model, this feature can be + enabled with the hydra overrides `++training.hp.distribution=student_t` and + `++training.hp.nu_student_t=`. For generation, this feature can be + enabled with similar overrides: `++generation.distribution=student_t` and + `++generation.nu_student_t=`. +- CorrDiff example: the parameters `P_mean` and `P_std` (used to compute the + noise level `sigma`) are now configurable. They can be set with the hydra + overrides `++training.hp.P_mean=` and + `++training.hp.P_std=` for training (and similar ones with + `training.hp` replaced by `generation` for generation). +- Diffusion utils: patch-based inference and lead time support with + deterministic sampler. +- Existing DGL-based XAeroNet example has been renamed to `xaeronet_dgl`. + Added new `xaeronet` example that uses PyTorch Geometric instead. +- Updated the deforming plate example to use the Hybrid MeshGraphNet model. +- ⚠️ **BC-breaking:** Refactored the `transolver` model to improve + readability and performance, and extend to more use cases. +- Diffusion models: improved lead time support for `SongUNetPosLtEmbd` and + `EDMLoss`. Lead-time embeddings can now be used with/without positional + embeddings. +- Diffusion models: consolidate `ApexGroupNorm` and `GroupNorm` in + `models/diffusion/layers.py` with a factory `get_group_norm` that can + be used to instantiate either one of them. `get_group_norm` is now the + recommended way to instantiate a GroupNorm layer in `SongUNet`-based and + other diffusion models. +- Physicsnemo models: improved checkpoint loading API in + `Module.from_checkpoint` that now exposes a `strict` parameter to raise error + on missing/unexpected keys, similar to that used in + `torch.nn.Module.load_state_dict`. +- Migrated Hybrid MGN and deforming plate example to PyTorch Geometric. + +### Fixed + +- Bug fixes in DoMINO model in sphere sampling and tensor reshaping +- Bug fixes in DoMINO utils random sampling and test.py +- Optimized DoMINO config params based on DrivAer ML + +## [1.1.1] - 2025-06-16 + +### Fixed + +- Fixed an inadvertent change to the deterministic sampler 2nd order correction +- Bug Fix in Domino model ball query layer +- Fixed bug models/unet/unet.py: setting num_conv_layers=1 gives errors + +## [1.1.0] - 2025-06-05 + +### Added + +- Added ReGen score-based data assimilation example +- General purpose patching API for patch-based diffusion +- New positional embedding selection strategy for CorrDiff SongUNet models +- Added Multi-Storage Client to allow checkpointing to/from Object Storage +- Added a new aerodynamics example using DoMINO to compute design sensitivities + (e.g., drag adjoint) with respect to underlying input geometry. + +### Changed + +- Simplified CorrDiff config files, updated default values +- Refactored CorrDiff losses and samplers to use the patching API +- Support for non-square images and patches in patch-based diffusion +- ERA5 download example updated to use current file format convention and + restricts global statistics computation to the training set +- Support for training custom StormCast models and various other improvements for StormCast +- Updated CorrDiff training code to support multiple patch iterations to amortize + regression cost and usage of `torch.compile` +- Refactored `physicsnemo/models/diffusion/layers.py` to optimize data type + casting workflow, avoiding unnecessary casting under autocast mode +- Refactored Conv2d to enable fusion of conv2d with bias addition +- Refactored GroupNorm, UNetBlock, SongUNet, SongUNetPosEmbd to support usage of + Apex GroupNorm, fusion of activation with GroupNorm, and AMP workflow. +- Updated SongUNetPosEmbd to avoid unnecessary HtoD Memcpy of `pos_embd` +- Updated `from_checkpoint` to accommodate conversion between Apex optimized ckp + and non-optimized ckp +- Refactored CorrDiff NVTX annotation workflow to be configurable +- Refactored `ResidualLoss` to support patch-accumlating training for + amortizing regression costs +- Explicit handling of Warp device for ball query and sdf +- Merged SongUNetPosLtEmb with SongUNetPosEmb, add support for batch>1 +- Add lead time embedding support for `positional_embedding_selector`. Enable +arbitrary positioning of probabilistic variables +- Enable lead time aware regression without CE loss +- Bumped minimum PyTorch version from 2.0.0 to 2.4.0, to minimize + support surface for `physicsnemo.distributed` functionality. + +### Dependencies + +- Made `nvidia.dali` an optional dependency + +## [1.0.1] - 2025-03-25 + +### Added + +- Added version checks to ensure compatibility with older PyTorch for distributed + utilities and ShardTensor + +### Fixed + +- `EntryPoint` error that occured during physicsnemo checkpoint loading + +## [1.0.0] - 2025-03-18 + +### Added + +- DoMINO model architecture, datapipe and training recipe +- Added matrix decomposition scheme to improve graph partitioning +- DrivAerML dataset support in FIGConvNet example. +- Retraining recipe for DoMINO from a pretrained model checkpoint +- Prototype support for domain parallelism of using ShardTensor (new). +- Enable DeviceMesh initialization via DistributedManager. +- Added Datacenter CFD use case. +- Add leave-in profiling utilities to physicsnemo, to easily enable torch/python/nsight + profiling in all aspects of the codebase. + +### Changed + +- Refactored StormCast training example +- Enhancements and bug fixes to DoMINO model and training example +- Enhancement to parameterize DoMINO model with inlet velocity +- Moved non-dimensionaliztion out of domino datapipe to datapipe in domino example +- Updated utils in `physicsnemo.launch.logging` to avoid unnecessary `wandb` and `mlflow` + imports +- Moved to experiment-based Hydra config in Lagrangian-MGN example +- Make data caching optional in `MeshDatapipe` +- The use of older `importlib_metadata` library is removed + +### Deprecated + +- ProcessGroupConfig is tagged for future deprecation in favor of DeviceMesh. + +### Fixed + +- Update pytests to skip when the required dependencies are not present +- Bug in data processing script in domino training example +- Fixed NCCL_ASYNC_ERROR_HANDLING deprecation warning + +### Dependencies + +- Remove the numpy dependency upper bound +- Moved pytz and nvtx to optional +- Update the base image for the Dockerfile +- Introduce Multi-Storage Client (MSC) as an optional dependency. +- Introduce `wrapt` as an optional dependency, needed when using + ShardTensor's automatic domain parallelism + +## [0.9.0] - 2024-12-04 + +### Added + +- Graph Transformer processor for GraphCast/GenCast. +- Utility to generate STL from Signed Distance Field. +- Metrics for CAE and CFD domain such as integrals, drag, and turbulence invariances and + spectrum. +- Added gradient clipping to StaticCapture utilities. +- Bistride Multiscale MeshGraphNet example. +- FIGConvUNet model and example. +- The Transolver model. +- The XAeroNet model. +- Incoporated CorrDiff-GEFS-HRRR model into CorrDiff, with lead-time aware SongUNet and + cross entropy loss. +- Option to offload checkpoints to further reduce memory usage +- Added StormCast model training and simple inference to examples +- Multi-scale geometry features for DoMINO model. + +### Changed + +- Refactored CorrDiff training recipe for improved usability +- Fixed timezone calculation in datapipe cosine zenith utility. +- Refactored EDMPrecondSRV2 preconditioner and fixed the bug related to the metadata +- Extended the checkpointing utility to store metadata. +- Corrected missing export of loggin function used by transolver model + +## [0.8.0] - 2024-09-24 + +### Added + +- Graph Transformer processor for GraphCast/GenCast. +- Utility to generate STL from Signed Distance Field. +- Metrics for CAE and CFD domain such as integrals, drag, and turbulence invariances and + spectrum. +- Added gradient clipping to StaticCapture utilities. +- Bistride Multiscale MeshGraphNet example. + +### Changed + +- Refactored CorrDiff training recipe for improved usability +- Fixed timezone calculation in datapipe cosine zenith utility. + +## [0.7.0] - 2024-07-23 + +### Added + +- Code logging for CorrDiff via Wandb. +- Augmentation pipeline for CorrDiff. +- Regression output as additional conditioning for CorrDiff. +- Learnable positional embedding for CorrDiff. +- Support for patch-based CorrDiff training and generation (stochastic sampling only) +- Enable CorrDiff multi-gpu generation +- Diffusion model for fluid data super-resolution (CMU contribution). +- The Virtual Foundry GraphNet. +- A synthetic dataloader for global weather prediction models, demonstrated on GraphCast. +- Sorted Empirical CDF CRPS algorithm +- Support for history, cos zenith, and downscaling/upscaling in the ERA5 HDF5 dataloader. +- An example showing how to train a "tensor-parallel" version of GraphCast on a +Shallow-Water-Equation example. +- 3D UNet +- AeroGraphNet example of training of MeshGraphNet on Ahmed body and DrivAerNet datasets. +- Warp SDF routine +- DLWP HEALPix model +- Pangu Weather model +- Fengwu model +- SwinRNN model +- Modulated AFNO model + +### Changed + +- Raise `PhysicsNeMoUndefinedGroupError` when querying undefined process groups +- Changed Indexing error in `examples/cfd/swe_nonlinear_pino` for `physicsnemo` loss function +- Safeguarding against uninitialized usage of `DistributedManager` + +### Removed + +- Remove mlflow from deployment image + +### Fixed + +- Fixed bug in the partitioning logic for distributing graph structures +intended for distributed message-passing. +- Fixed bugs for corrdiff diffusion training of `EDMv1` and `EDMv2` +- Fixed bug when trying to save DDP model trained through unified recipe + +### Dependencies + +- Update DALI to CUDA 12 compatible version. +- Update minimum python version to 3.10 + +## [0.6.0] - 2024-04-17 + +### Added + +- The citation file. +- Link to the CWA dataset. +- ClimateDatapipe: an improved datapipe for HDF5/NetCDF4 formatted climate data +- Performance optimizations to CorrDiff. +- Physics-Informed Nonlinear Shallow Water Equations example. +- Warp neighbor search routine with a minimal example. +- Strict option for loading PhysicsNeMo checkpoints. +- Regression only or diffusion only inference for CorrDiff. +- Support for organization level model files on NGC file system +- Physics-Informed Magnetohydrodynamics example. + +### Changed + +- Updated Ahmed Body and Vortex Shedding examples to use Hydra config. +- Added more config options to FCN AFNO example. +- Moved posiitonal embedding in CorrDiff from the dataloader to network architecture + +### Deprecated + +- `physicsnemo.models.diffusion.preconditioning.EDMPrecondSR`. Use `EDMPecondSRV2` instead. + +### Removed + +- Pickle dependency for CorrDiff. + +### Fixed + +- Consistent handling of single GPU runs in DistributedManager +- Output location of objects downloaded with NGC file system +- Bug in scaling the conditional input in CorrDiff deterministic sampler + +### Dependencies + +- Updated DGL build in Dockerfile +- Updated default base image +- Moved Onnx from optional to required dependencies +- Optional Makani dependency required for SFNO model. + +## [0.5.0] - 2024-01-25 ### Added @@ -18,26 +467,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Molecular Dynamics example. - Improved usage of GraphPartition, added more flexible ways of defining a partitioned graph. - Physics-Informed Stokes Flow example. +- Profiling markers, benchmarking and performance optimizations for CorrDiff inference. +- Unified weather model training example. ### Changed - MLFLow logging such that only proc 0 logs to MLFlow. - FNO given seperate methods for constructing lift and spectral encoder layers. -### Deprecated - ### Removed - The experimental SFNO -### Fixed - -### Security - ### Dependencies - Removed experimental SFNO dependencies -- Added CorrDiff dependencies (cftime, einops, pyspng) +- Added CorrDiff dependencies (cftime, einops, pyspng, nvtx) - Made tqdm a required dependency ## [0.4.0] - 2023-11-20 @@ -49,7 +494,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 weather models - Added distributed FFT utility. - Added ruff as a linting tool. -- Ported utilities from Modulus Launch to main package. +- Ported utilities from PhysicsNeMo Launch to main package. - EDM diffusion models and recipes for training and sampling. - NGC model registry download integration into package/filesystem. - Denoising diffusion tutorial. @@ -57,12 +502,12 @@ weather models ### Changed - The AFNO input argument `img_size` to `inp_shape` -- Integrated the network architecture layers from Modulus-Sym. +- Integrated the network architecture layers from PhysicsNeMo-Sym. - Updated the SFNO model, and the training and inference recipes. ### Fixed -- Fixed modulus.Module `from_checkpoint` to work from custom model classes +- Fixed physicsnemo.Module `from_checkpoint` to work from custom model classes ### Dependencies @@ -85,11 +530,11 @@ weather models ### Changed -- Updating file system cache location to modulus folder +- Updating file system cache location to physicsnemo folder ### Fixed -- Fixed modulus uninstall in CI docker image +- Fixed physicsnemo uninstall in CI docker image ### Security @@ -130,7 +575,7 @@ weather models ### Fixed - Fixed issue with torch-harmonics version locking -- Fixed the Modulus editable install +- Fixed the PhysicsNeMo editable install - Fixed AMP bug in static capture ### Security diff --git a/CITATION.cff b/CITATION.cff new file mode 100644 index 0000000000..4cfc7d6680 --- /dev/null +++ b/CITATION.cff @@ -0,0 +1,7 @@ +cff-version: 1.2.0 +message: "If you use this software, please cite it as below." +title: "NVIDIA PhysicsNeMo: An open-source framework for physics-based deep learning in science and engineering" +date-released: "2023-02-24" +authors: + - name: "PhysicsNeMo Contributors" +repository-code: "https://github.com/NVIDIA/physicsnemo" diff --git a/CODE_OF_CONDUCT.MD b/CODE_OF_CONDUCT.MD new file mode 100644 index 0000000000..81cf777856 --- /dev/null +++ b/CODE_OF_CONDUCT.MD @@ -0,0 +1,92 @@ + +# Contributor Covenant 3.0 Code of Conduct + +## Our Pledge + +We pledge to make our community welcoming, safe, and equitable for all. + +We are committed to fostering an environment that respects and promotes the dignity, rights, and contributions of all individuals, regardless of characteristics including race, ethnicity, caste, color, age, physical characteristics, neurodiversity, disability, sex or gender, gender identity or expression, sexual orientation, language, philosophy or religion, national or social origin, socio-economic position, level of education, or other status. The same privileges of participation are extended to everyone who participates in good faith and in accordance with this Covenant. + +## Encouraged Behaviors + +While acknowledging differences in social norms, we all strive to meet our community's expectations for positive behavior. We also understand that our words and actions may be interpreted differently than we intend based on culture, background, or native language. + +With these considerations in mind, we agree to behave mindfully toward each other and act in ways that center our shared values, including: + +1. Respecting the **purpose of our community**, our activities, and our ways of gathering. +2. Engaging **kindly and honestly** with others. +3. Respecting **different viewpoints** and experiences. +4. **Taking responsibility** for our actions and contributions. +5. Gracefully giving and accepting **constructive feedback**. +6. Committing to **repairing harm** when it occurs. +7. Behaving in other ways that promote and sustain the **well-being of our community**. + + +## Restricted Behaviors + +We agree to restrict the following behaviors in our community. Instances, threats, and promotion of these behaviors are violations of this Code of Conduct. + +1. **Harassment.** Violating explicitly expressed boundaries or engaging in unnecessary personal attention after any clear request to stop. +2. **Character attacks.** Making insulting, demeaning, or pejorative comments directed at a community member or group of people. +3. **Stereotyping or discrimination.** Characterizing anyone’s personality or behavior on the basis of immutable identities or traits. +4. **Sexualization.** Behaving in a way that would generally be considered inappropriately intimate in the context or purpose of the community. +5. **Violating confidentiality**. Sharing or acting on someone's personal or private information without their permission. +6. **Endangerment.** Causing, encouraging, or threatening violence or other harm toward any person or group. +7. Behaving in other ways that **threaten the well-being** of our community. + +### Other Restrictions + +1. **Misleading identity.** Impersonating someone else for any reason, or pretending to be someone else to evade enforcement actions. +2. **Failing to credit sources.** Not properly crediting the sources of content you contribute. +3. **Promotional materials**. Sharing marketing or other commercial content in a way that is outside the norms of the community. +4. **Irresponsible communication.** Failing to responsibly present content which includes, links or describes any other restricted behaviors. + + +## Reporting an Issue + +Tensions can occur between community members even when they are trying their best to collaborate. Not every conflict represents a code of conduct violation, and this Code of Conduct reinforces encouraged behaviors and norms that can help avoid conflicts and minimize harm. + +When an incident does occur, it is important to report it promptly. To report a possible violation, **please contact physicsnemo-team@nvidia.com** + +Community Moderators take reports of violations seriously and will make every effort to respond in a timely manner. They will investigate all reports of code of conduct violations, reviewing messages, logs, and recordings, or interviewing witnesses and other participants. Community Moderators will keep investigation and enforcement actions as transparent as possible while prioritizing safety and confidentiality. In order to honor these values, enforcement actions are carried out in private with the involved parties, but communicating to the whole community may be part of a mutually agreed upon resolution. + + +## Addressing and Repairing Harm + +**** + +If an investigation by the Community Moderators finds that this Code of Conduct has been violated, the following enforcement ladder may be used to determine how best to repair harm, based on the incident's impact on the individuals involved and the community as a whole. Depending on the severity of a violation, lower rungs on the ladder may be skipped. + +1) Warning + 1) Event: A violation involving a single incident or series of incidents. + 2) Consequence: A private, written warning from the Community Moderators. + 3) Repair: Examples of repair include a private written apology, acknowledgement of responsibility, and seeking clarification on expectations. +2) Temporarily Limited Activities + 1) Event: A repeated incidence of a violation that previously resulted in a warning, or the first incidence of a more serious violation. + 2) Consequence: A private, written warning with a time-limited cooldown period designed to underscore the seriousness of the situation and give the community members involved time to process the incident. The cooldown period may be limited to particular communication channels or interactions with particular community members. + 3) Repair: Examples of repair may include making an apology, using the cooldown period to reflect on actions and impact, and being thoughtful about re-entering community spaces after the period is over. +3) Temporary Suspension + 1) Event: A pattern of repeated violation which the Community Moderators have tried to address with warnings, or a single serious violation. + 2) Consequence: A private written warning with conditions for return from suspension. In general, temporary suspensions give the person being suspended time to reflect upon their behavior and possible corrective actions. + 3) Repair: Examples of repair include respecting the spirit of the suspension, meeting the specified conditions for return, and being thoughtful about how to reintegrate with the community when the suspension is lifted. +4) Permanent Ban + 1) Event: A pattern of repeated code of conduct violations that other steps on the ladder have failed to resolve, or a violation so serious that the Community Moderators determine there is no way to keep the community safe with this person as a member. + 2) Consequence: Access to all community spaces, tools, and communication channels is removed. In general, permanent bans should be rarely used, should have strong reasoning behind them, and should only be resorted to if working through other remedies has failed to change the behavior. + 3) Repair: There is no possible repair in cases of this severity. + +This enforcement ladder is intended as a guideline. It does not limit the ability of Community Managers to use their discretion and judgment, in keeping with the best interests of our community. + + +## Scope + +This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public or other spaces. Examples of representing our community include using an official email address, posting via an official social media account, or acting as an appointed representative at an online or offline event. + + +## Attribution + +This Code of Conduct is adapted from the Contributor Covenant, version 3.0, permanently available at [https://www.contributor-covenant.org/version/3/0/](https://www.contributor-covenant.org/version/3/0/). + +Contributor Covenant is stewarded by the Organization for Ethical Source and licensed under CC BY-SA 4.0. To view a copy of this license, visit [https://creativecommons.org/licenses/by-sa/4.0/](https://creativecommons.org/licenses/by-sa/4.0/) + +For answers to common questions about Contributor Covenant, see the FAQ at [https://www.contributor-covenant.org/faq](https://www.contributor-covenant.org/faq). Translations are provided at [https://www.contributor-covenant.org/translations](https://www.contributor-covenant.org/translations). Additional enforcement and community guideline resources can be found at [https://www.contributor-covenant.org/resources](https://www.contributor-covenant.org/resources). The enforcement ladder was inspired by the work of [Mozilla’s code of conduct team](https://github.com/mozilla/inclusion). + diff --git a/CODING_STANDARDS/EXTERNAL_IMPORTS.md b/CODING_STANDARDS/EXTERNAL_IMPORTS.md new file mode 100644 index 0000000000..4a9473ac1d --- /dev/null +++ b/CODING_STANDARDS/EXTERNAL_IMPORTS.md @@ -0,0 +1,179 @@ + + +# EXTERNAL_IMPORTS - Coding Standards + +## Overview + +This document defines the policies for managing external dependencies within +`physicsnemo`. The objectives are to maintain a predictable dependency surface, +prevent accidental coupling across modules, and ensure that optional +accelerations never compromise default functionality. + +**Important:** These requirements are enforced rigorously. Any deviation must be +explicitly justified in code comments and approved during code review. + +## Rule Index + +| Rule ID | Summary | Apply When | +|---------|---------|------------| +| `EXT-001` | Keep `pyproject.toml` as the single source of truth for dependencies | Declaring or modifying package requirements | +| `EXT-002` | Preserve the dependency hierarchy via optional dependency groups | Adding dependencies to any `physicsnemo` submodule | +| `EXT-003` | Classify every external import as hard or optional and guard optional ones | Importing third-party packages anywhere in the codebase | +| `EXT-004` | Use the delayed-error pattern for locally necessary optional packages | Implementing features that absolutely require an optional dependency | +| `EXT-005` | Provide guarded accelerated paths alongside a reference implementation | Adding performance-oriented backends that rely on optional packages | + +## Source of Truth for Dependencies + +The `pyproject.toml` file is the single authoritative record of every supported +dependency for the Python package and the test suite. Example applications may +list additional packages under `examples/**/requirements.txt`, but those +requirements must not leak into the core package. + +## Introducing new external dependencies + +Before you introduce new external dependencies to any `physicsnemo` component, +please consider carefully the following: + +- Does this package significantly increase install burden on any common platforms? + If it does, it should likely be an optional dependency and not a core dependency. +- Does this package add value beyond a single use case? If this is only for use + with one function, domain, etc., and not something that could be more broadly + used, consider making this an optional dependency. +- Is the dependency you want to add actively maintained and released? Packages + that do not have an active developer base should not be introduced into + physicsnemo. Such packages could be deprecated and removed in the future. +- Is the license for the package open and permissible for use in `physicsnemo`? + Do not introduce packages with restrictive licenses, `physicsnemo` is an open source + repository that needs to remain usable for all users, including commercial use cases. + In general, anything other than `MIT`, `Apache 2.0`, and `BSD` must be considered + carefully. + +When introducing a dependency, if it is a core dependency you must include a version +minimum. The easiest way to achieve this is with `uv add [package]`. Core +dependencies do not need to be protected in `physicsnemo`. For optional dependencies, +introduce it only where necessary, and ensure it is protected as shown below. + +> NOTE: Never import an optional dependency without protecting the import path. +> Choose an appropriate protection method from the examples below. Optional +> dependencies must never break imports or functionality in other parts of `physicsnemo`. + +## Dependency Hierarchy and Groups + +`physicsnemo` is structured as an acyclic hierarchy. Lower-level packages (for +example, `physicsnemo.core`) have strictly fewer dependencies than higher-level +packages (such as `physicsnemo.nn`). To enforce this layering, dependencies are +organized in `pyproject.toml` as follows: + +- **Core dependencies** are listed under `[project] dependencies` and are required + for all installations of physicsnemo. +- **Optional dependencies** are organized hierarchically under + `[project.optional-dependencies]`, where higher-level groups self-reference + lower-level groups to compose their dependencies. For example: + - `utils-extras` includes optional utilities + - `nn-extras` includes `utils-extras` plus neural network specific packages + - `model-extras` includes `nn-extras` plus model specific packages + - `datapipes-extras` includes `model-extras` plus data pipeline packages +- **Development dependencies** are organized under `[dependency-groups]` (e.g., + `dev` group) for testing and development tools, following PEP 735. +- **Use-case specific groups** like `gnns` and `healpix` provide targeted + dependency bundles for specific workflows. + +## Classification of External Imports + +Every import from a third-party package must fall into one of two categories: + +1. **Hard dependency.** The package is part of the mandatory dependency group + of the importing submodule or any lower-level submodule. Typical examples + include `torch` and `warp`. +2. **Optional dependency.** The package resides in an extras group or optional + dependency group. Its usage must be guarded so that importing the module + succeeds even when the package is absent. + +Packages such as `cuml`, `torch_geometric`, and `torch_scatter` remain optional +because of their installation complexity; they are surfaced only through extras +groups or per-example requirements. + +## Protecting Imports + +Two complementary patterns are used to guard optional dependencies. + +### Locally Necessary Imports + +Certain features cannot be delivered without a specific package (for example, +PyG for GraphCast backends). For such dependencies, follow the delayed-error +pattern: + +1. Perform a soft availability check via + `physicsnemo.core.version_check.check_version_spec`. +2. When the dependency is present, import it with `importlib.import_module` + inside the guarded block and expose the fully functional implementation. +3. When the dependency is absent, expose the same symbols, but raise an + informative exception upon instantiation or call. Static methods should be + treated as free functions for this purpose. + +Raised exceptions must explain who is raising the error, which package is +missing, the minimum required version, and where to find installation +instructions. + +```python +import importlib +import torch + +from physicsnemo.core.version_check import check_version_spec + +CUML_AVAILABLE = check_version_spec("cuml", "24.0.0", hard_fail=False) +CUPY_AVAILABLE = check_version_spec("cupy", "13.0.0", hard_fail=False) + +if CUML_AVAILABLE and CUPY_AVAILABLE: + cuml = importlib.import_module("cuml") + cp = importlib.import_module("cupy") + + def knn_impl(points, queries, k) -> torch.Tensor: + ... +else: + + def knn_impl(*args, **kwargs) -> torch.Tensor: + """ + Dummy implementation for when cuML or CuPy is unavailable. + """ + + raise ImportError( + "physicsnemo.nn.functional.knn: cuML>=24.0.0 and CuPy>=13.0.0 are required " + "for the accelerated kNN backend. Install both packages; see " + "https://docs.rapids.ai/install for instructions." + ) +``` + +### Locally Optional Imports + +Some dependencies simply provide accelerated code paths. In these situations, +always provide a reference implementation that only relies on core +dependencies, and add accelerated paths behind guarded imports. Two patterns +are acceptable: + +1. **Module-level runtime dispatch.** The dependency is a central part of the + implementation. Provide an entry-point that selects among backends + (`"auto"` should try accelerated paths first while falling back to the + reference path). Each backend implementation must live in its own module and + independently guard its imports. Example: `physicsnemo.nn.functional`. +2. **File-level runtime dispatch.** The dependency affects a small portion of + the implementation. Keep reference and accelerated code in the same module. + Use `check_version_spec` to pick the execution path automatically or to + respect a user override that demands the accelerated backend. + +In both cases the default behavior must rely exclusively on baseline +dependencies, and accelerated code paths must never raise at import time merely +because an optional dependency is missing. + +## Compliance + +- **Code review enforcement.** All pull requests must cite the relevant `EXT-00x` + rules when introducing new dependencies or optional backends. Reviewers block + changes that bypass `pyproject.toml`, break the dependency hierarchy, or ship + unguarded imports; deviations require explicit justification. +- **Import-linter enforcement.** `test/ci_tests/prevent_untracked_imports.py` + and `.importlinter` translate these rules into automated checks. Import Linter + fails CI when modules violate declared contracts (for example, high-level + packages importing from disallowed lower layers or pulling in unapproved + third-party modules). Keep dependency declarations synchronized so these + automated guards remain authoritative. diff --git a/CODING_STANDARDS/FUNCTIONAL_APIS.md b/CODING_STANDARDS/FUNCTIONAL_APIS.md new file mode 100644 index 0000000000..bd37b2bd93 --- /dev/null +++ b/CODING_STANDARDS/FUNCTIONAL_APIS.md @@ -0,0 +1,397 @@ + + + + + + + + + + + +# FUNCTIONAL_APIS - Coding Standards + +## Overview + +This document defines the conventions for functional APIs in PhysicsNeMo. These +rules are designed to ensure consistency, maintainability, and high code +quality across all functional implementations. + +**Important:** These rules are enforced as strictly as possible. Deviations +from these standards should only be made when absolutely necessary and must be +documented with clear justification in code comments and approved during code +review. + +## Document Organization + +This document is structured in two main sections: + +1. **Rule Index**: A quick-reference table listing all rules with their IDs, + one-line summaries, and the context in which they apply. Use this section + to quickly identify relevant rules when implementing or reviewing code. + +2. **Detailed Rules**: Comprehensive descriptions of each rule, including: + - Clear descriptions of what the rule requires + - Rationale explaining why the rule exists + - Examples demonstrating correct implementation + - Anti-patterns showing common mistakes to avoid + +## How to Use This Document + +- **When adding a new functional**: Review rules FNC-000 through FNC-006. +- **When reviewing code**: Use the Rule Index to quickly verify compliance. +- **When refactoring**: Ensure refactored code maintains or improves compliance. +- **For AI agents that generate code**: Each rule has a unique ID and structured + sections (Description, Rationale, Example, Anti-pattern) that can be extracted + and used as context. When generating code based on a rule, explicitly quote + the rule ID and the relevant extract being used as context. +- **For AI agents that review code**: Explicitly identify which rules are + violated, why, and quote the rule ID and relevant extract being used as + context. + +## Rule Index + +| Rule ID | Summary | Apply When | +|---------|---------|------------| +| [`FNC-000`](#fnc-000-functionals-must-use-functionspec) | Functionals must use FunctionSpec | Creating new functional APIs | +| [`FNC-001`](#fnc-001-functional-location-and-public-api) | Functional location and public API | Organizing or exporting functionals | +| [`FNC-002`](#fnc-002-file-layout-for-functionals) | File layout for functionals | Adding or refactoring functional files | +| [`FNC-003`](#fnc-003-registration-and-dispatch-rules) | Registration and dispatch rules | Registering implementations | +| [`FNC-004`](#fnc-004-optional-dependency-handling) | Optional dependency handling | Using optional backends | +| [`FNC-005`](#fnc-005-benchmarking-hooks) | Benchmarking hooks | Implementing `make_inputs`/`compare` | +| [`FNC-006`](#fnc-006-testing-functionals) | Testing functionals | Adding functional tests | +| [`FNC-007`](#fnc-007-benchmark-registry) | Benchmark registry | Adding a functional to ASV | + +--- + +## Detailed Rules + +### FNC-000: Functionals must use FunctionSpec + +**Description:** + +All functionals must be implemented with `FunctionSpec`, even if only a single +implementation exists. This ensures the operation participates in validation +and benchmarking via `make_inputs` and `compare`. + +**Rationale:** + +`FunctionSpec` provides a consistent structure for backend registration, +selection, benchmarking and verification across the codebase. + +**Example:** + +```python +import importlib +import torch + +from physicsnemo.core.function_spec import FunctionSpec +from physicsnemo.core.version_check import check_version_spec + +WARP_AVAILABLE = check_version_spec("warp", "0.6.0", hard_fail=False) + +if WARP_AVAILABLE: + wp = importlib.import_module("warp") + wp.init() + wp.config.quiet = True + + @wp.kernel + def _identity_kernel( + x: wp.array(dtype=wp.float32), + y: wp.array(dtype=wp.float32), + ): + i = wp.tid() + y[i] = x[i] + + @torch.library.custom_op("physicsnemo::identity_warp", mutates_args=()) + def identity_impl(x: torch.Tensor) -> torch.Tensor: + out = torch.empty_like(x) + device, stream = FunctionSpec.warp_launch_context(x) + wp_x = wp.from_torch(x, dtype=wp.float32, return_ctype=True) + wp_y = wp.from_torch(out, dtype=wp.float32, return_ctype=True) + with wp.ScopedStream(stream): + wp.launch( + kernel=_identity_kernel, + dim=x.numel(), + inputs=[wp_x, wp_y], + device=device, + stream=stream, + ) + return out + + @identity_impl.register_fake + def identity_impl_fake(x: torch.Tensor) -> torch.Tensor: + return torch.empty_like(x) +else: + + def identity_impl(*args, **kwargs) -> torch.Tensor: + raise ImportError( + "warp>=0.6.0 is required for the Warp identity implementation" + ) + +def identity_torch(x: torch.Tensor) -> torch.Tensor: + return x.clone() + +class Identity(FunctionSpec): + """Identity function with Warp and PyTorch backends.""" + + @FunctionSpec.register( + name="warp", + required_imports=("warp>=0.6.0",), + rank=0, + ) + def warp_forward(x: torch.Tensor) -> torch.Tensor: + return identity_impl(x) + + @FunctionSpec.register(name="torch", rank=1, baseline=True) + def torch_forward(x: torch.Tensor) -> torch.Tensor: + return identity_torch(x) + + @classmethod + def make_inputs(cls, device: torch.device | str = "cpu"): + device = torch.device(device) + yield ("small", (torch.randn(1024, device=device),), {}) + yield ("medium", (torch.randn(4096, device=device),), {}) + yield ("large", (torch.randn(16384, device=device),), {}) + + @classmethod + def compare(cls, output: torch.Tensor, reference: torch.Tensor) -> None: + torch.testing.assert_close(output, reference) + +identity = Identity.make_function("identity") + +x = torch.arange(8, device="cuda") +y = identity(x) +``` + +**Anti-pattern:** + +```python +def my_op(x): + return x +``` + +--- + +### FNC-001: Functional location and public API + +**Description:** + +Functionals live under `physicsnemo/nn/functional` and must be re-exported from +`physicsnemo/nn/functional/__init__.py`. + +**Rationale:** + +Keeping functionals in a single location makes them easy to discover and keeps +the public API consistent. + +**Example:** + +```python +# physicsnemo/nn/functional/__init__.py +from .knn import knn +__all__ = ["knn"] +``` + +**Anti-pattern:** + +```python +# Function defined in a random model module and not exported. +``` + +--- + +### FNC-002: File layout for functionals + +**Description:** + +- Single-file functionals go in `physicsnemo/nn/functional/.py`. +- When implementations get too large for a single file, use + `physicsnemo/nn/functional//`. + - Keep each backend in its own module (e.g., `_torch_impl.py`). + - Keep shared helpers in `utils.py`. + +**Rationale:** + +Separating backend-specific code keeps optional dependencies isolated and makes +maintenance easier. + +**Example:** + +```text +physicsnemo/nn/functional/knn/ + __init__.py + knn.py + _torch_impl.py + _cuml_impl.py + _scipy_impl.py + utils.py +``` + +**Anti-pattern:** + +```text +physicsnemo/nn/functional/knn.py # all backends mixed in one file +``` + +--- + +### FNC-003: Registration and dispatch rules + +**Description:** + +Use `@FunctionSpec.register` inside the class body for every implementation. +`rank` selects the default implementation (lower is preferred). Exactly one +implementation should be marked `baseline=True`. Baseline implementations are +usually the straight PyTorch backend. + +**Rationale:** + +Consistent registration and rank-based dispatch keep functional selection +predictable and debuggable. + +**Example:** + +```python +class MyOp(FunctionSpec): + @FunctionSpec.register(name="warp", rank=0) + def warp_forward(x): + return x + + @FunctionSpec.register(name="torch", rank=1, baseline=True) + def torch_forward(x): + return x +``` + +**Anti-pattern:** + +```python +def warp_forward(x): + return x +``` + +--- + +### FNC-004: Optional dependency handling + +**Description:** + +Backend modules must guard optional imports and expose a stub that raises a +clear `ImportError` when called if the dependency is missing. Do not raise at +import time. + +**Rationale:** + +Optional backends should not prevent importing the package or unrelated +functionals. + +**Example:** + +```python +if has_dep: + def knn_impl(...): + ... +else: + def knn_impl(*args, **kwargs): + raise ImportError("missing dependency") +``` + +**Anti-pattern:** + +```python +import missing_dep # raises at import time +``` + +--- + +### FNC-005: Benchmarking hooks + +**Description:** + +Implement `make_inputs` and `compare` for every functional. `make_inputs` should +yield labeled inputs ordered from smaller to larger cases. Labels do not have to +be exactly "small/medium/large", and you can provide more than three cases. +`compare` should validate output consistency. Labels are used for benchmark +plots and summaries. + +**Rationale:** + +This enables automated benchmarking, labeling, and correctness testing across +backends. + +**Example:** + +```python +@classmethod +def make_inputs(cls, device="cpu"): + yield ("small", (torch.randn(1024, device=device),), {}) + yield ("medium", (torch.randn(4096, device=device),), {}) + yield ("large", (torch.randn(16384, device=device),), {}) +``` + +**Anti-pattern:** + +```python +@classmethod +def make_inputs(cls, device="cpu"): + pass +``` + +--- + +### FNC-006: Testing functionals + +**Description:** + +Add tests under `test/nn/functional/` to validate selection, optional +dependencies, and output correctness. + +**Rationale:** + +Functional APIs are public entry points and need coverage for both the API and +backend behavior. + +**Example:** + +```python +def test_knn_cpu(): + indices, distances = knn(points, queries, k=4) +``` + +**Anti-pattern:** + +```python +# No tests for a new functional. +``` + +--- + +### FNC-007: Benchmark registry + +**Description:** + +Functionals that should be benchmarked must be added to +`benchmarks/physicsnemo/nn/functional/registry.py`. Only add a functional once +its `make_inputs` implementation yields labeled inputs. + +**Rationale:** + +Centralizing the benchmark list keeps ASV configuration minimal and ensures +every benchmarked functional provides the inputs and labels needed for +consistent plotting across small-to-large cases. + +**Example:** + +```python +from physicsnemo.nn.functional.knn.knn import KNN +from physicsnemo.nn.functional.radius_search.radius_search import RadiusSearch + +FUNCTIONAL_SPECS = (KNN, RadiusSearch) +``` + +**Anti-pattern:** + +```python +# Adding a functional before make_inputs is implemented. +FUNCTIONAL_SPECS = (MyFunctionalWithoutInputs,) +``` diff --git a/CODING_STANDARDS/MODELS_IMPLEMENTATION.md b/CODING_STANDARDS/MODELS_IMPLEMENTATION.md new file mode 100644 index 0000000000..85a600255f --- /dev/null +++ b/CODING_STANDARDS/MODELS_IMPLEMENTATION.md @@ -0,0 +1,1944 @@ + + + + + + + + + + + +# MODELS_IMPLEMENTATION - Coding Standards + +## Overview + +This document defines the coding standards and best practices for implementing +model classes in the PhysicsNeMo repository. These rules are designed to ensure +consistency, maintainability, and high code quality across all model +implementations. + +**Important:** These rules are enforced as strictly as possible. Deviations +from these standards should only be made when absolutely necessary and must be +documented with clear justification in code comments and approved during code +review. + +## Document Organization + +This document is structured in two main sections: + +1. **Rule Index**: A quick-reference table listing all rules with their IDs, + one-line summaries, and the context in which they apply. Use this section + to quickly identify relevant rules when implementing or reviewing code. + +2. **Detailed Rules**: Comprehensive descriptions of each rule, including: + - Clear descriptions of what the rule requires + - Rationale explaining why the rule exists + - Examples demonstrating correct implementation + - Anti-patterns showing common mistakes to avoid + +## How to Use This Document + +- **When creating new models**: Review all rules before starting implementation, + paying special attention to rules MOD-000 through MOD-003. +- **When reviewing code**: Use the Rule Index to quickly verify compliance with + all applicable rules. +- **When refactoring**: Ensure refactored code maintains or improves compliance + with these standards. +- **For AI agents that generate code**: This document is formatted for easy parsing. Each rule has + a unique ID and structured sections (Description, Rationale, Example, + Anti-pattern) that can be extracted and used as context. When generating code + based on a rule, an AI agent should explicitly quote the rule ID that it is + following, and explicitly quote the relevant extract from the rule that it is + using as context. For example, "Following rule MOD-000, the new model class + should be ..." +- **For AI agents that review code**: When reviewing code, the AI agent should + explicitly identify which rules are violated by the code, and provide a clear + explanation of why the code violates the rule. The AI agent should explicitly + quote the rule ID that the code is violating, and explicitly quote the relevant + extract from the rule that it is using as context. For example, "Code violates + rule MOD-000, because the new model class is not..." + +## Rule Index + +| Rule ID | Summary | Apply When | +|---------|---------|------------| +| [`MOD-000a`](#mod-000a-reusable-layersblocks-belong-in-physicsnemonn) | Reusable layers/blocks belong in physicsnemo.nn (stored in physicsnemo/nn/module) | Creating or refactoring reusable layer classes | +| [`MOD-000b`](#mod-000b-complete-models-belong-in-physicsnemomodels) | Complete models belong in physicsnemo.models | Creating or refactoring complete model classes | +| [`MOD-001`](#mod-001-use-physicsnemomodule-as-model-base-classes) | Use physicsnemo.Module as model base classes | Creating or refactoring new model classes | +| [`MOD-002a`](#mod-002a-new-models-and-layers-belong-in-physicsnemoexperimental) | New models and layers belong in physicsnemo.experimental | Creating new model or layer classes | +| [`MOD-002b`](#mod-002b-add-deprecation-warnings-to-deprecating-model-class) | Add deprecation warnings to deprecating model class | Deprecating existing model classes | +| [`MOD-002c`](#mod-002c-remove-deprecated-model-from-codebase) | Remove deprecated model from codebase | Removing deprecated models after warning period | +| [`MOD-003a`](#mod-003a-missing-or-incomplete-docstring-for-modellayer-code) | Missing or incomplete docstring for model/layer code | Creating or editing any model or layer code | +| [`MOD-003b`](#mod-003b-docstring-must-use-raw-string-prefix-r) | Docstring must use raw string prefix r""" | Writing any model or method docstring | +| [`MOD-003c`](#mod-003c-missing-required-class-docstring-sections) | Missing required class docstring sections | Writing class docstrings | +| [`MOD-003d`](#mod-003d-missing-required-method-docstring-sections) | Missing required method docstring sections | Writing method docstrings | +| [`MOD-003e`](#mod-003e-tensor-shapes-must-use-latex-math-notation) | Tensor shapes must use LaTeX math notation | Documenting tensors in docstrings | +| [`MOD-003f`](#mod-003f-callback-functions-must-have-code-block-specification) | Callback functions must have code-block specification | Documenting callback function parameters | +| [`MOD-003g`](#mod-003g-inline-code-must-use-double-backticks) | Inline code must use double backticks | Writing inline code in docstrings | +| [`MOD-003h`](#mod-003h-parameters-must-be-documented-on-single-line) | Parameters must be documented on single line | Documenting function/method parameters | +| [`MOD-003i`](#mod-003i-docstrings-should-include-cross-references) | Docstrings should include cross-references | Writing comprehensive docstrings | +| [`MOD-003j`](#mod-003j-docstrings-should-include-examples-section) | Docstrings should include Examples section | Writing model class docstrings | +| [`MOD-003k`](#mod-003k-add-high-level-comments-for-complex-tensor-operations) | Add high-level comments for complex tensor operations | Writing model code with complex tensor operations | +| [`MOD-004`](#mod-004-model-code-is-not-self-contained) | Model code is not self-contained | Organizing or refactoring model code | +| [`MOD-005`](#mod-005-invalid-or-missing-tensor-shape-validation-logic) | Invalid or missing tensor shape validation logic | Implementing model forward or public methods | +| [`MOD-006`](#mod-006-invalid-or-missing-jaxtyping-tensor-annotations-in-public-function-signature) | Invalid or missing jaxtyping tensor annotations in public function signature | Adding type hints to model methods | +| [`MOD-007a`](#mod-007a-cannot-add-required-parameters-without-defaults) | Cannot add required parameters without defaults | Modifying production model signatures | +| [`MOD-007b`](#mod-007b-cannot-remove-or-rename-parameters-without-compat-mapper) | Cannot remove or rename parameters without compat mapper | Modifying production model signatures | +| [`MOD-007c`](#mod-007c-cannot-change-return-types-of-public-methods) | Cannot change return types of public methods | Modifying production model method signatures | +| [`MOD-008a`](#mod-008a-model-missing-constructorattributes-tests) | Model missing constructor/attributes tests | Adding CI tests for models | +| [`MOD-008b`](#mod-008b-model-missing-non-regression-test-with-reference-data) | Model missing non-regression test with reference data | Adding CI tests for models | +| [`MOD-008c`](#mod-008c-model-missing-checkpoint-loading-test) | Model missing checkpoint loading test | Adding CI tests for models | +| [`MOD-009`](#mod-009-avoid-string-based-class-selection-in-model-constructors) | Avoid string-based class selection in model constructors | Designing model constructor APIs | +| [`MOD-010`](#mod-010-avoid-splatted-kwargs-in-model-constructors) | Avoid splatted kwargs in model constructors | Designing model constructor APIs | +| [`MOD-011`](#mod-011-use-proper-optional-dependency-handling) | Use proper optional dependency handling | Implementing models with optional dependencies | + +--- + +## Detailed Rules + +### MOD-000a: Reusable layers/blocks belong in physicsnemo.nn + +**Description:** + +Reusable layers that are the building blocks of more complex architectures +should live in `physicsnemo/nn/module` and be re-exported from +`physicsnemo/nn/__init__.py` so users can still import them from +`physicsnemo.nn`. Those include, for instance, `FullyConnected`, various variants +of attention layers, and `UNetBlock` (a block of a U-Net). + +All layers that are directly exposed to the user should be imported in +`physicsnemo/nn/__init__.py`, such that they can be used as follows: + +```python +from physicsnemo.nn import MyLayer +``` + +The only exception to this rule is for layers that are highly specific to a +single example. In this case, it may be acceptable to place them in a module +specific to the example code, such as `examples//utils/nn.py`. + +**Rationale:** + +Ensures consistency in the organization of reusable layers in the repository. +Keeping all reusable components in a single location makes them easy to find +and promotes code reuse across different models. + +**Example:** + +```python +# Good: Reusable layer in physicsnemo/nn/module/attention_layers.py +class MultiHeadAttention(Module): + """A reusable attention layer that can be used in various architectures.""" + pass + +# Good: Import in physicsnemo/nn/__init__.py +from physicsnemo.nn import MultiHeadAttention + +# Good: Example-specific layer in examples/weather/utils/nn.py +class WeatherSpecificLayer(Module): + """Layer highly specific to the weather forecasting example.""" + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Reusable layer placed in physicsnemo/models/ +# File: physicsnemo/models/attention.py +class MultiHeadAttention(Module): + """Should be in physicsnemo/nn/module/ not physicsnemo/models/""" + pass +``` + +--- + +### MOD-000b: Complete models belong in physicsnemo.models + +**Description:** + +More complete models, composed of multiple layers and/or other sub-models, +should go into `physicsnemo/models`. All models that are directly exposed to +the user should be imported in `physicsnemo/models/__init__.py`, such that they +can be used as follows: + +```python +from physicsnemo.models import MyModel +``` + +The only exception to this rule is for models that are highly specific to a +single example. In this case, it may be acceptable to place them in a module +specific to the example code, such as `examples//utils/nn.py`. + +**Rationale:** + +Ensures consistency and clarity in the organization of models in the repository, +in particular a clear separation between reusable layers and more complete +models that are applicable to a specific domain or specific data modality. + +**Example:** + +```python +# Good: Complete model in physicsnemo/models/transformer.py +class TransformerModel(Module): + """A complete transformer model composed of attention and feedforward layers.""" + def __init__(self): + super().__init__() + self.attention = MultiHeadAttention(...) + self.ffn = FeedForward(...) + +# Good: Import in physicsnemo/models/__init__.py +from physicsnemo.models.transformer import TransformerModel +``` + +**Anti-pattern:** + +```python +# WRONG: Complete model placed in physicsnemo/nn/module/ +# File: physicsnemo/nn/module/transformer_model.py +class TransformerModel(Module): + """Should be in physicsnemo/models/ not physicsnemo/nn/module/""" + pass +``` + +--- + +### MOD-001: Use physicsnemo.Module as model base classes + +**Description:** + +All model classes must inherit from `physicsnemo.Module`. Direct subclasses of +`torch.nn.Module` are not allowed. Direct subclasses of `physicsnemo.Module` +are allowed (note that `physicsnemo.Module` is a subclass of `torch.nn.Module`). +Ensure proper initialization of parent classes using `super().__init__()`. Pass +the `meta` argument to the `super().__init__()` call if appropriate, otherwise +set it manually with `self.meta = meta`. + +**Rationale:** +Ensures invariants and functionality of the `physicsnemo.Module` class for all +models. In particular, instances of `physicsnemo.Module` benefit from features +that are not available in `torch.nn.Module` instances. Those include serialization +for checkpointing and loading modules and submodules, versioning system to +handle backward compatibility, as well as ability to be registered in the +`physicsnemo.registry` for easy instantiation and use in any codebase. + +**Example:** + +```python +from physicsnemo import Module + +class MyModel(Module): + def __init__(self, input_dim: int, output_dim: int): + super().__init__(meta=MyModelMetaData()) + self.linear = nn.Linear(input_dim, output_dim) +``` + +**Anti-pattern:** + +```python +from torch import nn + +class MyModel(nn.Module): + def __init__(self, input_dim: int, output_dim: int): + self.linear = nn.Linear(input_dim, output_dim) +``` + +--- + +### MOD-002a: New models and layers belong in physicsnemo.experimental + +**Description:** + +For the vast majority of models, new classes are created in +`physicsnemo/experimental/models` (including reusable layers). The +`experimental` folder is used to store models that are still under development +(beta or alpha releases), where backward compatibility is not guaranteed. + +One exception is when the developer is highly confident that the model is +sufficiently mature and applicable to many domains or use cases. In this case +the model class can be created in `physicsnemo/nn/module` or +`physicsnemo/models` directly (and re-exported from `physicsnemo/nn`), and +backward compatibility is guaranteed. + +Another exception is when the model class is highly specific to a single +example. In this case, it may be acceptable to place it in a module specific to +the example code, such as `examples//utils/nn.py`. + +After staying in experimental for a sufficient amount of time (typically at +least 1 release cycle), the model class can be promoted to production. It is +then moved to `physicsnemo/nn/module` or `physicsnemo/models`, based on whether +it's a reusable layer (MOD-000a) or complete model (MOD-000b). During the +production stage, backward compatibility is guaranteed. + +**Note:** Per MOD-008a, MOD-008b, and MOD-008c, it is forbidden to move a model +out of the experimental stage/directory without the required CI tests. + +**Rationale:** + +The experimental stage allows rapid iteration without backward compatibility +constraints, enabling developers to refine APIs based on user feedback. This +protects users from unstable APIs while allowing innovation. + +**Example:** + +```python +# Good: New experimental model +# File: physicsnemo/experimental/models/new_diffusion.py +class DiffusionModel(Module): + """New diffusion model under active development. API may change.""" + pass + +# Good: After 1+ release cycles, promoted to production +# File: physicsnemo/models/diffusion.py (moved from experimental/) +class DiffusionModel(Module): + """Stable diffusion model with backward compatibility guarantees.""" + pass +``` + +**Anti-pattern:** + +```python +# WRONG: New model directly in production folder +# File: physicsnemo/models/brand_new_model.py (should be in experimental/ first) +class BrandNewModel(Module): + """Skipped experimental stage - risky for stability""" + pass +``` + +--- + +### MOD-002b: Add deprecation warnings to deprecating model class + +**Description:** + +For a model class being deprecated in `physicsnemo/nn/module` or +`physicsnemo/models`, the developer must add warning messages indicating that +the model class is +deprecated and will be removed in a future release. + +The warning message should be clear and concise, explaining why the model class +is being deprecated and what the user should do instead. The deprecation message +must be added to both: +1. The docstring using `.. deprecated::` directive +2. Runtime using `warnings.warn(..., DeprecationWarning)` + +The developer is free to choose the mechanism to raise the deprecation warning. +A model class cannot be deprecated without staying in the pre-deprecation stage +for at least 1 release cycle before it can be deleted (see MOD-002c). + +**Rationale:** + +Ensures users have sufficient time to migrate to newer alternatives, preventing +breaking changes that could disrupt their workflows. This graduated approach +balances innovation with stability. + +**Example:** + +```python +# Good: Pre-deprecation with proper warnings +# File: physicsnemo/models/old_diffusion.py +class DiffusionModel(Module): + """ + Legacy diffusion model. + + .. deprecated:: 0.5.0 + ``OldDiffusionModel`` is deprecated and will be removed in version 0.7.0. + Use :class:`~physicsnemo.models.NewDiffusionModel` instead. + """ + def __init__(self): + import warnings + warnings.warn( + "OldDiffusionModel is deprecated. Use NewDiffusionModel instead.", + DeprecationWarning, + stacklevel=2 + ) + super().__init__() +``` + +**Anti-pattern:** + +```python +# WRONG: No runtime warning +# File: physicsnemo/models/old_model.py +class OldModel(Module): + """Will be removed next release.""" # Docstring mentions it but no runtime warning + def __init__(self): + # Missing: warnings.warn(..., DeprecationWarning) + super().__init__() + +# WRONG: Deprecation without sufficient warning period +# (Model deprecated and removed in same release) +``` + +--- + +### MOD-002c: Remove deprecated model from codebase + +**Description:** + +After staying in the pre-deprecation stage for at least 1 release cycle, the +model class is considered deprecated and can be deleted from the codebase. + +A model class cannot be deleted without first spending at least 1 release cycle +in the pre-deprecation stage with proper deprecation warnings (see MOD-002b). + +**Rationale:** + +This ensures users have sufficient warning and time to migrate their code to +newer alternatives. Premature deletion of models would break user code without +adequate notice, violating the framework's commitment to stability. + +**Example:** + +```python +# Good: Proper deprecation timeline +# v0.5.0: Added deprecation warnings (Stage 3 - pre-deprecation) +# v0.6.0: Model can be safely removed (Stage 4 - deprecation) +# File: physicsnemo/models/old_diffusion.py - DELETED +``` + +**Anti-pattern:** + +```python +# WRONG: Deleting model without deprecation period +# v0.5.0: Model exists without warnings +# v0.6.0: Model deleted - BREAKS USER CODE! + +# WRONG: Breaking changes without deprecation +# File: physicsnemo/models/diffusion.py +class DiffusionModel(Module): + def __init__(self, new_required_param): # Breaking change! + # Changed API without deprecation warning - breaks user code + pass +``` + +--- + +### MOD-003a: Missing or incomplete docstring for model/layer code + +**Description:** + +Every new model or modification of any model code should be documented with a +comprehensive docstring following all the sub-rules MOD-003b through MOD-003k. +All docstrings should be written in the NumPy style and adopt formatting to be +compatible with our Sphinx restructured text (RST) documentation. + +**Rationale:** + +Comprehensive and well-formatted documentation is essential for scientific +software. It enables users to understand model capabilities, expected inputs, +and outputs without inspecting source code. + +**Example:** + +```python +class MyEncoder(Module): + r""" + A simple encoder network. + + Parameters + ---------- + input_dim : int + Dimension of input features. + output_dim : int + Dimension of output features. + + Forward + ------- + x : torch.Tensor + Input tensor of shape :math:`(B, D_{in})`. + + Outputs + ------- + torch.Tensor + Output tensor of shape :math:`(B, D_{out})`. + + Examples + -------- + >>> model = MyEncoder(input_dim=784, output_dim=128) + >>> x = torch.randn(32, 784) + >>> output = model(x) + >>> output.shape + torch.Size([32, 128]) + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Missing all required sections +class BadEncoder(Module): + '''A simple encoder.''' # Wrong quotes, no sections + pass +``` + +--- + +### MOD-003b: Docstring must use raw string prefix r""" + +**Description:** + +Each docstring should be prefixed with `r"""` (not `"""` or `'''`). The `r` +prefix creates a raw string that prevents Python from interpreting backslashes, +which is essential for LaTeX math notation to render correctly in Sphinx +documentation. + +**Rationale:** + +LaTeX commands in docstrings use backslashes (e.g., `\math`, `\text`). Without +the raw string prefix, Python interprets these as escape sequences, breaking the +documentation rendering. + +**Example:** + +```python +class MyModel(Module): + r""" + A model with LaTeX notation. + + Parameters + ---------- + dim : int + Dimension :math:`D` of input features. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Using ''' instead of r""" +class MyModel(Module): + ''' + A model with LaTeX notation. + ''' + pass +``` + +--- + +### MOD-003c: Missing required class docstring sections + +**Description:** + +The class docstring should at least contain three sections: `Parameters`, +`Forward`, and `Outputs`. The forward method should be documented in the +docstring of the model class, instead of being in the docstring of the forward +method itself. A docstring for the forward method is still possible but it +should be concise and to the point. + +Other sections such as `Notes`, `Examples`, or `..important::` or +`..code-block::python` +are possible. Other sections are not recognized by our Sphinx restructured text +(RST) documentation and are prohibited. + +**Rationale:** + +Standardized sections ensure documentation is consistent and complete across all +models. The Forward and Outputs sections in the class docstring provide a +centralized place to document the model's primary behavior, making it easier for +users to understand the model's API. + +**Example:** + +```python +class MyModel(Module): + r""" + A simple encoder model. + + Parameters + ---------- + input_dim : int + Dimension of input features. + + Forward + ------- + x : torch.Tensor + Input tensor of shape :math:`(B, D_{in})`. + + Outputs + ------- + torch.Tensor + Output tensor of shape :math:`(B, D_{out})`. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Missing required sections +class BadModel(Module): + r""" + A simple encoder model. + + No proper sections defined. + """ + pass +``` + +--- + +### MOD-003d: Missing required method docstring sections + +**Description:** + +All methods should be documented with a docstring, with at least a `Parameters` +section and a `Returns` section. Other sections such as `Notes`, `Examples`, or +`..important::` or `..code-block:: python` are possible. Other sections are not +recognized by our Sphinx documentation and are prohibited. + +Note: The forward method is a special case - its full documentation should be in +the class docstring (see MOD-003c), though a concise forward method docstring is +permitted. + +**Rationale:** + +Complete method documentation ensures users understand how to call methods and +what to expect in return. Standardized sections make documentation consistent +and easier to parse for both humans and AI agents. + +**Example:** + +```python +def compute_loss( + self, + pred: torch.Tensor, + target: torch.Tensor, +) -> torch.Tensor: + r""" + Compute mean squared error loss. + + Parameters + ---------- + pred : torch.Tensor + Predicted values of shape :math:`(B, D)`. + target : torch.Tensor + Target values of shape :math:`(B, D)`. + + Returns + ------- + torch.Tensor + Scalar loss value. + """ + return torch.nn.functional.mse_loss(pred, target) +``` + +**Anti-pattern:** + +```python +# WRONG: No docstring +def helper_method(self, x): + return x * 2 + +# WRONG: Using wrong section names +def compute_loss(self, pred, target): + """ + Args: + pred: predictions + Returns: + loss + """ + pass +``` + +--- + +### MOD-003e: Tensor shapes must use LaTeX math notation + +**Description:** + +All tensors should be documented with their shape, using LaTeX math notation +such as `:math:`(N, C, H_{in}, W_{in})``. There is flexibility for naming the +dimensions, but the math format should be enforced. + +Our documentation is rendered using LaTeX, and supports a rich set of LaTeX +commands, so it is recommended to use LaTeX commands whenever possible for +mathematical variables in the docstrings. The mathematical notations should be +to some degree consistent with the actual variable names in the code. + +**Rationale:** + +LaTeX math notation ensures tensor shapes render correctly and consistently in +Sphinx documentation. This is critical for scientific software where precise +mathematical notation is expected. Plain text shapes don't render properly and +can be ambiguous. + +**Example:** + +```python +def forward(self, x: torch.Tensor) -> torch.Tensor: + r""" + Process input tensor. + + Parameters + ---------- + x : torch.Tensor + Input of shape :math:`(B, C, H_{in}, W_{in})` where :math:`B` is batch + size, :math:`C` is channels, and :math:`H_{in}, W_{in}` are spatial dims. + + Returns + ------- + torch.Tensor + Output of shape :math:`(B, C_{out}, H_{out}, W_{out})`. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Not using :math: notation +def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Parameters + ---------- + x : torch.Tensor + Input of shape (B, C, H, W) # Missing :math:`...` + """ + pass +``` + +--- + +### MOD-003f: Callback functions must have code-block specification + +**Description:** + +For arguments or variables that are callback functions (e.g. Callable), the +docstring should include a clear separated `..code-block::` that specifies the +required signature and return type of the callback function. This is not only +true for callback functions, but for any type of parameters or arguments that +has some complex type specification or API requirements. + +The explanation code block should be placed in the top or bottom section of the +docstrings, but not in the `Parameters` or `Forward` or `Outputs` sections, for +readability and clarity. + +**Rationale:** + +Callback functions have complex type signatures that are difficult to express +clearly in the Parameters section alone. A dedicated code-block provides a clear +visual reference for the expected signature, making it much easier for users to +implement compatible callbacks. + +**Example:** + +```python +class MyModel(Module): + r""" + Model with callback function. + + .. code-block:: python + + def preprocess_fn(x: torch.Tensor) -> torch.Tensor: + '''Preprocessing function signature.''' + ... + return y + + where ``x`` is input of shape :math:`(B, D_{in})` and ``y`` is output + of shape :math:`(B, D_{out})`. + + Parameters + ---------- + preprocess_fn : Callable[[torch.Tensor], torch.Tensor], optional + Optional preprocessing function. See code block above for signature. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: No code-block specification +class MyModel(Module): + r""" + Parameters + ---------- + preprocess_fn : Callable, optional + Preprocessing function. # No specification! + """ + pass +``` + +--- + +### MOD-003g: Inline code must use double backticks + +**Description:** + +Inline code should be formatted with double backticks, such as ``my_variable``. +Single backticks are not allowed as they don't render properly in our Sphinx +documentation. + +**Rationale:** + +Sphinx uses reStructuredText, which requires double backticks for inline code +literals. Single backticks are interpreted differently and don't produce the +expected code formatting in the rendered documentation. + +**Example:** + +```python +class MyModel(Module): + r""" + Model with inline code references. + + If ``True``, enables dropout. Set ``model.training`` to control behavior. + The parameter ``hidden_dim`` controls layer size. + + Parameters + ---------- + hidden_dim : int + Size of hidden layer. Access via ``self.hidden_dim``. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Using single backticks +class MyModel(Module): + r""" + If `True`, enables dropout. # WRONG + """ + pass +``` + +--- + +### MOD-003h: Parameters must be documented on single line + +**Description:** + +All parameters should be documented with their type and default values on a +single line, following the NumPy docstring style format: + +``` +parameter_name : type, optional, default=value +``` + +The description then follows on the next line(s), indented. + +**Rationale:** + +This standardized format makes parameter documentation consistent and easy to +parse. It provides all key information (name, type, optionality, default) at a +glance, improving readability. + +**Example:** + +```python +class MyModel(Module): + r""" + Parameters + ---------- + input_dim : int + Dimension of input features. + hidden_dim : int, optional, default=128 + Dimension of hidden layer. + dropout : float, optional, default=0.1 + Dropout probability. + """ + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Type and default not on same line +class MyModel(Module): + r""" + Parameters + ---------- + hidden_dim : int + optional, default=128 # Should be on line above + Dimension of hidden layer. + """ + pass +``` + +--- + +### MOD-003i: Docstrings should include cross-references + +**Description:** + +When possible, docstrings should use links to other docstrings using Sphinx +cross-reference syntax: +- Classes: `:class:`~physicsnemo.models.some_model.SomeModel`` +- Functions: `:func:`~physicsnemo.utils.common_function`` +- Methods: `:meth:`~physicsnemo.models.some_model.SomeModel.some_method`` + +When referencing external resources, such as papers, websites, or other +documentation, docstrings should use links to the external resource in the +format `some link text `_. + +**Rationale:** + +Cross-references create a navigable documentation structure where users can +easily jump between related classes, methods, and functions. External links +provide context and attribution for algorithms and techniques. + +**Example:** + +```python +class MyEncoder(Module): + r""" + Encoder using attention. + + Based on `Transformer Architecture `_. + See :class:`~physicsnemo.nn.MultiHeadAttention` for attention details. + + Parameters + ---------- + activation : str + Activation function. See :func:`~torch.nn.functional.relu`. + """ + pass +``` + +**Anti-pattern:** + +```python +# Not wrong, but missing opportunities for useful links +class MyEncoder(Module): + r""" + Uses MultiHeadAttention. # Could link to class + Based on Transformer paper. # Could link to paper + """ + pass +``` + +--- + +### MOD-003j: Docstrings should include Examples section + +**Description:** + +Docstrings are strongly encouraged to have an `Examples` section that +demonstrates basic construction and usage of the model. These example sections +serve as both documentation and tests, as our CI system automatically tests +these code sections for correctness when present. + +Examples should be executable Python code showing typical use cases, including +model instantiation, input preparation, and forward pass execution. The examples +should use realistic tensor shapes and demonstrate key features of the model. + +**Rationale:** + +Example sections provide immediate value to users by showing concrete usage +patterns. By automatically testing these examples in CI, we ensure that +documentation stays synchronized with code and that examples remain correct as +the codebase evolves. + +**Example:** + +```python +class MyEncoder(Module): + r""" + A simple encoder network. + + Parameters + ---------- + input_dim : int + Dimension of input features. + + Examples + -------- + >>> import torch + >>> from physicsnemo.models import MyEncoder + >>> model = MyEncoder(input_dim=784, output_dim=128) + >>> x = torch.randn(32, 784) + >>> output = model(x) + >>> output.shape + torch.Size([32, 128]) + """ + pass +``` + +**Anti-pattern:** + +```python +# Not wrong, but discouraged - no Examples section +class MyEncoder(Module): + r""" + Parameters + ---------- + input_dim : int + Dimension of input features. + """ + pass +``` + +--- + +### MOD-003k: Add high-level comments for complex tensor operations + +**Description:** + +Model code that involves complex tensor operations should include high-level +comments that explain what blocks of code accomplish semantically. One-line +comments every few lines of tensor operations is sufficient. + +Comments should focus on high-level semantic explanations rather than low-level +syntactic details. For example, use "Compute the encodings" instead of "Doing a +concatenation followed by a linear projection, followed by a nonlinear +activation". The goal is to give a high-level overview of what a block of tensor +operations accomplishes. + +When multiple tensor operations are chained, it is welcomed to add short inline +comments with the tensor shapes of computed tensors, e.g.: + +```python +x = torch.cat([y, z], dim=1) # (B, 2*C_in, H, W) +``` + +The symbols chosen in the comments should be consistent with the docstring +(possibly shortened versions of dimension names for explicitness). + +**Rationale:** + +High-level comments make complex tensor manipulation code more understandable +without cluttering it with excessive detail. Shape annotations help developers +track tensor dimensions through complex operations, catching shape mismatches +early. Consistency with docstring notation creates a unified mental model. + +**Example:** + +```python +def forward(self, x: torch.Tensor, context: torch.Tensor) -> torch.Tensor: + """Process input with context conditioning.""" + # Encode input features + h = self.encoder(x) # (B, C_enc, H, W) + + # Combine with context information + c = self.context_proj(context) # (B, C_enc) + c = c[:, :, None, None].expand(-1, -1, h.shape[2], h.shape[3]) # (B, C_enc, H, W) + h = torch.cat([h, c], dim=1) # (B, 2*C_enc, H, W) + + # Apply attention mechanism + h = self.attention(h) # (B, 2*C_enc, H, W) + + # Decode to output + out = self.decoder(h) # (B, C_out, H, W) + + return out +``` + +**Anti-pattern:** + +```python +# WRONG: No comments +def forward(self, x: torch.Tensor, context: torch.Tensor) -> torch.Tensor: + h = self.encoder(x) + c = self.context_proj(context) + c = c[:, :, None, None].expand(-1, -1, h.shape[2], h.shape[3]) + h = torch.cat([h, c], dim=1) + return self.decoder(self.attention(h)) + +# WRONG: Too low-level, syntactic comments +def forward(self, x, context): + # Pass x through encoder layer + h = self.encoder(x) + # Project context using linear layer + c = self.context_proj(context) + # Add two None dimensions and expand + c = c[:, :, None, None].expand(-1, -1, h.shape[2], h.shape[3]) +``` + +--- + +### MOD-004: Model code is not self-contained + +**Description:** + +All utility functions for a model class should be organized together with the +model class in a clear and logical structure. Acceptable patterns include: + +1. A single self-contained file: `physicsnemo//model_name.py` +2. A subdirectory: `physicsnemo//model_name/` containing: + - `model_name.py` with the main model class + - Additional modules for utility functions specific to this model + +What should be avoided is a flat organization where model files and their +utility files are all mixed together in `physicsnemo//`, making it +unclear which utilities belong to which models. + +The only exception is when a utility function is used across multiple models. In +that case, the shared utility should be placed in an appropriate shared module. + +**Rationale:** + +Self-contained modules are easier to understand, maintain, and navigate. Having +all model-specific code in one place reduces cognitive load and makes it clear +which utilities are model-specific versus shared. This also simplifies code +reviews and reduces the likelihood of orphaned utility files when models are +refactored or removed. + +**Example:** + +```python +# Good Pattern 1: Single self-contained file +# File: physicsnemo/models/my_simple_model.py + +def _compute_attention_mask(seq_length: int) -> torch.Tensor: + """Helper function specific to MySimpleModel.""" + mask = torch.triu(torch.ones(seq_length, seq_length), diagonal=1) + return mask.masked_fill(mask == 1, float('-inf')) + +class MySimpleModel(Module): + """A simple model with utilities in same file.""" + def forward(self, x: torch.Tensor) -> torch.Tensor: + mask = _compute_attention_mask(x.shape[1]) + return self._apply_attention(x, mask) + +# Good Pattern 2: Subdirectory organization +# File: physicsnemo/models/my_complex_model/my_complex_model.py +from physicsnemo.models.my_complex_model.utils import helper_function + +class MyComplexModel(Module): + """A complex model with utilities in subdirectory.""" + pass + +# File: physicsnemo/models/my_complex_model/utils.py +def helper_function(x): + """Utility specific to MyComplexModel.""" + pass +``` + +**Anti-pattern:** + +```python +# WRONG: Flat organization with utilities mixed in main directory +# File: physicsnemo/models/my_transformer.py +from physicsnemo.models.my_transformer_utils import _compute_mask # WRONG + +class MyTransformer(Module): + pass + +# File: physicsnemo/models/my_transformer_utils.py (WRONG: mixed with other models) +# File: physicsnemo/models/other_model.py +# File: physicsnemo/models/other_model_utils.py (WRONG: utilities scattered) +# All mixed together in flat structure - unclear organization! +``` + +--- + +### MOD-005: Invalid or missing tensor shape validation logic + +**Description:** + +All forward methods and other public methods that accept tensor arguments must +validate tensor shapes at the beginning of the method. This rule applies to: +- Individual tensor arguments +- Containers of tensors (lists, tuples, dictionaries) + +For containers, validate their length, required keys, and the shapes of +contained tensors. Validation statements should be concise (ideally one check +per argument). Error messages must follow the standardized format: +`"Expected tensor of shape (B, D) but got tensor of shape {actual_shape}"`. + +To avoid interactions with `torch.compile`, all validation must be wrapped in a +conditional check using `torch.compiler.is_compiling()`. Follow the "fail-fast" +approach by validating inputs before any computation. + +**Rationale:** + +Early shape validation catches errors at the API boundary with clear, actionable +error messages, making debugging significantly easier. Without validation, shape +mismatches result in cryptic errors deep in the computation graph. The +`torch.compile` guard ensures that validation overhead is eliminated in +production compiled code while preserving debug-time safety. + +**Example:** + +```python +def forward(self, x: torch.Tensor, mask: Optional[torch.Tensor] = None) -> torch.Tensor: + """Forward pass with shape validation.""" + ### Input validation + # Skip validation when running under torch.compile for performance + if not torch.compiler.is_compiling(): + # Extract expected dimensions + B, C, H, W = x.shape if x.ndim == 4 else (None, None, None, None) + + # Validate x shape + if x.ndim != 4: + raise ValueError( + f"Expected 4D input tensor (B, C, H, W), got {x.ndim}D tensor with shape {tuple(x.shape)}" + ) + + if C != self.in_channels: + raise ValueError( + f"Expected {self.in_channels} input channels, got {C} channels" + ) + + # Validate optional mask + if mask is not None: + if mask.shape != (B, H, W): + raise ValueError( + f"Expected mask shape ({B}, {H}, {W}), got {tuple(mask.shape)}" + ) + + # Actual computation happens after validation + return self._process(x, mask) + +def process_list(self, tensors: List[torch.Tensor]) -> torch.Tensor: + """Process a list of tensors with validation.""" + ### Input validation + if not torch.compiler.is_compiling(): + if len(tensors) == 0: + raise ValueError("Expected non-empty list of tensors") + + # Validate all tensors have consistent shapes + ref_shape = tensors[0].shape + for i, t in enumerate(tensors[1:], start=1): + if t.shape != ref_shape: + raise ValueError( + f"All tensors must have the same shape. " + f"Tensor 0 has shape {tuple(ref_shape)}, " + f"but tensor {i} has shape {tuple(t.shape)}" + ) + + return torch.stack(tensors) +``` + +**Anti-pattern:** + +```python +# WRONG: No validation at all +def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.layer(x) # Will fail with cryptic error if shape is wrong + +# WRONG: Validation not guarded by torch.compiler.is_compiling() +def forward(self, x: torch.Tensor) -> torch.Tensor: + if x.ndim != 4: # Breaks torch.compile + raise ValueError(f"Expected 4D tensor, got {x.ndim}D") + return self.layer(x) + +# WRONG: Validation after computation has started +def forward(self, x: torch.Tensor) -> torch.Tensor: + h = self.layer1(x) # Computation started + if x.shape[1] != self.in_channels: # Too late! + raise ValueError(f"Wrong number of channels") + return self.layer2(h) + +# WRONG: Non-standard error message format +def forward(self, x: torch.Tensor) -> torch.Tensor: + if not torch.compiler.is_compiling(): + if x.ndim != 4: + raise ValueError("Input must be 4D") # Missing actual shape info + return self.layer(x) +``` + +--- + +### MOD-006: Invalid or missing jaxtyping tensor annotations in public function signature + +**Description:** + +All tensor arguments and variables in model `__init__`, `forward`, and other +public methods must have type annotations using `jaxtyping`. This provides +runtime-checkable shape information in type hints. + +Use the format `Float[torch.Tensor, "shape_spec"]` where shape_spec describes +tensor dimensions using space-separated dimension names (e.g., `"batch channels height width"` +or `"b c h w"`). + +**Rationale:** + +Jaxtyping annotations provide explicit, machine-readable documentation of +expected tensor shapes. This enables better IDE support, catches shape errors +earlier, and makes code more self-documenting. The annotations serve as both +documentation and optional runtime checks when jaxtyping's validation is +enabled. + +**Example:** + +```python +from jaxtyping import Float +import torch + +class MyConvNet(Module): + def __init__(self, in_channels: int, out_channels: int): + super().__init__() + self.conv = torch.nn.Conv2d(in_channels, out_channels, kernel_size=3) + + def forward( + self, + x: Float[torch.Tensor, "batch in_channels height width"] + ) -> Float[torch.Tensor, "batch out_channels height width"]: + """Process input with convolution.""" + return self.conv(x) + +def process_attention( + query: Float[torch.Tensor, "batch seq_len d_model"], + key: Float[torch.Tensor, "batch seq_len d_model"], + value: Float[torch.Tensor, "batch seq_len d_model"] +) -> Float[torch.Tensor, "batch seq_len d_model"]: + """Compute attention with clear shape annotations.""" + pass +``` + +**Anti-pattern:** + +```python +# WRONG: No jaxtyping annotations +def forward(self, x: torch.Tensor) -> torch.Tensor: + return self.layer(x) + +# WRONG: Using plain comments instead of jaxtyping +def forward(self, x: torch.Tensor) -> torch.Tensor: + # x: (batch, channels, height, width) # Use jaxtyping instead + return self.layer(x) + +# WRONG: Incomplete annotations (missing jaxtyping for tensor arguments) +def forward( + self, + x: Float[torch.Tensor, "b c h w"], + mask: torch.Tensor # Missing jaxtyping annotation +) -> Float[torch.Tensor, "b c h w"]: + return self.layer(x, mask) +``` + +--- + +### MOD-007a: Cannot add required parameters without defaults + +**Description:** + +For any model in `physicsnemo/nn/module` or `physicsnemo/models`, adding new +required parameters (parameters without default values) to `__init__` or any +public method is strictly forbidden. This breaks backward compatibility. + +New parameters must have default values to ensure existing code and checkpoints +continue to work. If a new parameter is truly required, increment the model +version number using `__model_checkpoint_version__` and add appropriate +versioning support. + +**Rationale:** + +Adding required parameters breaks all existing code that instantiates the model, +and breaks loading of old checkpoints. This violates PhysicsNeMo's commitment to +backward compatibility and would disrupt user workflows. + +**Example:** + +```python +# Good: Adding parameter with default value (backward compatible) +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + __supported_model_checkpoint_version__ = { + "1.0": "Loading checkpoint from version 1.0 (current is 2.0). Still supported." + } + + def __init__( + self, + input_dim: int, + output_dim: int, + dropout: float = 0.0, # New parameter with default - backward compatible + new_feature: bool = False # New parameter with default - backward compatible + ): + super().__init__(meta=MyModelMetaData()) +``` + +**Anti-pattern:** + +```python +# WRONG: Adding required parameter without default +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + + def __init__( + self, + input_dim: int, + output_dim: int, + new_param: int # WRONG: No default! Breaks old checkpoints + ): + super().__init__(meta=MyModelMetaData()) +``` + +--- + +### MOD-007b: Cannot remove or rename parameters without compat mapper + +**Description:** + +For any model in `physicsnemo/nn/module` or `physicsnemo/models`, removing or +renaming parameters is strictly forbidden without proper backward compatibility +support. + +If a parameter must be renamed or removed, the developer must: +1. Increment `__model_checkpoint_version__` +2. Add the old version to `__supported_model_checkpoint_version__` dict +3. Implement `_backward_compat_arg_mapper` classmethod to handle the mapping +4. Maintain support for the old API for at least 2 release cycles + +**Rationale:** + +Removing or renaming parameters breaks existing checkpoints and user code. +Proper version management and argument mapping ensures old checkpoints can still +be loaded and users have time to migrate to the new API. + +**Example:** + +```python +from typing import Any, Dict + +# Good: Proper backward compatibility for parameter rename +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + __supported_model_checkpoint_version__ = { + "1.0": ( + "Loading checkpoint from version 1.0 (current is 2.0). " + "Parameter 'hidden_dim' renamed to 'hidden_size'." + ) + } + + @classmethod + def _backward_compat_arg_mapper( + cls, version: str, args: Dict[str, Any] + ) -> Dict[str, Any]: + """Map arguments from older versions.""" + args = super()._backward_compat_arg_mapper(version, args) + + if version == "1.0": + # Map old parameter name to new name + if "hidden_dim" in args: + args["hidden_size"] = args.pop("hidden_dim") + + # Remove deprecated parameters + if "legacy_param" in args: + _ = args.pop("legacy_param") + + return args + + def __init__( + self, + input_dim: int, + hidden_size: int = 128, # Renamed from 'hidden_dim' + ): + super().__init__(meta=MyModelMetaData()) +``` + +**Anti-pattern:** + +```python +# WRONG: Renaming without backward compat +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + # Missing: __supported_model_checkpoint_version__ and _backward_compat_arg_mapper + + def __init__(self, input_dim: int, hidden_size: int): # Renamed! + super().__init__(meta=MyModelMetaData()) + # WRONG: Old checkpoints with 'hidden_dim' will fail! + +# WRONG: Not calling super() +class MyModel(Module): + @classmethod + def _backward_compat_arg_mapper(cls, version: str, args: Dict[str, Any]) -> Dict[str, Any]: + # WRONG: Missing super()._backward_compat_arg_mapper(version, args) + if "hidden_dim" in args: + args["hidden_size"] = args.pop("hidden_dim") + return args +``` + +--- + +### MOD-007c: Cannot change return types of public methods + +**Description:** + +For any model in `physicsnemo/nn/module` or `physicsnemo/models`, changing the return +type of any public method (including `forward`) is strictly forbidden. This +includes: +- Changing from returning a single value to returning a tuple +- Changing from a tuple to a single value +- Changing the number of elements in a returned tuple +- Changing the type of returned values + +If a return type change is absolutely necessary, create a new method with a +different name and deprecate the old method following MOD-002b. + +**Rationale:** + +Changing return types is a breaking change that silently breaks user code. Users +who unpack return values or depend on specific return structures will experience +runtime errors. Unlike parameter changes (which can be managed with versioning), +return type changes affect runtime behavior and are harder to detect. + +**Example:** + +```python +# Good: Keeping consistent return type +class MyModel(Module): + def forward(self, x: torch.Tensor) -> torch.Tensor: + """Always returns single tensor.""" + return self.process(x) + +# Good: If new return is needed, add new method +class MyModel(Module): + def forward(self, x: torch.Tensor) -> torch.Tensor: + """Returns output tensor.""" + output, loss = self._forward_with_loss(x) + return output + + def forward_with_loss(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + """New method for returning both output and loss.""" + return self._forward_with_loss(x) +``` + +**Anti-pattern:** + +```python +# WRONG: Changing return type +class MyModel(Module): + __model_checkpoint_version__ = "2.0" + + def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]: + # WRONG: v1.0 returned single tensor, v2.0 returns tuple - breaks user code! + return output, loss +``` + +--- + +### MOD-008a: Model missing constructor/attributes tests + +**Description:** + +Every model in `physicsnemo/nn/module` or `physicsnemo/models` must have tests that +verify model instantiation and all public attributes (excluding buffers and +parameters). + +These tests should: +- Use `pytest` parameterization to test at least 2 configurations +- Test one configuration with all default arguments +- Test another configuration with non-default arguments +- Verify all public attributes have expected values + +**Rationale:** + +Constructor tests ensure the model can be instantiated correctly with various +configurations and that all attributes are properly initialized. This catches +issues early in the development cycle. + +**Example:** + +```python +@pytest.mark.parametrize( + "config", + ["default", "custom"], + ids=["with_defaults", "with_custom_args"] +) +def test_my_model_constructor(config): + """Test model constructor and attributes.""" + if config == "default": + model = MyModel(input_dim=64, output_dim=32) + assert model.hidden_dim == 128 # Default value + assert model.dropout == 0.0 # Default value + else: + model = MyModel( + input_dim=64, + output_dim=32, + hidden_dim=256, + dropout=0.1 + ) + assert model.hidden_dim == 256 + assert model.dropout == 0.1 + + # Test common attributes + assert model.input_dim == 64 + assert model.output_dim == 32 +``` + +**Anti-pattern:** + +```python +# WRONG: Only testing default configuration +def test_my_model_bad(): + model = MyModel(input_dim=64, output_dim=32) + # Only tests defaults +``` + +--- + +### MOD-008b: Model missing non-regression test with reference data + +**Description:** + +Every model must have non-regression tests that: +1. Instantiate the model with reproducible random parameters +2. Run forward pass with test data +3. Compare outputs against reference data saved in a `.pth` file + +Requirements: +- Use `pytest` parameterization to test multiple configurations +- Test tensors must have realistic shapes (no singleton dimensions except batch) +- Test data should be meaningful and representative of actual use cases +- Compare actual tensor values, not just shapes +- All public methods (not just forward) need similar non-regression tests + +**Critical:** Per MOD-002a, models cannot move out of experimental without these +tests. + +**Rationale:** + +Non-regression tests with reference data catch subtle numerical changes that +could break reproducibility. Simply checking output shapes is insufficient to +detect algorithmic changes or numerical instabilities. + +**Example:** + +```python +import pytest +import torch +from physicsnemo.models import MyModel + +def _instantiate_model(cls, seed: int = 0, **kwargs): + """Helper to create model with reproducible parameters.""" + model = cls(**kwargs) + gen = torch.Generator(device="cpu") + gen.manual_seed(seed) + with torch.no_grad(): + for param in model.parameters(): + param.copy_(torch.randn(param.shape, generator=gen, dtype=param.dtype)) + return model + +@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +@pytest.mark.parametrize("config", ["default", "custom"]) +def test_my_model_non_regression(device, config): + """Test model forward pass against reference output.""" + if config == "default": + model = _instantiate_model(MyModel, input_dim=64, output_dim=32) + else: + model = _instantiate_model( + MyModel, + input_dim=64, + output_dim=32, + hidden_dim=256 + ) + + model = model.to(device) + + # Load reference data (meaningful shapes, no singleton dimensions) + data = torch.load(f"test/models/data/my_model_{config}_v1.0.pth") + x = data["x"].to(device) # Shape: (4, 64), not (1, 64) + out_ref = data["out"].to(device) + + # Run forward and compare values + out = model(x) + assert torch.allclose(out, out_ref, atol=1e-5, rtol=1e-5) +``` + +**Anti-pattern:** + +```python +# WRONG: Only testing output shapes +def test_my_model_bad(device): + model = MyModel(input_dim=64, output_dim=32).to(device) + x = torch.randn(4, 64).to(device) + out = model(x) + assert out.shape == (4, 32) # NOT SUFFICIENT! + +# WRONG: Using singleton dimensions +def test_my_model_bad(device): + x = torch.randn(1, 1, 64) # WRONG: Trivial shapes +``` + +--- + +### MOD-008c: Model missing checkpoint loading test + +**Description:** + +Every model must have tests that load the model from a checkpoint file +(`.mdlus`) using `physicsnemo.Module.from_checkpoint()` and verify that: +1. The model loads successfully +2. All public attributes have expected values +3. Forward pass outputs match reference data + +This ensures the model's serialization and deserialization work correctly. + +**Critical:** Per MOD-002a, models cannot move out of experimental without these +tests. + +**Rationale:** + +Checkpoint tests verify that the model's custom serialization logic works +correctly and that saved models can be loaded in different environments. This is +critical for reproducibility and for users who need to save and load trained +models. + +**Example:** + +```python +@pytest.mark.parametrize("device", ["cuda:0", "cpu"]) +def test_my_model_from_checkpoint(device): + """Test loading model from checkpoint and verify outputs.""" + model = physicsnemo.Module.from_checkpoint( + "test/models/data/my_model_default_v1.0.mdlus" + ).to(device) + + # Verify attributes after loading + assert model.input_dim == 64 + assert model.output_dim == 32 + + # Load reference data and verify outputs + data = torch.load("test/models/data/my_model_default_v1.0.pth") + x = data["x"].to(device) + out_ref = data["out"].to(device) + out = model(x) + assert torch.allclose(out, out_ref, atol=1e-5, rtol=1e-5) +``` + +**Anti-pattern:** + +```python +# WRONG: No checkpoint loading test +# (Missing test_my_model_from_checkpoint entirely) +``` + +--- + +### MOD-009: Avoid string-based class selection in model constructors + +**Description:** + +Passing a string that represents a class name, which is then used to instantiate +an internal submodule, should be avoided unless there are only a few choices (2 +or 3 maximum) for the class name. + +When there are more than 2-3 choices, the recommended practice is to pass an +already instantiated instance of a submodule instead of a string primitive for +dependency injection. This promotes better type safety, clearer APIs, and easier +testing. + +**Rationale:** + +String-based class selection makes code harder to type-check, debug, and test. +It obscures dependencies and makes it difficult for static analysis tools to +understand the code. Direct instance injection provides better IDE support, +type safety, and makes testing easier by allowing mock object injection. + +**Example:** + +```python +# Good: Limited choices (2-3 max) - string selection acceptable +class MyModel(Module): + def __init__( + self, + activation: Literal["relu", "gelu"] = "relu" + ): + if activation == "relu": + self.act = nn.ReLU() + elif activation == "gelu": + self.act = nn.GELU() + +# Good: Many choices - use instance injection +class MyModel(Module): + def __init__( + self, + encoder: Module, # Pass instance, not string + decoder: Module # Pass instance, not string + ): + self.encoder = encoder + self.decoder = decoder + +# Usage: +model = MyModel( + encoder=MyCustomEncoder(dim=128), + decoder=MyCustomDecoder(dim=128) +) +``` + +**Anti-pattern:** + +```python +# WRONG: String selection with many choices +class MyModel(Module): + def __init__( + self, + encoder_type: str = "transformer" # Many possible values + ): + # String-based factory pattern with 10+ choices + if encoder_type == "transformer": + self.encoder = TransformerEncoder() + elif encoder_type == "cnn": + self.encoder = CNNEncoder() + elif encoder_type == "rnn": + self.encoder = RNNEncoder() + # ... many more options + # WRONG: Should accept encoder instance instead +``` + +--- + +### MOD-010: Avoid splatted kwargs in model constructors + +**Description:** + +Passing splatted arguments like `**kwargs_for_submodules` should be avoided in +model constructors as it might create conflicts in the names of these kwargs and +makes the API unclear. + +Instead, it is recommended to pass non-splatted arguments in the form of a +`Dict` when configuration for submodules needs to be passed through. This makes +parameter passing explicit and avoids naming conflicts. + +**Rationale:** + +Splatted kwargs obscure the actual parameters being passed, make type checking +impossible, and can lead to subtle bugs from name conflicts. Explicit dictionary +parameters make the API clearer and enable better IDE support and error +detection. + +**Example:** + +```python +# Good: Explicit dict parameter +class MyModel(Module): + def __init__( + self, + input_dim: int, + output_dim: int, + encoder_config: Optional[Dict[str, Any]] = None + ): + encoder_config = encoder_config or {} + self.encoder = Encoder(input_dim=input_dim, **encoder_config) + +# Usage: +model = MyModel( + input_dim=64, + output_dim=32, + encoder_config={"hidden_dim": 128, "num_layers": 3} +) +``` + +**Anti-pattern:** + +```python +# WRONG: Splatted kwargs +class MyModel(Module): + def __init__( + self, + input_dim: int, + output_dim: int, + **encoder_kwargs # WRONG: Unclear what's accepted + ): + self.encoder = Encoder(input_dim=input_dim, **encoder_kwargs) + # Risk of name conflicts, unclear API + +# Usage - unclear what parameters are valid: +model = MyModel(input_dim=64, output_dim=32, hidden_dim=128, num_layers=3) +# Are hidden_dim and num_layers for MyModel or Encoder? Unclear! +``` + +--- + +### MOD-011: Use proper optional dependency handling + +**Description:** + +When a model requires optional dependencies (packages not installed by default), +use the PhysicsNeMo APIs for dependency handling: + +1. **`check_min_version(package, version, hard_fail=False)`**: Use this function + to check if a package is installed and available without actually importing + it. Set `hard_fail=True` for hard requirements, `hard_fail=False` for soft + requirements. This is the primary method for handling optional dependencies. + +2. **`@require_version(package, version)`**: Use this decorator when core code + must always be available but certain features need to be protected against + older versions. This is rare and should only be used when you need to protect + specific methods or classes against version incompatibilities. + +3. **`pyproject.toml`**: This file is the one, only, and universal source of + truth for all dependencies in PhysicsNeMo. All optional dependencies must be + declared there. + +**Rationale:** + +Centralized dependency handling ensures consistent error messages and version +checking across the codebase. Checking availability without importing prevents +import errors and allows graceful degradation when optional packages are not +available. Using `pyproject.toml` as the single source of truth prevents +dependency specification from becoming scattered and inconsistent. + +**Example:** + +```python +import torch +from physicsnemo.core import Module +from physicsnemo.core.version_check import check_min_version, require_version + +# Check optional dependency availability without importing +APEX_AVAILABLE = check_min_version("apex", "0.1.0", hard_fail=False) + +class MyModel(Module): + def __init__( + self, + input_dim: int, + use_apex: bool = False + ): + super().__init__() + self.use_apex = use_apex + + if use_apex and not APEX_AVAILABLE: + raise RuntimeError( + "apex is required for use_apex=True but is not installed. " + "Install with: pip install apex>=0.1.0" + ) + + if use_apex: + import apex # Only import when actually needed + self.fused_layer = apex.FusedLayer() + else: + self.fused_layer = None + +# Using @require_version for protecting version-specific features +class AdvancedModel(Module): + @require_version("torch", "2.4.0") + def use_device_mesh(self): + """This feature requires torch>=2.4.0.""" + from torch.distributed.device_mesh import DeviceMesh + # Protected code that needs torch>=2.4.0 +``` + +**Anti-pattern:** + +```python +# WRONG: Direct import without checking availability +import apex # Will fail if apex not installed! + +class MyModel(Module): + def __init__(self, use_apex: bool = False): + if use_apex: + self.layer = apex.FusedLayer() # Already failed at import! + +# WRONG: Try/except for dependency checking +try: + import apex + APEX_AVAILABLE = True +except ImportError: + APEX_AVAILABLE = False +# Use check_min_version instead! + +# WRONG: Hardcoded version strings in multiple places +if version.parse(apex.__version__) < version.parse("0.1.0"): + raise ImportError("apex>=0.1.0 required") +# Should use check_min_version or require_version! + +# WRONG: Not declaring dependency in pyproject.toml +# All optional dependencies must be in pyproject.toml! +``` + +--- + +## Compliance + +When implementing models, ensure all rules are followed. Code reviews should +verify each rule is followed and enforce the rules as strictly as possible. +For exceptions to these rules, document the reasoning in code comments and +obtain approval during code review. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 46b8873402..4f1fe46c98 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,10 +1,10 @@ -# Modulus (Beta) Contribution Guide +# PhysicsNeMo Contribution Guide ## Introduction -Welcome to Project Modulus! We're excited you're here and want to contribute. +Welcome to Project PhysicsNeMo! We're excited you're here and want to contribute. This documentation is intended for individuals and institutions interested in -contributing to Modulus. Modulus is an open-source project and, as such, its +contributing to PhysicsNeMo. PhysicsNeMo is an open-source project and, as such, its success relies on its community of contributors willing to keep improving it. Your contribution will be a valued addition to the code base; we simply ask that you read this page and understand our contribution process, whether you @@ -13,27 +13,31 @@ contributor. ### Communicate with Us -We are happy to talk with you about your needs for Modulus and your ideas for +We are happy to talk with you about your needs for PhysicsNeMo and your ideas for contributing to the project. One way to do this is to create an issue discussing your thoughts. It might be that a very similar feature is under development or already exists, so an issue is a great starting point. If you are looking for an issue to resolve that will help, refer to the -[issue](https://github.com/NVIDIA/modulus/issues) section. +[issue](https://github.com/NVIDIA/physicsnemo/issues) section. +If you are considering collaborating with NVIDIA PhysicsNeMo team to enhance PhysicsNeMo, +fill this [proposal form](https://forms.gle/fYsbZEtgRWJUQ3oQ9) and +we will get back to you. -We are happy to talk with you about your needs for Modulus and your ideas for -contributing to the project. One way to do this is to create an issue discussing your -thoughts. It might be that a very similar feature is under development or already -exists, so an issue is a great starting point. If you are looking for an issue to -resolve that will help, refer to the [issue](https://github.com/NVIDIA/modulus/issues) section. +## Contribute to PhysicsNeMo-Core -## Contribute to Modulus-Core +### Coding Style + +Beyond using standard tools for formatting and linting, we document and enforce +guidelines for how the PhysicsNeMo codebase is organized. Please consult the +`CODING_STANDARDS` directory for details on how and where to contribute code, +including a "ruleset" for developing models. ### Pull Requests Developer workflow for code contributions is as follows: 1. Developers must first [fork](https://help.github.com/en/articles/fork-a-repo) -the [upstream](https://github.com/NVIDIA/Modulus) Modulus repository. +the [upstream](https://github.com/NVIDIA/physicsnemo) PhysicsNeMo repository. 2. Git clone the forked repository and push changes to the personal fork. @@ -45,7 +49,7 @@ to merge the changes from a branch of the fork into a selected branch of upstrea - Exercise caution when selecting the source and target branches for the PR. - Ensure that you update the [`CHANGELOG.md`](CHANGELOG.md) to reflect your contributions. - Creation of a PR creation kicks off CI and a code review process. - - Atleast one Modulus engineer will be assigned for the review. + - Atleast one PhysicsNeMo engineer will be assigned for the review. 4. The PR will be accepted and the corresponding issue closed after adequate review and testing has been completed. Note that every PR should correspond to an open issue and @@ -56,7 +60,9 @@ should be linked on Github. All source code files should start with this paragraph: ```bash -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -133,11 +139,13 @@ committing your changes: ### Pre-commit -For Modulus development, [pre-commit](https://pre-commit.com/) is **required**. +For PhysicsNeMo development, [pre-commit](https://pre-commit.com/) is **required**. This will not only help developers pass the CI pipeline, but also accelerate reviews. Contributions that have not used pre-commit will *not be reviewed*. -To install `pre-commit` follow the below steps inside the Modulus repository folder: +`pre-commit` is installed as part of the `dev` optional dependencies defined in `pyproject.toml`. +To install `pre-commit` in an existing environment, follow the below steps inside the PhysicsNeMo +repository folder: ```bash pip install pre-commit @@ -147,19 +155,18 @@ pre-commit install Once the above commands are executed, the pre-commit hooks will be activated and all the commits will be checked for appropriate formatting. -### CI +### Continuous Integration (CI) -To ensure quality of the code, your merge request will pass through several CI checks. +To ensure quality of the code, your merge request (MR) will pass through several CI checks. It is mandatory for your MRs to pass these pipelines to ensure a successful merge. Please keep checking this document for the latest guidelines on pushing code. Currently, The pipeline has following stages: 1. `format` - *Pre-commit will check this for you!* - Checks for formatting of your python code. - Refer [black](https://black.readthedocs.io/en/stable/) for more information. - If your MR fails this test, run `black .py` on problematic scripts and - black will take care of the rest. + *Pre-commit will check this for you!* Checks for formatting of your + Python code, using `ruff format` via [Ruff](https://docs.astral.sh/ruff/). + If your MR fails this test, run `ruff format .py` on + problematic scripts and Ruff will take care of the rest. 2. `interrogate` *Pre-commit will check this for you!* @@ -191,17 +198,20 @@ The pipeline has following stages: --ignore-regex MetaData \ -vv \ --color \ - ./modulus/ + ./physicsnemo/ ``` 3. `lint` *Pre-commit will check this for you!* - Linters will perform static analysis to check the style, complexity, errors and more. - For markdown files `markdownlint` is used, its suggested to use the vscode, - neovim or sublime [extensions](https://github.com/DavidAnson/markdownlint#related). - Modulus uses [Ruff](https://docs.astral.sh/ruff/) for linting of various types. - Currently we use flake8/pycodestyle (`E`), Pyflakes (`F`), flake8-bandit (`S`), - isort (`I`), and performance 'PERF' rules with the isort rules being fixable. + Linters will perform static analysis to check the style, complexity, errors + and more. For markdown files `markdownlint` is used, its suggested to use + the vscode, neovim or sublime + [extensions](https://github.com/DavidAnson/markdownlint#related). + PhysicsNeMo uses `ruff check` via[Ruff](https://docs.astral.sh/ruff/) for + linting of various types. Currently we use flake8/pycodestyle (`E`), + Pyflakes (`F`), flake8-bandit (`S`), isort (`I`), and performance 'PERF' + rules. Many rule violations will be automatically fixed by Ruff; others may + require manual changes. 4. `license` *Pre-commit will check this for you!* @@ -215,6 +225,17 @@ The pipeline has following stages: test, you will have to review your changes and fix the issues. To run pytest locally you can simply run `pytest` inside the `test` folder. + While writing these tests, we encourage you to make use of the [`@import_of_fail`](https://github.com/NVIDIA/physicsnemo/blob/main/test/pytest_utils.py#L25) + decorator to appropriately skip your tests for developers and users not having your + test specific dependencies. This mechanism helps us provide a better developer and + user experience when working with the unit tests. + + Some of the tests require test data to be run; otherwise, they will be skipped. + To get the data (available to NVIDIANs only), set the `TEST_DATA_DIR` environment variable + to a desired value and run make get-data. After that, pytest will use the same + variable to find the test data. Alternatively, you can pass it explicitly using + `pytest --nfs-data-dir=`. + 6. `doctest` Checks if the examples in the docstrings run and produce desired outputs. It is highly recommended that you provide simple examples of your functions/classes @@ -223,7 +244,7 @@ The pipeline has following stages: Refer [doctest](https://docs.python.org/3/library/doctest.html) for more information. If your MR fails this test, check your changes and the docstrings. To run doctest locally, you can simply run `pytest --doctest-modules` inside the - `modulus` folder. + `physicsnemo` folder. 7. `coverage` Checks if your code additions have sufficient coverage. diff --git a/Dockerfile b/Dockerfile index adfa024c53..c5b732080f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -11,138 +13,294 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +# +# Image stages: builder -> ci | deploy -> docs +# Builder: all custom if-else deps + all pyproject extras (no dev), project installed non-editable. +# Deploy: uninstall mlflow/wandb only; physicsnemo stays non-editable from builder. +# CI: add dev group, netcdf4 hack, FigNet/Makani, other CI-only packages; physicsnemo uninstalled +# Python packages use uv (UV_SYSTEM_PYTHON=1). Optional: RUN --mount=type=cache,target=/root/.cache/uv and ENV UV_LINK_MODE=copy for faster rebuilds. -ARG BASE_CONTAINER=nvcr.io/nvidia/pytorch:23.10-py3 -FROM ${BASE_CONTAINER} as builder +ARG BASE_CONTAINER=nvcr.io/nvidia/pytorch:26.01-py3 +FROM ${BASE_CONTAINER} AS builder ARG TARGETPLATFORM +# Install uv (use system Python for installs; set so --system is default) +COPY --from=ghcr.io/astral-sh/uv:0.10.3 /uv /uvx /bin/ +ENV UV_SYSTEM_PYTHON=1 +# Base image Python is PEP 668 externally-managed; allow system installs in container +ENV UV_BREAK_SYSTEM_PACKAGES=1 + # Update pip and setuptools -RUN pip install "pip==23.2.1" "setuptools==68.2.2" +RUN uv pip install "pip>=23.2.1" "setuptools>=77.0.3" # Setup git lfs, graphviz gl1(vtk dep) RUN apt-get update && \ - apt-get install -y git-lfs graphviz libgl1 && \ + apt-get install -y git-lfs graphviz libgl1 zip unzip && \ git lfs install ENV _CUDA_COMPAT_TIMEOUT=90 -# Install other dependencies -RUN pip install --no-cache-dir "h5py>=3.7.0" "netcdf4>=1.6.3" "ruamel.yaml>=0.17.22" "scikit-learn>=1.0.2" "cftime>=1.6.2" "einops>=0.7.0" "pyspng>=0.1.0" -RUN pip install --no-cache-dir "hydra-core>=1.2.0" "termcolor>=2.1.1" "wandb>=0.13.7" "mlflow>=2.1.1" "pydantic>=1.10.2" "imageio>=2.28.1" "moviepy>=1.0.3" "tqdm>=4.60.0" +# Copy physicsnemo source +COPY . /physicsnemo/ + +####################################################################### +# Step 1: Dependencies that need custom if-else handling (wheels, etc.) +####################################################################### + +# Remove packaging==23.2 from constraint.txt in the PyTorch container +RUN FILE="/etc/pip/constraint.txt" && \ + if [ -f "$FILE" ]; then \ + sed -i '/packaging/d' "$FILE"; \ + else \ + echo "File not found: $FILE"; \ + fi + +# Tell uv to respect the container's constraint file (inherited by all stages) +# Create an empty constraint file if one does not exist to avoid uv errors +RUN [ -f /etc/pip/constraint.txt ] || touch /etc/pip/constraint.txt +ENV UV_CONSTRAINT=/etc/pip/constraint.txt -# copy modulus source -COPY . /modulus/ +# Install pyspng for arm64 +ARG PYSPNG_ARM64_WHEEL +ENV PYSPNG_ARM64_WHEEL=${PYSPNG_ARM64_WHEEL:-unknown} + +RUN if [ "$TARGETPLATFORM" = "linux/arm64" ] && [ "$PYSPNG_ARM64_WHEEL" != "unknown" ]; then \ + echo "Custom pyspng wheel for $TARGETPLATFORM exists, installing!" && \ + uv pip install /physicsnemo/deps/${PYSPNG_ARM64_WHEEL}; \ + else \ + echo "No custom wheel for pyspng found. Installing pyspng for: $TARGETPLATFORM from pypi" && \ + uv pip install "pyspng>=0.1.0"; \ + fi + +# Install Numcodecs (separate install: Numcodecs ARM pip has issues) +ARG NUMCODECS_ARM64_WHEEL +ENV NUMCODECS_ARM64_WHEEL=${NUMCODECS_ARM64_WHEEL:-unknown} -# Install Numcodecs (This needs a separate install because Numcodecs ARM pip install has issues) -# A fix is being added here: https://github.com/zarr-developers/numcodecs/pull/315 but the public release is not ready yet. RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then \ echo "Pip install for numcodecs for $TARGETPLATFORM exists, installing!" && \ - pip install --no-cache-dir numcodecs; \ - elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ -e "/modulus/deps/numcodecs-0.11.0-cp310-cp310-linux_aarch64.whl" ]; then \ + uv pip install numcodecs; \ + elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ "$NUMCODECS_ARM64_WHEEL" != "unknown" ]; then \ echo "Numcodecs wheel for $TARGETPLATFORM exists, installing!" && \ - pip install --force-reinstall --no-cache-dir /modulus/deps/numcodecs-0.11.0-cp310-cp310-linux_aarch64.whl; \ + uv pip install --reinstall /physicsnemo/deps/${NUMCODECS_ARM64_WHEEL}; \ else \ - echo "Numcodecs wheel for $TARGETPLATFORM is not present, attempting to build from pip, but might fail" && \ - pip install --no-cache-dir numcodecs; \ + echo "Numcodecs wheel for $TARGETPLATFORM is not present. Will attempt to install from PyPi index, but might fail" && \ + uv pip install numcodecs; \ fi -# install vtk and pyvista -RUN if [ "$TARGETPLATFORM" = "linux/arm64" ] && [ -e "/modulus/deps/vtk-9.2.6.dev0-cp310-cp310-linux_aarch64.whl" ]; then \ - echo "VTK wheel for $TARGETPLATFORM exists, installing!" && \ - pip install --no-cache-dir /modulus/deps/vtk-9.2.6.dev0-cp310-cp310-linux_aarch64.whl; \ +# Install vtk and pyvista +ARG VTK_ARM64_WHEEL +ENV VTK_ARM64_WHEEL=${VTK_ARM64_WHEEL:-unknown} + +RUN if [ "$TARGETPLATFORM" = "linux/arm64" ] && [ "$VTK_ARM64_WHEEL" != "unknown" ]; then \ + echo "VTK wheel $VTK_ARM64_WHEEL for $TARGETPLATFORM exists, installing!" && \ + uv pip install /physicsnemo/deps/${VTK_ARM64_WHEEL}; \ elif [ "$TARGETPLATFORM" = "linux/amd64" ]; then \ - echo "Installing vtk for: $TARGETPLATFORM" && \ - pip install --no-cache-dir "vtk>=9.2.6"; \ + echo "Installing vtk for: $TARGETPLATFORM" && \ + uv pip install "vtk>=9.2.6"; \ else \ - echo "Installing vtk for: $TARGETPLATFORM from source" && \ - apt-get update && apt-get install -y libgl1-mesa-dev && \ - git clone https://gitlab.kitware.com/vtk/vtk.git && cd vtk && git checkout tags/v9.2.6 && git submodule update --init --recursive && \ - mkdir build && cd build && cmake -GNinja -DVTK_WHEEL_BUILD=ON -DVTK_WRAP_PYTHON=ON /workspace/vtk/ && ninja && \ - python setup.py bdist_wheel && \ - pip install --no-cache-dir dist/vtk-9.2.6.dev0-cp310-cp310-linux_aarch64.whl && \ - cd ../../ && rm -r vtk; \ + echo "No custom wheel or wheel on PyPi found. Installing vtk for: $TARGETPLATFORM from source" && \ + apt-get update && apt-get install -y libgl1-mesa-dev && \ + git clone https://gitlab.kitware.com/vtk/vtk.git && cd vtk && git checkout tags/v9.4.1 && git submodule update --init --recursive && \ + mkdir build && cd build && cmake -GNinja -DVTK_WHEEL_BUILD=ON -DVTK_WRAP_PYTHON=ON /workspace/vtk/ && ninja && \ + python setup.py bdist_wheel && \ + uv pip install dist/vtk-*.whl && \ + cd ../../ && rm -r vtk; \ fi -RUN pip install --no-cache-dir "pyvista>=0.40.1" - -# Install DGL from source -ARG DGL_BACKEND=pytorch -ENV DGL_BACKEND=$DGL_BACKEND -ENV DGLBACKEND=$DGL_BACKEND -RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] && [ -e "/modulus/deps/dgl-1.1.2-cp310-cp310-linux_x86_64.whl" ]; then \ - echo "DGL wheel for $TARGETPLATFORM exists, installing!" && \ - pip install --force-reinstall --no-cache-dir /modulus/deps/dgl-1.1.2-cp310-cp310-linux_x86_64.whl; \ - elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ -e "/modulus/deps/dgl-1.1.2-cp310-cp310-linux_aarch64.whl" ]; then \ - echo "DGL wheel for $TARGETPLATFORM exists, installing!" && \ - pip install --force-reinstall --no-cache-dir /modulus/deps/dgl-1.1.2-cp310-cp310-linux_aarch64.whl; \ +RUN uv pip install "pyvista>=0.40.1" + +# Install onnxruntime (custom wheel for ARM) +ARG ONNXRUNTIME_ARM64_WHEEL +ENV ONNXRUNTIME_ARM64_WHEEL=${ONNXRUNTIME_ARM64_WHEEL:-unknown} + +RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then \ + uv pip install "onnxruntime-gpu>1.19.0"; \ + elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ "$ONNXRUNTIME_ARM64_WHEEL" != "unknown" ]; then \ + uv pip install --no-deps /physicsnemo/deps/${ONNXRUNTIME_ARM64_WHEEL}; \ else \ - echo "No DGL wheel present, building from source" && \ - git clone https://github.com/dmlc/dgl.git && cd dgl/ && git checkout tags/1.1.2 && git submodule update --init --recursive && \ - DGL_HOME="/workspace/dgl" bash script/build_dgl.sh -g && \ - cd python && \ - python setup.py install && \ - python setup.py build_ext --inplace; \ + echo "Skipping onnxruntime_gpu install."; \ fi -RUN rm -rf /workspace/dgl - -# Install custom onnx -# TODO: Find a fix to eliminate the custom build -# Forcing numpy update to over ride numba 0.56.4 max numpy constraint -RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] && [ -e "/modulus/deps/onnxruntime_gpu-1.15.1-cp310-cp310-linux_x86_64.whl" ]; then \ - echo "Custom onnx wheel for $TARGETPLATFORM exists, installing!" && \ - pip install --force-reinstall --no-cache-dir /modulus/deps/onnxruntime_gpu-1.15.1-cp310-cp310-linux_x86_64.whl; \ - elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ -e "/modulus/deps/onnxruntime_gpu-1.15.1-cp310-cp310-linux_aarch64.whl" ]; then \ - echo "Custom onnx wheel for $TARGETPLATFORM exists, installing!" && \ - pip install --force-reinstall --no-cache-dir /modulus/deps/onnxruntime_gpu-1.15.1-cp310-cp310-linux_aarch64.whl; \ + +# Install torch-geometric and torch-scatter +RUN uv pip install "torch_geometric>=2.6.1" + +ARG TORCH_SCATTER_ARM64_WHEEL +ENV TORCH_SCATTER_ARM64_WHEEL=${TORCH_SCATTER_ARM64_WHEEL:-unknown} + +ARG TORCH_SCATTER_AMD64_WHEEL +ENV TORCH_SCATTER_AMD64_WHEEL=${TORCH_SCATTER_AMD64_WHEEL:-unknown} + +ENV TORCH_CUDA_ARCH_LIST="7.5 8.0 8.6 9.0 10.0 12.0+PTX" + +RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] && [ "$TORCH_SCATTER_AMD64_WHEEL" != "unknown" ]; then \ + echo "Installing torch_scatter for: $TARGETPLATFORM" && \ + uv pip install --reinstall /physicsnemo/deps/${TORCH_SCATTER_AMD64_WHEEL}; \ + elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ "$TORCH_SCATTER_ARM64_WHEEL" != "unknown" ]; then \ + echo "Installing torch_scatter for: $TARGETPLATFORM" && \ + uv pip install --reinstall /physicsnemo/deps/${TORCH_SCATTER_ARM64_WHEEL}; \ else \ - echo "No custom wheel present, skipping" && \ - pip install --no-cache-dir "numpy==1.22.4"; \ + echo "No custom wheel present for scatter, building from source"; \ + mkdir -p /physicsnemo/deps/; \ + cd /physicsnemo/deps/; \ + git clone https://github.com/rusty1s/pytorch_scatter.git; \ + cd pytorch_scatter; \ + git checkout tags/2.1.2; \ + FORCE_CUDA=1 MAX_JOBS=64 python setup.py bdist_wheel && \ + uv pip install --reinstall dist/*.whl && \ + cd ../ && rm -r pytorch_scatter; \ fi -# cleanup of stage -RUN rm -rf /modulus/ +# Install pyg-lib +ARG PYGLIB_ARM64_WHEEL +ENV PYGLIB_ARM64_WHEEL=${PYGLIB_ARM64_WHEEL:-unknown} -# CI image -FROM builder as ci +ARG PYGLIB_AMD64_WHEEL +ENV PYGLIB_AMD64_WHEEL=${PYGLIB_AMD64_WHEEL:-unknown} -ARG TARGETPLATFORM +RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] && [ "$PYGLIB_AMD64_WHEEL" != "unknown" ]; then \ + echo "Installing pyg_lib for: $TARGETPLATFORM" && \ + uv pip install --reinstall /physicsnemo/deps/${PYGLIB_AMD64_WHEEL}; \ + elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ "$PYGLIB_ARM64_WHEEL" != "unknown" ]; then \ + echo "Installing pyg_lib for: $TARGETPLATFORM" && \ + uv pip install --reinstall /physicsnemo/deps/${PYGLIB_ARM64_WHEEL}; \ + else \ + echo "No custom wheel present for pyg_lib, building from source"; \ + uv pip install ninja wheel && \ + uv pip install --no-build-isolation "git+https://github.com/pyg-team/pyg-lib.git@0.5.0"; \ + fi -COPY . /modulus/ -RUN cd /modulus/ && pip install -e . && pip uninstall nvidia-modulus -y && rm -rf /modulus/ -RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then \ - echo "Installing tensorflow and warp-lang for: $TARGETPLATFORM" && \ - pip install --no-cache-dir "tensorflow==2.9.0" "warp-lang>=0.6.0"; \ - elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then \ - echo "Installing tensorflow and warp-lang for: $TARGETPLATFORM is not supported presently"; \ +# Install torch_cluster +ARG TORCH_CLUSTER_ARM64_WHEEL +ENV TORCH_CLUSTER_ARM64_WHEEL=${TORCH_CLUSTER_ARM64_WHEEL:-unknown} + +ARG TORCH_CLUSTER_AMD64_WHEEL +ENV TORCH_CLUSTER_AMD64_WHEEL=${TORCH_CLUSTER_AMD64_WHEEL:-unknown} + +RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] && [ "$TORCH_CLUSTER_AMD64_WHEEL" != "unknown" ]; then \ + echo "Installing torch_cluster for: $TARGETPLATFORM" && \ + uv pip install --reinstall /physicsnemo/deps/${TORCH_CLUSTER_AMD64_WHEEL}; \ + elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ "$TORCH_CLUSTER_ARM64_WHEEL" != "unknown" ]; then \ + echo "Installing torch_cluster for: $TARGETPLATFORM" && \ + uv pip install --reinstall /physicsnemo/deps/${TORCH_CLUSTER_ARM64_WHEEL}; \ + else \ + echo "No custom wheel present for cluster, building from source"; \ + mkdir -p /physicsnemo/deps/; \ + cd /physicsnemo/deps/; \ + git clone --branch 1.6.3 --depth 1 https://github.com/rusty1s/pytorch_cluster.git; \ + cd pytorch_cluster; \ + FORCE_CUDA=1 MAX_JOBS=64 python setup.py bdist_wheel && \ + uv pip install --reinstall dist/*.whl && \ + cd ../ && rm -r pytorch_cluster; \ fi -RUN pip install --no-cache-dir "black==22.10.0" "interrogate==1.5.0" "coverage==6.5.0" "protobuf==3.20.3" -# Deployment image -FROM builder as deploy -COPY . /modulus/ -RUN cd /modulus/ && pip install . -RUN pip install --no-cache-dir "protobuf==3.20.3" + +# natten and torch_sparse need torch at build time (--no-build-isolation) +ENV NATTEN_CUDA_ARCH="8.0;8.6;9.0;10.0;12.0" + +ARG NATTEN_ARM64_WHEEL +ENV NATTEN_ARM64_WHEEL=${NATTEN_ARM64_WHEEL:-unknown} + +ARG NATTEN_AMD64_WHEEL +ENV NATTEN_AMD64_WHEEL=${NATTEN_AMD64_WHEEL:-unknown} + +RUN if [ "$TARGETPLATFORM" = "linux/amd64" ] && [ "$NATTEN_AMD64_WHEEL" != "unknown" ]; then \ + echo "Installing natten for: $TARGETPLATFORM" && \ + uv pip install --reinstall /physicsnemo/deps/${NATTEN_AMD64_WHEEL}; \ + elif [ "$TARGETPLATFORM" = "linux/arm64" ] && [ "$NATTEN_ARM64_WHEEL" != "unknown" ]; then \ + echo "Installing natten for: $TARGETPLATFORM" && \ + uv pip install --reinstall /physicsnemo/deps/${NATTEN_ARM64_WHEEL}; \ + else \ + echo "No custom wheel present for natten, building from source"; \ + mkdir -p /physicsnemo/deps/; \ + cd /physicsnemo/deps/; \ + git clone --recursive --branch v0.21.5 --depth 1 https://github.com/SHI-Labs/NATTEN.git; \ + cd NATTEN; \ + MAX_JOBS=64 python setup.py bdist_wheel && \ + uv pip install --reinstall dist/*.whl && \ + cd ../ && rm -r NATTEN; \ + fi + +RUN uv pip install --no-build-isolation "torch_sparse" + +# All pyproject extras (no dev); installs physicsnemo non-editable +RUN cd /physicsnemo && uv pip install ".[cu13,utils-extras,mesh-extras,datapipes-extras,gnns,perf]" + +# Cleanup builder stage +RUN rm -rf /physicsnemo/ + +####################################################################### +# CI image: builder + dev group + netcdf4 hack + FigNet/Makani + CI-only packages +####################################################################### +FROM builder AS ci + +ARG TARGETPLATFORM + +# UV: use system Python and respect container constraint (same as builder) +ENV UV_SYSTEM_PYTHON=1 +ENV UV_BREAK_SYSTEM_PACKAGES=1 +ENV UV_CONSTRAINT=/etc/pip/constraint.txt + +# TODO: Remove hacky downgrade of netCDF4. netCDF4 v1.7.1 issue: https://github.com/Unidata/netcdf4-python/issues/1343 +RUN uv pip install "netcdf4>=1.6.3,<1.7.1" + +COPY . /physicsnemo/ + +# Dev dependency-group (pytest, ruff, etc.) +RUN cd /physicsnemo && uv pip install --group dev + +# FigNet/Makani and related CI-only deps +RUN FORCE_CUDA_EXTENSION=1 uv pip install --no-build-isolation "torch-harmonics==0.8.0" +RUN uv pip install "tensorly>=0.8.1" "tensorly-torch>=0.4.0" "torchinfo>=1.8" "webdataset>=0.2" +# Install Makani via direct URL +RUN uv pip install --no-deps "git+https://github.com/NVIDIA/makani.git@v0.2.1#egg=makani" + +# Other CI-only specs (moto, scikit-image, etc.) +RUN uv pip install "moto[s3]>=5.0.28" +RUN uv pip install "numpy-stl" "scikit-image>=0.24.0" "sparse-dot-mkl" "shapely" +RUN uv pip install "multi-storage-client[boto3]>=0.33.0" + +# E2Grid install +RUN uv pip install --no-deps --no-build-isolation "git+https://github.com/NVlabs/earth2grid.git@11dcf1b0787a7eb6a8497a3a5a5e1fdcc31232d3" + +# Uninstall the non-editable physicsnemo from builder +RUN uv pip uninstall nvidia-physicsnemo + +# Cleanup +RUN rm -rf /physicsnemo/ + +####################################################################### +# Deploy image: builder with mlflow/wandb removed; physicsnemo already non-editable from builder +####################################################################### +FROM builder AS deploy + +# UV: use system Python and respect container constraint (same as builder) +ENV UV_SYSTEM_PYTHON=1 +ENV UV_BREAK_SYSTEM_PACKAGES=1 +ENV UV_CONSTRAINT=/etc/pip/constraint.txt + +# Remove mlflow and wandb (CVE concerns) +RUN uv pip uninstall mlflow wandb # Set Git Hash as a environment variable -ARG MODULUS_GIT_HASH -ENV MODULUS_GIT_HASH=${MODULUS_GIT_HASH:-unknown} +ARG PHYSICSNEMO_GIT_HASH +ENV PHYSICSNEMO_GIT_HASH=${PHYSICSNEMO_GIT_HASH:-unknown} -# Clean up -RUN rm -rf /modulus/ +# Remove uv cache to save image size +RUN uv cache clean -# Docs image -FROM deploy as docs +####################################################################### +# Docs image: deploy + docs build dependencies +####################################################################### +FROM deploy AS docs ARG TARGETPLATFORM -# Install CI packages -RUN pip install --no-cache-dir "protobuf==3.20.3" -RUN if [ "$TARGETPLATFORM" = "linux/amd64" ]; then \ - echo "Installing tensorflow and warp-lang for: $TARGETPLATFORM" && \ - pip install --no-cache-dir "tensorflow==2.9.0" "warp-lang>=0.6.0"; \ - elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then \ - echo "Installing tensorflow and warp-lang for: $TARGETPLATFORM is not supported presently"; \ - fi +# UV: use system Python and respect container constraint (same as builder) +ENV UV_SYSTEM_PYTHON=1 +ENV UV_BREAK_SYSTEM_PACKAGES=1 +ENV UV_CONSTRAINT=/etc/pip/constraint.txt + # Install packages for Sphinx build -RUN pip install --no-cache-dir "recommonmark==0.7.1" "sphinx==5.1.1" "sphinx-rtd-theme==1.0.0" "pydocstyle==6.1.1" "nbsphinx==0.8.9" "nbconvert==6.4.3" "jinja2==3.0.3" -RUN wget https://github.com/jgm/pandoc/releases/download/3.1.6.2/pandoc-3.1.6.2-1-amd64.deb && dpkg -i pandoc-3.1.6.2-1-amd64.deb +RUN uv pip install "protobuf==3.20.3" +RUN uv pip install "recommonmark==0.7.1" "sphinx==5.1.1" "sphinx-rtd-theme==1.0.0" "pydocstyle==6.1.1" "nbsphinx==0.8.9" "nbconvert==6.4.3" "jinja2==3.0.3" +RUN wget https://github.com/jgm/pandoc/releases/download/3.1.6.2/pandoc-3.1.6.2-1-amd64.deb && dpkg -i pandoc-3.1.6.2-1-amd64.deb diff --git a/FAQ.md b/FAQ.md index 0d7e3fa643..498ee33014 100644 --- a/FAQ.md +++ b/FAQ.md @@ -1,60 +1,60 @@ -# Frequently Asked Questions about Modulus +# Frequently Asked Questions about PhysicsNeMo ## Table of contents -- [What is the recommended hardware for training using Modulus framework?](#what-is-the-recommended-hardware-for-training-using-modulus-framework) -- [What model architectures are in Modulus?](#what-model-architectures-are-in-modulus) -- [What is the difference between Modulus Core and Symbolic?](#what-is-the-difference-between-modulus-core-and-symbolic) -- [What can I do if I dont see a PDE in Modulus?](#what-can-i-do-if-i-dont-see-a-pde-in-modulus) +- [What is the recommended hardware for training using PhysicsNeMo framework?](#what-is-the-recommended-hardware-for-training-using-physicsnemo-framework) +- [What model architectures are in PhysicsNeMo?](#what-model-architectures-are-in-physicsnemo) +- [What is the difference between PhysicsNeMo Core and Symbolic?](#what-is-the-difference-between-physicsnemo-core-and-symbolic) +- [What can I do if I dont see a PDE in PhysicsNeMo?](#what-can-i-do-if-i-dont-see-a-pde-in-physicsnemo) - [What is the difference between the pip install and the container?](#what-is-the-difference-between-the-pip-install-and-the-container) -## What is the recommended hardware for training using Modulus framework? +## What is the recommended hardware for training using PhysicsNeMo framework? Please refer to the recommended hardware section: -[System Requirments](https://docs.nvidia.com/deeplearning/modulus/getting-started/index.html#system-requirements) +[System Requirements](https://docs.nvidia.com/deeplearning/physicsnemo/getting-started/index.html#system-requirements) -## What model architectures are in Modulus? +## What model architectures are in PhysicsNeMo? -Nvidia Modulus is built on top of PyTorch and you can build and train any model -architecture you want in Modulus. Modulus however has a catalog of models that have been -packaged in a configurable form to make it easy to retrain with new data or certain +Nvidia PhysicsNeMo is built on top of PyTorch and you can build and train any model +architecture you want in PhysicsNeMo. PhysicsNeMo however has a catalog of models that +have been packaged in a configurable form to make it easy to retrain with new data or certain config parameters. Examples include GNNs like MeshGraphNet or Neural Operators like FNO. -Modulus samples have more models that illustrate how a specific approach with a specifc +PhysicsNeMo samples have more models that illustrate how a specific approach with a specific model architecture can be applied to a specific problem. These are reference starting points for users to get started. You can find the list of built in model architectures -[here](https://github.com/NVIDIA/modulus/tree/main/modulus/models) and -[here](https://github.com/NVIDIA/modulus-sym/tree/main/modulus/sym/models) +[here](https://github.com/NVIDIA/physicsnemo/tree/main/physicsnemo/models) and +[here](https://github.com/NVIDIA/physicsnemo-sym/tree/main/physicsnemo/sym/models) -## What is the difference between Modulus Core and Symbolic? +## What is the difference between PhysicsNeMo Core and Symbolic? -Modulus core is the foundational module that provides the core algorithms, network +PhysicsNeMo core is the foundational module that provides the core algorithms, network architectures and utilities that cover a broad spectrum of Physics-ML approaches. -Modulus Symbolic provides pythonic APIs, algorithms and utilities to be used with -Modulus core, to explicitly physics inform the model training. This includes symbolic +PhysicsNeMo Symbolic provides pythonic APIs, algorithms and utilities to be used with +PhysicsNeMo core, to explicitly physics inform the model training. This includes symbolic APIs for PDEs, domain sampling and PDE-based residuals. It also provides higher level abstraction to compose a training loop from specification of the geometry, PDEs and constraints like boundary conditions using simple symbolic APIs. So if you are familiar with PyTorch and want to train model from a dataset, you start -with Modulus core and you import Modulus symbolic to bring in explicit domain knowledge. -Please refer to the [DeepONet example](https://github.com/modulus/tree/main/examples/cfd/darcy_deeponet_physics) +with PhysicsNeMo core and you import PhysicsNeMo symbolic to bring in explicit domain knowledge. +Please refer to the [DeepONet example](https://github.com/physicsnemo/tree/main/examples/cfd/darcy_deeponet_physics) that illustrates the concept. If you are an engineer or domain expert accustomed to using numerical solvers, you can -use Modulus Symbolic to define your problem at a higher level of abstraction. Please -refer to the [Lid Driven cavity](https://docs.nvidia.com/deeplearning/modulus/modulus-sym/user_guide/basics/lid_driven_cavity_flow.html) +use PhysicsNeMo Symbolic to define your problem at a higher level of abstraction. Please +refer to the [Lid Driven cavity](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-sym/user_guide/basics/lid_driven_cavity_flow.html) that illustrates the concept. -## What can I do if I dont see a PDE in Modulus? +## What can I do if I dont see a PDE in PhysicsNeMo? -Modulus Symbolic provides a well documeted -[example](https://docs.nvidia.com/deeplearning/modulus/modulus-sym/user_guide/foundational/1d_wave_equation.html#writing-custom-pdes-and-boundary-initial-conditions) -that walks you through how to define a custom PDE. Please see the source [here](https://github.com/NVIDIA/modulus-sym/tree/main/modulus/sym/eq/pdes) +PhysicsNeMo Symbolic provides a well documented +[example](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-sym/user_guide/foundational/1d_wave_equation.html#writing-custom-pdes-and-boundary-initial-conditions) +that walks you through how to define a custom PDE. Please see the source [here](https://github.com/NVIDIA/physicsnemo-sym/tree/main/physicsnemo/sym/eq/pdes) to see the built-in PDE implementation as an additional reference for your own implementation. ## What is the difference between the pip install and the container? There is no functional difference between the two. This is to simplify the ease of -installing and setting up the Modulus environment. Please refer to the -[getting started guide](https://docs.nvidia.com/deeplearning/modulus/getting-started/index.html#modulus-with-docker-image-recommended) +installing and setting up the PhysicsNeMo environment. Please refer to the +[getting started guide](https://docs.nvidia.com/deeplearning/physicsnemo/getting-started/index.html#physicsnemo-with-docker-image-recommended) on how to install using Pip or using the container. diff --git a/Makefile b/Makefile index d97961a265..27025f18e4 100644 --- a/Makefile +++ b/Makefile @@ -1,52 +1,61 @@ install: pip install --upgrade pip && \ pip install -e . + pip install tfrecord # Putting this here till we update the container. + +editable-install: + pip install --upgrade pip && \ + pip install -e .[dev] --config-settings editable_mode=strict get-data: - mkdir -p /data && \ - mkdir -p /data/nfs/ && \ - git -C /data/nfs/modulus-data pull || \ - git clone https://gitlab-master.nvidia.com/modulus/modulus-data.git /data/nfs/modulus-data + test -n "$(TEST_DATA_DIR)" || { echo "Error: TEST_DATA_DIR should be set"; exit 1; } + mkdir -p $(TEST_DATA_DIR) && \ + rm -rf $(TEST_DATA_DIR)/modulus-data && \ + git clone https://gitlab-master.nvidia.com/modulus/modulus-data.git $(TEST_DATA_DIR)/modulus-data && \ + echo "Test data has been saved in ${TEST_DATA_DIR}" setup-ci: pip install pre-commit && \ pre-commit install black: - pre-commit run black -a + pre-commit run ruff-format -a interrogate: pre-commit run interrogate -a lint: + pre-commit run ruff-check -a && \ pre-commit run markdownlint -a && \ - pre-commit run ruff -a && \ pre-commit run check-added-large-files -a license: - pre-commit run license -a + python test/ci_tests/header_check.py --all-files doctest: coverage run \ --rcfile='test/coverage.docstring.rc' \ -m pytest \ - --doctest-modules modulus/ --ignore-glob=*internal* --ignore-glob=*experimental* + --doctest-modules physicsnemo/ --ignore-glob=*internal* --ignore-glob=*experimental* pytest: coverage run \ --rcfile='test/coverage.pytest.rc' \ - -m pytest --ignore-glob=*docs* + -m pytest --ignore-glob=*docs* --ignore-glob=*examples* pytest-internal: cd test/internal && \ pytest && \ cd ../../ +# NOTE: temporarily omitting diffusion coverage until we have a better way to test it. coverage: coverage combine && \ - coverage report --show-missing --omit=*test* --omit=*internal* --omit=*experimental* --fail-under=70 && \ + coverage report --show-missing --omit=*test* --omit=*internal* --omit=*experimental* --omit=*diffusion* --fail-under=60 && \ coverage html +all-ci: get-data setup-ci black interrogate lint license install pytest doctest coverage + # For arch naming conventions, refer # https://docs.docker.com/build/building/multi-platform/ # https://github.com/containerd/containerd/blob/v1.4.3/platforms/platforms.go#L86 @@ -56,18 +65,20 @@ ifeq ($(ARCH), x86_64) TARGETPLATFORM := "linux/amd64" else ifeq ($(ARCH), aarch64) TARGETPLATFORM := "linux/arm64" +else ifeq ($(ARCH), arm) + TARGETPLATFORM := "linux/arm64" else $(error Unknown CPU architecture ${ARCH} detected) endif -MODULUS_GIT_HASH = $(shell git rev-parse --short HEAD) +PHYSICSNEMO_GIT_HASH = $(shell git rev-parse --short HEAD) container-deploy: - docker build -t modulus:deploy --build-arg TARGETPLATFORM=${TARGETPLATFORM} --build-arg MODULUS_GIT_HASH=${MODULUS_GIT_HASH} --target deploy -f Dockerfile . + docker build -t physicsnemo:deploy --build-arg TARGETPLATFORM=${TARGETPLATFORM} --build-arg PHYSICSNEMO_GIT_HASH=${PHYSICSNEMO_GIT_HASH} --target deploy -f Dockerfile . container-ci: - docker build -t modulus:ci --build-arg TARGETPLATFORM=${TARGETPLATFORM} --target ci -f Dockerfile . + docker build -t physicsnemo:ci --build-arg TARGETPLATFORM=${TARGETPLATFORM} --target ci -f Dockerfile . container-docs: - docker build -t modulus:docs --build-arg TARGETPLATFORM=${TARGETPLATFORM} --target docs -f Dockerfile . + docker build -t physicsnemo:docs --build-arg TARGETPLATFORM=${TARGETPLATFORM} --target docs -f Dockerfile . diff --git a/README.md b/README.md index 5fb12a135b..769fae52a9 100644 --- a/README.md +++ b/README.md @@ -1,63 +1,198 @@ -# Modulus (Beta) +# NVIDIA PhysicsNeMo + +📝 NVIDIA PhysicsNeMo is undergoing an update to v2.0 - all the features, with easier installation and integration to external packages. See the [migration guide](https://github.com/NVIDIA/physicsnemo/blob/main/v2.0-MIGRATION-GUIDE.md) for more details! + [![Project Status: Active - The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active) -[![GitHub](https://img.shields.io/github/license/NVIDIA/modulus)](https://github.com/NVIDIA/modulus/blob/master/LICENSE.txt) +[![GitHub](https://img.shields.io/github/license/NVIDIA/physicsnemo)](https://github.com/NVIDIA/physicsnemo/blob/master/LICENSE.txt) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) +[![Install CI](https://github.com/NVIDIA/physicsnemo/actions/workflows/install-ci.yml/badge.svg?event=schedule)](https://github.com/NVIDIA/physicsnemo/actions/workflows/install-ci.yml) + +[**NVIDIA PhysicsNeMo**](#what-is-physicsnemo) +| [**Documentation**](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/index.html) +| [**Install Guide**](#installation) +| [**Getting Started**](#getting-started-with-physicsnemo) +| [**Contributing Guidelines**](#contributing-to-physicsnemo) +| [**Dev blog**](https://nvidia.github.io/physicsnemo/blog/) -Modulus is an open source deep-learning framework for building, training, and fine-tuning -deep learning models using state-of-the-art Physics-ML methods. +## What is PhysicsNeMo? -Whether you are exploring the use of Neural operators like Fourier Neural Operators or -interested in Physics informed Neural Networks or a hybrid approach in between, Modulus -provides you with the optimized stack that will enable you to train your models at real -world scale. +NVIDIA PhysicsNeMo is an open-source deep-learning framework for building, training, +fine-tuning, and inferring Physics AI models using state-of-the-art SciML methods for +AI4Science and engineering. -This package is the core module that provides the core algorithms, network architectures -and utilities that cover a broad spectrum of physics-constrained and data-driven -workflows to suit the diversity of use cases in the science and engineering disciplines. +PhysicsNeMo provides Python modules to compose scalable and optimized training and +inference pipelines to explore, develop, validate, and deploy AI models that combine +physics knowledge with data, enabling real-time predictions. -Detailed information on features and capabilities can be found in the [Modulus documentation](https://docs.nvidia.com/modulus/index.html#core). +Whether you are exploring the use of neural operators, GNNs, or transformers, or are +interested in Physics-Informed Neural Networks or a hybrid approach in between, PhysicsNeMo +provides you with an optimized stack that will enable you to train your models at scale.

- Modulus + PhysicsNeMo

-## Modulus Packages + + +- [More About PhysicsNeMo](#more-about-physicsnemo) + - [Scalable GPU-Optimized Training Library](#scalable-gpu-optimized-training-library) + - [A Suite of Physics-Informed ML Models](#a-suite-of-physics-informed-ml-models) + - [Seamless PyTorch Integration](#seamless-pytorch-integration) + - [Easy Customization and Extension](#easy-customization-and-extension) + - [AI4Science Library](#ai4science-library) + - [Domain-Specific Packages](#domain-specific-packages) +- [Who is Using and Contributing to PhysicsNeMo](#who-is-using-and-contributing-to-physicsnemo) +- [Why Use PhysicsNeMo](#why-are-they-using-physicsnemo) +- [Getting Started](#getting-started-with-physicsnemo) +- [Resources](#resources) +- [Installation](#installation) +- [Contributing](#contributing-to-physicsnemo) +- [Communication](#communication) +- [License](#license) + + + +## More About PhysicsNeMo + +At a granular level, PhysicsNeMo is developed as modular functionality and therefore +provides built-in composable modules that are packaged into a few key components: + + +Component | Description | +---- | --- | +[**physicsnemo.models**](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.models.html) | A collection of optimized, customizable, and easy-to-use families of model architectures such as Neural Operators, Graph Neural Networks, Diffusion models, Transformer models, and many more| +[**physicsnemo.datapipes**](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.datapipes.html) | Optimized and scalable built-in data pipelines fine-tuned to handle engineering and scientific data structures like point clouds, meshes, etc.| +[**physicsnemo.distributed**](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.distributed.html) | A distributed computing sub-module built on top of `torch.distributed` to enable parallel training with just a few steps| +[**physicsnemo.curator**](https://github.com/NVIDIA/physicsnemo-curator) | A sub-module to streamline and accelerate the process of data curation for engineering datasets| +[**physicsnemo.sym.geometry**](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-sym/user_guide/features/csg_and_tessellated_module.html) | A sub-module to handle geometry for DL training using Constructive Solid Geometry modeling and CAD files in STL format| +[**physicsnemo.sym.eq**](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-sym/user_guide/features/nodes.html) | A sub-module to use PDEs in your DL training with several implementations of commonly observed equations and easy ways for customization| + + +For a complete list, refer to the PhysicsNeMo API documentation for +[PhysicsNeMo](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/index.html). -- [Modulus (Beta)](https://github.com/NVIDIA/modulus): Open-source deep-learning - framework for building, training, and fine-tuning deep learning models using - state-of-the-art Physics-ML methods. -- [Modulus Symbolic (Beta)](https://github.com/NVIDIA/modulus-sym): Framework - providing pythonic APIs, algorithms and utilities to be used with Modulus - core to physics inform model training as well as higher level abstraction - for domain experts. +## AI4Science Library -### Domain Specific Packages +Usually, PhysicsNeMo is used either as: -- [Earth-2 MIP (Beta)](https://github.com/NVIDIA/earth2mip): Python framework - to enable climate researchers and scientists to explore and experiment with +- A complementary tool to PyTorch when exploring AI for SciML and AI4Science applications. +- A deep learning research platform that provides scale and optimal performance on +NVIDIA GPUs. + +### Domain-Specific Packages + +The following are packages dedicated to domain experts of specific communities, catering +to their unique exploration needs: + +- [PhysicsNeMo CFD](https://github.com/NVIDIA/physicsnemo-cfd): Inference sub-module of PhysicsNeMo + to enable CFD domain experts to explore, experiment, and validate using pretrained + AI models for CFD use cases. +- [PhysicsNeMo Curator](https://github.com/NVIDIA/physicsnemo-curator): Inference sub-module + of PhysicsNeMo to streamline and accelerate the process of data curation for engineering + datasets. +- [Earth-2 Studio](https://github.com/NVIDIA/earth2studio): Inference sub-module of PhysicsNeMo + to enable climate researchers and scientists to explore and experiment with pretrained AI models for weather and climate. - -## Installation -### PyPi +### Scalable GPU-Optimized Training Library + +PhysicsNeMo provides a highly optimized and scalable training library for maximizing the +power of NVIDIA GPUs. +[Distributed computing](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.distributed.html) +utilities allow for efficient scaling from a single GPU to multi-node GPU clusters with +a few lines of code, ensuring that large-scale +physics-informed machine learning (ML) models can be trained quickly and effectively. +The framework includes support for advanced +[optimization utilities](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.utils.html#module-physicsnemo.utils.capture), +[tailor-made datapipes](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.datapipes.html), +and [validation utilities](https://github.com/NVIDIA/physicsnemo-sym/tree/main/physicsnemo/sym/eq) +to enhance end-to-end training speed. + +### A Suite of Physics-Informed ML Models + +PhysicsNeMo offers a library of state-of-the-art models specifically designed +for Physics-ML applications. Users can build any model architecture by using the underlying +PyTorch layers and combining them with curated PhysicsNeMo layers. + +The [Model Zoo](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.models.html#model-zoo) +includes optimized implementations of families of model architectures such as +Neural Operators: + +- [Fourier Neural Operators (FNOs)](physicsnemo/models/fno) +- [DeepONet](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-sym/user_guide/neural_operators/deeponet.html) +- [DoMINO](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/examples/cfd/external_aerodynamics/domino/readme.html) +- [Graph Neural Networks (GNNs)](physicsnemo/models/gnn_layers) +- [MeshGraphNet](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/examples/cfd/vortex_shedding_mgn/readme.html) +- [MeshGraphNet for Lagrangian](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/examples/cfd/lagrangian_mgn/readme.html) +- [XAeroNet](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/examples/cfd/external_aerodynamics/xaeronet/readme.html) +- [Diffusion Models](physicsnemo/models/diffusion) +- [Correction Diffusion Model](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/examples/generative/corrdiff/readme.html) +- [DDPM](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/examples/generative/diffusion/readme.html) +- [PhysicsNeMo GraphCast](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/examples/weather/graphcast/readme.html) +- [Transsolver](https://github.com/NVIDIA/physicsnemo/tree/main/examples/cfd/darcy_transolver) +- [RNNs](https://github.com/NVIDIA/physicsnemo/tree/main/physicsnemo/models) +- [SwinVRNN](https://github.com/NVIDIA/physicsnemo/tree/main/physicsnemo/models/swinvrnn) +- [Physics-Informed Neural Networks (PINNs)](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-sym/user_guide/foundational/1d_wave_equation.html) + +And many others. + +These models are optimized for various physics domains, such as computational fluid +dynamics, structural mechanics, and electromagnetics. Users can download, customize, and +build upon these models to suit their specific needs, significantly reducing the time +required to develop high-fidelity simulations. + +### Seamless PyTorch Integration + +PhysicsNeMo is built on top of PyTorch, providing a familiar and user-friendly experience +for those already proficient with PyTorch. +This includes a simple Python interface and modular design, making it easy to use +PhysicsNeMo with existing PyTorch workflows. +Users can leverage the extensive PyTorch ecosystem, including its libraries and tools, +while benefiting from PhysicsNeMo's specialized capabilities for physics-ML. This seamless +integration ensures users can quickly adopt PhysicsNeMo without a steep learning curve. + +For more information, refer to [Converting PyTorch Models to PhysicsNeMo Models](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.models.html#converting-pytorch-models-to-physicsnemo-models). + +### Easy Customization and Extension + +PhysicsNeMo is designed to be highly extensible, allowing users to add new functionality +with minimal effort. The framework provides Pythonic APIs for +defining new physics models, geometries, and constraints, making it easy to extend its +capabilities to new use cases. +The adaptability of PhysicsNeMo is further enhanced by key features such as +[ONNX support](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.deploy.html) +for flexible model deployment, +robust [logging utilities](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.launch.logging.html) +for streamlined error handling, +and efficient +[checkpointing](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.launch.utils.html#module-physicsnemo.launch.utils.checkpoint) +to simplify model loading and saving. + +This extensibility ensures that PhysicsNeMo can adapt to the evolving needs of researchers +and engineers, facilitating the development of innovative solutions in the field of physics-ML. + +Detailed information on features and capabilities can be found in the [PhysicsNeMo documentation](https://docs.nvidia.com/physicsnemo/index.html#core). + +[Reference samples](examples/README.md) cover a broad spectrum of physics-constrained +and data-driven +workflows to suit the diversity of use cases in the science and engineering disciplines. -The recommended method for installing the latest version of Modulus is using PyPi: +> [!TIP] +> Have questions about how PhysicsNeMo can assist you? Try our [Experimental] chatbot, +> [PhysicsNeMo Guide](https://chatgpt.com/g/g-PXrBv20SC-modulus-guide), for answers. -```Bash -pip install nvidia-modulus -``` +### Hello World -The installation can be verified by running a simple python code snippet as shown below: +You can start using PhysicsNeMo in your PyTorch code as simply as shown here: ```python -python >>> import torch ->>> from modulus.models.mlp.fully_connected import FullyConnected +>>> from physicsnemo.models.mlp.fully_connected import FullyConnected >>> model = FullyConnected(in_features=32, out_features=64) >>> input = torch.randn(128, 32) >>> output = model(input) @@ -65,91 +200,331 @@ python torch.Size([128, 64]) ``` -#### Optional dependencies +To use the distributed module, you can do the following (example for +distributed data parallel training; for a more in-depth tutorial, refer to +[PhysicsNeMo Distributed](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.distributed.html#)): + +```python +import torch +from torch.nn.parallel import DistributedDataParallel +from physicsnemo.distributed import DistributedManager +from physicsnemo.models.mlp.fully_connected import FullyConnected + +def main(): + DistributedManager.initialize() + dist = DistributedManager() + + arch = FullyConnected(in_features=32, out_features=64).to(dist.device) + + if dist.distributed: + ddps = torch.cuda.Stream() + with torch.cuda.stream(ddps): + arch = DistributedDataParallel( + arch, + device_ids=[dist.local_rank], + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + ) + torch.cuda.current_stream().wait_stream(ddps) + + # Set up the optimizer + optimizer = torch.optim.Adam( + arch.parameters(), + lr=0.001, + ) + + def training_step(invar, target): + pred = arch(invar) + loss = torch.sum(torch.pow(pred - target, 2)) + loss.backward() + optimizer.step() + return loss + + # Sample training loop + for i in range(20): + # Random inputs and targets for simplicity + input = torch.randn(128, 32, device=dist.device) + target = torch.randn(128, 64, device=dist.device) + + # Training step + loss = training_step(input, target) + +if __name__ == "__main__": + main() +``` + +To use the PDE module, you can do the following: + +```python +>>> from physicsnemo.sym.eq.pdes.navier_stokes import NavierStokes +>>> ns = NavierStokes(nu=0.01, rho=1, dim=2) +>>> ns.pprint() +continuity: u__x + v__y +momentum_x: u*u__x + v*u__y + p__x + u__t - 0.01*u__x__x - 0.01*u__y__y +momentum_y: u*v__x + v*v__y + p__y + v__t - 0.01*v__x__x - 0.01*v__y__y +``` + +## Who is Using and Contributing to PhysicsNeMo + +PhysicsNeMo is an open-source project and gets contributions from researchers in +the SciML and AI4Science fields. While the PhysicsNeMo team works on optimizing the +underlying software stack, the community collaborates and contributes model architectures, +datasets, and reference applications so we can innovate in the pursuit of +developing generalizable model architectures and algorithms. + +Some recent examples of community contributors are the [HP Labs 3D Printing team](https://developer.nvidia.com/blog/spotlight-hp-3d-printing-and-nvidia-physicsnemo-collaborate-on-open-source-manufacturing-digital-twin/), +[Stanford Cardiovascular research team](https://developer.nvidia.com/blog/enabling-greater-patient-specific-cardiovascular-care-with-ai-surrogates/), +[UIUC team](https://github.com/NVIDIA/physicsnemo/tree/main/examples/cfd/mhd_pino), +[CMU team](https://github.com/NVIDIA/physicsnemo/tree/main/examples/generative/diffusion), +etc. + +Recent examples of research teams using PhysicsNeMo are the +[ORNL team](https://arxiv.org/abs/2404.05768), +[TU Munich CFD team](https://www.nvidia.com/en-us/on-demand/session/gtc24-s62237/), etc. + +Please navigate to this page for a complete list of research work leveraging PhysicsNeMo. +For a list of enterprises using PhysicsNeMo, refer to the [PhysicsNeMo Webpage](https://developer.nvidia.com/physicsnemo). + +Using PhysicsNeMo and interested in showcasing your work on +[NVIDIA Blogs](https://developer.nvidia.com/blog/category/simulation-modeling-design/)? +Fill out this [proposal form](https://forms.gle/XsBdWp3ji67yZAUF7) and we will get back +to you! + +## Why Are They Using PhysicsNeMo + +Here are some of the key benefits of PhysicsNeMo for SciML model development: + + + | | +---|---|---| +|SciML Benchmarking and Validation|Ease of Using Generalized SciML Recipes with Heterogeneous Datasets |Out-of-the-Box Performance and Scalability +|PhysicsNeMo enables researchers to benchmark their AI models against proven architectures for standard benchmark problems with detailed domain-specific validation criteria.|PhysicsNeMo enables researchers to pick from state-of-the-art SciML architectures and use built-in data pipelines for their use case.| PhysicsNeMo provides out-of-the-box performant training pipelines, including optimized ETL pipelines for heterogeneous engineering and scientific datasets and out-of-the-box scaling across multi-GPU and multi-node GPUs. + + +See what your peer SciML researchers are saying about PhysicsNeMo (coming soon). + +## Getting Started with PhysicsNeMo + +The following resources will help you learn how to use PhysicsNeMo. The best +way is to start with a reference sample and then update it for your own use case. + +- [Using PhysicsNeMo with your PyTorch model](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/tutorials/simple_training_example.html#using-custom-models-in-physicsnemo) +- [Using PhysicsNeMo built-in models](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/tutorials/simple_training_example.html#using-built-in-models) +- [Getting Started Guide](https://docs.nvidia.com/deeplearning/physicsnemo/getting-started/index.html) +- [Reference Samples](https://github.com/NVIDIA/physicsnemo/blob/main/examples/README.md) +- [User Guide Documentation](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/index.html) + +## Learning AI Physics + +- [Explore Jupyter Notebooks on Hugging Face](https://huggingface.co/collections/nvidia/physicsnemo) +- [AI4Science PhysicsNeMo Bootcamp](https://github.com/openhackathons-org/End-to-End-AI-for-Science) +- [Self-Paced DLI Training](https://learn.nvidia.com/courses/course-detail?course_id=course-v1:DLI+S-OV-04+V1) +- [Deep Learning for Science and Engineering Lecture Series](https://www.nvidia.com/en-us/on-demand/deep-learning-for-science-and-engineering/) +- [Video Tutorials](https://www.nvidia.com/en-us/on-demand/search/?facet.mimetype[]=event%20session&layout=list&page=1&q=physicsnemo&sort=relevance&sortDir=desc) + +## Resources + +- [Getting Started Webinar](https://www.nvidia.com/en-us/on-demand/session/gtc24-dlit61460/?playlistId=playList-bd07f4dc-1397-4783-a959-65cec79aa985) +- [PhysicsNeMo: Purpose and Usage](https://www.nvidia.com/en-us/on-demand/session/dliteachingkit-setk5002/) +- [AI4Science PhysicsNeMo Bootcamp](https://github.com/openhackathons-org/End-to-End-AI-for-Science) +- [PhysicsNeMo Pretrained Models](https://catalog.ngc.nvidia.com/models?filters=&orderBy=scoreDESC&query=PhysicsNeMo&page=&pageSize=) +- [PhysicsNeMo Datasets and Supplementary Materials](https://catalog.ngc.nvidia.com/resources?filters=&orderBy=scoreDESC&query=PhysicsNeMo&page=&pageSize=) + +## Installation + +The following instructions help you install the base PhysicsNeMo modules to get +started. In addition to this, optional dependencies can be installed to provide +additional functionality. A complete list of optional dependencies is available +in the [`pyproject.toml`](./pyproject.toml) file. + +The [training recipes](./examples) are not packaged into the pip wheels or the +container to keep the footprint low. We recommend users clone the appropriate +training recipes and use them as a starting point. These training recipes may +require additional example-specific dependencies, as indicated through an +associated `requirements.txt` file in such cases. + +### CUDA Backend Selection + +> **Important:** To get GPU-accelerated RAPIDS packages (cuML, pylibraft, cupy) +> and a CUDA-matched PyTorch build, you **must** include either `cu13` or `cu12` +> when installing. Feature extras like `nn-extras` and `utils-extras` provide +> additional non-CUDA packages (scipy, natten, wandb, etc.) but do not include +> RAPIDS dependencies on their own. + +PhysicsNeMo supports both CUDA 12 and CUDA 13 backends. The backend is selected +via an extra that is orthogonal to the feature extras - combine them freely: + +| Extra | What it provides | +| --- | --- | +| `cu13` | PyTorch (CUDA 13.0), cuML-cu13, pylibraft-cu13, cupy-cuda13x | +| `cu12` | PyTorch (CUDA 12.8), cuML-cu12, pylibraft-cu12, cupy-cuda12x | +| *(neither)* | PyTorch from PyPI (default build), **no RAPIDS packages** | + +### PyPI + +The recommended method for installing the latest version of PhysicsNeMo is using PyPI: + +```Bash +pip install nvidia-physicsnemo +python -c "import physicsnemo; print('PhysicsNeMo version:', physicsnemo.__version__)" +``` + +To install with a specific CUDA backend and optional feature extras: + +```Bash +# CUDA 13 backend with nn-extras +pip install "nvidia-physicsnemo[cu13,nn-extras]" + +# CUDA 12 backend with nn-extras +pip install "nvidia-physicsnemo[cu12,nn-extras]" +``` + +Other feature extras (`utils-extras`, `mesh-extras`, `model-extras`, +`datapipes-extras`, `gnns`) can be combined in the same way. + +The installation can also be verified by running the [Hello World](#hello-world) example. + +### uv + +For development or to run examples, we recommend using [uv](https://docs.astral.sh/uv/) +to clone the repository and sync dependencies: + +```Bash +git clone https://github.com/NVIDIA/physicsnemo.git +cd physicsnemo +uv sync --extra cu13 +uv run python -c "import physicsnemo; print('PhysicsNeMo version:', physicsnemo.__version__)" +``` + +To install with optional feature extras (e.g., `nn-extras`): + +```Bash +uv sync --extra cu13 --extra nn-extras +``` + +For a CUDA 12 environment, replace `cu13` with `cu12`: -Modulus has many optional dependencies that are used in specific components. -When using pip, all dependencies used in Modulus can be installed with -`pip install modulus[all]`. If you are developing Modulus, developer dependencies -can be installed using `pip install modulus[dev]`. Otherwise, additional dependencies -can be installed on a case by case basis. A detailed information on installing the -optional dependencies can be found in the -[Getting Started Guide](https://docs.nvidia.com/deeplearning/modulus/getting-started/index.html). +```Bash +uv sync --extra cu12 --extra nn-extras +``` ### NVCR Container -The recommended Modulus docker image can be pulled from the -[NVIDIA Container Registry](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/modulus/containers/modulus): +The recommended PhysicsNeMo Docker image can be pulled from the +[NVIDIA Container Registry](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/physicsnemo/containers/physicsnemo) +(refer to the NGC registry for the latest tag): ```Bash -docker pull nvcr.io/nvidia/modulus/modulus:23.11 +docker pull nvcr.io/nvidia/physicsnemo/physicsnemo:25.06 ``` -Inside the container you can clone the Modulus git repositories and get started with the -examples. Below command show the instructions to launch the modulus container and run an -examples from this repo. +Inside the container, you can clone the PhysicsNeMo git repositories and get +started with the examples. The command below shows the instructions to launch +the PhysicsNeMo container and run examples from this repo: ```bash docker run --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 --runtime nvidia \ ---rm -it nvcr.io/nvidia/modulus/modulus:23.11 bash -git clone https://github.com/NVIDIA/modulus.git -cd modulus/examples/cfd/darcy_fno/ -pip install warp-lang # install NVIDIA Warp to run the darcy example +--rm -it nvcr.io/nvidia/physicsnemo/physicsnemo:25.06 bash +git clone https://github.com/NVIDIA/physicsnemo.git +cd physicsnemo/examples/cfd/darcy_fno/ +pip install warp-lang # install NVIDIA Warp to run the Darcy example python train_fno_darcy.py ``` -## From Source - -### Package +### From Source -For a local build of the Modulus Python package from source use: +For a local build of the PhysicsNeMo Python package from source, use: ```Bash -git clone git@github.com:NVIDIA/modulus.git && cd modulus +git clone git@github.com:NVIDIA/physicsnemo.git && cd physicsnemo pip install --upgrade pip pip install . +python -c "import physicsnemo; print('PhysicsNeMo version:', physicsnemo.__version__)" ``` -### Source Container +### Building Docker from Source -To build Modulus docker image: +To build the PhysicsNeMo Docker image: ```bash -docker build -t modulus:deploy \ +docker build -t physicsnemo:deploy \ --build-arg TARGETPLATFORM=linux/amd64 --target deploy -f Dockerfile . ``` -Alternatively, you can run `make container-deploy` +Alternatively, you can run `make container-deploy`. -To build CI image: +To build the CI image: ```bash -docker build -t modulus:ci \ +docker build -t physicsnemo:ci \ --build-arg TARGETPLATFORM=linux/amd64 --target ci -f Dockerfile . ``` Alternatively, you can run `make container-ci`. -Currently only `linux/amd64` and `linux/arm64` platforms are supported. If using -`linux/arm64`, some dependencies like `warp-lang` might not install correctly. +### Platform Support + +For pip or uv installation, Linux, macOS (ARM), and Windows are supported. + +Docker containers are available for `linux/amd64` and `linux/arm64` platforms only. +If using `linux/arm64`, some dependencies like `warp-lang` might not install correctly. + +## PhysicsNeMo Migration Guide -## Contributing +NVIDIA Modulus has been renamed to NVIDIA PhysicsNeMo. For migration: -Modulus is an open source collaboration and its success is rooted in community -contribution to further the field of Physics-ML. Thank you for contributing to the -project so others can build on your contribution. -For guidance on making a contribution to Modulus, please refer to the -[contributing guidelines](https://github.com/NVIDIA/modulus/blob/main/CONTRIBUTING.md). +- Use `pip install nvidia-physicsnemo` rather than `pip install nvidia-modulus` + for PyPI wheels. +- Use `nvcr.io/nvidia/physicsnemo/physicsnemo:` rather than + `nvcr.io/nvidia/modulus/modulus:` for Docker containers. +- Replace `nvidia-modulus` with `nvidia-physicsnemo` in your pip requirements + files (`requirements.txt`, `setup.py`, `setup.cfg`, `pyproject.toml`, etc.). +- In your code, change the import statements from `import modulus` to + `import physicsnemo`. + +The old PyPI registry and the NGC container registry will be deprecated soon +and will not receive any bug fixes/updates. The old checkpoints will remain +compatible with these updates. + +More details to follow soon. + +## DGL to PyTorch Geometric Migration Guide + +PhysicsNeMo supports a wide range of Graph Neural Networks (GNNs), +including MeshGraphNet and others. +Currently, PhysicsNeMo uses the DGL library as its GNN backend, +with plans to completely transition to PyTorch Geometric (PyG) in a future release. +For more details, please refer to the [DGL-to-PyG migration guide](https://github.com/NVIDIA/physicsnemo/blob/main/examples/dgl_to_pyg_migration.md). + +## Contributing to PhysicsNeMo + +PhysicsNeMo is an open-source collaboration, and its success is rooted in community +contributions to further the field of Physics-ML. Thank you for contributing to the +project so others can build on top of your contributions. + +For guidance on contributing to PhysicsNeMo, please refer to the +[contributing guidelines](CONTRIBUTING.md). + +## Cite PhysicsNeMo + +If PhysicsNeMo helped your research and you would like to cite it, please refer to the [guidelines](https://github.com/NVIDIA/physicsnemo/blob/main/CITATION.cff). ## Communication -- Github Discussions: Discuss new architectures, implementations, Physics-ML research, etc. +- GitHub Discussions: Discuss new architectures, implementations, Physics-ML research, etc. - GitHub Issues: Bug reports, feature requests, install issues, etc. -- Modulus Forum: The [Modulus Forum](https://forums.developer.nvidia.com/c/physics-simulation/modulus-physics-ml-model-framework) -hosts an audience of new to moderate level users and developers for general chat, online +- PhysicsNeMo Forum: The [PhysicsNeMo Forum](https://forums.developer.nvidia.com/t/welcome-to-the-physicsnemo-ml-model-framework-forum/178556) +hosts an audience of new to moderate-level users and developers for general chat, online discussions, collaboration, etc. +## Feedback + +Want to suggest some improvements to PhysicsNeMo? Use our [feedback form](https://docs.google.com/forms/d/e/1FAIpQLSfX4zZ0Lp7MMxzi3xqvzX4IQDdWbkNh5H_a_clzIhclE2oSBQ/viewform?usp=sf_link). + ## License -Modulus is provided under the Apache License 2.0, please see [LICENSE.txt](./LICENSE.txt) -for full license text. +PhysicsNeMo is provided under the Apache License 2.0. Please see [LICENSE.txt](./LICENSE.txt) +for the full license text. Enterprise SLA, support, and preview access are available +under NVAIE. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000000..76b0f06726 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,34 @@ +# Security + +NVIDIA is dedicated to the security and trust of our software products and +services, including all source code repositories managed through our organization. + +If you need to report a security issue, please use the appropriate contact points +outlined below. **Please do not report security vulnerabilities through GitHub/GitLab.** + +## Reporting Potential Security Vulnerability in an NVIDIA Product + +To report a potential security vulnerability in any NVIDIA product: + +- Web: [Security Vulnerability Submission Form](https://www.nvidia.com/object/submit-security-vulnerability.html) +- E-Mail: `psirt@nvidia.com` + - We encourage you to use the following PGP key for secure email communication: + [NVIDIA public PGP Key for communication](https://www.nvidia.com/en-us/security/pgp-key) + - Please include the following information: + - Product/Driver name and version/branch that contains the vulnerability + - Type of vulnerability (code execution, denial of service, buffer overflow, etc.) + - Instructions to reproduce the vulnerability + - Proof-of-concept or exploit code + - Potential impact of the vulnerability, including how an attacker could + exploit the vulnerability + +While NVIDIA currently does not have a bug bounty program, we do offer +acknowledgement when an externally reported security issue is addressed under our +coordinated vulnerability disclosure policy. Please visit our +[Product Security Incident Response Team (PSIRT)](https://www.nvidia.com/en-us/security/psirt-policies/) +policies page for more information. + +## NVIDIA Product Security + +For all security-related concerns, please visit NVIDIA's Product Security portal +at `https://www.nvidia.com/en-us/security` diff --git a/asv.conf.json b/asv.conf.json new file mode 100644 index 0000000000..d9ca69d317 --- /dev/null +++ b/asv.conf.json @@ -0,0 +1,25 @@ +{ + "version": 1, + "project": "physicsnemo", + "project_url": "https://github.com/NVIDIA/physicsnemo", + "repo": ".", + "branches": ["main"], + "dvcs": "git", + "environment_type": "virtualenv", + "install_timeout": 600, + "show_commit_url": "https://github.com/NVIDIA/physicsnemo/commit/", + "pythons": ["3.12"], + "build_command": ["python -m pip wheel --no-deps --wheel-dir {build_cache_dir} {build_dir}"], + "install_command": ["in-dir={env_dir} python -m pip install {wheel_file}"], + "uninstall_command": ["return-code=any python -m pip uninstall -y nvidia-{project}"], + "benchmark_dir": "benchmarks", + "env_dir": ".asv/env", + "results_dir": ".asv/results", + "html_dir": ".asv/html", + "matrix": { + "req": { + "cuml-cu13": [], + "scipy": [] + } + } +} diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 0000000000..159e9fcc8e --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,70 @@ +# PhysicsNeMo Benchmarks with ASV + +This directory contains ASV-based benchmarks for PhysicsNeMo. Benchmarks are +discovered from the `benchmarks/` tree and configured via `asv.conf.json` in the +repository root. + +Resources: + +* [ASV documentation](https://asv.readthedocs.io/en/latest/index.html) + +## Running a benchmark + +Run all benchmarks from the repo root: + +```sh +./benchmarks/run_benchmarks.sh +``` + +Note: the first run may take a while because ASV builds its virtual environment. + +Run a subset by name or regex: + +```sh +./benchmarks/run_benchmarks.sh -b knn +``` + +## Publishing and viewing results + +Publish results to the local HTML report: + +```sh +asv publish +``` + +Preview the report in a local web server: + +```sh +asv preview +``` + +The generated site is written to `.asv/html/` (open `index.html` if you prefer). + +## Adding a new benchmark + +1. Add a new file under `benchmarks/` following the package structure (for + example, `benchmarks/physicsnemo/nn/neighbors/my_benchmark.py`). +2. Define a benchmark class and at least one `time_*` method. + See [documentation](https://asv.readthedocs.io/en/latest/writing_benchmarks.html#benchmark-types) + for available benchmark types. +3. Use `setup()` to create inputs and keep benchmarks deterministic. + See [documentation](https://asv.readthedocs.io/en/latest/writing_benchmarks.html#benchmark-attributes) + for available benchmark attributes. + +Example: + +```py +import torch + + +class MyOpBenchmark: + params = [1024, 4096] + param_names = ["n"] + + def setup(self, n: int) -> None: + self.x = torch.randn(n, n, device="cuda") + + def time_my_op(self, n: int) -> None: + _ = self.x @ self.x + torch.cuda.synchronize() +``` diff --git a/benchmarks/__init__.py b/benchmarks/__init__.py new file mode 100644 index 0000000000..6397b6eea2 --- /dev/null +++ b/benchmarks/__init__.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ASV benchmarks for PhysicsNeMo.""" diff --git a/benchmarks/physicsnemo/__init__.py b/benchmarks/physicsnemo/__init__.py new file mode 100644 index 0000000000..6397b6eea2 --- /dev/null +++ b/benchmarks/physicsnemo/__init__.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ASV benchmarks for PhysicsNeMo.""" diff --git a/benchmarks/physicsnemo/nn/README.md b/benchmarks/physicsnemo/nn/README.md new file mode 100644 index 0000000000..ec95419d23 --- /dev/null +++ b/benchmarks/physicsnemo/nn/README.md @@ -0,0 +1,50 @@ +# PhysicsNeMo NN Benchmarks + +This directory contains ASV benchmarks for `physicsnemo.nn`. + +For functionals, the benchmark flow is intentionally simple: + +1. Implement or update the functional `FunctionSpec`. +2. Add representative `make_inputs(device=...)` cases to that `FunctionSpec`. +3. Register the `FunctionSpec` in `benchmarks/physicsnemo/nn/functional/registry.py`. +4. Run ASV and regenerate plots. + +## Where to read more + +- Functional benchmark rules and expectations: + - `CODING_STANDARDS/FUNCTIONAL_APIS.md` +- `FunctionSpec` behavior and required hooks: + - `physicsnemo/core/function_spec.py` + +## Where to edit + +- Benchmark registry (which functionals are benchmarked): + - `benchmarks/physicsnemo/nn/functional/registry.py` +- ASV benchmark runner for functionals: + - `benchmarks/physicsnemo/nn/functional/benchmark_functionals.py` +- Plot generation: + - `benchmarks/physicsnemo/nn/functional/plot_functional_benchmarks.py` + +## Example functionals to copy + +- `physicsnemo/nn/functional/interpolation/interpolation.py` +- `physicsnemo/nn/functional/radius_search/radius_search.py` +- `physicsnemo/nn/functional/knn/knn.py` + +## Common commands + +Run benchmarks (repo root): + +```bash +./benchmarks/run_benchmarks.sh +``` + +Run only selected functionals while iterating: + +```bash +PHYSICSNEMO_ASV_FUNCTIONALS=Interpolation,RadiusSearch ./benchmarks/run_benchmarks.sh +``` + +Plots are written under: + +- `docs/nn/functional//benchmark.png` diff --git a/benchmarks/physicsnemo/nn/__init__.py b/benchmarks/physicsnemo/nn/__init__.py new file mode 100644 index 0000000000..6397b6eea2 --- /dev/null +++ b/benchmarks/physicsnemo/nn/__init__.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ASV benchmarks for PhysicsNeMo.""" diff --git a/benchmarks/physicsnemo/nn/functional/__init__.py b/benchmarks/physicsnemo/nn/functional/__init__.py new file mode 100644 index 0000000000..7fea4c73cf --- /dev/null +++ b/benchmarks/physicsnemo/nn/functional/__init__.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ASV benchmarks for functional APIs.""" diff --git a/benchmarks/physicsnemo/nn/functional/benchmark_functionals.py b/benchmarks/physicsnemo/nn/functional/benchmark_functionals.py new file mode 100644 index 0000000000..2d530f149f --- /dev/null +++ b/benchmarks/physicsnemo/nn/functional/benchmark_functionals.py @@ -0,0 +1,129 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ASV benchmarks for PhysicsNeMo functionals.""" +# TODO: This code will likely evolve with CI/CD integration. + +from __future__ import annotations + +import os +from typing import Any, Iterable + +import torch + +from benchmarks.physicsnemo.nn.functional.registry import FUNCTIONAL_SPECS + + +def _resolve_device() -> torch.device: + """Resolve the device to benchmark on.""" + + # Allow the benchmark device to be overridden from the environment. + device_name = os.getenv("PHYSICSNEMO_ASV_DEVICE") + if device_name: + return torch.device(device_name) + + # Prefer CUDA when available; otherwise default to CPU. + if torch.cuda.is_available(): + return torch.device("cuda") + return torch.device("cpu") + + +def _filter_specs(specs: Iterable[type]) -> list[type]: + """Filter the specs to the requested subset, this is mostly used for debugging locally.""" + + # Allow selecting a subset of functionals for quick benchmark iteration. + spec_filter = os.getenv("PHYSICSNEMO_ASV_FUNCTIONALS") + if not spec_filter: + return list(specs) + + # Parse comma-separated spec names into a normalized lookup set. + requested = { + name.strip().lower() for name in spec_filter.split(",") if name.strip() + } + if not requested: + return list(specs) + + # Keep only specs explicitly requested by name. + selected = [spec for spec in specs if spec.__name__.lower() in requested] + if not selected: + available = ", ".join(sorted(spec.__name__ for spec in specs)) + raise ValueError( + "PHYSICSNEMO_ASV_FUNCTIONALS did not match any FunctionSpec. " + f"Requested: {spec_filter!r}. Available: {available}" + ) + return selected + + +# Resolve benchmark configuration and precompute all ASV parameter tuples. +_DEVICE = _resolve_device() +_PARAMS: list[tuple[str, str, int]] = [] +_SELECTED_SPECS = _filter_specs(FUNCTIONAL_SPECS) +_WORK_ITEMS: dict[ + tuple[str, str, int], tuple[type, str, tuple[Any, ...], dict[str, Any]] +] = {} + +# Build the ASV parameter triples: (spec_name, implementation_name, case_index). +for spec in _SELECTED_SPECS: + # Skip specs that currently have no dispatchable implementations. + implementations = spec.available_implementations() + if not implementations: + continue + + # Materialize inputs once so ASV setup can index by case id. + # TODO: This is not ideal, we should keep make_inputs as a generator + cases = list(spec.make_inputs(device=_DEVICE)) + if not cases: + continue + + # Build ASV parameter triples and cache resolved work items for setup(). + for impl in implementations: + for case_index, case in enumerate(cases): + label, args, kwargs = case + key = (spec.__name__, impl, case_index) + _PARAMS.append(key) + _WORK_ITEMS[key] = (spec, label, args, kwargs) + + +class FunctionalBenchmarks: + """Benchmark registered FunctionSpec implementations with ASV.""" + + # ASV expects params to be a list of parameter axes. + params = [_PARAMS] + param_names = ["spec_impl_case"] + timeout = 120 + + def setup(self, spec_impl_case: tuple[str, str, int]) -> None: + # Resolve the precomputed work item for this benchmark key. + spec, _, args, kwargs = _WORK_ITEMS[spec_impl_case] + + # Cache resolved objects on self to minimize per-iteration overhead. + self.spec = spec + self.implementation = spec_impl_case[1] + self.args = args + self.kwargs = kwargs + + # Synchronize before timing so previous CUDA work is excluded. + if _DEVICE.type == "cuda": + torch.cuda.synchronize() + + def time_functional(self, spec_impl_case: tuple[str, str, int]) -> None: + # Dispatch to the selected implementation for the selected input case. + self.spec.dispatch( + *self.args, **self.kwargs, implementation=self.implementation + ) + # Synchronize to ensure the measured time includes kernel execution. + if _DEVICE.type == "cuda": + torch.cuda.synchronize() diff --git a/benchmarks/physicsnemo/nn/functional/plot_functional_benchmarks.py b/benchmarks/physicsnemo/nn/functional/plot_functional_benchmarks.py new file mode 100644 index 0000000000..95fb74ba8b --- /dev/null +++ b/benchmarks/physicsnemo/nn/functional/plot_functional_benchmarks.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Generate bar plots for functional benchmarks from ASV results.""" +# TODO: This code is not meant to be a long term solution. As things progress we will +# update this script for better automated plot generation. + +from __future__ import annotations + +import argparse +import ast +import json +from pathlib import Path +from typing import Any + +from benchmarks.physicsnemo.nn.functional.registry import FUNCTIONAL_SPECS + +# Map each FunctionSpec to its docs output directory. +_SPEC_OUTPUT_SLUG = { + "DropPath": "drop_path", + "KNN": "knn", + "RFFT": "rfft", + "RFFT2": "rfft2", + "RadiusSearch": "radius_search", + "SignedDistanceField": "sdf", + "IRFFT": "irfft", + "IRFFT2": "irfft2", + "Interpolation": "interpolation", +} + +# Keep implementation order and colors stable across plots. +_IMPL_ORDER = ("warp", "cuml", "scipy", "torch") +_IMPL_COLORS = { + "warp": "#76B900", + "cuml": "#2E2E2E", + "scipy": "#5A5A5A", + "torch": "#111111", + "unknown": "#8A8A8A", +} + +# Match the ASV benchmark function used in benchmark_functionals.py. +_BENCHMARK_SUFFIX = "FunctionalBenchmarks.time_functional" + + +def _build_case_labels() -> dict[str, list[str]]: + # Build case labels directly from make_inputs for each plottable spec. + labels: dict[str, list[str]] = {} + for spec in FUNCTIONAL_SPECS: + if len(spec.available_implementations()) < 2: + continue + labels[spec.__name__] = [ + label for label, _, _ in spec.make_inputs(device="cpu") + ] + return labels + + +def _build_spec_implementations() -> dict[str, list[str]]: + # Build implementation lists for each plottable spec. + implementations: dict[str, list[str]] = {} + for spec in FUNCTIONAL_SPECS: + impls = spec.available_implementations() + if len(impls) < 2: + continue + implementations[spec.__name__] = impls + return implementations + + +def _build_params( + case_labels: dict[str, list[str]], + spec_implementations: dict[str, list[str]], +) -> list[tuple[str, str, int]]: + # Recreate ASV parameter ordering for fallback labels. + params: list[tuple[str, str, int]] = [] + for spec_name, impls in spec_implementations.items(): + for impl_name in impls: + params.extend( + (spec_name, impl_name, case_index) + for case_index in range(len(case_labels[spec_name])) + ) + return params + + +# Materialize spec metadata once. +_SPEC_CASE_LABELS = _build_case_labels() +_SPEC_IMPLEMENTATIONS = _build_spec_implementations() +_PARAMS = _build_params(_SPEC_CASE_LABELS, _SPEC_IMPLEMENTATIONS) + + +def _walk_dicts(value: Any): + # Walk nested dict/list containers and yield dict nodes. + if isinstance(value, dict): + yield value + for nested in value.values(): + yield from _walk_dicts(nested) + elif isinstance(value, list): + for nested in value: + yield from _walk_dicts(nested) + + +def _latest_result_file(results_dir: Path) -> Path: + # Pick the newest ASV result JSON, excluding metadata files. + candidates = [ + path + for path in results_dir.rglob("*.json") + if path.name not in {"benchmarks.json", "machine.json"} + ] + return max(candidates, key=lambda path: path.stat().st_mtime) + + +def _benchmark_entry(data: dict[str, Any]) -> Any: + # Find the benchmark entry for the functional benchmark suite. + for mapping in _walk_dicts(data): + for key, value in mapping.items(): + if isinstance(key, str) and _BENCHMARK_SUFFIX in key: + return value + raise KeyError(f"Unable to find benchmark entry for {_BENCHMARK_SUFFIX}") + + +def _entry_vectors(entry: Any) -> tuple[list[float | None], list[str]]: + # Normalize ASV payload into (values, labels). + if isinstance(entry, dict): + entry = entry.get("result", entry.get("results")) + + values = entry[0] + labels = ( + entry[1] if len(entry) > 1 else [str(param) for param in _PARAMS[: len(values)]] + ) + if labels and isinstance(labels[0], list): + labels = labels[0] + return values, labels + + +def _plot_benchmarks( + values: list[float | None], labels: list[str], output_root: Path +) -> None: + # Import plotting dependency only for plotting. + import matplotlib.pyplot as plt + + # Build spec -> case -> implementation -> value map from ASV vectors. + data: dict[str, dict[str, dict[str, float]]] = {} + for label, value in zip(labels, values): + if value is None: + continue + spec_name, impl_name, case_index = ast.literal_eval(label) + if spec_name not in _SPEC_CASE_LABELS: + continue + case_label = _SPEC_CASE_LABELS[spec_name][case_index] + data.setdefault(spec_name, {}).setdefault(case_label, {})[impl_name] = value + + # Render one grouped bar chart per spec. + for spec_name, case_map in data.items(): + output_dir = output_root / _SPEC_OUTPUT_SLUG.get(spec_name, spec_name.lower()) + output_dir.mkdir(parents=True, exist_ok=True) + + # Build case order and implementation order for this spec. + case_labels = [ + label for label in _SPEC_CASE_LABELS[spec_name] if label in case_map + ] + impl_names = sorted( + {impl for impl_map in case_map.values() for impl in impl_map}, + key=lambda name: (_IMPL_ORDER.index(name) if name in _IMPL_ORDER else 99), + ) + if len(impl_names) < 2: + continue + + # Create and style the figure. + fig, ax = plt.subplots(figsize=(8, 4)) + fig.patch.set_facecolor("white") + ax.set_facecolor("white") + + # Draw grouped bars for each implementation. + bar_width = 0.8 / len(impl_names) + x_positions = list(range(len(case_labels))) + for idx, impl_name in enumerate(impl_names): + offsets = [x + idx * bar_width for x in x_positions] + y_values = [ + case_map[label].get(impl_name, float("nan")) for label in case_labels + ] + ax.bar( + offsets, + y_values, + width=bar_width, + color=_IMPL_COLORS.get(impl_name, _IMPL_COLORS["unknown"]), + label=impl_name, + ) + + # Configure axes and legend. + tick_positions = [ + x + bar_width * (len(impl_names) - 1) / 2 for x in x_positions + ] + ax.set_xticks(tick_positions) + ax.set_xticklabels(case_labels, rotation=20, ha="right") + ax.set_ylabel("Time (s)") + ax.set_title(f"{spec_name} Benchmark", color="#111111") + ax.grid(axis="y", linestyle=":", color="#E0E0E0") + ax.spines["top"].set_visible(False) + ax.spines["right"].set_visible(False) + ax.tick_params(axis="x", colors="#111111") + ax.tick_params(axis="y", colors="#111111") + ax.legend( + frameon=False, fontsize="small", loc="upper left", bbox_to_anchor=(1.02, 1) + ) + + # Save figure to docs image path. + fig.tight_layout() + fig.savefig(output_dir / "benchmark.png") + plt.close(fig) + + +def main() -> int: + # Parse command-line paths. + parser = argparse.ArgumentParser( + description="Generate functional benchmark bar plots from ASV results." + ) + parser.add_argument("--results-dir", type=Path, default=Path(".asv/results")) + parser.add_argument("--output-root", type=Path, default=Path("docs/nn/functional")) + args = parser.parse_args() + + # Load the newest ASV result payload. + results_file = _latest_result_file(args.results_dir) + data = json.loads(results_file.read_text()) + + # Extract benchmark vectors from the ASV payload. + entry = _benchmark_entry(data) + values, labels = _entry_vectors(entry) + + # Generate all benchmark plots. + _plot_benchmarks(values, labels, args.output_root) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/benchmarks/physicsnemo/nn/functional/registry.py b/benchmarks/physicsnemo/nn/functional/registry.py new file mode 100644 index 0000000000..fa588f1a41 --- /dev/null +++ b/benchmarks/physicsnemo/nn/functional/registry.py @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Registry of FunctionSpec classes to benchmark with ASV.""" + +from physicsnemo.nn.functional.drop_path import DropPath +from physicsnemo.nn.functional.fft import IRFFT, IRFFT2, RFFT, RFFT2 +from physicsnemo.nn.functional.interpolation.interpolation import Interpolation +from physicsnemo.nn.functional.knn.knn import KNN +from physicsnemo.nn.functional.radius_search.radius_search import RadiusSearch +from physicsnemo.nn.functional.sdf import SignedDistanceField + +# FunctionSpec classes listed here must implement ``make_inputs`` for ASV. +FUNCTIONAL_SPECS = ( + DropPath, + KNN, + Interpolation, + RadiusSearch, + SignedDistanceField, + RFFT, + RFFT2, + IRFFT, + IRFFT2, +) + +__all__ = ["FUNCTIONAL_SPECS"] diff --git a/benchmarks/physicsnemo/nn/neighbors/__init__.py b/benchmarks/physicsnemo/nn/neighbors/__init__.py new file mode 100644 index 0000000000..6397b6eea2 --- /dev/null +++ b/benchmarks/physicsnemo/nn/neighbors/__init__.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""ASV benchmarks for PhysicsNeMo.""" diff --git a/benchmarks/physicsnemo/nn/neighbors/knn.py b/benchmarks/physicsnemo/nn/neighbors/knn.py new file mode 100644 index 0000000000..b8f70e023e --- /dev/null +++ b/benchmarks/physicsnemo/nn/neighbors/knn.py @@ -0,0 +1,75 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +ASV benchmarks for the knn function. +""" + +import torch + +from physicsnemo.nn.functional import knn + + +class KNNBenchmark: + """Benchmark suite for the knn function.""" + + bench_params = { + "n_points": [10000, 100000], + "n_queries": [1000, 5000], + "k": [5, 16], + "implementation": ["cuml", "scipy", "torch"], + } + + # ASV benchmark attributes. + # https://asv.readthedocs.io/en/latest/benchmarks.html#benchmark-attributes + params = list(bench_params.values()) + param_names = list(bench_params.keys()) + + # Timeout for each benchmark (seconds). + timeout = 60 + + def setup(self, n_points: int, n_queries: int, k: int, implementation: str) -> None: + """Set up test data for the benchmark.""" + # CUDA is required for the cuML implementation. + if not torch.cuda.is_available(): + raise RuntimeError("CUDA not available") + + # Determine device based on implementation. + if implementation in ["cuml", "torch"]: + self.device = "cuda" + elif implementation == "scipy": + self.device = "cpu" + else: + raise ValueError(f"Invalid implementation: {implementation}") + + # Generate random point clouds. + self.points = torch.randn(n_points, 3, device=self.device, dtype=torch.float32) + self.queries = torch.randn( + n_queries, 3, device=self.device, dtype=torch.float32 + ) + self.k = k + self.implementation = implementation + + if self.device == "cuda": + torch.cuda.synchronize() + + def time_knn( + self, n_points: int, n_queries: int, k: int, implementation: str + ) -> None: + """Benchmark the knn function execution time.""" + knn(self.points, self.queries, self.k, implementation=self.implementation) + if self.device == "cuda": + torch.cuda.synchronize() diff --git a/benchmarks/run_benchmarks.sh b/benchmarks/run_benchmarks.sh new file mode 100755 index 0000000000..07e432c0a3 --- /dev/null +++ b/benchmarks/run_benchmarks.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Run ASV benchmarks from the repository root directory. +# Usage: ./benchmarks/run_benchmarks.sh [additional asv arguments] + +set -e + +# Navigate to the repository root directory. +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT_DIR="$(dirname "$SCRIPT_DIR")" +cd "$REPO_ROOT_DIR" + +echo -e "\033[0;32mRunning ASV benchmarks from: $REPO_ROOT_DIR\033[0m" + +# Run ASV with spawn method for CUDA compatibility. +asv run --launch-method spawn "$@" + +# Generate functional benchmark plots if results exist. +python benchmarks/physicsnemo/nn/functional/plot_functional_benchmarks.py diff --git a/docs/Makefile b/docs/Makefile index 4cad7a96a7..7a896be8d1 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -14,21 +14,63 @@ help: .PHONY: help Makefile -# Catch-all target: route all unknown targets to Sphinx using the new -# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). -%: Makefile - @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) - -convert-markdown-to-rst: - @echo "Converting Markdown files to reStructuredText..." - @mkdir -p examples - @find ../examples -name '*.md' -type f -exec sh -c ' \ - mkdir -p "examples/$$(dirname {})"; \ - pandoc "{}" -o "examples/$$(dirname {})/$$(basename -s .md {}).rst"; \ - sed -i ":a; /:alt:/ { N; s/\n / /; ta }" "examples/$$(dirname {})/$$(basename -s .md {}).rst"; \ - sed -i "s|.. figure:: ../../../docs/img|.. figure:: ../../../img|g" "examples/$$(dirname {})/$$(basename -s .md {}).rst" \ - ' \; +EXAMPLE_SRCDIR := ../examples +EXAMPLE_BUILDDIR := examples + +# Find all Markdown files in ../examples +MD_FILES := $(shell find $(EXAMPLE_SRCDIR) -name '*.md') + +# Compute the corresponding .rst output files in examples/ +RST_FILES := $(patsubst $(EXAMPLE_SRCDIR)/%.md,$(EXAMPLE_BUILDDIR)/%.rst,$(MD_FILES)) + + +# Make sure the output dir exists +prepare-examples-dir/: + @mkdir -p $(EXAMPLE_BUILDDIR) + +# Rule to convert each example md to rst: +$(EXAMPLE_BUILDDIR)/%.rst: $(EXAMPLE_SRCDIR)/%.md | prepare-examples-dir + @mkdir -p $(dir $@) + @pandoc $< -o $@ --wrap=none --lua-filter=rewrite_examples_image_paths.lua + +clean-examples: + @rm -r $(EXAMPLE_BUILDDIR) + + +convert-markdown-to-rst: $(RST_FILES) + @echo "All Markdown files have been converted to reStructuredText and post-processed." + +.SECONDARY: $(RST_FILES) + + +.PHONY: convert-markdown-to-rst prepare-examples-dir clean-convert-markdown + html: convert-markdown-to-rst @echo "Running custom commands..." @$(SPHINXBUILD) -M html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +# package docs for CMS upload +package_docs: + apt-get update && apt-get install -y zip unzip + rm -rf deeplearning/ + mkdir deeplearning + mkdir deeplearning/physicsnemo + mkdir deeplearning/physicsnemo/physicsnemo-core + cp -r _build/html/* deeplearning/physicsnemo/physicsnemo-core/ + zip -r deeplearning.zip deeplearning + +.PHONY: clean + +clean: clean-examples + @echo "All generated files have been cleaned." + + +# Catch-all for Sphinx, but ignore file-like targets +%: Makefile + @if echo "$@" | grep -q '\.'; then \ + :; \ + else \ + echo "Running $@..."; \ + $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O); \ + fi diff --git a/docs/_templates/layout.html b/docs/_templates/layout.html new file mode 100644 index 0000000000..2f473f38ee --- /dev/null +++ b/docs/_templates/layout.html @@ -0,0 +1,13 @@ +{% extends "!layout.html" %} + +{% block extrahead %} + + + +{% endblock %} + +{% block footer %} + + + +{% endblock %} diff --git a/docs/api/models/convolutional.rst b/docs/api/models/convolutional.rst new file mode 100644 index 0000000000..4e59893b39 --- /dev/null +++ b/docs/api/models/convolutional.rst @@ -0,0 +1,42 @@ +Convolutional Networks +======================= + +.. autoclass:: physicsnemo.models.pix2pix.pix2pix.Pix2Pix + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.pix2pix.pix2pix.ResnetBlock + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.pix2pix.pix2pixunet.Pix2PixUnet + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.srrn.super_res_net.SRResNet + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.srrn.super_res_net.ConvolutionalBlock3d + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.srrn.super_res_net.PixelShuffle3d + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.srrn.super_res_net.ResidualConvBlock3d + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.srrn.super_res_net.SubPixel_ConvolutionalBlock3d + :show-inheritance: + :members: + :exclude-members: forward diff --git a/docs/api/models/diffusion.rst b/docs/api/models/diffusion.rst new file mode 100644 index 0000000000..0144dd3edf --- /dev/null +++ b/docs/api/models/diffusion.rst @@ -0,0 +1,420 @@ +.. _diffusion_models: + +Diffusion Models +================ + +PhysicsNeMo diffusion library provides three categories of models, that serve +different purposes. All models are based on the +:class:`~physicsnemo.models.module.Module` class. + + - :ref:`Model backbones `: + Those are highly configurable architectures that can be used as a + building block for more complex models. + + - :ref:`Specialized architectures `: + Those are models that usually inherit from the model backbones, with + some specific additional functionalities. + + - :ref:`Application-specific interfaces `: + These Modules are not truly architectures, but rather wrappers around + the model backbones or specialized architectures. Their intent is to + provide a more user-friendly interface for specific applications. + +In addition of these model architectures, PhysicsNeMo provides +:ref:`diffusion preconditioners `, which are +essentially wrappers around model architectures, that rescale the inputs and +outputs of diffusion models to improve their performance. + +.. _diffusion_architecture_backbones: + +Architecture Backbones +---------------------- + +Diffusion model backbones are highly configurable architectures that can be used +as a building block for more complex models. Backbones support +both conditional and unconditional modeling. There are two provided +backbones: the SongUNet, as implemented in the +:class:`~physicsnemo.models.diffusion_unets.SongUNet` class and the DhariwalUNet, +as implemented in the :class:`~physicsnemo.models.diffusion_unets.DhariwalUNet` +class. These models were introduced in the papers `Score-based generative modeling through stochastic +differential equations, Song et al. `_ and +`Diffusion models beat gans on image synthesis, Dhariwal et al. +`_. +The PhysicsNeMo implementation of these models follows closely that used in the paper +`Elucidating the Design Space of Diffusion-Based Generative Models, Karras et al. +`_. The original implementation of these +models can be found in the `EDM repository `_. + +Model backbones can be used as is, such as in in +`the StormCast example <../../examples/weather/stormcast/README.rst>`_, but they can also be used as a base class for +more complex models. + +One of the most common diffusion backbones for image generation is the +:class:`~physicsnemo.models.diffusion_unets.SongUNet` +class. Its latent state :math:`\mathbf{x}` is a tensor of shape :math:`(B, C, H, W)`, +where :math:`B` is the batch size, :math:`C` is the number of channels, +and :math:`H` and :math:`W` are the height and width of the feature map. The +model is conditional on the noise level, and can additionally be conditioned on +vector-valued class labels and/or images. The model is organized into *levels*, +whose number is determined by ``len(channel_mult)``, and each level operates at half the resolution of the +previous level (odd resolutions are rounded down). Each level is composed of a sequence of UNet blocks, that optionally contain +self-attention layers, as controlled by the ``attn_resolutions`` parameter. The feature map resolution +is halved at the first block of each level and then remains constant within the level. + +Here we start by creating a ``SongUNet`` model with three levels, that applies self-attention +at levels one and two. The model is unconditional, *that is,* it is not conditioned on any +class labels or images (but is still conditional on the noise level, as it is +standard practice for diffusion models). + +.. code:: python + + import torch + from physicsnemo.models.diffusion_unets import SongUNet + + B, C_x, res = 3, 6, 40 # Batch size, channels, and resolution of the latent state + + model = SongUNet( + img_resolution=res, + in_channels=C_x, + out_channels=C_x, # No conditioning on image: number of output channels is the same as the input channels + label_dim=0, # No conditioning on vector-valued class labels + augment_dim=0, + model_channels=64, + channel_mult=[1, 2, 3], # 3-levels UNet with 64, 128, and 192 channels at each level, respectively + num_blocks=4, # 4 UNet blocks at each level + attn_resolutions=[20, 10], # Attention is applied at level 1 (resolution 20x20) and level 2 (resolution 10x10) + ) + + x = torch.randn(B, C_x, res, res) # Latent state + noise_labels = torch.randn(B) # Noise level for each sample + + # The feature map resolution is 40 at level 0, 20 at level 1, and 10 at level 2 + out = model(x, noise_labels, None) + print(out.shape) # Shape: (B, C_x, res, res), same as the latent state + + # The same model can be used on images of different resolution + # Note: the attention is still applied at levels 1 and 2 + x_32 = torch.randn(B, C_x, 32, 32) # Lower resolution latent state + out_32 = model(x_32, noise_labels, None) # None means no conditioning on class labels + print(out_32.shape) # Shape: (B, C_x, 32, 32), same as the latent state + +.. _example_song_unet_conditional: + +The unconditional ``SongUNet`` can be extended to be conditional on class labels and/or +images. Conditioning on images is performed by channel-wise concatenation of the image +to the latent state :math:`\mathbf{x}` before passing it to the model. The model does not perform +conditioning on images internally, and this operation is left to the user. For +conditioning on class labels (or any vector-valued quantity whose dimension is ``label_dim``), +the model internally generates embeddings for the class labels +and adds them to intermediate activations within the UNet blocks. Here we +extend the previous example to be conditional on a 16-dimensional vector-valued +class label and a 3-channel image. + +.. code:: python + + import torch + from physicsnemo.models.diffusion_unets import SongUNet + + B, C_x, res = 3, 10, 40 + C_cond = 3 + + model = SongUNet( + img_resolution=res, + in_channels=C_x + C_cond, # Conditioning on an image with C_cond channels + out_channels=C_x, # Output channels: only those of the latent state + label_dim=16, # Conditioning on 16-dimensional vector-valued class labels + augment_dim=0, + model_channels=64, + channel_mult=[1, 2, 2], + num_blocks=4, + attn_resolutions=[20, 10], + ) + + x = torch.randn(B, C_x, res, res) # Latent state + cond = torch.randn(B, C_cond, res, res) # Conditioning image + x_cond = torch.cat([x, cond], dim=1) # Channel-wise concatenation of the conditioning image before passing to the model + noise_labels = torch.randn(B) + class_labels = torch.randn(B, 16) # Conditioning on vector-valued class labels + + out = model(x_cond, noise_labels, class_labels) + print(out.shape) # Shape: (B, C_x, res, res), same as the latent state + +.. _diffusion_specialized_architectures: + +Specialized Architectures +------------------------- + +Even though backbones can be used as is, some of the examples in the +PhysicsNeMo examples use specialized architectures. These specialized architectures +typically inherit from the backbones and implement additional functionalities for specific +applications. For example, the `CorrDiff example <../../examples/weather/corrdiff/README.rst>`_ +uses the specialized architectures :class:`~physicsnemo.models.diffusion_unets.SongUNetPosEmbd` +and :class:`~physicsnemo.models.diffusion_unets.SongUNetPosLtEmbd` to implement +the diffusion model. + +Positional Embeddings +~~~~~~~~~~~~~~~~~~~~~ + +Multi-diffusion (also called *patch-based* diffusion) is a technique to scale +diffusion models to large domains. The idea is to split the full domain into +patches, and run a diffusion model on each patch in parallel. The generated +patches are then fused back to form the final image. This technique is +particularly useful for domains that are too large to fit into the memory of +a single GPU. The `CorrDiff example <../../examples/weather/corrdiff/README.rst>`_ +uses patch-based diffusion for weather downscaling on large domains. A key +ingredient in the implementation of patch-based diffusion is the use of a +global spatial grid, that is used to inform each patch with their respective +position in the full domain. The :class:`~physicsnemo.models.diffusion_unets.SongUNetPosEmbd` +class implements this functionality by providing multiple methods to encode +global spatial coordinates of the pixels into a *global positional embedding grid*. +In addition to multi-diffusion, spatial positional embeddings have also been +observed to improve the quality of the generated images, even for diffusion models +that operate on the full domain. + +The following example shows how to use the specialized architecture +:class:`~physicsnemo.models.diffusion_unets.SongUNetPosEmbd` to implement a +multi-diffusion model: + +Create a ``SongUNetPosEmbd`` model similar to +the one in :ref:`the conditional SongUnet example ` +with a global positional embedding grid of shape ``(C_pos_emb, res, res)``. The model can be used with the entire latent state (full domain). + +.. code:: python + + import torch + from physicsnemo.models.diffusion_unets import SongUNetPosEmbd + + B, C_x, res = 3, 10, 40 + C_cond = 3 + C_PE = 8 # Number of channels in the positional embedding grid + + # Create a SongUNet with a global positional embedding grid of shape (C_PE, res, res) + model = SongUNetPosEmbd( + img_resolution=res, # Define the resolution of the global positional embedding grid + in_channels=C_x + C_cond + C_PE, # in_channels must include the number of channels in the positional embedding grid + out_channels=C_x, + label_dim=16, + augment_dim=0, + model_channels=64, + channel_mult=[1, 2, 2], + num_blocks=4, + attn_resolutions=[20, 10], + gridtype="learnable", # Use a learnable grid of positional embeddings + N_grid_channels=C_PE # Number of channels in the positional embedding grid + ) + + # Can pass the entire latent state to the model + x_global = torch.randn(B, C_x, res, res) # Entire latent state + cond = torch.randn(B, C_cond, res, res) # Conditioning image + x_cond = torch.cat([x_global, cond], dim=1) # Latent state with conditioning image + noise_labels = torch.randn(B) + class_labels = torch.randn(B, 16) + + # The model internally concatenates the global positional embedding grid to the + # input x_cond before the first UNet block. + # Note: global_index=None means use the entire positional embedding grid + out = model(x_cond, noise_labels, class_labels, global_index=None) + print(out.shape) # Shape: (B, C_x, res, res), same as the latent state + +The model can be used on local patches of the latent state +(multi-diffusion approach). We manually extract three patches from the latent +state. Patches are treated as individual samples, so they are concatenated along +the batch dimension. We also create a global grid of indices ``grid`` that +contains the indices of the pixels in the full domain, and we exctract *the same +three patches* from the global grid and pass them to the ``global_index`` +parameter. The model internally uses ``global_index`` to extract the corresponding +patches from the positional embedding grid and concatenate them to the input +``x_cond_patches`` before the first UNet block. Conditional +multi-diffusion still requires that each patch *be conditioned on the entire +conditioning image* ``cond``, which is why we interpolate the conditioning image +to the patch resolution and concatenate it to each individual patch. +In practice it is not necessary to manually extract the patches from the latent +state and the global grid, because PhysicsNeMo provides utilities to help with the +patching operations, in :mod:`~physicsnemo.diffusion.multi_diffusion`. For an example of how +to use these utilities, refer to the `CorrDiff example <../../examples/weather/corrdiff/README.rst>`_. + +.. code:: python + + # Can pass local patches to the model + # Create batch of 3 patches from `x_global` with resolution 16x16 + pres = 16 # Patch resolution + p1 = x_global[0:1, :, :pres, :pres] # Patch 1 + p2 = x_global[3:4, :, pres:2*pres, pres:2*pres] # Patch 2 + p3 = x_global[1:2, :, -pres:, pres:2*pres] # Patch 3 + patches = torch.cat([p1, p2, p3], dim=0) # Batch of 3 patches + + # Note: the conditioning image needs interpolation (or other operations) to + # match the patch resolution + cond1 = torch.nn.functional.interpolate(cond[0:1], size=(pres, pres), mode="bilinear") + cond2 = torch.nn.functional.interpolate(cond[3:4], size=(pres, pres), mode="bilinear") + cond3 = torch.nn.functional.interpolate(cond[1:2], size=(pres, pres), mode="bilinear") + cond_patches = torch.cat([cond1, cond2, cond3], dim=0) + + # Concatenate the patches and the conditioning image + x_cond_patches = torch.cat([patches, cond_patches], dim=1) + + # Create corresponding global indices for the patches + Ny, Nx = torch.arange(res).int(), torch.arange(res).int() + grid = torch.stack(torch.meshgrid(Ny, Nx, indexing="ij"), dim=0) + idx_patch1 = grid[:, :pres, :pres] # Global indices for patch 1 + idx_patch2 = grid[:, pres:2*pres, pres:2*pres] # Global indices for patch 2 + idx_patch3 = grid[:, -pres:, pres:2*pres] # Global indices for patch 3 + global_index = torch.stack([idx_patch1, idx_patch2, idx_patch3], dim=0) + + # The model internally extracts the corresponding patches from the global + # positional embedding grid and concatenates them to the input x_cond_patches + # before the first UNet block. + out = model(x_cond_patches, noise_labels, class_labels, global_index=global_index) + print(out.shape) # Shape: (3, C_x, pres, pres), same as the patches extracted from the latent state + +Lead-Time Aware Models +~~~~~~~~~~~~~~~~~~~~~~ + +In many diffusion applications, the latent state is time-dependent, and the +diffusion process should account for the time-dependence of the latent state. +For instance, a *forecast* model could provide latent states :math:`\mathbf{x}(T)` (current time), +:math:`\mathbf{x}(T + \Delta t)` (one time step forward), ..., up to :math:`\mathbf{x}(T + K \Delta t)` +(K time steps forward). Such prediction horizons are called *lead-times* (a term +adopted from the weather and climate forecasting community) and we want to apply +diffusion to each of these latent states while accounting for their associated +lead-time information. + +PhysicsNeMo provides a specialized architecture +:class:`~physicsnemo.models.diffusion_unets.SongUNetPosLtEmbd` that implements +lead-time aware models. This is an extension of the +:class:`~physicsnemo.models.diffusion_unets.SongUNetPosEmbd` class, and +additionally supports lead-time information. In its forward pass, the model +uses the ``lead_time_label`` parameter to internally retrieve the associated +lead-time embeddings; it then conditions the diffusion process on those with a +channel-wise concatenation to the latent-state before the first UNet block. + +Here we show an example extending the previous ones with lead-time information. +We assume that we have a batch of three latent states at times :math:`T + 2 \Delta t` +(two time intervals forward), :math:`T + 0 \Delta t` (current time), +and :math:`T + \Delta t` (one time interval forward). The associated lead-time labels are +``[2, 0, 1]``. In addition, the ``SongUNetPosLtEmbd`` model has the ability to +predict probabilities for some channels of the latent state, specified by the +``prob_channels`` parameter. Here we assume that channels one and three are +probability (that is, classification) outputs, while other channels are regression +outputs. + +.. code:: python + + import torch + from physicsnemo.models.diffusion_unets import SongUNetPosLtEmbd + + B, C_x, res = 3, 10, 40 + C_cond = 3 + C_PE = 8 + lead_time_steps = 3 # Maximum supported lead-time is 2 * dt + C_LT = 6 # 6 channels for each lead-time embeddings + + # Create a SongUNet with a lead-time embedding grid of shape + # (lead_time_steps, C_lt_emb, res, res) + model = SongUNetPosLtEmbd( + img_resolution=res, + in_channels=C_x + C_cond + C_PE + C_LT, # in_channels must include the number of channels in lead-time grid + out_channels=C_x, + label_dim=16, + augment_dim=0, + model_channels=64, + channel_mult=[1, 2, 2], + num_blocks=4, + attn_resolutions=[10, 5], + gridtype="learnable", + N_grid_channels=C_PE, + lead_time_channels=C_LT, + lead_time_steps=lead_time_steps, # Maximum supported lead-time horizon + prob_channels=[1, 3], # Channels 1 and 3 fromn the latent state are probability outputs + ) + + x = torch.randn(B, C_x, res, res) # Latent state at times T+2*dt, T+0*dt, and T + 1*dt + cond = torch.randn(B, C_cond, res, res) + x_cond = torch.cat([x, cond], dim=1) + noise_labels = torch.randn(B) + class_labels = torch.randn(B, 16) + lead_time_label = torch.tensor([2, 0, 1]) # Lead-time labels for each sample + + # The model internally extracts the lead-time embeddings corresponding to the + # lead-time labels 2, 0, 1 and concatenates them to the input x_cond before the first + # UNet block. In training mode, the model outputs logits for channels 1 and 3. + out = model(x_cond, noise_labels, class_labels, lead_time_label=lead_time_label) + print(out.shape) # Shape: (B, C_x, res, res), same as the latent state + + # If eval mode the model outputs probabilities for channels 1 and 3 + model.eval() + out = model(x_cond, noise_labels, class_labels, lead_time_label=lead_time_label) + +.. note:: + The ``SongUNetPosLtEmbd`` *is not* an autoregressive model that performs a rollout + to produce future predictions. From the point of view of the ``SongUNetPosLtEmbd``, + the lead-time information is *frozen*. The lead-time dependent latent state :math:`\mathbf{x}` + might however be produced by such an autoregressive/rollout model. + +.. note:: + - The ``SongUNetPosLtEmbd`` model cannot be scaled to very long lead-time + horizons (controlled by the ``lead_time_steps`` parameter). This is because + the lead-time embeddings are represented by a grid of learnable parameters of + shape ``(lead_time_steps, C_LT, res, res)``. For very long lead-time, the + size of this grid of embeddings becomes prohibitively large. + - In a given input batch ``x``, the associated lead-times might be not necessarily + consecutive or in order. The do not even need to originate from the same forecast + trajectory. For example, the lead-time labels might be ``[0, 1, 2]`` instead of ``[2, 0, 1]``, + or even ``[2, 2, 1]``. + +.. _diffusion_application_specific_interfaces: + +Application-Specific Interfaces +------------------------------- + +Application-specific interfaces are not true architectures, but rather wrappers +around the model backbones or specialized architectures that provide a more +user-friendly interface for specific applications. Not all these +classes are true diffusion models, but can also be used with +diffusion models. For instance, the CorrDiff example in +`CorrDiff example <../../examples/weather/corrdiff/README.rst>`_ uses the +:class:`~physicsnemo.models.diffusion_unets.CorrDiffRegressionUNet` class to +implement a regression model. + + +:code:`SongUNet` +---------------- + +.. autoclass:: physicsnemo.models.diffusion_unets.SongUNet + :show-inheritance: + :members: + :exclude-members: forward + +:code:`DhariwalUNet` +--------------------- + +.. autoclass:: physicsnemo.models.diffusion_unets.DhariwalUNet + :show-inheritance: + :members: + :exclude-members: forward + + +:code:`SongUNetPosEmbd` +----------------------- + +.. autoclass:: physicsnemo.models.diffusion_unets.SongUNetPosEmbd + :show-inheritance: + :members: + :exclude-members: forward + + +:code:`SongUNetPosLtEmbd` +------------------------- + +.. autoclass:: physicsnemo.models.diffusion_unets.SongUNetPosLtEmbd + :show-inheritance: + :members: + :exclude-members: forward + +:code:`UNet` +------------ + +.. autoclass:: physicsnemo.models.diffusion_unets.CorrDiffRegressionUNet + :show-inheritance: + :members: + :exclude-members: forward diff --git a/docs/api/models/diffusion_preconditioners.rst b/docs/api/models/diffusion_preconditioners.rst new file mode 100644 index 0000000000..68e0532527 --- /dev/null +++ b/docs/api/models/diffusion_preconditioners.rst @@ -0,0 +1,35 @@ +.. _diffusion_preconditioners: + +Diffusion Preconditioners +========================= + +Preconditioning is an essential technique to improve the performance of +diffusion models. It consists in scaling the latent state and the noise +level that are passed to a network. Some preconditioning also requires to +re-scale the output of the network. PhysicsNeMo provides a set of preconditioning +classes that are wrappers around backbones or specialized architectures. + +.. autoclass:: physicsnemo.diffusion.preconditioners.VPPrecond + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.diffusion.preconditioners.VEPrecond + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.diffusion.preconditioners.iDDPMPrecond + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.diffusion.preconditioners.EDMPrecond + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.diffusion.preconditioners.EDMPrecondSuperResolution + :show-inheritance: + :members: + :exclude-members: forward \ No newline at end of file diff --git a/docs/api/models/fnos.rst b/docs/api/models/fnos.rst new file mode 100644 index 0000000000..c42dbf366f --- /dev/null +++ b/docs/api/models/fnos.rst @@ -0,0 +1,56 @@ +Fourier Neural Operators +======================== + +.. autoclass:: physicsnemo.models.fno.fno.FNO + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.fno.fno.FNO1DEncoder + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.fno.fno.FNO2DEncoder + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.fno.fno.FNO3DEncoder + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.fno.fno.FNO4DEncoder + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.afno.afno.AFNO + :show-inheritance: + :members: + +.. autoclass:: physicsnemo.models.afno.afno.AFNO2DLayer + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.afno.afno.AFNOMlp + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.afno.afno.Block + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.afno.afno.PatchEmbed + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.afno.modafno.ModAFNO + :show-inheritance: + :members: + :exclude-members: forward \ No newline at end of file diff --git a/docs/api/models/fully_connected.rst b/docs/api/models/fully_connected.rst new file mode 100644 index 0000000000..feca4a5a9c --- /dev/null +++ b/docs/api/models/fully_connected.rst @@ -0,0 +1,7 @@ +Fully Connected Network +======================= + +.. autoclass:: physicsnemo.models.mlp.fully_connected.FullyConnected + :show-inheritance: + :members: + :exclude-members: forward \ No newline at end of file diff --git a/docs/api/models/gnns.rst b/docs/api/models/gnns.rst new file mode 100644 index 0000000000..16cafaae6e --- /dev/null +++ b/docs/api/models/gnns.rst @@ -0,0 +1,22 @@ +Graph Neural Networks +===================== + +.. autoclass:: physicsnemo.models.meshgraphnet.meshgraphnet.MeshGraphNet + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.meshgraphnet.meshgraphnet.MeshGraphNetProcessor + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.mesh_reduced.mesh_reduced.Mesh_Reduced + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.meshgraphnet.bsms_mgn.BiStrideMeshGraphNet + :show-inheritance: + :members: + :exclude-members: forward \ No newline at end of file diff --git a/docs/api/models/modules.rst b/docs/api/models/modules.rst new file mode 100644 index 0000000000..94ce66d7ba --- /dev/null +++ b/docs/api/models/modules.rst @@ -0,0 +1,555 @@ +PhysicsNeMo Modules +=================== + +.. automodule:: physicsnemo.models +.. currentmodule:: physicsnemo.models + +Basics +------ + +PhysicsNeMo contains its own Model class for constructing neural networks. This model class +is built on top of PyTorch's ``nn.Module`` and can be used interchangeably within the +PyTorch ecosystem. Using PhysicsNeMo models allows you to leverage various features of +PhysicsNeMo aimed at improving performance and ease of use. These features include, but are +not limited to, model zoo, automatic mixed-precision, CUDA Graphs, and easy checkpointing. +We discuss each of these features in the following sections. + +Model Zoo +--------- + +PhysicsNeMo contains several optimized, customizable and easy-to-use models. +These include some very general models like Fourier Neural Operators (FNOs), +ResNet, and Graph Neural Networks (GNNs) as well as domain-specific models like +Deep Learning Weather Prediction (DLWP) and Spherical Fourier Neural Operators (SFNO). + +For a list of currently available models, please refer the `models on GitHub `_. + +Below are some simple examples of how to use these models. + +.. code:: python + + >>> import torch + >>> from physicsnemo.models.mlp.fully_connected import FullyConnected + >>> model = FullyConnected(in_features=32, out_features=64) + >>> input = torch.randn(128, 32) + >>> output = model(input) + >>> output.shape + torch.Size([128, 64]) + +.. code:: python + + >>> import torch + >>> from physicsnemo.models.fno.fno import FNO + >>> model = FNO( + in_channels=4, + out_channels=3, + decoder_layers=2, + decoder_layer_size=32, + dimension=2, + latent_channels=32, + num_fno_layers=2, + padding=0, + ) + >>> input = torch.randn(32, 4, 32, 32) #(N, C, H, W) + >>> output = model(input) + >>> output.size() + torch.Size([32, 3, 32, 32]) + +How to write your own PhysicsNeMo model +--------------------------------------- + +There are a few different ways to construct a PhysicsNeMo model. If you are a seasoned +PyTorch user, the easiest way would be to write your model using the optimized layers and +utilities from PhysicsNeMo or Pytorch. Let's take a look at a simple example of a UNet model +first showing a simple PyTorch implementation and then a PhysicsNeMo implementation that +supports CUDA Graphs and Automatic Mixed-Precision. + +.. code:: python + + import torch.nn as nn + + class UNet(nn.Module): + def __init__(self, in_channels=1, out_channels=1): + super(UNet, self).__init__() + + self.enc1 = self.conv_block(in_channels, 64) + self.enc2 = self.conv_block(64, 128) + + self.dec1 = self.upconv_block(128, 64) + self.final = nn.Conv2d(64, out_channels, kernel_size=1) + + def conv_block(self, in_channels, out_channels): + return nn.Sequential( + nn.Conv2d(in_channels, out_channels, 3, padding=1), + nn.ReLU(inplace=True), + nn.MaxPool2d(2) + ) + + def upconv_block(self, in_channels, out_channels): + return nn.Sequential( + nn.ConvTranspose2d(in_channels, out_channels, 2, stride=2), + nn.Conv2d(out_channels, out_channels, 3, padding=1), + nn.ReLU(inplace=True) + ) + + def forward(self, x): + x1 = self.enc1(x) + x2 = self.enc2(x1) + x = self.dec1(x2) + return self.final(x) + +Now we show this model rewritten in PhysicsNeMo. First, let us subclass the model from +``physicsnemo.Module`` instead of ``torch.nn.Module``. The +``physicsnemo.Module`` class acts like a direct replacement for the +``torch.nn.Module`` and provides additional functionality for saving and loading +checkpoints, etc. Refer to the API docs of ``physicsnemo.Module`` for further +details. Additionally, we will add metadata to the model to capture the optimizations +that this model supports. In this case we will enable CUDA Graphs and Automatic Mixed-Precision. + +.. code:: python + + from dataclasses import dataclass + import physicsnemo + import torch.nn as nn + + @dataclass + class UNetMetaData(physicsnemo.ModelMetaData): + name: str = "UNet" + # Optimization + jit: bool = True + cuda_graphs: bool = True + amp_cpu: bool = True + amp_gpu: bool = True + + class UNet(physicsnemo.Module): + def __init__(self, in_channels=1, out_channels=1): + super(UNet, self).__init__(meta=UNetMetaData()) + + self.enc1 = self.conv_block(in_channels, 64) + self.enc2 = self.conv_block(64, 128) + + self.dec1 = self.upconv_block(128, 64) + self.final = nn.Conv2d(64, out_channels, kernel_size=1) + + def conv_block(self, in_channels, out_channels): + return nn.Sequential( + nn.Conv2d(in_channels, out_channels, 3, padding=1), + nn.ReLU(inplace=True), + nn.MaxPool2d(2) + ) + + def upconv_block(self, in_channels, out_channels): + return nn.Sequential( + nn.ConvTranspose2d(in_channels, out_channels, 2, stride=2), + nn.Conv2d(out_channels, out_channels, 3, padding=1), + nn.ReLU(inplace=True) + ) + + def forward(self, x): + x1 = self.enc1(x) + x2 = self.enc2(x1) + x = self.dec1(x2) + return self.final(x) + +Now that we have our PhysicsNeMo model, we can make use of these optimizations using the +``physicsnemo.utils.StaticCaptureTraining`` decorator. This decorator will capture the +training step function and optimize it for the specified optimizations. + +.. code:: python + + import torch + from physicsnemo.utils import StaticCaptureTraining + + model = UNet().to("cuda") + input = torch.randn(8, 1, 128, 128).to("cuda") + output = torch.zeros(8, 1, 64, 64).to("cuda") + + optim = torch.optim.Adam(model.parameters(), lr=0.001) + + # Create training step function with optimization wrapper + # StaticCaptureTraining calls `backward` on the loss and + # `optimizer.step()` so you don't have to do that + # explicitly. + @StaticCaptureTraining( + model=model, + optim=optim, + cuda_graph_warmup=11, + ) + def training_step(invar, outvar): + predvar = model(invar) + loss = torch.sum(torch.pow(predvar - outvar, 2)) + return loss + + # Sample training loop + for i in range(20): + # In place copy of input and output to support cuda graphs + input.copy_(torch.randn(8, 1, 128, 128).to("cuda")) + output.copy_(torch.zeros(8, 1, 64, 64).to("cuda")) + + # Run training step + loss = training_step(input, output) + +For the simple model above, you can observe ~1.1x speed-up due to CUDA Graphs and AMP. +The speed-up observed changes from model to model and is typically greater for more +complex models. + +.. note:: + The ``ModelMetaData`` and ``physicsnemo.Module`` do not make the model + support CUDA Graphs, AMP, etc. optimizations automatically. The user is responsible + to write the model code that enables each of these optimizations. + Models in the PhysicsNeMo Model Zoo are written to support many of these optimizations + and checked against PhysicsNeMo's CI to ensure that they work correctly. + +.. note:: + The ``StaticCaptureTraining`` decorator is still under development and may be + refactored in the future. + + +.. _physicsnemo-models-from-torch: + +Converting PyTorch Models to PhysicsNeMo Models +----------------------------------------------- + +In the above example we show constructing a PhysicsNeMo model from scratch. However, you +can also convert existing PyTorch models to PhysicsNeMo models in order to leverage +PhysicsNeMo features. To do this, you can use the ``Module.from_torch`` method as shown +below. + +.. code:: python + + from dataclasses import dataclass + import physicsnemo + import torch.nn as nn + + class TorchModel(nn.Module): + def __init__(self): + super(TorchModel, self).__init__() + self.conv1 = nn.Conv2d(1, 20, 5) + self.conv2 = nn.Conv2d(20, 20, 5) + + def forward(self, x): + x = self.conv1(x) + return self.conv2(x) + + @dataclass + class ConvMetaData(ModelMetaData): + name: str = "UNet" + # Optimization + jit: bool = True + cuda_graphs: bool = True + amp_cpu: bool = True + amp_gpu: bool = True + + PhysicsNeMoModel = physicsnemo.Module.from_torch(TorchModel, meta=ConvMetaData()) + + + + +.. _saving-and-loading-physicsnemo-models: + +Saving and Loading PhysicsNeMo Models +------------------------------------- + +As mentioned above, PhysicsNeMo models are interoperable with PyTorch models. This means that +you can save and load PhysicsNeMo models using the standard PyTorch APIs however, we provide +a few additional utilities to make this process easier. A key challenge in saving and +loading models is keeping track of the model metadata such as layer sizes, etc. PhysicsNeMo +models can be saved with this metadata to a custom ``.mdlus`` file. These files allow +for easy loading and instantiation of the model. We show two examples of this below. +The first example shows saving and loading a model from an already instantiated model. + +.. code:: python + + >>> from physicsnemo.models.mlp.fully_connected import FullyConnected + >>> model = FullyConnected(in_features=32, out_features=64) + >>> model.save("model.mdlus") # Save model to .mdlus file + >>> model.load("model.mdlus") # Load model weights from .mdlus file from already instantiated model + >>> model + FullyConnected( + (layers): ModuleList( + (0): FCLayer( + (activation_fn): SiLU() + (linear): Linear(in_features=32, out_features=512, bias=True) + ) + (1-5): 5 x FCLayer( + (activation_fn): SiLU() + (linear): Linear(in_features=512, out_features=512, bias=True) + ) + ) + (final_layer): FCLayer( + (activation_fn): Identity() + (linear): Linear(in_features=512, out_features=64, bias=True) + ) + ) + +The second example shows loading a model from a ``.mdlus`` file without having to +instantiate the model first. We note that in this case we don't know the class or +parameters to pass to the constructor of the model. However, we can still load the +model from the ``.mdlus`` file. + +.. code:: python + + >>> from physicsnemo import Module + >>> fc_model = Module.from_checkpoint("model.mdlus") # Instantiate model from .mdlus file. + >>> fc_model + FullyConnected( + (layers): ModuleList( + (0): FCLayer( + (activation_fn): SiLU() + (linear): Linear(in_features=32, out_features=512, bias=True) + ) + (1-5): 5 x FCLayer( + (activation_fn): SiLU() + (linear): Linear(in_features=512, out_features=512, bias=True) + ) + ) + (final_layer): FCLayer( + (activation_fn): Identity() + (linear): Linear(in_features=512, out_features=64, bias=True) + ) + ) + + + +.. note:: + In order to make use of this functionality, the model must have ``.json`` + serializable inputs to the ``__init__`` function. The only exception to this + rule is when the argument passed to the ``__init__`` function is itself a + ``physicsnemo.Module`` instance. In this case, it is possible to construct, + save and load nested Modules, with multiple levels of nesting and/or multiple + ``physicsnemo.Module`` instances at each level. See the section + :ref:`constructing-nested-modules` for more details. It is highly recommended + that all PhysicsNeMo models be developed with this requirement in mind. + +.. note:: + Using ``Module.from_checkpoint`` will not work if the model has any buffers or + parameters that are registered outside of the model's ``__init__`` function due to + the above requirement. In that case, one should use ``Module.load``, or ensure + that all model parameters and buffers are registered inside ``__init__``. + + +.. _constructing-nested-modules: + +Constructing Nested Modules +---------------------------- + +PhysicsNeMo supports constructing nested modules where one ``physicsnemo.Module`` +can accept another ``physicsnemo.Module`` as an argument to its ``__init__`` +function. This allows you to build complex, modular architectures while still +benefiting from PhysicsNeMo's checkpointing and model management features. + +**Simple Nesting with PhysicsNeMo Modules** + +The simplest case is nesting ``physicsnemo.Module`` instances directly: + +.. code:: python + + import physicsnemo + from physicsnemo.models.meta import ModelMetaData + + class EncoderModule(physicsnemo.Module): + def __init__(self, input_size, hidden_size): + super().__init__(meta=ModelMetaData()) + self.encoder = torch.nn.Linear(input_size, hidden_size) + self.input_size = input_size + self.hidden_size = hidden_size + + def forward(self, x): + return self.encoder(x) + + class DecoderModule(physicsnemo.Module): + def __init__(self, hidden_size, output_size): + super().__init__(meta=ModelMetaData()) + self.decoder = torch.nn.Linear(hidden_size, output_size) + self.hidden_size = hidden_size + self.output_size = output_size + + def forward(self, x): + return self.decoder(x) + + class AutoEncoder(physicsnemo.Module): + def __init__(self, encoder, decoder): + super().__init__(meta=ModelMetaData()) + self.encoder = encoder + self.decoder = decoder + + def forward(self, x): + encoded = self.encoder(x) + return self.decoder(encoded) + + # Create nested model + encoder = EncoderModule(input_size=64, hidden_size=32) + decoder = DecoderModule(hidden_size=32, output_size=64) + model = AutoEncoder(encoder=encoder, decoder=decoder) + + # Save and load with full structure preserved + model.save("autoencoder.mdlus") + loaded_model = physicsnemo.Module.from_checkpoint("autoencoder.mdlus") + +**Nesting Converted PyTorch Modules** + +You can also nest PyTorch ``nn.Module`` instances, but they must first be +converted to ``physicsnemo.Module`` using ``Module.from_torch``. All nested +PyTorch modules must be converted: + +.. code:: python + + import torch.nn as nn + import physicsnemo + from physicsnemo.models.meta import ModelMetaData + + # Define PyTorch modules + class TorchEncoder(nn.Module): + def __init__(self, input_size, hidden_size): + super().__init__() + self.encoder = nn.Linear(input_size, hidden_size) + self.input_size = input_size + self.hidden_size = hidden_size + + def forward(self, x): + return self.encoder(x) + + class TorchDecoder(nn.Module): + def __init__(self, hidden_size, output_size): + super().__init__() + self.decoder = nn.Linear(hidden_size, output_size) + self.hidden_size = hidden_size + self.output_size = output_size + + def forward(self, x): + return self.decoder(x) + + # Convert to PhysicsNeMo modules + PNMEncoder = physicsnemo.Module.from_torch( + TorchEncoder, meta=ModelMetaData() + ) + PNMDecoder = physicsnemo.Module.from_torch( + TorchDecoder, meta=ModelMetaData() + ) + + # Define top-level model + class AutoEncoder(physicsnemo.Module): + def __init__(self, encoder, decoder): + super().__init__(meta=ModelMetaData()) + self.encoder = encoder + self.decoder = decoder + + def forward(self, x): + encoded = self.encoder(x) + return self.decoder(encoded) + + # Create nested model with converted modules + encoder = PNMEncoder(input_size=64, hidden_size=32) + decoder = PNMDecoder(hidden_size=32, output_size=64) + model = AutoEncoder(encoder=encoder, decoder=decoder) + + # Save and load + model.save("autoencoder.mdlus") + loaded_model = physicsnemo.Module.from_checkpoint("autoencoder.mdlus") + +**What Does NOT Work** + +You cannot directly pass a ``torch.nn.Module`` instance to a +``physicsnemo.Module``'s ``__init__`` without converting it first: + +.. code:: python + + # This will NOT work and raise an error during save/load: + class AutoEncoder(physicsnemo.Module): + def __init__(self, encoder): + super().__init__(meta=ModelMetaData()) + self.encoder = encoder # encoder is a torch.nn.Module + + torch_encoder = TorchEncoder(input_size=64, hidden_size=32) + model = AutoEncoder(encoder=torch_encoder) # This creates the model + + # But this will fail: + model.save("autoencoder.mdlus") + # Error: Cannot serialize torch.nn.Module arguments. + # You must use Module.from_torch() to convert it first. + + +PhysicsNeMo Model Registry and Entry Points +------------------------------------------- + +PhysicsNeMo contains a model registry that allows for easy access and ingestion of +models. Below is a simple example of how to use the model registry to obtain a model +class. + +.. code:: python + + >>> from physicsnemo.registry import ModelRegistry + >>> model_registry = ModelRegistry() + >>> model_registry.list_models() + ['AFNO', 'DLWP', 'FNO', 'FullyConnected', 'GraphCastNet', 'MeshGraphNet', 'One2ManyRNN', 'Pix2Pix', 'SFNO', 'SRResNet'] + >>> FullyConnected = model_registry.factory("FullyConnected") + >>> model = FullyConnected(in_features=32, out_features=64) + +The model registry also allows exposing models via entry points. This allows for +integration of models into the PhysicsNeMo ecosystem. For example, suppose you have a +package ``MyPackage`` that contains a model ``MyModel``. You can expose this model +to the PhysicsNeMo registry by adding an entry point to your ``toml`` file. For +example, suppose your package structure is as follows: + +.. code:: python + + # setup.py + + from setuptools import setup, find_packages + + setup() + +.. code:: python + + # pyproject.toml + + [build-system] + requires = ["setuptools", "wheel"] + build-backend = "setuptools.build_meta" + + [project] + name = "MyPackage" + description = "My Neural Network Zoo." + version = "0.1.0" + + [project.entry-points."physicsnemo.models"] + MyPhysicsNeMoModel = "mypackage.models.MyPhysicsNeMoModel:MyPhysicsNeMoModel" + +.. code:: python + + # mypackage/models.py + + import torch.nn as nn + from physicsnemo.models import Module + + class MyModel(nn.Module): + def __init__(self): + super(MyModel, self).__init__() + self.conv1 = nn.Conv2d(1, 20, 5) + self.conv2 = nn.Conv2d(20, 20, 5) + + def forward(self, x): + x = self.conv1(x) + return self.conv2(x) + + MyPhysicsNeMoModel = Module.from_pytorch(MyModel) + + +Once this package is installed, you can access the model via the PhysicsNeMo model +registry. + + +.. code:: python + + >>> from physicsnemo.registry import ModelRegistry + >>> model_registry = ModelRegistry() + >>> model_registry.list_models() + ['MyPhysicsNeMoModel', 'AFNO', 'DLWP', 'FNO', 'FullyConnected', 'GraphCastNet', 'MeshGraphNet', 'One2ManyRNN', 'Pix2Pix', 'SFNO', 'SRResNet'] + >>> MyPhysicsNeMoModel = model_registry.factory("MyPhysicsNeMoModel") + + +For more information on entry points and potential use cases, see +`this `_ blog post. + +.. autosummary:: + :toctree: generated \ No newline at end of file diff --git a/docs/api/models/operators.rst b/docs/api/models/operators.rst new file mode 100644 index 0000000000..c7bafba32d --- /dev/null +++ b/docs/api/models/operators.rst @@ -0,0 +1,7 @@ +Operator Models +=============== + +.. autoclass:: physicsnemo.models.domino.model.DoMINO + :show-inheritance: + :members: + :exclude-members: forward diff --git a/docs/api/models/recurrent.rst b/docs/api/models/recurrent.rst new file mode 100644 index 0000000000..574fa32b1e --- /dev/null +++ b/docs/api/models/recurrent.rst @@ -0,0 +1,12 @@ +Recurrent Neural Networks +========================= + +.. autoclass:: physicsnemo.models.rnn.rnn_one2many.One2ManyRNN + :show-inheritance: + :members: + :exclude-members: forward + +.. autoclass:: physicsnemo.models.rnn.rnn_seq2seq.Seq2SeqRNN + :show-inheritance: + :members: + :exclude-members: forward \ No newline at end of file diff --git a/docs/api/models/weather.rst b/docs/api/models/weather.rst new file mode 100644 index 0000000000..129e73f546 --- /dev/null +++ b/docs/api/models/weather.rst @@ -0,0 +1,28 @@ +Weather / Climate Models +======================== + +.. autoclass:: physicsnemo.models.dlwp.dlwp.DLWP + :show-inheritance: + :members: + +.. autoclass:: physicsnemo.models.dlwp_healpix.HEALPixRecUNet.HEALPixRecUNet + :show-inheritance: + :members: + +.. autoclass:: physicsnemo.models.graphcast.graph_cast_net.GraphCastNet + :show-inheritance: + :members: + +.. autofunction:: physicsnemo.models.graphcast.graph_cast_net.get_lat_lon_partition_separators + +.. autoclass:: physicsnemo.models.fengwu.fengwu.Fengwu + :show-inheritance: + :members: + +.. autoclass:: physicsnemo.models.pangu.pangu.Pangu + :show-inheritance: + :members: + +.. autoclass:: physicsnemo.models.swinvrnn.swinvrnn.SwinRNN + :show-inheritance: + :members: \ No newline at end of file diff --git a/docs/api/modulus.datapipes.rst b/docs/api/modulus.datapipes.rst deleted file mode 100644 index 93e697c8ce..0000000000 --- a/docs/api/modulus.datapipes.rst +++ /dev/null @@ -1,38 +0,0 @@ -Modulus Datapipes -================= - -.. autosummary:: - :toctree: generated - -Benchmark datapipes -------------------- - -.. automodule:: modulus.datapipes.benchmarks.darcy - :members: - :show-inheritance: - -.. automodule:: modulus.datapipes.benchmarks.kelvin_helmholtz - :members: - :show-inheritance: - -Weather and climate datapipes ------------------------------ - -.. automodule:: modulus.datapipes.climate.era5_hdf5 - :members: - :show-inheritance: - -Graph datapipes ---------------- - -.. automodule:: modulus.datapipes.gnn.vortex_shedding_dataset - :members: - :show-inheritance: - -.. automodule:: modulus.datapipes.gnn.ahmed_body_dataset - :members: - :show-inheritance: - -.. automodule:: modulus.datapipes.gnn.utils - :members: - :show-inheritance: diff --git a/docs/api/modulus.deploy.rst b/docs/api/modulus.deploy.rst deleted file mode 100644 index 5c61bf3258..0000000000 --- a/docs/api/modulus.deploy.rst +++ /dev/null @@ -1,13 +0,0 @@ -Modulus Deploy -============== - -.. autosummary:: - :toctree: generated - -ONNX ----- -.. automodule:: modulus.deploy.onnx.utils - :members: - :show-inheritance: - - diff --git a/docs/api/modulus.distributed.rst b/docs/api/modulus.distributed.rst deleted file mode 100644 index 4d361ce2e4..0000000000 --- a/docs/api/modulus.distributed.rst +++ /dev/null @@ -1,169 +0,0 @@ -Modulus Distributed -=================== - -.. automodule:: modulus.distributed -.. currentmodule:: modulus.distributed - -Distributed utilites in Modulus are designed to simplify implementation of parallel training and -make inference scripts easier by providing a unified way to configure and query parameters associated -with the distributed environment. The utilites in ``modulus.distributed`` build on top of the -utilites from ``torch.distributed`` and abstract out some of the complexities of setting up a -distributed execution environment. - -The example below shows how to setup a simple distributed data parallel training recipe using the -distributed utilites in Modulus. -`DistributedDataParallel `_ -in PyTorch provides the framework for data parallel training by reducing parameter gradients -across multiple worker processes after the backwards pass. The code below shows how to specify -the ``device_ids``, ``output_device``, ``broadcast_buffers`` and ``find_unused_parameters`` -arguments of the ``DistributedDataParallel`` utility using the ``DistributedManager``. - -.. code:: python - - import torch - from torch.nn.parallel import DistributedDataParallel - from modulus.distributed import DistributedManager - from modulus.models.mlp.fully_connected import FullyConnected - - def main(): - # Initialize the DistributedManager. This will automatically - # detect the number of processes the job was launched with and - # set those configuration parameters appropriately. Currently - # torchrun (or any other pytorch compatible launcher), mpirun (OpenMPI) - # and SLURM based launchers are supported. - DistributedManager.initialize() - - # Since this is a singleton class, you can just get an instance - # of it anytime after initialization and not need to reinitialize - # each time. - dist = DistributedManager() - - # Set up model on the appropriate device. DistributedManager - # figures out what device should be used on this process - arch = FullyConnected(in_features=32, out_features=64).to(dist.device) - - # Set up DistributedDataParallel if using more than a single process. - # The `distributed` property of DistributedManager can be used to - # check this. - if dist.distributed: - ddps = torch.cuda.Stream() - with torch.cuda.stream(ddps): - arch = DistributedDataParallel( - arch, - device_ids=[dist.local_rank], # Set the device_id to be - # the local rank of this process on - # this node - output_device=dist.device, - broadcast_buffers=dist.broadcast_buffers, - find_unused_parameters=dist.find_unused_parameters, - ) - torch.cuda.current_stream().wait_stream(ddps) - - # Set up the optimizer - optimizer = torch.optim.Adam( - arch.parameters(), - lr=0.001, - ) - - def training_step(input, target): - pred = arch(invar) - loss = torch.sum(torch.pow(pred - target, 2)) - loss.backward() - optimizer.step() - return loss - - # Sample training loop - for i in range(20): - # Random inputs and targets for simplicity - input = torch.randn(128, 32, device=dist.device) - target = torch.randn(128, 64, device=dist.device) - - # Training step - loss = training_step(input, target) - - if __name__ == "__main__": - main() - -This training script can be run on a single GPU -using ``python train.py`` or on multiple GPUs using - -.. code-block:: bash - - torchrun --standalone --nnodes=1 --nproc_per_node= train.py -or - -.. code-block:: bash - - mpirun -np python train.py -if using OpenMPI. The script can also -be run on a SLURM cluster using - -.. code-block:: bash - - srun -n python train.py - -How does this work? -""""""""""""""""""" - -An important aspect of the ``DistributedManager`` is that it is follows the -`Borg pattern `_. -This means that ``DistributedManager`` essentially functions like a singleton -class and once configured, all utilities in Modulus can access the same configuration -and adapt to the specified distributed structure. - -For example, see the constructor of the ``DistributedAFNO`` class: - -.. literalinclude:: ../../modulus/models/afno/distributed/afno.py - :pyobject: DistributedAFNO.__init__ - -This model parallel implementation can just instantiate ``DistributedManager`` and query -if the process group named ``"model_parallel"`` exists and if so, what is it's size. Similarly, -other utilities can query what device to run on, the total size of the distributed run, etc. -without having to explicitly pass those params down the call stack. - -.. note:: - - This singleton/borg pattern is very useful for the ``DistributedManager`` since it takes charge - of bootstrapping the distributed run and unifies how all utilities become aware of the distributed - configuration. However, the singleton/borg pattern is not just a way to avoid passing parameters - to utilities. Use of this pattern should be limited and have good justification to avoid losing - tracability and keep the code readable. - - -.. autosummary:: - :toctree: generated - -modulus.distributed.manager ----------------------------- - -.. automodule:: modulus.distributed.manager - :members: - :show-inheritance: - -modulus.distributed.utils ----------------------------- - -.. automodule:: modulus.distributed.utils - :members: - :show-inheritance: - -modulus.distributed.autograd ----------------------------- - -.. automodule:: modulus.distributed.autograd - :members: - :show-inheritance: - -modulus.distributed.fft ----------------------------- - -.. automodule:: modulus.distributed.fft - :members: - :show-inheritance: - -modulus.distributed.mappings ----------------------------- - -.. automodule:: modulus.distributed.mappings - :members: - :show-inheritance: diff --git a/docs/api/modulus.launch.logging.rst b/docs/api/modulus.launch.logging.rst deleted file mode 100644 index 5d13e6db00..0000000000 --- a/docs/api/modulus.launch.logging.rst +++ /dev/null @@ -1,44 +0,0 @@ -Modulus Launch Logging -====================== - -.. automodule:: modulus.launch.logging -.. currentmodule:: modulus.launch.logging - -.. autosummary:: - :toctree: generated - -Launch Logger -------------- - -.. automodule:: modulus.launch.logging.launch - :members: - :show-inheritance: - -Console Logger --------------- - -.. automodule:: modulus.launch.logging.console - :members: - :show-inheritance: - -MLflow Logger -------------- - -.. automodule:: modulus.launch.logging.mlflow - :members: - :show-inheritance: - -Weights and Biases Logger -------------------------- - -.. automodule:: modulus.launch.logging.wandb - :members: - :show-inheritance: - -Logging utils -------------- - -.. automodule:: modulus.launch.logging.utils - :members: - :show-inheritance: - diff --git a/docs/api/modulus.launch.utils.rst b/docs/api/modulus.launch.utils.rst deleted file mode 100644 index 992384b31b..0000000000 --- a/docs/api/modulus.launch.utils.rst +++ /dev/null @@ -1,16 +0,0 @@ -Modulus Launch Utils -==================== - -.. automodule:: modulus.launch.utils -.. currentmodule:: modulus.launch.utils - -.. autosummary:: - :toctree: generated - -Checkpointing -------------- - -.. automodule:: modulus.launch.utils.checkpoint - :members: - :show-inheritance: - diff --git a/docs/api/modulus.metrics.rst b/docs/api/modulus.metrics.rst deleted file mode 100644 index 8c3904ccac..0000000000 --- a/docs/api/modulus.metrics.rst +++ /dev/null @@ -1,277 +0,0 @@ -Modulus Metrics -=============== - -.. automodule:: modulus.metrics -.. currentmodule:: modulus.metrics - -Basics -------- - -Modulus provides several general and domain-specific metric calculations you can -leverage in your custom training and inference workflows. These metrics are optimized to -operate on PyTorch tensors. - -General Metrics and Statistical Methods -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Below is a summary of general purpose statistical methods and metrics that are available: - -.. list-table:: - :widths: 20 80 - :header-rows: 1 - - * - Metric - - Description - * - `modulus.metrics.general.mse.mse <#modulus.metrics.general.mse.mse>`_ - - Mean Squared error between two tensors - * - `modulus.metrics.general.mse.rmse <#modulus.metrics.general.mse.rmse>`_ - - Root Mean Squared error between two tensors - * - `modulus.metrics.general.histogram.histogram <#modulus.metrics.general.histogram.histogram>`_ - - Histogram of a set of tensors over the leading dimension - * - `modulus.metrics.general.histogram.cdf <#modulus.metrics.general.histogram.cdf>`_ - - Cumulative density function of a set of tensors over the leading dimension - * - `modulus.metrics.general.histogram.normal_cdf <#modulus.metrics.general.histogram.normal_cdf>`_ - - Cumulative density function of a normal variable with given mean and standard deviation - * - `modulus.metrics.general.histogram.normal_pdf <#modulus.metrics.general.histogram.normal_pdf>`_ - - Probability density function of a normal variable with given mean and standard deviation - * - `modulus.metrics.general.calibration.find_rank <#modulus.metrics.general.calibration.find_rank>`_ - - Find the rank of the observation with respect to the given counts and bins - * - `modulus.metrics.general.calibration.rank_probability_score <#modulus.metrics.general.calibration.rank_probability_score>`_ - - Rank Probability Score for the passed ranks - * - `modulus.metrics.general.entropy.entropy_from_counts <#modulus.metrics.general.entropy.entropy_from_counts>`_ - - Computes the statistical entropy of a random variable using a histogram. - * - `modulus.metrics.general.entropy.relative_entropy_from_counts <#modulus.metrics.general.entropy.relative_entropy_from_counts>`_ - - Computes the relative statistical entropy, or KL Divergence of two random variables using their histograms. - * - `modulus.metrics.general.crps.crps <#modulus.metrics.general.crps.crps>`_ - - Local Continuous Ranked Probability Score (CRPS) by computing a histogram and CDF of the predictions - * - `modulus.metrics.general.wasserstein.wasserstein <#modulus.metrics.general.wasserstein.wasserstein>`_ - - 1-Wasserstein distance between two discrete CDF functions - * - `modulus.metrics.general.reduction.WeightedMean <#modulus.metrics.general.reduction.WeightedMean>`_ - - Weighted Mean - * - `modulus.metrics.general.reduction.WeightedStatistic <#modulus.metrics.general.reduction.WeightedStatistic>`_ - - Weighted Statistic - * - `modulus.metrics.general.reduction.WeightedVariance <#modulus.metrics.general.reduction.WeightedVariance>`_ - - Weighted Variance - -Below shows some examples of how to use these metrics in your own workflows. - - -To compute RMSE metric: - -.. code:: python - - >>> import torch - >>> from modulus.metrics.general.mse import rmse - >>> pred_tensor = torch.randn(16, 32) - >>> targ_tensor = torch.randn(16, 32) - >>> rmse(pred_tensor, targ_tensor) - tensor(1.4781) - - -To compute the histogram of samples: - -.. code:: python - - >>> import torch - >>> from modulus.metrics.general import histogram - >>> x = torch.randn(1_000) - >>> bins, counts = histogram.histogram(x, bins = 10) - >>> bins - tensor([-3.7709, -3.0633, -2.3556, -1.6479, -0.9403, -0.2326, 0.4751, 1.1827, - 1.8904, 2.5980, 3.3057]) - >>> counts - tensor([ 3, 9, 43, 150, 227, 254, 206, 81, 24, 3]) - - -To use compute the continuous density function (CDF): - -.. code:: python - - >>> bins, cdf = histogram.cdf(x, bins = 10) - >>> bins - tensor([-3.7709, -3.0633, -2.3556, -1.6479, -0.9403, -0.2326, 0.4751, 1.1827, - 1.8904, 2.5980, 3.3057]) - >>> cdf - tensor([0.0030, 0.0120, 0.0550, 0.2050, 0.4320, 0.6860, 0.8920, 0.9730, 0.9970, - 1.0000]) - -To use the histogram for statistical entropy calculations: - -.. code:: python - - >> from modulus.metrics.general import entropy - >>> entropy.entropy_from_counts(counts, bins) - tensor(0.4146) - -Many of the functions operate over batches. For example, if one has a collection of two dimensional -data, then we can compute the histogram over the collection: - -.. code:: python - - >>> import torch - >>> from modulus.metrics.general import histogram, entropy - >>> x = torch.randn((1_000, 3, 3)) - >>> bins, counts = histogram.histogram(x, bins = 10) - >>> bins.shape, counts.shape - (torch.Size([11, 3, 3]), torch.Size([10, 3, 3])) - >>> entropy.entropy_from_counts(counts, bins) - tensor([[0.5162, 0.4821, 0.3976], - [0.5099, 0.5309, 0.4519], - [0.4580, 0.4290, 0.5121]]) - -There are additional metrics to compute differences between distributions: Ranks, Continuous Rank -Probability Skill, and Wasserstein metric. - -CRPS: - -.. code:: python - - >>> from modulus.metrics.general import crps - >>> x = torch.randn((1_000,1)) - >>> y = torch.randn((1,)) - >>> crps.crps(x, y) - tensor([0.8023]) - -Ranks: - -.. code:: python - - >>> from modulus.metrics.general import histogram, calibration - >>> x = torch.randn((1_000,1)) - >>> y = torch.randn((1,)) - >>> bins, counts = histogram.histogram(x, bins = 10) - >>> ranks = calibration.find_rank(bins, counts, y) - tensor([0.1920]) - -Wasserstein Metric: - -.. code:: python - - >>> from modulus.metrics.general import wasserstein, histogram - >>> x = torch.randn((1_000,1)) - >>> y = torch.randn((1_000,1)) - >>> bins, cdf_x = histogram.cdf(x) - >>> bins, cdf_y = histogram.cdf(y, bins = bins) - >>> wasserstein(bins, cdf_x, cdf_y) - >>> wasserstein.wasserstein(bins, cdf_x, cdf_y) - tensor([0.0459]) - - -Weighted Reductions -^^^^^^^^^^^^^^^^^^^ -Modulus currently offers classes for weighted mean and variance reductions. - -.. code:: python - - >>> from modulus.metrics.general import reduction - >>> x = torch.randn((1_000,)) - >>> weights = torch.cos(torch.linspace(-torch.pi/4, torch.pi/4, 1_000)) - >>> wm = reduction.WeightedMean(weights) - >>> wm(x, dim = 0) - tensor(0.0365) - >>> wv = reduction.WeightedVariance(weights) - >>> wv(x, dim = 0) - tensor(1.0148) - - -Online Statistical Methods -^^^^^^^^^^^^^^^^^^^^^^^^^^ -Modulus current offers routines for computing online, or out-of-memory, means, -variances, and histograms. - -.. code:: python - - >>> import torch - >>> from modulus.metrics.general import ensemble_metrics as em - >>> x = torch.randn((1_000, 2)) # Interpret as 1_000 members of size (2,). - >>> torch.mean(x, dim = 0) # Compute mean of entire data. - tensor([-0.0545, 0.0267]) - >>> x0, x1 = x[:500], x[500:] # Split data into two. - >>> M = em.Mean(input_shape = (2,)) # Must pass shape of data - >>> M(x0) # Compute mean of initial batch. - tensor([-0.0722, 0.0414]) - >>> M.update(x1) # Update with second batch. - tensor([-0.0545, 0.0267]) - - -Climate Related Metrics -^^^^^^^^^^^^^^^^^^^^^^^ - -To compute the Anomaly Correlation Coefficient, a metric widely used in weather and -climate sciences: - -.. code:: python - - >>> import torch - >>> import numpy as np - >>> from modulus.metrics.climate.acc import acc - >>> channels = 1 - >>> img_shape = (32, 64) - >>> time_means = np.pi / 2 * np.ones((channels, img_shape[0], img_shape[1]), dtype=np.float32) - >>> x = np.linspace(-180, 180, img_shape[1], dtype=np.float32) - >>> y = np.linspace(-90, 90, img_shape[0], dtype=np.float32) - >>> xv, yv = np.meshgrid(x, y) - >>> pred_tensor_np = np.cos(2 * np.pi * yv / (180)) - >>> targ_tensor_np = np.cos(np.pi * yv / (180)) - >>> pred_tensor = torch.from_numpy(pred_tensor_np).expand(channels, -1, -1) - >>> targ_tensor = torch.from_numpy(targ_tensor_np).expand(channels, -1, -1) - >>> means_tensor = torch.from_numpy(time_means) - >>> lat = torch.from_numpy(y) - >>> acc(pred_tensor, targ_tensor, means_tensor, lat) - tensor([0.9841]) - - -.. autosummary:: - :toctree: generated - -General ---------- - -.. automodule:: modulus.metrics.general.mse - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.general.histogram - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.general.entropy - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.general.calibration - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.general.crps - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.general.ensemble_metrics - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.general.reduction - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.general.wasserstein - :members: - :show-inheritance: - -Weather and climate metrics ---------------------------- - -.. automodule:: modulus.metrics.climate.acc - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.climate.efi - :members: - :show-inheritance: - -.. automodule:: modulus.metrics.climate.reduction - :members: - :show-inheritance: - - diff --git a/docs/api/modulus.models.rst b/docs/api/modulus.models.rst deleted file mode 100644 index 78df8e7922..0000000000 --- a/docs/api/modulus.models.rst +++ /dev/null @@ -1,508 +0,0 @@ - -Modulus Models -============== - -.. automodule:: modulus.models -.. currentmodule:: modulus.models - -Basics -^^^^^^ - -Modulus contains its own Model class for constructing neural networks. This model class -is built on top of PyTorch's ``nn.Module`` and can be used interchangeably within the -PyTorch ecosystem. Using Modulus models allows you to leverage various features of -Modulus aimed at improving performance and ease of use. These features include, but are -not limited to, model zoo, automatic mixed-precision, CUDA Graphs, and easy checkpointing. -We discuss each of these features in the following sections. - -Model Zoo -^^^^^^^^^ - -Modulus contains several optimized, customizable and easy-to-use models. -These include some very general models like Fourier Neural Operators (FNOs), -ResNet, and Graph Neural Networks (GNNs) as well as domain-specific models like -Deep Learning Weather Prediction (DLWP) and Spherical Fourier Neural Operators (SFNO). - -Currently available models include: - -.. list-table:: - :widths: 20 40 40 - :header-rows: 1 - - * - Model Name - - Inputs - - Outputs - * - FullyConnected - - torch.Tensor [N, in_features] - - torch.Tensor [N, out_features] - * - FourierNeuralOperator - - torch.Tensor [N, in_channels, H, W] - - torch.Tensor [N, out_channels, H, W] - * - AdaptiveFourierNeuralOperator - - torch.Tensor [N, in_channels, H, W] - - torch.Tensor [N, out_channels, H, W] - * - MeshGraphNet - - torch.Tensor [num_nodes, input_dim_nodes], torch.Tensor [num_edges, input_dim_edges], dgl.DGLGraph [num_nodes, num_edges] - - torch.Tensor [num_nodes, output_dim] - * - GraphCastNet - - torch.Tensor [N, C_in, H, W] - - torch.Tensor [N, C_out, H, W] - * - Pix2PixNet - - torch.Tensor [N, in_channels, H, W] - - torch.Tensor [N, out_channels, H, W] - * - One2ManyRNN - - torch.Tensor [N, C, 1, H, W] - - torch.Tensor [N, C, T, H, W] - * - Seq2SeqRNN - - torch.Tensor [N, C, T, H, W] - - torch.Tensor [N, C, T, H, W] - * - SRResNet - - torch.Tensor [N, C_in, D, H, W] - - torch.Tensor [N, C_out, D_out, H_out, W_out] - * - DLWP - - torch.Tensor [N, C_in, 6, Res, Res] - - torch.Tensor [N, C_out, 6, Res, Res] - * - SphericalFourierNeuralOperatorNet - - torch.Tensor [N, C_in, H, W] - - torch.Tensor [N, C_out, H, W] - - -Below are some simple examples of how to use these models. - -.. code:: python - - >>> import torch - >>> from modulus.models.mlp.fully_connected import FullyConnected - >>> model = FullyConnected(in_features=32, out_features=64) - >>> input = torch.randn(128, 32) - >>> output = model(input) - >>> output.shape - torch.Size([128, 64]) - -.. code:: python - - >>> import torch - >>> from modulus.models.fno.fno import FNO - >>> model = FNO( - in_channels=4, - out_channels=3, - decoder_layers=2, - decoder_layer_size=32, - dimension=2, - latent_channels=32, - num_fno_layers=2, - padding=0, - ) - >>> input = torch.randn(32, 4, 32, 32) #(N, C, H, W) - >>> output = model(input) - >>> output.size() - torch.Size([32, 3, 32, 32]) - -How to write your own Modulus model -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -There are a few different ways to construct a Modulus model. If you are a seasoned -PyTorch user, the easiest way would be to write your model using the optimized layers and -utilities from Modulus or Pytorch. Lets take a look at a simple example of a UNet model -first showing a simple PyTorch implementation and then a Modulus implementation that -supports CUDA Graphs and Automatic Mixed-Precision. - -.. code:: python - - import torch.nn as nn - - class UNet(nn.Module): - def __init__(self, in_channels=1, out_channels=1): - super(UNet, self).__init__() - - self.enc1 = self.conv_block(in_channels, 64) - self.enc2 = self.conv_block(64, 128) - - self.dec1 = self.upconv_block(128, 64) - self.final = nn.Conv2d(64, out_channels, kernel_size=1) - - def conv_block(self, in_channels, out_channels): - return nn.Sequential( - nn.Conv2d(in_channels, out_channels, 3, padding=1), - nn.ReLU(inplace=True), - nn.MaxPool2d(2) - ) - - def upconv_block(self, in_channels, out_channels): - return nn.Sequential( - nn.ConvTranspose2d(in_channels, out_channels, 2, stride=2), - nn.Conv2d(out_channels, out_channels, 3, padding=1), - nn.ReLU(inplace=True) - ) - - def forward(self, x): - x1 = self.enc1(x) - x2 = self.enc2(x1) - x = self.dec1(x2) - return self.final(x) - -Now we show this model rewritten in Modulus. First, let's subclass the model from -``modulus.Module`` instead of ``torch.nn.Module``. The -``modulus.Module`` class acts like a direct replacement for the -``torch.nn.Module`` and provides additional functionality for saving and loading -checkpoints, etc. Refer to the API docs of ``modulus.Module`` for further -details. Additionally we will add metadata to the model to capture the optimizations -that this model supports. In this case we will enable CUDA Graphs and Automatic Mixed-Precision. - -.. code:: python - - from dataclasses import dataclass - import modulus - import torch.nn as nn - - @dataclass - class UNetMetaData(modulus.ModelMetaData): - name: str = "UNet" - # Optimization - jit: bool = True - cuda_graphs: bool = True - amp_cpu: bool = True - amp_gpu: bool = True - - class UNet(modulus.Module): - def __init__(self, in_channels=1, out_channels=1): - super(UNet, self).__init__(meta=UNetMetaData()) - - self.enc1 = self.conv_block(in_channels, 64) - self.enc2 = self.conv_block(64, 128) - - self.dec1 = self.upconv_block(128, 64) - self.final = nn.Conv2d(64, out_channels, kernel_size=1) - - def conv_block(self, in_channels, out_channels): - return nn.Sequential( - nn.Conv2d(in_channels, out_channels, 3, padding=1), - nn.ReLU(inplace=True), - nn.MaxPool2d(2) - ) - - def upconv_block(self, in_channels, out_channels): - return nn.Sequential( - nn.ConvTranspose2d(in_channels, out_channels, 2, stride=2), - nn.Conv2d(out_channels, out_channels, 3, padding=1), - nn.ReLU(inplace=True) - ) - - def forward(self, x): - x1 = self.enc1(x) - x2 = self.enc2(x1) - x = self.dec1(x2) - return self.final(x) - -Now that we have our Modulus model, we can make use of these optimizations using the -``modulus.utils.StaticCaptureTraining`` decorator. This decorator will capture the -training step function and optimize it for the specified optimizations. - -.. code:: python - - import torch - from modulus.utils import StaticCaptureTraining - - model = UNet().to("cuda") - input = torch.randn(8, 1, 128, 128).to("cuda") - output = torch.zeros(8, 1, 64, 64).to("cuda") - - optim = torch.optim.Adam(model.parameters(), lr=0.001) - - # Create training step function with optimization wrapper - # StaticCaptureTraining calls `backward` on the loss and - # `optimizer.step()` so you don't have to do that - # explicitly. - @StaticCaptureTraining( - model=model, - optim=optim, - cuda_graph_warmup=11, - ) - def training_step(invar, outvar): - predvar = model(invar) - loss = torch.sum(torch.pow(predvar - outvar, 2)) - return loss - - # Sample training loop - for i in range(20): - # In place copy of input and output to support cuda graphs - input.copy_(torch.randn(8, 1, 128, 128).to("cuda")) - output.copy_(torch.zeros(8, 1, 64, 64).to("cuda")) - - # Run training step - loss = training_step(input, output) - -For the simple model above, you can observe ~1.1x speed-up due to CUDA Graphs and AMP. -The speed-up observed changes from model to model and is typically greater for more -complex models. - -.. note:: - The ``ModelMetaData`` and ``modulus.Module`` do not make the model - support CUDA Graphs, AMP, etc. optimizations automatically. The user is responsible - to write the model code that enables each of these optimizations. - Models in the Modulus Model Zoo are written to support many of these optimizations - and checked against Modulus's CI to ensure that they work correctly. - -.. note:: - The ``StaticCaptureTraining`` decorator is still under development and may be - refactored in the future. - - -.. _modulus-models-from-torch: - -Converting PyTorch Models to Modulus Models -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In the above example we show constructing a Modulus model from scratch. However you -can also convert existing PyTorch models to Modulus models in order to leverage -Modulus features. To do this, you can use the ``Module.from_torch`` method as shown -below. - -.. code:: python - - from dataclasses import dataclass - import modulus - import torch.nn as nn - - class TorchModel(nn.Module): - def __init__(self): - super(TorchModel, self).__init__() - self.conv1 = nn.Conv2d(1, 20, 5) - self.conv2 = nn.Conv2d(20, 20, 5) - - def forward(self, x): - x = self.conv1(x) - return self.conv2(x) - - @dataclass - class ConvMetaData(ModelMetaData): - name: str = "UNet" - # Optimization - jit: bool = True - cuda_graphs: bool = True - amp_cpu: bool = True - amp_gpu: bool = True - - ModulusModel = modulus.Module.from_torch(TorchModel, meta=ConvMetaData()) - - - - -.. _saving-and-loading-modulus-models: - -Saving and Loading Modulus Models -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -As mentioned above, Modulus models are interoperable with PyTorch models. This means that -you can save and load Modulus models using the standard PyTorch APIs however, we provide -a few additional utilities to make this process easier. A key challenge in saving and -loading models is keeping track of the model metadata such as layer sizes, etc. Modulus -models can be saved with this metadata to a custom ``.mdlus`` file. These files allow -for easy loading and instantiation of the model. We show two examples of this below. -The first example shows saving and loading a model from an already instantiated model. - -.. code:: python - - >>> from modulus.models.mlp.fully_connected import FullyConnected - >>> model = FullyConnected(in_features=32, out_features=64) - >>> model.save("model.mdlus") # Save model to .mdlus file - >>> model.load("model.mdlus") # Load model weights from .mdlus file from already instantiated model - >>> model - FullyConnected( - (layers): ModuleList( - (0): FCLayer( - (activation_fn): SiLU() - (linear): Linear(in_features=32, out_features=512, bias=True) - ) - (1-5): 5 x FCLayer( - (activation_fn): SiLU() - (linear): Linear(in_features=512, out_features=512, bias=True) - ) - ) - (final_layer): FCLayer( - (activation_fn): Identity() - (linear): Linear(in_features=512, out_features=64, bias=True) - ) - ) - -The second example shows loading a model from a ``.mdlus`` file without having to -instantiate the model first. We note that in this case we don't know the class or -parameters to pass to the constructor of the model. However, we can still load the -model from the ``.mdlus`` file. - -.. code:: python - - >>> from modulus import Module - >>> fc_model = Module.from_checkpoint("model.mdlus") # Instantiate model from .mdlus file. - >>> fc_model - FullyConnected( - (layers): ModuleList( - (0): FCLayer( - (activation_fn): SiLU() - (linear): Linear(in_features=32, out_features=512, bias=True) - ) - (1-5): 5 x FCLayer( - (activation_fn): SiLU() - (linear): Linear(in_features=512, out_features=512, bias=True) - ) - ) - (final_layer): FCLayer( - (activation_fn): Identity() - (linear): Linear(in_features=512, out_features=64, bias=True) - ) - ) - - - -.. note:: - In order to make use of this functionality, the model must have json serializable - inputs to the ``__init__`` function. It is highly recommended that all Modulus - models be developed with this requirement in mind. - - -Modulus Model Registry and Entry Points -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Modulus contains a model registry that allows for easy access and ingestion of -models. Below is a simple example of how to use the model registry to obtain a model -class. - -.. code:: python - - >>> from modulus.registry import ModelRegistry - >>> model_registry = ModelRegistry() - >>> model_registry.list_models() - ['AFNO', 'DLWP', 'FNO', 'FullyConnected', 'GraphCastNet', 'MeshGraphNet', 'One2ManyRNN', 'Pix2Pix', 'SFNO', 'SRResNet'] - >>> FullyConnected = model_registry.factory("FullyConnected") - >>> model = FullyConnected(in_features=32, out_features=64) - -The model registry also allows exposing models via entry points. This allows for -integration of models into the Modulus ecosystem. For example, suppose you have a -package ``MyPackage`` that contains a model ``MyModel``. You can expose this model -to the Modulus registry by adding an entry point to your ``toml`` file. For -example, suppose your package structure is as follows: - -.. code:: python - - # setup.py - - from setuptools import setup, find_packages - - setup() - -.. code:: python - - # pyproject.toml - - [build-system] - requires = ["setuptools", "wheel"] - build-backend = "setuptools.build_meta" - - [project] - name = "MyPackage" - description = "My Neural Network Zoo." - version = "0.1.0" - - [project.entry-points."modulus.models"] - MyModulusModel = "mypackage.models.MyModulusModel:MyModulusModel" - -.. code:: python - - # mypackage/models.py - - import torch.nn as nn - from modulus.models import Model - - class MyModel(nn.Module): - def __init__(self): - super(MyModel, self).__init__() - self.conv1 = nn.Conv2d(1, 20, 5) - self.conv2 = nn.Conv2d(20, 20, 5) - - def forward(self, x): - x = self.conv1(x) - return self.conv2(x) - - MyModulusModel = Model.from_pytorch(MyModel) - - -Once this package is installed, you can access the model via the Modulus model -registry. - - -.. code:: python - - >>> from modulus.registry import ModelRegistry - >>> model_registry = ModelRegistry() - >>> model_registry.list_models() - ['MyModulusModel', 'AFNO', 'DLWP', 'FNO', 'FullyConnected', 'GraphCastNet', 'MeshGraphNet', 'One2ManyRNN', 'Pix2Pix', 'SFNO', 'SRResNet'] - >>> MyModulusModel = model_registry.factory("MyModulusModel") - - -For more information on entry points and potential use cases, see -`this `_ blog post. - -.. autosummary:: - :toctree: generated - -Fully Connected Network ------------------------ - -.. automodule:: modulus.models.mlp.fully_connected - :members: - :show-inheritance: - -Fourier Neural Operators ------------------------- - -.. automodule:: modulus.models.fno.fno - :members: - :show-inheritance: - -.. automodule:: modulus.models.afno.afno - :members: - :show-inheritance: - -Graph Neural Networks ---------------------- - -.. automodule:: modulus.models.meshgraphnet.meshgraphnet - :members: - :show-inheritance: - -.. automodule:: modulus.models.graphcast.graph_cast_net - :members: - :show-inheritance: - -Pix2Pix Net ------------ - -.. automodule:: modulus.models.pix2pix.pix2pix - :members: - :show-inheritance: - -Recurrent Neural Networks -------------------------- - -.. automodule:: modulus.models.rnn.rnn_one2many - :members: - :show-inheritance: - -.. automodule:: modulus.models.rnn.rnn_seq2seq - :members: - :show-inheritance: - -Super Resolution Network ------------------------- - -.. automodule:: modulus.models.srrn.super_res_net - :members: - :show-inheritance: - -DLWP Model ----------- - -.. automodule:: modulus.models.dlwp.dlwp - :members: - :show-inheritance: - diff --git a/docs/api/modulus.utils.rst b/docs/api/modulus.utils.rst deleted file mode 100644 index 22bed63839..0000000000 --- a/docs/api/modulus.utils.rst +++ /dev/null @@ -1,43 +0,0 @@ -Modulus Utils -============= - -.. autosummary:: - :toctree: generated - -Optimization utils ------------------- - -.. automodule:: modulus.utils.capture - :members: - :show-inheritance: - -GraphCast utils ---------------- - -.. automodule:: modulus.utils.graphcast.data_utils - :members: - :show-inheritance: - -.. automodule:: modulus.utils.graphcast.graph - :members: - :show-inheritance: - -.. automodule:: modulus.utils.graphcast.graph_utils - :members: - :show-inheritance: - -.. automodule:: modulus.utils.graphcast.icospheres - :members: - :show-inheritance: - -.. automodule:: modulus.utils.graphcast.loss - :members: - :show-inheritance: - -Filesystem utils ----------------- - -.. automodule:: modulus.utils.filesystem - :members: - :show-inheritance: - diff --git a/docs/api/nn/functionals/fourier_spectral.rst b/docs/api/nn/functionals/fourier_spectral.rst new file mode 100644 index 0000000000..a3f076df96 --- /dev/null +++ b/docs/api/nn/functionals/fourier_spectral.rst @@ -0,0 +1,18 @@ +FFT Functionals +=============== + +.. autofunction:: physicsnemo.nn.functional.rfft + +.. autofunction:: physicsnemo.nn.functional.rfft2 + +.. autofunction:: physicsnemo.nn.functional.irfft + +.. autofunction:: physicsnemo.nn.functional.irfft2 + +.. rubric:: Utilities + +.. autofunction:: physicsnemo.nn.functional.view_as_complex + +.. autofunction:: physicsnemo.nn.functional.real + +.. autofunction:: physicsnemo.nn.functional.imag diff --git a/docs/api/nn/functionals/geometry.rst b/docs/api/nn/functionals/geometry.rst new file mode 100644 index 0000000000..d87a9294e5 --- /dev/null +++ b/docs/api/nn/functionals/geometry.rst @@ -0,0 +1,4 @@ +Geometry Functionals +==================== + +.. autofunction:: physicsnemo.nn.functional.signed_distance_field diff --git a/docs/api/nn/functionals/neighbors.rst b/docs/api/nn/functionals/neighbors.rst new file mode 100644 index 0000000000..426081c4f8 --- /dev/null +++ b/docs/api/nn/functionals/neighbors.rst @@ -0,0 +1,18 @@ +Neighbor Functionals +==================== + +KNN +--- + +.. autofunction:: physicsnemo.nn.functional.knn + +Radius Search +------------- + +.. autofunction:: physicsnemo.nn.functional.radius_search + +.. rubric:: Benchmarks (ASV) + +.. figure:: /nn/functional/radius_search/benchmark.png + :alt: Radius search benchmark comparison + :width: 100% diff --git a/docs/api/nn/functionals/regularization_parameterization.rst b/docs/api/nn/functionals/regularization_parameterization.rst new file mode 100644 index 0000000000..3d64624d43 --- /dev/null +++ b/docs/api/nn/functionals/regularization_parameterization.rst @@ -0,0 +1,6 @@ +Regularization and Parameterization Functionals +=============================================== + +.. autofunction:: physicsnemo.nn.functional.drop_path + +.. autofunction:: physicsnemo.nn.functional.weight_fact diff --git a/docs/api/nn/functionals/resampling_interpolation.rst b/docs/api/nn/functionals/resampling_interpolation.rst new file mode 100644 index 0000000000..43b7b85b32 --- /dev/null +++ b/docs/api/nn/functionals/resampling_interpolation.rst @@ -0,0 +1,10 @@ +Interpolation Functionals +========================= + +.. autofunction:: physicsnemo.nn.functional.interpolation + +.. rubric:: Benchmarks (ASV) + +.. figure:: /nn/functional/interpolation/benchmark.png + :alt: Interpolation benchmark comparison + :width: 100% diff --git a/docs/api/nn/layers/activations.rst b/docs/api/nn/layers/activations.rst new file mode 100644 index 0000000000..352e321e4d --- /dev/null +++ b/docs/api/nn/layers/activations.rst @@ -0,0 +1,14 @@ +Activations +=========== + +.. automodule:: physicsnemo.nn.module.activations + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.fused_silu + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.gumbel_softmax + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/attention_transformers.rst b/docs/api/nn/layers/attention_transformers.rst new file mode 100644 index 0000000000..36e5ab2067 --- /dev/null +++ b/docs/api/nn/layers/attention_transformers.rst @@ -0,0 +1,18 @@ +Attention and Transformers +========================== + +.. automodule:: physicsnemo.nn.module.attention_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.transformer_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.transformer_decoder + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.physics_attention + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/convolutional.rst b/docs/api/nn/layers/convolutional.rst new file mode 100644 index 0000000000..5ea5f8d3b4 --- /dev/null +++ b/docs/api/nn/layers/convolutional.rst @@ -0,0 +1,6 @@ +Convolutional Layers +==================== + +.. automodule:: physicsnemo.nn.module.conv_layers + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/embeddings.rst b/docs/api/nn/layers/embeddings.rst new file mode 100644 index 0000000000..870084db4b --- /dev/null +++ b/docs/api/nn/layers/embeddings.rst @@ -0,0 +1,6 @@ +Embeddings +========== + +.. automodule:: physicsnemo.nn.module.embedding_layers + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/fourier_spectral.rst b/docs/api/nn/layers/fourier_spectral.rst new file mode 100644 index 0000000000..2288f01915 --- /dev/null +++ b/docs/api/nn/layers/fourier_spectral.rst @@ -0,0 +1,22 @@ +Fourier, FFT, and Spectral Layers +================================= + +.. automodule:: physicsnemo.nn.module.fourier_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.spectral_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.fft + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.afno_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.fno_layers + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/fully_connected_mlp.rst b/docs/api/nn/layers/fully_connected_mlp.rst new file mode 100644 index 0000000000..961d1e491f --- /dev/null +++ b/docs/api/nn/layers/fully_connected_mlp.rst @@ -0,0 +1,10 @@ +Fully Connected and MLP Layers +============================== + +.. automodule:: physicsnemo.nn.module.fully_connected_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.mlp_layers + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/graph_geometry.rst b/docs/api/nn/layers/graph_geometry.rst new file mode 100644 index 0000000000..7e6163909c --- /dev/null +++ b/docs/api/nn/layers/graph_geometry.rst @@ -0,0 +1,14 @@ +Graph and Geometry Layers +========================= + +.. automodule:: physicsnemo.nn.module.ball_query + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.dgm_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.gnn_layers + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/normalization.rst b/docs/api/nn/layers/normalization.rst new file mode 100644 index 0000000000..b6ee5a195a --- /dev/null +++ b/docs/api/nn/layers/normalization.rst @@ -0,0 +1,10 @@ +Normalization +============= + +.. automodule:: physicsnemo.nn.module.group_norm + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.layer_norm + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/regularization.rst b/docs/api/nn/layers/regularization.rst new file mode 100644 index 0000000000..d8ae036507 --- /dev/null +++ b/docs/api/nn/layers/regularization.rst @@ -0,0 +1,6 @@ +Regularization +============== + +.. automodule:: physicsnemo.nn.module.drop + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/resampling_interpolation.rst b/docs/api/nn/layers/resampling_interpolation.rst new file mode 100644 index 0000000000..7e162e159f --- /dev/null +++ b/docs/api/nn/layers/resampling_interpolation.rst @@ -0,0 +1,6 @@ +Resampling and Interpolation +============================ + +.. automodule:: physicsnemo.nn.module.resample_layers + :members: + :show-inheritance: diff --git a/docs/api/nn/layers/specialized.rst b/docs/api/nn/layers/specialized.rst new file mode 100644 index 0000000000..d60dce2881 --- /dev/null +++ b/docs/api/nn/layers/specialized.rst @@ -0,0 +1,26 @@ +Specialized Layers +================== + +.. automodule:: physicsnemo.nn.module.kan_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.siren_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.unet_layers + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.weight_norm + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.weight_fact + :members: + :show-inheritance: + +.. automodule:: physicsnemo.nn.module.hpx + :members: + :show-inheritance: diff --git a/docs/api/physicsnemo.active_learning.rst b/docs/api/physicsnemo.active_learning.rst new file mode 100644 index 0000000000..dc2fe6afb8 --- /dev/null +++ b/docs/api/physicsnemo.active_learning.rst @@ -0,0 +1,87 @@ +PhysicsNeMo Active Learning +=========================== + +.. currentmodule:: physicsnemo.active_learning + +Developing Active Learning Workflows +------------------------------------- + +For a high level overview and understanding of how to construct active +learning workflows using PhysicsNeMo, users should consult the `User +Guide `_ +. The guide will motivate the need for active learning, the abstraction +provided by PhysicsNeMo, and some additional tips for developing custom +components like querying and labeling strategies. + +API Reference +------------- + +Protocols +^^^^^^^^^ + +.. automodule:: physicsnemo.active_learning.protocols + :members: + :undoc-members: + :show-inheritance: + +Configuration Classes +^^^^^^^^^^^^^^^^^^^^^ + +These data structures are used to modify the behavior of different components +of the active learning workflow. The general pattern is to ensure that they +are JSON-serializable so that they can be checkpointed and restarted. + +.. automodule:: physicsnemo.active_learning.config + :members: + :undoc-members: + :show-inheritance: + + +Default Training Loop +^^^^^^^^^^^^^^^^^^^^^ + +This module and corresponding +:class:`~physicsnemo.active_learning.loop.DefaultTrainingLoop` class +implements the :class:`~physicsnemo.active_learning.protocols.TrainingLoop` interface, +and should provide most of the necessary boilerplate for model training +and fine-tuning; users will need to provide the training, validation, +and testing step protocols when configuring the loop. + +.. automodule:: physicsnemo.active_learning.loop + :members: + :undoc-members: + :show-inheritance: + +Active Learning Driver +^^^^^^^^^^^^^^^^^^^^^^ + +This module and class implements the +:class:`~physicsnemo.active_learning.protocols.DriverProtocol` interface, +and is usable out-of-the-box for most active learning workflows. The +:class:`~physicsnemo.active_learning.driver.Driver` class is configured +by :class:`~physicsnemo.active_learning.config.DriverConfig`, and serves +as the focal point for orchestrating the active learning. + +.. automodule:: physicsnemo.active_learning.driver + :members: + :undoc-members: + :show-inheritance: + +Active Learning Registry +^^^^^^^^^^^^^^^^^^^^^^^ + +The registry provides a centralized location for registering and constructing +custom active learning strategies. It enables string-based lookups for +checkpointing and provides argument validation when constructing protocol +instances. + +.. note:: + + Users should not use the class directly, but rather the instance of the + class through the :data:`~physicsnemo.active_learning.registry` object, + documented below. + +.. autodata:: physicsnemo.active_learning.registry + :annotation: = ActiveLearningRegistry() + + Global registry instance for active learning protocols. diff --git a/docs/api/physicsnemo.datapipes.rst b/docs/api/physicsnemo.datapipes.rst new file mode 100644 index 0000000000..47773f8791 --- /dev/null +++ b/docs/api/physicsnemo.datapipes.rst @@ -0,0 +1,201 @@ +PhysicsNeMo Datapipes +====================== + +.. automodule:: physicsnemo.datapipes +.. currentmodule:: physicsnemo.datapipes + +PhysicsNeMo Datapipes provides a collection of data loading and processing utilities designed +to handle various types of physics-based datasets. The datapipes are organized into several +categories to support different types of physics simulations and machine learning tasks. + +The datapipes in PhysicsNeMo are built on top of PhysicsNeMo's DataPipe class and provide +specialized implementations for handling physics simulation data, climate data, +graph-based data, and mesh-based data. Each category of datapipes is designed to efficiently +load and preprocess specific types of physics datasets. + +The example below shows how to use a typical datapipe in PhysicsNeMo: + +.. code:: python + + import torch + from physicsnemo.datapipes.benchmarks.darcy import Darcy2D + + def main(): + # Create a datapipe for Darcy flow simulation data + datapipe = Darcy2D( + batch_size=32, + device="cuda" if torch.cuda.is_available() else "cpu" + ) + + # Iterate through the datapipe + for batch in datapipe: + # batch contains input features and target values + input_features = batch["permeability"] + target_values = batch["darcy"] + + # Use the data for training or inference + ... + + if __name__ == "__main__": + main() + +Here's another example showing how to use the ERA5HDF5Datapipe for weather data processing: + +.. code:: python + + import torch + from physicsnemo.datapipes.climate.era5_hdf5 import ERA5HDF5Datapipe + + def main(): + # Create a datapipe for ERA5 weather data in HDF5 format + datapipe = ERA5HDF5Datapipe( + data_dir="path/to/era5/data", + stats_dir="path/to/era5/stats", + channels=[0, 1], + latlon_resolution=(721, 1440), + shuffle=True, + ) + + # Iterate through the datapipe + for batch in datapipe: + invar = batch[0]["invar"] + outvar = batch[0]["outvar"] + + # Use the data for weather prediction or analysis + ... + + if __name__ == "__main__": + main() + +Available Datapipes +""""""""""""""""""" + +PhysicsNeMo provides several categories of datapipes: + +1. Benchmark Datapipes + - Designed for standard physics benchmark problems + - Include implementations for Darcy flow and Kelvin-Helmholtz instability + +2. Weather and Climate Datapipes + - Specialized for handling climate and weather data + - Support ERA5 HDF5 format and synthetic climate data + - Include utilities for HEALPix time series data + +3. Graph Datapipes + - Handle graph-based physics data + - Support vortex shedding, Ahmed body, DrivaerNet, and Stokes flow datasets + - Include utility functions for graph data processing + +4. CAE (Computer-Aided Engineering) Datapipes + - Specialized for mesh-based data + - Support various mesh formats and configurations + +Each category of datapipes is designed to handle specific data formats and preprocessing +requirements. The datapipes automatically handle data loading, preprocessing, +and device placement, making it easy to integrate them into training or inference pipelines. + +.. autosummary:: + :toctree: generated + +Benchmark datapipes +------------------- + +.. automodule:: physicsnemo.datapipes.benchmarks.darcy + :members: + :show-inheritance: + +The Darcy2D provides data loading and preprocessing utilities for 2D Darcy +flow simulations. It handles permeability fields and pressure solutions, supporting +various boundary conditions and mesh resolutions. + +.. automodule:: physicsnemo.datapipes.benchmarks.kelvin_helmholtz + :members: + :show-inheritance: + +The KelvinHelmholtz2D manages data for Kelvin-Helmholtz instability simulations, +including velocity fields and density distributions. It supports both 2D and 3D simulation +data with various initial conditions. + +Weather and climate datapipes +----------------------------- + +.. automodule:: physicsnemo.datapipes.climate.era5_hdf5 + :members: + :show-inheritance: + +The ERA5HDF5Datapipe handles ERA5 reanalysis data stored in HDF5 format, providing access to +atmospheric variables like temperature, pressure, and wind fields at various pressure levels. + +.. automodule:: physicsnemo.datapipes.climate.climate + :members: + :show-inheritance: + +The ClimateDataPipe provides a general interface for climate data processing, supporting +various climate datasets and variables with standardized preprocessing and normalization. + +.. automodule:: physicsnemo.datapipes.climate.synthetic + :members: + :show-inheritance: + +The SyntheticWeatherDataset generates synthetic climate data for testing and development +purposes, supporting various climate patterns and noise models. + +.. automodule:: physicsnemo.datapipes.healpix.timeseries_dataset + :members: + :show-inheritance: + +The TimeSeriesDataset handles spherical harmonic data in HEALPix format, +supporting time series analysis of global climate variables. + +Graph datapipes +--------------- + +.. automodule:: physicsnemo.datapipes.gnn.vortex_shedding_dataset + :members: + :show-inheritance: + +The VortexSheddingDataset processes flow field data around bluff bodies, +capturing vortex shedding patterns and flow structures for graph-based learning. + +.. automodule:: physicsnemo.datapipes.gnn.ahmed_body_dataset + :members: + :show-inheritance: + +The AhmedBodyDataset manages flow field data around Ahmed bodies, supporting aerodynamic +analysis and drag prediction tasks. + +.. automodule:: physicsnemo.datapipes.gnn.drivaernet_dataset + :members: + :show-inheritance: + +The DrivaerNetDataset handles automotive aerodynamics data, providing access to flow field +measurements and surface pressure distributions. + +.. automodule:: physicsnemo.datapipes.gnn.stokes_dataset + :members: + :show-inheritance: + +The StokesDataset processes Stokes flow simulations, supporting various boundary conditions +and geometry configurations for microfluidic applications. + +.. automodule:: physicsnemo.datapipes.gnn.utils + :members: + :show-inheritance: + +The GNN utilities provide helper functions for graph construction, feature extraction, +and data preprocessing in graph-based physics learning tasks. + +CAE datapipes +------------- + +.. automodule:: physicsnemo.datapipes.cae.mesh_datapipe + :members: + :show-inheritance: + +The MeshDataPipe handles mesh data for physics simulations, +supporting various mesh formats and providing utilities for mesh preprocessing +and feature extraction. + +.. automodule:: physicsnemo.datapipes.cae.domino_datapipe + :members: + :show-inheritance: diff --git a/docs/api/physicsnemo.deploy.rst b/docs/api/physicsnemo.deploy.rst new file mode 100644 index 0000000000..e1ae30a713 --- /dev/null +++ b/docs/api/physicsnemo.deploy.rst @@ -0,0 +1,53 @@ +PhysicsNeMo Deploy +================== + +.. automodule:: physicsnemo.deploy +.. currentmodule:: physicsnemo.deploy + +Deploying models trained in PhysicsNeMo +--------------------------------------- + +Application developers can deploy PhysicsNeMo either as the training framework or +deploy inference recipes using models trained in PhysicsNeMo into their applications. +PhysicsNeMo is written natively in python and you can use the standard python packaging +and deployment practices to productize your applications. It is provided under the +`Apache License 2.0 `_. + +physicsnemo.deploy.onnx +----------------------- + +ONNX is a standard format for representing and exchanging machine learning models in +other frameworks or environments without significant re-work. The +physicsnemo.deploy.onnx module translates a model from physicsnemo.model and converts +it into an ONNX graph. + +The exported model can be consumed by any of the many runtimes that support ONNX, including Microsoft’s ONNX Runtime. + +Next example shows how to export a simple model. + +.. code:: python + + from physicsnemo.deploy.onnx import export_to_onnx_stream, run_onnx_inference + from physicsnemo.models.mlp import FullyConnected + + model = FullyConnected( + in_features=32, + out_features=8, + num_layers=1, + layer_size=8, + ) + x = torch.randn(4, 32).to(device) + y = model(x) # Get PyTorch output + + onnx_stream = export_to_onnx_stream(model) + ort_y = run_onnx_inference(onnx_stream, x) + ort_y = torch.Tensor(ort_y[0]) + + # test the output + assert torch.allclose(y, ort_y, atol=1e-4) + +ONNX +---- +.. automodule:: physicsnemo.deploy.onnx.utils + :members: + :show-inheritance: diff --git a/docs/api/physicsnemo.distributed.rst b/docs/api/physicsnemo.distributed.rst new file mode 100644 index 0000000000..5a8b25f543 --- /dev/null +++ b/docs/api/physicsnemo.distributed.rst @@ -0,0 +1,211 @@ +PhysicsNeMo Distributed +======================== + +.. automodule:: physicsnemo.distributed +.. currentmodule:: physicsnemo.distributed + +Distributed utilities in PhysicsNeMo are designed to simplify the implementation of parallel training and +make inference scripts easier by providing a unified way to configure and query parameters associated +with the distributed environment. The utilities in ``physicsnemo.distributed`` build on top of the +utilities from ``torch.distributed`` and abstract out some of the complexities of setting up a +distributed execution environment. + +The example below shows how to set up a simple distributed data parallel training recipe using the +distributed utilities in PhysicsNeMo. +`DistributedDataParallel `_ +in PyTorch provides the framework for data parallel training by reducing parameter gradients +across multiple worker processes after the backwards pass. The code below shows how to specify +the ``device_ids``, ``output_device``, ``broadcast_buffers`` and ``find_unused_parameters`` +arguments of the ``DistributedDataParallel`` utility using the ``DistributedManager``. + +.. code:: python + + import torch + from torch.nn.parallel import DistributedDataParallel + from physicsnemo.distributed import DistributedManager + from physicsnemo.models.mlp.fully_connected import FullyConnected + + def main(): + # Initialize the DistributedManager. This will automatically + # detect the number of processes the job was launched with and + # set those configuration parameters appropriately. Currently + # torchrun (or any other pytorch compatible launcher), mpirun (OpenMPI) + # and SLURM based launchers are supported. + DistributedManager.initialize() + + # Since this is a singleton class, you can just get an instance + # of it anytime after initialization and not need to reinitialize + # each time. + dist = DistributedManager() + + # Set up model on the appropriate device. DistributedManager + # figures out what device should be used on this process + arch = FullyConnected(in_features=32, out_features=64).to(dist.device) + + # Set up DistributedDataParallel if using more than a single process. + # The `distributed` property of DistributedManager can be used to + # check this. + if dist.distributed: + ddps = torch.cuda.Stream() + with torch.cuda.stream(ddps): + arch = DistributedDataParallel( + arch, + device_ids=[dist.local_rank], # Set the device_id to be + # the local rank of this process on + # this node + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + ) + torch.cuda.current_stream().wait_stream(ddps) + + # Set up the optimizer + optimizer = torch.optim.Adam( + arch.parameters(), + lr=0.001, + ) + + def training_step(input, target): + pred = arch(invar) + loss = torch.sum(torch.pow(pred - target, 2)) + loss.backward() + optimizer.step() + return loss + + # Sample training loop + for i in range(20): + # Random inputs and targets for simplicity + input = torch.randn(128, 32, device=dist.device) + target = torch.randn(128, 64, device=dist.device) + + # Training step + loss = training_step(input, target) + + if __name__ == "__main__": + main() + +This training script can be run on a single GPU +using ``python train.py`` or on multiple GPUs using + +.. code-block:: bash + + torchrun --standalone --nnodes=1 --nproc_per_node= train.py + +or + +.. code-block:: bash + + mpirun -np python train.py + +if using OpenMPI. The script can also +be run on a SLURM cluster using + +.. code-block:: bash + + srun -n python train.py + +How does this work? +""""""""""""""""""" + +An important aspect of the ``DistributedManager`` is that it follows the +`Borg pattern `_. +This means that ``DistributedManager`` essentially functions like a singleton +class and once configured, all utilities in PhysicsNeMo can access the same configuration +and adapt to the specified distributed structure. + +For example, see the constructor of the ``DistributedAFNO`` class: + +.. code-block:: python + + def __init__( + self, + inp_shape: Tuple[int, int], + in_channels: int, + out_channels: Union[int, Any] = None, + patch_size: int = 16, + embed_dim: int = 256, + depth: int = 4, + num_blocks: int = 4, + channel_parallel_inputs: bool = False, + channel_parallel_outputs: bool = False, + ) -> None: + super().__init__() + + out_channels = out_channels or in_channels + + if DistributedManager().group("model_parallel") is None: + raise RuntimeError( + "Distributed AFNO needs to have model parallel group created first. " + "Check the MODEL_PARALLEL_SIZE environment variable" + ) + + comm_size = DistributedManager().group_size("model_parallel") + if channel_parallel_inputs: + if not (in_channels % comm_size == 0): + raise ValueError( + "Error, in_channels needs to be divisible by model_parallel size" + ) + + self._impl = DistributedAFNONet( + inp_shape=inp_shape, + patch_size=(patch_size, patch_size), + in_chans=in_channels, + out_chans=out_channels, + embed_dim=embed_dim, + depth=depth, + num_blocks=num_blocks, + input_is_matmul_parallel=False, + output_is_matmul_parallel=False, + ) + +This model parallel implementation can just instantiate ``DistributedManager`` and query +if the process group named ``"model_parallel"`` exists and if so, what its size is. Similarly, +other utilities can query what device to run on, the total size of the distributed run, etc. +without having to explicitly pass those parameters down the call stack. + +.. note:: + + This singleton/borg pattern is very useful for the ``DistributedManager`` since it takes charge + of bootstrapping the distributed run and unifies how all utilities become aware of the distributed + configuration. However, the singleton/borg pattern is not just a way to avoid passing parameters + to utilities. Use of this pattern should be limited and have good justification to avoid losing + traceability and keep the code readable. + + +.. autosummary:: + :toctree: generated + +physicsnemo.distributed.manager +-------------------------------- + +.. automodule:: physicsnemo.distributed.manager + :members: + :show-inheritance: + +physicsnemo.distributed.utils +----------------------------- + +.. automodule:: physicsnemo.distributed.utils + :members: + :show-inheritance: + +physicsnemo.distributed.autograd +-------------------------------- + +.. automodule:: physicsnemo.distributed.autograd + :members: + :show-inheritance: + +physicsnemo.distributed.fft +---------------------------- + +.. automodule:: physicsnemo.distributed.fft + :members: + :show-inheritance: + +physicsnemo.distributed.mappings +-------------------------------- + +.. automodule:: physicsnemo.distributed.mappings + :members: + :show-inheritance: diff --git a/docs/api/physicsnemo.distributed.shardtensor.rst b/docs/api/physicsnemo.distributed.shardtensor.rst new file mode 100644 index 0000000000..957c0319d6 --- /dev/null +++ b/docs/api/physicsnemo.distributed.shardtensor.rst @@ -0,0 +1,121 @@ + +PhysicsNeMo ``ShardTensor`` +=========================== + +In scientific AI applications, the parallelization techniques to enable state of the art +models are different from those used in training large language models. PhysicsNeMo +introduces a new parallelization primitive called a ``ShardTensor`` that is designed for +large-input AI applications to enable domain parallelization. + +``ShardTensor`` provides a distributed tensor implementation that supports uneven sharding across devices. +It builds on PyTorch's DTensor while adding flexibility for cases where different ranks may have +different local tensor sizes. + +The example below shows how to create and work with ``ShardTensor``: + +.. code:: python + + import torch + from torch.distributed.device_mesh import DeviceMesh + from torch.distributed.tensor.placement_types import Shard + from physicsnemo.distributed import DistributedManager + from physicsnemo.distributed.shard_tensor import ShardTensor, scatter_tensor + + def main(): + # Initialize distributed environment + DistributedManager.initialize() + dm = DistributedManager() + + # Create a 1D device mesh - by default, a -1 will use all devices + # (For a 2D mesh, -1 will work to infer a single dimension in a mesh tensor) + mesh = dm.initialize_mesh((-1,), mesh_dim_names=["spatial"]) + + # Create a tensor on rank 0 + if dist.rank == 0: + tensor = torch.randn(100, 64) + else: + tensor = None + + # Scatter the tensor across devices with uneven sharding + # This will automatically determine appropriate local sizes + sharded = scatter_tensor( + tensor, + global_src=0, + mesh=mesh, + placements=(Shard(0),) # Shard along first dimension + ) + + # Work with local portions + local_tensor = sharded.to_local() + + # Redistribute to different sharding scheme + new_sharded = sharded.redistribute( + placements=(Shard(1),) # Change to shard along second dimension + ) + +How does this work? +""""""""""""""""""" + +``ShardTensor`` extends PyTorch's ``DTensor`` to support uneven sharding where different ranks can have different +local tensor sizes. It tracks shard size information and handles redistribution between different +sharding schemes while maintaining gradient flow. + +Key differences from ``DTensor`` include: +- Support for uneven sharding where ranks have different local sizes +- Tracking and propagation of shard size information +- Custom collective operations optimized for uneven sharding +- Flexible redistribution between different sharding schemes + +Operations work by: + +1. Converting inputs to local tensors + +2. Performing operations locally + +3. Constructing new ``ShardTensor`` with appropriate sharding + +4. Handling any needed communication between ranks + +.. autosummary:: + :toctree: generated + +``ShardTensor`` +--------------- + +.. autoclass:: physicsnemo.distributed.shard_tensor.ShardTensor + :members: + :show-inheritance: + +Utility Functions +----------------- + +.. autofunction:: physicsnemo.distributed.shard_tensor.scatter_tensor + + +Why do we need this? +"""""""""""""""""""" + +During deep learning training, memory usage can grow significantly when working with large input data, even if the model itself is relatively small. This is because many operations create intermediate tensors that temporarily consume memory. + +For example, consider a 2D convolution operation on a high-resolution image. If we have a batch of 1024x1024 images, even a simple 3x3 convolution needs to save the entire input image in memory for computing the gradients in the backward pass. + +For high resolution images, this can easily lead to out of memory errors as model depth grows, even if the number of parameters is small - this is a significant contrast from LLM model training, where the memory usage is dominated by the number of parameters and the corresponding optimizer states. In software solutions like DeepSpeed and ZeRO, this is handled by partitioning the model across GPUs, but this is not a solution for large-input applications. + +``ShardTensor`` helps address this by: +- Distributing the input data across multiple devices +- Performing operations on smaller local portions +- Coordinating the necessary communication between devices in the forward and backward passes + +``ShardTensor`` is built as an extension of PyTorch's DTensor, and gains substantial functionality by leveraging the utilities already implemented in the PyTorch distributed package. However, some operations on sharded input data are not trivial to implement correctly, nor relevant to the model sharding problem. In PhysicsNeMo, we have implemented parallelized versions of several key operations, including (so far): + +- Convolution (1D, 2D, 3D) +- Neighborhood Attention (2D) + +These operations are implemented in the ``physicsnemo.distributed.shard_utils`` module, and are enabled by dynamically intercepting calls to (for example) ``torch.nn.functional.conv2d``. When the function is called with ShardTensor inputs, the operation is automatically parallelized across the mesh associated with the input. When the function is called with non-ShardTensor inputs, the operation is executed in a non-parallelized manner, exactly as expected. + +To enable these operations, you must import ``patch_operations`` from ``physicsnemo.distributed.shard_utils``. This will patch the relevant functions in the distributed package to support ``ShardTensor`` inputs. + +We are continuing to add more operations, and contributions are welcome! + + + diff --git a/docs/api/physicsnemo.launch.logging.rst b/docs/api/physicsnemo.launch.logging.rst new file mode 100644 index 0000000000..6a100d5957 --- /dev/null +++ b/docs/api/physicsnemo.launch.logging.rst @@ -0,0 +1,148 @@ +PhysicsNeMo Launch Logging +=========================== + +.. automodule:: physicsnemo.launch.logging +.. currentmodule:: physicsnemo.launch.logging + +The PhysicsNeMo Launch Logging module provides a comprehensive and flexible logging system for machine learning experiments +and physics simulations. It offers multiple logging backends including console output, MLflow, and Weights & Biases (W&B), +allowing users to track metrics, artifacts, and experiment parameters across different platforms. The module is designed to +work seamlessly in both single-process and distributed training environments. + +Key Features: +- Unified logging interface across different backends +- Support for distributed training environments +- Automatic metric aggregation and synchronization +- Flexible configuration and customization options +- Integration with popular experiment tracking platforms + +Consider the following example usage: + +.. code:: python + + from physicsnemo.launch.logging import LaunchLogger + + # Initialize the logger + logger = LaunchLogger.initialize(use_mlflow=True) + + # Training loop + for epoch in range(num_epochs): + + # Training logger + with LaunchLogger( + "train", epoch = epoch, num_mini_batch = len(training_datapipe), epoch_alert_freq = 1 + ) as logger: + for batch in training_datapipe: + # Training loop + ... # training code + logger.log_metrics({"train_loss": training_loss}) + + # Validation logger + with LaunchLogger( + "val", epoch = epoch, num_mini_batch = len(validation_datapipe), epoch_alert_freq = 1 + ) as logger: + for batch in validation_datapipe: + # Validation loop + ... # validation code + logger.log_minibatch({"val_loss": validation_loss}) + + learning_rate = ... # get the learning rate at the end of the epoch from the optimizer + logger.log_epoch({"learning_rate": learning_rate}) # log the learning rate at the end of the epoch + +This example shows how to use the LaunchLogger to log metrics during training and +validation. The LaunchLogger is initialized with the MLflow backend, and the logger +is created for each epoch, a separate logger is created for training and validation. +We can use the `.log_minibatch` method to log metrics during training and validation. +We can use the `.log_epoch` method to log the learning rate at the end of the epoch. + +For a more detailed example, please refer to the `Logging and Checkpointing recipe <../../user-guide/simple_logging_and_checkpointing.html>`_ . + +.. autosummary:: + :toctree: generated + +Launch Logger +------------- + +The LaunchLogger serves as the primary interface for logging in PhysicsNeMo. It provides a unified API that works +consistently across different logging backends and training environments. The logger automatically handles metric +aggregation in distributed settings and ensures proper synchronization across processes. + +.. automodule:: physicsnemo.launch.logging.launch + :members: + :show-inheritance: + +Console Logger +-------------- + +A simple but powerful console-based logger that provides formatted output to the terminal. It's particularly useful +during development and debugging, offering clear visibility into training progress and metrics. + +.. automodule:: physicsnemo.launch.logging.console + :members: + :show-inheritance: + +MLflow Logger +------------- + +Integration with MLflow for experiment tracking and model management. This utility enables systematic tracking of +experiments, including metrics, parameters, artifacts, and model versions. It's particularly useful for teams +that need to maintain reproducibility and compare different experiments. Users should initialize the MLflow backend +before using the LaunchLogger. + +.. automodule:: physicsnemo.launch.logging.mlflow + :members: + :show-inheritance: + +Example usage: + +.. code:: python + + from physicsnemo.launch.logging.mlflow import initialize_mlflow + from physicsnemo.launch.logging import LaunchLogger + + # Initialize MLflow + initialize_mlflow( + experiment_name="weather_prediction", + user_name="physicsnemo_user", + mode="offline", + ) + + # Create logger with MLflow backend + logger = LaunchLogger.initialize(use_mlflow=True) + +Weights and Biases Logger +------------------------- + +Integration with Weights & Biases (W&B) for experiment tracking and visualization. This utility provides rich +visualization capabilities and easy experiment comparison, making it ideal for projects that require detailed +analysis of training runs and model performance. Users should initialize the W&B backend before using the LaunchLogger. + +.. automodule:: physicsnemo.launch.logging.wandb + :members: + :show-inheritance: + +Example usage: + +.. code:: python + + from physicsnemo.launch.logging.wandb import initialize_wandb + from physicsnemo.launch.logging import LaunchLogger + + # Initialize W&B + initialize_wandb( + project="physics_simulation", + entity="my_team" + ) + + # Create logger with W&B backend + logger = LaunchLogger.initialize(use_wandb=True) + +Logging utils +------------- + +Utility functions and helpers for logging operations. + +.. automodule:: physicsnemo.launch.logging.utils + :members: + :show-inheritance: + diff --git a/docs/api/physicsnemo.launch.utils.rst b/docs/api/physicsnemo.launch.utils.rst new file mode 100644 index 0000000000..c3531b2e49 --- /dev/null +++ b/docs/api/physicsnemo.launch.utils.rst @@ -0,0 +1,20 @@ +PhysicsNeMo Launch Utils +========================= + +The PhysicsNeMo Launch Utils module utilities that support the saving and loading +of model checkpoints. These utilities are used internally by the LaunchLogger, but +can also be used by users to save and load model checkpoints. + +.. automodule:: physicsnemo.launch.utils +.. currentmodule:: physicsnemo.launch.utils + +.. autosummary:: + :toctree: generated + +Checkpointing +------------- + +.. automodule:: physicsnemo.launch.utils.checkpoint + :members: + :show-inheritance: + diff --git a/docs/api/physicsnemo.metrics.rst b/docs/api/physicsnemo.metrics.rst new file mode 100644 index 0000000000..9cc681cd57 --- /dev/null +++ b/docs/api/physicsnemo.metrics.rst @@ -0,0 +1,277 @@ +PhysicsNeMo Metrics +==================== + +.. automodule:: physicsnemo.metrics +.. currentmodule:: physicsnemo.metrics + +Basics +------- + +PhysicsNeMo provides several general and domain-specific metric calculations you can +leverage in your custom training and inference workflows. These metrics are optimized to +operate on PyTorch tensors. + +General Metrics and Statistical Methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Below is a summary of general purpose statistical methods and metrics that are available: + +.. list-table:: + :widths: 20 80 + :header-rows: 1 + + * - Metric + - Description + * - `physicsnemo.metrics.general.mse.mse <#physicsnemo.metrics.general.mse.mse>`_ + - Mean Squared error between two tensors + * - `physicsnemo.metrics.general.mse.rmse <#physicsnemo.metrics.general.mse.rmse>`_ + - Root Mean Squared error between two tensors + * - `physicsnemo.metrics.general.histogram.histogram <#physicsnemo.metrics.general.histogram.histogram>`_ + - Histogram of a set of tensors over the leading dimension + * - `physicsnemo.metrics.general.histogram.cdf <#physicsnemo.metrics.general.histogram.cdf>`_ + - Cumulative density function of a set of tensors over the leading dimension + * - `physicsnemo.metrics.general.histogram.normal_cdf <#physicsnemo.metrics.general.histogram.normal_cdf>`_ + - Cumulative density function of a normal variable with given mean and standard deviation + * - `physicsnemo.metrics.general.histogram.normal_pdf <#physicsnemo.metrics.general.histogram.normal_pdf>`_ + - Probability density function of a normal variable with given mean and standard deviation + * - `physicsnemo.metrics.general.calibration.find_rank <#physicsnemo.metrics.general.calibration.find_rank>`_ + - Find the rank of the observation with respect to the given counts and bins + * - `physicsnemo.metrics.general.calibration.rank_probability_score <#physicsnemo.metrics.general.calibration.rank_probability_score>`_ + - Rank Probability Score for the passed ranks + * - `physicsnemo.metrics.general.entropy.entropy_from_counts <#physicsnemo.metrics.general.entropy.entropy_from_counts>`_ + - Computes the statistical entropy of a random variable using a histogram. + * - `physicsnemo.metrics.general.entropy.relative_entropy_from_counts <#physicsnemo.metrics.general.entropy.relative_entropy_from_counts>`_ + - Computes the relative statistical entropy, or KL Divergence of two random variables using their histograms. + * - `physicsnemo.metrics.general.crps.crps <#physicsnemo.metrics.general.crps.crps>`_ + - Local Continuous Ranked Probability Score (CRPS) by computing a histogram and CDF of the predictions + * - `physicsnemo.metrics.general.wasserstein.wasserstein_from_cdf <#physicsnemo.metrics.general.wasserstein.wasserstein_from_cdf>`_ + - Wasserstein distance between two discrete CDF functions + * - `physicsnemo.metrics.general.reduction.WeightedMean <#physicsnemo.metrics.general.reduction.WeightedMean>`_ + - Weighted Mean + * - `physicsnemo.metrics.general.reduction.WeightedStatistic <#physicsnemo.metrics.general.reduction.WeightedStatistic>`_ + - Weighted Statistic + * - `physicsnemo.metrics.general.reduction.WeightedVariance <#physicsnemo.metrics.general.reduction.WeightedVariance>`_ + - Weighted Variance + +Below shows some examples of how to use these metrics in your own workflows. + + +To compute RMSE metric: + +.. code:: python + + >>> import torch + >>> from physicsnemo.metrics.general.mse import rmse + >>> pred_tensor = torch.randn(16, 32) + >>> targ_tensor = torch.randn(16, 32) + >>> rmse(pred_tensor, targ_tensor) + tensor(1.4781) + + +To compute the histogram of samples: + +.. code:: python + + >>> import torch + >>> from physicsnemo.metrics.general import histogram + >>> x = torch.randn(1_000) + >>> bins, counts = histogram.histogram(x, bins = 10) + >>> bins + tensor([-3.7709, -3.0633, -2.3556, -1.6479, -0.9403, -0.2326, 0.4751, 1.1827, + 1.8904, 2.5980, 3.3057]) + >>> counts + tensor([ 3, 9, 43, 150, 227, 254, 206, 81, 24, 3]) + + +To use compute the continuous density function (CDF): + +.. code:: python + + >>> bins, cdf = histogram.cdf(x, bins = 10) + >>> bins + tensor([-3.7709, -3.0633, -2.3556, -1.6479, -0.9403, -0.2326, 0.4751, 1.1827, + 1.8904, 2.5980, 3.3057]) + >>> cdf + tensor([0.0030, 0.0120, 0.0550, 0.2050, 0.4320, 0.6860, 0.8920, 0.9730, 0.9970, + 1.0000]) + +To use the histogram for statistical entropy calculations: + +.. code:: python + + >> from physicsnemo.metrics.general import entropy + >>> entropy.entropy_from_counts(counts, bins) + tensor(0.4146) + +Many of the functions operate over batches. For example, if one has a collection of two dimensional +data, then we can compute the histogram over the collection: + +.. code:: python + + >>> import torch + >>> from physicsnemo.metrics.general import histogram, entropy + >>> x = torch.randn((1_000, 3, 3)) + >>> bins, counts = histogram.histogram(x, bins = 10) + >>> bins.shape, counts.shape + (torch.Size([11, 3, 3]), torch.Size([10, 3, 3])) + >>> entropy.entropy_from_counts(counts, bins) + tensor([[0.5162, 0.4821, 0.3976], + [0.5099, 0.5309, 0.4519], + [0.4580, 0.4290, 0.5121]]) + +There are additional metrics to compute differences between distributions: Ranks, Continuous Rank +Probability Skill, and Wasserstein metric. + +CRPS: + +.. code:: python + + >>> from physicsnemo.metrics.general import crps + >>> x = torch.randn((1_000,1)) + >>> y = torch.randn((1,)) + >>> crps.crps(x, y) + tensor([0.8023]) + +Ranks: + +.. code:: python + + >>> from physicsnemo.metrics.general import histogram, calibration + >>> x = torch.randn((1_000,1)) + >>> y = torch.randn((1,)) + >>> bins, counts = histogram.histogram(x, bins = 10) + >>> ranks = calibration.find_rank(bins, counts, y) + tensor([0.1920]) + +Wasserstein Metric: + +.. code:: python + + >>> from physicsnemo.metrics.general import wasserstein, histogram + >>> x = torch.randn((1_000,1)) + >>> y = torch.randn((1_000,1)) + >>> bins, cdf_x = histogram.cdf(x) + >>> bins, cdf_y = histogram.cdf(y, bins = bins) + >>> wasserstein(bins, cdf_x, cdf_y) + >>> wasserstein.wasserstein(bins, cdf_x, cdf_y) + tensor([0.0459]) + + +Weighted Reductions +^^^^^^^^^^^^^^^^^^^ +PhysicsNeMo currently offers classes for weighted mean and variance reductions. + +.. code:: python + + >>> from physicsnemo.metrics.general import reduction + >>> x = torch.randn((1_000,)) + >>> weights = torch.cos(torch.linspace(-torch.pi/4, torch.pi/4, 1_000)) + >>> wm = reduction.WeightedMean(weights) + >>> wm(x, dim = 0) + tensor(0.0365) + >>> wv = reduction.WeightedVariance(weights) + >>> wv(x, dim = 0) + tensor(1.0148) + + +Online Statistical Methods +^^^^^^^^^^^^^^^^^^^^^^^^^^ +PhysicsNeMo current offers routines for computing online, or out-of-memory, means, +variances, and histograms. + +.. code:: python + + >>> import torch + >>> from physicsnemo.metrics.general import ensemble_metrics as em + >>> x = torch.randn((1_000, 2)) # Interpret as 1_000 members of size (2,). + >>> torch.mean(x, dim = 0) # Compute mean of entire data. + tensor([-0.0545, 0.0267]) + >>> x0, x1 = x[:500], x[500:] # Split data into two. + >>> M = em.Mean(input_shape = (2,)) # Must pass shape of data + >>> M(x0) # Compute mean of initial batch. + tensor([-0.0722, 0.0414]) + >>> M.update(x1) # Update with second batch. + tensor([-0.0545, 0.0267]) + + +Climate Related Metrics +^^^^^^^^^^^^^^^^^^^^^^^ + +To compute the Anomaly Correlation Coefficient, a metric widely used in weather and +climate sciences: + +.. code:: python + + >>> import torch + >>> import numpy as np + >>> from physicsnemo.metrics.climate.acc import acc + >>> channels = 1 + >>> img_shape = (32, 64) + >>> time_means = np.pi / 2 * np.ones((channels, img_shape[0], img_shape[1]), dtype=np.float32) + >>> x = np.linspace(-180, 180, img_shape[1], dtype=np.float32) + >>> y = np.linspace(-90, 90, img_shape[0], dtype=np.float32) + >>> xv, yv = np.meshgrid(x, y) + >>> pred_tensor_np = np.cos(2 * np.pi * yv / (180)) + >>> targ_tensor_np = np.cos(np.pi * yv / (180)) + >>> pred_tensor = torch.from_numpy(pred_tensor_np).expand(channels, -1, -1) + >>> targ_tensor = torch.from_numpy(targ_tensor_np).expand(channels, -1, -1) + >>> means_tensor = torch.from_numpy(time_means) + >>> lat = torch.from_numpy(y) + >>> acc(pred_tensor, targ_tensor, means_tensor, lat) + tensor([0.9841]) + + +.. autosummary:: + :toctree: generated + +General +--------- + +.. automodule:: physicsnemo.metrics.general.mse + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.general.histogram + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.general.entropy + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.general.calibration + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.general.crps + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.general.ensemble_metrics + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.general.reduction + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.general.wasserstein + :members: + :show-inheritance: + +Weather and climate metrics +--------------------------- + +.. automodule:: physicsnemo.metrics.climate.acc + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.climate.efi + :members: + :show-inheritance: + +.. automodule:: physicsnemo.metrics.climate.reduction + :members: + :show-inheritance: + + diff --git a/docs/api/physicsnemo.nn.functionals.rst b/docs/api/physicsnemo.nn.functionals.rst new file mode 100644 index 0000000000..6608736d45 --- /dev/null +++ b/docs/api/physicsnemo.nn.functionals.rst @@ -0,0 +1,25 @@ +PhysicsNeMo Functionals +======================= + +PhysicsNeMo functionals follow the ``torch.nn.functional`` pattern: stateless +operations designed for direct use in model code, training loops, and +pre/post-processing pipelines. They are intended to be easy to compose and to +behave consistently across CPU and GPU execution paths. + +Many functionals are optimized for NVIDIA GPUs and can dispatch to accelerated +implementations when those backends are installed. For operations with multiple +implementations, PhysicsNeMo selects a preferred implementation by default and +falls back to another supported one when needed, emitting a warning so behavior +is explicit. Functionals with multiple implementations have plots available +in the documentation for performance comparisons. + +.. toctree:: + :maxdepth: 2 + :caption: PhysicsNeMo Functionals + :name: PhysicsNeMo Functionals + + nn/functionals/neighbors + nn/functionals/geometry + nn/functionals/fourier_spectral + nn/functionals/regularization_parameterization + nn/functionals/resampling_interpolation diff --git a/docs/api/physicsnemo.nn.layers.rst b/docs/api/physicsnemo.nn.layers.rst new file mode 100644 index 0000000000..977155f7b0 --- /dev/null +++ b/docs/api/physicsnemo.nn.layers.rst @@ -0,0 +1,19 @@ +PhysicsNeMo Layers +================== + +.. toctree:: + :maxdepth: 2 + :caption: PhysicsNeMo Layers + :name: PhysicsNeMo Layers + + nn/layers/activations + nn/layers/attention_transformers + nn/layers/convolutional + nn/layers/embeddings + nn/layers/fourier_spectral + nn/layers/fully_connected_mlp + nn/layers/normalization + nn/layers/resampling_interpolation + nn/layers/regularization + nn/layers/specialized + nn/layers/graph_geometry diff --git a/docs/api/physicsnemo.optim.rst b/docs/api/physicsnemo.optim.rst new file mode 100644 index 0000000000..86fe5f820f --- /dev/null +++ b/docs/api/physicsnemo.optim.rst @@ -0,0 +1,23 @@ +PhysicsNeMo Optim +================= + +.. automodule:: physicsnemo.optim +.. currentmodule:: physicsnemo.optim + +The PhysicsNeMo Optim module provides optimization utilities for training physics-informed +machine learning models. These utilities are designed to work seamlessly with PyTorch's +optimizer ecosystem while providing additional functionality for complex training scenarios. + +CombinedOptimizer +----------------- + +The :class:`CombinedOptimizer` allows combining multiple PyTorch optimizers into a unified +interface. This is particularly useful when different parts of a model require different +optimization strategies - for example, using Adam for encoder layers and SGD with momentum +for decoder layers. + +.. autoclass:: physicsnemo.optim.CombinedOptimizer + :members: + :show-inheritance: + :special-members: __init__ + diff --git a/docs/api/physicsnemo.utils.rst b/docs/api/physicsnemo.utils.rst new file mode 100644 index 0000000000..f8caed6377 --- /dev/null +++ b/docs/api/physicsnemo.utils.rst @@ -0,0 +1,137 @@ +PhysicsNeMo Utils +================== + +.. automodule:: physicsnemo.utils +.. currentmodule:: physicsnemo.utils + +The PhysicsNeMo Utils module provides a comprehensive set of utilities that support various aspects of scientific computing, +machine learning, and physics simulations. These utilities range from optimization helpers and distributed computing tools +to specialized functions for weather and climate modeling, and geometry processing. The module is designed to simplify common +tasks while maintaining high performance and scalability. + +.. autosummary:: + :toctree: generated + +Optimization Utils +------------------ + +The optimization utilities provide tools for capturing and managing training states, gradients, and optimization processes. +These are particularly useful when implementing custom training loops or specialized optimization strategies. + +.. automodule:: physicsnemo.utils.capture + :members: + :show-inheritance: + + +GraphCast Utils +--------------- + +A collection of utilities specifically designed for working with the GraphCast model, including data processing, +graph construction, and specialized loss functions. These utilities are essential for implementing and +training GraphCast-based weather prediction models. + +.. automodule:: physicsnemo.utils.graphcast.data_utils + :members: + :show-inheritance: + +.. automodule:: physicsnemo.utils.graphcast.graph + :members: + :show-inheritance: + +.. automodule:: physicsnemo.utils.graphcast.graph_utils + :members: + :show-inheritance: + +.. automodule:: physicsnemo.utils.graphcast.loss + :members: + :show-inheritance: + +Filesystem Utils +---------------- + +Utilities for handling file operations, caching, and data management across different storage systems. +These utilities abstract away the complexity of dealing with different filesystem types and provide +consistent interfaces for data access. + +.. automodule:: physicsnemo.utils.filesystem + :members: + :show-inheritance: + +.. _diffusion_utils: + +Diffusion Utils +--------------- + +Tools for working with diffusion models and other generative approaches, +including deterministic and stochastic sampling utilities. + +.. automodule:: physicsnemo.diffusion.samplers.deterministic_sampler + :members: + :show-inheritance: + +.. automodule:: physicsnemo.diffusion.samplers.stochastic_sampler + :members: + :show-inheritance: + +.. automodule:: physicsnemo.diffusion.utils + :members: + :show-inheritance: + +Weather / Climate Utils +----------------------- + +Specialized utilities for weather and climate modeling, including calculations for solar radiation +and atmospheric parameters. These utilities are used extensively in weather prediction models. + +.. automodule:: physicsnemo.utils.insolation + :members: + :show-inheritance: + +.. automodule:: physicsnemo.utils.zenith_angle + :show-inheritance: + +.. _patching_utils: + +Patching Utils +-------------- + +Patching utilities are particularly useful for *patch-based* diffusion, also called +*multi-diffusion*. This approach is used to scale diffusion to very large images. +The following patching utilities extract patches from 2D images, and typically gather +them in the batch dimension. A batch of patches is therefore composed of multiple +smaller patches that are extracted from each sample in the original batch of larger +images. Diffusion models can then process these patches independently. These +utilities also support fusing operations to reconstruct the entire predicted +image from the individual predicted patches. + +.. automodule:: physicsnemo.diffusion.multi_diffusion + :members: + :show-inheritance: + +Domino Utils +------------ + +Utilities for working with the Domino model, including data processing and grid construction. +These utilities are essential for implementing and training Domino-based models. + +.. automodule:: physicsnemo.utils.domino.utils + :members: + :show-inheritance: + +CorrDiff Utils +-------------- + +Utilities for working with the CorrDiff model, particularly for the diffusion and regression steps. + +.. automodule:: physicsnemo.diffusion.samplers + :members: + :show-inheritance: + +Profiling Utils +--------------- + +Utilities for profiling the performance of a model. + +.. automodule:: physicsnemo.utils.profiling + :members: + :show-inheritance: diff --git a/docs/api_index.rst b/docs/api_index.rst new file mode 100644 index 0000000000..2aa8d438be --- /dev/null +++ b/docs/api_index.rst @@ -0,0 +1,22 @@ +API Reference +============= + +.. toctree:: + :maxdepth: 2 + :caption: API Reference + :name: API Reference + + api_models + api/physicsnemo.nn.layers.rst + api/physicsnemo.nn.functionals.rst + + api/physicsnemo.datapipes.rst + api/physicsnemo.metrics.rst + api/physicsnemo.deploy.rst + api/physicsnemo.distributed.rst + api/physicsnemo.distributed.shardtensor.rst + api/physicsnemo.optim.rst + api/physicsnemo.utils.rst + api/physicsnemo.launch.logging.rst + api/physicsnemo.launch.utils.rst + api/physicsnemo.active_learning.rst diff --git a/docs/api_models.rst b/docs/api_models.rst new file mode 100644 index 0000000000..261e7aa86b --- /dev/null +++ b/docs/api_models.rst @@ -0,0 +1,20 @@ +PhysicsNeMo Models +=================== + +.. currentmodule:: physicsnemo.models + +.. toctree:: + :maxdepth: 2 + :caption: PhysicsNeMo Models + :name: PhysicsNeMo Models + + api/models/modules.rst + api/models/fully_connected.rst + api/models/fnos.rst + api/models/gnns.rst + api/models/convolutional.rst + api/models/recurrent.rst + api/models/operators.rst + api/models/diffusion.rst + api/models/diffusion_preconditioners.rst + api/models/weather.rst diff --git a/docs/conf.py b/docs/conf.py index 151e39abd1..224fdfa5da 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -21,13 +23,11 @@ # https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information import os -import sphinx_rtd_theme +from physicsnemo import __version__ as version -from modulus import __version__ as version - -project = "NVIDIA Modulus" -copyright = "2023, NVIDIA Modulus Team" -author = "NVIDIA Modulus Team" +project = "NVIDIA PhysicsNeMo" +copyright = "2023, NVIDIA PhysicsNeMo Team" +author = "NVIDIA PhysicsNeMo Team" release = version # -- General configuration --------------------------------------------------- @@ -51,13 +51,12 @@ ("index", "rst2pdf", "Sample rst2pdf doc", "Your Name"), ] -napoleon_custom_sections = ["Variable Shape"] +napoleon_custom_sections = [("Variable Shape", "notes"), ("Forward", "params_style"), ("Outputs", "returns_style")] # -- Options for HTML output ------------------------------------------------- # HTML theme options -html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] -html_theme = "sphinx_rtd_theme" +html_theme = "nvidia_sphinx_theme" html_theme_options = { "logo_only": True, "display_version": True, @@ -82,7 +81,7 @@ # html_last_updated_fmt = '' # Additional sphinx switches -math_number_all = True +math_number_all = False todo_include_todos = True numfig = True @@ -121,9 +120,12 @@ "LICENSE.txt", ] +# Fake imports +autodoc_mock_imports = ["torch_scatter", "torch_cluster"] # install of these packages takes very long + source_suffix = {".rst": "restructuredtext", ".md": "markdown"} pdf_documents = [ ("index", "rst2pdf", "Sample rst2pdf doc", "Your Name"), ] -napoleon_custom_sections = ["Variable Shape"] +# napoleon_custom_sections = ["Variable Shape"] diff --git a/docs/examples_additive_manufacturing.rst b/docs/examples_additive_manufacturing.rst new file mode 100644 index 0000000000..1c660a43d2 --- /dev/null +++ b/docs/examples_additive_manufacturing.rst @@ -0,0 +1,11 @@ +Additive Manufacturing Examples +=============================== + +Additive manufacturing and materials science examples using PhysicsNeMo. + +.. toctree:: + :maxdepth: 2 + :caption: Examples: Additive Manufacturing + :name: Examples: Additive Manufacturing + + examples/additive_manufacturing/sintering_physics/README.rst \ No newline at end of file diff --git a/docs/examples_cfd.rst b/docs/examples_cfd.rst new file mode 100644 index 0000000000..7fb2e51fdc --- /dev/null +++ b/docs/examples_cfd.rst @@ -0,0 +1,28 @@ +CFD Examples +============ + +Computational Fluid Dynamics (CFD) examples using PhysicsNeMo. + +.. toctree:: + :maxdepth: 2 + :caption: Examples: CFD + :name: Examples: CFD + + examples/cfd/vortex_shedding_mgn/README.rst + examples/cfd/external_aerodynamics/aero_graph_net/README.rst + examples/cfd/external_aerodynamics/domino/README.rst + examples/cfd/external_aerodynamics/figconvnet/README.rst + examples/cfd/external_aerodynamics/xaeronet/README.rst + examples/cfd/navier_stokes_rnn/README.rst + examples/cfd/gray_scott_rnn/README.rst + examples/cfd/lagrangian_mgn/README.rst + examples/cfd/darcy_nested_fnos/README.rst + examples/cfd/darcy_physics_informed/README.rst + examples/cfd/stokes_mgn/README.rst + examples/cfd/lid_driven_cavity/README.rst + examples/cfd/swe_distributed_gnn/README.rst + examples/cfd/vortex_shedding_mesh_reduced/README.rst + examples/cfd/darcy_transolver/README.rst + examples/cfd/flow_reconstruction_diffusion/README.rst + examples/cfd/datacenter/README.rst + examples/cfd/external_aerodynamics/domino_nim_finetuning/README.rst diff --git a/docs/examples_geophysics.rst b/docs/examples_geophysics.rst new file mode 100644 index 0000000000..6ae0a9f8ac --- /dev/null +++ b/docs/examples_geophysics.rst @@ -0,0 +1,11 @@ +Geophysics Examples +=================== + +Geophysics applications using PhysicsNeMo. + +.. toctree:: + :maxdepth: 2 + :caption: Examples: Geophysics + :name: Examples: Geophysics + + examples/geophysics/diffusion_fwi/README.rst \ No newline at end of file diff --git a/docs/examples_healthcare.rst b/docs/examples_healthcare.rst new file mode 100644 index 0000000000..fae0035ca7 --- /dev/null +++ b/docs/examples_healthcare.rst @@ -0,0 +1,12 @@ +Healthcare Examples +=================== + +Healthcare and medical applications using PhysicsNeMo. + +.. toctree:: + :maxdepth: 2 + :caption: Examples: Healthcare + :name: Examples: Healthcare + + examples/healthcare/bloodflow_1d_mgn/README.rst + examples/healthcare/brain_anomaly_detection/README.rst \ No newline at end of file diff --git a/docs/examples_index.rst b/docs/examples_index.rst new file mode 100644 index 0000000000..ec10f44d07 --- /dev/null +++ b/docs/examples_index.rst @@ -0,0 +1,17 @@ +Examples +======== + +PhysicsNeMo examples organized by application domain. + +.. toctree:: + :maxdepth: 2 + :caption: Contents + :name: Contents + + examples_introductory + examples_cfd + examples_weather + examples_healthcare + examples_additive_manufacturing + examples_molecular_dynamics + examples_geophysics \ No newline at end of file diff --git a/docs/examples_introductory.rst b/docs/examples_introductory.rst new file mode 100644 index 0000000000..3e4fa277a5 --- /dev/null +++ b/docs/examples_introductory.rst @@ -0,0 +1,17 @@ +Introductory Examples +===================== + +These examples are designed to help you learn the key concepts of PhysicsNeMo. + +.. toctree:: + :maxdepth: 2 + :caption: Introductory Examples + :name: Introductory Examples + + examples/cfd/darcy_fno/README.rst + examples/cfd/darcy_physics_informed/README.rst + examples/cfd/ldc_pinns/README.rst + examples/cfd/vortex_shedding_mgn/README.rst + examples/weather/fcn_afno/README.rst + examples/cfd/lagrangian_mgn/README.rst + examples/cfd/stokes_mgn/README.rst \ No newline at end of file diff --git a/docs/examples_molecular_dynamics.rst b/docs/examples_molecular_dynamics.rst new file mode 100644 index 0000000000..b0c51e4d5a --- /dev/null +++ b/docs/examples_molecular_dynamics.rst @@ -0,0 +1,11 @@ +Molecular Dynamics Examples +=========================== + +Molecular dynamics and computational chemistry examples using PhysicsNeMo. + +.. toctree:: + :maxdepth: 2 + :caption: Examples: Molecular Dynamics + :name: Examples: Molecular Dynamics + + examples/molecular_dynamics/lennard_jones/README.rst \ No newline at end of file diff --git a/docs/examples_weather.rst b/docs/examples_weather.rst new file mode 100644 index 0000000000..1bd2c81ddf --- /dev/null +++ b/docs/examples_weather.rst @@ -0,0 +1,20 @@ +Weather and Climate Examples +============================ + +Weather and climate modeling examples using PhysicsNeMo. + +.. toctree:: + :maxdepth: 2 + :caption: Examples: Weather and Climate + :name: Examples: Weather and Climate + + examples/weather/dataset_download/README.rst + examples/weather/graphcast/README.rst + examples/weather/fcn_afno/README.rst + examples/weather/dlwp/README.rst + examples/weather/dlwp_healpix/README.rst + examples/weather/diagnostic/README.rst + examples/weather/unified_recipe/README.rst + examples/weather/corrdiff/README.rst + examples/weather/stormcast/README.rst + examples/weather/temporal_interpolation/README.rst \ No newline at end of file diff --git a/docs/img/MHD_0_0.png b/docs/img/MHD_0_0.png new file mode 100644 index 0000000000..af7d46c524 Binary files /dev/null and b/docs/img/MHD_0_0.png differ diff --git a/docs/img/SWE_0.png b/docs/img/SWE_0.png new file mode 100644 index 0000000000..8916f52d6c Binary files /dev/null and b/docs/img/SWE_0.png differ diff --git a/docs/img/corrdiff_cold_front.png b/docs/img/corrdiff_cold_front.png new file mode 100644 index 0000000000..d33c8cd383 Binary files /dev/null and b/docs/img/corrdiff_cold_front.png differ diff --git a/docs/img/corrdiff_demo.gif b/docs/img/corrdiff_demo.gif deleted file mode 100644 index 2b31e54b72..0000000000 Binary files a/docs/img/corrdiff_demo.gif and /dev/null differ diff --git a/docs/img/corrdiff_training_loss.png b/docs/img/corrdiff_training_loss.png new file mode 100644 index 0000000000..d6e9659e3a Binary files /dev/null and b/docs/img/corrdiff_training_loss.png differ diff --git a/docs/img/crash/crash_case4_reduced.gif b/docs/img/crash/crash_case4_reduced.gif new file mode 100644 index 0000000000..95187bb882 Binary files /dev/null and b/docs/img/crash/crash_case4_reduced.gif differ diff --git a/docs/img/crash/crushcan.gif b/docs/img/crash/crushcan.gif new file mode 100644 index 0000000000..f7eeabaeff Binary files /dev/null and b/docs/img/crash/crushcan.gif differ diff --git a/docs/img/datacenter_design_cfd.gif b/docs/img/datacenter_design_cfd.gif new file mode 100644 index 0000000000..d9bf89bfaf Binary files /dev/null and b/docs/img/datacenter_design_cfd.gif differ diff --git a/docs/img/datacenter_hybrid_training.png b/docs/img/datacenter_hybrid_training.png new file mode 100644 index 0000000000..e990fff8a4 Binary files /dev/null and b/docs/img/datacenter_hybrid_training.png differ diff --git a/docs/img/deforming_plate.gif b/docs/img/deforming_plate.gif new file mode 100644 index 0000000000..6a1d2d7217 Binary files /dev/null and b/docs/img/deforming_plate.gif differ diff --git a/docs/img/diffusion_fwi_intro.png b/docs/img/diffusion_fwi_intro.png new file mode 100644 index 0000000000..f2dff02dec Binary files /dev/null and b/docs/img/diffusion_fwi_intro.png differ diff --git a/docs/img/diffusion_fwi_pi_predictions.png b/docs/img/diffusion_fwi_pi_predictions.png new file mode 100644 index 0000000000..08b2b865e5 Binary files /dev/null and b/docs/img/diffusion_fwi_pi_predictions.png differ diff --git a/docs/img/diffusion_fwi_predictions.png b/docs/img/diffusion_fwi_predictions.png new file mode 100644 index 0000000000..a5846813c9 Binary files /dev/null and b/docs/img/diffusion_fwi_predictions.png differ diff --git a/docs/img/diffusion_fwi_variance.png b/docs/img/diffusion_fwi_variance.png new file mode 100644 index 0000000000..7f323da565 Binary files /dev/null and b/docs/img/diffusion_fwi_variance.png differ diff --git a/docs/img/domain_parallelism/inference_latency_vs_sequence_length_8_heads_256_dim.png b/docs/img/domain_parallelism/inference_latency_vs_sequence_length_8_heads_256_dim.png new file mode 100644 index 0000000000..f73c9bc95c Binary files /dev/null and b/docs/img/domain_parallelism/inference_latency_vs_sequence_length_8_heads_256_dim.png differ diff --git a/docs/img/domain_parallelism/training_latency_vs_sequence_length_8_heads_256_dim_backward.png b/docs/img/domain_parallelism/training_latency_vs_sequence_length_8_heads_256_dim_backward.png new file mode 100644 index 0000000000..e710e15326 Binary files /dev/null and b/docs/img/domain_parallelism/training_latency_vs_sequence_length_8_heads_256_dim_backward.png differ diff --git a/docs/img/domino/combined-training-curve.png b/docs/img/domino/combined-training-curve.png new file mode 100644 index 0000000000..9a56f9d76d Binary files /dev/null and b/docs/img/domino/combined-training-curve.png differ diff --git a/docs/img/domino/drag-r2.jpg b/docs/img/domino/drag-r2.jpg new file mode 100644 index 0000000000..1411bd0277 Binary files /dev/null and b/docs/img/domino/drag-r2.jpg differ diff --git a/docs/img/domino/lift-r2.jpg b/docs/img/domino/lift-r2.jpg new file mode 100644 index 0000000000..de7813af24 Binary files /dev/null and b/docs/img/domino/lift-r2.jpg differ diff --git a/docs/img/domino/surface-training-curve.png b/docs/img/domino/surface-training-curve.png new file mode 100644 index 0000000000..992acbd424 Binary files /dev/null and b/docs/img/domino/surface-training-curve.png differ diff --git a/docs/img/domino_perf.png b/docs/img/domino_perf.png new file mode 100644 index 0000000000..0038267354 Binary files /dev/null and b/docs/img/domino_perf.png differ diff --git a/docs/img/domino_result_rtwt.jpg b/docs/img/domino_result_rtwt.jpg new file mode 100644 index 0000000000..4221acb8d0 Binary files /dev/null and b/docs/img/domino_result_rtwt.jpg differ diff --git a/docs/img/drivaernet_results.png b/docs/img/drivaernet_results.png new file mode 100644 index 0000000000..040b7894e5 Binary files /dev/null and b/docs/img/drivaernet_results.png differ diff --git a/docs/img/graphcast_architecture.png b/docs/img/graphcast_architecture.png new file mode 100644 index 0000000000..bfa79a9df0 Binary files /dev/null and b/docs/img/graphcast_architecture.png differ diff --git a/docs/img/hydrographnet.gif b/docs/img/hydrographnet.gif new file mode 100644 index 0000000000..53d8b0cb9f Binary files /dev/null and b/docs/img/hydrographnet.gif differ diff --git a/docs/img/lagrangian_meshgraphnet_compressed.gif b/docs/img/lagrangian_meshgraphnet_compressed.gif new file mode 100644 index 0000000000..f28febb277 Binary files /dev/null and b/docs/img/lagrangian_meshgraphnet_compressed.gif differ diff --git a/docs/img/lagrangian_meshgraphnet_multi.png b/docs/img/lagrangian_meshgraphnet_multi.png new file mode 100644 index 0000000000..42c2980268 Binary files /dev/null and b/docs/img/lagrangian_meshgraphnet_multi.png differ diff --git a/docs/img/lj_system_modulus_results.png b/docs/img/lj_system_physicsnemo_results.png similarity index 100% rename from docs/img/lj_system_modulus_results.png rename to docs/img/lj_system_physicsnemo_results.png diff --git a/docs/img/moe_scores.png b/docs/img/moe_scores.png new file mode 100644 index 0000000000..ba731f75bd Binary files /dev/null and b/docs/img/moe_scores.png differ diff --git a/docs/img/profiling/FA-fp16.png b/docs/img/profiling/FA-fp16.png new file mode 100644 index 0000000000..66db34a1da Binary files /dev/null and b/docs/img/profiling/FA-fp16.png differ diff --git a/docs/img/profiling/FA-fp32.png b/docs/img/profiling/FA-fp32.png new file mode 100644 index 0000000000..233f7c0e85 Binary files /dev/null and b/docs/img/profiling/FA-fp32.png differ diff --git a/docs/img/profiling/endOfSync.png b/docs/img/profiling/endOfSync.png new file mode 100644 index 0000000000..93673286e2 Binary files /dev/null and b/docs/img/profiling/endOfSync.png differ diff --git a/docs/img/profiling/nsys-application-zoom.png b/docs/img/profiling/nsys-application-zoom.png new file mode 100644 index 0000000000..a93780574c Binary files /dev/null and b/docs/img/profiling/nsys-application-zoom.png differ diff --git a/docs/img/profiling/nsys-overview.png b/docs/img/profiling/nsys-overview.png new file mode 100644 index 0000000000..f55bd477fa Binary files /dev/null and b/docs/img/profiling/nsys-overview.png differ diff --git a/docs/img/profiling/nsys-triton-kernel.png b/docs/img/profiling/nsys-triton-kernel.png new file mode 100644 index 0000000000..84efeb833f Binary files /dev/null and b/docs/img/profiling/nsys-triton-kernel.png differ diff --git a/docs/img/profiling/softmax_bwd.png b/docs/img/profiling/softmax_bwd.png new file mode 100644 index 0000000000..93db556e43 Binary files /dev/null and b/docs/img/profiling/softmax_bwd.png differ diff --git a/docs/img/profiling/softmax_fwd.png b/docs/img/profiling/softmax_fwd.png new file mode 100644 index 0000000000..36ad2c195c Binary files /dev/null and b/docs/img/profiling/softmax_fwd.png differ diff --git a/docs/img/profiling/whole-application.png b/docs/img/profiling/whole-application.png new file mode 100644 index 0000000000..565dd2ad85 Binary files /dev/null and b/docs/img/profiling/whole-application.png differ diff --git a/docs/img/regen_method.gif b/docs/img/regen_method.gif new file mode 100644 index 0000000000..5b03005ce2 Binary files /dev/null and b/docs/img/regen_method.gif differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_21_PRED.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_21_PRED.png new file mode 100644 index 0000000000..905bf620e6 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_21_PRED.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_21_TRUE.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_21_TRUE.png new file mode 100644 index 0000000000..be4b4ddc0b Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_21_TRUE.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_42_PRED.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_42_PRED.png new file mode 100644 index 0000000000..79cb3aa86b Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_42_PRED.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_42_TRUE.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_42_TRUE.png new file mode 100644 index 0000000000..a8aa3440b7 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_42_TRUE.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_DIFF_21.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_DIFF_21.png new file mode 100644 index 0000000000..78d701de08 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_DIFF_21.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_DIFF_42.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_DIFF_42.png new file mode 100644 index 0000000000..47b0f1752f Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/PRES_DIFF_42.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_21_PRED.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_21_PRED.png new file mode 100644 index 0000000000..c347124f9b Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_21_PRED.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_21_TRUE.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_21_TRUE.png new file mode 100644 index 0000000000..ad6f5d9178 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_21_TRUE.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_42_PRED.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_42_PRED.png new file mode 100644 index 0000000000..98d6e5b421 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_42_PRED.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_42_TRUE.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_42_TRUE.png new file mode 100644 index 0000000000..b177d4d654 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_42_TRUE.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_DIFF_21.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_DIFF_21.png new file mode 100644 index 0000000000..98007fe25c Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_DIFF_21.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_DIFF_42.png b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_DIFF_42.png new file mode 100644 index 0000000000..db5bbb5061 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/inference/SWAT_DIFF_42.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/static/PARTITION.png b/docs/img/reservoir_simulation/xmgn/Norne/static/PARTITION.png new file mode 100644 index 0000000000..bdf20b7193 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/static/PARTITION.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/static/PERMX.png b/docs/img/reservoir_simulation/xmgn/Norne/static/PERMX.png new file mode 100644 index 0000000000..3314d99778 Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/static/PERMX.png differ diff --git a/docs/img/reservoir_simulation/xmgn/Norne/static/PORO.png b/docs/img/reservoir_simulation/xmgn/Norne/static/PORO.png new file mode 100644 index 0000000000..83d6d6ac7c Binary files /dev/null and b/docs/img/reservoir_simulation/xmgn/Norne/static/PORO.png differ diff --git a/docs/img/stormcast_rollout.gif b/docs/img/stormcast_rollout.gif new file mode 100644 index 0000000000..9a12ca1a09 Binary files /dev/null and b/docs/img/stormcast_rollout.gif differ diff --git a/docs/img/swe_distributed_gnn_figures/dist_message_passing.png b/docs/img/swe_distributed_gnn_figures/dist_message_passing.png new file mode 100644 index 0000000000..b54664fa5f Binary files /dev/null and b/docs/img/swe_distributed_gnn_figures/dist_message_passing.png differ diff --git a/docs/img/swe_distributed_gnn_figures/dist_mlp.png b/docs/img/swe_distributed_gnn_figures/dist_mlp.png new file mode 100644 index 0000000000..44838fbe48 Binary files /dev/null and b/docs/img/swe_distributed_gnn_figures/dist_mlp.png differ diff --git a/docs/img/swe_distributed_gnn_figures/global_vs_local_graph.png b/docs/img/swe_distributed_gnn_figures/global_vs_local_graph.png new file mode 100644 index 0000000000..a07d4d5bba Binary files /dev/null and b/docs/img/swe_distributed_gnn_figures/global_vs_local_graph.png differ diff --git a/docs/img/swe_distributed_gnn_figures/memory_scaling_dim128.png b/docs/img/swe_distributed_gnn_figures/memory_scaling_dim128.png new file mode 100644 index 0000000000..06400955c8 Binary files /dev/null and b/docs/img/swe_distributed_gnn_figures/memory_scaling_dim128.png differ diff --git a/docs/img/swe_distributed_gnn_figures/val_loss_latlon.png b/docs/img/swe_distributed_gnn_figures/val_loss_latlon.png new file mode 100644 index 0000000000..5979815618 Binary files /dev/null and b/docs/img/swe_distributed_gnn_figures/val_loss_latlon.png differ diff --git a/docs/img/tank_filling.gif b/docs/img/tank_filling.gif new file mode 100644 index 0000000000..88438c73e9 Binary files /dev/null and b/docs/img/tank_filling.gif differ diff --git a/docs/img/tank_filling_velocity.gif b/docs/img/tank_filling_velocity.gif new file mode 100644 index 0000000000..9154c9ed05 Binary files /dev/null and b/docs/img/tank_filling_velocity.gif differ diff --git a/docs/img/tank_sim.png b/docs/img/tank_sim.png new file mode 100644 index 0000000000..fa7863d6b9 Binary files /dev/null and b/docs/img/tank_sim.png differ diff --git a/docs/img/temporal_interpolation.gif b/docs/img/temporal_interpolation.gif new file mode 100644 index 0000000000..7df5db2176 Binary files /dev/null and b/docs/img/temporal_interpolation.gif differ diff --git a/docs/img/topodiff_doc/grid_topology.png b/docs/img/topodiff_doc/grid_topology.png new file mode 100644 index 0000000000..3fc9909d9d Binary files /dev/null and b/docs/img/topodiff_doc/grid_topology.png differ diff --git a/docs/img/topodiff_doc/topodiff.png b/docs/img/topodiff_doc/topodiff.png new file mode 100644 index 0000000000..9b8ef23240 Binary files /dev/null and b/docs/img/topodiff_doc/topodiff.png differ diff --git a/docs/img/topodiff_doc/topology_generated.png b/docs/img/topodiff_doc/topology_generated.png new file mode 100644 index 0000000000..88bc975f21 Binary files /dev/null and b/docs/img/topodiff_doc/topology_generated.png differ diff --git a/docs/img/transolver.png b/docs/img/transolver.png new file mode 100644 index 0000000000..07e31966e3 Binary files /dev/null and b/docs/img/transolver.png differ diff --git a/docs/img/value_prop/Knowledge_guided_models.gif b/docs/img/value_prop/Knowledge_guided_models.gif new file mode 100644 index 0000000000..406c95daea Binary files /dev/null and b/docs/img/value_prop/Knowledge_guided_models.gif differ diff --git a/docs/img/value_prop/Knowledge_guided_models.png b/docs/img/value_prop/Knowledge_guided_models.png new file mode 100644 index 0000000000..6170285b2d Binary files /dev/null and b/docs/img/value_prop/Knowledge_guided_models.png differ diff --git a/docs/img/value_prop/benchmarking.svg b/docs/img/value_prop/benchmarking.svg new file mode 100644 index 0000000000..b314cfe1d4 --- /dev/null +++ b/docs/img/value_prop/benchmarking.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/img/value_prop/performance.svg b/docs/img/value_prop/performance.svg new file mode 100644 index 0000000000..09c485c54e --- /dev/null +++ b/docs/img/value_prop/performance.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/img/value_prop/recipe.svg b/docs/img/value_prop/recipe.svg new file mode 100644 index 0000000000..277ab9bc5b --- /dev/null +++ b/docs/img/value_prop/recipe.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/img/vfgn_doc/4-parts-final.png b/docs/img/vfgn_doc/4-parts-final.png new file mode 100644 index 0000000000..a411b52db4 Binary files /dev/null and b/docs/img/vfgn_doc/4-parts-final.png differ diff --git a/docs/img/vfgn_doc/HP-MetalJet-process.png b/docs/img/vfgn_doc/HP-MetalJet-process.png new file mode 100644 index 0000000000..7747106a87 Binary files /dev/null and b/docs/img/vfgn_doc/HP-MetalJet-process.png differ diff --git a/docs/img/vfgn_doc/busbar.gif b/docs/img/vfgn_doc/busbar.gif new file mode 100644 index 0000000000..1ccfd8b571 Binary files /dev/null and b/docs/img/vfgn_doc/busbar.gif differ diff --git a/docs/img/vfgn_doc/pushing-grip.gif b/docs/img/vfgn_doc/pushing-grip.gif new file mode 100644 index 0000000000..4f96f8d3fa Binary files /dev/null and b/docs/img/vfgn_doc/pushing-grip.gif differ diff --git a/docs/img/vfgn_doc/screw.gif b/docs/img/vfgn_doc/screw.gif new file mode 100644 index 0000000000..8024c6910b Binary files /dev/null and b/docs/img/vfgn_doc/screw.gif differ diff --git a/docs/img/vfgn_doc/usb.gif b/docs/img/vfgn_doc/usb.gif new file mode 100644 index 0000000000..05378540ca Binary files /dev/null and b/docs/img/vfgn_doc/usb.gif differ diff --git a/docs/img/xaeronet_s_results.png b/docs/img/xaeronet_s_results.png new file mode 100644 index 0000000000..0af1fbb2d5 Binary files /dev/null and b/docs/img/xaeronet_s_results.png differ diff --git a/docs/img/xaeronet_v_results.png b/docs/img/xaeronet_v_results.png new file mode 100644 index 0000000000..d4c4ee38f2 Binary files /dev/null and b/docs/img/xaeronet_v_results.png differ diff --git a/docs/index.rst b/docs/index.rst index d10c37e07b..5261c57eaf 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,70 +1,12 @@ -Welcome to Modulus Core's documentation! -======================================== +PhysicsNeMo +================= -.. toctree:: - :maxdepth: 2 - :caption: Modulus Tutorials - :name: Modulus Tutorials - - tutorials/simple_training_example.rst - tutorials/simple_logging_and_checkpointing.rst +Welcome to the PhysicsNeMo documentation. This section contains the API reference and examples for the main PhysicsNeMo repository. .. toctree:: :maxdepth: 2 - :caption: Modulus API - :name: Modulus API - - api/modulus.models.rst - api/modulus.datapipes.rst - api/modulus.metrics.rst - api/modulus.deploy.rst - api/modulus.distributed.rst - api/modulus.utils.rst - api/modulus.launch.logging.rst - api/modulus.launch.utils.rst - - -.. toctree:: - :maxdepth: 1 - :caption: Examples: Weather and Climate - :name: Examples: Weather and Climate - - examples/weather/dataset_download/README.rst - examples/weather/fcn_afno/README.rst - examples/weather/dlwp/README.rst - examples/weather/graphcast/README.rst - - -.. toctree:: - :maxdepth: 1 - :caption: Examples: CFD - :name: Examples: CFD - - examples/cfd/ahmed_body_mgn/README.rst - examples/cfd/vortex_shedding_mgn/README.rst - examples/cfd/darcy_fno/README.rst - examples/cfd/darcy_nested_fnos/README.rst - examples/cfd/navier_stokes_rnn/README.rst - examples/cfd/gray_scott_rnn/README.rst - - -.. toctree:: - :maxdepth: 1 - :caption: Examples: Healthcare - :name: Examples: Healthcare - - examples/healthcare/bloodflow_1d_mgn/README.rst - -.. toctree:: - :maxdepth: 1 - :caption: Examples: Molecular Dynamics - :name: Examples: Molecular Dynamics - - examples/molecular_dynamics/lennard_jones/README.rst - - -Indices and tables -================== - -* :ref:`genindex` + :caption: Contents + :name: Contents + api_index + examples_index diff --git a/docs/nn/functional/interpolation/benchmark.png b/docs/nn/functional/interpolation/benchmark.png new file mode 100644 index 0000000000..c1df9ad154 Binary files /dev/null and b/docs/nn/functional/interpolation/benchmark.png differ diff --git a/examples/generative/corrdiff/docs/2023-arxiv/generation_scripts/out b/docs/nn/functional/knn/.gitkeep similarity index 100% rename from examples/generative/corrdiff/docs/2023-arxiv/generation_scripts/out rename to docs/nn/functional/knn/.gitkeep diff --git a/docs/nn/functional/radius_search/.gitkeep b/docs/nn/functional/radius_search/.gitkeep new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/nn/functional/radius_search/benchmark.png b/docs/nn/functional/radius_search/benchmark.png new file mode 100644 index 0000000000..bd37a88252 Binary files /dev/null and b/docs/nn/functional/radius_search/benchmark.png differ diff --git a/docs/nn/functional/sdf/.gitkeep b/docs/nn/functional/sdf/.gitkeep new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/rewrite_examples_image_paths.lua b/docs/rewrite_examples_image_paths.lua new file mode 100644 index 0000000000..3a80674057 --- /dev/null +++ b/docs/rewrite_examples_image_paths.lua @@ -0,0 +1,7 @@ +-- rewrite_image_paths.lua +function Image(el) + -- Example: change '../../docs/img/foo.png' → '../../img/foo.png' + el.src = string.gsub(el.src, "/docs/img/", "/img/") + return el + end + \ No newline at end of file diff --git a/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_baseline.py b/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_baseline.py new file mode 100644 index 0000000000..a1181e1443 --- /dev/null +++ b/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_baseline.py @@ -0,0 +1,53 @@ +import torch +import time + +# This time, let's make two moderately large tensors since we'll have to, at least briefly, +# construct a tensor of their point-by-point difference. +N_points_to_search = 234_567 +N_target_points = 12_345 +num_neighbors = 17 + + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +# We'll make these 3D tensors to represent 3D points +a = torch.randn(N_points_to_search, 3, device=device) +b = torch.randn(N_target_points, 3, device=device) + +def knn(x, y, n): + # Return the n nearest neighbors in x for each point in y. + # Returns the + + # First, compute the pairwise difference between all points in x and y. + displacement_vec = x[None, :, :] - y[:, None, :] + + # Use the norm to compute the distance: + distance = torch.norm(displacement_vec, dim=2) + + distances, indices = torch.topk(distance, k=n, dim=1, largest=False) + + x_results = x[indices] + # distance = distances[indices] + + return x_results, distances + +y_neighbors_to_x, neighbor_disances = knn(a,b, num_neighbors) +print(y_neighbors_to_x.shape) # should be (N_target_points, num_neighbors, 3) +print(neighbor_disances.shape) # should be (N_target_points, num_neighbors) + +# run a couple times to warmup: +for i in range(5): + _ = knn(a,b, num_neighbors) + +# Optional: Benchmark it if you like: + +# Measure execution time +torch.cuda.synchronize() +start_time = time.time() +for i in range(10): + _ = knn(a,b, num_neighbors) +torch.cuda.synchronize() +end_time = time.time() +elapsed_time = end_time - start_time + +print(f"Execution time for 10 runs: {elapsed_time:.4f} seconds") \ No newline at end of file diff --git a/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_ring_sharded.py b/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_ring_sharded.py new file mode 100644 index 0000000000..40ab03d66b --- /dev/null +++ b/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_ring_sharded.py @@ -0,0 +1,210 @@ +import torch +import torch.distributed as dist +from torch.overrides import handle_torch_function, has_torch_function +import time + +from physicsnemo.distributed import DistributedManager, scatter_tensor, ShardTensor +from torch.distributed.tensor.placement_types import Shard, Replicate + +from physicsnemo.distributed.shard_utils.ring import perform_ring_iteration, RingPassingConfig + +# This time, let's make two moderately large tensors since we'll have to, at least briefly, +# construct a tensor of their point-by-point difference. +N_points_to_search = 234_567 +N_target_points = 12_345 +num_neighbors = 17 + +DistributedManager.initialize() +dm = DistributedManager() + +device = dm.device + + + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +# We'll make these 3D tensors to represent 3D points +a = torch.randn(N_points_to_search, 3, device=device) +b = torch.randn(N_target_points, 3, device=device) + +def knn(x, y, n): + # This is to enable torch to track this knn function and route it correctly in ShardTensor: + if has_torch_function((x, y)): + return handle_torch_function( + knn, (x, y), x, y, n + ) + + # Return the n nearest neighbors in x for each point in y. + + # First, compute the pairwise difference between all points in x and y. + displacement_vec = x[None, :, :] - y[:, None, :] + + # Use the norm to compute the distance: + distance = torch.norm(displacement_vec, dim=2) + + distances, indices = torch.topk(distance, k=n, dim=1, largest=False) + + x_results = x[indices] + + return x_results, distances + +# Get the baseline result +y_neighbors_to_x, neighbor_disances = knn(a,b, num_neighbors) + +if dm.rank == 0: + + print(y_neighbors_to_x.shape) # should be (N_target_points, num_neighbors, 3) + print(neighbor_disances.shape) # should be (N_target_points, num_neighbors) + +# DeviceMesh is a pytorch object - you can initialize it directly, or for added +# flexibility physicsnemo can infer up to one mesh dimension for you +# (as a -1, like in a tensor.reshape() call...) +mesh = dm.initialize_mesh(mesh_shape = [-1,], mesh_dim_names = ["domain"]) +# Shard(i) indicates we want the final tensor to be sharded along the tensor dimension i +# But the placements is a tuple or list, indicating the desired placement along the mesh. +placements = (Shard(0),) +# This function will distribute the tensor from global_src to the specified mesh, +# using the input placements. +# Note that in multi-level parallelism, the source is the _global_ rank not the mesh group rank. +a_sharded = scatter_tensor(tensor = a, global_src = 0, mesh = mesh, placements = placements) +b_sharded = scatter_tensor(tensor = b, global_src = 0, mesh = mesh, placements = placements) + + +def knn_ring(func, types, args, kwargs): + # Wrapper to intercept knn and compute it in a ring. + # Never fully realizes the distance product. + + def extract_args(x, y, n, *args, **kwargs): + return x, y, n + x, y, n = extract_args(*args, **kwargs) + + + # Each tensor has a _spec attribute, which contains information about the tensor's placement + # and the devices it lives on: + x_spec = x._spec + y_spec = y._spec + + # ** In general ** you want to do some checking on the placements, since each + # point cloud might be sharded differently. By construction, I know they're both + # sharded along the points axis here (and not, say, replicated). + + if not x_spec.mesh == y_spec.mesh: + raise NotImplementedError("Tensors must be sharded on the same mesh") + + mesh = x_spec.mesh + local_group = mesh.get_group(0) + local_size = dist.get_world_size(group=local_group) + mesh_rank = mesh.get_local_rank() + + # x and y are both sharded - and since we're returning the nearest + # neighbors to x, let's make sure the output keeps that sharding too. + + # One memory-efficient way to do this is with with a ring computation. + # We'll compute the knn on the local tensors, get the distances and outputs, + # then shuffle the y shards along the mesh. + + # we'll need to sort the results and make sure we have just the top-k, + # which is a little extra computation. + + # Physics nemo has a ring passing utility we can use. + ring_config = RingPassingConfig( + mesh_dim = 0, + mesh_size = local_size, + ring_direction = "forward", + communication_method = "p2p" + ) + + local_x, local_y = x.to_local(), y.to_local() + current_dists = None + current_topk_y = None + + x_sharding_shapes = x._spec.sharding_shapes()[0] + + + for i in range(local_size): + source_rank = (mesh_rank - i) % local_size + + # For point clouds, we need to pass the size of the incoming shard. + next_source_rank = (source_rank - 1) % local_size + recv_shape = x_sharding_shapes[next_source_rank] + if i != local_size - 1: + # Don't do a ring on the last iteration. + next_local_x = perform_ring_iteration( + local_x, + mesh, + ring_config, + recv_shape=recv_shape, + ) + + # Compute the knn on the local tensors: + local_x_results, local_distances = func(local_x, local_y, n) + + + if current_dists is None: + current_dists = local_distances + current_topk_y = local_x_results + else: + # Combine with the topk so far: + current_dists = torch.cat([current_dists, local_distances], dim=1) + current_topk_y = torch.cat([current_topk_y, local_x_results], dim=1) + # And take the topk again: + current_dists, running_indexes = torch.topk(current_dists, k=n, dim=1, largest=False) + + # This creates proper indexing to select specific elements along dim 1 + current_topk_y = torch.gather(current_topk_y, 1, + running_indexes.unsqueeze(-1).expand(-1, -1, 3)) + + + + if i != local_size - 1: + # Don't do a ring on the last iteration. + local_x = next_local_x + + # Finally, return the outputs as ShardTensors. + topk_y = ShardTensor.from_local( + current_topk_y, + device_mesh = mesh, + placements = y._spec.placements, + sharding_shapes = y._spec.sharding_shapes(), + ) + + distances = ShardTensor.from_local( + current_dists, + device_mesh = mesh, + placements = y._spec.placements, + sharding_shapes = y._spec.sharding_shapes(), + ) + + return topk_y, distances + + +ShardTensor.register_function_handler(knn, knn_ring) + +# Get the sharded result +y_neighbors_to_x_sharded, neighbor_disances_sharded = knn(a_sharded,b_sharded, num_neighbors) + +# Check for agreement: +y_neighbors_to_x_sharded = y_neighbors_to_x_sharded.full_tensor() +neighbor_disances_sharded = neighbor_disances_sharded.full_tensor() + +if dm.rank == 0: + print(f"Neighbors agreement? {torch.allclose(y_neighbors_to_x, y_neighbors_to_x_sharded)}") + print(f"Distances agreement? {torch.allclose(neighbor_disances, neighbor_disances_sharded)}") + + +# run a couple times to warmup: +for i in range(5): + _ = knn(a_sharded,b_sharded, num_neighbors) +# Optional: Benchmark it if you like: + +# Measure execution time +torch.cuda.synchronize() +start_time = time.time() +for i in range(10): + _ = knn(a_sharded,b_sharded, num_neighbors) +torch.cuda.synchronize() +end_time = time.time() +elapsed_time = end_time - start_time + +if dm.rank == 0: + print(f"Execution time for 10 runs: {elapsed_time:.4f} seconds") diff --git a/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_sharded.py b/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_sharded.py new file mode 100644 index 0000000000..1084a81823 --- /dev/null +++ b/docs/test_scripts/domain_parallelism/new_layers/knn_brute_force_sharded.py @@ -0,0 +1,86 @@ +import torch +import torch.distributed as dist +import time + +from physicsnemo.distributed import DistributedManager, scatter_tensor, ShardTensor +from torch.distributed.tensor.placement_types import Shard, Replicate + +from physicsnemo.distributed.shard_utils.ring import perform_ring_iteration, RingPassingConfig + +# This time, let's make two moderately large tensors since we'll have to, at least briefly, +# construct a tensor of their point-by-point difference. +N_points_to_search = 234_567 +N_target_points = 12_345 +num_neighbors = 17 + +DistributedManager.initialize() +dm = DistributedManager() + +# We'll make these 3D tensors to represent 3D points +a = torch.randn(N_points_to_search, 3, device=dm.device) +b = torch.randn(N_target_points, 3, device=dm.device) + +def knn(x, y, n): + # Return the n nearest neighbors in x for each point in y. + + # First, compute the pairwise difference between all points in x and y. + displacement_vec = x[None, :, :] - y[:, None, :] + + # Use the norm to compute the distance: + distance = torch.norm(displacement_vec, dim=2) + + distances, indices = torch.topk(distance, k=n, dim=1, largest=False) + + x_results = x[indices] + + return x_results, distances + +# Get the baseline result +y_neighbors_to_x, neighbor_distances = knn(a, b, num_neighbors) + +if dm.rank == 0: + print(y_neighbors_to_x.shape) # should be (N_target_points, num_neighbors, 3) + print(neighbor_distances.shape) # should be (N_target_points, num_neighbors) + +# DeviceMesh is a pytorch object - you can initialize it directly, or for added +# flexibility physicsnemo can infer up to one mesh dimension for you +# (as a -1, like in a tensor.reshape() call...) +mesh = dm.initialize_mesh(mesh_shape = [-1,], mesh_dim_names = ["domain"]) +# Shard(i) indicates we want the final tensor to be sharded along the tensor dimension i +# But the placements is a tuple or list, indicating the desired placement along the mesh. +placements = (Shard(0),) +# This function will distribute the tensor from global_src to the specified mesh, +# using the input placements. +# Note that in multi-level parallelism, the source is the _global_ rank not the mesh group rank. +a_sharded = scatter_tensor(tensor = a, global_src = 0, mesh = mesh, placements = placements) +b_sharded = scatter_tensor(tensor = b, global_src = 0, mesh = mesh, placements = placements) + +# Get the sharded result +y_neighbors_to_x_sharded, neighbor_distances_sharded = knn(a_sharded, b_sharded, num_neighbors) + +# Check for agreement: +y_neighbors_to_x_sharded = y_neighbors_to_x_sharded.full_tensor() +neighbor_distances_sharded = neighbor_distances_sharded.full_tensor() + +if dm.rank == 0: + # Note - do the ``full_tensor`` call outside this if-block or it will hang! + print(f"Neighbors agreement? {torch.allclose(y_neighbors_to_x, y_neighbors_to_x_sharded)}") + print(f"Distances agreement? {torch.allclose(neighbor_distances, neighbor_distances_sharded)}") + +# run a couple times to warmup: +for i in range(5): + _ = knn(a_sharded, b_sharded, num_neighbors) + +# Optional: Benchmark it if you like: +# Measure execution time +torch.cuda.synchronize() +start_time = time.time() +for i in range(10): + _ = knn(a_sharded, b_sharded, num_neighbors) +torch.cuda.synchronize() +end_time = time.time() +elapsed_time = end_time - start_time + +if dm.rank == 0: + print(f"Execution time for 10 runs: {elapsed_time:.4f} seconds") + diff --git a/docs/test_scripts/domain_parallelism/new_layers/reshape_subtract.py b/docs/test_scripts/domain_parallelism/new_layers/reshape_subtract.py new file mode 100644 index 0000000000..a2ed4fec6a --- /dev/null +++ b/docs/test_scripts/domain_parallelism/new_layers/reshape_subtract.py @@ -0,0 +1,49 @@ +import torch +import torch.distributed as dist +import time + +from physicsnemo.distributed import DistributedManager, scatter_tensor, ShardTensor +from torch.distributed.tensor.placement_types import Shard, Replicate + +from physicsnemo.distributed.shard_utils.ring import perform_ring_iteration, RingPassingConfig + +# This time, let's make two moderately large tensors since we'll have to, at least briefly, +# construct a tensor of their point-by-point difference. +N1 = 234_567 +N2 = 12_345 +num_neighbors = 17 + +DistributedManager.initialize() +dm = DistributedManager() + +# We'll make these 3D tensors to represent 3D points +a = torch.randn(N1, 3, device=dm.device) +b = torch.randn(N2, 3, device=dm.device) + +# DeviceMesh is a pytorch object - you can initialize it directly, or for added +# flexibility physicsnemo can infer up to one mesh dimension for you +# (as a -1, like in a tensor.reshape() call...) +mesh = dm.initialize_mesh(mesh_shape = [-1,], mesh_dim_names = ["domain"]) +# Shard(i) indicates we want the final tensor to be sharded along the tensor dimension i +# But the placements is a tuple or list, indicating the desired placement along the mesh. +placements = (Shard(0),) +# This function will distribute the tensor from global_src to the specified mesh, +# using the input placements. +# Note that in multi-level parallelism, the source is the _global_ rank not the mesh group rank. +a_sharded = scatter_tensor(tensor = a, global_src = 0, mesh = mesh, placements = placements) +b_sharded = scatter_tensor(tensor = b, global_src = 0, mesh = mesh, placements = placements) + +if dm.rank == 0: + print(f"a_sharded shape and placement: {a_sharded.shape}, {a_sharded.placements}") + print(f"b_sharded shape and placement: {b_sharded.shape}, {b_sharded.placements}") +a_sharded = a_sharded[None, :, :] +b_sharded = b_sharded[:, None, :] + +if dm.rank == 0: + print(f"a_sharded shape and placement: {a_sharded.shape}, {a_sharded.placements}") + print(f"b_sharded shape and placement: {b_sharded.shape}, {b_sharded.placements}") + +distance_vec = a_sharded - b_sharded +if dm.rank == 0: + print(f"distance_vec shape and placement: {distance_vec.shape}, {distance_vec.placements}") + diff --git a/docs/test_scripts/domain_parallelism/new_layers/vector_add_baseline.py b/docs/test_scripts/domain_parallelism/new_layers/vector_add_baseline.py new file mode 100644 index 0000000000..898384b918 --- /dev/null +++ b/docs/test_scripts/domain_parallelism/new_layers/vector_add_baseline.py @@ -0,0 +1,31 @@ +import torch +import time + +# Make a really big tensor: +N = 1_000_000_000 + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +a = torch.randn(N, device=device) +b = torch.randn(N, device=device) + +def f(a, b): + # This is a truly local operation: no communication is needed. + return a + b + +# run a couple times to warmup: +for i in range(5): + c = f(a,b) + +# Optional: Benchmark it if you like: + +# Measure execution time +torch.cuda.synchronize() +start_time = time.time() +for i in range(10): + c = f(a,b) +torch.cuda.synchronize() +end_time = time.time() +elapsed_time = end_time - start_time + +print(f"Execution time for 10 runs: {elapsed_time:.4f} seconds") \ No newline at end of file diff --git a/docs/test_scripts/domain_parallelism/new_layers/vector_add_sharded.py b/docs/test_scripts/domain_parallelism/new_layers/vector_add_sharded.py new file mode 100644 index 0000000000..df8a98d8dc --- /dev/null +++ b/docs/test_scripts/domain_parallelism/new_layers/vector_add_sharded.py @@ -0,0 +1,57 @@ +import torch +import time + +from physicsnemo.distributed import DistributedManager, scatter_tensor +from torch.distributed.tensor.placement_types import Shard + +# Another really big tensor: +N = 1_000_000_000 + +DistributedManager.initialize() +dm = DistributedManager() + +device = dm.device + +a = torch.randn(N, device=device) +b = torch.randn(N, device=device) + +def f(x, y): + return x + y + +# Get the baseline result +c_baseline = f(a,b) + +# DeviceMesh is a pytorch object - you can initialize it directly, or for added +# flexibility physicsnemo can infer up to one mesh dimension for you +# (as a -1, like in a tensor.reshape() call...) +mesh = dm.initialize_mesh(mesh_shape = [-1,], mesh_dim_names = ["domain"]) +# Shard(i) indicates we want the final tensor to be sharded along the tensor dimension i +# But the placements is a tuple or list, indicating the desired placement along the mesh. +placements = (Shard(0),) +# This function will distribute the tensor from global_src to the specified mesh, +# using the input placements. +# Note that in multi-level parallelism, the source is the _global_ rank not the mesh group rank. +a_sharded = scatter_tensor(tensor = a, global_src = 0, mesh = mesh, placements = placements) +b_sharded = scatter_tensor(tensor = b, global_src = 0, mesh = mesh, placements = placements) +c_sharded = f(a_sharded,b_sharded) + +# Comparison requires that we coalesce the results: +c_sharded = c_sharded.full_tensor() + +# Now, performance measurement: +# Warm up: +for i in range(5): + c = f(a_sharded,b_sharded) + +# Measure execution time +torch.cuda.synchronize() +start_time = time.time() +for i in range(10): + c = f(a_sharded,b_sharded) +torch.cuda.synchronize() +end_time = time.time() +elapsed_time = end_time - start_time + +if dm.rank == 0: + print(f"Rank {dm.rank}, Tensor agreement? {torch.allclose(c_baseline, c_sharded)}") + print(f"Execution time for 10 runs: {elapsed_time:.4f} seconds") diff --git a/docs/test_scripts/domain_parallelism/new_layers/vector_dot_baseline.py b/docs/test_scripts/domain_parallelism/new_layers/vector_dot_baseline.py new file mode 100644 index 0000000000..1c1b3a83a3 --- /dev/null +++ b/docs/test_scripts/domain_parallelism/new_layers/vector_dot_baseline.py @@ -0,0 +1,31 @@ +import torch +import time + +# Make a really big tensor: +N = 1_000_000_000 + +device = torch.device("cuda" if torch.cuda.is_available() else "cpu") + +a = torch.randn(N, device=device) +b = torch.randn(N, device=device) + +def f(a, b): + # This is a truly non-local operation: full reduction is needed. + return torch.dot(a, b) + +# run a couple times to warmup: +for i in range(5): + c = f(a,b) + +# Optional: Benchmark it if you like: + +# Measure execution time +torch.cuda.synchronize() +start_time = time.time() +for i in range(10): + c = f(a,b) +torch.cuda.synchronize() +end_time = time.time() +elapsed_time = end_time - start_time + +print(f"Execution time for 10 runs: {elapsed_time:.4f} seconds") \ No newline at end of file diff --git a/docs/test_scripts/domain_parallelism/new_layers/vector_dot_sharded.py b/docs/test_scripts/domain_parallelism/new_layers/vector_dot_sharded.py new file mode 100644 index 0000000000..b1c19b8a34 --- /dev/null +++ b/docs/test_scripts/domain_parallelism/new_layers/vector_dot_sharded.py @@ -0,0 +1,146 @@ +import torch +import torch.distributed as dist +import time + +from physicsnemo.distributed import DistributedManager, scatter_tensor, ShardTensor +from torch.distributed.tensor.placement_types import Shard, Replicate + +def sharded_dot_product(func: Callable, types: Tuple, args: Tuple, kwargs: Dict): + """ + Overload for torch.dot to support sharded tensors. + + This function enables mutli-gpu dot product operations on ShardTensors, + by computing the dot product locally on each rank and then summin across + all GPUs. Requires the placements and mesh to agree across the two tensors. + + This is tutorial code: it does not handle all cases and you should + not use it in production. + + Note the function signature: we are using this function in the + __torch_function__ protocol and it has to follow the specific signature + requirements. + + Args: + func (Callable): The function to overload (e.g., torch.dot). + types (Tuple): Tuple of types passed by __torch_function__ protocol. + args (Tuple): Positional arguments passed to the function. + kwargs (Dict): Keyword arguments passed to the function. + + In general, torch will use the values in `types` to determine which + path of execution to take. In this function, we don't have to worry + about that as much because it's already selected for execution. + """ + # NOTE: all functions overloaded and used by __torch_function__ will have + # the same input signature. You can use python argument unpacking to + # extract what you need: + def extract_args(x, y, *args, **kwargs): + return x, y + x, y = extract_args(*args, **kwargs) + + # Each tensor has a _spec attribute, which contains information about the tensor's placement + # and the devices it lives on: + x_spec = x._spec + y_spec = y._spec + + # IT'S usually good to ensure the tensor placements work: + if not x_spec.placements == y_spec.placements: + raise NotImplementedError("Tensors must be sharded on the same device") + + if not x_spec.mesh == y_spec.mesh: + raise NotImplementedError("Tensors must be sharded on the same mesh") + + # And, you might want to check placements are valid in more complex cases + + # Extract the mesh - we'll want it for the all reduce: + mesh = x_spec.mesh + + # This is a straightforward implementation, for clarity + # Get the local values of each tensor: + local_x = x.to_local() + local_y = y.to_local() + + # This is a purely single-gpu operation: + local_dot_product = torch.dot(local_x, local_y) + # If you wanted to write a generic sharding handler for this type of operation, + # you could do: + # local_dot_product = func(local_x, local_y) + # But it's over kill here... + + # SUM_Reduce the local result across all ranks: + dist.all_reduce(local_dot_product, op=dist.ReduceOp.SUM, group=mesh.get_group()) + + # We do want to return the result as a ShardTensor, for consistency. + # We can easily create one on the same mesh as a "Replicated" tensor: + + + # The output placements are now Replicated, not sharded. We have used all_reduce + # to sum the local results across all ranks, and each rank has the full data - + # exactly what the Replicate() placement expects. + # (Even though it's a scalar output, we still have to specify a placement) + output = ShardTensor.from_local( + local_tensor = local_dot_product, + device_mesh = mesh, + placements = (Replicate(),) + ) + + return output + +# Register the implementation with ShardTensor's function dispatch: +ShardTensor.register_function_handler(torch.dot, sharded_dot_product) + + +# Another really big tensor: +N = 1_000_000_000 + +DistributedManager.initialize() +dm = DistributedManager() + +device = dm.device + +a = torch.randn(N, device=device) +b = torch.randn(N, device=device) + +def f(x, y): + return torch.dot(x , y) + +# Get the baseline result +c_baseline = f(a,b) + +# DeviceMesh is a pytorch object - you can initialize it directly, or for added +# flexibility physicsnemo can infer up to one mesh dimension for you +# (as a -1, like in a tensor.reshape() call...) +mesh = dm.initialize_mesh(mesh_shape = [-1,], mesh_dim_names = ["domain"]) +# Shard(i) indicates we want the final tensor to be sharded along the tensor dimension i +# But the placements is a tuple or list, indicating the desired placement along the mesh. +placements = (Shard(0),) +# This function will distribute the tensor from global_src to the specified mesh, +# using the input placements. +# Note that in multi-level parallelism, the source is the _global_ rank not the mesh group rank. +a_sharded = scatter_tensor(tensor = a, global_src = 0, mesh = mesh, placements = placements) +b_sharded = scatter_tensor(tensor = b, global_src = 0, mesh = mesh, placements = placements) + + +c_sharded = f(a_sharded,b_sharded) + +# Comparison requires that we coalesce the results: +c_sharded = c_sharded.full_tensor() + + +# Now, performance measurement: + +# Warm up: +for i in range(5): + c = f(a_sharded,b_sharded) + +# Measure execution time +torch.cuda.synchronize() +start_time = time.time() +for i in range(10): + c = f(a_sharded,b_sharded) +torch.cuda.synchronize() +end_time = time.time() +elapsed_time = end_time - start_time + +if dm.rank == 0: + print(f"Rank {dm.rank}, Tensor agreement? {torch.allclose(c_baseline, c_sharded)}") + print(f"Execution time for 10 runs: {elapsed_time:.4f} seconds") \ No newline at end of file diff --git a/docs/test_scripts/domain_parallelism/sharded_conv_example.py b/docs/test_scripts/domain_parallelism/sharded_conv_example.py new file mode 100644 index 0000000000..ae8887bc7e --- /dev/null +++ b/docs/test_scripts/domain_parallelism/sharded_conv_example.py @@ -0,0 +1,119 @@ +import torch + +from torch.distributed.tensor import ( + Shard, + distribute_module, +) + + +from physicsnemo.distributed import ( + DistributedManager, + ShardTensor, + scatter_tensor, +) + +DistributedManager.initialize() +dm = DistributedManager() + +########################### +# Single GPU - Create input +########################### +original_tensor = torch.randn(1, 8, 1024, 1024, device=dm.device, requires_grad=True) + +########################################### +# Single GPU - Create a single-layer model: +########################################### +conv = torch.nn.Conv2d(8, 8, 3, stride=1, padding=1).to(dm.device) + +######################################## +# Single GPU - forward + loss + backward +######################################## +single_gpu_output = conv(original_tensor) + +# This isn't really a loss, just a pretend one that's scalar! +single_gpu_output.mean().backward() +# Copy the gradients produced here - so we don't overwrite them later. +original_tensor_grad = original_tensor.grad.data.clone() + +#################### +# Single GPU - DONE! +#################### + + +################# +# Sharded - Setup +################# + + +# DeviceMesh is a pytorch object - you can initialize it directly, or for added +# flexibility physicsnemo can infer up to one mesh dimension for you +# (as a -1, like in a tensor.reshape() call...) +mesh = dm.initialize_mesh(mesh_shape=(-1,), mesh_dim_names=("domain_parallel",)) + +# A mesh, by the way, refers to devices and not data: it's a mesh of connected +# GPUs in this case, and the python DeviceMesh can be reused as many times as needed. +# That said, it can be decomposed similar to a tensor - multiple mesh axes, and +# you can axis sub-meshes. Each mesh also has ways to access process groups +# for targeted collectives. + + +########################### +# Sharded - Distribute Data +########################### + +# This is now a tensor across all GPUs, spread on the "height" dimension == 2 +# In general, to create a ShardTensor (or DTensor) you need to specify placements. +# Placements must be a list or tuple of `Shard()` or `Replicate()` objects +# from torch.distributed.tensor. +# +# Each index in the tuple represents the placement over the corresponding mesh dimension +# (so, mesh.ndim == len(placements)! ) +# `Shard()` takes an argument representing the **tensor** index that is sharded. +# So below, the tensor is sharded over the tensor dimension 2 on the mesh dimension 0. +sharded_tensor = scatter_tensor(original_tensor, 0, mesh, (Shard(2),), requires_grad=True) + + +################################ +# Sharded - distribute the model +################################ + +# We tell pytorch that the convolution will work on distributed tensors: +# And, over the same mesh! +distributed_conv = distribute_module(conv, mesh) + + +##################################### +# Sharded - forward + loss + backward +##################################### + +# Now, we can do the distributed convolution: +sharded_output = distributed_conv(sharded_tensor) +sharded_output.mean().backward() + + +############################################ +# Sharded - gather up outputs to all devices +############################################ + +# This triggers a collective allgather. +full_output = sharded_output.full_tensor() +full_grad = sharded_tensor.grad.full_tensor() + + + +################# +# Accuracy Checks +################# + +if dm.rank == 0: + # Only check on rank 0 because we used it's data and weights for the sharded tensor. + # Check that the output is the same as the single-device output: + assert torch.allclose(full_output, single_gpu_output) + print(f"Global operation matches local! ") + + # Check that the gradient is correct: + assert torch.allclose(original_tensor_grad, full_grad) + print(f"Gradient check passed!") + + +print(f"Distributed grad sharding and local shape: {sharded_tensor.grad._spec.placements}, {sharded_tensor.grad.to_local().shape}") diff --git a/docs/test_scripts/profiling/annotated_code/attn.py b/docs/test_scripts/profiling/annotated_code/attn.py new file mode 100644 index 0000000000..feb08946e4 --- /dev/null +++ b/docs/test_scripts/profiling/annotated_code/attn.py @@ -0,0 +1,108 @@ +import torch +from torch import nn + +from physicsnemo.utils.profiling import profile, annotate + +class Attention(nn.Module): + """Dummy example Attention mechanism. Meant not for efficienct computation + but to show how to use the profiling tools! + + """ + + def __init__( + self, + dim: int, + num_heads: int = 8, + qkv_bias: bool = False, + attn_drop: float = 0., + proj_drop: float = 0., + ) -> None: + super().__init__() + assert dim % num_heads == 0, 'dim should be divisible by num_heads' + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.scale = self.head_dim ** -0.5 + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + + B, N, C = x.shape + qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4) + q, k, v = qkv.unbind(0) + + + # This is not optimal code right here ... + q = q * self.scale + attn = q @ k.transpose(-2, -1) + attn = attn.softmax(dim=-1) + attn = self.attn_drop(attn) + x = attn @ v + + x = x.transpose(1, 2).reshape(B, N, C) + x = self.proj(x) + x = self.proj_drop(x) + return x + + + + +class MLP(nn.Module): + def __init__(self, in_features, hidden_features=None, out_features=None, drop=0): + super().__init__() + hidden_features = hidden_features or in_features + out_features = out_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.gelu = nn.GELU() + self.drop1 = nn.Dropout(drop) + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop2 = nn.Dropout(drop) + + @profile + def forward(self, x): + x = self.fc1(x) + x = self.gelu(x) + x = self.drop1(x) + x = self.fc2(x) + x = self.gelu(x) + x = self.drop2(x) + return x + +class Block(nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + mlp_ratio: float = 4., + qkv_bias: bool = False, + proj_drop: float = 0., + attn_drop: float = 0., + norm_layer: nn.Module = nn.LayerNorm, + mlp_layer: nn.Module = MLP, + ) -> None: + super().__init__() + self.norm1 = norm_layer(dim) + self.attn = Attention( + dim, + num_heads=num_heads, + qkv_bias=qkv_bias, + attn_drop=attn_drop, + proj_drop=proj_drop, + ) + + self.norm2 = norm_layer(dim) + self.mlp = mlp_layer( + in_features=dim, + hidden_features=int(dim * mlp_ratio), + drop=proj_drop, + ) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + self.attn(self.norm1(x)) + x = x + self.mlp(self.norm2(x)) + return x \ No newline at end of file diff --git a/docs/test_scripts/profiling/annotated_code/dataset.py b/docs/test_scripts/profiling/annotated_code/dataset.py new file mode 100644 index 0000000000..92eb90479f --- /dev/null +++ b/docs/test_scripts/profiling/annotated_code/dataset.py @@ -0,0 +1,46 @@ +import torch +import numpy as np +from torch.utils.data import Dataset + + +class RandomNoiseDataset(Dataset): + """ + Random normal distribution dataset. + + Mean AND STD of the distribution is set to the index + of the sample requested. + + Length is hardcoded to 64. + + (Don't use this anywhere that isn't an example of how to write non-performant python code!) + + """ + + def __init__(self, image_shape, ): + """ + Arguments: + image_shape (string): Shape of a single example to generate + """ + self.shape = image_shape + + self.rng = np.random.default_rng() + + def __len__(self): + return 64 + + def __getitem__(self, idx): + if torch.is_tensor(idx): + idx = idx.tolist() + + # Generate the raw data: + raw = self.gen_single_image(idx) + + sample = { + 'image' : raw + } + + return sample + + def gen_single_image(self, idx): + + return self.rng.normal(loc=idx, scale=idx, size=self.shape).astype(np.float32) diff --git a/docs/test_scripts/profiling/annotated_code/workload.py b/docs/test_scripts/profiling/annotated_code/workload.py new file mode 100644 index 0000000000..7075c5ce1f --- /dev/null +++ b/docs/test_scripts/profiling/annotated_code/workload.py @@ -0,0 +1,111 @@ +import time + +import torch + +# For hydra: +import hydra +from omegaconf import DictConfig +from pathlib import Path + +# For dataloader +from torch.utils.data import DataLoader + +# Import the dataset: +from dataset import RandomNoiseDataset + +# For the model code: +from attn import Block + +# Import profiling hooks from physicsnemo: +from physicsnemo.utils.profiling import Profiler, profile, annotate + +def loss_fn(output_data): + # All except the first dim: + dims = tuple(range(len(output_data.shape))) + # Just a silly loss function: + output_data = output_data**2. + loss = torch.sum(output_data, dims[1:]) + return loss.mean() + +@profile +def workload(cfg): + + ds = RandomNoiseDataset(cfg["shape"]) + + loader = DataLoader( + ds, + batch_size=cfg["batch_size"], + shuffle = True, + ) + + + # Initialize the model: + model = Block( + dim = cfg["shape"][-1], + num_heads = cfg.model["num_heads"], + qkv_bias = cfg.model["qkv_bias"] , + attn_drop = cfg.model["attn_drop"], + proj_drop = cfg.model["proj_drop"], + ).to("cuda") + + if cfg["train"]: + opt = torch.optim.SGD(model.parameters(), lr=0.0001, momentum=0.9) + + times = [] + with Profiler() as p: + start = time.perf_counter() + for i, batch in enumerate(loader): + image = batch["image"] + image = image.to("cuda") + with annotate(domain="forward", color="blue"): + output = model(image) + if cfg["train"]: + opt.zero_grad() + # Compute the loss: + loss = loss_fn(output) + # Do the gradient calculation: + with annotate(domain="backward", color="green"): + loss.backward() + # Apply the gradients + opt.step() + p.step() + torch.cuda.synchronize() + end = time.perf_counter() + print(f"Finished step {i} in {end - start:.4f} seconds") + times.append(end - start) + start = time.perf_counter() + + times = torch.tensor(times) + # Drop first and last: + avg_time = times[1:-1].mean() + # compute throughput too: + throughput = cfg["batch_size"] / avg_time + print(f"Average time per iteration: {avg_time:.3f} ({throughput:.3f} examples / s)") + + +@hydra.main(version_base="1.3", config_path="../", config_name="cfg") +def main(config: DictConfig): + + # configure the profiling tools: + p = Profiler() + + for key, val in config.profile.items(): + # This is not the mandatory way to enable tools + # I've set up the config to have the keys match + # the registered profilers. You can do it manually + # too such as `p.enable("torch")` + if val: p.enable(key) + + # The profiler has to be initilized before use. Using it in a context + # will do it automatically, but to use it as a decorator we should do + # it manually here: + p.initialize() + print(p) + + workload(config) + + + +if __name__ == "__main__": + + main() \ No newline at end of file diff --git a/docs/test_scripts/profiling/cfg.yaml b/docs/test_scripts/profiling/cfg.yaml new file mode 100644 index 0000000000..c5e2eb3102 --- /dev/null +++ b/docs/test_scripts/profiling/cfg.yaml @@ -0,0 +1,23 @@ +defaults: + - _self_ + - override hydra/hydra_logging: disabled + - override hydra/job_logging: disabled + +hydra: + output_subdir: null + run: + dir: . + +shape: [2048, 1024] +batch_size: 8 +train: true +model: + num_heads: 32 + qkv_bias: False + attn_drop: 0. + proj_drop: 0. + +profile: + torch: False + nvtx: False + line_profiler: False \ No newline at end of file diff --git a/docs/test_scripts/profiling/fixed_data_loader/attn.py b/docs/test_scripts/profiling/fixed_data_loader/attn.py new file mode 100644 index 0000000000..feb08946e4 --- /dev/null +++ b/docs/test_scripts/profiling/fixed_data_loader/attn.py @@ -0,0 +1,108 @@ +import torch +from torch import nn + +from physicsnemo.utils.profiling import profile, annotate + +class Attention(nn.Module): + """Dummy example Attention mechanism. Meant not for efficienct computation + but to show how to use the profiling tools! + + """ + + def __init__( + self, + dim: int, + num_heads: int = 8, + qkv_bias: bool = False, + attn_drop: float = 0., + proj_drop: float = 0., + ) -> None: + super().__init__() + assert dim % num_heads == 0, 'dim should be divisible by num_heads' + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.scale = self.head_dim ** -0.5 + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + + B, N, C = x.shape + qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4) + q, k, v = qkv.unbind(0) + + + # This is not optimal code right here ... + q = q * self.scale + attn = q @ k.transpose(-2, -1) + attn = attn.softmax(dim=-1) + attn = self.attn_drop(attn) + x = attn @ v + + x = x.transpose(1, 2).reshape(B, N, C) + x = self.proj(x) + x = self.proj_drop(x) + return x + + + + +class MLP(nn.Module): + def __init__(self, in_features, hidden_features=None, out_features=None, drop=0): + super().__init__() + hidden_features = hidden_features or in_features + out_features = out_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.gelu = nn.GELU() + self.drop1 = nn.Dropout(drop) + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop2 = nn.Dropout(drop) + + @profile + def forward(self, x): + x = self.fc1(x) + x = self.gelu(x) + x = self.drop1(x) + x = self.fc2(x) + x = self.gelu(x) + x = self.drop2(x) + return x + +class Block(nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + mlp_ratio: float = 4., + qkv_bias: bool = False, + proj_drop: float = 0., + attn_drop: float = 0., + norm_layer: nn.Module = nn.LayerNorm, + mlp_layer: nn.Module = MLP, + ) -> None: + super().__init__() + self.norm1 = norm_layer(dim) + self.attn = Attention( + dim, + num_heads=num_heads, + qkv_bias=qkv_bias, + attn_drop=attn_drop, + proj_drop=proj_drop, + ) + + self.norm2 = norm_layer(dim) + self.mlp = mlp_layer( + in_features=dim, + hidden_features=int(dim * mlp_ratio), + drop=proj_drop, + ) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + self.attn(self.norm1(x)) + x = x + self.mlp(self.norm2(x)) + return x \ No newline at end of file diff --git a/docs/test_scripts/profiling/fixed_data_loader/dataset.py b/docs/test_scripts/profiling/fixed_data_loader/dataset.py new file mode 100644 index 0000000000..7027190d49 --- /dev/null +++ b/docs/test_scripts/profiling/fixed_data_loader/dataset.py @@ -0,0 +1,45 @@ +import torch +import numpy as np +from torch.utils.data import Dataset + + +class RandomNoiseDataset(Dataset): + """ + Random normal distribution dataset. + + Mean AND STD of the distribution is set to the index + of the sample requested. + + Length is hardcoded to 64. + + (Don't use this anywhere that isn't an example of how to write non-performant python code!) + + """ + + def __init__(self, image_shape, ): + """ + Arguments: + image_shape (string): Shape of a single example to generate + """ + self.shape = list(image_shape) + + + def __len__(self): + return 256 + + def __getitem__(self, idx): + if torch.is_tensor(idx): + idx = idx.tolist() + + # Generate the raw data: + raw = self.gen_single_image(idx) + + sample = { + 'image' : raw + } + + return sample + + def gen_single_image(self, idx): + + return torch.normal(idx, idx, self.shape, device="cuda" ) diff --git a/docs/test_scripts/profiling/fixed_data_loader/workload.py b/docs/test_scripts/profiling/fixed_data_loader/workload.py new file mode 100644 index 0000000000..7075c5ce1f --- /dev/null +++ b/docs/test_scripts/profiling/fixed_data_loader/workload.py @@ -0,0 +1,111 @@ +import time + +import torch + +# For hydra: +import hydra +from omegaconf import DictConfig +from pathlib import Path + +# For dataloader +from torch.utils.data import DataLoader + +# Import the dataset: +from dataset import RandomNoiseDataset + +# For the model code: +from attn import Block + +# Import profiling hooks from physicsnemo: +from physicsnemo.utils.profiling import Profiler, profile, annotate + +def loss_fn(output_data): + # All except the first dim: + dims = tuple(range(len(output_data.shape))) + # Just a silly loss function: + output_data = output_data**2. + loss = torch.sum(output_data, dims[1:]) + return loss.mean() + +@profile +def workload(cfg): + + ds = RandomNoiseDataset(cfg["shape"]) + + loader = DataLoader( + ds, + batch_size=cfg["batch_size"], + shuffle = True, + ) + + + # Initialize the model: + model = Block( + dim = cfg["shape"][-1], + num_heads = cfg.model["num_heads"], + qkv_bias = cfg.model["qkv_bias"] , + attn_drop = cfg.model["attn_drop"], + proj_drop = cfg.model["proj_drop"], + ).to("cuda") + + if cfg["train"]: + opt = torch.optim.SGD(model.parameters(), lr=0.0001, momentum=0.9) + + times = [] + with Profiler() as p: + start = time.perf_counter() + for i, batch in enumerate(loader): + image = batch["image"] + image = image.to("cuda") + with annotate(domain="forward", color="blue"): + output = model(image) + if cfg["train"]: + opt.zero_grad() + # Compute the loss: + loss = loss_fn(output) + # Do the gradient calculation: + with annotate(domain="backward", color="green"): + loss.backward() + # Apply the gradients + opt.step() + p.step() + torch.cuda.synchronize() + end = time.perf_counter() + print(f"Finished step {i} in {end - start:.4f} seconds") + times.append(end - start) + start = time.perf_counter() + + times = torch.tensor(times) + # Drop first and last: + avg_time = times[1:-1].mean() + # compute throughput too: + throughput = cfg["batch_size"] / avg_time + print(f"Average time per iteration: {avg_time:.3f} ({throughput:.3f} examples / s)") + + +@hydra.main(version_base="1.3", config_path="../", config_name="cfg") +def main(config: DictConfig): + + # configure the profiling tools: + p = Profiler() + + for key, val in config.profile.items(): + # This is not the mandatory way to enable tools + # I've set up the config to have the keys match + # the registered profilers. You can do it manually + # too such as `p.enable("torch")` + if val: p.enable(key) + + # The profiler has to be initilized before use. Using it in a context + # will do it automatically, but to use it as a decorator we should do + # it manually here: + p.initialize() + print(p) + + workload(config) + + + +if __name__ == "__main__": + + main() \ No newline at end of file diff --git a/docs/test_scripts/profiling/optimized_code_1/attn.py b/docs/test_scripts/profiling/optimized_code_1/attn.py new file mode 100644 index 0000000000..a1e0185335 --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_1/attn.py @@ -0,0 +1,103 @@ +import torch +from torch import nn +torch.backends.cuda.enable_flash_sdp(True) +from physicsnemo.utils.profiling import profile, annotate + +class Attention(nn.Module): + """Dummy example Attention mechanism using Flash Attention for efficient computation. + """ + + def __init__( + self, + dim: int, + num_heads: int = 8, + qkv_bias: bool = False, + attn_drop: float = 0., + proj_drop: float = 0., + ) -> None: + super().__init__() + assert dim % num_heads == 0, 'dim should be divisible by num_heads' + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.scale = self.head_dim ** -0.5 + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + B, N, C = x.shape + qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4) + q, k, v = qkv.unbind(0) + + # Use scaled_dot_product_attention with flash attention + x = torch.nn.functional.scaled_dot_product_attention( + q, k, v, + dropout_p=self.attn_drop.p if self.training else 0.0, + is_causal=False + ) + + x = x.transpose(1, 2).reshape(B, N, C) + x = self.proj(x) + x = self.proj_drop(x) + return x + + + +class MLP(nn.Module): + def __init__(self, in_features, hidden_features=None, out_features=None, drop=0): + super().__init__() + hidden_features = hidden_features or in_features + out_features = out_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.gelu = nn.GELU() + self.drop1 = nn.Dropout(drop) + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop2 = nn.Dropout(drop) + + @profile + def forward(self, x): + x = self.fc1(x) + x = self.gelu(x) + x = self.drop1(x) + x = self.fc2(x) + x = self.gelu(x) + x = self.drop2(x) + return x + +class Block(nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + mlp_ratio: float = 4., + qkv_bias: bool = False, + proj_drop: float = 0., + attn_drop: float = 0., + norm_layer: nn.Module = nn.LayerNorm, + mlp_layer: nn.Module = MLP, + ) -> None: + super().__init__() + self.norm1 = norm_layer(dim) + self.attn = Attention( + dim, + num_heads=num_heads, + qkv_bias=qkv_bias, + attn_drop=attn_drop, + proj_drop=proj_drop, + ) + + self.norm2 = norm_layer(dim) + self.mlp = mlp_layer( + in_features=dim, + hidden_features=int(dim * mlp_ratio), + drop=proj_drop, + ) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + self.attn(self.norm1(x)) + x = x + self.mlp(self.norm2(x)) + return x \ No newline at end of file diff --git a/docs/test_scripts/profiling/optimized_code_1/dataset.py b/docs/test_scripts/profiling/optimized_code_1/dataset.py new file mode 100644 index 0000000000..7027190d49 --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_1/dataset.py @@ -0,0 +1,45 @@ +import torch +import numpy as np +from torch.utils.data import Dataset + + +class RandomNoiseDataset(Dataset): + """ + Random normal distribution dataset. + + Mean AND STD of the distribution is set to the index + of the sample requested. + + Length is hardcoded to 64. + + (Don't use this anywhere that isn't an example of how to write non-performant python code!) + + """ + + def __init__(self, image_shape, ): + """ + Arguments: + image_shape (string): Shape of a single example to generate + """ + self.shape = list(image_shape) + + + def __len__(self): + return 256 + + def __getitem__(self, idx): + if torch.is_tensor(idx): + idx = idx.tolist() + + # Generate the raw data: + raw = self.gen_single_image(idx) + + sample = { + 'image' : raw + } + + return sample + + def gen_single_image(self, idx): + + return torch.normal(idx, idx, self.shape, device="cuda" ) diff --git a/docs/test_scripts/profiling/optimized_code_1/workload.py b/docs/test_scripts/profiling/optimized_code_1/workload.py new file mode 100644 index 0000000000..a9dcc839da --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_1/workload.py @@ -0,0 +1,112 @@ +import time + +import torch + +# For hydra: +import hydra +from omegaconf import DictConfig +from pathlib import Path + +# For dataloader +from torch.utils.data import DataLoader + +# Import the dataset: +from dataset import RandomNoiseDataset + +# For the model code: +from attn import Block + +# Import profiling hooks from physicsnemo: +from physicsnemo.utils.profiling import Profiler, profile, annotate + +def loss_fn(output_data): + # All except the first dim: + dims = tuple(range(len(output_data.shape))) + # Just a silly loss function: + output_data = output_data**2. + loss = torch.sum(output_data, dims[1:]) + return loss.mean() + +@profile +def workload(cfg): + + ds = RandomNoiseDataset(cfg["shape"]) + + loader = DataLoader( + ds, + batch_size=cfg["batch_size"], + shuffle = True, + ) + + + # Initialize the model: + model = Block( + dim = cfg["shape"][-1], + num_heads = cfg.model["num_heads"], + qkv_bias = cfg.model["qkv_bias"] , + attn_drop = cfg.model["attn_drop"], + proj_drop = cfg.model["proj_drop"], + ).to("cuda") + + if cfg["train"]: + opt = torch.optim.SGD(model.parameters(), lr=0.0001, momentum=0.9) + + times = [] + with Profiler() as p: + start = time.perf_counter() + for i, batch in enumerate(loader): + image = batch["image"] + image = image.to("cuda") + with annotate(domain="forward", color="blue"): + output = model(image) + if cfg["train"]: + opt.zero_grad() + # Compute the loss: + loss = loss_fn(output) + # Do the gradient calculation: + with annotate(domain="backward", color="green"): + loss.backward() + # Apply the gradients + opt.step() + p.step() + torch.cuda.synchronize() + end = time.perf_counter() + print(f"Finished step {i} in {end - start:.4f} seconds") + times.append(end - start) + start = time.perf_counter() + + times = torch.tensor(times) + # Drop first and last: + avg_time = times[1:-1].mean() + # compute throughput too: + throughput = cfg["batch_size"] / avg_time + print(f"Average time per iteration: {avg_time:.3f} ({throughput:.3f} examples / s)") + + +@hydra.main(version_base="1.3", config_path="../", config_name="cfg") +def main(config: DictConfig): + + # configure the profiling tools: + p = Profiler() + + for key, val in config.profile.items(): + # This is not the mandatory way to enable tools + # I've set up the config to have the keys match + # the registered profilers. You can do it manually + # too such as `p.enable("torch")` + if val: p.enable(key) + + # The profiler has to be initilized before use. Using it in a context + # will do it automatically, but to use it as a decorator we should do + # it manually here: + + p.initialize() + print(p) + + workload(config) + + + +if __name__ == "__main__": + + main() \ No newline at end of file diff --git a/docs/test_scripts/profiling/optimized_code_2/attn.py b/docs/test_scripts/profiling/optimized_code_2/attn.py new file mode 100644 index 0000000000..a1e0185335 --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_2/attn.py @@ -0,0 +1,103 @@ +import torch +from torch import nn +torch.backends.cuda.enable_flash_sdp(True) +from physicsnemo.utils.profiling import profile, annotate + +class Attention(nn.Module): + """Dummy example Attention mechanism using Flash Attention for efficient computation. + """ + + def __init__( + self, + dim: int, + num_heads: int = 8, + qkv_bias: bool = False, + attn_drop: float = 0., + proj_drop: float = 0., + ) -> None: + super().__init__() + assert dim % num_heads == 0, 'dim should be divisible by num_heads' + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.scale = self.head_dim ** -0.5 + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + B, N, C = x.shape + qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4) + q, k, v = qkv.unbind(0) + + # Use scaled_dot_product_attention with flash attention + x = torch.nn.functional.scaled_dot_product_attention( + q, k, v, + dropout_p=self.attn_drop.p if self.training else 0.0, + is_causal=False + ) + + x = x.transpose(1, 2).reshape(B, N, C) + x = self.proj(x) + x = self.proj_drop(x) + return x + + + +class MLP(nn.Module): + def __init__(self, in_features, hidden_features=None, out_features=None, drop=0): + super().__init__() + hidden_features = hidden_features or in_features + out_features = out_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.gelu = nn.GELU() + self.drop1 = nn.Dropout(drop) + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop2 = nn.Dropout(drop) + + @profile + def forward(self, x): + x = self.fc1(x) + x = self.gelu(x) + x = self.drop1(x) + x = self.fc2(x) + x = self.gelu(x) + x = self.drop2(x) + return x + +class Block(nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + mlp_ratio: float = 4., + qkv_bias: bool = False, + proj_drop: float = 0., + attn_drop: float = 0., + norm_layer: nn.Module = nn.LayerNorm, + mlp_layer: nn.Module = MLP, + ) -> None: + super().__init__() + self.norm1 = norm_layer(dim) + self.attn = Attention( + dim, + num_heads=num_heads, + qkv_bias=qkv_bias, + attn_drop=attn_drop, + proj_drop=proj_drop, + ) + + self.norm2 = norm_layer(dim) + self.mlp = mlp_layer( + in_features=dim, + hidden_features=int(dim * mlp_ratio), + drop=proj_drop, + ) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + self.attn(self.norm1(x)) + x = x + self.mlp(self.norm2(x)) + return x \ No newline at end of file diff --git a/docs/test_scripts/profiling/optimized_code_2/dataset.py b/docs/test_scripts/profiling/optimized_code_2/dataset.py new file mode 100644 index 0000000000..7027190d49 --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_2/dataset.py @@ -0,0 +1,45 @@ +import torch +import numpy as np +from torch.utils.data import Dataset + + +class RandomNoiseDataset(Dataset): + """ + Random normal distribution dataset. + + Mean AND STD of the distribution is set to the index + of the sample requested. + + Length is hardcoded to 64. + + (Don't use this anywhere that isn't an example of how to write non-performant python code!) + + """ + + def __init__(self, image_shape, ): + """ + Arguments: + image_shape (string): Shape of a single example to generate + """ + self.shape = list(image_shape) + + + def __len__(self): + return 256 + + def __getitem__(self, idx): + if torch.is_tensor(idx): + idx = idx.tolist() + + # Generate the raw data: + raw = self.gen_single_image(idx) + + sample = { + 'image' : raw + } + + return sample + + def gen_single_image(self, idx): + + return torch.normal(idx, idx, self.shape, device="cuda" ) diff --git a/docs/test_scripts/profiling/optimized_code_2/workload.py b/docs/test_scripts/profiling/optimized_code_2/workload.py new file mode 100644 index 0000000000..2fedb8b5d2 --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_2/workload.py @@ -0,0 +1,115 @@ +import time + +import torch +torch.backends.cudnn.benchmark = True + +# For hydra: +import hydra +from omegaconf import DictConfig +from pathlib import Path + +# For dataloader +from torch.utils.data import DataLoader + +# Import the dataset: +from dataset import RandomNoiseDataset + +# For the model code: +from attn import Block + +# Import profiling hooks from physicsnemo: +from physicsnemo.utils.profiling import Profiler, profile, annotate + +def loss_fn(output_data): + # All except the first dim: + dims = tuple(range(len(output_data.shape))) + # Just a silly loss function: + output_data = output_data**2. + loss = torch.sum(output_data, dims[1:]) + return loss.mean() + +@profile +def workload(cfg): + + ds = RandomNoiseDataset(cfg["shape"]) + + loader = DataLoader( + ds, + batch_size=cfg["batch_size"], + shuffle = True, + ) + + + # Initialize the model: + model = Block( + dim = cfg["shape"][-1], + num_heads = cfg.model["num_heads"], + qkv_bias = cfg.model["qkv_bias"] , + attn_drop = cfg.model["attn_drop"], + proj_drop = cfg.model["proj_drop"], + ).to("cuda") + + if cfg["train"]: + opt = torch.optim.SGD(model.parameters(), lr=0.0001, momentum=0.9) + + times = [] + with Profiler() as p: + start = time.perf_counter() + for i, batch in enumerate(loader): + image = batch["image"] + image = image.to("cuda") + with torch.amp.autocast(device_type="cuda", dtype=torch.float16): + with annotate(domain="forward", color="blue"): + output = model(image) + if cfg["train"]: + opt.zero_grad() + # Compute the loss: + loss = loss_fn(output) + # Do the gradient calculation: + with annotate(domain="backward", color="green"): + loss.backward() + # Apply the gradients + opt.step() + p.step() + torch.cuda.synchronize() + end = time.perf_counter() + print(f"Finished step {i} in {end - start:.4f} seconds") + times.append(end - start) + start = time.perf_counter() + + times = torch.tensor(times) + # Drop first and last: + avg_time = times[1:-1].mean() + # compute throughput too: + throughput = cfg["batch_size"] / avg_time + print(f"Average time per iteration: {avg_time:.3f} ({throughput:.3f} examples / s)") + + +@hydra.main(version_base="1.3", config_path="../", config_name="cfg") +def main(config: DictConfig): + + # configure the profiling tools: + p = Profiler() + + for key, val in config.profile.items(): + # This is not the mandatory way to enable tools + # I've set up the config to have the keys match + # the registered profilers. You can do it manually + # too such as `p.enable("torch")` + if val: p.enable(key) + + # The profiler has to be initilized before use. Using it in a context + # will do it automatically, but to use it as a decorator we should do + # it manually here: + + + p.initialize() + print(p) + + workload(config) + + + +if __name__ == "__main__": + + main() \ No newline at end of file diff --git a/docs/test_scripts/profiling/optimized_code_3/attn.py b/docs/test_scripts/profiling/optimized_code_3/attn.py new file mode 100644 index 0000000000..a1e0185335 --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_3/attn.py @@ -0,0 +1,103 @@ +import torch +from torch import nn +torch.backends.cuda.enable_flash_sdp(True) +from physicsnemo.utils.profiling import profile, annotate + +class Attention(nn.Module): + """Dummy example Attention mechanism using Flash Attention for efficient computation. + """ + + def __init__( + self, + dim: int, + num_heads: int = 8, + qkv_bias: bool = False, + attn_drop: float = 0., + proj_drop: float = 0., + ) -> None: + super().__init__() + assert dim % num_heads == 0, 'dim should be divisible by num_heads' + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.scale = self.head_dim ** -0.5 + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + B, N, C = x.shape + qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4) + q, k, v = qkv.unbind(0) + + # Use scaled_dot_product_attention with flash attention + x = torch.nn.functional.scaled_dot_product_attention( + q, k, v, + dropout_p=self.attn_drop.p if self.training else 0.0, + is_causal=False + ) + + x = x.transpose(1, 2).reshape(B, N, C) + x = self.proj(x) + x = self.proj_drop(x) + return x + + + +class MLP(nn.Module): + def __init__(self, in_features, hidden_features=None, out_features=None, drop=0): + super().__init__() + hidden_features = hidden_features or in_features + out_features = out_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.gelu = nn.GELU() + self.drop1 = nn.Dropout(drop) + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop2 = nn.Dropout(drop) + + @profile + def forward(self, x): + x = self.fc1(x) + x = self.gelu(x) + x = self.drop1(x) + x = self.fc2(x) + x = self.gelu(x) + x = self.drop2(x) + return x + +class Block(nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + mlp_ratio: float = 4., + qkv_bias: bool = False, + proj_drop: float = 0., + attn_drop: float = 0., + norm_layer: nn.Module = nn.LayerNorm, + mlp_layer: nn.Module = MLP, + ) -> None: + super().__init__() + self.norm1 = norm_layer(dim) + self.attn = Attention( + dim, + num_heads=num_heads, + qkv_bias=qkv_bias, + attn_drop=attn_drop, + proj_drop=proj_drop, + ) + + self.norm2 = norm_layer(dim) + self.mlp = mlp_layer( + in_features=dim, + hidden_features=int(dim * mlp_ratio), + drop=proj_drop, + ) + + @profile + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + self.attn(self.norm1(x)) + x = x + self.mlp(self.norm2(x)) + return x \ No newline at end of file diff --git a/docs/test_scripts/profiling/optimized_code_3/dataset.py b/docs/test_scripts/profiling/optimized_code_3/dataset.py new file mode 100644 index 0000000000..2a7bf54d99 --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_3/dataset.py @@ -0,0 +1,45 @@ +import torch +import numpy as np +from torch.utils.data import Dataset + + +class RandomNoiseDataset(Dataset): + """ + Random normal distribution dataset. + + Mean AND STD of the distribution is set to the index + of the sample requested. + + Length is hardcoded to 64. + + (Don't use this anywhere that isn't an example of how to write non-performant python code!) + + """ + + def __init__(self, image_shape, ): + """ + Arguments: + image_shape (string): Shape of a single example to generate + """ + self.shape = list(image_shape) + + + def __len__(self): + return 256 + + def __getitem__(self, idx): + if torch.is_tensor(idx): + idx = idx.tolist() + + # Generate the raw data: + raw = self.gen_single_image(idx) + + sample = { + 'image' : raw + } + + return sample + + def gen_single_image(self, idx): + + return torch.normal(idx, idx, self.shape, device="cuda", dtype=torch.float16 ) diff --git a/docs/test_scripts/profiling/optimized_code_3/workload.py b/docs/test_scripts/profiling/optimized_code_3/workload.py new file mode 100644 index 0000000000..94a06a3818 --- /dev/null +++ b/docs/test_scripts/profiling/optimized_code_3/workload.py @@ -0,0 +1,117 @@ +import time + +import torch +torch.backends.cudnn.benchmark = True + +# For hydra: +import hydra +from omegaconf import DictConfig +from pathlib import Path + +# For dataloader +from torch.utils.data import DataLoader + +# Import the dataset: +from dataset import RandomNoiseDataset + +# For the model code: +from attn import Block + +# Import profiling hooks from physicsnemo: +from physicsnemo.utils.profiling import Profiler, profile, annotate + +def loss_fn(output_data): + # All except the first dim: + dims = tuple(range(len(output_data.shape))) + # Just a silly loss function: + output_data = output_data**2. + loss = torch.sum(output_data, dims[1:]) + return loss.mean() + +@profile +def workload(cfg): + + ds = RandomNoiseDataset(cfg["shape"]) + + loader = DataLoader( + ds, + batch_size=cfg["batch_size"], + shuffle = True, + ) + + + # Initialize the model: + model = Block( + dim = cfg["shape"][-1], + num_heads = cfg.model["num_heads"], + qkv_bias = cfg.model["qkv_bias"] , + attn_drop = cfg.model["attn_drop"], + proj_drop = cfg.model["proj_drop"], + ).to("cuda") + + model = torch.compile(model) + + if cfg["train"]: + opt = torch.optim.SGD(model.parameters(), lr=0.0001, momentum=0.9) + + times = [] + with Profiler() as p: + start = time.perf_counter() + for i, batch in enumerate(loader): + image = batch["image"] + image = image.to("cuda") + with torch.amp.autocast(device_type="cuda", dtype=torch.float16): + with annotate(domain="forward", color="blue"): + output = model(image) + if cfg["train"]: + opt.zero_grad() + # Compute the loss: + loss = loss_fn(output) + # Do the gradient calculation: + with annotate(domain="backward", color="green"): + loss.backward() + # Apply the gradients + opt.step() + p.step() + torch.cuda.synchronize() + end = time.perf_counter() + print(f"Finished step {i} in {end - start:.4f} seconds") + times.append(end - start) + start = time.perf_counter() + + times = torch.tensor(times) + # Drop first and last: + avg_time = times[1:-1].mean() + # compute throughput too: + throughput = cfg["batch_size"] / avg_time + print(f"Average time per iteration: {avg_time:.3f} ({throughput:.3f} examples / s)") + + +@hydra.main(version_base="1.3", config_path="../", config_name="cfg") +def main(config: DictConfig): + + # configure the profiling tools: + p = Profiler() + + for key, val in config.profile.items(): + # This is not the mandatory way to enable tools + # I've set up the config to have the keys match + # the registered profilers. You can do it manually + # too such as `p.enable("torch")` + if val: p.enable(key) + + # The profiler has to be initilized before use. Using it in a context + # will do it automatically, but to use it as a decorator we should do + # it manually here: + + + p.initialize() + print(p) + + workload(config) + + + +if __name__ == "__main__": + + main() \ No newline at end of file diff --git a/docs/test_scripts/profiling/original_code/attn.py b/docs/test_scripts/profiling/original_code/attn.py new file mode 100644 index 0000000000..aef4c0f0d4 --- /dev/null +++ b/docs/test_scripts/profiling/original_code/attn.py @@ -0,0 +1,103 @@ +import torch +from torch import nn + +class Attention(nn.Module): + """Dummy example Attention mechanism. Meant not for efficienct computation + but to show how to use the profiling tools! + + """ + + def __init__( + self, + dim: int, + num_heads: int = 8, + qkv_bias: bool = False, + attn_drop: float = 0., + proj_drop: float = 0., + ) -> None: + super().__init__() + assert dim % num_heads == 0, 'dim should be divisible by num_heads' + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.scale = self.head_dim ** -0.5 + + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.attn_drop = nn.Dropout(attn_drop) + self.proj = nn.Linear(dim, dim) + self.proj_drop = nn.Dropout(proj_drop) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + + B, N, C = x.shape + qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4) + q, k, v = qkv.unbind(0) + + + # This is not optimal code right here ... + q = q * self.scale + attn = q @ k.transpose(-2, -1) + attn = attn.softmax(dim=-1) + attn = self.attn_drop(attn) + x = attn @ v + + x = x.transpose(1, 2).reshape(B, N, C) + x = self.proj(x) + x = self.proj_drop(x) + return x + + + + +class MLP(nn.Module): + def __init__(self, in_features, hidden_features=None, out_features=None, drop=0): + super().__init__() + hidden_features = hidden_features or in_features + out_features = out_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.gelu = nn.GELU() + self.drop1 = nn.Dropout(drop) + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop2 = nn.Dropout(drop) + + def forward(self, x): + x = self.fc1(x) + x = self.gelu(x) + x = self.drop1(x) + x = self.fc2(x) + x = self.gelu(x) + x = self.drop2(x) + return x + +class Block(nn.Module): + def __init__( + self, + dim: int, + num_heads: int, + mlp_ratio: float = 4., + qkv_bias: bool = False, + proj_drop: float = 0., + attn_drop: float = 0., + norm_layer: nn.Module = nn.LayerNorm, + mlp_layer: nn.Module = MLP, + ) -> None: + super().__init__() + self.norm1 = norm_layer(dim) + self.attn = Attention( + dim, + num_heads=num_heads, + qkv_bias=qkv_bias, + attn_drop=attn_drop, + proj_drop=proj_drop, + ) + + self.norm2 = norm_layer(dim) + self.mlp = mlp_layer( + in_features=dim, + hidden_features=int(dim * mlp_ratio), + drop=proj_drop, + ) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + x = x + self.attn(self.norm1(x)) + x = x + self.mlp(self.norm2(x)) + return x \ No newline at end of file diff --git a/docs/test_scripts/profiling/original_code/dataset.py b/docs/test_scripts/profiling/original_code/dataset.py new file mode 100644 index 0000000000..92eb90479f --- /dev/null +++ b/docs/test_scripts/profiling/original_code/dataset.py @@ -0,0 +1,46 @@ +import torch +import numpy as np +from torch.utils.data import Dataset + + +class RandomNoiseDataset(Dataset): + """ + Random normal distribution dataset. + + Mean AND STD of the distribution is set to the index + of the sample requested. + + Length is hardcoded to 64. + + (Don't use this anywhere that isn't an example of how to write non-performant python code!) + + """ + + def __init__(self, image_shape, ): + """ + Arguments: + image_shape (string): Shape of a single example to generate + """ + self.shape = image_shape + + self.rng = np.random.default_rng() + + def __len__(self): + return 64 + + def __getitem__(self, idx): + if torch.is_tensor(idx): + idx = idx.tolist() + + # Generate the raw data: + raw = self.gen_single_image(idx) + + sample = { + 'image' : raw + } + + return sample + + def gen_single_image(self, idx): + + return self.rng.normal(loc=idx, scale=idx, size=self.shape).astype(np.float32) diff --git a/docs/test_scripts/profiling/original_code/workload.py b/docs/test_scripts/profiling/original_code/workload.py new file mode 100644 index 0000000000..be3ff62875 --- /dev/null +++ b/docs/test_scripts/profiling/original_code/workload.py @@ -0,0 +1,86 @@ +import time + +import torch + +# For hydra: +import hydra +from omegaconf import DictConfig + +# For dataloader +from torch.utils.data import DataLoader + +# Import the dataset: +from dataset import RandomNoiseDataset + +# For the model code: +from attn import Block + +def loss_fn(output_data): + # All except the first dim: + dims = tuple(range(len(output_data.shape))) + # Just a silly loss function: + output_data = output_data**2. + loss = torch.sum(output_data, dims[1:]) + return loss.mean() + +def workload(cfg): + ds = RandomNoiseDataset(cfg["shape"]) + + loader = DataLoader( + ds, + batch_size=cfg["batch_size"], + shuffle = True, + ) + + + # Initialize the model: + model = Block( + dim = cfg["shape"][-1], + num_heads = cfg.model["num_heads"], + qkv_bias = cfg.model["qkv_bias"] , + attn_drop = cfg.model["attn_drop"], + proj_drop = cfg.model["proj_drop"], + ).to("cuda") + + if cfg["train"]: + opt = torch.optim.SGD(model.parameters(), lr=0.0001, momentum=0.9) + + times = [] + start = time.perf_counter() + for i, batch in enumerate(loader): + image = batch["image"] + image = image.to("cuda") + output = model(image) + if cfg["train"]: + opt.zero_grad() + # Compute the loss: + loss = loss_fn(output) + # Do the gradient calculation: + loss.backward() + # Apply the gradients + opt.step() + torch.cuda.synchronize() + end = time.perf_counter() + print(f"Finished step {i} in {end - start:.4f} seconds") + times.append(end - start) + start = time.perf_counter() + + times = torch.tensor(times) + # Drop first and last: + avg_time = times[1:-1].mean() + # compute throughput too: + throughput = cfg["batch_size"] / avg_time + print(f"Average time per iteration: {avg_time:.3f} ({throughput:.3f} examples / s)") + +@hydra.main(version_base="1.3", config_path="../", config_name="cfg") +def main(config: DictConfig): + + + + workload(config) + + + +if __name__ == "__main__": + + main() \ No newline at end of file diff --git a/docs/test_scripts/test_basic.py b/docs/test_scripts/test_basic.py index a0223a3d39..1b9ccada08 100644 --- a/docs/test_scripts/test_basic.py +++ b/docs/test_scripts/test_basic.py @@ -1,16 +1,16 @@ # [imports] import torch -import modulus -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.metrics.general.mse import mse -from modulus.models.fno.fno import FNO +import physicsnemo +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.metrics.general.mse import mse +from physicsnemo.models.fno.fno import FNO # [imports] # [code] -normaliser = { - "permeability": (1.25, 0.75), +normaliser = { # Dictionary with mean and std of the permeability and darcy fields + "permeability": (1.25, 0.75), "darcy": (4.52e-2, 2.79e-2), } dataloader = Darcy2D( @@ -34,11 +34,12 @@ ) # run for 20 iterations +dataloader = iter(dataloader) for i in range(20): - batch = next(iter(dataloader)) - true = batch["darcy"] + batch = next(dataloader) + truth = batch["darcy"] pred = model(batch["permeability"]) - loss = mse(pred, true) + loss = mse(pred, truth) loss.backward() optimizer.step() scheduler.step() diff --git a/docs/test_scripts/test_basic_checkpointing.py b/docs/test_scripts/test_basic_checkpointing.py index 6d5aa0ceb2..d5472b6c81 100644 --- a/docs/test_scripts/test_basic_checkpointing.py +++ b/docs/test_scripts/test_basic_checkpointing.py @@ -1,11 +1,11 @@ # [imports] import torch -import modulus -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.launch.utils import load_checkpoint, save_checkpoint -from modulus.metrics.general.mse import mse -from modulus.models.fno.fno import FNO +import physicsnemo +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.launch.utils import load_checkpoint, save_checkpoint +from physicsnemo.metrics.general.mse import mse +from physicsnemo.models.fno.fno import FNO # [imports] @@ -34,8 +34,9 @@ optimizer, lr_lambda=lambda step: 0.85**step ) -# load the epoch and optimizer, model ans scheduler parameters from the checkpoint if -# it exists +# load the epoch and optimizer, model and scheduler parameters from the checkpoint if +# it exists. Here we will use the `load_checkpoint` function to load the checkpoint, +# optimizer, and scheduler parameters from the checkpoint. loaded_epoch = load_checkpoint( "./checkpoints", models=model, @@ -46,10 +47,11 @@ # we will setup the training to run for 20 epochs each epoch running for 5 iterations # starting with the loaded epoch +dataloader = iter(dataloader) for i in range(max(1, loaded_epoch), 20): # this would be iterations through different batches for _ in range(5): - batch = next(iter(dataloader)) + batch = next(dataloader) true = batch["darcy"] pred = model(batch["permeability"]) loss = mse(pred, true) diff --git a/docs/test_scripts/test_basic_inference.py b/docs/test_scripts/test_basic_inference.py index 1b86c3e6ed..a302435a5c 100644 --- a/docs/test_scripts/test_basic_inference.py +++ b/docs/test_scripts/test_basic_inference.py @@ -1,8 +1,8 @@ # [imports] import torch -import modulus -from modulus.models.fno.fno import FNO +import physicsnemo +from physicsnemo.models.fno.fno import FNO # [imports] @@ -27,7 +27,7 @@ # Inference code # The parameters to instantitate the model will be loaded from the checkpoint -model_inf = modulus.Module.from_checkpoint("untrained_checkpoint.mdlus").to("cuda") +model_inf = physicsnemo.Module.from_checkpoint("untrained_checkpoint.mdlus").to("cuda") # put the model in evaluation mode model_inf.eval() diff --git a/docs/test_scripts/test_console_logger.py b/docs/test_scripts/test_console_logger.py index 49e4866c3b..99ee38331b 100644 --- a/docs/test_scripts/test_console_logger.py +++ b/docs/test_scripts/test_console_logger.py @@ -1,11 +1,11 @@ # [imports] import torch -import modulus -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.launch.logging import LaunchLogger, PythonLogger -from modulus.metrics.general.mse import mse -from modulus.models.fno.fno import FNO +import physicsnemo +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.launch.logging import LaunchLogger, PythonLogger +from physicsnemo.metrics.general.mse import mse +from physicsnemo.models.fno.fno import FNO # [imports] @@ -42,15 +42,16 @@ logger.info("Starting Training!") # we will setup the training to run for 20 epochs each epoch running for 5 iterations +dataloader = iter(dataloader) for i in range(20): # wrap the epoch in launch logger to control frequency of output for console logs with LaunchLogger("train", epoch=i) as launchlog: # this would be iterations through different batches for _ in range(5): - batch = next(iter(dataloader)) - true = batch["darcy"] + batch = next(dataloader) + truth = batch["darcy"] pred = model(batch["permeability"]) - loss = mse(pred, true) + loss = mse(pred, truth) loss.backward() optimizer.step() scheduler.step() diff --git a/docs/test_scripts/test_custom_model_demo_1.py b/docs/test_scripts/test_custom_model_demo_1.py index 59abbfed2d..8de8ba9cfe 100644 --- a/docs/test_scripts/test_custom_model_demo_1.py +++ b/docs/test_scripts/test_custom_model_demo_1.py @@ -5,9 +5,9 @@ # [pytorch model] import torch.nn as nn -import modulus -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.metrics.general.mse import mse +import physicsnemo +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.metrics.general.mse import mse class UNet(nn.Module): @@ -45,14 +45,14 @@ def forward(self, x): # [pytorch model] -# [modulus model] +# [physicsnemo model] from dataclasses import dataclass import torch.nn as nn -from modulus.models.meta import ModelMetaData -from modulus.models.module import Module +from physicsnemo.models.meta import ModelMetaData +from physicsnemo.models.module import Module @dataclass @@ -67,14 +67,14 @@ class MdlsUNetMetaData(ModelMetaData): MdlsUNet = Module.from_torch(UNet, meta=MdlsUNetMetaData) -# [modulus model] +# [physicsnemo model] -# [modulus sym model] +# [physicsnemo sym model] from typing import Dict, Optional -from modulus.sym.key import Key -from modulus.sym.models.arch import Arch +from physicsnemo.sym.key import Key +from physicsnemo.sym.models.arch import Arch class MdlsSymUNet(Arch): @@ -102,14 +102,14 @@ def forward(self, dict_tensor: Dict[str, torch.Tensor]): return self.split_output(out, self.output_key_dict, dim=1) -# [modulus sym model] +# [physicsnemo sym model] # [code] import time -from modulus.utils import StaticCaptureTraining +from physicsnemo.utils import StaticCaptureTraining normaliser = { "permeability": (1.25, 0.75), @@ -142,10 +142,11 @@ def training_step(invar, outvar): # run for 20 iterations +dataloader = iter(dataloader) for i in range(20): - batch = next(iter(dataloader)) - true = batch["darcy"] + batch = next(dataloader) + truth = batch["darcy"] input = batch["permeability"] - loss = training_step(input, true) + loss = training_step(input, truth) scheduler.step() # [code] diff --git a/docs/test_scripts/test_mlflow_logger.py b/docs/test_scripts/test_mlflow_logger.py index 4b63ae9392..aad103b3b3 100644 --- a/docs/test_scripts/test_mlflow_logger.py +++ b/docs/test_scripts/test_mlflow_logger.py @@ -1,11 +1,12 @@ # [imports] import torch -import modulus -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.launch.logging import LaunchLogger, PythonLogger, initialize_mlflow -from modulus.metrics.general.mse import mse -from modulus.models.fno.fno import FNO +import physicsnemo +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.launch.logging import LaunchLogger, PythonLogger +from physicsnemo.launch.logging.mlflow import initialize_mlflow +from physicsnemo.metrics.general.mse import mse +from physicsnemo.models.fno.fno import FNO # [imports] @@ -39,11 +40,11 @@ # Initialize the MLFlow logger initialize_mlflow( - experiment_name="Modulus Tutorials", - experiment_desc="Simple Modulus Tutorials", - run_name="Modulus MLFLow Tutorial", - run_desc="Modulus Tutorial Training", - user_name="Modulus User", + experiment_name="PhysicsNeMo Tutorials", + experiment_desc="Simple PhysicsNeMo Tutorials", + run_name="PhysicsNeMo MLFLow Tutorial", + run_desc="PhysicsNeMo Tutorial Training", + user_name="PhysicsNeMo User", mode="offline", ) LaunchLogger.initialize(use_mlflow=True) @@ -52,14 +53,15 @@ logger.info("Starting Training!") # we will setup the training to run for 20 epochs each epoch running for 5 iterations +dataloader = iter(dataloader) for i in range(20): # wrap the epoch in launch logger to control frequency of output for console logs with LaunchLogger("train", epoch=i) as launchlog: for _ in range(5): - batch = next(iter(dataloader)) - true = batch["darcy"] + batch = next(dataloader) + truth = batch["darcy"] pred = model(batch["permeability"]) - loss = mse(pred, true) + loss = mse(pred, truth) loss.backward() optimizer.step() scheduler.step() diff --git a/docs/test_scripts/test_simple_distributed.py b/docs/test_scripts/test_simple_distributed.py index d3c751bd93..8b9523825b 100644 --- a/docs/test_scripts/test_simple_distributed.py +++ b/docs/test_scripts/test_simple_distributed.py @@ -2,12 +2,12 @@ import torch from torch.nn.parallel import DistributedDataParallel -import modulus -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.distributed import DistributedManager -from modulus.metrics.general.mse import mse -from modulus.models.fno.fno import FNO -from modulus.utils import StaticCaptureTraining +import physicsnemo +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.distributed import DistributedManager +from physicsnemo.metrics.general.mse import mse +from physicsnemo.models.fno.fno import FNO +from physicsnemo.utils import StaticCaptureTraining # [imports] @@ -77,11 +77,12 @@ def training_step(invar, outvar): return loss # run for 20 iterations + dataloader = iter(dataloader) for i in range(20): - batch = next(iter(dataloader)) - true = batch["darcy"] + batch = next(dataloader) + truth = batch["darcy"] input = batch["permeability"] - loss = training_step(input, true) + loss = training_step(input, truth) scheduler.step() diff --git a/docs/test_scripts/test_wandb_logger.py b/docs/test_scripts/test_wandb_logger.py index 787fe7aee8..ecf494a6be 100644 --- a/docs/test_scripts/test_wandb_logger.py +++ b/docs/test_scripts/test_wandb_logger.py @@ -1,11 +1,12 @@ # [imports] import torch -import modulus -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.launch.logging import LaunchLogger, PythonLogger, initialize_wandb -from modulus.metrics.general.mse import mse -from modulus.models.fno.fno import FNO +import physicsnemo +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.launch.logging import LaunchLogger, PythonLogger +from physicsnemo.launch.logging.wandb import initialize_wandb +from physicsnemo.metrics.general.mse import mse +from physicsnemo.models.fno.fno import FNO # [imports] @@ -39,9 +40,9 @@ # Initialize the MLFlow logger initialize_wandb( - project="Modulus Tutorials", - name="Simple Modulus Tutorials", - entity="Modulus MLFLow Tutorial", + project="PhysicsNeMo Tutorials", + name="Simple PhysicsNeMo Tutorials", + entity="PhysicsNeMo MLFLow Tutorial", mode="offline", ) LaunchLogger.initialize(use_wandb=True) @@ -50,15 +51,16 @@ logger.info("Starting Training!") # we will setup the training to run for 20 epochs each epoch running for 10 iterations +dataloader = iter(dataloader) for i in range(20): # wrap the epoch in launch logger to control frequency of output for console logs with LaunchLogger("train", epoch=i) as launchlog: # this would be iterations through different batches for _ in range(10): - batch = next(iter(dataloader)) - true = batch["darcy"] + batch = next(dataloader) + truth = batch["darcy"] pred = model(batch["permeability"]) - loss = mse(pred, true) + loss = mse(pred, truth) loss.backward() optimizer.step() scheduler.step() diff --git a/docs/tutorials/simple_logging_and_checkpointing.rst b/docs/tutorials/simple_logging_and_checkpointing.rst deleted file mode 100644 index e000d284e8..0000000000 --- a/docs/tutorials/simple_logging_and_checkpointing.rst +++ /dev/null @@ -1,238 +0,0 @@ -Simple Logging and Checkpointing recipe -======================================== - -Logging and checkpointing are important comonents of model training workflow. It allows -users to keep a record of the model hyper-parameters and its performance on training. - -In this tutorial we will look at some of the utilities from Modulus to simplify this -important aspect of model training. - -Logging in Modulus -------------------- - -Modulus provides utilities to standardize the logs of different training runs. Using the -logging utilites from Modulus, you would have the flexibility of choosing between the -good-old console logging to more advanced ML experiments trackers like MLFlow and -Weights & Biases. You can always implement these loggers yourself, but in this example, -we will use the utilites from Modulus that will not only simplify this process but also -provide a standardized output format. Let's get started. - -Console logging -^^^^^^^^^^^^^^^^^ - -The below example shows a simple setup using the console logging. - -.. literalinclude:: ../test_scripts/test_console_logger.py - :language: python - :start-after: [imports] - :end-before: [imports] - -.. literalinclude:: ../test_scripts/test_console_logger.py - :language: python - :start-after: [code] - :end-before: [code] - -The logger output can be seen below. - -.. code-block:: bash - - Warp 0.10.1 initialized: - CUDA Toolkit: 11.5, Driver: 12.2 - Devices: - "cpu" | x86_64 - "cuda:0" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:1" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:2" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:3" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:4" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:5" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:6" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:7" | Tesla V100-SXM2-16GB-N (sm_70) - Kernel cache: /root/.cache/warp/0.10.1 - /usr/local/lib/python3.10/dist-packages/pydantic/_internal/_fields.py:128: UserWarning: Field "model_server_url" has conflict with protected namespace "model_". - - You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`. - warnings.warn( - /usr/local/lib/python3.10/dist-packages/pydantic/_internal/_config.py:317: UserWarning: Valid config keys have changed in V2: - * 'schema_extra' has been renamed to 'json_schema_extra' - warnings.warn(message, UserWarning) - [21:23:57 - main - INFO] Starting Training! - Module modulus.datapipes.benchmarks.kernels.initialization load on device 'cuda:0' took 73.06 ms - Module modulus.datapipes.benchmarks.kernels.utils load on device 'cuda:0' took 314.91 ms - Module modulus.datapipes.benchmarks.kernels.finite_difference load on device 'cuda:0' took 149.86 ms - [21:24:02 - train - INFO] Epoch 0 Metrics: Learning Rate = 4.437e-03, Loss = 1.009e+00 - [21:24:02 - train - INFO] Epoch Execution Time: 5.664e+00s, Time/Iter: 1.133e+03ms - [21:24:06 - train - INFO] Epoch 1 Metrics: Learning Rate = 1.969e-03, Loss = 6.040e-01 - [21:24:06 - train - INFO] Epoch Execution Time: 4.013e+00s, Time/Iter: 8.025e+02ms - ... - [21:25:32 - train - INFO] Epoch 19 Metrics: Learning Rate = 8.748e-10, Loss = 1.384e-01 - [21:25:32 - train - INFO] Epoch Execution Time: 4.010e+00s, Time/Iter: 8.020e+02ms - [21:25:32 - main - INFO] Finished Training! - - -MLFlow logging -^^^^^^^^^^^^^^^^^ - -The below example shows a simple setup using the MLFlow logging. The only difference from -the previous example is that, we will use ``initialize_mlflow`` function to initialize -the MLFlow client and also set ``use_mlflow=True`` when initializing the ``LaunchLogger``. - -.. literalinclude:: ../test_scripts/test_mlflow_logger.py - :language: python - :start-after: [imports] - :end-before: [imports] - -.. literalinclude:: ../test_scripts/test_mlflow_logger.py - :language: python - :start-after: [code] - :end-before: [code] - -During the run, you will notice a directory named as ``mlruns_0`` created which stores -the mlflow logs. To visulaize the logs interactively, you can run the following: - -.. code-block:: bash - - mlflow ui --backend-store-uri mlruns_0/ - -And then navigate to localhost:5000 in your favorite browser. - -.. warning:: - - Currently the MLFlow logger will log the output of each processor separately. So in - multi-processor runs, you will see multiple directories being created. This is a known - issue and will be fixed in the future releases. - - -Weight and Biases logging -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The below example shows a simple setup using the Weights and Biases logging. The only -difference from the previous example is that, we will use ``initialize_wandb`` function -to initialize the Weights and Biases logger and also set ``use_wandb=True`` when -initializing the ``LaunchLogger``. - -.. literalinclude:: ../test_scripts/test_wandb_logger.py - :language: python - :start-after: [imports] - :end-before: [imports] - -.. literalinclude:: ../test_scripts/test_wandb_logger.py - :language: python - :start-after: [code] - :end-before: [code] - -During the run, you will notice a directory named as ``wandb`` created which stores -the wandb logs. - -The logger output can also be seen below. - -.. code-block:: bash - - Warp 0.10.1 initialized: - CUDA Toolkit: 11.5, Driver: 12.2 - Devices: - "cpu" | x86_64 - "cuda:0" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:1" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:2" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:3" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:4" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:5" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:6" | Tesla V100-SXM2-16GB-N (sm_70) - "cuda:7" | Tesla V100-SXM2-16GB-N (sm_70) - Kernel cache: /root/.cache/warp/0.10.1 - /usr/local/lib/python3.10/dist-packages/pydantic/_internal/_fields.py:128: UserWarning: Field "model_server_url" has conflict with protected namespace "model_". - - You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`. - warnings.warn( - /usr/local/lib/python3.10/dist-packages/pydantic/_internal/_config.py:317: UserWarning: Valid config keys have changed in V2: - * 'schema_extra' has been renamed to 'json_schema_extra' - warnings.warn(message, UserWarning) - wandb: Tracking run with wandb version 0.15.12 - wandb: W&B syncing is set to `offline` in this directory. - wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing. - [21:26:38 - main - INFO] Starting Training! - Module modulus.datapipes.benchmarks.kernels.initialization load on device 'cuda:0' took 74.11 ms - Module modulus.datapipes.benchmarks.kernels.utils load on device 'cuda:0' took 310.06 ms - Module modulus.datapipes.benchmarks.kernels.finite_difference load on device 'cuda:0' took 151.24 ms - [21:26:48 - train - INFO] Epoch 0 Metrics: Learning Rate = 1.969e-03, Loss = 7.164e-01 - [21:26:48 - train - INFO] Epoch Execution Time: 9.703e+00s, Time/Iter: 9.703e+02ms - ... - [21:29:47 - train - INFO] Epoch 19 Metrics: Learning Rate = 7.652e-17, Loss = 3.519e-01 - [21:29:47 - train - INFO] Epoch Execution Time: 1.125e+01s, Time/Iter: 1.125e+03ms - [21:29:47 - main - INFO] Finished Training! - wandb: Waiting for W&B process to finish... (success). - wandb: - wandb: Run history: - wandb: epoch ▁▁▂▂▂▃▃▄▄▄▅▅▅▆▆▇▇▇██ - wandb: train/Epoch Time (s) ▃▁▃▃▃▃▁█▁▁▁▃▃▃▃▆▁▃▃▆ - wandb: train/Learning Rate █▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ - wandb: train/Loss █▁▂▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃ - wandb: train/Time per iter (ms) ▃▁▃▃▃▃▁█▁▁▁▃▃▃▃▆▁▃▃▆ - wandb: - wandb: Run summary: - wandb: epoch 19 - wandb: train/Epoch Time (s) 11.24806 - wandb: train/Learning Rate 0.0 - wandb: train/Loss 0.35193 - wandb: train/Time per iter (ms) 1124.80645 - wandb: - wandb: You can sync this run to the cloud by running: - wandb: wandb sync /workspace/modulus/docs/test_scripts/wandb/wandb/offline-run-20231115_212638-ib4ylq4e - wandb: Find logs at: ./wandb/wandb/offline-run-20231115_212638-ib4ylq4e/logs - - -To visulaize the logs interactively, simply follow the instructions printed in the outputs. - - -Checkpointing in Modulus --------------------------- - -Modulus provides easy utilities to save and load the checkpoints of the model, optimizer, -scheduler, and scaler during training and inference. Similar to logging, custom -implementation can be used, but in this example we will see the utilites from Modulus and -some of its benefits. - -Loading and saving checkpoints during training -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The example below shows how you can save and load a checkpoint during training. The implementation -allows the model training to be resumed from the last saved checkpoint. Here, we will -demonstrate the use of ``load_checkpoint`` and the ``save_checkpoint`` functions. - -.. literalinclude:: ../test_scripts/test_basic_checkpointing.py - :language: python - :start-after: [imports] - :end-before: [imports] - -.. literalinclude:: ../test_scripts/test_basic_checkpointing.py - :language: python - :start-after: [code] - :end-before: [code] - -The output of the above script when loaded from a partially trained model will be -something like below. - -.. code-block:: bash - - >>> python test_scripts/test_basic_checkpointing.py - ... - [23:11:09 - checkpoint - INFO] Loaded model state dictionary /workspace/release_23.11/docs_upgrade/modulus/docs/checkpoints/FourierNeuralOperator.0.10.mdlus to device cuda - [23:11:09 - checkpoint - INFO] Loaded checkpoint file /workspace/release_23.11/docs_upgrade/modulus/docs/checkpoints/checkpoint.0.10.pt to device cuda - [23:11:09 - checkpoint - INFO] Loaded optimizer state dictionary - [23:11:09 - checkpoint - INFO] Loaded scheduler state dictionary - ... - [23:11:11 - checkpoint - INFO] Saved model state dictionary: /workspace/release_23.11/docs_upgrade/modulus/docs/checkpoints/FourierNeuralOperator.0.10.mdlus - [23:11:12 - checkpoint - INFO] Saved training checkpoint: /workspace/release_23.11/docs_upgrade/modulus/docs/checkpoints/checkpoint.0.10.pt - [23:11:16 - checkpoint - INFO] Saved model state dictionary: /workspace/release_23.11/docs_upgrade/modulus/docs/checkpoints/FourierNeuralOperator.0.15.mdlus - [23:11:16 - checkpoint - INFO] Saved training checkpoint: /workspace/release_23.11/docs_upgrade/modulus/docs/checkpoints/checkpoint.0.15.pt - [23:11:21 - checkpoint - INFO] Saved model state dictionary: /workspace/release_23.11/docs_upgrade/modulus/docs/checkpoints/FourierNeuralOperator.0.20.mdlus - [23:11:21 - checkpoint - INFO] Saved training checkpoint: /workspace/release_23.11/docs_upgrade/modulus/docs/checkpoints/checkpoint.0.20.pt - - -Loading checkpoints during inference -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For loading the checkpoint in inference, the process is simple and you can refer the samples -provided in :ref:`running-inference-on-trained-models` and :ref:`saving-and-loading-modulus-models` . - diff --git a/docs/tutorials/simple_training_example.rst b/docs/tutorials/simple_training_example.rst deleted file mode 100644 index 948467a3a8..0000000000 --- a/docs/tutorials/simple_training_example.rst +++ /dev/null @@ -1,204 +0,0 @@ -Simple Training and Inference recipe -===================================== - -In this tutorial, we will see how to use utilites from Modulus to setup a simple model -training pipeline. Once the initial setup is complete, we will look into optimizing -the training loop, and also run it in a distributed fashion. -We will finish the tutorial with an inference workflow that will demonstrate how to use -Modulus models in inference. - - -Basic Training workflow ------------------------- - -Let's get started. For the purposes of this tutorial, we will focus more on the Modulus -utilities and not the correctness of the problem definition or the results. A typical -training workflow requires data, a trainable model and an optimizer to update the model -parameters. - - -Using built-in models -^^^^^^^^^^^^^^^^^^^^^^ - -In this example, we will look at different ways one can interact with Models in Modulus. -Modulus presents a library of models suitable for Physics-ML applications for you to -use directly in your training workflows. In this tutorial we will see how to use a -simple model in Modulus to setup a data-driven training. Using the models from Modulus -will enable us to use various other Modulus features like optimization and -quality-of-life functionalites like checkpointing and model entrypoints. - -Later we will also see how to customize these models in Modulus. - -In this example we will use the -FNO model from Modulus. To demonstrate the training using this model, we would need some -dataset to train the model. To allow for fast prototyping of models, Modulus provides -a set of benchmark datasets that can be used out of the box without the need to setup -data-loading pipelines. In this example, we will use one such datapipe called `Darcy2D` -to get the training data. - -Let's start with importing a few utils and packages. - -.. literalinclude:: ../test_scripts/test_basic.py - :language: python - :start-after: [imports] - :end-before: [imports] - -In this example we want to develop a mapping between the permeability and its subsequent -pressure field for a given forcing function. Refer :ref:`Modulus Datapipes` for -additional details. - -Then a simple training loop for this example can be written as follows: - -.. literalinclude:: ../test_scripts/test_basic.py - :language: python - :start-after: [code] - :end-before: [code] - -That's it! This shows how to use a model from Modulus. Most of the models in Modulus are -highly configurable allowing you to use them out-of-the-box for different applications. -Refer :ref:`Modulus Models` for a more complete list of available models. - -Using custom models in Modulus -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Modulus provides a lot of pre-built optimized models. However, -there might be times where the shipped models might not serve your application. In such -cases, you can easily write your own models and have them interact with the other Modulus -utilites and features. Modulus uses PyTorch in the backend and most Modulus models are, -at the core, PyTorch models. In this section we will see how to go from a typical PyTorch -model to a Modulus model. - -Let's get started with the same application of Darcy problem. Let's write a simple UNet -to solve the problem. A simple PyTorch model for a UNet can be written as shown below: - -.. literalinclude:: ../test_scripts/test_custom_model_demo_1.py - :language: python - :start-after: [pytorch model] - :end-before: [pytorch model] - -Let's now convert this to a Modulus Model. Modulus provides ``Module`` class that is -designed to be a drop-in replacement for the ``torch.nn.module``. Along with that, you -need to also pass a ``MetaData`` that captures the optimizations and other features -supported by the model. Using the ``Module`` subclass allows using these optimizations, -and other features like checkpointing etc. from Modulus. - -Thus, converting a PyTorch model to a Modulus model is very simple. For the above model, -the diff would look something like below: - -.. code-block:: diff - - - import torch.nn as nn - + from dataclasses import dataclass - + from modulus.models.meta import ModelMetaData - + from modulus.models.module import Module - - - class UNet(nn.Module): - + @dataclass - + class MetaData(ModelMetaData): - + name: str = "UNet" - + # Optimization - + jit: bool = False - + cuda_graphs: bool = True - + amp_cpu: bool = True - + amp_gpu: bool = True - + - + class UNet(Module): - def __init__(self, in_channels=1, out_channels=1): - - super(UNet, self).__init__() - + super(UNet, self).__init__(meta=MetaData()) - - self.enc1 = self.conv_block(in_channels, 64) - self.enc2 = self.conv_block(64, 128) - - -With simple changes like this you can convert a PyTorch model to a Modulus Model! - -.. note:: - - The optimizations are not automatically applied. The user is responsible for writing - the model with the optimizations supported. However, if the models supports the - optimization and the same is captured in the MetaData, then the downstream features - will work out-of-the-box. - -.. note:: - - For utilizing the checkpointing functionality of Modulus, the Model instantiation - arguments must be json serializable. - - -You can also use a Modulus model as a standard PyTorch model as they are interoperable. - - -Let's say you don't want to make changes to the code, but you have a PyTorch model -already. You can convert it to a Modulus model by using the ``modulus.Module.from_torch`` -method. This is described in detail in :ref:`modulus-models-from-torch`. - -.. literalinclude:: ../test_scripts/test_custom_model_demo_1.py - :language: python - :start-after: [modulus model] - :end-before: [modulus model] - - -And just like that you can use your existing PyTorch model as a Modulus Model. -A very similar process can be followed to convert a Modulus model to a Modulus Sym model -so that you can use the Constraints and other defitions from the Modulus Sym repository. -Here you will use the ``Arch`` class from Modulus Sym that provides utilites and methods -to go from a tensor data to a dict format which Modulus Sym uses. - -.. literalinclude:: ../test_scripts/test_custom_model_demo_1.py - :language: python - :start-after: [modulus sym model] - :end-before: [modulus sym model] - - -Optimized Training workflow ----------------------------- - -Once we have a model defined in the Modulus style, we can use the optimizations -like AMP, CUDA Graphs, and JIT using the ``modulus.utils.StaticCaptureTraining`` decorator. -This decorator will capture the training step function and optimize it for the specified -optimizations. - -.. note:: - The ``StaticCaptureTraining`` decorator is still under development and may be - refactored in the future. - - -.. literalinclude:: ../test_scripts/test_custom_model_demo_1.py - :language: python - :start-after: [code] - :end-before: [code] - - -Distributed Training workflow ------------------------------- - -Modulus has several Distributed utilites to simplify the implementation of parallel training -and make inference scripts easier by providing a unified way to configure and query parameters -associated with distributed environment. - -In this example, we will see how to convert our existing workflow to use data-parallelism. -For an deep-dive on Modulus Distributed utilities, refer :ref:`Modulus Distributed`. - - -.. literalinclude:: ../test_scripts/test_simple_distributed.py - :language: python - :start-after: [code] - :end-before: [code] - - - -.. _running-inference-on-trained-models: - -Running infernece on trained models ------------------------------------- - -Running inference on trained model is simple! This is shown by the code below. - -.. literalinclude:: ../test_scripts/test_basic_inference.py - :language: python - :start-after: [code] - :end-before: [code] - -The static capture and distributed utilities can also be used during inference for -speeding up the inference workflow, but that is out of the scope for this tutorial. \ No newline at end of file diff --git a/examples/.markdownlint.yaml b/examples/.markdownlint.yaml index e190fcb25b..9655e0dc2d 100644 --- a/examples/.markdownlint.yaml +++ b/examples/.markdownlint.yaml @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 0000000000..f210eb2ba6 --- /dev/null +++ b/examples/README.md @@ -0,0 +1,130 @@ + +# NVIDIA PhysicsNeMo Examples + +## Introduction + +This repository provides sample applications demonstrating use of specific Physics-ML +model architectures that are easy to train and deploy. These examples aim to show how +such models can help solve real world problems. + +## Introductory examples for learning key ideas + +|Use case|Concepts covered| +| --- | --- | +|[Darcy Flow](./cfd/darcy_fno/)|Introductory example for learning basics of data-driven models on Physics-ML datasets| +|[Darcy Flow (Data + Physics)](./cfd/darcy_physics_informed/)|Data-driven training with physics-based constraints| +|[Lid Driven Cavity Flow](./cfd/ldc_pinns/)|Purely physics-driven (no external simulation/experimental data) training| +|[Vortex Shedding](./cfd/vortex_shedding_mgn/)|Introductory example for learning the basics of MeshGraphNets in PhysicsNeMo| +|[Medium-range global weather forecast using FCN-AFNO](./weather/fcn_afno/)|Introductory example on training data-driven models for global weather forecasting (auto-regressive model)| +|[Lagrangian Fluid Flow](./cfd/lagrangian_mgn/)|Introductory example for data-driven training on Lagrangian meshes| +|[Stokes Flow (Physics Informed Fine-Tuning)](./cfd/stokes_mgn/)|Data-driven training followed by physics-based fine-tuning| + +## Domain-specific examples + +The several examples inside PhysicsNeMo can be classified based on their domains as below: + +> **NOTE:** The below classification is not exhaustive by any means! + One can classify single example into multiple domains and we encourage + the users to review the entire list. + +> **NOTE:** * Indicates externally contributed examples. + +### CFD + +|Use case|Model|Transient| +| --- | --- | --- | +|[Drag prediction - External Aero](./cfd/external_aerodynamics/)|MeshGraphNet, UNet, DoMINO, FigConvNet, Transolver|NO| +|[Drag prediction - External Aero - Mixture of Experts](./cfd/external_aerodynamics/)|MoE Model|NO| +|[Navier-Stokes Flow](./cfd/navier_stokes_rnn/)|RNN|YES| +|[Gray-Scott System](./cfd/gray_scott_rnn/)|RNN|YES| +|[Lagrangian Fluid Flow](./cfd/lagrangian_mgn/)|MeshGraphNet|YES| +|[Darcy Flow (Data + Physics Driven) using DeepONet approach](./cfd/darcy_physics_informed/)|FNO (branch) and MLP (trunk)|NO| +|[Darcy Flow (Data + Physics Driven) using PINO approach (Numerical gradients)](./cfd/darcy_physics_informed/)|FNO|NO| +|[Magnetohydrodynamics using PINO (Data + Physics Driven)*](./cfd/mhd_pino/)|FNO|YES| +|[Shallow Water Equations using PINO (Data + Physics Driven)*](./cfd/swe_nonlinear_pino/)|FNO|YES| +|[Shallow Water Equations using Distributed GNNs](./cfd/swe_distributed_gnn/)|GraphCast|YES| +|[Vortex Shedding with Temporal Attention](./cfd/vortex_shedding_mesh_reduced/)|MeshGraphNet|YES| +|[Data Center Airflow](./cfd/datacenter/)|3D UNet|NO| +|[Fluid Super-resolution*](./cfd/flow_reconstruction_diffusion/)|Denoising Diffusion Probablistic Model|YES| +|[Pre-trained DPOT for Navier-Stokes*](./cfd/navier_stokes_dpot/)|Denoising Operator Transformer|YES| +|[Fine-tuning of DoMINO NIM](./cfd/external_aerodynamics/domino_nim_finetuning/)|DoMINO|NO| +|[Transolver for External Aerodynamics on Irregular Meshes](./cfd/external_aerodynamics/transolver/)|Transolver|NO| + + +### Weather + +|Use case|Model| +| --- | --- | +|[Medium-range global weather forecast using FCN-SFNO](https://github.com/NVIDIA/modulus-makani)|FCN-SFNO| +|[Medium-range global weather forecast using GraphCast](./weather/graphcast/)|GraphCast| +|[Medium-range and S2S global weather forecast using DLWP](./weather/dlwp/)|DLWP| +|[Coupled Ocean-Atmosphere Medium-range and S2S global weather forecast using DLWP-HEALPix](./weather/dlwp_healpix/)|DLWP-HEALPix| +|[Diagonistic (Precipitation) model using AFNO](./weather/diagnostic/)|AFNO| +|[Unified Recipe for training several Global Weather Forecasting models](./weather/unified_recipe/)|AFNO, FCN-SFNO, GraphCast| +|[Generative Correction Diffusion Model for Km-scale Atmospheric Downscaling](./weather/corrdiff/)|CorrDiff| +|[StormCast: Generative Diffusion Model for Km-scale, Convection allowing Model Emulation](./weather/stormcast/)|StormCast| +|[Medium-range global weather forecast using Mixture of Experts](./weather/mixture_of_experts/)|MoE Model| +|[Generative Data Assimilation of Sparse Weather Observations](./weather/regen/)|Denoising Diffusion Model| +|[Flood Forecasting](./weather/flood_modeling/)|GNN + KAN| +|[Temporal Interpolation of Weather Forecasts](./weather/temporal_interpolation/)|ModAFNO| + +### Structural Mechanics + +|Use case|Model| +| --- | --- | +|[Deforming Plate](./structural_mechanics/deforming_plate/)|MeshGraphNet| +|[Machine Learning Surrogates for Automotive Crash Dynamics](./structural_mechanics/crash)|Transolver, MeshGraphNet| + +### Healthcare + +|Use case|Model| +| --- | --- | +|[Cardiovascular Simulations*](./healthcare/bloodflow_1d_mgn/)|MeshGraphNet| +|[Brain Anomaly Detection](./healthcare/brain_anomaly_detection/)|FNO| + +### Additive Manufacturing + +|Use case|Model| +| --- | --- | +|[Metal Sintering Simulation*](./additive_manufacturing/sintering_physics/)|MeshGraphNet| + +### Molecular Dymanics + +|Use case|Model| +| --- | --- | +|[Force Prediciton for Lennard Jones system](./molecular_dynamics/lennard_jones/)|MeshGraphNet| + +### Geophysics + +|Use case|Model| +| --- | --- | +|[Diffusion model for full-waveform inversion](./geophysics/diffusion_fwi/)|UNet, Global Filter Net| +|[Reservoir Simulation using X-MeshGraphNet](./reservoir_simulation/)|MeshGraphNet| + +### Generative + +|Use case|Model| +| --- | --- | +|[TopoDiff*](./generative/topodiff)|Conditional diffusion-model| + +### Active Learning + +1. [Classify the famous two-moons data distribution using Active learning](./active_learning/moons/) + +## Additional examples + +In addition to the examples in this repo, more Physics-ML usecases and examples +can be referenced from the [PhysicsNeMo-Sym examples](https://github.com/NVIDIA/physicsnemo-sym/blob/main/examples/README.md). + +## NVIDIA support + +In each of the example READMEs, we indicate the level of support that will be provided. +Some examples are under active development/improvement and might involve rapid changes. +For stable examples, please refer the tagged versions. + +## Feedback / Contributions + +We're posting these examples on GitHub to better support the community, facilitate +feedback, as well as collect and implement contributions using +[GitHub issues](https://github.com/NVIDIA/physicsnemo/issues) and pull requests. +We welcome all contributions! diff --git a/examples/active_learning/moons/.gitignore b/examples/active_learning/moons/.gitignore new file mode 100644 index 0000000000..d535416f2e --- /dev/null +++ b/examples/active_learning/moons/.gitignore @@ -0,0 +1 @@ +active_learning_logs/ \ No newline at end of file diff --git a/examples/active_learning/moons/README.md b/examples/active_learning/moons/README.md new file mode 100644 index 0000000000..f7dc64b3ff --- /dev/null +++ b/examples/active_learning/moons/README.md @@ -0,0 +1,286 @@ +# Moons with Active Learning + +This example is intended to give a quick, high level overview of one kind of active learning +experiments that can be put together using the `physicsnemo` active learning modules and +protocols. + +The experiment that is being done in `moon_example.py` is to use a simple MLP classifier +to label 2D coordinates from the famous two-moons data distribution. The platform for +the experiment is to initially show the MLP a minimal set of data (with some class imbalance), +and use the prediction uncertainties from the model to query points that will be the most +informative to it. + +The main thing to monitor in this experiment is the `f1_metrology.json` output, which is +a product of the `F1Metrology` strategy: in here, we compute precision/recall/F1 values +as a function of the number of active learning cycles. Due to class imbalance, the initial +precision will be quite poor as the predictions will heavily bias towards false negatives, +but as more samples (chosen by `ClassifierUQQuery`) are added to the training set, the +precision and subsequently F1 scores will improve. + +## Quick Start + +To run this example: + +```bash +python moon_example.py +``` + +This will create an `active_learning_logs//` directory containing: + +- **Model checkpoints**: `.mdlus` files saved according to `checkpoint_interval` +- **Driver logs**: `driver_log.json` tracking the active learning process +- **Metrology outputs**: `f1_metrology.json` with precision/recall/F1 scores over iterations +- **Console logs**: `console.log` with detailed execution logs + +The `` is an 8-character UUID prefix that uniquely identifies each run. You can +specify a custom `run_id` in `DriverConfig` if needed. + +## Implementation notes + +To illustrate a simple active learning process, this example implements the bare necessary +ingredients: + +1. **Training step logic** - `training_step` function defines the per-batch training logic, +computing the loss from model predictions. This is passed to the `Driver` which uses it +within the training loop. + +2. **Training loop** - The example uses `DefaultTrainingLoop`, a built-in training loop +that handles epoch iteration, progress bars, validation, and static capture optimizations. +For reference, a custom `training_loop` function is also defined in the example but not used, +showing how you could implement your own if needed. + +3. **Query strategy** - `moon_strategies.ClassifierUQQuery` uses the classifier uncertainty +to rank data indices from the full (not in training) sample set, selecting points where +the model is most uncertain (predictions closest to 0.5). + +4. **Label strategy** - `DummyLabelStrategy` handles obtaining data labels. +Because ground truths are already known for this dataset, it's essentially a +no-op but the `Driver` pipeline relies on it to append labeled data to the +training set. + +5. **Metrology strategy** - `F1Metrology` computes precision/recall/F1 scores and serializes +them to JSON. This makes it easy to track how model performance improves with each active +learning iteration, helping inform hyperparameter choices for future experiments. + +## Configuration + +The rest is configuration: we take the components we've written, and compose them in +the various configuration dataclasses in `moon_example.py::main`. + +### TrainingConfig + +The `train_datapool` specifies what set of data to train on. We configure the training +loop using `DefaultTrainingLoop` with progress bars enabled. The `OptimizerConfig` specifies +which optimizer to use and its hyperparameters. You can configure different epoch counts +for initial training vs. subsequent fine-tuning iterations. + +```python +# configure how training/fine-tuning is done within active learning +training_config = c.TrainingConfig( + train_datapool=dataset, + optimizer_config=c.OptimizerConfig( + torch.optim.SGD, + optimizer_kwargs={"lr": 0.01}, + ), + # configure different times for initial training and subsequent + # fine-tuning + max_training_epochs=10, + max_fine_tuning_epochs=5, + # this configures the training loop + train_loop_fn=DefaultTrainingLoop( + use_progress_bars=True, + ), +) +``` + +**Key options:** + +- `max_training_epochs`: Epochs for initial training (step 0) +- `max_fine_tuning_epochs`: Epochs for subsequent fine-tuning steps +- `DefaultTrainingLoop(use_progress_bars=True)`: Built-in loop with tqdm progress bars +- `val_datapool`: Optional validation dataset (not used in this example) + +### StrategiesConfig + +The `StrategiesConfig` localizes all of the different active learning components +into one place. The `queue_cls` is used to pipeline query samples to label processes. +Because we're carrying out a single process workflow, `queue.Queue` is sufficient, +but multiprocess variants, up to constructs like Redis Queue, can be used to pass +data around the pipeline. + +```python +strategy_config = c.StrategiesConfig( + query_strategies=[ClassifierUQQuery(max_samples=10)], + queue_cls=queue.Queue, + label_strategy=DummyLabelStrategy(), + metrology_strategies=[F1Metrology()], +) +``` + +**Key components:** + +- `query_strategies`: List of strategies for selecting samples (can have multiple) +- `queue_cls`: Queue implementation for passing data between phases (e.g., `queue.Queue`) +- `label_strategy`: Single strategy for labeling queried samples +- `metrology_strategies`: List of strategies for measuring model performance +- `unlabeled_datapool`: Optional pool of unlabeled data for query strategies (not shown here) + +### DriverConfig + +Finally, the `DriverConfig` specifies orchestration parameters that control the overall +active learning loop execution: + +```python +driver_config = c.DriverConfig( + batch_size=16, + max_active_learning_steps=70, + fine_tuning_lr=0.005, + device=torch.device("cpu"), # set to other accelerators if needed +) +driver = Driver( + config=driver_config, + learner=uq_model, + strategies_config=strategy_config, + training_config=training_config, +) +# our model doesn't implement a `training_step` method but in principle +# it could be implemented, and we wouldn't need to pass the step function here +driver(train_step_fn=training_step) +``` + +**Key parameters:** + +- `batch_size`: Batch size for training and validation dataloaders +- `max_active_learning_steps`: Total number of active learning iterations +- `fine_tuning_lr`: Learning rate to switch to after the first AL step (optional) +- `device`: Device for computation (e.g., `torch.device("cpu")`, `torch.device("cuda:0")`) +- `dtype`: Data type for tensors (defaults to `torch.get_default_dtype()`) +- `skip_training`: Set to `True` to skip training phase (default: `False`) +- `skip_metrology`: Set to `True` to skip metrology phase (default: `False`) +- `skip_labeling`: Set to `True` to skip labeling phase (default: `False`) +- `checkpoint_interval`: Save model every N steps (default: 1, set to 0 to disable) +- `root_log_dir`: Directory for logs and checkpoints (default: `"active_learning_logs"`) +- `dist_manager`: Optional `DistributedManager` for multi-GPU training + +### Running the Driver + +The final `driver(...)` call is syntactic sugar for `driver.run(...)`, which executes the +full active learning loop. The `train_step_fn` argument provides the per-batch training logic. + +**Two ways to provide training logic:** + +1. **Pass as function** (shown in example): + + ```python + driver(train_step_fn=training_step) + ``` + +1. **Implement in model** (alternative): + + ```python + class MLP(Module): + def training_step(self, data): + # training logic here + ... + + driver() # no train_step_fn needed + ``` + +**Optional validation step:** + +You can also provide a `validate_step_fn` parameter: + +```python +driver(train_step_fn=training_step, validate_step_fn=validation_step) +``` + +### Active Learning Workflow + +Under the hood, `Driver.active_learning_step` is called repeatedly for the number of +iterations specified in `max_active_learning_steps`. Each iteration follows this sequence: + +1. **Training Phase**: Train model on current `train_datapool` using the +training loop +2. **Metrology Phase**: Compute performance metrics via metrology strategies +3. **Query Phase**: Select new samples to label via query strategies → +`query_queue` +4. **Labeling Phase**: Label queued samples via label strategy → `label_queue` +→ append to `train_datapool` + +The logic for each phase is in methods like `Driver._training_phase`, +`Driver._query_phase`, etc. + +## Advanced Customization + +### Custom Training Loops + +While `DefaultTrainingLoop` is suitable for most use cases, you can write +custom training loops that implement the `TrainingLoop` protocol, which is the +overarching logic for how to carry out model training and validation over some +number of epochs. Custom loops are useful when you need: + +- Specialized training logic (e.g., alternating, or multiple optimizers) +- Custom logging or checkpointing within the loop +- Non-standard epoch/batch iteration patterns + +### Custom Strategies + +All strategies must implement their respective protocols: + +- **QueryStrategy**: Implement `sample(query_queue, *args, **kwargs)` and +`attach(driver)` +- **LabelStrategy**: Implement `label(queue_to_label, serialize_queue, *args, +**kwargs)` and `attach(driver)` +- **MetrologyStrategy**: Implement `compute(*args, **kwargs)`, +`serialize_records(*args, **kwargs)`, and `attach(driver)` + +The `attach(driver)` method gives your strategy access to the driver's +attributes like `driver.learner`, `driver.train_datapool`, +`driver.unlabeled_datapool`, etc. + +### Static Capture and Performance + +The `DefaultTrainingLoop` supports static capture via CUDA graphs for +performance optimization: + +```python +train_loop_fn=DefaultTrainingLoop( + enable_static_capture=True, # Enable CUDA graph capture (default) + use_progress_bars=True, +) +``` + +For custom training loops, you can use: + +- `StaticCaptureTraining` for training steps +- `StaticCaptureEvaluateNoGrad` for validation/inference steps + +### Distributed Training + +To use multiple GPUs, provide a `DistributedManager` in `DriverConfig`: + +```python +from physicsnemo.distributed import DistributedManager + +dist_manager = DistributedManager() +driver_config = c.DriverConfig( + batch_size=16, + max_active_learning_steps=70, + dist_manager=dist_manager, # Handles device placement and DDP +) +``` + +The driver will automatically wrap the model in `DistributedDataParallel` and use +`DistributedSampler` for dataloaders. + +## Experiment Ideas + +Here, we perform a relatively straightforward experiment without a baseline; suitable +ones could be to train a model using the full data, and see how the precision/recall/F1 +scores differ between the `ClassifierUQQuery` learner to the full data model (i.e. use +the latter as a roofline). + +A suitable baseline to compare against would be random selection: to check the efficacy +of `ClassifierUQQuery`, samples could be chosen uniformly and see if and how the same +metrology scores differ. If the UQ is performing as intended, then precision/recall/F1 +should improve at a faster rate. diff --git a/examples/active_learning/moons/moon_data.py b/examples/active_learning/moons/moon_data.py new file mode 100644 index 0000000000..642cf0fcfa --- /dev/null +++ b/examples/active_learning/moons/moon_data.py @@ -0,0 +1,134 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Module that defines the classic two-moon classification dataset to +use as a demonstration dataset for a minimal active learning workflow. +""" + +import torch +from torch.utils.data import Dataset + +__all__ = ["MoonsDataset"] + + +def make_moons(n_samples: int = 2000, sigma: float = 0.25) -> torch.Tensor: + """ + Make the classic two-moon dataset. Code was adapted from + ``sklearn``, but modified slightly for the purposes of + this particular example. + + Parameters + ---------- + n_samples: int + The number of samples to generate. + + Returns + ------- + X_values: torch.Tensor + The input features. + y_values: torch.Tensor + The target labels. + """ + outer_grid = torch.linspace(0.0, torch.pi, int(n_samples * 0.7)) + inner_grid = torch.linspace(0.0, torch.pi, int(n_samples * 0.3)) + outer_x = torch.cos(outer_grid) + outer_y = torch.sin(outer_grid) + inner_x = 1 - torch.cos(inner_grid) + inner_y = 1 - torch.sin(inner_grid) - 0.5 + outer = torch.stack([outer_x, outer_y], dim=-1) + inner = torch.stack([inner_x, inner_y], dim=-1) + X_values = torch.cat([outer, inner], dim=0) + # add some noise to the coordinates + X_values += torch.randn_like(X_values) * sigma + y_values = torch.zeros(n_samples) + y_values[outer_grid.shape[0] :] = 1 + return X_values, y_values + + +class MoonsDataset(Dataset): + """ + Generate the classic two-moon dataset, repurposed for a minimal + active learning example. + + This class implements the `DataPool` protocol by subclassing + ``Dataset``, which provides all the methods except for ``append``, + which we implement here. + + The intuition is to have one of the moons be data poor, and a quasi- + intelligent query strategy will help overcome class imbalance to + some extent, as it will hopefully have higher uncertainty in its + classifier output to reflect this. + + Attributes + ---------- + initial_samples: float + The initial number of samples to hold out for training. + total_samples: int + The total number of samples to generate. + train_indices: torch.LongTensor | None + The indices of the samples to use for training. + X_values: torch.Tensor + The full set of input features; i.e. the coordinates + of a point in 2D space. + y_values: torch.Tensor + The target labels; 0 for the outer moon, 1 for the inner moon. + sigma: float + The standard deviation of the noise to add to the coordinates. + """ + + def __init__( + self, + initial_samples: float = 0.05, + total_samples: int = 1000, + train_indices: torch.LongTensor | None = None, + sigma: float = 0.25, + ): + super().__init__() + self.initial_samples = initial_samples + self.total_samples = total_samples + # this holds the full dataset for training + self.X_values, self.y_values = make_moons(total_samples, sigma) + # this corresponds to the subset that is actually exposed + # during training; it grows as we 'label' more samples + if train_indices is None: + # initial hold out for training + train_indices = torch.randperm(total_samples)[ + : int(total_samples * initial_samples) + ] + self.train_indices = train_indices + + def __len__(self) -> int: + """Return the length of the training subset.""" + return len(self.train_indices) + + def _sample_indices(self) -> torch.LongTensor: + """Return the indices that are not currently in training.""" + all_indices = torch.arange(self.total_samples) + mask = ~torch.isin(all_indices, self.train_indices) + return all_indices[mask] + + def append(self, item: int) -> None: + """Append a single index to the training set; needed for 'labeling'.""" + self.train_indices = torch.cat( + [self.train_indices, torch.tensor([item])], dim=0 + ) + + def __getitem__(self, index: int) -> tuple[torch.Tensor, torch.Tensor]: + """Retrieve a single coordinate-label pair from the dataset.""" + actual_index = self.train_indices[index] + x_val = self.X_values[actual_index, :] + y_val = self.y_values[actual_index] + return x_val, y_val diff --git a/examples/active_learning/moons/moon_example.py b/examples/active_learning/moons/moon_example.py new file mode 100644 index 0000000000..525fe03412 --- /dev/null +++ b/examples/active_learning/moons/moon_example.py @@ -0,0 +1,205 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Minimal example to show how the active learning workflow can be put together, +comprising the minimum of model training, and running some query strategy +to select data samples for labeling. + +This example implements a simple MLP that takes in 2D coordinates and +outputs logits for a binary classification task. The active learning +workflow here uses a query strategy that looks for samples that have +the highest classification uncertainty (i.e. closest to 0.5), and iterates +by adding those samples to the training set. Ideally, if uncertainty is +well-adjusted to this problem, then the query strategy will select samples +that are more likely to improve the model's general performance, as compared +to a random selection baseline. +""" + +import queue +import time + +import torch +from moon_data import MoonsDataset +from moon_strategies import ClassifierUQQuery, DummyLabelStrategy, F1Metrology +from torch import nn + +from physicsnemo import ModelMetaData, Module +from physicsnemo.active_learning import Driver, registry +from physicsnemo.active_learning import config as c +from physicsnemo.active_learning.loop import DefaultTrainingLoop + +torch.manual_seed(216167) + + +@registry.register("MLP") +class MLP(Module): + """ + Define a trivial MLIP model that will classify a 2D coordinate + into one of two classes, producing logits as the output. + + There is nothing to configure here, so focus on the active learning + components. + """ + + def __init__(self): + super().__init__(meta=ModelMetaData(amp=False)) + self.layers = nn.Sequential( + nn.Linear(2, 16), + nn.SiLU(), + nn.Linear(16, 1), + ) + self.loss_fn = nn.BCEWithLogitsLoss() + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Forward pass of the model. + + Parameters + ---------- + x: torch.Tensor + The input tensor, shape [B, 2] for a batch + size of B. + + Returns + ------- + torch.Tensor + The output tensor, shape [B, 1] for a batch + size of B. Remember to ``squeeze`` the output. + """ + return self.layers(x) + + +# this implements the `TrainingProtocol` interface +@registry.register("training_step") +def training_step(model: MLP, data: tuple[torch.Tensor, torch.Tensor]) -> torch.Tensor: + """ + Implements the training logic for a single batch of data. + + Parameters + ---------- + model: MLP + The model to train. + data: tuple[torch.Tensor, torch.Tensor] + The data to train on. + + Returns + ------- + torch.Tensor + The loss tensor. + """ + x, y = data + logits = model(x).squeeze() + loss = model.loss_fn(logits, y) + return loss + + +def main(): + """ + Configure an end-to-end active learning workflow. + + The code below primarily demonstrates how to compose things together + to form the full workflow. There are three configurations structures + that ultimately dictate the behavior of ``Driver``, which orchestrates + the workflow: + + 1. ``TrainingConfig``: everything to do with the model training process. + 2. ``StrategiesConfig``: comprises the query, label, and metrology strategies. + 3. ``DriverConfig``: decides things like batch size, logging, and ``DistributedManager``. + + The workflow should completely quickly: an `active_learning_logs` folder will + be created, and within it, run-specific logs. You will find the model weights, + alongside JSON logs of the process and from the ``F1Metrology`` strategy, which + will records how precision/recall progresses as more data points are added to the + strategy. + """ + # instantiate the model and data + dataset = MoonsDataset() + uq_model = MLP() + + # configure how training/fine-tuning is done within active learning + training_config = c.TrainingConfig( + train_datapool=dataset, + optimizer_config=c.OptimizerConfig( + torch.optim.SGD, + optimizer_kwargs={"lr": 0.01}, + ), + # configure different times for initial training and subsequent + # fine-tuning + max_training_epochs=30, + max_fine_tuning_epochs=30, + # this configures the training loop + train_loop_fn=DefaultTrainingLoop( + use_progress_bars=False, + enable_static_capture=False, + ), + ) + # this configuration packs all the strategy components together + strategy_config = c.StrategiesConfig( + query_strategies=[ClassifierUQQuery(max_samples=10)], + queue_cls=queue.Queue, + label_strategy=DummyLabelStrategy(), + metrology_strategies=[F1Metrology()], + ) + # this driver class handles the active learning loop + driver_config = c.DriverConfig( + batch_size=16, + max_active_learning_steps=70, + fine_tuning_lr=0.005, + device=torch.device("cpu"), # set to other accelerators if needed + ) + driver = Driver( + config=driver_config, + learner=uq_model, + strategies_config=strategy_config, + training_config=training_config, + ) + # our model doesn't implement a `training_step` method but in principle + # it could be implemented, and we wouldn't need to pass the step function here + driver(train_step_fn=training_step) + + # just some sanity checks + if not ( + len(dataset.train_indices) + == int(dataset.initial_samples * dataset.total_samples) + + driver_config.max_active_learning_steps + * strategy_config.query_strategies[0].max_samples + ): + raise RuntimeError( + "Number of samples added to the training pool inconsistent with expected value." + ) + + # restart the driver from a checkpoint; in practice the path would be provided + # train_datapool must be provided since it's not serialized + # learner must nominally have the same architecture as the one used to create the checkpoint + new_driver = Driver.load_checkpoint( + driver.log_dir / "checkpoints" / "step_42" / "labeling", + learner=uq_model, + train_datapool=dataset, + ) + assert new_driver.active_learning_step_idx == 42 + # enable this to re-run the driver training: be aware that this will overwrite subsequent checkpoints!! + RERUN = True + if RERUN: + new_driver.logger.info( + f"Rerunning driver from checkpoint {new_driver.last_checkpoint}" + ) + time.sleep(5) + new_driver(train_step_fn=training_step) + + +if __name__ == "__main__": + main() diff --git a/examples/active_learning/moons/moon_strategies.py b/examples/active_learning/moons/moon_strategies.py new file mode 100644 index 0000000000..285d3a88f4 --- /dev/null +++ b/examples/active_learning/moons/moon_strategies.py @@ -0,0 +1,242 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json +from queue import Queue +from typing import Any + +import torch + +from physicsnemo.active_learning import registry +from physicsnemo.active_learning.protocols import ( + DriverProtocol, + LabelStrategy, + MetrologyStrategy, + QueryStrategy, +) + +__all__ = ["ClassifierUQQuery", "DummyLabelStrategy", "F1Metrology"] + + +@registry.register("ClassifierUQQuery") +class ClassifierUQQuery(QueryStrategy): + """ + This query strategy is representative of a more complex + uncertainty-based query strategy: since our model produces + logits, we can use the model's confidence in class label + predictions to select data points for labeling: specifically, + we pick ``max_samples`` each active learning iteration of + the data points with the most uncertainty (closest to 0.5). + """ + + def __init__(self, max_samples: int): + """ + Initialize the query strategy. + + Parameters + ---------- + max_samples: int + The maximum number of samples to query. + """ + self.max_samples = max_samples + + def sample(self, query_queue: Queue) -> None: + """ + Identify which data points that need labels by the query strategy. + + At a high level, this method will: + 1. Slice out the data indices not currently in the training set, + 2. Query the model for predictions on the 'unlabeled' data, + 3. Enqueue indices of data points with the class predictions closest to 0.5. + + Parameters + ---------- + query_queue: Queue + The queue to enqueue data to be labeled. + """ + # strategy will be attached to a driver to access model and data + model = self.driver.learner + data = self.driver.train_datapool + unlabeled_indices = data._sample_indices() + # grab all of the data that's currently not labeled and obtain + # predictions from the model + unlabeled_coords = data.X_values[unlabeled_indices] + unlabeled_coords = unlabeled_coords.to(model.device) + model.eval() + with torch.no_grad(): + pred_logits = model(unlabeled_coords) + pred_probs = torch.sigmoid(pred_logits).squeeze() + # find probabilities that are closet to 0.5; the lower this + # value is, the more uncertain the model is + uncertainties = torch.abs(pred_probs - 0.5) + chosen_indices = torch.argsort(uncertainties)[: self.max_samples] + # enqueue indices of the chosen data points + for idx in chosen_indices: + query_queue.put(unlabeled_indices[idx]) + + def attach(self, driver: DriverProtocol) -> None: + """Attach the driver to the query strategy.""" + self.driver = driver + + +@registry.register("DummyLabelStrategy") +class DummyLabelStrategy(LabelStrategy): + """ + Since we have labels for all of our data already, this label strategy + will simply just add the data points our model has chosen to the + training set. + """ + + __is_external_process__ = False + + def __init__(self): + super().__init__() + + def label(self, query_queue: Queue, serialize_queue: Queue) -> None: + """ + Label the data points in the query queue. + + This is trivial because we are just passing indices from one queue + to another, but in a real implementation this might call an external + process to obtain ground truth data for a set of data points. + + Parameters + ---------- + query_queue: Queue + The queue to dequeue data from. + serialize_queue: Queue + The queue to enqueue labeled data to. + """ + while not query_queue.empty(): + selected_idx = query_queue.get() + serialize_queue.put(selected_idx) + + def attach(self, driver: DriverProtocol) -> None: + """Attach the driver to the label strategy.""" + self.driver = driver + + +@registry.register("F1Metrology") +class F1Metrology(MetrologyStrategy): + """ + While metrology is optional in the workflow, this provides observability + into how the model is performing over the course of active learning. + + For a simple use case like the Moons dataset, the margin between validation + and metrology is small, but for more complex use cases this strategy can + potentially represent a workflow beyond simple metrics (e.g. using the model + as a surrogate in a simulation loop). + """ + + def __init__(self): + self.records = [] + + def compute(self, *args: Any, **kwargs: Any) -> None: + """Compute the F1 score of the model on the validation set.""" + model = self.driver.learner + data = self.driver.train_datapool # this can be any `DataPool` + model.eval() + indices = torch.arange(data.total_samples) + input_data, labels = data.X_values[indices], data.y_values[indices] + input_data = input_data.to(model.device) + labels = labels.to(model.device) + with torch.no_grad(): + # pack the entire dataset into a single batch + pred_logits = model(input_data) + pred_probs = torch.sigmoid(pred_logits).squeeze() + pred_labels = torch.round(pred_probs) + precision = self.precision(pred_labels, labels) + recall = self.recall(pred_labels, labels) + # compute the F1 score + f1 = 2 * (precision * recall) / (precision + recall + 1e-8) + iteration = self.driver.active_learning_step_idx + num_train_samples = len(self.driver.train_datapool.train_indices) + report = { + "precision": precision, + "recall": recall, + "f1": f1, + "step": iteration, + "num_train_samples": num_train_samples, + } + self.append(report) + + @staticmethod + def precision(pred_labels: torch.Tensor, true_labels: torch.Tensor) -> float: + """ + Calculate precision for class 0. + + Precision is the ratio of true positives to all predicted positives: + how many of the samples predicted as class 0 are actually class 0. + + Parameters + ---------- + pred_labels : torch.Tensor + Predicted binary labels (0 or 1). + true_labels : torch.Tensor + Ground truth binary labels (0 or 1). + + Returns + ------- + float + Precision score for class 0. + """ + true_positives = ((true_labels == 1) & (pred_labels == 1)).sum().item() + predicted_positives = (pred_labels == 1).sum().item() + if predicted_positives == 0: + return 0.0 + return true_positives / predicted_positives + + @staticmethod + def recall(pred_labels: torch.Tensor, true_labels: torch.Tensor) -> float: + """ + Calculate recall for class 0. + + Recall is the ratio of true positives to all actual positives: + how many of the actual class 0 samples were predicted as class 0. + + Parameters + ---------- + pred_labels : torch.Tensor + Predicted binary labels (0 or 1). + true_labels : torch.Tensor + Ground truth binary labels (0 or 1). + + Returns + ------- + float + Recall score for class 0. + """ + true_positives = ((pred_labels == 0) & (true_labels == 0)).sum().item() + actual_positives = (true_labels == 0).sum().item() + if actual_positives == 0: + return 0.0 + return true_positives / actual_positives + + def attach(self, driver: DriverProtocol) -> None: + """Attach the driver to the metrology strategy.""" + self.driver = driver + + @property + def is_attached(self) -> bool: + """Check if the metrology strategy is attached to a driver.""" + return hasattr(self, "driver") + + def serialize_records(self, *args: Any, **kwargs: Any) -> None: + """Serialize the records of the metrology strategy.""" + output_path = self.strategy_dir / f"step_{self.driver.active_learning_step_idx}" + output_path.mkdir(parents=True, exist_ok=True) + with open(output_path / "f1_metrology.json", "w") as f: + json.dump(self.records, f, indent=2) diff --git a/examples/additive_manufacturing/sintering_physics/README.md b/examples/additive_manufacturing/sintering_physics/README.md new file mode 100644 index 0000000000..44eb57854f --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/README.md @@ -0,0 +1,220 @@ + + +# Virtual Foundary GraphNet + +## Introduction + +Metal sintering is a necessary step for Metal Injection Molded parts and +binder jetting such as +[HP’s metal 3D printer (MetJet)](https://www.hp.com/us-en/printers/3d-printers/products/metal-jet.html). +The metal sintering process introduces large deformation varying from 25% to 50% +depending on +the green part porosity. The final part's geometrical accuracy and consistency +remain the top challenge to manufacturing yield. +This is due to: + +1. Green parts out of MetJet printer are much more porous than +other technologies (e.g., MIM); +2. Such shrinkage is not isotropic depending on non-uniform stress +built up during the sintering process, +e.g., gravitational sag, gravitational slump, surface drag. + +

+ +

+ +In this work, we use a graph-based deep learning approach to +predict the part deformation, +which can speed up the deformation simulation substantially +at the voxel level. Running a well-trained Metal sintering +inferencing engine only takes a range of seconds to +obtain the final sintering deformation value. +The tested accuracy on example complex geometry achieves 0.7um mean +deviation for a 63mm testing part, for a single sintering step +(equivalent to 8.3 minutes physical sintering time), and a 0.3mm +mean deviation for the complete sintering cycle (~4 hrs physical sintering time). +

+ +

+ +Full paper on: +[Virtual Foundry Graphnet for Metal Sintering Deformation Prediction](https://arxiv.org/abs/2404.11753) + +For more sample parts simulation: +

+ +

+

+ +

+

+ +

+

+ +

+ +## Setup with PhysicsNeMo package + +- Download PhysicsNeMo, make or install + +- Find the matching torch-scatter version with torch and cuda version enabled: + - i.e. pip install torch-scatter-f `https://data.pyg.org/whl/torch-2.2.0%2Bcu121/torch_scatter-2.1.2%2Bpt22cu121-cp311-cp311-linux_x86_64.whl` + (replace the torch-scatter wheel with the matching cuda, torch version ) + - torch-scatter installation guide: `https://pypi.org/project/torch-scatter/` + - wheels source: `https://data.pyg.org/whl/` + +- pip install tensorflow + + - test version: tensorflow-2.15.0.post1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl + +- for logging: + - pip install wandb + - pip install mlflow + +- For training with mixed precision: + - `https://github.com/NVIDIA/apex` +- pyvista is required only if need to run data proprocessing with the raw +simulation data files + +- Dev: + + - install pytest + - pip install importlib-metadata + - pip install hydra-core --upgrade + +## Train + +Change the params in conf/config.yaml for training: + +- mode: "train" +- ckpt_path_vfgn={path to save model trained ckpt}, i.e. "models/test24" +- data_path: {data path for the pre-processed data in tfrecord}, i.e. "./data/test_validation" +- noise_std: i.e.1e-9 +- loss: i.e. me loss # options: ['standard', 'anchor', +'me', 'weighted', '''correlation', 'anchor_me'] + +Then run: + +```bash +python train.py +``` + +Currently default params: + +- INPUT_SEQUENCE_LENGTH = 5 +- PREDICT_LENGTH = 1 +- NUM_PARTICLE_TYPES = 3 +- Provided ckpt trained at every 100 step of Physcis sintering simulation data + +## Test (with the provided sample data) + +Change the params in conf/config.yaml for testing: + +- mode: "eval_rollout" +- eval_split: "test" +- batch_size: 1 +- noise_std: 0 +- ckpt_path_vfgn={path to model trained ckpt}, i.e. "models/ckpt/model_loss-4.17E-06_step-1113000.pt" +- output_path: {path to store outputs}, i.e. "rollouts/test24" +- data_path: {preprocessed test data tfrecord}, i.e. "./data/test_validation" + +Then run: + +```bash +python train.py +``` + +## Visualize test result + +Change the params in conf/config.yaml: + +- rollout_path={selected_prediction_output.json}, i.e. "rollouts/rollout_test_0.json" +- metadata_path={metadata path}, i.e. "./data/test_validation" +- test_build_name={test file name}, i.e. "test0" + +```bash +python render_rollout.py +``` + +## Inference + +Change the params in conf/config.yaml for inference run: + +(model tested with spliting the entire sintering profile into 2 stages, +can combine the entire sintering profile inferencing according to train schema) + +- mode: "rollout" +- eval_split: "inference" # name of the tfrecord dataset +- noise_std: 0 +- batch_size: 1 +- ckpt_path_vfgn={path to model trained ckpt}, i.e. "models/ckpt/models/ckpt/model_loss-4.17E-06_step-1113000.pt" +- output_path: {path to store outputs}, i.e. "rollouts/test24" +- data_path: {preprocessed test data tfrecord}, i.e. "./data/test_validation" + +```bash +python inference.py +``` + +## Data + +- Test data + + - Same voxel resolution as train + +- To generate your own tfrecord from Physical simulation output: + +```bash +python data_process/rawdata2tfrecord.py +``` + +Defition of step_context & methods tried: + +- appending only the previous step global context / ( sinter temperature) + + ```bash + tensor_dict['step_context'] =tensor_dict['step_context'][-predict_length - 1][tf.newaxis] + ``` + +- appending previous sequence of global context / (sequence of sinter temperature) + + ```bash + tensor_dict['step_context'] = tf.reshape(tensor_dict['step_context'][:-1], [1, -1]) + ``` + +- appending the entire sequence of sintering profile + + ```bash + tensor_dict['step_context'] = tf.reshape(tensor_dict['step_context'],[1, -1]) + ``` + +## Disclaimer and future work + +With the model prediction accuracy and fast inference speed, +this work, as a component of HP’s Digital Twin effort, +Virtual Foundry Graphnet led by HP Labs, aims to apply Physics-ML +to significantly accelerate the computation that predicts the +metal powder material phase transition. It has achieved orders +of magnitude speed-up compared to physics simulation software while +preserving reasonable accuracy. Furthermore, Virtual Foundry Graphnet +has demonstrated an outstanding path forward to scaling for +diverse parts of arbitrary geometrical complexity +and scaling for different process parameter configurations. + +## Reference + +[Learning to Simulate Complex Physics with Graph Networks](https://arxiv.org/abs/2002.09405) + +```text +@inproceedings{sanchezgonzalez2020learning, + title={Learning to Simulate Complex Physics with Graph Networks}, + author={Alvaro Sanchez-Gonzalez and + Jonathan Godwin and + Tobias Pfaff and + Rex Ying and + Jure Leskovec and + Peter W. Battaglia}, + booktitle={International Conference on Machine Learning}, + year={2020} +} +``` diff --git a/examples/additive_manufacturing/sintering_physics/conf/config.yaml b/examples/additive_manufacturing/sintering_physics/conf/config.yaml new file mode 100644 index 0000000000..453b60312d --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/conf/config.yaml @@ -0,0 +1,61 @@ +# ignore_header_test +# ignore_header_test +# ruff: noqa: E402 + +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +general: + mode: "rollout" # Train model, one step evaluation or rollout evaluation, options: ['train', 'eval_rollout', 'rollout'] + eval_split: "inference" # Stored output dataset name, options: ['train', 'valid', 'test'] + device: "cuda:0" + message_passing_devices: "['cuda:0']" + fp16: False # performance configs + +train_options: + batch_size: 1 + num_steps: 2e7 + eval_steps: 1 + prefetch_buffer_size: 100 + input_seq_len: 5 # calculate the last 5 velocities. [options: 5, 10] + pred_len: 1 # [options: 5] + loss: "me" + loss_decay_factor: 0.6 #range (0, 1] + l_plane: 30 + l_me: 3 + +data_options: + data_path: "./data/2024-3-absnorm" + noise_std: 0 + ckpt_path_vfgn: "models/new24-gpus-mean_me_final/model_loss-6.42E-04_step-350.pt" + output_path: "rollouts/test2404" + NUM_PARTICLE_TYPES: 3 + KINEMATIC_PARTICLE_ID: 0 # refers to anchor point + METAL_PARTICLE_ID: 2 # refers to normal particles + ANCHOR_PLANE_PARTICLE_ID: 1 # refers to anchor plane + +test_options: + rollout_refine: False # Set False for: rollout the predictions, True for: single-step prediction for all steps + rollout_path: "rollouts/test24/rollout_test_0.json" + metadata_path: "./data/2024-3-absnorm" + step_stride: 3 + block_on_show: True + ds_type: str = "standard" # test data type: ['standard', 'train', 'test'] + test_build_name: "test0" + plot_tolerance_range: True + plot_3d: False diff --git a/examples/additive_manufacturing/sintering_physics/data_process/acceleration_profile.py b/examples/additive_manufacturing/sintering_physics/data_process/acceleration_profile.py new file mode 100644 index 0000000000..4cf8d87f4c --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/data_process/acceleration_profile.py @@ -0,0 +1,95 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +""" +File to visualize and check the acceleration profile +""" + +import logging +import os + +import hydra +import matplotlib.pyplot as plt +import numpy as np +import pyvista as pv +from omegaconf import DictConfig +from utils import get_data_position, read_raw_folder, time_diff + +logging.basicConfig(filename="data_analysis.log", level=logging.DEBUG) + + +@hydra.main(version_base=None, config_path=".", config_name="config") +def main(cfg: DictConfig): + """The function to visualize and check the acceleration profile, builds to be analyzed are read from config""" + raw_data_dir = cfg.data_options.raw_data_dir + # build_name = "2024-02-27-Overhangs_t3_theta0_V3-deformation" + + for build_name in cfg.data_options.builds_train: + solution_list = read_raw_folder(os.path.join(raw_data_dir, build_name)) + logging.info( + f"\n\nRead solution files from {build_name}, cnt= {len(solution_list)}" + ) + + pos_list, pos_max_list, pos_mean_list = [], [], [] + step = cfg.data_options.step_size + logging.info(f"Process for every {step} files ...... ") + for i in range(0, len(solution_list), step): + # Read the raw solution file with step size + solution_data = pv.read(solution_list[i]) + # pos_array: np array stores displacement, dim: (num_nodes, 3) + pos_array, _ = get_data_position(solution_data) + pos_list.append(pos_array) + + pos_mean_list.append(np.mean(pos_array, axis=0)) + pos_max_list.append(np.max(pos_array)) + + logging.info(f"Computed pos_list, shape {np.asarray(pos_list).shape}") + + # Compute the velovity, acceleration for each node + velocity_array = time_diff(np.array(pos_list)) + acceleration_array = time_diff(velocity_array) + logging.info(f"Computed velocity_array, shape {velocity_array.shape}") + logging.info(f"Computed acceleration_array, shape {acceleration_array.shape}") + + acc_3d_mean = np.mean(acceleration_array, axis=1) + logging.info(f"Computed Mean(acce), shape {acc_3d_mean.shape}") + + # Visualize + fig, ax = plt.subplots() + logging.info("Plot Mean(acce) ......... ") + sol_index = [i for i in range(acc_3d_mean.shape[0])] + ax.plot(sol_index, acc_3d_mean[:, 0], "b-", linewidth=1, label="x-dim velocity") + ax.plot(sol_index, acc_3d_mean[:, 1], "y-", linewidth=1, label="y-dim velocity") + ax.plot(sol_index, acc_3d_mean[:, 2], "g-", linewidth=1, label="z-dim velocity") + ax.set_xlabel("time steps", color="blue", fontsize=14) + ax.set_ylabel("acce", color="blue", fontsize=14) + # ax.set_ylim(0, 3e-6) + ax.set_title(build_name) + + ax.legend(loc="lower right") + fig_name = "acc_3d_" + build_name + "_step" + str(step) + fig.savefig(fig_name + ".jpg", format="png", dpi=100, bbox_inches="tight") + logging.info(f"Saved figure at {fig_name}. ") + + logging.info(f"Mean(acce) {np.mean(np.abs(acc_3d_mean), axis=0)}") + + +""" +Perform data analyis on voxel moving speed, acceleration +""" +if __name__ == "__main__": + main() diff --git a/examples/additive_manufacturing/sintering_physics/data_process/conf/config.yaml b/examples/additive_manufacturing/sintering_physics/data_process/conf/config.yaml new file mode 100644 index 0000000000..2fc04ff356 --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/data_process/conf/config.yaml @@ -0,0 +1,33 @@ +# ignore_header_test +# ruff: noqa: E402 + +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +data_options: + raw_data_dir: "/mnt/archive/pablo/2024-02-28-rachel" + metadata_json_path: "../data/2024-3-absnorm" + mode: "stats" # options: ["train", "test", "stats"] + builds_train: ["2024-02-26-bridge3-4x27-36_3up-deformation", + "2024-02-27-incomplete-table-deformation", "2024-02-26-DoubleCantilever3X17-deformation", + "2024-02-26-I_3mm-deformation", "2024-02-27-TBar-deformation", "2024-02-27-HandAkaFingers-deformation" + ] + builds_test: ["2024-02-26-Circle-10mmThickness-with-base-deformation", "2024-02-27-Overhangs_t3_theta0_V3-deformation"] + f_basename: "volumn-deformation" # other options before: ["solution", "volumn-deformation"] + add_anchor: True + step_size: 5 diff --git a/examples/additive_manufacturing/sintering_physics/data_process/deform_analyze.py b/examples/additive_manufacturing/sintering_physics/data_process/deform_analyze.py new file mode 100644 index 0000000000..b77f91638d --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/data_process/deform_analyze.py @@ -0,0 +1,243 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# © Copyright 2023 HP Development Company, L.P. + + +import glob +import json +import os +import sys + +import matplotlib.pyplot as plt +import numpy as np +import pyvista as pv +import rawdata2tfrecord_large_ts as rawdata2tfrecord +from utils import get_solution_id, read_configs + + +def plot_temperature_curve(temp_curve_list): + """Read from the list of sintering time-temp, visualize the sintering profile""" + t_list = [] + temp_list = [] + for idx in range(len(temp_curve_list) // 2): + t_list.append(int(temp_curve_list[idx * 2])) + temp_list.append(int(temp_curve_list[idx * 2 + 1])) + + print("time list: ", t_list) + print("temp_list : ", temp_list) + + fig = plt.figure(figsize=(10, 6)) + plt.plot(t_list, temp_list, color="blue", marker=".") + fig.savefig("temperature_profile.png", format="png", dpi=100, bbox_inches="tight") + + return t_list, temp_list + + +def plot_temperature_curve2(key_list, temp_list): + """Read from the list of sintering step_idx-temp, visualize the sintering profile""" + fig = plt.figure(figsize=(10, 6)) + plt.plot(key_list, temp_list, color="blue", marker=".") + fig.savefig( + "temperature_profile_complete.png", format="png", dpi=100, bbox_inches="tight" + ) + + +def read_sol_time(series_file): + """ + Read the solution.pvtu.series file + Returns: + + """ + dict_sol_time = {} + time_list = [] + with open(series_file, "r") as fobj: + data = json.load(fobj) + + for idx, item in enumerate(data["files"]): + time_list.append(item["time"]) + dict_sol_time[item["name"]] = [item["time"]] + + return dict_sol_time + + +# plot +def plot_p_deform( + temp_list, key_list, stage_keys, del_u, del_v, del_w, pid=0, split_stages=False +): + """Read from selected point-id deformation, visualize this point's deformation over the sintering profile""" + # sol_list = [i for i in range(len(temp_list))] + sol_list = key_list + + fig, ax = plt.subplots() + # ax.plot(sol_list, del_u, color="blue", marker=".") + ax.plot(sol_list, del_u, "b-", linewidth=2) + # ax.plot(sol_list, del_v, color="g", marker=".") + ax.plot(sol_list, del_v, "g-", linewidth=2) + ax.plot(sol_list, del_w, "y-", linewidth=2) + ax.set_xlabel("Solution index ", fontsize=14) + ax.set_ylabel("Sample point deformation (mm)", color="blue", fontsize=14) + + # twin object for two different y-axis on the sample plot + ax2 = ax.twinx() + # make a plot with different y-axis using second axis object + ax2.plot(sol_list, temp_list, "r-", linewidth=2) + ax2.set_ylabel("Temperature", color="red", fontsize=14) + + # plot cut-off regions + x_lb = min(np.asarray(del_u + del_v + del_w)) + x_ub = max(np.asarray(del_u + del_v + del_w)) + x_lim = np.arange(x_lb, x_ub + 0.0005, 0.0005).tolist() + + if split_stages: + ax.fill_betweenx( + x_lim, + x1=stage_keys[5], + x2=stage_keys[6], + where=None, + step=None, + color="gainsboro", + interpolate=True, + label="stage-separation-1", + ) + ax.fill_betweenx( + x_lim, + x1=stage_keys[7], + x2=stage_keys[8], + where=None, + step=None, + color="gainsboro", + interpolate=True, + label="stage-separation-2", + ) + ax.fill_betweenx( + x_lim, + x1=stage_keys[9], + x2=max(sol_list), + where=None, + step=None, + color="gainsboro", + interpolate=True, + label="stage-separation-3", + ) + + fig_name = "point_deform_curve_p" + str(pid) + ax.set_title("p" + str(pid), fontsize=14) + ax.legend(loc="lower right") + fig.savefig(fig_name + ".jpg", format="png", dpi=100, bbox_inches="tight") + + +# Read a point id-x, from each file in solution list, plot its uvw +def read_point(file_name, p_id): + """Read the sintering point id value, with pv library, from path: file_name""" + data = pv.read(file_name) + uvw_values = data["displacement_U"] + + p_xyz = data.GetPoint(p_id) + p_uvw = uvw_values[p_id, ...] + + return p_xyz, p_uvw + + +def read_solutions_data_temp_anchor( + raw_data_path, build_name, start_temp=500, end_temp=2000 +): + """1st version""" + build_path = os.path.join(raw_data_path, "out") + solution_list = glob.glob(build_path + "/displacement-*.pvtu") + solution_list = sorted(solution_list, key=get_solution_id) + + # read configs from params.prm only, bypass the solution.pvtu.series file + time_params, temp_curve_list = read_configs(raw_data_path) + + # For each build, read the displacement.pvtu.series file + series_file = os.path.join(raw_data_path, "out", "displacement.pvtu.series") + print("Find and read series file: ", series_file) + assert os.path.exists(series_file), "displacement.pvtu.series not exists!" + dict_sol_time = read_sol_time(series_file) + print("dict_sol_time: ", dict_sol_time) + + key_list = [] + temp_list = [] + + del_u, del_v, del_w = [], [], [] + + # todo: sample point id, move to input variable + read_point_id = 8000 + for solution_idx, solution_path in enumerate(solution_list): + # For each solution-*.pvtu file, get the solution-id, read data points + # filter out the repeated data + solution_id = get_solution_id(solution_path) + solution_temp = rawdata2tfrecord.get_solution_temperature_customer( + solution_path, dict_sol_time, temp_curve_list + ) + + if solution_temp >= start_temp and solution_temp < end_temp: + key_list.append(solution_id) + temp_list.append(solution_temp) + + p_xyz, p_uvw = read_point(file_name=solution_path, p_id=read_point_id) + del_u.append(p_uvw[0]) + del_v.append(p_uvw[1]) + del_w.append(p_uvw[2]) + + plot_p_deform( + build_name, + temp_list, + key_list=key_list, + del_u=del_u, + del_v=del_v, + del_w=del_w, + pid=read_point_id, + ) + + return key_list, temp_list + + +def main(argv): + """ + Args: + raw_data_dir: raw data directory + metadata_json_path: path of metadata.json + mode: there are three mode [train, test, stats] + i.e. data path on server to test 69 parts generalization: + """ + raw_data_dir = argv[0] + + # sample builds to perform analysis, include the build names + # i.e. build_list = ['10007564-slide-53710', 'Bearing_Support-slide-53710', 'GRF_SH20-slide-53710', '2289x1D_N_acc-slide-53710', 'S8001-slide-53710'] + build_list = ["MK-M-1045-rA-slide-53710"] + + # input the start and end point for ploting + start_temp, end_temp = 600, 1310 + + n_steps = 0 + for build_name in build_list: + key_list, temp_list = read_solutions_data_temp_anchor( + os.path.join(raw_data_dir, build_name), + build_name, + start_temp=start_temp, + end_temp=end_temp, + ) + + if n_steps != 0 and len(key_list) != n_steps: + print(build_name, " read failed") + continue + + +if __name__ == "__main__": + argv = sys.argv[1:] + main(argv) diff --git a/examples/additive_manufacturing/sintering_physics/data_process/out2pvtu.py b/examples/additive_manufacturing/sintering_physics/data_process/out2pvtu.py new file mode 100644 index 0000000000..da38d98458 --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/data_process/out2pvtu.py @@ -0,0 +1,352 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# © Copyright 2023 HP Development Company, L.P. + +""" +This file takes the predicted output stored in /rollouts +convert into the .vtu format, that can be processed by simulation software +""" + +import glob +import logging as log +import os +import pickle +import sys + +import numpy as np +import pyvista as pv +import vtk +from rawdata2tfrecord_large_ts import get_solution_id + +log.basicConfig( + format="%(asctime)s - %(levelname)s\t%(message)s", + datefmt="%I:%M:%S %p", + level=log.INFO, +) + + +def write(name, obj): + """Write output to vtk""" + writer = vtk.vtkUnstructuredGridWriter() + writer.SetFileName(name) + writer.SetInputData(obj) + writer.Update() + writer.Write() + log.info(f"Saved {name}") + + +def pv_get_data_position(data): + """ + Read the sample solution.pvtu file, get the position indexes for predicted data updates + """ + + points = data.points + n_points = points.shape[0] + print( + "complete points shape: ", points.shape + ) # i.e. all points shape: (134464, 3) + uvw_values = data["displacement_U"] + print("complete ori uvw_values shape: ", uvw_values.shape) + + pos_deformed_list = [] # [all unique deformed_positions] + index_list = [] # [corresponding original first pid of the unique location] + position_ids_dict = {} # stores {xyz: [pid list of the same xyz location]} + for p_id in range(n_points): + # Read the point xyz-location + point_xyz = data.GetPoint(p_id) + + if point_xyz not in position_ids_dict: + # if the xyz-location point not stored before, read its uvw value, init dict + position_ids_dict[point_xyz] = [p_id] + uvw = uvw_values[p_id] + + # Compute the deformed physical location from original physical location + # todo: can postentially skip the deformation compute here + deformed_pos = point_xyz + uvw + + index_list.append(p_id) + pos_deformed_list.append(deformed_pos) + else: + # if point of location xyz already appeared, update the id list + position_ids_dict[point_xyz].append(p_id) + + print("index_list: ", len(index_list)) + return position_ids_dict, np.array(pos_deformed_list), index_list + + +def vtk_get_data_position(file_path): + """ + Read the basemesh geometry, return points with xyz location recording + Args: + file_path: + + Returns: + + """ + # baseMeshReader = vtk.vtkGenericDataObjectReader() + baseMeshReader = vtk.vtkXMLPUnstructuredGridReader() + baseMeshReader.SetFileName(file_path) + baseMeshReader.Update() + + basemesh = baseMeshReader.GetOutput() + x0Points = basemesh.GetPoints() + print("vtk_get_data_position X0points : ", x0Points) + + return x0Points + + +def update_points(basemesh_points, metadata, new_pos, position_ids_dict, index_list): + """ + new_pos: example_rollout['predicted_rollout'], shape:(predicted_time_steps, num_particles, dim) + i.e. (5, 21969, 3) + :return + location_deform_map:{(xyz, ): array([uvw_ list])} + i.e. check map: {(4.0, 48.0, 1.0): array([-0.0399062 , -0.01175825, -0.01040107])} + """ + # initialize uvw of same node number as basemesh + new_uvw_array = np.zeros((basemesh_points.GetNumberOfPoints(), 3)) + print("new_uvw_array shape: ", new_uvw_array.shape) + pos_mean, pos_std = metadata["pos_mean"], metadata["pos_std"] + # need to store last timestep prediction only + # todo: may change per requirements + new_pos = new_pos[-1, ...] + denormed_new_pos = new_pos * pos_std + pos_mean + print("predicted_rollout shape: ", new_pos.shape) + + location_deform_map = {} + # update the deformed position for each point + for p_id, point_new_pos in enumerate(denormed_new_pos): + # iterate the predicted non-duplicate points set, i.e. 13500 + # get the xyz index (in mm) for each point id, from the match in basemesh + xyz = basemesh_points.GetPoint(index_list[p_id]) + + # get the point p_ids of the same xyz location + xyz_dup_pids = position_ids_dict[xyz] + + # denormed_point_new_pos = point_new_pos * pos_std + pos_mean + uvw_ = point_new_pos - xyz + for id_ in xyz_dup_pids: + # update the new pos value for all nodes of this same xyz location + new_uvw_array[id_, :] = uvw_ + + location_deform_map[xyz] = uvw_ + return new_uvw_array, location_deform_map + + +def write_output(basemesh_path, new_uvw_array, end_inference_index, outPath): + """ + + Args: + basemesh_path: out/mesh.pvtu file that contains the xyz geometry information + new_uvw_array: + + Returns: + + """ + # new uvw value= [ 0.25397945 0.55836469 -0.79864266] + # prepare vtk array that will be added to our points + uvw_vtk_array = vtk.vtkDoubleArray() + uvw_vtk_array.SetNumberOfComponents(3) + uvw_vtk_array.SetName("displacement_U") + # add uvw-displacement values to the array + for index in range(len(new_uvw_array)): + uvw = new_uvw_array[index, :] # [ 0.25397945 0.55836469 -0.79864266] + uvw_vtk_array.InsertNextTuple(uvw) + # uvw_vtk_array.InsertTuple(point_index, uvw) + + ##### read mesh file (to copy its cells) + mesh_vtu_file_reader = vtk.vtkXMLPUnstructuredGridReader() + mesh_vtu_file_reader.SetFileName(basemesh_path) + mesh_vtu_file_reader.Update() + mesh_vtu_file = mesh_vtu_file_reader.GetOutput() + + ##### prepare new solution object + predicted_vtu_solution = vtk.vtkUnstructuredGrid() + # copy cells of mesh vtu file + predicted_vtu_solution.DeepCopy(mesh_vtu_file) + # add uvw-displacement array to our new solution vtk object + predicted_vtu_solution.GetPointData().AddArray(uvw_vtk_array) + + ##### save vtu file + predicted_vtu_solution_writer = vtk.vtkXMLUnstructuredGridWriter() + predicted_vtu_solution_writer.SetInputData(predicted_vtu_solution) + predicted_file_path = os.path.join( + outPath, "predicted-displacement-{}.vtu".format(end_inference_index) + ) + predicted_vtu_solution_writer.SetFileName(predicted_file_path) + predicted_vtu_solution_writer.Write() + print("wrote ", predicted_file_path) + return predicted_file_path + + +def save_volume_deformation( + basemesh_path, new_uvw_array, end_inference_index, outPath, core_id +): + """Save the predicted deformation""" + uvw_vtk_array = vtk.vtkDoubleArray() + uvw_vtk_array.SetNumberOfComponents(3) + uvw_vtk_array.SetName("displacement_U") + # add uvw-displacement values to the array + for index in range(len(new_uvw_array)): + uvw = new_uvw_array[index, :] # [ 0.25397945 0.55836469 -0.79864266] + uvw_vtk_array.InsertNextTuple(uvw) + + # read mesh file (to copy its cells) + # todo: check the local_ugrid/ mesh_vtu_file consistent with C code + mesh_vtu_file_reader = vtk.vtkXMLUnstructuredGridReader() + mesh_vtu_file_reader.SetFileName(basemesh_path) + mesh_vtu_file_reader.Update() + mesh_vtu_file = mesh_vtu_file_reader.GetOutput() + + predicted_vtu_solution = vtk.vtkUnstructuredGrid() + predicted_vtu_solution.SetPoints(mesh_vtu_file.GetPoints()) + predicted_vtu_solution.SetCells( + mesh_vtu_file.GetCellTypesArray(), mesh_vtu_file.GetCells() + ) + predicted_vtu_solution.GetPointData().SetVectors(uvw_vtk_array) + + ##### Write + predicted_vtu_solution_writer = vtk.vtkXMLUnstructuredGridWriter() + predicted_vtu_solution_writer.SetInputData(predicted_vtu_solution) + # update names with leading-0 + predicted_file_path = os.path.join( + outPath, + "predicted-displacement-{}-{}.vtu".format( + str(core_id).rjust(4, "0"), end_inference_index + ), + ) + predicted_vtu_solution_writer.SetFileName(predicted_file_path) + predicted_vtu_solution_writer.Write() + + return predicted_file_path + + +def save_volume_deformation_master_record(outPath, vtu_list, end_inference_index): + """Save the predicted deformation master file""" + + print("process solution: ", vtu_list[0]) + master_path = os.path.join( + outPath, "predicted-displacement-{}.pvtu".format(end_inference_index) + ) + master_file = open(master_path, "w") + + master_file.write('\n') + master_file.write("\n") + master_file.write( + '\n' + ) + master_file.write('\n') + master_file.write(' \n') + master_file.write( + ' \n' + ) + master_file.write(" \n") + master_file.write(" \n") + master_file.write(' \n') + master_file.write(" \n") + + for i, vtu_name in enumerate(vtu_list): + vtu_name = os.path.basename(vtu_name) + master_file.write( + ' \n' + ) # displacement-0000-1505.vtu + + master_file.write("\n") + master_file.write("\n") + print("complete writing to master file: ", master_path) + return + + +def post_process( + raw_data_path, metadata, example_rollout, end_inference_index, outPath +): + """ + Args: + raw_data_path: Virtual Foundry output solution file folder, + i.e."/home/rachel_chen/dataset/ladder_fast" + metadata: metadata path with corresponding VFGN trained model ckpt version + example_rollout: predicted rollout map data structure, contains keys below + {'initial_positions':, 'predicted_rollout':, 'particle_types':, 'global_context':, + ''ground_truth_rollout':, ''metadata': } + end_inference_index: + outPath: VFGN predicted output store path, + i.e."learning_to_simulate/rollouts/test" + + Returns: + predicted_file_path: + """ + print(example_rollout.keys()) + print( + example_rollout["predicted_rollout"].shape + ) # i.e. (47, 21969, 3): (predicted_time_steps, num_nodes, xyz-dim) + + # Read sample geometry with xyz node locations + # solution_list, dict_sol_time, temp_list = read_configs(raw_data_path) + + ### Get the basemesh to build geometry from + build_path = os.path.join(raw_data_path, "out") + solution_list = glob.glob(build_path + "/displacement-*.pvtu") + solution_list = sorted(solution_list, key=get_solution_id) + + print("\nread basemesh from ", solution_list[0]) + solution_data = pv.read(solution_list[0]) + position_ids_dict, _, index_list = pv_get_data_position(solution_data) + # print(pos_deformed_array.shape) # (21969, 3) + + # Compare the readings from VTK library function + print( + "\nCompare VTK read basemesh ", + ) + basemesh_path = os.path.join(raw_data_path, "mesh", "mesh.pvtu") + assert os.path.exists(basemesh_path), print( + f"basemesh does not exist: {basemesh_path}" + ) + basemesh_points = vtk_get_data_position(basemesh_path) # mesh_0.vtu + + # Match each xyz-location node with its uvw-displacement value + new_uvw_array, _ = update_points( + basemesh_points, + metadata, + example_rollout["predicted_rollout"], + position_ids_dict, + index_list, + ) + + # write to a new vtu file + predicted_file_path = write_output( + basemesh_path, new_uvw_array, end_inference_index, outPath + ) + print("Complete writing to new vtu file, stored in ", outPath) + return predicted_file_path + + +if __name__ == "__main__": + argv = sys.argv[1:] + raw_data_path, rollout_data_path, end_inference_index, outPath = argv + + if not rollout_data_path: + raise ValueError("A `rollout_path` must be passed.") + with open(rollout_data_path, "rb") as file: + example_rollout = pickle.load(file) + + post_process( + raw_data_path, + example_rollout["metadata"], + example_rollout, + end_inference_index, + outPath, + ) diff --git a/examples/additive_manufacturing/sintering_physics/data_process/rawdata2tfrecord_large_ts.py b/examples/additive_manufacturing/sintering_physics/data_process/rawdata2tfrecord_large_ts.py new file mode 100644 index 0000000000..1fa2b9fba5 --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/data_process/rawdata2tfrecord_large_ts.py @@ -0,0 +1,782 @@ +# ignore_header_test +# ruff: noqa: E402 + +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +""" +Include some basic functions for simulation data processings. +Convert simulation output displacement-*.pvtu files to tfRecord, +Store for model training + +usage: +python rawdata2tfrecord.py data-root meta-file-root +i.e. +python rawdata2tfrecord.py "/home/rachel_chen/dataset/Virtual-Foundry" "./" +python rawdata2tfrecord.py "/home/lopezca/repos/sintervox-models/dl-models" +""" + +import csv +import glob +import json +import logging +import os +import sys + +import numpy as np +import pyvista as pv +import tensorflow as tf +from natsort import natsorted +from sklearn import neighbors + +logging.basicConfig(filename="DS-retrain-2403.log", level=logging.DEBUG) + +import hydra +import utils +from omegaconf import DictConfig +from utils import time_diff + +# Create a description of the features. +_FEATURE_DESCRIPTION = { + "position": tf.io.VarLenFeature(tf.string), +} + +_FEATURE_DESCRIPTION_WITH_GLOBAL_CONTEXT = _FEATURE_DESCRIPTION.copy() +_FEATURE_DESCRIPTION_WITH_GLOBAL_CONTEXT["step_context"] = tf.io.VarLenFeature( + tf.string +) + +_FEATURE_DTYPES = { + "position": {"in": np.float64, "out": tf.float64}, + "step_context": {"in": np.float64, "out": tf.float64}, +} + +_CONTEXT_FEATURES = { + "key": tf.io.FixedLenFeature([], tf.int64, default_value=0), + "particle_type": tf.io.VarLenFeature(tf.string), + "senders": tf.io.VarLenFeature(tf.string), + "receivers": tf.io.VarLenFeature(tf.string), +} + + +def arrange_data(data): + """ + Organizes data from a structured format into a dictionary keyed by + point coordinates. + """ + arranged_data = {} + + name2array = {} + for array_name in data.array_names: + name2array[array_name] = data[array_name] + + # Get all points in the solution file + points = data.points + + # Number of points in one solution file + n_points = points.shape[0] + for point_index in range(n_points): + point = data.GetPoint(point_index) + if point not in arranged_data: + data_item = {} + for array_name in name2array: + data_item[array_name] = name2array[array_name][point_index] + arranged_data[point] = data_item + return arranged_data + + +def get_data_position(data): + """ + For the data read from one displacement-id.pvtu file, + iterate each point data, filter out the points in existed physical xyz-location + store the non-repeating point's uvw_values + Args: + data: data read from displacement-id.pvtu file with pv.read + format: UnstructuredGrid i.e. (0x7f1636fe5520) + N Cells: 21970 + N Points: 175760 + X Bounds: -4.600e+01, 3.000e+00 + Y Bounds: -4.500e+00, 4.500e+00 + Z Bounds: 0.000e+00, 1.300e+01 + N Arrays: 11 + + Returns: array of non-repeating nodes' current physical location (original location + displacement) + pos_list -> np.array + index_list -> list of same size + """ + # each points' coordinate, and displacement, shape [# point, dim], i.e. (175760, 3) + points = data.points + + uvw_values = data["u__v__w"] + + try: + points.shape == uvw_values.shape + except: + logging.error( + f"pv.read solution file field failed {data['u__v__w']} dimension not matching " + ) + raise + + # Construct a dictionary, store physical location {xyz: boolean} + arranged_data = {} + position_list = [] + index_list = [] + for point_index in range(points.shape[0]): + # Read coordinates of each point (x_coor, y_coor, z_coor) + # i.e. point_index: 168395, point_coor = (-31.0, -1.5, 0.0) + point_coor = data.GetPoint(point_index) + + # if there's not record of this point_coordinate, add; else skip to avoid duplicated points + if point_coor not in arranged_data: + # read displacement of this point + uvw = uvw_values[point_index] + # Compute the deformed physical location from original physical location + pos = point_coor + uvw + + index_list.append(point_index) + position_list.append(pos) + arranged_data[point_coor] = True + + return np.array(position_list), index_list + + +def read_sol_time(series_file): + """ + Read the solution.pvtu.series file + Returns: + Dictionary contains the sintering simulation file name, and corresponding timestamp for the simulation file + """ + dict_sol_time = {} + time_list = [] + with open(series_file, "r") as fobj: + data = json.load(fobj) + + for idx, item in enumerate(data["files"]): + time_list.append(item["time"]) + dict_sol_time[item["name"]] = [item["time"]] + + return dict_sol_time + + +def read_temperature(temp_file): + """ + Open and read the temperature profile from params.prm file under each build data. + Save Temp for each time solution file. + Returns: + list of temperature value, corresponding to each solution file (sorted with time) + """ + # reading csv file + logging.info(f"read temperature file from path: {temp_file}") + with open(temp_file, "r") as csvfile: + # creating a csv reader object + csvreader = csv.reader(csvfile) + + # extracting each data row one by one + for idx, row in enumerate(csvreader): + if row[0].find("temperature_curve") != -1: + temp_row = row[1:] + break + + # Add temperature data to each solution file + temp_row.insert(0, "0") + return temp_row + + +def get_solution_temperature_customer(solution_path, dict_sol_time, temp_curve_list): + """ + This function read each solution file, determine the time, Temperature of this file + Args: + solution_path: the solution.pvtu file to be processed + dict_sol_time: Dictionary contains the {solution_fname: simulation_time} + temp_curve_list: Read from the params file, + i.e. ['0', '0', '600', '20', '6000', '200', '18000', '400', '32400', '400', '38400', '600', '43800', '1050', '51000', '1050', '55140', '1395', '62340', '1395', '70680', '700'] + + Returns: + Temperature at the simulation time -> float + """ + t_list = [] + temp_list = [] + # Get the stage separation time-temperature pairs + for idx in range(len(temp_curve_list) // 2): + t_list.append(int(temp_curve_list[idx * 2]) / 3600) + temp_list.append(int(temp_curve_list[idx * 2 + 1])) + + sol_name = os.path.basename(solution_path) + sol_time = int(dict_sol_time[sol_name][0]) + + # search range + sol_time = sol_time / 3600 + + def find_nearest(array, value): + """Find the nearest value from the set of time lists""" + array = np.asarray(array) + idx = (np.abs(array - value)).argmin() + return idx, array[idx] + + nearest_id, nearest_value = find_nearest(t_list, sol_time) + + # determine left / right / == + if nearest_value == sol_time: + return temp_list[nearest_id] + elif nearest_value < sol_time: + start_temp, end_temp = temp_list[nearest_id], temp_list[nearest_id + 1] + start_time, end_time = nearest_value, t_list[nearest_id + 1] + else: + start_temp, end_temp = temp_list[nearest_id - 1], temp_list[nearest_id] + start_time, end_time = t_list[nearest_id - 1], nearest_value + + temp = ((end_temp - start_temp) / (end_time - start_time)) * ( + sol_time - start_time + ) + start_temp + + logging.info(f"solname {sol_name} | soltime {sol_time} | temp: {temp}") + return temp + + +def get_solution_temperature(solution_path, dict_sol_time, temp_curve_list): + """ + This temperature point compute only works for 1st-version temp curve + :param solution_path: + :param dict_sol_time: + :param temp_curve_list: + :return: + """ + sol_name = os.path.basename(solution_path) + sol_time = int(dict_sol_time[sol_name][0]) + + start_time, start_temp = int(temp_curve_list[0]), int(temp_curve_list[1]) + equil_time, equil_temp = int(temp_curve_list[2]), int(temp_curve_list[3]) + + if sol_time >= equil_time: + return equil_temp + else: + temp = ( + (equil_temp - start_temp) / (equil_time - start_time) + ) * sol_time + start_temp + return temp + + +def get_solution_time(solution_path, dict_sol_time): + """ + Retrieves the time associated with a solution from a dictionary using the + solution's file name. + """ + sol_name = os.path.basename(solution_path) + sol_time = int(dict_sol_time[sol_name][0]) + return sol_time + + +def _compute_connectivity(positions, radius): + """Get the indices of connected edges with radius connectivity. + + Args: + positions: Positions of nodes in the graph. Shape: + [num_nodes_in_graph, num_dims]. i.e. (10000, 3) + radius: Radius of connectivity. i.e. 1.2 + + Returns: + senders indices [num_edges_in_graph] + receiver indices [num_edges_in_graph] + + """ + # Construct tree from the points' positions + tree = neighbors.KDTree(positions) + + # For each point, get the list of nodes' indices within r + # return -> list[array(all connecting node indices)] + # i.e. receivers_list: [array([ 300, 2, 0, 1, 4, 1371, 310]) + # array([ 3, 301, 0, 1, 5, 311, 8]) + # array([1372, 6, 2, 24, 358, 3, 0]) ... + receivers_list = tree.query_radius(positions, r=radius) + + # For each node with sorted index, repeat its len(receiver-nodes) times, to form the matching sender indices array + senders = np.repeat(range(len(positions)), [len(a) for a in receivers_list]) + receivers = np.concatenate(receivers_list, axis=0) + + try: + senders.shape == receivers.shape + except: + logging.error("Sender, receiver indices not match") + raise + + return senders, receivers + + +def get_anchor_point(ds, index_list): + """ + pvtu_file_name: + with the given build files, return the index of point, that is the anchor point of the build + Query criteria: point xyz-displacement == 0 + Ideally, should only exist 1 anchor point + Args: + data: data from pv.read + format: UnstructuredGrid i.e. (0x7f1636fe5520) + N Cells: 21970 + N Points: 175760 + X Bounds: -4.600e+01, 3.000e+00 + Y Bounds: -4.500e+00, 4.500e+00 + Z Bounds: 0.000e+00, 1.300e+01 + N Arrays: 11 + index_list: the non-duplicated point index + + Returns: + anchor point index: int + """ + + # Read the Digital sintering software raw data, field version name: "u_v_w" + ds1 = ds["u__v__w"] + + ## return index of the anchor point + ds1_ax = np.where(ds1[:, 0] == 0)[0] + ds1_ay = np.where(ds1[:, 1] == 0)[0] + ds1_az = np.where(ds1[:, 2] == 0)[0] + + # Intersect 3 array to get the point with xyz-displacement == 0 + listx_as_set = set(ds1_ax) + intersection = listx_as_set.intersection(ds1_ay) + anchor_pset = intersection.intersection(ds1_az) + + # Intersect with the non-duplicated point index + anchor_point = anchor_pset.intersection(index_list) + + try: + len(anchor_point) == 1 + except: + logging.error( + f"Find non-unique anchor points {len(anchor_point)}, id list: {anchor_point}!" + ) + raise + + # find the point id in the non-duplicated point list + p_idx = list(anchor_point)[0] + index_list_str = list(map(str, index_list)) + p_i = index_list_str.index(str(p_idx)) + + return p_i + + +def get_anchor_zplane(ds, index_list): + """ + with the given build files, return the index of point that are on the anchor z-plane of the build + Query criteria: point z-position == 0, z-displacement == 0 + Args: + data: data from pv.read + format: UnstructuredGrid i.e. (0x7f1636fe5520) + N Cells: 21970 + N Points: 175760 + X Bounds: -4.600e+01, 3.000e+00 + Y Bounds: -4.500e+00, 4.500e+00 + Z Bounds: 0.000e+00, 1.300e+01 + N Arrays: 11 + index_list: the non-duplicated point index + + Returns: + anchor plane points index: List[int] + """ + # each points' coordinate, shape [# point, dim], i.e. (175760, 3) + n_points, _ = ds.points.shape + uvw_values = ds["u__v__w"] + + z_plane = [] + for i, p_idx in enumerate(index_list): + # for ip in range(n_points): + point = ds.GetPoint(p_idx) + + # Filter the points on z==0 z-plane + if point[2] == 0: + # for all the points fall on the z-plane, collected the point id in the non-duplicated point list + z_plane.append(i) + uvw_ = uvw_values[p_idx] + + # set non-0 threshold for corner-cases + threshold_z = 0.0002 + try: + uvw_values[p_idx][2] < threshold_z + except: + logging.info(f"wrong anchor_zplane id: {p_idx} - z-displacement {uvw_}") + raise + + return set(z_plane) + + +def read_solutions_data(raw_data_path=None, init_idx=0, metadata=None): + """From the raw simulation files, read the deformation value at every time-step for pre-processing""" + build_path = os.path.join(raw_data_path, "out") + solution_list = glob.glob(build_path + "/volume-deformation-*.pvtu") + # solution_list = sorted(solution_list, key=get_solution_id) + solution_list = natsorted(solution_list) + + try: + pv.read(solution_list[0]) + except: + logging.error(f"solution_list not found from: {build_path}") + raise + + # For each build, read the displacement.pvtu.series file + # series_file = os.path.join(raw_data_path, 'out', "solution.pvtu.series") + series_file = os.path.join(raw_data_path, "out", "volume-deformation.pvtu.series") + try: + dict_sol_time = read_sol_time(series_file) + except: + logging.error(f"solution.pvtu.series not exists!, {series_file}") + raise + + # For each build, read the temperature profile at every time step + temp_profile_path = os.path.join(raw_data_path, "params.prm") + try: + temp_curve_list = read_temperature(temp_profile_path) + except: + logging.error("Temperature profile file params.prm not exists!") + raise + logging.info(f"check temp_curve_list: {temp_curve_list}") + + # Record stage ids + # For example, for an entire sintering duration of 3393 simulation steps, can choose the data process window + # can either be the entire sintering duration, for process window for detailed accuracy + # for the default sintering profile, there are 3393 simulation steps, with + # stage 1: temperature increase window, start_index, end_index =[0, 596], + # stage 2: temperature stable window, start_index, end_index = [596,1321] + # stage 3: T increase: [> 1325] + # if to consider the transition stage separately: [1200, 1700] + + particles_list = [] + temp_list = [] + + # index for the step 100 model, 2 stages + step = 5 + + # solution_data_end = pv.read(solution_list[-1]) + solution_data_t0 = pv.read(solution_list[0]) + radius_begin = utils.get_radius(solution_data_t0) + logging.info(f"get simulation radius: {radius_begin}") + + # Get the non-duplicated points' start position, and the matching point index + pos_array_begin, index_list = get_data_position(solution_data_t0) + # Get the connecting points' matching sending-receiving indices + # return -> np.array of shape (#edges, ) + senders_graph_t0, receivers_graph_t0 = _compute_connectivity( + pos_array_begin, radius_begin + ) + logging.info(f"Computed the connected edges: {senders_graph_t0.shape}") + + # Proces each solution.pvtu file, with step / gap to skip + solution_step_list = solution_list[init_idx::step] + logging.info(f"process with start solution file idx: {init_idx}") + logging.info(f"solution list len: {len(solution_step_list)}") + + for solution_path in solution_step_list: + # For each displacement-*.pvtu file, get the displacement-id, read data points + # filter out the repeated data + time_ = int(dict_sol_time[os.path.basename(solution_path)][0]) + + # if start_index <= solution_id <= end_index: + solution_temp = get_solution_temperature_customer( + solution_path, dict_sol_time, temp_curve_list + ) + logging.info(f"process {solution_path}, time: {time_}, temp: {solution_temp}") + + solution_data_ = pv.read(solution_path) + pos_array_, index_list_ = get_data_position(solution_data_) + # todo: assert index_list_ matches with index_list + + particles_list.append(pos_array_) + temp_list.append(solution_temp) + + # ensure the processed sequence window have same length, to confrom with other train data + # todo: move this outside + if init_idx != 0 and len(particles_list) != metadata["sequence_length"]: + skip = True + else: + skip = False + + return ( + particles_list, + temp_list, + senders_graph_t0, + receivers_graph_t0, + radius_begin, + skip, + ) + + +def compute_metadata_stats( + metadata, + particles_list_builds, + velocity_list_builds, + acceleration_list_builds, + radius_list, + temp_list_builds, +): + """Compute stats of the train dataset, to update metadata for normalization in data processing""" + # Compute position mean, std + # todo: check why use different norm dimension + # todo: change the pos stats to 3d as well + position_stats_array = np.concatenate(particles_list_builds) + position_std = position_stats_array.std() + metadata["pos_mean"] = position_stats_array.mean() + metadata["pos_std"] = position_stats_array.std() + + # Compute velocity mean, std + velocity_stats_array = np.concatenate(velocity_list_builds) + velocity_stats_array = velocity_stats_array / position_std + metadata["vel_mean"] = [i for i in velocity_stats_array.mean(axis=0).tolist()] + metadata["vel_std"] = [i for i in velocity_stats_array.std(axis=0).tolist()] + + # Compute acceleration mean, std + acceleration_stats_array = np.concatenate(acceleration_list_builds) + acceleration_stats_array = acceleration_stats_array / position_std + metadata["acc_mean"] = [i for i in acceleration_stats_array.mean(axis=0).tolist()] + metadata["acc_std"] = [i for i in acceleration_stats_array.std(axis=0).tolist()] + + # Compute radius mean, std + radius_array = np.array(radius_list) / position_std + metadata["default_connectivity_radius"] = radius_array.mean() + + # Compute temperature mean, std + metadata["context_mean"] = [np.array(temp_list_builds).mean()] + metadata["context_std"] = [np.array(temp_list_builds).std()] + if np.array(temp_list_builds).ndim > 1: + metadata["context_feat_len"] = np.array(temp_list_builds).shape[1] + else: + metadata["context_feat_len"] = 1 + + return metadata + + +def write_tfrecord_entry(writer, features, particles_array, times_array): + """ + Write data into entry + Args: + writer: + features: contains context_features = { + 'particle_type': _bytes_feature(particles_type), particles_type dim=[#nodes, ], i.e. (26487,) + 'key': create_int_feature(key_i), + 'senders': _bytes_feature(senders_graph_i.tobytes()), + 'receivers': _bytes_feature(receivers_graph_i.tobytes()), + } + particles_array: + times_array: of float64, dim=[sim_steps,] i.e. (56,) + + Returns: + + """ + tf_sequence_example = tf.train.SequenceExample(context=features) + position_list = tf_sequence_example.feature_lists.feature_list["position"] + timestep_list = tf_sequence_example.feature_lists.feature_list["step_context"] + + for i in range(len(particles_array)): + position_list.feature.add().bytes_list.value.append( + particles_array[i].tobytes() + ) + timestep_list.feature.add().bytes_list.value.append(times_array[i].tobytes()) + + writer.write(tf_sequence_example.SerializeToString()) + + +@hydra.main(version_base=None, config_path="conf", config_name="config") +def main(cfg: DictConfig): + """ + Args: + raw_data_dir: raw data directory + metadata_json_path: path of metadata.json + mode: there are three mode [train, test, stats] + i.e. data path on server to test 69 parts generalization: + """ + mode = cfg.data_options.mode + raw_data_dir = cfg.data_options.raw_data_dir + metadata_json_path = cfg.data_options.metadata_json_path + + with open(os.path.join(metadata_json_path, "metadata.json"), "r") as f: + metadata = json.load(f) + logging.info(f"meta: {metadata}") + if mode != "stats": + writer = tf.io.TFRecordWriter( + os.path.join(metadata_json_path, mode + ".tfrecord") + ) + + # State the build names + try: + mode in ["train", "stats", "test"] + except: + logging.error("Mode not implemented, insert from [train|test|stats]") + raise + + if mode in ["train", "stats"]: + # for expanded version data + build_list = cfg.data_options.builds_train + elif mode == "test": + build_list = cfg.data_options.builds_test + + key_i = 0 + n_steps = 0 + temp_list_builds = [] + particles_list_builds = [] + velocity_list_builds = [] + acceleration_list_builds = [] + radius_list = [] + # Read and process each build data set + for build_name in build_list: + logging.info(f"\n\nProcess build: {build_name}") + # Get information for each build + # todo: move the compute anchor information outside + build_path = os.path.join(os.path.join(raw_data_dir, build_name), "out") + solution_list = glob.glob(build_path + "/solution-*.pvtu") + # solution_list = sorted(solution_list, key=get_solution_id) + solution_list = natsorted(solution_list) + logging.info(f"computing points from : {solution_list[-1]}") + solution_data_end = pv.read(solution_list[-1]) + _, index_list = get_data_position(solution_data_end) + + if cfg.data_options.add_anchor: + # Compute the anchor points from the sinter-end data + zplane_anchors = get_anchor_zplane(solution_data_end, index_list) + logging.info(f"Found points on the z-plane, cnt= {len(zplane_anchors)}") + zplane_anchors = list(zplane_anchors) + + anchor_point = get_anchor_point(solution_data_end, index_list) + logging.info( + f"Found anchor point with 0-displacement, p_id: {anchor_point}" + ) + else: + anchor_point = None + + for init_idx in range( + 0, 4, 1 + ): # for testing purpose, need to cover start point (92, 93) + logging.info(f"\n\n process sequence with init_idx: {init_idx}") + ( + particles_list, + temp_list, + senders_graph_i, + receivers_graph_i, + radius, + skip, + ) = read_solutions_data( + os.path.join(raw_data_dir, build_name), + init_idx=init_idx, + metadata=metadata, + ) + + if skip: + print("skip length different sequence ") + continue + + # Gather information across builds, prep for builds stats calculation + particles_list_builds += particles_list + + velocity_array = time_diff(np.array(particles_list)) + velocity_list = [velocity_array[i] for i in range(velocity_array.shape[0])] + velocity_list_builds += velocity_list + + acceleration_array = time_diff(velocity_array) + acceleration_list = [ + acceleration_array[i] for i in range(acceleration_array.shape[0]) + ] + acceleration_list_builds += acceleration_list + + radius_list.append(radius) + + temp_list_builds += temp_list + + # particles_array.shape(num_time_steps, nodes_per_build, xyz-dim) i.e. (12, 1107, 3) + particles_array = np.array(particles_list) + n_steps, n_particles, _ = particles_array.shape + logging.info(f"particles_array shape: {particles_array.shape}") + if init_idx == 0: + metadata["sequence_length"] = n_steps + + key_i += 1 + + ##### Write to TFRECORD ##### + if mode != "stats": + # TODO: reshape reason + particles_array = particles_array.reshape((n_steps, -1)).astype( + np.float64 + ) + + # # normalize data + particles_mean, particles_std = ( + metadata["pos_mean"], + metadata["pos_std"], + ) + particles_array = (particles_array - particles_mean) / particles_std + + # TODO: Compute and add the boundary condition here + # for normal particles, assign value = 2 + particles_type = np.repeat(2, n_particles) # [5 5 5 ... 5 5 5] (1107,) + if cfg.data_options.add_anchor: + particles_type[zplane_anchors] = 1 + particles_type[anchor_point] = 0 + particles_type = particles_type.tobytes() + + # Add global features + context_features = { + "particle_type": utils._bytes_feature(particles_type), + "key": utils.create_int_feature(key_i), + "senders": utils._bytes_feature(senders_graph_i.tobytes()), + "receivers": utils._bytes_feature(receivers_graph_i.tobytes()), + } + + features = tf.train.Features(feature=context_features) + + # Write to tfrecord + start_idx, end_idx = 0, particles_array.shape[0] + logging.info(f"write range: {start_idx}-{end_idx}") + write_tfrecord_entry( + writer, + features, + particles_array[start_idx:end_idx], + np.array(temp_list)[start_idx:end_idx], + ) + logging.info( + f"Finished writing to tfrecord, finale feature shape: {particles_array.shape}" + ) + + # Write metadata file + if mode == "stats": + metadata = compute_metadata_stats( + metadata, + particles_list_builds, + velocity_list_builds, + acceleration_list_builds, + radius_list, + temp_list_builds, + ) + + metadata["sequence_length"] = n_steps - 1 + metadata["dim"] = 3 + + with open(os.path.join(metadata_json_path, "metadata.json"), "w") as f: + json.dump(metadata, f) + elif mode == "test" or mode == "generalization": + logging.info(f"Finale feature shape: {particles_array.shape}") + metadata["sequence_length"] = particles_array.shape[0] - 1 + with open(os.path.join(metadata_json_path, "metadata.json"), "w") as f: + json.dump(metadata, f) + + +""" + Perform data processing over all builds defined in the raw_dir_path. + + Arguments: + cfg: Dictionary of parameters. + + """ +if __name__ == "__main__": + argv = sys.argv[1:] + main() diff --git a/examples/additive_manufacturing/sintering_physics/data_process/utils.py b/examples/additive_manufacturing/sintering_physics/data_process/utils.py new file mode 100644 index 0000000000..fd307d99e5 --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/data_process/utils.py @@ -0,0 +1,193 @@ +# ignore_header_test +# ruff: noqa: E402 + +# © Copyright 2023 HP Development Company, L.P. +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import csv +import glob +import os +import re + +import numpy as np +import tensorflow as tf +from natsort import natsorted + + +def read_raw_folder(data_dir): + """ + data_dir: Physics Simulation Engine raw output folder path, contains folder /out/solution-*.pvtu + Read the Physics Simulation Engine raw output folder + sort all timestep deformation files in time-series + """ + build_path = os.path.join(data_dir, "out") + solution_list = glob.glob(build_path + "/volume-deformation-*.pvtu") + # solution_list = sorted(solution_list, key=get_solution_id) + solution_list = natsorted(solution_list) + assert len(solution_list) >= 3, ( + "Need to have at least 3 solution files as input to start prediction or analysis!" + ) + return solution_list + + +def get_solution_id(solution_name): + """ + Read the Physics simulation file, current version file name: solution-xx.pvtu (2023-June) + Previous used version solution_name: volume-deformation, displacement-xx.pvtu + + return: sorted keys by int index + """ + m = re.search("solution-(\d+)", solution_name) + if m: + id = int(m.group(1)) + return id + return -1 + + +def time_diff(sequence_array): + """ + sequence_array: Position/ velocity/ acceleration numpy array, + return: step-wise difference + """ + return sequence_array[1:, :] - sequence_array[:-1, :] + + +def _bytes_feature(value): + """ + Returns a bytes_list from a string / byte. + """ + if isinstance(value, type(tf.constant(0))): + value = value.numpy() # BytesList won't unpack a string from an EagerTensor. + return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value])) + + +def create_int_feature(values): + """Create/ convert tf.Int64 feature""" + return tf.train.Feature(int64_list=tf.train.Int64List(value=[values])) + + +def get_radius(data): + """ + From data read from simulation path by pv.read, compute the radius by max(XYZ-distance) of a cell + Args: + data: data from pv.read + format: UnstructuredGrid i.e. (0x7f1636fe5520) + N Cells: 21970 + N Points: 175760 + X Bounds: -4.600e+01, 3.000e+00 + Y Bounds: -4.500e+00, 4.500e+00 + Z Bounds: 0.000e+00, 1.300e+01 + N Arrays: 11 + Returns: + float, i.e. 0.6 for meshsize=500 + """ + # Cell: vtkHexahedron i.e. cell_0 (0x55f4e9928f20) + # Debug: Off + # Modified Time: 85931 + # Reference Count: 2 + # Registered Events: (none) + # Number Of Points: 8 + # Bounds: + # Xmin,Xmax: (-2.5, -2) + # Ymin,Ymax: (4, 4.5) + # Zmin,Zmax: (4, 4.5) + # Point ids are: 0, 1, 3, 2, 4, 5, 7, 6 + # Merge Tolerance: 0.01 + # .... + cell_0 = data.GetCell(0) + + bounds = np.array((cell_0.GetBounds())) + + # compute the distance of each xyz-dimension + len_s = bounds[1::2] - bounds[0:-1:2] + radius = 1.2 * np.max(len_s) + return radius + + +def get_data_position(data): + """ + For the data read from one displacement-id.pvtu file, + iterate each point data, filter out the points in existed physical xyz-location + store the non-repeating point's uvw_values in + Args: + data: data read from displacement-id.pvtu file + + Returns: array of non-repeating nodes' current physical location (original location + displacement) + + """ + points = data.points + n_points = points.shape[0] + + # uvw_values: the feature name storing voxel deformation, + # depend on physics engine version, could also be data['u__v__w'], or other version i.e. data["displacement_U"] + uvw_values = data["u__v__w"] + + pos_list = [] + index_list = [] + + for point_index in range(n_points): + uvw = uvw_values[point_index] + + # Compute the deformed physical location from original physical location + pos = uvw + + index_list.append(point_index) + pos_list.append(pos) + + return np.array(pos_list), index_list + + +def read_configs(raw_data_path): + """ + Read the dataset information (without using solution.pvtu.series file) + + """ + # For each build, read the temperature profile at every time step + params_prm_path = os.path.join(raw_data_path, "params.prm") + assert os.path.exists(params_prm_path), ( + f"Temperature profile file params.prm not exists! {params_prm_path}" + ) + + # reading csv file + with open(params_prm_path, "r") as csvfile: + # creating a csv reader object + csvreader = csv.reader(csvfile) + + # extracting each data row one by one + for idx, row in enumerate(csvreader): + if "sintering_temperature_curve" in row[0]: + # i.e. ['set sintering_temperature_curve=0', '20', '16320', '1380', '23520', '1380'] + temp_row = row[1:] + elif "initial_time" in row[0]: + initial_time = int(float(row[0].split("=")[1].strip())) + elif "final_time" in row[0]: + final_time = int(float(row[0].split("=")[1].strip())) + elif "time_step" in row[0]: + time_step = int(float(row[0].split("=")[1].strip())) + elif "save_every_n_steps" in row[0]: + save_every_n_steps = int(float(row[0].split("=")[1].strip())) + + # temp_curve_list = read_temperature(params_prm_path) + # Add temperature data to each solution file + temp_row.insert(0, "0") + temp_curve_list = [float(i) for i in temp_row] + # Get the stage separation time-temperature pairs + stage_t_list = [] + stage_temp_list = [] + for idx in range(len(temp_curve_list) // 2): + # t_list.append(int(temp_curve_list[idx*2]) / 3600) + stage_t_list.append(int(temp_curve_list[idx * 2])) + stage_temp_list.append(int(temp_curve_list[idx * 2 + 1])) + + return (initial_time, final_time, time_step, save_every_n_steps), temp_curve_list diff --git a/examples/additive_manufacturing/sintering_physics/graph_dataset.py b/examples/additive_manufacturing/sintering_physics/graph_dataset.py new file mode 100644 index 0000000000..5814662c9c --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/graph_dataset.py @@ -0,0 +1,271 @@ +# ignore_header_test +# ruff: noqa: E402 + +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import functools +import os + +import tree + +try: + import tensorflow as tf +except ImportError: + raise ImportError( + "Mesh Graph Net Datapipe requires the Tensorflow library. Install the " + + "package at: https://www.tensorflow.org/install" + ) + +from reading_utils import parse_serialized_simulation_example, split_trajectory +from utils import _read_metadata, tf2torch + +INPUT_SEQUENCE_LENGTH = 5 # calculate the last 5 velocities. [options: 5, 10] +PREDICT_LENGTH = 1 # [options: 5] +LOSS_DECAY_FACTOR = 0.6 + +NUM_PARTICLE_TYPES = 3 +KINEMATIC_PARTICLE_ID = 0 # refers to anchor point +METAL_PARTICLE_ID = 2 # refers to normal particles +ANCHOR_PLANE_PARTICLE_ID = 1 # refers to anchor plane + + +def batch_concat(dataset, batch_size): + """We implement batching as concatenating on the leading axis.""" + + # We create a dataset of datasets of length batch_size. + windowed_ds = dataset.window(batch_size) + + # The plan is then to reduce every nested dataset by concatenating. We can + # do this using tf.data.Dataset.reduce. This requires an initial state, and + # then incrementally reduces by running through the dataset + + # Get initial state. In this case this will be empty tensors of the + # correct shape. + initial_state = tree.map_structure( + lambda spec: tf.zeros( # pylint: disable=g-long-lambda + shape=[0] + spec.shape.as_list()[1:], dtype=spec.dtype + ), + dataset.element_spec, + ) + + # We run through the nest and concatenate each entry with the previous state. + def reduce_window(initial_state, ds): + """reduce every nested dataset by concatenating, done using tf.data.Dataset.reduce""" + return ds.reduce(initial_state, lambda x, y: tf.concat([x, y], axis=0)) + + return windowed_ds.map( + lambda *x: tree.map_structure(reduce_window, initial_state, x) + ) + + +def get_input_fn(data_path, batch_size, prefetch_buffer_size, mode, split): + """Gets the learning simulation input function for tf.estimator.Estimator. + + Args: + data_path: the path to the dataset directory. + batch_size: the number of graphs in a batch. + mode: either 'one_step_train', 'one_step' or 'rollout' + split: either 'train', 'valid' or 'test. + + Returns: + The input function for the learning simulation model. + """ + + def input_fn(): + """Gets the learning simulation input function for tf.estimator""" + # Load the metadata of the dataset. + metadata = _read_metadata(data_path) + + # Create a tf.data.Dataset from the TFRecord. + # todo: try data exists + ds = tf.data.TFRecordDataset([os.path.join(data_path, f"{split}.tfrecord")]) + ds = ds.map( + functools.partial(parse_serialized_simulation_example, metadata=metadata) + ) + + if mode.startswith("one_step"): + # Splits an entire trajectory into chunks of n steps. (n=INPUT_SEQUENCE_LENGTH) + # Previous steps are used to compute the input velocities + split_with_window = functools.partial( + split_trajectory, + window_length=INPUT_SEQUENCE_LENGTH, + predict_length=PREDICT_LENGTH, + ) + ds = ds.flat_map(split_with_window) + # Splits a chunk into input steps and target steps + ds = ds.map(prepare_inputs) + # If in train mode, repeat dataset forever and shuffle. + if mode == "one_step_train": + ds.prefetch(buffer_size=prefetch_buffer_size) + ds = ds.repeat() + ds = ds.shuffle(512) + + # Custom batching on the leading axis. + print("before apply batch_concat ds: ", ds) + ds = batch_concat(ds, batch_size) + elif mode == "rollout": + if not batch_size == 1: + raise ValueError("Rollout evaluation only available for batch size 1") + + ds = ds.map(prepare_rollout_inputs) + else: + raise ValueError(f"mode: {mode} not recognized") + + return ds + + return input_fn + + +def prepare_inputs(tensor_dict): + """Prepares a single stack of inputs by calculating inputs and targets. + + Computes n_particles_per_example, which is a tensor that contains information + about how to partition the axis - i.e. which nodes belong to which graph. + + Adds a batch axis to `n_particles_per_example` and `step_context` so they can + later be batched using `batch_concat`. This batch will be the same as if the + elements had been batched via stacking. + + Note that all other tensors have a variable size particle axis, + and in this case they will simply be concatenated along that + axis. + + + + Args: + tensor_dict: A dict of tensors containing positions, and step context ( + if available). + + Returns: + A tuple of input features and target positions. + + """ + predict_length = PREDICT_LENGTH + + pos = tensor_dict["position"] + pos = tf.transpose(pos, perm=[1, 0, 2]) + + # The target position is the final step of the stack of positions. + target_position = pos[:, -predict_length:] + + # Remove the target from the input. + tensor_dict["position"] = pos[:, :-predict_length] + + # Compute the number of particles per example. + num_particles = tf.shape(pos)[0] + # Add an extra dimension for stacking via concat. + tensor_dict["n_particles_per_example"] = num_particles[tf.newaxis] + + num_edges = tf.shape(tensor_dict["senders"])[0] + tensor_dict["n_edges_per_example"] = num_edges[tf.newaxis] + + if "step_context" in tensor_dict: + # Take the input global context. We have a stack of global contexts, + # and we take the penultimate since the final is the target. + + # Method: input the entire sequence of sintering profile + tensor_dict["step_context"] = tf.reshape(tensor_dict["step_context"], [1, -1]) + + # if mode== inference: + # if "step_context" in tensor_dict: + # tensor_dict["step_context"] = tensor_dict["step_context"][-predict_length - 1] + # # Add an extra dimension for stacking via concat. + # tensor_dict["step_context"] = tensor_dict["step_context"][tf.newaxis] + + print( + "prepare inputs, tensor_dict['step_context'] shape: ", + tensor_dict["step_context"].shape, + ) + + return tensor_dict, target_position + + +def prepare_rollout_inputs(context, features): + """Prepares an inputs trajectory for rollout.""" + out_dict = {**context} + + pos = tf.transpose(features["position"], [1, 0, 2]) + # The target position is the final step of the stack of positions. + target_position = pos[:, -1] + + # can change whether to Remove the target from the input, with: out_dict['position'] = pos[:, :-1] + out_dict["position"] = pos + # if mode == "inference + # out_dict["position"] = pos[:, :-1] + + # Compute the number of nodes + out_dict["n_particles_per_example"] = [tf.shape(pos)[0]] + out_dict["n_edges_per_example"] = [tf.shape(context["senders"])[0]] + if "step_context" in features: + out_dict["step_context"] = tf.dtypes.cast(features["step_context"], tf.float64) + + out_dict["is_trajectory"] = tf.constant([True], tf.bool) + return out_dict, target_position + + +class GraphDataset: + """ + A dataset class for graph-based models, handling data loading and iteration + for training or evaluation in different modes. + """ + + # todo: update the size + def __init__( + self, + size=1000, + mode="one_step_train", + split="train", + data_path="None", + batch_size=1, + prefetch_buffer_size=100, + ): + self.mode = mode + self.dataset = get_input_fn( + data_path, batch_size, prefetch_buffer_size, mode=mode, split=split + )() + + if mode == "rollout": + # test / inference with test data size: + self.size = len(list(self.dataset)) + else: + # train + self.size = size + + self.dataset = iter(self.dataset) + self.pos = 0 + + def __len__(self): + return self.size + + def __next__(self): + # print("get next ds: pos/ size: ", self.pos, self.size) + if self.pos < self.size: + features, targets = self.dataset.get_next() + for key in features: + if key != "key": + features[key] = tf2torch(features[key]) + + targets = tf2torch(targets) + self.pos += 1 + return features, targets + else: + raise StopIteration + + def __iter__(self): + return self diff --git a/examples/additive_manufacturing/sintering_physics/inference.py b/examples/additive_manufacturing/sintering_physics/inference.py new file mode 100644 index 0000000000..2a870aa631 --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/inference.py @@ -0,0 +1,301 @@ +# ignore_header_test +# ruff: noqa: E402 + +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import json +import os +import time + +from tqdm import tqdm + +try: + import tensorflow as tf +except ImportError: + raise ImportError( + "Mesh Graph Net Datapipe requires the Tensorflow library. Install the " + + "package at: https://www.tensorflow.org/install" + ) +physical_devices = tf.config.list_physical_devices("GPU") + +try: + for device_ in physical_devices: + tf.config.experimental.set_memory_growth(device_, True) +except: + # Invalid device or cannot modify virtual devices once initialized. + pass + +import hydra +import torch +from graph_dataset import GraphDataset +from omegaconf import DictConfig +from utils import _combine_std, _read_metadata, Stats, cast + +from physicsnemo.distributed.manager import DistributedManager +from physicsnemo.utils.logging import ( + LaunchLogger, + PythonLogger, + RankZeroLoggingWrapper, +) +from physicsnemo.models.vfgn.graph_network_modules import VFGNLearnedSimulator + + +def Inference(rank_zero_logger, dist, cfg): + """ + Executes the testing phase for a graph-based model, generating and + storing predictions. + """ + rank_zero_logger.info( + "\n\n.......... Start calling model inference with defined data path ........\n\n" + ) + + # config test dataset + dataset = GraphDataset( + # size=C.num_steps, + mode="rollout", + split=cfg.general.eval_split, + data_path=cfg.data_options.data_path, + batch_size=cfg.train_options.batch_size, + ) + rank_zero_logger.info( + f"Initialized inference dataset with mode {dataset.mode}, dataset size {dataset.size}..." + ) + + metadata = _read_metadata(cfg.data_options.data_path) + acceleration_stats = Stats( + torch.DoubleTensor(cast(metadata["acc_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["acc_std"]), cfg.data_options.noise_std) + ), + ) + velocity_stats = Stats( + torch.DoubleTensor(cast(metadata["vel_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["vel_std"]), cfg.data_options.noise_std) + ), + ) + context_stats = Stats( + torch.DoubleTensor(cast(metadata["context_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["context_std"]), cfg.data_options.noise_std) + ), + ) + + normalization_stats = { + "acceleration": acceleration_stats, + "velocity": velocity_stats, + "context": context_stats, + } + + model = VFGNLearnedSimulator( + num_dimensions=metadata["dim"] * cfg.train_options.pred_len, + num_seq=cfg.train_options.input_seq_len, + boundaries=torch.DoubleTensor(metadata["bounds"]), + num_particle_types=cfg.data_options.NUM_PARTICLE_TYPES, + particle_type_embedding_size=16, + normalization_stats=normalization_stats, + ) + rank_zero_logger.info("Initialized model with VFGNLearnedSimulator") + + loaded = False + example_index = 0 + device = torch.device(cfg.general.device if torch.cuda.is_available() else "cpu") + model.setMessagePassingDevices([device]) + model.to(device) + + with torch.no_grad(): + for features, targets in tqdm(dataset): + if loaded is False: + # input feature size is dynamic, so need to forward model in CPU before loading into GPU + global_context = features["step_context"].to(device) + if global_context is None: + global_context_step = None + else: + global_context_step = global_context[:-1] + global_context_step = torch.reshape(global_context_step, [1, -1]) + + model.inference( + position_sequence=features["position"][ + :, 0 : cfg.train_options.input_seq_len + ].to(device), + n_particles_per_example=features["n_particles_per_example"].to( + device + ), + n_edges_per_example=features["n_edges_per_example"].to(device), + senders=features["senders"].to(device), + receivers=features["receivers"].to(device), + predict_length=cfg.train_options.pred_len, + particle_types=features["particle_type"].to(device), + global_context=global_context_step.to(device), + ) + + # Loading the pretrained model from model ckpt_path_vfgn + # For provided ckpt with missing keys, ignore + model.load_state_dict( + torch.load(cfg.data_options.ckpt_path_vfgn), strict=False + ) + # device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") + rank_zero_logger.info(f"Device: {device}") + rank_zero_logger.info( + f"Loaded model from ckpt path: {cfg.data_options.ckpt_path_vfgn}" + ) + + # config optimizer + # # todo: check msg passing device + # model.setMessagePassingDevices(["cuda:0"]) + # model = model.to(device) + model.eval() + loaded = True + + initial_positions = features["position"][ + :, : cfg.train_options.input_seq_len + ].to(device) + + global_context = features["step_context"].to(device) + rank_zero_logger.info( + f"\n Read global_context shape: {global_context.shape}" + ) + + rank_zero_logger.info( + f"\n Initial_positions shape: {initial_positions.shape}" + ) + + # num_steps = ground_truth_positions.shape[1] + num_steps = global_context.shape[0] - cfg.train_options.input_seq_len + rank_zero_logger.info(f"\n Start prediction for {num_steps} steps...... ") + + current_positions = initial_positions + updated_predictions = [] + + start_time = time.time() + rank_zero_logger.info(f"Start time: {start_time}\n") + + for step in range(num_steps): + rank_zero_logger.info(f"start predictiong step: {step}") + if global_context is None: + global_context_step = None + rank_zero_logger.info("global_context_step is None") + else: + read_step_context = global_context[ + : step + cfg.train_options.input_seq_len + ] + zero_pad = torch.zeros( + [global_context.shape[0] - read_step_context.shape[0] - 1, 1], + dtype=features["step_context"].dtype, + ).to(device) + + global_context_step = torch.cat([read_step_context, zero_pad], 0) + global_context_step = torch.reshape(global_context_step, [1, -1]) + + predict_positions = model.inference( + position_sequence=current_positions.to(device), + n_particles_per_example=features["n_particles_per_example"].to( + device + ), + n_edges_per_example=features["n_edges_per_example"].to(device), + senders=features["senders"].to(device), + receivers=features["receivers"].to(device), + predict_length=cfg.train_options.pred_len, + particle_types=features["particle_type"].to(device), + global_context=global_context_step.to(device), + ) + + # kinematic_mask = ( + # get_kinematic_mask(features["particle_type"]) + # .to(torch.bool) + # .to(device) + # ) + + predict_positions = predict_positions[:, 0].squeeze(1) + + # todo: implement the masking for predicted results for different particle types + # kinematic_mask = torch.repeat_interleave( + # kinematic_mask, repeats=predict_positions.shape[-1] + # ) + # kinematic_mask = torch.reshape( + # kinematic_mask, [-1, predict_positions.shape[-1]] + # ) + # next_position = torch.where( + # kinematic_mask, positions_ground_truth, predict_positions + # ) + next_position = predict_positions + + updated_predictions.append(next_position) + current_positions = torch.cat( + [current_positions[:, 1:], next_position.unsqueeze(1)], axis=1 + ) + + updated_predictions = torch.stack(updated_predictions) + rank_zero_logger.info( + f"\n\n finished running all stages, initial_positions shape: {initial_positions.shape},\n" + f"\n updated_predictions shape: {updated_predictions.shape}" + ) + + initial_positions_list = initial_positions.cpu().numpy().tolist() + updated_predictions_list = updated_predictions.cpu().numpy().tolist() + particle_types_list = features["particle_type"].cpu().numpy().tolist() + global_context_list = global_context.cpu().numpy().tolist() + + rollout_op = { + "initial_positions": initial_positions_list, + "predicted_rollout": updated_predictions_list, + "particle_types": particle_types_list, + "global_context": global_context_list, + } + + # Add a leading axis, since Estimator's predict method insists that all + # tensors have a shared leading batch axis fo the same dims. + # rollout_op = tree.map_structure(lambda x: x.numpy(), rollout_op) + + rollout_op["metadata"] = metadata + filename = f"rollout_{cfg.general.eval_split}_{example_index}.json" + filename = os.path.join(cfg.data_options.output_path, filename) + if not os.path.exists(cfg.data_options.output_path): + os.makedirs(cfg.data_options.output_path) + with open(filename, "w") as file_object: + json.dump(rollout_op, file_object) + + example_index += 1 + rank_zero_logger.info(f"prediction time: {time.time() - start_time}\n") + + +@hydra.main(version_base=None, config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + """ + Triggers the train or test phase based on the configuration. + """ + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + rank_zero_logger = RankZeroLoggingWrapper(logger, dist) # Rank 0 logger + + if cfg.general.mode == "rollout": + Inference(rank_zero_logger, dist, cfg) + else: + raise NotImplementedError("Mode not implemented ") + + +if __name__ == "__main__": + # tf.disable_v2_behavior() + LaunchLogger.initialize() # PhysicsNeMo launch logger + logger = PythonLogger("main") # General python logger + logger.file_logging() + + main() diff --git a/examples/additive_manufacturing/sintering_physics/reading_utils.py b/examples/additive_manufacturing/sintering_physics/reading_utils.py new file mode 100644 index 0000000000..e748d104d3 --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/reading_utils.py @@ -0,0 +1,201 @@ +# ignore_header_test +# ruff: noqa: E402 + +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import functools + +import numpy as np + +try: + import tensorflow as tf +except ImportError: + raise ImportError( + "Mesh Graph Net Datapipe requires the Tensorflow library. Install the " + + "package at: https://www.tensorflow.org/install" + ) + +# Create a description of the features. +_FEATURE_DESCRIPTION = { + "position": tf.io.VarLenFeature(tf.string), +} + +_FEATURE_DESCRIPTION_WITH_GLOBAL_CONTEXT = _FEATURE_DESCRIPTION.copy() +_FEATURE_DESCRIPTION_WITH_GLOBAL_CONTEXT["step_context"] = tf.io.VarLenFeature( + tf.string +) + +_FEATURE_DTYPES = { + "position": {"in": np.float64, "out": tf.float64}, + "step_context": {"in": np.float64, "out": tf.float64}, +} + +_CONTEXT_FEATURES = { + "key": tf.io.FixedLenFeature([], tf.int64, default_value=0), + "particle_type": tf.io.VarLenFeature(tf.string), + "senders": tf.io.VarLenFeature(tf.string), + "receivers": tf.io.VarLenFeature(tf.string), + # 'temperature': tf.io.VarLenFeature(tf.string) +} + + +def convert_to_tensor(x, encoded_dtype): + """Convert inputs to tensor""" + if len(x) == 1: + out = np.frombuffer(x[0].numpy(), dtype=encoded_dtype) + else: + out = [] + for el in x: + out.append(np.frombuffer(el.numpy(), dtype=encoded_dtype)) + out = tf.convert_to_tensor(np.array(out)) + return out + + +def parse_serialized_simulation_example(example_proto, metadata): + """ + Parses a serialized simulation tf.SequenceExample. + + Args: + example_proto: A string encoding of the tf.SequenceExample proto. + metadata: A dict of metadata for the dataset. + + Returns: + context: A dict, with features that do not vary over the trajectory. + parsed_features: A dict of tf.Tensors representing the parsed examples + across time, where axis zero is the time axis. + + """ + if "context_mean" in metadata: + feature_description = _FEATURE_DESCRIPTION_WITH_GLOBAL_CONTEXT + else: + feature_description = _FEATURE_DESCRIPTION + + context, parsed_features = tf.io.parse_single_sequence_example( + example_proto, + context_features=_CONTEXT_FEATURES, + sequence_features=feature_description, + ) + + for feature_key, item in parsed_features.items(): + print("feature_key", feature_key) + convert_fn = functools.partial( + convert_to_tensor, encoded_dtype=_FEATURE_DTYPES[feature_key]["in"] + ) + parsed_features[feature_key] = tf.py_function( + convert_fn, inp=[item.values], Tout=_FEATURE_DTYPES[feature_key]["out"] + ) + + # There is an extra frame at the beginning so we can calculate pos change + # for all frames used in the paper. + position_shape = [metadata["sequence_length"] + 1, -1, metadata["dim"]] + print(f"\n\nposition shape: {position_shape}") + print(f"parsed_features['position'] shape: {parsed_features['position'].shape}") + + # Reshape positions to correct dim: + parsed_features["position"] = tf.reshape( + parsed_features["position"], position_shape + ) + + # Set correct shapes of the remaining tensors. + sequence_length = metadata["sequence_length"] + 1 + if "context_mean" in metadata: + context_feat_len = len(metadata["context_mean"]) + parsed_features["step_context"] = tf.reshape( + parsed_features["step_context"], [sequence_length, context_feat_len] + ) + + # Decode particle type explicitly + print("decode particle_type") + context["particle_type"] = tf.py_function( + functools.partial(convert_fn, encoded_dtype=np.int64), + inp=[context["particle_type"].values], + Tout=[tf.int64], + ) + context["particle_type"] = tf.reshape(context["particle_type"], [-1]) + + context["senders"] = tf.py_function( + functools.partial(convert_fn, encoded_dtype=np.int64), + inp=[context["senders"].values], + Tout=[tf.int64], + ) + context["senders"] = tf.reshape(context["senders"], [-1]) + + context["receivers"] = tf.py_function( + functools.partial(convert_fn, encoded_dtype=np.int64), + inp=[context["receivers"].values], + Tout=[tf.int64], + ) + context["receivers"] = tf.reshape(context["receivers"], [-1]) + + return context, parsed_features + + +def split_trajectory(context, features, window_length=7, predict_length=10): + """Splits trajectory into sliding windows.""" + # Our strategy is to make sure all the leading dimensions are the same size, + # then we can use from_tensor_slices. + + trajectory_length = features["position"].get_shape().as_list()[0] + + # We then stack window_length position changes so the final + # trajectory length will be - window_length +1 (the 1 to make sure we get + # the last split). + input_trajectory_length = trajectory_length - window_length - predict_length + 1 + + model_input_features = {} + # Prepare the context features per step. + # Repeat the particle types for each window step + model_input_features["particle_type"] = tf.tile( + tf.expand_dims(context["particle_type"], axis=0), [input_trajectory_length, 1] + ) + + model_input_features["senders"] = tf.tile( + tf.expand_dims(context["senders"], axis=0), [input_trajectory_length, 1] + ) + + model_input_features["receivers"] = tf.tile( + tf.expand_dims(context["receivers"], axis=0), [input_trajectory_length, 1] + ) + + # todo: change the hard-coded trajectory length to be the entire global context (/ sintering profile) sequence length + # sequence length here is the default sintering 2-stage total length + # trajectory_length = 14 + 24 + + # Process the parsed_features + if "step_context" in features: + global_stack = [] + for idx in range(input_trajectory_length): + # append all the previous temperature history, use an additional module to concat to final vector as global features + read_step_context = features["step_context"][: idx + window_length] + zero_pad = tf.zeros( + [trajectory_length - read_step_context.shape[0] - 1, 1], + dtype=features["step_context"].dtype, + ) + + read_step_context = tf.concat([read_step_context, zero_pad], 0) + global_stack.append(read_step_context) + model_input_features["step_context"] = tf.stack(global_stack) + + pos_stack = [ + features["position"][idx : idx + window_length + predict_length] + for idx in range(input_trajectory_length) + ] + model_input_features["position"] = tf.stack(pos_stack) + + return tf.data.Dataset.from_tensor_slices(model_input_features) diff --git a/examples/additive_manufacturing/sintering_physics/render_rollout.py b/examples/additive_manufacturing/sintering_physics/render_rollout.py new file mode 100644 index 0000000000..6520f0e62c --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/render_rollout.py @@ -0,0 +1,490 @@ +# ignore_header_test +# ruff: noqa: E402 + +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ============================================================================ +"""Simple matplotlib rendering of a rollout prediction against ground truth. + +Usage (from parent directory): + +`python -m learning_to_simulate.render_rollout --rollout_path={OUTPUT_PATH}/rollout_test_1.json` + +Where {OUTPUT_PATH} is the output path passed to `train.py` in "eval_rollout" +mode. + +It may require installing Tkinter with `sudo apt-get install python3.7-tk`. + +""" + +import json +import os + +import hydra +import matplotlib.pyplot as plt +import numpy as np +from matplotlib import animation +from omegaconf import DictConfig + +from physicsnemo.distributed.manager import DistributedManager +from physicsnemo.utils.logging import ( + LaunchLogger, + PythonLogger, + RankZeroLoggingWrapper, +) + + +def compute_accuracy_percent(rollout_data_tuple, trajectory_len): + """ + This function compute the percentage accuracy of uvw-channels: + (abs(gt- pred)) / gt_dispalcement + + :param rollout_data: + :param trajectory_len: num of rollout steps + :return: + percentage_rollout_list: mean accuracy (%) of each timestep, across 3-channels + percent_uvw: (dim1: timesteps, dim2: 3-dimensions) + """ + (init_position, gt_position, pred_position) = rollout_data_tuple + init_position = init_position[0, ...] + gt_position = gt_position + pred_position = pred_position + + percentage_rollout_list = [] + percent_uvw = [] + for i in range(trajectory_len): + # Iterate for each rollout trajectory step, each step shape, i.e.:(21969, 3) + # Compute the different percentage: (abs(gt- pred)) / gt_dispalcement + + # Compute the mean diff of 3-channels + diff = np.float64((gt_position[i, ...] - pred_position[i, ...]) ** 2) + diff_point = np.sqrt(np.sum(diff, axis=1)) + + gt_displacement = np.float64((gt_position[i, ...] - init_position) ** 2) + gt_displacement_point = np.sqrt(np.sum(gt_displacement, axis=1)) + + # Set the episilon (mean diff_point: e-05, mean gt_displacement_point: e-03) + e = 1e-07 + # Exclude the point where displacement == 0 + nonzero_index = np.where(gt_displacement_point != 0)[0] + zero_index = np.where(gt_displacement_point == 0)[0] + # percent_point = np.mean(diff_point[nonzero_index] / gt_displacement_point[nonzero_index]) + percent_point_2 = np.mean( + np.mean(diff_point[nonzero_index] / gt_displacement_point[nonzero_index]) + + np.mean(diff_point[zero_index] / e) + ) + + percent_point = np.sum(diff_point) / (np.sum(gt_displacement_point) + e) + print( + f"mean diff_point: {np.sum(diff_point)}, mean gt_displacement_point: {np.sum(gt_displacement_point)}" + ) + print(percent_point, percent_point_2) + + # Compute mean for 3 axis, represent u, v, w values + diff_abs = np.absolute( + np.float64((gt_position[i, ...] - pred_position[i, ...])) + ) + gt_displacement_abs = np.absolute( + np.float64((gt_position[i, ...] - init_position)) + ) + percent_uvw_time = np.sum(diff_abs, axis=0) / np.sum( + gt_displacement_abs, axis=0 + ) + + percentage_rollout_list.append(percent_point) + percent_uvw.append(percent_uvw_time) + print(percentage_rollout_list) + return percentage_rollout_list, np.array(percent_uvw) + + +def plot_rollout_percentage( + percentage_rollout_list, percent_uvw, save_path, save_name, build_name="ladder" +): + """ + bar plot of rollout percentage loss + Args: + percentage_rollout_list: ave of 3-dims percentage loss + percent_uvw: (trajectory_length, dim=3) + Returns: + None + """ + print( + "plot_rollout_percentage, num of rollout steps: ", len(percentage_rollout_list) + ) + n = len(percentage_rollout_list) + + # creating the bar plot, plot the mean accuracy across uvw 3-channels + # x-axis: rollout timsteps, y-axis: accuracy + name_list = [str(x) for x in range(len(percentage_rollout_list))] + name_values = [x for x in range(len(percentage_rollout_list))] + plt.bar(name_list, percentage_rollout_list, color="silver") + + # Add u, v, w accuracy curves on the plot + plt.plot(name_values, percent_uvw[:, 0], "b-", label="u-displacement") + plt.plot(name_values, percent_uvw[:, 1], "g-", label="v-displacement") + plt.plot(name_values, percent_uvw[:, 2], "y-", label="w-displacement") + plt.legend( + ["u-displacement", "v-displacement", "w-displacement"], loc="lower right" + ) + plt.legend() + + # Add 10% cut-off line, this is the accuracy tolerance requirement + cutoff_line = [0.1 for i in range(len(percentage_rollout_list))] + plt.plot(name_values, cutoff_line, "r-") + + cutoff_line = [0.03 for i in range(len(percentage_rollout_list))] + plt.plot(name_values, cutoff_line, "r.") + + # Set x-y axis range + plt.xlim(0, n) + plt.ylim(0, 0.2) + + plt.xlabel("Rollout steps") + plt.ylabel("(abs(gt- pred)) / gt (%)") + plt.title("Percent loss as compare to VF " + build_name) + plt.savefig(os.path.join(save_path, save_name + "_" + build_name + ".png")) + plt.close() + + +def plot_3Danime(rollout_data, pred_denorm, save_name): + print("\n\nplot_3Danime: ") + fig = plt.figure(figsize=(10, 5)) + plot_info = [] + # choose the bounds set in the metadata, or manually set plot bounds + bounds = rollout_data["metadata"]["bounds"] + bounds = [[-1.5, 1.5], [-1.5, 1.5], [-1, 0]] + + for ax_i, (label, rollout_field) in enumerate( + [("Ground truth", "ground_truth_rollout"), ("Prediction", "predicted_rollout")] + ): + # Append the initial positions to get the full trajectory. + ax = fig.add_subplot(1, 2, (ax_i + 1), projection="3d") + # title = label + title = ax.set_title(label) + ax.set_xlim3d(bounds[0][0], bounds[0][1]) + # ax.set_xticks(np.arange(bounds[0][0]-0.25, bounds[0][1]+0.25, 0.5)) + ax.set_xlabel("X") + ax.set_ylim3d(bounds[1][0], bounds[1][1]) + # ax.set_yticks(np.arange(bounds[1][0]-0.25, bounds[1][1]+0.25, 0.5)) + ax.set_ylabel("Y") + ax.set_zlim3d(bounds[2][0], bounds[2][1]) + ax.set_zlabel("Z") + # ax.set_xticks([]) + # ax.set_yticks([]) + # ax.set_zticks([]) + # ax.view_init(40, 50) + ax.auto_scale_xyz + ax.view_init(40, 55) + + data = rollout_data[rollout_field][0, ...] + (graph,) = ax.plot( + data[:, 0], data[:, 1], data[:, 2], linestyle="", marker="o", ms=1 + ) + points = rollout_data[rollout_field] + # points = { + # particle_type: ax.scatter3D([], [], [], "o", color=color)[0] + # for particle_type, color in TYPE_TO_COLOR.items()} + plot_info.append((ax, label, points, graph)) + + num_steps = pred_denorm.shape[0] + print("predicted shape: ", num_steps, pred_denorm.shape) + + def update_graph(num): + outputs = [] + for _, label, points, graph in plot_info: + data = points[num, ...] + graph.set_data(data[:, 0], data[:, 1]) + graph.set_3d_properties(data[:, 2]) + title.set_text("{}, time={}".format(label, num)) + outputs.append(graph) + return outputs + # return title, graph, + + ani = animation.FuncAnimation( + fig, update_graph, num_steps, interval=70, blit=False, repeat=True + ) + + # Save gif + save = True + if save: + Writer = animation.writers["ffmpeg"] + writer = Writer( + fps=30, + metadata=dict(artist="Me"), + bitrate=1800, + extra_args=["-vcodec", "libx264"], + ) + ani.save(save_name + "-3d-animated.mp4", writer=writer) + + plt.close() + + +def plot_mean_error(rollout_data_tuple, metadata, plot_steps, rollout_path, build_name): + (init_position, gt_position, pred_position) = rollout_data_tuple + + pos_mean = metadata["pos_mean"] + pos_std = metadata["pos_std"] + + gt_position = gt_position * pos_std + pos_mean + pred_position = pred_position * pos_std + pos_mean + + rollout_list = [] + rollout_list_max = [] + rollout_uvw = [] + for i in range(plot_steps): + # Compute the mean diff of 3-channels + diff = np.absolute(gt_position[i, ...] - pred_position[i, ...]) + me_ = np.mean(diff) + max_ = np.max(diff) + print("step me, max: ", i, me_, max_) + + # Compute mean for 3 axis, represent u, v, w values + uvw_time = np.mean(diff, axis=0) + + rollout_list.append(me_) + rollout_list_max.append(max_) + rollout_uvw.append(uvw_time) + + print("rolloutlist shape :", np.array(rollout_list).shape) + print("rollout_uvw shape :", np.array(rollout_uvw).shape) + + ########## Plot ########## + # x-axis: rollout timsteps, y-axis: accuracy + name_list = [str(x) for x in range(len(rollout_list))] + name_values = [x for x in range(len(rollout_list))] + # plt.bar(name_list, rollout_list, color="silver") + plt.bar(name_list, rollout_list, color="silver") + plt.plot(name_values, rollout_list_max, "r.") + + # Add u, v, w accuracy curves on the plot + ######### Comment out for adding the xyz-deformation curves + # rollout_uvw = np.array(rollout_uvw) + # plt.plot(name_values, rollout_uvw[:, 0], "b-", label='u-displacement') + # plt.plot(name_values, rollout_uvw[:, 1], "g-", label='v-displacement') + # plt.plot(name_values, rollout_uvw[:, 2], "y-", label='w-displacement') + # plt.legend(["u-displacement", "v-displacement", "w-displacement"], + # loc="lower right", prop={'size': 10}) + # plt.legend() + + # cutoff_line = [0.001 for i in range(len(name_values))] + # plt.plot(name_values, cutoff_line, "r.") + + # Set x-y axis range + plt.xlim(0, plot_steps) + # plt.ylim(0, 0.05) + plt.xticks(np.arange(0, plot_steps, 10)) + # plt.yticks(np.arange(0, 250, 50)) + plt.xticks(size=30) + plt.yticks(size=30) + + plt.xlabel("Rollout steps", fontsize=30) + plt.ylabel("Accuracy (Mean error/mm)", fontsize=30) + # plt.title("Mean error as compare to VF " + build_name) + plt.savefig( + os.path.join(os.path.dirname(rollout_path), "mean_error_" + build_name + ".png") + ) + plt.close() + + return rollout_list, rollout_uvw + + +def plot_mean_error_temperature( + rollout_data, metadata, plot_steps, rollout_path, build_name +): + gt_position = rollout_data["ground_truth_rollout"] + pred_position = rollout_data["predicted_rollout"] + temperatures = rollout_data["global_context"][3:] + print("rollout shape: ", pred_position.shape) # rollout shape: (164, 100764, 3) + print("temperature: ", temperatures.shape) + + pos_mean = metadata["pos_mean"] + pos_std = metadata["pos_std"] + + gt_position = gt_position * pos_std + pos_mean + pred_position = pred_position * pos_std + pos_mean + + rollout_list = [] + rollout_uvw = [] + for i in range(plot_steps): + # Compute the mean diff of 3-channels + diff = np.absolute(gt_position[i, ...] - pred_position[i, ...]) + me_ = np.mean(diff) + # print("step me: ", i, me_) + + # Compute mean for 3 axis, represent u, v, w values + uvw_time = np.mean(diff, axis=0) + + rollout_list.append(me_ * 1000) + rollout_uvw.append(uvw_time * 1000) + + print("rolloutlist shape :", np.array(rollout_list).shape) + print("rollout_uvw shape :", np.array(rollout_uvw).shape) + + ########## Plot ########## + fig, ax = plt.subplots(figsize=(20, 7)) + name_values = [x for x in range(len(rollout_list))] + + # Add u, v, w accuracy curves on the plot + rollout_uvw = np.array(rollout_uvw) + ax.plot(name_values, rollout_uvw[:, 0], "b-", label="u-displacement") + ax.plot(name_values, rollout_uvw[:, 1], "g-", label="v-displacement") + ax.plot(name_values, rollout_uvw[:, 2], "y-", label="w-displacement") + ax.legend( + ["u-displacement", "v-displacement", "w-displacement"], + loc="lower right", + prop={"size": 10}, + ) + ax.legend() + + ax.set_xlabel("Rollout steps", fontsize=20) + ax.set_ylabel("Accuracy (Mean error/mm)", fontsize=20) + + # twin object for two different y-axis on the sample plot + ax2 = ax.twinx() + # make a plot with different y-axis using second axis object + temperatures_list = temperatures.flatten() + name_values = [x for x in range(len(temperatures_list))] + # print("temperature: ", temperatures_list, temperatures_list.shape) + ax2.plot(name_values, temperatures_list, "r-", linewidth=2) + ax2.set_ylabel("Temperature", color="red", fontsize=14) + + plt.title("Mean error as compare to VF " + build_name) + plt.savefig( + os.path.join( + os.path.dirname(rollout_path), "mean_error_wtemp" + build_name + ".png" + ) + ) + plt.close() + + return rollout_list, rollout_uvw + + +@hydra.main(version_base=None, config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + """ + Read from the prediction output, which is saved in rollout_path, i.e. rollouts/rollout_test_0.json + initial_positions.shape:(time_Step_size for init input, partical_size, dim), i.e.(5, 1394, 3) + predicted_rollout.shape:(time_Step_size for predicted steps, partical_size, dim), i.e.(29, 1394, 3) + ground_truth_rollout.shape:(time_Step_size for GT steps, partical_size, dim), i.e.(29, 1394, 3) + """ + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + rank_zero_logger = RankZeroLoggingWrapper(logger, dist) # Rank 0 logger + + if not cfg.test_options.rollout_path: + raise ValueError("A `rollout_path` must be passed.") + with open(cfg.test_options.rollout_path, "rb") as file: + rollout_data = json.load(file) + + metadata_path_json = os.path.join(cfg.test_options.metadata_path, "metadata.json") + rank_zero_logger.info(f"load metadata from path: {metadata_path_json}") + print("load metadata from path: ", metadata_path_json) + with open(metadata_path_json, "r") as f: + metadata = json.load(f) + + pos_mean = metadata["pos_mean"] + pos_std = metadata["pos_std"] + rank_zero_logger.info( + f"load from the metadata partical position mean={pos_mean}/ std={pos_std}" + ) + + # after transpose, vector shape + initial_positions = np.asarray((rollout_data["initial_positions"])) + predicted_rollout = np.asarray((rollout_data["predicted_rollout"])) + ground_truth_rollout = np.asarray((rollout_data["ground_truth_rollout"])) + + initial_positions = np.transpose(initial_positions, [1, 0, 2]) + ground_truth_rollout = np.transpose(ground_truth_rollout, [1, 0, 2]) + + # metadata recorded sequence_length = len(initial_positions) + len(predicted_rollout) -1 + rank_zero_logger.info(f"initial steps #= {len(initial_positions)}") + rank_zero_logger.info(f"pred steps #= {len(predicted_rollout)}") + n = len(predicted_rollout) + + # Compute prediction accuracy if ground-truth data available + for i in range(n): + # Denormalize with saved metadata + gt_step = ground_truth_rollout[i] + gt_step = pos_std * gt_step + pos_mean + + pred_step = predicted_rollout[i] + pred_step = pos_std * pred_step + pos_mean + + diff_step = gt_step - pred_step + diff_step = diff_step.reshape((-1, 3)) + mse0 = np.square(diff_step) + me0 = np.absolute(diff_step) + + mse = np.mean(mse0) + me = np.mean(me0) + # print(f"ground truth: {A}, me: {me}") + rank_zero_logger.info(f"{i} step, me: {me}, mse: {mse}") + + # Compute the entire sintering profile prediction accuracy + gt_seq = ground_truth_rollout[:n, ...] + gt_seq = pos_std * gt_seq + pos_mean + + pred_seq = predicted_rollout[:n, ...] + pred_seq = pos_std * pred_seq + pos_mean + diff_seq = gt_seq - pred_seq + diff_seq = diff_seq.reshape((-1, 3)) + mse0 = np.square(diff_seq) + me0 = np.absolute(diff_seq) + + mse = np.mean(mse0) + me = np.mean(me0) + rank_zero_logger.info(f"rollout shape: {gt_seq.shape}, total me: {me}") + + ############ PLOT ############ + # If plot tolerance range + print("Compute percentage rollout \n\n") + rollout_data_tuple = (initial_positions, ground_truth_rollout, predicted_rollout) + if cfg.test_options.plot_tolerance_range: + percentage_rollout_list, percent_uvw = compute_accuracy_percent( + rollout_data_tuple, n + ) + plot_rollout_percentage( + percentage_rollout_list, + percent_uvw, + os.path.dirname(cfg.test_options.rollout_path), + "rollout_acc_percent", + build_name=cfg.test_options.test_build_name, + ) + + print("\n\n plot mean error") + plot_mean_error( + rollout_data_tuple, + metadata, + n, + cfg.test_options.rollout_path, + cfg.test_options.test_build_name, + ) + + if cfg.test_options.plot_3d: + # Plot 3D visualization + pred_denorm = predicted_rollout * pos_std + pos_mean + plot_3Danime(rollout_data, pred_denorm, cfg.test_options.rollout_path[:-4]) + + +if __name__ == "__main__": + LaunchLogger.initialize() # PhysicsNeMo launch logger + logger = PythonLogger("main") # General python logger + logger.file_logging() + + main() diff --git a/examples/additive_manufacturing/sintering_physics/requirements.txt b/examples/additive_manufacturing/sintering_physics/requirements.txt new file mode 100644 index 0000000000..667a0024d5 --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/requirements.txt @@ -0,0 +1,3 @@ +# pyvista is optional, required if need to run data preprocessing from raw simulation +# pyvista==0.32.1 +tensorflow>=2.15,<3.0 # generate tfrecord diff --git a/examples/additive_manufacturing/sintering_physics/train.py b/examples/additive_manufacturing/sintering_physics/train.py new file mode 100644 index 0000000000..4062337f6e --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/train.py @@ -0,0 +1,734 @@ +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import ast +import json +import math +import os +import random +import time + +import numpy as np +from tqdm import tqdm + +try: + import tensorflow as tf +except ImportError: + raise ImportError( + "Mesh Graph Net Datapipe requires the Tensorflow library. Install the " + + "package at: https://www.tensorflow.org/install" + ) + +import hydra +import torch +from graph_dataset import GraphDataset +from omegaconf import DictConfig +from torch.utils.tensorboard import SummaryWriter +from utils import ( + Stats, + cast, + _combine_std, + _read_metadata, + get_anchor_z_mask, + get_kinematic_mask, + get_metal_mask, + weighted_square_error, +) + +from physicsnemo.distributed.manager import DistributedManager +from physicsnemo.utils.logging import ( + LaunchLogger, + PythonLogger, + RankZeroLoggingWrapper, +) +from physicsnemo.models.vfgn.graph_network_modules import VFGNLearnedSimulator + +physical_devices = tf.config.list_physical_devices("GPU") +try: + for device_ in physical_devices: + tf.config.experimental.set_memory_growth(device_, True) +except: + # Invalid device or cannot modify virtual devices once initialized. + pass + + +def Train(rank_zero_logger, dist, cfg: DictConfig): + """ + Trains a graph-based model, evaluating and saving its performance periodically. + """ + + # config dataset + dataset = GraphDataset( + size=cfg.train_options.num_steps, + data_path=cfg.data_options.data_path, + batch_size=cfg.train_options.batch_size, + prefetch_buffer_size=cfg.train_options.prefetch_buffer_size, + ) + rank_zero_logger.info( + f"Initialized train dataset with mode {dataset.mode}, dataset size {dataset.size}..." + ) + + testDataset = GraphDataset( + size=cfg.train_options.num_steps, + split="test", + data_path=cfg.data_options.data_path, + batch_size=cfg.train_options.batch_size, + ) + rank_zero_logger.info( + f"Initialized testDataset with mode {testDataset.mode}, dataset size {testDataset.size}..." + ) + + # config model + metadata = _read_metadata(cfg.data_options.data_path) + acceleration_stats = Stats( + torch.DoubleTensor(cast(metadata["acc_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["acc_std"]), cfg.data_options.noise_std) + ), + ) + velocity_stats = Stats( + torch.DoubleTensor(cast(metadata["vel_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["vel_std"]), cfg.data_options.noise_std) + ), + ) + context_stats = Stats( + torch.DoubleTensor(cast(metadata["context_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["context_std"]), cfg.data_options.noise_std) + ), + ) + + normalization_stats = { + "acceleration": acceleration_stats, + "velocity": velocity_stats, + "context": context_stats, + } + model = VFGNLearnedSimulator( + num_dimensions=metadata["dim"] * cfg.train_options.pred_len, + num_seq=cfg.train_options.input_seq_len, + boundaries=torch.DoubleTensor(metadata["bounds"]), + num_particle_types=cfg.data_options.NUM_PARTICLE_TYPES, + particle_type_embedding_size=16, + normalization_stats=normalization_stats, + ) + + writer = SummaryWriter(log_dir=cfg.data_options.ckpt_path_vfgn) + + optimizer = None + # todo : check device + device = "cpu" + step = 0 + running_loss = 0.0 + best_loss = 1000.0 + + rank_zero_logger.info("Training started...") + + for features, targets in tqdm(dataset): + inputs = features["position"] + particle_types = features["particle_type"] + + sampled_noise = model.get_random_walk_noise_for_position_sequence( + inputs, noise_std_last_step=cfg.data_options.noise_std + ) + if cfg.train_options.loss.startswith("anchor"): + rank_zero_logger.info("Compute noise_mask...") + + non_kinematic_mask = get_metal_mask(features["particle_type"]) + noise_mask = ( + non_kinematic_mask.to(sampled_noise.dtype).unsqueeze(-1).unsqueeze(-1) + ) + + anchor_plane_mask = get_anchor_z_mask(features["particle_type"]) + noise_anchor_plane_mask = ( + anchor_plane_mask.to(sampled_noise.dtype).unsqueeze(-1).unsqueeze(-1) + ) + zero_mask = torch.zeros( + noise_anchor_plane_mask.shape, dtype=noise_anchor_plane_mask.dtype + ) + noise_anchor_plane_mask = torch.cat( + [noise_anchor_plane_mask, noise_anchor_plane_mask, zero_mask], axis=-1 + ) + + noise_mask = torch.repeat_interleave(noise_mask, repeats=3, dim=-1) + noise_mask += noise_anchor_plane_mask + + else: + non_kinematic_mask = torch.logical_not( + get_kinematic_mask(particle_types).bool() + ) + noise_mask = ( + non_kinematic_mask.to(sampled_noise.dtype).unsqueeze(-1).unsqueeze(-1) + ) + + sampled_noise *= noise_mask + + pred_target = model( + next_positions=targets.to(device), + position_sequence=inputs.to(device), + position_sequence_noise=sampled_noise.to(device), + n_particles_per_example=features["n_particles_per_example"].to(device), + n_edges_per_example=features["n_edges_per_example"].to(device), + senders=features["senders"].to(device), + receivers=features["receivers"].to(device), + predict_length=cfg.train_options.pred_len, + particle_types=features["particle_type"].to(device), + global_context=features.get("step_context").to(device), + ) + + if optimizer is None: + # first data need to inference the feature size + device = torch.device( + cfg.general.device if torch.cuda.is_available() else "cpu" + ) + rank_zero_logger.info( + f"*******************device: {device} ****************" + ) + # print("*******************device: {} ****************".format(device)) + # config optimizer + message_passing_devices = ast.literal_eval( + cfg.general.message_passing_devices + ) + model.setMessagePassingDevices(message_passing_devices) + model = model.to(device) + optimizer = torch.optim.Adam(model.parameters(), lr=1e-5) + if cfg.general.fp16: + # double check if amp installed + try: + from apex import amp + + model, optimizer = amp.initialize(model, optimizer, opt_level="O1") + except ImportError as e: + print("Apex package not available -> ", e) + exit() + + scheduler = torch.optim.lr_scheduler.ExponentialLR( + optimizer, gamma=0.1, verbose=True + ) + decay_steps = int(5e6) + # input feature size is dynamic, so need to forward model in CPU before loading into GPU + # first step is forwarded in CPU, so skip the first step + continue + + pred_acceleration, target_acceleration = pred_target + # Calculate the L2 loss and mask out loss on kinematic particles + loss = (pred_acceleration - target_acceleration) ** 2 + + decay_fators_1 = torch.DoubleTensor( + [ + math.pow(cfg.train_options.loss_decay_factor, i) + for i in range(cfg.train_options.pred_len) + ] + ).to(device) + decay_fators_3 = torch.repeat_interleave(decay_fators_1, repeats=3) + + loss = loss * decay_fators_3 # torch.Size([num_nodes, input_dim]) + loss = torch.sum(loss, dim=-1) # torch.Size([num_nodes]) + print("overall loss: ", loss.shape, loss) + + # todo: check device + + if cfg.train_options.loss.startswith("anchor"): + rank_zero_logger.info("processing anchor loss\n\n") + # print("processing anchor loss\n\n") + # omit anchor point in loss + non_kinematic_mask = ( + torch.logical_not(get_kinematic_mask(particle_types)) + .to(torch.bool) + .to(device) + ) + num_non_kinematic = torch.sum(non_kinematic_mask) + + loss = torch.where( + non_kinematic_mask, + loss, + torch.zeros(loss.shape, dtype=inputs.dtype).to(device), + ) + loss = torch.sum(loss) / torch.sum(num_non_kinematic) + + # compute the loss in z-axis of anchor plane points + loss_plane = pred_acceleration[..., 2] ** 2 + decay_fator = torch.DoubleTensor( + [math.pow(cfg.train_options.loss_decay_factor, i) for i in range(1)] + ).to(device) + loss_plane = loss_plane * decay_fator + + anchor_plane_mask = anchor_plane_mask.to(torch.bool).to(device) + num_anchor_plane = torch.sum(anchor_plane_mask) + + loss_plane = torch.where( + anchor_plane_mask, + loss_plane, + torch.zeros(loss_plane.shape, dtype=inputs.dtype).to(device), + ) + loss_plane = torch.sum(loss_plane) / torch.sum(num_anchor_plane) + rank_zero_logger.info(f"loss: {loss}, loss_plane: {loss_plane}") + + loss = loss + cfg.train_options.l_plane * loss_plane + + if cfg.train_options.loss == "anchor_me": + loss_l1 = torch.nn.functional.l1_loss( + pred_acceleration * decay_fators_3, + target_acceleration * decay_fators_3, + ) + + loss = loss + cfg.train_options.l_me * loss_l1 + + elif cfg.train_options.loss.startswith("weighted"): + loss = weighted_square_error(pred_acceleration, target_acceleration, device) + + if cfg.train_options.loss == "weighted_anchor": + loss_plane = pred_acceleration[..., 2] ** 2 + + anchor_plane_mask = anchor_plane_mask.to(torch.bool).to(device) + num_anchor_plane = torch.sum(anchor_plane_mask) + loss_plane = torch.where( + anchor_plane_mask, + loss_plane, + torch.zeros(loss_plane.shape, dtype=inputs.dtype).to(device), + ) + + loss_plane = torch.sum(loss_plane) / torch.sum(num_anchor_plane) + rank_zero_logger.info(f"loss: {loss}, loss_plane: {loss_plane}") + loss = loss + cfg.train_options.l_plane * loss_plane + + elif cfg.train_options.loss == "correlation": + """ + Compute the correlation of random neighboring point pairs + to be optimized: + - todo: get random surface point id list + - todo: fix the pid list for each build + """ + rank_zero_logger.info("processing correlation loss\n\n") + + loss_corr_factor = 1 + k = 100 # OR 1/ 100 * particle num, whichever smaller + + pid_list = [pid for pid in range(target_acceleration.shape[0])] + random_pids = random.choices(pid_list, k=k) + + loss_corr = 0 + for idx_i in range(len(random_pids)): + for idx_j in range(idx_i, len(random_pids)): + i, j = random_pids[idx_i], random_pids[idx_j] + + corr_gt = torch.nn.functional.cosine_similarity( + target_acceleration[i], target_acceleration[j], dim=0 + ) + corr_pred = torch.nn.functional.cosine_similarity( + pred_acceleration[i], pred_acceleration[j], dim=0 + ) + + loss_corr_ = (corr_gt - corr_pred) ** 2 + loss_corr += loss_corr_ + + loss_corr /= k**2 + + non_kinematic_mask = non_kinematic_mask.to(torch.bool).to(device) + num_non_kinematic = torch.sum(non_kinematic_mask) + loss = torch.where( + non_kinematic_mask, + loss, + torch.zeros(loss.shape, dtype=loss.dtype).to(device), + ) + loss = torch.sum(loss) / torch.sum(num_non_kinematic) + + loss = loss + (loss_corr_factor * loss_corr) + + elif cfg.train_options.loss == "me": + # adding the L1 loss component with weight defined by "cfg.train_options.l_me" + rank_zero_logger.info("processing ME loss\n\n") + loss_l1 = torch.nn.functional.l1_loss( + pred_acceleration, target_acceleration + ) + loss_l1 = loss_l1 * decay_fators_3 + print("loss_l1 shape: ", loss_l1.shape) + loss_l1 = torch.sum(loss_l1, dim=-1) + print("loss_l1 shape: sum ", loss_l1.shape, loss_l1) + + non_kinematic_mask = non_kinematic_mask.to(torch.bool).to(device) + num_non_kinematic = torch.sum(non_kinematic_mask) + print( + "non_kinematic_mask/ num_non_kinematic: ", + non_kinematic_mask.shape, + num_non_kinematic, + num_non_kinematic.shape, + ) + loss = torch.where( + non_kinematic_mask, + loss, + torch.zeros(loss.shape, dtype=loss.dtype).to(device), + ) + loss = torch.sum(loss) / torch.sum(num_non_kinematic) + print("loss shape: sum ", loss.shape, loss) + + loss = loss + cfg.train_options.l_me * loss_l1 + + else: + # standard loss with applying mask + non_kinematic_mask = non_kinematic_mask.to(torch.bool).to(device) + num_non_kinematic = torch.sum(non_kinematic_mask) + loss = torch.where( + non_kinematic_mask, + loss, + torch.zeros(loss.shape, dtype=loss.dtype).to(device), + ) + loss = torch.sum(loss) / torch.sum(num_non_kinematic) + + rank_zero_logger.info(f"loss: {loss}") + # back propogation + optimizer.zero_grad() + if cfg.general.fp16: + with amp.scale_loss(loss, optimizer) as scaled_loss: + scaled_loss.backward() + else: + loss.backward() + optimizer.step() + + running_loss += loss.item() + + step += 1 + + if step % decay_steps == 0: + scheduler.step() + + if step % 10 == 0: + mean_loss = round(running_loss / 10, 5) + writer.add_scalar("loss", mean_loss, step) + writer.flush() + + running_loss = 0.0 + + if step % 50 == 0: + model.eval() + with torch.no_grad(): + test_loss = 0.0 + position_loss = 0.0 + for j in range(cfg.train_options.eval_steps): + features, targets = next(testDataset) + # test inference features.get('step_context') shape: torch.Size([2, 5]) + + predicted_positions = model.inference( + position_sequence=features["position"].to(device), + n_particles_per_example=features["n_particles_per_example"].to( + device + ), + n_edges_per_example=features["n_edges_per_example"].to(device), + senders=features["senders"].to(device), + receivers=features["receivers"].to(device), + predict_length=cfg.train_options.pred_len, + particle_types=features["particle_type"].to(device), + global_context=features.get("step_context").to(device), + ) + inputs = features["position"] + sampled_noise = torch.zeros(inputs.shape, dtype=inputs.dtype) + # sampled_noise = model.get_random_walk_noise_for_position_sequence(inputs, noise_std_last_step=FLAGS.noise_std) + + pred_target = model( + next_positions=targets.to(device), + position_sequence=inputs.to(device), + position_sequence_noise=sampled_noise.to(device), + n_particles_per_example=features["n_particles_per_example"].to( + device + ), + n_edges_per_example=features["n_edges_per_example"].to(device), + senders=features["senders"].to(device), + receivers=features["receivers"].to(device), + predict_length=cfg.train_options.pred_len, + particle_types=features["particle_type"].to(device), + global_context=features.get("step_context").to(device), + ) + + test_mse = torch.nn.functional.mse_loss(*pred_target) + p_mse = torch.nn.functional.mse_loss( + predicted_positions, targets.to(device) + ) + test_loss += test_mse.item() + position_loss += p_mse.item() + + writer.add_scalar("loss_mse", test_loss, step) + writer.add_scalar("position_mse", position_loss, step) + writer.flush() + + if test_loss < best_loss: + torch.save( + model.state_dict(), + os.path.join( + cfg.data_options.ckpt_path_vfgn, + "model_loss-{:.2E}_step-{}.pt".format(test_loss, step), + ), + ) + best_loss = test_loss + model.train() + + writer.close() + + +def Test(rank_zero_logger, dist, cfg): + """ + Executes the testing phase for a graph-based model, generating and + storing predictions. + """ + rank_zero_logger.info( + "\n\n.......... Start Testing model with defined data path ........\n\n" + ) + + # config test dataset + dataset = GraphDataset( + # size=C.num_steps, + mode="rollout", + split=cfg.general.eval_split, + data_path=cfg.data_options.data_path, + batch_size=cfg.train_options.batch_size, + ) + + metadata = _read_metadata(cfg.data_options.data_path) + acceleration_stats = Stats( + torch.DoubleTensor(cast(metadata["acc_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["acc_std"]), cfg.data_options.noise_std) + ), + ) + velocity_stats = Stats( + torch.DoubleTensor(cast(metadata["vel_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["vel_std"]), cfg.data_options.noise_std) + ), + ) + context_stats = Stats( + torch.DoubleTensor(cast(metadata["context_mean"])), + torch.DoubleTensor( + _combine_std(cast(metadata["context_std"]), cfg.data_options.noise_std) + ), + ) + + normalization_stats = { + "acceleration": acceleration_stats, + "velocity": velocity_stats, + "context": context_stats, + } + + model = VFGNLearnedSimulator( + num_dimensions=metadata["dim"] * cfg.train_options.pred_len, + num_seq=cfg.train_options.input_seq_len, + boundaries=torch.DoubleTensor(metadata["bounds"]), + num_particle_types=cfg.data_options.NUM_PARTICLE_TYPES, + particle_type_embedding_size=16, + normalization_stats=normalization_stats, + ) + + loaded = False + example_index = 0 + device = "cpu" + with torch.no_grad(): + for features, targets in tqdm(dataset): + if loaded is False: + # input feature size is dynamic, so need to forward model in CPU before loading into GPU + global_context = features["step_context"].to(device) + if global_context is None: + global_context_step = None + else: + global_context_step = global_context[:-1] + global_context_step = torch.reshape(global_context_step, [1, -1]) + + model.inference( + position_sequence=features["position"][ + :, 0 : cfg.train_options.input_seq_len + ].to(device), + n_particles_per_example=features["n_particles_per_example"].to( + device + ), + n_edges_per_example=features["n_edges_per_example"].to(device), + senders=features["senders"].to(device), + receivers=features["receivers"].to(device), + predict_length=cfg.train_options.pred_len, + particle_types=features["particle_type"].to(device), + global_context=global_context_step.to(device), + ) + + # Loading the pretrained model from model ckpt_path_vfgn + # For provided ckpt with missing keys, ignore with strict=False + model.load_state_dict( + torch.load(cfg.data_options.ckpt_path_vfgn), strict=False + ) + device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") + rank_zero_logger.info(f"Device: {device}") + # config optimizer + # todo: check msg passing device + model.setMessagePassingDevices(["cuda:0"]) + model = model.to(device) + model.eval() + loaded = True + + initial_positions = features["position"][ + :, : cfg.train_options.input_seq_len + ].to(device) + ground_truth_positions = features["position"][ + :, cfg.train_options.input_seq_len : + ].to(device) + global_context = features["step_context"].to(device) + rank_zero_logger.info( + f"\n Initial_positions shape: {initial_positions.shape}" + ) + rank_zero_logger.info( + f"\n Ground_truth_positions shape: {ground_truth_positions.shape}" + ) + + num_steps = ground_truth_positions.shape[1] + + current_positions = initial_positions + updated_predictions = [] + + start_time = time.time() + rank_zero_logger.info(f"start time: {start_time}\n") + + for step in range(num_steps): + rank_zero_logger.info(f"start predictiong step: {step}") + if global_context is None: + global_context_step = None + else: + read_step_context = global_context[ + : step + cfg.train_options.input_seq_len + ] + zero_pad = torch.zeros( + [global_context.shape[0] - read_step_context.shape[0] - 1, 1], + dtype=features["step_context"].dtype, + ).to(device) + + global_context_step = torch.cat([read_step_context, zero_pad], 0) + global_context_step = torch.reshape(global_context_step, [1, -1]) + + predict_positions = model.inference( + position_sequence=current_positions.to(device), + n_particles_per_example=features["n_particles_per_example"].to( + device + ), + n_edges_per_example=features["n_edges_per_example"].to(device), + senders=features["senders"].to(device), + receivers=features["receivers"].to(device), + predict_length=cfg.train_options.pred_len, + particle_types=features["particle_type"].to(device), + global_context=global_context_step.to(device), + ) + + kinematic_mask = ( + get_kinematic_mask(features["particle_type"]) + .to(torch.bool) + .to(device) + ) + positions_ground_truth = ground_truth_positions[:, step] + + predict_positions = predict_positions[:, 0].squeeze(1) + kinematic_mask = torch.repeat_interleave( + kinematic_mask, repeats=predict_positions.shape[-1] + ) + kinematic_mask = torch.reshape( + kinematic_mask, [-1, predict_positions.shape[-1]] + ) + + next_position = torch.where( + kinematic_mask, positions_ground_truth, predict_positions + ) + + updated_predictions.append(next_position) + if cfg.test_options.rollout_refine is False: + # False: rollout the predictions + current_positions = torch.cat( + [current_positions[:, 1:], next_position.unsqueeze(1)], axis=1 + ) + else: + # True: single-step prediction for all steps + current_positions = torch.cat( + [current_positions[:, 1:], positions_ground_truth.unsqueeze(1)], + axis=1, + ) + + updated_predictions = torch.stack(updated_predictions) + rank_zero_logger.info( + f"\n Updated_predictions shape: {updated_predictions.shape}" + ) + rank_zero_logger.info( + f"\n Ground_truth_positions shape: {ground_truth_positions.shape}" + ) + + initial_positions_list = initial_positions.cpu().numpy().tolist() + updated_predictions_list = updated_predictions.cpu().numpy().tolist() + ground_truth_positions_list = ground_truth_positions.cpu().numpy().tolist() + particle_types_list = features["particle_type"].cpu().numpy().tolist() + global_context_list = global_context.cpu().numpy().tolist() + + rollout_op = { + "initial_positions": initial_positions_list, + "predicted_rollout": updated_predictions_list, + "ground_truth_rollout": ground_truth_positions_list, + "particle_types": particle_types_list, + "global_context": global_context_list, + } + + # Add a leading axis, since Estimator's predict method insists that all + # tensors have a shared leading batch axis fo the same dims. + # rollout_op = tree.map_structure(lambda x: x.numpy(), rollout_op) + + rollout_op["metadata"] = metadata + filename = f"rollout_{cfg.general.eval_split}_{example_index}.json" + filename = os.path.join(cfg.data_options.output_path, filename) + if not os.path.exists(cfg.data_options.output_path): + os.makedirs(cfg.data_options.output_path) + with open(filename, "w") as file_object: + json.dump(rollout_op, file_object) + + example_index += 1 + rank_zero_logger.info(f"Prediction time: {time.time() - start_time}\n") + + +@hydra.main(version_base=None, config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + """ + Triggers the train or test phase based on the configuration. + """ + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + # save constants to JSON file + # todo: test the disk.rank init and save + # if dist.rank == 0: + # print('check main', C.ckpt_path) + # os.makedirs(C.ckpt_path, exist_ok=True) + # with open( + # os.path.join(C.ckpt_path, C.ckpt_name.replace(".pt", ".json")), "w" + # ) as json_file: + # json_file.write(C.json(indent=4)) + + rank_zero_logger = RankZeroLoggingWrapper(logger, dist) # Rank 0 logger + print("check cfg loading: ", cfg) + if cfg.general.mode == "train": + Train(rank_zero_logger, dist, cfg) + elif cfg.general.mode == "eval_rollout": + Test(rank_zero_logger, dist, cfg) + else: + raise NotImplementedError("Mode not implemented ") + + +if __name__ == "__main__": + # tf.disable_v2_behavior() + LaunchLogger.initialize() # PhysicsNeMo launch logger + logger = PythonLogger("main") # General python logger + logger.file_logging() + + main() diff --git a/examples/additive_manufacturing/sintering_physics/utils.py b/examples/additive_manufacturing/sintering_physics/utils.py new file mode 100644 index 0000000000..eab96f85ba --- /dev/null +++ b/examples/additive_manufacturing/sintering_physics/utils.py @@ -0,0 +1,166 @@ +# © Copyright 2023 HP Development Company, L.P. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +import json +import math +import os + +import numpy as np +import torch + +try: + import tensorflow as tf +except ImportError: + raise ImportError( + "Mesh Graph Net Datapipe requires the Tensorflow library. Install the " + + "package at: https://www.tensorflow.org/install" + ) + +NUM_PARTICLE_TYPES = 3 +KINEMATIC_PARTICLE_ID = 0 # refers to anchor point +METAL_PARTICLE_ID = 2 # refers to normal particles +ANCHOR_PLANE_PARTICLE_ID = 1 # refers to anchor plane + + +class Stats: + """ + Represents statistical attributes with methods for device transfer. + """ + + def __init__(self, mean, std): + self.mean = mean + self.std = std + + def to(self, device): + """Transfers the mean and standard deviation to a specified device.""" + self.mean = self.mean.to(device) + self.std = self.std.to(device) + return self + + +def cast(v): + return np.array(v, dtype=np.float64) + + +def _read_metadata(data_path): + """reads metadata""" + with open(os.path.join(data_path, "metadata.json"), "rt") as fp: + return json.load(fp) + + +def _combine_std(std_x, std_y): + """combine standard deviation with l2 norm""" + return np.sqrt(std_x**2 + std_y**2) + + +def tf2torch(t): + """ + Converts a TensorFlow tensor to a PyTorch tensor. + """ + t = torch.from_numpy(t.numpy()) + return t + + +def torch2tf(t): + """ + Converts a PyTorch tensor to a TensorFlow tensor. + """ + t = tf.convert_to_tensor(t.cpu().numpy()) + return t + + +def get_kinematic_mask(particle_types): + """Returns a boolean mask, set to true for kinematic (obstacle) particles.""" + # return tf.equal(particle_types, KINEMATIC_PARTICLE_ID) + # return size: num_particles_in_batch + + return particle_types == torch.ones(particle_types.shape) * KINEMATIC_PARTICLE_ID + + +def get_metal_mask(particle_types): + """Returns a boolean mask, set to true for metal particles.""" + # get free particles + return particle_types == torch.ones(particle_types.shape) * METAL_PARTICLE_ID + + +def get_anchor_z_mask(particle_types): + """ + Generates a mask identifying anchor plane particles in a tensor of particle types. + """ + # get anchor plane particles + return particle_types == torch.ones(particle_types.shape) * ANCHOR_PLANE_PARTICLE_ID + + +def cos_theta(p1, p2): + """compute cosine of two non-zero vectors""" + return (torch.dot(p1, p2)) / ( + (torch.sqrt(torch.dot(p1, p1))) * (math.sqrt(torch.dot(p2, p2))) + ) + + +def weighted_square_error(y_pre, y, device): + """ + Calculates a weighted square error for predictions, emphasizing larger errors + by sorting and applying diminishing weights. + """ + k = y_pre - y + print("weighted_square_error k shape: ", k.shape) + + k = k.view(-1) + k = torch.square(k) + sorted, indices = torch.sort(k, descending=True) + print("weight: ", sorted.size()) + n = sorted.size()[0] + + weights = [] + dw = 1.0 / n + for i in range(n): + weights.append(dw) + dw = dw * 0.99 + weights = torch.FloatTensor(weights).to(device) + + out = weights * sorted + print("weighted_square_error out shape: ", out.shape) + + out = torch.mean(out) + # out = torch.sum(out) + print("mean out: ", out, out.shape) + return out + + +def weighted_loss(loss_, device): + """ + Computes a loss value where individual components are weighted, with higher weights + assigned to larger loss components. + """ + loss_ = loss_.view(-1) + sorted, indices = torch.sort(loss_, descending=True) + n = sorted.size()[0] + + weights = [] + dw = 1.0 / n + for i in range(n): + weights.append(dw) + dw = dw * 0.99 + weights = torch.FloatTensor(weights).to(device) + + out = weights * sorted + + out = torch.sum(out) + print("out: ", out) + return out diff --git a/examples/cfd/ahmed_body_mgn/README.md b/examples/cfd/ahmed_body_mgn/README.md deleted file mode 100644 index 1aacb62171..0000000000 --- a/examples/cfd/ahmed_body_mgn/README.md +++ /dev/null @@ -1,119 +0,0 @@ -# AeroGraphNet for external aerodynamic evaluation - -This example demonstrates how to train the AeroGraphNet model for external aerodynamic -analysis of simplified (Ahmed body-type) car geometries. AeroGraphNet is based on the -MeshGraphNet architecture. It achieves good accuracy on predicting the pressure and -wall shear stresses on the surface mesh of the Ahmed body-type geometries, as well as -the drag coefficient. - -## Problem overview - -To goal is to develop an AI surrogate model that can use simulation data to learn the -external aerodynamic flow over parameterized Ahmed body shape. This serves as a baseline -for more refined models for realistic car geometries. The trained model can be used to -predict the change in drag coefficient,and surface pressure and wall shear stresses due -to changes in the car geometry. This is a stepping stone to applying similar approaches -to other application areas such as aerodynamic analysis of aircraft wings, real car -geometries, etc. - -## Dataset - -Industry-standard Ahmed-body geometries are characterized by six design parameters: -length, width, height, ground clearance, slant angle, and fillet radius. Refer -to the [wiki](https://www.cfd-online.com/Wiki/Ahmed_body) for details on Ahmed -body geometry. In addition to these design parameters, we include the inlet velocity to -address a wide variation in Reynolds number. We identify the design points using the -Latin hypercube sampling scheme for space filling design of experiments and generate -around 500 design points. - -The aerodynamic simulations were performed using the GPU-accelerated OpenFOAM solver -for steady-state analysis, applying the SST K-omega turbulence model. These simulations -consist of 7.2 million mesh points on average, but we use the surface mesh as the input -to training which is roughly around 70k mesh nodes. - -To request access to the full dataset, please reach out to the -[NVIDIA Modulus team](modulus-team@nvidia.com). - -## Model overview and architecture - -The AeroGraphNet model is based on the MeshGraphNet architecture which is instrumental -for learning from mesh-based data using GNNs. The inputs to the model are: - -- Ahmed body surface mesh -- Reynolds number -- Geometry parameters (optional, including length, width, height, ground clearance, -slant angle, and fillet radius) -- surface normals (optional) - -Output of the model are: - -- Surface pressure -- Wall shear stresses -- Drag coefficient - -![Comparison between the AeroGraphNet prediction and the -ground truth for surface pressure, wall shear stresses, and the drag coefficient for one -of the samples from the test dataset.](../../../docs/img/ahmed_body_results.png) - -The input to the model is in form of a `.vtp` file and is then converted to -bi-directional DGL graphs in the dataloader. The final results are also written in the -form of `.vtp` files in the inference code. A hidden dimensionality of 256 is used in -the encoder, processor, and decoder. The encoder and decoder consist of two hidden -layers, and the processor includes 15 message passing layers. Batch size per GPU is -set to 1. Summation aggregation is used in the -processor for message aggregation. A learning rate of 0.0001 is used, decaying -exponentially with a rate of 0.99985. Training is performed on 8 NVIDIA A100 -GPUs, leveraging data parallelism. Total training time is 4 hours, and training is -performed for 500 epochs. - -## Getting Started - -The dataset for this example is not publicly available. To get access, please reach out -to the [NVIDIA Modulus team](modulus-team@nvidia.com). - -This example requires the `pyvista` and `vtk` libraries. Install with - -```bash -pip install pyvista vtk -``` - -To train the model, run - -```bash -python train.py -``` - -Data parallelism is also supported with multi-GPU runs. To launch a multi-GPU training, -run - -```bash -mpirun -np python train.py -``` - -If running in a docker container, you may need to include the `--allow-run-as-root` in -the multi-GPU run command. - -Progress and loss logs can be monitored using Weights & Biases. To activate that, -set `wandb_mode` to `online` in the `constants.py`. This requires to have an active -Weights & Biases account. You also need to provide your API key. There are multiple ways -for providing the API key but you can simply export it as an environment variable - -```bash -export WANDB_API_KEY= -``` - -The URL to the dashboard will be displayed in the terminal after the run is launched. -Alternatively, the logging utility in `train.py` can be switched to MLFlow. - -Once the model is trained, run - -```bash -python inference.py -``` - -This will save the predictions for the test dataset in `.vtp` format in the `results` -directory. Use Paraview to open and explore the results. - -## References - -- [Learning Mesh-Based Simulation with Graph Networks](https://arxiv.org/abs/2010.03409) diff --git a/examples/cfd/ahmed_body_mgn/constants.py b/examples/cfd/ahmed_body_mgn/constants.py deleted file mode 100644 index 9646b0b8b7..0000000000 --- a/examples/cfd/ahmed_body_mgn/constants.py +++ /dev/null @@ -1,49 +0,0 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import os -from pathlib import Path -from pydantic import BaseModel -from typing import Tuple, Optional - - -class Constants(BaseModel): - """Ahmed Body model constants""" - - ckpt_path: str = "./checkpoints" - ckpt_name: str = "./ahmed_body" - data_dir: str = "../dataset" - results_dir: str = "./results" - - input_dim_nodes: int = 11 - input_dim_edges: int = 4 - output_dim: int = 4 - aggregation: int = "sum" - hidden_dim_node_encoder = 256 - hidden_dim_edge_encoder = 256 - hidden_dim_node_decoder = 256 - - batch_size: int = 1 - epochs: int = 500 - num_training_samples: int = 683 - num_validation_samples: int = 100 - num_test_samples: int = 100 - - lr: float = 1e-4 - lr_decay_rate: float = 0.99985 - - amp: bool = False - jit: bool = False - - wandb_mode = "disabled" diff --git a/examples/cfd/ahmed_body_mgn/inference.py b/examples/cfd/ahmed_body_mgn/inference.py deleted file mode 100644 index a34dd564c7..0000000000 --- a/examples/cfd/ahmed_body_mgn/inference.py +++ /dev/null @@ -1,197 +0,0 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import os - -import torch -import numpy as np -import wandb as wb -from modulus.models.meshgraphnet import MeshGraphNet -from modulus.datapipes.gnn.ahmed_body_dataset import AhmedBodyDataset -from modulus.launch.utils import load_checkpoint -from modulus.launch.logging import PythonLogger - -from utils import compute_drag_coefficient, relative_lp_error -from constants import Constants - -try: - from dgl.dataloading import GraphDataLoader - from dgl import DGLGraph -except: - raise ImportError( - "Ahmed Body example requires the DGL library. Install the " - + "desired CUDA version at: \n https://www.dgl.ai/pages/start.html" - ) - -try: - import pyvista as pv -except: - raise ImportError( - "Ahmed Body Dataset requires the pyvista library. Install with " - + "pip install pyvista" - ) - - -C = Constants() - - -def dgl_to_pyvista(graph: DGLGraph): - """ - Converts a DGL graph to a PyVista graph. - - Parameters: - ----------- - graph: DGLGraph - The input DGL graph. - - Returns: - -------- - pv_graph: - The output PyVista graph. - """ - - # Convert the DGL graph to a NetworkX graph - nx_graph = graph.to_networkx( - node_attrs=["pos", "p_pred", "p", "s_pred", "wallShearStress"] - ).to_undirected() - - # Initialize empty lists for storing data - points = [] - lines = [] - p_pred = [] - s_pred = [] - p = [] - wallShearStress = [] - - # Iterate over the nodes in the NetworkX graph - for node, attributes in nx_graph.nodes(data=True): - # Append the node and attribute data to the respective lists - points.append(attributes["pos"].numpy()) - p_pred.append(attributes["p_pred"].numpy()) - s_pred.append(attributes["s_pred"].numpy()) - p.append(attributes["p"].numpy()) - wallShearStress.append(attributes["wallShearStress"].numpy()) - - # Add edges to the lines list - for edge in nx_graph.edges(): - lines.extend([2, edge[0], edge[1]]) - - # Initialize a PyVista graph - pv_graph = pv.PolyData() - - # Assign the points, lines, and attributes to the PyVista graph - pv_graph.points = np.array(points) - pv_graph.lines = np.array(lines) - pv_graph.point_data["p_pred"] = np.array(p_pred) - pv_graph.point_data["s_pred"] = np.array(s_pred) - pv_graph.point_data["p"] = np.array(p) - pv_graph.point_data["wallShearStress"] = np.array(wallShearStress) - - return pv_graph - - -class AhmedBodyRollout: - """MGN inference on Ahmed Body dataset""" - - def __init__(self, wb, logger): - # set device - self.device = "cuda" if torch.cuda.is_available() else "cpu" - logger.info(f"Using {self.device} device") - - # instantiate dataset - self.dataset = AhmedBodyDataset( - name="ahmed_body_test", - data_dir=C.data_dir, - split="test", - num_samples=C.num_test_samples, - compute_drag=True, - ) - - # instantiate dataloader - self.dataloader = GraphDataLoader( - self.dataset, - batch_size=C.batch_size, - shuffle=False, - drop_last=False, - ) - - # instantiate the model - self.model = MeshGraphNet( - C.input_dim_nodes, - C.input_dim_edges, - C.output_dim, - aggregation=C.aggregation, - hidden_dim_node_encoder=C.hidden_dim_node_encoder, - hidden_dim_edge_encoder=C.hidden_dim_edge_encoder, - hidden_dim_node_decoder=C.hidden_dim_node_decoder, - ) - self.model = self.model.to(self.device) - - # enable train mode - self.model.eval() - - # load checkpoint - self.model.load( - os.path.join(C.ckpt_path, C.ckpt_name, "MeshGraphNet.0.499.mdlus") - ) - - def predict(self, save_results=False): - """ - Run the prediction process. - - Parameters: - ----------- - save_results: bool - Whether to save the results in form of a .vtp file, by default False - - - Returns: - -------- - None - """ - - self.pred, self.exact, self.faces, self.graphs = [], [], [], [] - - for i, (graph, sid, normals, areas, coeff) in enumerate(self.dataloader): - graph = graph.to(self.device) - normals = normals.to(self.device, torch.float32).squeeze() - areas = areas.to(self.device, torch.float32).squeeze() - coeff = coeff.to(self.device, torch.float32).squeeze() - sid = sid.item() - logger.info(f"Processing sample ID {sid}") - pred = self.model(graph.ndata["x"], graph.edata["x"], graph).detach() - - gt = graph.ndata["y"] - graph.ndata["p_pred"] = pred[:, 0] - graph.ndata["s_pred"] = pred[:, 1:] - graph.ndata["p"] = gt[:, 0] - graph.ndata["wallShearStress"] = gt[:, 1:] - - error = relative_lp_error(pred, gt) - logger.info(f"Test error (%): {error}") - - if save_results: - # Convert DGL graph to PyVista graph and save it - os.makedirs(C.results_dir, exist_ok=True) - pv_graph = dgl_to_pyvista(graph.cpu()) - pv_graph.save(os.path.join(C.results_dir, f"graph_{sid}.vtp")) - - -if __name__ == "__main__": - logger = PythonLogger("main") # General python logger - logger.file_logging() - - logger.info("Rollout started...") - rollout = AhmedBodyRollout(wb, logger) - rollout.predict(save_results=True) diff --git a/examples/cfd/ahmed_body_mgn/train.py b/examples/cfd/ahmed_body_mgn/train.py deleted file mode 100644 index a6f89c4bd3..0000000000 --- a/examples/cfd/ahmed_body_mgn/train.py +++ /dev/null @@ -1,259 +0,0 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import os -import time - -import torch -from torch.cuda.amp import autocast, GradScaler -from torch.nn.parallel import DistributedDataParallel -import wandb as wb - -from modulus.models.meshgraphnet import MeshGraphNet -from modulus.datapipes.gnn.ahmed_body_dataset import AhmedBodyDataset -from modulus.distributed.manager import DistributedManager - -from modulus.launch.logging import ( - PythonLogger, - initialize_wandb, - RankZeroLoggingWrapper, -) -from modulus.launch.utils import load_checkpoint, save_checkpoint -from constants import Constants - -try: - from dgl.dataloading import GraphDataLoader -except: - raise ImportError( - "Ahmed Body example requires the DGL library. Install the " - + "desired CUDA version at: \n https://www.dgl.ai/pages/start.html" - ) - -try: - import apex -except ImportError: - pass - -# Instantiate constants -C = Constants() - - -class MGNTrainer: - def __init__(self, wb, dist, rank_zero_logger): - self.dist = dist - self.wb = wb - self.rank_zero_logger = rank_zero_logger - - # instantiate dataset - rank_zero_logger.info("Loading the training dataset...") - self.dataset = AhmedBodyDataset( - name="ahmed_body_train", - data_dir=C.data_dir, - split="train", - num_samples=C.num_training_samples, - ) - - # instantiate validation dataset - rank_zero_logger.info("Loading the validation dataset...") - self.validation_dataset = AhmedBodyDataset( - name="ahmed_body_validation", - data_dir=C.data_dir, - split="validation", - num_samples=C.num_validation_samples, - ) - - # instantiate dataloader - self.dataloader = GraphDataLoader( - self.dataset, - batch_size=C.batch_size, - shuffle=True, - drop_last=True, - pin_memory=True, - use_ddp=dist.world_size > 1, - ) - - # instantiate validation dataloader - self.validation_dataloader = GraphDataLoader( - self.validation_dataset, - batch_size=C.batch_size, - shuffle=False, - drop_last=True, - pin_memory=True, - use_ddp=False, - ) - - # instantiate the model - self.model = MeshGraphNet( - C.input_dim_nodes, - C.input_dim_edges, - C.output_dim, - aggregation=C.aggregation, - hidden_dim_node_encoder=C.hidden_dim_node_encoder, - hidden_dim_edge_encoder=C.hidden_dim_edge_encoder, - hidden_dim_node_decoder=C.hidden_dim_node_decoder, - ) - if C.jit: - self.model = torch.jit.script(self.model).to(dist.device) - else: - self.model = self.model.to(dist.device) - - # distributed data parallel for multi-node training - if dist.world_size > 1: - self.model = DistributedDataParallel( - self.model, - device_ids=[dist.local_rank], - output_device=dist.device, - broadcast_buffers=dist.broadcast_buffers, - find_unused_parameters=dist.find_unused_parameters, - ) - - # enable train mode - self.model.train() - - # instantiate optimizer, and scheduler - try: - self.optimizer = apex.optimizers.FusedAdam(self.model.parameters(), lr=C.lr) - rank_zero_logger.info("Using FusedAdam optimizer") - except: - self.optimizer = torch.optim.Adam(self.model.parameters(), lr=C.lr) - self.scheduler = torch.optim.lr_scheduler.LambdaLR( - self.optimizer, lr_lambda=lambda epoch: C.lr_decay_rate**epoch - ) - self.scaler = GradScaler() - - # load checkpoint - if dist.world_size > 1: - torch.distributed.barrier() - self.epoch_init = load_checkpoint( - os.path.join(C.ckpt_path, C.ckpt_name), - models=self.model, - optimizer=self.optimizer, - scheduler=self.scheduler, - scaler=self.scaler, - device=dist.device, - ) - - def train(self, graph): - self.optimizer.zero_grad() - loss = self.forward(graph) - self.backward(loss) - self.scheduler.step() - return loss - - def forward(self, graph): - # forward pass - with autocast(enabled=C.amp): - pred = self.model(graph.ndata["x"], graph.edata["x"], graph) - diff_norm = torch.norm( - torch.flatten(pred) - torch.flatten(graph.ndata["y"]), p=2 - ) - y_norm = torch.norm(torch.flatten(graph.ndata["y"]), p=2) - loss = diff_norm / y_norm - return loss - - def backward(self, loss): - # backward pass - if C.amp: - self.scaler.scale(loss).backward() - self.scaler.step(self.optimizer) - self.scaler.update() - else: - loss.backward() - self.optimizer.step() - lr = self.get_lr() - self.wb.log({"lr": lr}) - - def get_lr(self): - # get the learning rate - for param_group in self.optimizer.param_groups: - return param_group["lr"] - - @torch.no_grad() - def validation(self): - error = 0 - for graph in self.validation_dataloader: - graph = graph.to(self.dist.device) - pred = self.model(graph.ndata["x"], graph.edata["x"], graph) - pred, gt = self.dataset.denormalize( - pred, graph.ndata["y"], self.dist.device - ) - error += ( - torch.mean(torch.norm(pred - gt, p=2) / torch.norm(gt, p=2)) - .cpu() - .numpy() - ) - error = error / len(self.validation_dataloader) * 100 - self.wb.log({"val_error (%)": error}) - self.rank_zero_logger.info(f"Denormalized validation error (%): {error}") - - -if __name__ == "__main__": - # initialize distributed manager - DistributedManager.initialize() - dist = DistributedManager() - - # save constants to JSON file - if dist.rank == 0: - os.makedirs(C.ckpt_path, exist_ok=True) - with open(os.path.join(C.ckpt_path, C.ckpt_name + ".json"), "w") as json_file: - json_file.write(C.model_dump_json(indent=4)) - - # initialize loggers - initialize_wandb( - project="Aero", - entity="Modulus", - name="Aero-Training", - group="Aero-DDP-Group", - mode=C.wandb_mode, - ) # Wandb logger - - logger = PythonLogger("main") # General python logger - rank_zero_logger = RankZeroLoggingWrapper(logger, dist) # Rank 0 logger - logger.file_logging() - - trainer = MGNTrainer(wb, dist, rank_zero_logger) - start = time.time() - rank_zero_logger.info("Training started...") - - for epoch in range(trainer.epoch_init, C.epochs): - loss_agg = 0 - for graph in trainer.dataloader: - graph = graph.to(dist.device) - loss = trainer.train(graph) - loss_agg += loss.detach().cpu().numpy() - loss_agg /= len(trainer.dataloader) - rank_zero_logger.info( - f"epoch: {epoch}, loss: {loss_agg:10.3e}, lr: {trainer.get_lr()}, time per epoch: {(time.time()-start):10.3e}" - ) - wb.log({"loss": loss_agg}) - - # validation - if dist.rank == 0: - trainer.validation() - - # save checkpoint - if dist.world_size > 1: - torch.distributed.barrier() - if dist.rank == 0: - save_checkpoint( - os.path.join(C.ckpt_path, C.ckpt_name), - models=trainer.model, - optimizer=trainer.optimizer, - scheduler=trainer.scheduler, - scaler=trainer.scaler, - epoch=epoch, - ) - logger.info(f"Saved model on rank {dist.rank}") - start = time.time() - rank_zero_logger.info("Training completed!") diff --git a/examples/cfd/ahmed_body_mgn/utils.py b/examples/cfd/ahmed_body_mgn/utils.py deleted file mode 100644 index c7f0abfe42..0000000000 --- a/examples/cfd/ahmed_body_mgn/utils.py +++ /dev/null @@ -1,73 +0,0 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import numpy as np -import torch -from torch import Tensor - - -def compute_drag_coefficient(normals, area, coeff, p, s): - """ - Compute drag coefficient for a given mesh. - - Parameters: - ----------- - normals: Tensor - The surface normals mapped onto nodes - area: Tensor - The surface areas of each cell mapped onto nodes - coeff: Tensor - Dynamic pressure times the frontal area - p: Tensor - Pressure distribution on the mesh - s: Tensor - Wall shear stress distribution on the mesh - - Returns: - -------- - c_drag: float: - Computed drag coefficient - """ - - # Compute coefficients - c_p = coeff * torch.dot(normals[:, 0], area * p) - c_f = -coeff * torch.dot(s[:, 0], area) - - # Compute total drag coefficients - c_drag = c_p + c_f - - return c_drag - - -def relative_lp_error(pred, y, p=2): - """ - Calculate relative L2 error norm - Parameters: - ----------- - pred: torch.Tensor - Prediction - y: torch.Tensor - Ground truth - Returns: - -------- - error: float - Calculated relative L2 error norm (percentage) on cpu - """ - - error = ( - torch.mean(torch.linalg.norm(pred - y, ord=p) / torch.linalg.norm(y, ord=p)) - .cpu() - .numpy() - ) - return error * 100 diff --git a/examples/cfd/darcy_fno/README.md b/examples/cfd/darcy_fno/README.md index 81a8017fbc..d39a5939c3 100644 --- a/examples/cfd/darcy_fno/README.md +++ b/examples/cfd/darcy_fno/README.md @@ -1,40 +1,27 @@ -# Fourier Neural Operater for Darcy Flow +# Fourier Neural Operator for Darcy Flow This example demonstrates how to set up a data-driven model for a 2D Darcy flow using -the Fourier Neural Operator (FNO) architecture inside of Modulus. -Training progress can be tracked through [MLFlow](https://mlflow.org/docs/latest/index.html). +the Fourier Neural Operator (FNO) architecture inside of PhysicsNeMo. This example runs on a single GPU, go to the `darcy_nested_fno` example for exploring a multi-GPU training. -## Getting Started +## Prerequisites -To train the model, run +Install the required dependencies by running below: ```bash -python train_fno_darcy.py +pip install -r requirements.txt ``` -training data will be generated on the fly. - -Progress can be monitored using MLFlow. Open a new terminal and navigate to the training -directory, then run: - -```bash -mlflow ui -p 2458 -``` - -View progress in a browser at +## Getting Started -If training on a remote machine, set up a ssh tunnel to -the server with `LocalForward 8080 your_remote_machine_addr:8080`. -ssh to the server via the specified port, in this case `8080`, navigate to the training -directory and launch mlflow server +To train the model, run ```bash -mlflow server --host 0.0.0.0 --port 8080 +python train_fno_darcy.py ``` -On your local machine, open a browser and connect to `localhost:8080`. +training data will be generated on the fly. ## Additional Information diff --git a/examples/cfd/darcy_fno/config.yaml b/examples/cfd/darcy_fno/config.yaml index 1e805317c6..8ee65983a9 100644 --- a/examples/cfd/darcy_fno/config.yaml +++ b/examples/cfd/darcy_fno/config.yaml @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/examples/cfd/darcy_fno/requirements.txt b/examples/cfd/darcy_fno/requirements.txt new file mode 100644 index 0000000000..ca21ae7ff2 --- /dev/null +++ b/examples/cfd/darcy_fno/requirements.txt @@ -0,0 +1,3 @@ +hydra-core>=1.2.0 +warp-lang>=1.6.0 +termcolor>=2.1.1 diff --git a/examples/cfd/darcy_fno/train_fno_darcy.py b/examples/cfd/darcy_fno/train_fno_darcy.py index cdd5c49ac1..b6ed942b72 100644 --- a/examples/cfd/darcy_fno/train_fno_darcy.py +++ b/examples/cfd/darcy_fno/train_fno_darcy.py @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -19,12 +21,12 @@ from torch.nn import MSELoss from torch.optim import Adam, lr_scheduler -from modulus.models.fno import FNO -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.distributed import DistributedManager -from modulus.utils import StaticCaptureTraining, StaticCaptureEvaluateNoGrad -from modulus.launch.utils import load_checkpoint, save_checkpoint -from modulus.launch.logging import PythonLogger, LaunchLogger, initialize_mlflow +from physicsnemo.models.fno import FNO +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils import StaticCaptureTraining, StaticCaptureEvaluateNoGrad +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, LaunchLogger from validator import GridValidator @@ -35,7 +37,7 @@ def darcy_trainer(cfg: DictConfig) -> None: This training script demonstrates how to set up a data-driven model for a 2D Darcy flow using Fourier Neural Operators (FNO) and acts as a benchmark for this type of operator. - Training data is generated in-situ via the Darcy2D data loader from Modulus. Darcy2D + Training data is generated in-situ via the Darcy2D data loader from PhysicsNeMo. Darcy2D continuously generates data previously unseen by the model, i.e. the model is trained over a single epoch of a training set consisting of (cfg.training.max_pseudo_epochs*cfg.training.pseudo_epoch_sample_size) unique samples. @@ -47,15 +49,7 @@ def darcy_trainer(cfg: DictConfig) -> None: # initialize monitoring log = PythonLogger(name="darcy_fno") log.file_logging() - initialize_mlflow( - experiment_name=f"Darcy_FNO", - experiment_desc=f"training an FNO model for the Darcy problem", - run_name=f"Darcy FNO training", - run_desc=f"training FNO for Darcy", - user_name="Gretchen Ross", - mode="offline", - ) - LaunchLogger.initialize(use_mlflow=True) # Modulus launch logger + LaunchLogger.initialize() # PhysicsNeMo launch logger # define model, loss, optimiser, scheduler, data loader model = FNO( @@ -107,12 +101,12 @@ def darcy_trainer(cfg: DictConfig) -> None: if cfg.training.pseudo_epoch_sample_size % cfg.training.batch_size != 0: log.warning( f"increased pseudo_epoch_sample_size to multiple of \ - batch size: {steps_per_pseudo_epoch*cfg.training.batch_size}" + batch size: {steps_per_pseudo_epoch * cfg.training.batch_size}" ) if cfg.validation.sample_size % cfg.training.batch_size != 0: log.warning( f"increased validation sample size to multiple of \ - batch size: {validation_iters*cfg.training.batch_size}" + batch size: {validation_iters * cfg.training.batch_size}" ) # define forward passes for training and inference @@ -133,7 +127,7 @@ def forward_eval(invars): if loaded_pseudo_epoch == 0: log.success("Training started...") else: - log.warning(f"Resuming training from pseudo epoch {loaded_pseudo_epoch+1}.") + log.warning(f"Resuming training from pseudo epoch {loaded_pseudo_epoch + 1}.") for pseudo_epoch in range( max(1, loaded_pseudo_epoch + 1), cfg.training.max_pseudo_epochs + 1 diff --git a/examples/cfd/darcy_fno/validator.py b/examples/cfd/darcy_fno/validator.py index d036613251..bec8762a25 100644 --- a/examples/cfd/darcy_fno/validator.py +++ b/examples/cfd/darcy_fno/validator.py @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,7 +16,7 @@ import matplotlib.pyplot as plt from torch import FloatTensor -from modulus.launch.logging import LaunchLogger +from physicsnemo.utils.logging import LaunchLogger class GridValidator: diff --git a/examples/cfd/darcy_nested_fnos/README.md b/examples/cfd/darcy_nested_fnos/README.md index 160b867a02..147b828b73 100644 --- a/examples/cfd/darcy_nested_fnos/README.md +++ b/examples/cfd/darcy_nested_fnos/README.md @@ -1,14 +1,22 @@ -# Nested Fourier Neural Operater for Darcy Flow +# Nested Fourier Neural Operator for Darcy Flow This example demonstrates how to set up a data-driven model for a 2D Darcy flow using -the Nested Fourier Neural Operator (FNO) architecture inside of Modulus. +the Nested Fourier Neural Operator (FNO) architecture inside of PhysicsNeMo. Training progress can be tracked through [MLFlow](https://mlflow.org/docs/latest/index.html). This case is parallelised to run in multi-GPU settings. ## Getting Started +### Prerequisites + +Install the required dependencies by running below: + +```bash +pip install -r requirements.txt +``` + Start with generating the dataset for training: ```bash diff --git a/examples/cfd/darcy_nested_fnos/config.yaml b/examples/cfd/darcy_nested_fnos/config.yaml index f127ba518b..af8a6d225a 100644 --- a/examples/cfd/darcy_nested_fnos/config.yaml +++ b/examples/cfd/darcy_nested_fnos/config.yaml @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/examples/cfd/darcy_nested_fnos/evaluate_nested_darcy.py b/examples/cfd/darcy_nested_fnos/evaluate_nested_darcy.py index 212abc6878..4daeaf3c58 100644 --- a/examples/cfd/darcy_nested_fnos/evaluate_nested_darcy.py +++ b/examples/cfd/darcy_nested_fnos/evaluate_nested_darcy.py @@ -1,273 +1,275 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import hydra -from torch import cat, FloatTensor -import numpy as np -import matplotlib.pyplot as plt -from os.path import join -from omegaconf import DictConfig, open_dict -from torch.utils.data import DataLoader - -from modulus.models.mlp import FullyConnected -from modulus.models.fno import FNO -from modulus.utils import StaticCaptureEvaluateNoGrad -from modulus.distributed import DistributedManager -from modulus.launch.logging import PythonLogger -from modulus.launch.utils import load_checkpoint - -from utils import NestedDarcyDataset, PlotNestedDarcy - - -def plot_assembled(perm, darc): - """Utility for plotting""" - headers = ["permeability", "darcy"] - plt.rcParams.update({"font.size": 28}) - fig, ax = plt.subplots(1, 2, figsize=(15 * 2, 15), sharey=True) - im = [] - im.append(ax[0].imshow(perm)) - im.append(ax[1].imshow(darc)) - - for ii in range(len(im)): - fig.colorbar(im[ii], ax=ax[ii], location="bottom", fraction=0.046, pad=0.04) - ax[ii].set_title(headers[ii]) - - fig.savefig(join("./", f"test_test.png")) - - -def EvaluateModel( - cfg: DictConfig, - model_name: str, - norm: dict = {"permeability": (0.0, 1.0), "darcy": (0.0, 1.0)}, - parent_result: FloatTensor = None, - log: PythonLogger = None, -): - """Utility for running inference on trained model""" - # define model and load weights - dist = DistributedManager() - log.info(f"evaluating model {model_name}") - model_cfg = cfg.arch[model_name] - model = FNO( - in_channels=model_cfg.fno.in_channels, - out_channels=model_cfg.decoder.out_features, - decoder_layers=model_cfg.decoder.layers, - decoder_layer_size=model_cfg.decoder.layer_size, - dimension=model_cfg.fno.dimension, - latent_channels=model_cfg.fno.latent_channels, - num_fno_layers=model_cfg.fno.fno_layers, - num_fno_modes=model_cfg.fno.fno_modes, - padding=model_cfg.fno.padding, - ).to(dist.device) - load_checkpoint( - path=f"./checkpoints/best/{model_name}", device=dist.device, models=model - ) - - # prepare data for inference - dataset = NestedDarcyDataset( - mode="eval", - data_path=cfg.inference.inference_set, - model_name=model_name, - norm=norm, - log=log, - parent_prediction=parent_result, - ) - dataloader = DataLoader(dataset, batch_size=cfg.inference.batch_size, shuffle=False) - with open_dict(cfg): - cfg.ref_fac = dataset.ref_fac - cfg.fine_res = dataset.fine_res - cfg.buffer = dataset.buffer - - # store positions of insets if refinement level > 0, ie if not global model - if int(model_name[-1]) > 0: - pos = dataset.position - else: - pos = None - - # define forward method - @StaticCaptureEvaluateNoGrad( - model=model, logger=log, use_amp=False, use_graphs=False - ) - def forward_eval(invars): - return model(invars) - - # evaluate and invert normalisation - invars, result = [], [] - for batch in dataloader: - invars.append(batch["permeability"]) - result.append(forward_eval(batch["permeability"])) - invars = cat(invars, dim=0).detach() - result = cat(result, dim=0).detach() - - return pos, invars, result - - -def AssembleSolutionToDict(cfg: DictConfig, perm: dict, darcy: dict, pos: dict): - """Assemble solution to easily interpretable dict""" - dat, idx = {}, 0 - for ii in range(perm["ref0"].shape[0]): - samp = str(ii) - dat[samp] = { - "ref0": { - "0": { - "permeability": perm["ref0"][ii, 0, ...], - "darcy": darcy["ref0"][ii, 0, ...], - } - } - } - - # insets - dat[samp]["ref1"] = {} - for ins, ps in pos["ref1"][samp].items(): - dat[samp]["ref1"][ins] = { - "permeability": perm["ref1"][idx, 1, ...], - "darcy": darcy["ref1"][idx, 0, ...], - "pos": ps, - } - idx += 1 - - if cfg.inference.save_result: - np.save( - "./nested_darcy_results.npy", - dat, - ) - return dat - - -def AssembleToSingleField(cfg: DictConfig, dat: dict): - """Assemble multiple fields to a single dict""" - ref_fac = cfg.ref_fac - glob_size = dat["0"]["ref0"]["0"]["darcy"].shape[0] - inset_size = dat["0"]["ref1"]["0"]["darcy"].shape[0] - size = ref_fac * glob_size - min_offset = (cfg.fine_res * (ref_fac - 1) + 1) // 2 + cfg.buffer * ref_fac - - perm = np.zeros((len(dat), size, size), dtype=np.float32) - darc = np.zeros_like(perm) - for ii, (_, field) in enumerate(dat.items()): - # extract global premeability and expand to size x size - perm[ii, ...] = np.kron( - field["ref0"]["0"]["permeability"], - np.ones((ref_fac, ref_fac), dtype=field["ref0"]["0"]["permeability"].dtype), - ) - darc[ii, ...] = np.kron( - field["ref0"]["0"]["darcy"], - np.ones((ref_fac, ref_fac), dtype=field["ref0"]["0"]["darcy"].dtype), - ) - - # overwrite refined regions - for __, inset in field["ref1"].items(): - pos = inset["pos"] * ref_fac + min_offset - perm[ - ii, pos[0] : pos[0] + inset_size, pos[1] : pos[1] + inset_size - ] = inset["permeability"] - darc[ - ii, pos[0] : pos[0] + inset_size, pos[1] : pos[1] + inset_size - ] = inset["darcy"] - - return {"permeability": perm, "darcy": darc}, ref_fac - - -def GetRelativeL2(pred, tar): - """Compute L2 error""" - div = 1.0 / tar["darcy"].shape[0] * tar["darcy"].shape[1] - err = pred["darcy"] - tar["darcy"] - - l2_tar = np.sqrt(np.einsum("ijk,ijk->i", tar["darcy"], tar["darcy"]) * div) - l2_err = np.sqrt(np.einsum("ijk,ijk->i", err, err) * div) - - return np.mean(l2_err / l2_tar) - - -def ComputeErrorNorm(cfg: DictConfig, pred_dict: dict, log: PythonLogger, ref0_pred): - """Compute relative L2-norm of error""" - # assemble ref1 and ref2 solutions alongside gound truth to single scalar field - log.info("computing relative L2-norm of error...") - tar_dict = np.load(cfg.inference.inference_set, allow_pickle=True).item()["fields"] - pred, ref_fac = AssembleToSingleField(cfg, pred_dict) - tar = AssembleToSingleField(cfg, tar_dict)[0] - - assert np.all( - tar["permeability"] == pred["permeability"] - ), "Permeability from file is not equal to analysed permeability" - - # compute l2 norm of error - rel_l2_err = GetRelativeL2(pred, tar) - log.log(f" ...which is {rel_l2_err}.") - - if cfg.inference.get_ref0_error_norm: - ref0_pred = np.kron( - ref0_pred, np.ones((ref_fac, ref_fac), dtype=ref0_pred.dtype) - ) - rel_l2_err = GetRelativeL2({"darcy": ref0_pred}, tar) - log.log(f"The error with ref_0 only would be {rel_l2_err}.") - - return - - -@hydra.main(version_base="1.3", config_path=".", config_name="config") -def nested_darcy_evaluation(cfg: DictConfig) -> None: - """Inference of the nested 2D Darcy flow benchmark problem. - - This inference script consecutively evaluates the models of nested FNO for the - nested Darcy problem, taking into account the result of the model associated - with the parent level. All results are stored in a numpy file and a selection - of samples can be plotted in the end. - """ - # initialize monitoring, models and normalisation - DistributedManager.initialize() # Only call this once in the entire script! - log = PythonLogger(name="darcy_fno") - - model_names = sorted(list(cfg.arch.keys())) - norm = { - "permeability": ( - cfg.normaliser.permeability.mean, - cfg.normaliser.permeability.std, - ), - "darcy": (cfg.normaliser.darcy.mean, cfg.normaliser.darcy.std), - } - - # evaluate models and revoke normalisation - perm, darcy, pos, result, ref0_pred = {}, {}, {}, None, None - for name in model_names: - position, invars, result = EvaluateModel(cfg, name, norm, result, log) - perm[name] = ( - (invars * norm["permeability"][1] + norm["permeability"][0]) - .detach() - .cpu() - .numpy() - ) - darcy[name] = ( - (result * norm["darcy"][1] + norm["darcy"][0]).detach().cpu().numpy() - ) - pos[name] = position - - if cfg.inference.get_ref0_error_norm and int(name[-1]) == 0: - ref0_pred = np.copy(darcy[name]).squeeze() - - # port solution format to dict structure like in input files - pred_dict = AssembleSolutionToDict(cfg, perm, darcy, pos) - - # compute error norm - if cfg.inference.get_error_norm: - ComputeErrorNorm(cfg, pred_dict, log, ref0_pred) - - # plot some fields - if cfg.inference.n_plots > 0: - log.info("plotting results") - for idx in range(cfg.inference.n_plots): - PlotNestedDarcy(pred_dict, idx) - - -if __name__ == "__main__": - nested_darcy_evaluation() +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import hydra +from torch import cat, FloatTensor +import numpy as np +import matplotlib.pyplot as plt +from os.path import join +from omegaconf import DictConfig, open_dict +from torch.utils.data import DataLoader + +from physicsnemo.models.mlp import FullyConnected +from physicsnemo.models.fno import FNO +from physicsnemo.utils import StaticCaptureEvaluateNoGrad +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils.logging import PythonLogger +from physicsnemo.utils import load_checkpoint + +from utils import NestedDarcyDataset, PlotNestedDarcy + + +def plot_assembled(perm, darc): + """Utility for plotting""" + headers = ["permeability", "darcy"] + plt.rcParams.update({"font.size": 28}) + fig, ax = plt.subplots(1, 2, figsize=(15 * 2, 15), sharey=True) + im = [] + im.append(ax[0].imshow(perm)) + im.append(ax[1].imshow(darc)) + + for ii in range(len(im)): + fig.colorbar(im[ii], ax=ax[ii], location="bottom", fraction=0.046, pad=0.04) + ax[ii].set_title(headers[ii]) + + fig.savefig(join("./", f"test_test.png")) + + +def EvaluateModel( + cfg: DictConfig, + model_name: str, + norm: dict = {"permeability": (0.0, 1.0), "darcy": (0.0, 1.0)}, + parent_result: FloatTensor = None, + log: PythonLogger = None, +): + """Utility for running inference on trained model""" + # define model and load weights + dist = DistributedManager() + log.info(f"evaluating model {model_name}") + model_cfg = cfg.arch[model_name] + model = FNO( + in_channels=model_cfg.fno.in_channels, + out_channels=model_cfg.decoder.out_features, + decoder_layers=model_cfg.decoder.layers, + decoder_layer_size=model_cfg.decoder.layer_size, + dimension=model_cfg.fno.dimension, + latent_channels=model_cfg.fno.latent_channels, + num_fno_layers=model_cfg.fno.fno_layers, + num_fno_modes=model_cfg.fno.fno_modes, + padding=model_cfg.fno.padding, + ).to(dist.device) + load_checkpoint( + path=f"./checkpoints/best/{model_name}", device=dist.device, models=model + ) + + # prepare data for inference + dataset = NestedDarcyDataset( + mode="eval", + data_path=cfg.inference.inference_set, + model_name=model_name, + norm=norm, + log=log, + parent_prediction=parent_result, + ) + dataloader = DataLoader(dataset, batch_size=cfg.inference.batch_size, shuffle=False) + with open_dict(cfg): + cfg.ref_fac = dataset.ref_fac + cfg.fine_res = dataset.fine_res + cfg.buffer = dataset.buffer + + # store positions of insets if refinement level > 0, ie if not global model + if int(model_name[-1]) > 0: + pos = dataset.position + else: + pos = None + + # define forward method + @StaticCaptureEvaluateNoGrad( + model=model, logger=log, use_amp=False, use_graphs=False + ) + def forward_eval(invars): + return model(invars) + + # evaluate and invert normalisation + invars, result = [], [] + for batch in dataloader: + invars.append(batch["permeability"]) + result.append(forward_eval(batch["permeability"])) + invars = cat(invars, dim=0).detach() + result = cat(result, dim=0).detach() + + return pos, invars, result + + +def AssembleSolutionToDict(cfg: DictConfig, perm: dict, darcy: dict, pos: dict): + """Assemble solution to easily interpretable dict""" + dat, idx = {}, 0 + for ii in range(perm["ref0"].shape[0]): + samp = str(ii) + dat[samp] = { + "ref0": { + "0": { + "permeability": perm["ref0"][ii, 0, ...], + "darcy": darcy["ref0"][ii, 0, ...], + } + } + } + + # insets + dat[samp]["ref1"] = {} + for ins, ps in pos["ref1"][samp].items(): + dat[samp]["ref1"][ins] = { + "permeability": perm["ref1"][idx, 1, ...], + "darcy": darcy["ref1"][idx, 0, ...], + "pos": ps, + } + idx += 1 + + if cfg.inference.save_result: + np.save( + "./nested_darcy_results.npy", + dat, + ) + return dat + + +def AssembleToSingleField(cfg: DictConfig, dat: dict): + """Assemble multiple fields to a single dict""" + ref_fac = cfg.ref_fac + glob_size = dat["0"]["ref0"]["0"]["darcy"].shape[0] + inset_size = dat["0"]["ref1"]["0"]["darcy"].shape[0] + size = ref_fac * glob_size + min_offset = (cfg.fine_res * (ref_fac - 1) + 1) // 2 + cfg.buffer * ref_fac + + perm = np.zeros((len(dat), size, size), dtype=np.float32) + darc = np.zeros_like(perm) + for ii, (_, field) in enumerate(dat.items()): + # extract global premeability and expand to size x size + perm[ii, ...] = np.kron( + field["ref0"]["0"]["permeability"], + np.ones((ref_fac, ref_fac), dtype=field["ref0"]["0"]["permeability"].dtype), + ) + darc[ii, ...] = np.kron( + field["ref0"]["0"]["darcy"], + np.ones((ref_fac, ref_fac), dtype=field["ref0"]["0"]["darcy"].dtype), + ) + + # overwrite refined regions + for __, inset in field["ref1"].items(): + pos = inset["pos"] * ref_fac + min_offset + perm[ii, pos[0] : pos[0] + inset_size, pos[1] : pos[1] + inset_size] = ( + inset["permeability"] + ) + darc[ii, pos[0] : pos[0] + inset_size, pos[1] : pos[1] + inset_size] = ( + inset["darcy"] + ) + + return {"permeability": perm, "darcy": darc}, ref_fac + + +def GetRelativeL2(pred, tar): + """Compute L2 error""" + div = 1.0 / tar["darcy"].shape[0] * tar["darcy"].shape[1] + err = pred["darcy"] - tar["darcy"] + + l2_tar = np.sqrt(np.einsum("ijk,ijk->i", tar["darcy"], tar["darcy"]) * div) + l2_err = np.sqrt(np.einsum("ijk,ijk->i", err, err) * div) + + return np.mean(l2_err / l2_tar) + + +def ComputeErrorNorm(cfg: DictConfig, pred_dict: dict, log: PythonLogger, ref0_pred): + """Compute relative L2-norm of error""" + # assemble ref1 and ref2 solutions alongside gound truth to single scalar field + log.info("computing relative L2-norm of error...") + tar_dict = np.load(cfg.inference.inference_set, allow_pickle=True).item()["fields"] + pred, ref_fac = AssembleToSingleField(cfg, pred_dict) + tar = AssembleToSingleField(cfg, tar_dict)[0] + + assert np.all(tar["permeability"] == pred["permeability"]), ( + "Permeability from file is not equal to analysed permeability" + ) + + # compute l2 norm of error + rel_l2_err = GetRelativeL2(pred, tar) + log.log(f" ...which is {rel_l2_err}.") + + if cfg.inference.get_ref0_error_norm: + ref0_pred = np.kron( + ref0_pred, np.ones((ref_fac, ref_fac), dtype=ref0_pred.dtype) + ) + rel_l2_err = GetRelativeL2({"darcy": ref0_pred}, tar) + log.log(f"The error with ref_0 only would be {rel_l2_err}.") + + return + + +@hydra.main(version_base="1.3", config_path=".", config_name="config") +def nested_darcy_evaluation(cfg: DictConfig) -> None: + """Inference of the nested 2D Darcy flow benchmark problem. + + This inference script consecutively evaluates the models of nested FNO for the + nested Darcy problem, taking into account the result of the model associated + with the parent level. All results are stored in a numpy file and a selection + of samples can be plotted in the end. + """ + # initialize monitoring, models and normalisation + DistributedManager.initialize() # Only call this once in the entire script! + log = PythonLogger(name="darcy_fno") + + model_names = sorted(list(cfg.arch.keys())) + norm = { + "permeability": ( + cfg.normaliser.permeability.mean, + cfg.normaliser.permeability.std, + ), + "darcy": (cfg.normaliser.darcy.mean, cfg.normaliser.darcy.std), + } + + # evaluate models and revoke normalisation + perm, darcy, pos, result, ref0_pred = {}, {}, {}, None, None + for name in model_names: + position, invars, result = EvaluateModel(cfg, name, norm, result, log) + perm[name] = ( + (invars * norm["permeability"][1] + norm["permeability"][0]) + .detach() + .cpu() + .numpy() + ) + darcy[name] = ( + (result * norm["darcy"][1] + norm["darcy"][0]).detach().cpu().numpy() + ) + pos[name] = position + + if cfg.inference.get_ref0_error_norm and int(name[-1]) == 0: + ref0_pred = np.copy(darcy[name]).squeeze() + + # port solution format to dict structure like in input files + pred_dict = AssembleSolutionToDict(cfg, perm, darcy, pos) + + # compute error norm + if cfg.inference.get_error_norm: + ComputeErrorNorm(cfg, pred_dict, log, ref0_pred) + + # plot some fields + if cfg.inference.n_plots > 0: + log.info("plotting results") + for idx in range(cfg.inference.n_plots): + PlotNestedDarcy(pred_dict, idx) + + +if __name__ == "__main__": + nested_darcy_evaluation() diff --git a/examples/cfd/darcy_nested_fnos/generate_nested_darcy.py b/examples/cfd/darcy_nested_fnos/generate_nested_darcy.py index ee50b71e9b..ef640462d9 100644 --- a/examples/cfd/darcy_nested_fnos/generate_nested_darcy.py +++ b/examples/cfd/darcy_nested_fnos/generate_nested_darcy.py @@ -1,136 +1,138 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from os.path import isdir -from os import mkdir -import numpy as np -from utils import DarcyInset2D, PlotNestedDarcy - - -def nested_darcy_generator() -> None: - """Dataset Generator for the nested Darcy Problem - - This script generates the training, validation and out-of-sample data sets - for the nested FNO problem and stores them in ./data, where trainer and - inferencer will find it. - """ - out_dir = "./data/" - file_names = ["training_data.npy", "validation_data.npy", "out_of_sample.npy"] - sample_size = [8192, 2048, 2048] - max_batch_size = 128 - resolution = 1024 - glob_res = 256 - fine_res = 128 - buffer = 32 - permea_freq = 3 - max_n_insets = 2 - fine_permeability_freq = 2 - min_dist_frac = 1.8 - device = "cuda" - n_plots = 10 - fill_val = -99999 - - perm_norm = (0.0, 1.0) - darc_norm = (0.0, 1.0) - - if not isdir(out_dir): - mkdir(out_dir) - - assert resolution % glob_res == 0, "resolution needs to be multiple of glob_res" - ref_fac = resolution // glob_res - inset_size = fine_res + 2 * buffer - min_offset = (fine_res * (ref_fac - 1) + 1) // 2 + buffer * ref_fac - - # force inset on coarse grid - if not min_offset % ref_fac == 0: - min_offset += ref_fac - min_offset % ref_fac - - for dset in range(len(file_names)): - # compute batch size and number of iterations - batch_size = min(max_batch_size, sample_size[dset]) - nr_iterations = (sample_size[dset] - 1) // max_batch_size + 1 - - datapipe = DarcyInset2D( - resolution=resolution, - batch_size=batch_size, - nr_permeability_freq=permea_freq, - max_permeability=2.0, - min_permeability=0.5, - max_iterations=30000, - iterations_per_convergence_check=10, - nr_multigrids=3, - normaliser={"permeability": perm_norm, "darcy": darc_norm}, - device=device, - max_n_insets=max_n_insets, - fine_res=fine_res, - fine_permeability_freq=fine_permeability_freq, - min_offset=min_offset, - ref_fac=ref_fac, - min_dist_frac=min_dist_frac, - fill_val=fill_val, - ) - - dat = {} - samp_ind = -1 - for _, sample in zip(range(nr_iterations), datapipe): - permea = sample["permeability"].cpu().detach().numpy() - darcy = sample["darcy"].cpu().detach().numpy() - pos = (sample["inset_pos"].cpu().detach().numpy()).astype(int) - assert ( - np.where(pos == fill_val, 0, pos) % ref_fac - ).sum() == 0, "inset off coarse grid" - - # crop out refined region, allow for surrounding area, save in extra array - for ii in range(batch_size): - samp_ind += 1 - samp_str = str(samp_ind) - - # global fields - dat[samp_str] = { - "ref0": { - "0": { - "permeability": permea[ii, 0, ::ref_fac, ::ref_fac], - "darcy": darcy[ii, 0, ::ref_fac, ::ref_fac], - } - } - } - - # insets - dat[samp_str]["ref1"] = {} - for pp in range(pos.shape[1]): - if pos[ii, pp, 0] == fill_val: - continue - xs = pos[ii, pp, 0] - buffer - ys = pos[ii, pp, 1] - buffer - - dat[samp_str]["ref1"][str(pp)] = { - "permeability": permea[ - ii, 0, xs : xs + inset_size, ys : ys + inset_size - ], - "darcy": darcy[ - ii, 0, xs : xs + inset_size, ys : ys + inset_size - ], - "pos": (pos[ii, pp, :] - min_offset) // ref_fac, - } - meta = {"ref_fac": ref_fac, "buffer": buffer, "fine_res": fine_res} - - np.save(out_dir + file_names[dset], {"meta": meta, "fields": dat}) - - # plot some fields - for idx in range(n_plots): - PlotNestedDarcy(dat, idx) - - -if __name__ == "__main__": - nested_darcy_generator() +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from os.path import isdir +from os import mkdir +import numpy as np +from utils import DarcyInset2D, PlotNestedDarcy + + +def nested_darcy_generator() -> None: + """Dataset Generator for the nested Darcy Problem + + This script generates the training, validation and out-of-sample data sets + for the nested FNO problem and stores them in ./data, where trainer and + inferencer will find it. + """ + out_dir = "./data/" + file_names = ["training_data.npy", "validation_data.npy", "out_of_sample.npy"] + sample_size = [8192, 2048, 2048] + max_batch_size = 128 + resolution = 1024 + glob_res = 256 + fine_res = 128 + buffer = 32 + permea_freq = 3 + max_n_insets = 2 + fine_permeability_freq = 2 + min_dist_frac = 1.8 + device = "cuda" + n_plots = 10 + fill_val = -99999 + + perm_norm = (0.0, 1.0) + darc_norm = (0.0, 1.0) + + if not isdir(out_dir): + mkdir(out_dir) + + assert resolution % glob_res == 0, "resolution needs to be multiple of glob_res" + ref_fac = resolution // glob_res + inset_size = fine_res + 2 * buffer + min_offset = (fine_res * (ref_fac - 1) + 1) // 2 + buffer * ref_fac + + # force inset on coarse grid + if not min_offset % ref_fac == 0: + min_offset += ref_fac - min_offset % ref_fac + + for dset in range(len(file_names)): + # compute batch size and number of iterations + batch_size = min(max_batch_size, sample_size[dset]) + nr_iterations = (sample_size[dset] - 1) // max_batch_size + 1 + + datapipe = DarcyInset2D( + resolution=resolution, + batch_size=batch_size, + nr_permeability_freq=permea_freq, + max_permeability=2.0, + min_permeability=0.5, + max_iterations=30000, + iterations_per_convergence_check=10, + nr_multigrids=3, + normaliser={"permeability": perm_norm, "darcy": darc_norm}, + device=device, + max_n_insets=max_n_insets, + fine_res=fine_res, + fine_permeability_freq=fine_permeability_freq, + min_offset=min_offset, + ref_fac=ref_fac, + min_dist_frac=min_dist_frac, + fill_val=fill_val, + ) + + dat = {} + samp_ind = -1 + for _, sample in zip(range(nr_iterations), datapipe): + permea = sample["permeability"].cpu().detach().numpy() + darcy = sample["darcy"].cpu().detach().numpy() + pos = (sample["inset_pos"].cpu().detach().numpy()).astype(int) + assert (np.where(pos == fill_val, 0, pos) % ref_fac).sum() == 0, ( + "inset off coarse grid" + ) + + # crop out refined region, allow for surrounding area, save in extra array + for ii in range(batch_size): + samp_ind += 1 + samp_str = str(samp_ind) + + # global fields + dat[samp_str] = { + "ref0": { + "0": { + "permeability": permea[ii, 0, ::ref_fac, ::ref_fac], + "darcy": darcy[ii, 0, ::ref_fac, ::ref_fac], + } + } + } + + # insets + dat[samp_str]["ref1"] = {} + for pp in range(pos.shape[1]): + if pos[ii, pp, 0] == fill_val: + continue + xs = pos[ii, pp, 0] - buffer + ys = pos[ii, pp, 1] - buffer + + dat[samp_str]["ref1"][str(pp)] = { + "permeability": permea[ + ii, 0, xs : xs + inset_size, ys : ys + inset_size + ], + "darcy": darcy[ + ii, 0, xs : xs + inset_size, ys : ys + inset_size + ], + "pos": (pos[ii, pp, :] - min_offset) // ref_fac, + } + meta = {"ref_fac": ref_fac, "buffer": buffer, "fine_res": fine_res} + + np.save(out_dir + file_names[dset], {"meta": meta, "fields": dat}) + + # plot some fields + for idx in range(n_plots): + PlotNestedDarcy(dat, idx) + + +if __name__ == "__main__": + nested_darcy_generator() diff --git a/examples/cfd/darcy_nested_fnos/requirements.txt b/examples/cfd/darcy_nested_fnos/requirements.txt new file mode 100644 index 0000000000..c51f257854 --- /dev/null +++ b/examples/cfd/darcy_nested_fnos/requirements.txt @@ -0,0 +1 @@ +mlflow>=2.1.1 \ No newline at end of file diff --git a/examples/cfd/darcy_nested_fnos/train_nested_darcy.py b/examples/cfd/darcy_nested_fnos/train_nested_darcy.py index 80d17bee1d..bbd7620c3a 100644 --- a/examples/cfd/darcy_nested_fnos/train_nested_darcy.py +++ b/examples/cfd/darcy_nested_fnos/train_nested_darcy.py @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -23,17 +25,16 @@ from torch.nn.parallel import DistributedDataParallel from torch.utils.data.distributed import DistributedSampler -from modulus.models.fno import FNO -from modulus.distributed import DistributedManager -from modulus.utils import StaticCaptureTraining, StaticCaptureEvaluateNoGrad -from modulus.launch.utils import load_checkpoint, save_checkpoint -from modulus.launch.logging import ( +from physicsnemo.models.fno import FNO +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils import StaticCaptureTraining, StaticCaptureEvaluateNoGrad +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import ( PythonLogger, RankZeroLoggingWrapper, LaunchLogger, - initialize_mlflow, ) - +from physicsnemo.utils.logging.mlflow import initialize_mlflow from utils import NestedDarcyDataset, GridValidator @@ -69,7 +70,7 @@ def InitializeLoggers(cfg: DictConfig) -> Tuple[DistributedManager, PythonLogger user_name="Gretchen Ross", mode="offline", ) - LaunchLogger.initialize(use_mlflow=True) # Modulus launch logger + LaunchLogger.initialize(use_mlflow=True) # PhysicsNeMo launch logger return dist, RankZeroLoggingWrapper(logger, dist) @@ -301,7 +302,7 @@ def nested_darcy_trainer(cfg: DictConfig) -> None: if loaded_epoch == 0: logger.success("Training started...") else: - logger.warning(f"Resuming training from epoch {loaded_epoch+1}.") + logger.warning(f"Resuming training from epoch {loaded_epoch + 1}.") # train model TrainModel(cfg, base, loaded_epoch) diff --git a/examples/cfd/darcy_nested_fnos/utils.py b/examples/cfd/darcy_nested_fnos/utils.py index 5276ddae47..811abe8bea 100644 --- a/examples/cfd/darcy_nested_fnos/utils.py +++ b/examples/cfd/darcy_nested_fnos/utils.py @@ -1,590 +1,594 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import torch -import os.path -import warp as wp -import numpy as np -import matplotlib.pyplot as plt -from typing import Union, Tuple, Dict -from torch import FloatTensor, Tensor -from torch.nn import MSELoss -from modulus.distributed import DistributedManager -from modulus.launch.logging import PythonLogger, LaunchLogger -from modulus.datapipes.benchmarks.darcy import Darcy2D -from modulus.datapipes.benchmarks.kernels.initialization import init_uniform_random_4d -from modulus.datapipes.benchmarks.kernels.utils import ( - fourier_to_array_batched_2d, - threshold_3d, -) - - -class NestedDarcyDataset: - """Nested Darcy Dataset - - A Dataset class for loading nested Darcy data generated with generate_nested_darcy.py - during training. The method takes care of loading the correct level and associated - information from its parent level. - - Parameters - ---------- - data_path : str - Path to numpy dict file containing the data - level : int, optional - Refinement level which shall be loaded - norm : Dict, optional - mean and standard deviation for each channel to normalise input and target - log : PythonLogger - logger for command line output - - """ - - def __init__( - self, - mode: str, - data_path: str = None, - model_name: str = None, - norm: dict = {"permeability": (0.0, 1.0), "darcy": (0.0, 1.0)}, - log: PythonLogger = None, - parent_prediction: FloatTensor = None, - ) -> None: - self.dist = DistributedManager() - self.data_path = os.path.abspath(data_path) - self.model_name = model_name - # self.level = level - self.norm = norm - self.log = log - self.mode = mode - assert self.mode in [ - "train", - "eval", - ], "mode in NestedDarcyDataset must be train or eval." - - if mode == "eval" and int(self.model_name[-1]) > 0: - assert ( - parent_prediction is not None - ), f"pass parent result to evaluate level {int(self.model_name[-1])}" - parent_prediction = parent_prediction.detach().cpu().numpy() - self.load_dataset(parent_prediction) - - def load_dataset(self, parent_prediction: FloatTensor = None) -> None: - try: - contents = np.load(self.data_path, allow_pickle=True).item() - except IOError as err: - self.log.error(f"Unable to find or load file {self.data_path}") - exit() - - # load input varibales, copy to device and normalise - dat = contents["fields"] - self.ref_fac = contents["meta"]["ref_fac"] - self.buffer = contents["meta"]["buffer"] - self.fine_res = contents["meta"]["fine_res"] - - mod = self.model_name - perm, darc, par_pred, self.position = [], [], [], {} - for id, samp in dat.items(): - if int(mod[-1]) > 0: - self.position[id] = {} - for jd, fields in samp[mod].items(): - perm.append(fields["permeability"][None, None, ...]) - darc.append(fields["darcy"][None, None, ...]) - - if int(mod[-1]) > 0: # if not on global level - xy_size = perm[-1].shape[-1] - pos = fields["pos"] - self.position[id][jd] = pos - if self.mode == "eval": - parent = parent_prediction[int(id), 0, ...] - elif self.mode == "train": - parent = ( - samp[f"ref{int(mod[-1])-1}"]["0"]["darcy"] - - self.norm["darcy"][0] - ) / self.norm["darcy"][1] - par_pred.append( - parent[ - pos[0] : pos[0] + xy_size, - pos[1] : pos[1] + xy_size, - ][None, None, ...] - ) - - perm = ( - np.concatenate(perm, axis=0) - self.norm["permeability"][0] - ) / self.norm["permeability"][1] - darc = (np.concatenate(darc, axis=0) - self.norm["darcy"][0]) / self.norm[ - "darcy" - ][1] - - if int(mod[-1]) > 0: - par_pred = np.concatenate(par_pred, axis=0) - perm = np.concatenate((par_pred, perm), axis=1) - - self.invars = torch.from_numpy(perm).float().to(self.dist.device) - self.outvars = torch.from_numpy(darc).float().to(self.dist.device) - - self.length = self.invars.size()[0] - - def __getitem__(self, idx: int): - return {"permeability": self.invars[idx, ...], "darcy": self.outvars[idx, ...]} - - def __len__(self): - return self.length - - -class GridValidator: - """Grid Validator - - The validator compares model output and target, inverts normalisation and plots a sample - - Parameters - ---------- - loss_fun : MSELoss - loss function for assessing validation error - norm : Dict, optional - mean and standard deviation for each channel to normalise input and target - font_size : float, optional - font size used in figures - - """ - - def __init__( - self, - loss_fun: MSELoss, - norm: dict = {"permeability": (0.0, 1.0), "darcy": (0.0, 1.0)}, - font_size: float = 28.0, - ) -> None: - self.norm = norm - self.criterion = loss_fun - self.font_size = font_size - self.headers = ("invar", "truth", "prediction", "relative error") - - def compare( - self, - invar: FloatTensor, - target: FloatTensor, - prediction: FloatTensor, - step: int, - logger: LaunchLogger, - ) -> float: - """compares model output, target and plots everything - - Parameters - ---------- - invar : FloatTensor - input to model - target : FloatTensor - ground truth - prediction : FloatTensor - model output - step : int - iteration counter - logger : LaunchLogger - logger to which figure is passed - - Returns - ------- - float - validation error - """ - loss = self.criterion(prediction, target) - norm = self.norm - - # pick first sample from batch - invar = invar * norm["permeability"][1] + norm["permeability"][0] - target = target * norm["darcy"][1] + norm["darcy"][0] - prediction = prediction * norm["darcy"][1] + norm["darcy"][0] - invar = invar.cpu().numpy()[0, -1, :, :] - target = target.cpu().numpy()[0, 0, :, :] - prediction = prediction.detach().cpu().numpy()[0, 0, :, :] - - plt.close("all") - plt.rcParams.update({"font.size": self.font_size}) - fig, ax = plt.subplots(1, 4, figsize=(15 * 3.5, 15), sharey=True) - im = [] - im.append(ax[0].imshow(invar)) - im.append(ax[1].imshow(target)) - im.append(ax[2].imshow(prediction)) - im.append(ax[3].imshow((prediction - target) / norm["darcy"][1])) - - for ii in range(len(im)): - fig.colorbar(im[ii], ax=ax[ii], location="bottom", fraction=0.046, pad=0.04) - ax[ii].set_title(self.headers[ii]) - - logger.log_figure(figure=fig, artifact_file=f"validation_step_{step:03d}.png") - - return loss - - -def PlotNestedDarcy(dat: dict, idx: int) -> None: - """Plot fields from the nested Darcy case - - Parameters - ---------- - dat : dict - dictionary containing fields - target : FloatTensor - index of example to plot - """ - fields = dat[str(idx)] - n_insets = len(fields["ref1"]) - - fig, ax = plt.subplots(n_insets + 1, 4, figsize=(20, 5 * (n_insets + 1))) - - vmin = fields["ref0"]["0"]["darcy"].min() - vmax = fields["ref0"]["0"]["darcy"].max() - - ax[0, 0].imshow(fields["ref0"]["0"]["permeability"]) - ax[0, 0].set_title("permeability glob") - ax[0, 1].imshow(fields["ref0"]["0"]["darcy"], vmin=vmin, vmax=vmax) - ax[0, 1].set_title("darcy glob") - ax[0, 2].axis("off") - ax[0, 3].axis("off") - - for ii in range(n_insets): - loc = fields["ref1"][str(ii)] - inset_size = loc["darcy"].shape[1] - ax[ii + 1, 0].imshow(loc["permeability"]) - ax[ii + 1, 0].set_title(f"permeability fine {ii}") - ax[ii + 1, 1].imshow(loc["darcy"], vmin=vmin, vmax=vmax) - ax[ii + 1, 1].set_title(f"darcy fine {ii}") - ax[ii + 1, 2].imshow( - fields["ref0"]["0"]["permeability"][ - loc["pos"][0] : loc["pos"][0] + inset_size, - loc["pos"][1] : loc["pos"][1] + inset_size, - ] - ) - ax[ii + 1, 2].set_title(f"permeability zoomed {ii}") - ax[ii + 1, 3].imshow( - fields["ref0"]["0"]["darcy"][ - loc["pos"][0] : loc["pos"][0] + inset_size, - loc["pos"][1] : loc["pos"][1] + inset_size, - ], - vmin=vmin, - vmax=vmax, - ) - ax[ii + 1, 3].set_title(f"darcy zoomed {ii}") - - fig.tight_layout() - plt.savefig(f"sample_{idx:02d}.png") - plt.close() - - -@wp.kernel -def fourier_to_array_batched_2d_cropped( - array: wp.array3d(dtype=float), - fourier: wp.array4d(dtype=float), - nr_freq: int, - lx: int, - ly: int, - bounds: wp.array3d(dtype=int), - fill_val: int, -): # pragma: no cover - """Array of Fourier amplitudes to batched 2d spatial array - - Parameters - ---------- - array : wp.array3d - Spatial array - fourier : wp.array4d - Array of Fourier amplitudes - nr_freq : int - Number of frequencies in Fourier array - lx : int - Grid size x - ly : int - Grid size y - x_start : int - lowest x-index - y_start : int - lowest y-index - """ - b, p, x, y = wp.tid() - - if bounds[b, p, 0] == fill_val: - return - - x += bounds[b, p, 0] - y += bounds[b, p, 1] - - array[b, x, y] = 0.0 - dx = 6.28318 / wp.float32(lx) - dy = 6.28318 / wp.float32(ly) - rx = dx * wp.float32(x) - ry = dy * wp.float32(y) - for i in range(nr_freq): - for j in range(nr_freq): - ri = wp.float32(i) - rj = wp.float32(j) - ss = fourier[0, b, i, j] * wp.sin(ri * rx) * wp.sin(rj * ry) - cs = fourier[1, b, i, j] * wp.cos(ri * rx) * wp.sin(rj * ry) - sc = fourier[2, b, i, j] * wp.sin(ri * rx) * wp.cos(rj * ry) - cc = fourier[3, b, i, j] * wp.cos(ri * rx) * wp.cos(rj * ry) - wp.atomic_add( - array, b, x, y, 1.0 / (wp.float32(nr_freq) ** 2.0) * (ss + cs + sc + cc) - ) - - -class DarcyInset2D(Darcy2D): - """2D Darcy flow benchmark problem datapipe. - - This datapipe continuously generates solutions to the 2D Darcy equation with variable - permeability. All samples are generated on the fly and is meant to be a benchmark - problem for testing data driven models. Permeability is drawn from a random Fourier - series and threshold it to give a piecewise constant function. The solution is obtained - using a GPU enabled multi-grid Jacobi iterative method. - - Parameters - ---------- - resolution : int, optional - Resolution to run simulation at, by default 256 - batch_size : int, optional - Batch size of simulations, by default 64 - nr_permeability_freq : int, optional - Number of frequencies to use for generating random permeability. Higher values - will give higher freq permeability fields., by default 5 - max_permeability : float, optional - Max permeability, by default 2.0 - min_permeability : float, optional - Min permeability, by default 0.5 - max_iterations : int, optional - Maximum iterations to use for each multi-grid, by default 30000 - convergence_threshold : float, optional - Solver L-Infinity convergence threshold, by default 1e-6 - iterations_per_convergence_check : int, optional - Number of Jacobi iterations to run before checking convergence, by default 1000 - nr_multigrids : int, optional - Number of multi-grid levels, by default 4 - normaliser : Union[Dict[str, Tuple[float, float]], None], optional - Dictionary with keys `permeability` and `darcy`. The values for these keys are two floats corresponding to mean and std `(mean, std)`. - device : Union[str, torch.device], optional - Device for datapipe to run place data on, by default "cuda" - - Raises - ------ - ValueError - Incompatable multi-grid and resolution settings - """ - - def __init__( - self, - resolution: int = 256, - batch_size: int = 64, - nr_permeability_freq: int = 5, - max_permeability: float = 2.0, - min_permeability: float = 0.5, - max_iterations: int = 30000, - convergence_threshold: float = 1e-6, - iterations_per_convergence_check: int = 1000, - nr_multigrids: int = 4, - normaliser: Union[Dict[str, Tuple[float, float]], None] = None, - device: Union[str, torch.device] = "cuda", - max_n_insets: int = 3, - fine_res: int = 32, - fine_permeability_freq: int = 10, - min_offset: int = 48, - ref_fac: int = None, - min_dist_frac: float = 1.7, - fill_val: int = -99999, - ): - super().__init__( - resolution, - batch_size, - nr_permeability_freq, - max_permeability, - min_permeability, - max_iterations, - convergence_threshold, - iterations_per_convergence_check, - nr_multigrids, - normaliser, - device, - ) - - self.max_n_insets = max_n_insets - self.fine_res = fine_res - self.fine_freq = fine_permeability_freq - self.ref_fac = ref_fac - assert ( - resolution % self.ref_fac == 0 - ), "simulation res must be multiple of ref_fac" - - # force inset on coarse grid - if not min_offset % self.ref_fac == 0: - min_offset += self.ref_fac - min_offset % self.ref_fac - self.beg_min = min_offset - self.beg_max = resolution - min_offset - fine_res - self.ref_fac - self.bounds = None - self.min_dist_frac = min_dist_frac - self.fill_val = fill_val - - assert ( - self.max_n_insets <= 3 - ), f"at most 3 insets supported, change max_n_insets accordingly" - assert (self.beg_max - self.beg_min) % ref_fac == 0, "lsdhfgn3x!!!!" - - def initialize_batch(self) -> None: - """Initializes arrays for new batch of simulations""" - - # initialize permeability - self.permeability.zero_() - seed = np.random.randint(np.iinfo(np.uint64).max, dtype=np.uint64) - wp.launch( - kernel=init_uniform_random_4d, - dim=self.fourier_dim, - inputs=[self.rand_fourier, -1.0, 1.0, seed], - device=self.device, - ) - wp.launch( - kernel=fourier_to_array_batched_2d, - dim=self.dim, - inputs=[ - self.permeability, - self.rand_fourier, - self.nr_permeability_freq, - self.resolution, - self.resolution, - ], - device=self.device, - ) - - rr = np.random.randint( - low=0, - high=(self.beg_max - self.beg_min) // self.ref_fac, - size=(self.batch_size, self.max_n_insets, 2), - ) - n_insets = np.random.randint( - low=1, - high=rr.shape[1] + 1, - size=(self.batch_size,), - ) - - # check that regions do not overlap and have distance - min_dist = self.min_dist_frac * self.fine_res // self.ref_fac + 1 - print("adjusting inset positions") - for ib in range(self.batch_size): - if n_insets[ib] <= 1: - rr[ib, 1:, :] = self.fill_val - continue - else: - while ( - abs(rr[ib, 0, 0] - rr[ib, 1, 0]) < min_dist - and abs(rr[ib, 0, 1] - rr[ib, 1, 1]) < min_dist - ): - rr[ib, 0, :] = np.random.randint( - low=0, - high=(self.beg_max - self.beg_min) // self.ref_fac, - size=(2,), - ) - rr[ib, 1, :] = np.random.randint( - low=0, - high=(self.beg_max - self.beg_min) // self.ref_fac, - size=(2,), - ) - if n_insets[ib] <= 2: - rr[ib, 2:, :] = self.fill_val - continue - else: - while ( - abs(rr[ib, 0, 0] - rr[ib, 2, 0]) < min_dist - and abs(rr[ib, 0, 1] - rr[ib, 2, 1]) < min_dist - ) or ( - abs(rr[ib, 1, 0] - rr[ib, 2, 0]) < min_dist - and abs(rr[ib, 1, 1] - rr[ib, 2, 1]) < min_dist - ): - rr[ib, 2, :] = np.random.randint( - low=0, - high=(self.beg_max - self.beg_min) // self.ref_fac, - size=(2,), - ) - print("done") - - rr = np.where(rr != self.fill_val, (rr * self.ref_fac) + self.beg_min, rr) - self.bounds = wp.array(rr, dtype=int, device=self.device) - - wp.launch( - kernel=fourier_to_array_batched_2d_cropped, - dim=(self.batch_size, self.bounds.shape[1], self.fine_res, self.fine_res), - inputs=[ - self.permeability, - self.rand_fourier, - self.fine_freq, - self.fine_res, - self.fine_res, - self.bounds, - self.fill_val, - ], - device=self.device, - ) - - wp.launch( - kernel=threshold_3d, - dim=self.dim, - inputs=[ - self.permeability, - 0.0, - self.min_permeability, - self.max_permeability, - ], - device=self.device, - ) - - # zero darcy arrays - self.darcy0.zero_() - self.darcy1.zero_() - - def batch_generator(self) -> Tuple[Tensor, Tensor]: - # run simulation - self.generate_batch() - - # convert warp arrays to pytorch - permeability = wp.to_torch(self.permeability) - darcy = wp.to_torch(self.darcy0) - - # add channel dims - permeability = torch.unsqueeze(permeability, axis=1) - darcy = torch.unsqueeze(darcy, axis=1) - - # crop edges by 1 from multi-grid - permeability = permeability[:, :, : self.resolution, : self.resolution] - darcy = darcy[:, :, : self.resolution, : self.resolution] - - # normalize values - if self.normaliser is not None: - permeability = ( - permeability - self.normaliser["permeability"][0] - ) / self.normaliser["permeability"][1] - darcy = (darcy - self.normaliser["darcy"][0]) / self.normaliser["darcy"][1] - - # CUDA graphs static copies - if self.output_k is None: - self.output_k = permeability - self.output_p = darcy - else: - self.output_k.data.copy_(permeability) - self.output_p.data.copy_(darcy) - - return {"permeability": self.output_k, "darcy": self.output_p} - - def __iter__(self) -> Tuple[Tensor, Tensor, Tensor]: - """ - Yields - ------ - Iterator[Tuple[Tensor, Tensor, Tensor]] - Infinite iterator that returns a batch of (permeability, darcy pressure) - fields of size [batch, resolution, resolution] - """ - # infinite generator - while True: - batch = self.batch_generator() - batch["inset_pos"] = wp.to_torch(self.bounds) - yield batch +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +import os.path +import warp as wp +import numpy as np +import matplotlib.pyplot as plt +from typing import Union, Tuple, Dict +from torch import FloatTensor, Tensor +from torch.nn import MSELoss +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils.logging import PythonLogger, LaunchLogger +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.datapipes.benchmarks.kernels.initialization import ( + init_uniform_random_4d, +) +from physicsnemo.datapipes.benchmarks.kernels.utils import ( + fourier_to_array_batched_2d, + threshold_3d, +) + + +class NestedDarcyDataset: + """Nested Darcy Dataset + + A Dataset class for loading nested Darcy data generated with generate_nested_darcy.py + during training. The method takes care of loading the correct level and associated + information from its parent level. + + Parameters + ---------- + data_path : str + Path to numpy dict file containing the data + level : int, optional + Refinement level which shall be loaded + norm : Dict, optional + mean and standard deviation for each channel to normalise input and target + log : PythonLogger + logger for command line output + + """ + + def __init__( + self, + mode: str, + data_path: str = None, + model_name: str = None, + norm: dict = {"permeability": (0.0, 1.0), "darcy": (0.0, 1.0)}, + log: PythonLogger = None, + parent_prediction: FloatTensor = None, + ) -> None: + self.dist = DistributedManager() + self.data_path = os.path.abspath(data_path) + self.model_name = model_name + # self.level = level + self.norm = norm + self.log = log + self.mode = mode + assert self.mode in [ + "train", + "eval", + ], "mode in NestedDarcyDataset must be train or eval." + + if mode == "eval" and int(self.model_name[-1]) > 0: + assert parent_prediction is not None, ( + f"pass parent result to evaluate level {int(self.model_name[-1])}" + ) + parent_prediction = parent_prediction.detach().cpu().numpy() + self.load_dataset(parent_prediction) + + def load_dataset(self, parent_prediction: FloatTensor = None) -> None: + try: + contents = np.load(self.data_path, allow_pickle=True).item() + except IOError as err: + self.log.error(f"Unable to find or load file {self.data_path}") + exit() + + # load input varibales, copy to device and normalise + dat = contents["fields"] + self.ref_fac = contents["meta"]["ref_fac"] + self.buffer = contents["meta"]["buffer"] + self.fine_res = contents["meta"]["fine_res"] + + mod = self.model_name + perm, darc, par_pred, self.position = [], [], [], {} + for id, samp in dat.items(): + if int(mod[-1]) > 0: + self.position[id] = {} + for jd, fields in samp[mod].items(): + perm.append(fields["permeability"][None, None, ...]) + darc.append(fields["darcy"][None, None, ...]) + + if int(mod[-1]) > 0: # if not on global level + xy_size = perm[-1].shape[-1] + pos = fields["pos"] + self.position[id][jd] = pos + if self.mode == "eval": + parent = parent_prediction[int(id), 0, ...] + elif self.mode == "train": + parent = ( + samp[f"ref{int(mod[-1]) - 1}"]["0"]["darcy"] + - self.norm["darcy"][0] + ) / self.norm["darcy"][1] + par_pred.append( + parent[ + pos[0] : pos[0] + xy_size, + pos[1] : pos[1] + xy_size, + ][None, None, ...] + ) + + perm = ( + np.concatenate(perm, axis=0) - self.norm["permeability"][0] + ) / self.norm["permeability"][1] + darc = (np.concatenate(darc, axis=0) - self.norm["darcy"][0]) / self.norm[ + "darcy" + ][1] + + if int(mod[-1]) > 0: + par_pred = np.concatenate(par_pred, axis=0) + perm = np.concatenate((par_pred, perm), axis=1) + + self.invars = torch.from_numpy(perm).float().to(self.dist.device) + self.outvars = torch.from_numpy(darc).float().to(self.dist.device) + + self.length = self.invars.size()[0] + + def __getitem__(self, idx: int): + return {"permeability": self.invars[idx, ...], "darcy": self.outvars[idx, ...]} + + def __len__(self): + return self.length + + +class GridValidator: + """Grid Validator + + The validator compares model output and target, inverts normalisation and plots a sample + + Parameters + ---------- + loss_fun : MSELoss + loss function for assessing validation error + norm : Dict, optional + mean and standard deviation for each channel to normalise input and target + font_size : float, optional + font size used in figures + + """ + + def __init__( + self, + loss_fun: MSELoss, + norm: dict = {"permeability": (0.0, 1.0), "darcy": (0.0, 1.0)}, + font_size: float = 28.0, + ) -> None: + self.norm = norm + self.criterion = loss_fun + self.font_size = font_size + self.headers = ("invar", "truth", "prediction", "relative error") + + def compare( + self, + invar: FloatTensor, + target: FloatTensor, + prediction: FloatTensor, + step: int, + logger: LaunchLogger, + ) -> float: + """compares model output, target and plots everything + + Parameters + ---------- + invar : FloatTensor + input to model + target : FloatTensor + ground truth + prediction : FloatTensor + model output + step : int + iteration counter + logger : LaunchLogger + logger to which figure is passed + + Returns + ------- + float + validation error + """ + loss = self.criterion(prediction, target) + norm = self.norm + + # pick first sample from batch + invar = invar * norm["permeability"][1] + norm["permeability"][0] + target = target * norm["darcy"][1] + norm["darcy"][0] + prediction = prediction * norm["darcy"][1] + norm["darcy"][0] + invar = invar.cpu().numpy()[0, -1, :, :] + target = target.cpu().numpy()[0, 0, :, :] + prediction = prediction.detach().cpu().numpy()[0, 0, :, :] + + plt.close("all") + plt.rcParams.update({"font.size": self.font_size}) + fig, ax = plt.subplots(1, 4, figsize=(15 * 3.5, 15), sharey=True) + im = [] + im.append(ax[0].imshow(invar)) + im.append(ax[1].imshow(target)) + im.append(ax[2].imshow(prediction)) + im.append(ax[3].imshow((prediction - target) / norm["darcy"][1])) + + for ii in range(len(im)): + fig.colorbar(im[ii], ax=ax[ii], location="bottom", fraction=0.046, pad=0.04) + ax[ii].set_title(self.headers[ii]) + + logger.log_figure(figure=fig, artifact_file=f"validation_step_{step:03d}.png") + + return loss + + +def PlotNestedDarcy(dat: dict, idx: int) -> None: + """Plot fields from the nested Darcy case + + Parameters + ---------- + dat : dict + dictionary containing fields + target : FloatTensor + index of example to plot + """ + fields = dat[str(idx)] + n_insets = len(fields["ref1"]) + + fig, ax = plt.subplots(n_insets + 1, 4, figsize=(20, 5 * (n_insets + 1))) + + vmin = fields["ref0"]["0"]["darcy"].min() + vmax = fields["ref0"]["0"]["darcy"].max() + + ax[0, 0].imshow(fields["ref0"]["0"]["permeability"]) + ax[0, 0].set_title("permeability glob") + ax[0, 1].imshow(fields["ref0"]["0"]["darcy"], vmin=vmin, vmax=vmax) + ax[0, 1].set_title("darcy glob") + ax[0, 2].axis("off") + ax[0, 3].axis("off") + + for ii in range(n_insets): + loc = fields["ref1"][str(ii)] + inset_size = loc["darcy"].shape[1] + ax[ii + 1, 0].imshow(loc["permeability"]) + ax[ii + 1, 0].set_title(f"permeability fine {ii}") + ax[ii + 1, 1].imshow(loc["darcy"], vmin=vmin, vmax=vmax) + ax[ii + 1, 1].set_title(f"darcy fine {ii}") + ax[ii + 1, 2].imshow( + fields["ref0"]["0"]["permeability"][ + loc["pos"][0] : loc["pos"][0] + inset_size, + loc["pos"][1] : loc["pos"][1] + inset_size, + ] + ) + ax[ii + 1, 2].set_title(f"permeability zoomed {ii}") + ax[ii + 1, 3].imshow( + fields["ref0"]["0"]["darcy"][ + loc["pos"][0] : loc["pos"][0] + inset_size, + loc["pos"][1] : loc["pos"][1] + inset_size, + ], + vmin=vmin, + vmax=vmax, + ) + ax[ii + 1, 3].set_title(f"darcy zoomed {ii}") + + fig.tight_layout() + plt.savefig(f"sample_{idx:02d}.png") + plt.close() + + +@wp.kernel +def fourier_to_array_batched_2d_cropped( + array: wp.array3d(dtype=float), + fourier: wp.array4d(dtype=float), + nr_freq: int, + lx: int, + ly: int, + bounds: wp.array3d(dtype=int), + fill_val: int, +): # pragma: no cover + """Array of Fourier amplitudes to batched 2d spatial array + + Parameters + ---------- + array : wp.array3d + Spatial array + fourier : wp.array4d + Array of Fourier amplitudes + nr_freq : int + Number of frequencies in Fourier array + lx : int + Grid size x + ly : int + Grid size y + x_start : int + lowest x-index + y_start : int + lowest y-index + """ + b, p, x, y = wp.tid() + + if bounds[b, p, 0] == fill_val: + return + + x += bounds[b, p, 0] + y += bounds[b, p, 1] + + array[b, x, y] = 0.0 + dx = 6.28318 / wp.float32(lx) + dy = 6.28318 / wp.float32(ly) + rx = dx * wp.float32(x) + ry = dy * wp.float32(y) + for i in range(nr_freq): + for j in range(nr_freq): + ri = wp.float32(i) + rj = wp.float32(j) + ss = fourier[0, b, i, j] * wp.sin(ri * rx) * wp.sin(rj * ry) + cs = fourier[1, b, i, j] * wp.cos(ri * rx) * wp.sin(rj * ry) + sc = fourier[2, b, i, j] * wp.sin(ri * rx) * wp.cos(rj * ry) + cc = fourier[3, b, i, j] * wp.cos(ri * rx) * wp.cos(rj * ry) + wp.atomic_add( + array, b, x, y, 1.0 / (wp.float32(nr_freq) ** 2.0) * (ss + cs + sc + cc) + ) + + +class DarcyInset2D(Darcy2D): + """2D Darcy flow benchmark problem datapipe. + + This datapipe continuously generates solutions to the 2D Darcy equation with variable + permeability. All samples are generated on the fly and is meant to be a benchmark + problem for testing data driven models. Permeability is drawn from a random Fourier + series and threshold it to give a piecewise constant function. The solution is obtained + using a GPU enabled multi-grid Jacobi iterative method. + + Parameters + ---------- + resolution : int, optional + Resolution to run simulation at, by default 256 + batch_size : int, optional + Batch size of simulations, by default 64 + nr_permeability_freq : int, optional + Number of frequencies to use for generating random permeability. Higher values + will give higher freq permeability fields., by default 5 + max_permeability : float, optional + Max permeability, by default 2.0 + min_permeability : float, optional + Min permeability, by default 0.5 + max_iterations : int, optional + Maximum iterations to use for each multi-grid, by default 30000 + convergence_threshold : float, optional + Solver L-Infinity convergence threshold, by default 1e-6 + iterations_per_convergence_check : int, optional + Number of Jacobi iterations to run before checking convergence, by default 1000 + nr_multigrids : int, optional + Number of multi-grid levels, by default 4 + normaliser : Union[Dict[str, Tuple[float, float]], None], optional + Dictionary with keys `permeability` and `darcy`. The values for these keys are two floats corresponding to mean and std `(mean, std)`. + device : Union[str, torch.device], optional + Device for datapipe to run place data on, by default "cuda" + + Raises + ------ + ValueError + Incompatable multi-grid and resolution settings + """ + + def __init__( + self, + resolution: int = 256, + batch_size: int = 64, + nr_permeability_freq: int = 5, + max_permeability: float = 2.0, + min_permeability: float = 0.5, + max_iterations: int = 30000, + convergence_threshold: float = 1e-6, + iterations_per_convergence_check: int = 1000, + nr_multigrids: int = 4, + normaliser: Union[Dict[str, Tuple[float, float]], None] = None, + device: Union[str, torch.device] = "cuda", + max_n_insets: int = 3, + fine_res: int = 32, + fine_permeability_freq: int = 10, + min_offset: int = 48, + ref_fac: int = None, + min_dist_frac: float = 1.7, + fill_val: int = -99999, + ): + super().__init__( + resolution, + batch_size, + nr_permeability_freq, + max_permeability, + min_permeability, + max_iterations, + convergence_threshold, + iterations_per_convergence_check, + nr_multigrids, + normaliser, + device, + ) + + self.max_n_insets = max_n_insets + self.fine_res = fine_res + self.fine_freq = fine_permeability_freq + self.ref_fac = ref_fac + assert resolution % self.ref_fac == 0, ( + "simulation res must be multiple of ref_fac" + ) + + # force inset on coarse grid + if not min_offset % self.ref_fac == 0: + min_offset += self.ref_fac - min_offset % self.ref_fac + self.beg_min = min_offset + self.beg_max = resolution - min_offset - fine_res - self.ref_fac + self.bounds = None + self.min_dist_frac = min_dist_frac + self.fill_val = fill_val + + assert self.max_n_insets <= 3, ( + f"at most 3 insets supported, change max_n_insets accordingly" + ) + assert (self.beg_max - self.beg_min) % ref_fac == 0, "lsdhfgn3x!!!!" + + def initialize_batch(self) -> None: + """Initializes arrays for new batch of simulations""" + + # initialize permeability + self.permeability.zero_() + seed = np.random.randint(np.iinfo(np.uint64).max, dtype=np.uint64) + wp.launch( + kernel=init_uniform_random_4d, + dim=self.fourier_dim, + inputs=[self.rand_fourier, -1.0, 1.0, seed], + device=self.device, + ) + wp.launch( + kernel=fourier_to_array_batched_2d, + dim=self.dim, + inputs=[ + self.permeability, + self.rand_fourier, + self.nr_permeability_freq, + self.resolution, + self.resolution, + ], + device=self.device, + ) + + rr = np.random.randint( + low=0, + high=(self.beg_max - self.beg_min) // self.ref_fac, + size=(self.batch_size, self.max_n_insets, 2), + ) + n_insets = np.random.randint( + low=1, + high=rr.shape[1] + 1, + size=(self.batch_size,), + ) + + # check that regions do not overlap and have distance + min_dist = self.min_dist_frac * self.fine_res // self.ref_fac + 1 + print("adjusting inset positions") + for ib in range(self.batch_size): + if n_insets[ib] <= 1: + rr[ib, 1:, :] = self.fill_val + continue + else: + while ( + abs(rr[ib, 0, 0] - rr[ib, 1, 0]) < min_dist + and abs(rr[ib, 0, 1] - rr[ib, 1, 1]) < min_dist + ): + rr[ib, 0, :] = np.random.randint( + low=0, + high=(self.beg_max - self.beg_min) // self.ref_fac, + size=(2,), + ) + rr[ib, 1, :] = np.random.randint( + low=0, + high=(self.beg_max - self.beg_min) // self.ref_fac, + size=(2,), + ) + if n_insets[ib] <= 2: + rr[ib, 2:, :] = self.fill_val + continue + else: + while ( + abs(rr[ib, 0, 0] - rr[ib, 2, 0]) < min_dist + and abs(rr[ib, 0, 1] - rr[ib, 2, 1]) < min_dist + ) or ( + abs(rr[ib, 1, 0] - rr[ib, 2, 0]) < min_dist + and abs(rr[ib, 1, 1] - rr[ib, 2, 1]) < min_dist + ): + rr[ib, 2, :] = np.random.randint( + low=0, + high=(self.beg_max - self.beg_min) // self.ref_fac, + size=(2,), + ) + print("done") + + rr = np.where(rr != self.fill_val, (rr * self.ref_fac) + self.beg_min, rr) + self.bounds = wp.array(rr, dtype=int, device=self.device) + + wp.launch( + kernel=fourier_to_array_batched_2d_cropped, + dim=(self.batch_size, self.bounds.shape[1], self.fine_res, self.fine_res), + inputs=[ + self.permeability, + self.rand_fourier, + self.fine_freq, + self.fine_res, + self.fine_res, + self.bounds, + self.fill_val, + ], + device=self.device, + ) + + wp.launch( + kernel=threshold_3d, + dim=self.dim, + inputs=[ + self.permeability, + 0.0, + self.min_permeability, + self.max_permeability, + ], + device=self.device, + ) + + # zero darcy arrays + self.darcy0.zero_() + self.darcy1.zero_() + + def batch_generator(self) -> Tuple[Tensor, Tensor]: + # run simulation + self.generate_batch() + + # convert warp arrays to pytorch + permeability = wp.to_torch(self.permeability) + darcy = wp.to_torch(self.darcy0) + + # add channel dims + permeability = torch.unsqueeze(permeability, axis=1) + darcy = torch.unsqueeze(darcy, axis=1) + + # crop edges by 1 from multi-grid + permeability = permeability[:, :, : self.resolution, : self.resolution] + darcy = darcy[:, :, : self.resolution, : self.resolution] + + # normalize values + if self.normaliser is not None: + permeability = ( + permeability - self.normaliser["permeability"][0] + ) / self.normaliser["permeability"][1] + darcy = (darcy - self.normaliser["darcy"][0]) / self.normaliser["darcy"][1] + + # CUDA graphs static copies + if self.output_k is None: + self.output_k = permeability + self.output_p = darcy + else: + self.output_k.data.copy_(permeability) + self.output_p.data.copy_(darcy) + + return {"permeability": self.output_k, "darcy": self.output_p} + + def __iter__(self) -> Tuple[Tensor, Tensor, Tensor]: + """ + Yields + ------ + Iterator[Tuple[Tensor, Tensor, Tensor]] + Infinite iterator that returns a batch of (permeability, darcy pressure) + fields of size [batch, resolution, resolution] + """ + # infinite generator + while True: + batch = self.batch_generator() + batch["inset_pos"] = wp.to_torch(self.bounds) + yield batch diff --git a/examples/cfd/darcy_physics_informed/README.md b/examples/cfd/darcy_physics_informed/README.md index 60a1c36b1c..9a44d2c568 100644 --- a/examples/cfd/darcy_physics_informed/README.md +++ b/examples/cfd/darcy_physics_informed/README.md @@ -1,4 +1,4 @@ -# Physics Informed DeepONet for Darcy flow +# Physics Guided models for Darcy flow This example demonstrates physics informing of a data-driven model using to approaces - the DeepONet approach which computes the gradients using Autograd and an approach using @@ -6,17 +6,17 @@ Numerical derivatives (PINO). ## Problem overview -This is an extension of the 2D darcy flow data-driven problem. In addition to the +This is an extension of the 2D Darcy flow data-driven problem. In addition to the data loss, we will demonstrate the use of physics constranints, specifically -the equation residual loss. [Modulus Sym](https://github.com/NVIDIA/modulus-sym) +the equation residual loss. [PhysicsNeMo Sym](https://github.com/NVIDIA/physicsnemo-sym) has utilities tailored for physics-informed machine learning. It also presents an abstracted APIs that allows users to think and model the problem from the lens of equations, constraints, etc. In this example, we will only levarage the physics-informed -utilites to see how we can add physics to an existing data-driven model with ease while +utilities to see how we can add physics to an existing data-driven model with ease while still maintaining the flexibility to define our own training loop and other details. For a more abstracted definition of these type of problems, where the training loop definition and other things is taken care of implictily, you may refer -[Modulus Sym](https://github.com/NVIDIA/modulus-sym) +[PhysicsNeMo Sym](https://github.com/NVIDIA/physicsnemo-sym) ## Dataset @@ -31,7 +31,7 @@ python download_data.py Do demonstrate the usefulness of the Physics loss, we will deliberately choose a smaller dataset size of 100 samples. In such regiemes, the effect of physics loss is more -evident, as it regularizes the model in the absense of large data. +evident, as it regularizes the model in the absence of large data. ## Model overview and architecture @@ -42,7 +42,7 @@ field and the input to the trunk network is the x, y coordinates. The output of the model is the pressure field. Having the mapping between the pressure field and the input x and y through a fully-differentiable network will allow us to compute the gradients of the pressure field w.r.t input x and y through automatic differentiation -through Modulus sym utils. +through PhysicsNeMo sym utils. In the second case, we will use just FNO and then compute the derivatives in a PINO style, using Numerical differentiation. Both approaches are viable ways to introduce physics in @@ -50,30 +50,37 @@ the loss function and the use of one over the other can change from case-to-case With this example, we intend to demonstrate both such cases so that the users can compare and contrast the two approaches. -In this example we will also use the `PDE` class from Modulus-Sym to symbolically define -the PDEs. This is very convinient and most natural way to define these PDEs and allows -us to print the equations to check for correctness. This also abstracts out the -complexity of converting the equation into a pytorch representation. Modulus Sym also +In this example we will use the `PDE` class from PhysicsNeMo-Sym to symbolically define +the PDEs and use the `PhysicsInformer` utility to introduce the PDE +constraints. Defining the PDEs sympolically is very convenient and most natural way to +define these PDEs and allows us to print the equations to check for correctness. +This also abstracts out the +complexity of converting the equation into a pytorch representation. PhysicsNeMo Sym also provides several complex, well tested PDEs like 3D Navier-Stokes, Linear elasticity, Electromagnetics, etc. pre-defined which can be used directly in physics-informing -applications. +applications. ## Getting Started To get started with the example, simply run, ```bash -python +python darcy_physics_informed_deeponet.py ``` or ```bash -python +python darcy_physics_informed_fno.py ``` +### Note + +If you are running this example outside of the PhysicsNeMo container, install +PhysicsNeMo Sym using the instructions from [here](https://github.com/NVIDIA/physicsnemo-sym?tab=readme-ov-file#pypi) + ## References - [Fourier Neural Operator for Parametric Partial Differential Equations](https://arxiv.org/abs/2010.08895) diff --git a/examples/cfd/darcy_physics_informed/conf/config.yaml b/examples/cfd/darcy_physics_informed/conf/config.yaml deleted file mode 100644 index 7f5a93d8f6..0000000000 --- a/examples/cfd/darcy_physics_informed/conf/config.yaml +++ /dev/null @@ -1,42 +0,0 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -hydra: - job: - chdir: True - run: - dir: ./outputs_phy_deepo - -start_lr: 0.001 -gamma: 0.99948708 -max_epochs: 50 - -phy_wt: 0.1 - -model: - fno: - in_channels: 1 - out_channels: 1 - decoder_layers: 1 - decoder_layer_size: 32 - dimension: 2 - latent_channels: 32 - num_fno_layers: 4 - num_fno_modes: 12 - padding: 9 - fc: - in_features: 2 - out_features: 1 - layer_size: 128 - num_layers: 3 \ No newline at end of file diff --git a/examples/cfd/darcy_physics_informed/conf/config_deeponet.yaml b/examples/cfd/darcy_physics_informed/conf/config_deeponet.yaml new file mode 100644 index 0000000000..ac0ade11fb --- /dev/null +++ b/examples/cfd/darcy_physics_informed/conf/config_deeponet.yaml @@ -0,0 +1,44 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +hydra: + job: + chdir: True + run: + dir: ./outputs_phy_deepo + +start_lr: 0.001 +gamma: 0.99948708 +max_epochs: 50 + +physics_weight: 0.1 + +model: + fno: + in_channels: 1 # k-prime + out_channels: 2 # u_branch, k_branch + decoder_layers: 1 + decoder_layer_size: 32 + dimension: 2 + latent_channels: 32 + num_fno_layers: 4 + num_fno_modes: 12 + padding: 9 + fc: + in_features: 2 # x, y + out_features: 2 # u_trunk, k_trunk + layer_size: 128 + num_layers: 3 \ No newline at end of file diff --git a/examples/cfd/darcy_physics_informed/conf/config_pino.yaml b/examples/cfd/darcy_physics_informed/conf/config_pino.yaml index cba7051d21..76495d527c 100644 --- a/examples/cfd/darcy_physics_informed/conf/config_pino.yaml +++ b/examples/cfd/darcy_physics_informed/conf/config_pino.yaml @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -22,7 +24,7 @@ start_lr: 0.001 gamma: 0.99948708 max_epochs: 50 -phy_wt: 0.1 +physics_weight: 0.1 model: fno: diff --git a/examples/cfd/darcy_physics_informed/darcy_pde.py b/examples/cfd/darcy_physics_informed/darcy_pde.py deleted file mode 100644 index c3fba8f3da..0000000000 --- a/examples/cfd/darcy_physics_informed/darcy_pde.py +++ /dev/null @@ -1,45 +0,0 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -from modulus.sym.eq.pde import PDE -from sympy import Symbol, Function - - -class Darcy(PDE): - """Darcy PDE using Modulus Sym""" - - name = "Darcy" - - def __init__(self): - - # time - x, y = Symbol("x"), Symbol("y") - - # make input variables - input_variables = {"x": x, "y": y} - - # make sol function - u = Function("sol")(*input_variables) - k = Function("K")(*input_variables) - f = 1.0 - - # set equation - self.equations = {} - self.equations["darcy"] = ( - f - + k.diff(x) * u.diff(x) - + k * u.diff(x).diff(x) - + k.diff(y) * u.diff(y) - + k * u.diff(y).diff(y) - ) diff --git a/examples/cfd/darcy_physics_informed/darcy_physics_informed_deeponet.py b/examples/cfd/darcy_physics_informed/darcy_physics_informed_deeponet.py index 8348bbaaae..a528a32174 100644 --- a/examples/cfd/darcy_physics_informed/darcy_physics_informed_deeponet.py +++ b/examples/cfd/darcy_physics_informed/darcy_physics_informed_deeponet.py @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -12,61 +14,59 @@ # See the License for the specific language governing permissions and # limitations under the License. +from itertools import chain +from typing import Dict + import hydra -from omegaconf import DictConfig -import torch -import numpy as np import matplotlib.pyplot as plt -from hydra.utils import to_absolute_path +import numpy as np +import torch import torch.nn.functional as F +from hydra.utils import to_absolute_path +from physicsnemo.utils.logging import LaunchLogger +from physicsnemo.utils.checkpoint import save_checkpoint +from physicsnemo.models.fno import FNO +from physicsnemo.models.mlp import FullyConnected +from physicsnemo.sym.eq.pdes.diffusion import Diffusion +from physicsnemo.sym.eq.phy_informer import PhysicsInformer +from physicsnemo.sym.key import Key +from physicsnemo.sym.models.arch import Arch +from omegaconf import DictConfig from torch.utils.data import DataLoader -from itertools import chain - -from modulus.models.mlp import FullyConnected -from modulus.models.fno import FNO -from modulus.launch.logging import LaunchLogger -from modulus.launch.utils.checkpoint import save_checkpoint from utils import HDF5MapStyleDataset -from darcy_pde import Darcy -def validation_step(model_branch, model_trunk, dataloader, epoch): +def validation_step(graph, dataloader, epoch): """Validation Step""" - model_branch.eval() - model_trunk.eval() with torch.no_grad(): loss_epoch = 0 for data in dataloader: invar, outvar, x_invar, y_invar = data - coords = torch.cat( - (x_invar.squeeze(dim=2), y_invar.squeeze(dim=2)), dim=0 - ).reshape(-1, 2) - - branch_out = model_branch(invar[:, 0].unsqueeze(dim=1)) - trunk_out = model_trunk(coords) - branch_out = branch_out.reshape(-1, 240 * 240) - trunk_out = trunk_out.reshape(-1, 240 * 240) - deepo_out = trunk_out * branch_out - deepo_out = deepo_out.reshape(-1, 1, 240, 240) - loss_epoch += F.mse_loss(outvar, deepo_out) + out = graph.forward( + {"k_prime": invar[:, 0].unsqueeze(dim=1), "x": x_invar, "y": y_invar} + ) + + deepo_out_u = out["u"] + + loss_epoch += F.mse_loss(outvar, deepo_out_u) # convert data to numpy outvar = outvar.detach().cpu().numpy() - predvar = deepo_out.detach().cpu().numpy() + predvar = deepo_out_u.detach().cpu().numpy() # plotting fig, ax = plt.subplots(1, 3, figsize=(25, 5)) - d_min = np.min(outvar[0, 0, ...]) - d_max = np.max(outvar[0, 0, ...]) + d_min = np.min(outvar[0, 0]) + d_max = np.max(outvar[0, 0]) - im = ax[0].imshow(outvar[0, 0, ...], vmin=d_min, vmax=d_max) + im = ax[0].imshow(outvar[0, 0], vmin=d_min, vmax=d_max) plt.colorbar(im, ax=ax[0]) - im = ax[1].imshow(predvar[0, 0, ...], vmin=d_min, vmax=d_max) + im = ax[1].imshow(predvar[0, 0], vmin=d_min, vmax=d_max) plt.colorbar(im, ax=ax[1]) - im = ax[2].imshow(np.abs(predvar[0, 0, ...] - outvar[0, 0, ...])) + im = ax[2].imshow(np.abs(predvar[0, 0] - outvar[0, 0])) plt.colorbar(im, ax=ax[2]) ax[0].set_title("True") @@ -78,22 +78,98 @@ def validation_step(model_branch, model_trunk, dataloader, epoch): return loss_epoch / len(dataloader) -@hydra.main(version_base="1.3", config_path="conf", config_name="config.yaml") +class MdlsSymWrapper(Arch): + """ + Wrapper model to convert PhysicsNeMo model to PhysicsNeMo-Sym model. + + PhysicsNeMo Sym relies on the inputs/outputs of the model being dictionary of tensors. + This wrapper converts the input dictionary of tensors to a tensor inputs that can + be processed by the PhysicsNeMo model that operate on tensors. Appropriate + transformations are performed in the forward pass of the model to translate between + these two input/output definitions. + + These transformations can differ based on the models. For e.g. typically for a fully + connected network, the input tensors are combined by concatenating them along + appropriate dimension before passing them as an input to the PhysicsNeMo model. + During the output, the process is reversed, the output tensor from pytorch model is + split across appropriate dimensions and then converted to a dictionary with + appropriate keys to produce the final output. + + Having the model wrapped in a wrapper like this allows gradient computation using + the PhysicsNeMo Sym's optimized gradient computing backend. + + For more details on PhysicsNeMo Sym models, refer: + https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/tutorials/simple_training_example.html#using-custom-models-in-physicsnemo + For more details on Key class, refer: + https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-sym/api/physicsnemo.sym.html#module-physicsnemo.sym.key + """ + + def __init__( + self, + input_keys=[Key("k"), Key("x"), Key("y")], + output_keys=[Key("k_prime"), Key("u")], + trunk_net=None, + branch_net=None, + ): + super().__init__( + input_keys=input_keys, + output_keys=output_keys, + ) + + self.branch_net = branch_net + self.trunk_net = trunk_net + + def forward(self, dict_tensor: Dict[str, torch.Tensor]): + # Concatenate x, y inputs to feeed in the trunk network which has a MLP + xy_input_shape = dict_tensor["x"].shape + xy = self.concat_input( + { + k: dict_tensor[k].view(xy_input_shape[0], -1, 1) for k in ["x", "y"] + }, # flatten the coordinate dimensions + ["x", "y"], + detach_dict=self.detach_key_dict, + dim=-1, # concat along the last dimension to form the feature vector. + ) + fc_out = self.trunk_net(xy) + + # Pass the k-prime for the FNO input + fno_out = self.branch_net(dict_tensor["k_prime"]) + + # reshape the fc_out + fc_out = fc_out.view( + xy_input_shape[0], -1, xy_input_shape[-2], xy_input_shape[-1] + ) + + # multiply the outputs of branch and trunk networks to get the final output + out = fc_out * fno_out + + return self.split_output( + out, self.output_key_dict, dim=1 + ) # Split along the channel dimension to get a dictionary of tensors + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config_deeponet.yaml") def main(cfg: DictConfig): + # CUDA support + if torch.cuda.is_available(): + device = torch.device("cuda") + else: + device = torch.device("cpu") LaunchLogger.initialize() - darcy = Darcy() - darcy_node = darcy.make_nodes() + # Use Diffusion equation for the Darcy PDE + forcing_fn = 1.0 * 4.49996e00 * 3.88433e-03 # after scaling + darcy = Diffusion(T="u", time=False, dim=2, D="k", Q=forcing_fn) dataset = HDF5MapStyleDataset( - to_absolute_path("./datasets/Darcy_241/train.hdf5"), device="cuda" + to_absolute_path("./datasets/Darcy_241/train.hdf5"), device=device ) validation_dataset = HDF5MapStyleDataset( - to_absolute_path("./datasets/Darcy_241/validation.hdf5"), device="cuda" + to_absolute_path("./datasets/Darcy_241/validation.hdf5"), device=device ) - dataloader = DataLoader(dataset, batch_size=1, shuffle=True) + dataloader = DataLoader(dataset, batch_size=2, shuffle=True) validation_dataloader = DataLoader(validation_dataset, batch_size=1, shuffle=False) @@ -107,20 +183,38 @@ def main(cfg: DictConfig): num_fno_layers=cfg.model.fno.num_fno_layers, num_fno_modes=cfg.model.fno.num_fno_modes, padding=cfg.model.fno.padding, - ).to("cuda") + ) model_trunk = FullyConnected( in_features=cfg.model.fc.in_features, out_features=cfg.model.fc.out_features, layer_size=cfg.model.fc.layer_size, num_layers=cfg.model.fc.num_layers, - ).to("cuda") + ) + + # Define k-prime as an auxiliary variable that is a copy of k. + # Having k as the output of the model will allow gradients of k (for pde loss) + # to be computed using Sym's gradient backend + model = MdlsSymWrapper( + input_keys=[Key("k_prime"), Key("x"), Key("y")], + output_keys=[Key("k"), Key("u")], + trunk_net=model_trunk, + branch_net=model_branch, + ).to(device) + + phy_informer = PhysicsInformer( + required_outputs=["diffusion_u"], + equations=darcy, + grad_method="autodiff", + device=device, + ) optimizer = torch.optim.Adam( chain(model_branch.parameters(), model_trunk.parameters()), betas=(0.9, 0.999), lr=cfg.start_lr, weight_decay=0.0, + fused=True if torch.cuda.is_available() else False, ) scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, gamma=cfg.gamma) @@ -135,73 +229,42 @@ def main(cfg: DictConfig): ) as log: for data in dataloader: optimizer.zero_grad() - invar = data[0] outvar = data[1] - x_invar = data[2].squeeze(dim=2).reshape(-1, 1).requires_grad_(True) - y_invar = data[3].squeeze(dim=2).reshape(-1, 1).requires_grad_(True) - coords = torch.cat((x_invar, y_invar), dim=1) + coords = torch.stack([data[2], data[3]], dim=1).requires_grad_(True) # compute forward pass - branch_out = model_branch(invar[:, 0].unsqueeze(dim=1)) - trunk_out = model_trunk(coords) - branch_out = branch_out.reshape(-1, 1) - trunk_out = trunk_out.reshape(-1, 1) - deepo_out = trunk_out * branch_out - - # Compute physics loss - # note: the derivative computation can be done using Modulus-Sym - # utilities. However, for the purposes of this example, we show it using - # torch.autograd. - grad_sol = torch.autograd.grad( - deepo_out.sum(), - [x_invar, y_invar], - create_graph=True, # grad_outputs=torch.ones_like(deepo_out) - ) - sol_x = grad_sol[0] - sol_y = grad_sol[1] - - sol_x_x = torch.autograd.grad( - sol_x.sum(), - [x_invar], - create_graph=True, # grad_outputs=torch.ones_like(sol_x) - )[0] - sol_y_y = torch.autograd.grad( - sol_y.sum(), - [y_invar], - create_graph=True, # grad_outputs=torch.ones_like(sol_y) - )[0] - - k, k_x, k_y = ( - invar[:, 0].reshape(-1, 1), - invar[:, 1].reshape(-1, 1), - invar[:, 2].reshape(-1, 1), + out = model.forward( + { + "k_prime": data[0][:, 0].unsqueeze(dim=1), + "x": coords[:, 0:1], + "y": coords[:, 1:2], + } ) - pde_out = darcy_node[0].evaluate( + residuals = phy_informer.forward( { - "sol__x": sol_x, - "sol__y": sol_y, - "sol__x__x": sol_x_x, - "sol__y__y": sol_y_y, - "K": k, - "K__x": k_x, - "K__y": k_y, + "coordinates": coords, + "u": out["u"], + "k": out["k"], } ) + pde_out_arr = residuals["diffusion_u"] - pde_out_arr = pde_out["darcy"] - pde_out_arr = pde_out_arr.reshape(-1, 240, 240) + # Boundary condition pde_out_arr = F.pad( - pde_out_arr[:, 2:-2, 2:-2], [2, 2, 2, 2], "constant", 0 + pde_out_arr[..., 2:-2, 2:-2], [2, 2, 2, 2], "constant", 0 ) loss_pde = F.l1_loss(pde_out_arr, torch.zeros_like(pde_out_arr)) # Compute data loss - deepo_out = deepo_out.reshape(-1, 1, 240, 240) - loss_data = F.mse_loss(outvar, deepo_out) + deepo_out_u = out["u"] + deepo_out_k = out["k"] + loss_data = F.mse_loss(outvar, deepo_out_u) + F.mse_loss( + data[0][:, 0].unsqueeze(dim=1), deepo_out_k + ) # Compute total loss - loss = loss_data + cfg.phy_wt * loss_pde + loss = loss_data + cfg.physics_weight * loss_pde # Backward pass and optimizer and learning rate update loss.backward() @@ -214,9 +277,7 @@ def main(cfg: DictConfig): log.log_epoch({"Learning Rate": optimizer.param_groups[0]["lr"]}) with LaunchLogger("valid", epoch=epoch) as log: - error = validation_step( - model_branch, model_trunk, validation_dataloader, epoch - ) + error = validation_step(model, validation_dataloader, epoch) log.log_epoch({"Validation error": error}) save_checkpoint( diff --git a/examples/cfd/darcy_physics_informed/darcy_physics_informed_fno.py b/examples/cfd/darcy_physics_informed/darcy_physics_informed_fno.py index 645814a6bd..2035787452 100644 --- a/examples/cfd/darcy_physics_informed/darcy_physics_informed_fno.py +++ b/examples/cfd/darcy_physics_informed/darcy_physics_informed_fno.py @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,23 +15,20 @@ # limitations under the License. import hydra -from omegaconf import DictConfig -import torch -import numpy as np - import matplotlib.pyplot as plt -from hydra.utils import to_absolute_path +import numpy as np +import torch import torch.nn.functional as F +from hydra.utils import to_absolute_path +from physicsnemo.utils.logging import LaunchLogger +from physicsnemo.utils.checkpoint import save_checkpoint +from physicsnemo.models.fno import FNO +from physicsnemo.sym.eq.pdes.diffusion import Diffusion +from physicsnemo.sym.eq.phy_informer import PhysicsInformer +from omegaconf import DictConfig from torch.utils.data import DataLoader -import torch.nn.functional as F - -from modulus.models.fno import FNO -from modulus.launch.logging import LaunchLogger -from modulus.launch.utils.checkpoint import save_checkpoint from utils import HDF5MapStyleDataset -from ops import dx, ddx -from darcy_pde import Darcy def validation_step(model, dataloader, epoch): @@ -51,14 +50,14 @@ def validation_step(model, dataloader, epoch): # plotting fig, ax = plt.subplots(1, 3, figsize=(25, 5)) - d_min = np.min(outvar[0, 0, ...]) - d_max = np.max(outvar[0, 0, ...]) + d_min = np.min(outvar[0, 0]) + d_max = np.max(outvar[0, 0]) - im = ax[0].imshow(outvar[0, 0, ...], vmin=d_min, vmax=d_max) + im = ax[0].imshow(outvar[0, 0], vmin=d_min, vmax=d_max) plt.colorbar(im, ax=ax[0]) - im = ax[1].imshow(predvar[0, 0, ...], vmin=d_min, vmax=d_max) + im = ax[1].imshow(predvar[0, 0], vmin=d_min, vmax=d_max) plt.colorbar(im, ax=ax[1]) - im = ax[2].imshow(np.abs(predvar[0, 0, ...] - outvar[0, 0, ...])) + im = ax[2].imshow(np.abs(predvar[0, 0] - outvar[0, 0])) plt.colorbar(im, ax=ax[2]) ax[0].set_title("True") @@ -72,17 +71,23 @@ def validation_step(model, dataloader, epoch): @hydra.main(version_base="1.3", config_path="conf", config_name="config_pino.yaml") def main(cfg: DictConfig): + # CUDA support + if torch.cuda.is_available(): + device = torch.device("cuda") + else: + device = torch.device("cpu") LaunchLogger.initialize() - darcy = Darcy() - darcy_node = darcy.make_nodes() + # Use Diffusion equation for the Darcy PDE + forcing_fn = 1.0 * 4.49996e00 * 3.88433e-03 # after scaling + darcy = Diffusion(T="u", time=False, dim=2, D="k", Q=forcing_fn) dataset = HDF5MapStyleDataset( - to_absolute_path("./datasets/Darcy_241/train.hdf5"), device="cuda" + to_absolute_path("./datasets/Darcy_241/train.hdf5"), device=device ) validation_dataset = HDF5MapStyleDataset( - to_absolute_path("./datasets/Darcy_241/validation.hdf5"), device="cuda" + to_absolute_path("./datasets/Darcy_241/validation.hdf5"), device=device ) dataloader = DataLoader(dataset, batch_size=1, shuffle=True) @@ -99,7 +104,15 @@ def main(cfg: DictConfig): num_fno_layers=cfg.model.fno.num_fno_layers, num_fno_modes=cfg.model.fno.num_fno_modes, padding=cfg.model.fno.padding, - ).to("cuda") + ).to(device) + + phy_informer = PhysicsInformer( + required_outputs=["diffusion_u"], + equations=darcy, + grad_method="finite_difference", + device=device, + fd_dx=1 / 240, # Unit square with resoultion as 240 + ) optimizer = torch.optim.Adam( model.parameters(), @@ -123,39 +136,18 @@ def main(cfg: DictConfig): invar = data[0] outvar = data[1] - # compute forward pass + # Compute forward pass out = model(invar[:, 0].unsqueeze(dim=1)) - dxf = 1.0 / out.shape[-2] - dyf = 1.0 / out.shape[-1] - - sol_x = dx(out, dx=dxf, channel=0, dim=1, order=1, padding="zeros") - sol_y = dx(out, dx=dyf, channel=0, dim=0, order=1, padding="zeros") - sol_x_x = ddx(out, dx=dxf, channel=0, dim=1, order=1, padding="zeros") - sol_y_y = ddx(out, dx=dyf, channel=0, dim=0, order=1, padding="zeros") - - k_x = dx(invar, dx=dxf, channel=0, dim=1, order=1, padding="zeros") - k_y = dx(invar, dx=dxf, channel=0, dim=0, order=1, padding="zeros") - - k, _, _ = ( - invar[:, 0], - invar[:, 1], - invar[:, 2], - ) - - pde_out = darcy_node[0].evaluate( + # print(out.shape, invar[:,0:1].shape) + residuals = phy_informer.forward( { - "sol__x": sol_x, - "sol__y": sol_y, - "sol__x__x": sol_x_x, - "sol__y__y": sol_y_y, - "K": k, - "K__x": k_x, - "K__y": k_y, + "u": out, + "k": invar[:, 0:1], } ) + pde_out_arr = residuals["diffusion_u"] - pde_out_arr = pde_out["darcy"] pde_out_arr = F.pad( pde_out_arr[:, :, 2:-2, 2:-2], [2, 2, 2, 2], "constant", 0 ) @@ -165,7 +157,7 @@ def main(cfg: DictConfig): loss_data = F.mse_loss(outvar, out) # Compute total loss - loss = loss_data + 1 / 240 * cfg.phy_wt * loss_pde + loss = loss_data + 1 / 240 * cfg.physics_weight * loss_pde # Backward pass and optimizer and learning rate update loss.backward() diff --git a/examples/cfd/darcy_physics_informed/download_data.py b/examples/cfd/darcy_physics_informed/download_data.py index 3b340f7f88..3ce3c4f0be 100644 --- a/examples/cfd/darcy_physics_informed/download_data.py +++ b/examples/cfd/darcy_physics_informed/download_data.py @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,6 +16,7 @@ import h5py import numpy as np + from utils import download_FNO_dataset download_FNO_dataset("Darcy_241", outdir="datasets/") @@ -28,9 +31,10 @@ # split_percentage = [80, 20] split_percentage = [10, 10] -with h5py.File(output_filenames[0], "w") as f_part1, h5py.File( - output_filenames[1], "w" -) as f_part2: +with ( + h5py.File(output_filenames[0], "w") as f_part1, + h5py.File(output_filenames[1], "w") as f_part2, +): with h5py.File(filename, "r") as f: # Loop through all the datasets in the input file for key in f.keys(): diff --git a/examples/cfd/darcy_physics_informed/ops.py b/examples/cfd/darcy_physics_informed/ops.py deleted file mode 100644 index f2278e41e6..0000000000 --- a/examples/cfd/darcy_physics_informed/ops.py +++ /dev/null @@ -1,101 +0,0 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -import torch -import torch.nn.functional as F - - -def dx(inpt, dx, channel, dim, order=1, padding="zeros"): - "Compute first order numerical derivatives of input tensor" - - var = inpt[:, channel : channel + 1, :, :] - - # get filter - if order == 1: - ddx1D = torch.Tensor( - [ - -0.5, - 0.0, - 0.5, - ] - ).to(inpt.device) - elif order == 3: - ddx1D = torch.Tensor( - [ - -1.0 / 60.0, - 3.0 / 20.0, - -3.0 / 4.0, - 0.0, - 3.0 / 4.0, - -3.0 / 20.0, - 1.0 / 60.0, - ] - ).to(inpt.device) - ddx3D = torch.reshape(ddx1D, shape=[1, 1] + dim * [1] + [-1] + (1 - dim) * [1]) - # apply convolution - if padding == "zeros": - var = F.pad(var, 4 * [(ddx1D.shape[0] - 1) // 2], "constant", 0) - elif padding == "replication": - var = F.pad(var, 4 * [(ddx1D.shape[0] - 1) // 2], "replicate") - output = F.conv2d(var, ddx3D, padding="valid") - output = (1.0 / dx) * output - if dim == 0: - output = output[:, :, :, (ddx1D.shape[0] - 1) // 2 : -(ddx1D.shape[0] - 1) // 2] - elif dim == 1: - output = output[:, :, (ddx1D.shape[0] - 1) // 2 : -(ddx1D.shape[0] - 1) // 2, :] - - return output - - -def ddx(inpt, dx, channel, dim, order=1, padding="zeros"): - "Compute second order numerical derivatives of input tensor" - - var = inpt[:, channel : channel + 1, :, :] - - # get filter - if order == 1: - ddx1D = torch.Tensor( - [ - 1.0, - -2.0, - 1.0, - ] - ).to(inpt.device) - elif order == 3: - ddx1D = torch.Tensor( - [ - 1.0 / 90.0, - -3.0 / 20.0, - 3.0 / 2.0, - -49.0 / 18.0, - 3.0 / 2.0, - -3.0 / 20.0, - 1.0 / 90.0, - ] - ).to(inpt.device) - ddx3D = torch.reshape(ddx1D, shape=[1, 1] + dim * [1] + [-1] + (1 - dim) * [1]) - - # apply convolution - if padding == "zeros": - var = F.pad(var, 4 * [(ddx1D.shape[0] - 1) // 2], "constant", 0) - elif padding == "replication": - var = F.pad(var, 4 * [(ddx1D.shape[0] - 1) // 2], "replicate") - output = F.conv2d(var, ddx3D, padding="valid") - output = (1.0 / dx**2) * output - if dim == 0: - output = output[:, :, :, (ddx1D.shape[0] - 1) // 2 : -(ddx1D.shape[0] - 1) // 2] - elif dim == 1: - output = output[:, :, (ddx1D.shape[0] - 1) // 2 : -(ddx1D.shape[0] - 1) // 2, :] - - return output diff --git a/examples/cfd/darcy_physics_informed/utils.py b/examples/cfd/darcy_physics_informed/utils.py index e837fe8f77..6af2b91b71 100644 --- a/examples/cfd/darcy_physics_informed/utils.py +++ b/examples/cfd/darcy_physics_informed/utils.py @@ -1,4 +1,6 @@ -# Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -20,13 +22,14 @@ except: gdown = None -import scipy.io -import numpy as np -import h5py from typing import Union + +import h5py +import numpy as np +import scipy.io import torch +from physicsnemo.sym.hydra import to_absolute_path from torch.utils.data import Dataset -from modulus.sym.hydra import to_absolute_path # list of FNO dataset url ids on drive: https://drive.google.com/drive/folders/1UnbQh2WWc6knEHbLn-ZaXrKUZhp7pjt- _FNO_datatsets_ids = { @@ -218,23 +221,23 @@ def __getitem__(self, idx): invar = torch.cat( [ - torch.from_numpy( - (data["Kcoeff"][:, :240, :240] - 7.48360e00) / 4.49996e00 - ), - torch.from_numpy(data["Kcoeff_x"][:, :240, :240]), - torch.from_numpy(data["Kcoeff_y"][:, :240, :240]), + torch.from_numpy((data["Kcoeff"][:, :240, :240]) / 4.49996e00), + torch.from_numpy(data["Kcoeff_x"][:, :240, :240]) / 4.49996e00, + torch.from_numpy(data["Kcoeff_y"][:, :240, :240]) / 4.49996e00, ] ) - outvar = torch.from_numpy( - (data["sol"][:, :240, :240] - 5.74634e-03) / 3.88433e-03 - ) + outvar = torch.from_numpy((data["sol"][:, :240, :240]) / 3.88433e-03) x = np.linspace(0, 1, 240) y = np.linspace(0, 1, 240) xx, yy = np.meshgrid(x, y) - x_invar = torch.from_numpy(xx.astype(np.float32)).reshape(-1, 1) - y_invar = torch.from_numpy(yy.astype(np.float32)).reshape(-1, 1) + x_invar = torch.from_numpy(xx.astype(np.float32)).view( + 1, 240, 240 + ) # add channel dimension + y_invar = torch.from_numpy(yy.astype(np.float32)).view( + 1, 240, 240 + ) # add channel dimension if self.device.type == "cuda": # Move tensors to GPU diff --git a/examples/cfd/darcy_transolver/README.md b/examples/cfd/darcy_transolver/README.md new file mode 100644 index 0000000000..fcfa96da19 --- /dev/null +++ b/examples/cfd/darcy_transolver/README.md @@ -0,0 +1,103 @@ + + +# Transolver for Darcy Flow + +This example demonstrates how to set up a data-driven model for a 2D Darcy flow +using the Transolver inside of PhysicsNeMo. + +

+ +

+ +Training progress can be tracked through +[MLFlow](https://mlflow.org/docs/latest/index.html). This example runs on a +single GPU. + +## Problem overview + +This example is based on a 2D Darcy flow problem, which is often used to model +flow diffusion through a porous medium. The equation describes the steady-state +pressure and velocity field (related by the constitutive relation, based on +Darcy's law) for some fluid of interest. As an applied example, one might +consider a geological example, where the fluid is oil which is both a) generated +by a source term $f(\mathbf{x})$ and b) diffusing through porous bedrock with +permeability $k(\mathbf{x})$. For isotropic media, the governing equation is +given as the following second-order elliptic PDE: + +$$ \nabla \cdot \left(\,k(\mathbf{x})\ \nabla p \left(\mathbf{x}\right)\,\right) += f(\mathbf{x}) $$ + +where $\mathbf{x} \in \Omega \subset \mathbb{R}^2$ is the spatial coordinate, +$p(\mathbf{x})$ is the pressure, $k(\mathbf{x})$ is the local permeability and +$f(\mathbf{x})$ is the source term. + +## Dataset + +For the baseline case defined in `train_transolver_darcy.py`, the dataset is +generated on-the-fly at train time via the `Darcy2D` Datapipe described in +`./physicsnemo/datapipes/benchmarks/darcy.py`. + +For the modified case with a fixed (i.e., precomputed) dataset defined in +`train_transolver_darcy_fix.py`, the dataset is pre-generated, and loaded via +the `Darcy2D_fix` Datapipe. In the fixed case, extra data is needed for training +and the data path should be added when `Darcy_2D_fix` dataset is constructed. +You can download the data +[here](https://huggingface.co/datasets/lkuang/example_data). + +The `fix` dataset training (which uses a fixed dataset) requires you to convert +data from matlab to numpy format, for faster startup of the training. Just +use the `convert_mat_to_npz.py` script to port your data. + +## Model overview and architecture + +## Getting Started + +### Prerequisites + +Install the required dependencies by running below: + +```bash +pip install -r requirements.txt +``` + +### Training + +To train the model following PhysicsNeMo's settings, simply run: + +```bash +python train_transolver_darcy.py +``` + +Each batch is a new data generated by equation, which is different from +commonly-used settings. + +To reproduce the results in the paper, run: + +```bash +python train_transolver_darcy_fix.py +``` + +In this case, the train set and test set are fixed after the construction of +Dataset, corresponding to Transolver's setting. + +### Customization + +To train Transolver on the same 2D Darcy flow problem with different physical +parameters or ML hyperparameters, modify the `config.yaml` file. + +To train Transolver on your own physics problem, modify the `dataloader` in +`train_transolver_darcy.py` to use your own pre-computed data or on-the-fly +solver. + +## Additional Information + +More components are added for convenience. `Validators` calculate the loss +between ground-truth and prediction, and visualize them in `./mlruns`. Below is +a simple example of visualization. + +[![visualization](https://s21.ax1x.com/2024/09/26/pAlis3T.png)](https://imgse.com/i/pAlis3T) + +## References + +- [Transolver: A Fast Transformer Solver for PDEs on General + Geometries](https://arxiv.org/abs/2402.02366) diff --git a/examples/cfd/darcy_transolver/config.yaml b/examples/cfd/darcy_transolver/config.yaml new file mode 100644 index 0000000000..afbe9a02a6 --- /dev/null +++ b/examples/cfd/darcy_transolver/config.yaml @@ -0,0 +1,67 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: MIT License +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +model: + embedding_dim: 2 + n_layers: 3 + n_hidden: 128 + dropout: 0.0 + n_head: 8 + time_input: False + act: gelu + mlp_ratio: 1 + functional_dim: 1 + out_dim: 1 + slice_dim: 32 + ref: 8 + unified_pos: true + slice_num: 32 + use_te: false + + +data: + resolution: 256 + + +normaliser: + permeability: + mean: 1.25 + std_dev: .75 + darcy: + mean: 4.52E-2 + std_dev: 2.79E-2 + +scheduler: + initial_lr: 1.E-3 + decay_rate: .85 + decay_pseudo_epochs: 8 + +training: + resolution: 256 + batch_size: 8 + rec_results_freq : 8 + max_pseudo_epochs: 256 + pseudo_epoch_sample_size: 2048 + +validation: + sample_size: 256 + validation_pseudo_epochs: 4 \ No newline at end of file diff --git a/examples/cfd/darcy_transolver/config_fix.yaml b/examples/cfd/darcy_transolver/config_fix.yaml new file mode 100644 index 0000000000..ee90d84855 --- /dev/null +++ b/examples/cfd/darcy_transolver/config_fix.yaml @@ -0,0 +1,71 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: MIT License +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +output_dir: ./output/darcy_transolver_fix +run_id: bf16_dev_r85_b8_s64 + +data: + train_path: /user_data/datasets/darcy_fix/example_data/piececonst_r421_N1024_smooth1.npz + test_path: /user_data/datasets/darcy_fix/example_data/piececonst_r421_N1024_smooth2.npz + resolution: 85 #421, 211, 141, 106, 85 all viable + batch_size: 8 # This is the GLOBAL batch size + +model: + functional_dim: 1 + out_dim: 1 + embedding_dim: 2 + n_layers: 4 + n_hidden: 128 + dropout: 0.0 + n_head: 4 + act: gelu + mlp_ratio: 4 + unified_pos: False + ref: 8 + slice_num: 64 + use_te: False + time_input: False + +precision: bf16 + +normaliser: + permeability: + mean: 1.25 + std_dev: .75 + darcy: + mean: 4.52E-2 + std_dev: 2.79E-2 + +scheduler: + initial_lr: 1.E-3 + decay_rate: 5.E-5 + weight_decay: 1.E-5 + decay_pseudo_epochs: 8 + +training: + rec_results_freq : 100 + max_pseudo_epochs: 1000 + pseudo_epoch_sample_size: 1024 + +validation: + sample_size: 200 + validation_pseudo_epochs: 1 diff --git a/examples/cfd/darcy_transolver/convert_mat_to_npz.py b/examples/cfd/darcy_transolver/convert_mat_to_npz.py new file mode 100644 index 0000000000..ba8096a85f --- /dev/null +++ b/examples/cfd/darcy_transolver/convert_mat_to_npz.py @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: MIT License +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: + +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. + +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +import sys +import numpy as np +from scipy.io import loadmat + + +def main(mat_file, npz_file): + # Load the .mat file + data = loadmat(mat_file) + + # Extract 'coeff' and 'sol' + coeff = data["coeff"] + sol = data["sol"] + + # Save to .npz file + np.savez(npz_file, coeff=coeff, sol=sol) + print(f"Saved 'coeff' and 'sol' to {npz_file}") + + +if __name__ == "__main__": + if len(sys.argv) != 2: + print("Usage: python convert_mat_to_npz.py input.mat") + sys.exit(1) + mat_file = sys.argv[1] + npz_file = mat_file.replace(".mat", ".npz") + main(mat_file, npz_file) diff --git a/examples/cfd/darcy_transolver/darcy_datapipe_fix.py b/examples/cfd/darcy_transolver/darcy_datapipe_fix.py new file mode 100644 index 0000000000..c043f3f78c --- /dev/null +++ b/examples/cfd/darcy_transolver/darcy_datapipe_fix.py @@ -0,0 +1,274 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from dataclasses import dataclass +from typing import Dict, Tuple, Union + +import numpy as np +import torch +import scipy.io as scio +import torch.distributed as dist + +from physicsnemo.datapipes.datapipe import Datapipe +from physicsnemo.datapipes.meta import DatapipeMetaData + +Tensor = torch.Tensor + +from physicsnemo.utils.profiling import profile + + +class UnitTransformer: + """Unit transformer class for normalizing and denormalizing data.""" + + def __init__(self, X): + self.mean = X.mean(dim=(0, 1), keepdim=True) + self.std = X.std(dim=(0, 1), keepdim=True) + 1e-8 + + def to(self, device): + self.mean = self.mean.to(device) + self.std = self.std.to(device) + return self + + def cuda(self): + self.mean = self.mean.cuda() + self.std = self.std.cuda() + + def cpu(self): + self.mean = self.mean.cpu() + self.std = self.std.cpu() + + def encode(self, x): + x = (x - self.mean) / (self.std) + return x + + def decode(self, x): + return x * self.std + self.mean + + def transform(self, X, inverse=True, component="all"): + if component == "all" or "all-reduce": + if inverse: + orig_shape = X.shape + return (X * (self.std - 1e-8) + self.mean).view(orig_shape) + else: + return (X - self.mean) / self.std + else: + if inverse: + orig_shape = X.shape + return ( + X * (self.std[:, component] - 1e-8) + self.mean[:, component] + ).view(orig_shape) + else: + return (X - self.mean[:, component]) / self.std[:, component] + + +@dataclass +class MetaData(DatapipeMetaData): + name: str = "Darcy2D" + # Optimization + auto_device: bool = True + cuda_graphs: bool = True + # Parallel + ddp_sharding: bool = False + + +class Darcy2D_fix(Datapipe): + """2D Darcy flow benchmark problem datapipe. + + This datapipe continuously generates solutions to the 2D Darcy equation with variable + permeability. All samples are generated on the fly and is meant to be a benchmark + problem for testing data driven models. Permeability is drawn from a random Fourier + series and threshold it to give a piecewise constant function. The solution is obtained + using a GPU enabled multi-grid Jacobi iterative method. + + Parameters + ---------- + resolution : int, optional + Resolution to run simulation at, by default 256 + batch_size : int, optional + Batch size of simulations, by default 64 + nr_permeability_freq : int, optional + Number of frequencies to use for generating random permeability. Higher values + will give higher freq permeability fields., by default 5 + max_permeability : float, optional + Max permeability, by default 2.0 + min_permeability : float, optional + Min permeability, by default 0.5 + max_iterations : int, optional + Maximum iterations to use for each multi-grid, by default 30000 + convergence_threshold : float, optional + Solver L-Infinity convergence threshold, by default 1e-6 + iterations_per_convergence_check : int, optional + Number of Jacobi iterations to run before checking convergence, by default 1000 + nr_multigrids : int, optional + Number of multi-grid levels, by default 4 + normaliser : Union[Dict[str, Tuple[float, float]], None], optional + Dictionary with keys `permeability` and `darcy`. The values for these keys are two floats corresponding to mean and std `(mean, std)`. + device : Union[str, torch.device], optional + Device for datapipe to run place data on, by default "cuda" + + Raises + ------ + ValueError + Incompatable multi-grid and resolution settings + """ + + @profile + def __init__( + self, + resolution: int = 256, + batch_size: int = 64, + nr_permeability_freq: int = 5, + max_permeability: float = 2.0, + min_permeability: float = 0.5, + max_iterations: int = 30000, + convergence_threshold: float = 1e-6, + iterations_per_convergence_check: int = 1000, + # nr_multigrids: int = 4, + # normaliser: Union[Dict[str, Tuple[float, float]], None] = None, + device: Union[str, torch.device] = "cuda", + train_path: str = None, + is_test: bool = False, + x_normalizer: UnitTransformer = None, + y_normalizer: UnitTransformer = None, + downsample: int = 5, + ): + super().__init__(meta=MetaData()) + + # simulation params + self.resolution = resolution + self.batch_size = batch_size + self.nr_permeability_freq = nr_permeability_freq + self.max_permeability = max_permeability + self.min_permeability = min_permeability + self.max_iterations = max_iterations + self.convergence_threshold = convergence_threshold + self.iterations_per_convergence_check = iterations_per_convergence_check + + # Set up device for warp, warp has same naming convention as torch. + if isinstance(device, torch.device): + device = str(device) + self.device = device + + # spatial dims + self.dx = 1.0 / (self.resolution + 1) # pad edges by 1 for multi-grid + self.dim = (self.batch_size, self.resolution + 1, self.resolution + 1) + + self.train_path = train_path + self.native_resolution = 421 # Native grid size + + # Calculate downsampling factor + if (self.native_resolution - 1) % (self.resolution - 1) != 0: + raise ValueError( + f"Resolution {self.resolution} is not achievable by strided sampling from native resolution {self.native_resolution}." + ) + self.r = (self.native_resolution - 1) // (self.resolution - 1) + self.s = self.resolution + self.dx = 1.0 / self.s + # Output tenors + self.output_k = None + self.output_p = None + + self.is_test = is_test + + if not self.is_test: + self.n_train = 1024 + else: + self.n_train = 200 + + if self.train_path is not None: + self.__get_data__() + + if not self.is_test: + self.x_normalizer = UnitTransformer(self.x_train) + self.y_normalizer = UnitTransformer(self.y_train) + + self.x_train = self.x_normalizer.encode(self.x_train) + self.y_train = self.y_normalizer.encode(self.y_train) + else: + self.x_train = x_normalizer.encode(self.x_train) + self.y_train = y_normalizer.encode(self.y_train) + + @profile + def __get_normalizer__(self): + return self.x_normalizer, self.y_normalizer + + @profile + def __get_data__(self): + if self.train_path.endswith(".mat"): + data_dict = scio.loadmat(self.train_path) + elif self.train_path.endswith(".npz"): + data_dict = np.load(self.train_path) + + # Extract data from dicts: + self.x_train = data_dict["coeff"] + self.y_train = data_dict["sol"] + + x = np.linspace(0, 1, self.s) + y = np.linspace(0, 1, self.s) + x, y = np.meshgrid(x, y) + pos = np.c_[x.ravel(), y.ravel()] + pos = torch.tensor(pos, dtype=torch.float).cuda() + + # Downsampling logic + if self.r > 1: + # Downsample by slicing + self.x_train = self.x_train[: self.n_train, :: self.r, :: self.r][ + :, : self.s, : self.s + ] + self.y_train = self.y_train[: self.n_train, :: self.r, :: self.r][ + :, : self.s, : self.s + ] + else: + # No downsampling, use full resolution + self.x_train = self.x_train[: self.n_train, : self.s, : self.s] + self.y_train = self.y_train[: self.n_train, : self.s, : self.s] + + # Flatten them: + self.x_train = self.x_train.reshape(self.n_train, -1) + self.y_train = self.y_train.reshape(self.n_train, -1) + + self.x_train = torch.from_numpy(self.x_train).float().cuda() + self.y_train = torch.from_numpy(self.y_train).float().cuda() + # Why are we repeating the postion? + # print(f"pos shape: {pos.shape}") + self.pos_train = pos + self.pos_train_batched = pos.repeat(self.batch_size, 1, 1).cuda() + # print(f"pos shape post repeat: {self.pos_train.shape}") + # self.pos_train = pos + + @profile + def __iter__(self): + """ + Yields + ------ + Iterator[Tuple[Tensor, Tensor]] + Infinite iterator that returns a batch of (permeability, darcy pressure) + fields of size [batch, resolution, resolution] + """ + + while True: + # Sample batch_size indices from this rank's shard + idx = np.random.choice(self.n_train, self.batch_size) + # All tensors are already on GPU, so no .cuda() needed + x = self.x_train[idx] + y = self.y_train[idx] + yield self.pos_train_batched, x, y + + def __getitem__(self, idx): + return self.pos_train, self.x_train[idx], self.y_train[idx] + + def __len__(self): + return self.n_train diff --git a/examples/cfd/darcy_transolver/requirements.txt b/examples/cfd/darcy_transolver/requirements.txt new file mode 100644 index 0000000000..8c1435d428 --- /dev/null +++ b/examples/cfd/darcy_transolver/requirements.txt @@ -0,0 +1,9 @@ +einops>=0.7.0 +hydra-core>=1.2.0 +omegaconf>=2.3.0 +scipy>=1.15.0 +warp-lang>=1.7.0 +termcolor>=2.1.1 +mlflow>=2.1.1 +torchinfo +tensorboard diff --git a/examples/cfd/darcy_transolver/train_transolver_darcy.py b/examples/cfd/darcy_transolver/train_transolver_darcy.py new file mode 100644 index 0000000000..012d1e4cf3 --- /dev/null +++ b/examples/cfd/darcy_transolver/train_transolver_darcy.py @@ -0,0 +1,181 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import hydra +from omegaconf import DictConfig +from math import ceil + +from torch.nn import MSELoss +from utils.testloss import TestLoss +from torch.optim import Adam, lr_scheduler + +from physicsnemo.models.transolver import Transolver +from physicsnemo.datapipes.benchmarks.darcy import Darcy2D +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils import StaticCaptureTraining, StaticCaptureEvaluateNoGrad +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, LaunchLogger +from physicsnemo.utils.logging.mlflow import initialize_mlflow + +from validator import GridValidator +from einops import rearrange + + +@hydra.main(version_base="1.3", config_path=".", config_name="config.yaml") +def darcy_trainer(cfg: DictConfig) -> None: + """Training for the 2D Darcy flow benchmark problem.""" + DistributedManager.initialize() # Only call this once in the entire script! + dist = DistributedManager() # call if required elsewhere + + # initialize monitoring + log = PythonLogger(name="darcy_transolver") + log.file_logging() + initialize_mlflow( + experiment_name=f"Darcy_Transolver", + experiment_desc=f"training a Transformer-based PDE solver for the Darcy problem", + run_name=f"Darcy Transolver training", + run_desc=f"training Transolver for Darcy", + user_name="Haixu Wu, Huakun Luo, Haowen Wang", + mode="offline", + ) + LaunchLogger.initialize(use_mlflow=True) # PhysicsNeMo launch logger + + # define model, loss, optimiser, scheduler, data loader + model = Transolver( + out_dim=cfg.model.out_dim, + embedding_dim=cfg.model.embedding_dim, + n_layers=cfg.model.n_layers, + n_hidden=cfg.model.n_hidden, + dropout=cfg.model.dropout, + n_head=cfg.model.n_head, + act=cfg.model.act, + mlp_ratio=cfg.model.mlp_ratio, + functional_dim=cfg.model.functional_dim, + slice_num=cfg.model.slice_num, + unified_pos=True, + ref=cfg.model.ref, + structured_shape=[cfg.data.resolution, cfg.data.resolution], + use_te=cfg.model.use_te, + time_input=cfg.model.time_input, + ).to(dist.device) + + loss_fun = TestLoss(size_average=False) + optimizer = Adam(model.parameters(), lr=cfg.scheduler.initial_lr) + scheduler = lr_scheduler.LambdaLR( + optimizer, lr_lambda=lambda step: cfg.scheduler.decay_rate**step + ) + norm_vars = cfg.normaliser + normaliser = { + "permeability": (norm_vars.permeability.mean, norm_vars.permeability.std_dev), + "darcy": (norm_vars.darcy.mean, norm_vars.darcy.std_dev), + } + dataloader = Darcy2D( + resolution=cfg.training.resolution, + batch_size=cfg.training.batch_size, + normaliser=normaliser, + ) + validator = GridValidator(loss_fun=TestLoss(size_average=False), norm=normaliser) + + ckpt_args = { + "path": f"./checkpoints", + "optimizer": optimizer, + "scheduler": scheduler, + "models": model, + } + loaded_pseudo_epoch = load_checkpoint(device=dist.device, **ckpt_args) + + # calculate steps per pseudo epoch + steps_per_pseudo_epoch = ceil( + cfg.training.pseudo_epoch_sample_size / cfg.training.batch_size + ) + validation_iters = ceil(cfg.validation.sample_size / cfg.training.batch_size) + log_args = { + "name_space": "train", + "num_mini_batch": steps_per_pseudo_epoch, + "epoch_alert_freq": 1, + } + if cfg.training.pseudo_epoch_sample_size % cfg.training.batch_size != 0: + log.warning( + f"increased pseudo_epoch_sample_size to multiple of \ + batch size: {steps_per_pseudo_epoch * cfg.training.batch_size}" + ) + if cfg.validation.sample_size % cfg.training.batch_size != 0: + log.warning( + f"increased validation sample size to multiple of \ + batch size: {validation_iters * cfg.training.batch_size}" + ) + + # define forward passes for training and inference + @StaticCaptureTraining( + model=model, optim=optimizer, logger=log, use_amp=False, use_graphs=False + ) + def forward_train(invars, target): + invars_shape = invars.shape + invars = rearrange(invars, "b c h w -> b (h w) c") + pred = model(invars) + loss = loss_fun(pred, target) + return loss + + @StaticCaptureEvaluateNoGrad( + model=model, logger=log, use_amp=False, use_graphs=False + ) + def forward_eval(invars): + return model(invars) + + if loaded_pseudo_epoch == 0: + log.success("Training started...") + else: + log.warning(f"Resuming training from pseudo epoch {loaded_pseudo_epoch + 1}.") + + for pseudo_epoch in range( + max(1, loaded_pseudo_epoch + 1), cfg.training.max_pseudo_epochs + 1 + ): + # Wrap epoch in launch logger for console / MLFlow logs + with LaunchLogger(**log_args, epoch=pseudo_epoch) as logger: + for _, batch in zip(range(steps_per_pseudo_epoch), dataloader): + loss = forward_train(batch["permeability"], batch["darcy"]) + logger.log_minibatch({"loss": loss.detach()}) + logger.log_epoch({"Learning Rate": optimizer.param_groups[0]["lr"]}) + + # save checkpoint + if pseudo_epoch % cfg.training.rec_results_freq == 0: + save_checkpoint(**ckpt_args, epoch=pseudo_epoch) + + # validation step + if pseudo_epoch % cfg.validation.validation_pseudo_epochs == 0: + with LaunchLogger("valid", epoch=pseudo_epoch) as logger: + total_loss = 0.0 + for _, batch in zip(range(validation_iters), dataloader): + val_loss = validator.compare( + batch["permeability"], + batch["darcy"], + forward_eval(batch["permeability"]), + pseudo_epoch, + logger, + ) + total_loss += val_loss + logger.log_epoch({"Validation error": total_loss / validation_iters}) + + # update learning rate + if pseudo_epoch % cfg.scheduler.decay_pseudo_epochs == 0: + scheduler.step() + + save_checkpoint(**ckpt_args, epoch=cfg.training.max_pseudo_epochs) + log.success("Training completed *yay*") + + +if __name__ == "__main__": + darcy_trainer() diff --git a/examples/cfd/darcy_transolver/train_transolver_darcy_fix.py b/examples/cfd/darcy_transolver/train_transolver_darcy_fix.py new file mode 100644 index 0000000000..0f528f6205 --- /dev/null +++ b/examples/cfd/darcy_transolver/train_transolver_darcy_fix.py @@ -0,0 +1,475 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Configuration imports: +import hydra +from omegaconf import DictConfig, OmegaConf +import json +import time +from math import ceil + +# Base PyTorch imports: +import torchinfo +import torch +import torch.distributed as dist + + +from torch.optim import lr_scheduler, AdamW +from torch.nn.parallel import DistributedDataParallel as DDP + +# PyTorch Data tools +from torch.utils.data import DataLoader, DistributedSampler + +from torch.utils.tensorboard import SummaryWriter + +from utils.testloss import TestLoss + +# Model imports from PhysicsNeMo +from physicsnemo.models.transolver import Transolver +from physicsnemo.distributed import DistributedManager + +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, RankZeroLoggingWrapper + +from darcy_datapipe_fix import Darcy2D_fix +from validator_fix import GridValidator + +from physicsnemo.utils.profiling import Profiler +from contextlib import nullcontext + + +prof = Profiler() + + +def forward_train_full_loop( + model: torch.nn.Module, + loss_fun: callable, + optimizer: torch.optim.Optimizer, + pos: torch.Tensor, + x: torch.Tensor, + y: torch.Tensor, + y_normalizer, + precision_context, + scaler: torch.cuda.amp.GradScaler = None, +) -> torch.Tensor: + """ + Forward and backward pass for one iteration, with optional mixed precision training. + + Args: + model (torch.nn.Module): The model to train. + loss_fun (callable): Loss function. + optimizer (torch.optim.Optimizer): Optimizer. + pos (torch.Tensor): Position tensor (embedding). + x (torch.Tensor): Input tensor. + y (torch.Tensor): Target tensor. + y_normalizer: Normalizer for the target tensor. + precision_context: Context manager for precision (e.g., autocast). + scaler (torch.cuda.amp.GradScaler, optional): GradScaler for mixed precision. + + Returns: + torch.Tensor: The computed loss for this minibatch. + """ + dm = DistributedManager() + with precision_context: + pred = model(embedding=pos, fx=x.unsqueeze(-1)).squeeze(-1) + pred = y_normalizer.decode(pred) + loss = loss_fun(pred, y) + if scaler is not None: + scaler.scale(loss).backward() + scaler.step(optimizer) + scaler.update() + else: + loss.backward() + optimizer.step() + optimizer.zero_grad() + return loss + + +def train_epoch( + model: torch.nn.Module, + optimizer: torch.optim.Optimizer, + scheduler: torch.optim.lr_scheduler._LRScheduler, + train_dataloader: DataLoader, + loss_fun: callable, + y_normalizer, + precision_context, + scaler: torch.cuda.amp.GradScaler, +) -> torch.Tensor: + """ + One epoch of training. Returns the loss from the last minibatch used, averaged across replicas. + + Args: + model (torch.nn.Module): The model to train. + optimizer (torch.optim.Optimizer): Optimizer. + scheduler (torch.optim.lr_scheduler._LRScheduler): Learning rate scheduler. + train_dataloader (DataLoader): Training data loader. + loss_fun (callable): Loss function. + y_normalizer: Normalizer for the target tensor. + precision_context: Context manager for precision (e.g., autocast). + scaler (torch.cuda.amp.GradScaler): GradScaler for mixed precision. + + Returns: + torch.Tensor: The averaged loss from the last minibatch. + """ + for i, batch in enumerate(train_dataloader): + pos, x, y = batch + loss = forward_train_full_loop( + model, + loss_fun, + optimizer, + pos, + x, + y, + y_normalizer, + precision_context, + scaler, + ) + scheduler.step() + + # At the end of the epoch, reduce the last local loss if needed: + dm = DistributedManager() + if dm.world_size > 1: + dist.all_reduce(loss.detach(), op=dist.ReduceOp.SUM) + loss = loss / dm.world_size + + return loss + + +def val_epoch( + model: torch.nn.Module, + test_dataloader: DataLoader, + loss_fun: callable, + y_normalizer, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """ + One epoch of validation. Returns the loss averaged across the entire validation set. + + Args: + model (torch.nn.Module): The model to validate. + test_dataloader (DataLoader): Validation data loader. + loss_fun (callable): Loss function. + y_normalizer: Normalizer for the target tensor. + + Returns: + tuple: (val_loss, pred, y, RL2) + val_loss (torch.Tensor): Averaged validation loss. + pred (torch.Tensor): Last batch predictions. + y (torch.Tensor): Last batch targets. + RL2 (torch.Tensor): Averaged relative L2 error. + """ + val_loss = None + RL2 = None + for i, batch in enumerate(test_dataloader): + pos, x, y = batch + with torch.no_grad(): + pred = model(embedding=pos, fx=x.unsqueeze(-1)).squeeze(-1) + pred = y_normalizer.decode(pred) + loss = loss_fun(pred, y) + + # Compute per-sample relative L2 error + diff = pred.reshape(y.shape) - y + rel_l2 = torch.norm(diff.view(diff.shape[0], -1), dim=1) / torch.norm( + y.view(y.shape[0], -1), dim=1 + ) + rel_l2_mean = rel_l2.mean() + + if RL2 is None: + RL2 = rel_l2_mean + else: + RL2 += rel_l2_mean + if val_loss is None: + val_loss = loss + else: + val_loss += loss + + val_loss = val_loss / len(test_dataloader) + RL2 = RL2 / len(test_dataloader) + + dm = DistributedManager() + if dm.world_size > 1: + dist.all_reduce(val_loss, op=dist.ReduceOp.SUM) + dist.all_reduce(RL2, op=dist.ReduceOp.SUM) + val_loss = val_loss / dm.world_size + RL2 = RL2 / dm.world_size + return val_loss, pred, y, RL2 + + +@hydra.main(version_base="1.3", config_path=".", config_name="config_fix.yaml") +def darcy_trainer(cfg: DictConfig) -> None: + """ + Training entry point for the 2D Darcy flow benchmark problem. + + Args: + cfg (DictConfig): Configuration object loaded by Hydra. + """ + ######################################################################## + # Initialize distributed tools + ######################################################################## + DistributedManager.initialize() # Only call this once in the entire script! + dm = DistributedManager() # call if required elsewhere + + ######################################################################## + # Initialize monitoring and logging + ######################################################################## + logger = RankZeroLoggingWrapper(PythonLogger(name="darcy_transolver"), dm) + logger.file_logging() + + # === TensorBoard SummaryWriter === + # Only rank 0 writes logs to avoid duplication in DDP + writer = None + if dm.rank == 0: + log_dir = f"{cfg.output_dir}/runs/{cfg.run_id}" + writer = SummaryWriter(log_dir=log_dir) + + ######################################################################## + # Print the configuration to log + ######################################################################## + logger.info(json.dumps(OmegaConf.to_container(cfg), indent=4)) + + ######################################################################## + # define model + ######################################################################## + model = Transolver( + functional_dim=cfg.model.functional_dim, + out_dim=cfg.model.out_dim, + embedding_dim=cfg.model.embedding_dim, + n_layers=cfg.model.n_layers, + n_hidden=cfg.model.n_hidden, + dropout=cfg.model.dropout, + n_head=cfg.model.n_head, + act=cfg.model.act, + mlp_ratio=cfg.model.mlp_ratio, + slice_num=cfg.model.slice_num, + unified_pos=cfg.model.unified_pos, + ref=cfg.model.ref, + structured_shape=[cfg.data.resolution, cfg.data.resolution], + use_te=cfg.model.use_te, + time_input=cfg.model.time_input, + ).to(dm.device) + + logger.info(f"\n{torchinfo.summary(model, verbose=0)}") + + if dm.world_size > 1: + model = DDP(model, device_ids=[dm.rank]) + + ######################################################################## + # define loss and optimizer + ######################################################################## + loss_fun = TestLoss(size_average=True) + optimizer = AdamW( + model.parameters(), + lr=cfg.scheduler.initial_lr, + weight_decay=cfg.scheduler.weight_decay, + ) + + ######################################################################## + # Create the data pipes and samplers + ######################################################################## + + train_datapipe = Darcy2D_fix( + resolution=cfg.data.resolution, + batch_size=cfg.data.batch_size, + train_path=cfg.data.train_path, + is_test=False, + ) + # Sampler ensures disjoint instances on each rank + train_sampler = DistributedSampler( + train_datapipe, num_replicas=dm.world_size, rank=dm.rank, shuffle=True + ) + # DataLoader handles the batching + train_dataloader = DataLoader( + train_datapipe, + batch_size=cfg.data.batch_size // dm.world_size, + sampler=train_sampler, + drop_last=True, + ) + # Reuse the train normalizer for the test data: + # (The normalizer puts the inputs and targets to mean 0, std=1.0) + x_normalizer, y_normalizer = train_datapipe.__get_normalizer__() + + test_datapipe = Darcy2D_fix( + resolution=cfg.data.resolution, + batch_size=cfg.data.batch_size, + train_path=cfg.data.test_path, + is_test=True, + x_normalizer=x_normalizer, + y_normalizer=y_normalizer, + ) + test_sampler = DistributedSampler( + test_datapipe, num_replicas=dm.world_size, rank=dm.rank, shuffle=False + ) + test_dataloader = DataLoader( + test_datapipe, + batch_size=cfg.data.batch_size // dm.world_size, + sampler=test_sampler, + drop_last=True, + ) + + # calculate steps per pseudo epoch + steps_per_pseudo_epoch = ceil( + cfg.training.pseudo_epoch_sample_size / cfg.data.batch_size + ) + + scheduler = lr_scheduler.OneCycleLR( + optimizer, + max_lr=cfg.scheduler.initial_lr, + steps_per_epoch=steps_per_pseudo_epoch, + epochs=cfg.training.max_pseudo_epochs, + ) + + validator = GridValidator(output_dir=f"{cfg.output_dir}/runs/{cfg.run_id}/plots") + + ckpt_args = { + "path": f"{cfg.output_dir}/runs/{cfg.run_id}/checkpoints", + "optimizer": optimizer, + "scheduler": scheduler, + "models": model, + } + loaded_pseudo_epoch = load_checkpoint(device=dm.device, **ckpt_args) + + validation_iters = ceil(cfg.validation.sample_size / cfg.data.batch_size) + + if cfg.training.pseudo_epoch_sample_size % cfg.data.batch_size != 0: + logger.warning( + f"increased pseudo_epoch_sample_size to multiple of \ + batch size: {steps_per_pseudo_epoch * cfg.data.batch_size}" + ) + if cfg.validation.sample_size % cfg.data.batch_size != 0: + logger.warning( + f"increased validation sample size to multiple of \ + batch size: {validation_iters * cfg.data.batch_size}" + ) + + # Initialize GradScaler for mixed precision training + if cfg.precision == "fp16": + precision_context = torch.amp.autocast(device_type="cuda", dtype=torch.float16) + scaler = torch.amp.GradScaler("cuda") + elif cfg.precision == "bf16": + precision_context = torch.amp.autocast(device_type="cuda", dtype=torch.bfloat16) + scaler = None + else: + precision_context = nullcontext() + scaler = None + + if loaded_pseudo_epoch == 0: + logger.success("Training started...") + else: + logger.warning( + f"Resuming training from pseudo epoch {loaded_pseudo_epoch + 1}." + ) + + # Get the first batch of the test dataset for plotting + + with prof: + for pseudo_epoch in range( + max(1, loaded_pseudo_epoch + 1), cfg.training.max_pseudo_epochs + 1 + ): + # --- TRAINING --- + train_start = time.time() + loss = train_epoch( + model, + optimizer, + scheduler, + train_dataloader, + loss_fun, + y_normalizer, + precision_context, + scaler, + ) + train_time = time.time() - train_start + + # After training epoch, e.g. after loss, train_time, optimizer, etc. are available: + if torch.cuda.is_available(): + gpu_mem_reserved = torch.cuda.memory_reserved() / 1024**3 + else: + gpu_mem_reserved = 0 + + lr = optimizer.param_groups[0]["lr"] + + header = "mode\tEpoch\tloss\ttime\tLR\t\tGPU_mem" + values = f"train\t{pseudo_epoch}\t{loss.item():.4f}\t{train_time:.2f}\t{lr:.4e}\t{gpu_mem_reserved:.2f}" + + log_string = f"\n{header}\n{values}" + logger.info(log_string) + + # --- TensorBoard logging (only on rank 0) --- + if dm.rank == 0 and writer is not None: + # Images/sec/GPU: (num images processed in train_epoch) / train_time / num_gpus + # Each batch processes batch_size // world_size images, for steps_per_pseudo_epoch steps + images_per_epoch = len(train_dataloader) * ( + cfg.data.batch_size // dm.world_size + ) + images_per_sec_per_gpu = images_per_epoch / train_time + + writer.add_scalar("loss/train", loss.item(), pseudo_epoch) + writer.add_scalar("time_per_epoch/train", train_time, pseudo_epoch) + writer.add_scalar( + "images_per_sec_per_gpu/train", images_per_sec_per_gpu, pseudo_epoch + ) + writer.add_scalar("learning_rate/train", lr, pseudo_epoch) + + # save checkpoint + if pseudo_epoch % cfg.training.rec_results_freq == 0 and dm.rank == 0: + save_checkpoint(**ckpt_args, epoch=pseudo_epoch) + + # --- VALIDATION --- + if pseudo_epoch % cfg.validation.validation_pseudo_epochs == 0: + val_start = time.time() + val_loss, pred, y, RL2 = val_epoch( + model, test_dataloader, loss_fun, y_normalizer + ) + val_time = time.time() - val_start + + header = "mode\tEpoch\tloss\tRL2\ttime" + values = f"val\t{pseudo_epoch}\t{val_loss.item():.4f}\t{RL2.item():.4f}\t{val_time:.2f}" + + log_string = f"\n{header}\n{values}" + logger.info(log_string) + + # --- TensorBoard logging (only on rank 0) --- + if dm.rank == 0 and writer is not None: + # Validation images/sec/GPU + val_images = validation_iters * ( + cfg.data.batch_size // dm.world_size + ) + val_images_per_sec_per_gpu = val_images / val_time + writer.add_scalar("loss/val", val_loss.item(), pseudo_epoch) + writer.add_scalar("RL2/val", RL2.item(), pseudo_epoch) + writer.add_scalar("time_per_epoch/val", val_time, pseudo_epoch) + writer.add_scalar( + "images_per_sec_per_gpu/val", + val_images_per_sec_per_gpu, + pseudo_epoch, + ) + + if dm.rank == 0: + validator.make_plot(pred, y, pseudo_epoch, test_datapipe.s) + + # update learning rate + # if pseudo_epoch % cfg.scheduler.decay_pseudo_epochs == 0: + + if dm.rank == 0 and writer is not None: + writer.close() + logger.success("Training completed *yay*") + + +if __name__ == "__main__": + # prof.enable("line_profile") + # prof.enable("torch") + # prof.initialize() + darcy_trainer() + + # prof.finalize() diff --git a/examples/cfd/darcy_transolver/utils/__init__.py b/examples/cfd/darcy_transolver/utils/__init__.py new file mode 100644 index 0000000000..8d8571af20 --- /dev/null +++ b/examples/cfd/darcy_transolver/utils/__init__.py @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .testloss import TestLoss diff --git a/examples/cfd/darcy_transolver/utils/testloss.py b/examples/cfd/darcy_transolver/utils/testloss.py new file mode 100644 index 0000000000..81e149670a --- /dev/null +++ b/examples/cfd/darcy_transolver/utils/testloss.py @@ -0,0 +1,80 @@ +# ignore_header_test +# ruff: noqa: E402 +"""""" + +""" +Transolver model. This code was modified from, https://github.com/thuml/Transolver + +The following license is provided from their source, + +MIT License + +Copyright (c) 2024 THUML @ Tsinghua University + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +""" + +import torch + + +class TestLoss(object): + def __init__(self, d=2, p=2, size_average=True, reduction=True): + super(TestLoss, self).__init__() + + assert d > 0 and p > 0 + + self.d = d + self.p = p + self.reduction = reduction + self.size_average = size_average + + def abs(self, x, y): + num_examples = x.size()[0] + + h = 1.0 / (x.size()[1] - 1.0) + + all_norms = (h ** (self.d / self.p)) * torch.norm( + x.view(num_examples, -1) - y.view(num_examples, -1), self.p, 1 + ) + + if self.reduction: + if self.size_average: + return torch.mean(all_norms) + else: + return torch.sum(all_norms) + + return all_norms + + def rel(self, x, y): + num_examples = x.size()[0] + + diff_norms = torch.norm( + x.reshape(num_examples, -1) - y.reshape(num_examples, -1), self.p, 1 + ) + y_norms = torch.norm(y.reshape(num_examples, -1), self.p, 1) + if self.reduction: + if self.size_average: + return torch.mean(diff_norms / y_norms) + else: + return torch.sum(diff_norms / y_norms) + + return diff_norms / y_norms + + def __call__(self, x, y): + return self.rel(x, y) diff --git a/examples/cfd/darcy_transolver/validator.py b/examples/cfd/darcy_transolver/validator.py new file mode 100644 index 0000000000..bec8762a25 --- /dev/null +++ b/examples/cfd/darcy_transolver/validator.py @@ -0,0 +1,103 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import matplotlib.pyplot as plt +from torch import FloatTensor +from physicsnemo.utils.logging import LaunchLogger + + +class GridValidator: + """Grid Validator + + The validator compares model output and target, inverts normalisation and plots a sample + + Parameters + ---------- + loss_fun : MSELoss + loss function for assessing validation error + norm : Dict, optional + mean and standard deviation for each channel to normalise input and target + font_size : float, optional + font size used in figures + + """ + + def __init__( + self, + loss_fun, + norm: dict = {"permeability": (0.0, 1.0), "darcy": (0.0, 1.0)}, + font_size: float = 28.0, + ): + self.norm = norm + self.criterion = loss_fun + self.font_size = font_size + self.headers = ("invar", "truth", "prediction", "relative error") + + def compare( + self, + invar: FloatTensor, + target: FloatTensor, + prediction: FloatTensor, + step: int, + logger: LaunchLogger, + ) -> float: + """compares model output, target and plots everything + + Parameters + ---------- + invar : FloatTensor + input to model + target : FloatTensor + ground truth + prediction : FloatTensor + model output + step : int + iteration counter + logger : LaunchLogger + logger to which figure is passed + + Returns + ------- + float + validation error + """ + loss = self.criterion(prediction, target) + norm = self.norm + + # pick first sample from batch + invar = invar * norm["permeability"][1] + norm["permeability"][0] + target = target * norm["darcy"][1] + norm["darcy"][0] + prediction = prediction * norm["darcy"][1] + norm["darcy"][0] + invar = invar.cpu().numpy()[0, -1, :, :] + target = target.cpu().numpy()[0, 0, :, :] + prediction = prediction.detach().cpu().numpy()[0, 0, :, :] + + plt.close("all") + plt.rcParams.update({"font.size": self.font_size}) + fig, ax = plt.subplots(1, 4, figsize=(15 * 4, 15), sharey=True) + im = [] + im.append(ax[0].imshow(invar)) + im.append(ax[1].imshow(target)) + im.append(ax[2].imshow(prediction)) + im.append(ax[3].imshow((prediction - target) / norm["darcy"][1])) + + for ii in range(len(im)): + fig.colorbar(im[ii], ax=ax[ii], location="bottom", fraction=0.046, pad=0.04) + ax[ii].set_title(self.headers[ii]) + + logger.log_figure(figure=fig, artifact_file=f"validation_step_{step:03d}.png") + + return loss diff --git a/examples/cfd/darcy_transolver/validator_fix.py b/examples/cfd/darcy_transolver/validator_fix.py new file mode 100644 index 0000000000..3329dd92de --- /dev/null +++ b/examples/cfd/darcy_transolver/validator_fix.py @@ -0,0 +1,117 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import matplotlib.pyplot as plt +from torch import FloatTensor +import threading +import os + + +class GridValidator: + """Grid Validator + + The validator compares model output and target, inverts normalisation and plots a sample + + Parameters + ---------- + loss_fun : MSELoss + loss function for assessing validation error + norm : Dict, optional + mean and standard deviation for each channel to normalise input and target + font_size : float, optional + font size used in figures + + """ + + def __init__( + self, + font_size: float = 28.0, + output_dir: str = "./plots/", + ): + self.font_size = font_size + self.headers = ("true", "prediction", "error") + self._plot_thread = None + self.output_dir = output_dir + os.makedirs(self.output_dir, exist_ok=True) + + def plot_figure( + self, target: FloatTensor, prediction: FloatTensor, step: int, resolution: int + ): + target = target.cpu().numpy().reshape(-1, resolution, resolution)[0, :, :] + prediction = ( + prediction.reshape(-1, resolution, resolution) + .detach() + .cpu() + .numpy()[0, :, :] + ) + + plt.close("all") + plt.rcParams.update({"font.size": self.font_size}) + fig, ax = plt.subplots(1, 3, figsize=(15 * 3, 15), sharey=True) + im = [] + im.append(ax[0].imshow(target)) + im.append(ax[1].imshow(prediction)) + im.append(ax[2].imshow((prediction - target))) + + for ii in range(len(im)): + fig.colorbar(im[ii], ax=ax[ii], location="bottom", fraction=0.046, pad=0.04) + ax[ii].set_title(self.headers[ii]) + + plt.savefig(f"{self.output_dir}/validation_step_{step:03d}.png") + + def _plot_figure_thread(self, target, prediction, step, resolution): + self.plot_figure(target, prediction, step, resolution) + + def make_plot( + self, + prediction: FloatTensor, + target: FloatTensor, + step: int, + resolution: int, + ) -> float: + """compares model output, target and plots everything + + Parameters + ---------- + invar : FloatTensor + input to model + target : FloatTensor + ground truth + prediction : FloatTensor + model output + step : int + iteration counter + logger : LaunchLogger + logger to which figure is passed + + Returns + ------- + float + validation error + """ + + # Wait for previous plot thread if still running + if self._plot_thread is not None and self._plot_thread.is_alive(): + self._plot_thread.join() + + # Start new plot thread + self._plot_thread = threading.Thread( + target=self._plot_figure_thread, + args=(target, prediction, step, resolution), + ) + self._plot_thread.start() + + return diff --git a/examples/cfd/datacenter/README.md b/examples/cfd/datacenter/README.md new file mode 100644 index 0000000000..4dd0e1819d --- /dev/null +++ b/examples/cfd/datacenter/README.md @@ -0,0 +1,97 @@ +# Thermal and airflow surrogate model for Datacenter design + +This example demonstrates the use of a Deep Learning model (3D UNet) for training a +surrogate model for datacenter airflow to enable real-time datacenter design. +The aim of this workflow is to train a Deep Learning model that can predict the +temperature and airflow distribution within a hot aisle of a typical datacenter. +For any given geometry of the hot aisle (height, width, and length) and the number +of IT racks inside it, the trained model can predict the temperature, velocity, +and pressure distribution inside the hot aisle instantaneously. Such an approach +can be very useful from a datacenter design perspective where iterating through +various design combinations is crucial to obtain optimal cooling and minimize +hotspots. + +![Design study using the AI surrogate model](../../../docs/img/datacenter_design_cfd.gif) + +## Dataset + +The model is trained on OpenFOAM simulation data. Based on the variables, i.e., +Length, Height, Width, and Number of Racks, several hot aisle configurations are +generated. These configurations are solved with OpenFOAM assuming maximum flow +rate and rack exit temperature (max load condition). Steady state simulations +are used and the resulting OpenFOAM data is exported in VTK format for training +of the AI surrogate. The dataset is then normalized using the mean and standard +deviation statistics of the dataset. The normalized dataset, along with a sample +OpenFOAM configuration, can be downloaded from NGC link +[here](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/physicsnemo/resources/physicsnemo_datacenter_cfd_dataset) + +After downloading, place the datasets directory into the current directory. +Running below commands should setup the directory structure required to run the +training +([Requires NGC CLI](https://docs.ngc.nvidia.com/cli/index.html)). + +```bash +ngc registry resource download-version "nvidia/physicsnemo/physicsnemo_datacenter_cfd_dataset:v1" +mv physicsnemo_datacenter_cfd_dataset_vv1/datasets . +``` + +**Note:** Access to NVAIE is required to download the dataset +and the reference OpenFOAM configuration. + +**Note:** The OpenFOAM configuration provided is only representative. +Several key aspects have been masked to protect the IP. +Users should not expect to generate the training data +exactly using this setup, and one will have to change +it using their own geometries and boundary conditions. + +## Training + +**Note:** A minimum GPU memory of 80GB is required for this example. + +A UNet model is used in this problem. The hex-dominant mesh used in this problem +makes this model an attractive choice offering good speed and accuracy. Since +the model is primarily trained to capture the changes in geometry, we use the +Signed Distance Field of the interior of the hot aisle to capture the parameter +variation. Additionally, we add sinusoidal embeddings to enable the model to +capture sharp features in the flow field. Finally, to make the different +datacenter sizes uniform (for ingestion into UNet), we pad the geometry for the +maximum hot aisle dimensions. This padding is removed before computing the loss. + +The model can be trained by executing the below commands: + +```bash +python train.py +``` + +To train on multiple GPUs, + +```bash +mpirun -np <#GPUs> python train.py +``` + +Once the model is trained, you can use the inference.py script to compute the +model inference. For generating the Signed Distance Field and geometry for the +inference, we make use of the utilities from PhysicsNeMo-Sym. + +### Training of Physics-Informed model + +We also train a variant where we add the physics losses to the data loss. +The physics-informed training can be executed using the below commands: + +```bash +python train_physics_informed.py +``` + +Addition of such physics data losses proves very beneficial in the low-data +regime where the physics losses can compensate for the lack of enough data. + +![Comparison of data+physics driven training with pure data driven training](../../../docs/img/datacenter_hybrid_training.png) + +## Contributors + +This example was developed as a part of collaboration between NVIDIA and Wistron. + +## Resources + +1. [Wistron Uses NVIDIA Omniverse and NVIDIA PhysicsNeMo to Build Digital Twin Platform, Transforming Factory Planning and Operations](https://www.wistron.com/en/Newsroom/2024-03-19-1) +2. [Model Innovators: How Digital Twins Are Making Industries More Efficient](https://blogs.nvidia.com/blog/digital-twins-nvidia-physicsnemo-wistron/) diff --git a/examples/cfd/datacenter/conf/config.yaml b/examples/cfd/datacenter/conf/config.yaml new file mode 100644 index 0000000000..f129a59062 --- /dev/null +++ b/examples/cfd/datacenter/conf/config.yaml @@ -0,0 +1,33 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +hydra: + job: + chdir: True + run: + dir: ./outputs + +start_epoch: 1 +max_epochs: 260 + +start_lr: 1e-3 +lr_scheduler_gamma: 0.99975 + +train_batch_size: 2 +val_batch_size: 2 + +train_num_samples: 768 +val_num_samples: 192 diff --git a/examples/cfd/datacenter/conf/config_inference.yaml b/examples/cfd/datacenter/conf/config_inference.yaml new file mode 100644 index 0000000000..83fb2dd40d --- /dev/null +++ b/examples/cfd/datacenter/conf/config_inference.yaml @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +hydra: + job: + chdir: True + run: + dir: ./outputs diff --git a/examples/cfd/datacenter/conf/config_physics_informed.yaml b/examples/cfd/datacenter/conf/config_physics_informed.yaml new file mode 100644 index 0000000000..daea1f7157 --- /dev/null +++ b/examples/cfd/datacenter/conf/config_physics_informed.yaml @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +hydra: + job: + chdir: True + run: + dir: ./outputs + +start_epoch: 1 +max_epochs: 260 + +start_lr: 1e-3 +lr_scheduler_gamma: 0.99975 + +phy_wt: 0.001 + +train_batch_size: 2 +val_batch_size: 2 + +train_num_samples: 64 +val_num_samples: 192 \ No newline at end of file diff --git a/examples/cfd/datacenter/inference.py b/examples/cfd/datacenter/inference.py new file mode 100644 index 0000000000..d3bed356b5 --- /dev/null +++ b/examples/cfd/datacenter/inference.py @@ -0,0 +1,346 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from physicsnemo.datapipes.cae.mesh_datapipe import MeshDatapipe +from physicsnemo.distributed import DistributedManager +import vtk +from physicsnemo.models.unet import UNet +import matplotlib.pyplot as plt +from omegaconf import DictConfig +import torch +import hydra +import matplotlib.pyplot as plt +import torch.nn.functional as F +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, LaunchLogger +from hydra.utils import to_absolute_path +from torch.nn.parallel import DistributedDataParallel +from physicsnemo.utils import StaticCaptureTraining, StaticCaptureEvaluateNoGrad +from apex import optimizers +import os +import numpy as np +from vtk.util.numpy_support import vtk_to_numpy, numpy_to_vtk +from physicsnemo.sym.geometry.primitives_3d import Box, Channel +from physicsnemo.sym.utils.io.vtk import var_to_polyvtk +import itertools + + +def reshape_fortran(x, shape): + """Based on https://stackoverflow.com/questions/63960352/reshaping-order-in-pytorch-fortran-like-index-ordering""" + if len(x.shape) > 0: + x = x.permute(*reversed(range(len(x.shape)))) + return x.reshape(*reversed(shape)).permute(*reversed(range(len(shape)))) + + +def generate_mask(points, sample): + """ + Generate a mask + """ + num_racks, width, gap, translate, length, height = ( + sample[1], + sample[2], + sample[3], + sample[4], + sample[5], + sample[6], + ) + + rack_x = 600 / 1000 + rack_y = 50 / 1000 + rack_z = 2200 / 1000 + + width = width * 2 / 1000 + length = length / 1000 + height = height / 1000 + + origin = (0, 0.05, 0) + + w1_x = gap / 2 / 1000 # the x distance of the left wall + geo = Box( + (origin[0] + w1_x, origin[1], origin[2]), + (origin[0] + w1_x + rack_x, origin[1] + rack_y, origin[2] + rack_z), + ) + geo = geo.repeat( + gap / 1000 + rack_x, + repeat_lower=(0, 0, 0), + repeat_higher=(int(num_racks - 1), 0, 0), + center=( + origin[0] + w1_x + rack_x / 2, + origin[1] + rack_y / 2, + origin[2] + rack_z / 2, + ), + ) + + geo_block_pos_y = Box( + (origin[0] - w1_x, origin[1] - rack_y, origin[2]), + (origin[0] + w1_x, origin[1] + 2, origin[2] + rack_z), + ) + geo_block_neg_y = Box( + (origin[0] - w1_x, origin[1] - width - 2 * rack_y - 2, origin[2]), + (origin[0] + w1_x, origin[1] - width - rack_y, origin[2] + rack_z), + ) + + geo_block_pos_y = geo_block_pos_y.repeat( + gap / 1000 + rack_x, + repeat_lower=(0, 0, 0), + repeat_higher=(int(num_racks), 0, 0), + center=(origin[0], origin[1] - rack_y / 2 + 1, origin[2] + rack_z / 2), + ) + + geo_block_neg_y = geo_block_neg_y.repeat( + gap / 1000 + rack_x, + repeat_lower=(0, 0, 0), + repeat_higher=(int(num_racks), 0, 0), + center=( + origin[0], + origin[1] - width - 3 * rack_y / 2 - 1, + origin[2] + rack_z / 2, + ), + ) + + geo_block = geo_block_pos_y + geo_block_neg_y + + rack_top_pos_x = Box( + (origin[0] - 5, origin[1] - rack_y, origin[2] + rack_z), + (origin[0] + length + 5, origin[1] + 2, origin[2] + height + 10), + ) + rack_top_neg_x = Box( + (origin[0] - 5, origin[1] - width - 2 * rack_y - 2, origin[2] + rack_z), + (origin[0] + length + 5, origin[1] - width - rack_y, origin[2] + height + 10), + ) + + geo_block = geo_block + rack_top_pos_x + rack_top_neg_x + + hot_aisle_bounds = ( + (origin[0], origin[1] - width - 2 * rack_y, origin[2]), + (origin[0] + length, origin[1], origin[2] + height), + ) + + hot_aisle = Channel( + (origin[0] - 5, origin[1] - width - 2, origin[2]), + (origin[0] + length + 5, origin[1] + 2, origin[2] + height + 10), + ) + + hot_aisle = hot_aisle - geo_block + + # Compute SDF on the points + sdf = hot_aisle.sdf(points, params={}) + + return sdf["sdf"], hot_aisle_bounds + + +def save_to_vtu(data_dict, bounds, output_file): + num_cells_x, num_cells_y, num_cells_z = next(iter(data_dict.values())).shape + x_min, x_max, y_min, y_max, z_min, z_max = bounds + dx = (x_max - x_min) / (num_cells_x - 1) + dy = (y_max - y_min) / (num_cells_y - 1) + dz = (z_max - z_min) / (num_cells_z - 1) + + # Create an unstructured grid + points = vtk.vtkPoints() + grid = vtk.vtkUnstructuredGrid() + + # Insert points + for k in range(num_cells_z): + for j in range(num_cells_y): + for i in range(num_cells_x): + points.InsertNextPoint(x_min + i * dx, y_min + j * dy, z_min + k * dz) + + grid.SetPoints(points) + + # Create cells + for k in range(num_cells_z - 1): + for j in range(num_cells_y - 1): + for i in range(num_cells_x - 1): + pt_ids = [ + i + j * num_cells_x + k * num_cells_x * num_cells_y, + (i + 1) + j * num_cells_x + k * num_cells_x * num_cells_y, + (i + 1) + (j + 1) * num_cells_x + k * num_cells_x * num_cells_y, + i + (j + 1) * num_cells_x + k * num_cells_x * num_cells_y, + i + j * num_cells_x + (k + 1) * num_cells_x * num_cells_y, + (i + 1) + j * num_cells_x + (k + 1) * num_cells_x * num_cells_y, + (i + 1) + + (j + 1) * num_cells_x + + (k + 1) * num_cells_x * num_cells_y, + i + (j + 1) * num_cells_x + (k + 1) * num_cells_x * num_cells_y, + ] + grid.InsertNextCell(vtk.VTK_HEXAHEDRON, 8, pt_ids) + + # Add data arrays to the grid + for var_name, array in data_dict.items(): + array = np.asfortranarray(array) + flat_array = array.flatten(order="F") + vtk_array = numpy_to_vtk(flat_array, deep=True) + vtk_array.SetName(var_name) + grid.GetPointData().AddArray(vtk_array) + + # Write the unstructured grid to a VTU file + writer = vtk.vtkXMLUnstructuredGridWriter() + writer.SetFileName(output_file) + writer.SetInputData(grid) + writer.Write() + + +@hydra.main(version_base="1.2", config_path="conf", config_name="config_inference") +def main(cfg: DictConfig) -> None: + print("Inference Started!") + + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + nx, ny, nz = 960, 96, 80 + + # Compute positional embeddings + x = np.linspace(-1, 1, nx) + y = np.linspace(-1, 1, ny) + z = np.linspace(-1, 1, nz) + + xv, yv, zv = np.meshgrid(x, y, z, indexing="ij") + x_freq_sin = np.sin(xv * 72 * np.pi / 2) + x_freq_cos = np.cos(xv * 72 * np.pi / 2) + y_freq_sin = np.sin(yv * 8 * np.pi / 2) + y_freq_cos = np.cos(yv * 8 * np.pi / 2) + z_freq_sin = np.sin(zv * 8 * np.pi / 2) + z_freq_cos = np.cos(zv * 8 * np.pi / 2) + pos_embed = np.stack( + ( + xv, + x_freq_sin, + x_freq_cos, + yv, + y_freq_sin, + y_freq_cos, + zv, + z_freq_sin, + z_freq_cos, + ), + axis=0, + ) + + model = UNet( + in_channels=10, + out_channels=5, + model_depth=5, + feature_map_channels=[32, 32, 64, 64, 128, 128, 256, 256, 512, 512], + num_conv_blocks=2, + ).to(dist.device) + + loaded_epoch = load_checkpoint( + to_absolute_path("./outputs/checkpoints/"), + models=model, + device=dist.device, + ) + + grid_dims = (nx, ny, nz) # dimensions of the grid + bounds = (0, 40, -3.95, 0.05, 0, 3.2) # bounding box coordinates + + # Define the bounds and resolution of the Cartesian grid + x_min, x_max, y_min, y_max, z_min, z_max = bounds + num_cells_x, num_cells_y, num_cells_z = grid_dims + dx = (x_max - x_min) / (num_cells_x - 1) + dy = (y_max - y_min) / (num_cells_y - 1) + dz = (z_max - z_min) / (num_cells_z - 1) + + x = np.linspace(x_min, x_max, num_cells_x) + y = np.linspace(y_min, y_max, num_cells_y) + z = np.linspace(z_min, z_max, num_cells_z) + + xv, yv, zv = np.meshgrid(x, y, z, indexing="ij") + + points = { + "x": xv, + "y": yv, + "z": zv, + } + + # Generate custom samples + racks = np.linspace(35, 55, 6) + length = 40000 + widths = 3500 / 2 + heights = 2900 + combinations = list(itertools.product(racks)) + + # Define mean and std dictionaries + mean_dict = { + "T": 39, + "U": 1.5983600616455078, + "p": 6.1226935386657715, + "wallDistance": 0.6676982045173645, + } + std_dict = { + "T": 4, + "U": 1.3656059503555298, + "p": 4.166020393371582, + "wallDistance": 0.45233625173568726, + } + + model.eval() + + for design in combinations: + print("Computing: ", design) + rack, width, height = design[0], widths, heights + gap = (length / rack) - 600 + sample = ( + 0, + rack, + width, + gap, + 0, + length, + height, + ) # case num and translate var dont matter + + sdf, hot_aisle_bounds = generate_mask(points, sample) + mask = np.where( + (sdf > 0) + & (zv < hot_aisle_bounds[1][2]) + & (yv > hot_aisle_bounds[0][1]) + & (xv < hot_aisle_bounds[1][0]), + 1, + 0, + ) + + sdf = ((sdf - mean_dict["wallDistance"]) / std_dict["wallDistance"]) * mask + + invar_np = np.concatenate( + (np.expand_dims(sdf, 0), pos_embed), axis=0 + ) # concat along channel dim + invar_np = np.expand_dims(invar_np, 0) # add batch dim + invar_tensor = torch.from_numpy(invar_np).to(dist.device).to(torch.float) + + with torch.no_grad(): + pred_outvar = model(invar_tensor) + + pred_outvar_np = pred_outvar.detach().cpu().numpy() + + output_filename = f"results_{rack}_{length}_{width}_{height}.vtu" + var = { + "u_x_pred": pred_outvar_np[0, 0], + "u_y_pred": pred_outvar_np[0, 1], + "u_z_pred": pred_outvar_np[0, 2], + "T_pred": pred_outvar_np[0, 3], + "p_pred": pred_outvar_np[0, 4], + "wallDistance": invar_np[0, 0], + "mask": mask, + } + save_to_vtu(var, bounds, output_filename) + + print("Inference complete") + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/datacenter/requirements.txt b/examples/cfd/datacenter/requirements.txt new file mode 100644 index 0000000000..f2cdfd272d --- /dev/null +++ b/examples/cfd/datacenter/requirements.txt @@ -0,0 +1,4 @@ +vtk +omegaconf +hydra-core +nvidia-physicsnemo.sym \ No newline at end of file diff --git a/examples/cfd/datacenter/train.py b/examples/cfd/datacenter/train.py new file mode 100644 index 0000000000..35a4738e96 --- /dev/null +++ b/examples/cfd/datacenter/train.py @@ -0,0 +1,316 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from physicsnemo.datapipes.cae.mesh_datapipe import MeshDatapipe +from physicsnemo.distributed import DistributedManager +import vtk +from physicsnemo.models.unet import UNet +import matplotlib.pyplot as plt +from omegaconf import DictConfig +import torch +import hydra +import matplotlib.pyplot as plt +import torch.nn.functional as F +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, LaunchLogger +from hydra.utils import to_absolute_path +from torch.nn.parallel import DistributedDataParallel +from physicsnemo.utils import StaticCaptureTraining, StaticCaptureEvaluateNoGrad +from apex import optimizers +import os +import numpy as np + + +def reshape_fortran(x, shape): + """Based on https://stackoverflow.com/questions/63960352/reshaping-order-in-pytorch-fortran-like-index-ordering""" + if len(x.shape) > 0: + x = x.permute(*reversed(range(len(x.shape)))) + return x.reshape(*reversed(shape)).permute(*reversed(range(len(shape)))) + + +@torch.no_grad() +def validation_step( + model, dataset, pos_embed_tensor, epoch, plotting=False, device=None, name="default" +): + loss_epoch = 0.0 + num_samples = 0.0 + + nx, ny, nz = 960, 96, 80 + for i, data in enumerate(dataset): + bs, _, chans = data[0]["x"].shape + + var = reshape_fortran(data[0]["x"], (bs, nx, ny, nz, chans)) + + mask = torch.permute(var[..., 6:7], (0, 4, 1, 2, 3)) + invar = torch.permute(var[..., 5:6], (0, 4, 1, 2, 3)) # Grab Wall Distance + invar = torch.cat((invar, pos_embed_tensor), axis=1) + outvar = torch.permute( + var[..., 0:5], (0, 4, 1, 2, 3) + ) # Grab U components, T and P + pred_outvar = model(invar) + outvar = outvar * mask + pred_outvar = pred_outvar * mask + loss_epoch += F.mse_loss(outvar, pred_outvar) + + num_samples += invar.shape[0] + + if plotting: + if i == 0: + for chan in range(outvar.size(1)): + fig, ax = plt.subplots(1, 3) + vmin, vmax = ( + np.min(outvar[i, chan, :, :, nz // 2].detach().cpu().numpy()), + np.max(outvar[i, chan, :, :, nz // 2].detach().cpu().numpy()), + ) + # plot z slices + im = ax[0].imshow( + outvar[i, chan, :, :, nz // 2].detach().cpu().numpy(), + vmin=vmin, + vmax=vmax, + ) + fig.colorbar(im, ax=ax[0]) + im = ax[1].imshow( + pred_outvar[i, chan, :, :, nz // 2].detach().cpu().numpy(), + vmin=vmin, + vmax=vmax, + ) + fig.colorbar(im, ax=ax[1]) + im = ax[2].imshow( + ( + pred_outvar[i, chan, :, :, nz // 2] + - outvar[i, chan, :, :, nz // 2] + ) + .detach() + .cpu() + .numpy() + ) + fig.colorbar(im, ax=ax[2]) + + ax[0].set_aspect("equal") + ax[1].set_aspect("equal") + ax[2].set_aspect("equal") + + ax[0].set_title("True") + ax[1].set_title("Pred") + ax[2].set_title("Diff") + + plt.savefig(f"chan_{chan}_epoch_{epoch}_mid_z_slice_{name}.png") + plt.close() + + return loss_epoch.detach() / num_samples + + +@hydra.main(version_base="1.2", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + logger = PythonLogger("main") # General python logger + LaunchLogger.initialize() + + nx, ny, nz = 960, 96, 80 + + # Compute positional embeddings + x = np.linspace(-1, 1, nx) + y = np.linspace(-1, 1, ny) + z = np.linspace(-1, 1, nz) + + xv, yv, zv = np.meshgrid(x, y, z, indexing="ij") + x_freq_sin = np.sin(xv * 72 * np.pi / 2) + x_freq_cos = np.cos(xv * 72 * np.pi / 2) + y_freq_sin = np.sin(yv * 8 * np.pi / 2) + y_freq_cos = np.cos(yv * 8 * np.pi / 2) + z_freq_sin = np.sin(zv * 8 * np.pi / 2) + z_freq_cos = np.cos(zv * 8 * np.pi / 2) + pos_embed = np.stack( + ( + xv, + x_freq_sin, + x_freq_cos, + yv, + y_freq_sin, + y_freq_cos, + zv, + z_freq_sin, + z_freq_cos, + ), + axis=0, + ) + + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + pos_embed_tensor = torch.from_numpy(pos_embed).to(torch.float).to(dist.device) + pos_embed_tensor = pos_embed_tensor.repeat( + cfg.train_batch_size, 1, 1, 1, 1 + ) # repeat along the batch size dim + + model = UNet( + in_channels=10, + out_channels=5, + model_depth=5, + feature_map_channels=[32, 32, 64, 64, 128, 128, 256, 256, 512, 512], + num_conv_blocks=2, + ).to(dist.device) + + # Distributed learning (Data parallel) + if dist.world_size > 1: + model = DistributedDataParallel( + model, + device_ids=[dist.local_rank], + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + ) + + # Initialize the dataset + data_dir = to_absolute_path("./datasets/train/") + dataset = MeshDatapipe( + data_dir=data_dir, + file_format="vtu", + variables=["U", "T", "p", "wallDistance", "vtkValidPointMask"], + num_variables=7, + num_samples=cfg.train_num_samples, + batch_size=cfg.train_batch_size, + num_workers=1, + device=dist.device, + process_rank=dist.rank, + world_size=dist.world_size, + shuffle=True, + parallel=False, + ) + + # Initialize the validation dataset + if dist.rank == 0: + pos_embed_tensor_val = ( + torch.from_numpy(pos_embed).to(torch.float).to(dist.device) + ) + pos_embed_tensor_val = pos_embed_tensor_val.repeat( + cfg.val_batch_size, 1, 1, 1, 1 + ) # repeat along the batch size dim + val_data_dir = to_absolute_path("./datasets/test/") + val_dataset = MeshDatapipe( + data_dir=val_data_dir, + file_format="vtu", + variables=["U", "T", "p", "wallDistance", "vtkValidPointMask"], + num_variables=7, + num_samples=cfg.val_num_samples, + batch_size=cfg.val_batch_size, + num_workers=1, + device=dist.device, + process_rank=dist.rank, + world_size=dist.world_size, + shuffle=False, + parallel=False, + ) + + train_dataset_plotting = MeshDatapipe( + data_dir=data_dir, + file_format="vtu", + variables=["U", "T", "p", "wallDistance", "vtkValidPointMask"], + num_variables=7, + num_samples=16, + batch_size=cfg.val_batch_size, + num_workers=0, + device=dist.device, + process_rank=dist.rank, + world_size=dist.world_size, + shuffle=False, + parallel=False, + ) + + optimizer = optimizers.FusedAdam( + model.parameters(), betas=(0.9, 0.999), lr=cfg.start_lr, weight_decay=0.0 + ) + + scheduler = torch.optim.lr_scheduler.ExponentialLR( + optimizer, gamma=cfg.lr_scheduler_gamma + ) + + # Attempt to load latest checkpoint if one exists + loaded_epoch = load_checkpoint( + "./checkpoints", + models=model, + optimizer=optimizer, + scheduler=scheduler, + device=dist.device, + ) + + for epoch in range(max(1, loaded_epoch + 1), cfg.max_epochs + 1): # epochs + with LaunchLogger( + "train", epoch=epoch, num_mini_batch=len(dataset), epoch_alert_freq=1 + ) as log: + for i, data in enumerate(dataset): + optimizer.zero_grad() + bs, _, chans = data[0]["x"].shape + + var = reshape_fortran(data[0]["x"], (bs, nx, ny, nz, chans)) + + mask = torch.permute(var[..., 6:7], (0, 4, 1, 2, 3)) + invar = torch.permute( + var[..., 5:6], (0, 4, 1, 2, 3) + ) # Grab Wall Distance + invar = torch.cat( + (invar, pos_embed_tensor), axis=1 + ) # Concat along channel dim + outvar = torch.permute( + var[..., 0:5], (0, 4, 1, 2, 3) + ) # Grab U components, T and P + pred_outvar = model(invar) + + outvar = outvar * mask + pred_outvar = pred_outvar * mask + loss = F.mse_loss(outvar, pred_outvar) + loss.backward() + optimizer.step() + scheduler.step() + + log.log_minibatch({"Mini-batch loss": loss.detach()}) + log.log_epoch({"Learning Rate": optimizer.param_groups[0]["lr"]}) + + if dist.world_size > 1: + torch.distributed.barrier() + + if dist.rank == 0: + with LaunchLogger("valid", epoch=epoch) as log: + train_loss = validation_step( + model, + train_dataset_plotting, + pos_embed_tensor_val, + epoch, + plotting=True, + name="train", + ) + val_loss = validation_step( + model, + val_dataset, + pos_embed_tensor_val, + epoch, + plotting=True, + name="val", + ) + log.log_epoch({"Val loss": val_loss, "Train loss": train_loss}) + + if epoch % 2 == 0 and dist.rank == 0: + save_checkpoint( + "./checkpoints", + models=model, + optimizer=optimizer, + scheduler=scheduler, + epoch=epoch, + ) + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/datacenter/train_physics_informed.py b/examples/cfd/datacenter/train_physics_informed.py new file mode 100644 index 0000000000..b6b527d771 --- /dev/null +++ b/examples/cfd/datacenter/train_physics_informed.py @@ -0,0 +1,382 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from physicsnemo.datapipes.cae.mesh_datapipe import MeshDatapipe +from physicsnemo.distributed import DistributedManager +import vtk +from physicsnemo.models.unet import UNet +import matplotlib.pyplot as plt +from omegaconf import DictConfig +import torch +import hydra +import matplotlib.pyplot as plt +import torch.nn.functional as F +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, LaunchLogger +from hydra.utils import to_absolute_path +from torch.nn.parallel import DistributedDataParallel +from physicsnemo.utils import StaticCaptureTraining, StaticCaptureEvaluateNoGrad +from apex import optimizers +import os +import numpy as np +from physicsnemo.sym.eq.phy_informer import PhysicsInformer +from physicsnemo.sym.eq.pdes.navier_stokes import NavierStokes + + +def dilate_mask_3d(mask, padding_size): + """Dilate a 3D mask by a specified padding size.""" + + inverted_mask = (~mask.bool()).float() + + kernel_size = 2 * padding_size + 1 + kernel = torch.ones( + (kernel_size, kernel_size, kernel_size), dtype=torch.float32 + ).to(mask.device) + kernel = kernel.unsqueeze(0).unsqueeze(0) + + dilated_result = torch.clamp( + torch.nn.functional.conv3d(inverted_mask, kernel, padding=padding_size), 0, 1 + ) + dilated_result = (~dilated_result.bool()).float() + + return dilated_result + + +def reshape_fortran(x, shape): + """Based on https://stackoverflow.com/questions/63960352/reshaping-order-in-pytorch-fortran-like-index-ordering""" + if len(x.shape) > 0: + x = x.permute(*reversed(range(len(x.shape)))) + return x.reshape(*reversed(shape)).permute(*reversed(range(len(shape)))) + + +@torch.no_grad() +def validation_step( + model, dataset, pos_embed_tensor, epoch, plotting=False, device=None, name="default" +): + loss_epoch = 0.0 + num_samples = 0.0 + + nx, ny, nz = 960, 96, 80 + for i, data in enumerate(dataset): + bs, _, chans = data[0]["x"].shape + + var = reshape_fortran(data[0]["x"], (bs, nx, ny, nz, chans)) + + mask = torch.permute(var[..., 6:7], (0, 4, 1, 2, 3)) + invar = torch.permute(var[..., 5:6], (0, 4, 1, 2, 3)) # Grab Wall Distance + invar = torch.cat((invar, pos_embed_tensor), axis=1) + outvar = torch.permute( + var[..., 0:5], (0, 4, 1, 2, 3) + ) # Grab U components, T and P + pred_outvar = model(invar) + outvar = outvar * mask + pred_outvar = pred_outvar * mask + loss_epoch += F.mse_loss(outvar, pred_outvar) + + num_samples += invar.shape[0] + + if plotting: + if i == 0: + for chan in range(outvar.size(1)): + fig, ax = plt.subplots(1, 3) + vmin, vmax = ( + np.min(outvar[i, chan, :, :, nz // 2].detach().cpu().numpy()), + np.max(outvar[i, chan, :, :, nz // 2].detach().cpu().numpy()), + ) + # plot z slices + im = ax[0].imshow( + outvar[i, chan, :, :, nz // 2].detach().cpu().numpy(), + vmin=vmin, + vmax=vmax, + ) + fig.colorbar(im, ax=ax[0]) + im = ax[1].imshow( + pred_outvar[i, chan, :, :, nz // 2].detach().cpu().numpy(), + vmin=vmin, + vmax=vmax, + ) + fig.colorbar(im, ax=ax[1]) + im = ax[2].imshow( + ( + pred_outvar[i, chan, :, :, nz // 2] + - outvar[i, chan, :, :, nz // 2] + ) + .detach() + .cpu() + .numpy() + ) + fig.colorbar(im, ax=ax[2]) + + ax[0].set_aspect("equal") + ax[1].set_aspect("equal") + ax[2].set_aspect("equal") + + ax[0].set_title("True") + ax[1].set_title("Pred") + ax[2].set_title("Diff") + + plt.savefig(f"chan_{chan}_epoch_{epoch}_mid_z_slice_{name}.png") + plt.close() + + return loss_epoch.detach() / num_samples + + +@hydra.main( + version_base="1.2", config_path="conf", config_name="config_physics_informed" +) +def main(cfg: DictConfig) -> None: + logger = PythonLogger("main") # General python logger + LaunchLogger.initialize() + + nx, ny, nz = 960, 96, 80 + + # Compute positional embeddings + x = np.linspace(-1, 1, nx) + y = np.linspace(-1, 1, ny) + z = np.linspace(-1, 1, nz) + + xv, yv, zv = np.meshgrid(x, y, z, indexing="ij") + x_freq_sin = np.sin(xv * 72 * np.pi / 2) + x_freq_cos = np.cos(xv * 72 * np.pi / 2) + y_freq_sin = np.sin(yv * 8 * np.pi / 2) + y_freq_cos = np.cos(yv * 8 * np.pi / 2) + z_freq_sin = np.sin(zv * 8 * np.pi / 2) + z_freq_cos = np.cos(zv * 8 * np.pi / 2) + pos_embed = np.stack( + ( + xv, + x_freq_sin, + x_freq_cos, + yv, + y_freq_sin, + y_freq_cos, + zv, + z_freq_sin, + z_freq_cos, + ), + axis=0, + ) + + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + pos_embed_tensor = torch.from_numpy(pos_embed).to(torch.float).to(dist.device) + pos_embed_tensor = pos_embed_tensor.repeat( + cfg.train_batch_size, 1, 1, 1, 1 + ) # repeat along the batch size dim + + model = UNet( + in_channels=10, + out_channels=5, + model_depth=5, + feature_map_channels=[32, 32, 64, 64, 128, 128, 256, 256, 512, 512], + num_conv_blocks=2, + ).to(dist.device) + + bounds = (0, 40, -3.95, 0.05, 0, 3.2) # bounding box coordinates + nx, ny, nz = 960, 96, 80 + + # Define mean and std dictionaries + mean_dict = { + "T": 39, + "U": 1.5983600616455078, + "p": 6.1226935386657715, + "wallDistance": 0.6676982045173645, + } + std_dict = { + "T": 4, + "U": 1.3656059503555298, + "p": 4.166020393371582, + "wallDistance": 0.45233625173568726, + } + + ns = NavierStokes(nu=0.01, rho=1.0, dim=3, time=False) + + phy_informer = PhysicsInformer( + required_outputs=["continuity", "momentum_x", "momentum_y", "momentum_z"], + equations=ns, + grad_method="finite_difference", + device=dist.device, + fd_dx=[ + (bounds[1] - bounds[0]) / nx, + (bounds[3] - bounds[2]) / ny, + (bounds[5] - bounds[4]) / nz, + ], + ) + + # Distributed learning (Data parallel) + if dist.world_size > 1: + model = DistributedDataParallel( + model, + device_ids=[dist.local_rank], + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + ) + + # Initialize the dataset + data_dir = to_absolute_path("./datasets/train/") + dataset = MeshDatapipe( + data_dir=data_dir, + file_format="vtu", + variables=["U", "T", "p", "wallDistance", "vtkValidPointMask"], + num_variables=7, + num_samples=cfg.train_num_samples, + batch_size=cfg.train_batch_size, + num_workers=1, + device=dist.device, + process_rank=dist.rank, + world_size=dist.world_size, + shuffle=True, + ) + + # Initialize the validation dataset + if dist.rank == 0: + pos_embed_tensor_val = ( + torch.from_numpy(pos_embed).to(torch.float).to(dist.device) + ) + pos_embed_tensor_val = pos_embed_tensor_val.repeat( + cfg.val_batch_size, 1, 1, 1, 1 + ) # repeat along the batch size dim + val_data_dir = to_absolute_path("./datasets/test/") + val_dataset = MeshDatapipe( + data_dir=val_data_dir, + file_format="vtu", + variables=["U", "T", "p", "wallDistance", "vtkValidPointMask"], + num_variables=7, + num_samples=cfg.val_num_samples, + batch_size=cfg.val_batch_size, + num_workers=1, + device=dist.device, + process_rank=dist.rank, + world_size=dist.world_size, + shuffle=False, + ) + + train_dataset_plotting = MeshDatapipe( + data_dir=data_dir, + file_format="vtu", + variables=["U", "T", "p", "wallDistance", "vtkValidPointMask"], + num_variables=7, + num_samples=16, + batch_size=cfg.val_batch_size, + num_workers=1, + device=dist.device, + process_rank=dist.rank, + world_size=dist.world_size, + shuffle=False, + ) + + optimizer = optimizers.FusedAdam( + model.parameters(), betas=(0.9, 0.999), lr=cfg.start_lr, weight_decay=0.0 + ) + + scheduler = torch.optim.lr_scheduler.ExponentialLR( + optimizer, gamma=cfg.lr_scheduler_gamma + ) + + # Attempt to load latest checkpoint if one exists + loaded_epoch = load_checkpoint( + "./checkpoints", + models=model, + optimizer=optimizer, + scheduler=scheduler, + device=dist.device, + ) + + for epoch in range(max(1, loaded_epoch + 1), cfg.max_epochs + 1): # epochs + with LaunchLogger( + "train", epoch=epoch, num_mini_batch=len(dataset), epoch_alert_freq=1 + ) as log: + for i, data in enumerate(dataset): + optimizer.zero_grad() + bs, _, chans = data[0]["x"].shape + + var = reshape_fortran(data[0]["x"], (bs, nx, ny, nz, chans)) + + mask = torch.permute(var[..., 6:7], (0, 4, 1, 2, 3)) + mask_dilated = dilate_mask_3d(mask, 3) + invar = torch.permute( + var[..., 5:6], (0, 4, 1, 2, 3) + ) # Grab Wall Distance + invar = torch.cat( + (invar, pos_embed_tensor), axis=1 + ) # Concat along channel dim + outvar = torch.permute( + var[..., 0:5], (0, 4, 1, 2, 3) + ) # Grab U components, T and P + pred_outvar = model(invar) + phy_losses = phy_informer.forward( + { + "u": pred_outvar[:, 0:1] * std_dict["U"] + mean_dict["U"], + "v": pred_outvar[:, 1:2] * std_dict["U"] + mean_dict["U"], + "w": pred_outvar[:, 2:3] * std_dict["U"] + mean_dict["U"], + "p": pred_outvar[:, 4:5] * std_dict["p"] + mean_dict["p"], + } + ) + + phy_loss = 0.0 + for key in phy_losses.keys(): + phy_loss += torch.mean(mask_dilated * phy_losses[key] ** 2) + + outvar = outvar * mask + pred_outvar = pred_outvar * mask + data_loss = F.mse_loss(outvar, pred_outvar) + loss = data_loss + cfg.phy_wt * phy_loss + loss.backward() + optimizer.step() + scheduler.step() + + log.log_minibatch({"Mini-batch data loss": data_loss.detach()}) + log.log_minibatch({"Mini-batch phy loss": phy_loss.detach()}) + log.log_epoch({"Learning Rate": optimizer.param_groups[0]["lr"]}) + + if dist.world_size > 1: + torch.distributed.barrier() + + if dist.rank == 0: + with LaunchLogger("valid", epoch=epoch) as log: + train_loss = validation_step( + model, + train_dataset_plotting, + pos_embed_tensor_val, + epoch, + plotting=True, + name="train", + ) + val_loss = validation_step( + model, + val_dataset, + pos_embed_tensor_val, + epoch, + plotting=True, + name="val", + ) + log.log_epoch({"Val loss": val_loss, "Train loss": train_loss}) + + if epoch % 20 == 0 and dist.rank == 0: + save_checkpoint( + "./checkpoints", + models=model, + optimizer=optimizer, + scheduler=scheduler, + epoch=epoch, + ) + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/README.md b/examples/cfd/external_aerodynamics/aero_graph_net/README.md new file mode 100644 index 0000000000..fde0e601c1 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/README.md @@ -0,0 +1,297 @@ +# AeroGraphNet for external aerodynamic evaluation + +This example demonstrates how to train the AeroGraphNet model for external aerodynamic +analysis of both simplified (Ahmed body-type) and more realistic (DrivAerNet dataset) +car geometries. AeroGraphNet is based on the MeshGraphNet architecture. +It achieves good accuracy on predicting the pressure and +wall shear stresses on the surface mesh of the respective geometries, as well as +the drag coefficient. + +1. [Problem overview](#problem-overview) +2. [Datasets](#datasets) + 1. [Ahmed Body](#ahmed-body) + 2. [DrivAerNet](#drivaernet) +3. [Model](#model-overview-and-architecture) + 1. [MeshGraphNet](#meshgraphnet) + 2. [Bistride Multiscale (BSMS) MGN](#bistride-multiscale-bsms-mgn) +4. [Training](#model-training) + 1. [Ahmed Body](#ahmed-body-training) + 1. [BSMS MGN](#bsms-mgn-training) + 2. [DrivAerNet](#drivaer-training) +5. [Inference](#inference) + +## Problem overview + +To goal is to develop an AI surrogate model that can use simulation data to learn the +external aerodynamic flow over parameterized car body shape. The trained model can be used +to predict the change in drag coefficient,and surface pressure and wall shear stresses due +to changes in the car geometry. This is a stepping stone to applying similar approaches +to other application areas such as aerodynamic analysis of aircraft wings, more complex +real car geometries, and so on. + +## Datasets + +AeroGraphNet currently supports two datasets: [Ahmed Body](#ahmed-body) and +[DrivAerNet](#drivaernet). + +### Ahmed Body + +Industry-standard Ahmed-body geometries are characterized by six design parameters: +length, width, height, ground clearance, slant angle, and fillet radius. Refer +to the [[2, 3](#references)] for details on Ahmed +body geometry. In addition to these design parameters, we include the inlet velocity to +address a wide variation in Reynolds number. We identify the design points using the +Latin hypercube sampling scheme for space filling design of experiments and generate +around 500 design points. + +The aerodynamic simulations were performed using the GPU-accelerated OpenFOAM solver +for steady-state analysis, applying the SST K-omega turbulence model. These simulations +consist of 7.2 million mesh points on average, but we use the surface mesh as the input +to training which is roughly around 70k mesh nodes. + +To request access to the full dataset, please reach out to the +[NVIDIA PhysicsNeMo team](mailto:physicsnemo-team@nvidia.com). + +### DrivAerNet + +DrivAerNet [[5](#references)] is a larger dataset which contains around 4000 high-quality +car meshes, coefficients and flow information. +The dataset can be downloaded by following the instructions on the [DrivAerNet GitHub](https://github.com/Mohamedelrefaie/DrivAerNet) +Please see the corresponding [paper](#references) for more details. + +## Model overview and architecture + +### MeshGraphNet + +The AeroGraphNet model is based on the MeshGraphNet [[1](#references)] architecture +which is instrumental for learning from mesh-based data using GNNs. + +### Bistride Multiscale (BSMS) MGN + +PhysicsNeMo BSMS MGN implementation is based on the BSMS GNN paper [[6](#references)]. +The model has two major building blocks: + +1. Bi-Stride Pooling and Adjacency Enhancement which precomputes different levels of meshes + consecutively from the input mesh as the top level. +2. Transition between levels which determines how to do message passing across levels, + computing the edge weight and node updating after pooling and returning. + +Depending on the dataset, the model takes different inputs: + +### Ahmed Body dataset + +- Ahmed body surface mesh +- Reynolds number +- Geometry parameters (optional, including length, width, height, ground clearance, +slant angle, and fillet radius) +- surface normals (optional) + +Output of the model are: + +- Surface pressure +- Wall shear stresses +- Drag coefficient - optional, computed using pressure and shear stress outputs. + +![Comparison between the AeroGraphNet prediction and the +ground truth for surface pressure, wall shear stresses, and the drag coefficient for one +of the samples from the test dataset.](../../../../docs/img/ahmed_body_results.png) + +The input to the model is in form of a `.vtp` file and is then converted to +bi-directional graphs in the dataloader. The final results are also written in the +form of `.vtp` files in the inference code. A hidden dimensionality of 256 is used in +the encoder, processor, and decoder. The encoder and decoder consist of two hidden +layers, and the processor includes 15 message passing layers. Batch size per GPU is +set to 1. Summation aggregation is used in the +processor for message aggregation. A learning rate of 0.0001 is used, decaying +exponentially with a rate of 0.99985. Training is performed on 8 NVIDIA A100 +GPUs, leveraging data parallelism. Total training time is 4 hours, and training is +performed for 500 epochs. + +### DrivAerNet dataset + +- Surface mesh + +Output of the model are: + +- Surface pressure +- Wall shear stresses +- Drag coefficient - optional, can be learned by the model along with other outputs. + +The input to the model is the original DrivAerNet dataset. It is recommended to enable +dataset caching (on by default) to speed up the subsequent data loading and training. + +![Comparison between the AeroGraphNet prediction and the +ground truth for surface pressure, wall shear stresses, and absolute error for one +of the samples from the test dataset.](../../../../docs/img/drivaernet_results.png) + +## Model training + +### Prerequisites + +This example also requires the `pyvista`, `shapely` and `vtk` libraries. Install with + +```bash +pip install pyvista shapely vtk +``` + +BSMS MGN model requires additional dependency: + +```bash +pip install sparse_dot_mkl +``` + +Additionally, if you are using the [PhysicsNeMo Docker Container](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/physicsnemo/containers/physicsnemo), +install a few system-level packages with + +```bash +apt install libosmesa6 libosmesa6-dev +apt install mesa-utils libgl1-mesa-dev +``` + +> [!NOTE] +> If you are running this example using the +> [PhysicsNeMo Docker Container](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/physicsnemo/containers/physicsnemo) +> you may have to pass a few additional flags to your `docker run` command. +> Specifically, include `--ulimit nofile=65535:65535` and `--shm-size=4g` as +> additional flags while launching the container. These flags are required +> because a process pool is used to parallelize dataset creation prior to +> starting training, which requires additional resources inside the container. + +### Running the experiments + +The example uses [Hydra](https://hydra.cc/docs/intro/) for experiment configuration. +Hydra provides a convenient way to change almost any experiment parameter, +such as dataset configuration, model and optimizer settings and so on. + +For the full set of training script options, run the following command: + +```bash +python train.py --help +``` + +In case of issues with Hydra config, you may get a Hydra error message +that is not particularly useful. In such case, use `HYDRA_FULL_ERROR=1` +environment variable: + +```bash +HYDRA_FULL_ERROR=1 python train.py ... +``` + +### Ahmed Body training + +The Ahmed Body dataset for this example is not publicly available. To get access, +please reach out to the [NVIDIA PhysicsNeMo team](mailto:physicsnemo-team@nvidia.com). + +To train the model, run + +```bash +python train.py +experiment=ahmed/mgn data.data_dir=/data/ahmed_body/ +``` + +Make sure to set `data.data_dir` to a proper location. + +The following example demonstrates how to change some of the parameters: + +```bash +python train.py \ + +experiment=ahmed/mgn \ + data.data_dir=/data/ahmed_body/ \ + model.processor_size=10 \ + optimizer.lr=0.0003 \ + loggers.wandb.mode=online +``` + +This will change the number of model message passing layers to 10, set learning rate to 0.0003 +and enable Weights & Biases logger. + +Data parallelism is also supported with multi-GPU runs. To launch a multi-GPU training, run + +```bash +mpirun -np python train.py +experiment=ahmed/mgn data.data_dir=/data/ahmed_body/ +``` + +If running in a docker container, you may need to include the `--allow-run-as-root` in +the multi-GPU run command. + +Progress and loss logs can be monitored using Weights & Biases. To activate that, +add `loggers.wandb.mode=online` to the train script command line. This requires to +have an active Weights & Biases account. You also need to provide your API key. +There are multiple ways for providing the API key but you can simply export it as +an environment variable + +```bash +export WANDB_API_KEY= +``` + +The URL to the dashboard will be displayed in the terminal after the run is launched. + +#### BSMS MGN training + +To train BSMS MGN, provide additional parameters, such as number of multi-scale layers, +and, optionally, location of the BSMS cache which would greatly speed up +the training process. +For example, for 6-layer BSMS model, use the following command line: + +```bash +python train.py +experiment=ahmed/bsms_mgn \ + data.data_dir=. \ + data.train.num_layers=6 \ + data.val.num_layers=6 \ + data.train.cache_dir=./cache_dir \ + data.val.cache_dir=./cache_dir \ + model.num_mesh_levels=6 \ +``` + +When trained using provided experiment, `ahmed/bsms_mgn`, results should look something like: + +| Model | RRMSE | +| :--- | ---: | +| Baseline MGN | 0.21 | +| Level 4 BSMS MGN | 0.16 | +| Level 6 BSMS MGN | 0.11 | + +### DrivAer training + +To train the MeshGraphNet model, run + +```bash +python train.py +experiment=drivaernet/mgn data.data_dir=/data/DrivAerNet/ +``` + +Make sure to set `data.data_dir` to a proper location. + +Another option is to train an extended version of MGN, called AeroGraphNet. This model +predicts a drag coefficient directly, along with pressure and WSS. +To use AGN instead of MGN, use `drivaernet/agn` experiment + +```bash +python train.py +experiment=drivaernet/agn data.data_dir=/data/DrivAerNet/ +``` + +## Inference + +Once the model is trained, run + +```bash +python inference.py +experiment=drivaernet/mgn \ + data.data_dir=/data/DrivAerNet/ \ + data.test.num_samples=2 \ + resume_dir=./outputs/ +``` + +Update experiment and data directory as needed. `resume_dir` directory should point +to the output directory of the training which contains model checkpoints. +This example will run inference for only 2 samples, this is just to demonstrate +how various options can be set from the command line. + +The inference script will save the predictions for the test dataset in `.vtp` format +in the output directory. Use ParaView or VTK.js to open and explore the results. + +## References + +1. [Learning Mesh-Based Simulation with Graph Networks](https://arxiv.org/abs/2010.03409) +2. [Some Salient Features Of The Time-Averaged Ground Vehicle Wake](https://doi.org/10.4271/840300) +3. [Ahmed body wiki](https://www.cfd-online.com/Wiki/Ahmed_body) +4. [Deep Learning for Real-Time Aerodynamic Evaluations of Arbitrary Vehicle Shapes](https://arxiv.org/abs/2108.05798) +5. [DrivAerNet: A Parametric Car Dataset for Data-driven Aerodynamic Design and Graph-Based Drag Prediction](https://arxiv.org/abs/2403.08055) +6. [Efficient Learning of Mesh-Based Physical Simulation with Bi-Stride Multi-Scale Graph Neural Network](https://arxiv.org/pdf/2210.02573) diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/config.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/config.yaml new file mode 100644 index 0000000000..4271cb9723 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/config.yaml @@ -0,0 +1,96 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - /visualizer@visualizers.mesh_p: mesh + - /visualizer@visualizers.mesh_wss: mesh + + - /logging/python: default + - override hydra/job_logging: disabled # We use rank-aware logger configuration instead. + - _self_ + +hydra: + run: + dir: ${output} + output_subdir: hydra # Default is .hydra which causes files not being uploaded in W&B. + +# Main output directory. +output: outputs/${now:%Y-%m-%d}/${now:%H-%M-%S} + +# The directory to search for checkpoints to continue training. +resume_dir: ${output} + +# The dataset directory must be set either in command line or config. +data: + data_dir: ??? + +# The loss should be set in the experiment. +loss: ??? + +# The optimizer should be set in the experiment. +optimizer: ??? + +# The scheduler should be set in the experiment. +lr_scheduler: ??? + +train: + batch_size: 1 + epochs: 50 + checkpoint_save_freq: 10 + dataloader: + shuffle: true + num_workers: 1 + pin_memory: true + drop_last: true + +val: + batch_size: 1 + dataloader: + shuffle: false + num_workers: 1 + pin_memory: true + drop_last: false + +test: + batch_size: 1 + dataloader: + shuffle: false + num_workers: 1 + pin_memory: true + drop_last: false + +compile: + enabled: false + args: + backend: inductor + +amp: + enabled: false + autocast: + dtype: torch.float16 + scaler: + _target_: torch.amp.GradScaler + enabled: ${..enabled} + +loggers: + wandb: + _target_: loggers.WandBLogger + project: car-cfd # aero-graph-net + entity: physicsnemo + name: agn + group: + mode: disabled + dir: ${output} diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/ahmed.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/ahmed.yaml new file mode 100644 index 0000000000..96147b0828 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/ahmed.yaml @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: physicsnemo.datapipes.gnn.ahmed_body_dataset.AhmedBodyDataset +_convert_: all + +name: ??? +data_dir: ${data.data_dir} +split: ??? +num_samples: ??? +# number of workers used by dataset during pre-loading (null - auto-select). +num_workers: null diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/bsms_ahmed.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/bsms_ahmed.yaml new file mode 100644 index 0000000000..252bb3d7f8 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/bsms_ahmed.yaml @@ -0,0 +1,37 @@ +# @package _global_ + +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - ahmed # use Ahmed experiment as a base and change only required parameters. + +_target_: physicsnemo.datapipes.gnn.bsms.BistrideMultiLayerGraphDataset + +# TODO(akamenev): is there a way to avoid duplication of configs? +dataset: + _target_: physicsnemo.datapipes.gnn.ahmed_body_dataset.AhmedBodyDataset + _convert_: all + + name: ${..name} + data_dir: ${..data_dir} + split: ${..split} + num_samples: ${..num_samples} + # number of workers used by dataset during pre-loading (null - auto-select). + num_workers: ${..num_workers} + +num_layers: 2 +cache_dir: ${.dataset.data_dir}/bsms_cache diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/drivaernet.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/drivaernet.yaml new file mode 100644 index 0000000000..aee3e0f942 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/data/drivaernet.yaml @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: physicsnemo.datapipes.gnn.drivaernet_dataset.DrivAerNetDataset +_convert_: all + +name: ??? +data_dir: ${data.data_dir} +split: ??? +num_samples: ??? diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/ahmed/bsms_mgn.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/ahmed/bsms_mgn.yaml new file mode 100644 index 0000000000..e4d361db29 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/ahmed/bsms_mgn.yaml @@ -0,0 +1,23 @@ +# @package _global_ + +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - ahmed/mgn # use MGN experiment as a base and change only required parameters. + - override /data@data.train: bsms_ahmed + - override /data@data.val: bsms_ahmed + - override /model: bsms_mgn diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/ahmed/mgn.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/ahmed/mgn.yaml new file mode 100644 index 0000000000..1a6cea43c3 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/ahmed/mgn.yaml @@ -0,0 +1,69 @@ +# @package _global_ + +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - /data@data.train: ahmed + - /data@data.val: ahmed + - /data@data.test: ahmed + - /model: mgn + - /loss@loss.graph: rrmseloss + - /optimizer: adam + - /lr_scheduler: exponentiallr + +data: + train: + name: ahmed_body_train + split: train + num_samples: 408 + val: + name: ahmed_body_val + split: validation + num_samples: 50 + test: + name: ahmed_body_test + split: test + num_samples: 50 + compute_drag: true + +train: + epochs: 500 + +visualizers: + mesh_p: + scalar: p + tag: pressure + camera_positions: + - [ + [-2.2, -0.7, 0.5], + [0.46, 0.76, -0.16], + [0.23, 0.01, 0.97], + ] + - [ + [-2.1, 0.86, 0.35], + [0.18, -0.36, 0.02], + [0.18, 0.05, 0.98], + ] + - [ + [-2.6, 0.5, -1.18], + [-0.04, -0.14, 0.58], + [-0.47, 0.31, 0.82], + ] + mesh_wss: + scalar: wallShearStress + tag: wall_shear_stress + camera_positions: ${..mesh_p.camera_positions} diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/drivaernet/agn.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/drivaernet/agn.yaml new file mode 100644 index 0000000000..90ed8df82b --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/drivaernet/agn.yaml @@ -0,0 +1,24 @@ +# @package _global_ + +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - drivaernet/mgn # use MGN experiment as a base and change only required parameters. + - /loss@loss.c_d: mseloss + +model: + _target_: models.AeroGraphNet diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/drivaernet/mgn.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/drivaernet/mgn.yaml new file mode 100644 index 0000000000..d3c478bf09 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/experiment/drivaernet/mgn.yaml @@ -0,0 +1,72 @@ +# @package _global_ + +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - /data@data.train: drivaernet + - /data@data.val: drivaernet + - /data@data.test: drivaernet + - /model: mgn + - /loss@loss.graph: rrmseloss + - /optimizer: adam + - /lr_scheduler: exponentiallr + +data: + train: + name: drivaernet_train + split: train + num_samples: 2766 + val: + name: drivaernet_val + split: val + num_samples: 593 + test: + name: drivaernet_test + split: test + num_samples: 595 + +model: + input_dim_nodes: 3 + processor_size: 10 + +train: + epochs: 50 + +visualizers: + mesh_p: + scalar: p + tag: pressure + camera_positions: + - [ + [-8.9, -4.5, 4.9], + [1.4, 0.11, 0.64], + [0.34, 0.1, 0.93], + ] + - [ + [-8.0, 5.3, 6.1], + [1.4, -0.004, 0.62], + [0.43, -0.17, 0.86], + ] + - [ + [-5.3, 4.1, -8.5], + [1.4, 0.11, 0.64], + [-0.8, 0.11, 0.65], + ] + mesh_wss: + scalar: wallShearStress + tag: wall_shear_stress + camera_positions: ${..mesh_p.camera_positions} diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/logging/python/default.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/logging/python/default.yaml new file mode 100644 index 0000000000..711984bafa --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/logging/python/default.yaml @@ -0,0 +1,48 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Standard Python logging configuration, as described here: +# https://docs.python.org/3.10/library/logging.config.html + +version: 1 +disable_existing_loggers: false + +output: ??? +rank: ??? +rank0_only: true + +formatters: + default: + format: "[%(asctime)s - %(name)s - %(levelname)s] %(message)s" + datefmt: "%H:%M:%S" + +handlers: + console: + class: logging.StreamHandler + level: ${...loggers.agnet.level} + formatter: default + + file: + class: logging.FileHandler + filename: ${...output}/train_${...rank}.log + level: ${...loggers.agnet.level} + formatter: default + +loggers: + agnet: + handlers: [console, file] + level: INFO + propagate: false diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/loss/mseloss.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/loss/mseloss.yaml new file mode 100644 index 0000000000..4f4af047c3 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/loss/mseloss.yaml @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.nn.MSELoss diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/loss/rrmseloss.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/loss/rrmseloss.yaml new file mode 100644 index 0000000000..62a7d0e253 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/loss/rrmseloss.yaml @@ -0,0 +1,17 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: utils.RRMSELoss diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/lr_scheduler/exponentiallr.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/lr_scheduler/exponentiallr.yaml new file mode 100644 index 0000000000..7c8e32d244 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/lr_scheduler/exponentiallr.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.lr_scheduler.ExponentialLR +gamma: 0.99985 diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/lr_scheduler/steplr.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/lr_scheduler/steplr.yaml new file mode 100644 index 0000000000..93a03de409 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/lr_scheduler/steplr.yaml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.lr_scheduler.ExponentialLR +step_size: 8 +gamma: 0.99985 diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/model/bsms_mgn.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/model/bsms_mgn.yaml new file mode 100644 index 0000000000..2fe9c4bc1b --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/model/bsms_mgn.yaml @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - mgn # use MGN model as a base and change only required parameters. + +_target_: physicsnemo.models.meshgraphnet.BiStrideMeshGraphNet + +num_mesh_levels: 2 +bistride_unet_levels: 1 diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/model/mgn.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/model/mgn.yaml new file mode 100644 index 0000000000..cb9bc7a393 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/model/mgn.yaml @@ -0,0 +1,33 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: physicsnemo.models.meshgraphnet.MeshGraphNet +_convert_: all + +input_dim_nodes: 11 +input_dim_edges: 4 +output_dim: 4 +processor_size: 15 +aggregation: sum +hidden_dim_node_encoder: 256 +hidden_dim_edge_encoder: 256 +hidden_dim_node_decoder: 256 +mlp_activation_fn: relu +do_concat_trick: False +num_processor_checkpoint_segments: 0 +recompute_activation: false + +# See MeshGraphNet implementation for more details and additional arguments. diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/optimizer/adam.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/optimizer/adam.yaml new file mode 100644 index 0000000000..60963e639a --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/optimizer/adam.yaml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.Adam +lr: 0.0001 +# weight_decay: 1e-4 diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/optimizer/fusedadam.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/optimizer/fusedadam.yaml new file mode 100644 index 0000000000..05c296d3e7 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/optimizer/fusedadam.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: apex.optimizers.FusedAdam +lr: 0.0001 diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/conf/visualizer/mesh.yaml b/examples/cfd/external_aerodynamics/aero_graph_net/conf/visualizer/mesh.yaml new file mode 100644 index 0000000000..1e9e4e046f --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/conf/visualizer/mesh.yaml @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: visualizers.MeshVisualizer +_convert_: all + +scalar: ??? +tag: ??? +camera_positions: ??? diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/inference.py b/examples/cfd/external_aerodynamics/aero_graph_net/inference.py new file mode 100644 index 0000000000..1a059dfde4 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/inference.py @@ -0,0 +1,178 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import logging +from pathlib import Path + +from torch_geometric.data import Data as PyGData +from torch_geometric.loader import DataLoader as PyGDataLoader + +import hydra +from hydra.utils import instantiate, to_absolute_path + +import numpy as np +import pyvista as pv +import torch + +from omegaconf import DictConfig + +from physicsnemo.distributed.manager import DistributedManager +from physicsnemo.utils import load_checkpoint + +from loggers import init_python_logging +from utils import batch_as_dict + + +logger = logging.getLogger("agnet") + + +def pyg_to_pyvista(graph: PyGData): + """ + Converts a PyG graph to a PyVista graph. + + Parameters: + ----------- + graph: PyGData + The input graph. + + Returns: + -------- + pv_graph: + The output PyVista graph. + """ + + pv_graph = pv.PolyData() + + # Assuming "pos" is in the source graph node data. + assert "pos" in graph, f"pos data does not exist" + pv_graph.points = graph.pos.numpy() + + # Create lines from edges. + edges = np.column_stack(graph.edge_index) + lines = np.empty((edges.shape[0], 3), dtype=np.int64) + lines[:, 0] = 2 + lines[:, 1:] = edges + + pv_graph.lines = lines.flatten() + pv_graph.point_data["p_pred"] = graph.p_pred.numpy() + pv_graph.point_data["p"] = graph.p.numpy() + pv_graph.point_data["wallShearStress_pred"] = graph.wallShearStress_pred.numpy() + pv_graph.point_data["wallShearStress"] = graph.wallShearStress.numpy() + + return pv_graph + + +class EvalRollout: + """MGN inference with a given experiment.""" + + def __init__(self, cfg: DictConfig): + self.output_dir = Path(to_absolute_path(cfg.output)) + logger.info(f"Storing results in {self.output_dir}") + + self.device = DistributedManager().device + logger.info(f"Using {self.device} device") + + # instantiate dataset + logger.info("Loading the test dataset...") + self.dataset = instantiate(cfg.data.test) + logger.info(f"Using {len(self.dataset)} test samples.") + + # instantiate dataloader + logger.info("Creating the dataloader...") + self.dataloader = PyGDataLoader( + self.dataset, + **cfg.test.dataloader, + ) + + # instantiate the model + logger.info("Creating the model...") + self.model = instantiate(cfg.model).to(self.device) + + # enable train mode + self.model.eval() + + # load checkpoint + load_checkpoint( + to_absolute_path(cfg.resume_dir), + models=self.model, + device=self.device, + ) + + # instantiate losses. + logger.info("Creating the losses...") + self.loss = instantiate(cfg.loss) + + @torch.inference_mode() + def predict(self, save_results=False): + """ + Run the prediction process. + + Parameters: + ----------- + save_results: bool + Whether to save the results in form of a .vtp file, by default False + + Returns: + -------- + None + """ + + for batch in self.dataloader: + graph, case_id, normals, areas, coeff = batch + assert len(case_id) == 1, "Only batch size 1 is currently supported." + + case_id = case_id[0].item() + graph = graph.to(self.device) + normals = normals.to(self.device)[0] + areas = areas.to(self.device)[0] + coeff = coeff.to(self.device)[0] + + logger.info(f"Processing case id {case_id}") + pred = self.model(graph.x, graph.edge_attr, graph) + gt = graph.y + pred, gt = self.dataset.denormalize(pred, gt, pred.device) + + num_out_c = gt.shape[1] + if num_out_c in [1, 4]: + graph.p_pred = pred[:, 0] + graph.p = gt[:, 0] + if num_out_c in [3, 4]: + graph.wallShearStress_pred = pred[:, num_out_c - 3 :] + graph.wallShearStress = gt[:, num_out_c - 3 :] + + error = self.loss.graph(pred, gt) + logger.info(f"Error (%): {error * 100:.4f}") + + if save_results: + # Convert graph to PyVista graph and save it + pv_graph = pyg_to_pyvista(graph.cpu()) + pv_graph.save(self.output_dir / f"{case_id}.vtp") + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + # initialize distributed manager + DistributedManager.initialize() + + init_python_logging(cfg, DistributedManager().rank) + + logger.info("Rollout started...") + rollout = EvalRollout(cfg) + rollout.predict(save_results=True) + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/inference_analysis/ahmed_body.ipynb b/examples/cfd/external_aerodynamics/aero_graph_net/inference_analysis/ahmed_body.ipynb new file mode 100644 index 0000000000..1ace122d7d --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/inference_analysis/ahmed_body.ipynb @@ -0,0 +1,532 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# AeroGraphNet inference (Ahmed Body)\n", + "\n", + "This notebook uses the [PhysicsNeMo AeroGraphNet checkpoint](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/physicsnemo/models/modulus_ahmed_body_meshgraphnet) to run inference on different Ahmed bodies. The training code and documentation for this checkpoint/model can be found on the GitHub repo [here](https://github.com/NVIDIA/physicsnemo/tree/main/examples/cfd/external_aerodynamics/aero_graph_net/). \n", + "\n", + "This notebook will use the model that was trained on a dataset of Ahmed bodies of various sizes to infer results and perform some scientific analysis on unseen Ahmed body like geometries. " + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Running Inference\n", + "\n", + "This notebook will use the `AhmedBodyDataset` util from PhysicsNeMo and PyTorch Geometric's `DataLoader` to prepare and load the data. \n", + "\n", + "The inputs to the model are:\n", + "* Ahmed body surface mesh\n", + "* Reynolds number\n", + "* Geometry parameters (optional, including length, width, height, ground clearance, slant angle, and fillet radius)\n", + "* surface normals (optional)\n", + "\n", + "Output of the model are:\n", + "* Surface pressure\n", + "* Wall shear stresses" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The input to the model is in form of a `.vtp` file and is then converted to bi-directional PyG graphs in the dataloader. The final results are also written in the form of `.vtp` files." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's begin by first downloading the model package." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "from pathlib import Path\n", + "\n", + "if not Path(\"ahmed_body_mgn.zip\").is_file():\n", + " !wget 'https://api.ngc.nvidia.com/v2/models/nvidia/modulus/modulus_ahmed_body_meshgraphnet/versions/v0.2/files/ahmed_body_mgn.zip'\n", + " !unzip ahmed_body_mgn.zip\n", + " !mv ahmed_body_mgn/* .\n", + " !rm utils.py inference.py # TODO: hacky, remove the old utils.py" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can see that the model checkpoint in the `checkpoints` folder under the name `MeshGraphNet.0.499.mdlus`. We also have a few sample dataset to do the inference on inside the `dataset` directory. Let's start with a few imports" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import sys\n", + "\n", + "import torch\n", + "import numpy as np\n", + "import wandb as wb\n", + "\n", + "from torch_geometric.data import Data as PyGData\n", + "from torch_geometric.loader import DataLoader as PyGDataLoader\n", + "\n", + "from physicsnemo.models.meshgraphnet import MeshGraphNet\n", + "from physicsnemo.datapipes.gnn.ahmed_body_dataset import AhmedBodyDataset\n", + "from physicsnemo.launch.logging import PythonLogger\n", + "\n", + "import pyvista as pv\n", + "\n", + "if sys.path[0] != \"..\":\n", + " sys.path.insert(0, \"..\")\n", + "\n", + "# Helper function to convert PyG graph to PyVista graph for visualization.\n", + "# PyVista graph can also be saved as .vtp file for visualization using tools like ParaView.\n", + "from inference import pyg_to_pyvista" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Next, let's write some steps to load the data and compute the model inference. This portion can be briefly broken down into three major steps, load the data, instantiate the model and load the trained weights and finally compute the model inference on the data. " + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING:root:Missing keys when loading MeshGraphNet: ['edge_encoder.model.5._extra_state', 'node_encoder.model.5._extra_state', 'processor.processor_layers.0.edge_mlp.model.5._extra_state', 'processor.processor_layers.1.node_mlp.model.5._extra_state', 'processor.processor_layers.2.edge_mlp.model.5._extra_state', 'processor.processor_layers.3.node_mlp.model.5._extra_state', 'processor.processor_layers.4.edge_mlp.model.5._extra_state', 'processor.processor_layers.5.node_mlp.model.5._extra_state', 'processor.processor_layers.6.edge_mlp.model.5._extra_state', 'processor.processor_layers.7.node_mlp.model.5._extra_state', 'processor.processor_layers.8.edge_mlp.model.5._extra_state', 'processor.processor_layers.9.node_mlp.model.5._extra_state', 'processor.processor_layers.10.edge_mlp.model.5._extra_state', 'processor.processor_layers.11.node_mlp.model.5._extra_state', 'processor.processor_layers.12.edge_mlp.model.5._extra_state', 'processor.processor_layers.13.node_mlp.model.5._extra_state', 'processor.processor_layers.14.edge_mlp.model.5._extra_state', 'processor.processor_layers.15.node_mlp.model.5._extra_state', 'processor.processor_layers.16.edge_mlp.model.5._extra_state', 'processor.processor_layers.17.node_mlp.model.5._extra_state', 'processor.processor_layers.18.edge_mlp.model.5._extra_state', 'processor.processor_layers.19.node_mlp.model.5._extra_state', 'processor.processor_layers.20.edge_mlp.model.5._extra_state', 'processor.processor_layers.21.node_mlp.model.5._extra_state', 'processor.processor_layers.22.edge_mlp.model.5._extra_state', 'processor.processor_layers.23.node_mlp.model.5._extra_state', 'processor.processor_layers.24.edge_mlp.model.5._extra_state', 'processor.processor_layers.25.node_mlp.model.5._extra_state', 'processor.processor_layers.26.edge_mlp.model.5._extra_state', 'processor.processor_layers.27.node_mlp.model.5._extra_state', 'processor.processor_layers.28.edge_mlp.model.5._extra_state', 'processor.processor_layers.29.node_mlp.model.5._extra_state']\n" + ] + } + ], + "source": [ + "import os\n", + "\n", + "from utils import relative_lp_error\n", + "\n", + "\n", + "class AhmedBodyRollout:\n", + " def __init__(self, wb, logger):\n", + " # set device\n", + " self.device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + "\n", + " logger.info(f\"Using {self.device} device\")\n", + "\n", + " # instantiate dataset\n", + " self.dataset = AhmedBodyDataset(\n", + " name=\"ahmed_body_test\",\n", + " data_dir=\"./dataset\",\n", + " split=\"test\",\n", + " num_samples=4,\n", + " compute_drag=True,\n", + " )\n", + "\n", + " # instantiate dataloader\n", + " self.dataloader = PyGDataLoader(\n", + " self.dataset,\n", + " batch_size=1,\n", + " shuffle=False,\n", + " drop_last=False,\n", + " )\n", + "\n", + " # instantiate the model\n", + " self.model = MeshGraphNet(\n", + " 11,\n", + " 4,\n", + " 4,\n", + " aggregation=\"sum\",\n", + " hidden_dim_node_encoder=256,\n", + " hidden_dim_edge_encoder=256,\n", + " hidden_dim_node_decoder=256,\n", + " )\n", + " self.model = self.model.to(self.device)\n", + "\n", + " # enable train mode\n", + " self.model.eval()\n", + "\n", + " self.model.load(\n", + " \"./checkpoints/ahmed_body/MeshGraphNet.0.499.mdlus\", strict=False\n", + " )\n", + "\n", + " def predict(self, save_results=False):\n", + " \"\"\"\n", + " Run the prediction process.\n", + "\n", + " Parameters:\n", + " -----------\n", + " save_results: bool\n", + " Whether to save the results in form of a .vtp file, by default False\n", + "\n", + "\n", + " Returns:\n", + " --------\n", + " None\n", + " \"\"\"\n", + "\n", + " self.pred, self.exact, self.faces, self.graphs = [], [], [], []\n", + "\n", + " for i, (graph, sid, normals, areas, coeff) in enumerate(self.dataloader):\n", + " graph = graph.to(self.device)\n", + " normals = normals.to(self.device, torch.float32).squeeze()\n", + " areas = areas.to(self.device, torch.float32).squeeze()\n", + " coeff = coeff.to(self.device, torch.float32).squeeze()\n", + " sid = sid.item()\n", + " logger.info(f\"Processing sample ID {sid}\")\n", + " pred = self.model(graph.x, graph.edge_attr, graph).detach()\n", + "\n", + " gt = graph.y\n", + " graph.p_pred = pred[:, 0]\n", + " graph.wallShearStress_pred = pred[:, 1:]\n", + " graph.p = gt[:, 0]\n", + " graph.wallShearStress = gt[:, 1:]\n", + "\n", + " error = relative_lp_error(pred, gt)\n", + " logger.info(f\"Test error (%): {error}\")\n", + "\n", + " if save_results:\n", + " # Convert PyG graph to PyVista graph and save it\n", + " os.makedirs(\"./results_nbk\", exist_ok=True)\n", + " pv_graph = pyg_to_pyvista(graph.cpu())\n", + " pv_graph.save(os.path.join(\"./results_nbk\", f\"graph_{sid}.vtp\"))\n", + "\n", + "\n", + "logger = PythonLogger(\"main\") # General python logger\n", + "logger.file_logging()\n", + "\n", + "logger.info(\"Rollout started...\")\n", + "rollout = AhmedBodyRollout(wb, logger)\n", + "rollout.predict(save_results=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Post Processing\n", + "\n", + "Once the results are written, we can visualize them using `trame` which supports interactive visualizations in notebooks.\n", + "\n", + "This might require installing a few additional dependencies, which can be done by uncommenting the below code block:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [], + "source": [ + "# %%capture\n", + "# %pip install trame-vtk trame-vuetify trame-jupyter-extension" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Once you have verified that you have the right dependencies, the results can be visualized by running the following:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCASwBLADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDrNO0zwdovw08M6pf+D7DUJ7u1sodsOnwPNNNKigEl8ZJY8knvVu2j8DtqNrZal8PItIe6kEUEl/pFuI5JD0QOhYBjjgHGe1U9Rlkh+EfgKWGBriVJ9HZIUYKZGHlkKCxABPTJIFbOpx+I/Fs2nWM/h19IsYL6G8nubm7ikciJg4VFjZuSQBkkYGetAG3/AMIJ4P8A+hU0P/wXQ/8AxNH/AAgng/8A6FTQ/wDwXQ//ABNclpF1qFqPGPia+1bULm30e+vvs2niY+UVRM4bqT6AdFxnFZlrfzX+nRX17qfjxNXmQS+ZZ6XcLbRMRkKkXl7GQdPmyT1zzQB6B/wgng//AKFTQ/8AwXQ//E1m674e8D+H9KbUbvwjpDwrLFEVi02AtmSRYx1A4ywz7ZrKi8QeINesPDGkS/aNF1DVFuGvpzAYpVjgIB8tHHymTcpBI+UE1R8feGrzRfD0c9nr+pXNo99aLdW2o3BnDDz4yrIzcqwYDgHBGeKAOmtPDPgu71bUNOXwbpSSWPl75H0yEJJvXcNhxzjv05q//wAIJ4P/AOhU0P8A8F0P/wATXL6r4k1DRNT8dXEEhmktzp8VpHM5MUTyqEBx2G5gxxjOK1pPBWqJa/aLbxhrX9sAbhPNKGt2fuDBjYEPoOQO9ABqHh7wPpuqaVp83hHSGl1KV4YSmmwFVKxtId2RwMKemeafaeGfBd3q2oacvg3SkksfL3yPpkISTeu4bDjnHfpzWJ4v8PC98ZeEJLu/1GO4uriVJhaX0scaMts5JiAb5MkdRyQTnqaTUNfv/D9742e2mkne1GnW9mlzIzpG8qhAxz2ywY+uKAOr/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JrHvvCeq6fpM+o2ni3WH1iCJpvMuJg1vK4GSrQ42qh5HGCPXismC81Hxl4u0cxarf6Zp174cjv57e1lKks0gwAf4Tz94DJAxnBoA67/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiazNEa90PxzN4bk1K71Cwn077dbveyeZLCyyBGTf1ZTuUjOSOa6TRYNRtdGtYdXvEvNQRMT3CIEEjeoAAA/KgDjNasfBej61BpMXw9ttSvJrdrkJY6XanbGrBSTvK92H51Lomn+BNavptPbwTYafqMMYla0vtJhjkMZON64BDLnjIJwetN16+vdP+LGny2Gkz6pKdDnUwwSxxlV8+I7syMoxwBjOeah1G31+9uNW8UX9gdGWy0O6trSEXKyTs7AOZGZDhQNi4AJOcnigDpf+EE8H/wDQqaH/AOC6H/4mj/hBPB//AEKmh/8Aguh/+JripdR1jw18P9GvG1bU77Vdee1gacxm4NsGjLsYogDuYKCOcljgnNVrrVLvR4lv9Afx3e30TKXtNR0+5liulyNy/MmI2xnBXAB7YoA77/hBPB//AEKmh/8Aguh/+Jo/4QTwf/0Kmh/+C6H/AOJrj/EM+o23i28uNW1nWNGsjJB/ZV5CpexRcLuWdRxuLbhl8DBGCKXxHNqNp4vvZ9W1rV9Hs/Mg/su9gBexVcLvWdRxlm3DL4GCMEUAdf8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTVbQru5m8e+LbaW4leCD7H5MTOSse6Ik7R0GT1x1rk/GOrapb/APCwBZ6jcxPawab9m2zMBCzsd23B+XPGcdaAO1/4QTwf/wBCpof/AILof/iaP+EE8H/9Cpof/guh/wDia5bxLeah4a/sbw+mq63cyapJNPeX8Fubm5VEVdyxIqkICWGMDCjPc1UtNVvNL1vTW0YeMry3uLlIL221awuHQRsceasjplCpIJGcEZ4FAHaf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAcH408F+FbXwL4huLfw1o0M8WmXLxyR2ESsjCJiCCFyCDzmofC/hfwbF8OdE1PU9A0IKNKt5ri5uLKL/nkpZmYr16kk10Xjv/AJJ54l/7BV1/6KauNuArfDT4dpd/8gtpNOF9n7mzyfk39tvmeXnPHSgCVG8EzIbi3+Gcs1h1F4mgRbGX+8qnEhHfha6LTPC/gLWNOh1DTvDvh+4tZhuSRNPiwex/h4IPBB5Bq9qetavZXzwWnhW/1CFQCLiG5tkVsjnh5Fbjp0rh/EmrQ618OvFFva6NJo0lvewxT/6o7p2mjZmBjJVmGRk5znrzQB2v/CCeD/8AoVND/wDBdD/8TR/wgng//oVND/8ABdD/APE1gapotpa6v4e8IW73FvpN4Lm5u8Tv5l20YTCNJncd28s3PIXHTNOvtHsvBviLw9PoCG0j1C9+xXVlG58qZDG7b9hOAylAdw7E5oA3f+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJrP8Ahw6x/DLS2dgAkMm4nth3z/KuJ0fSodZ034XWdy8ot20+5MqRyFDIojT5SRzgnGR3HHegD0b/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4msW10y28J/EDTrHR42t9P1SxuXls1cmMSxGMq6qThSQ5BxweKqeFPDGleL/DNr4h16Nr/AFLUVaZ5WmcfZ8k/u4sEeWE6cc5BzQBqaV4d8D6w1+tv4R0hTY3b2cnmabCMuoUkjAPHzDrj6Vh6ivhHTdZi0t/haZZp3kS3aLS7IrPsGWK5kBxjnkCtL4Xwm2sPEVu17JemLXbmP7RI2532qgyx7njBPc1e8Qf8lB8HfW9/9EigCxbeCvCNxawzN4O0eBpEVzFLp0G9CRna2ARkdDgke9S/8IJ4P/6FTQ//AAXQ/wDxNYGk6Haah8VPFGoXfmSNZyWht4/MIRHMAy+0HBboBnOOcdav/Dh1j+GWls7ABIZNxPbDvn+VACav4d8D6LHZvc+EdIYXV3FZp5emwnDyNtUnIHGev8qLLw74HvtX1PTIvCOkCbTmjWZm02Ha29N428ZPB5yBXDjQNJ1L4Z/D+5vdOtridrjT7YySRgkxM/zJn0OTxW7pXgzRNR8eeKbe7sxJYWgs4rey3FYU/cDnYDgkAADPTnHWgDrP+EE8H/8AQqaH/wCC6H/4mj/hBPB//QqaH/4Lof8A4msbw5a3aQ+LvDFlqM8C2Nx5NhcuTK9sssCOoG45YIznAJ6YGa7aJGSFEd97qoDPjG4+tAHmFtP4Mu7Q3sHwtkksQzj7Smj2jqQjFWIVXLEZU/w54rqrDwn4G1TT7e/svDWgzWtxGJIpF06LDKRkH7tcz4HvPFo8HQW+laNpbQedciK7utQcf8t5OWiWI9DnjdzjqKg0zwTZweOdM0W9mluoNO0CNmUMUWaT7Q53MAeQCSQp46elAHa/8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXCRJN4m1bWrzVvB9/ryw6jPaW4F5AkNvHG20BY3lUhjjcWIyc8HGKQaT4g1PRJNLlsJJ7Kw1VZItKvdSiaa5tvLyYGdHblGYMA55AXNAHef8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNcC+k22raZph07TL3VdJ02W7jvNAu7kJcW8hYY2gkBvL5Cgt0YYNbHhG4srjxnYyafdXNzaN4ajWKW7OZmC3DAh+B8wPB96AOm/4QTwf/ANCpof8A4Lof/iaP+EE8H/8AQqaH/wCC6H/4muX8SureJvGgBBK+FFDexzcH+oqhe2UnhzwD4ettKjvZrnW7i1hvpobkLPKDEzkK7sAhO3aMEAA4HOKAO3/4QTwf/wBCpof/AILof/iaP+EE8H/9Cpof/guh/wDia4abTbzR7mxvtA8F32h3SXUSyyPqVsIriNmAdJR5x3kgnBwW3AYr1mgDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJrD8aeC/Ctr4F8Q3Fv4a0aGeLTLl45I7CJWRhExBBC5BB5zXeVz/jv/knniX/ALBV1/6KagDI0XQv7e+GvgqL7T5H2SHTb3Ozdv8AKVH29RjOMZ7ehrt65/wJ/wAk88Nf9gq1/wDRS10FAGJo/hyLTLfWIJ5Vu4tTvZrp0aPACyYBQjJyMDrxnPSsmHwfrdjajTdO8X3VvpKjZHE1qkk8Mf8AcSYngAcAlWI454rsaKAObvvBdjPpGm2dlPcWFxpZ3WN5E26WJsYOS2d4bJ3A9ayNU8Bar4hjiTXfFUlylvNHPbxwWawIro4bc4DEuSAR1AG4nGcV3dFAHNyeD7S7vvEcl+4ubXW0hSS32bfLEabchs8nuDgYIqi/hHxBNbf2fN41vG0zGxglqi3TJ/dM4PXHG4KD712VFAGJfeHI7rUvD91FOYY9Hld1iKlzIGiaMDcTkY3ZzznH41Wl8HWl5e+I5L+T7Ra62kKSQbNpjEabchs8nPIOBgiukooA4ybwdrl5Ztpd94xup9JdfLkjFoiXEkfTY0wPccEhQTzzzWvb+GobTxRFq9vIscEOmLp0doseAqh9wIOegHGMfjW5RQBjtoW7xjF4g+0/6vT3svI2dd0ivu3Z/wBnGMd+tWdFsrvTtGtbO+1B9Quok2yXTpsMp9SMnH51fooAx30Lf4xh8QfaceVp8ll5GzrukR927PbZjGO/WrmrWP8AamjX2n+Z5X2q3kg8zbu27lK5xxnGauUUAYM/hW1vPCllodzPLmzjhEN1D+7kjkjACyJ12nI9+pHIqh/wier37wxa94mlv7CJ1k+zQ2i2/nFSCPNZSSwyM4G0HvXW0UAcZrPga81WbUrdfEVxDo+qOHvLJoRI3QBhHIT+7DBRkYPfGM0mr+BbvU5NRtY/ENxBoupuGu7FoRI3QBljkJ/dqwUZGD3xjNdpRQBzOp+Frt9bbWNC1htKvJoUguVa3WeKdUJ2EqSCGGSAQehrIb4bPLY+IY59enuLvXPszT3M0AO1omzwoIGCMADjAA613tFAGP4g8Px65HbSJdTWV/ZyGW0vIMb4mIwRgghlI4Kng1QtPDGpTalbXuv6++pC0fzLe3ithbxK+CN7gEl2APGTgdcV09FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imqHwhZ22ofDDw/Z3kEc9tNo9skkUi5VlMK5BFTeO/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0AUv+EDiRPIg8R+IoLHoLSO++RV/uhypkA+jVpXXhPSrjwwfD0MLWmn5QhbcgEFXD5yQcksOSck5PfmtuigDM1vQbLX7WOG8EqvDIJYJ4JDHLC44DIw5BwT7HPOapaZ4RtbDVE1O5v8AUNTvokMcM1/MH8lT97YqhVBPAJxkgda6CigDlJfAGmvLcpFqGq29hdSNLPp0Fztt5GY5bjG5QxzkKwByeKt2Hg/TtN/sL7PJc40WCSC2DODuV1CnfxycKMYxXQUUAZ1zo1vda7p+ru8ouLGKaKJVI2ES7N2RjOfkGMEd+tY9x4Fs2uLmSw1XV9LiunMk9vYXISN3P3mAKnYT3KFa6migDH8OeGdN8K2U9npUbx2807TlGbdtYgA4PXHyjrk9eanvNGt77WNN1OV5RPp5lMSqRtbzF2ndxk8dMEVo0UAZ1lo1vY6vqepxPKZtRaNplYjauxNg28ZHA5yTWLL4A015blItQ1W3sLqRpZ9OgudtvIzHLcY3KGOchWAOTxXV0UAc+ng/To9B0fR0kuRbaVNDNbneNxMRyoY45HrjH4VfstGt7HV9T1OJ5TNqLRtMrEbV2JsG3jI4HOSa0aKAMSXwxaSDX9tzdxPrYHnvHIFaIiIRAxnHBwoPOefyrYijEMKRBmYIoUFjknHrT6KAM7Q9Gt9A0iLTbR5XhiZ2DSkFsu7OegA6se1A0a3HiNtc3y/amtBZlMjZsDl84xnOT6/hWjRQBzt/4PtbrUp9Qs9R1LSrm4x9pawnCLOQMAsrKy7sDG4AHHeo5fAmkHTbS0tnu7Oa0na5ivYJv9IErAh3Z2B3FgSDuBB/AV01FAHIL8O9MhWGSzv9Us7+Myl9QgnAnn81gz+YSpDZIB6cYGMVYk8CaSLHTLexlvNOl02Nora6tJtsqo33gxIIYEgE7geea6eigDlbXwDpdqdWf7VqE1xqtmbS7uJ5w8jr8w3ZIwGw2BxgADita48P6feeH49Eu4mms440jXcxDjZjawZcEMCAcjHNalFAHN2ngy1hv7a7vtT1XVWtW320d/cB0hfoGCqo3MOcFskZ4rpKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiio2nhT78sa/VgKL2Akoqo+qWCHDXtuCO3mCoH1/S4/vXsf4ZP8qzdWmt5L7yXOK3ZpUVjt4o0gf8ALyT9Eb/CoX8X6Wo4Mzf7qVP1il/MiHXpr7SN6iubbxpYfwwXJ+qgf1qJvG0A+5ZSn6sBU/WqP8xLxNJfaOporkD43f8Ah08D6zf/AFqibxrdH7tnCPqxNT9co9yfrlHv+DO0orhG8Y6mekVqv/AG/wAajbxZqzdHhX6R/wCNS8dRJ+u0vM7+ivP08V6sv3pI3/3ox/Spl8Yakp5jt2+qH/Gl9fogsbT8zuqK4tfGd2PvWsB+hIqVfGsn8Vgp+kuP6VSx1DuUsXS7nX0Vyq+NYz96xcfSQH+lTL4zs/4ra4H0wf61SxdF/aKWJpPqdJRWAvi/TW6rcL9UH9DUq+KtKPWV1+sZq1iaT+0ilXp/zI2qKyl8SaQ5wLsA+6MP6VKuuaY/S+h/FsfzqlWpvaS+8pVYPZo0KKqLqlg/3b23P/bUf41Mtzbv92eJvo4NUpRezKUk9mS0UgYN0IP0paoYUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/jv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtHjv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRUbTwp9+WNfqwFQPqlghw17bgjt5gqXOK3YnJLdluis19f0uP717H+GT/KoW8UaQP8Al5J+iN/hUOvSW8l95DqwW8kbFFYL+L9LUcGZv91KhbxpYfwwXJ+qgf1qXiaS+0S8RSX2kdJRXLN42gH3LKU/VgKhPjd/4dPA+s3/ANapeLor7RLxdFdTr6K4tvGt0fu2cI+rE1A3jHUz0itV/wCAN/jUvG0V1JeMpHd0VwDeLNWbo8K/SP8AxpE8V6sv3pI3/wB6Mf0qfr9En67T8z0CiuFXxhqSnmO3b6of8amXxndj71rAfoSKax1HuUsZSO0orkF8ayfxWCn6S4/pUq+NYz96xcfSQH+lUsbQf2vzKWKpdzqqK5tfGdn/ABW1wPpg/wBalXxfprdVuF+qD+hq1iqL+0UsRSf2jforFXxVpR6yuv1jNSr4k0hzgXYB90Yf0qlXpP7S+8r21P8AmRq0VnrrmmP0vofxbH86lXVLB/u3tuf+2o/xqlUg9miueL6luiolubd/uzxN9HBqQMG6EH6VV0yri0UUUwCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/x3/yTzxL/wBgq6/9FNXQVz/jv/knniX/ALBV1/6KagA8Cf8AJPPDX/YKtf8A0UtdBXP+BP8Aknnhr/sFWv8A6KWugoAKKKKACiiigAooooAKKKKACiiigDL1rVzpEMTiHzTIxABbGP0Nc+fGl0w+W0hU+7E1d8Zrm1tT6Of5Vx4XgZxXkYvFVKdVxi9DzsRWqRqWi9DdPi/UyT8tsB2wh/xqCTxPqzMMXSoB1Cxr/UVkYX160vA5yK43iqz+0zm9rUf2maD67qkg5vZhn+7gf0qu2pX7ffvbg59ZW/xquMcd6UEDFZutN7ti55PdivLLKcvJI/P8TE0zAzn196XI4780uR1x3qG2TuIOo45B9KVAC65AOT6CgMOn8qFbkcZoV7giwkiJGVEcLEx4yRnB9aia3G87d33gACP6daf5nGN4HyjhBmnZUn7pI3AAfhnH+RW1lvYe6syEQg9wDzxkdv8AJprxEfd54Ht1qxvC8fKo+bp1698f54pG+YgAOfu8Zx370RWuwnFWKpikHVl7jjH/ANb3poST2PI6e9WiAD0QEbh835c0qgM2Qzk8ZI4/n/8AWrW0exHIVSkozlBxz/n0puXHBXB7j0q2UGzlccEjefU9aUtkn51zljlVyBS5Y9g5fMqZfODG2fTj0zTDMFAypGcYHc59qvgfMBhj838Tbe3+f5VGAoCkeWowoYjk/wCf89qahT6oTi+5VEysCcNgYySp4zT8/LuwcYznFXI4920jeeF6DA6n6fn9aTylC8xKOhBduvPU/wCfSpcIPYpRkVUJkYKg3MegHJpzKycMjA+4xVyAmJ98T4Y7RvCnP3vY/wD1jU3mzOQWZ24TkqD3P94modK70ZpGCtruZeaM1prOyxCNmUq4AYNtGeD2C5/Wqf2dNpPzHjOAuP4c5yT/AJ/GkqT6kyja3KQZFGRVl4AMjYExnq/07f54zTTbhjw/8WPkGf4scde3+TR7LW1xWZDxRxR9mkKqRvOR14HYn09v/wBVNMMinDSov459Pp6mr+ry7kXfYcODkVKtzOgws8ij2cioBDLuxy3rtT6+v0o8qcdVUDuScY4H+P6UKhUWwc7XQtrqV8pyt7cD/tq3+NTJrepp0vpj9Wz/ADrLy+cfL+BJ9fb2pR5pbaImPIH48ev1FUoV1s395SrNbNmuviPVl6Xjfiqn+lSr4p1Zes6t9Y1/oKw2ZkGXUqOvJApN7ZICOcdcKT6/4H8qd8SurH9YmvtM6NfF+pDqsDfVD/jUieMr4fft7dvoCP61zO5s4KOD7qab58f94fnT9tiV1ZX1qovtHWr41lH3rJD9JCP6VMvjZf4rAjntLn+lcb5qf3h+dL5i+tV9axC6/gUsZV/mO2Xxpa/xWsw+hBqRfGWnn70Nyv8AwEH+tcLvX1pdw9aaxtf+kUsbU7nfr4t0turyr9Y/8KlXxPpDHH2rH1jb/CvO8j1ozVLH1eyLWOqeR6Suv6U44vYvxyP51Kusaa3S/tvxkArzGjI9atZjP+VFfXpdkeqLe2jnC3ULfSQGpRIjfddT9DXkZljXq6j8aabmLs2foM1osdN/Y/r7g/tFLdfiewUV5El5MuBCJ/bbkVYjvNXIxE1wo/66kVTx6XxK3zRUcwT2iz1WivNoZ/EZxtvJVHu5b+dXon8RcbtTcf8AAVP8xWUs5wkfikvzOmFWpPanI7uiuatNQ1SLb588cwHXcmCfyrag1GGbAY+W3ox4/OqoZxg68uSM9fPQ6/Z1LXcbFuiiivTICiiigAooooAKKKKAOf8AHf8AyTzxL/2Crr/0U1HgT/knnhr/ALBVr/6KWjx3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+iloA6CiiigAooooAKKKKACiiigAooooAKy9a1c6RDE4h80yMQAWxj9DWpXMeM1za2p9HP8AKsMTOUKTlHcyrScaba3KR8aXTD5bSFT7sTUJ8X6mSfltgO2EP+NYQXgZxSYX1614rxlZ/aPLdeq95GvJ4n1ZmGLpUA6hY1/qKhfXdUkHN7MM/wB3A/pWfwOcigY471m8RVf2mJ1JveT+8sNqV+33724OfWVv8aheWWU5eSR+f4mJpAQMUmRx35rNzk9ybt7sTAzn196UdRxyD6UuR1x3oDDp/KpuxaAgBdcgHJ9BVhJESMqI4WJjxkjOD61XVuRxmpvM4xvA+UcIM1pFXTTQ07O6GNbjedu77wABH9OtIIQe4B54yO3+TU2VJ+6SNwAH4Zx/kUbwvHyqPm6devfH+eKtryDlRXeIj7vPA9utMMUg6svccY/+t71ab5iAA5+7xnHfvTSAD0QEbh835c1cUraohxVyqEk9jyOnvSlJRnKDjn/PpVpQGbIZyeMkcfz/APrUhQbOVxwSN59T1quWPYXIVMuOCuD3HpS5fODG2fTj0zVstkn51zljlVyBSgfMBhj838Tbe3+f5UuSHYOV9ygZgoGVIzjA7nPtSiZWBOGwMZJU8Zq0AoCkeWowoYjk/wCf89qljj3bSN54XoMDqfp+f1puFOwlGT6lPPy7sHGM5xSoTIwVBuY9AOTVrylC8xKOhBduvPU/59KkgJiffE+GO0bwpz972P8A9Y1m6aezLjF3VymysnDIwPuMU3NanmzOQWZ24TkqD3P94mkWdliEbMpVwAwbaM8HsFz+tR7KWhpyRvu/u/4JmZoyKn+zptJ+Y8ZwFx/DnOSf8/jTngAyNgTGer/Tt/njNP2XmZ2ZWyKXipjbhjw/8WPkGf4scde3+TUP2aQqpG85HXgdifT2/wD1U40G9mS7roHFA4ORTTDIpw0qL+OfT6epoEMu7HLeu1Pr6/Sq+ry7i5n2J1uZ0GFnkUezkVIupXynK3twP+2rf41U8qcdVUDuScY4H+P6UzL5x8v4En19van7Kqtn+I/aNGomt6mnS+mP1bP86kXxHqy9LxvxVT/SsgeaW2iJjyB+PHr9RQzMgy6lR15IFVbELq/vH7eS6v8AE3F8U6svWdW+sa/0FTL4v1IdVgb6of8AGuc3tkgI5x1wpPr/AIH8qXc2cFHB91NPnxK6spYmf8zOmTxlfD79vbt9AR/WpV8ayj71kh+khH9K5Lz4/wC8PzpfNT+8PzprEYldSli6nSR2S+Nl/isCOe0uf6VKvjS1/itZh9CDXE+YvrRvX1prGYhdfwKWMq9zul8ZaefvQ3K/8BB/rUy+LdLbq8q/WP8AwrgNw9aMj1qljqyKWNqeR6IvifSGOPtWPrG3+FSrr+lOOL2L8cj+debZoqlmFTqkWsdPsj05dY01ul/bfjIBUy3to5wt1C30kBryvI9aYZY16uo/GrWYTf2R/X2t0euCRG+66n6GnV4+bmLs2foM09LyZcCET+23IrVY2XWH4jWZR7fieu0V5VHeauRiJrhR/wBdSKuQz+IzjbeSqPdy386l5nRh8bS+aNYYtz+GDfyPSaK4SJ/EXG7U3H/AVP8AMVrWmoapFt8+eOYDruTBP5VjLPcFHeX4HZTVWf2GvW3+Z0tFVINRhmwGPlt6MePzq3XpUMRSrx56Uk0XKLi7MKKKK2JCiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigDmvGWRY2xGP9bj9K43eT/+uuz8ZD/iW25x/wAtsfoa4o9ea8LHpe3Z5WLv7Vic468UevWl9OKPWuS5yWDoaAORgUfTpR+tK47B2H8qAORgZNKO3ajrjv8AWi4WEHbmlAGOmRigcY7cUY479O9Fx2JicbhuUcAYQe9KR8x4A56t9P5fh2/IJ5IBHUcKPfr7+lC4LYAUkMSQ546d66B2EzgAblX5W4Az37//AKqdgn5drEEjqetNziMgMq/IeFGSfrTiPmPyk/MM7zTQWEyFGMqvD8Dk9eh/z0p5BLf8tD8w4PA6Z6fmaYSQp+cAFW4Aznnv6U7aC3K8Bud5x27+3/1qLgN+6Mny1OOSfr/OnsdwPLkfMen+NNVtoA3KvA4UZPXoaVgTk7XP3icsBjH07j+dFwFPDDcMNv8A4mz/AA/5/Glj5K7S3G3hFxnr0zSfLvO0oMN/ACf4e2PxoUblBwxAwOTtU8d6aeowGcDI4wn324xk9vSnI4UgKQPu42rk9fX/AD+lNTG4bdqthTwNzdT+vTNKucgAv0UY6DqfxoQK49WPGVY4C8khe/tzSKULDhdwC5H3yOTTQAGGdgPynByxzn0/+v6U5MttxvIO3lVAA5P+fzoSQ7scjHEYUyY4+6uB0Prz/iKaWwnzEjj+Nz/cpuFyobbk7eGbtg9qUHCHG77v8MeP4Pf/ADj60WC4/Kj7u3kHG2Pr06Hp/n3pzMcEsWxuPLMF/j9qjkzl9xPIIO+Tr06gf579qOAxIAB39VTJ+/6/57U1HUdxmFIH+rPHu/8AC3+fanjdk7d/fGxQv93pn/PSjDFV/wBZjjuAPut6U0lTnPlnP1k7L/nP1qyegpA34JHJAw0uO7f5/H3poVeoCfURn0Xv/nH41KobcQA+M9lA7t6/j+tMU9i3OOhl5+6vbof8+lMLCkNuOfMx/tYHd/8A6/v1puEZh9w8/wB4yd04/wA9elOVRuOFA5/hiPq/c/z9valO4EbmYf77Kn9zuP8AI/CnfQLaEYj/AHR2rxt/hix/CfX+X40NGNxL5HzHl5Av8Tdx9f8AOaV9vlkDBO0jPzPj5W6f54qXaysSqlF3HlUVR95vX+X19qpMSRAIYyy/LGSGBHDSEcr/AI/jSRwHYhHmDheVUDHCevHp9ePepQwLLl8/Mp5mJPVOw/z+VIiBgpCAnA6xEnoO54/pz71XMxcquQPEoUgkAgE4eXHY84H/AOuo5raL5yFjxhjkRFj0fv8AgPrjPbFXGLIrZLDAPBKL2bHTmmSNuEnzZBDciRnHIk9B/n86pTaE4oz5rHDSDy3x83RFXGPNH17D8vpUUlqiNJtCDG7/AJbnP8eOPwH6Hua1XiBaQiNSTuztj2/89B3P+cewp0oU7wzjPzcOwGfv56D3P5+9bKt3Rn7FGU8SIsmcgru+6snbf65/ufz7io57ZozJtW5AXcclhjjzPf8A2B+vqM6k1sGV9uc4bB+d/wC/7/5/Cqc9oIvM/chM7+UiVccSep98D/8AZxpTVOWpMqXkJ9gUBv8ASpCQSMMyAdSOzn+7/kc1Zk062tmQSmKcMOWil3YPcdauQqJA+JQclufMUfxPz8q/jx1zx1rZMck9zJJNK6sIiAYy3YKAOAPzrz8TKnJ8qk4v7jro4ZNXivv1Mq10nTZ0DRjB9DV6PSLWP+AH8Kd9iDQEImyRVkdpDIw3YIx1Hb/OKmSV4Mx3O4sDtLEEZPXuM9K+cxmHxEVzRqOUfU9nCqhop00mKlpAnSMVKEVRwoH4VJg7ASO460g7gsBj8a8Zxl16nrRjGOyEpMU8kAg5J9u2KTdtXj370OKT1ZVxAMkCjHrTgcsBkD6DNJ1655HHOKOVJAX7K/aMpFLyh4VjxitfrXNMvPJAI4POa1dNut4MLsMj7vGOPSvq8kzOXN9WrO/Z/p39PuOOvSVueJoUUUV9UcgUUUUAFFFFAHP+O/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtHjv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABXNeMsixtiMf63H6V0tc34yH/ABLbc4/5bY/Q1z4v+DIxxH8JnGbyf/10znHXilPXmj04r55WR4zuJ69aXoaPWj6dKdwsAHIwKOw/lR+tKO3alcLCAcjAyaB25peuO/1oHGO3FFx2AAY6ZGKmJxuG5RwBhB71Djjv071OTyQCOo4Ue/X39K0pvcLAR8x4A56t9P5fh2/JM4AG5V+VuAM9+/8A+qlXBbACkhiSHPHTvSZxGQGVfkPCjJP1rS4WHYJ+XaxBI6nrSZCjGVXh+ByevQ/56UpHzH5SfmGd5pCSFPzgAq3AGc89/SmOw8glv+Wh+YcHgdM9PzNM+6Mny1OOSfr/ADp20FuV4Dc7zjt39v8A61IrbQBuVeBwoyevQ0BYcx3A8uR8x6f40HhhuGG3/wATZ/h/z+NIwJydrn7xOWAxj6dx/Ol+XedpQYb+AE/w9sfjTuMWPkrtLcbeEXGevTNIM4GRxhPvtxjJ7elCjcoOGIGBydqnjvQmNw27VbCngbm6n9emaAHI4UgKQPu42rk9fX/P6U5WPGVY4C8khe/tzTFzkAF+ijHQdT+NAADDOwH5Tg5Y5z6f/X9KLDuxylCw4XcAuR98jk05GOIwpkxx91cDofXn/EU1MttxvIO3lVAA5P8An86bhcqG25O3hm7YPamkF2OLYT5iRx/G5/uU7Kj7u3kHG2Pr06Hp/n3pgOEON33f4Y8fwe/+cfWlkzl9xPIIO+Tr06gf579qVtAuSMxwSxbG48swX+P2qDCkD/Vnj3f+Fv8APtT+AxIAB39VTJ+/6/57UYYqv+sxx3AH3W9KtIHqA3ZO3f3xsUL/AHemf89KQgb8EjkgYaXHdv8AP4+9ISpznyzn6ydl/wA5+tSKG3EAPjPZQO7ev4/rVJ6i6kQVeoCfURn0Xv8A5x+NPIbcc+Zj/awO7/8A1/frSKexbnHQy8/dXt0P+fSlVRuOFA5/hiPq/c/z9vammCWo3CMw+4ef7xk7px/nr0pBH+6O1eNv8MWP4T6/y/GpDuBG5mH++yp/c7j/ACPwpj7fLIGCdpGfmfHyt0/zxTuJoRoxuJfI+Y8vIF/ibuPr/nNIIYyy/LGSGBHDSEcr/j+NT7WViVUou48qiqPvN6/y+vtTAwLLl8/Mp5mJPVOw/wA/lVczBxViKOA7EI8wcLyqgY4T149Prx70x4lCkEgEAnDy47HnA/8A11OiBgpCAnA6xEnoO54/pz70rFkVslhgHglF7NjpzVKTFyopzW0XzkLHjDHIiLHo/f8AAfXGe2KgmscNIPLfHzdEVcY80fXsPy+laEjbhJ82QQ3IkZxyJPQf5/Oh4gWkIjUk7s7Y9v8Az0Hc/wCcewrSNVmbppmVJaojSbQgxu/5bnP8eOPwH6Huac8SIsmcgru+6snbf65/ufz7itWUKd4Zxn5uHYDP389B7n8/eoprYMr7c5w2D87/AN/3/wA/hVqpCWkkL2KWxlz2zRmTatyAu45LDHHme/8AsD9fUZn+wKA3+lSEgkYZkA6kdnP93/I5pZ7QReZ+5CZ38pEq44k9T74H/wCzjUhUSB8Sg5Lc+Yo/ifn5V/HjrnjrVV/ZwjcIUU5WaKcmnW1syCUxThhy0Uu7B7jrWha6Tps6Boxg+hrVMck9zJJNK6sIiAYy3YKAOAPzqL7EGgIRNkirI7SGRhuwRjqO3+cV4OKpOpf2VVp+v6HqUaMacryppr0Gx6Rax/wA/hVlLSBOkYpEleDMdzuLA7SxBGT17jPSrWDsBI7jrXzVenXjPlqN3PcoRoON4RRGEVRwoH4U6lHcFgMfjSkgEHJPt2xXOoaXZ1bbDMUoGSBS7tq8e/elBywGQPoM01GN1qFxuPWtCyv2jKRS8oeFY8Yqh1655HHOKVl55IBHB5zXThMTVws/a0n/AMH16EzipqzOl60Vn6bdbwYXYZH3eMcelaFfoeExUMVSVWHU8ycHB2YUUUV0khRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQBz3jEf8AEqgOcfvx/wCgtXEetdz4uGdHTA6TKf0NcMe9eHmH8b5HlYv+IGPxo7elHegdsda4TmD1pe/Wk59aXHXjFAB+HbvScfp2peMnNBHXoPoKQB09Bx60nr3p2OvHb0o6560XHYlOclW+U5GVwP5f/WpV+Yhdu8bicEYH8zRwpP3EG703EYpQu4gbXfknDHA69a6QGZATlkA2fwjJp+MtnaeGHLnkcf5/SkzgYDKDsH3Bn/J/xp235s7SBv8AvyHnp3oQDc/IQDn5D06Y/wA/nTiAX3YH3+Gd8/w0mcr1JIXoo4/z/hTsBZQTgZb69v8APemMaDlB8x+6v3R056f570MAFPAGQ3LNkn/P+NKAWVeHb5UHQDvSHAyMqCQ4I6nr3ouA4ncTyzAt0VRg/LQFG5SwUPlcb2z29O9KwPmH77tknn5e1CY3LhkByD8i57U+oxEyyqoJPC5wAv8AP/GgYBGcDO3G85PU9qVRlFDA7MLkyNgUicAY6AL91ckcnv0oTAVckgAnovCLgdT3NAxvG/GTt/1jZyMnt/n9aBzjdyuF3b2wOpojyNu0joo+Rc+vfpmmgCM/Km3J6cKnfB9f89aCMKNw7dWfHOz0oXIWPcOMLy747HsO3/1qQHCZBUZGMon+x3P+f1ovoA75c/LjocFE9h+H+RSnIOW3AeZ/y0bH8ftx/k0j7ssW3Z5H7x8Z4H4f40KArAJgEOOETn7/AGJ4/wAiqAQBflK7RnuELc4Pf/PrTzu5JL9+SQvZe4/z+VJkjBYv0HLybcDB7Dt/+qkyMkjYTg8qmey9z/n86L6B0BdpccR9eMAv/Ef8/l70AMIzw4XHPAA+6vXuP8+9PywOWLDn+Ngv8Tdx/n86jUKUyApO3jahP8I79v8A9frTAdkGRgSuQSSN5Pduw/z096EBBG0EDH8CAf3PXr/jS5PJJYLk/eYKPvN3HP8Ak+tNAVuynp2L54T/AD+neqAHOY2UnB2nhpcHo3b0p20GQsFX73Xyyf4j3PX8fb0ocEL/ABgYPHAH3X/zil3bn3ZVju67mk7/AP16aAQMVK5cr0PzOqf88/T6fXv2pAA0YIAYEY/ikHQf5/yKVAy4IVh0+6oA/wCWZ/i/z07ZpCQVPz7iFwf3h9OhAoAJFZEb5WAw33Y1UdH4/SkkO53+cMRu/wCWpz/y09Bjv/kE0rqMPtQchh8sZHZ/U/8A16dI3zMC/OTjdIFOcv6Dnr+vvTuIRow27Cg9cbojkfe7k/5xQxwSNxXOflZ1X19Mnufz9xS7A2QBuyecoz569Cf8/rS/czktGvqdiD9M07jsNI3AnJbPOSXcd/p/n8RVS7hCoxEQHB5WNF7N/e9c4/P0FXh8/oS3oXcH8sCq10mEz5eCc5ZUVcf99f4fyFa0Ze+hSWhY0+cMWAmEnU5EqnvIf4R9D+Xrx1OEZpGVcDaf4mPYVxlpOC7BZxJ1z+9U/wDPT+6Pf9ffjsI5jM07NjJBzjOOg9hXm5pBxb9V+p6mVzTXKyY24aInC4KvxznGabPCJVYNz8xJJVsn5cdal2rsPzHO1uNjetIQArllJGGxhSK4tYrS2y63/I9VxjLdFIW5jRQpJAGec9OfWlwFyQ/IPGB1q0oBLZ5/d+47/rWdM0kch2kAFRuGevANeTWw6iueP9f1/XY051FWJ2I2DAbGeCTSEgjIAGD0pisGUGnsV42rjHcnOa4ebmu2bIQsSSc8nrRyx7k0YLNwOvYUAZIGQM9zUu7eoDlZgDggd+adDJ5c6OWOAdxxTAFxkt+AFJxjvmtI1JwcZJ7a7iaTOmByAR0NLVexbfZRHGMDH5VYr9Oo1Pa041F1Sf3nlSVm0FFFFaiCiiigDn/Hf/JPPEv/AGCrr/0U1HgT/knnhr/sFWv/AKKWjx3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+iloA6CiiigAooooAKKKKACiiigAooooAK57xiP8AiVQHOP34/wDQWroawPFwzo6YHSZT+hrDFfwZehjX/hs4b1ox+NB70d6+cPGDt6UetA7Y60c+tAC9+tH4du9GOvGKOMnNAxOP07UvT0HHrQR16D6Clx147elK47DfXvU5JBIb5TuHGB/n9Ki6561NwpP3EG703EYrSmwBfmIXbvG4nBGB/M03ICcsgGz+EZNPC7iBtd+ScMcDr1pM4GAyg7B9wZ/yf8a0AXGWztPDDlzyOP8AP6UmfkIBz8h6dMf5/OnbfmztIG/78h56d6TOV6kkL0Ucf5/wpjFIBfdgff4Z3z/DSA5QfMfur90dOen+e9OwFlBOBlvr2/z3pACyrw7fKg6Ad6YCMAFPAGQ3LNkn/P8AjTidxPLMC3RVGD8tNOBkZUEhwR1PXvT2B8w/fdsk8/L2oGIFG5SwUPlcb2z29O9CZZVUEnhc4AX+f+NKmNy4ZAcg/Iue1CjKKGB2YXJkbApoBBgEZwM7cbzk9T2pVySACei8IuB1Pc0icAY6AL91ckcnv0pRzjdyuF3b2wOpppggGN434ydv+sbORk9v8/rRGflTbk9OFTvg+v8AnrRHkbdpHRR8i59e/TNC5Cx7hxheXfHY9h2/+tQgAjCjcO3Vnxzs9KX5c/LjocFE9h+H+RTQcJkFRkYyif7Hc/5/WnPuyxbdnkfvHxngfh/jTuApyDltwHmf8tGx/H7cf5NNAX5Su0Z7hC3OD3/z60qgKwCYBDjhE5+/2J4/yKXJGCxfoOXk24GD2Hb/APVTCwp3ckl+/JIXsvcf5/KmrtLjiPrxgF/4j/n8vejIySNhODyqZ7L3P+fzp+WByxYc/wAbBf4m7j/P5076h1GAMIzw4XHPAA+6vXuP8+9LkGRgSuQSSN5Pduw/z096aoUpkBSdvG1Cf4R37f8A6/Wn5PJJYLk/eYKPvN3HP+T600wQiAgjaCBj+BAP7nr1/wAaRzmNlJwdp4aXB6N29KAFbsp6di+eE/z+nenOCF/jAweOAPuv/nFMA2gyFgq/e6+WT/Ee56/j7elAYqVy5XofmdU/55+n0+vftS7tz7sqx3ddzSd//r0iBlwQrDp91QB/yzP8X+enbNMBAA0YIAYEY/ikHQf5/wAiiRWRG+VgMN92NVHR+P0oJBU/PuIXB/eH06ECh1GH2oOQw+WMjs/qf/r0AJIdzv8AOGI3f8tTn/lp6DHf/IJpzRht2FB643RHI+93J/zilkb5mBfnJxukCnOX9Bz1/X3o2BsgDdk85Rnz16E/5/WmmFhGOCRuK5z8rOq+vpk9z+fuKCNwJyWzzkl3Hf6f5/EU77mclo19TsQfpmlHz+hLehdwfywKq47FG7hCoxEQHB5WNF7N/e9c4/P0FXdPnDFgJhJ1ORKp7yH+EfQ/l68V7pMJny8E5yyoq4/76/w/kKbaTguwWcSdc/vVP/PT+6Pf9ffjWrFzw7Jg+SomdnhGaRlXA2n+Jj2FPNuGiJwuCr8c5xmoY5jM07NjJBzjOOg9hVnauw/Mc7W42N61861epO9uvZdT6iDTgmiKeESqwbn5iSSrZPy461WFuY0UKSQBnnPTn1q6QArllJGGxhSKaoBLZ5/d+47/AK1liaftnyzt/X9eZUIqL5kVcBckPyDxgdaGI2DAbGeCTUEzSRyHaQAVG4Z68A09WDKDXjVV7P3bf1fzNoTUmPJBGQAMHpSFiSTnk9aVivG1cY7k5zSYLNwOvYVnK97J39P6+RYcse5NOVmAOCB35poGSBkDPc0oC4yW/AClByTun+IMfDJ5c6OWOAdxxXRA5AI6GuZ4x3zW/YtvsojjGBj8q+o4axD5p0X6nJio6KRYooor644wooooAK5/x3/yTzxL/wBgq6/9FNXQVz/jv/knniX/ALBV1/6KagA8Cf8AJPPDX/YKtf8A0UtdBXP+BP8Aknnhr/sFWv8A6KWugoAKKKKACiiigAooooAKKKKACiiigDD8WDOiH2lX+tcGfrXfeKv+QFIfR1/nXA98da8TMf4y9Dy8Z/E+QUemaOmaOnsK4DlFGR7c0cen50g7UZ49s0AKT1yaX16AUmcfT60evpSGLwc/1NHft1o5560c9M/lSAmBwQcbct6exoA3YyC5OTycZoxghh8vzH5i35cUmQVO4ZO09/6//rrpAdnGRuVeFGEHvS7fm3bQPmPzynrxSE4yNwH3eEHvRja27YB8xG6Q9KBgCGTALN8o+VeBS8K2DtT5iMdWPFNzlQu5mO1cAcDr09//AK9PHytgbU5PCjLdP/19qaAQLlR8uRhcFuM80ZwrbTtGGztHv/8Aq9KQDjO3cRtyZTwKXk8bmY4PCjA6+9ACkY3ZUAEnl2Hp+VOBBYYJxuH3VAH3ff6U1sAtuCA/MMdT0796d1cZ3E7ujEL/AA0xiLhSm8KMBeGOSeP88UqclT8zHC4H3R379aSMsAvPp8qD696VRjacYJC4LnI/IdqECCM/MuCA2F6Dc3U/5/KlUEhSQxAC98DvSR5O1fmIwv3RjHJ70KBuXIXovJBY9+3WmugCRgKEIwM4+6pPY9/6fWlOQmSGXjGWYKPue36UR7sIfnPC4zgDofy+tINu35Sn3ecDd/B6/wCe9C2DoKNoJ27e/wBxCew7n/P504gg/MpBLf8ALQ4z8/oP89aRyfmLbgDkfOdo6D0/zzQuAwCjA3fwL33jueKoYLkYKnsOEXno3c/560HIDZJ6Hl3x/COwozjbu4Jxw78/xdh/np70gPynBb/gCbf4V9aOgdB6A7/kB+91SP8A2j/EeP8APtTM4Tkj7mRufGflHYcH/PpTsAyENtJ3HIZiT98/w/4Uig+TlQfuc7Vx/CO5pgOVT5hKqc7jyic/eP8AEf8AP5U3Ix8xwcDhn2/wr2H+fypx2mTB2E56Elz94/w/5z170ibvLG3cBj+EDH3V9efT9PemAm0HICrznO2PB6P69f8APpTyxDgOTndxvkCnqOgFNYhiwyrfUl/7/b/PY+tSLlXGNy4YfdAUfeX15FNAiFFBwQAcAdIieydz/n86c5KKd24cEfO4XseBikQq0agkH5RgAl+y9v8APp6UrZWNiuQNp+6FUdG/GhMQkgDF+AwAbqrP/wA9O/8AnpntUjEoWyWRck/MVUdW44z6/wCeKazAs53BuW6Oz55f0/z39acFKEkLtGScKqr3PryP8+1UhoT75PRmPTO6T8z+NKB5ZHWPoeioD09cnFAYMR8+4E9dzP6dhxQiFNpCbeRyqKuD8vrRcYgYPjPzDgnDs/8Ad7Cq9wmEB2YJ/jWJVx067zVlGDFfmyBjGJCf7nZarToQqNswTtG9Y1Uj7ndj7/r0rWi7zRMthtq4dwDKCehUyjI4fsB7/wCQa662jTzLjy3ZlUHl2bJ4Hr/n9K5Kxky6KX3twCplUHOD2Ue/6/l1sA2yzDYyf7JZ2I/E1z5nbrb+kzvy7WRbRC46DAHOQR396dKAA6nJUB8KAfl/HpSEEiPIXGB0X37018q7BQVGHGNp6ZHcV5kvdg9O39f0/O3b2t2NKEen+qz39qglVS0hAI+UDBY+gHepflCnIP3BjAI9KaSAZecZTAyfcetcFVq2ll+ezLSvuUlBR1dhkN27GpQSOlLKoKocg5BP600dK8mrpPTy/JGkFZWFyfXpSUUmR61lcsWik3Ck3ikK6N7TP+PJfqauVT0wg2KEY6np9auV+nZcrYSl/hX5Hl1PjYUUUV2EBRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABWH4sGdEPtKv8AWtysXxV/yApD6Ov86xxH8KXozKt/Dl6HAn60Ud8daOma+bPFD0zSjI9uaTp7CgdqBi8en50E9cmkzx7Zpc4+n1pAL69AKODn+ppPX0peeetAw79utTA4IONuW9PY1Dz0z+VTYwQw+X5j8xb8uK0pgAG7GQXJyeTjNLnGRuVeFGEHvTcgqdwydp7/ANf/ANdOJxkbgPu8IPetAF2/Nu2gfMfnlPXikBDJgFm+UfKvAoxtbdsA+YjdIelJnKhdzMdq4A4HXp7/AP16BjuFbB2p8xGOrHigLlR8uRhcFuM80o+VsDanJ4UZbp/+vtTQOM7dxG3JlPApgLnCttO0YbO0e/8A+r0pSMbsqACTy7D0/Kk5PG5mODwowOvvStgFtwQH5hjqenfvQA4EFhgnG4fdUAfd9/pSLhSm8KMBeGOSeP8APFL1cZ3E7ujEL/DSRlgF59PlQfXvTGKnJU/MxwuB90d+/WiM/MuCA2F6Dc3U/wCfyoUY2nGCQuC5yPyHaiPJ2r8xGF+6MY5PemgQqgkKSGIAXvgd6bGAoQjAzj7qk9j3/p9aVQNy5C9F5ILHv260R7sIfnPC4zgDofy+tC6AByEyQy8YyzBR9z2/SgbQTt29/uIT2Hc/5/OkG3b8pT7vOBu/g9f896c5PzFtwByPnO0dB6f55pjFIIPzKQS3/LQ4z8/oP89aRcjBU9hwi89G7n/PWhcBgFGBu/gXvvHc8UZxt3cE44d+f4uw/wA9Pen1ADkBsk9Dy74/hHYU5Ad/yA/e6pH/ALR/iPH+famA/KcFv+AJt/hX1p2AZCG2k7jkMxJ++f4f8Ka3DqNzhOSPuZG58Z+Udhwf8+lPVT5hKqc7jyic/eP8R/z+VNUHycqD9znauP4R3NOO0yYOwnPQkufvH+H/ADnr3oQDcjHzHBwOGfb/AAr2H+fyo2g5AVec52x4PR/Xr/n0pU3eWNu4DH8IGPur68+n6e9DEMWGVb6kv/f7f57H1piHFiHAcnO7jfIFPUdAKjRQcEAHAHSInsnc/wCfzqZcq4xuXDD7oCj7y+vIqJCrRqCQflGACX7L2/z6elNsbFclFO7cOCPncL2PAxSSAMX4DABuqs//AD07/wCeme1K2VjYrkDafuhVHRvxpWYFnO4Ny3R2fPL+n+e/rTEOYlC2SyLkn5iqjq3HGfX/ADxSffJ6Mx6Z3SfmfxpQpQkhdoyThVVe59eR/n2oDBiPn3AnruZ/TsOKZQAeWR1j6HoqA9PXJxSBg+M/MOCcOz/3ewpUQptITbyOVRVwfl9aEYMV+bIGMYkJ/udlouBWuEwgOzBP8axKuOnXeabauHcAygnoVMoyOH7Ae/8AkGnToQqNswTtG9Y1Uj7ndj7/AK9KSxky6KX3twCplUHOD2Ue/wCv5di/gMz+0dbbRp5lx5bsyqDy7Nk8D1/z+lW0QuOgwBzkEd/eqkA2yzDYyf7JZ2I/E1cIJEeQuMDovv3r5+STqS077ep9NR0poWUAB1OSoD4UA/L+PSoihHp/qs9/anPlXYKCow4xtPTI7io/lCnIP3BjAI9K568k5O6t/wAN/Xp2No7EUqqWkIBHygYLH0A71VUFHV2GQ3bsaukgGXnGUwMn3HrUEqgqhyDkE/rXl4jZ/f6e8Wo6poQEjpRk+vSkHSivPuzYKKTI9aNwpBcWtzTP+PJfqawd4re0wg2KEY6np9a+h4a/3t/4X+aObEv3C5RRRX3JwBRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQBjeKR/xIJz6Mp/8AHhXANmvQfE//ACL9z7FP/QxXnhIzXjZiv3q9Dy8b/E+QtHekzwODRn8MVwWOO4uemaXP4c0n6DvQOg/nSsO4vuKPU9efSk6+5z0pev59MUrDF7/j0xR+Q59KTPvgZpR2PTnrmkNEyHlSFGeTnBI/CkJ/d8Mx+THC8D6/rSjJ2/eY8nk9/akzlcfOQFxhf89K6LAPJOSO+R8qj/P0oHDAgL1I3Oab0bbjbkgkLzkcf56UA98ID83JOe9MBQcgjczcKOOAOf8A6/pTs9Rnjn5VHH9f84/FuQQAWDfdA4wOv+e1BbIIy3G7hR098/57UDH8BgWIB+Uc8/5/OjqmMHGD1OO/t/8AXpPunoqDI68npSD5hwC3y8ljgdev0pgO4AYLgfe6ClIw3GD83Vzn+H0prNkfePRsbBgfXP6/lS8hiDhDuwQPmb7vT/P/AOsGKuSyD5yTtxgYHSkj2jZtKqxC8ou49/1oXgL8mFyMlyAOnpSo25VA3HAX7vA79TQgFXopIBGFyXPy/lRH1UKWH3chQPfvQhwyYO1vlGerd/y/z6UKchdwJXC5Lthe/p2poEIuPkLBM4XGTuOMH/Pt+FLlyhyZG/Hbj5Pb/PSkjYkR7WIHHCrjnB7+v/16D0O5VBx3Jc/d/wA/Tn0oWwdBeNx2lO/3RuPQd/8AP60pOOTxz/y0bH8ft2/+vQwYgk7mHOS2F7D3z/n2oGM/IR97+Fc/xjv0pjBAQwVAcDGAqf73r1//AF0hxgqxHTo77j91ewpTjID7RnHDtn17D/PSj5tjAbwuD90bR91fXkU+gdByg7hw33scAKPvn1/z2qIbTF1QkJ7vj5f0/pUgx5g4U/PjvJ/F+n+fWmsSLf5i2NmBubA+52x/WhgSfMD8xYDdzucIPvH05/yajXBQNhWIUc7ST90d+30+vrUi4EuUHO7jy0yfvf3jx/n60wH92oJ5KDq/P3R/COv+fSqBjmyCQxYDoMsF7v6fX9fehQCylAM7hgohP8S/xNx+dA+8do6k/dX3f1/pQSCw3EEkjOcv3T+H/OaOgdABOxQzHoMBn2/wjsPoPy9qHGVbCj7rDiMjHD9zQuRF0IG30CjoOuee1I4Vg33SOfVx0f8Az+tAD3OXbLk8n70mD1bsvXr+vvQEIJZVPXOViwevq1K25WIORyeGKqOrenP+fpSIqu4woY5HO0v3XoTVDAPyo3gnI4Mhz/D2WiIZKMAMnbysYB6x92+tCsRsDORkrwzhf7npSRruMZ2bj8p+4Sf+WfcnH/6/Q0mAsb7vLAck/LxvP/TPsv8Ang9xVSWMYjPlhs7fuxAnGYv731/r3q3Gf9Wpcn7vBfB/5Z+nPb9PUc1Jo9xiKpvwUIAjy3WLoW4/pz6HjWh8aIqbC6dISYlEwLYX78yjGQP7o6c111uCHl3NuOeodnB57GuR0mRv3CpLufCEIXUE5CY4Ufpz/OuvgLtJMZBht3I+b+8e1YZm9f67M9DLNWWADhSVOPlz8h9aXb8rlQuCHyGTkDdSg/ugoLElY8fK2Bz6d6YV+RtxAwDgbDzzXk6W2vp38+9z2xso2g4GPkXuT6UyUjdJt4GP7x9R61I21UlAPZeSp9u9MkA3S7SSo7k+/vXLXWjtb8P7xUf6/AgkHCc9v6monbAqV/upxjC/1NV5jwK8ar8X3fkat2QwyGm76jJpu6osczmS76N1RbqXNHKLnZ1elf8AIOi69+v1q5VTSxjTYOMfL6Y71br9QwSthqa/ur8jkluwooorpEFFFFAHP+O/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLR47/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFY3ikf8AEgnPoyn/AMeFbNZHif8A5F+59in/AKGKyr/wpejM638OXozz5s0lISM0Z4HBr5xI8O4vejPTNJn8MUv6DvSsO4ufw5o9xSDoP50dfc56UWHcX1PXn0pe/wCPTFJ1/PpijPvgZpWAX8hz6VMh5UhRnk5wSPwqEdj0565qYZO37zHk8nv7VdNDEJ/d8Mx+THC8D6/rTyTkjvkfKo/z9KZnK4+cgLjC/wCelL0bbjbkgkLzkcf56VqA4cMCAvUjc5pAcgjczcKOOAOf/r+lID3wgPzck570uQQAWDfdA4wOv+e1Ax2eozxz8qjj+v8AnH4rwGBYgH5Rzz/n86YWyCMtxu4UdPfP+e1O+6eioMjryelMBeqYwcYPU47+3/16OAGC4H3ugpo+YcAt8vJY4HXr9KVmyPvHo2NgwPrn9fyoAcRhuMH5urnP8PpQuSyD5yTtxgYHSk5DEHCHdggfM33en+f/ANYvAX5MLkZLkAdPSjqMI9o2bSqsQvKLuPf9acvRSQCMLkufl/KkRtyqBuOAv3eB36mlQ4ZMHa3yjPVu/wCX+fShAgj6qFLD7uQoHv3pFx8hYJnC4ydxxg/59vwpVOQu4Erhcl2wvf07UkbEiPaxA44Vcc4Pf1/+vTXQBcuUOTI347cfJ7f56UcbjtKd/ujceg7/AOf1pD0O5VBx3Jc/d/z9OfSnMGIJO5hzkthew98/59qYwJxyeOf+WjY/j9u3/wBehAQwVAcDGAqf73r1/wD10DGfkI+9/Cuf4x36UHGQH2jOOHbPr2H+elPqAhxgqxHTo77j91ewp6g7hw33scAKPvn1/wA9qb82xgN4XB+6No+6vryKUY8wcKfnx3k/i/T/AD60LcCMbTF1QkJ7vj5f0/pU3zA/MWA3c7nCD7x9Of8AJqNiRb/MWxswNzYH3O2P61IuBLlBzu48tMn73948f5+tNAiNcFA2FYhRztJP3R37fT6+tPbIJDFgOgywXu/p9f196aD+7UE8lB1fn7o/hHX/AD6U4feO0dSfur7v6/0poSBQCylAM7hgohP8S/xNx+dAJ2KGY9BgM+3+Edh9B+XtQSCw3EEkjOcv3T+H/OaFyIuhA2+gUdB1zz2o6jBxlWwo+6w4jIxw/c05zl2y5PJ+9Jg9W7L16/r70xwrBvukc+rjo/8An9akbcrEHI5PDFVHVvTn/P0poEIEIJZVPXOViwevq1AflRvBORwZDn+HstCKruMKGORztL916E0KxGwM5GSvDOF/uelMYRDJRgBk7eVjAPWPu31ojfd5YDkn5eN5/wCmfZf88HuKSNdxjOzcflP3CT/yz7k4/wD1+hpYz/q1Lk/d4L4P/LP057fp6jmQKksYxGfLDZ2/diBOMxf3vr/XvS6dISYlEwLYX78yjGQP7o6c0k0e4xFU34KEAR5brF0Lcf059DwukyN+4VJdz4QhC6gnITHCj9Of513R/gMyTtM663BDy7m3HPUOzg89jVkA4UlTj5c/IfWq8BdpJjIMNu5Hzf3j2q0D+6CgsSVjx8rYHPp3r52dnN7df1PqKPwITb8rlQuCHyGTkDdUco2g4GPkXuT6U4r8jbiBgHA2HnmkbaqSgHsvJU+3esKtuTZLTuuz6fL+tDZbkcpG6TbwMf3j6j1qGQcJz2/qankA3S7SSo7k+/vUD/dTjGF/qa8zF/FK/wDwPiNIkTtgVCZDT5jwKrE159rkVJWZJvpN9RbqN1PlMudku6un0r/kHRde/X61yma6zSxjTYOMfL6Y719HwzH/AGmb/u/qjKrK6LdFFFfbGAUUUUAFc/47/wCSeeJf+wVdf+imroK5/wAd/wDJPPEv/YKuv/RTUAHgT/knnhr/ALBVr/6KWugrn/An/JPPDX/YKtf/AEUtdBQAUUUUAFFFFABRRRQAUUUUAFFFFAGX4iXfoN0PZf8A0IV50V5OK9I14btDux/sf1rzbd1rx8xv7RW7HmY23OvQbz0xR60pXnNNPv8AlXEtTgY7v/nil9O/4UzOM80bhRyhcf1x/KjP4c0zeP8AGjf9KOUOZEgPQ9s9aUduPxxUW45znmjNLlHzIsArjkBvlzgDH/66ViOVO37o4U8f5/xqvvbj5jx05pRKwGAcD0rS6DmLJPJxtXL/AFPT/PrRkY4CjhuD9cf56VB57Bs4HXP6YoE3GCOxA/Gquh8yLGcnG4nOOo4Pv/kGk3Ajq3AboOnNR+cpzkk8g4I4P+fxpTKpBG9jwwGBjr/n/Pc0HzE33ZNuApLdByx4/wA/jn0pOqgsR0HLH37en+H14arKSQuBl+EQ5PT1/wA96BwAflBIHLck8/5/SnYdx5OVcjcchznGM/5/z0oyN5GQfm+6oz2/z+tNY5DZ3N8r8twOvUCn7mZiF5GT90YH3fw/n6UWHcASrqWCqc/xfMelKpLKuQduF5Y4HT09KajBWAQjqPuc/wAP+f1pVIUoX2ggL97k4+lFhpixtkALnGF4Xj174/X60qfeQDAJ25JyT3/z/wDqpsZLKu4EqAoO44GMH9KIiGCqMkEKDsGPX/P50JbAnsOBJVNwO3Cgl2wOnt/npQDuU7CTwfupgfd9/wDJ/GmowHlgEA4XpkkcGlflfn4Hq7Afw/55otoO4r4BcsFzhufvHoKcxY53Z5Y/6x9v8Y9P1pmdwfbu2gH7o2gcD16f59KcCC5ZShJbqoLH7w74/LNOwXFTO7CblxjhFx/e9f8APWmcbGJCg46lt/8AD/n/ACKUFc87RuCnDtnP3uw6il58s4DY25yuFA+Uev8Anp607aBcechsnIAb7xIQff8Az/yaj48olcHK8lRn+E9T+FOUgybk2lt3WMbm+92J/DrTWJ2KG4O3+NsHG09hx6fpQwbJG++Qw6t/Ec5+b+6OtNXIiyNxUJ/CMD7vvz+NL6gZClscJ5Y+8PXmmZDIM7Sdh9XP3fXpVPcGPGCeNrcnsX/ibv2+v096XJDKCWAPTc4TunQjrSE4fDkj5j/rH2/xHsOvX/DrSIvICqR/uIFz9z+9+H5+9FtAFAyPlAJ254Uk/d7E9aVz8zZPr958Hq/Yf5H50xtrIVyjHB4yz9j27f0p5yC3UDJxkhAOXHbnv0/xFFgHBcZ2rgE9VQKOvv8A0pAyuRkq2SBnJcdV7fjQuGYELv56hSx6+p4NAf50DOSQV4L9OU7LQ9EMEYx7Dyq/LzkKP+WZ69f8/WkjAITABxt6gv8A3Pw9f1HQiliDAIVBBwvKoF/uHkn/AD0Pc0IQwAGGOAMAs4/h7Djt/SkCHKdmwMSBx1cL/c9Oew/L1FVJ0L7CEEhBQgeWW6FO7ED+nJ7Hi2uYwvJUYHdUX+H8fSqtwgdk+VWO5TgozjqvQnAHT+ncVvQ+NCmrqwmkkIYFeV1A2Z3FFI+WPsOen8vrXXWuG851kLqGGCzEk/Me1cnpBEM8G52iUMhw2wYx5XI654GeT29Qa623ZXEz72YmTI5OD857CuXM37/3/kz0Mr3ZZG3apEjEhUx8jetI5DRktjIH9w8804DYyYBbheqMaZJuWI5A+YA8KfX9K8yWlN3S2d/6ue0txhICTA9Tt42n/IpjMMyfN36bvenkf6w+hXGSR+lMbJWTrgN657muGq3+f/t3/BLRC4xt47VWn7VZfqPpVW4615FT4ip/CVz1pKDRTRxhRRinAUNjSOs0wY02D/dq3VexGLCAf7A75qxX6fhlahBeS/I53uFFFFbiCiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACsvxEu/Qboey/+hCtSs7Xhu0O7H+x/Ws638OXoyKmsH6Hm5Xk4pnPTFO3daQrzmvnI+Z4L8hPWl7/54pp9/wAqM4zzV2Fcf6d/wo64/lTNwo3j/GlysOZD8/hzSg9D2z1qPf8ASjcc5zzS5R8yJR24/HFPBXHIDfLnAGP/ANdV80u9uPmPHTmqirBzFhiOVO37o4U8f5/xpxPJxtXL/U9P8+tVhKwGAcD0p3nsGzgdc/pirug5kT5GOAo4bg/XH+elLnJxuJzjqOD7/wCQaribjBHYgfjT/OU5ySeQcEcH/P409BqSJNwI6twG6DpzT/uybcBSW6Dljx/n8c+lQmVSCN7HhgMDHX/P+e71ZSSFwMvwiHJ6ev8AnvTsUmO6qCxHQcsfft6f4fXhScq5G45DnOMZ/wA/56UwcAH5QSBy3JPP+f0pWOQ2dzfK/LcDr1AosO47I3kZB+b7qjPb/P60oJV1LBVOf4vmPSjczMQvIyfujA+7+H8/SkRgrAIR1H3Of4f8/rRYdxyksq5B24XljgdPT0ojbIAXOMLwvHr3x+v1pFIUoX2ggL97k4+lEZLKu4EqAoO44GMH9KEguOT7yAYBO3JOSe/+f/1UAkqm4HbhQS7YHT2/z0psRDBVGSCFB2DHr/n86EYDywCAcL0ySODQlsCY4Hcp2Eng/dTA+77/AOT+ND4BcsFzhufvHoKR+V+fgersB/D/AJ5ozuD7d20A/dG0DgevT/PpTsO49ixzuzyx/wBY+3+Men60JndhNy4xwi4/vev+etICC5ZShJbqoLH7w74/LNICuedo3BTh2zn73YdRT6hfUTjYxIUHHUtv/h/z/kVKchsnIAb7xIQff/P/ACaZz5ZwGxtzlcKB8o9f89PWlUgybk2lt3WMbm+92J/DrQkFxvHlErg5XkqM/wAJ6n8Kkb75DDq38Rzn5v7o61GxOxQ3B2/xtg42nsOPT9Kf6gZClscJ5Y+8PXmmkCEXIiyNxUJ/CMD7vvz+NKME8bW5PYv/ABN37fX6e9MyGQZ2k7D6ufu+vSnk4fDkj5j/AKx9v8R7Dr1/w60JAhckMoJYA9NzhO6dCOtIBkfKATtzwpJ+72J60iLyAqkf7iBc/c/vfh+fvSNtZCuUY4PGWfse3b+lMB7n5myfX7z4PV+w/wAj86cFxnauAT1VAo6+/wDSmnILdQMnGSEA5cdue/T/ABFKuGYELv56hSx6+p4NNIaAMrkZKtkgZyXHVe340Ixj2HlV+XnIUf8ALM9ev+frQH+dAzkkFeC/TlOy0RBgEKgg4XlUC/3DyT/noe5pAJGAQmADjb1Bf+5+Hr+o6EU9Ts2BiQOOrhf7npz2H5eopqEMABhjgDALOP4ew47f0py5jC8lRgd1Rf4fx9KENFSdC+whBIQUIHlluhTuxA/pyex4NJIQwK8rqBszuKKR8sfYc9P5fWluEDsnyqx3KcFGcdV6E4A6f07il0giGeDc7RKGQ4bYMY8rkdc8DPJ7eoNdu1BmSXvpnWWuG851kLqGGCzEk/Me1Wht2qRIxIVMfI3rVa3ZXEz72YmTI5OD857CrQGxkwC3C9UY185b376W/wA2/NfqfU0vgQ1yGjJbGQP7h55qMkBJgep28bT/AJFPk3LEcgfMAeFPr+lMI/1h9CuMkj9KwrX5raXt+kvPsaxGMwzJ83fpu96hcY28dqmbJWTrgN657moX6j6V5mJ1V3/WrNYlaftVY9asXHWqxriic9X4goooxVmQV12mDGmwf7tcmBXX2IxYQD/YHfNfR8Na15vy/UipsWKKKK+yMgooooAK5/x3/wAk88S/9gq6/wDRTV0Fc/47/wCSeeJf+wVdf+imoAPAn/JPPDX/AGCrX/0UtdBXP+BP+SeeGv8AsFWv/opa6CgAooooAKKKKACiiigAooooAKKKKAK1/btdafPAmNzoQMmuDl8MavEf+PYOP7yOD/XNei0Vz1sNCq7yMK2HjVs5Hlsmn30JPmWk6diTGaonrXsFRS2tvOMTQRSA/wB9Af51zf2eltI5JZffaR5HxRivTpvD2kz/AH7KMf7mV/lVCbwbpknMbTxH2YEfqKh4KotjGWX1Fs0zgcGiuwl8D8kw33HYPH/UH+lUZfBupx5KNDJ6bXwf1FYyw9VfZMZYSsvsnO5NGa1JvD+qwAl7KQgf3MN/LNUpbWeDiWGSPt8ykfzrJxcfiVjKVOcd0QZozSlaQgVKsRqGaM0hFNP41SSFdj80bqj79fzFGT60+UXMSZpwcjoSPpUOT2GaMnOMHNHIPmLHmtzls5z15604zsxO/wCbJzzUG1sZIx35o5HUY+tTbzK5mWhdHj5QMHOFOO2PrTkuETbhSMYzgDP51TzS5p3Y1NlxJo8qWI4Cj5hk8Z/z+XpT0lVgoJBACg7mwOnp3qhmjNHMylUZoxsCqBd2OPuDaM4OefWkDKDhCAf9nk/d9f8A63rVANjkU4SuowHYD2NPmH7Qvv0cMV7n94cnoOwpzZLHIfAY5JOwfe79/wDDms8TuM4OM9eKet0wxlVJznJJJ65/pT5kP2iLqN84CsRyAdi+7dc/zphI8snClgv94ufuj8v5dari6HG5S3T7zZ6Z7dD1pftCGPZ8w4PTgZ2gfj/+r3p3Vh86sW2J35bPLf8ALQ7c/OOwpoGIxt3YC4+VePun16/1/GoxNEX+V1HzDAA5+8O5o3IyfwnjHzEufut2HT+lUVcm4MgI2lt46ZkP3h3PQ/570jHEJU5HycZO3+E9qaxOVDEnLdHOz+JeMD+R6Uc+UdoOCp+6nB+U9z1p2C5MvDnYCBux+7XaPvH+9yPw6VEpViPuE8dAZT0T/wCv+o9Kdw0g4VsN7y4+b9P8+tNUnaNxboPvMB/CvYfT9PUUNDY6QnyyGLD5TgFwv8LdhTyBucgDOTkpHju3c/5x9KjP3G25wQRlVAB4f15zTsq75G18E4IzKep70WGmOBVmG7D8gfMS/OR1A4oTK7ASVHy8Fgg/5Zntz/8Aq9qFJ3oGLD5h96QA9U7L/KmwjCoVBAwuSqhR0TqT+f6+tJgPjUnaVHzYB4TJH3e54/8Ar/WlBBGC3Axkb8nt2X6UxcOo6N9QZOw/Af5FPJKjaWYHsGfb+WOTRYaHD5RkZQd2ChR+pJ/zmqdwPMwFCvng/K0wPt2HoKtkH5mxyMglUxjr1LH61TvSJAd/zdQAC0nr1A4rbD/GhT2JNNXypUHzRjI4wigY8v8Aw/T1BrqrP5rd23Ljd2kJJ5P51zWkxmG6i24g+bO8KqKPu+uT29PTpzXTW77kcl1cljyJOvJPYVxZm1za+f5HpZYtWXgehOUHy5JyWHHbv+VV3KmJ8gliowcEdzU3yblO5n5HB3c8etRPtMBwjcoozk9ea8+u242uvtefTy0PXiNYjLjHO4c7iPWmM2BJwOWHfPrUjEpIQeMOOrHjr61E5Lb2POWHTB9a8+s7adddl6miIn4YD0A7YqrP1q3J9/8AAdvaoJE3GvJq/G/UuSvEqbaUJVgRU8RgVN2ZKkVxGTUiw1MABUsEfmzxp1y3I9qqnB1JKC3ZfIoq7Ogt12W8S9MKB09qkoAwMCiv1WEeWKj2PMYUUUVQBRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABVa/t2utPngTG50IGTVmilJKSaYmrqx51L4Y1eI/8AHsHH95HB/rmqUmn30JPmWk6diTGa9Sorhll8HszieAh0bPHz1pOK9cltbecYmgikB/voD/OqM3h7SZ/v2UY/3Mr/ACqHgZLZnPLLpdJHmOKXBrvpvBumScxtPEfZgR+oqjL4H5JhvuOweP8AqD/SspYSqulzKWBrLpc4+jJropfBupx5KNDJ6bXwf1FUZvD+qwAl7KQgf3MN/LNYypVI7xZi8PVjvFmXmjNTy2s8HEsMkfb5lI/nUJWs9Opm00JmjNBApCKasLUXNGaYfxpO/X8xVcpNyTdRmo8n1oyewzRyBzEwcjoSPpTvNbnLZznrz1qvk5xg5p+1sZIx35pctilJ9CczsxO/5snPNPF0ePlAwc4U47Y+tVeR1GPrRmlqVztFxLhE24UjGM4Az+dKk0eVLEcBR8wyeM/5/L0qnmjNO7GqjL6SqwUEggBQdzYHT0706NgVQLuxx9wbRnBzz61nZpQ2ORQpMaqF8MoOEIB/2eT931/+t605+jhivc/vDk9B2FUBK6jAdgPY0oncZwcZ68U+YftEaDZLHIfAY5JOwfe79/8ADmhG+cBWI5AOxfduuf51SW6YYyqk5zkkk9c/0pwuhxuUt0+82eme3Q9armVyvaK5YJHlk4UsF/vFz90fl/LrUjE78tnlv+Wh25+cdhVT7Qhj2fMOD04GdoH4/wD6vepRNEX+V1HzDAA5+8O5pqw1JEgGIxt3YC4+VePun16/1/GncGQEbS28dMyH7w7nof8APeodyMn8J4x8xLn7rdh0/pT2JyoYk5bo52fxLxgfyPSnbQaeg5jiEqcj5OMnb/Ce1SLw52Agbsfu12j7x/vcj8OlQ8+UdoOCp+6nB+U9z1p/DSDhWw3vLj5v0/z600hpjVKsR9wnjoDKeif/AF/1HpTpCfLIYsPlOAXC/wALdhTVJ2jcW6D7zAfwr2H0/T1FKfuNtzggjKqADw/rzmiwXJCBucgDOTkpHju3c/5x9KAVZhuw/IHzEvzkdQOKblXfI2vgnBGZT1PenKTvQMWHzD70gB6p2X+VFhgmV2AkqPl4LBB/yzPbn/8AV7Usak7So+bAPCZI+73PH/1/rTIRhUKggYXJVQo6J1J/P9fWlXDqOjfUGTsPwH+RSsNDwQRgtwMZG/J7dl+lOHyjIyg7sFCj9ST/AJzTSSo2lmB7Bn2/ljk0pB+ZscjIJVMY69Sx+tA0VLgeZgKFfPB+VpgfbsPQU/TV8qVB80YyOMIoGPL/AMP09QajvSJAd/zdQAC0nr1A4qzpMZhuotuIPmzvCqij7vrk9vT06c12P+ARFXmdNZ/Nbu25cbu0hJPJ/OrwPQnKD5ck5LDjt3/KqNu+5HJdXJY8iTryT2FXPk3KdzPyODu549a+fpy97Rr56d/60PpoL3EQuVMT5BLFRg4I7mkYjLjHO4c7iPWnPtMBwjcoozk9eaRiUkIPGHHVjx19a456NN2+7/F3N1/X4EbNgScDlh3z61C/DAegHbFSuS29jzlh0wfWo5Pv/gO3tXmYl3j/AF3ZpEqT9ag21bkTcaaIq4k7Gc4NsrhKeIyasCMCnAAUXY1SIVhrq7ddlvEvTCgdPaufgj82eNOuW5HtXSAYGBX13DFJr2lT0X6/5HPibKyQUUUV9YcoUUUUAFc/47/5J54l/wCwVdf+imroK5/x3/yTzxL/ANgq6/8ARTUAHgT/AJJ54a/7BVr/AOilroK5/wACf8k88Nf9gq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFBAIwRkUUUAVZdNsZ/wDW2cDH1MYz+dUZvC+kzEn7MUJ7o5H6dK2KKiVOEt0iJU4S3SOYm8E2bZ8m5mjPbcAwH8qz5vA1yAfJvInP+2pX/Gu3orJ4Wk+hjLB0ZdDzmbwhq0f3YUlH+xIP64rPm0TUYDiSynGOciMkfmK9WoqHg49GzCWXUns2eOtA6MVZSGHUGkCsCDyMelegeKlAk0+UgELKQQe4IxVJ9PtHTm3j69hg/pXg43GrC1nSkr2JhlMp35JbHG8joSKMV3lv4W027tFkIlRjnlH9/fNQzeB4icw3rr7Omf5EV30qVSrTjUitGk/vOeeX1ou25xO00YNdRL4Lv0OYpoJB7kg/yqjL4a1aHrZlh6owb+tKVKqt4mLw1WO8WYvNHNXJbO6gGZrWaMf7cZFQfLWTbW6MnC25FRmpcKaTYtLnQuVkeaM1J5fvR5Zp88Q5WR5ozT/LPpR5R9DRzRFysZmjdTvLPoaaUNNOIWYokK9CR9DS+c5zls5/vc9v/r0wqaaQatWFdos/bZc5JBOc+nfPanJeFQAEUADHyjGenP6VS5pM1VmHtJGibtGHKYOCORu7Nxkngcj+fapvtkLtkyORn+M++egGP8+9ZO6jfT95FKs0bCTRnYFdQMgcEIOqd+o6dfbjpT48EKV5IA+6m4jAXueP8+hrG30oYUrvqilW7m3u9eWA53MSfxVfpTjmNT8xRcHGMIP4vxzxWMs0gGBIwHoCakW6lXO1gM8EhRk5/wD10cxoqyNUgHdhclQeQpYj7/c/Sql63L4fcecguSf4+qp/X+tRG9dgd6I2c/eyeTn1P+1+lVL3U8qQUY5yQBIVH8Xpj+9+lbYZpzFOtFI3NHUpextGRCd/+tWIIOvXnJrpIH/cvmQEk8/P+vArj9Ov7fzPOkBCqSciIEgZ9yc9a663niFrt3ybzyOO2MdhXn5nUSdr23/I9jK2nexd3ArtzyX6An+769ajYBYwAxIKrnLEU5JUZ98Tgru4DEg/d9D2ppDbAQOMqOGPXH0rz5tNPZ79Oh7CFkI3Hlc+b/z0ye/rVdx8h69Rz+dTs2SdzYfzeQCPWqU8u2MBRl92e3SuPEtSv8yk+VXYshzKFGMkDp9KCMHGQfpUUanG49TUleRKSbbtuaxvYDjtR0ooqSgrU0u2O4zsMDotV7Oxe4YO3EWevrW2qhFCqAAOABX0+Q5XKU1iaqslt5vv6HJiKqS5ULRRRX2RxBRRRQAUUUUAc/47/wCSeeJf+wVdf+imo8Cf8k88Nf8AYKtf/RS0eO/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAEAjBGRVWXTbGf8A1tnAx9TGM/nVqik0nuJpPcx5vC+kzEn7MUJ7o5H6dKoTeCbNs+TczRntuAYD+VdPRWTw9J/ZMpYelLeKOIm8DXIB8m8ic/7alf8AGqE3hDVo/uwpKP8AYkH9cV6NRUPCU3toYSwFF+R5TNomowHEllOMc5EZI/MVTaB0YqykMOoNexVznipQJNPlIBCykEHuCMVx4ul7Ck6qd7GE8tjumefhWBB5GPSl5HQkV2T6faOnNvH17DB/SrVv4W027tFkIlRjnlH9/fNeXgsYsZU9nBa2v/X3lVcoq01dSTODxRtNdtN4HiJzDeuvs6Z/kRVGXwXfocxTQSD3JB/lXoyw9ZfZON4Osuhy+DRzW1L4a1aHrZlh6owb+tUZbO6gGZrWaMf7cZFYyjOO8TJ0ZR3TRT5oqX5aMKajmI5SLNGak2LR5fvRzoOVkeaM1J5ZpPLPpT5ohysZmjNP8o+hpPLPoafNEVmN3UokK9CR9DSFDSFTTVhaof5znOWzn+9z2/8Ar1J9tlzkkE5z6d89qrEGm81aQc8kXUvCoACKABj5RjPTn9KebtGHKYOCORu7Nxkngcj+fas7NLuqrMFVZrfbIXbJkcjP8Z989AMf596ck0Z2BXUDIHBCDqnfqOnX246Vj76dvobkX7d9TZjwQpXkgD7qbiMBe54/z6Gn7vXlgOdzEn8VX6ViBhUizSAYEjAegJpc3kWqyNk5jU/MUXBxjCD+L8c8UEA7sLkqDyFLEff7n6VlLdSrnawGeCQoyc//AK6lN67A70Rs5+9k8nPqf9r9KXMi1ViS3rcvh9x5yC5J/j6qn9f61c0dSl7G0ZEJ3/61Ygg69ecmsO91PKkFGOckASFR/F6Y/vfpWhp1/b+Z50gIVSTkRAkDPuTnrXZOSVFMVKrB1NzsIH/cvmQEk8/P+vAq3uBXbnkv0BP93161St54ha7d8m88jjtjHYVbSVGffE4K7uAxIP3fQ9q+bjUTlfm3a/U+sgvdSGsAsYAYkFVzliKWQjceVz5v/PTJ7+tIQ2wEDjKjhj1x9KGbJO5sP5vIBHrWTaSstPl69zQgcfIevUc/nUUhzKFGMkDp9KSeXbGAoy+7PbpUcanG49TXkYhrRd/+D/mXGV3ZEpGDjIP0pDjtRRXK32NQ6UUVcs7F7hg7cRZ6+tb4fDVMRUVOkrsmUlFXZY0u2O4zsMDotalIqhFCqAAOABS1+jYHBxwlBUo/PzZ5lSbnK4UUUV2EBRRRQAVz/jv/AJJ54l/7BV1/6Kaugrn/AB3/AMk88S/9gq6/9FNQAeBP+SeeGv8AsFWv/opa6Cuf8Cf8k88Nf9gq1/8ARS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBzniz5o7KPGS8pAprD9yDu5z0/CrHiOO4b7G8Nq86o7F9gyV461C0MvkA+RJvI3Y29jXxmd4etLESkoNxtv8AJf5G+GlFSld66GtpP/IOjx7/AM6u1S0mKSHTYklUq/JIPXrV2vqMDFwwtOMt1FfkZSd3cKKKK6yQqCaxtLj/AF1rDJ3+dAanopNJ7iaT3Mmbw1pE3WzVT6oxX+RqjL4L09yTHNcR+24EfyrpKKzdGm94mUsPSlvFHGyeCJBnyr9T6Bo8f1qlL4S1aIZQQy+yP/jiu/orGWCovpYyeCpPbQ80l0XVIPv2Mx/3V3fyqo6vEdskbIfRhivVqRlDDDAEehFYSy2L2kQ8CukjyoFaXAPevSZdK0+YkyWUBJ6nYAT+NUpfC2lSZ2wvGT/cc/1zXNLLKn2ZEPBzWzRwZRT2FMaBD2x9K7GbwXbkHybyVD23qG/liqM3gy+X/U3UL/72V/xrL6jiI7GM8NU6xOXa2Xs2PrUTW7diK3JvDetQni18wY6o6n+ZzWfNZ38BImsblMdzG2Pz6VSp4iO6OWdK28WjOaFx2phV+y5qz5yZwTgj8KMo3f8AWrVSS3RzuEXsyrk+hpc4PpVrb/tNj68f54qN44+pyT7YUVSqRYnTa6jQr4zjj1P+fanYcdQf8/8A6jSNLt3beBgjgc/xdz9abJdbi2cHr1yf73+NHLJ7IfuLqPLYqhdElwvc++OxqWW76jPr3x6+lV42Dz5PAz6YHeumhTcLyZjUaeiN7S4zFCWCliB0BxnpXUADzRwOB6E/xVzNtOogKhyuR1VsEdK1Bex/aPNKKX27d3Ocbs18tjoTnUcrd/0PpMurU6VO1+xZVLh0UMqxnaPmVc87umCOmKmiVYSSuQfmOSOeT9azGvH6xgEbQNpXjr+dOgWa6m77CCD8oHf1rnqRkott2R2QxEXJKKbZpi5kZ9okOCnBOfvfTP8A9erBG5tzHJptpZFUVI0LlRjgVpRaVM/MjKg/M1zQw+JxbtRi2vw+/Y74tQV6j1KFORGc4RSx9AK2YtMgj5bLn36VbVFQYRQo9AMV62H4arS1rSUfTV/5fmTLFRXwoxYtMuJPvAIP9o1fg0yGI7n/AHje/SrtFe7hckweHaly8z7vX8NjnnXnIOlFFFeuYhRRRQAUUUUAFFFFAHP+O/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtHjv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc54s+aOyjxkvKQK6OsLxHHcN9jeG1edUdi+wZK8da4cyhOeFmoK70/NEytbXYrsP3IO7nPT8K2NJ/5B0ePf8AnWS0MvkA+RJvI3Y29jWtpMUkOmxJKpV+SQevWvnsioVIYtucWvd8+6O2vOLiki7RRRX15yBRRRQBBNY2lx/rrWGTv86A1Qm8NaRN1s1U+qMV/ka1qKlwjLdESpwl8Suc3L4L09yTHNcR+24EfyqjJ4IkGfKv1PoGjx/WuyorKWFpS3Rk8JRf2TgJfCWrRDKCGX2R/wDHFU5dF1SD79jMf91d38q9LorCWX0ns2ZPA0+jZ5S6vEdskbIfRhigFa9VZQwwwBHoRVSXStPmJMllASep2AE/jXNLLH9mRDwTWzPNsA96CinsK7yXwtpUmdsLxk/3HP8AXNUZvBduQfJvJUPbeob+WKweXV1syJYWoulzjmgQ9sfSomtl7Nj611E3gy+X/U3UL/72V/xrOm8N61CeLXzBjqjqf5nNJYbEx6HNPDyW8P6+Rhtbt2IqJoXHatGazv4CRNY3KY7mNsfn0qp5yZwTgj8KtOrH4kcsoRW+hWKv2XNJk+hq1lG7/rS7f9psfXj/ADxVe17oz9n2ZVzg+lSBXxnHHqf8+1OeOPqck+2FFI0u3dt4GCOBz/F3P1qr8y90FFL4mLhx1B/z/wDqNBbFMkutxbOD165P97/GoJbvqM+vfHr6URpTk9gcorZkV0SXC9z747GtvS4zFCWCliB0BxnpWDGwefJ4GfTA710FtOogKhyuR1VsEdKjMeZUlBIvB8vtOZnTADzRwOB6E/xVWVLh0UMqxnaPmVc87umCOmKrC9j+0eaUUvt27uc43ZqNrx+sYBG0DaV46/nXzFOlNdO259TPFU31+404lWEkrkH5jkjnk/WnC5kZ9okOCnBOfvfTP/16zIFmupu+wgg/KB39a2bSyKoqRoXKjHArGq5xlyxd5PojejJ1FdK0RxG5tzHJpavxaVM/MjKg/M1ci0yCPlsuffpW9DJMbX1ceVef+W/4HQ69OOxjIjOcIpY+gFWotMuJPvAIP9o1tKioMIoUegGKdXt4fhqjHWtJy9NF/n+RhLFSfwopQaZDEdz/ALxvfpV3pRRXv4fC0cPHlpRsjnlOUndsKKKK3JCiiigAooooAK5/x3/yTzxL/wBgq6/9FNXQVz/jv/knniX/ALBV1/6KagA8Cf8AJPPDX/YKtf8A0UtdBXP+BP8Aknnhr/sFWv8A6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigCKa2guFKzQxyAjBDqD/ADqhN4c0ecYbT4V/3Bs/litSik4p7kSpxl8SucfqvhXToDGYBLHu3ZAfPp61mSeGgEDJdOAezLnv65rrtZPMH/Av6Vn4BRen/fXvXxeZYurSxtSEHZJKy+SN4ZfhpwTcDmk8KXt0ZBC8LlMZDcZzVO48J6vGCTYM3+427+Wa7/Rf9Zc/8B/rWvXu5XzV8LCrN6u/5tHn18roc7SujxS40u5gB863miH+0uKqrbMrEj88V7rVWbTLC4IM1lbyEdC8SnH6V6Xs5pWucU8oX2ZHkUEUzEKo5PvWtZaVf3hBihdge4Bx+fSvR4tNsYDuisrdD6rEAf5VaxgYFcssE5v3pfcv6/I2pZco/EzkbDwg4w13Io/2V5rooNLtLdQFjDf73NXKKuGX4eL5nG789f8AgHo04qmrQ0EChRhQAPQUtFFdqVigooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaPHf/JPPEv/AGCrr/0U1HgT/knnhr/sFWv/AKKWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAqKa2guFKzQxyAjBDqD/OpaKAavuZc3hzR5xhtPhX/cGz+WKw9V8K6dAYzAJY927ID59PWuwrK1k8wf8C/pXlZzJ0sHOpDRq2vzQqeGo1JpSijkZPDQCBkunAPZlz39c1AnhS9ujIIXhcpjIbjOa6XAKL0/7696vaL/AKy5/wCA/wBa+fy3F1quKp0pPRr9L9h4jLMNytqNvmcBceE9XjBJsGb/AHG3fyzWVcaXcwA+dbzRD/aXFe10V9cqLW0jyp5VTfwto8KW2ZWJH54q9BFMxCqOT7167NplhcEGayt5COheJTj9KItNsYDuisrdD6rEAf5VFWjOatdGccqcZfFoecWWlX94QYoXYHuAcfn0ro7Dwg4w13Io/wBlea67GBgUVisupv423+H/AAfxO+lhIU9d2U4NLtLdQFjDf73NWwoUYUAD0FLRXVSw9KirU4peh1uTe4UUUVsIKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv8A2Crr/wBFNXQVz/jv/knniX/sFXX/AKKagA8Cf8k88Nf9gq1/9FLXQVz/AIE/5J54a/7BVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAyNbcL5OTjhsfpVDcMIA2STjAP+cVrarpn9oxqFk2OucH/P0qqdGnWPCzRlgOMgjv8AX0r5TMcpxFbE1K0Fo0vwS/yOinX5UotDtDffLd85wyjP51sVR03TvsHnfvN/mEHpjAH/AOur1e5llCdDCxpzWqv+bMZS5ncKKKK7yQooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArI1twvk5OOGx+la9Z+q6Z/aMahZNjrnB/wA/SvPzTDzxOFlShu7fmmVCfI+ZGTuGEAbJJxgH/OKvaG++W75zhlGfzpp0adY8LNGWA4yCO/19Kuabp32Dzv3m/wAwg9MYA/8A114mX5ViKGLhUmtEu/k0a1K3PGxeooor6swCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAorL8S3s+m+FdYvrVgtxbWU00TEA4ZUJBwevIrnrK08a32iW+qjxBbxX0sCzLYfY0NvkqCEZvvknoWBHU4HagDtaKzPDusJ4g8O6fq8cZiW7gWXyyclCRyM98HIrE8U6jrsfifw/pGi3MFut+l0Z5Zog+wIIyGA7kbiAM4yecgUAddRXIG51vw3relwajqv9radqc5tQ8lukUsE2xnUjYAGQhGGCMjjk81zFj4u8R2XgfSdX1LU0nu9amS2gCWBkW2GHZpSkQ3SMVQnaMDJHYE0AerUV5vp/iu+tda0+GPU9U1u3u51gnjudCmtmg3cCRXESrtBxkN2Oc8c6tnPr/iqe/vLLWhpOn293LaW0cVqkrymJijPIXBwCwbCqAcAc80AdRaalaX1ze29vLvlspRDcLtI2OUVwORz8rKeM9at1xHw/Oof2l4tGqGBr1dVVZHgBCPi3hAYAk4yADjJxnFdvQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/47/5J54l/7BV1/wCimo8Cf8k88Nf9gq1/9FLR47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRXnvhU+MfFHhay1i48Rx2Es0ZMUUFlHIrAEgNIW5OeuF24GOc0AehUVi+FtZn1vRjNeRRxXtvcS2l0kRJTzYnKMVzztOMj61tUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFZeq69a6PJGlxbalMZASDaafPcAfUxowH40AQeIfEieHFt5bjTb64tpZEjee3EZSEu6ou/c4PJYdAa265P4guJPBLuAwDXdiQGUg/8AH1F1B5FdZQBy+reMZdHS9muPC+uNaWgdpLpPs3llFyS4zMDjAzyAfapbrxjawNZwQ6dqV5fXVst19it4VMsUbdDJlgq85HLckHGcVW8Xf8TjUtJ8LJyl5J9qvgO1rEQSD/vuUX3BanaP/wAlI8UZ+99ksMf7v77+uaANnRdbs9ds3uLTzVMUjQzQzIUkhkHVHU9DyD9CCODWjXK+G8f8Jn4z2/d+2W2cf3vssef6V1VABRRRQAUUUUAFcvdeObGC4u1g03Vb22snaO6vLW23xRMv3h1DNt77Q2K0r3xDaWGoLZS2upvI23DwabPLGM+siIVHvzx3rK+HWP8AhCoc4z9qvN+fX7TLnNAHS211BeWkN3bSpLbzIJI5FOVZSMgg+mK5n/hYOl7Bdmy1MaQX2DVTb/6N1xuzndszxv27e+cc0zwBbi6+Fej2zMypLYBAw6hSCAR+BrmtQvddsPCVl4Bm0Em/vbc6TBeLNGbd4xHtabbu3jCDcQVwDgZ5GQDtNX8XWuk3z2aWGoX80MIuLgWUIcQRknDNkjrtbAGScHir82sIdEi1XT7W41OKZEkhjtNm+RGwQw3soxg55IrJ1HT5dU0S707w5rMVhfQYtLi6+ziV8rHwjZxg4dTnnGeOtTeBbqK68FaX5VqLVYIzaeSr7ghhYxHDHqMocHuKANHQ9Yt9f0Oy1a1SVILuISosoAcA+oBIz+NaFcr8Nf8Akm3h7/rySuqoAKKKKAMXxhG83gnXookZ5H064VVUZLExtgAdzWDpnhbXB4etNOj8U3MGmtbIhjNqpuY0KjKLMTxjkAlSw9cjNdxWXquvWujyRpcW2pTGQEg2mnz3AH1MaMB+NAHP/wDCWWeg3s3hrSvDGt3sekwxITYRRPGilAVXLSA7sdiM9+QQTjXN1L481zwpqunR6npKRpfGG4dY2ZGAiHzKrOuCd6lWweG4GAa6HwS4kuvFL4YM2tOSGUhgDDDjIPI4xTvAeP7P1jb9z+27/b6f69s/rmgDLmF1p/jPRj4kmv8AU2aUxafNb2sUFnFM6PksvmFy+xWGcYAJxyam1PQ9P8OfDa1stQvrgnStj213axgTeeGwnloSQWJbbtOQdxB4NXvGX/IQ8Jf9hxP/AEnnrN1qwu/FvjoacupzafaaFFFd5gRGeW4l3hW+dWXCKpxweW9RQBJeJdaQLHV/E+tXN95UqrZ6daWYiaWdgQAyqzeY4GeMhRgntkVdLW4utc1KDQ9RvfD9zOftlzpmp2CS8ucGaIh8ckc4Zhu6gZ50dIu1lm1KPxLPBcXHhq73JqLARKVeAMHYD5QwSRlPbuAM07w+LnxB4kfxZJC9tYLaNZ6dFIu2SaNnV2mYfwhii7QecDJ60AX/AAt4XHhkakTqE99Jf3X2qSWcDdvKKrdOOSpPAAGcAYFdBRRQAUUUUAZGt+IrXQ3toXguru8uiwt7S0j3yybRljgkAAZGSSByPWl0XxDa621zDHDc2t5alRcWl1Hslj3DKkgEgg4OCCQcH0rLm/5KxZbv+gJPsz/13izj9KIMf8LYvtv/AEBLfdj/AK7zY/rQBta3rNroGkT6lebzFEBhI13PIxOFRR3ZiQAPU1Hq2v2uiadDd3kc4ed1ihto08yaSRhkIqqTluD3xwTnHNcvrVhd+LfHQ05dTm0+00KKK7zAiM8txLvCt86suEVTjg8t6is6HUb+bx3omk6rOl1caXqk8Iu1QJ5weyaRCyjgOAxBxxxkAZoA66w8WQX01zaNpuo2uowwG4FjcxKksyDjKHcUbnA+9wSM4zVM+Nmhv7C1vPDGuWf264W2ikmW3K7yCedsxOAAScA8A0ur4/4WT4Xx977Hf5/3f3H9cU21/wCJ98Qbm7+9ZaDGbWH0a6kAaU/8BTYv/A2oA6yiiigAooooAKKKKAMT/hJEXxTHoM2m30Mk0ckkF04j8mUIF3bSHLcbwOVFbTusaM7sFVRkknAArltX/wCSk+F/+vLUP/aFdDqF3aWGn3F1fypFaRIWmeT7qr3J9qAMPSvG2n6rfWlulnqFvHfKzWNzcwBIrsAbvkOSR8oLDcFyBkUmpeONP0y7vI2s9RuLewIF9eW8AeG1JAY7jkE4UgnaGwDzisrxHa3ml+KvD+sS3Mdxo8V5HaQaekIj+zPMvkrKGH38biNpAwGOOlYU2la/r1j4tutDv4LPSL26njeymTdJO0f7qYrJ/wAst+wjkNjrxmgD1dWV0V0YMrDIIOQRS1m+H7+31Tw3pl/aRNFb3NrFLFG3VFZQQPwHFaVAHP8Ajv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLR47/wCSeeJf+wVdf+imo8Cf8k88Nf8AYKtf/RS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFYniHxInhxbeW402+uLaWRI3ntxGUhLuqLv3ODyWHQGp9V1610eSNLi21KYyAkG00+e4A+pjRgPxrH+ILiTwS7gMA13YkBlIP8Ax9RdQeRQB1lcvq3jGXR0vZrjwvrjWloHaS6T7N5ZRckuMzA4wM8gH2rqK5Pxd/xONS0nwsnKXkn2q+A7WsRBIP8AvuUX3BagCzdeMbWBrOCHTtSvL66tluvsVvCplijboZMsFXnI5bkg4zitLRdbs9ds3uLTzVMUjQzQzIUkhkHVHU9DyD9CCODWNo//ACUjxRn732Swx/u/vv65o8N4/wCEz8Z7fu/bLbOP732WPP8ASgDqqKKKACiiigAoorJvfENpYagtlLa6m8jbcPBps8sYz6yIhUe/PHegDNuvHNjBcXawabqt7bWTtHdXlrbb4omX7w6hm299obFdFbXUF5aQ3dtKktvMgkjkU5VlIyCD6Yrmvh1j/hCoc4z9qvN+fX7TLnNR+ALcXXwr0e2ZmVJbAIGHUKQQCPwNAD/+Fg6XsF2bLUxpBfYNVNv/AKN1xuzndszxv27e+cc1c1fxda6TfPZpYahfzQwi4uBZQhxBGScM2SOu1sAZJweK4vUL3XbDwlZeAZtBJv723OkwXizRm3eMR7Wm27t4wg3EFcA4GeRnrdR0+XVNEu9O8OazFYX0GLS4uvs4lfKx8I2cYOHU55xnjrQBrTawh0SLVdPtbjU4pkSSGO02b5EbBDDeyjGDnkil0PWLfX9DstWtUlSC7iEqLKAHAPqASM/jWd4FuorrwVpflWotVgjNp5KvuCGFjEcMeoyhwe4qt8Nf+SbeHv8ArySgDqqKKKACiiigArntQ8XW1nqk+nWum6nqdxbBWuRYwBxBuGQGLMBkjnaMnGOOau6rr1ro8kaXFtqUxkBINpp89wB9TGjAfjWP4JcSXXil8MGbWnJDKQwBhhxkHkcYoA6DStUs9a0yHULCXzbaYEqxBBBBwQQeQQQQQehFeTeF/wC3dC0Pw5pMF/qlhb6mBGJJrGKZIJmDsVVmkV14QthkYDPHoO78B4/s/WNv3P7bv9vp/r2z+uaPGX/IQ8Jf9hxP/SeegC2TpfgPwmzEzvbW2WJ/1k1xK7Z9t0ju3tyewq3q+vWuiaZFeXsc4aZ0iito03zSSt0jVQcFuvfHBOcc1y+tWF34t8dDTl1ObT7TQoorvMCIzy3Eu8K3zqy4RVOODy3qKt6RdrLNqUfiWeC4uPDV3uTUWAiUq8AYOwHyhgkjKe3cAZoA19H8TW2rXs1g9ne6ffwxiVrW9jCOYycB1KkqwzxwTg9cVtVyHh8XPiDxI/iySF7awW0az06KRdsk0bOrtMw/hDFF2g84GT1rr6ACiiigArI1vxFa6G9tC8F1d3l0WFvaWke+WTaMscEgADIySQOR61r1ys3/ACViy3f9ASfZn/rvFnH6UAami+IbXW2uYY4bm1vLUqLi0uo9kse4ZUkAkEHBwQSDg+lTa3rNroGkT6lebzFEBhI13PIxOFRR3ZiQAPU1iwY/4Wxfbf8AoCW+7H/XebH9azdasLvxb46GnLqc2n2mhRRXeYERnluJd4VvnVlwiqccHlvUUAdRq2v2uiadDd3kc4ed1ihto08yaSRhkIqqTluD3xwTnHNVLDxZBfTXNo2m6ja6jDAbgWNzEqSzIOModxRucD73BIzjNcjDqN/N470TSdVnS6uNL1SeEXaoE84PZNIhZRwHAYg444yAM10er4/4WT4Xx977Hf5/3f3H9cUAIfGzQ39ha3nhjXLP7dcLbRSTLbld5BPO2YnAAJOAeAa6uuTtf+J98Qbm7+9ZaDGbWH0a6kAaU/8AAU2L/wADausoAKKKKACiiigArE/4SRF8Ux6DNpt9DJNHJJBdOI/JlCBd20hy3G8DlRW3XK6v/wAlJ8L/APXlqH/tCgDqXdY0Z3YKqjJJOABXN6V420/Vb60t0s9Qt475WaxubmAJFdgDd8hySPlBYbguQMitzULu0sNPuLq/lSK0iQtM8n3VXuT7VxviO1vNL8VeH9YluY7jR4ryO0g09IRH9meZfJWUMPv43EbSBgMcdKANXUvHGn6Zd3kbWeo3FvYEC+vLeAPDakgMdxyCcKQTtDYB5xXSqyuiujBlYZBByCK8om0rX9esfFt1od/BZ6Re3U8b2UybpJ2j/dTFZP8Allv2EchsdeM16N4fv7fVPDemX9pE0Vvc2sUsUbdUVlBA/AcUAaVc/wCO/wDknniX/sFXX/opq6Cuf8d/8k88S/8AYKuv/RTUAHgT/knnhr/sFWv/AKKWugrn/An/ACTzw1/2CrX/ANFLXQUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBn61pFvrumNYXTypE0sUpMRAbMciyDqDxlRn2zUl5p63l1YztcXERs5jKEifaspKMm1xjlfmzj1AParlFAGdb6Nb2+u3usb5ZLq7ijhO8grHGmcKnHAJZic5yT9Kq6t4ZttUv49RjvL3T79IvJ+02UgVmjznawYMrAHJGRxk46185fF7/kqms/9sP/AERHXGrXZDCc0U7nUsNdJ3PsjRNDtNBs5Le1MztLK00808heSaRsZZmPU8AegAAFaVfFa1MtX9S/vfgS6Fup9nUV8bLUq0fUf734EOnbqfYlFfH61IKf1H+9+H/BM2rH15XMXfgeyuJ7swalqtjbXrtJdWlpcBIpWb7x5UlS3faVz1r5uFSCl9R/vfgZuVj6ih0W2tbyxmtnmghsrVrWK1jfEOw7MEr3ICAA54BPrQmiWy+Ipdcd5ZbtrcW0YcjZCmcsEAHBY4JJJztHQCvmAU8UfUv734EOtbofRuo+Eba+1Ka/ttS1LTJ7lQtybCYIJ8DALAqfmA43Lg4xzwK19N0600jTbfT7GEQ2tugSNAScD6nkn1J5NfLwpwpfUv734GbxNuh9M6HpFvoGh2ek2ryvBaRCKNpSCxA9SABn8K0K+WRTxS+p/wB4h4y32fxPqOivl4U8UvqvmZvH2+z+P/APp6ivmQU4VP1bzIeZW+z+P/APedQ8JW95qU+oWupalplzcqq3LWMyoJ9owCwZWGQONwwcY54FT2vhmxsLXSraykubeDTZWlSOOU4mLKwPmE5L5Llv97BrwEU8Uvq/mQ80t9j8f+AfQmp6Rb6rNp0s7yq1hdC7i2EAFwjphsg8Yc9Mdqqat4YttU1CPUYry907UEj8n7VZSBWePOdjBgysM8jI4ycYzXhIpw6VLoeZLze32Px/4B7Df/DvTL7RotMF/qUEYu/tk0qSo73Uv96XzFYPyAcYxwOMAVraVol1pt0003iHVdRUoVEN55GwHIO4bIlOeMdccnivCxTxUunbqT/bP9z8f+AfRNFfPQp61DjYX9tf3Px/4B9BUV4EKlWs3Kwf2z/c/H/gHsWteHrXW3tpnmubW8tSxt7u1k2Sx7hhgCQQQcDIIIOB6VTt/B9nb6dqlv8Ab9Re61OPy7nUHnH2ggKQu1gMLtycAAAEnivLlqVaylWt0LWb3+x+P/APUtU8LW2pXsV/Fe32n38cXk/arOUK7x5ztYMGVhnkZHGTjGag/wCEJ0j+x/7P/wBJ3faftn2zzz9o+0f89fM67scemOMY4rzlamWsZYy32S1ml/sfj/wD0fSfDFvpeoyalLe32o37xeSLm9lDMkec7FChVUEgE4GTgZ6Vb0TRrfQtO+x2zyyAyyTSSzEF5JHYszMQACSSe1eZLUq1jLMbfZ/H/gGqx9/s/j/wD1mivLFqVawlnHL9j8f+AbRxV+h6dRXmy1Ktc8s/5f8Al3+P/ANo1b9D0SiuAWpVrCXEtv8Al1/5N/wDaKudVc6Rb3WuWGrO8onsopoo1UjaRJs3ZGM5+QY5HfrSS6NbXF5fT3LzTxXtsttLaytuh2DfnC46tvIPqAPSubWpkrP/AFp/6df+Tf8AAN40L9SzY+CbKzu7SabUdUvobJt9nbXlwHigbGAQMAsQCQC5bHakvPA9jdXF40Wo6pZ2185ku7O1uAkU7N94ngspbvsK5+tWNM/4/ovx/ka369zLMf8AXqLq8vLZ23v0T7LuZ1afI7XI7e3htLaK2t41ighQRxxqMBVAwAPYCpKKK9EzOf8AHf8AyTzxL/2Crr/0U1HgT/knnhr/ALBVr/6KWjx3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+iloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKz9a0i313TGsLp5UiaWKUmIgNmORZB1B4yoz7ZrQooAp3mnreXVjO1xcRGzmMoSJ9qykoybXGOV+bOPUA9qit9Gt7fXb3WN8sl1dxRwneQVjjTOFTjgEsxOc5J+laNfK/xe/wCSqaz/ANsP/REda0aXtJWuaUqftHa59G6t4ZttUv49RjvL3T79IvJ+02UgVmjznawYMrAHJGRxk461Z0TQ7TQbOS3tTM7SytNPNPIXkmkbGWZj1PAHoAABXxutTLXT9S/vfgbPDW6n2pRXxitSrT+o/wB78DN0bdT7Jor47WpFp/Uf734f8EhwsfYFFfIYqQUvqP8Ae/D/AIJm3Y+kbvwPZXE92YNS1Wxtr12kurS0uAkUrN948qSpbvtK561qw6LbWt5YzWzzQQ2Vq1rFaxviHYdmCV7kBAAc8An1r5dFPFH1H+9+Bm6luh9PpolsviKXXHeWW7a3FtGHI2QpnLBABwWOCSSc7R0AqhqPhG2vtSmv7bUtS0ye5ULcmwmCCfAwCwKn5gONy4OMc8CvnIU8UvqX978CHiLdD6h03TrTSNNt9PsYRDa26BI0BJwPqeSfUnk1DoekW+gaHZ6TavK8FpEIo2lILED1IAGfwr5mFPFL6n/e/AzeLt0PqaivlwU4UvqnmQ8db7P4n1DRXzCKeKX1XzIeYW+z+P8AwD6brn9Q8JW95qU+oWupalplzcqq3LWMyoJ9owCwZWGQONwwcY54FeDCnCl9W8yHmdvsfj/wD3618M2Nha6VbWUlzbwabK0qRxynExZWB8wnJfJct/vYNWdT0i31WbTpZ3lVrC6F3FsIALhHTDZB4w56Y7V89inip9h5kPNrfY/H/gHu2reGLbVNQj1GK8vdO1BI/J+1WUgVnjznYwYMrDPIyOMnGM1mX/w70y+0aLTBf6lBGLv7ZNKkqO91L/el8xWD8gHGMcDjAFePDpTxUul5kvOP7n4/8A900rRLrTbpppvEOq6ipQqIbzyNgOQdw2RKc8Y645PFbFfOwqQVLhYn+2f7n4/8A+haK+fVqUVm9A/tn+5+P/APfayda8PWutvbTPNc2t5aljb3drJslj3DDAEggg4GQQQcD0rx1alWsnUt0Gs4/ufj/wAA9Rt/B9nb6dqlv9v1F7rU4/LudQecfaCApC7WAwu3JwAAASeKm1TwtbalexX8V7faffxxeT9qs5QrvHnO1gwZWGeRkcZOMZry1alWspYm3QtZrf7H4/8AAPRv+EJ0j+x/7P8A9J3faftn2zzz9o+0f89fM67scemOMY4qfSfDFvpeoyalLe32o37xeSLm9lDMkec7FChVUEgE4GTgZ6V5wtSrWMsdb7P4/wDANVmV/s/j/wAA9N0TRrfQtO+x2zyyAyyTSSzEF5JHYszMQACSSe1aNeTLUy1hLNbfY/H/AIBrHG3+z+J6nRXmK1KtYSzvl/5d/j/wDaOIv0PSaK87WpVrCXEXL/y7/H/gG0ZXO/rPudIt7rXLDVneUT2UU0UaqRtIk2bsjGc/IMcjv1rlVqVaxfE9v+XX/k3/AADeNO/U6SXRra4vL6e5eaeK9tltpbWVt0Owb84XHVt5B9QB6VlWPgmys7u0mm1HVL6GybfZ215cB4oGxgEDALEAkAuWx2qslaGmf8f0X4/yNXh+JPbVo0vZW5ml8Xd+ho8PaLdyveeB7G6uLxotR1Sztr5zJd2drcBIp2b7xPBZS3fYVz9a6O3t4bS2itreNYoIUEccajAVQMAD2AqSivqDmCuf8d/8k88S/wDYKuv/AEU1dBXP+O/+SeeJf+wVdf8AopqADwJ/yTzw1/2CrX/0UtdBXP8AgT/knnhr/sFWv/opa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigD5X+L3/JVNZ/7Yf+iI641a+xbzwxoGoXb3V7oemXNzJjfNNaRu7YGBkkZPAA/Cof+EN8Lf8AQtaP/wCAMX/xNd0MWoxSsdccQlFKx8jLUy19af8ACHeGP+hc0j/wBi/+Jo/4Q/wz/wBC5pH/AIBR/wDxNV9dj2JddPofKC1KtfVf/CI+Gv8AoXdJ/wDAKP8A+Jpf+ES8Nf8AQvaT/wCAUf8A8TT+ux7GbqJnyutSCvqT/hE/Df8A0L+lf+AUf+FL/wAIp4c/6AGlf+Acf+FH16PYzbufLwqQV9O/8Ir4d/6AGl/+Acf+FH/CK+Hf+gDpf/gHH/hR9dj2M3G58yinivpj/hFvD3/QB0v/AMA4/wDCj/hF/D//AEAtM/8AASP/AApfXY9jN0W+p81CnCvpT/hF/D//AEAtM/8AASP/AAo/4RjQP+gHpn/gJH/hR9cj2M3hm+p83Cnivo7/AIRnQP8AoB6b/wCAkf8AhS/8IzoP/QE03/wEj/wqfrcexm8HJ9T5yFPFfRX/AAjWg/8AQE03/wABU/wo/wCEb0L/AKAum/8AgKn+FL60uxm8DJ9T54FOFfQ3/CN6F/0BdO/8BU/wo/4RzQ/+gLp3/gKn+FL6yuxDy6T+0fPYp4r6B/4RzQ/+gNp3/gKn+FL/AMI7of8A0BtP/wDAVP8ACp+sLsQ8rm/tI8AFOHSvff8AhHdE/wCgPp//AICp/hR/wj2if9AfT/8AwGT/AAqXWXYh5TN/aR4KKeK94/4R/Rf+gRp//gMn+FH/AAj+i/8AQIsP/AZP8Kl1EyHk8/5keFinrXuP9gaN/wBAmw/8Bk/wpf7B0f8A6BNj/wCA6f4Vm5XF/Y0/5keJCpVr2j+wtI/6BVj/AOA6f4Uv9h6T/wBAuy/8B0/wrNxuNZPP+ZHja1Ktewf2JpX/AEDLL/wHX/Cj+xtK/wCgZZ/9+F/wrGVFvqWspmvtI8kWplr1b+x9L/6Btn/34X/Cl/sjTP8AoHWn/fhf8KxlhJPqaLLJr7R5atSrXpv9k6b/ANA+1/78r/hS/wBlad/z4Wv/AH5X/CueWXSf2jWOAkup5utSrXon9maf/wA+Nt/36X/Cj+zbD/nytv8Av0v+Fc8som/tI2jhWupwC1Ktd3/Z1j/z52//AH6X/Cl/s+y/59Lf/v2P8K55ZFUl9tG8abRxC1Ktdl9gs/8An0g/79il+xWn/PrD/wB+xXPLhuq/to6IuxyK1MldR9jtf+faH/vgUv2S2/594v8AvgVk+GK3/PxfczeNdLoY2mf8f0X4/wAjW/UawQowZIkVh3CgVJX0GVYCWBounJ3u7/gv8jOrUU3dBRRRXpmRz/jv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtHjv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXyv8Xv+Sqaz/wBsP/REdfVFZV54Y0DULt7q90PTLm5kxvmmtI3dsDAySMngAfhW1Goqcrs1pVFCV2fHS1MtfXP/AAhvhb/oWtH/APAGL/4ml/4Q7wx/0Lmkf+AMX/xNdX1yPY2eIT6HyWtSrX1f/wAIf4Z/6FzSP/AKP/4ml/4RHw1/0Luk/wDgFH/8TT+ux7Gbqp9D5UWpFr6o/wCES8Nf9C9pP/gFH/8AE0f8In4b/wChf0r/AMAo/wDCj69HsZudz5bFSCvqH/hFPDn/AEANK/8AAOP/AAo/4RXw7/0ANL/8A4/8KPrsexm1c+YhTxX01/wivh3/AKAOl/8AgHH/AIUv/CLeHv8AoA6X/wCAcf8AhR9dj2M3TbPmcU8V9K/8Iv4f/wCgFpn/AICR/wCFL/wi/h//AKAWmf8AgJH/AIUvrkexm6DfU+axTxX0j/wjGgf9APTP/ASP/Cj/AIRnQP8AoB6b/wCAkf8AhS+uR7GbwrfU+cRThX0b/wAIzoP/AEBNN/8AASP/AAo/4RrQf+gJpv8A4Cp/hS+trsZvBSfU+dRTxX0P/wAI3oX/AEBdN/8AAVP8KX/hG9C/6Aunf+Aqf4VP1pdiHl8n1PnkU4V9Cf8ACOaH/wBAXTv/AAFT/Cj/AIRzQ/8AoDad/wCAqf4UniF2M3ls39o+fhTxXv8A/wAI7of/AEBtP/8AAVP8KP8AhHdE/wCgPp//AICp/hU+3XYh5VN/aR4EOlPFe9f8I9on/QH0/wD8Bk/wpf8AhH9F/wCgRp//AIDJ/hUuqiHlE/5keDipBXun/CP6L/0CLD/wGT/Cj+wNG/6BNh/4DJ/hUOdyf7Hn/Mjw5alFe2/2Do//AECbH/wHT/Cj+wtI/wCgVY/+A6f4Vm9Q/saf8yPF1qVa9k/sPSf+gXZf+A6f4Uv9iaV/0DLL/wAB1/wrKVNspZRP+ZHj61Ktet/2NpX/AEDLP/vwv+FL/Y+l/wDQNs/+/C/4VjLDN9TRZXNfaR5StSrXqX9kaZ/0DrT/AL8L/hR/ZOm/9A+1/wC/K/4VhLAyfU0jl0l9o8yWplr0j+ytO/58LX/vyv8AhS/2Zp//AD423/fpf8K55ZXN/aRtHByXU87WpVrv/wCzbD/nytv+/S/4Uv8AZ1j/AM+dv/36X/CueWS1H9pG8aDXU4RalWu3/s+y/wCfS3/79j/Cj7BZ/wDPpB/37Fc8uHqr+2jeMbHGrUq1132K0/59Yf8Av2KPsdr/AM+0P/fArB8M1X/y8X3M3jUSOXStDTP+P6L8f5Gtn7Jbf8+8X/fApywQowZIkVh3CgVphuHKtGtCo5r3Wn9zNXiE4tWJKKKK+tOUK5/x3/yTzxL/ANgq6/8ARTV0Fc/47/5J54l/7BV1/wCimoAPAn/JPPDX/YKtf/RS10Fc/wCBP+SeeGv+wVa/+ilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8k88S/8AYKuv/RTUeBP+SeeGv+wVa/8AopaPHf8AyTzxL/2Crr/0U1HgT/knnhr/ALBVr/6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/x3/yTzxL/ANgq6/8ARTV0Fc/47/5J54l/7BV1/wCimoAw/BfjTwra+BfD1vceJdGhni0y2SSOS/iVkYRKCCC2QQeMVuf8J34P/wChr0P/AMGMP/xVcbpmleENI+HPhfUL3wdY6jc3trZwhYdPgeWWWSIHJL7QcnOSTWlp9h4Ju9Vi0y88A2ul3c6s0CX2lW4E23lgrJuUkDnGc45oA6D/AITvwf8A9DXof/gxh/8AiqP+E78H/wDQ16H/AODGH/4qj/hBPB//AEKmh/8Aguh/+Jo/4QTwf/0Kmh/+C6H/AOJoAP8AhO/B/wD0Neh/+DGH/wCKo/4Tvwf/ANDXof8A4MYf/iqP+EE8H/8AQqaH/wCC6H/4msMaZ4FfUZ7KPwTp8kkGoJYSMmlQFVdohKHPHCYIBPqenegDc/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4ms/W/DngfQNHn1O68I6O8MO3csWmwljlgoxkAdSO9AGh/wnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVUx4V8GnWX0z/hDNLDrbrceedLh8ogsV2hsfe+XOMdCKuf8ACCeD/wDoVND/APBdD/8AE0AH/Cd+D/8Aoa9D/wDBjD/8VR/wnfg//oa9D/8ABjD/APFUf8IJ4P8A+hU0P/wXQ/8AxNH/AAgng/8A6FTQ/wDwXQ//ABNAB/wnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVRXPgvwfbWk0/wDwiGjSeUjPsj06Es2BnAyByayLjT/h7beDP+Eqbwro7ad9kW7AXTYN5VgCFAxjccgYz170Abn/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VWPe6R4BsvC6+ID4T0iWzeKOWNYtNgLyCQqEABAGSWA6962P+EE8H/8AQqaH/wCC6H/4mgA/4Tvwf/0Neh/+DGH/AOKo/wCE78H/APQ16H/4MYf/AIqs/U/DngjSZNPjn8I6OxvrtbSLy9NhOHKswLZA4wh6Z7cUaT4c8EazHdSW/hHR1Ftdy2j+ZpsIy8bbWIwDxkcfyoA0P+E78H/9DXof/gxh/wDiqP8AhO/B/wD0Neh/+DGH/wCKrm9Vt/AemX91aJ4Etb5rJFe8ez0eF1tgRuG7IBJ284UE4xxzV6y0bwBqOqpYWnhbRpTJYx6hHMumw+W8TsQuDjOflz06EUAa3/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNAB/wnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATXK2r/Dy6ubUL4Js0sbucW9tqUmjwi3mkJwoU43YJGASoB7HmgDqv+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qs+08OeB73WNR0yPwjo4m0/yvNZtNh2t5i7ht4z0HOQK0P+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqP8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaAD/hO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKo/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JoAP+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqP8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaAD/hO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKo/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JoAP+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqP8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaAD/hO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKo/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JoAP+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqP8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaAD/hO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKo/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JoAP+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqP8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaAD/hO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKo/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JoAP+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqP8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaAD/hO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKo/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JoAP+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqP8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaAD/hO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKo/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JoAP+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqP8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/iaAD/hO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKo/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JoAw/GnjTwrdeBfENvb+JdGmnl0y5SOOO/iZnYxMAAA2SSeMUeC/GnhW18C+Hre48S6NDPFplskkcl/ErIwiUEEFsgg8Yo8aeC/Ctr4F8Q3Fv4a0aGeLTLl45I7CJWRhExBBC5BB5zVbw34c8FWnw20TVtX0HQ1QaXbS3FzPYxMWYxrkklckkn6kmgDpf+E78H/9DXof/gxh/wDiqP8AhO/B/wD0Neh/+DGH/wCKrlingmKE3V18N/sunjk3s2hwbFX+8yjMij3KDHfFdNF4J8FzwpND4Y0CSKRQyOlhCQwPIIO3kUAP/wCE78H/APQ16H/4MYf/AIqj/hO/B/8A0Neh/wDgxh/+Ko/4QTwf/wBCpof/AILof/iaP+EE8H/9Cpof/guh/wDiaAD/AITvwf8A9DXof/gxh/8AiqP+E78H/wDQ16H/AODGH/4qj/hBPB//AEKmh/8Aguh/+Jo/4QTwf/0Kmh/+C6H/AOJoAP8AhO/B/wD0Neh/+DGH/wCKo/4Tvwf/ANDXof8A4MYf/iqP+EE8H/8AQqaH/wCC6H/4mqGi+GvBGu6Rb6la+EdHSGcEqsumwhhgkc4BHb1oAv8A/Cd+D/8Aoa9D/wDBjD/8VR/wnfg//oa9D/8ABjD/APFVy1tb+FLjXf7IPwsEVyqRyyGTTrDbHG7MockSHjKN0yeOnSup/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4msG3sfAFzpui36eD9MEWrzi3tw2mQblYq7Zf0GIz0z2oA3v+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4mgA/4Tvwf/wBDXof/AIMYf/iqP+E78H/9DXof/gxh/wDiqxoNL8AzeGLnXz4T0iK0tknaZJNNgDp5RZXBAGMgoe9MbT/h+vg6PxOvhHSnspIEmSNdMg8078BUxjG7JC4z170Abn/Cd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVR/wgng//AKFTQ/8AwXQ//E1Q1Lw34I0ubTop/COjs1/dC1i2abCQHKO+WyBxhD0z2oAv/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVVDRfDfgjXdEs9WtfCWjpb3cQlRZdNhDAH1wCM/jWPYr8P766s0XwRZRWl9IYrO/l0iAQXDYJAU/eGQCQWUA9s0AdP/wnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVUNE8N+CNe0Sz1W18I6OkF3EJY1l02EMAfUAEZ/Gr/8Awgng/wD6FTQ//BdD/wDE0AH/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNYeraX4J0vUBp8XgSy1G88n7RJDZaVAxjjJIDMW2jkhgACScHAoA3P8AhO/B/wD0Neh/+DGH/wCKo/4Tvwf/ANDXof8A4MYf/iqxbXTPh9fS6Qlr4W0eVNWhkmt5BpkIUKgUsGyAQfmAxjqDnFbX/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNAB/wnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNAB/wnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNAB/wnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNAB/wnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNAB/wnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNAB/wnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNAB/wnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VWH408aeFbrwL4ht7fxLo008umXKRxx38TM7GJgAAGySTxitz/hBPB//QqaH/4Lof8A4msPxp4L8K2vgXxDcW/hrRoZ4tMuXjkjsIlZGETEEELkEHnNAGXcPcx/DD4ePaQRz3Cy6YY4pJPLV28rgFsHA98Guji03xBrPiPS9R1m20/T7XTGkligtrlrh5ZGQx5ZiiBVCs3ABycUzwzo9vq/w88G/aHlX7JaWN3H5ZAy6RLgHIPHP/167CgDx7T9Q1zT/h/oF6NT1jUNQ124S2d0kR3hjAkbEQkIXeQmNzEnJJ7AVrWdxrOma3pjWOneKUt57lYbtNZu4JomRuCynzmZXXrheCARj062LwjpcfhS38OSCaayt1URu8m2VWU7lcMuMMDyCMUyx8I21tqMF/eajqWqT22fsxvplZYCRglVVVG7BI3HJxnnk0AZGl2Vz4wl1LUb3WdTtYob6e0trSxuTAIVicpufbyzsVLfNkAEDFcvpv22y1yaCfUXu5h4whikuBhTKgsuAwXAJwFz2yDxXfXng+2m1G4vrLUtT0uW6Ia5FjMFSZsY3FWVgGwANy4Jx1qCx+H+jab5f2VrtAmorqXzS7y0wj8vLFgSQQSTzksc5oAz9NsrnxjJqV/e6zqdpFDfT2lta2FybcQrE5Tc+3lnYqW+YkAEDHrR8d6FdD4bXKapq93eT2jr5c8UjW5kRpUAEqowV2A7kYzyADXS3vg+2n1C4vbLUtT0qa6Ia5FhMFWZgMbirKwDYAG4AE4609/B2lHwvN4fjE0NpM2+WRZN0rvvDl2ds7mJHJOaAObvryTwnr+tSW0l1dx6d4Z+1Rx3Nw8xdhLM3zMxJPTGfQe1WJ/D2pQeG31lfFWptrEdsbnzzP8A6KzBd23yfueX26Zx3zzXUtolo+uzas+95prNbJ4mwYzGGZumM5JYg84x2rE/4V9p5txYnU9XOkDj+yzdfuNv9zON+ztt3YxxjFAGNbyX3i7xbYl9T1HT9PuPDttey2lpcNHmR3f+IcrgcEjBOBzgYrS0kautx4n8NW+sTNNaJE9hfXSiWSHzUbAbI+faykjPODg5ro49FtYtebWEMiztaJZ+WMCMIrMwwMZzlj3xjHFVbrwza3M2tTC6vIJdXt0t5pIZArRBAwDRnHDfOeTntQBtKCEAY5YDk+teV2oDQ2PgYgH7P4hdHjPe0i/0tM+2GiSvVFXagXJOBjJ61hx+E9Oj8Zy+KVM32+S1+zFCw8sDI+YDGd2FUZz0HSgDiNKJuI/D3g1iSdN1mcTA/wDPC1PmRZHv5ltW3pdlc+MJtS1G91nU7WKG+ntLa0sbkwLCsTlNz7eWdipb5sgAgYrdtPCenWXi+/8AE0Rm+3XsKwyKzDy1ACjKjGQSETPP8IqC78HW02o3F9Y6lqely3RDXK2M4VJmxjcVZWAbAA3LgnHWgDhLSXUf7VjtdR1SXUntPGMcEc0h6ILQkDA4B55wAN2Tjmu08Cf8eeuf9hy+/wDRposPh/o2miMWrXaBNRXU/ml3lphH5ZLFgSQQSTzkk5zW1pOj2+jR3Udu8rC5u5bt/MIOHkbcwGAOMnj+dAHO3Om6nBrGrar4V1HT7priRRf6bdDchmRFXAkU5jYoEBDAjoeM1z+mxxeMPGVld2dzeaXp0vh2CRrezfyX/wBdIAm9eVCnP3cZwO1ddqHgy0vNRur231LVNOa8A+1x2NwEScgbcsCpw2ABuXacAc1d0/w1p2l6lHe2SPD5djHYRwKR5aRIzMuBjOfmPOaAOKv9e1fwtp3ijT4Lu5v5LCSzFlPcbZJY1uWCYYkgPtO4gse4BOOarXE+u6TEl9puneMjeRuhk/tS8tnt5xkBlZfOITIzgoBg44PSu9k8NadcXmrz3MbXC6rDHBcwyEFCqBgMDGR949/TGKoweCbRZ7drzVNW1C2tnWSC0vLkPEjLypOAGcjqN5bB560AUvJuvFXinWrWfVL6ysNKkit47aynMDSO0ayGR3X5sfOAACB8pPOa5HW5td0m48XadD4jvpjbwaZ9ilklO+ASXJBDYwCexPVl2g5r0PVPCtrqGpHU7e9v9Nv2jEUlxYyhDKg6B1YMrYycEjIz1rOT4c6Miah+/v2l1DyDdTyT75JGhk8xWJYHnPB7YAAAxQB0FhZJpOleRNe3FykYZpLi9l3Me5LHgAewwAK4qNb3wZothtnsNb8IpLAkJZMT28buoiZWBKSqpZcHCnHOTXobKrqVYAqRggjgiuVtfAGmWr28QvdTl021lE1vpktxut4mU5XAxuIU4IUsQMDjigDkpNGTStS+IV5a6hqyz2Wnh4XbUZm+Y2znLZb5iD93P3e2Kmu7nWNNtvDmkrea9qMmrQSXl7Payx+ediR/u4jIyrGuXycfNgepJruLjwxZXLa8XluAdbgEFzhl+RRGY8pxwcE9c807UPDVjqOm2dnK08bWW021zDJsmhYLt3Kw7kZBHQ55FAHKaHc61ZeJbWzgstfi026ilEo1y5hm8p1XKvGwlaQjPBU5HIPFUNGHiC3hvdOl1TVbfxVNpkxWHUJBLbTzgjE0DjKqoLAbABgMMrxmuzsPCVla3M11eXN5ql3LC1uZ7+QOViP3kUKFVQe+Bk4GScVnQ/DfRlge3u7nUb+3Fo1lbxXdxuFrC2MrHgAg/KvJJPyjmgCPwNMY7m/sZ7zWRdxxxPLYau/myQk7gXSXJDoxB6HAK9s4rtKxNE8MW+i3dxem+vtQvZ0SJrm+lDuI1yVQYAAALE9MknkmtugAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/JPPEv8A2Crr/wBFNXJKVi+H/wAOby7/AOQXa/YZLwn7qD7OVjdv9kSNGfbg9q63x3/yTzxL/wBgq6/9FNTPBMUc/wAN/DkU0ayRvpFsrI4yGBhXIIPUUAGq6r4gszcyxaPo8mnRqW+0XOrNFlMZJYeQwH5muTaGDxjqvgiTUtM+w209jfSmwjlOwxgw7ASApKkbWxgdgRXVL4A8MK6H+zN0UZDJbvPK0CkdMQltg/75rbl020m1G11CSLN1apJHC+4jar7dwxnBzsXr6UAcZoeg6X4sGq6hr0Avb9dRubbZK7f6IkchVEjAPyHYFbcME7856Vn+Hbye51/ws1xcvcpF/a9pb3UjZa4jSWMRsT/ESiHnvgmux1LwfoWrXkl3d2R8+VQszwzyQ+cBwBIEYBxjj5s1YvvDmkajpsGn3FhF9ltypgSPMZhKjAKFSChA44I4oA878fbLzXfEFskzIwsNHido2w0ZN+54PY4YH8q3dR0Sw8J674dvNDtvsz3d61ncxo5xco0Erjfk/MwaNSGPPXnmtuLwT4dhjmRNOA8/yvOczSF5DHJ5iFmLZYh+ck5PQ5HFa15p1pfy2klzF5j2c/2iA7iNkm1lzweeHYYPHNAHC+HPDui6/wCDrXX9WdpNVuoTcXOpGZkltperKjZ/diMgrtGANvI61r/DBt3w20RvM8zMJO8jG7525q9c+CfDt5eS3U2mqWmfzJo1ldYpX/vPGGCOfcg5rW0/T7XSrGKysoRDbRAhIwSQoJz39zQBgW3/ACVLVP8AsC2f/o65rh/DWk2mm+EPAN+gYz3epW7XE8jkk/uJ1ReegG8KAPX1NerLp1qmqy6msWLyWFLd5Nx5jRmZRjOOC7c4zz9Kqf8ACN6QfD0egtYo+mRxrGkDksFA6YJOcg8g5yKAMHxDbWGq/EPQtNvoILqJtMvmlt5lDqVL2+3KntlT+Vcdp/h7TH8CeBYIbVLZb7Vo2umtv3TTYhnzuZcHkDH0Nelab4S0TSb5L60syLxEZBcSzSSyFW25BZ2JI+VcA9O2MmpovDulQWem2kdrtg02UTWi+Yx8twrLnOcnh26560Ac5HpFn4d8dWNjpCGystU0+6M8ELEIJImi2yKvQNiRgSOvGelddp9t9i0y1tftM1z5EKR+fO+6SXaANzHuxxkn1NMutKsry9gvLiHfcQRSwxvuYbUk27xgHHOxfpjipLGyt9N0+2sLSPy7a2iWGFNxO1FACjJ5PAHWgDzfVf3Or6r4O7avq1rcxL3MEoLz49s282f9+i2+fWLXwV/Daa9LeMn/AE6oBcx/h5ksS/8AAa7+fQdMudetdbltVbUrWJoYZ9xyiN1GM4PfqO59aE0LTI/EEuvJaqNTltxbPcbjkxg5C4zjrjnGeBQByeh6Dpfiw6tqOvQC9v11G5ttkrt/oaRyFURAD8hKBW3DBO/Oelc74eWNLLR0hu3vIk8Z3KpcSPvaRRFOAS38RwOvfrXoepeD9C1W9kvLuxP2iVQszwzSRecBwBIEYBxjj5s1Lb+F9FtIraK20+OGK2ujeQpGWVUlKldwAOOjEY6c9KAMr4fyRxfDDQ5JjiJNPVn4zwBzxWXaGfwvY6E9hqMGreGLm4t7e1hni/fQJKQImjkH3lXcOGXO0deK7bTdOtNI0230+xi8q1t0EcUe4ttUdBkkk/jWXZeDPD+nX6XtrpyxyxszxL5rtHExzkpGTsQ8nlQOtAHnmleF9OtvgrHr8KyJrVtpT3sN+JG82N0Quqg54QYC7ehHar8kGo+KPFWsC70HTdXitfIW3tr7UXhSGN4VfesYicEsxb5yc/Ljjbz6BHoOmReHToCW2NLNu1qYPMb/AFRBUruzu6E85zUOo+FtG1R4JLq0bzYI/KjmhmkhkCf3d6MGK+xOKAOAgkvxp1tol5exw6Vca/8AYZPsl/JO1vF5TN9mMxVG5lUJ6gNtzV3xL4c0TQPEHhGTS0Wwkm1eNGtYXIScBHO4pnBZTj5uvzEHrXcf8I9o40P+xP7Otv7M27fs2wbMZznHrnnPXPPWqNr4I8PWl3FdpYF7mF1eKae4kldCM4Cs7Egc9Bx7cUAZHgDSLSO513VyrPey6vfwh3cnZH9ob5FHQAkZPqau6jp7Xnie6utB1pbLW4LaKO6glg82KWMl2j3rwepfDKw6nOa6Cx06002OaO0i8tZp5LhxuJzI7FmPJ7kk46VS1bwvo+t3MdzfWrNcRoYxNFNJC5QnO0sjAlc9jkUAcFa2Wj+JvEvhTVbzQdOW6uI75bkLCrK8kDKgYEjkAqSpPY1Ve1fS/CninXbBnXUptbubVrhp2T7PbtehXCtg+WMZYsASPvc4FenRaHpkEthJDZxxHT42itRHlViRgAwCjjoo6jtT7bSLC0tbm1itk+z3Uss00b5dXaRiz5DZ4JJ46c0AedS6NfaHfaVdWeiaLoU73sMX2iHWJZGulZvmjdDAPNLLuwScggHPFP0i2tdH8UJdXrxu+ozXjWuvWl2GSdSHcxzqeB5aqduMqNnbpXZ6d4P0HSr2O7tLHbNCCsJkmkkEIIwRGrsQnHHygcUWvhDQbLVH1G305FuXLnl2ZFL/AHyqE7VLdyAM5OaAOV8A2MOh6wmm3FsqajNp3nC+tboy2+pxqygzsDyJMuuc54bgkdPRqxtG8KaJoE7z6ZYiGR0Ee4yO+xAc7F3E7Fz/AArge1bNABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHP+O/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLR47/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXP+O/+SeeJf+wVdf8Aopq6Cuf8d/8AJPPEv/YKuv8A0U1AB4E/5J54a/7BVr/6KWugrn/An/JPPDX/AGCrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/jv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0eO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtHjv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHP+O/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLR47/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXP+O/+SeeJf+wVdf8Aopq6Cuf8d/8AJPPEv/YKuv8A0U1AB4E/5J54a/7BVr/6KWugrn/An/JPPDX/AGCrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/jv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0eO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtHjv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHP+O/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLR47/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXP+O/+SeeJf+wVdf8Aopq6Cuf8d/8AJPPEv/YKuv8A0U1AB4E/5J54a/7BVr/6KWugrn/An/JPPDX/AGCrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/jv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0eO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtHjv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHP+O/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLR47/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXP+O/+SeeJf+wVdf8Aopq6Cuf8d/8AJPPEv/YKuv8A0U1AB4E/5J54a/7BVr/6KWugrn/An/JPPDX/AGCrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/jv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0eO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtHjv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFYviLXv7CghkFv5zSsQAW24x+B9a5lviFdn7ljAv1cn/Coc4rRnLVxlGlLlm9T0CivNW8eauc4S0X6Rn+rVWk8Za454u1jHcLEv8AUVPtYmDzSgu56nRXkj+Jtak66jMP93A/kKrNq+pP9/Ubo59Zm/xpe2Rm81p9Is9kqJ7mCP788a/7zgV4vJPNMf3ssj/7zE/zqPvml7byM3m3aH4/8A9jfWtLjJD6jagjqPNXP86ryeJ9Fi+9qER/3ct/IV5L3HH6U6MAypkA5Ipe2ZH9q1HtFHp7eM9CHS7Y/SJv8Khfx1oyAkG4f2WP/E1wKtEsZUJCWMWM/e57n/P4d8xvar5jbA33wACP0x1/nS9rIJZhXtpb+vmd03xB0wfdtbs/VVH9ahf4h2o+5YTN9XAriBApwCQDyCMjjH+TSGDHQM3CnsOv+R/Oj2kjJ4/E9/wOwb4iv/DpYH1n/wDsahb4h3Z+5YQj6uTXJm3YfedRyQQCO3ucehpPJbHUE5HTvn+vtT55dzN43FfzfkdK3j7V2zths1Hb5G/+KqJvHGtN0eBfpF/jXPm3lGcjAAJ59M45x0/GmlGBw2AckEdxS5pmbxWI6yZ0MfjfW0+9JC/+9EP6YqZPHurqRuitGHujf/FVzHly7tvlnPoSPTP8qadygEr1AIGRk56cUXmCxeIX2mdivxBvR9+ytz9CRUy/ESQfe0xT9Jsf+y1xAbIJ2tgYycdM9KdtYLuKkDGckdqOeaKWPxP835Hdp8Q4T9/TpB9JQf6VOvxA08/etLofTaf6154uXbaoJJ7ClKlTggg+9L2strlrMcR3/A9JTx3pDfeW5T6xj+hqZfGuiN1nkX6xNXl9FP2si1mlddj1ZPF2hOcC/AP+1G4/mKnTxHo79NRt/wAWx/OvIqKftmWs2q9Yo9kTWdMk+5qNocf9Nl/xqdbu2cZS4hb6ODXilFP2z7FrNpdY/ie4KysMqwP0NLXhwJByDgip0vLqMYS5mUegcij23kWs3XWH4/8AAPaqK8bXV9TQgrqF2Mekzf41PH4j1mMYXUbg8Y+Zt386ftl2LWbU+sWeu0V5QvizXE6X7fiin+YqZPGmtpjNyj/70S/0FP20SlmtHs/6+Z6jRXmq+PNXXqlq31jP+NTR/EDUR/rLW1b/AHQw/qaftYlrM8O+r+49Eorgl+Ic4+9p0Z+kpH9KnT4iIfv6aw5/hmzx+VP2kS1mGHf2vwZ21FcevxCsjjdZXA9cEGpU8f6U33oLtf8AgCn/ANmo549y1jcO/tI6uiucTxxordZJl+sZ/pUyeMdCc4+27T7xP/hT5o9y1iaL+2vvN2islPE2iyAEajCM/wB4kfzqZde0humpWn4yqKd0WqtN7SX3mhRVZdQsnOEvLdj7Sqf61MssbfdkU/Q0y009h9FFFAwooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAoorF8Ra9/YUEMgt/OaViAC23GPwPrSbSV2ROcacXKWyNqivP2+IV2fuWMC/Vyf8Kqt481c5wlov0jP9WqPaxON5lh1s/wAD0qivLJPGWuOeLtYx3CxL/UVXfxNrUnXUZh/u4H8hS9tEzea0eif4f5nrdFeNtq+pP9/Ubo59Zm/xqtJPNMf3ssj/AO8xP86n2y7GbzaPSP4ntD3MEf3541/3nAqs+taXGSH1G1BHUeauf51453zS9xx+lJ1n2M3m0ukfxPWpPE+ixfe1CI/7uW/kKrN4z0IdLtj9Im/wrzCMAypkA5Iq2rRLGVCQljFjP3ue5/z+HfKdaVtBLMqstkl9/wDmd8/jrRkBINw/ssf+Jqu3xB0wfdtbs/VVH9a4V7VfMbYG++AAR+mOv86YIFOASAeQRkcY/wAmn7WRDzDE+R27/EO1H3LCZvq4FV2+Ir/w6WB9Z/8A7GuPMGOgZuFPYdf8j+dNNuw+86jkggEdvc49DRzyfUzeOxT6/gjrG+Id2fuWEI+rk1Xbx9q7Z2w2ajt8jf8AxVc15LY6gnI6d8/19qDbyjORgAE8+mcc46fjS5pdzN4zEv7TOgbxxrTdHgX6Rf40kfjfW0+9JC/+9EP6YrnijA4bAOSCO4o8uXdt8s59CR6Z/lReZH1rEfzM6dPHurqRuitGHujf/FVYX4g3o+/ZW5+hIrjjuUAleoBAyMnPTigNkE7WwMZOOmelHNMpY3EL7TO3X4iSD72mKfpNj/2Wpk+IcJ+/p0g+koP9K4TawXcVIGM5I7Ui5dtqgknsKXtJItZhiV9r8Eehr8QNPP3rS6H02n+tTp470hvvLcp9Yx/Q15sVKnBBB96Smqsi1meIW9vuPUF8a6I3WeRfrE1TJ4u0JzgX4B/2o3H8xXlNFP2silmtbsvx/wAz11PEejv01G3/ABbH86nTWdMk+5qNocf9Nl/xrxuij2z7FrNp9Yo9rW7tnGUuIW+jg1KrKwyrA/Q14fQCQcg4Ip+28i1m76w/H/gHuNFeKpeXUYwlzMo9A5FTLq+poQV1C7GPSZv8aftl2LWbR6xPZKK8ij8R6zGMLqNweMfM27+dTL4s1xOl+34op/mKftkWs1pdU/w/zPV6K8uTxpraYzco/wDvRL/QVOvjzV16pat9Yz/jR7WJazSg+56VRXncfxA1Ef6y1tW/3Qw/qanX4hzj72nRn6Skf0p+1iWsxw76/gzvaK4lPiIh+/prDn+GbPH5VMvxCsjjdZXA9cEGn7SPctY7Dv7X5nYUVyieP9Kb70F2v/AFP/s1WE8caK3WSZfrGf6U+ePctYug/to6OisJPGOhOcfbdp94n/wqdPE2iyAEajCM/wB4kfzp8y7lqvSe0l95rUVnrr2kN01K0/GVRU66hZOcJeW7H2lU/wBaLopTi9mWaKYssbfdkU/Q0+mWFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAcb8QlzY2R9JG/lXA4r0H4gf8gy0PH+uI6e3/1q8+3GuWrfmPnMy/3h/IOPWim0VnY4BwxRkCk70AHjiiwC5FGRSUY9qLALkf8A6qVT8w4z7etNoA9Rkd6LDLvmcFfNAOxQBGufXjP4+vcU44JIKErvAAz2xnGMfnxTGJBZdyjhRtiHv/PP+RSlfmIwo+bq3Q8dSPTjPTqO/arHQKHC8ZRB833fvDnvg/5ApTyQoEh5TjOOp78jn/OaZu2qBvRDsYAAZ79//wBVPIJ+UKxBKgbj978P/wBfeiwIQhVPSJSNw+ce+Of6fQ0q4d9ys7H5QWUbffv/AIgU3KouAY1yJBgcnr0P+elSEFnz+9YbhwcADjPI/M9PSmNDDGNmSgA2kgSMe569/wCmKUyZYjzFzljlEyF/n/nvTfujJESMV+Zjzznt6n86kdtwb5pGGHJ+Xjj688d6ABRlwArk7x959uflz/genoKjj2KFwYkGE3FRuPOePr6j/CpW4cb1wwkH32J/h/n/AFoiySmxn42cRqRnGehPP+fei47aiRoX2lfMb5U+6MD7x78H8frSeUFX5oVHQgyN1+Y8n+v4Uqg4Uso2kRk+Y+BjJ7en8vxpY3ClQrKPuAbEyfvep/n7+2KASXUfA3lvvjkGSVG8IT/EeuD/APWPTtUqvI7DcGYbU52A92/vE/16e1RKWOMq5wE+YkJ/Ee45Hf8AyaahQuv3AwC5H+sI5Of6fpSLTtZEqSBUSNvLw4UMH2jPB7Bc9feqn2aIoT8xGM4VMY+TOck/549asxscQqplxhfuIAPut689PzFNJxGN5I4z+8kOR8nGM8f59qEktiZLmSuRSWyLkGJUxnrJ344x+P5ZNM+yxuww3G7B8oE5O7GB17fzHWrXyj7mzkHHlxEZ+70PT/PvTix2ksXxvPzO4T/lp7duvT39qYvZxben9fIofY2KIVWQ7gDnIH8JP9M/4U1rTa2GmROv8QP936epP4VcIQhf9UeM45kP3X/z7GpBvz8hkHBx5aBePkPGffH6AVRPsotGcLNy2AzMe4WM+/rj0/zik+yT/wCyB3LHGOFJ/wDQv0NaLY8zBYckDDzY6M/bof6Z96aqrkkLHx3ERz91O/bvx259aNOwexiZvkvkjchx6En17Y9v8ijyJy+0RseQPTk445+orXO/zG3ebj/a2r3kH+Pv1pnyMy/cbn++0vdOP/rd+lFkL2KsZbRSoMuu0YzywHv60myTJAjc464XI7/4H8q1Qn7ltin7hzsgx/CfX+X40rJgsXyPmbmSQL/E/dfr2/kaLRD2CMny5QSDFICP9k1H5i+orYEUZZfliJDAj5WkI5T8+vbrRFE3lIw8wYVeVUDHyx/3uPT68e9HLEToa6MyN49RRuHrWrIiqpBYBgCdrzEdjzhf/wBdMnhiy5CxYAY5EJY9JO/TsPrjPbFNQQnQa6mbkUuRV2ayAaQeU+Pm5CIuMeaOB1/hH5delRSWcKtIFSMbd3/Lc5/jxx+A5/3T3NV7JdyfZSK+aM1ZktYlWTOAV3coknbf65/ufz7imT2HlmTalxhNxyWXHHme/wDsD9fUZXsWJ0pkNFWX0tVV8XLFlJGHMYHBI6iQ/wB0/wD6uaLnT47NkDyQzBgMtFKGwcZI69s1E48u4nSmldoql1HVhTfNT1z9K0YbG1lXK5HbmrceloOgBriqY6jT0lc1hhak9jHju54wBE8ygHI2kirKalqoGI7m6Ue0zD+tbCaco/hqwlioHQVxzzqjHZHXTy+u+tjJi1bxAMBdQuAB/ekLfzq5Fq/iMY/4mUv4qp/mK0ls17Cp0sx6CuOpxBJaRR30ssrdZv72Ns/EmtxFPPaKdR13JtJ/EV09jrtteAK4MEmOVc8Z9jWBHaAsBipVtlHX8q54cRYiLva6PTo4KpHRyv6nXUVj2F80ISGU5j6Kx4wK2OtfVYHHUsZT56e/VdhzpuDswooortICiiigAooooA5/x3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCilo8d/wDJPPEv/YKuv/RTUeBP+SeeGv8AsFWv/opaAOgooooAKKKKACiiigAooooAKKKKACuN+IS5sbI+kjfyrsq5D4gf8gy0PH+uI6e3/wBaoqfCzlx3+7yPPsUcetG402uSzPlx1AxTaXvRYBcgUZFIAeOKKLALkUZH/wCqkx7UUWAcp+YcZ9vWrfmcFfNAOxQBGufXjP4+vcVSA9Rkd6uMSCy7lHCjbEPf+ef8imkaQHnBJBQld4AGe2M4xj8+KQOF4yiD5vu/eHPfB/yBSFfmIwo+bq3Q8dSPTjPTqO/ZN21QN6IdjAADPfv/APqp2NbjzyQoEh5TjOOp78jn/OaQhVPSJSNw+ce+Of6fQ0pBPyhWIJUDcfvfh/8Ar703KouAY1yJBgcnr0P+elMY5cO+5Wdj8oLKNvv3/wAQKQxjZkoANpIEjHuevf8ApinkFnz+9YbhwcADjPI/M9PSo/ujJESMV+Zjzznt6n86AY4yZYjzFzljlEyF/n/nvTlGXACuTvH3n25+XP8AgenoKHbcG+aRhhyfl44+vPHelbhxvXDCQffYn+H+f9aBkUexQuDEgwm4qNx5zx9fUf4VJGhfaV8xvlT7owPvHvwfx+tLFklNjPxs4jUjOM9Cef8APvSKDhSyjaRGT5j4GMnt6fy/Gi4JbCeUFX5oVHQgyN1+Y8n+v4VLA3lvvjkGSVG8IT/EeuD/APWPTtTI3ClQrKPuAbEyfvep/n7+2KepY4yrnAT5iQn8R7jkd/8AJoHGy2JVeR2G4Mw2pzsB7t/eJ/r09qakgVEjby8OFDB9ozwewXPX3qJChdfuBgFyP9YRyc/0/SnxscQqplxhfuIAPut689PzFKyfQpS2K32aIoT8xGM4VMY+TOck/wCePWnSWyLkGJUxnrJ344x+P5ZNSk4jG8kcZ/eSHI+TjGeP8+1O+Ufc2cg48uIjP3eh6f596COSNtir9ljdhhuN2D5QJyd2MDr2/mOtM+xsUQqsh3AHOQP4Sf6Z/wAKvljtJYvjefmdwn/LT27denv7VEQhC/6o8ZxzIfuv/n2NUhOnEptabWw0yJ1/iB/u/T1J/CkFm5bAZmPcLGff1x6f5xWiN+fkMg4OPLQLx8h4z74/QCkbHmYLDkgYebHRn7dD/TPvT0D2Ubmd9kn/ANkDuWOMcKT/AOhfoaZ5L5I3IcehJ9e2Pb/IrSVVySFj47iI5+6nft347c+tSnf5jbvNx/tbV7yD/H360adhexTZkeROX2iNjyB6cnHHP1FDRSoMuu0YzywHv61qfIzL9xuf77S904/+t36UBP3LbFP3DnZBj+E+v8vxoshewXRmVskyQI3OOuFyO/8Agfyo8uUEgxSAj/ZNazJgsXyPmbmSQL/E/dfr2/kaaIoyy/LESGBHytIRyn59e3WjliL2CtuY/mL6il3j1Fa8UTeUjDzBhV5VQMfLH/e49Prx702RFVSCwDAE7XmI7HnC/wD66OVC9g7bmVuHrRkVpTwxZchYsAMciEseknfp2H1xntio5rIBpB5T4+bkIi4x5o4HX+Efl16VSpp9ROi+hSyKM1Yks4VaQKkY27v+W5z/AB44/Ac/7p7mlktYlWTOAV3coknbf65/ufz7ij2XYn2Uitmipp7DyzJtS4wm45LLjjzPf/YH6+oy99LVVfFyxZSRhzGBwSOokP8AdP8A+rmh0rB7Kp2K1IXUdWFWrnT47NkDyQzBgMtFKGwcZI69s1LDY2sq5XI7c1zVKsKavK/3AqU3Ll6md5qeufpUkd3PGAInmUA5G0kVsR6Wg6AGrKaco/hrinmtCOx0wwNZ7GOmpaqBiO5ulHtMw/rVqLVvEAwF1C4AH96Qt/OtZLFQOgqwtmvYVx1M+jH4UdtPLa7+0zNi1fxGMf8AEyl/FVP8xWvZ+JNbiKee0U6jruTaT+IpyWY9BU0doCwGK458QVm/dPRpZdVjrzv7zfsddtrwBXBgkxyrnjPsa1K5FbZR1/KtawvmhCQynMfRWPGBXdl/ECqTVPEK1+v+Z3/VpqN9zYoo60V9OYBRRRQAUUUUAFc/47/5J54l/wCwVdf+imroK5/x3/yTzxL/ANgq6/8ARTUAHgT/AJJ54a/7BVr/AOilroK5/wACf8k88Nf9gq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQByXxAH/ABKLY4/5bgf+OmvO+9ej+PhnRLc5x/pI/wDQWrziuar8R87mX8d/IKKKO1ZHnhRRilx70AJRS4oxQMKKMUYoAuE5ZgCOqjCL7jn39OlC4L4VUJDliJGyOnf/AD+NIc5ZXO07hlcD+Q/w/KlTLMqBfMG5jhgQOPXk/wAqo6OomSsRAZF/dnKquSfY+n6dKeVJZsKT8wz5h/T/AD69KjyBEAZI1Hl8hFyfofTt+dSYJfJVvlccytyvHP8An6UAhpJCt+8ABR+AuSeRwfT17U/aC5yhwHG4yHHbv7d+noKYTiMhW3YjbG0cY/zn0zTyAZA4Cj95w8kmf4f1/wD1UDQ1X2ADcqnavCDJ69D7/h1pzAncdsjHDk5YDGD147j6dfpSKxMS4dh8qj5B056Z/l7/AEocYRuAAVfl3yTg4wPp/jQHQcNvmNsMS4fpGpORs7EfjQg3qDhio2j5mCoeO/8An+tKSWY/MzAv91EAB+T/AD3pAqh0LhFkyuDI2TjHp3GfSmMI8Bht2K+EJ2gs3U/r0zxTkzkAGQ/KgAHyj7x9Of8AJpsRLqigsflTO1Qvr1J6/mKFIBXJVchMeY2T1P8ACP5cdvWgF0FUKGG7ywRsOGy7Z3dcfl39KdHligXzCpCcqoUDlvXn/JpE3MVCs2cJxGmAeT3P+T+FA2+YPMAyQgxK+cjJ7f5/Wga6CYUFA+zJ2ZDv2weo5/IdPwoBxEdpb7vOyLHHl/7Xb+Y+tLC3yRbcnhflSPHO0569ff8AH2oYYQbwRx1aTac+X6evv+HagOg6XIL72PIIPmy4z93qB39u/XtQMB2IVQfMPKJk/wCs/vZ/X6UfIG+QDo2DHHjsvfp/kUrbhgtvA83J819vHme3H+TTH1bDDFUGZdvHcKPusOMcj8fpTSUYnd5RBz6y54Xr+vP19RQoQbCmwZxkrGWycN/F6/8A6+1SZbksX6Hliq9k7j/P5UDtdCqH3kKJMZ/hRQPvP1B7df19qjQ8lSwzgcGfk/IvYcH/AD6UJsMg4iPzcYBk/ibvnjtz3496VQwiPEgXbyMKqj5E69x+Ht70B0QIo3nCqMnGFhPq/c9fr7Y7UrFgw3s69h5jqneM9R/kfUUAgyuCUyCSR5jMRy3Yf56e9CBlZSgZRj/lnGq5/wBX/e6/4/jTD7Og2Qr5LAbSQhGcNJj5W4Gen9PwqYKyOSisi7jysaoPvN6/yHv7VFK2Y2VmIJU8NNg8K46DtTwgaVmVUPz9fJJ/iP8AEev0Pt6UDW4m8M6Zk3AMp5nJP3o+do/z+IojjV1QhATtH3oST0XPzHjH6cn1oDsjJukKdD88ip/zy7r9Pr0PUUABogQFYEY/jlHQf5/yKA3Y52ZFbJZcA8EomeGx057d/wA6jlfcsh35BD5IkaQciX0HT/8AXjrUkqtHG+EdV2twkaKMYfjn6f5NNlbfJIfMDEb+fPORzL6DHf8AyCadwd9gkiyZSIlJJbO2PaTzKO5/zg+gp0uPnDOM/NxI6rn/AFmeg/2j+fvStGHLYUNy2C0JyM7+5P8AkClZtpI3Fck/K8ip/ez0BPc/n7g07jSsRSxAq4GejYPzydN+f8+v0NQz2ojWUCHZkPykSr2l9TnvgfT/AHcWz8yk5LZ5BJkkB69+P8/iKhnhCIxEKrkNykSLz8/971zj8/QVcJO9h2v0Jg4KyYmU8tz5qj+KQ5G1foeOueOtapka5u5pLmZ0ZYGwYWY8hVAHygfnWTFOD5gWcSDDHiZfWU/wjk9D+Xrxty7JLido1Kr5bDHmSMPup3/z29q5a9V02bUbyW5SayiuLRlSMRyok0rytKw37SMfeU5x+f0qP97YMyXG9irbXJBALHB43AHofT8xWs1k5tmYFcGKY4LHONwzxng/5waS5EkySJIC3zuxJjfcfkxjPJ/A/wAq5KvJWjaaNfZSi+ZaMS3KyRKxGOR19OanEQAPzAAfjWOsL2+0x5I2liCCMDJ6Z+nvVyO8VCxE2WUjaVHB/wAK+VxOCnCd4ar+vQ9TD4uDjaejL4Kgg5J7Edse1PVwqjHv3qs0yOilQ2M4BJ/z607eGGQFXB6VwTvF76/1/X5HfCaexbVwWAyBx2GaM5655HHOKreaxJO7k9aXeWPcms3O+hspllsZ5Kgjg85rV0278wGFmGR93jHHpWGkrKDtKjHPPWnwXAiuUkLnAbccV6OX436tXjUjs9/T7+noZ1LTjZnVUUikMoYdCM0tfopwBRRRQAUUUUAc/wCO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtHjv/AJJ54l/7BV1/6KajwJ/yTzw1/wBgq1/9FLQB0FFFFABRRRQAUUUUAFFFFABRRRQAVyXxAH/Eotjj/luB/wCOmutrlfHwzoluc4/0kf8AoLVE/hZy43+BI8470UUVyHy4UUdqMUAFFLj3oxQAlLRijFAwq4TlmAI6qMIvuOff06VTxVwlgxVjsO4ZXA46dh+Hb8qaNIAuC+FVCQ5YiRsjp3/z+NJkrEQGRf3Zyqrkn2Pp+nSlTLMqBfMG5jhgQOPXk/ypuQIgDJGo8vkIuT9D6dvzplkhUlmwpPzDPmH9P8+vSmkkK37wAFH4C5J5HB9PXtTsEvkq3yuOZW5Xjn/P0ppOIyFbdiNsbRxj/OfTNA2P2gucocBxuMhx27+3fp6Cmq+wAblU7V4QZPXoff8ADrTiAZA4Cj95w8kmf4f1/wD1UisTEuHYfKo+QdOemf5e/wBKBisCdx2yMcOTlgMYPXjuPp1+lKNvmNsMS4fpGpORs7EfjTXGEbgAFX5d8k4OMD6f408ksx+ZmBf7qIAD8n+e9AxEG9QcMVG0fMwVDx3/AM/1ojwGG3Yr4QnaCzdT+vTPFAVQ6FwiyZXBkbJxj07jPpRES6ooLH5UztUL69Sev5imCHJnIAMh+VAAPlH3j6c/5NIoUMN3lgjYcNl2zu64/Lv6UikArkquQmPMbJ6n+Efy47etOTcxUKzZwnEaYB5Pc/5P4UDXQWPLFAvmFSE5VQoHLevP+TTcKCgfZk7Mh37YPUc/kOn4Uo2+YPMAyQgxK+cjJ7f5/WiFvki25PC/KkeOdpz16+/4+1A10EBxEdpb7vOyLHHl/wC12/mPrT5cgvvY8gg+bLjP3eoHf279e1NYYQbwRx1aTac+X6evv+HanfIG+QDo2DHHjsvfp/kUB0sAwHYhVB8w8omT/rP72f1+lLhiqDMu3juFH3WHGOR+P0obcMFt4Hm5Pmvt48z24/yaaoQbCmwZxkrGWycN/F6//r7Ux9gJRid3lEHPrLnhev68/X1FSqH3kKJMZ/hRQPvP1B7df19qTLcli/Q8sVXsncf5/KmJsMg4iPzcYBk/ibvnjtz3496AWjBDyVLDOBwZ+T8i9hwf8+lKijecKoycYWE+r9z1+vtjtQoYRHiQLt5GFVR8ide4/D296UEGVwSmQSSPMZiOW7D/AD096AW+oMWDDezr2HmOqd4z1H+R9RTZCvksBtJCEZw0mPlbgZ6f0/CnIGVlKBlGP+Wcarn/AFf97r/j+NJK2Y2VmIJU8NNg8K46DtTB9SUKyOSisi7jysaoPvN6/wAh7+1M3hnTMm4BlPM5J+9HztH+fxFKEDSsyqh+fr5JP8R/iPX6H29KQOyMm6Qp0PzyKn/PLuv0+vQ9RQN7BHGrqhCAnaPvQknoufmPGP05PrTnZkVsllwDwSiZ4bHTnt3/ADpoAaIEBWBGP45R0H+f8inSq0cb4R1Xa3CRooxh+Ofp/k0DWiI5X3LId+QQ+SJGkHIl9B0//XjrTpIsmUiJSSWztj2k8yjuf84PoKJW3ySHzAxG/nzzkcy+gx3/AMgmntGHLYUNy2C0JyM7+5P+QKdyUriS4+cM4z83Ejquf9ZnoP8AaP5+9MliBVwM9Gwfnk6b8/59foalZtpI3Fck/K8ip/ez0BPc/n7g0H5lJyWzyCTJID178f5/EU1JosqT2ojWUCHZkPykSr2l9TnvgfT/AHcWw4KyYmU8tz5qj+KQ5G1foeOueOtQzwhEYiFVyG5SJF5+f+965x+foKlinB8wLOJBhjxMvrKf4Ryeh/L14qpJ8lxLR9jWMjXN3NJczOjLA2DCzHkKoA+UD86rtZRXFoypGI5USaV5WlYb9pGPvKc4/P6Vdl2SXE7RqVXy2GPMkYfdTv8A57e1StZObZmBXBimOCxzjcM8Z4P+cGuCWIbbi9TsdJyv1Mn97YMyXG9irbXJBALHB43AHofT8xWpblZIlYjHI6+nNLciSZJEkBb53YkxvuPyYxnk/gf5VmrC9vtMeSNpYggjAyemfp714+YYSFRc1Pc3oVJUZWesTYEQAPzAAfjTwVBByT2I7Y9qoR3ioWImyykbSo4P+FWWmR0UqGxnAJP+fWvnp0alNNSX6nrwr056xLKuFUY9+9SK4LAZA47DNVN4YZAVcHpTvNYkndyetYc/L1/r8DpjMs5z1zyOOcUrYzyVBHB5zVbeWPcmnJKyg7Soxzz1pRkno/6/r1K5zc0278wGFmGR93jHHpWjXKwXAiuUkLnAbccV1KkMoYdCM19zkOMliKDhN6x0+XQ4q0UpXQtFFFe4YhRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQBzHjtc6AnGcTqfpw1ebV6Z44GfDje0qf1rzOuWr8R89mf8f5BRRRWZ5wUetFFIAo/KiigYUtJRQBb4QnBjjXfxxuIx39/wCfNOVd5UbJJOWOGOB16/5Hc0inBBxsy/Xb7EY/z+VAXdjKtISGOGOM9Of19/8ACzdf1/X/AABAxC7RIqnywP3a5/yfy60/b8wbYQPMyHlbJ+73H6/54bnGV3qvyqMRjPGf8/iRTtuH37Avzn95M3Xj/Pega/r+v+GE3ZTqzEJ91BwP8/0p4+ScE4BL9+f4f/r+ppikPGAGdhsHyJkD6fT/AOvTshHIbZF85BA+Zjx7c/5/MGujBQWReJHO2MdAAOenb8DSEgBgSikrICOp69/89qFXKj5crtTBc4B57/5P60AgI2w7BtcnYvPUcfoPSgCRt3mk4kkbcSM4XoPbn+lIhXegVo1bIOY0z29v6+9IVA3ZQBSxy0rDrj8s05SC4AY7dwB2KAPu+/0/L60FdRE+aNA6kxgICZWwvf07URYAG3oAn3FyRyeSenX/ADxQuEMfmKikBOGO5jwfx/D/AApYwSUPzOcIFGdg79D16d/SgS6ABnG/lMJu8x8Act2H/wBb9aIcjbsI5CAeWmT374IzRE3zptID4TOAXbqc/wBMjHpTkBIQlXKhUx821TwfzH4Z4oGujGrkLFuHG1MmSXbj5T2Hb0P0FNB2xZUquRjMcX/TPuTn8e/U96dCFQREbQTtPyKT/Ce4/UH39Kcdwj3MGX5cZZwgP7v27+n5etMOgSFgXL7gTuH76TAPC/gf68+lChVYCMKCHHEceT98dGPH+QKAEDHZsH3seXGT2XoTx/ketOIKsAylSXH+tbGR5noOv/6z3oH3E3EbWcyY2jl5duBhuw7f/WHrTfl3Fh5ZbBwVjJ7L/EeP8/7VKhIwVPYHEac9H7nr9fr3FK2QH3E9CMySY/hXqo4PegOg4MwO5yy88+Y4X+NupH+fzqNApQEBCQvBWMn+EdGPT6H39aliB8wmMHJc8xxDP3z/ABHj/PtUeQI8kjmPI3PjI2D+EcEfr27UxvZEmW5ZiwTcRlnCL99+4+bv/P1FMAVzjCN042mTPCf5/TvT0UiUlEO7eeY4ufvt/E3B/wA+lNDDHzsAcAkO+3+FOw/z+VAPZXFlDbcfvAuDgcAfdk9eSfb/AOtTg29ywZWIbGd7Tfxenfr/AJzUexSSFVcnOdsRBI2ydzwfr/hUxciQB2IO4AeZKFP3h0A6f59qCl3GRh12kIwxt+6iqB/qj/F/np2zQWUof3m4hcH96SemMFR/n8qZEittZVBwF6QkkcR9zgH8f60+RmjQ7tw+UjEkip2PAxTEhrouJNiLyHHyxFe0ndj14+v5U+ViGYF+ctjdKEOcyf3Rz1/X0amSqrF8hWADclXfp5v8R/P8M9jUrFoy+4vGuSfmKIOr8cZ9f85FARDYHyAu/J5zG0gPJ6Fj/njjrS5Mecl4l9SUQD8s/wCfwpv+syOHY9M75eD6n8acB5RHWIcEnCID931ycUF+YZ3nPBZvRpJAfywKjkj2qT5WCf4kjVcfXd/h+eBT1cSY3fMOCQJHk/u9himPGQoYx4LY+dYVXHTqXNVHcNxizBiwWdZODnMy+kv90e/059Ccb7ztNc3cj43FWzt34+6o6AD+X17VhJIHIUyg9mVpRno/YL7/AOQa3UhRri98mVnRFJzI7gkYXsT/AI/yrDGRuma0b30ZoJHE0Od5zsf5TE/HzdM5/XtUht02yF4yRtfGFK84qO1iMi5AGAOQVIzlvc1oOoCyKclQHwgB+X8eleLVTUXJJLT+up7NGCktTJkiVfNyucW57svf9evT/wCvWDP5iE+WyqCi7xu+9wDnn3/nXSXUDc9OIN3fpke/6Vj3SgSzFQVHlqMGQ+gHcfX/APVTpT920jixdOz0KsFweATWqZo2VSiFfcnOaxSpiaOVl3ByTg9CAfWrkExCcHtivMxuGS1XUeErte6y+HLt8q9TwBQHywGQM9zVTzjgDccA5HNJ5tea8Pqd31guh1wCX/ACmGQbe+c81TMtNM1UsM30JeKO80qUS6ZAwGMLt/LirlZPhxt2jof9pu3vWtX6FhG3h4X7L8hp3VwoooroGFFFFAHP+O/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0eO/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAVzHjtc6AnGcTqfpw1dPXN+OBnw43tKn9amfws58X/An6HmdFFFcZ8qFFFFIA9aKKKAD8qKKKBi1a4QnBjjXfxxuIx39/581Uq4pwQcbMv12+xGP8/lTRcBVXeVGySTljhjgdev+R3NNDELtEiqfLA/drn/ACfy60oXdjKtISGOGOM9Of19/wDAzjK71X5VGIxnjP8An8SKo0HbfmDbCB5mQ8rZP3e4/X/PCbsp1ZiE+6g4H+f6Uu3D79gX5z+8mbrx/nvSKQ8YAZ2GwfImQPp9P/r0DHj5JwTgEv35/h/+v6mkUFkXiRztjHQADnp2/A0ZCOQ2yL5yCB8zHj25/wA/mirlR8uV2pgucA89/wDJ/Wgf/BAkAMCUUlZAR1PXv/ntUjbvNJxJI24kZwvQe3P9KjBARth2Da5Oxeeo4/QelOKgbsoApY5aVh1x+WaBioV3oFaNWyDmNM9vb+vvQnzRoHUmMBATK2F7+nalUguAGO3cAdigD7vv9Py+tIuEMfmKikBOGO5jwfx/D/CgaCLAA29AE+4uSOTyT06/54pQM438phN3mPgDluw/+t+tEYJKH5nOECjOwd+h69O/pRE3zptID4TOAXbqc/0yMelALoEORt2EchAPLTJ798EZoXIWLcONqZMku3HynsO3ofoKcgJCEq5UKmPm2qeD+Y/DPFNhCoIiNoJ2n5FJ/hPcfqD7+lALoNB2xZUquRjMcX/TPuTn8e/U96kkLAuX3AncP30mAeF/A/159KDuEe5gy/LjLOEB/d+3f0/L1oAQMdmwfex5cZPZehPH+R60w6WBQqsBGFBDjiOPJ++OjHj/ACBS7iNrOZMbRy8u3Aw3Ydv/AKw9aUgqwDKVJcf61sZHmeg6/wD6z3pqEjBU9gcRpz0fuev1+vcUFdUJ8u4sPLLYOCsZPZf4jx/n/aqQMwO5yy88+Y4X+NupH+fzprZAfcT0IzJJj+Feqjg96fED5hMYOS55jiGfvn+I8f59qYLciQKUBAQkLwVjJ/hHRj0+h9/WpctyzFgm4jLOEX779x83f+fqKjyBHkkcx5G58ZGwfwjgj9e3apEUiUlEO7eeY4ufvt/E3B/z6UBEYArnGEbpxtMmeE/z+nenShtuP3gXBwOAPuyevJPt/wDWpAwx87AHAJDvt/hTsP8AP5UmxSSFVcnOdsRBI2ydzwfr/hQBIG3uWDKxDYzvab+L079f85pIw67SEYY2/dRVA/1R/i/z07Zp5ciQB2IO4AeZKFP3h0A6f59qhiRW2sqg4C9ISSOI+5wD+P8AWmUx5ZSh/ebiFwf3pJ6YwVH+fyprouJNiLyHHyxFe0ndj14+v5U6RmjQ7tw+UjEkip2PAxTZVVi+QrABuSrv083+I/n+GexoFIfKxDMC/OWxulCHOZP7o56/r6NTtgfIC78nnMbSA8noWP8AnjjrQxaMvuLxrkn5iiDq/HGfX/ORSf6zI4dj0zvl4PqfxoKQ7JjzkvEvqSiAfln/AD+FGd5zwWb0aSQH8sCgDyiOsQ4JOEQH7vrk4pFcSY3fMOCQJHk/u9higfkMkj2qT5WCf4kjVcfXd/h+eBTFmDFgs6ycHOZl9Jf7o9/pz6E4e8ZChjHgtj51hVcdOpc0iSByFMoPZlaUZ6P2C+/+Qa0+wDXyN152mubuR8birZ278fdUdAB/L69q0EjiaHO852P8pifj5umc/r2rPSFGuL3yZWdEUnMjuCRhexP+P8q0LWIyLkAYA5BUjOW9zXjYiL53ZXuelQu3qSG3TbIXjJG18YUrziqskSr5uVzi3Pdl7/r16f8A161nUBZFOSoD4QA/L+PSs+6gbnpxBu79Mj3/AEricpRq27f1/X6HbUpLkukc3P5iE+WyqCi7xu+9wDnn3/nToLg8AmrV0oEsxUFR5ajBkPoB3H1//VVAqYmjlZdwck4PQgH1qq9KFWPoePeVOd0zaM0bKpRCvuTnNKHLt8q9TwBVCCYhOD2xUnnHAG44ByOa+fqULydz14YjRMth8sBkDPc0odcAl/wAql5tNMtRHD+Q3iS4ZBt75zzXXaVKJdMgYDGF2/lxXBmauy8ONu0dD/tN2969/IKbp4iXmv8AIlVud2NaiiivrSwooooAK5/x3/yTzxL/ANgq6/8ARTV0Fc/47/5J54l/7BV1/wCimoAPAn/JPPDX/YKtf/RS10Fc/wCBP+SeeGv+wVa/+ilroKACiiigAooooAKKKKACiiigAooooA57xqM+GpvZ0/nXmBr1HxmM+GLknPDIeP8AeFeWmuat8R8/mn8Zen+YUuaSisjzRaKSigBaKM0ZpAL+dFJRQMubcMGHynecuzZ6ZxxTchlO4bjsb+L+vfH40sRBKEIufmIOCR+H+fyppb91w7E+XjCrwPqfz/WrN+hIxxld6jG0bYxnvz+v+fVdu1t/lqPnI3ytnGP6j+lIT8zL0OV+VRz7f4dD/SlHysGAXqw3yN/9f8voaBjQdyqm53OxcKBtHXGPfrj8SaeuFfClI+Twg3N0+n17UwHcpG53wEHHAHPTp059O9P9Vz8p3fIoyB79xjp0Hp+ICEA+UnbvI25aZuAfTH/16Ukn5dzOcNhVG0A59T/T/wDWuAHBYgElR8xyf556+/b2o+9FjDbcHOTjv7e319aB2B9oZtwjU/MMHk9O/enctICdzEuMBmCfw98f5xx600kAOEKgfN91Rkcd/b14pSMOMYI38GRs87PT6mgYQsyqnPTGEiGfXv0z/n3pUGNhxglUwZCSOvoOgoQMXjA81iduCF2j7p6dD+I7U2HaPL2MiMQgzGu49/yNA10HxZbanzkbUHyDaBye5/yeaRAAyZVOi8sCzd+3XPt0oXopIBUqm5pD8vft0pYeqKpYD5MhFHv3/wA9/pQC6BFvKxt+8PC4JwoHyn8vr3/Gmrt2kKYx8vOBu/5Z+o/H9aE25iLiPIVcZbccbT/nA6fhTsyNGctK3ryEwdntzn/630oBbIdIW+csHAO4fvG2jovXH+Tk+lCgKwVBtG/pGvGfMHc8Z/8ArA0g272KGP8AiHyAsei/xf5/WlY7fmPGW/5asBwJB6dRnr+Jpj7sTO0oG4JA4kkwT97oo49P096AcKwUsOw8uMLj5V/vf57iljDB1SMHAxgLHjP3+7dfr9aacYKOVzj7sjlm+4vYdaA6EgCtKQ4VjvOQ7Fj989V/wpqA/ZsqD/q+Sqhf4B3NOQHePlf7+DhQg++397/PY9qiXYYOsZIjHXdLj5PTt/T8KB9iU7GlIPlsd3Ri0h++f4Rx/jwe5ojDeUNm8Lt6ooAHyp68+nP096dllJ3F1G853OI1++fTn/DmokwYwxCsQo52Fj91f4u30+vrQPsOchmcblYehZpP+enbp6fofWpVDK67d6YYfdAQfeX15FMfIcqxcDOBuYJ3k9Pr+voaFUF1aNVzuXDRxkn7y9Gbg/jTBaDImVokBIb5RgBmk/hTt/nHI9Ke2UiYruVdrfcCqvR+PWhSfLVWY5KjAaQJ/CvZfTA/L2pJRlXwgPyuPliIx8r9269KA2QrMGaQ7lbLNgiRpM8ydh9c8fX1qQIYySqFRknCIqDqeueR/wDWx6U1yfMYFyTlvvS4P3n7KOev059zhVQ5Loh65ysGG6ju3+fyoKiAkDsPn3gnkh2k7jsOKI0KbCse3lfmWNVwfk/vc/8A66BJlkG8FsrgGY5wSnZf6U2EZMbKqgnbykQB6xDq3147d+9ANjo5AxT58gbcbZWY/wDLPso9/wA+1RmMqEbZtJCjesSqR9z++ff9enSnxPuMSiQk/Jx5h6fuv4V/l7HuKiWMERMIlYkJ92HJx+64+Y+4/PPerhuK9yS3lyyIZPMY4BQzKDkjptUe/r3/AC2iNt1dL5bx442FpXI6dSefz/wrEtZT+6QTqWIX786rjIHXaOnNbYDC4uAz7yD95ZJJAfm7HvWOL2Oik7mpGrFIAyR4KjG1Dkgt354/GrDEpI4UFRhxjaTxkdwapQIQqFkO3CZzGw6t2NWwn7uQxhNpEmQ6HKjd6k8mvCrLm0S1t3/r+uh7VF2RE4Qxksp/1a4wGGDkVnXAVWuRuIzCFAJJzyPWtCcbVIAx+7XoSeuOciq9yqF5ioIG0dXOeo7nrXHzOk3e2n39QrR50YVzHhIiGByCSPTk9aYjbVxV66jVVj6HKn/0I1myHBrRT9q7PyPJqQ9m7khlpDLVfdSbvetVQRi6rJzJSGSod1GapUkiHUZ6H4YwdChIxyzZwc9zWxWT4aBGgW2c9GPP1Na1fTUFalFeSPbpfw4+gUUUVqaBRRRQBz/jv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0eO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtAHQUUUUAFFFFABRRRQAUUUUAFFFFABXPeNRnw1N7On866GsDxmM+GLknPDIeP94VM/hZhif4M/RnlxpKDRXGfKC5opKKAFopKXNABS/nSZopDFq3twwYfKd5y7NnpnHFU6txEEoQi5+Yg4JH4f5/Kmi6YmQyncNx2N/F/Xvj8aexxld6jG0bYxnvz+v+fWMt+64difLxhV4H1P5/rUhPzMvQ5X5VHPt/h0P9Ko0Qu3a2/wAtR85G+Vs4x/Uf0poO5VTc7nYuFA2jrjHv1x+JNOHysGAXqw3yN/8AX/L6Gmg7lI3O+Ag44A56dOnPp3oGPXCvhSkfJ4Qbm6fT69qQD5Sdu8jblpm4B9Mf/XpfVc/Kd3yKMge/cY6dB6fiuAHBYgElR8xyf556+/b2oHuIST8u5nOGwqjaAc+p/p/+tX2hm3CNT8wweT0796PvRYw23Bzk47+3t9fWgkAOEKgfN91Rkcd/b14oGO5aQE7mJcYDME/h74/zjj1pIWZVTnpjCRDPr36Z/wA+9BGHGMEb+DI2ednp9TSoGLxgeaxO3BC7R909Oh/EdqBrcEGNhxglUwZCSOvoOgpYsttT5yNqD5BtA5Pc/wCTzTIdo8vYyIxCDMa7j3/I05eikgFSqbmkPy9+3SgF0BAAyZVOi8sCzd+3XPt0pYt5WNv3h4XBOFA+U/l9e/40Q9UVSwHyZCKPfv8A57/SmptzEXEeQq4y2442n/OB0/CgF0Bdu0hTGPl5wN3/ACz9R+P61JIW+csHAO4fvG2jovXH+Tk+lNzI0Zy0revITB2e3Of/AK30pRt3sUMf8Q+QFj0X+L/P60AthVAVgqDaN/SNeM+YO54z/wDWBpM7SgbgkDiSTBP3uijj0/T3pWO35jxlv+WrAcCQenUZ6/iaIwwdUjBwMYCx4z9/u3X6/WmPqhAcKwUsOw8uMLj5V/vf57ingK0pDhWO85DsWP3z1X/CozjBRyucfdkcs33F7DrUiA7x8r/fwcKEH32/vf57HtQNbjUB+zZUH/V8lVC/wDuacdjSkHy2O7oxaQ/fP8I4/wAeD3NRLsMHWMkRjrulx8np2/p+FT5ZSdxdRvOdziNfvn05/wAOaAiNjDeUNm8Lt6ooAHyp68+nP096RyGZxuVh6Fmk/wCenbp6fofWmpgxhiFYhRzsLH7q/wAXb6fX1qR8hyrFwM4G5gneT0+v6+hphuPUMrrt3phh90BB95fXkVDEytEgJDfKMAM0n8Kdv845HpT1UF1aNVzuXDRxkn7y9Gbg/jQpPlqrMclRgNIE/hXsvpgfl7UFPcGykTFdyrtb7gVV6Px60MwZpDuVss2CJGkzzJ2H1zx9fWklGVfCA/K4+WIjHyv3br0p7k+YwLknLfelwfvP2Uc9fpz7nALqOCGMkqhUZJwiKg6nrnkf/Wx6UgkDsPn3gnkh2k7jsOKFQ5Loh65ysGG6ju3+fyoEmWQbwWyuAZjnBKdl/pQXewRoU2FY9vK/Msarg/J/e5//AF0RyBinz5A242ysx/5Z9lHv+famwjJjZVUE7eUiAPWIdW+vHbv3pYn3GJRISfk48w9P3X8K/wAvY9xQTcYYyoRtm0kKN6xKpH3P759/16dKdby5ZEMnmMcAoZlByR02qPf17/lGsYIiYRKxIT7sOTj91x8x9x+ee9PtZT+6QTqWIX786rjIHXaOnNa/YHF/I2yNt1dL5bx442FpXI6dSefz/wAK1I1YpAGSPBUY2ockFu/PH41lgMLi4DPvIP3lkkkB+bse9X4EIVCyHbhM5jYdW7GvGxL1Z6eH3LrEpI4UFRhxjaTxkdwagcIYyWU/6tcYDDByKlCfu5DGE2kSZDocqN3qTyahnG1SAMfu16EnrjnIryq0GrzXb9P6/wAj1ObSxn3AVWuRuIzCFAJJzyPWsq5jwkRDA5BJHpyetbtyqF5ioIG0dXOeo7nrWbdRqqx9DlT/AOhGj6xytx7f5nm16N9SijbVxQZajkODUW6rjSUtTgdRx0LBlppkqDd70bq0VFEOqyYyV3vhjB0KEjHLNnBz3NeeZr0Xw0CNAts56Mefqa9LLYWqt+R1YKV6j9DWooor2j1AooooAK5/x3/yTzxL/wBgq6/9FNXQVz/jv/knniX/ALBV1/6KagA8Cf8AJPPDX/YKtf8A0UtdBXP+BP8Aknnhr/sFWv8A6KWugoAKKKKACiiigAooooAKKKKACiiigDC8Y/8AIrXZ9DHx/wADWvKu9es+K03+Gb1cD7qnn/eFeT4rnq/EeBmq/fJ+X6sSikorM8sdRmkzzRSsO4vWik9KKLALS02lFIC4AzFceYzYLZYgDOe3b/8AX+SZypGZGAQDanTvxnuKapUDJVXGzOFXHHue9D45VivKrwhwPbJ/L8xVG9x5wH2Y2ZIJVTnIwP6exz/JRjOSsYb58szZHXp/n1NITgtgomZPqeg/r9etHAXgIPlfhvc4/DuO1Ax2QQAXD/cAGNq9ehH/ANagtkMNzcb/AJUHTnrn9T+FBJY7d5YsV6jAb3/yDxSFwR1fgOflGMc/5Pagdx/3G6JEMqMn5mPHb+XekHzjhWc7eWkOAOev0/wpQNs2zaEJYHAG5jx/9f8AMn0pp5RWcg/KuCx46549P04+vAP+v6/4Ycz5H3zyHwI1wOOcgn8zz6UuCHZTtjYtghcs33enH+ev4tJ3LIRubKyEnGAf/r+v/wBalO3zCuQx3/dRc/wj9P8AA0AOQYCfu8JlQWlIA4Hcen/1qI23KgBY4CD5AFHQ9T1/H60gykiM4VTkff8Ambp2/lSx7mRCQdoVBlzheh6gdvr1/Wgpbjo8h49p2t8g3YJbv+X+fShDkJvUlAFyZG2rjn07UyFsqoTdtwuQmF9epx+v1+lOjyHjAIBO3JOWbv8Al/8AX9qAXQWFsrEEcgfLwiYGdp7nv/8AXpGxtJdUUgH7zGQ/c/l/Ln0oUkpHvU7QEBMj4H3TwQD09M+1Cnch2FiMHhI8D7nv/Tr+NAdB7hiGJ3MPmyz4TjC++f5/pSDGf3ZGN2fkTJP7wfxHgn/61NkCgyMypuwwzy7dF/z+eeuaexc5LknLn/WybR98Zzjr75+vpTH1YHbuCyBVzj5ZHzn738I/D9KT5xG4XzAu0/dUKv3Fz15FOjzv2x71AxxGmAfv+v8Anrio+PKckIGweS5lx8oxx2+v19KA6Egx5qnCH95jnMuPn9e3/wCr1prki0+Zn2+XgFnCr9ztjr9D1796kO4MWbIUOfmZgi/f5z39f19Kh4+zsVIJKHJVdx+4c5P4UDZOoCzZjHJfgxR5P3+zNwf/ANXvUakeUoJGTGPvSc42D+EdRx9fyqST/WEOOr4w5zn5x/AOtRoCtvldxURjO0BVPyDqTz+Ipje9h6j5m2LySeEQf3nH8X9OetI5VnBYqxLDOS0n8Uf8P9O9IuGOBsfkno0v8T9+3178e9OJZXVWZwCeN0gTvGeCOvB/yaA+yIu5YOAyrtHGFRc7R1zz260koV1b7jDnnJkHSTv/AJ9aVVJX5VBOzPCEnG3sx4P1+tDkh3yfX78uD1k7Lx/h9M0AyRt6OwO5RuPDlUXlm4457/T9KRFSR12qHORhijP3XoTgUKpXOxMAk8pGEXqeu7+n0pAySMMsjlioyWaUYynUDgdenrxmgroKrkeWHkZQxUbXcJ/zz/u/n6496SFS5iJj3kbD/qyT/wAse7HHt6c+h4SJzH5bDKINmCSqr/yyPXk/j+PrREoYRhQGxs6hpP8Ann+A7/qOhFBO4sR/1SmQn7nDSBSP9V0C89vrx6jmIRbjCVTfgpgCLLdYc4ZsD+nPoeJ0PliNXJUfKOZFT+50xz2HvxnORUWwv5ZCCUgoQPLLdDH3Ygf05PY8XDcGrjtPlb/R1jl3SfuyqGRQTkJjhR79Of51uBpHurlpQVfeNwG/j5z2AHp/+usLT2VDAjzOgBjzuKKQMRdhyeP0H1rdhVZHupUlMiK64LSMWYGRhwO9YYv+vuOig29DUhJ+zpHmRi0UOPkcAfN0weo+lP2DynLYBAOB5Tc/N9aih2mOJhM7MscW3Mb8cngHP/6+1TuweEliAwHaM5J3V4NWzV3bZ9fXzPbpbEbhVjmCt/CvJQj07/5zUFwo3TlCWUcEknPUdc1MxAiuAeCduBtPp6npUErDM3zDGcY3e/vXFUkno7dfzkaNaGbdH5IgVxhcfXk1kzH5q1boYCcY+X068msmb7xrTCb6nkYwizRSUV6Vjzhc0UlFAHpXh0Y0C04xlSfu47mtSs3QBjQrPBB/d9s/1rSr6Cl8C9D6Gl8C9AoooqywooooA5/x3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWjx3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCiloA6CiiigAooooAKKKKACiiigAooooAKwvGP/IrXZ9DHx/wNa3axvFab/DN6uB91Tz/vCplszHEK9GS8n+R5N3opcU2uRHyQtLTaXPNOwC5o60lHpSsO4tFJRRYB1WwGYrjzGbBbLEAZz27f/r/KmKsqVAyVVxszhVxx7nvQjSDHZypGZGAQDanTvxnuKU4D7MbMkEqpzkYH9PY5/kx8cqxXlV4Q4Htk/l+Yp5OC2CiZk+p6D+v160zQUYzkrGG+fLM2R16f59TS5BABcP8AcAGNq9ehH/1qbwF4CD5X4b3OPw7jtTiSx27yxYr1GA3v/kHigYFshhubjf8AKg6c9c/qfwp33G6JEMqMn5mPHb+XemFwR1fgOflGMc/5PangbZtm0ISwOANzHj/6/wCZPpQO4g+ccKznby0hwBz1+n+FKz5H3zyHwI1wOOcgn8zz6U08orOQflXBY8dc8en6cfXhSdyyEbmyshJxgH/6/r/9agB2CHZTtjYtghcs33enH+ev4qgwE/d4TKgtKQBwO49P/rU07fMK5DHf91Fz/CP0/wADSjKSIzhVOR9/5m6dv5UDQsbblQAscBB8gCjoep6/j9adHkPHtO1vkG7BLd/y/wA+lNj3MiEg7QqDLnC9D1A7fXr+tJC2VUJu24XITC+vU4/X6/Sga6D0OQm9SUAXJkbauOfTtRC2ViCOQPl4RMDO09z3/wDr0keQ8YBAJ25Jyzd/y/8Ar+1CklI96naAgJkfA+6eCAenpn2oBdAbG0l1RSAfvMZD9z+X8ufSnuGIYncw+bLPhOML75/n+lMU7kOwsRg8JHgfc9/6dfxokCgyMypuwwzy7dF/z+eeuaYdP6/4YcMZ/dkY3Z+RMk/vB/EeCf8A61Kdu4LIFXOPlkfOfvfwj8P0oYuclyTlz/rZNo++M5x198/X0pY879se9QMcRpgH7/r/AJ64oKW435xG4XzAu0/dUKv3Fz15FOGPNU4Q/vMc5lx8/r2//V61Hx5TkhA2DyXMuPlGOO31+vpUx3BizZChz8zMEX7/ADnv6/r6UAiNyRafMz7fLwCzhV+52x1+h69+9SqAs2YxyX4MUeT9/szcH/8AV71Bx9nYqQSUOSq7j9w5yfwqeT/WEOOr4w5zn5x/AOtALuRqR5SgkZMY+9JzjYP4R1HH1/KnqPmbYvJJ4RB/ecfxf0560xAVt8ruKiMZ2gKp+QdSefxFKuGOBsfkno0v8T9+3178e9MFuK5VnBYqxLDOS0n8Uf8AD/TvQu5YOAyrtHGFRc7R1zz260pLK6qzOATxukCd4zwR14P+TSKpK/KoJ2Z4Qk429mPB+v1oH9oSUK6t9xhzzkyDpJ3/AM+tStvR2B3KNx4cqi8s3HHPf6fpUbkh3yfX78uD1k7Lx/h9M09VK52JgEnlIwi9T13f0+lA4gipI67VDnIwxRn7r0JwKFcjyw8jKGKja7hP+ef938/XHvSBkkYZZHLFRks0oxlOoHA69PXjNJE5j8thlEGzBJVV/wCWR68n8fx9aAbFhUuYiY95Gw/6sk/8se7HHt6c+h4Ij/qlMhP3OGkCkf6roF57fXj1HKRKGEYUBsbOoaT/AJ5/gO/6joRT0PliNXJUfKOZFT+50xz2HvxnORQJIgEW4wlU34KYAiy3WHOGbA/pz6HiTT5W/wBHWOXdJ+7KoZFBOQmOFHv05/nTdhfyyEEpBQgeWW6GPuxA/pyex4XT2VDAjzOgBjzuKKQMRdhyeP0H1rb7A46O5uhpHurlpQVfeNwG/j5z2AHp/wDrrUhJ+zpHmRi0UOPkcAfN0weo+lZcKrI91KkpkRXXBaRizAyMOB3rSh2mOJhM7MscW3Mb8cngHP8A+vtXh4l+901T/U9bDdyXYPKctgEA4HlNz831prhVjmCt/CvJQj07/wCc1I7B4SWIDAdozkndUTECK4B4J24G0+nqeleXVaWitt38peZ6KIbhRunKEso4JJOeo65rMuj8kQK4wuPrya0pWGZvmGM4xu9/esy6GAnGPl9OvJrmk05XX9anPX2MqY/NUOalm+8ahr1qS91Hg1PiFozSUVoQLXpPh0Y0C04xlSfu47mvNa9M0AY0KzwQf3fbP9a7sB8b9DvwHxv0NKiiivVPVCiiigArn/Hf/JPPEv8A2Crr/wBFNXQVz/jv/knniX/sFXX/AKKagA8Cf8k88Nf9gq1/9FLXQVz/AIE/5J54a/7BVr/6KWugoAKKKKACiiigAooooAKKKKACiiigDJ8TDPhu+H/TP+oryPNezarave6Vc20ZAeSMqMnHNeaz+DtchJxaCQf3o5FP9c1hVi2zxszo1JzjKCb0MMjmkq5LpWo2+fNsblMcEtEwH54qiazseNKLjuh1FNoosTcdmjNNoosFxcmikzRmiwXH+Y/Hztx056U4TSAEBsA9h+dR5ozQHMyYXLhs4HXPv0x1pRcfKQwJ+UgH6j/PTFQZpM0yueXcuG4QliSzZIOCMA4/z3BpTOhDAyOflcDAA6/5/wA96WaM0D9qzRRkZiFwoMgAjjOScLjg/wBfc57UDACnCgsq/M/Jbn+f4+lZ+aUMV+6xGfQ0FKr5Gg+GViQz/LIcvwOvUD/H+lSZZ2ITlQTjYNq/d9sfXr6Vm+a/OWJyCOeevWnG4dyTJh8kn5qRSqovxsEdRGV+8P8AV4P8J7/n+v4CYRozJsBAXG/lsY9Of8/TFVBeNx8oABzhDtGcYHv+tPju449u1SoBGcAZ4B7/AOf5UFqpHuWYsuibwxQKgO8kLjB/T8P55pIDvCIu4qQgIQAevfH659e/FQx3EWULEfKFHzLuIwD0/wD1+npT450YICwIAQHe+BwDngdf8/iFRktNR8TAeSFZQcLnALMPlPf+nbPtSyAspEgwp7yOBzs9P6/4UkTBkiC78ALnYNoztOeeueuce/rSKyqcRMoP+x8x+56/0x607DT0Q8tuEmwPtAbhVCAcL68j/wDV6U4EM5ZTGSX4KKXP3xj5sfTGfbPpTJOkgcp/ER5rZPQdgMZ9/wDJe2SxyshAc5Jbyx98Zz3HfOenP4Fiu4gKlhu2jcFIErZz97sOo6Uo3eS2A2NucphAPkHrz/kepojb94AjsMlQdiY7v1z396Zx5TNhCwTj5jIfuD8v5dewFAdCVcGQsm0tv6xDe2d/Zjx0xjPt71G+fKQOcHZ0kbacbD0UcenP096kcnfltxy//LY7c/vBngdR/wDrpijEICBsBSPkj4+43c9f6/jQD7EuOoGQhbHyx+Uv3x68gVHkNEpO1iYz1BkP3Mden/16XgyAgKzCQfd3Stncvc9D/P8AGkclbdlOQBHxubb/AAHt3/H+dA2SE4fDlh854lk2/wATZ4Xr1/w602NTkBEI9fLjCZ/1f978PzB7mnoCHPlggb8Hyk2D7567uR+HSoUKMR/q2JxwAZj0Tt09f1HcUA9l/X4jn2vGy5Rjg/KS0mOD2/h/pUh3KX4KjcwG4hAPmkHGPmHXp68dCKZMzeUyuzj5DhWkC/wv2HJ/r+dP2jfIVVc7jkpFju/Usf8A6+PoKBrccu13UhN/I5CFj1H8TYBpokxJGHdiQV4MnIGY/wCFefX9e9AKuwDkSfMB8xMnOV6gcCkjyuwElR8o2s4jH/LI9uc9/wAOOlA5MLcMqxMoIOEyyoFxxEeS3+eh7mljKsAq7WOAMAtKP4f4V47Hp9KIkJ2soy+1SCI9zD7n8R4/pk+/DgwI2lsgYyDJk9v4VGD0/wAmgEhVzEqnJQYHdY1J+Xvyew/So2QOyfIrHcpwUaQdV6E4A6ent3FSj5BuAMY7uqqg/Uk/5z61Cw8wKF2vng/K8wPt2HoO3pVw3K8hdPcW9zbl5GhQOh+byxjHk8jrngZ5Pb1BrdjaOVrqUzOzGYMnLEMDK3JAOBx2/LmsO1HlTR/eiGRkYRAMeXznnHTP4ZxkHOzajzLeWQSR4Eg6XBLHLn0HPX+veufGPRmtDSVjWhTyni2hnO2M4aJzz/I/5xUsm5IDlVw6qeEOR836UsROEJzEBsBLbmccdu+PpxUTFDBICCWKLhsMO5zmvDxNoxautn+R7lLYRh/rWzgArgEkevb8Kgl3FJupAfnDZ6k+tTuQGkGOSy8l2Hr61XlbCTZAOXByTn1z9a8+dl177fPyNHsZt3/D/ujtWTL941q3fDYx0UdsdqyZfvGt8IveZ42M3IqKXFGK9G559hKWnBacEqHNIpRZ6XowxotmCc4iX+LPb1q9VbT4zFp1shBBWJQQQBjj2qzX0kNIo+ggrRSCiiiqKCiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigArJ8TDPhu+H/AEz/AKitaqeq2r3ulXNtGQHkjKjJxzSlsyKqvBpdjxnNBHNbk/g7XIScWgkH96ORT/XNZ0ulajb582xuUxwS0TAfniuVxaPlJUasfii18inRTTRRYxuOozTaKLBcdmkyaSjNFguLTvMfj5246c9KZmlzQFyQTSAEBsA9h+dOFy4bOB1z79MdahzRmmNTfcnFx8pDAn5SAfqP89MVIbhCWJLNkg4IwDj/AD3BqnmjNA1UZdM6EMDI5+VwMADr/n/PeRGRmIXCgyACOM5JwuOD/X3Oe1Z2aXNBSqu5oDACnCgsq/M/Jbn+f4+lK+GViQz/ACyHL8Dr1A/x/pWeGK/dYjPoad5r85YnII5569aRXtUaWWdiE5UE42Dav3fbH16+lJGwR1EZX7w/1eD/AAnv+f6/hQNw7kmTD5JPzVILxuPlAAOcIdozjA9/1oLVWN7ltMI0Zk2AgLjfy2MenP8An6YpYsuibwxQKgO8kLjB/T8P55qtHdxx7dqlQCM4AzwD3/z/ACp0dxFlCxHyhR8y7iMA9P8A9fp6UylOOmpNAd4RF3FSEBCAD174/XPr34oiYDyQrKDhc4BZh8p7/wBO2famRzowQFgQAgO98DgHPA6/5/F8TBkiC78ALnYNoztOeeueuce/rRYqLWgsgLKRIMKe8jgc7PT+v+FKW3CTYH2gNwqhAOF9eR/+r0pisqnETKD/ALHzH7nr/THrTpOkgcp/ER5rZPQdgMZ9/wDJLD7jwQzllMZJfgopc/fGPmx9MZ9s+lICpYbto3BSBK2c/e7DqOlK2SxyshAc5Jbyx98Zz3HfOenP4EbfvAEdhkqDsTHd+ue/vRYfVAN3ktgNjbnKYQD5B68/5HqacuDIWTaW39YhvbO/sx46Yxn296i48pmwhYJx8xkP3B+X8uvYCpXJ35bccv8A8tjtz+8GeB1H/wCugERvnykDnB2dJG2nGw9FHHpz9PepsdQMhC2Plj8pfvj15AqJRiEBA2ApHyR8fcbuev8AX8adwZAQFZhIPu7pWzuXueh/n+NA0JkNEpO1iYz1BkP3Mden/wBepCcPhyw+c8Sybf4mzwvXr/h1qNyVt2U5AEfG5tv8B7d/x/nUqAhz5YIG/B8pNg++eu7kfh0oGtxkanICIR6+XGEz/q/734fmD3NI+142XKMcH5SWkxwe38P9KahRiP8AVsTjgAzHonbp6/qO4p8zN5TK7OPkOFaQL/C/Ycn+v50C6DzuUvwVG5gNxCAfNIOMfMOvT146EU5drupCb+RyELHqP4mwDTdo3yFVXO45KRY7v1LH/wCvj6CgFXYByJPmA+YmTnK9QOBQUtEAkxJGHdiQV4MnIGY/4V59f170luGVYmUEHCZZUC44iPJb/PQ9zRHldgJKj5RtZxGP+WR7c57/AIcdKWJCdrKMvtUgiPcw+5/EeP6ZPvwC3YRlWAVdrHAGAWlH8P8ACvHY9PpT1zEqnJQYHdY1J+Xvyew/SkDAjaWyBjIMmT2/hUYPT/Jpw+QbgDGO7qqoP1JP+c+tBSImQOyfIrHcpwUaQdV6E4A6ent3FGnuLe5ty8jQoHQ/N5YxjyeR1zwM8nt6g0jDzAoXa+eD8rzA+3Yeg7elOtR5U0f3ohkZGEQDHl855x0z+GcZBzr9gLdUbkbRytdSmZ2YzBk5YhgZW5IBwOO35c1pwp5TxbQznbGcNE55/kf84rJtB5lvLJ5keBIOBcEscsfQc9f6963IicITmIDYCW3M447d8fTivGrK8r/11PVw2wkm5IDlVw6qeEOR836VEw/1rZwAVwCSPXt+FKxQwSAglii4bDDuc5pHIDSDHJZeS7D19a8eo1KV7q1vX+b/AIf7j0lsQS7ik3UgPzhs9SfWsy7/AIf90dq0pWwk2QDlwck59c/Wsy74bGOijtjtXPvb+urObEbGVL941DUsv3jUeK9in8J4E9xKKXFOC1TkibDa9P0YY0WzBOcRL/Fnt615oEr1DT4zFp1shBBWJQQQBjj2rvy53lI9HARtJss0UUV6p6YUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABUM1nbXIInt4ZQeodA386mooE0nuZE/hbRLj7+nxLzn93lP8A0Eis2fwFpEvMT3EJ/wBlwR+orqaKXKjGWGoy3ivuOFm+HRyTBqPHZZIv6g/0rNm8B6vHkxtbyjttcgn8wK9MoqXTic8stw72VvmeRT+GNat1JfTpSB/zzw//AKDms6a0uLb/AF9vLFzj50K/zr26ggEYIyKn2S7nPLKIfZk/6+48LxRivaJ9J0655msbZz6mIZ/Os2fwdoc5Y/ZDGx7xyMMfhnFL2bOeWUVF8MkzyjFJXo0/w9sH3eRd3EZPTcAwH8qzZ/h3dqD5F/DIfR0K/wAs0uSRzyy7ER+zc4r8aM10c/gjW4vuwRzD/YkH9cVmz6BqtucS6fcjAzkRkj8xxRY5pYetH4ov7jO5ozTnieNyrqVYdQRgikwRS0MrC4PcfnRzSc+tGKQC5ozSYNGDSAdmikoosAoODkcEU8TSKMLI4HsTUdFFhptEwuZV3YbG7OcAVIt664yisQc7jknqD1z7VUozRYpTkupeW9XjejP0+82RwT2xg9f0o+1RGLy/nHB6cDO0D6nn+Q96o5ozTH7WRrC5hL5R0HzjAC8n51P3mx/kUEo6fwtkY+ZjIfuv/COnPp061k5ozQX7d9UbTk7kDsTl+kjbAfmU4wP5Hp+AoIPkHYrYKEHbHwflb+I9fw/rWOsjLjaxGDng07z5Dnc5bP8Ae57Ed/qaCvbrsbWA0o4RsP15mI+b9P8APrTUztXeW6D77Afwp2H06e3PK1mjUJ85YhjnPcd89scUsd+0YAEagAY+UAZ4Xn/x2gv28GaDAGNtgOCCMogAPyyf3uc9af8ALI+QUkwTgjMxHzMepxjtz+PY1Qa+jccx4O0j5hvPIfjJPA+Yfz7CrIv4HbLSyEE/8tD756KMf596C41IPqWFLeZGHLg7h9+UA9U4wv8AL0qO3UhYyoIGFJKIEA4jPJP4H9e5qNLiE7AsigZUcERjrHnn7w6dfbjpUkIDBSoBIVeVTeRgJ/EePw9fY0FXTeg5NsiL91sjPIMuOB/wEdP5j0qQkqNhdgcYAeQJ+W3k9P1pm7rk5YDnc5YjHqq/Tv8AjSsDEhG4xrg4xtjX+L8c8e3egtDiCA7Y5GQWSPGPvdSx579qiuGEmfMw3UAAtL/e6gDFPYKd2FyVDchC5A+c8M30/Oop2+Z8SbyN24GQseN/VU4/OqhuFy3pyNBewbCtvhwfMCoijlfXJ7c8enTmtGOXcZSZVkLSH5hN15J4AFZ+mKyahE0TfZzv/wBcsKoOvX5sn9P61oRykpLumDMX5/egZ464A/zmubGO0ToomzF5O9D5skvKjBL8jb680r7WtW2xvkxoM5bGefamwupjCbhkyg4BJ/g/vdalCKIlCsSpRM5Zhzg/hXhYmV72tbX8vme3RjdEMhMcpB+UiQfeY8Hn1qpO+4SOTnLgnGDnr/n/ACK0LoqpJyoPndfNJPf17c1iXcn7p+vLjnAx39K5JQc20npqKtP2a1ILxh5nYfKo4GOwrNcZNWLjc1wqZBJVCMH1UU1otrYJB+hzXRGHsm2zx6svaMrhKeEqfyxn5c4x3p4jI9qUq6RMaJAI6t2FoLi9gixnc4BHTj/OaQR10nhzTSJDeyLhcERg9/U08LzYisoL5+h008PqdIAFAAGAOAKWiivsD0AooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAhms7a5BE9vDKD1DoG/nWdP4W0S4+/p8S85/d5T/wBBIrXopWREqcJ/Ekzlp/AWkS8xPcQn/ZcEfqKzpvh0ckwajx2WSL+oP9K7qilyROeWBw8t4/oeZzeA9XjyY2t5R22uQT+YFZ0/hjWrdSX06Ugf888P/wCg5r12ipdNHPLKqL2bR4jNaXFt/r7eWLnHzoV/nUOK90IBGCMiqc+k6dc8zWNs59TEM/nU+y8znllH8svwPF8UmK9Xn8HaHOWP2Qxse8cjDH4ZxWbP8PbB93kXdxGT03AMB/Kl7ORzyyqutrM85pPxrtZ/h3dqD5F/DIfR0K/yzWbP4I1uL7sEcw/2JB/XFHKznlgsRHeD/P8AI5zNHNaM+garbnEun3IwM5EZI/McVQeJ43KupVh1BGCKWhzyhKPxKw3NOwe4/OkwRRz60iReaM0mKMGgBc0uabg0tIBaAcHI4IpKKLASCaRRhZHA9iacLmVd2GxuznAFQ0lFiuZ9y2t664yisQc7jknqD1z7U9b1eN6M/T7zZHBPbGD1/SqOaM07FKrJdS99qiMXl/OOD04GdoH1PP8AIe9WBcwl8o6D5xgBeT86n7zY/wAisnNGaClWkjWJR0/hbIx8zGQ/df8AhHTn06dakcncgdicv0kbYD8ynGB/I9PwFYuacsjLjaxGDng0FKv5GwQfIOxWwUIO2Pg/K38R6/h/WpMBpRwjYfrzMR836f59axfPkOdzls/3uexHf6mphqE+csQxznuO+e2OKC4149TSTO1d5boPvsB/CnYfTp7c8rSMAY22A4IIyiAA/LJ/e5z1rPjv2jAAjUADHygDPC8/+O1I19G45jwdpHzDeeQ/GSeB8w/n2FA1Wj3L/wAsj5BSTBOCMzEfMx6nGO3P49jTlLeZGHLg7h9+UA9U4wv8vSq4v4HbLSyEE/8ALQ++eijH+fehLiE7AsigZUcERjrHnn7w6dfbjpQa88baMkt1IWMqCBhSSiBAOIzyT+B/Xuacm2RF+62RnkGXHA/4COn8x6U2EBgpUAkKvKpvIwE/iPH4evsafu65OWA53OWIx6qv07/jQOK0HklRsLsDjADyBPy28np+tBBAdscjILJHjH3upY89+1NYGJCNxjXBxjbGv8X4549u9DBTuwuSobkIXIHznhm+n50i7jLhhJnzMN1AALS/3uoAxVnTkaC9g2FbfDg+YFRFHK+uT2549OnNVJ2+Z8SbyN24GQseN/VU4/OrmmKyahE0TfZzv/1ywqg69fmyf0/rWv2Bx1aNCOXcZSZVkLSH5hN15J4AFbMXk70PmyS8qMEvyNvrzWNHKSku6YMxfn96BnjrgD/Oa2IXUxhNwyZQcAk/wf3uteJWklLp8z08LqOfa1q22N8mNBnLYzz7UyQmOUg/KRIPvMeDz61MEURKFYlSiZyzDnB/CkuiqknKg+d180k9/XtzXky11dk9P18j03GyuZ877hI5OcuCcYOev+f8is28YeZ2HyqOBjsKnu5P3T9eXHOBjv6VnXG5rhUyCSqEYPqoohQbV3/W55mIr9EV3GTTQlWGi2tgkH6HNO8sZ+XOMd63dXlVjg9m2yAJTxHU4jI9qeI6wniDaNAWwtBcXsEWM7nAI6cf5zXo4AUAAYA4Arm/DmmkSG9kXC4IjB7+prpa+iyqnJUeeX2vyO6hT5EFFFFeobhRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAcZ48G2XS5CAQrvn9Koy2mnyWBb7LCHx1UAH9KveP+V05f7zv/AOy1Va326VkvyD07dK+ZzltVU1Jp+XyOOCvWqrlTVlv6FjS/CGlalpMM7+dHI2cmN/f3BpJ/h3CTm31B19pIw36git7wv/yALfHq38zWxXvUIp0ot9kKODoTgm4rY84n+H+pof3NxbSr7kqf5f1rNm8Ja5CebFnHqjq38jXrNFaezRlLK6D2ujxSbT723GZ7O4iHq8TL/MVW4r3Sq0+n2V1/x8WkEvOfnjDfzqfZeZzyyj+Wf4HinFGBXrU/hLQ7j71iiH1jYr/I4rOm8AaW5JimuYvbcCP1Gf1pezZzyyqstrM8220bTXby/DuUZ8nUlb0DxY/UE1nT+BtahGUWCb2jkx/6FipcZIwlgcRHeH6nM7TRtPpWrP4e1m25k06cj/YXf/LNUJIpIW2yxPG3oykVLbRzypSj8SaINtJipuKXilzk8pBikxVjaD2FNMY9KaqC5SCjNSmMetMMZ7GrUkxWY3NG6gowpuG9KrRiH7qM1HmlzRygWFnlUYWVwOOAxqRbyZQdrKpIIJCgE598e9VQD1xTsEdqlotSktmXDqDuG8yNHzk/Nk4Jz2Jx/F+gps+rZVg0bnJJAEpUD7/YAf3/AMcVUzUEpycVdNaj9tNbM3rXU7XzhNIrABs5EQJAye5Jyef51v2lzA1oV86fzS2QMYBG3HYcc1yFqpVCcE+2cVv2xCuOnA9D615WY1nFPlPSwVSUn7x1kEkZl82CQMm/gOxDfcxnae1PDFY1ZQMZUZDHrjp0xXJCaXywCEjO0fMq553dOe2Kb9oETExlgfmOe/J+teS+ad0+72PaWMjBaI6a7uAxIZ9son5UEdj7/wCFc9cSL5BwpMpkz2+7g56c/jVU3s7tjzX6Zzk/e9etSeW8rBmJya2TVJXmcNau6790SBCx3GrQjpyIEQCpY0eV9kSM7eijNeZUrSqT901p0VFakYSn4rTt9Bv58FkWFfVzz+QrZs/D9tbkPMxnf/aGFH4V00crxdZ6rlXn/kdMY9kY+l6RJeusrjbbg8k9W9hXWxxpFGqIoVVGAB2pwAAAAwB2or6fBYGnhIWjq3uzVKwUUUV2jCiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArjPHg2y6XIQCFd8/pXZ1xnj/ldOX+87/wDstYYn+Ezlxv8AAl8vzKMtpp8lgW+ywh8dVAB/Srml+ENK1LSYZ386ORs5Mb+/uDVdrfbpWS/IPTt0rpfC/wDyALfHq38zXgZK26souTat3HOjCc0pwW3YwZ/h3CTm31B19pIw36gis6f4f6mh/c3FtKvuSp/l/WvR6K+j9nEyll2Hl9mx5NN4S1yE82LOPVHVv5Gs2bT723GZ7O4iHq8TL/MV7XRU+yRzyymm/hk0eF8UcV7XPp9ldf8AHxaQS85+eMN/Os6fwlodx96xRD6xsV/kcUvZPuc8spqL4ZJ/18zyXAo216TN4A0tyTFNcxe24EfqM/rWbL8O5RnydSVvQPFj9QTU+zkYSy3ELpf5nEbTRtNdNP4G1qEZRYJvaOTH/oWKzp/D2s23MmnTkf7C7/5ZqWpI55YarHeL+4ytp9KTbU8kUkLbZYnjb0ZSKbxU8zMeUhxRip+KNoPYUc4cpXxSVOYx6U0xj1qlNE8rIs0ZpxjPY00owq00xWDdS7qZhvSkzTsBJmpVnlUYWVwOOAxqvmnAHrik0NNrYtLeTKDtZVJBBIUAnPvj3qQ6g7hvMjR85PzZOCc9icfxfoKp4I7UmamxaqSXUtz6tlWDRuckkASlQPv9gB/f/HFXLXU7XzhNIrABs5EQJAye5Jyef51gynJxV61UqhOCfbOKuq+WmXTr1OY6+0uYGtCvnT+aWyBjAI247DjmtyCSMy+bBIGTfwHYhvuYztPauTtiFcdOB6H1pwml8sAhIztHzKued3TntivmalapKenRr9T6LD1Ywj7x1oYrGrKBjKjIY9cdOmKrXdwGJDPtlE/Kgjsff/CuZ+0CJiYywPzHPfk/Wojezu2PNfpnOT97161MKTevQurj1blsWriRfIOFJlMme33cHPTn8aggQsdxpfLeVgzE5NW0QIgFRiMRFR5YnHTpucuZ7DRHTwlSRo8r7IkZ29FGa0rfQb+fBZFhX1c8/kK46dHEV3+7i2dsYLoZmK1NL0iS9dZXG23B5J6t7Ctiz8P21uQ8zGd/9oYUfhWuAAAAMAdq9nB5I+ZTxD+X+ZtGI2ONIo1RFCqowAO1Ooor6RK2iLCiiigAooooAK5/x3/yTzxL/wBgq6/9FNXQVz/jv/knniX/ALBV1/6KagA8Cf8AJPPDX/YKtf8A0UtdBXP+BP8Aknnhr/sFWv8A6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOW8Z2V3dLYyW1s84iZy4TkqMDn9DWbPDeGw2GyuPNIBwEyNp6dO9d3RXDicvpYiXNPcwdF80pJ25jJ8N209rocEVxGY5BuJU9RknFa1FFdkIqMVFdDWEeWKiugUUUVRQUUUUAFFFFABRRRQAUjKrqVZQwPYjNLRQBQm0TS7hi0un2zMereWAT+NZ03gvRJclbd4ie6SH+ua6Cik4p9DKVClL4op/I46f4e2jA/Z76dD28xQ38sVmz/D+/T/j3vbeT/fBT/GvQ6Kn2cexzyy/Dy+zY8qn8H67CeLQSrj70cin+ZBrMn0vUrYkT6fdJjuYmx+fSvaKKXsonPLKaT+GTR4UWGcHgjt0oyD3r3Ca2t7lSs8EUqkYIdAwP51mz+F9EuFIfTYFz/zzGz/0HFL2fY55ZRP7Mk/6+Z5CPqaaVX/PFdr4l8L6dYG3Noske8OSN+RxjHX61mjwukluksd4wLD7rR8Z+ua4quMo0ZuE3ZryOb+zsQ5OKV2vP/M5svjOP5fWkaXJOeevv61uWnhXUNQedLcxM0ONwY4znPT8qin8I65CpLadIQP7jBv5E11U5QqRU46pnM8PXS+FmK0vX/PrUanL5P8AnrVq4066th+/tpof99CtVhGQcit1y20MGmnZmhFIAhAYj3BxVsXAEu/A3YxnnpnNZcaOTgfzrTstH1G/I+z2ssgPRgp2/n0rhq0IyZvSlN6RQ0zueV54Axj3p0SyTScZ2kEHj3rp9P8AAVw2Gvp0iH9xPmP+H866iz8N6bZqAIPMYfxSHP6dKzeGm1aCt6npUcDWqaz0Rw9jpsshVI4nkI7Kua6C28N3kuDKyQL7/MfyH+NdYiJGoVFVVHQKMCnVEcppyfNWbk/uR6tPDxpqyMe28N2cODKXnb/aOB+QrVihigXbFGiL6KMU+ivQpUKVJWpxSN1FLYKKKK2GFFFFABRRRQAUUUUAFFFFAHP+O/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtHjv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFct4zsru6WxktrZ5xEzlwnJUYHP6GupoqJwU4uLM6tNVIODOEnhvDYbDZXHmkA4CZG09Oneul8N209rocEVxGY5BuJU9RknFa1FcuFwFLDSbp9RKm+fnbvpYKKKK7TUKKKKACiiigAooooAKKKKAEZVdSrKGB7EZqjNoml3DFpdPtmY9W8sAn8av0UWuTKMZbq5z83gvRJclbd4ie6SH+uazp/h7aMD9nvp0PbzFDfyxXY0VLhF9DCWDoS3ivyPPJ/h/fp/wAe97byf74Kf41mz+D9dhPFoJVx96ORT/Mg16rRU+yic8ssoPa6+f8AmeLz6XqVsSJ9Pukx3MTY/PpVIsM4PBHbpXutRTW1vcqVngilUjBDoGB/Oj2SOeWUL7M/wPD8g96B9TXr0/hfRLhSH02Bc/8APMbP/QcVyniXwvp1gbc2iyR7w5I35HGMdfrWNZqlBzk9EctXLK0FzJpnFFV/zxSF8Zx/L610g8LpJbpLHeMCw+60fGfrmq1p4V1DUHnS3MTNDjcGOM5z0/KsKOMo1pckZXfYwngcRC3u79jDaXJOeevv60xpev8An1ran8I65CpLadIQP7jBv5E1l3GnXVsP39tND/voVrtSijnnTqx+KLXyKqnL5P8AnrWjFIAhAYj3BxWcIyDkVYjRycD+dKtGMluRFtPQ1BcAS78DdjGeemc0wzueV54Axj3p1lo+o35H2e1lkB6MFO38+ldPp/gK4bDX06RD+4nzH/D+dcKwy6K530qder8KOYiWSaTjO0gg8e9a9jpsshVI4nkI7Kua7iz8N6bZqAIPMYfxSHP6dK1ERI1Coqqo6BRgVMsBUq6Slyry3PUoYDl1m9Tk7bw3eS4MrJAvv8x/If41rW3huzhwZS87f7RwPyFbFFbUcsw1LVRu/PU71TihkUMUC7Yo0RfRRin0UV3JJaIsKKKKYBRRRQAUUUUAFFFFABXP+O/+SeeJf+wVdf8Aopq6Cuf8d/8AJPPEv/YKuv8A0U1AB4E/5J54a/7BVr/6KWugrn/An/JPPDX/AGCrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBzHi44NoD/dk/kKp2Ck2UHAxk/8ALTtn0xVjxnIENp6lXH8qp211FFawruycngc//q/Gvkc4jJ15tK//AAyM6E4rEPmdv6RqeGcG91LH95P610dct4Pl8241Ns5+dOf++q6mvoMtg4YWEX0ClJTjzLu/zCqk+ladckGewtpSOheJSR+lW6K7i3FPRlOHSdOt23Q2FrG3qsSg/wAquAYGB0oooBRS2QUUUUDCiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/JPPEv/AGCrr/0U1HgT/knnhr/sFWv/AKKWjx3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+iloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuY8XHBtAf7sn8hXT1ynjOQIbT1KuP5VxZiubDSXp+aMa7tTbK9gpNlBwMZP/LTtn0xWh4Zwb3Usf3k/rWXbXUUVrCu7JyeBz/+r8aveD5fNuNTbOfnTn/vqvncqpyeNU2rK35mk6keWnBO7/4B1NFFFfXlFSfStOuSDPYW0pHQvEpI/Skh0nTrdt0Nhaxt6rEoP8quUUWJ5I3vYAMDA6UUUUFBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/AI7/AOSeeJf+wVdf+imroK5/x3/yTzxL/wBgq6/9FNQAeBP+SeeGv+wVa/8Aopa6Cuf8Cf8AJPPDX/YKtf8A0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBja9oX9spEVlEbxggEjPXH+H61myeEphAVjuY2YD5QwbGd3Pf09v8AGurorGdCnNtyRjLD05Nya1ZkaHon9jm5PnCQzMDwuMAZ/wAa16KK0jFRVkaQhGEeWOwUUUVRQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/jv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0eO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVja9oX9spEVlEbxggEjPXH+H61s0VMoqSsyZwjOPLLY5STwlMICsdzGzAfKGDYzu57+nt/jWpoeif2Obk+cJDMwPC4wBn/Gteis44enCXNFamcaFOMlJLUKKKK2NgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv/AGCrr/0U1dBXP+O/+SeeJf8AsFXX/opqADwJ/wAk88Nf9gq1/wDRS10Fc/4E/wCSeeGv+wVa/wDopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv/AGCrr/0U1dBXP+O/+SeeJf8AsFXX/opqADwJ/wAk88Nf9gq1/wDRS10Fc/4E/wCSeeGv+wVa/wDopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv/AGCrr/0U1dBXP+O/+SeeJf8AsFXX/opqADwJ/wAk88Nf9gq1/wDRS10Fc/4E/wCSeeGv+wVa/wDopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv/AGCrr/0U1dBXP+O/+SeeJf8AsFXX/opqADwJ/wAk88Nf9gq1/wDRS10Fc/4E/wCSeeGv+wVa/wDopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv/AGCrr/0U1dBXP+O/+SeeJf8AsFXX/opqADwJ/wAk88Nf9gq1/wDRS10Fc/4E/wCSeeGv+wVa/wDopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv/AGCrr/0U1dBXP+O/+SeeJf8AsFXX/opqADwJ/wAk88Nf9gq1/wDRS10Fc/4E/wCSeeGv+wVa/wDopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv/AGCrr/0U1dBXP+O/+SeeJf8AsFXX/opqADwJ/wAk88Nf9gq1/wDRS10Fc/4E/wCSeeGv+wVa/wDopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKgvbyDTrC4vbqTy7e3iaaV8E7UUZJwOTwD0qtFrmmzXZtUul80Wi3pDKVAhYkBySMDlTx14oA0KKw9K8Y6Brd6LOw1ASzshkjVonQSqOrRlgA491yK0dM1K01jTYNQsJfOtZ13xybSu4fQgEfjQBbooooAKKKKACisG38aeHrrVV02HUka4eRokPluI5HGcoshGxmGDwCTxWnb6laXWoXljDNuubPZ56bSNm8ZXkjByPSgC3RRRQAUUUUAFFFFABRRRQAUUVUu9StLG4s4LmXZLezGC3XaTvcKz44HHyqx5x0oAt0UUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/47/5J54l/7BV1/wCimo8Cf8k88Nf9gq1/9FLR47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUVlWfiTSdQ/s37Ld+Z/aUTy2n7tx5ipjceRxjcOuOtAGrRWDb+NPD11qq6bDqSNcPI0SHy3EcjjOUWQjYzDB4BJ4rTsNStNTSd7OXzFgne3kO0jbIh2sORzg9+lAFuiiigAooooAKKwbrxp4es9UbTrjUkS4WRYnPluY43OMK0gGxWORwSDzWn/aVp/a39l+d/pvkfaPK2n/V7tuc4x14xnNAFuiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/JPPEv8A2Crr/wBFNXQVz/jv/knniX/sFXX/AKKagA8Cf8k88Nf9gq1/9FLXQVz/AIE/5J54a/7BVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAorn9Y8W2ukX8lkthqN/NDCLi4FlCH+zxkkBnyR12tgDJ4PFbNndwahY297ayiW3uI1likXoysMg/iDQBPRWJqXiRNL1zT9Nn02+KX0ohivEEZhEhVm2n592cIT93HvSax4nt9K1CLTo7K+1G/kjM32ayjDMkecb2LMqqM8DJ5wcZxQBuUVnaLrVpr1h9rszIoWRopYpkKSRSKcMjqejD/6/Q1o0AFFFFABRRRQAUUVk+I9dHhzSZdSk069vYIVZ5haCMtGiqWLne68ADtk+1AGtXP6pqV3pnizR0eXOl6gsloyFR8lwBvjbdjPzKsi4zjO2ty3mW5top0BCyIHAPXBGa4/xvNd6jYXOk2+h600iNHNb6laJbssUqEOrqHmQnBGCCBnkdKAIrH4hJLF4muLu2EdvpYaa0Knm6hDPGGHqTJE4GPVfXkt7vxPfXll4d/tOO0voNOjvNUvhbo775GYLHGpG0YKPliDwF4ySa5t7fRLnRPA9/ajVNqReVFpkdsrz6jHGyOC/z7VUPGsm4sRhsZ+aupVI/FOpvquj3l3omvWC/ZLqG6tgx8s/Oqyxk4YcllZW7nB6igDA8S3fiTT7nVtD1HVI77TpfDeoXMcv2dI5HdQi4fAx8u44K4B3cjjNW49MXWtcutKeVolvfCEFuZF6qHeVc/rWhP4Au7++vdQ1LxBLc3t1pdxppP2YJFGku3BRA3GCCeSSc9eBV648EWt9LcC9uWltrjR49KkiVNpwrMfMDZOD83AxxjqaAKNrd3djquhaf4p0a2WaNzFp2p2cpaEy+Ww2lSA0ZZA3HzDtnpWB4St/FUHwwstWs9chi+zWjSwaebRWikRMkK7n58tjqpGM9DjnrrTwrqUl/p8+t+IG1OHTn822iFqsJMm0qHkYE7yAzYwFGTnFXNI8Of2V4Mi8PfavN8u1a38/y9ucgjO3J9emaAOW1Xx1Jd3+nWVrd3OmQT6bFqM1zb6bJey4lzsjUKjKv3WJZgc8ADrh+meOLixj1n7a97qdlYae9/Hey6ZLZu2z70TB0VS3QgqBxnI4rZHg6a1tdJk0zVms9T0+xjsTc+QHjuIlA4kjJ6ZBIwwIyeeTVuw8MsFv5Nbv31W4v4fs825PLiWLBGxIwTtB3HJJJPrwKAMlrXxquitq/wDb9t9u8kz/ANnCzT7MOM+WH+/7bs9ecdqyPDms+J/EOpaFAdYWG2l0C11C8b7MheWRnYMF4AXcBye2OACc1DqK6haNceFrXWtb1Owt4ljuY7LT43uIImHEbTl1BJX0Uvjnqc1ueDvD+nGxsNQs9Q+3WJ0WLSgrQlCwjZ9xbJyDklSuOCDzQBnwQXPhLSNL0TxDpUF7oVncwx22p20mGiYSAQtLGQCp3FcspIyeRzVBbfWNH1rx9qEfiK8lmsbFJgJLeDEjfZ3KFsRj7pxjGM45zUV0ItKI0K91bVL7wvo00IuGjsE22+wq8cU027LKvyE7UyBjcetbfiJLa01LxFYR/br+/wDENiF+zWNsHa2jEbReYxZ1Ugk8AkZwQM0AU7vxNrmk2OiWd/qpN9rETXUl1Bpjzi0iVUykccYJZizgbm4HJx0FW9B8UX58QwaYt3qGs2t1HIRPd6RLZvbyKu4BmMaIysAR0BBx1zwmp3mmXPgyx1y0ub+C90Z1tYnhgUXCTErC0LxSEA7iVypI7EHoateG7TVtV1G4u/EC6xHcJA0MBlihtoUV/vFFimkO/gfMx47Y5oAytD8Q+KJ9P1Jm1CO612PTpJm0S5s/s8tvcj7oj4HmRds5Ofl+YZre8D6pdait4tzro1PyxHmOe0+zXNs5B3JJHgYHTacevJ61XTwHfzsran4ovbmS3spLOynhjEM0Icrl2cEl3+ReeAecjmrFrpWp6BfXOuXrXXiLUrmOK022UENuI4k3sDteQA/Mxyd2eRgAZoA2vEt7PpvhXWL61YLcW1lNNExAOGVCQcHryK56ytPGt9olvqo8QW8V9LAsy2H2NDb5KghGb75J6FgR1OB2p3iHV9R1HwtrVm3hbWLXzdOuVEszWzKD5TYGI5mYknAGFPJFRaZ4W1weHrTTo/FNzBprWyIYzaqbmNCoyizE8Y5AJUsPXIzQBT07xLrPijxHpzaffjT9Kl0aDUp4jAsr7zK4ZFJHcLgnn7vAyc1l2fje+1qxXVV17UNOacGS3sofDs9xCqfwh5PLJckYyUZRzx613eneF7bStcS+s3EVtFpsWnRWoThFjdmB3Z5+9jGO2c1ymlW2o/21qHhjSdS1LRbOBPPSG6s4ZCkTuy/6PIsh2rlWwHUkDH0oAvR+KdY1y08P2VjCNL1LVIpprmS4gY/Zo4WCuVRsElmZdu7oDkjtVPV7PXbDxh4Phv8AVV1OxfUZGWWSBYpo5BbzcfJhWUgnsCMd88a/iLRtM0jw9ZXo1G502TRVxbXqjzpPmwpRlOfM3naCOpOMEGuf05LjWPGOh3PiG/1W31C2kklsbe509baCceWyuFCyP8+GDfM2QFOFGTQB6fRXIX/juTS7c3F74S8QQwB1j8wrbEbmYKoAE2TkkDgd66+gAooooAKKKKACiiigAork0+IOlvi4+xakNKaXyl1U24+yk7toOc7tu7jft2++K6ygAorC1PxZpul+JNL0CbznvtSLeWIlBWMAEgucjAO1gOucH0NVNW8Yy6Ol7NceF9ca0tA7SXSfZvLKLklxmYHGBnkA+1AHUUVBZXQvbC3u1ikiE8SyCOUYdNwzhh2IzzU9AHP+O/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtHjv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRXP6x4ttdIv5LJbDUb+aGEXFwLKEP8AZ4ySAz5I67WwBk8HigDoKKgs7uDULG3vbWUS29xGssUi9GVhkH8Qay9S8SJpeuafps+m3xS+lEMV4gjMIkKs20/PuzhCfu496ANuisPWPE9vpWoRadHZX2o38kZm+zWUYZkjzjexZlVRngZPODjOKt6LrVpr1h9rszIoWRopYpkKSRSKcMjqejD/AOv0NAGjRRRQAUUUUAFFFFABRWT4j10eHNJl1KTTr29ghVnmFoIy0aKpYud7rwAO2T7VpW8y3NtFOgIWRA4B64IzQBJRWFqXiG70+9kgj8M6zexoAftFt9n8tsjPG+VTx05HaqcfjzT5tD0rUYrHUZJdVDGzsY4la4kC9WwG2hcYO4tjBHPNAHU1z/ifUrvRpNJ1COXGni8WC/jKgjy5fkV84yNshTp2JzVvRPEFrrguUihuba6tXCXFpdR7JYiRkZGSCCOQQSD68VY1jS4Na0W90u5/1N3C0LEdRuGMj3HUfSgDAtfGYl8b6no08KRWFrCWiuycB5IwjTL/AMBEsf5N6cZdpr/iHU7bRrOC4jtr3W/tF+s00Ib7HZqy7FVeNzlZI/vZwSxOcAVLdfDYXnhax0mbWZTdw3Ms91fiHD3Qm3idSN3yh1cjqcYHpXQa34cTVPsNxaXcmnX+nkm0uYUDbAwwyMp4ZCAMjjoCCMUAc9dz+KNC8WeG7GbW11DTdQupI5nkto45gVhkYL8oAKkgHIAI24yQaxvB4yPhyOedNvun/bKumHg7UbvxBpWtav4he7n02VnhhitRDDhkZG+XcTuO4HJJ+7gAZNO0zwPHYQ+H4Zb0XEek2lxasph2icS7QT947cbenOc9qAMaCC58JaRpeieIdKgvdCs7mGO21O2kw0TCQCFpYyAVO4rllJGTyOaZ4Z03xFff8JC9jry6bAmtXogjS0SXe3mHJkLc4zxhdvTrzxq2/ga8SztNIuvEEtzoFpJG0Vm1sqyssbBo43lz8ygqvRQTjk02x1W28M32q6Va2up6zcG9lvbn7Dagi285i4RizAE4PQZJGDgZoAy7r4gXk+jeHkBNheamk7XVxBZS3fk+QwR/LjQEklyMFuAOuTjNvQPFN4PEdvpr3uoavY3McjG5utHltHtnVd3zMY0RlYAgcAg4654m0Tw5Z6j4T0i40jWZVubSWea01COHaV8yRi8bxtnI52spwcoDwRxTXWk0fxDdXuuX2par/ZwFvNdWtmIrKx3hWO5d5ZmwVJb5toPbmgC7pDeKfFGkQ6/Br0emx3iedZ2S2aSosZ5TzWPzMxGCdpXGcDpWFpXiHxfrQ8O2n9pwWd1fT6ml5ItukgQQyhVCAgZIHygnrnJzireowXnhe6XQNA1jUXhmRp49OtNPjuJrSIsclJHdVRNxIUPu9BnFQaJbaR4e8Pabr1rd391baNPd27Wj24Fy0tzMg8pgzDDqxA5+9nP1ALl9ZXvhXStZt9T0qHW/C9zPPd3MkUm24iSRi8m+MjDhSScqwOB04p0Wn3s3xbjuIvEN21u+lC4VBDCVMRm4izszt987v9qodN8O6pqcGo6PG+saL4fuWd5LW8trdn2ysTJHFKkrbVJLdVJG7g12MWgJB4nTV4pgscenixW3CcAB9wO7P4Yx+NAHA2Pi7xHZeB9J1fUtTSe71qZLaAJYGRbYYdmlKRDdIxVCdowMkdgTV7T/ABXfWutafDHqeqa3b3c6wTx3OhTWzQbuBIriJV2g4yG7HOeOegt/BcMXg3TtBkvpfO0/a9vfQqEeOVSSHUHI7kEHIIJHepbTw9qkmpW13revNfJaMXgt4LYW8ZfBG6TDMXIBOBkDvjpgAwtI1zXJvF01pqGqJbz/AGi4VdHubURrJAufKeCXH7wkBS3J4LcDFSeBta1jUb9otX1dXvPs5e70q4s/s8tpLuH+r4+eLGRuJbPynPOKtjwTeTX9t9u8RXV1p1ncSXNrA0eJkd1dRmfduYKJG28A9Mk4qfRvCV9ZaxaahquvS6o1hbPa2e+ARsquV3NIwJMjkIozx3OMmgDqqKxNO8SJf6/e6M+m31nc2sSzZuBHtljZmUMhV2OCUPUA+1XtW1Wz0TTJtRv5THbwgFiFLEkkAAAckkkAAdSaALtFc9p3i62vdSg0+607UtMublWa2W/hCCfaMkKVZhkDnacHHbiqN/47k0u3Nxe+EvEEMAdY/MK2xG5mCqABNk5JA4HegDr6KKKACiiigAooooAKKK5NPiDpb4uPsWpDSml8pdVNuPspO7aDnO7bu437dvvigDrKKKwtT8WabpfiTS9Am8577Ui3liJQVjABILnIwDtYDrnB9DQBu0Vy+reMZdHS9muPC+uNaWgdpLpPs3llFyS4zMDjAzyAfauhsroXthb3axSRCeJZBHKMOm4Zww7EZ5oAnrn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKy9V1610eSNLi21KYyAkG00+e4A+pjRgPxrUooA5vVoZfEWl38PhzV4tOv8Ad5NxcfZQ7qdmQjA4IOHU56jNP8C3UV14K0vyrUWqwRm08lX3BDCxiOGPUZQ4PcUaj4Rtr7Upr+21LUtMnuVC3JsJggnwMAsCp+YDjcuDjHPAq3D4dsrSDSbeyae0ttMfdDDDJhX+RlxJnJYfMW653AHNAGX4y/5CHhL/ALDif+k89XdavtJ8MmfWZoGa+uxHbJHCC010ylvLjRe5yzfmSeBV3U9It9Vm06Wd5VawuhdxbCAC4R0w2QeMOemO1ZGr+C49V8Rxa4uuatZ3cMJgiW3aEpGp+8VEkbYJ7kdQMUAT+EdJvdOsby61PYuoandte3EUZykJZVURg99qooJ7nJroKo6Vp82m2rQz6pe6izOWEt55e9RgDaNiKMcZ6Z5PNXqACiiigAooooAoarq8GkRRyXEF9KHbaBaWctwR9RGrED3NZPi26S9+Gmv3MaTIkmk3LBZomicfum6qwBB9iK6Wqmq6dDrGj3umXDOsN5byW8jRkBgrqVJGQRnB9KADSv8AkD2X/XvH/wCgisPxxczNpMOiWchS91qYWUbL1jjIJlk/4DGHOfXFa91pEd1ptvYi5uYUgeF1eF9rt5bKwUnHQ7cEdwTSSaNby+IYNakeV7iC2e3hjJHloGYFmAxncdqjOeg+tAFW+TQvDdpFq90kVtFp1qbaKTnKREr8ijuSUQAAZOABVLwrZX82oan4j1K3a0n1QRJFZt96CCMNs3/7ZLsSO2QO1TeI/CMPiO/0+8k1XUbOSwYvCtq0RTeeN5WRGBYDoe2Tir+kaTcaX53n61qOpeZtx9t8r93jP3fLjTrnnOeg6UAadFFFABRRRQAVl6rr1ro8kaXFtqUxkBINpp89wB9TGjAfjWpRQByfglxJdeKXwwZtackMpDAGGHGQeRxineA8f2frG37n9t3+30/17Z/XNWtQ8JW95qU+oWupalplzcqq3LWMyoJ9owCwZWGQONwwcY54FT2vhmxsLXSraykubeDTZWlSOOU4mLKwPmE5L5Llv97BoAwvF+nzeI/Cd2+galbR2beabyARDbe7Dh42kGCmdjKWGT+VXDr2i2GiW3iw2jrc6ta26wwRjdNOSpaOFV7n526e5PAp154FsLue7KX+p2tneuZLuxtrgJBOzfeJGNy7v4trLnJz1pupeBre+1231aDWNU0+a1t/s1vHamHy4U77FeNtpPAJHYAdKAMSXwxfXGjxWt9q0Gmazq+srqjIqiUK0YVhEgPDFViQknIJUnBFbNjcaroni200S+1STVbW/tZpoZZ4o0mheIpuB8tVUqRIMHGQRjnNXJfCcN3pP2HUdU1K/dJxcQXczok9u4GAUaNFAxz2P3iDkHFS6R4ZttKv5NQkvL3UNQkj8n7VeyBnWPOdihQqqM8nA575xQBt0UUUAFFFFABXKwf8lXv/APsB23/o+euqrPTSLdPEM2tB5ftMtqloy5GzYju4IGM5y57+lAGL47/48dG3fc/tux3Z6f65cfrijxlj+0PCWPv/ANtpj1/1E2f0zWlc+GrG9tdVtb17i5t9RlErxyy8REKoHl4wUwUDDuGyarWHhG3tNUg1G61LU9TuLZWW2N9MGEG4YJUKqgkjjccnBPPNAFTU/wDie+OtO0ofNaaQg1G79DM2VgQ/T53/AOArXWVnaZo1vpdxqNxE8sk9/cm4mklIJztChRgD5VVQAP51o0AFFFFABRRRQAU10WSNkcZVgQR6inU2RDJE6B2QspG5eq+496AOG16ODVrT/hX+gQqsCwpDfTLzHY23HyZ7yMowq9R94+/Va7rNp4c0G81e+Yi2tIjI2Op7AD3JwPxrndM+HzaNam20/wAW+IIYmdpW5tWZ3Y5ZmYwEsT6kk10q6ZGNUub5pp5PtEMcLQSMDEoQsdyrjhju5PfA9KAPKodW0T+0PDWoXXiHR7nWr/Whc3/kX0biFfs0ypEMNwiblQHuxJ6tXb+Lv+JxqWk+Fk5S8k+1XwHa1iIJB/33KL7gtWvf+HtP1G6064kj2PYXP2mIRqoDNsdMNxyMOTxjkCpLfRre31291jfLJdXcUcJ3kFY40zhU44BLMTnOSfpQBo0UUUAc/wCO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtHjv/AJJ54l/7BV1/6KajwJ/yTzw1/wBgq1/9FLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBl6rr1ro8kaXFtqUxkBINpp89wB9TGjAfjWfq0MviLS7+Hw5q8WnX+7ybi4+yh3U7MhGBwQcOpz1Ga6Sue1HwjbX2pTX9tqWpaZPcqFuTYTBBPgYBYFT8wHG5cHGOeBQAeBbqK68FaX5VqLVYIzaeSr7ghhYxHDHqMocHuKreMv8AkIeEv+w4n/pPPWpD4dsrSDSbeyae0ttMfdDDDJhX+RlxJnJYfMW653AHNTanpFvqs2nSzvKrWF0LuLYQAXCOmGyDxhz0x2oApa1faT4ZM+szQM19diO2SOEFprplLeXGi9zlm/Mk8CmeEdJvdOsby61PYuoandte3EUZykJZVURg99qooJ7nJqDV/Bceq+I4tcXXNWs7uGEwRLbtCUjU/eKiSNsE9yOoGK2dK0+bTbVoZ9UvdRZnLCW88veowBtGxFGOM9M8nmgC9RRRQAUUUUAFUNV1eDSIo5LiC+lDttAtLOW4I+ojViB7mr9FAHNeLbpL34aa/cxpMiSaTcsFmiaJx+6bqrAEH2Ira0r/AJA9l/17x/8AoIo1XTodY0e90y4Z1hvLeS3kaMgMFdSpIyCM4PpUV1pEd1ptvYi5uYUgeF1eF9rt5bKwUnHQ7cEdwTQBkeOLmZtJh0SzkKXutTCyjZescZBMsn/AYw5z64q5fJoXhu0i1e6SK2i061NtFJzlIiV+RR3JKIAAMnAAq1Jo1vL4hg1qR5XuILZ7eGMkeWgZgWYDGdx2qM56D61neI/CMPiO/wBPvJNV1GzksGLwratEU3njeVkRgWA6Htk4oAh8K2V/NqGp+I9St2tJ9UESRWbfeggjDbN/+2S7EjtkDtXUVmaRpNxpfnefrWo6l5m3H23yv3eM/d8uNOuec56DpWnQAUUUUAFFFFAGXquvWujyRpcW2pTGQEg2mnz3AH1MaMB+NY/glxJdeKXwwZtackMpDAGGHGQeRxiusrn9Q8JW95qU+oWupalplzcqq3LWMyoJ9owCwZWGQONwwcY54FAFXwHj+z9Y2/c/tu/2+n+vbP65qn4v0+bxH4Tu30DUraOzbzTeQCIbb3YcPG0gwUzsZSwyfyrdtfDNjYWulW1lJc28GmytKkccpxMWVgfMJyXyXLf72DVC88C2F3PdlL/U7WzvXMl3Y21wEgnZvvEjG5d38W1lzk560ANOvaLYaJbeLDaOtzq1rbrDBGN005Klo4VXufnbp7k8CsSXwxfXGjxWt9q0Gmazq+srqjIqiUK0YVhEgPDFViQknIJUnBFbepeBre+1231aDWNU0+a1t/s1vHamHy4U77FeNtpPAJHYAdKtS+E4bvSfsOo6pqV+6Ti4gu5nRJ7dwMAo0aKBjnsfvEHIOKAKdjcaroni200S+1STVbW/tZpoZZ4o0mheIpuB8tVUqRIMHGQRjnNdZWJpHhm20q/k1CS8vdQ1CSPyftV7IGdY852KFCqozycDnvnFbdABRRRQAUUUUAcrB/yVe/8A+wHbf+j56PHf/Hjo277n9t2O7PT/AFy4/XFbSaRbp4hm1oPL9pltUtGXI2bEd3BAxnOXPf0qvc+GrG9tdVtb17i5t9RlErxyy8REKoHl4wUwUDDuGyaAM3xlj+0PCWPv/wBtpj1/1E2f0zTdT/4nvjrTtKHzWmkINRu/QzNlYEP0+d/+ArVuw8I29pqkGo3WpanqdxbKy2xvpgwg3DBKhVUEkcbjk4J55rQ0zRrfS7jUbiJ5ZJ7+5NxNJKQTnaFCjAHyqqgAfzoA0aKKKACiiigAooooAa6LJGyOMqwII9RXDa9HBq1p/wAK/wBAhVYFhSG+mXmOxtuPkz3kZRhV6j7x9+5kQyROgdkLKRuXqvuPeuO0z4fNo1qbbT/FviCGJnaVubVmd2OWZmMBLE+pJNAHRa7rNp4c0G81e+Yi2tIjI2Op7AD3JwPxry2HVtE/tDw1qF14h0e51q/1oXN/5F9G4hX7NMqRDDcIm5UB7sSerV6qumRjVLm+aaeT7RDHC0EjAxKELHcq44Y7uT3wPSoL/wAPafqN1p1xJHsewuftMQjVQGbY6YbjkYcnjHIFAGR4u/4nGpaT4WTlLyT7VfAdrWIgkH/fcovuC1dZWdb6Nb2+u3usb5ZLq7ijhO8grHGmcKnHAJZic5yT9K0aACuf8d/8k88S/wDYKuv/AEU1dBXP+O/+SeeJf+wVdf8AopqADwJ/yTzw1/2CrX/0UtdBXP8AgT/knnhr/sFWv/opa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigD5X+L3/JVNZ/7Yf+iI641a7L4vf8lU1n/th/6IjrjVr2KXwL0PTh8C9CZamWoVqZa1IkSrUq1EtSrVGEiRakFRrUgoMJEgqQVGKkFBhIeKeKYKeKlnPIeKcKaKcKRhIeKeKYKeKlmEhwp4pgp4qGYyHinCminCpZjIcKeKYKeKhmMh4pw6U0U4dKzZix4p4pgp4rNmbJBT1pgp61jIklFSrUQqVa55FIlWpVqJalWuWZrElWplqFamWuOZvElWpVqJalWuOodMCZalWolqVa4Kh1wJVqVaiWpVrhqHbTJVqVaiWpVrhmdtMlWpkqFamSuZnZA0NM/wCP6L8f5Gt+sDTP+P6L8f5Gt+vuOGf90l/if5IwxPxr0CiiivojnOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAr5X+L3/ACVTWf8Ath/6Ijr6or5X+L3/ACVTWf8Ath/6Ijrqwnxv0OnC/G/Q41amWoVqZa9JHVImWpVqJalWqMZEq1ItRrUi0zCRIKkFRipBQc8iQU8UwU8UjCQ8U8UwU8VLOeQ4U8UwU8UmYyHinCminCoZhIeKeKYKeKlmMhwpwpopwqGYyHinimCnioZjIcOlPFMHSnis2ZMeKkFRipBWUiGPWpRUS1KKwkCJVqVaiWpVrmmWiValWolqVa5Jm0SZalWolqVa46h0QJVqZahWplriqHVAlWpVqJalWuCodlMlWpVqJalWuCodtMlWpVqJalWuOZ20yZK0NM/4/ovx/kaz0rQ0z/j+i/H+Rroy/wD3ul/ij+aOmXwP0N+iiiv0880K5/x3/wAk88S/9gq6/wDRTV0Fc/47/wCSeeJf+wVdf+imoAPAn/JPPDX/AGCrX/0UtdBXP+BP+SeeGv8AsFWv/opa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDKvPDGgahdvdXuh6Zc3MmN801pG7tgYGSRk8AD8Kh/wCEN8Lf9C1o/wD4Axf/ABNbdFVzPuPmfcxf+EO8Mf8AQuaR/wCAMX/xNH/CH+Gf+hc0j/wCj/8Aia2qKOeXcLsxv+ER8Nf9C7pP/gFH/wDE0v8AwiXhr/oXtJ/8Ao//AImtiijnl3C7Mf8A4RPw3/0L+lf+AUf+FL/winhz/oAaV/4Bx/4Vr0Uc8u4jI/4RXw7/ANADS/8AwDj/AMKP+EV8O/8AQB0v/wAA4/8ACteijnl3Ayf+EW8Pf9AHS/8AwDj/AMKP+EX8P/8AQC0z/wABI/8ACtaijnl3FZGV/wAIv4f/AOgFpn/gJH/hR/wjGgf9APTP/ASP/CtWijml3DlXYyv+EZ0D/oB6b/4CR/4Uv/CM6D/0BNN/8BI/8K1KKOaXcXLHsZf/AAjWg/8AQE03/wABU/wo/wCEb0L/AKAum/8AgKn+FalFLmfcOSPYzP8AhG9C/wCgLp3/AICp/hR/wjmh/wDQF07/AMBU/wAK06KOZ9w5I9jM/wCEc0P/AKA2nf8AgKn+FL/wjuh/9AbT/wDwFT/CtKijmYvZw7Izf+Ed0T/oD6f/AOAqf4Uf8I9on/QH0/8A8Bk/wrSoouw9nDsjO/4R/Rf+gRp//gMn+FH/AAj+i/8AQIsP/AZP8K0aKLsPZQ/lRnf2Bo3/AECbD/wGT/Cl/sHR/wDoE2P/AIDp/hWhRSF7KH8q+4z/AOwtI/6BVj/4Dp/hS/2HpP8A0C7L/wAB0/wq/RQP2UP5UUf7E0r/AKBll/4Dr/hR/Y2lf9Ayz/78L/hV6ilZB7OHZFL+x9L/AOgbZ/8Afhf8KX+yNM/6B1p/34X/AAq5RS5Y9g9nDsU/7J03/oH2v/flf8KX+ytO/wCfC1/78r/hVuijkj2HyR7FX+zNP/58bb/v0v8AhR/Zth/z5W3/AH6X/CrVFL2UOyHyrsVv7Osf+fO3/wC/S/4Uv9n2X/Ppb/8Afsf4VYopexp/yr7gsiv9gs/+fSD/AL9il+xWn/PrD/37FT0UvYUv5V9wyD7Ha/8APtD/AN8Cl+yW3/PvF/3wKmopfV6P8q+5DuyNYIUYMkSKw7hQKkoorSMIwVoqwNt7hRRRVCOf8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOilo8d/8k88S/8AYKuv/RTUeBP+SeeGv+wVa/8AopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArKvPDGgahdvdXuh6Zc3MmN801pG7tgYGSRk8AD8K1aKabWw02tjE/4Q3wt/wBC1o//AIAxf/E0v/CHeGP+hc0j/wAAYv8A4mtqinzS7hzPuYv/AAh/hn/oXNI/8Ao//iaX/hEfDX/Qu6T/AOAUf/xNbNFHPLuF2Y//AAiXhr/oXtJ/8Ao//iaP+ET8N/8AQv6V/wCAUf8AhWxRRzy7iuZH/CKeHP8AoAaV/wCAcf8AhR/wivh3/oAaX/4Bx/4Vr0Uc8u4GR/wivh3/AKAOl/8AgHH/AIUv/CLeHv8AoA6X/wCAcf8AhWtRRzy7isjJ/wCEX8P/APQC0z/wEj/wpf8AhF/D/wD0AtM/8BI/8K1aKOaXcLIyv+EY0D/oB6Z/4CR/4Uf8IzoH/QD03/wEj/wrVoo5pdw5V2Mv/hGdB/6Amm/+Akf+FH/CNaD/ANATTf8AwFT/AArUoo5n3FyR7GX/AMI3oX/QF03/AMBU/wAKX/hG9C/6Aunf+Aqf4Vp0UuZ9w5I9jM/4RzQ/+gLp3/gKn+FH/COaH/0BtO/8BU/wrToo5n3F7OHYzf8AhHdD/wCgNp//AICp/hR/wjuif9AfT/8AwFT/AArSoouw9nDsjN/4R7RP+gPp/wD4DJ/hS/8ACP6L/wBAjT//AAGT/CtGii7D2UOyM7/hH9F/6BFh/wCAyf4Uf2Bo3/QJsP8AwGT/AArRopXF7KH8qM/+wdH/AOgTY/8AgOn+FH9haR/0CrH/AMB0/wAK0KKA9lD+VfcUP7D0n/oF2X/gOn+FL/Ymlf8AQMsv/Adf8KvUUrIfsodkUf7G0r/oGWf/AH4X/Cl/sfS/+gbZ/wDfhf8ACrtFHKuwezh2RT/sjTP+gdaf9+F/wo/snTf+gfa/9+V/wq5RS5I9h8kexU/srTv+fC1/78r/AIUv9maf/wA+Nt/36X/CrVFL2cOyDlj2Kv8AZth/z5W3/fpf8KX+zrH/AJ87f/v0v+FWaKXsqf8AKvuHZFf+z7L/AJ9Lf/v2P8KPsFn/AM+kH/fsVYopexpfyr7hkH2K0/59Yf8Av2KPsdr/AM+0P/fAqeij6vS/lX3Id2Q/ZLb/AJ94v++BTlghRgyRIrDuFAqSihUKSd1FfcHMwooorUQVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK4PwX408K2vgXw9b3HiXRoZ4tMtkkjkv4lZGESgggtkEHjFbn/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQB0FFc/8A8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVQAeO/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLWH408aeFbrwL4ht7fxLo008umXKRxx38TM7GJgAAGySTxijwX408K2vgXw9b3HiXRoZ4tMtkkjkv4lZGESgggtkEHjFAHeUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQUVz/8Awnfg/wD6GvQ//BjD/wDFUf8ACd+D/wDoa9D/APBjD/8AFUAdBRXP/wDCd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVQB0FFc//AMJ34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVAHQVz/jv/AJJ54l/7BV1/6Kaj/hO/B/8A0Neh/wDgxh/+KrD8aeNPCt14F8Q29v4l0aaeXTLlI447+JmdjEwAADZJJ4xQB//Z", + "image/png": "iVBORw0KGgoAAAANSUhEUgAABLAAAASwCAIAAABkQySYAAEAAElEQVR4Aey9B2Bc13XmPw2DQgBsYG8iKZJqlqjmoi5LsuMiO66J4yROHMex03cT/zfJZjfJJruJk82mblySeNNsxzXuTZLVJVvdsip7bwALigjMADPz/933AZePA5Ai6ZGlob6n0eV555573n3fG8yZb84tmYwPI2AEjIARMAJGwAgYASNgBIyAEXhRIpDlrmu12ovy3n3TRsAIGAEj8GJHIJt1HHyxvwd8/0bACBiBFzMCxMHci/n+fe9GwAgYASNgBIyAETACRsAIGIEXMwImhC/mp+97NwJGwAgYASNgBIyAETACRuBFjYAJ4Yv68fvmjYARMAJGwAgYASNgBIyAEXgxI1BPCP/n//yfDCSNR3t7+4oVK37hF37hkUceeU5heuaZZ7joX/3VX+kq11577Q033HCyV/zIRz6Ck/3798eGp+YnNv9BhOcLyR+kz25rBIyAETACEYHh4eE//dM/veyyy3p6eqZNm3buuee+/e1v/+Y3v9lEE+/f//73X3jhhfGOorB8+fIY6KPQ1dUVDU5B+O3f/m3cxoZcl6vH01MQ/tN/+k+xby0tLQsWLHjpS1/6v//3/z58+PApeHMTI2AEjIAROBYChSkr/v3f/x19uVzu7+/fsmXLF77whY9//ON33HHHRRddNKW9lcdCwEgeCxnrjYARMAIvZARuvfXW9773vXv37n3d6173xje+kd9G9+zZc8stt9x4442Ews9//vMLFy58Iff/+H3753/+Z+guNn/0R380NDT0J3/yJ8j5fP74rX74tYsXL/6Hf/gHGPjIyMjg4CC/Tf/hH/7hbbfdxteSQmHqLzA//E76ikbACBiBZkdg6s/TH/uxH0vf2H/9r//1yiuvfMc73vH000+n9c+d/BM/8RO5XH328hQu1yg/p3BpNXnekTzlnruhETACRuBFi8CXvvSlH/3RH339618PLYSTRBx+5Vd+5fHHH3/b295Gqurhhx+eM2dOrGou4aqrrlKHoVuHDh169atf/cLsf2dnZ7pvP/VTP/WWt7zl8ssv/9SnPvXOd77zhdln98oIGAEj0HQInBDpmjlz5vve975169b19fX9cO7wDW94A5H4B79Wo/z84D2Rhx8+ko3quf0YASNgBF4kCOzbt+/nf/7n3/SmN0EL02xQt8/A0W9961vI/FQaAdHwyG984xuwl9/7vd+L+gcffPDHf/zH16xZM2PGjKuvvvp//a//xdCbWDvloEoG5jBOEsIjM9k88cQTECFykmeccQa/Mz722GPRSRQeffRRONLZZ5/Nta677ro//uM/Hh0djbUnLky+l2ft5yte8Qr6TJqRIUUa5MmN64r3338/1Hr+/PmMJmXA7fe+970T78mUlozgXbVq1X333Uft5K6qyQMPPADsmDEI9uKLL/7Lv/zLNOzr16/nMXV3d59zzjmf/vSn1WRK5bPe+Cn3QRd1aQSMgBF4gSBwQoSQvhICKYlJlJqY9+STT5IzJOrEO9m+fTsTBvjwZa7FypUrCaj8khpro8Com1/7tV8jfvBJTZAgQnz/+9+PtRL4KP/Jn/zJtBIu+hu/8Rv8LsiH+LJly2B6N910UzQggUkQgrWiYbIHMpMZkCf72bFjxy/+4i9eeumlXP0lL3nJT//0T9ddXXd38OBBvHHXs2bNYjbjhz/84XitH1BoCJJgCPizZ89eunTpf/tv/21sbEy9Opb+WHMp+bV7yZIlavsDPtYfEBY3NwJGwAi8QBD4rd/6LYIIk9KP1R9YIqHkox/9aNqA0Yxvfetbi8Ui8Uh6CNLLX/5y6CXz8P/2b/8WdgEhJPrQNt3wWeV7770XFgTNg+P90i/9EsEXt1//+tfTDf/iL/7ikksugY+9+93v/r//9/9yrT/7sz/jU3337t1psxOUJ9/L8Rtydcgw0XzevHkIHJAxmtxzzz2vfOUr6fkHP/hBvh7wFYIb+epXvxq9MWz1H//xH7du3Ro1JyIQ5UFVlpO7yiRDvmCQ9mQKIrEbHg51v+KKK7S+ACSZn5v5usLsUAghw4geeuihKZUn0pNT68OJe7alETACRuCHhwCj8+PBdAIuHE+j8Pd///foN2/ejOaaa64577zziIgKObJhkiEUix8v/7//7/8jGhFQ4YStra00jE4Q+DWO2EDMIDb8zd/8zW/+5m/Cyli6Bhn//IwnYy5x/fXXx4Zf/OIXIT8MziGsYsl6LXzcY/+e97yHz3HMbr75Zmbr/dzP/RxKgjQyP0Oir/Pzr//6r3SSnv/yL/8yfv7zf/7Pq1evZqo6gSFeiyb8mkvnuQQ/9BLGmECCW0IdvCuaMXvh93//9+PpZOG5Q7Jarb7sZS8jIv7u7/4urJsvLurJsfT0rQ6H2FtwAA2dYnPKjzU6tGAEjIARaC4E+HjniH3mc54gxQ+CUXMiwtq1a3HyiU98Ihoz25APZyJI1CAw0Ibf4F772tdKSavJF4LJ4IooFm04JXRGPyypAtM788wzK5WKlHfeeSeTLP77f//vRIFotnPnTn6fpS1XicrJAiQ2HW0xmHwvJ9JPGhL3+Q4QLyE///RP/xQ1rB4HMSO8xp7zIyY9ZEJgtEkLv/7rv37WWWelNcgHDhzgOwPcGHlyV2+//XZgJ76nW23atIlfn/kdGaVSl1BWZPpzwQUX/N3f/d2USvk/wQeUfvTP2od03ywbASNgBJ53BPgcDke6H1PSGGacM4eez1NZwhxoBfGLsQf2BfeDj5VKpeiNERrwDaIUqTwpSRi2tbXxocwHejQj+vLJHvpxDELIxEVSjrAykoSxFQKxhyZ0IyqVx0ubpYkQvyPy2y05w4GBgdiEeepQU+IHlFJK3R2/LMa7Qy8+DJ+MDU+NEDYESc3khHWrM9yRwvmx9JilcYi3gFBHCIVnvPETfKxph5aNgBEwAs2FQIg9qTio9J0WMjnxG4GZwL7S9vxgygiatEbyf/zHf3A5OAOnJ0K0sCH1V+eHgIWTu+66S3qyjuTB4kd3NH7qqacIzXiImsnClISw7l5OpJ94nkwIJ19ao20Z0qmenDghhEMycgfc+D2UX4e3bduGB/zXdRUoRPzq7pTRvyDGgCAG1iLwbSFtMKVS/k+EEJ5sH9KXtmwEjIAReN4R4FNx6kVlPve5z1FH/1jUi49dfqrkd830ABUGUrLSFzwKMw7oE1k1Btik1ygj8/bXf/3X/PDGz5YwFsz4tY9lo//t3/6NNF3SLhQ0IYrg///9v/8XlWnhV3/1V0kP8vMbg0XTemgkwy9ZAg5KeSKrjdFJxpoyOqWjoyP6IVjSye985zv0LY4d5e5wHu8OY1KRDPiBFsaBrBBRfumMfo4lPEdIcjnFUYRPfvKT6asfS5+2OZZ8ao/1WN6sNwJGwAg0HQL8rEmfme8de86QQqhIPE0LmMXFtzUdQLWMhIRjfOhDH0obS4aukOBiZl1c1mWyTZ2GLFmdRtdipCXTKOgwP3cSodIxS/bEZQ2erGv+rKfpe3lW4+MYMCazrpYxQWhg3ZAoBIYL8U2jziZ9Cqetuy/u6Ctf+Uqc7JDuKlCw0g+DVEnPpp0gE+v5ZZnEIHNPXvWqVzFzhC8nxH31kNExk5V1Ho5zerJ94HLH8eYqI2AEjMAPH4GpCSE/GaorkDrIGMwHHsXQ/9g/wgzZNp3yux2EiiDHqI9oEAU++OBRjN0ncDKdAHaXZoPRjHkRUxJCfvK8++67+eyuY4NqGOlZ9HMsAdJIJxkjmmaDMoZMMhH/Ax/4AOxXfWMoDsGjztX555+fjjEMW60zmPL0uUCSYa6sdgAhZyYJuPHDsOLlsfRTdmxK5Sk8Vt4eU7qy0ggYASPQjAiwcAuBD8L25je/Wf2HQrDPwZT3wgATBhyqKh1coGpERnaqmNyKUTOMmWQJE1VNpkM0rGsFk6nT6INX+0Ywb5CpE1Nei1Z8qn/3u9+ta/6sp+l7kfGJ9HOy28nhnskOmJ34AnVMaiBbK88QaX7V5UhfKN1VoCDWM0yXI20TZX4wJVwyCYVJj9B1vpzw7QKZ7zNTKml4Ijd+sn2I/bFgBIyAEXiBIDA1IZz8CVjX3XR84qc+pjSQReSoM4unfAozWJ9DE82jPgosjBbltIBzhllO/n00bXMiMrMpiJ0wvSmNCZncMllK/WY5OYbRijEqp7AZ7nOBJF8FmE/CVEkGvTCThNUFSJ9qO+Bj6en/lD0hdqYBOYXHakKYBtCyETACzY4A8xpIFrHwCT+6aewJO09MvikYHYFJUwwm10LPYJVMXUuvuxbNGHfD9gmcMlmRHyKjXsJkslSXIsMsvS0T0/a4Fv3RSJw6b8S1Os0pnJ5gPyd7ntzzyZrJrdKaum0n0lWTZaDgkfFVhBVZJ9dGDY+YZef4FZj5jVB6spSMY5pSeQo3foJ9iJ2xYASMgBF4ISCQ+8E7wcgNPrJZ1wvKcayDARWLFi2CbzD8Y8orkuyaUs+vgzj/wfc/VCfjj7J112KxHDT8aiv9lBErHYDrmjfq9ASR5HLEPH7XZEgSv1uziFzcpulYesaCspT55H7GhdomV514Zya3tcYIGAEj0LwIEM7gclqqesq7ICPH6AwGizJYY0oDMk6sVvLZz352ci0f2sxj14+PzKGYHN0Y/zm51XE0jGdhviLruhF/68wIeaxQWqc8hdOG9PMUrnuyTQTFZz7zmckNGSLE7pH8xEzuVwyfcMlkELbEYELmlEqcnMKNn0gfJnfPGiNgBIzA84tAAwghTIlpDCSp6tJNujGi5u/8zu8gM1eQQaf8dDclM5lyrgWtcM6AVVoRQScjRWQ91nXrjPHDTHTihMbYpGsJoiTWSBLyW2Ba/8OXTxBJxv9oWQJ6yHjR//E//gdRn68vx9JjBhvfsGFDeiMmlAxMYmb/sW7zBDtzrObWGwEjYASaFAFSf4y/+PM//3M25tGan+kbIef2mte8BgrBfHhSc+mqtMzi1Syo9n/+z/9JK3ft2kVKikluyhyyWBoz2dLz85msGFdZSzc8vsxVIH5sQcQki2jZ29vLvkqaEhmVpyY0qp+Tr05EPoVtJyb7iRqgINDXfaOAB77rXe9ijBI/MTO3hVVe42MlMjImaEolPk/txp+1D7G3FoyAETACLxAEGkAIuRO2iyDFxxYIaboF0fov/+W/sNJMHDPzV3/1VwyGYcKetgMSBJih/+d//udjIcKiL9izWVC6FcYM9sDzHXfcwe98x2qb1jNPgIE6/CJIVIh6SBGbN7BWW13YjgbHEtiRYspxRMeyP0H9iSDJVEYmt8ScqrJ8sNlj6bk0UxmJf1pCNvYEJgkg8XSycCKdmdzKGiNgBIxAsyPABHVIAgNHmebw3ve+lzD0+c9/Ho5H/NKyKAzQ4GfE49wm4/nZdZAgyE62xDh+diTW0BbOpmWraUvOCnJImpE1z/jdU/sWapbdcTxPruI3WS5BYhOBEkYEOWTeO9Mc2LF2sv3JahrVz8nX5adegvLJJkUn+4kafkGGybPXMQNnmN7Jum78JE22FtZHEhUzBuvyrQOqTP6WxeG49M/8zM9MqcT41G78WfsQe2vBCBgBI/ACQeCEqNSz9pXfU9mSgY/1b3/72+zXRwRl0iDhjb1f/+Vf/iUSQiYK8pMqZtjzOct6M6wR+s1vfpM8FZsosLjzlBfCGwtpstlubIVb1p7mB1GcEKenbDVZSSTml0h+naUbTDAgljOxkHhPXGcfP34ynNzkOBoiPTk0Qv5xbE6h6kSQJHTxveTHfuzH+JrCb8AEP3aeYFDosfR0g+XsuHGmxNx33318/yAcsvY3GUW+qURiObm3J9KZya2sMQJGwAicBgiwUhqTCVm9kx0LGJTB2BamTMOy+JWTdNOUMwvq7poFtPHABvHkG0lSEfL4HGaTdBZHkSUjS4mA/FTH6peEJ4IdIxjZnpef/E6WFrKHEHMIyS7y6ypZL+bF8SvqH/zBH3DpuAJcXfdO/LSB/Tzxi56yJWuHEuZYV4avB6RkgR3MoYhaBZ24BtqMB/7Zn/1ZFhACH75IcK0plad848fvwynfmhsaASNgBJ5DBKAH8ZhyH8JYK+FYm9rxyUtMYhcgZv2xlDMUjpXW6tpyCodh0TZ+yGThUJYeZewNCTpSi7jlF7tjXYL0IFPAaTV9+nRGfdCKOFrnnE92YDrWPoQyJioTktVJluWEvsKL0n6OdXcEabJw0ZK05KltTB89IBzrWs+KJCtr8z2D/gAFgYd1d+T2WHrVso4rrfiewYIHfFfAmB+ViYWqPeXOqLlLI2AEjEAzIqDg2ow9d5+NgBEwAkbACPzgCBAHs/yPI0VEl0bACBgBI2AEXlQIKNfnOPiieui+WSNgBIyAEYgIEAcbM4cwerRgBIyAETACRsAIGAEjYASMgBEwAs2CgAlhszwp99MIGAEjYASMgBEwAkbACBgBI9BgBEwIGwyo3RkBI2AEjIARMAJGwAgYASNgBJoFARPCZnlS7qcRMAJGwAgYASNgBIyAETACRqDBCJgQNhhQuzMCRsAIGAEjYASMgBEwAkbACDQLAiaEzfKk3E8jYASMgBEwAkbACBgBI2AEjECDETAhbDCgdmcEjIARMAJGwAgYASNgBIyAEWgWBEwIm+VJuZ9GwAgYASNgBIyAETACRsAIGIEGI2BC2GBA7c4IGAEjYASMgBEwAkbACBgBI9AsCJgQNsuTcj+NgBEwAkbACBgBI2AEjIARMAINRsCEsMGA2p0RMAJGwAgYASNgBIyAETACRqBZEDAhbJYn5X4aASNgBIyAETACRsAIGAEjYAQajIAJYYMBtTsjYASMgBEwAkbACBgBI2AEjECzIGBC2CxPyv00AkbACBgBI2AEjIARMAJGwAg0GAETwgYDandGwAgYASNgBIyAETACRsAIGIFmQcCEsFmelPtpBIyAETACRsAIGAEjYASMgBFoMAImhA0G1O6MgBEwAkbACBgBI2AEjIARMALNgoAJYbM8KffTCBgBI2AEjIARMAJGwAgYASPQYARMCBsMqN0ZASNgBIyAETACRsAIGAEjYASaBQETwmZ5Uu6nETACRsAIGAEjYASMgBEwAkagwQiYEDYYULszAkbACBgBI2AEjIARMAJGwAg0CwImhM3ypNxPI2AEjIARMAJGwAgYASNgBIxAgxEwIWwwoHZnBIyAETACRsAIGAEjYASMgBFoFgRMCJvlSbmfRsAIGAEjYASMgBEwAkbACBiBBiNgQthgQO3OCBgBI2AEjIARMAJGwAgYASPQLAiYEDbLk3I/jYARMAJGwAgYASNgBIyAETACDUbAhLDBgNqdETACRsAIGAEjYASMgBEwAkagWRAwIWyWJ+V+GgEjYASMgBEwAkbACBgBI2AEGoyACWGDAbU7I2AEjIARMAJGwAgYASNgBIxAsyBgQtgsT8r9NAJGwAgYASNgBIyAETACRsAINBgBE8IGA2p3RsAIGAEjYASMgBEwAkbACBiBZkHAhLBZnpT7aQSMgBEwAkbACBgBI2AEjIARaDACJoQNBtTujIARMAJGwAgYASNgBIyAETACzYKACWGzPCn30wgYASNgBIyAETACRsAIGAEj0GAETAgbDKjdGQEjYASMgBEwAkbACBgBI2AEmgUBE8JmeVLupxEwAkbACBgBI2AEjIARMAJGoMEImBA2GFC7MwJGwAgYASNgBIyAETACRsAINAsCJoTN8qTcTyNgBIyAETACRsAIGAEjYASMQIMRMCFsMKB2ZwSMgBEwAkbACBgBI2AEjIARaBYETAib5Um5n0bACBgBI2AEjIARMAJGwAgYgQYjYELYYEDtzggYASNgBIyAETACRsAIGAEj0CwImBA2y5NyP42AETACRsAIGAEjYASMgBEwAg1GwISwwYDanREwAkbACBgBI2AEjIARMAJGoFkQMCFsliflfhoBI2AEjIARMAJGwAgYASNgBBqMgAlhgwG1OyNgBIyAETACRsAIGAEjYASMQLMgYELYLE/K/TQCRsAIGAEjYASMgBEwAkbACDQYARPCBgNqd0bACBgBI2AEjIARMAJGwAgYgWZBwISwWZ6U+2kEjIARMAJGwAgYASNgBIyAEWgwAiaEDQbU7oyAETACRsAIGAEjYASMgBEwAs2CgAlhszwp99MIGAEjYASMgBEwAkbACBgBI9BgBEwIGwyo3RkBI2AEjIARMAJGwAgYASNgBJoFARPCZnlS7qcRMAJGwAgYASNgBIyAETACRqDBCJgQNhhQuzMCRsAIGAEjYASMgBEwAkbACDQLAiaEzfKk3E8jYASMgBEwAkbACBgBI2AEjECDETAhbDCgdmcEjIARMAJGwAgYASNgBIyAEWgWBEwIm+VJuZ9GwAgYASNgBIyAETACRsAIGIEGI2BC2GBA7c4IGAEjYASMgBEwAkbACBgBI9AsCJgQNsuTcj+NgBEwAkbACBgBI2AEjIARMAINRsCEsMGA2p0RMAJGwAgYASNgBIyAETACRqBZEDAhbJYn5X4aASNgBIyAETACRsAIGAEjYAQajIAJYYMBtTsjYASMgBEwAkbACBgBI2AEjECzIGBC2CxPyv00AkbACBgBI2AEjIARMAJGwAg0GAETwgYDandGwAgYASNgBIyAETACRsAIGIFmQcCEsFmelPtpBIyAETACRsAIGAEjYASMgBFoMAImhA0G1O6MgBEwAkbACBgBI2AEjIARMALNgoAJYbM8KffTCBgBI2AEjIARMAJGwAgYASPQYARMCBsMqN0ZASNgBIyAETACRsAIGAEjYASaBQETwmZ5Uu6nETACRsAIGAEjYASMgBEwAkagwQiYEDYYULszAkbACBgBI2AEjIARMAJGwAg0CwImhM3ypNxPI2AEjIARMAJGwAgYASNgBIxAgxEwIWwwoHZnBIyAETACRsAIGAEjYASMgBFoFgRMCJvlSbmfRsAIGAEjYASMgBEwAkbACBiBBiNgQthgQO3OCBgBI2AEjIARMAJGwAgYASPQLAiYEDbLk3I/jYARMAJGwAgYASNgBIyAETACDUbAhLDBgNqdETACRsAIGAEjYASMgBEwAkagWRAwIWyWJ+V+GgEjYASMgBEwAkbACBgBI2AEGoyACWGDAbU7I2AEjIARMAJGwAgYASNgBIxAsyBgQtgsT8r9NAJGwAgYASNgBIyAETACRsAINBgBE8IGA2p3RsAIGAEjYASMgBEwAkbACBiBZkHAhLBZnpT7aQSMgBEwAkbACBgBI2AEjIARaDACJoQNBtTujIARMAJGwAgYASNgBIyAETACzYKACWGzPCn30wgYASNgBIyAETACRsAIGAEj0GAETAgbDKjdGQEjYASMgBEwAkbACBgBI2AEmgUBE8JmeVLupxEwAkbACBgBI2AEjIARMAJGoMEImBA2GFC7MwJGwAgYASNgBIyAETACRsAINAsCJoTN8qTcTyNgBIyAETACRsAIGAEjYASMQIMRMCFsMKB2ZwSMgBEwAkbACBgBI2AEjIARaBYETAib5Um5n0bACBgBI2AEjIARMAJGwAgYgQYjYELYYEDtzggYASNgBIyAETACRsAIGAEj0CwImBA2y5NyP42AETACRsAIGAEjYASMgBEwAg1GwISwwYDanREwAkbACBgBI2AEjIARMAJGoFkQMCFsliflfhoBI2AEjIARMAJGwAgYASNgBBqMgAlhgwG1OyNgBIyAETACRsAIGAEjYASMQLMgYELYLE/K/TQCRsAIGAEjYASMgBEwAkbACDQYARPCBgNqd0bACBgBI2AEjIARMAJGwAgYgWZBwISwWZ6U+2kEjIARMAJGwAgYASNgBIyAEWgwAiaEDQbU7oyAETACRsAIGAEjYASMgBEwAs2CgAlhszwp99MIGAEjYASMgBEwAkbACBgBI9BgBEwIGwyo3RkBI2AEjIARMAJGwAgYASNgBJoFARPCZnlS7qcRMAJGwAgYASNgBIyAETACRqDBCJgQNhhQuzMCRsAIGAEjYASMgBEwAkbACDQLAiaEzfKk3E8jYASMgBEwAkbACBgBI2AEjECDETAhbDCgdmcEjIARMAJGwAgYASNgBIyAEWgWBEwIm+VJuZ9GwAgYASNgBIyAETACRsAIGIEGI2BC2GBA7c4IGAEjYASMgBEwAkbACBgBI9AsCJgQNsuTcj+NgBEwAkbACBgBI2AEjIARMAINRsCEsMGA2p0RMAJGwAgYASNgBIyAETACRqBZEDAhbJYn5X4aASNgBIyAETACRsAIGAEjYAQajIAJYYMBtTsjYASMgBEwAkbACBgBI2AEjECzIGBC2CxPyv00AkbACBgBI2AEjIARMAJGwAg0GAETwgYDandGwAgYASNgBIyAETACRsAIGIFmQcCEsFmelPtpBIyAETACRsAIGAEjYASMgBFoMAImhA0G1O6MgBEwAkbACBgBI2AEjIARMALNgoAJYbM8KffTCBgBI2AEjIARMAJGwAgYASPQYARMCBsMqN0ZASNgBIyAETACRsAIGAEjYASaBQETwmZ5Uu6nETACRsAIGAEjYASMgBEwAkagwQiYEDYYULszAkbACBgBI2AEjIARMAJGwAg0CwImhM3ypNxPI2AEjIARMAJGwAgYASNgBIxAgxEwIWwwoHZnBIyAETACRsAIGAEjYASMgBFoFgRMCJvlSbmfRsAIGAEjYASMgBEwAkbACBiBBiNgQthgQO3OCBgBI2AEjIARMAJGwAgYASPQLAiYEDbLk3I/jYARMAJGwAgYASNgBIyAETACDUbAhLDBgNqdETACRsAIGAEjYASMgBEwAkagWRAwIWyWJ+V+GgEjYASMgBEwAkbACBgBI2AEGoyACWGDAbU7I2AEjIARMAJGwAgYASNgBIxAsyBgQtgsT8r9NAJGwAgYASNgBIyAETACRsAINBgBE8IGA2p3RsAIGAEjYASMgBEwAkbACBiBZkHAhLBZnpT7aQSMgBEwAkbACBgBI2AEjIARaDACJoQNBtTujIARMAJGwAgYASNgBIyAETACzYKACWGzPCn30wgYASNgBIyAETACRsAIGAEj0GAETAgbDKjdGQEjYASMgBEwAkbACBgBI2AEmgUBE8JmeVLupxEwAkbACBgBI2AEjIARMAJGoMEImBA2GFC7MwJGwAgYASNgBIyAETACRsAINAsCJoTN8qTcTyNgBIyAETACRsAIGAEjYASMQIMRMCFsMKB2ZwSMgBEwAkbACBgBI2AEjIARaBYETAib5Um5n0bACBgBI2AEjIARMAJGwAgYgQYjYELYYEDtzggYASNgBIyAETACRsAIGAEj0CwImBA2y5NyP42AETACRsAIGAEjYASMgBEwAg1GwISwwYDanREwAkbACBgBI2AEjIARMAJGoFkQMCFsliflfhoBI2AEjIARMAJGwAgYASNgBBqMgAlhgwG1OyNgBIyAETACRsAIGAEjYASMQLMgYELYLE/K/TQCRsAIGAEjYASMgBEwAkbACDQYARPCBgNqd0bACBgBI2AEjIARMAJGwAgYgWZBwISwWZ6U+2kEjIARMAJGwAgYASNgBIyAEWgwAiaEDQbU7oyAETACRsAIGAEjYASMgBEwAs2CgAlhszwp99MIGAEjYASMgBEwAkbACBgBI9BgBEwIGwyo3RkBI2AEjIARMAJGwAgYASNgBJoFARPCZnlS7qcRMAJGwAgYASNgBIyAETACRqDBCJgQNhhQuzMCRsAIGAEjYASMgBEwAkbACDQLAiaEzfKk3E8jYASMgBEwAkbACBgBI2AEjECDETAhbDCgdmcEjIARMAJGwAgYASNgBIyAEWgWBEwIm+VJuZ9GwAgYASNgBIyAETACRsAIGIEGI2BC2GBA7c4IGAEjYASMgBEwAkbACBgBI9AsCJgQNsuTcj+NgBEwAkbACBgBI2AEjIARMAINRsCEsMGA2p0RMAJGwAgYASNgBIyAETACRqBZEDAhbJYn5X4aASNgBIyAETACRsAIGAEjYAQajIAJYYMBtTsjYASMgBEwAkbACBgBI2AEjECzIGBC2CxPyv00AkbACBgBI2AEjIARMAJGwAg0GAETwgYDandGwAgYASNgBIyAETACRsAIGIFmQcCEsFmelPtpBIyAETACRsAIGAEjYASMgBFoMAImhA0G1O6MgBEwAkbACBgBI2AEjIARMALNgoAJYbM8KffTCBgBI2AEjIARMAJGwAgYASPQYARMCBsMqN0ZASNgBIyAETACRsAIGAEjYASaBQETwmZ5Uu6nETACRsAIGAEjYASMgBEwAkagwQiYEDYYULszAkbACBgBI2AEjIARMAJGwAg0CwImhM3ypNxPI2AEjIARMAJGwAgYASNgBIxAgxEwIWwwoHZnBIyAETACRsAIGAEjYASMgBFoFgRMCJvlSbmfRsAIGAEjYASMgBEwAkbACBiBBiNgQthgQO3OCBgBI2AEjIARMAJGwAgYASPQLAiYEDbLk3I/jYARMAJGwAgYASNgBIyAETACDUbAhLDBgNqdETACRsAIGAEjYASMgBEwAkagWRAwIWyWJ+V+GgEjYASMgBEwAkbACBgBI2AEGoyACWGDAbU7I2AEjIARMAJGwAgYASNgBIxAsyBgQtgsT8r9NAJGwAgYASNgBIyAETACRsAINBgBE8IGA2p3RsAIGAEjYASMgBEwAkbACBiBZkHAhLBZnpT7aQSMgBEwAkbACBgBI2AEjIARaDACJoQNBtTujIARMAJGwAgYASNgBIyAETACzYKACWGzPCn30wgYASNgBIyAETACRsAIGAEj0GAETAgbDKjdGQEjYASMgBEwAkbACBgBI2AEmgUBE8JmeVLupxEwAkbACBgBI2AEjIARMAJGoMEImBA2GFC7MwJGwAgYASNgBIyAETACRsAINAsCJoTN8qTcTyNgBIyAETACRsAIGAEjYASMQIMRMCFsMKB2ZwSMgBEwAkbACBgBI2AEjIARaBYETAib5Um5n0bACBgBI2AEjIARMAJGwAgYgQYjYELYYEDtzggYASNgBIyAETACRsAIGAEj0CwImBA2y5NyP42AETACRsAIGAEjYASMgBEwAg1GwISwwYDanREwAkbACBgBI2AEjIARMAJGoFkQMCFsliflfhoBI2AEjIARMAJGwAgYASNgBBqMgAlhgwG1OyNgBIyAETACRsAIGAEjYASMQLMgYELYLE/K/TQCRsAIGAEjYASMgBEwAkbACDQYARPCBgNqd0bACBgBI2AEjIARMAJGwAgYgWZBwISwWZ6U+2kEjIARMAJGwAgYASNgBIyAEWgwAiaEDQbU7oyAETACRsAIGAEjYASMgBEwAs2CgAlhszwp99MIGAEjYASMgBEwAkbACBgBI9BgBEwIGwyo3RkBI2AEjIARMAJGwAgYASNgBJoFARPCZnlS7qcRMAJGwAgYASNgBIyAETACRqDBCJgQNhhQuzMCRsAIGAEjYASMgBEwAkbACDQLAoVm6aj7+SJHYGh0QAgMlAb6Rw/VatlqLXuofLBWy+wvHaBqX6mXcldpdyZTW9y2APns7jWZbGg0szgz/JPJTC9Op+xOyq5iV6JzYQSMgBEwAkagCRBwHGyCh+QuGoHmRCB8X67xndqHEfghIhCj2tBoP5cdKB+iTNhd9mC5D6bXW+rlTYmwe2TXtvKGSi2HXMkEEshbNpetZTM1yF4uMy6gRBPezeiRs+E0nAX9+BE1iSoYL21dpbr5rQsl9LTOQ5hVnD29OLO/fHBp53Lpu1q6EbqKofRhBIzA6YRANnyoOA6eTo+0Oe7FcbA5npN7aQReBAgQBx0IXwTP+Tm+xSmjGtccGD1wqNyHcKC8FyIHP9tb2rmjtF7sbqyWg9Khr2ayFchejdHLgcIlZC+UyKJwgenVE7ykNrx5jxC/cDJO/0JtOEKrow5OqTvCDMftj7JJTnT1yfrM4tbVO0rrKOe1LtpX2kk5uxg45MzAIWf1lw8s6VwBv+0uzuieYI+dCZmcwpdVRsAIvAAQcBx8ATyEpu+C4yCP0HGw6d/HvoEXKwImhC/WJ3/s+35mdEB0Srm7wSR3h3lkd8gQPMq+8vY95ScgdXA5csyVWh5qF/J4tdxYLY8TaB61EL+Q1ksIXkjoJWk9mksTKV+ioZiKrYV8oKpE55CDJqF2ScV4cUwWR/0EM9TNpVpN5ShWT7SKilMRFreu2VF6mnJucRH9wOes4jzKGcXZ0d3SaSskKwlpDhmRsWAEnmsETAifa4Sbzr/jYPqROQ6m0bBsBE5LBEwIT8vHOvVN6fdL0bwh5uBlMoPlA4dGmXfHKM09sDua7Ss/HgnTePoukDoIXi6wPthdNcjjVUlmD5mGSuLB9wJVS7J5SQgJso4J4ofp+BWokU3SfNws+ScYTLQ7oo/GdaopjY/YnIA0ca2JW584n6IpJtSqjNVHK2P9sazU7jgXiY4RFgUauVgapSKhkTOKsw6VD9RxSGxMI9PQWTYCJ4iACeEJAnUamDkOHushToQkAldyTJxPYX90yBs3OFrpODgFblYZgRcwAiaEL+CHczJd08+ZkL2Q0MsyJQ+m1wcHO5ik8vaXt5HK41TkDcek8liUJeF1uWqYqqcZejC9MG0vGcYJAwwGGCdxIQQJCeJ4InjpPo6bJf9MGI/XJ0Qx2kZXUXOUQOZwwsdR+ilPUpbx5o4YpmqPKE9jCfYY7040Eg4Z85DQSNUumrZ0sDxAKtIEMsJl4cWMgAnh6fH0HQfDr5WTwt4kxenxtI95F46Dx4TGFUbg2AiYEB4bmxdGzZQ/Z/aPsuDKeFpvLzm9ZBoepIskHuM2R6uFXLbakhvT3DzdB1WcDo21HRjpXNK5v5U8Xy1bDsZ5Mn7YEDMSJsZJjfl7LdkKGqhbPlsrVfNheCcnIbDAJCWEs8DtwlmopUtRk6hPrghOgn8VJ9o2aXEqDU/0Ase2m0yJj22bQJTcmhCbsBRgyU1MgDlRVf/v+E3WqxtzHiPo2Z0XyyNTQfScmRjZ3TJDSg9nbQzc9vICQ8CE8AX2QOq74zhYj8jR546DR+NximeOg6cInJudFgiYED6fj1E/Zz4z2q81NvuTJVjgaQdH91Squb7R7b2jj0PkYAqQAZJ4fEE/PNYG2esolLY/M3tW21BLdowbgL8prdeZrT021MPptEJ5btsArqidlRs9UG2ZnRvrzo0dqLQ+PTzjDTO2DlQL1PVWWqfjuJbbV2nhfHq+XMqev6v89Ir2MwfHHlk9bdWy3Jcubs3j5G8PLt1amj1cKdIHKGIuU4UiciFIEf1DDl3EYygD5UlEUTvRCnRUjh/0FhNOgyb8z+m4NN4ynMpdkOIhu3h6fGGcsE1aV+b4repq5WSirKt8nk8jnhMAHkE43bPngkkSOHeWnqaMw1lXdp4TL1qXilwwbXzIazSwYAReUAiYED6Pj8Nx0HHwB3n7OQ7+IOi5rRGICJgQRigaKTAeD3Y0UB7A6eAoG+VlE7K3v6+0Fz37KGw4vGW0UmhvGenIlxmZCfcReeO7O8xq2+FZTNVrK4wt69iPBiKHn8C4MpmD5e6ewkg2P7JvZGatMNKSqyTkKrOs5fDBSpE9JbePdpxbPHygUuDaeJ2dH+urtMzMjR6stuBkdm50f7WFa6U+QzMzc2MoZyXlktZV20vrz2g98+yui2478PErZ151VrE6N7Phr/q2LGt7w739d9DbAjwwW6UzEEIsd5afXtK2etvIOtxWMrlCpiIo4Y3VTBh3OlbVYjOB0HIk98HdhNuBPSYcLyGIqcmH0tN/MTFaBbY5wXeSJsHVKRzjDlMuUuKz+IudwS5ppc4/S6sXYPXE0x+/9YnTIz1NntGR0wZK4pD7yjtgknVTIqGRMRUJhyQn4BGtDUTero6DgAnhccA55SrHQaBzHDzl989z3XAi8DkOPtdI239zIGBCeNLPie+psBnI3sGRAbJ2z1R6w2y9bKavtA9fe0s72JNgcKx930hnS67a0zbYDW3LVkdDIi3TlR07VG0pZmobDs+en4cSljsLI2J003Oj/dWWGWTzKsW5ubGWTPbp0WkzWg6rNvaSb+o9udHeSstYJrcgX65mavsrLbPzo5TwuoPVwsx8hY85TkkMHgpMLAcb7K8WOA21CTNEnp4f7a+0kDlc3nJYwqz86NKWw721l7Ie5kVdr9xb3slFt45sWNK6mpwkLDGfqbLXwrbShsXFVfAhdo+AoeUDiwufq/Qz7sfAKSBAFxmMurC45unDW5e2rGLA6lPDW9e0nfH0yBaVuilGqNKeqDl+mmQdkdEktHCcK+INpXKSfH7XMUadYiDOJvY4/jEvvydTJg3H+yPmp9Yn4nASaaRpcHUibU+mjz9U2wSLcAcToISrP3ekse7eJg/j0azI9Mo6dGyhOWQdcD49YQRMCE8YqnFDx0HHweO8ZxwHjwPOqVU5Dp4abm514giYEJ44VsHy+/tv/eSOvxmutjxTae0vTTtcLSybFubjyUtrtjJcLZDBG6q07ih1zSuUpxWGhxIqyBdWvlBTQtsOVFqY6UferzXLwM0q7A5Gx7hManlxOicfKN+MfOVQJQzX5IDmHazkMYM8YUO6b3quAp2D2lErIZZo0MdaGUhJHw4EcjjG5zX8UM2p4sCeElLHkNFEWLOdnRKK44uUbC+te3XPj+8v731o8Nsyo5Qln1OMHoT44XxX+SnyhNCzhcWzKJG3ltaXKi0LWtY8PbxlRevKdcNbaAivo2pV+xnrRzaval2OBnRUhbiq7QzKp0e2UkIdKZ8cDjKHwkxCDiF9YcyqlBLgllwUOZDJgNM4gURD36ShDDZJwjVoUgNKOaUNlid1pOyPokgp/dT+1J90XdIkdJvjWZvL7IVTJv0OvdYNHIVFI3oZ3CbzVJ/VGe/hxCZ05uzOi3jTkopkSmRsGDeKRMNekTx1zY2MBhZebAiYEJ7UE3ccdByse8OkAtZRn/0pfV2L8VPHwalxOYbWcfAYwFjdAARMCE8OxFt2/OVjg33rDm+enS/vGWvnk29O4TAJtGeqjNYMx9z8KPPx5uVHd1eKnGKAvK9SQCChxx8z8pw8BDK7t9LSw5IutUx/Nc9IS4z5ZISoBC/sUJcjl1hAPztX1icmJJD8HlUSDlSL2IjFkbtDP54hDDbhQ5hRoGgS+zBd8IyW4UOVQPloooGjlKQQ0UT2SJUIIRnCfWV2kA9b52EwIdSSDdnXwPpEAqm6sOu6hwdvQeCg7e5SqELeXX6qkK3ML55D5nDzyAZWr6kkA0e5PUaZbhrZuLz1TGRo4aaRTSQPabKibQV3v2FkM6NMz2xbjmb9yBZKDoGyqm05pPHaGa/YObJ73ciWRJld3XYGwtPDW0HyrPYzMI7sMdC7MFMxG+hfQvykgS6GczbTYO2c8CxCvrGYG8vDz3NVpkciKD9JQ3om/NUHjDnwKUElVUedp+smySnLxOXE+cS/kxpMKNQNnWGcumjw86zNJ9w8J//yBOswaexlwh2eGBs82esmzyA0ir99IJ/TdVFfQiNnTdDIsLJOcUawS1KRWqNVpy5PAwRMCE/qIToOEhkdB+s+81Mh6dnfTamAlXwGT5xP/HtMD46DdbAfE6mTqXAcPBm0TltbE8KTe7Rf3PKr3+nf1T/WNq9QIokHo+PzC2FuvryvUlSeSpwQv1BBptMxhU+MCwLGXx2kK03AdHk+4+AeYnRoRPwQEmMYS/hr1SRAjflUK1lGokjKjNwjIz9Dw4T7yYBSbRHioT5wGoaMVluwh+LOyI+Vs2uT8ZbjhhNUMKT+UIkHivJxuqf0JD1b2HrWrtLTa7uu6y3vCMryk2i4o31hz/rMvOI5W0Y2zGs5G2a4ZWTjstYz57Uuum/g9qWtZ2K8tbQBzfzWRXtLOyVLScOlrauiBtJ4WfeVO0d2bRrZTBUfiLBH2N3Go9ljHF5LjhEySYmlxqlS9fjh7We1LkPz0DO7bpx+6abDvd8Z2s0j4xV4ICQwF2TSvNlctT0/xvzMfJaHWikEAQ6ZkK5kvqcykNwjWCk+qaTzIZ7xf5LITf4N56HP/B/Ow//yREWo0z/hESfmE+epqmcxk4NUu3GPE3o8H3Uklxm/rq4auhsO1RxlPKEP/9Y7mjBM386E7jn8V90IlL5BB8/j1G5hIhUZ+gGHVHdmFnsQZhRnHiofXNq5XEpKRrSaRkY0XmiCCeFJPRHHQS2pJdAcBx0HeSecWhA5qb+7tLHjYBoNyw1BgDg4ntpqiLvT3sm0wopCZmdPkuvjZiF7umWxQVE+MoSiW/C/SP8w4w+YUwRSguT0KHtypQmaN876qIXRQc/ECZPvvIztDK3ENvtrhVn5MhMRI2kUpeHDiIwiMiVVsDu1wp5T9LKX8owwbzDQV5QsN6qGXGvzaPvi1pAPRCZpllw0UEExQJKBrAUS+Q8RcXHrWXOKi+GBKB8duinpLY0gFjVoIdMLz+v8kUeHvpXJhP7jkKVoEOB+iVzbNrJheduZ20rrdpTXoc9BrrK1baX1/Puy6Vff1397juRe0g0ShnvLO3aObljdEdjj7sAen17WuvLsaUu2ljburQSy2tUSco9bShuWFlctaJ27c3T9rrF1y4pntudHSQPCH1h5taVQqVZz7YXRrw9+B6GrNXthx8LHS9vWdix6YmQbpPFHZl26fXgvsxzp6FgY1LpySdv87wzezrWgDeuH0Y/fJcYsAHtguHPNjD2/OivYl6utfWOFnZUMePZXiyNVBvjmSsyyTELFBJ4kMvmjY+4n6nCgD4EkcJKgQuZ8XAj/1h26upxhx2loHY/kPK2gJlQnJrLnLLlAqlXK5ihtdPtCEHiCSb8b3Bf55LZPwTkJ89ibtByVUwrQyHnFxbyfqU3TSDhktA8bRY4OdLV0eyxrxMTCCwcBx0HHQb0bHQd/yH+VjoM/ZMBfbJcLXwFD+sPHCSDwpW1/c8eBOxkAiS0fhaIrGoGJhiDB13OxQUYeooBxQQJJ/UH/5B6NWJlSeShjWk9cjtpooyaU0kAFE03gfowX5ZkdnGCkGFAF/VMTFqeBNFLSB4ylxIbuqZPS6IrI6lK8LgwQszAzMMn1YcC7ZGfpqeQ0NN2VyJRCIDGoUQsPnF88W/ak0XaVniRDyNIyo9Ucq0p+NyQGV0H5VAZHyRGVOsUAASUl6UQIpJpwqioEPhaXtZ1JCpFM4/yiUo5hqRt9rwcZql7WffXu0q5Exd4amS2ljSwCFAJYhoVPc2e2rQhPMGmwfXT9FdMvv+3QvStbV+AcUrawdeGu0i6acBcr2pYjoGa5VDyvaFuJt2Roa25h5uxr5p95fcdXi4Uzs5WNvZmVg7XaE4MfLxaugk/2lh8ARtb1KWYr3bnhrmzubPYJqbU+Vc59eWDpaC0svopDmDMw8oah5A5ge7RCiNhyC7ozSg6acPsIoTknyXEULxzXTfEPnse18pUymaRI1SVvgOSc5hgedbXkPGgbckz0L7nPxOsRTUMucAJOxq94qpnDE7jCSZuIRsZmZ3adc7Dcp2xkOg8Jh8TGNDICdYKCM4QnCJTMHAf1cec46DgY/3AcByMUz53gOPjcYYtnZwhPDt6RCl/imXQXcnR8WYUK0p5lWhAOVvPKEEKuJlJ/IV/HJEDRLbJ2uhhf9EW9VCY0YDwNCKPjNKYBsRepQ0jMuGZYDAYaA8OEE8ZM4GCN09F8rYoyYRnQjOrg+BRBeltYWhgeDJnG0CV1gxKbgWoevspeEYwXRcOaooHmZWqk/pgQOK+4SANBUS5qPQsDkUDk3eUnyektKJ5Fuaf8xEs6X9VX3k73wmzIMHD0CWbizS+ex3drmpSreRga77fIBuF7kL37B24T8VNJQ1G+S7uvEQ+MDFANMZAlpzQPVyLlWN4JOeTWxB7RsCbq8rZVDwzemtRD3XMMTGX9nqVtK+nPxuHNV8+8fMcI40UDquQYr5l1GcStu2WE5CFNWAhn31jIOnbkM2e0rdw6su7MtsBON5U2vrzraoji1tKmV8684t6BO8/t7n586KG7+/dfPes19xy8t6el58xpl3znmZVXzXrlQHlve+Hsi2ZdccveT1w469WHx3rL5du6Z7y5beQzT2bOXdmx58lntpYqBdgyi+sA+NbyBsgnArjBCRHYi5LewicTEhdII33AhjtNzAIpDKdhOGuiDT9SJE2S0/CVJfnakhSciFUmXqgJzo558Bz120G0wAktwgjZeCTnOlNtShGNmlUQ5X7h9J4MZDoJ+WCyttNxukfg5KHMbV20r7STLWSw1B3NKPZ0t8w6UDo0o2XWjOJ0Hur0tq7pbZ3HceUqI1CHgOOg46Dj4PgfRSrsOQ7WfVA0/NRxsOGQ1jk8Qg/qKnw6JQLtpPuSGXdk9liUhUQf387DRu3QqyzDQccZnb6+40EcDwFaGIb6JWuZRMqn8Z+ygdFpfwh9wmipGEjd9LDATEvIIAUeWIAEUuKQ9UupaslWKPV1n4sOBrIQOo68pDA8kJBAaMPOSltga2EFzjBqMSxIw0KmYbnRcCTf/oNAii8hfjXYHcbfH7oJJU24fO/o42T/mFmX5ABrZBHDuNCEWOAZP7wWFM/BGZYLW8/dW3583+jjc1vOxQNT8OnU5pGNYnrhSuNsJQjpNCB8D7IHUQwVKfqXbhj1CJEoIvAKbZJW0otzvmz6VWGcKt1rXbintHMlK9YkUweZmnhm+3JeDw7eCtpFtogcfRJKzD1e0nUtyuTnqIU7y+sWtC3Aw8q2Fb2j23cxcrV9FeC0wsBz1Zd0r318sLZ3pHdF+2u4xL5S7/7RabsO799T2kte8ZYD99Kfhwc/hj0P5jsDdx2u7lvVcdH0wrIzWlt6WhbvLO3ePLIp1GYzsFZ29bgkIcO8VUJOtf+Ol3ZffU//HfDb87rWfqn3c+QnQRJOC5MMfLLKTo/sNMIvO4EW4odnwWRIPevwzkzuFCGfq9A9BCxVS8fQ6FSyaCLKiYeTiOG0/jgysDW8ecKRMpUirTmquSwxSoRx43AyoZJ1yuFRzX8IJ1w64EmPwvu6mY7QZ6AMKzltZEjzE89sG60W7jvUy1uFH2XKlXyJJayqOX7V4uCPl9mz/FBySff8l806hzfz3PaZy7rOWNCxpJnu2X19PhBwHHQcdBx0HHw+Pnue/ZqOg8+O0TEsTAiPAcxU6lnFeXyvogY2RakEHd+5D9UKfLsa4Ft3tiZex5cyMm+8LzkmRm8Gma9rJOUQYHpwOUhdzM6h5MsZThDgeLhKmF4gmfBGSqpY4IQmMLTOMB8P9+ELq/ghV0S/aDwTyGjS4vaxdmrjgbeQ2ISe5UdYSIYlTOkMLpRy5LvvoVqe7SKYNJj+FiwNrS7our4vmfgkh7DBBcWz5xSXJInBzGND35QehwuK5+0qPaH0oL56qooSWkUJ8+HKZPCWta1iQ4v0HEKSeyjJTGojxOQeMyQAmb4YqyB1afqHw+3ldUtJiWRr2Cwh91hciM+E6oRv9hjADEUXubVt5Q3k+mpwpZDwCrMWz2w/E5khsheyvGppJ6yJvCjUFPqOPaNSOV3Qupi0DBSR9Wx2lJ7cM/rk8rbVj0yssMqqqlwFPLl6G+OEx7YV8xmo5nndF36t99Ov7Xk7tb3lfas7z75zP4Q8mNJnXogdhSyAsOLOBd0X7Ox7mjuN9GpZ20pudjlZytIGXvnkwUAaEZa3rWCDx2umv+JbB+979YyXbxvZS6ulbfO+dvD+n537mo/t+sZ57eFr/WPD289tW0oZLpr8TADREWPEGXwgoZFhedVAIxKSwPo6vPd4LwVNwmPRCyucUIU+/KyR2Mtt0i+JiSlX4izRIk78GzrAQcMJISCWaPScJ8pQHWzqGgb1xDFRFS86XiF96uITDU7mX25PHTuZRj9sW3rImGeVDD8eZc5qpVCqQvwKI2OUjNPO83sBULAhKrAE7perTG8dYUFdlkqaxi9Z2Vp7vjwtX2rPb1g3fFsxO7anVF7XP7qg9cK1Pe+ZM23tD/uWfL0mQcBxMD4ox0HHQT5dFYeS6BND0sRZ8i9vmIl/x987joPxj+gHERwHfxD0Jrc1IZyMyfE0XfnRQ5UiEwXJ+DH8ki9VjPwUx5uZrPDJGxReB/tigh/kEA4mOhedhq/a2TAFEZvA5ZK8YqydGVasybJFRMLfwo4RHKJ8aPgQQcaAEaqxCRvKJ8NBg9mOsQ48U0V+gDlmwUmlwOWWFYa3jbXRVeo2j06bmR+bkR3rHWslwzk9G7Y03Mc+h7lRVgTlqyO0Bw9sM8jugpVaTdMC4XKsJprLBhqpKMioUV7UaqmVeS3n7Cw/BUtkqmJPy7mV8K067D8xv/WcnpYwFTBwiYSeQdvwD3cSf0OOAkpOYYMT1C4wQ8lLA3tczwsD6EiqeQ3yBrFEE5tPCCx4kyHXJ0zuH7yNJWdofln3VXf337WidcWC1gU7yixjkzmn8yIxYXq4tG313OIiNt6AImqvRV0RM0YsAI4M2Irqku5XUoVww+x3PDH40NzWxbOLc7G5YvYNGwafYOGQ/aW9Z7Sd+cTQQ0CBnp5gD1mFAfJzwI8t/OV/3/W3r+p5R19pL9dlq4Nk7GsgsfFOSV1yxfZcmXIn99g+n30vGdS6tH0+bwaoHV/uc7kab7nHh7ctbZuL8C+7v3p+x7Inks0boY5kfni9Zual3zh434/MfCnU8fuHt7921qVfO/DAq7pf9tWD979u5svp3lcPPPC6WZdQ0sN4JPSPd6jyiuNskN8+6GEok98gcA4ygUZOmCX2UH68hiiIcXRYJ2A5STNlcA1+MKXkSLXirZ0c427G/5Gxapq95Jb4i4O38wsOf9Tk90ZrBTJ+IekXyoQKVvnxKaBS4J2Qq8AAeU1rGUVmUaVp+XJbfozfKVpyYx25UTLbvIXas6MduXJHvtSSqSBw2poda8tWWWW3LbulduiLB/szM7r/X7bz+mYH0P1/LhBwHHQcdBx0HHwuPlum9Ok4OCUsDVeaEJ4EpPCQw7U8X4VJ0/FdmZZ8XyfXJ1m0Te6QZ+RH+ysBXnijvrZOD4NCWwgk3dmwvgsshYF/kDQZY4NAZo8mVPVVWrlAmBM4QQgPVEJaBrfQxcUFsnxFvnLDSzeWu3TR8M27luxhmOx8CNNjUhlb2FO7sTyNfe35Qok9PnvHitr8cF+lFT0ZBrbQQCPipJL0AjMJyZjRMe0/wQoxCFtH1i9qPXfLyNOLWs+jP9tGWJU0z4oyXGYsGb6YdCYLlWJY6dziOdtH1m0bWZeMvsuJ2mEAYSMNKHanU0olBmWDQSRFicNx0phKHgYnavXA4G1yqLxinVuZbR3Z8NKuq/eEZWYYGbtTG1fALcjCgfeTQw9xm48M3UKfWT5H22xAPRjsyi1rY0bgvbjrlUpyqksqsWH3c6gJiURevBk+s/tvVMUD210e387xjLZV7JP+zb5/f1XPj8t+w9ATjFB9avBBHvSe8tN7kjQjDXFI3ji59LqLu6/lpiDDcNQ9o2EVH5KTL51+fa1W3TPGaeYlnYvvG7qdLOL0ysh9h2+f3pJZ2b5i48hTF3Wv4I7gD4vbezaPboC2nT9t8a7yrnwus7Zz4fy2HkbGbi/tPr9j8TcO3cdFmZm6o7SH9/ZrZl0CV3ztzEtJNr521iXbhvcmzy7z/eHtMMavHHjg9TMv2UpCMvnd4dHh7RN3Gv4cOPAAM+SElBSn9IFfQOh28goDFBOb8JsITAdL9DLjvSSlTik5xp3y1zJxlghJ3YQUmFDqmFCPt5k4DRbIMo1N0rUpHxLHO8YJ0nEtJzU9ScXEDYBF+Fjg75Q/Jf4AyfKNVFooEwaYZ3hCSPphlMzm5JkCMovodhXH2gpslALr4zOi2pEfJd3XUShNz48gQAXbAuUrI3fnym3Z0bYsQ80ZI11hQaOOXLY9y2iFfCHD0AOWTWJ4+Pi91obeXxu+LpNfmW2/Idu+9iTvyeanLQJ87DgOOg7Wvb8dBwFE4QzBcbDu7XEip46DJ4LSc2pjQngS8PI9kh/Uu/OjQ2T2alWlBFEO1Fj2k13mw/c56B+Mi/TggUqR1BmJODiY1hTtq4QcHelExitShX76RJqud6xN/UhIXRbWd6ASlo2B0fF1lG+BXIuvaVyLtS7hb5vKnXyzDiuOJIeWNtWiMiKBfInfXy1yOT6j9leYeTgK95NxL8nA/GjvWCCBaLAMPa+Fi/ZWitCnOcmITfghGwaqCZ9vW0YYjbmar6oLiqtntyzaOrwemkeX2HwCG/KB81oXbxtZjytSYUkrshmSswuT3erhaZd0XSPOhkEUdIlj0T+RPWzE62iFZaR/sVUUZBltEOCQKCGEPa1zoYLLW1fOZRHRkV1byxt3jK4vkBLJjS5uCxxsQStLpDJmt/rK2e98auhBnhE0jLRnMowz8DT88F0ciojAKblBBB3izJIvSjKH+0o7aI6eEj0pR3ggv6o+NfQQp6QfeclenkkV3rz/E2F7j9KOi7tfObt1HiNIZxfncZUlbat7Wuflh6oM0OUpfH/wJp5O8g2+ygo3Z7ZDjCssTsOg0y/s+4/FbfMhWgtbF/DxyjqrhCgSRPl8pZisG4SS/Ru37V/PRh3tLWU2bLy4aznvOt42i1oDdYRjdBTKuVx17bRFCYurPTay7WcWvOaJXVu3j+y5oGPx1w/dT7fxc17bUgxe0r5EXFE5xq3DDF5NjvFP9+zLZp7z0Z3funH2xV/e/+B4VfKPnENvECCK0EHe3gjSa+Sqso4EV6oSFhScYsAp//H2Sy5CEdoGr1IlfIkFXieaTBgmf0Epq8Q+NAt/Wcm/wUFyBJ9BmDg/WkxMkiI2PKJKNVLthK/xLnLKn9s498vkGOQJb6dkjh9laazAhiXITA1NwOEuamT8oH+dLeXAAMn7FcpM4mLAJ/qY8YMQdubKnflhKF97rtSRHSPpl7wqrdkqc4j5XYpxBR3ZlkI2lyNznGXaLOSzmme13YkjdbuJqnJLpnJLrfzRTOVDzhZOgPRi/5d3teOg46DjIB8EfJg7DoKD4+DpERVMCE/uOTLaimF4jPcjx8EKLozJZCs/VnwhuReoGhPzatl9yaIvkLH9lWJfhkRfti9ZvQVayChNvt6N1aqM0gyWmcADGa45zugy1RlhW4SwXCRfkCn3V1sYyUmWr5cvdUf3lCb7K/yiz1TGkG9kyiL1fdViT7K9BA3R9CX5xqCvFDHbX2mJJDA6YzfFOQkzXFeexthLfviEOyW14Us29IMhjiJjXIvv2JRMbIOQLCmuYsuHZHZfYIawQT4WmOq2pHWN+BiUFSqIHK+FEBODsknXchUpMYv0T8JxkoexFTYxVSi34pwqYRFf7/s0ngO2mez28ga+DZ/RupI1Y5JM5mL2FyYjqq7C3BACG8nUxNY4jZlDfiDnIULnLu6+Dv3Dg7cwRzH5QKyd1XUxBLKnOO/JoQcDG2Foa9vq3tEdmJEynUg8Zrgi/bl+9k9EBgjOuiiWDD3F56LRNbS95cAnlrSRtHya7lGVDNY9iy7NLi55YODW87ouZPLhwrYFVO0e2dVX3gePmts6h0Qo1JcntaW0iY7Bsr4zcCc2HMtbV6xuP4Mmdx26e0nbArjWgtb56Fk1DhIIqZjfPvvc8pJ9ozta8pm7Bu9c1bac9Vc/3/cf3S3ZtsIomzRe3HUGDxrCuXNkN9RxxbSeQp7EVCAtC9pnQxqXts2HFoY9PWrkFbc9ums7OJBpBLQbZ1385QMP/vyiV3/3wBPL2udhtrR9Hm8eBPG7l8089yM7v/XG2Rd9cX+gzfFI4B0neIVkNGqgSfyJJOwLmkQekujMowEBbgo9B57pRRhxGVbcwUB/QyG5qYZUIQez8L4eP7hWYhc04SEmPC7RoEiUR5qHJjKYEGgQbLhiqAuWgeyG0Z5h/xVoHtwvPzLWwseFFnqB+GEg7gfrS5CstLdUWvO8RjXfj+eCwIcPP150F0b4FaMjW2YTE9gg9K8rF7J/cD8+l9qzY8UkSduRy7Rmci1ZXjxJbj/cCjQ63FDqSLPBlLpeDNnCoUym+N7c9Pdm8tPrq33+IkPAcZA/csdBx0HHwRBOkpDiOHgaBAETwpN4iHy3Izs3XM3xDVsz+mCD5AYZQQrN4OvWjEwFJkbijvVmKpmw0RysDFo4I888qxqkTsQPBshVE0YXpuStbDm8cbQDwsb6LpxSBXOj7EtSeZHUEYFE6vgUnhVWrMkq8YgxerVCIMuHHE/likswWlV6SkigZLHB3olTrbxClVb7hKXMbw0LtMg4kjfRNsoJTU60TfpI+WgVNWJlELboajKRo2oy/UOZulBoXeczttIlRB3lJ1hP2CNohh50lw9xfXixRuiu/WTwFoMtDwsbxnzCvlhVPLRMvuvvSyZG8sW+kKlem2QO0ZNEhaGRr6OhplzqM5G8Il6geUnrUOAN+odAeXbnxVTBAyP3U5UYIB44fXTwJox5f8Bk6MaSkIANtPyszotvO/BvpC6fHnoQ8sBVoIX4SRaeCeNgWSyHr/xwoa/2fZZTOnZG65kr25czT5I9Ld4w5y2PDjxyfvfa3lLv7tJOSCPEY27bHGS+1mCfUMdMMVv9Yu/nOeVgVO1ZHcsWts6HpaztvuBz+77wqnlXb96+8eIZL3mk/1H4FRMaqYJA3j5wD2mrtZ0LPt37RRpuGdqwpu2Mp0a2UF5cWLCkfd439t9/Wc9Z63ZtXtgxq9hf2VvqfaK0lZUtt43sgUAm7I2+1753eMejh3eQroI9cvtij+9d9KoP77zpDbMv3np437L2uVuG9yW9y8Anucnx04R93f/MriTPyI4dCfkJWbhQkVCgwBX10GGP4Af7jdlIQGO6XTK0NdhwUwk9C8NZOU2Y5LgXTgEWvf5OA9s86uBqIYfPe4msPnP8kFnYczSk/sL0v4QWhjtl8mdC/KqtBX5gCiM/mezHC2f0hKcwrVCC+3XmQwnZg/shBE2OWX9VhoAWw9w/Jv4x5a/Gj0zFhPvB7lp4hlnI+BTcD+d1bPCovp/ISfmj1d6P5uY8YE54Imidrja8+R0HebiKRKnw5DgYPmI4HAcBwXHQcTD8MTTVYUJ4Eo+rqzBnTsu5g6OPjWSypATVMtkTotaVq4xVM1C+GSGbF/IA7E8o2sZ3RrEy7MXuxOtQ6mul2KBs0CDIAPu0DKnTMqG4Teuj8bHsMeASyg3CABF4IYgWig0qSYgMFdR9QTt5wbIic5MQoyCnMXGnJiqP1YRa8TQ8pMeOqpVcIafia6jR5dKCnHAVVaUbUqUkIVU0kWXdtaCFgS5kWV90Q24QrrWKVBvfzvmef1//bbA78jbM3Ht48NvM3Av7MZafmtM6njk8MLqXb/N4xkPC5QLfwGBv+UkxAxZlZejpBV03YEPuVLyRHuoU/ob/p4Ye4CnDJ8eTfqUnmYHJOFXMvjd4M6u5YgzV5NLkA+GBTw89EIJsEmgPhNRlGDzZW96+q0wud10ybzPHU2OhGvbVIEP7+jlv/f7AI3BdShrxSohN6PP3Bx8JJKq0MTMQMmlf6g3UkYNMKbtxkCqk6t1L3vOx7f/wo3Pf9OjA9wLvSXJo2EB+nh58ioGpjw4+wojDewbvSppm8kO11vzY0o55tx+6550L3vhw/6MLWxeSPLxh5sugnQ8d+v68tp6XTFuMn9bCaG+5lxmMu8s7L+pc8KmEPULhzmpb9uTw1rPbl13SNZ+E4dcPPPCKnrOe2rVl4bRZLYcqe0p9sM0FbbNhidwMd6HjCxMpxAs7Fj94eMdFHYsunbYQxrj5cO8ZE+Wmob7lHXM2De2/omf1H22683dXXPmHG+989/zzPrbn8XEvyT/hHpNlcuBp7M+RLM2iNXLCBEjo3ChbNZDlC4wuDK9EDkPGIV1MuctXA79M3hbUtuQrjPZsZ4PJkJQLyJA4JaXZ1lLGnLG7tCK7EpQJBSWnx7KfrPvC+E8m9QXul6z4ggAb5DeIabmROOsP+scozxYWDg15vzx7obZm28JquUdQOXJbPyj3O+KpXqr2fzQ36wP1Wp+/aBBwHFSQInakY5CCTnwXOA46DjoOOg7GD4SmEEwIT+Ix8c1vcxhO2cbXtTG+K7LES7JSKAKTf8bCRg5hFwc8pglbvEBkbuxagYxem9rzfQ77nnyZcaRhE4jxvARpwGDDgQZ581g7JTI2Mo5N5uTLZCaxXNFyeFPgfmXkTWPts1mYNFvjKvA9yJ544H6aJ8uN0oewESLMkD0Sk9SHvhmLYumLdwxyiny4TfMrZfyIi5GVYRCbSBYrQ04fBEtOVSUKF5OH0sfLxeYSdCHVokGIl9MpJTYoYz/jtWJbNeH5cb/Iy1pX8zWdNTwv6goruDDMlfF+i4prkrmU21lGtbe0AxrGgE+tRKobgQcuKJK+C8c1s34S2sZzFIFE8+jQTdTyXR3eCLtDszfhjcE67MxxFs/x/K4byAee33kD1BEzAU7JaZCzwWzd0ANYwgz3HniCDn9/6Fta9FXGrE/z2MAjZDiZK5g4PlLABuGH3xm4AxWpwq8kaUNkeCObGSKQFhN1JAkcmiesBpZ4oLR/Rdvy3hKJuMA8yR9uK2/c0ruRDBvpRKYsMuiU5lfNuIxy18huUoi8eeB+JNzQ0DEYIA55MQx1U3lzbqC6ubx5c+/GYg7yvHNTacs1M15x26F7b5gZPHAQNX9k9kvx8GD/9+e1wx7D+FtGS+4r7Yvs8eP7vozl+tFN5yS7aJzbvvStcy8guwg3W9rWs3xaTyC6w70sgkvqjzzkZ/oeuWzWWZ/c970V0+bcc3jnXVt2kze7s5fNSDKbnunjuSfk8K7/tvKKP9x4F/L/3HzHby2/6k+33vaby6+598CTl89Zc/+hx5h1SbbwieFtUEEWdxmpFKpk+sOfSraakEP0cEIoYj4fhnpiDOXrbGEjh9EZxcMwug6m+WWrM/KHQzZSPyVksh25EVZzKdUKvOuYLsiHRsj7BbIXEoCtYcFPBoizOnCtSH4yk2MIaGsu35bJF5Lpf+Dw3JG98Dye9SBPuOej2Vk3Z4vLntXWBqcfAo6DPNMYX5AdB/Umdxx0HHQcbOoPfL7M+DhRBJLvzCE/ABvsyla6slWGg1IyfmYag0izFWYAai4fHsUJ5RrqBUnjmyTfRIOQUL5wGjhhC4QNDQLGUD5IneSkitqgp6RKZfoUGXtNNUSGDTLnUA1hfXgW94MEzilAOEMZUoUJF+J0w2gHbHBCE0aKQpAIb6JMlKJhlFFIMzeuiP5YTVRLyRFdwcqkkVJ6lYRYlPFCMqOKV5Rj39DUtYqe9dMsBlEj49jzeCEZ8L0cgZKFW2SJB76mPzjwbSZGsnYO8ypJxPGUz5v2KvKHc1vOmV1cyiklLxgaqT/eGz0aesrU0PIOlAwWfWTgZkpe1KJ5SecNuIUB8pU+kkmeBUNGoZcIYcjo0E28MfaNPvH40DcpoaC837jiuZ2vJu18XueraAshWd15Ccb7A20Lh3KDkr/S+1ntWyiWqKqXd1/FS1lEzMgfypiSCYfk9MKI04QlMuKUCZbfHbwDBqiD/CEJQ7jiwraFcMULpl/A21VVdHVu21zyioxHJcf4pd7P44eFbYgKKOGca9rPYKYi3Ob9S9/dkq3dMO8a0n3zWuesbl8W/gqS1+L2+bf338M+jRtLmxiMurG0+dP7vgSP2lPetbG8aVnHPMp3LXztedOWvGb2JZCun130I8s62N5jN+nEMzrmZrPVbx66b0HH7CdLW2CDj5e2/sPub7S3jN438HhHcXR7eXd7y9jvnHlFW8vYVfPPZIjmW1ZcQh6P/uOK8pruuTsPH7iqc/5dvesun7bgzzbffu/gnr/acut3B/YOj7Z+p3/fldOuPFTqeNec1w2Viu9b8Orh0ZYfnXFJaSz/1lkXhnVfGB0apghm8TatpdzVUgK2wbHW3cPTd4/MODQ6DR5YzI3OKxxa1brr4vbNV3asv6x9+2Xtu17Zsf2Gjm3XT9t8Tcfm89t2n13ce1Zx/5LCEDvErGypLS8UVhTalhTal7S0zS90zMy1tudayAqGP1zeKCd+5K87cdsjlsdqldLXDlxfG34kU+k/0srSiwMBx0GCVIwmeuaOg46DjoMnGwdf0b7bcfAFFTRMCE/icfAtuJxsHE+bsGBMpch3dFbvhBvsq7QhsKYL3IxvwxhQKt0nAQ08ja9ykdohxGtD9iSjhNTpFKaHkK5CVqso0EokUJwQNhjJoYRxfbKiDMbQVL2QV7UchgoqbagRpKJeomQYiDgh6BA3U+RTiT42Uaspm2Ame4wVR9VK+jRtQxNPow1KjnghZF1LZVI5vmWFNFjy4kLKDUY/kQdGxhhdyfIbfZ+KxqyVyjaMF3ZexyNe23kdP4ozHBR7hneSBmTzCU4peUEXGXSqEnLFm4GPRV4EyGtm/ySr71D2FJfA69jQmX0aeXNAHWGJvaXt0EKShJo3yCm0UFeBAV4566e5NL+58vWL16yWefDAmS3z6QnclUka7OfBxoZM+/zuwO3ifirJ+0H8oH/0FoG1VREoqYUcIkMX+xImmaaOWKr5Ty/6BWxe0X2VOB/TDjnVMad1DgLcj5LcIy9mJz468HDCFcNMRXgjMw+vnHE5RPGKGZez6ikCreCK+0v7kZ8eeOrMtuUMOoUr3j1wl14QP1gi5Vntyxh6ivzLZ7yLEZU3zLsaM/KN6EO6MlODScIhH+7//q7Sbubd7Szt+nb/vbcO3Muoy4f6H107bSH7arAWy++uejuay2avubRzwRsWvwL6N6djJivfzO2Y0VKofG//xiu75929/6mru+f+yZbb7xza8897H6ME3jsG9r5j9tryWP6XFt4A8bu067yBUtu8lrnPlNniJVseK8xpnzlSbjl/5kpqF7T1jI7lP/eKX3nn3As+87Jf+6XF13/gjHe+c8HrXtK5aH7bwOhoy7un97J20NOH535vaOmjw0u3lOcdrEzjmbZls7PzLfMLbfMSyreipX1NS+vqYtvyQvvylvb5hc7uhP6xFihzAk+O/uk5Fd+rf0PJMqEne8D66lpFHog+yjyS/rdVD/yXWnnryV7B9k2NAJ8MjoOKaCp5mgocnCoGOQ46DjoOPmsc7MkXHAdfULHAhPAkHseM4kyWkKGB0m4kA/lqDgeLLqJMGjByOQmRyGGMjFKtohmnUsYqmB6WaijjtAxdRMkVIwmE+6VJILWcqozpyjNbDpOu1IDV9aQHJ2YSihlirHimwCbyhhyjHQbI0lAS9ihR1h1SypIqNUdAr6p0vIz0jMtRS4l9bKJWsYxC5I11reJF4yUk6Lo0xyDdNn2t2D02itBWGdjrmFdcDAFj/CRDBKF/pA31CvSsloU3UkIg4YoITPyjhBOyMT0EjxLeuL207lt9/57mjbNalnJ6buereBE+V3deSsnyobBHGOCB0l4oJbvV8x7D7PHBh/kSxjaG5Co51aw2hoyS1H1Z99WQJRhgOu9Ht5UbZOwoZE/cDwon1qebSlNHNGp+oLw/0MjiXO6UA/rHD59f6v1clIM2mXbIQjUkD0Uy13SfLT00gRe5RC5E282lTfLAWjX8MsKgU7zRW15XTb+MF/yQNX6umn752u7zIXuwx1XtgT1CBZ8aeGpN+7JHBr6H/nO9X9hc3vi53v8gEbeJ3GOy4R7Jw3M6lv7kgte9atZLL515HmavmX8lI077Rg5eMG0xiTQ2zPj+ofUwzK/svptVOn9/w2cot5X2PjSyY2X37PtGdv72yishin953o307S3LXkr2be3sFQwHnds+49LWxZDAi1qW7hg4dHHL0tt3bXpF6+JffOjrEMJffPjr5dHC/1h/F4/4xjv/7hO933vrd/76L7be+qUd935o6y2HR1sHx9r+fPXoVQt+/OfnPXBe+35SiHvK3Y8eXnTvM6vuGznjiXLnnrHycG1sAjF6GpaB0UKgJ5P+SxykGNq4Q3aJONYx2Xiy5dGsL9SnNeKK0Q+bUgx9erIPa05jBBwHebgKNCodBx0HHQcdB0+Dz3wTwpN7iFrWhWGWsEHRM5EuETP5QlYVJeyOUqfUpoVkk8Dxq0tPWWdPdWSJCFFGHxODkfXJVzyNXUKAB6qWMaIxQ4gmDBadKGUQyxjwEKSUIH6VfKcPewnKbDLFis1hLLz4mqsm8ftubIKlXvAxtYoUDmPaxqtjoFaYRd44uZVslPSjrYTYRBpOEWhLqcshnNt1UVQiwMTuG7wNZkjJi1xf4GYlFildc2HXKzmlRBPmHCaLiIbVSpP0HeRNluTumHR638BtUIKxWm5+8ayYb4Qxwvo43TOyc+/IzgUtZ/Ul7DHsGFHaBeW7P8k6PjhwK/PWaDi7ZTGE8ILO63F1Qed1Z3ddzHVpQj855rcuYtNI8n4MUPxa72e4u5g2FC1MDxmNaUO1pZSN0oZf7v3sppGNX0wYYDT4mcUh6YQZDFA5Q+RknmFmTpKBvK33FqrgjfC9ewfuhPhtHtlEEyYcLmgLacl3L34P5QXT16KhRA4jH7MZNsDg37v672L5U5qwks3mkY1h0GmGmZO78HbR9PMpr5lxGa+3zXsjnO3t89+wqG0BL3yw6eK+Ui8jTikZa3rTvts3lDZ/ct+X1pU2MyMxLI6fqxbzlVfMXkPm8A9Wv4U1YN645OXFwhjED/3c9plXdM3fN3zoyq55j+zfdEXn/M9tvY+Bo7/22JdvH9x7554Nt/b3bug/cPPB3itnr77pYO//Pe/1Y6P5UI7lPnnpO8fG8l+64pde1rr4oxf87Ntnr335zLN/e+WPLy7Of+ucN+6q/XR2+k88Uvj7Vy74jTPazmSZUManHqq0Pjk8777DK+8YXrF1tLx7rDRaC4NXf6BDDO0EXaSMs+1/cIKNglmaE+o0Ni5/1EnCCMaLRHAcdBx0HOSP3XHQcXD8M/+0iIMmhCcdwUkSwm2YgMfKkD25stpDw/TiFEG8DjkyQORI50QF2ZhebWWjWnFC6dFIqVM2jo/2aiLuhxLKR5fEACmlmT5hj5l4oJorPRhsks3upUyzU+gEL+5uV+lpyAMCJV/QKXmxTzrf4DnlijKgZE1OSixlnG6S8MCwax/Ujn0LOaUWe14I6UssKC5Ck3jGf1XGuoqunr5Q7NjRrWqRZ+q+KMX9EKB/ekmmFANEUKvHBx9Cxn4y2xR1xEwCZjrQYK9N/+4fvI0XNE+7d2CpLw0Xd12LMeuLcjmIIuyRRCLRlF9VQwktzGS3lDaIPUIgoXwMJb1+9jtIQIXRqoE5hYOE1eyQuGOF0l2PDTzM/hmQQK5FSQYSA5WvS4aMspCM+BuEDQzrhoxOThuCp5KHN855K6vOXDb9yuSaGegfTI+VZjgV99OQUVgfL6iaTrkW8hu4dDGMVgWEy6ZfIeI3t3UuJPCpwacoWScGP9/rf4Tynv679II9MtCUFWhQvnLOKynfsySwxzcvejMlx5XTryBzyDDR7w18b1Hrgu8NPLq7tFvDTe/ov5uGrAXKcjWkE9nigrfKr53x0wwofenMc2GD581YxYIua2asYHohmcOLpi187ND6i6ct/Mqeu1/aNf8PNn3q4dL27xx84v7Sjmy++t2RnVfNW3XP4V1/dd6NgP6WMy7l6m9ZHsq506Zf3zl/3zP913fNU/lE73a4+meevAvqeMv2Rz6x79Fdwwd+f/1nWNvm73fc9LXdd7357vd/cP0nv7Djro0jm1h0CvpL33j6hyptm8szvzf6ju211z9VzhysjLI7fLjPhh8xjzeV59rw702lntBNbntsMlnr/+Pa0M0TLf3viwIBx0HHwfhGdxx0HHQcPA3ioAlh/Ew7UUE/jjIoVCSKb9uRg+Ei8rooS0hTO6ggbdU8skFRQVwx85ASt/3ssJ2pHWLd0UCTqltH29BAmfSakyshwEh5YcZ3TWxm5kuHqnmqKLGnPGKfD/Zz8iUsWZWUFwJpsLn50sFKftNoGyUGkaHR7Yu7wrfzeDCtjhd74nE5lMixRNB2fHVNogG12D84+G00HJxOtt9bZjHPcMA5uQqlriglMk3wr7aM4ZSeVnKVtFqVpprcIHf00q5rJBC/Ra1E/2jOpEFoG/SPF6eROv5Iz49xSpUsKXmlqSD26YZqi0YMMKYc0XPEtWognLEV11J+VTaUP7HwFym5NNcCX/JvLBszu3Ue8raRDVAsar/e92kYoAgnp6/peTvly6ZfxWjJl3VfdV5qvt+81kVUsbUG1BFyqLShqCNowN+omnLIKK04Ao1MwNKAz5MZMhqa43/3yC6IH0NGVcL9kFFSKmcIY7ysm9eVSjlCBaGFDBalOSXy53d+nvI/9v0H+UNyhjDijSOboYXhApnMyrYVb577o6vbly9uW3D19MvWJkvdnN29BlpIc7aywIaBoE8MPM10xP/+9F+35ioPDDz2VJmFZyrry5su61nDmqV/sOatHS3lH1368raJnCGEnRmG+w4fuqpr3sO9m6/pmve5Lfe/smve+x752k0D+3736Xu/1b/v9l0bb+rv3fnMwVyutqhrRmvL2IVzz7i8e96izpnMWnzF7LNu7LnwJ894/ecv/9A75r/+wukXwJB/ctH7f3PVn79pwa+xlSXTUPkLWj/8dF/trEfLc+8rdW8dGxtp+fnktljrZqplYNLTAsftJv6ZbI9G9nVpPbWIrhBi2yhgg0xDXtFSSjVPLOvJZBg4+n7nCYXQi6R0HHQcdBzk19JjTJ0IHwOOg46DzRUOTAhP5XnxlZqtI+B1YdmYDIvH5KFwyteJ4MlplCXEEipI24EUWxPTg7/xmp0rU+IWM9jgrMD32M2iBZ/ikJQS0BCTeNENCCf+N492oIzXFQulTNNRvljzOlApJK+wxox6q8yhyJVIV+RvGIiPSZA9NqJ/MkaJAD2LVK2OzmEwmS6iTNvjQRdSyancRhv8U8Vp5IG6SrqVrpK+lmqVcoxpRlFfSlKX5Me4kIgcnE2pQrFB9DA3kTfiH0SOFzaU2KtJHV2MDJBWWKbXqlErlLhVfpK2ar5u8AldOn0t2mIJ32PlUoRLu6/hRYfhe5SMTaVEzynJyb7yPtZ3gTRuL697YPDWZPsKFmJZxDDXl3ZfzYvb5JSShFvyzgnvH5oTt+pWmmHOIe86ljThxXS+n138XsoLutcubzvekFEGmsL34H4ifniOA0Q1ZPTaOa9EQ84wcMJw5Sz5PfJ+m0Y2fbv325TQP7TSsPYpmjfNfdPythWXT78CYxggEw4v6D6fFwKjTDHmXsIipSP7Noxs/sj2j60f2cJeiAwcZRApmcP5bT3Qwv+8/CeRXzv/ikK28pKZq1lmpqdtFrsj9o0cuJCcYf86iNxX99718u55H9r9LXKG9x56kpzhlpG99wzvZG3S2w/v/vCFr83mqp+/8u3w5PevvQbeeMPy89lt4uyepVDB3uGDyOF+p83ZO9J3W/+9M1u7OeXSMOp7+++8o+/mj2/96Kd2/e3N+z+5c+Spnpbzzun8kYcGbprR+s5N5dnfHZn/xNBHDlVHaDI+ODNNxlDWTQtM87c0c5MeTbRP1wbviSs5r2wM1+IQA5RAVWyCQTxkyWkUYpUEGpbX1+l8eroiQHji1hwHAYEg5TjoOJieOuE46DjYpJ/8DGTycRII8GUadgF32lPLT8+O8aUZ9kWpXJxYGe7OaDkMN2OeIUQOJcETateTK4nXQQWjHKmdLGmLgJIyCiijWexrpJcie2kSiHw0A0wahQgeDqisBMpZ+dG4KWJUShCJSstoOCX4QcbqjMXQZBDLyO4wFrUTnZMsD5Fz6nIq8YbAi2tR1tE/msdW8pxuhXHsm/yIOsqhLk2ZbgJRpAmLvvCZvq+8k6pFravhZkobJlWBASKQvotcEWoHc0OpKgQOaZR4lEYMUFUQP7VCQMMMQNqqCXpeaoKSFzZ0gECLASUkk8Qga71AAiF1WHIqNnj/wG1qqCGjMEYJMoMvwRIpHwsNAzhwPzam55381d7PJH42IPPdjlaUjDhlb0PasoIOted1X4gxK80gQzh5qz/G7vaZDONFKcMxEAooJRrSfeEkw46Fu6BwXBRqJ82B8gGo4K29IUX8hX2fp+T9uLx1JZSPNCAv0T8Gi/7D9n8gW7hp+yaao0+aZ5GTIaOPcnpn/92JMiQJGY3JyFtWppHmfUve/Xfb/t+F08+HFp4zfc2th+6d3TaLLe/7SgdYfoZs4bkdS27adyvl3+/4B5oUhqrbKpuXF+Zsra57x9w3sPfgry679jsHnrx0xup8vnLjgpdmt30X0nzt9LnkDG+YMfemzY++avqcDz9666u6577jvo8XC9VfePRjbENfHBhdP7q5bajEUFVea7sX3rL3q3MOzV1/+KFLZ13YPTRyduf5M4s99x6osOAQW1a+afH7F0xb8rktmSeGvpLNtDKV9Gn+8CtDZ7b0Lyq0FSBddbwr0K2PMuuvPjWn266jf/E01kbKFzWJw3CW9hyvKw91fVBbNaGsu0plY23oo7Xh63KzPpjJT4+2Fk5LBPiA4nPScbDu4SpUgQz6WMZwg1IByHHQcZA3g+Og42DdB8jzfuoM4Uk8gunFGXyQ8as/WTW2pIewRRqW9gJ525JsHQEJVPoOAQPsKamN7A5BylhGQZacyjitl6voJNI/SGAdD4zdQ5AM/YME4oFD65QiiCrGVCGRLAazxHA8jBHMeCmYSQ8xU7RDUBP0inmUaW4Wfco+nk5pzyXQi3liL+dqouaULLCptrqKIjF62qo5rXhhEzsWf8eNDbGMTaA6rALKXEceGbxI+UOGmzLo9NKua6jlOxBskDeA6BxOdKCJmUNp9HOpSJ00kVuiRAPHm9xK41TVVq0olWykOeyRPnxz/6fggTDASPk4TacNY8NIGpVFDGnD0M+QNoQfzm9jyOh6RpnyInNIKzhhesQpGhaq4YrfZ7JiaQMJQzTQvEAmk8wh8wzfOPctIW04fS2as5JVRmHUmogYhsqUdpIcw/5LSVsNHCUxSAoxDBadfgWMEai1O0Xi4Szon4aMsr4ossgkOUNY3+f3fQEbHRosSkmS8Mrpl0P/0J/dfRa0sK98YHX7GTJ7cuAphL/b+k/rRzY/1P/9dSNbSLSuL225eMZLNpU2vX3eG9kj8ZIZ57GnBVnEczuWUksWkQQg647uK++jZG3SQqHyVztvuq+04+4DT905sjNbqN7yzJ6rFqxk+OjfvuT1NPno2p9lFdN3rXzt63oufs38q9oLo29Z/ObW/NiarrOXd57JDvXcI7vMa/ztgrZFB0f3AJpO6d4Nc37rXWf8fblW2D/WuWNsRr7jg1vGRgaqo1V4GiyOUkfCvo6wwWNRNYyxVJ5wvGX4J1u8avwMh/IZ6Vyd51SrI6J6EjtD5nDyVdSlyi3V/o8eaWjpdETAcVBRI0YWAg0aThWq9MzRIFA6DsaI6TjoOMgfhePgCzYsOEN4co+GIWeazcUozUjnYo5OpEssLpa6QDQWtZMy2sRa9GmZU9lEgdO0hzoGiFmaBFIrKiiB2uVJ6lKcMGx4mB9VkpCSn3vhhEQ1xTbFM5rEQ/EPvQx0GpUS0vlD0S2aUxWbyK1K9MexF53T1dWckuCqCz2RLAATeWCdMa3kOV6CtvRHPVGrYzWJV5RACS8ii4g9TiBOMapBsUTwlDkk7MHfUFIq70dbDJAp0XOqUq2YNPiJXX9H27hYqPTKDaqthowiy6HWp4ENiu+hV5IQgYP8IaW4YkwbohFpVJUaYgmZYaUZ5Rvnty787sAd80oh8fjVhL9B5LhTTvnFCHr8U4ve9687P8xkv+xADWP0HKxKKhsYzu19t0AR2dQePavLMLMCAVqIzAql/7Tjo9fOfeXWHRvCAjNtYcgocRFyyIvBpby2lDZ9bMdGVl4NfpNDg0XhhFeEwaJhXCj7T5BFVC2DRWGDyGQLr+i+nMGiG7ZvVhWyhLkhNVp7/9Kf/b9b/+nV865et/Wfz52+5rZD91KrjCJUrbe8j4VnyJ0y8vvL+z/Lgj0tz1S2j21mv/stlfU/PvcN9x144jeWX3Pv/qdeOmNVLl9dOG3m9TPmsMDMDdPnsDYpq5I+dXALA1Dv2vfdB565/VBl64HKY/+249GOXPn2/fe3ZCu8DpZGqrWHL5/7Ox0t3fu2PXJwdGtHPvutXX81s7h0y+GvPDO2+bEBRpaS5swOVVvv6X9gbfefbXnmt+bm1vfUbgmpwnjAyvIr6/NysRZBPJAmlVtCLrESEryyD0ySWqoiD0w3TMuwPmxURn3iM56NZy/TfaNO/pM+MJkwW1x2xN7SaYeA4yCxgBcBRfFIpTQ87ePENTXBEjOVaI5jXxenMKaV4yDoOQ46Dk7xyeo4OAUoJ6oyITxRpGTH4C6+ZSLz1ZW83+x8mR3qOYV3RVrIaWR6yBA8Sh0ie+nyWS1pGBmghLRDXXdKEjhxzSOrm6JhIOt4PydShRpBGgeOpiMT9opzEhTtFJDQxGAmQSX6eMS2ajjZQJYxtk22p0o0LNrUkczJwTL2H0tapTujDugq8S4QYpNoTEOac6pSfeBU/VnWuvqcrotIJ5Kl1E4P7DSYGIephgiRxSFH0gjZg+lxGnOAcdKgzCJjFC2UUsa4pS3LiqKE6ZESRIDmUXKkuZ80lBDFc7suJDFIiWXdWFMZUItALcZwEs05fO2ct2nIaExCYvP04JNwP9a54W0vvVKF4ceRAeoD92OTQ1ZPQYZnYslSqOTB4gqlTw88icwu9nhQzpCN7BPeGEigphoyhIa97OGQZ3WfxUIywW9CBZUqxI70I1ckYagqkoS8uAqM9OeX/NyHt32MpWUghG9b/OY/3vDns1tnrWo7g5QjOcMn+p9mmZmb9t62uu2MT+/9In/C8OHNI5vpNntd/OjcN0NQ35SU53efDwGe395zTinJGebCurCsULpnpA9CSJ4wm8v90qNfrZBK3Fu5+/C+M6fPglguaJ/dOjwG6f3eoeGzui4ZruycWVywqGPNhv5vd7cs7B0eLVcO0snO/NiKztc92f/FNdN/dFph1vDY5msW/Oq0wowvb/+96xf8Jz5WPrP1p+/aDyf81d2lv87kaz35Fran180eNYhU7Gu8Ihn2KXmCy9XKdxxlT20df4tt+Sjr/BBLwowr5CE9ezBlGcR46Rh3YY8cNIxVw3dnTAgDKKft4TgYI0gMKxImhzlFHN4KVMVWk98ZMcZNtlfcoXm0cRx0HHQcPBJx9OekSITsODj58+XENCaEJ4bT0VYk0/jZH0IohlbHymRbx+KiUmyQ07RBVEY9tWkl+ngaG6JksiLDU1UVaSF6sT6EeKg2pgrRx+GjShjyfZT7iuRHoUsxTHJdMCM44UTRC5kQhaxT6dPxD1k9kRPZo4mMTvpYG/1gI2N5kxPK6FzcjFPM1CoKdV1SW/VEJW3TTUT/0MSryywyxhiGlZ/cX9oHiaqzZ4gpq4NyLfAUA2TxGM08RAnZE2mE/okBSkkJ9yP3SIkBZUw2qi2nImOQtzQDrMv+cQoDVN6PKX8Yq0xzP65FnjAGVP3OigbSSHNqdaChihfcT/QPfcwcwg8ZYko6kbTh9t71q7vOphYqyIYWGjWqJtvLG7b3bWC13L3lXciMUL23/45XJLyR8aXwRozxoDW7oYUaS3lr7y0r25azlz0+N/duZKoh/FDzDNFAwOCNrBlDDpC0IclDaCHEb9XETMJ7ekMakAVmKDP9mQ0jW9iukGVm3j7vDaw3w2BRWrEsDZ7ow7LWlWEvCGhfjUeWJWcIV2SgbCGX/ULf50la5ocqm8a2kzN8orzlPfNfc9e+db+3+vI7dm34hXNeecf9/3btwou/8dR3z+xeue7wyoPlPoaP0nz/yMd7ir+xbeCW4dI/Tsu9u1D71uO9Fa4xNvbtvup17bXbdw+NBS5dvber5ffbC93TcpW79vx5NZMho7is4y3bRjZ3ZN8wWvnatFy5K9sW7oIDukWGULxOOcC4bwTxLx6ST5D+Ja2OsEE5SXuLbiUQcSsbx6cycolwmso6TvStVmYrldd5JmEdeKffqeMgz5RwQxnjSF3QQU8QUYmZAorskRXaOHUcBAT9Euo46DjoOMifw/NymBCeBOxdLd2yZmhlF6mDhKSlM4QiXdAzmYnUIYvCibalZVmi4QUr4yspgszUClmre2uVGjE3iFxsCBtM+9d1KfmKy3frccpXG09g0j1e4oRTMMZksZkYvcRzFMAiI1LYwz96RT7Zp/XUchqbxC6hxDjaI8jPZPvIu8ROY7CUvfxE1hfHjupC0VhmKNUTTtWcMoZhsnx0o64Jmmigq9AftY16TjnUPQQuIVn2UDJ6BX/jBSUTo5vMABksSi5RDJBYCOXLDAa3GjK6pByWlkknG2EavDDAp7J/Gv8pRhdajl/uYYQ60kiTpD5QQQQNLk2nDSM//Frfp2u1XCR+2CNryGiYZDgw7gEnjFdJhoyGLv3bzg/LPxq1RSD/hvzTi37hX3Z+5Oqe6/5l5wZm1mE2pzj30cFx3kg+EJ+UW3o3UsWQVHKVC9sW3tN/54/OfQuJO+YoUr553pv2jezTJRg4yqXv7r+L0aTk9wCZ7Sj0c2l2IGQL57bNyQ+EnOFHtn3s+rnXbNz+Mbaj0GBReWBo6L6RXoZpPtT/KNf89z1fRs/2j1BH+rxhePOb576JtORb5/4o1PHanquze29/ycwzQ1seQfKHT97uI4/fctW0+e/73seumLHkj9b95ZzWwWnPlCvVh/uGRztzI9nqju5cbVrrlV35alvumQXT30I/9w9Vutsu3z345cUz3gRqGw989dAz9x+A61W/feOKx4bH+r+x9erlnS8ZGJ3/WP9f9mdnVmrPnF88PK3tl8OlWVSmyAaP48eR+YQTmqP+jcm6o7ThpJ7+TTIIitgcypfmh8j5645cOl11tJ9q7yW5+Ud+XDi60mfNjYDjoKIJT5Fg4TjoOOg46Dg45Wd608VBE8Ipn+PxlN25sY7c2L5KK0bwN40g5Yua2KCUInWR2skdxgiBy/HdkO+WydoSYceIZOUJ1eJNL1E+mSFvHWtneCqnmCFEYzTxKv21Qvi+SOoh8awRrbJUEzwvKwwPJN1QH2CMtOAQdaQUj6JkJ0BVUcZP/EiWYkREIGOzuHXNjtLT4cqxTSJgz7/RGHlnmVVbshd1X7svWcQSDZRDBmJTaXua70x6Mrd10YMDt9IlBIxFzKJzhHQP5UE8kKoYsNWK5rE/1CJHP5HEqgmnuIqHWkV79PESyDjBv4zFUbFUE9USMzC4f/DWS7quJQcI2YtTB8UAMUMZJyhyCodEI5/pZCOz+PYniSxVkdaLmT0Ecb9IGjVkNKYNaSJjDOrShntGdlVrubjKKDxT/un5xJDRXjTSw/RUqyGj6VbIEDYWKSVhSFutULouGXQKM4w+t5XDNEVmHia8MUw1/H87PvqmRW/9i00ffHnPK1iThrcy76f0XvYsSKPm/IZKwlA72pMtZMIhy5OyccW1c67duH3T7NbZmIWcYfty1qchc3hL762r25d9NlmZBs65sbR5UduC9cNb3jbvjUnOMGQOr5tzTaaX1XHOGr+v5B/ez9/uvTWXzf7ttn8aq+YHR1tLlUKlr3DbM/t+bt55H931xIcvfssfbv7UH6z5me8c+ubqzgvGqvNnF+eXyqOz29Z2tsze37++s+2iTDn8eMTRwTYemcyM/JhOZ4RNQTPU9eSH0CDPLb5iQrhkefebH93/gcO1wtXZD2vg6BEmpvaTyjj4M1BHPi2OnSc80rSO8nHKAdOTfvLY0bRP8cbx1OXEJhZ8snH14Vtqw49k29ceuZCl0wuBF2Ec1C+DPEZ92itM6KnyUe84GJERJo6DjoOOg80VB00I9dl1oiWcaqBaGElScJGJibxBh0SrlOjDY5q2zZjI9cHWlPTjuyacjUydzNSD6DOtRI56uY2nurTKeA/Y4zl2IBqjgVhyyl1ggCCiiNyTL0No0V8UdpXYiSBuB81b2hYGQEqg3FV+GvqHAaX0EpYkZnOLi9i5gdqHh46sh4EHGmIGX12StOot74QZyjNytGfXO/xHe7xxSk/AChuoI86Xtq2e7PyS7vFuJzsr1MTEIGAiiiJ+aXqGrKCOEIkct5Pc0Xgmk7bSxDJaaqzp7Na56e8E0V7O5R9l5Jn4YX4a9wK7e3DwVk4nM0AShuh5BGKAyAicKn+oZKNII37CgNXy+mWQRr6M1LLgA9mL3C89ZJRBpLiqSxtqLmJMMJ7btXZr7wTNS4g6G9krQxjpn7J/uILLkTBktuFVPdf/687xVozexBtKDNj4nnzd9tIGBpTCERgaStvXz3krY0phgJwy4VBDTBluyim8keTef+z8LANE4YR4UM5wS0lTDVFkVrQtZ3wpUZYLkcG7tueV2seC8aXQP2jhwoGFIPyP2/8BcvVo2N5wM61YU5QNDL838Cj0jzIZLJqZ19YDXQxOJ46b993GO42JiFKw8Mxn9n4Rz+E0W3vngtc/eOix63qu/fLOe14799La5gfWzln+89UsS8usbV+MyZaRjRdlzt/0zBc68zeWx2458ExlOFtrq355bPAr03P54vBnGDfbnrkpN5btzLdlRj5Dk5Z8Pjf2EInKhfnSnt4/fWZsw6x85syZ1w2MbB08/F8PDlVmF19+eOyOh8u1lxSrbXEyYejQ1Edt+LOqOCZ1rKN/so75QE5j0k9Cmv4dfc0wcLR8x/jqNfDGOJyVz8ZkLGut/23ZdicJj0btdDnj09hx0HHQcdBx0HHwdIqDfHHycRIIaCFEvovzipxNArQKfkVJpo54iUzJC94FG9Q1JFBGjciearFEiGUUUKbNlrUc1ikGurQMkPVKe0vL2KuJPMe2OtUg0ocGv82tcUDhKIl5CKJzKmGMSX0oIGyxRMCA5pShLmmrUmbIeKZWBtFzME2M0SCk/UP/pMQtgpzH5mol51iq273lHZFhwhvBH0qZbLBee3XPj5Nj5AXJCZecOCB14nWQt0gaReqUJBQPhOBNtMjot09GnE5pX5dahDfSEEuughOuouVbFkB3S0+zp0WyrUVN+9TD/XjpQkoPckqqUGNHNe0QvfaooAkytwNphBKCAHm/yP2gf/BDGJpSgvik9jU9b6e8sud6ThkyGtkgNhBIhmsyLRDmBhWkiminnjBkFAEGiCZcbgJALEkeUqVWcYVSNIqUCKxSQ/nGhW+ljIc4IeQQD6QNKZGpXcR+GOUNbGgRtrWY85blbSt/dvF74WnwQF60SqYdjm9oARskZ8hUQ0pmIfKswwIw2dp7lvwcrl4551owuW7OtSLhyW844frwSd4PMEMsyRnyA41KBGqvnn6ZXsi/sOTdlNfPuXZRsqIpHh7vX8cCM1/Y8R0u9P5HvvbhnU/+8cY7vzO452u77yrX8gCIi2mFhWxfcfa8P12z8KMdLTfO7P6Xadli+7S3FbveXyy8utDxtpZsIchd75+WbW3vuIFXd644Y9oN86a/f2YuU6n0l8vr5xavO2vWe6fnKis6frqYGTtQOTxWG08z0qVjHpG/wfE4VCbWJA/HW6WUQQPxi63GLU7on3HWB/eTh8gkY+u6C0W9heZHwHEwHaccBx0HHQcdB4+aUR8/5JsqDpoQxud2EgI7zotc0UaCyFWkWOijLEFZQcr4iterszz+Ka22TqwigyVXp4xNRO2wSZNGXShtqYbo1VClpheKp4mG1ZE37NGIlckAGoYyzdM4VVU6XorXpYmcZIyjGTE18sYoqAOUsVey57QuBuNKR/SMTbonyE8OPRQGrCb5RtKMEEUxRvYehCHApiByvET8VELhJhwHRifGGIMfNnphI3sZRwaoU3hj2kkdXeS6InLap16WED/xQJ1GBigeiFKpQg06FYfcXg7bDO4or+MFtxH3w5KEYSwhcpAWuN+dfTdjwJBRZHgj3E8XomTA5xltK1n9Bc4jJdSubsgo9lA4ZQLVllaYaUtDyjDbMDlQSvjirs8is58hDVmPVAlDqlCSNkR4SdJEi9OoCbiADBtaUH6p93O87h24k4Zh2iHLKbWuXNi2AAb77iXvIXPICp/Ir+gJQy4PlA+Q2Xtq8ClK1qdhcCmbGTLbkJzh5pGNeKNk1wpe18y4DFr4vqXvprx+7rWMI2X+4a7Sbkr83Nx7K+WHt3/s1kPf+fTeL9FwZ2nXkyNbr5izmhVH/2DV5Td0zfvni39qrJL7iWU3Vqq5C2deOavlwo7C3PZCwsfY+J6HW91byOazlX2Z4ZuSgBHWWQ3y8E2MRM0NfaQw8PbOltcUhj6S639Tdy57qO8D5cOfZnDpM4dvmpWvZst/0d3yIzOm/+tIrTxcG/9dCQfHPDTmk3xd8b1Hth8MWbskeXiq9G/qyx2fSVJb6Z+6obWnBQKOgwoxjoOOg46DjoNTf6g3VRzkG4uPE0Wgq9gtU4aGduUqaTaFXoxLgszSvIuUoDghVWm5rpWaUMq5aiknO1RVtMcAEiiuiBwF+VFzlbFhVGqpUi02cyDFviBdhDpRLJgYxI9SwS/yLjkhLoq2xVPMaC6ztPFkM9XKLc25ipzIUmV0pSp1jIvWGVMbexht0t7UTzSxGwgXd4ebWty2OlkyNJDPuRPbTqgtPFB8Dy4nmZwhtJABM0oVcirqqFJr1dBWp7RCwEN6UVN5RolDyZS7JiY0wtY0ZJQy8TNO2OK0Q6UKRRqVSKQkYUgiEdIIXYQTMs8QPxo4GtOG8VpQwboho1ppZtwgGzamJ0/IKRSOl4RI8BCUJ4T7xYGmGjJKaBRjRM+mDdWBwIBI7sV9KTiF1+EBEqjEYHQL/UOGN4bL9Y7vhYh8+fQrKXeVduGHJWfoKinBNd3JWNPSfiji0wMLMWDEKbRQ21qAANsbvjFsJvE91qRhZZrzuy8gZ0gnoYisUMqgU+Yf0hMyjXSRgaZ44KD2C/v+g5JrccWrZ1yGkoVw2N3+1fOvru6+MzDVCbZ887bvvaxj0e27Hxir5b6+598ztUcf2n/XgpZDm/Z9oCuX6a5+vVBZRfPxXQHhh/lF8GzKoERohwy/lYVe8p0faq8w8e+O9o63jZYeHqtsKBYXl2sbs8VfHyv9bb58ZiX/mkrlG6MZ9jbM0/aodV+CnxsYohn0HPwkCfFj9Zd01u745E0Nf8CS66aukowp3exphD8gqC/A5o6DMcoQPtIPyHHQcZD3g+Og42D8WGiuOOgMYXxwJyrwoT8zP9qfzMGLbcSydCoOFksJYoMaLIosoa45ltEPAq9o0J0PmUDVRs/UQgJlQ1WaBKYtMZArSglqGE+1VGlcFEdBjlKpP+4XexE2aWSAXlWUsC91QwIGao4SIZpxKln66EenlBwYyIagKw1lIGxJN+gAQuzGZONILNVW3dCF0Kgzsbdnd14kMzTkCUkeMhOSLCKZQ/KHcazpxd0MPkxygNlQxpmEcdJ8XeZQOUAxQPwrYYiAvRig7ImdsEo0vKiFLlLinFcc/zme/ZsYMhqXnNFUQ+hiGDJaDENGaascYySNShjCcrcnK6xiUJc2RANzE6+DH0Ia69KGcLP5bQuhWD+16BfgL6QBQw9bFykBSNhTnhCzKYeMYs9wTYZ60koLzFBCwDiFa1FqyKjShmKG6PFJzhCf6cwhxhysL8qD+O7AHQwchQSy3z2ltqyQwTVzQmqO9WkoWZ9GtI3HSucp2VICCXvgZaBpLBO0a1fMuJwX1JFTBp1SQhoXtS6c1zpn18hudrmg7bf23k75dztvun9455371t88uBfHt/Xvm9s6d6RaeNmMHxmptpw/64OQQ8Z/zuh+fy7/mkzxQppA2HiNj+GEOBXPyeTnjp+SPDx6VAmzDVtbr84Wzylms7w62n6l2PEqhLbWX8mHpWfCEZikkoEJ66sNfVj6I0NAU9xsvGrinyPDRyc0p/6v+qD2XDF1CgemV2xSf+rO3fIFjACfvY6DPJ8YgxBiZHEcBBnHQUAgkDkOOg42Sxx0hvCkQy6kopgNuImYiWJFtlZ3Ku8xJRhpYRTS7E4+8YAyOpSyzkw2VE0mgbpibK7+SEkJsRxIVhalYVQiyCFJwo6W8zjlHqf8EZSYp9p0W4KfCButeMUqjDlVmEzrpVStZJrEyyHjUPZRQClN2ie08FjG6EUdEWhCqSPKEhhEil5+EGLHIIo37f8kNggoe4rzwio4EyNLY04v0rmYIURAqcVmcBinGk5c/ahFTZUwpIlq2dUQZqhfWHHCK7YiqGDGCqUXJyuUSk/JSjNijMjkDJU2pFTOUBs6kTZEc173hY8PPDS3dbEShiphg/BAXvBAPMD0GD7KJEOYITK1uhC164aePKONjekDKUKvzKHWF8WSU2Je3M6e3sIYNaAUGfLGOjHie+Opv9J46k+0UHlCSlKCL519mRgjF+orBf6mhmQX1Rkc8oLdMWQUqsk4UkqWn3l5z2UMKD1Q2o/+6YEnKckWUmIQGg6QLdy4oHXB1vKGV3SHZCNpRi707sXvubXv2yxOw/hSSB1ZxEAeM5nb+r4NmfxyX0hUVgdyLGq6p7yTmcGMMmXxnoUti6rV7Hmda2rVB9fOOeNnsrWzZi4b68sxyHZ2y/mHx/aNMp9w6KbuXKaj8vXM0K0hWSfCVmHtzVs4DXmzSj/DOLOd14dQMfx7teFkUdCwqcNnszArKOLw74XOiNdVNrZWv53h1QozDxP/wu4RkUamM4GhOjmoPQYnjGvPTJj+YP+mL5TuieThq7xJ/Q+G7wu0NR+MjoPpZ+M46DjoOOg4GD4TmjYOmhCmP9JPQobj9U3sPEGzyNDE3NBIQM/AMKUEVVIVT0UL1ZZSlxeXk4foltM6M2nSZpEEoiRzKOKHzBGr1Cspo/N4abqqqIZBpEnIok98AxBlkuZYlthgkCZ4Mb+nKmo56swif5vsNl60zidOTsQ42kjAW+yG7iveXfQPUZRSjBFymHQ5FFocFVd0+EIG0JZ2ApqOGA7jpEGl/lRigwE0j1L2x2GAGPzYwl/+1K6/1XBTTiGZucGaGCCn8EDKuEshDDAOGdXGhtRyoFTaMMw5zLJ9yK0otSCEeGCkguiPGjI6wQ/heyKNGKiJspGcUkVJnpAyHmKSsMEw0HRizVJkJhbCG1/efTWW5Akhe5H4ocFgQWkhwl9v/pNwmjRECfcTaVSptlBE9jNUcygfSg7oH+U4/WPNz9LGN8x5C6eUpARDlnKArs7FnlIadq24LZmg+E87P5r4CCTz3oE7KHU5SCN6Bot+b+CRtV1r2ZOQbOEDhx67YNrae/qeXnegP5PN/eIjX4M6fq7/wRltLTfvva2UXbeovKhUaymzxE9+4dhoTVtBKClXG7olO/0zjO1kKepwxcAPH8lU+gJF7HxfpvLWhCK+L1O+ivQamlqYdhgW8ExGhGojeAaRju9GWAt8L7z7svnrJ3M/7f0QrjL5OAZRnGyY1oShL8nyoWnlUZHvqIqJExKhPk5fBBwHY3ypi5h1AY63gOOg/g4cBx0HHQdfmDFhfADSC7NzL8xesQkhO4lpyCjfBWFTx6Jq9B8CFumfbkckUCXRNJIxGVPKm5ibSl1CzVWin2wWPSDEBGD0IIEqXTGWEqTHhqhWx5eoigeUCRmyhM1ky8ivJMRWGOsVDeSEU2IkVQixSQyrUamL0gTj6FPCsYzTntUq3WFdi5JYjh9KXlJyGnuLoKtQpdq0GVVkDklVaXEaypg5hAHqJe6nsaATrkIOUINF6waawgCxeVXPj9OK5hsGn4hMEhopknlT3ycZhMnapMnKmTWmC0L5KNNDRsMGFcmhGYaMMo3JQxmzgAoJw1Am682I10H2Jg8ZFV3E2TsXvZ+S5GFwHKYXJkxp0pBRKJ/yiviEy/FzKeYaXwqFgxNqcsXjAw8n2wyGmYSwPuUAg+dMRsNENWSUHe2ljCX2GNOEhUnZ1uKxwUe2lzeELRkzYRlSzOJgUTVRMhCZLCWlBpfCEqGLkTqix4wX1BEeqJLHGl7hXjPfH3wkKR9mWZrP9oZVSf9h9zeyubCiKeXvr7zi2o4F/3P1W0rVwpsWhXJR+/msOLpsxtuntZ1TYWnQo0eEBo5HSq38xBGh/DB94FSJuyCU7wg8sPzEEbqFnBxhnRhRrPx1lVrlcLX8TLU0CmOMi4jKLjj8vQmxMf+emkMymePstzG9sJcXCgKOgwSFdFjhwShMxFJCfGAxskQDNeHUcTCOiHEc5F3hOOg4GD83fpiCM4Qnhzbf0fnqP8bP/wnZ41u1hFiKX0XCxml3fmzraLtGjUYeKCG2koBxpG1o4mlaKc/phjKjRCm95NiH2GTcJvT9yJF2Li33yKdzwkbYbHA1SoiQaJUIGN+JWZaTY1HrGgZSXth13b7SDpRMu5rTuhiZNFoyJS+Y7So/FSbmlZ5eWFzD11S5TS6UJUASC3mJeqXCZEAVYxZ3URUdkJDox8d2IsecHgacRmP6WeeZ03gXdd4iY6R57INkWTJwNOYM1QE10bBSNLTC/pLusOM8bwlGaTJYlM7LOE41jDFPqCoHCAMkVYilGGCdsR6E7JGVMKQkYRi9acXRyPrikFFYIvJPLPzFT+z6O600gybOM4QcPj7E2jMhbbi4GJ4yLG7KIaNUMRQTcshAU2RsKDmyg+GJ06puyKhqY8lAU8jhv+4cn+SGrNQfzxdeSuJO6TjShjSZVZyNgdqywAxCerCoGmLA+FJGn1LSnIZgzmb07FGhIaNHDRZNfMEAY95Pl2NBmtt6bxGNTNNFJR7hmeqDOgNKtOJVqeW2D+/5nZU//qUd975l2Usf3rclMWM3woMXtC25a993Ryotm5/5/nC12NW2jDz9wIHa6OHfZZXRMCJUB0yPxGAlSWymhLCPX9jKb2NYdQYb0KlM/PxR2ai/hzBSVOMz89eNjH0jU/zFZ0b+slyrtQ7/RUft19vDviP09GQOvHGcUsLwZC5j29MNAT6RHAcdBx0H+cN2HHQcPG0+300IT+VRkiE8WAkbu7PloAgVsoTIr6IAG+QakQrG64kiRvKGnibRT3rJ0NgkXkJmzOlXVSBj2Ux3sq7MrLAXYgt6CZSJTZYvzfq2yE6Jicze9MEMmXLcYW6so3AuaZFoTBVcjtifWFKMk0CoYDiJR1htJbA+qCC68K00WztCF5PNG+J3VcySqqdEF5G5nFglws7yU7hCYIt5FkRBeCTZ436yMRfBQF1QZEKOOcPI7iJpxAalTkX8ZBwtb5j9DhG/OsaogaOY8dLl1Dy60lRDqmBc3B323Cwr06Ch22CyuAjvDauMkug7PgOkiegfZhjHIaOacKjZicoxYknCkJJc1aKEt8P6GERaN2R03eATGjiKZVyTBjnyRmrnFRfyFmJsJPp4iB9SomGDCkoNGa1bm5SkXzZbTbciScgL8qaRn5LTkyuQNR0RJgalhNEF1jcQ5t+T+guuBsIgUpEx9q9HAU8TJWNfChFF1WKmsaZqyBsPLkcTHL5x7ltJJ76ka+2jg49c03Md9C8OFiVVuLt3F25jnhC6GEkjz47tEKnV0FYx1YQ0hvc1y8x8bc9dLOX5G09+oTxWKJfzzCrM7x19aHTPmV2zh6stXYWFu0eK6w/e0p6rjdQyuRmfy/S/PYwI5Un13xIGglKywEylr1ZhMuERIckZQgtZbGZllqVoiqtq/cmG7/wxsYgo0xLYSSKsTPPbtf4/LtXG8i0XDZVeP1z5WjFbGz78F23Z7PR8a5EtLpK/Py5Xf7DiC4STEacxeVihJwwEDW/U5+7IFpc/d87t+XlEwHHQcZC3n+MgIDgOOg4e/6O4KeKgh4we/yFOUTsN7lcJRFr0DCrFS3YS0qVs4H4ygIPpxSkCegxkI4Mop1eLkc2hZL97CB4vvrPClPgyzUuDV6F528ba+Vovmif/KpMrivlTW1BndHVddFkh5i9YWXE8rxXYXfKSjSIfZUx8ReHhwVuiZQyQEDw1pIyW0Sz6jG6hdhAnTsW1MMBtnVn0iTH9DAgkOREE8pDJ77W1JW2reaFhMwnAYRt6ORGXwzkkUNSOMvI6bJAhcnWULzJGGRD5MOMFY5RblaKRGMsepZxjjwypC/3MZPaX99IfcoBTDhmFAcqYEk6otCGyZlzQCiUUkcRgzA3SBOUlXWHvdXgdxpRTDhkVFaQW0ogZOcxYargp/JCE4bK2M7+5/1MQP21OOPWQUVomB3MONdB0WbKl4fbyOtY1DRQ9kPkjQ0bfsPBtEy3GJxwywxCWSF6RFyM/GUFK/+F7yO9KhomS+oPvhQTgxEEtIjQPNggJTC9MKsKm8aVX94R3HaRRNBKZrCbI04pH8NW+z0IXKWHpeudcNv1KXgwTZbqgSj0pXRY+CRHlWrxY2pRhrnceuputCJcwKDdb+6m55797/rkfXvs6diP8+bOuH6vkr5x36Vg1N6PYM71wSeCOmQxTCEuHwxjRZEToTWjC6jLJGjNhVZjkNAphgCjJQDYMRMj3ZIvkGLVx/MawpUQYaIp+ZlBOHPyWUyi8lj9qPiJGMrWBanm4Ov5RM2Ey/i/ET2xwfMTpRPURcjihOel/lWacaBYudPRRG77/aIXPTgcEHAf1FGN0cxx0HOSXU8dBx0E+GZo0DoonnA7x6Yd2D4erhZZsdUZubF8lz0UjhZMMG5RGJafjw70mth+MfAx7WFm0F42M3qIeboMlrcT0VKLhEKmLVWjoVUL5xpJ6DAqQPRijjGUvD+i3joWBrFQhcDld8eAEDUMPPYPCRY6HhuDHN37pqUKjI22Ztqc28jq5ig7TZlLKCU1UFXKGSQJQEVeXQ5Ygt9Fb2lg9DB0jOzeRtEw4QI1Txq+e3Xnxzfs/QROQJSE5uzhPyUORPdFFWsdT/QKKJuYYg/NUNhLLSC8RIl1U3lL+wzDaJH1KQ+x5rLiFQGplmsAAkwGZ1MIANR0RZZyXiF5DRhHgk2KJyDrUBFLHqVJ/MQGYHjJKLeNLoYVHVpqZlDbUhhZgt2dilVFaTTlkdPzSqYGm53ZdCAGD7IWfKiZ26vvSrs/A7tiXAntyhsrsISMwvRA9JXomGdJwfzksE1q3viiWShgiiBnCFWFr0L9/2bmBUaahh+X91K4bfJJyPM2Y9E8N1UplOu+nCzEXEds9yX4Y2CftQvegqciRW7In4Vvm/ejekb5SObOsbd7cwsw7926ojYbf1D765M0vn7bgXzZ9jTmEn9z5t3OLAx2HxmYWRrtqhRI+Rr/env/l8R0ISdBVduZmfYDFRWtDnw7C8CPQRVKCtfJKXAWB32eYZMhiMwzmZL2ZMG8w7FWYMMlPw+swy2eypeFP1ypf5orVTIFU5OFqpZqr5jJjbZkWFjzFJn0k8xKTNU7T2h9EnhhuGlKOjEKYGHd6ZKTrhPNa+eFs5vqJM/97miDgOKiApXikhyqN4pfK+LAdBx0HiSmOg46D8TPhBSg4Q3gqD4XveOTrYktxOZWRB6qWUwiYOFjkYzqNBpGMoZETBAajkgmMbBCNyBulXmoeXSXKcR5IVddEThKyl7aJfsQGqeKV7nkMYzEdR5MpfwRFL+O0JUoOYiTxL0bKGCCjn2gm41giRG9ig2gQJONzMg/EQDaUsfPpDshh7ANmTw49qNPe8g6+ON9y4BMkGIEaDpPMeFx9Q887gtMwiXER/xDJKGF6lHA5WJyYnhgjpcwoeXGKWfqIi5SqSmnDaEYHyKcxKJfOkD+EAYoEivKRXYQW8uIUnxoyKplTJQyxhyJy+tKua/KZKiVM5tKua0gDwv3SQ0ZhiXppvRl1UmlDyitn34BmYiHTGhvTkwCUDT98Km3IKWlDVpeh1GhS0nGyoWRZGkgdAv6VPGQPQ3YyRGaYKOGQdWVkrJVmkFFKc0ffzZCxr/V+hjwhmUPlDKFw8DHoJWWSMKzFy2EM/aOtRpBqsRlO0WtNmvTKNJBAcn0q03k/Uo7oRQK5li7HtWiLTCdjP/GM5pH+R7cP7/3qwQc+1/fIn2y5HcK4cWg/VVf0rL5rYO/rF17OhhPvWfZ7pWpx7ex3r5n53sO1lmLrRWP5V2fCUM+JjQdpUHckHO+ILr/yCK2SPLEyDXQx2cs+w4r/3V3vbyvc2N7ymp629/PuHctkh6v5w7XaSJUfoQKVPeqYIGxHKU/yJP7qGdawCWQ1MNiQY0w7J41ZdzD1Uauq1ul92uQIOA7qASqaxMgVn2o6DKGMMchxEDQcBwEhxhfHwfBX4zgYUHg+jyOs5vnsRVNde0X7qm0jT0Oi9lZDhhBBJLCOCsYqETmxMkpOJSNABmgVG0oQCaS5zCTEVmk9ct1BSlCa7WPt6ap4dSljl+SW64oTqlTcgnoh8CKqRWpHc2lko1PYmhJ3abNoSZhMUzs1rLPEWGbUYlxXy6laxbZpn7o0HjiwjMay4RSHaoileoJSxnVuY5w+wNjOtjDWtHc0TGKEMSIngwlrbPIuBqhMoJghtSgloxd1lJJSCUZKiKWq4I0aXEqrOMoUS1KIl3S/EjYVejiRYYP+ie9BC2GAmJEqREaALsIGNc8QDXQxMzhOGrlTts4T/cNSCUNYHwNEGT6qzQkpSRhqpRkM7twfRpxqKwvZM4JUBE+bE5IA1BaFWl1G21Tc2Xcz5BDGSAnxo+RycEVk7CmhcOPr0CR3RBU+j9qXYoAWITSSJ9RUw6t6rocfBm1yQE1pwmt7acO/7QwZvNwAg6XDQcn8wH/euUHZQihfjKxJ/ThdZMwnp+n8ZMz7QQihf0oVqgklo1gp4aXSAAukcUlx1cK2RWzaMTe/uFrLzc3N2/nMwbWzVj68dwvvjcs75/eOHFzTdsZ9++/pKZ73VP9tuer67sKrdh786ZnF17cPfqjALoKBPoWB0NU9DP4M+wRWe8MaMyH1x0jRcSFqQiYwGSZ6XZhAmBzZ9g+GEZiVW1oKry4N35SvfLklf2NpbMOMtl+c1rJo9+Dv5GrV6RkYIaMXcvnsKf3el3RMl6srI03VMNfQtxM5QpJz+YkY2qaJEHAcjLGGp0Yo4SPXcdBx0HHQcXDqj/FmiIMmhFM/u2Np+XbOGic9+TAyUzZiU+JyaKIgOdLF6DDNCaWUjfwo8SjCRq2Mo8BpurnoHBr50WBRSvVtYvhoGBTKgVl6+CgaNRcJTEzGC/ElIpwEaXVax8QIgbKJljKjjK1imEQzpZmMMZMBp3KCsariRUXY5EShVwnD2BBBxjRU3xCwlFx39bRbDHQtuUpbyielephMSqxJRskxmQFqcRqqtNiMhspA/AiWdQyQtvqhlFpYougiDA2ew2tp22qtTIMrKMfOCR7IKe9DaCEMcHE5MENOlVeMpBElMglDLU7DVeCZ/KIfZhgOUhm4n5YnrVtpRkNGMaYblJAxJkWEBsl8PMq4ymiiG1+blJVmIGzwRg0ZhSuKGWITGSMyPDPRZ5VdhATCDKF//xpGfvZELic2GPlY1CPQRKNMWXgmzOPP1DTKlGwhY03J/nGVY9E/DRbFIArI5P3USnlCNMnlsuiRuRzr32B/Xi0IFRAUFa1l7updx/b0H9oaflxo2z/a2jLWur+8aWzbqq7ZW4fXQbY78vP2Df3Rkhl/NlK6vdr+tmz2bfCosC1h2I+ezQafYKRlEFLbD+Iq2/m6av9HGVeTKZ7D8jMsJJMoWYomTCOEDTICE4HN64vFq4rZLH/e1cpXc9nX9ZX+Jp/NTstmqrDO2mhrpoXOFk6cE6Z4IJnAqecWniADpMfpIyymeiibn57WWW5qBBwHYwiI8YUHmg4cyDGmKB4pZh3LTMZyq7ZoEKKfdMCKThwHHQcdBx0HGxVNTAhPGsn2ZJ7erDwZwnpOKGoniiVmSCnexWVE80TqOOUbf7ThNLTKZmZmx/OHaNLcT6eUcnIsnxgkNmPJv3whPOpgmGg8jx5ERNUTahV+YhyK0Ygq4tOzpg1jzJscJqNnRT6V0UzBL3YAIca/GHqjEHmgeoXnNA+MTuSTWjQcOqVUJ6M31WJWZxk7gIFuXL3llGdH5pCfBvBGLhH9gwPfViYQysfiMfLJJHuEOGQUOc0ARRfJK8aBppPpIhq5wobuwaMeChe6TkrtUQEVVMJQtJCEIbUkFbX/IYvTUEuH5xcXiTRSqzQgnBCZEvaCRmlDGKPShlTB38ROWWmGUyUAYXQxVRgTgDBACCH5Q5mhR9BBn+GQWpsUszjJULXQsKcHFyGP70uRWl9U40txq6Si2COW4oeBvyWpRUaZouQGGWuqBOONc94Cx7u65/rb+26mfHrwSdYX/X7Y/HCXLhoWNU384ETewnjUgQwlqULRRaIsNpBA5lKynAyJQVZRWze89XClZbDcdnbL8lqGH2Ky71t4DmaLZ3Tfvf+pd624/vfW/TUZ0ScPf3NRx5rRSt9gS9hEnr7x0oaEgc5xhCmCO/k3rCyqTSYSIfDDhDiF2uFgQBYuoWfJ7oWkECthzdVw5K/LFc8pZnPDfCa0/xr+Z7f/ycDQb7YXbhytfIX6sUyJyYSFbDGxnlRMzACMFYxErSXr4pJ+DHMCG3sM353RGjmNdWtvzx8CjoN18YVHURdi0MQAh6z4QhnN1CRtpioZIyPEMBQDVhQcB8GHw3EQEBwHHQeTv4YfqDAhPGn4mKhTyFYnsnCjfE2ETUVqJ06YdprmdZKpRVCTtP3BSkvdeFHM0q7UMHI5nUYDccXkNFDB2DYKKNNtJYu+qht0KQakGIei/xjJpJnSklglvYJf2nJKh9EsOpcZTnAVa+VHNsjUUupU7C7JwgX72CvJnEYWx1oyKHUqD3Irsod8/eyfYHohgnxGYfJNqXn6jrSGisaCKtFHc9G5NKmjKs4/hDeqKo4apVYaBNFFHOInNukt72RRHNKGmPFWIX0HEWU9VVgTrC8OGVWApOTXU154oMSAFwSPfe1JG1Ir57xX0psTpoeMwhVJEtJc2xhqiCngHH/IqBgjg0jTQ0ahgnGGIQ5xu728XoyOU46fWvQ+OGE6cyi92CDysUaZqiFrmf7N5j9WE0oIEmlGyvT+h6QTqcKhsn8wPQaOwhXZUgIeiJISJHmls4XwzEAas+yauHasllvbfcHoaGHX4QNzu+fxt58Zy92xZwOEEM//vOlrZ7cvvWP/zbw9/mPnB+YW+mcUDvcOfmV6fnTkMCu95MKg0OQIfC/hgZphjxyEsMToTWE/eqrYeQINxmQF2ac+bFEY2oaGpOmUzeu/pTXb0lr9SqVUayncmK1umJknR5hhGiE/CDFalAVmWmt8WIWR7fXHpA0nxkeBym6in/WtTvm8GGizj9MJAcdBPU3+2AkryOlwMDlkYCDLtJk80FweOEVImyE7DhINHQcdBx0H9XHxnJYmhKcAb7Y7Vxmq8Y0rkC5SB0eRuqkWDhUlo0zzMTExXV4e2FeQ76yYyV7GlLJRc11UVWmHMlPDY9nEIaPROPZcnVFJEMJDpEnIk8OVbFRiEI9oSZXkuvinJnVm0Q8C8Q9vGCDzQpCMMrqCk8hG7A6ZVUMpOWRMGY0VUHGltWTqflWliSxlEK/FabSUEk1cnhT+iTKyUJzA0JQwhKet7bqOPB4awpioXXoL+9DLJFWoEjNedQyQKtHFunmJWFIVPdMB+CG0h2jx4MCtSdX4PEPJlOJ+JA81ERFNPAS1GGMlMIiQEqwbMqq0oZrAGxHYyF6n2aFxfqUEIDwqDhnFYPKQUTTKMVILV6Sc15rsf5jk4g6U+5SsQw/3o6wbMso9olSeUKNMV3SuyvSyvmhomF7LFJ4ZU39KM5Lu086HeIhHnJqYpn/UxgspWzivtEikEQ+LWhfsGenbMHCATSb+se8p9iEcLRdu6J5757713xnZ/bOzzv724N3vmfPqDYfvv37eb7dB0Q792uKZHzv8zKc7ut+frexn+dCQiGPIaNh+kNVEPxtWmqnsDOuIQpkQkpGimfIdYR/CiUOryLBpIYpgwHhOKGLxvaxQWtyzqrvlVdWxb+Zbzhwd2zBcqx0uf7lWuLE69mXeFbwV2zJMdM7lpn82jDg9+ph6UOjRNg07O3q3jIa5taPnEwHHwRCkeAIq04+C6KBTqiTHeBT1CHVm0Q+C46CAchx0HNQ7wXFQODynpQnhycFLEqb4TIWNHGbny2X2AKyF7emV64uOOI1K5Mi+MEhTuHSr8SaVFhJNfJPjUKtEDEW6oU4pZRP9I/DCUrRQZdpGK4vKJjidGLOaFpAVpWJw4vRYUU3GMapFS7WNHjDjiPwKWQ7TZqJ20QM2khFiKFWMRBOFyFrliiaR3WHGISfxcjrlWjSMbiOvw0yWsaGEaElwwoCrqwORhdaZ8QTJxelHTaq0OxOUjxenInUI6Rwgp1NGPtnTBGNYJSXe0p4xUIJRe+vtGFkXh4xCAiF76f0MGTKKkiZanCZxHtKGDC5Fc17XhSQPJw8ZxawubZgeYgpZgnTNPoEho5Ex4jDmGCGTEDAWIyWjyEotVNVxOTTQOSii9qWQgUaZKiWozCF9IHBqyCikEU5ICYVTWy5NQx3Hon8yVoklAmycTCZpQJpvLm2ssKkg+7i0rMpmW5Z3zHnXvIVULSz2sLTMRQuX5rZXrl5wye7RHeTo9Cc8PLavs+XKwyMPFcgjHr4pX92E24ndIz4cMn5xLZnOD2Xb14ZxpKzYOZx0s/jb2fyM2nBYeCbsQpEYB7lCqlCrzqzU7fCRkWn5kWxhcZUZg5nXFauZw2NfLhRuHB77SnutNloba8nks2x++LwebLCRnfWB57ULvngjEXAcjDFIsBJZeEU5Rhw0yGnoHQcdB/XLKe8Kx0HHwfSHw/MumxCe9CMo1xiYFYaM8u2QnSEOVI7M0lGGDXYnsjdO8yYN++SSYmXRHk00hhOmOZuoIAYieLGtNNEyCug5OKWkiZqrbZ2NWGu6DzQRZVJ4UyRLl5GAyZJSh+JfJE4oI8GLYRKlzCiplX6yWexApHaE3rQT/MgGIX1F+afkwCDdRPbxorKhlKaO18XuUStXkTFGn9hQxdUp6Z78SJCGb+kkDJHhacQ/GF0kdZHOHYcB0vDtC37507v/lvVpMIOZyFiTEuMQU8zkXHTxnM6LQg+zmb2lsFEhtRwkBqGFGjKqU0qlDeGB0MW4hKlmG97U90kMWMBmUbKQ6eQhoww61RBTzDiYcEheEYHxpQiUdUNGGSzKBMK6SYZJ06OKdy56/8d3fggWlx2sXTk7TPxTNWSMFyyOhWeCZuBIKw0WVbYQuhgr4HLIIooI5A9FMpX6o8QhpdKGlNjQJF5Ifs5oWzmez2SwaNeFNRhm14W7h/uYYbhjsH9e6+ztpQEWmGG8KPabt+zNFTK/+fiHu1pG+nu/f/mMK9fOvubQyNa7d/1ROT8GIczlF8HZAt/TIjHpXQfLK8PEwqEMeb/s9M/ADKu9LDmzPqRfEx4Y8ocZ9vq7A4VGliZDRjdWD/wZmpyGdw7fkq9VchDA6hismrbdbb8+VvqLWo3JhJV8hdGzYqm0eB6OkBH1cXoh4Dio5xmDAqcKcOmoRGhQ/JpsFgMNDSebYZ/2huw46DjoOOg4+JyGERPCk4a3mK2MZXKs4cmiMiJaYlY4EqlDSGsiDZOgJpTRpq4H07MsYTp1XlE0T/aSZUkZBWq16TzCsWzQ1309VGfEYxWK6nqleEZYkj6Gq6hBT8SilGby76AxTCr41ZlxqkvILLrVhVQrmUtEV4qR1E4OqNhE+hq9SVA/62pxjhPdHQZYygwNjFENKdUHLBGoqnOiu06boWGO377SDr6jnzgDxPOGoSfgkGKAsD79qoo+zS3hgWg45Hncf+koBqg8obamoOSnfY0RpZWmGoouJm7CVMNUgnERSliQ+B7y5FVJUZItpNSSpEw1hCvCIXl3sWUFeg6mDmqNmThkVIxRO1hgwCoskLEDpf3o9Yvpx3f9XdI0jDtlqiElpA6uGNcXhf6RD9RgUagjtXGuIA0j/RMDxDmZQ7FEuRUnRI+lNHhAkD0XUt841So4jB1lfGlved/u8t5tI3vvO7TvmdFi/3DbFe0LwxbxmczKztnZXO2aGSseGnj0DYsuv//g1/cd3j5aPTit5arZnW8ulx7Kt56TKT8JGwyMLlC7C3Xd+jIOrWTjQQ4t/ZI2SkaWjs8/bL8hbEeR2EAa8+U78qNfY2dCjRctVDZUwhqk/BxVrdSqU88kTHt+TuV4X8/pVez8h4iA4yBgKxyojNgrcChkKCKoSnEhBi/HQWCpC21o0r+Ecuo4CAiOg4Bw1OE4eBQcDTsxITxpKEu1PJNzJhaVGeM3eB1pTiVmJT1ULQowMU6jZeSECNioFSWJx9hKbWOJHifyE90iiASiR9bQUIToREKcQyg2GDsZBbrR0XIe4U1sR/4pY9iLkU/RTmU0k6VOFfyQI8EjTIpHTY6Rk72l+xAvmnYVlbpQ+pSrcAmROjqguIsQr6vLTRmY1Xm8SUhbIqv/lKrVJTCWPQa8kNMGWCrByGhMdp/nlEmnkDoEsn+MAo1rkx4/8pFXJDEoY9rC/WKJEHOGShiyX0W1NvXWFBdnXon9lENGL5/9qk/tCivQkDNUgpE+L0hWtdGQUapIAPJKDxnVxEJtZy/GCIGMHJK/lPO6LyTph0MWcXltsi8FfzHpoTIf37kez3FhUgQtSRp6Utqp7CIMjVOydpSwuDv6+DcDPwz/JJo4WJTEoLYx1GBRSB0vlDIL1knOkDLSP7iijOOo1MQqbLBBJKY/jGWlz4uLq0jILW5dMNJRvKT7Jdv7Bxa2z9p5qJ/c28Jp3ffsf3LpjC4yw+sHn+Dv6/6+/8jV1k/LZXYf+pl5XR8stF9Yy8/KHLhenxYaBcoWFNnO6zP9+5hASGKQPGGt/4+DQVznM8n+ZTs/WCtvzpBdJEOokaUJD9RGFCGLGI6rcgwZLbxmdPQrHS031vJnVsc2QFXzmWwlU2MjCraqfx6ThB4ymjyj06pwHORxKkCoTD/ddASRnA5eiiMxHskPZpO9oYyuokHaVVQq9KRPuYrjoOOg46DjYPqj6fiyCeHx8Zm6dnpu9HAy3+lgtRBWGkyONK2SLJrHNzyxOKxEzKIlQuSEcqJThqGywAwzFVGK46mh/EiWfWSGIoGqisrJNuqSrqvayX1AT3QhnEB4CDCKMTHSqFUsYy2aKE8Z2GSghopeabP05aJzmaXjX7yKqjiVkzS7IwqinOxQTVRFV7m72FxuUSJwyDJ9Xd2aqmIglxJjLTbDFZPW9YUSjLoubZUwZOaeEnpYi+ylc4Ao03SRU1glNDLhljujJUNoxAAhinBCEoacap5haJLsU680YNyaQgNEJw8ZxX7DIDnJ1XVDTNFzxFVGRf/SQ0bDeqEJA4xDRusYozww0hLPjyf7cGwd2SClSiZUMIGQEtZ3Zc/1dyZkT1VwNjghL1hZZIxMOCSJt71vnUaZvmHRW//v5v+VdqhtDEUXlS1UrdKMon8q0SOQbGQKIpYalQp91bVgg1r8hpIkIeCzokxPsac8WkDY9Ez/xoH9tVE4V+1fDj5SLIy1HSxtHtsyq+1waz4zvWVBZ37u7qE/PKPzVytjO8eGbs6T9Atb017FSqFhzRg2kKjsq5W3htPKXAQ6o9VlakOZ7PTf5rR2IJlnCBtk6VHGmpY/Wht+JCw0SlaQZUhxlbBETkPWscKo0WpLNj9W+Vpbxz+VM5nRij6aapVaLZ+phRVIn6fDQ0afJ+Cf28s6DkZ8FQsUQaLMZz4GKmUZDXRKlYyj2eSwpSrM0vEIpRrKQ2zuOOg4yJtBh+Og4+DEe+FE/zUhPFGkZDez2IMA72LADLwLzhYpVnQExZIsrlXH4kT5Yqu0Ma10GmwqLROckK9xR2YVRm8iirqQ5FgVBWpjVhCb6bkxeKM6oL6pxEz9kTfFmLpAFaMaepmlBTWJ0UgGlNhEM05lRknYQ6/gdyyzGP/kDXs1lz0yQrQRd0VZl7XDRq0odVF5i6V6SK0MxBKjJYLopZzQSt2QPt4v1A4DtZVD+UeJEM3iRee2Brd8/4e/idFRJVIXE4ZT0kUZwwzhkMhxOiLNUaqJRpmSjcwNhQuSmoMNnsiQUVFBOCEvNaSEVeYGWa8kHDEBqFOV6SGjrFCqNWmoghaKN6YXpFETRpPSCm9XzL6e9OCs1tlQL1Wxiz0CqTmdSq9UHqxMjJEmK7rOzPRlNMr0y7s+gxmUkiawxOjqZdOZepfRmE9KZLgfLzHMusGiKEUX42BRBrKqM+rJdwfuuKTr6rv67zlcKQ6U284qrMhkWlZ29tRGc2QIVxVnMGT0ovlLv7Vv7NyuNRueuX/htDUDI0/Maf/ZXKaa5mGBuYU9JKCCDBwlGRj2n6gNb8yUVwa+B+tj+0H2hMh/kM3cA/ELi8qEvOj4bhMwQ9KDmLG0DKwyMaCSDQwxy5U+0lp4TaHyzerhz+QmGCCJQdKDlUyVjj6PScJwCz5OFwQcB/mc18NMCwoTkz/w00GBVjKjVKChnBw7olmMcfFyas4pbpERog1+pESQWeyeTil1UXmLpXpIrQzUn2iJ4DjoOMi7xXEw/smcloIJ4Uk/VjJ+M3Njh2tZeJfIVXQRaVUkWpF3RZsopG2Q0cvbESeVoORy/RMMU225bhSgebzSGqrSJFCZQ3mOHY6XkB+V6sPBiTiHUrGEkniTEJgQyXjF4IHAqZojYINlbBWdq4mCFkrFP9nX2aS9qUrXinLaFXKMo8iqwlKyrsKpolq8KBrdS+x87JiiqU5xIjPsZYlGrtDo0FjQdA+xoapuP8NoRpXcygyes6v81NrO60j0Kfsnt0oYIqfpIgacxiGjslTCEFmpQgyQsVQp0gijW1weJ3iBGU41ZFSkkTVmmENISVJRQ0Y123BJ6+rjDBlVTzRklGxhTBhKT/JQ2UVNMqREjw0lVesHWV+09slkxiCbWLDjn+hcHDIKkYOhUULYaELibl2yi71SglShwRgCqQRjJI3rJtakwYaGaYYJKwJ5lm6S8blda+eXF67uPPvO/YGLcnA56GVdWhK6SMMVbSte0rV2x9ChecU52w4NLmyfGYaMZhgyOuueA0/u3r29WKhxxbZ87cs7//P8lkMz8s9Myw/MLL5u7PDGXPXbgb+FNOCFYaTo0Nyw/0Sy/WBI9yGoNtl+sDb01dCV8eGgCOxMmJBDVhnlgBPyyaB95GWWDxnFXDYcOVYbTXahyGby1cw4mWcaIftPPG9JQngsg119nEYIOA7qk58P8xhKeLz6hHccBArHwTh1wnHQcTB89r/g4yATT3ycHAJFlhitjBPpSOpwIZaFZpxZTQjyLuaGHGtF0nSKnCZpKKOGVCHLzDDxjAAcneAHdicqGNkgp7oWJFBCtNFV0ldHk1ZGWVyFUi+aKMJJT+STZwKehFgqKNZxrbSTaImNQikauU2XkC70GMhMrWSgi0qWTV3JaeyY2B0aBJrEi6qf0ZKqyCpRyhJBZrpW9CkDSjWJy5OmzZDj7vbRDG+04kCI3qBzaKAZYm5k9uByjHiB3aFX9g8GiEyp9WPURMQPPVMQRRqROZiOTxlXmpHxk4MPoYQK8kKIQ0ZJA5I2FPELg0uTA2UcYipLDKCF0C0yflA7GB0vpf5iApBoF9OAuJElwusXvF1uKWGMlJExRoaJ8icW/iIlOUMyh6xhg6wDSibKBz2D+InaqYrBogiML6UklRcJpGxYrRQGyEs0Eht4JreAGSRQHmLJbEYsYYO6hErssaxrFRcy/cqBB9iY/l/2Pvo/Ntz1oZ1PbRzq+1+bQrpvfOP74qJV0y79tbNvvmr+P8xu+7nZnX+ay5/Z0vU+ZgwGG5YMZexoJdDIkCpkcnzCBoOQcEUJ4yUkMIwyvVBbMAW58+0hPYie9WlwmHDI4Gr47vGVZiq3QAtzlY2Fys2FbC7HJoTJDpOhd8/XEbr9uufr4r7uc4SA4yDAxg/zCLICh+Og46DjoONg/FgIQjPEQWcIj3pkJ3IyWssxmJMMoYyhUqJ2nEqWoNpYJdom1if2VddQ9tGJbDhVE/mpBfoQaKG8Rb4XqSD2kqlSkyjIj/xrBRoZpLuxouVwCYvkUEoNUWRGcW68boIR8aEvfdpGmphMkx+UspGH2PA4Ztjo6hJUShNd6VqUBGYRv8jusOcVDZC5FnEagSl/8iBNvKl4OfWKsq5KrVDihDJetM6hrqtLR2MEuaWH8gxvxFJfHcJzTVKCMQcoOpe0CsvJwBV5xQ3uRRqZZwiNhDrKeMplafBMTy5OCDwcTwQvXCw5NExUMlQQ7qfxopRHuOJgqKeDk4eM1v3wSQjUWFA5/MruT6OJXBEZvZKEMgsscZCd5fdTBb2kNplkmIWhQcbgZpTiZjA9DRyF/lGrwaKkAamF/tEw2iOLH5Lo09I1aDiQ4YeiiJQ4R6lSl+CUS/BSWpJTsoiUXA6HDDHdsGPTWDU/Wimc374E/RWdC67oWXPHjk0sMXr1zOW7R/ZfPH/ZkwNs5Dg9yetjwpaAm4Zq6zpzmbHhbAtZPiqY9ceUvwNhyl8Q+kOuL5PsNsGYz7BLYXKM7yOvrODwRKowkMNloU+yYbMKHYE3jlP6oAinK/P4D58XWZKEzCHkbfDDHy9KWpLu1MZuasl8cPzjUh122fwIOA7yDBVu9IGvEqU++VXqMx8lAsYoecWHHxsexwyb6BNBp9JEV7oWZQxJjoN1C3Q7Dupd5zjoOKh3wpSlCeGUsDyLkiVGmUMoigWhEnlLMyvax1rxrng62fVkgzpX8h8a1jIHJsaLagt7MUP5jPxQp3Iiua5UClFuuTrf1WYl2cVNox2LWsdtRVQ4iQEphp+oqXOraCQlAQkzmsRWchWDFqeKkXVmOtUl1OQ4oTTGP10U52qePo2xVkpOlcGLN6irqFbXVedjTJVbmaX7ryaUaWoXlbjSpbXkjDyr1KWR5Y0maM4OIzbDdwXlBmPCMDqkVnQRZqj1Y1QFOUwPGUUp9oiN9rKPHhC00oxWGRX9E/dDFgOcPGQUA9nDL9icUKwPmgepi0Ni8AzTIwEIaUQZ9WnGiI0Gi6o/VIkcxmwhmku6roXwwMrgbNCwyM1oghxSeX2htWYMYgaj02BRZPQaBRosku0uJFCKW6rkVHlCmvCquwSeuS6WO8rr1Jy8JX8gDBk9t/vCvcN9s4tz7+59+oJZK9cdfBAD2CBz9O7atz5XqNy08Z7OlvKCkYE105b1bbyrpXb7/NaXcjuwomx+YfAWJv69l39J+rHQaMgW5t/KwNGwJX1SmyT9emCDbEgYlMk8Q9KDNWYYcjDJsP8TwYmGkoa5iMxEDXQx2/7B4Icj4ZAh95i/LuGEXJzJg0wgzJE5DAbPzVFNVltmsmK4kfGSj6vx09zw/QVGyfo4vRBwHOR58hle91QdB9OAOA6ChuOg4yBh8YUfB00I059dzy7PKM5ktQYGcPJ7e+RpYnQ0FgeTPtKtKU/T/DDdSj1QLbI8R+O0zxnZMQyS3S/GZzMmpy3RA8ayl6buKvJMlb4kimeKFqIkyImuqK1KkRnVUspASpViUDKOYVJmslcVpZjSlJQsWmKDQ/yoM1Ffd4peNgjxqOuVTrkcgkospYze0OiKCCjlSjaRdqYzgdioe3LFKWbIuika4k1ORBcxkFuqeCHLm0BDc6C8F+pFMlCtIHVigJzC9zSslEyguJ9II3wvGiPEHZz046iqWGkG4eGhW8KbduLQKqMwQI0LFUuk5DQOGaUzmGPDWNO4KimLbe5JRrQqp0d5rAQgeoaMfmjL/5y45jgbhBMSHaMSgSGjn9j1d9BLZBYjfWwgJAN5KaEHN+MV7eFynGp9UbgcyT1VIWOvIaPpBCO17HvBdvZsJMgLs8eSWYXhWslcQaUW4yU4ZeQnMxvp1Z37b7py9g2Uq7rOZs8MmtzVf9fLuq56dHj7ves+xz6EbAX/zb4Dr5rRs2p2z5bhva+bdUk+X13RPX3D0P2vmPMzlcrrdg99rqvtalaOybOxRDlJOKq7lPmVYX3RmNljdgHsLll6NJhwWneITDKyFJYYWWUlTCwMmcbh+yWEkj9qtigcCgnJfJZbYUFk/mnM7IDA8TiSvlFyqm4m3G9cRgM/PHKS/5G29ktl5vL0QMBxkOcYP88R+ABHo9JxECgcBwGBoOY46DjIOyEczRAHTQj1rE6i1Nfq2bmx3dU829Mfi3dFFhepF9eIDE30DI1q02ytzj59ilzXUXWAEWH72amCtUnDHoZHbNB058YGqgVdN15FfqI32CBUkJLXouQC6fCmK6JR/NNp+kdQaRQUZRPltJ9ItxQqop/I0NQqzQOnvDQ+p7SRMaUuGm1izyXEnmOgJtFSnZcH5MgDRfxoXpcJVIexxwNypHZyGG9TZjRHj5JS19XapJKxQWBp0MWtZyFABZUDhP7JgFJDRnUqvYaMohE/nHLIqAgkbbGBMnF1koH7S/vkR3xP8vGHjGIjrri9tCEyuvQPn+nsnxweZ8iocoOTh4yKXpKdg2rCa+a3hsSa8n7MGKxbX1T0T9lCGB1cDhqJvcgkmUOGx8DuGIya+Kk9MHgbbneU1+/YH7J/LGND9m9ecWGywMz4qqd37b9pddc53AsDWbk71rCh/PjODdhXajkWleHP/4KOJZfMP2/7oYEF7bNXtPYv6pqxe7Tv8jlrCsWxRwa+N5LZuKh98ZP9txUYLJqtHRz6wKzW148d+jAbQuCEg+xf2E6wPyw0GtYXhcK1/3YYC8qplh6FaCEkTA/7kPpLaB5l2ImeI6mqDSWyUoIwSQnUBqEvmCVHkhg88lvAhPpZ/oXgKemHXZr4ccquhoku+SfhfZH7xSp5JzsZGGPlq7XMn550D56lg65+nhFwHOQBxGgSH4Y+7fV5HmU+/DFQqejAaQwQ8uM4CA6Og5o64TjIm4HDcVA4/NBKE8KThpq5E/zqvr9amJEfY8WX2B5+xZg6TtlFUMpIwziN7EusDE26VqfYSClBpezrjNP+Jcdy/Jd5vuTWwiXUnNrYgWhJX5UYRIMwexLbRK8P6NhEIQ1ljF5pm3T8I+ypreIfsk6xj06QRaJkgz5tVhdrqU27kpO0K3UybSONSjXHf/z5VoKuKD/xijhRbxF0pxigiaX0cRU1LiEPCGooY5lFtzJLXzf2DUEJQ1E7TuO2ExpHigZBvA6Ohxmnyv6pCafREjNZkm+MDmnFF3clAzV1cMohoxo4qo5hgBCHjEqGdzGslIg15ZBRJQDFGGF9cLD0HoaRNEIFqZpqyOg1ujSWmsKnodHwveOvL5psYHgzST/NNoQTItBWPcEnVJCXrkjPibv0gQGugf7tD6wbS1Y9pfzErvXqA6Uo608uej97Ol0x+7pb9t16zvQ1G4fu4SeYf9r72Lt6zt84eGjj4b57Rre3Hxyd3Xb4oq4F28vrLu+54aVzr9o99MjGgx8ptry+ULy60PL2TJzylwwQTXYjvIrFY8KuEuX14lRh8Zg8Q0ZvYRuJTCUZTQqBRC5fJa4Ytpfof1ugfGFQKENGx4ePjgdO9TvsVJGQxngbxxU00w+ThM4F07p0HxpmIYaK5Egkzsc1/KOGfJ5IRZlOENKodPj+9i4PGR0H8PT4x3GQD3nHQRBwHIxRhj9tQgzBy3HQcbAZ46AJ4alEZ23ImwzXHE/x8exjjk5EjlNcS0aAcZGFS7O7WCuqJvKm3iAjqIxETq7qjOuUaoXygLasSL6gqSeUs/PoC8nyEkeoIE2UHtyv2YkToyVFbNSfSIQ4FdWRPp7KOM18oD3iSASM6ET2lGJcsscsbUOVKJOU1KJRw3hpeZZbStnQCgP1RErZ61poYpfUMHqL3ZAHTmUQeyVLXQU59lB8DA0vNVHH1GFpKJNxnvV9w1JzC4mm8kOJUqm/yO5E7XAiBqhpgSi1ioyyfxijwWbyXvYTVw+EZ5wuDiKGgaCQPW1OGM6TQ0NGNYhUVBC1UoiyhAeSWpQB5ApORanfMomIyIl94FQaBQrXgvWh5IgL0qRjp+iWjJPBmYGAYSnaRiuFVQR+gokjORlfCv375K51LDBDzjN4z2S0ZyCd4e2tBCPCcegfHbskcw2dmUz/UoNFM7OKs7kvsoUJI128ubTx+1t3DI62lsst9GfjMyTiClfNO3PVKPsQVpfN6Oot9y7vXnOo3PfM6MDw2P6e9itbqjsqbDbYMk/9hMLxR5ktw7E21gKd2xlmBmbYdP7hJAGYcLwwBPSmcbIXZbVnKClsEA/5kAsNHpIjIYph7Gg4UzpRFccuSQBC5MT9xO7SrA/fkR/KR3LKqNfkM2XCbcwiTijCv5WjbfKF15sNpvE5bWTHwfgoFSMUCBwHHQcJPY6DjoPx8wGhKeKgCWH6kZ2QzPbubAzYkmxMrwbMzgqz77LjLEtEjiqRNwkoIzFDwxdctVVJ7bGYXrpK9E9NovPoJ9bSRBwv9Co5oKPLWw5vHu2Qvg9ymBsVA1zZcpjbIeMBXZyRL5dyF4rh8Jk+ObzhLOr1oZ8uRah0RbWN9jLTKSWWlJFxUQuJ4lStKNEorCJHV+lAKxtxs7RNvJCuJQMpox+dyhuyOiOHnKZ7hRP06etGe/WfU17qfHRIlQ55ZncKbNBggPPoTQyQYahU6RZ4CrwtxO40IVD5PbFE9FppJmYCxRLRywyHastpnItIEzHGWAXZU56Q64rvKSuolWNQkhLkpdVlZCmuqCGjNOdVSTYzIOaltxmkLYcYFAKz7yJjlJ5TsnNpyidjSoyxqaOXGKMUrxN71JKk6wefoIk2MGT8JzJmHOPri7bO08Cb0IT1RUt7J9O/2GRK+sd9abBomi7iiiTYryx71/9a/6k3Ln75ksLWtbNXfObpR9CHI5u57+DjxcLY8MDGjsLogW13FjK3zcodXlQYbC2+vnJ4Y2FiH0JxwtBk0vaDgeaxD2H5jtysD9SGH6kNJeyOTKCOMGR0nO/Vhsd/Z0kGiJISDOhFKhjWoYm7FI43PuofsoJsVQ+702Q/0UIsEoo4bslbkXmHdSyROvSJ/VEOdcIYUR0xW8hpdewrlUp/Pj99otL/ng4IOA4qoOizPV06DoKAANEb3XGQcKPQBiDIvBwHHQdfgGHAhPCkH8pQLa82DBBVkpDTOPYysrIoUCtSF0s0SwvDW0Y7kjxeNq2nqo7pndFyGEt5S1siYxyP9GnkgbEWATZIqX5CAjeOdmiMqASa9FVaeC1uzcJvIQ8M+lpYDDRmbymQN6USlDgi97KouGbHyNMISgbsLIcUGTGAl7iNBEUFlZG8YalDetlTi1KUSW1juI0NFWiplXG8kDSKOjhBHzWR3aGX28jHYuJOenUptqWhriK93KpLckUZXUWb45hxlWiGZ/zXebtu1k+wSIw2HsRSEwLrEoboeTS8RAW5BcnoxRIpRf9S1wqpJDFGvM1KNj+E2kH5WC2Gklol/eJKM1Gjx43NZK7IzD1CGsQpXgjaFofKSIkmMLGua9BjLJIWqRpVaq5SDA1jtYXLYZk2Fi2MYZVWaS6q+Kq2MRvJnhYcCsAS1A3lISfPFayjfzSJ2ctaX5aE5I7h3ftL+89rX/K9Axv/ae9TVwwc+NbAgVsO725vK18zc057K38m2bmti1py1bO7zx4q88dU68xXC4VF+bZzs+WrGSaKz8D6GCk6tFECi8Sw6CgjRTPDmbD9YGr6H8bwPfYerA19Osi0ZcgoLDERKGtDHw4niU94WjzGVxyN50cLkEDYINyvWgtlTOjxJx+dpJmhWquKkj96jmipWmnSHqIe+1JpU0fHhVFj4TRAwHHQcVBBzXHQcdBxMH6kExmbNw6aEMbneEJCd3FGpRZW7sMaNsiaLowcU+pP7SMxi+QtcjkMYq04njR8icSDzJj1BGfTVy7VqonKSA5VJepIFS+l/tBzRHZaJ8esoEigMoRRmTSdGFgYTrLiGFCOROCUTeqCTIephQ8nE4mCGUuhUG4fCQbwyWR3hB0I8QvmztK6NP9BzwIqfKmc2xpm60UeKP6GhkgTeSBdkQZB5C2eavYCp9EmzRVlL/Ylh7JEgxC3lccsWspA3FIlGillg6z4p1iIUrWc0nkpY4DUhSLzlIdohrG8Qbnh3jG/FweCYs8BpLw0WDSuIqMho1pNNCYDxRK1NQUlNjyU5OphXwpccbptZB1CeiAow0FJAIr7URWHjEYNAjZUYRaHjHIKW4Nr1eX0oG1ER3FFeNeS8ipKCOERkkbLiQNGR/PIGGVDZR2xlDl+IHhiaOn1S6N/rsuLLqlEEEtUKTNcTc4WTnTnCP27cz/73Z/TW95BQpLadYNPMGT0X3ZuGq3mS5WWx4b3LGpZiP4tyy+tlh++euGK3aO9izpnthRHc7nacG2H/n5Jr+XZhG9sw1hlQ7W2K6vhnWT5GNiZrBkT0n1sS8jfUvsHM/nptf7M+PaDZNX2jOc8AyEsLgtpQ21gGIaSJrnB/G8z8zD0PFlgZnwfwjhkNEwvnEghBqOjDu1MWCH5l6QENewzzQBjou+oZhinGGOs4mZDCEyIohoip49C4UazwTQgp4HsOMhnqeMgIdJx0HHQcZCP9NMjDpoQnnR0JmECmWGA5Ui1pql6aRfidZGqUSUuJ41O07XptshsJhHYXbJS6MHq+NORT2pjqhA5KpGVEhQnlIwSWsgrEkXlA9FLgA2KCkahJ7koSUJsOAh4on9R4BS9SmWcqKqrlQEUBTNZYhDchV9NQuKR04QlhjzkHLY9KO1EL44EO4rRBXuCTdSIO8USoqVQxHjLOhsaSiMbZASUKhHEORF0rWiAJiolR89x0nx62wn8YCBLSuQ0UdSpatPMM20mb6BB3xJSF6i18KRh3YRA0UVKqmQmQTjzOPAglohep0o2premwBjMHxz4NtRucTnkBseZYSbwvclpQDQMHIUQYk9SMc0VyZJD26BeMafHKU7gXZBABA4RPCjflCSNtpqhFxkdTWBrk42VG0xcju9iP3n9UmxkRjfoEj6VYERJN7gWzVUiqKviltBFNDFbyNWhf1jevb8mSpxnHGXycf/Ti37hH7f/w8UzziuNtbAP4ZUHD+175hBVhIJNz/RtLe/dVF3fXRxZ0NY/vTBcqa5vzdy2oPVlM/MhVOTICkIIk8GfyfaDc4/sQzj0/rBpRHLk5q/PVPqrvZewIWFNi9Aww3Do5rBmzMTUwUC3wo72f6wmodQqMokgZdjnELY51UHzhPuFv0IGjoYypP+PZPyOHiM67iIaIBznoFZkGJvYJJc/c6zSX/CQ0eMA14RVjoM8NMdBx0HHQcfBus/v5o2DJoR1j/LZT8kVFLIsIdNSruUjK4u8S/QvksC0EI3TSq6HnjI2xBXO1Q+xOy1eGs3kB/t40TTxo2HMEGLA97M0AxQJxEaCqnQtleJsoiWRnKQNkCMhQcY+XUtVVEY57Qd7gqgSVuKN/PEsKp5F7kEZRXKGDw98OzhpWx0ZIDxK/E2luCI2nB7fJp17THvAobotJTKaOmUkeJrmh03cdiLtCj2Mjrb0CkGEEwOEEzFLqGDoC7AAlPJ7oBQThmJ3AlNIUptOGOpLSdI9BoKuiQ4FspKK2OMQ+s2+6lhqNiBDRsOFJ7KCcbPBmAbkO58Gl07mitA8loFhjCXUC8YlPgadw1vkXShDOi4ZMkoyUOM/0yXLw2CvJjAxBBmL0cWEoQyCXUIy8Y8BnjVkVFsFqpaSixKhFaSlxDMvET9qUWoBG01H5BSHpATVbUpsIIGXz37Vp3atU3np7Mt1a8tbV/D7xfcPbytt/86dQ/vZh/CmgQPZfCXfmlneMWdNcXo+Vx3JbnzNgh9b0rls79D9h0YeztTWtU17exhcrRmDE8vAqG+hhOlV9o2n+OBRlUPifqEqcMjrgsChmYQT+xaGWYJhHZrEIGxMf/SkQQagHuNIRrOwYAxzCMNgUcguf4B1qb/KVG0x41CokzyV1RFiGWtHSn8xNvZjJoQRkNNDcBzk01gfyzxQPnjTj1V6KaOMEM0QHAfT4TKGLcdB3kiOg46D6c+TH5psQnjSUOez1UBd+IbHvg61I6xMjiLr4xTOdiKnMsNexml2F6kdtXFgKgIjS1e0DG8ebRdjpDa2ioKUlDoiFeQ0yjE9KJuQHqyMMzqFMcUwydhEQfbSwGHQy5I4F2VqFfbUSvFPtWouY8nEhiQhk31oIGQXMeObNyWDUfkKy4hKshgh+ZAdTwDCtcTfKOt4F1ZSRlamU9E2ShkgiFjKD0r5pJQlGgTxOtnIII4IxSAeVCGLDSKkr0gmMFLKtJluUHk8wmFdfg8nwgo0BCM2ootUKVWoMoZSNZFD2CAkEA8oOTS4dHtpHS9pNBCUpB9JwvSQUXFFapUfgxPKUq2UKkTmaZBJg2ilh4xyCpuCU8G+KJUtTCcARbrkijKdo6OJuGKa0aXtxT8jC2WNUPnRajQibGjoA6WIn0pRVhkjYyn6J+eySTuH/j08QZUPlvtoSEKS8va+WzaXNve0LOFd+MYlr/j2ga9cOW/VitaBRd3Td5d7F7TN3F/dA/HTG6yj0H14dN/o2IbRzJdHD2eq2Vyuess4p2I9mEACkxVi8BsFiFnv+0I/oY6d12eG2IEwJBWRA/dLdiCssRO9mgyHf8fHhY5TyuBznEBO3tc+Mafgo6tSDWwQEphmg2kSSD+PRfnGbwEDFpaZyAemjZPPxYmLJf+2t/2nttZlR6l80vwIOA6mnyGftI6DAJIOcJw6DjoOOg7qg6Ip4qAJYfpT/dnlrpbusVqeBMKs3NhowgzF2Shn5sYOVAt91Va8iKcx5vNAtfj/s/ceAHId15lu5548CDOIjABIipmgghVIKpCULCtYlhWcJHklW8Frr/2en9dr73q99trrsN7VOqwly3K2ZFuSJSvZskRSJCUqWkySmJAzMMAAk6dzv+/U6S40ZgZDAEQDDeC/aNTUrTp1quq/t2/dv8+pKiYZRmZIhFyIn9cUqaCne0iWk7o5ES/iVkG4xJZSD26rvJf7ThIuTOhVu+nP+R6JROZwP+eELoazqK8o4xGvyFkZIaceD/TMqJonku6jIISkVYD0yFuIw0xiLqccfupxJJ3PRCWeS+iSQcyYYKP2AgyHqYnGFS+2lW8aI5Br8+HHyRspraMRp9C/GBKJFrzoEeo80F/oXYAwHgx1LuB6/LSVUpLCh0qjBi/rbJB0b5iL0QX/KRR8iNBfX0E0srvWH0pjG3xCIH33FAeNsm4wpEiUJOJ00V1GeWVnHSDYHS6g7gjqkk/rMuomssgDIZBNrphaxFUGagcxc19NRkTiHhKB+JHuBMxtdM4DXZhWtTK6VmFvsLPQhvBkQ5gsflIlhO+5NiIuD/3z7SWc/nmi81U3MJpkWHuGLJ8x6PTvH/b+ESkYSwm9tS8Zun3zrg/esuJ7PnUQu1/i1r5VsKv37Xn8Pekr//bwd3rGSuxD+JzBlZuLj0xWt2+fXJ2tPzWcW5+rv4Z9CNPZle4jatsP9r0pmV5SG/9AsvvOZO7y4CBqPLC5/eAhloSpl3aY2ZCD3SlmH7YIrA9yiC8oy4eGuCU6A3RK6ZMGQ1ivrrfchY4wDZBvkJkHuStwEHVaiCyR1sPcSY/PDF0yiLQWWphJymX0GIzO/RONgz4e+ZUk7gOZD14xS+Ng652ucVDjoMbBDveUYeEDHaeCwKFaDtrmFA5Wxir8pCxNNX5nJ8spIqvOkBVy84OpqsdhiV7WTYKuxFkidM4ZnWtw+ufta2QZOUowyZAPnqV8qB2LJRUR8mFLCSiTM0AkY4SVRV1PNA96BAFP9zmEDGl8PCUOb/E0Uo4YcQLjQ6AXJIyshoJk8SES1XrcT92t0Yt7GDV7pV7WQ0+JtdDxQA4Tu8NCKU63XMbZmsfhbJx6nDCKYejzRAYqT3RJiKIb+py5IQPBixpihCJRDErpYq2SpLiAK6dgFGMin1ftfQGu2E3eIfh4rocAQq4DFScEOkpR0smky6PNFXoRXEap5ea+2+F+CMBwoIXOc3AZda9RnxyIyyhZhC7p2h4Y/TyJcMgmD+Q6Xnnn0A+TC00ihFDBDIlgaoN0OVVzrujGN6dqsC8/RRIxeB0RrIvEozCSLkxWPKKw+5e6MIyOwdWFiUdhakcAEohmQuJOLwlpACFtjimUQrkTSFdOcRK9X++67FeIb+i/JiofDQvM/MPOT5LyX576xy9N7d87fRjGtLZ32S19q3724tu/b/lzbuy/8eaBlz176StvHvqB/uy6fKaFoqfX2yqjwdCXYEJddQts0CKomP2CrRcathz0BUJZVvToXvOY+7wUouZcavetLUzKEeJ2yo72J3bA+vAXhcjx5SHkgUXIx9kg68zEj6ej1XNDQRM72c904b0n1jRJnXsIaBz0a+ZDHg9qHrZxhPJEFyDLH/Ke64lRWOMggPiwRQSgNA76HRJDjYMJjYPxbmhbRITw1KHFJEhheFotODMSj+TKyZuTulgBp56OfK3JEqGIEBv/uCRirpbQi3gKoX2qtuQMYet6NuuaZM+JIuY+2ub8MEY2l3t4seOgkXOoYEg+GvjQNX8AI4VxizGMSHx2xyGNRFR4qTjmcerxGBJxMuOSXmuU59SriKpcgDBqIB41BLGn7M01kGH3LLXdMupmD3TyBh/zj6tyHkiWG/c8kVOnbYStzK2V4JEFwZuvCg1QSlfoOqMYieRG5onJjrYxkW+kuIfut2KIGFY+Qn8ziJ1Fhg/pfjgDjDwwepmSi0LKOuAIRJzJclOPG/dggFA754FO/wgxGzYqCNMLXTLOG3SXURYadcl4t0eXUcaqyL6c9TlJgyvC2dxY51QNSXcx9epwGSXi7K6VWJLoxM8ZWqsw64v6KWHUTNwbgH7ikf55M1gwxov4mOptIHTlrsRb0qp825Sp2hyIa4MuTjxEyp0rX0z4vhvfTnjT8OV3Dq5Y0TtInGPn7IGHJx7l+vZnlw13X9KTWXNo6r+UK5+ulO4zmgepCxMIIXu1w/8T4x5Gwvr4hynI/hO20gxTAVsiphG+h/UPGhkikMBk36vMKkji1HuCQouT21iBxvmhLUwa6KKpOOYI+weaSTDsNmFTB8nmlD/OAzklzqeV9ZnQMziWD/5Nh/8s+gw6p6I4vGgctOHJH9o8gbknCP1R7PdHjMcHe3w+u6SLea7HNQ6CQ4RL46DfFYTzhyqNgz4mahyMN8kpR+QyeirQ8WZcrn77QDUPnTsYfETd9xJdkRM6i3Pt8EAizuiIcErcQ04x7hGOBudSfnAdSpVaJefIOxsksfXYyo7zwWYYWaIvS4NVcCzQS5xLkcd46E6hB5sRtwoerGRt0QsTaMwhJM6zeBd0Lmchmf5oZvQyuXCQwml4pbRzSAhPbU9sHf/cwOVFCH38c20ekohM6/jnwq6qVdLFXENUSGS+DK+wbJPYoucp7DYjYUNFQlu3prnii8s4D4T+ESHFnVuIOMHzRE6hdhQkjGyTRC/rMs4A3QcVbRA20oOf52725OCDPAcN5jPHzxO66B2JDJCeOvcDHIfRXUYR8HQMhhRxMUJkUO4oEXG6+PnRvwsetuwIciWJGABher5+DKduKvQwMkAknfuR0uoySo++M/kQVIoPZZ2zRZsepyQyOLHKaKvLKA6ZbpSLXBEmZmPYZGPJULfRtc5FRI8Le9gqPH99URdGkgY4/0Sh6ySkqcfbXdD3rri8b0PiEDrs+OboA3iDu7Ooz6XEVzbudXHr0B1P7fogYtxdTxzeTuThg9u+MDHyxUc+250vZ0YrPV1GnXAR/+7Y/eXqoZHZL1/S9xs97EKRTCR7X16fXQ/fY/FPY2vpoToWwm4zsR7dfhA6xz6E/A5KhMVjmvvRI8amhYETjthUew4oH6vIQDL9ME4YfESJcNhpmIvYyG79E77nTgKDs6izQX4yoOnW+mOPkNsocmzOSZzNFh+sVJ8rTngSkJ0johoHuVD+BNY4qHHQv7UaBzUOLvj8PifGQRHCBa/dYomphK1Hn0k17HhO/KLZzY1vnhgpn9M/lHqK53pIoudGYRimKyELfuhxKm1tE/TPT91gaEqaC5MSh/45IbT9BtPleEoWlM9JoBf3cHm6QvpIpUEXScQcBOeBSFArc89I8f3rwuDXoBYMgWR5szB8/dvEF3kZDmJJimNlwkSJDAYTE6vjnmd0yN8ud8HWeHHmPJSH8BCF0hD6+EroFIgUZzjkukCrjMcXlwl6nsIoB5/hwmGMZd8FGoKjKfxtT+lJc+CkGaElzuhQyBFJY5xk6OmEqHIzYEwh0iCKuasen/wWvWN3QcgnIHz+kNE2BLxrcDnvrzPA2Pc5vfMlYTzXa6F4dBn1dFfl5NCrQDIq9FI4i3It8AIldE5I6AwQNPi4GKGzRPcXRdjTMZEh71xxR3Ez1I4Pljf4WzQDQvDgY3ww01HK6SKhc0Wnaq7NQ7JIJB6thVHYBRhTIYEeorNVGJ2k+PQ/GuBVeymviFJoo5GEpM9vLXzV5wpC/xB43/bfJPSVaYYD/Xvzmp+GE75q9Zv/ZPt/X5obovs+1/GpyceR/KMdf5VM5L8y+gTxNb3LCP/4xlc9cmTL2r4lj8w+tHHwhh2Fr13ae10yUevLrput7ElX3jvQ93s4hWLQq+feaca9WQoZbUvm5m4/yDRCyzLf0Y8Zr/M43qTG8TAD3m8bGAZrYaJ6m5FATI4hC5Jp6814EUotsqiMLedrNmOnghYJd34IrEIswDFu58/4KFQ2iw0+YxQ7ToHGQX+cahz0H0PjDapxUOOgxsH4dYiRc2IcFCGM1+uEIv25ATamTwdaw0jg/AYuh/ENTgXZwPiGIrfIcQof8DgMDUOBk0DkvTKngsQjObSyzVyPRE7opsiD1TwyhMPpEsvJRIOhRyJRND2BNDoz5NRTOHVOSOiJMdKcQ2h2pMgHiEfO4DTPSxHS+5jFKaOjsw4SKe5ZrTIIkNjKuFZkjaUAETSSXCIX568ctqloT1KQN3snOdFi5oTKEz1chCs6n3QBL+iMi9BPNwZixo4X9IQrRXqTstbXsrSpE7zibtisEzwngbQ/EkVGPlgf9wKUEusf9wORxiqpwT7pjXTaRtw5HljRKjrVygBJfNPqn/7Ivj9iyp8vBhMNhlGMiHM/KJ/3zg2GkQG6pFdKiIzTeJRzcHXcX5Q4FwLA+cQ1Zvx6BUG7+px6ise59ACyv7THOZiLzbfswb5gZTAxxAgjqeNSUsSpmpf10FcZZc4ey37Gw8WcznmiWwhdmHjc59Ab4KMvktRO6OyRkA/C7hfqWa6zVTPyrtathU7/WFkUZD677x8I3VqI2K7i5qt7n03kpy99229v/vu3X/G993z9Q9cMXXznwJaR6fG/PvDoj2euJffRyUcma09+fuShK7qv7ErcvSb3Qm6PKj6ffPndn5M9CZcZD2RB0bj9ILsOMjkQvmd2v9JD5jg6+CN4k1pK7jbitvpoWFEm2f1cFqGhIvMdnfpsvTm30DxIOZwrWuz4Bw3iWobAOGGLbXBBeyCPO9fVLLew5lRYd3nBvCW9b1owXYnnLgIaB+O148vU+vDkoaFxUOOgxkGNg/ER4ZFzYhwUIZxz1Z7+FPLQnyr1pGojYW0YCsAG3QrnISmtNCzG4YEeD+u+NCxkkSJ6JNoGTUlzuRePNIhi2AKRivghf0m66vwQYY+gn/c2aGFsSWwYMn6QwhjmvqNzyOGSVHlzMBkxqvkgRxjj0AnirmTOEMipf8h1GS8Vi8dEBOJ46bmkRKpJw6JFkVyvi9VEoTqR55DoNI9Ep0CkkOsyzpRcgMRIJqNMpGeufL4S2KArCe/NSfObRXnhSSgfe2Bg27w5uIxCX3mZDl2wKYtz6vLqGnq8prD3g4sRzm/P5ilscbbWKFlzWkVKVOX0z91Bo8HQFfqWg0gij5jLRHbnl8BdRiPgvs0gBkNy3QzoF4XTyBXRhuckzqIMcvArCJVzKmdubgYkTi6hkzeKwMGi8ByqRq6LueVt/r4RyM+x/lHEhZ3gtfqXegNQCD/0BniITOv6ojQPJZBPPEg9jLsLko4n6qX5DZH+AYIDFa2Ff7z9N5fll7MP4aHZI9d1X3LPHrtMb/7a35XL6VSmdkvfak45bhy4cfvs7HUDN6/qWjZdurg/uzpbqWdyNyczQ7h9Nux4LmrWvDfERWI8jRA2GOPHRMwYuDbuzlQvbWvMLQxCye5fq8/+arQQWnyhw7gd37EQOM8LZ5ZS9R+3nCg2SaBlNI8o2Uw45i9fh3iO7T919CwxNv2R5QN3xFxFzg8ENA5yHeMgRZxnJqf+8dPWRM8ljFkaBzUOahwMY5bGQZ4KHXGIEJ7KZRiv5abrVSbTQ8PYEhAVzvQi9yMlMrGY6JGY7qXMzNhCKQ/VskPBQhjZYKSIDa/UYNmLqrwiUxHawFo16I8skcSRYFEM+Y2gtaxbBaMTKQvPxFFt/gBGeR//CD2XlEgtiDhjdBkf+WJubICPl57rIVmQllYBj3sVhJjmsN2BE1QnLMfCoGvOpRyMKM4DPW4Cwa7YygNhgJy6bS0WnMO4XImTLidUKERVlI/6SQmULEnVXpfLuAbE/JTc+Xq8tZGRnrjBMNRonZ3TJO8UiaiiajctujC2QTcP8trh14uIyQTuB8FzzP1ihWYbBYIf+muKS8bwXw59hDg80KkdHpvQLTgbiQxpkDSnW9H45lRwQZdR5KNNz3kdSlDrwlEhEY9H4agTvocwYTAtfsElnShGskqiN5UINkA4J/VGZ1F3GfXtJchi3uDq3Fru3kj/orNotBai7W/2vL9cTxeq2W/P7nv28PVo/r83vOqd3/rcu6592Z8+9fkbl62/a3SHN4bt6acrIyOzfzmQemul8unqDAwpxchnO0bgkxlMfEQakwBbnEjNm9S9Qw/e7+yOOYf1/b/qaoknYHrBDMiMRFtUpml1dIFoIWzww0bq0T+sJcMSo5zzbaqGZCIh3iBw9jPHUfFTjEEVWKLGj/Rp0HeKzVCxdiOgcdAHKXCOI53GQcYpANE4CAhxzCLuP5gS8UPjoMbB5r3QQX9THdSWc6cp/WEZGGx6vPa0/jQOGaMThM76iEf25Yme4rlzsjglne0NcTrlg1oIJx/erqgF62JkhhEn1xBPPeJ6iHuNhMShhXBFQt/3gjDGXT+mQvcddfIQqVpUPp/IIcOH12hnGkQQnlPcE12JayAeB1GX99wo41UTulrSoS6e+K2JLzozZMoi+BCaw1swTTj7cj9M52ZOllqVO1OKXJFxCxkfvQjRgAARBFwmcjyyXMwFmk01UuoyXruHroqsOXpiOkpcMm486A3zBnuIcDQYkkstaCNELVlRFaeuDVVRBh7IrDAQg+AhAPeLYHIJ3GAY14/h9QUZX2XUJxZyCvJI+rVDnqmDEDA+kCJGtUi0ossoNAzaBkNzATQ4VyTih9NFBJB05uYpvhUEYVOQa73HB1G4YhSG+CHgoS8ZymhKSjQtumZSaIDr9AYgTAr0D1W0ltApqLcBhdiyXO1zlr8IlCL9I461EBBgyITgQ9m3rH035OnNF78O8jTcvdT2IQzHXTseeWB633/d9LHvzu56ZOKRvaUnrxp4/vNWvnlV99uMeyUS2Z432bYQthLMr3GaWrWJDxESk334gt6OE6lHCM191DxC351gwmHge8nBj1rE1g69zTWYAIbEwAzhjbZCafeLSHQyaZHjHLSchwk7EHq+0zZvpD/KGhnN4pZoSyLbhwdFjDfzn/4vHqdDA+95ejlJnIMIaBzkovGs4KNxMI5KGgc1DmocnPM4P1fGQRHCORfuhE9bXp2cepEAGYOfHKzkDlby7ElYq6f4kG5cq37UjzRyNipzUucphH7qLM5PKQ5LxMvUWaL/hE/KcuOKdngRD0PCwgFtcLXYA6NJENGRai6Sw6Vp2ylxZ3ETH96A/f0PPsCH9DX5q4h4pRTkRZmPj4VepQ+NPhPPQ1I8yyWJxxQiEA8/JdfFnJ94PPJAr2iOBi+I8XBt/kooEEvaEEIRN/bdDkT4eboS+JXTLed4oQH2+6UfnugDmDMuJ3iMbQhEbtYUN/rnH0+hoEtySsQrQsApHCmuAU/OqMHlI7tD2GUQcA9PIqglPbbHyzLD0LO8tS7sFXkvuC53LP9hEADDyKUp4qtl8r4Cnpx6VivOIOlmQ+eKLkmiy4AwwxsFIV3O4iKtchrWyhURc6oG+6KUCzhXDFUbmXRKafa6oJZ0p2cuhnJSyHKW6GzNnUV9lwt2gIgNQJKDBngICUS501Snfy7s+r1hWAsR9u0lUHtJ/gpfNQeTIDeY0z/nw1gLkWRpGULmFtIk5C/vWv/3Oz95bdclv/zkx9mH8B+3/Ru5vrTMr1/5hmu6L1mVW3NT/+17Z7bOVCbIGuzamMm8plI9QHyBg3mDLBjDwW6E6aX8tQ0Jw9qhvgiNLRvDwQoxYbUY8xGNB8uQcgTSaOmzDzh7bEgex2XUv7yEYXt6UxBS4In2w5bnWrzJ/SCBJhSOmMtZFPBIQ+I4f6aLjx0nR8nnPgItt4XGQb+cPDz5aBzUOKhxUONgfMSfE+OgCGG8Xica4SVysmauts6viED/IIHLU9VlKX5z5xXK3qJgaC5AIh/cOJ0fGkW07ePZdqJiSpoTCyOji2qJsG8E6a1ZJHpxftdHA69xfA5VcsQZmtEGUXRbn5dqLUt1HIiZZLAcIhnJ4RXNzQwDGXgpRIu+8PIdX9wpSxxLI2+BThFhiUQgik4XkYdJsuwKAoSeS9xzoRahfuMkVMH7t9ugSIz8xEmLi83ngU5mPETGNRCGBpuli7iXZWlNKCLsiNCW1K+zBOjL3DsXQuXTApGEnkXqFfkbAq2kKwpE8jafKM4hbxR34ucU0T05W/WQy8drQcblT8RgiLCPsq2zB+kmk/hZ/Ma5H/1yahfnBIaeNui37aPA1Q9hlHR5F4vXAhku3PJgi5tD7ZytObULShbmigg4XUQmHhAzbqpWM6BntfK6KDx//3o3ALow+t0SSIQWohNWyQf9nEZWiQzxZbnlUa2vL4q1kBvG++70jz02kFnQWojO70w8tK2w5dlLrvtOYef/uOr1SOIsSsjSMi/qXX1w5gj2xlVdQw9P3r1r5tHP7fqVkdm/2Dv2jqq5jP4nXzXUlgmFTR3+n7YPIV9D22CQlULvtlNjdOtTy37BbIl+YAAMa8Y0eCCWQLapwBLo5NCXIbW9DdmKcIvJEAkWRUpHQ2JDVcsff4F3qyDJ/jRoyTey13p6InGKuNoFhacL9y2YrsRzGgGNg9z2Ggd9pPM7WeMgIxFQMFi4K4qPmxoHNQ5yV5wT42DqnB6TzkrjfdJg/O0cZhVSjOM5E4smuEjJIiujwcgvS1fq9RRGP6OI8Kt6CnOiK2xlgJRi34jYx5iFBj7kRrWsImMvdpBD+2kfWjifKBpL9OahMJZFDwW9wZtCXbw68zjjrdrr5ZQIIR9/2HGKgKcT91dzF0ZV+GW0sejInFzWqHQb4zcm7/VxlFMaDOv45uQXI56uymkecRRGHsi7O6fQSHJdwNkLcS/FKbkuQ+gyFEc5a9XQPMjz/rANBuZEGFrQb9yMD7k8vok46SIrki5MfC7jtXg4h+CROJ8ouqRb8NDgpkLEnAqSSyKhj6NEIuckfRGDoauiwc/quxm6a/iE5W2ChqObzjs47jjqGAIIpBFkWrkipdxllNCRRBhizyoyZDlzI8KF9nsgXnq3uTkBI3QO5veDW/YQIJGyfue4pNvrnMi5To+7GPKRK5Lih5v10DmnaoQZd71qJKkFARgjaqNOL8Id65vLu5dpq7XQbYDuLLq4tfC1a94IVxrKL7+u65KR2TFcRr+w/RHq/aGvf/jLU/u+eviJx2Z3Pjz+KClLcqsu6nlhX/bFQ32/xWn30k81XUaN7LG0jO9ASBgjpNtRHbeQBUV9i0IzALbEq4cSpU3O+lxhEL7dQpaccQ9SK357g0NaxjEHhsGwquhczheXEuX7eEyBEz7hK3a8sisG33PCaiR4ziCgcVDjIDerxkGNgxoH41P7XB8HzdKl46QQyCRrA6lSd6oGrWKxdTYAHEqXiPscP9id8y64FmojZ3MKF1Nirgt4yM20qdQLPeOlbDQYIZGPBX1TwVYNXioKkBVziXgV0AaIIiEskUTahjGTKojYiF5nq0NbmJTDExNV89njYLTziIecxhR/yfbQ3/ijjEc8izgv6zE+pzi5nuICtNAtipwSIYQ1QUsQizzQlRM68YO9OM85Hg90ekPxWJAIiTAiIjiXkhXff2mAy5OFAKcwN9awwdT52NSDbLXEKcwNMuYhYr6qJ5HFl4cxPYEXuQ3Q44yjfEJdRkeJuxJy6XjkkK11scW8N4m22aeejO6g3iPQoAugEbtMus8MjMZYaowySLqp0F1GnStC1wGfWxF+xQVyLtcakugXzpmbM0BuA2QIbXScTLh7p/9KirBfYiRdjBTsdSQ6Y3Q9LkboCmF0pLs2dxl1Lteqkxq9YX4TujY6yF03f31R314CVskqo147asHHt5eIkyoBxxfXcZK8of8aFEIXae2n9n70stz639/+19OVXLGY+9LUoXVdZm79oxte/f898YkXLHvWlbUla3uX7ijMDmaHl+eXbi/dRy5HDZ9Jtp1g8ZiSuYCysYSvB2P+osGsFyO1w7/YTAkbDDZyt5gWtjFMvNMiLZbABgnMvdO2vMdNtDmr0MSOf/CbEfcPPyV4JApy0WP81CKona9ibPoLg70bT02hSnUsAhoH/bHmoT+C/GLxEPOIZxHXOAgIGgf9ftA4qHHQnw8dGMpCeNIXhYGwlEimEjUW0FuRLuRSlclaKpusjtfT8MOUmejMObGxNkz44ZxTCJgzNA/hcpHOkRKzwh7xPl0w5b6mRufqSZs0eKxVkHZHVVFD1EmuV+Ep3gBnqhA/cgmhiDSS0C2KbEzvkYtyV/KhDyFM7ihs5uO+qaQQ4ZQm7Sxs5sOrpecSei4hRG5nwWhAHBH9dZ9mXJy7giI+ZPJkRIB4HEGdDPglwaLI+zpmB8yJblHE7ROuCENzgePxQM91Gulxwkj2KMUpmp01+So1hAiQSJYL0MFEINK+waB7nxJiWgzTFM3YCINiFwo2TvzXQ3/vApgfSfd5jMGTcy2ny3IrAdP0JBIU58O7OLlM+SMXb0804PB5qHSAEElIIOnIxLoQJjc2iaa2mvhothNj2J13wZmhm1KdCpLupkLnOc4DQYAuoM0lgZoPVwSWFS+cUbJgO+Wdho+/97S6jHqKEzOuY6urDFfTP2jgcFUextl98U7wSx/rRZ4snxCIThcjdJdRb0DQejRwMWhkVAIJJO41ulzrREQk6btjFV1GHUCEnUiztAzg2IUublqVX7u9uOVHVr36hp6L3nHFK5C5afjylw8YJ3wR206kEl848vVDxYP7So9fMXjzJf03kj4ZvCVr1T1usvOVYPALTQ3b5MPU8PuTy+4iYua+hjHQIpySZeyORLMo3kbIabKPxWnudNbX2LkecsiBy2jcsD54jTaKWN4Ch3M27kY/uNmIcBpTmjmn5282s/b0KJKWTkJA46DGQftRSeNgGC41DvJw0ji4yBP6nBgHRQgXuYILZ2WSdcbCqbqRQA+HM8UVmcJwukjKikzRGVcszCk/vfOxVWGaoXmNhjcwJ2lO25zCOc2jOFnhk6OsLSpTyUMRzcu0nlqXKTBvkI9NGmwhipEZUtzjrg3NHnG1UTlto4qDYSuLOc1mj3jjdcVNzBVBntAjxJ878BLSedWmC1A1TyeRLD+e0/9SijNe+pDJmPHs/pfCZ+wToIA0kvWNiftIIRd5fxud8zsrb+F8/P2e0HMxoGHFwt3UQxoARWSCor/ORh7Iu75/vElO85oNbPBDBGJKpFVocH5FJAoQaRWgFKfUSNWsfUocgTlVmJLwqg15c9ppvxM0CSeRxlIuxd24syLQuoYquc7ciDhjQYZ4ZHde15wa4S0uQ8uJ0CrCKOPVeUiTaHDsHXQIpu1ulpHaUZbDzXdEwJ9P5IrEI9GazxXnuIwGTYlI2DiF0QEGXUMP9YYrazZJsohz0anXSxG6yyhiCHuiE0Wvl9ATnSu6EtqGMDZAVLmwu4y68uiDSkFsgADlLqMPjH6eFAfHrz4smlyuBdqu7L/6hYO3IsBN+8FNn8dl9D0Pf/bzEyP/uO2bD0zt2zc7StZwfphw8/iDOycfITLU90bCXP+b4XJG5I5/GFfE7XPOUWXrwtsSrB/jZatHYr6vSmrpnhXmIlo82A+Ptw+hF+cnq3jwfYzxORF2Jmz9hJt3jsgJnS7pveWE5CR0TiGgcVDjIDcsj0eeDBoHgULjoMbBRR7h58Q4KJfRRa7gwlkr8hfxnrSv+DjbuB+u5hAar6VdFN9RWF+xya+c7BE67/LQJT3LU1zAOSG5kby1RlpLEWfDQOdvvuVg1BzU4p3WeHmLOl3YOSEvgLFJRFakyyPV7Kp0eX9otlcE3/MIZC/GPYXwmxP3Ekau6OnOWFw+FmHI9PSY6wW9SGSYGNZ2FjdjC+FVO5Q1I+QlLNRWegoZXvF5rSfkgct7uYdOFD2L3mJOZEziTZd0lKcSdU73Fp+EDjk7ImwwqybNc47nLXHuhIxHnHfNF3BhQqdSCLtM1EwWia4HJa6NEIHYAGQ4JdeVeBFCWBn2KLKIczhz85Z4CmHsDnHUYuhDLXqiKiIuE3mgyyAfVbkMKYB2Tf9GfqRwu5lzMIeXXKffzq88JJEDhB1kj3NFMO4h3Cozx2UU/o/wK5a/mf0Mr+27mYu7FJfR3JXLcyu49CzWwtUPVxy+l1yZX8ttgDbCITOuJt1l1JsHq4z3gBsV0Qz9iwSSOF2ILfRcvFhbXUZdIY3kJmndjN6hc5dRFph5wCieHYeKI4R/tedPytX0qvSzHp3d94reF/zFkcfed9P3sw/hD17+3K8+sfulq28e2b2TzevX5J41VjpUrh6YLt83U7iMB0Q6dynF67bf4N1EavsbtNYigcId3X4wLBVTr97eMABSKhShFEd9KviLEmOW4OwXjPuFOKTRLJDOBi0prD4aFqTxszlhpICNx0TInsMM4z71sSz3Sbr5YImJJxKZmP5yT94Q0HE+IaBxUOOgjzuMRER8mCPCkMR9zqnGQY2DGgfjM/+cGAdFCOP1OtEIdAs22JusHarm/S0K91GWdRmrZWCG/rLlvpfYvmBi+JEGnmY0jDqcmxHhlI9zOWjksVl+GqYmNj08vX1exIt7iiv00KsIfqfMFcy7csSwAbLffSppkaiHyPW5GS8ClboyO3OkZtZCZ3qR1EWO5xEvvmBiaxHEOIUHxsQYoawr8RQ/naPQmSRkANMfnAFGgTkRtgBtuNQooq16ghJ4CPSASCQAnBJniVQYBYMTnCcKOyPyoYswkisEfAwj0RvmI9x8gTkaInnzkc/LRv1xgMS+hNGJUwQIIzFrrYW4r2jS2hJXRS0UQYk3z09RxakLUDbKeBv81PW7DOl059+aemCJ3518CFsrK8cAVyt6AOjUDosc6fArQHZyyKmjjS0OYuZirWFUtb+4F4733IGXensI4y8CTkI+tOd9nsV15FpDAqMMhjiE4YrfmLjXyGEi4YzxeQMv+bfJLzr5pAFu97NSoXmxalfr9M+74OuLOglsnYiIJD8ZtM4YjOSZrGgtdIW7iptfM/RDD088cnXP9ZVa5sbl63+8mhqZHsdldGRmDJm/3Pov26s7Dux+YkV+on+mvCRdyvPDRNIMcMWxD+Uy5nnLHoP1qffYToPpofr4G22DwdJjbDdvLqPV23D7JOLzK1h4xlgiRbAQ5q5BGL4XiB+rkobphb4jBRJNnmnSJ3YAqX/8YUUhvyix9Hw2GLMWiaAtKnQxTnn6LVJEWecuAhoHuXZzhi2/mnNGOk41DjIMaRzk9tA4CAgaB/1B0YGhXEZP+qJU6+zwkDpSyy0xB87kYLIymKqOVnOwF3ThFogh7mBgVrwKwa8OVHIs4sckQEI+nJLuNAz5SPA87iEaQtkcnqXuaMpugZBP43XBybO1lBchJSYSIdEp4sp0eTWfsO/FgUpj64tVTdfTR4q903WzKGIkfKrccySYOhnkqJ0QOxt6GM+It0Z4g+eUw9NjLjYfGumcIZaan4tSHzLRQy5xP0Vh1Mxz00kCv8K6AJKeSxVQRKsrzGaEIvp8RYpHMmONs9dcexv95uS9RPAs9e0xbP+JsCsGWUQYqKBbhHygTJQihFARgR4QtgpAsaKAxznl8OKEfkpxz3V2hyenczNOvRaqIIUBMsq31uKJ5NJCVDnbjAZDWkuzMdcQMk3f4/BeUjBkcQprwpkWF1BSCOkvkSBvjj0HiyPggKUOVLGkeV3uQgnL4tQJHvxqzvw9shxeQrII4YokwsfQwwcSyNcBMyDhtf0buV5DuRU+/vn7kFcKEaXUK4feZHXl1/rtQdxPaRVckdDF0EA62gihiLgWe6XYA71eQppKLu2h/dBF9yx1+kc6h5NAiiBJLine3+v6N8aLCLyku7Ooz6v0q+Muo1gL0W+66okDhUN8x//TE5/4iwPfuf/AJlxG904fJue1a14IH3r7RT/BU+CNl/7KKy7+9d7si3vzN7MPYSZ/jZXlqI6YHQ812PeIwAax7IX0MAkQsjfScBy1iBkAEWgIm1zzsD3u77QTJhmih5CPH24nPI55sIa+MMOZW4sjhvbkemYHGqJjgmsKE6qthuWDr35mulW6ExHQOBhHK3+IecizS+Og368aBzUOahw8t8ZBWQhPeqzlHQcP0XItxfqcvH9DBbnky9MlzGvoIuVwLTOcKh+o5oZhhk2LnFfjKSzf0poFRVzBOqW17Ibs9OZyL5KR0TnH89BVwQxdlTMB18PbmHt+IhmdP4nDBqu8sh/bhkuyMweqeVfCujhjTYWklOr2A4F5b4ZXxVoivb2whV/5CS+1nXY3Oen9+vj9l+Y3MOxtZ1GZRII40xqfN/Bi3vtNIKzFskguRTw3KOesUZZEjEIYghhTSXWHHCKMu04LnSKS4gJEGINtAA72QJghtdMGW7cmOPQT0mZe5QmjeyGJdI6UppKUG5083YnTSHB6iWY9+JtTO0iCsz6ndpEBOt9zscgxUOgChCTyYYAk7qrIdaIIwfMiFCcOx4PkwNlgd/C3fbY8qe0GScMwUj0nUFYE3PbFJoFQQW859wPx0NOXkhJl3MRHiucSQQZMIH7IOP1zzxZngJElIskBy4JKuRjyaEMADZ5LBOMtE0SBPV4dsg6VjF/BAAnBmcvEh9HRXEb7N5KCa6W/PyEQL/QiYmRBEf2SETfukbSdMGgSIa1yl1Ha1rq+KMo5fGXROeuLggAK9zS9bf3StFoLvayHwVqY/PShj5Vr6ZlU7tuz+37+0lffv3/T6y9+zue/9bk7Lr3pb0cfQfLqrktHi0YOe7MDhDPle/ePfRpzfWns8/lkeDiw9IsZ9IJWIlUz92H3Sw7+SGJqBW6i4WuHwZDtKG5qzAP0iYVODqP7KC6jvqM9xC8wzKDRXEmNRhJiP1yIE9qywtThk1kbZeyP19uSsEDUfx5aICNQQR5BrQenMWVi+pvLB+5ozVX8PECAe0bjoMZBRjQf5rilGciIaxyM326Ng0ChcdDvh3NiHBQhjF/eE43wKlqsp/tTlVQSNphNs85WPcFkwqXBaxTf0XKt7owLixxKV2RKhBA53mIhdU7hnN0RJ4u407wnS32ctnJFBl14HYkeunnQTX+ugXBNury3mt0feN0NuelHS73D6RJvY5SdQwXRw7Gz3OM/WmDJ4Vde4kTi254xIntpjK9zNh+PRCxsJBEGP7h6iJi2y7rW++slLp2MjpbUELNITFkw1zkkqqATrhDaCduMFJREZPx91WkhZMAJhoeQECcJ2AzJaq2aKYhkUYpm22qogSNBYJwOcepUJ6bDK7Alwi1DaFtfON+Am0WiBYUDmbgxwyuMLD2EBmbixUHxeAMkYhzO/YggFodP9HOx+DgdxZOzlXS5bcoZYKRw8B/SfUJd7FGkbR5ZUMb1EEaa55jQJNeDWnLRbLa4SWszB/USIhDOYGIvNX9OFvJhsl/w9iQda57zdpMPrsL+CzpxSKAD5dcIAXcZ9Yv4tGLIIxmNiihk/tJwfgUMlp7SNj6xbd5Udxl99eo3vW/7b8YZg3TNl5ZBIfIo5CsJyeeyYg1+duJlpHNEayFxv0y+ZNHVvc/GZfTWpS9enWv8YvLQyHZcRv/ku3ej579t/uiyruneySKlPr3zD1aHHSnWLPmLwuRbsv0fTLqPaN+76+N323ow1UPQQrPyVTeyCwVLFFvFYcvBMJ/wUGNDQmhVw2XUaF7wLB3x3e3NfRR+6AyQsj6f0EM75XecBY/wvTafAG63kzgWmT0YHx1RHT8zxTi5ocqYoMh5gsCC4yA/j/rsCY2DXOY4Smoc5EEaf1cFGY2D/hTQOKhxsKPGAxHCk74ctotAIonLqJfEcZTXq7F6+kg1w4vWkWq2DMviNcjeg2qwO7wxh9MVbIbO+mCJ2AMhcr6xL5FIC12hMz0SsRySgpUv8jpSeNlqVNzMgg06UUQYNkhjfEYip+G9z9qRS1WrNfwGU9BXTmGAhI0jvLwdfYMzZuIvjCaTTtZcmAjs0RVGAU7dykcE4cDNjDoy/sWQjMuaRBGZmI5MmBbYII1eJOY6V/QW7i/gi5gKZkmzSUIaV+bW7AjkBG7gHCNUbeLOGOEYZDWKhy3pqYss+uJZRJiZBiu4pMtoJFQBDkZIivNAEp1mkEgXGMCcKMLZXICIT8DjlAgE0okikUg1IYo0zGemEaF33iR0wjMhfqTABolQBRFq8WYghhJYGcq9iHtpEkeAsLWpyEDeaDmh06HjyVDQ9cAGkeeDfY+qX8lyL6PUZYY+NldABqZnfDJ3xT8fNCsfB+ghDAEjiyKc+pIwFIngIwbLcjMgcSgiPBDGSIjMHDESI1FEeEGxH137HoZMxBDww91QqdTtijSD29RJqdNgLIFfGk1AApH/5ugDhMDoZelvnFvoKTF0N9FIgEnn9YWQNUgxDw7lV3CJl2VX0ffPHbj/W5P7jsz2zJYzl+dW4jL6gee8gkVl/tuGN/7h7r/8gbVv+Nvdj3zP0A8UqocPTSWYUo/LqNVSeozAzHqN9WCMsB1v+8HGojJWjE0L70/ELSXCPMOQzFf6moRbGufMLfTsyAwb0o0/YBWMhFzAxY743EB+Pt9rLTknl1+XKBIPp5HLZB6MiJxHEY2DGgcZbjQOahzUOHjejIPmIqjjZBHgfWowVYFU8MFx9GAtB2PxKYXQPDgb+1KEXShqvCFxeqiaOdTcaJ66YIZQRIpADodSVUI+jC7GG9OVVekKMtDCSzJmP3Q2eFG6vDZdhiUyC3FXJTcbpiPuDbMTs7a8Dc6rkDcYG7/8H30li/FSLe2jVyXB/vR20XmOE3KwgARZ8UNKeFs0TsiHDgapBGwwZBkZoJboEdYQC9PV0MmHPHgOCgmDZGJbcQsfz4XJ8PoeqkvA+iIP9FqCPdCjjbd8iJzzKCT9vZ9spqtdEuYQQkf54CyKuyk6TVNN7AABAABJREFUoQdRxiOt7oiu1wkMkpz63hi85aMhsJ2L6C/KgYDTIGAUkQihtby5L4Jneejsi3gkb063EIZF0CoIUoDFmKSLuR6AQichGmItLkO68bHQSOelLuPNiC1BGwcCCBOB+bgeL4ihj0QW6qQ7dAoCTDcNz+BbSwety/m1ATpz4jVdiQSmP2RolcFS3AzBAytCx5PBD17nGLpHKEUQcOQdVcLoKoNwZG4oQbiV3bkYAv5xVS5265A5GR4ujs7R5pXSGIg9WWgD4SvC9vHOipkxCAgOCBo4fMYg1kLibi2E8RIHLn63diqInRA2DgkkJI7x02m8Ly3zrwFYXEa52y/uWvWa5c/++UtfelvfKrahf8/aqw/Ojt3St8q/T1879BU0r+i5uHUfwkzfHcm+V5GODZAP+xAmB38Jm16w/jEJcL2Z/mw2YIgEh0/L8oNE4oHgNSYcko67KcvMhIjFzRPVV5qxCYp2xCmFftoMmUPIleXwsJk89y9fdbgcnznjXKscWa25PCZ4/vjDwuKhuMsfmbirtaDi5w0C3EUaB/1qahzUOMh4xIexiZC7glDjoMZB/1WU++GcGAdlITzp0RlrWyZRwxLYeKmqJ1kD5kgtM1LNu/1teYoJgTlWHM2kqmNhz3pMc7y2VhI16J/Xh9kwRnz6H+/fGAAvzZR3VBpZ28v5VenSAZuaWN9VzeKkOlnL+CuXe5CigXeyQi3NqxjxUs04G3FII0wMokiEev29DU7iNbaGZqWE1toLoom5DBHU0WYkGefI8iJhzDPJSliHhggZMTfaEhEOyJg2+BWlSCE3GIhsV4kQSbDHNx+yWDWUFNpwcbAZXpLfgE44iVfqjJE4z1l4CKfEnSKGsmaK9KcwkQbRqifn80CYA1wiPqmpkbiHaPYm4XfqldIAT0EG1gQEHpJLj7hSMZeIr3oKCYGY0YBIyRggnZjB8ciKdBElzvFcklPEiHvjXYmXJZGPK3HTFmZDyAwyXsr1eJzQLZOuh1NmIdLsfw4T+Tj1vkebm9vuSKcL3h1CZPg4VlCyD+1p+Igi7JKEHI6Y21qJk8KlaXUZJQVtCHgWCl0DyokQOq/zGl0MSbcoEkHgqcnHiUTOGbVFGXKpGm1kxXaSGH1ige65y18Ekk4CP7PvIyDZai383qEfgjdixaWUH04Co7UQcujptosm1sK+a+8ZuWdlfuhrhx8f6l3FF+f+A5u/MD6SH63kMpXcXqOCK7uGNxcSD4/e22XfqsTyvjfNTLylVtrhX6GwdujHGu6gcLbcNclSsAEmjraBUoixDKkbAC3OAjPBNZTNDOtTDYMtfqcNThgIZHMpmru9wQtOIGxkNb6e9rXkcwqHP09iQbrmv/uQQtyfEs24SWUzy+2PjvMLAY2DGgd59nJT87gmonEQKDQOahzkNjh3x0ERwpMepbtTtaFMYaZiG837dD6nZ9wEsCp2imd5mGUpo4gk+DYUg2F/wkTdUvZXbBEarH87m8Sv20omMADuqWYjG7Qk3ulrGegHeiFUE2GWIO9wTrFQ4u9zWCNZ9dTlCSGBhPAWJjr6ixourKRAFJFvJYdG/5oGQGJRM+khgxTjkyHLQlMbTon44bnEibgJMaY3JJp/3N10Tq5zSIgcLd4RyCERZ4kodCrIa/WlXRswAzpbC9sVmpcppQihBERQ61zx6xP3hyqSl4R1ZZwi4ooJGs5D4gCGmDMZbxJDmkcIkfFTJzwuiX4iTlNdgNA1gAsfJ4pBoMEYmcQIN8Mc51fsmxP3QUgQQwb9bp2DuGIQGw1LsHgDnAR6HLXPhUyyTE7+CgxfhG74ijKk8KF6LFokeimzcFovbIkdbyrtRIm3FqJF7U7PqD32lHjsL79rkv6lQ3cRRkqGCZFTeoTOK/tsZ4jovUmKV01Id6CgkZSSQr1O21wMJUQQIzESRcRQyIesVm1OX6GmXzrUoIvIINxK+FHoeq7t20iWbzJBhIMJhISABjgQcsihc2bcSp+aNB7LAf0jZAEhQp8xCJh8OGX5H6yFTCmEKH574uHdhf38SvLX+//5iuy639vxxe/puui2VRsSteQbr9z4/z3xT29b98pf2/SU7UOYfxZfltnKSG/2JdOFB7syr6lOfSSNBc9WgnkPYe3wLzYsfs11X+pTRuTYhcLWkik9ZGJ+YP0LWUYIEWA/w/RaW2zG/U5DIqowMNZn7bcVE3Nn0RhxPTG0L/ECB9+74+QcI7wIFUSOXPSESPhj932ip+v/6eux66LjPENA42C8oBoHgYIHCB+Ng35XaBzUOBhfys+hcVCEMD7VTyIyW0+HCYHJaKmDGY7WMrU6rp6wxNKhWg6z4Vg17UohhziC7qxl/Bbhobm9nCOLDSGw0HkcNtjaAmSgcJHptdItbi+IWYP4USa8ymHri69rJDSIX5PvmVS94SZqTLIZJ51T6mJ2IEUY2EhxBkhkTrz1NMiYdCwSxBOhwQ1jI5oRCOZKU42kUdCmKZLqynilBeNh1OwjKyG5gRjY0pphpElCFL1hgQeuwQeVdFiik0OvnTgR44GBAhEP7MgqhVKSG1dDjR6SrQwEakER50Ue53SOAKdzBJBxfkKWsx2nXqSHK2ONdB5LCkfkXcSRd26GQKCOpJhVClVwEhgdULCASuQ/XgQy6UY8Nnn/F+YZDpj9iovYOt8PSWTQ42TJm8QpH7I4qJqPp0PtiDvBI8urAyKQfN6ATSy8buCmg8WDRJhNV5tI3X/obgi8Q4q8g8nEzvDjRcK8efkpoZ4M6wNZXSCPDErgircuv+NDexttiC6jJtRskjfY7ZPuMgo1JdcNvLSfpoKhFWgxKhI/WBoBw11BuTNnn4fp64te2X+NW1aRdLdSnzGINu6r1hmD0D9YIlQQQujWwvpkkp0tuZxru1YvT1+yffrg7z7rBz6+/Zt+gUdmx17Ut+refd9C81/t/sCy7PTdI9+8svvqTO3eRP7HK5VP17t/LxmIXLL713D7xGW0Pvswm0nYojLMLUyvSPbdUR//sO1CgQpu1u5fa+w9mHsnBUnxhWSQ8UiQMsfRRoSfP0rrLd5gg7Z4aSPr+H/sW9G8RZHiGxe+ngsXILc1w7/anoKSmOvPN0vhf9BfKr53YuLmAU0jbIXvfIlrHNQ4yPMz3s4aB4FC4yAgaBwEhHNxHIwkNn6pFXkaBPqyw72p8lC6Ym/hDZJjRXgDZj/6sL4LUwqTBys5ZushxnprDBsH8DNlI+ywTkwv8/jCsa+a3VvpaugJMxKhT5Tlwys1HilIEXfhGFIpSa2pvJBRO6SRUoE6UjYdjHK8xzWE/W2PEM7mcUrZm5y/6dWNy0HbaECMUDVKSPFWkc6HON1BgzWS0uHjWd5UU0vzOag7GA2seLO92DO9I+Gdkr4Z6aWga6YMkhRmZiMdJD0WtKaGA0rw9Yn7iEJXYIk+O5HJh9akpt3P7Ugu46V8diLMxLMoC12B0kAR4SoeYtfyLkCTqIXhzekTY55/3PAFcUJnFCDig6KzlOMNkKRHu5nrpyWU9SpQyG+KJhM8MIl4b2mDj7KuPBIhN+IxR9GWRZ241zvlGpBBCRpcxq1zrTJel8NCX2iAy0RjIGZJAIG/ITOcHyb8zsTDVAFonz34MU69kQ4pcT+FK7oAoQsjwAfMXRUF63UjkwCOHsDnEnDf0mBqJIQKIsNBk6LLqOND6H2nj14dYtQSW46Ag/DDa36KLJ8xiLUQC6pvqsH6otgJ3VqIWynp2AwJIX4sIto6Y9CthR5Cy+GKLCeLzpcM3b63sHdVfog2f2LX1wl/dfMDd00e+NLIpq9M71vTvQyZt619J+Hr1v6vjcvfTiSfWZvOvCaVXgnrcysfiY0Dzma70t/fPLctB40fBrNhc6PCD8Rci7CQDLMQ8TXl648rqR+5d5ofqc8h9JTWeENosT980fyIX7F4at/lY+cK8isVHy9CVpxnyEDCh3R8ImCDRDjlkcenK6zb1KxEf88TBDQOahzkVubBy+BCyIdBhxR/UHvc73UfwlyAdI2DGgePPgQ1Dh7F4uzHjKXoOFkEZmoZPD95MaIgC72wZoytGmorweRcVVh6O80rI7sR9qfKeJq1vmz5dn+heIPkMLRQMJA3GJYxLv7wXuX8xPhSLZVJ1aiCuBkDnXGFykg05hXexqiRiJciEaWkkEbtUDtKWfFA2OBOrIfJpEdnXK0hnM01NCQDK0MAJR5C3kLNFnjBeOoRirtwbEnU6SnkRiXEKUXIH3Ktj6E7HhrhDKybXlhWMJx6fwkNqNCwQBHNnGiukokkLNGznj9wW5wZyIBEFvwExkKNHiGEYJBCaO1IRLtWyrOgGfzmh58n3AZ7I0wJMkMWkVcNvwGmRBlb9bSw2S1s8Bw+pinBqjlsz2AFbe/BQPLhQk27GfQbUn10D0bbVp4lbQp7aQPtIY4VDg3oQQnKSQEL2vmN8fvoC2U5Z3z1IZYUN7ghT5s55WONCJY3wjgMI980SzZkHAF3T0WSbtJBTIKEzgBJJM4HakfKVf1X0yS4IhU5DTNIJ5BKgAkCKLSTcBAHJQR2HDTYOYg7dLcN3XH/obusX7S5sIe+u+ExXGqTnOMySl2x5fSCDnr33ajofBKjIjeJry/ausmEO4v6QjumGutiWJTVKWJ6stYyY9C2nWRPQt9/AkkqpZF/uftPirXsv00cwGV0Xc9wvZa8ZflV9+/d8s6rX/y1h/5quHvp1d2XMl8RV5m9009OFO8fyL54unBffypRm/loqnYPenyjiNpBtp3Az/PXjm4/WHrIbXrhDgzMsEnqwoIxDUtgY9Igisxl9GMNB9HSB4Lh0VxSyTEjIaTRI/bnmINFZeI5McPdrHk269jT/cvo8daQ7PjM4a6L33+PhFwTJ8LH7P4mb9/lfP7ncni66jgfEdA4GK+qxkGg0DiocVDjIF+Ec3ccjCN7fLIp8jQI8GoIr3Ah2CBUkL2nB5pGP0+fDq9Mbg2bCpMJGTAomE9V+fikPl7CgjXPZvdRitcsPkyrI+TUX7+cPnEKGySOElsGJpGAH9pp+Jj9LbzPkWF54Qh3ZGONUHKdfRFBrbEoa0yK5mFIpGaoBR+/jYmUmLjYpH8UNOHmLU4upzSYF0fisbjn0x0SYb+k8yHu6XQ8WBpDbijVyLVm0zZbATUIQKGtrNUSmCFd8dZan9waSd+tFrNk0ioA8b6HTh9lp/G9FqIIRUSMCASYCEZF1q0JS9fgNrnBC0J1YDh8OCUOhfN0T4EgQTyaExRtcVTPdZ4D4UGALkCEaDZZsB2nTy5mRrawYifaGDA8izCyJiKko5/QRpRwkIiM0y0i1OVtgDVRF1tEIOW/xVKpeXLWU2RFsxunP7rmpwgxqQEUbJBS5NJIp6yUgp2SiwxzAjl1Y6DXTjOompDO+ie2FgFYHE1ClQt7U11gWW6ILFflkDqGQP19w29E/rVrLPQDPUQQ8Lo88S1r301TaRLNg91hxIsuowg4ywUKqptjVIQPk9u0c9qFaF1f1JXH9UWxFvoCMyCDqXBj/8uoFHsgBkMkieAsCi381uQ9hMwtJPEFg7fdtuSFb1n1qkt7VqzuWbZl+pB935L1Dzx+9wv7Vv/mln94orD9O5MP7ys+vn3ma/3Zy6fL9/Z1YbhOpHvemOx7HxrMF9RWGf0dpgtiD6yXdoQFY1hHdKPlskO9LSfDfoN32jKkHGHvweBByhqkt6dWbQqLkRrxi0uPmlj3i4wERg9SS1r4ABRrcvMThSLZiyke4VFBln9I4WsVTIKmge+2f735WcKsgoEH8vsi2+mkk0lCi4NO6Q8qdFPHeYcATwy+Mt4tjYN8pXiQ+hdL4yD3BjeGxkGNg3PHQRsBGeA0DnboeCAL4UlfGF6Pru5Zvz8sBOJ7BvpMwkhCIFrFGgu62Fov/qaVT1Zn2YoiWSe9J1DH7lRlNhBF+CGJNCKXslKMJa0NcvYYWGLCt4vw3ECTbDR2jmdEkRe9cHiKPZHdcavpookwIuR6BNlyqNcJFVUwuntZ3gJbm0EXbOzHjABLNGMCXqMNvsqpMUNqCik1WsHsxEA7Q0Vm50StI4MwidZUI3L2ckl70GyRZqvIdYVwThcOUoH4eZEkTmg1WtLobeiUE2xSaAedtvY086ndgSFCroeRdHFKKcyJhL5ZIuwRpgFFgdU4w0GYuIekeFk/pRRZLkwc4gSxIcXNhqQ4nYM4kcjHzWuu1rO8OJIcXpxcV+KJhE6okKRSlyEknRTGG4RJd+ObN490IvT6ycnHiWPoAxXYJvE5tcQqcOMk19vptRDSWtRSRexOa0W0yhsTlfjg9zd73k9KbcLMm88P7dxX3HvtwE0symLpieQ/7f0YHO/TB//RC16WX48erLiE7DWPmfFw6RBZYZpio19khepsNqNzYC+LJxKn0RDqiW5UvKL/anQeb31RX/EVCyHmQQ+xHB4s7XYGiB63FkIL+bi1EPsttz138zfHvlOppv9y9PFCOfuFQ6O3969c17f8w4ce+d83vPkPdv7VDQM3bZkpvXD4h/LJ2tYxs9A27sSwUigk0OYEpgetqdUtDVdPtJbMslevQur2mPEw9zvk18PCMHWfHIgfqRsArWRgg/wJAhaWNjUmDTonbK5VE2SPCbAQNr8Zlk4cgufu2f6zFIn00rPCX4/bU4l0f+ElCRJoYdBAlA9xTIKEHGl7UJhEGs+I9CvT6SWWquP8QoDnucZBQOCqahxk+NA46N9vjYOLjYMBI42DHTsUiBCe9KVZ0b1h34FHE4lV4Q3p6AsWDNDoB8YrN2u1KHYfUedFuNmQ42yQCCTQyaFzRc8lHSrouYROz3LJaimswhLJpIsFOmTDknEwm8Bj8UgQjVY124iZsVzLZG0HRZgqpIw3OYStAFUENuVGRVvUlBRCZEIEkWNYYpUdFgNRpJUUNAukc8J6Mpuqop9K/a0RDXNyUWg8MGVVeC5tDvLW8sDZrFl+BAJpZzQGGQpCjJEPSuiwZTnTs0iIEeFKcDjfs34FhhhUmWZPN4lweHFf3RQdX5u4nxCWyIdxDvYCq6UsBd371Akb7IhcD52JubaYzikChE6cKA4pIoXiJCLmIblE4qxIVHk6AnY7mZIriF87sHHbQSOuCNDCUPUWUojQwusHbvrMwY9hJ8QQ5xrojvco9mJl3lbiQcZ7gWY4W9BpFJoq4G9OzKgFtfNdRlF+3cDGfz74UZgbcaejCNMjNEMC0fPq4TfQktX5NaT7QZyOo7/VZRQxEl88dPt9h+5GbFXoFP6925uupw6pe5ZSUaCLxn6DS62hd1nXeuyElHVrIRGf2+mVYlSEKO4+ZJa9OeuLsm29k0CXdDshBkNgXJ1bCwNkPqGvLOouo24t3F7YzBpIa7NXJRM5bqcfGb6R9UVvXLrhY1v+bePwulSqfrBwBJdReCNM6PGxeyeLf7MsMzk2/an+VL06wxcz6Ra8OjzwMKzPLqWtNFPdwgYSLC5aO2zNsRTuz6nP2km0+CEcuJ/7mpJjrqccThERqzY2KmykYC08DieMv5Q4i6MuDv8JJkQtMCYXEj3upzHkS+1lyXXXUOyBzgNT9v0zWggh5DlEIr3Odt2WdAJsWnWcPwhoHORaahzkURwGIwv9oe23eEznlCxCjYMaBzUO+rejY0MRwpO+NMu7L3n1mv/18mr6G6NfgZB8a/KLqIBUON+rJc2h0c4bb1YWgcbAkXCGdBLlVQaPTSNdkRzCBrEfOid0fhhPKeJskAjykS7CBtkPin3nSe8OO9oToSwhB1VbwwJr4hRClU7C1pwiWpMQwTLgM4jcKzWUw5TmnqLG6aC4boLzBhMiUwmd8hQv4voJMTxSoxsczJ014MApqBDHHxXGSHoxEGPkPSs20sTMxbTRBdLpBSn2qtmsyZmkIewWSOugmRlpmct4xxHnooQ8K+kNsNfUJkX0xKjWhFoOxBjVSLBqgq9pyDSuyAhHXxj/gk4jipxiCoNiIUmcTxggj3Iw6BkpRnjCAAn9cIbm9AxWQ3ESnexR3CkTTA+OQRFCigeyGloRAhgUpTyXkPGG2l0Dp87NXCYytDAmGVdEG6TOmSShzwCEmLlCOkIbjmflezRUhJUPJUBEFQhTBXHWcSER2ome2FpSaM9b177rr/f8yfevecMfbPttTjmcDcYGk8KFc+p7ZTDxBSlb0sYvhOshkQjzVfDU9XmbhL5uAVnuX8p8QgghC8x8aM/7oH/PSbzEVbG+KBFIIFD7Zo/RWujepOT6RhQ4i3oRrIVwQhaesZu5niykcitywzvHJrj0sEFkfu47n2IfwpfWlm+rbH/WwLK9xcdvHNh4Udf/yzTCvq7LuzJrswOvTpS21afMssecwLC4aJhjySIxmASbR/AXHUqUbrcVaHzvQS6neZNutBmDxNl7cDasIMpu9b7/RChrq9H44RzS5xA2kub8sS+S3/B8Dfje2J1t1I6/cw+XJPQvnp8ixDeTBxkRT4H+EW/yQNPKeQZWaN++ZHLmVxODP0pcx3mGgMZBLqjGQZ78GgcZjzQOahzkgXCuj4MihKcyTG8YvIFi1yy7lvBtiXdOliYmyxO7prbyKrR56rGHJu33e2cRvOLCXjBqhalxxhNINybjdrNgf4vshUQ2DyRkT3m4DcwQOoQoJkFCXtlIYYVSjyDG4cyQEBKIDJvUezohRJEQrui5kTfyUuvmwaDSJuM5GYPIuWGQFvorYCWkuEKUOw0jDK2yXmD6gxNCFzOB7yHjBy10Dmk2xsb+GWQmEUPA6SsRqCArqfKh0e4FiubA/fheNSKINfAKzQ25DQxRjlJLofYwoYXQ4fXitDBt5kprl1UcjkgRSUGA0+iLS1nKuwbKkOtFYsRPEXB+glmvNYtEHxicfSHmZA+LmRf0MLI4TpGnGZHt+KAClSI9sDvneCYQZVCOgLMv512e5cqJU9Yb4BxyDjdzse8JTJLuHWyyTdIjf7skZ626vv8m6CsReBcK5/+66R2hOtdJRUSgfIQYA0lHgDitdcKJAyenn9z7MbIiJsT5oN/bTBwZDrgidLQeXE9JdDsqeEJZXcBxc6uj/+DiJND3q2A/QDi271rxnYmHdpeeurh0hZsBmTTIhy0oWGYmkkDfpmJv8Unon+9G6BtRuLWQL6Mfu2f3z5RzHz/48JGZ7ud3XcT9xV3yy+tue2D0iVevec4f7twMCKsrV6/tvWqy+F1+bsmm1wIyJrJ6eikafE3R+tT7zS+0730YBm2h0fE31sdNvTmOhuPo3oP4kc7y3bDnCdY/W3q01Xe06TJq5DBaC5EM5segaW7QuqiMNcw4m301iNBHT4llfA9VTpHhY8L2A29DzMyA4dQJoXmH2mn88O2mUJg2GTUqcn4hoHHQBwuNgxoH48ilcVDj4Lk7DooQnoYhuj83wGdN70Xoet6K234k8e6p8gTxPdM7JkpHjpQOsdzFN8e/yHKDcA+jVY1tAI1FQRGR5D+vZRzQM8LIppyDsd2TpfJ+mWIuolG+3nTZ7YpIdiWN+HHABrtSVeeEyEfeZe9lHOGZjQayYI+wRGqPVkfyjbzZOx8t8LZYCg3wdHP7DDLkeSOd3NIAFNMCegWpc/YFoUWMl2Xe1FFIQWZRoqFUayhxtVSElTK6wiLG4cqJkEtTUUtF9gmqiNAep38ujHaW2PGBOfJACoaUxjzJoI0gUEtrT0A5wO69oAteHbWEsg2/Xx/nLMuI4tzDc2MqYs7N5hBF0t25Ec4TyRuRGGcsIe6MyMmeG81cwEcaN68hg4DXSDq0Ko5DJLoGz/WyhFtHtjBP7+LcBqa3fergP0LzrP/1xKrcmq9OfInrBOB7WI2muOUFtiIr2yqswSO0PpEYKY7ADNlogaythWOU4Op5b9gpfm9ojBd3Va8d/sHPHPoYVBBVERmoKS1pdRl1Mhntn0ge7bJ9dRJkIe89Ig4dJQ7gcELmeSJAx/n4tExCLJysZ8ONwX4VoI1h0LSERWhYOIctByGBnLInIUZCZgxCCH19URKxFppoInFz/8tGjl1f1K2FNJ4bA5fRNZmrLuladVF+9VBy1d7pI7iMPjSynYJY/r8ztunq7ksgnz2Z+mf2/tyqzPjS9PSR6U8tzb6qfHivfWnhdewrmNtouw5O3cUG9Ikpo4hhhv0KbIC2Tkx6BbvS2wo01RFnj7bpPNMIfdFRTH9OArmCjZTbURsmHwbSSC1M2ccO6T6lnM497C42IgdS9mW3bJsbaV80u8M9hdOQY8ZASzemZyl8Ig8kHnxE+T7iHWrfjyBgp6YuHix4o+PCQEDjoMZB7nSNgwypGgc1Dh7z1D9HxkERwmOu2uk66csOoOqqJddHhbDEydLkRGl8x9R2Enlt5dWWV+0G4QkMhZdOd+N0pgFr4lWN1yzjL4FWOT8koblxhRG2QqCIzrtI70mzXI0tXeNK4IfFkEiE10C30SHjvBFa6OTTKRwckqc5+uMB+6JVnGIJjBmQKGd0oT2Wi4GBRP7wKKRHJdbzhEOFNqMviKVCFUmsndRILmvsIEx7CrVsZJ5GINES1qpp0Mvm1EQagDwCYMFrK63hBZ3W0s3Q5mB9tRNYqJkNvfumLMiADykoD2/Cxn1pbgPexgzGxvqu1pnQFyIcFEcShY4wEdIaOk2zHaE9RwW8aisYNg6hle4pGpY/SX514n6IFn25PL8e79MXDNx6oMQqpmaUc21QI3KdX5ELs0KeFWKQgYBRPfSM3EjhMPfRhm2FLXhOOjdDgEZGDc4h3Rj4SFjcBSoIS6QULAs9kUnCBsmiUhL5IIAYp05TIxf94sG7yfrkwX8kNzTj1qiKCK2lJTDJB8a/xCmHt+THL3rnX+7+AGTyr/eYJQ1tbgWFbjmtpRnEnQq2up66OdHFwJbVSuFdzGYkRLmXJfdv9kAUbd8OTItvWfuev937Pt+IIjQhsTxnkychitwhraZCzyXEsxSDoc8Y5NTthERI4X64tu/mA8WDy7Ird0+NcQs9MPpUrZp63/ankql61+FyPlu5aumSJwo73jH8vWPlfT+87rfGCju+e/C3Vw7+YGHmIylIIBe3enus62gkvT7MsDfj6tMcML3uO+tTdlFgiYn0kLmPhrmCPvOwURx+2LQ0zldo9274OGnzW85/bYopCMADCbH4WV2BE7oZkK803A9jIOluLUzZvQ8JRNSEFzhY8EbbTiyAywWRpHGQ8ULjoMZB/7ZrHGSQ0DjYyY9+EcIzd3X6c/181vaZIfGFq27xip0l7jSWWH9y8glfUjK8qPFCb5/AcIwZ+kGEL1WgWI2wmWOOpnAtI34I2Gw9W+kUWoLZ0GUgY7xPOxUkxc2JMZf0SBSx2hXCHL/od9rSAHtPzDAXMcH6MUbSXHlokkmZLdHoky1gQxgoromUzbMzOVPNBmZlXXOro3NR+J7rQR3OqxV74zSv0WySCYdGIEu2XI150gb9Jks17CRhkebbKE0KfM9ynbO5ObHRRKYmBvQChs7aTJIiCBC2+o6STdlQEO3GRGmDSQfNHkEPYqGdxuVoJ5WSZn/4azpDGCJIWl6Qh72ga0fJCMzXJ9mc3dJhdEi7NS+Ssf0lM9lB0qCOpisIIONEjlN0fmXcKBwRGJoLuDHQZTD3Qb0iwUPAKRyJlOKUw2WoxU+d+MHf/mL3B1wVxkOq2lrYxge8wWpbcesLB20Go/uFuhJCjJCE2w+aRdH1I4MSWz60OEoWZBJz5SdHjExyysEpfSQCS3R2R9xdTykFVrEKkw7TKYELfktWw+o4YensV8Eap2xr8Yfbfgv31EtyV/r0S9+IAjaO5fA5Ay9BEr7q+xlaMRbFKdkcPKYUEnoD4uYTLDGK1yh3HUb+fcV9O2YOfGti/xXZ9V+ZPoTL6LvWXMN9dtGS/gdGn1zVtfyq8qVoiAeNHJv+SB8/CpQeSrqrZ+kDsDUmE9phLqBbzGzYjEQi5zMG3VMUg2FDIWLB19SMhOkhS3SdpiHwyabXKFtW1A/f3Sh17B/uTBgd1w8KxzcHU3vMJ8Y92To5kFMTti9HMAwSNxGjgs4DY25UMjdiJPa5cxN1fmEjoHGQ56fGQcZNjYMaB+PwFyMaB8/i+CBCeBbBt6pbR8cXrLr1xxM/CUUkfffUzrHSEZbgN3fTiXvDi5uxHeM8RjB4M+Ntzt7P4lG1d7UGb4EN5lIVuBN8EgshdItTIgj3pMvOCWFrXSkMdEfvAdIjP0TVkkzxcLkr6m/OVKxDF6ma6Y6wQTfuIUOEH8BokC1O0/QdZRZibBIycCpEKuxUyP5k9caURfQ4eQtKGivcGLkN/qVFjJqJeoZ32EAUvTEuD40KbLlJJUOe4WIvsgZO+GGyyaWN9Vki7Qw2P8CkgAWh1cRNgP+hLB2xbG9wiFoKZZkLCpI+39LKmNcr78emJyAAwmaijBeGdARMc1AeKqWcHUHMipPFaSie3Gks0YoTQQBNrG9JwuVd62GMkWIhA6GCazl1dAqHm0pY7iVB6LmEMDSnXpSF8kVjIBE3BqIKefSw8QaJGCFvGLhx68g2+NtluQ2PjD8CpX9g/MuX59ch+boVr/+nkY/fOHhTYpwzMzDuK5hr6LbiNrs56ykEHhl/eHWXu7YaDKSPFA9GMrk6vxYG62Sy2RhDLJo9/WdUd7nBWggnhCi6yyjd5B3CKg4Hrp4IfPbgR9ngPpgHLfVTez8KqYt7JGItfNXwm74z+RAyrDfjVNAXevUfX7gV0eMTCymOnfBw8QDri2IYDJUkPn/o7/nd4aLcldxWGwdvmC3l1uaHf6HvOlxG+QEGmb0zR7gPv3HksVzWlqsZSM9+eMum5ZlSqvbFZOZWyCQWPNs/kCMs9xIWlXksOIuaO6jtOpi7vHb4FyFyyfSS2sHnsFFhPSxCAydkx0KbZ1i6n9LmPjplTM+YoWkzf1E7DUvOeNzSq0csPP4BkePOoun2DWk5OOH2ddZHVjg1EsgvIoEEWsTiTZbYUnThaHCCHVw4T6lCoAUBjYMaBzUOahzUONjyUDwL0aNk4CxUrioXQoChkeSrw4o1nh/XrdkxtY2UTZOPf2PiXs/ipY4XO965LTQeYezKD0552SUO5XBbnKeziIub45wW5oMPJ1luMMQQlzOGZVrGKnnXQHwwXXL5gXRpopojBRYE8XNbJfyNOG+T7g5qquppahnIlA9X8ix/inGyWM1MB3bndImJf+xbiH4icEgUosFDNLtPrHcF+XKw1UGZqA6jJSmVWjKXDsZDEDBO5awtEGXi9jprdIrDs7y1WEjcjhdaS/tpsnUEQVpLIlgxzRJnVCKNuVXmv8rOh7aMvls7w2qxGCrTDZ0BeLJszqRp8F0xrO0uEDSHiuqhG4HmWcvCtUsHb1ukg3KjRhwGUeCVroE0/zE1mhOdB8L0nOzBsjzFCocjEDxcUre+MKwfE0x8lhGMgfu+Mv5lqyKR2lvYR70U5xQznVVeZ/bgQf7+08gnrEAi4VQQjkeciYUxRAlZVOECsMRtI1tdYAVmSXMZ/bKVbx5vv+gn/nz3B2kAbSbEp5QPmVhKvSPwQ5qNOREDo60Kk2PrixGa1OoyCtmDBGL6c5ZIcex++NMS8S0QWfIOpuf7VUAC7z8Uqq8n4+72yLu10Lc0hCKO2p6i1nVMhXxwKOWicOq7Eb586Ifgk9f2bdxfOMhWHBd3r+Ju+PLBp+q11BdGD90+uOIrR3a/eHDF85Ze89DkI+jMpqvPW35LtXpw/1Stt+tFWVDOXZPEMAip8/U/neARZylRrHx9b/KdCWGD1lYzAA5axI8mG2ycBjNgnG3onDASRT9t0MWmgviXHvJzh92l9puIJRNwMwaaZ6d2ywbLoZNA54HOAMkKYg2DoUmfyHGOTJw4ka5I5gwjoHGQh7/GQY2DGgftyaNx8Iw8f0UIzwjMz7iS1vn6L1h561sTZnBgddPd0zvHzZA4yvQnfN7CO58xQHLDG7+/8dlpOOw1EBZkL4LBrgUFgsI5P3Tnz55UeabGSyxGPHsnZjEYqBX+LYxMZh1I1mGDTh3JXZou8rsma5+yXqgb9EjkcOpIZKKaRTlr2MxWG3fautwMy+Hsq+RWZIv2+slCl4mskbRgsWu2vGHqdIoYVBqR40N8tmoepNC2QjVjXIs4RDFwOfieddgoYrP7VjigEroDx4OCuh5AoiwajJSG3rHlPQbJWZvTaLysK22WQFRSiLGZFA6K8HGeTArp4U3Z6oM0khWkLEDG6Fzwnq0FhlmzyVkhy5pkAkbUwys4ERMOBNW4oFsyjW9bXXA2lgCymvx/MCfCSLEiop9LyTorVNFK4Shyw8BGDH1QLyxyWwt4eN5CIha/QOHWmxGvsNdZnNMzTsnyFhI6f4sWP1LIdQGXRCGRFw7cQnHseH541pYD29Z1rWOSJMZGzI/uMvrE5BNQR+yHSGJmdBpJ/NJcwzJ51cDVe4v7IIHwQ2YbukIoorNZasFaDr6wQaZZ4kcaXUZhktwqrLvzt3vef1X/1W4ApPj9h+4ixFoIjQRxykLYyGWZGVS5fvcddYq4u2S7EfJh1RnshL7EaLAWroUTPjW7nVtuvNSNy2gymbtlxRWX51dsmRp977Wv/afdX7PbIHi9bp35xuPj92brT/Um60emfmEwlegqdGWZeQeXK33AFhftu8Psq7DE7puYQFgf/626kcO7a+Ohy0QO/8+4UqhtKRHcSgkbq49yQ5YwA5ptsJHlPfHQ+KRpa03zOM1jKiBAhYVhwi1oGSTb//Cx5WE45dvoPLDBDK1rln5CB7ZQWK4OIdAGBDQOahzUOKhx0B8tGgdP+yNWhPC0Q3rmFDI6Xp27Ltb31sS7fAOM8dIYLPGpycfc3GFkBtoQ3ul4LTb7WiAWXtAYRXjb402Rlz5ng+axWU+xlik0DzoHOYTxGCfkxRGek6wz2xCCMl7NLc0UWcPGfU0JoVKURbPHKeu1dAX/VWjPjnK3M8+JetrXQV2TKaF9fzW3Ms17rrXsYBWKSGMCawpMjHpTtj5qg8tRC61laDQzXXMjR6idbxOMEi+LKvPDZC83ZgMaPbNWkUWXMTMyO3G2noHcsvYpZaGspiFkoQFu5mzQCTMVkYgAIWWNQOLIGrbNoJm+pyJqaRg9AHAEXBjkySUMDbBmmzW1yRsbnBClgR+STilClNPlQAGNCYd8C9ytlAhdQ4b+etxlmNhJBJ0QRa72pw9+jBDqFd72E1A1hN3QR8QPN+gFGbMEOklDBuJ3uHQ48jc3BlLk+4df/08HP+4WRRoBE4PC8cEjlCbRcs9aE1xPwQKTml3Q4sEthW180HDL4C1fHv/yLYO34puK2IquYdqA5LbC1m27Ah2tmwPq61a8AeoIn4RqQgVhle4W2zSE2sqoaIMEskEF1kLiNm8wvwESGIkim1VgLfQNka/r3+hTCjEhtpJACpLiITZSX1qGn1f47uwL1kL4ITZDtqSH32In3DN9JOxDOEmEi7a+f/nHd3wjnan/2Z7PDWSL04cefuXwm19+0ev3Tz285cif9Hb9HNcp03Vz0lcNDY6j9dIOW0imymaD49TbWCGGHerZcpClYiCN3Xcmqs29B3Mb6+HXH5NkH8LxNxqxhPIZ6wtTEJ0okggJDDzQVhlFeqGDuX/2YweXxG4uk+IOCQ8GewQ4D+QUHmg//4TchdQsmhbZIE3SIQTaj4DGQY2D3GU81TQOgoPGQY2Dp/zQFSE8Zeg6sWDrD6jPX3mrN9FZ4s7gbrp58rG44zajiLM76ApEJYUx0ElIMJ1hJfTphcFyxVI0WZtDWLedD6FVrhlfUDghpfAIJSWEsK8qFJHTcj0JD0QJfCw8rMP8PSNpNlEQNuiTEvdWcqyJivxINQfF8q0pIIcY/Q5Vs0vTldGqWSxJOVDJEeEw1sTrcbrERERU1XHXZL9Be7m111toyfJ0uTeZYPrjRdnSnnJuU8U09Kcr/aEiHDdpcy8Ms56aqOSDSjxmKyTSjnyyhmET0yjp3lPej3l3pnfmEBsYHXwPt9Iwg8xK0x5S0BAGJEMR5uZ0l1ywgM2ShRjkEMxhmzQyvJCbnRBhyCf9tdd0kDJzq21w5wcCCINVWFUVTmhsk157LkXCx8TtPT/YMAkdcCLkuoZdNjsx9Y2whCnprcZANwNSPYwI3ojDJ7fExw98AjEObH2Y7OB+6ITawYs8/cvjD3gEASI3MvOwsHU4PwzT85BEuJ/LEP7ExT/xwV24jA4j7wJIugAp9BGBP9/9py8dftm23YEZmofqQVqC5RNqak6wXeYii70RKyIsMTlpgx/0b3VxDfrDTolbsByyAM+rh9/w3YmHqAXbICFWQd/dHjQu7dqAMCSQj9sS8Rr1RnLqv6FAAk0m7FQRphdexCKlBwqH6P7O2dEtk6NfHB0tVjKlUuaOweErlg4nU7WrupZwE24cWr9t6rvTYdeZ7sx61rt1Nt/wF61usYVkwjYSRGzuH3TOVx8l4lsOxsVjaAFxdqeIO0yUHvN2Qh0tPQiggXmGRhT9sCVqfrURX+gP935YOcZuFe4ZDmK+SAwRcp0KhpxnFEBfk1pf9BlBqMKniIDGQY2DNsJqHNQ4eJxHiMbBBYEJ3xleQnVcSAhAEenunmlbt4ZtEnGKgyUaEwnWMziPDycREl7KPW4c0jwbzajIA9eJipsBsZtBgYyVNY9oKoRiIU/yTHAcdUJoJMp3pwiWMG7BcC/aTEIEYKGwLzdU5lKQK3uvXpUp7a/krslPHajkkTcSmUhMhUVxsJWleSnPTddrRhrXBh74VOCBV2bKmyqZZ3dN7y1nIV196fKRanYskEwk6c6KbOFgpWsoU8D3lU+3zVE0KggHo17mWNKAqWoWWouf6nSzIGVJpzgRqOBUNdebLsH3YGukcEDz0IC3Lav4YH6kUyTyH2ZInIIY9Eq1TOtKrQgAqfO3sCYQ8zCr+HmCidNyH+QQcwpK6IA7pfSC5HokgyXTbLY2cdE0wMNrjSV2vNlcZSK0h34xk5OCHFwp6BnU7pbBF2H6g+Y9MvEIYpj+IrtrCtziRWBcyLhVkBTKEiJD6MU9jAIu4wLEOTwLsU+MfAITIkTRldDydV2XR+9WbIl2kyQSbN3ha+TADyGKrEDD2jnfH7ZAdJsh8wzZlAJh6CLy0EJCtwRa312LaUpAAsN2IDYXkaxrB25mf0JIIMwwblr4zwc/cnHuyk2z29dmryiUc2tyazYdGXvR8qt2j4+v7Vu6r3Qwla5dMjjwyOQjS/KzPZnyknQxV793aWZ6bXpqee7Vg+zUV7PVREOFGKxvc35o20WwD2Hr9oPgP/ux1PD7baPCsFSMxcc/bBzPDYPR/ua6CD09ksaw5f3RtUmjWDNSq3PPNJ74wGC3wGnlgc16bEv65OCPxNNOjtjzDuQ1DnbyRWpD2zQOahzkttI4CAgaB+1OOH2/h8bH1bk1DspCGC/cBRThB1R6+6wWd9Mfa7qbRkMiC+6zFZu/O/vuDpCZDIu5sDootMWeooksNMVi+FNi2TN7Y3B1NCuis0TkIHUUbJjako0VSqt1XE7NJXWymkOBsx1op3GmsIQMLAUC45ekWDXNpOwpdfGV/U6hHwLDKbkD6TIcydkgxG9Jqr6rVsfkCBXkFIHv6ZraWckNpStby/mazWnM4I/qagfT5fFq1jsICYQKOrnCvIkfKTL0gpAJhPCpsNBO3dkgPaIBGPHoOyZKzKSwPkx2GBthaJAK54dQZ3clDT63ZtaDm9FsmLNZNW32plkLnUCS4jwZzTBAwMUeSO2ESEaWCCaA3wDH5kPiX2rr3xB6pxw02oAG8OSU4m5OJBauYMOtFHlXa7U0Tb5WMMFq4Fu4Cl+b+BIFoVuhoDUYyfVdl+P5CX/z6vYV93AKhYO/8UE/ccLI/czEFxxHWwWCy6ixPsRQ6hZFEp0EOhv8gRU/AMl82fDL7jl4D+3GPumTDzFsYi0EogCLzZP0FWhoDxWxg0XLfhVmLXQSyA6Eq4JfKGLs60i4q/QUISSQkCmFOJGyaSFzM3AWJQXHUWbkfnjvJuIciF3bfxNXakV29eby6A1LN1SKO3AZ3TJ5ZOv0oa+UdvfkS0Mz0xv71wD4s5d833NX3IbL6P6puwbTzFBNpHtfnijdZrMBgZUVR8MuEe4gmgjLh4ZKQsAyM/FoOIWOm39p968F6ohhkN8h3mlMEm2QQ9ggbqJx0mBwGW0YD6OeYyMp20cerEDLxr9jMxc987oWFTmaibBWlDkKh2KdiIDGQa6KxkGNgxoHT+LxdF6PgyKEJ3EnnN+ix3OzcUMifWdSIm/JaZYstV/T8TIN5qZkY6tASB0yMCJCFjKF3uAy6rSNFLcfegjB8BT4VWCURy1aPJrJgg+xvgsRJIMGozWQH+Og4fC/K7OzMK4lcMI0C7okzB4YTIKIkLjXDGOJb5e6qI1lbKB87sg6kCpjVOxNVVCIQD+eqOXuoBWnUMhYklmR0FfjhKGd0Dknt8h4+7OpynQlh3mQBkIOsf4Rkus+pUTwkjXbXegLnAr260SRzrKIq/GremMtH5+aGIhiWN6mYT80D1L4pBM553jMeCzUM9YkupU0IkqIH6/lMksteKgSAheMjjYQMTobDtdAOUeeNG9b4IrGq52L0gxSgoxV40yVU1eIHyb8IUzhS2GgQ//mMCcQyT3FfXDpH1zxun8c+SeshVC7wBgN/y813EoBKPmTF7/jT3f9GQLrulhlFM9Sm3y4JZgTrd7gU/pnuz74+rWv/70tvxcanjA2aH6nDwSbpDFPTiGH2Azxa/UZj5DDP9/1Qbrtm2e8dvgNRK7vvwlJX4kUayHzD32TCRKxFrq/KHGaYSSwNEIKnBBToWNFli8tg7WQBWYwNnoLwe07szu/+tTHJgr5F3ZdxK26rm9oQ25JKlW/ZEn/wxOP+AaGFC9URu1yVzZ3ZTfUZ7/QWCQGv1CGEw6YW+6X8KhkT8Lo4RnNerX9RlP9YBlucyvlxN1HicADu3/Ncl0VF8A8SC3hxI+To4KuN7DNE60ivV7+oieKleQ6CQGNgxoHNQ5qHDzuM+m8HgftpU2uMse99sqYh4C72bC6Ke6mT4Xl/tkmkZdyBMNSn0YMfb1N3mIjpeE+42U6UAvjBsTJxeyGXZE9JygDB4OWOAmJdSJDllGSlgMlKEKDG9zIgY/hX+oibpyMIYlO52BrRPDPRB21UNblITZeHLMe6XzwEYWnkR1rdW1QR4pg0iTC2qq0PLJcpk0Gl9EyTDh6uiIMQwsKG4vQuOXQE1tzHaXQfauTJhF676iCnobTo7ZEP6UULQdMIOLxTUX+EEeeCCFl6TKNRxIKikK3OlKcw66TcW8rSBYCds7TwGy/hjntMYIYqINnodOzPPTqEPXqPM81EK4PLqMuuSa/Gmb4+hWve2TiUbKcBLrAjQM3kIhAcE+94eMj/xTS66wxgyQHVNDtjYFkmjkRz1JmFeJK6tZCQt8dsbks6g9ACCGNkMBAX43uQgJJQRszD5l/GBSbkwwRNxUS8V9JsROOlg5gIXQq6O6jkEluCVj6RcFl9NmD1+8am1jTvWz32Pja/qX7yweZQ3igviuXqcwkHl6SmRnKTq7JjK3Mv2BF/ZNLc68ZyAaCh1enkbfbsBDiDmqR3DVsEdFYVGb8jbblYPWQZfW9u7HKdnp9atkv+KKjyCf7XsUuhQ0eGIelyDD9l0tCDIbBs7RaxyubicHcJ2zmeUaP5LK7ziFCKJfRM3pznBeVaRzUOKhxUOPg4g+zc24clIVw8Quq3LkIuJuNr27KBhhkh20SJydK4zumtnMKS2RXACMhvP3apESjN8ZqwuMTasG8OJiFP0xZh4NcSBTLyZAFs4LhkEUiRIVEIljUnCPBYYz/BPdROAzvuM7lKO7cjD0SzYczVSE9mTIvVor7OqI4Z8LEqANOCBmkiDMuEpngRZrzIjitsyZTHhxTAyNFTQISiAwROCHV8SHuoZNHdHqNNJpeM3UQeTpOY5jLR8OYMUgRFmulUtpjhMr6Z4wFo6It22q/zhgTQzWhdT1wNtSaWTYclms6oXm85WMoxOqIE68l8nGSFgraKSWcD9OMQNeNJdIqZ4OQXgpSrU/mRJj0pgZvmrFiaCGX0msnpCF+gk7XE8ikmxadZ1pbnfX5pWTTeQpB9kjH6ZTw1jBBkQh+oWRhqYMxehEScUN9favLaIsPqruVXt61DoKHGAvMbDu4dVluGdZC39bC996gwTi+Ms8Qbb5fBREOuCIhEwtZGocQosjkwOvY2LA04nvWk4sZ/DmJl7CyKHEP4YqY/kYKB5dlVz5w8Ekg+JuRR39s6MY/2fdkeqSGv+htS4a7u4xG3zz4Mi731QNXJ2v7K9XdPen/J5Vdm8xfi5EQRmdTB1sPFomB11Vvsw0JOdhqKbiVzt1zyYsER1OPmqpqWGWUcyYN4lA621hi1BSGg4mC5QTXmtV0n44NnpaNIpyOet04tWo5GYdC4XmKgMZBjYMaBzUOzn28nePjoAjh3Auq81NAgE2E+aztw4ku8cJVt/x44icnS5PEd03tPMI2icVRzDJfnfgSKXAF+AMcIEQSrAQTnqowIMvigGbwNh94lBvH4IdmaoMCwQYpC5UiJYgZIYST2PYY9RQ8EPrBBwpUTJj5DjFnZVjJkKQoYk6QLCWs+NJsgBEwqGBoDHHbf4ICfDBgwsdIIQxkqY7mUHvDEEdfpgLzRDnWRU5paqlqa9vQ7FIizdxCaueUxjDbcICNOpjHaNY2CBXbKtpWjfSLXDxgc+mKETnmyAWzHk6h5Dq99GagB3xgyDSVXE45zGIa6DENMyYZyCGmO07Jpdmg5F3zFBKtg3Gd7uCga0AHYOkspBF6gwCVOesL+ASKbupDfSgh02pH1lxejTBbica1JJFPELBEp7KwOOL3jz/g7SE3HpgQIY0vG37pFt9/IpG85+AXUY9wMBsmMSQi7AbD4Hq6jlN6xH73rsSthcw2fOnQy754yKggDDDuV8EKNLBElioNXqNrcBPFp5TFSCn79fH72Z3CV5dBIc3GZTSsL7r2yv5rfD/DB8a//Ly+F3+3sLNSsccmswfv6F9x2+oNqWx1X2E031Vc07Nspr6H+5PckdkvDXDn1Gtdtc318pcabp8AA3PzMaM7uIyW7m+wxPTttgQLO1IcvqOxvRJipQ/UDoa9JSg4xT6EgGiTBlvZIFMKre+t41CYUliuGxvE79nm3Z6Bo0lEaUmy701noEJVIQQ6DQGNg1wRjYMaBzUOnqPjoAhhp40p50l7GBrpyTXLro39eXtgiRgStwdD4pMTj7tTX5Mz2Gs0/+FkRmnCYS6LgTrCiNzeBa8iDxsjRi1m8ZFopMsWViHd+B5sDf6DN4vZzdIVtyhCAsvVBnXpZ7OAwN8oC39DGDbDrEUoKadogF4igFUFVUSCTvMm7UZHPUm9gRbaPElvEi0yzhYYkaegjxY219Fhowljgyy1SghbcAskajkNaiu+wqevQONtsxEl5NIkt446j6WPVEGKC8DcnGXRVIgl5layXJLiadtHBNZqlBvmRpNweqwEsk2u98LQs9WAAtKBAUKHfGZjWLHGjJlMFjV5KKQ5xNoiN4a1XQdTQpTQW0X/woUgx1gIEoR0G0qJGP85mqBFAZPaPLsVATchMs8QsaY5cR3xaE4kjjRUMK5AQwrH2y/6STa0YL+KxEFPSHwxTD70/Sp2lDZDBbEWcr89f+BWJh9iD0SOUz5QxNpEkg0MScFauDJnRPHK/qtHRnfjOPq5Q//AcqM7i5sBFrMkvb6+++K1uaGLMmtuWrbuo5sepFtfPvhUOlPdPvtk/1RhTX78jWv+w3XLb9uerh2c+ijdT2Y2JPM3s6VEMOvt8UVl6uw8UdpkeIRNI/jLvhHsT5ioHgkDybvrs+tJxPkzNfhO27CeXQq735Dsfq65jKLTFpUJRDGQxoCriTdoYW5jucgmJXy4tPxwQCuaWa280VLDMX/l0mbOKfw111aZB08BOBU5HxHQOKhx0EY4jYMaB8+F55sI4blwlc6XNrb+gPqiVbd4t7AltrJEDIlwEjcD8i4L50EMaxa2sPBia5wKAV7QCSFysKngY2oBzMToBwu92P6EvArbFn9ESMEkCOUgBY9NPviRQpxMnrl2ySqek2xHEXaoh93VcSqFT/o2jEhi8ZvBQsmyMWFBUegR7pcwMS9OiFoYGtypnIAvGV2iJnTS8tBUszhBimgbksWaSTJ/0rhczYhl4GzWOad55BL3kMYTIZ1sQvrggMBOzczYPBAjapwwqMX4WQg7fMC+KEmuETbf8yMBwUubx2nTuEdBaB7tJEJf+NBy+KpZuuyftRmXV5xcA+CGGLU4mFBEsyI2TJFG+aiuIWZocxgIwIW89SpoI0AZBdFMOokeRq4YBC3YNLuNZmye3WYnybr7lGJj5IQlaliBBpdRnEhHi6NMNXRzItsn0gVfb+aFg7ewreILBm6DQ2Kj9omFbqmGIlI9xdn43tOv6r+aWYXYDD978GNMi6VC6109ecvyO3fs2czKNA+NPXr14LM2Tz2wsmv4f+z88paJw3dNHkyO1L5W3HfrkhWvWPo8fI8H8sXt09+5bPay6fJILrM+mdiUyd1sjU+vb6wLynowOIiWPmDbzYe9BBtLztj2gx/xtWcaKdjcsP6lBxsL0piWwOsakcAYkcEihwep7z0YzHT13BVmHkxwc7EMEVeE3zbCwqRmnMTR9G5X0I4w2fe+ZN8d7dAsnULgvEFA46DGQRDQOKhxsNOeaeFtzH+677SmqT0XKgLR3dS3SWQ38H+b/CJgNEhIIBJ+6nwD+yH3MRwGVoS9C7oC2XCOhJjbG3n6OpMhBWHIFVvPB+ufeZOy3ikem6T7xhJQRIQIoYXI+0FuNL7BuCCpuJvCu+JqNIhBRQhpALY4mIy5fVptDdpjMau9yYKCVTPmUoo2Y+vzlkMgnSC5vY4aob4hFzVGyRBDmwv7KTJAEZphXqbebJQ4As4SgxXUNg4h122V1prA4sAhsDtrPzB6X5zdxU651RSO6m1zIkePKEhx8CSLIijkQHMUIOLMFuUh0YAKlyL8bQbBLAmJNeiMRQYJ63CIk0jEmxSuZoOdhtoSkEBcRn2/ChaescVjQgackHRUsQ4NISQQP1IPbxy8ie3sIXufOfgxbwJzC4kwq/CfD33kx9a850ujd7HMDJvaD+fWfmnsK9PV3GQ5/6zMulIxe8vQVfft3rphcFkyW13bu/RwfR8W2v5cKZeu9qcrufpT/anq8vonl6TTg6mwjK1b5wh90Rd39Sx9gBmA1Aijs6mAzCHEGBhnGzondAteq3EvqgqNbhBCKGX1bqhgrfu/z07/R7tBE4m+VD4H8K1lvZ8nEp5sqfTtqWW/Ywz2XDv8fjVXAR1CoGMQ0DjIpdA4qHHQfsSME91bRyWNg6f1YcU4KAvhaUVUyk4HAvPdbJiUeGR250e2vueByTU/uHTHRLXrwaK9ZBvxwNExGJ/SgSDZijVmjMIiZ36esBTCBoUIWyk0uBNWwVQNP1LeAU04LOuCQiI+7dAZEephO04LoSJkYTbsS5eMSiXZc6KaTBtLhOowbnWnbCojJdAJ+6IZECIIWyAzRM13L3Ak5ynk03xS7UMbqAvmRmudoQVV1p580jpCFb7NBzLs5xH5HhVZs8PmhMiwrQWn0FFCJKFtkSU6hXMSSIhmPnQBu2IuWcG2GfSYbZB9MohzgBs6aZufIl8O+9cDCP3112fLs34FDt00chpzM/Jr7QcBWoKqYCdsMFgrEw7+IGcghALeL4RNpUHXWP60IW2SBhQfdFrNVipJq2CqTxV2pBKpjx74JG3DqEjVCJC7oesyswcm6pDARyYefmT8kTVda+CErEG6fWSLN4NZhdBCyOG1Azf5rMJLclc+OfnE9sIWPiiCEKLqpy9526f3fOX7Vr7gY1u+FVqRWNO79CtHHt9Z3L+tvunmgVVPFR+9smddf/dqigx03ZYr17LdL24sKmN+nusbLqNVzHobE+mhOmQvdw2Li1oz0isMDXxBWT8G0mgp65mPVx8PjSTe/QbbrR4vU2OM5m4aZG5382Cl8oV6+o5y5V/KxfsqdlNxV/AtMLjaahW0NoQjLJc62DzTXyEgBJ4RAhoHeXhpHNQ4qHHwGT1HTqawCOHJoCXZs4dAf25wTW5i19Q161c8efXg1h8NjGKq2rOn9rIds4/BQEZr2UfYuZ5Xal8NJRgJA6UxlkUEggZdhLExuY73dcsKS9q43Q/mA3ODNeHD2XTjrDPjjvf+sNxLurF0TbpsHp5hNh2+l+ghF1viRDWHVyochCkTTdNiPZOylWGi5Q1O4guNBhLUIELWMOMWbviyH0RDz+xF3siOk1sjgWbxw66HAFzILXWQWxKdcdEqOhJJIB2hs0lmPBrXMotfK82LLqMBHzizraZDFZAo+zQ9YJkw6W2jioq5H1rt8EOz8pkzKs0LDqL8Cc2IfaHxHifdLwentcAzrVPWtcbhfXQxC5vpdByZ4IJqBNl4dsgMcxTNNxV7aT5tVlMHgUsckDFAvGra7MqemNkZ2CMepw2WiMEQcyIskd3qUYt+JgoizLqjq4trCHcc3MwpkwkJ37L23fcduov1afBKHS0efnRm95rRrV+a2n/P+Ei5nE4esOv9wuVXX1Vbsr+057mDL13RtWJd34ZD0+ViZU+/IWbLh2L381l/dX7pNDPg3cwq9CPZfVOiOl6fhdcZ2cMFtBGhy92/xny8iMnRHefhkCxYyu+j4ah1/1oVElj9XLXyzyxjU6l8hmTQYwJhll8sTvlwwnkixYPbqpxFTwQqyQiBZ4KAxkHQ0ziocdC+RBoHn8mjZKGyIoQLoaK0zkOAF9zeVLk3UR+rwu5gMMYp+tIzV3Wvuapr1m0gP5JIQBEnq/nd2V9MpIe3TD02Utq9p/gkr9SQBBgCNAY+Y5GwOguMg6GFjeZJgQ3CDIzqhPmBECoID7PpIIp8ykZJwjb0tj9ECsqXTlaggmOVLhgFVArKQeiRwABTtk1TLQ3dgyhiUkM5Ifa6wNCYlWfsCyaWwSBnnqWNI5AZ65pzI5qBfogr6bSw4RFqxNJe9CPnYUIgDTCaR0vCvo7YykjBLkmTkIQucoo22oaljqqZWgg7hdNaXSZgkyeN74Upgg0KHSx+ZCGAcuyKbNpBpxCDKtNs0g3MurFE46ihg3SNdOeE1EgcnagwBDkhbOlvYC6kmRgdDD03GQ5jgNZ9SzOKbu8BxkW9hSSinFOa5y0MLQmdbRJaJMmizaYuNIMQzGk/LJFr8eTMzjxL1NoGldVthS3meRxqeevad33JSOAwlsPDpUMsPDOcu4jJisWKwcUOhHTkD69/9UP7dqwdgAce2js72tVVv7hn5YHS1unqtrHi/d31u4bSU2w1kinRq6PdMgthwMHsaURs7dC7fNsJMwCy2AwdZ6P5QBdtAdJx+mks0WYSNmcAsgINYo0jfXutsme68lnj6JlXFcufJh0UYIP5M+UAgsXSlkjVIQSEQJsR4FGicVDjoMZBvmcaB0/7w0aE8LRDKoVtQYB33KHM5LJU5UAVO5i99zeOY9dIhCLyWbPsOZhWvmfFbS4zVZ6YKGFd3Aq52Dz52IOT9/BCz/szBMNDRllLcbOcvU/zDh9MdoEgsQMhKbNhBwgU+qRBUthDAprkVXAKiWJeIgVJLFWxyBmJYjWaSljgFLGBdJH96yGKmBNZqwaakKvbvLuwwgpkBP6G+6vxIEgBa4NC4YjBc0iqwdds1ZxaFfbC0qe1DENCqJpS1gtYKw0ghTgpFglGNlgrGjilhezG0ZWu4PXqnp9oa9C8MCfQuVdQ0lhJlVKwKXiXMy4aEIgZJBOLotVLG4IvrFWGmAkENmgM0Np+1GXUWhUgNaLXcrgep4Kh5S15of2oQcYCo4LkUot1DVB4LTByaLMr4dVGFOG01lnW7HFP2lCjN4mCgUDalTUS3pgkSSk0GdBFXGdT1S+MfqMnk/qTHX/RkyluHdlq4+6E5XrbnrPk+mI5yxaXt/atGpkZ/5O9j787c9WHDn+7N1ca7p66eWD1slx9OH/xlX3Xjs2UVnbf0pfYk+y5M5VbVzv8i8HhE6KGO+hQogTBs5VOo5XPu23Lh9pehRtDlpFD+xEUcoi/KC6mNKNq206whb0tPMOmFLDB3K2V7DUzs//bbsTyp2kqaw1lklgGU9km2zc97TswD4oNtg9eaRYCLQjwKNc4qHFQ46DGwZanwmmL2suZJtOfNjilqG0I1KvjW/be8B+3veaO4Sdf3b/1ooyRHDvSt5cqd+fsRj56JJfdtfja91BEGMue6Z2+bs1oceQbk/dSnkTcFI3lGEkwhsPUvUBIjir3GPzKyQynTsbiFhH4amKFg5l4OnMO4WBoyyRqhXqGXIqQC3uEarohkRTWHcXCRgRhSAsRDkiO0yG4q1dHvaRTHJJWrGawB8JVYHGBm5mFk1zjS9ZyAwUNfBBGjHOGUtLN8skUSvO3NBMiWzgyk5DFV5HnwIpIYiBXjYJIgoJTPlMeiJY3MrTQFunxsoR+KRrkLdTb6ExTglNkPGymLfw3qDJBauSCuGY6Qi8ggZj1PKQN4IBKE7OL6NfO+g4gXooiZHnLQ98b7rUkAqapsp1FSsQpQgg5nK3kujOlmUqOXEJ4OyGfyVJ+spDHHbhaTv27NdduL++/deUVR+p7V3cv31f6yvOXfW93qlYpf6s7WV9Z/b/dy+9P5S6tHf6fZuKr3p0a/jeWXantv+Joh93zs2EGvPto+rExSGAivbQ+/ls2ENpO9L/KnV/BRTm9rpq9+eD4W5k+yiXrsUmnyWyCdXQzXfxt98FCMsPvb3cl7davRWXajbD0ny4ENA5qHNQ4qHHwdD1PWvVoUZlWNBTvaAR401+VTm7oO1ypZfdXBi/KTDaaW707lzGzyWSlvz8zWapXM8mU8apFj77sAPnPyl0XpX4s8a7J0sRkeWLn1Dbq2jT5+P4iG9BtwiEykDGjDS4cGJpxmrhzA1wCvuIbCSLj7Ci4aELXUixNSSL0qRy4pVMO42881+sJTIiQNwgYbND5TE8gkL4dIgY9GsPap7ZASLC6BYuWmd/Q0JWG5CTZ7SD2omLM05gbRrDpSr6LpVPDhooIGCFsmj1Da23eIBY/QigonAe+Cqdl3VR8XGkVa+RgFGUrDsqaGyq119ImUMtAlnCzDLUa9XImhgC2OFpFC0mhMuPVxziBNtpAJo3kONrucBrwaPBJSwj9Peoy6szWLIEpHBu4KHZd6FEoC5rmfxvMpIROVgGBTBpGLimwRxoGODW8jg1P87AFIi4xzQ5qEoVqrjdTKMPJQxUIhw4mcumKL9VDj7ozlclEfklXYbTc+8nJb5YTycf3bV2WnxmYKqzrOfi10a+uyoxdmjkylP/e7mQuMfWROm6ivnboLFMHv+kVpVZtYuog+wqa72h1hEViQuQ2FhpNdt/Z3IvCDIP1qffA/SyledhipNaTu2ZrxWLtlbOlzbBBegTPB3rWO8KIyeKiTfH2/E3bdheyDbYHXGkVAgsjwJNT4yAgaBzUOKhxcOFnxDNIbfNLwzNomYoKgVYEUulB5g0+Kz/1VKU7kyjHLLOWmJddAjYIASnWK8lE9mkJYSzeGunPDfBZ03sRiS9YeatnwRJ3T+3CkMjpk5OPf23C6mJAchJoBrfABo3gtXAZZIzwkZRsWs9sgRRjJjaLzyyB6OD1ndC26SOLrSCM4QTNkEkWp6H0TA1LqM1jZN4IjJEI8rBHBFBFhNN4oMS5kCsZzM14FrTHjGbmg8o0P0yUpsT2VQyNpEprqvGfwDyTOMeaeyp+obQQ8ybMjBZSahozpu1skfQZjJSh+85UmVjIvEoAaJC0wDOdORsGwU5o9aGxyQb91MOAkxNv47qNo9EqGkii+XWacorjnhoQoyXGroNOs4gGUhS6ZVcidMp8Sr32KhhbH82nFBJIiFpoMHUVqtneTJGQ7nAdybJ+Gd+m7w2j63QlBwJcONpAIr2YLuaK1dTobHdfV5HLATIwakxy3zhy5YFS9y+ufviJwq5875KHDn8N+Y0Du3aM7bq0e+mOfe89XO5+8bLERHFygKevTQ4c8YEt0EKWimHP+sdwB7WslsNcRk1yC2nGGFmqpvjHtfQrk6kN1q+ymQcztuBQgw1yXVpKn+4obHDwlxY3wp/uKqVPCAiBhMZBjYMaBzUOtulRKELYJmClti0IlBK1Jcn6viorMxa8gsaW3OGkUq9P1irZdCZRPZJIXHpaWgBFvHrZta7qBatuYQOM1u2hnph4nCz2Oje7U7Am+Ws4dMKoCO/uRvk8rcGPnFGQZLY/8lhVOhAexEmAdRB6DrkQIeKUnKphZmxsgAFpoQikiO3pYS9WBPsYrwpsPuErx7B0TY1lUVnVxugfxWF3fanSaKU7m67CG6kdDcV6pidVnq5lyZqt5bAWoo1KTVXYAIOy8F4+FCnZrve27QRxjIcUx8WUFNxWKULVzhKJI0AuobeNdGtt8DgN/TI4Qqe8m/Zbr/fUw5jlqDVkjVIaDgYyzW9aI4Gdk6DBbJhg6/WCrbUwLLtqNj5D0aimKbdOpTH34SCK/ycYehHIIcCmAwl0HKYrXVgLoYJ9mSJtQznFacBkMV9joVmYaCKxvHs2GWyleUyIdbw0y7AyxHpTTBmdHciMbRywfgxkjlzff4TIpd324agdvH0spLMXhZ2z7miIQPxYrtMoom1R+AaLT4XcqbtNjMO2sH+/eZ+mXw4oheJ7i3X8jZNLun6uVnxvdxJmzg8Hx/xS4OVOWyg2eNqglCIhcCoIaBwENY2DGgc1Dp7K4+P4ZUQIj4+NcjoMATNGsahMuXdtpS+RsNdrXo4bbQxL5MMNZnFsDJal9rW9dXuoF666hYreHlgi69Zsn9zG6ROTT3xl/MtEfAkW6IpzQVIC9zCTF3E/zOYVKJ8TRV8Dk2y3xUEsEYPRsNOgEaJAHUnBbxO2VjCLlBEVuoxmiBxGKkgaBZipaGuc+mYSiBj9M4dPpgvyCyuqYC19qSIFe1OlkFXuSdRZ9mZ/uffy7Myuct+G7PRIudvMhl6LMS/b4ZCWEFofzESZYD4e2thgA1XWKyOD5l5rzqhhJwxLw3JYS7KOjs9ahGFaadpWW+D50+SBlPPD1RoQ1MhLAP0Fsyo8l9shZLo2lttBJzJQO8MkGDBJsSJhmiX12oqjJFmbuTq2xCsI5x1kdivBv7eSx1pYquIWW6F6UmCMREDbcUYVu89PF3KD3YViKVuFh2IeTLPkbHF5dnpJZuJVQ99emSqtyCT6gtPmjtlhTIIbBxJEru9/6qGJS9D24mVHPj96Jc3+wZXfuG//15dlCkuyF39q9+O3DYwkxt573/jfvXXVtwd63jkx851EJd1Pgdw7bb2Z4ClqRLHvfSxDSteztvKteYpyydO1zfDAjK0lg4F3HoooeeYHVDBw1GeuSRqEgBA4NQQ0DtpjPjziNA5qHNQ4eGqPkQVLLfBCtqCcEoXAWUfAvOCSJV7ND1Tz1hhzt/MV+e8u1JNdmKQwkbXpVfjpOg9L5LO2by2CL1p9yzsSP+GGxJ2TrFszNlocPVg8+OXxLzdYE8TQhjTjh7UwSc/V21u880P6UjW2Y1sqhDxnNTnWLjXHSNv3ghERbgZLsUJBBuV0P7hIGk/jlFLkWC1NN0tzB7UqzIKHBrgTRrYl6RImROQnajnsimO1bH+6uK+aJ47Msuws66kuTRe3lQau6B4fqXTD8MYDS4T4oWqmkmfzBqxtTDVkGiH6jZIFAylOrUxxnC3n8pnyZDkPASubic+aREFcWCkPvUSeECobLH7mEUqzwccPTrwIKbTHEQisGloISua1S9Xe39BZw4CypCCMrZAI6QauJZvyQCwtnRYWbPEYs7Vy6jWG9rOujHFCZh4m07ZBCJV7Nkrgt4emenAZLVTTfdkK2siaLvem8tlCJftgYd3ly596aPKSmwcSh8vsjWm1+gGYSzIzE5Wl3zOwdzA7RWRd92EiO2ZWfHa6/7qeiY0DO78/g2nxyMTMRykyPv2penbpQOIDj05cxemlPSM4mt678/0vHh6CBNaqn6NZtuwpdVQ+253K5Lmkfjv4t6NR7TP+AxVkxmDfq1gR5xnrkgIhIAROHQGNgxoHNQ5qHDz1J8jxS4oQHh8b5XQYAuVErY9l+9PFy7Jj1rRgFWTOVaFyD2yQhHwy3Y8rX4ibwFk93JB47fKGuylteUfiHbDEYEjczukTE0/sK+5hk/RmM502GOkIzMQ4HwfEif9saAgFKthMOOhIIuyLaFmY5uAFEBL4DJ6cVjBM5LOCgVu60mBpNA6VbYKDUgSpDLEwF9Fwc3rJXhQY05aljCU29y2szmBgTJf3Voze4Gh6aWZmupYbyhT2lXs25Mc3Fwf6subEy/S8ctUWtmEj+0otCZvqTldgp5CpnqxNyaOFNBWeFsyM9WIJrliaLnXljJKFviPUILFm+XRq1+hU6Jr3yEgg+WYns36h0E6t+xbnEzhnWArISDH9tN0m+GvcuJ5kkwlbFZay0GszGBpDpvthU0prG9rYp5FToAB8SCAVYeGEChbKWfQv7S4WKpmebLkrw7RV22KEdUezCShx18a+3cuz4zcP7MRl9MXLxqz24CxKCBskhO+Nl3shgUuyM+/aef3bl4y+fPlTv7Pa+N54uW+s3HNxcCu1YkEY3hio4BEipLx42SNGmlk/yeZ2ZgFkorJ8SfZwjt8BQMoP/3Y0Tp7RH6bpigo+IwRVWAicPgQ0DmocZKjSOKhx8PQ9VBqaRAhPO6RS2C4EeG0/VO2Ct2SZMReP6pYqq2kkza+PJUbzZi3xR2WU6KDIsYbEF3nLGixxahunT00+/vWwbg2el94NXvCN9phDI+TF+BHp7HMPDWLmnlPAlM0ZS04HugjZwl8ULgc/ZBIdtCewJnSbN6fRSzugT8Y3g7eh6Q+JnNqBWqqxredt8qFVCiFKJqvwwKHMzHilG/sh0xEJD1VzXIl9zLVLl5ali6xKiuxMiu0ZjWSy3SIT6sxIiCMjezZAnZhoBz00g5YZCWFo2fyskdhEBYHpIrSwOlvO5jP4tRrfS6eNwvlBjyFyDStiWEfU0DBgrPH0y0x5Zlm1EOUI018ngVa2xjquKU6NZKasPZQBK5pCP6GXwbYJnkYCoZDu14oSisAPScElFes0bHZpV+HIVDca6Ahcl6pxK4UtL0sXejMTPelCtt4zWV66s7Di+v6x+w7fcKiSv3PZpocmLsZT9NOHL1uZKQ1mpz81evltgwewDd6QK6IBHvjk7BL4IX05XOly4gdjJPHi7l0k4mJKCG8k/MWtd7xn1RNrurfPVIeXZsfof3dukrWUyDqdR3AQTeSu0OIxpxNV6RICzwwBjYMaBzUO8h3SOPjMHiQLlBYhXAAUJXUmApADTDTTiXpXir26G0ey79191TfY0vwN3zmjCOfWcQxLXHXLv2uuW7N7CnfTw4dLo6OlkW9M3AvP4dWfj1EdW9/FJvLBZIzZGCuCB4YIC8CwgwWenHXWoYHGpIjzMbfMMI0QRgRjpIQRqcA2sW5xCg8EObe8wS8D+yKFVSvdk9M8PMeqeajRGBtpJOtT9XRXilmE9cty41uKS227RTZYyM7sSfRN1jJdqUpPItGTqkxU80uzs5PV3LJsYYr1PNMlqoCbEVIFNAxOxYo1kPxspoLnZ0/Olodh7wSoHZ9CGU/UaqWazmfLhWK+t2cGdkcXaBiUDCX0mY7YNQ/9MWtkOKwviWSlmqWzRKCXRgIDESXLqKMpsYNcVvYxDmm71QcHVMu15uHWSkWIMQCzvUSotD5V7KrXUnsn+zIZXJQTXZkyypnOV6oxpbDnwcnLqtX8hlWsNmTqUXFl91hYXQY73pLPzfRdmy3dmaj/y0wfuW/pPvjLFz+IwZC4Ez+o4MbslKdc2pO4fuBJUjjdX86vytraNhy/ve4LZcyZLGmTHW90O6SfnkA88PTgKC1CoC0IaBzkSc+H57PGQY2DfMc0Dp6uB40I4elCUnrajsBErb5pdtm63gND6fBmHGZJte7Phr8g79/+lt/21rSzAnc3jaubUtVbE++M2yRyunnysW9N3mNNoMOh12yKCOtzb0xSeGlgzVWyIVosNsPACfVCgM0VSC0lbMIeZXPkhuUyGV8dNxtgKF5P5iFTphNNjLvoqdbCvEG276NaVq9Bng8+pbvK/bDNg9UuGNOuSnd3qpRLlbuTFfSgbbaeHk6X8Ok9AuszA10im6rn4YHJ2mw1GxgjS32ayZdGEsJgYWgYEZNpc8WEDdJOazxbrufM6bRQgBbOTkz1DvZNzRTyXbkSYrUUfNXoKyqQpOvUBXXM2v6B7NJeZbIfJNDMgwEKy2WZ1rCtYi8+qxUoLrpxvzWfW6ZoQqrhtLSQDmLktD0M64mpUh430QzOofXkJYOTB2a7l3XNsqIMLJRFgPAs7UuPby71Hajk3pWoX9+/CYU3DZh9jwMb4MXdu39z1Vbi8MNfGNoDQybOMjPrunuJbJ1dzgRCX4QGWyI8cPvMitn60NX9j+4sDb946SYI/2ytWq3OhMVjaN5pNQmKB3INdAiBjkdA46BvF8yF0jiocVDj4Gl8YokQnkYwpaq9CIxW+irl/IpkHYLBwoteWbmyxbfU45SZasV6PRfe7NvblLOhvXWbxOevvPXHEu+iFbDEPdMYEo8cKR3CkPitCWOJgRXbr6fGD/nrmwrWzReUXNJhg8y3g9YUbSVMo38kGvkKdBrKhC2RFKNPgSfBnbzHPbYkaQZu1NjlAtOj7YFhB2UJqXG2noEvTdZz+F4yVY9VTEsYLRO1y/PjbJ4xmCoy+dAoVj3F6qbQRTOv2Yqp6ZX5qelqLp+sTlVzaMMESqIpZnYoi98kkl3ZEqWW9E1RC2yQWrrzDaMZNBJ56wVsELaZZhqkaS5XzQ2WSY3d6drq7MxUJX9xbnp7qffZ3RNfmFrWl0pM1VhUJgvhXJWfHin1rO2a2FMY6M0V2SgDEghQs2FfDfrPHh7Lc4XtxYGLeycLszmalM2Ere1TleW5qb709BKYcLL8fYM7k7Xu/szY/Ueuu23pd+86sh5g7li2eVsB58/dm2YHmXiJqZBWhQPQDDc/mFjIJENWJeX0SLm3PzO1a3bZ3tIQMptmh4ays725TaVaYmk6yd4Sp+E25yeV9Hp2NUzmLtdqMc2LoL9CoKMR0DgYtwvWOKhxUOPgaXxaiRCeRjClqr0IZNhgLW2TzfpT9pbvi8pENshCo7jtFeqJrqMv2O1tTydoZ2h8Vu662BJY4hxD4oHS7t3Fp6CCcBuIn2+fAMGAOeG3iUtkoII2mw7eaGZDSHXNrIg2ITAcCJEFB8NwF7YrtNl9FDfDYyCQmBlhiWh24NkFEiKZD2WrSabkVQ+V+vqyM5tnhnoyxYlEDlXZVJnNHpZmZvEgPVDqW5Ob3FFcgrUQ51YrHPY/7EqzpWQVT9SKbQJRHS93b8hP/tvMMogfumkq8xt7M2W4XE+6jIsptU9WM7YdImux8usARsNkYjBbRHpdbvpQuefZufLfFwaWJZOHKtlthQHcTW/rG/3I2OpXDu76+OF1q1hqNZWbKvX0pyojxR5g6UuXJyq54dzsvmLf8mwB3ojTbney/u2RoWolbewsW+rOlGwdnVqmXOs6Uu0uZ6cmq/mBVH28MsiCorTzef37CAcy47ctHZ+oDL5+xTcDMCwMM0YEZohVEINh8BSdxjX0cHlwfd/YaHU6l31ivLLyioGn4MzQ4Iv7HwS3kWru8gzQZdI2a/RUD4yBudsSuWuSbOepQwgIgXMKAY2D8y+XxkGNg/PviqdJ0Tg4DyARwnmQKKFTERhIlQZzs0vShe7G9LfQ0OZqijglsnzKZD3DEpDLOrULZ6BdcwyJXqOzxF1T5q+4eeqxByfv8d/VYHGQDWeJMAw2D4T1ucmKP4EKmkURU2GYXJcsM9MQyohQMB3CuyCHzhVR6CUhilBIXC59tVKWJSULe+TK7iNWNpEsVLrq9fREuQd6v29yaKhvdE9hCf6WMxBMHETrqaH0bIY5ipUclJXdLAr1VA8rrCYSB6t5dL526Z5PHLn4B5fs+frUymu7Jr49OzBbT+0rGwPtDmvG0B6qYfWannxxabq0dXbgynR5a7G7q5Zm1HxO99Rj5aXP75naPLFkDb2i7OwwZR+cXk64Lo/9sPvG3tGHp4bXdo/B9HhE5hP1bdODxYr9HDE6m79h6MjDI8sv6psYrWAnhCaylUdtWXY8Xcv2pQ8/e+DAkhSTJ/OXdCVhgP/v3qtB72Pr9waD4XcIl2aKl3YfvOvwFVd0jxH51sTFy7LDGAbvn1h9U9+esUrPY7NLb19+YFfhin2F5Q9Mfc9Prf3sZOnS1V2byon0QBLNzIq0q3DSh8a/k4ZMBYRAxyGgcfBELonGQY2DC98nGgcXxsVSRQiPj41yOgwB/BVX5I+szI02KEuYQzibelmxnlhSN1dJ1uPGKTEfHCM7rO1nuTmto+P3rLztRxPvnipP0CbWrRkvHT5i69YccJYIx4N04TIKJcPrEsbEHDxYXFheBUMihcy66Ev3WAS+1zyxPGN/7ptqK4FSHMLZnSkSmYQB4g5a7sGVlEJs9tCTnekeMDNakhVlcsUDEyv6uqamCv21/Ax7A2ax+6VZZqaaxroJ2UuXppjZmK58amwtLPST46tpy66pZdR4bdcll+ceg82OVNOHq9nJCoudJjO5En6kY6VexkWIJs14Qe9UZrpvIIMV2RrJ8X8Or6Jh+ytGJr9vcO9nxteuzU8dqeSK1Vxfurp9duloOTdbzo+Wctf0jn/98Irr+8YPTPaWy+llXaXpqu05gdp8qrw0O8Nmh7M1VtxJ/vqeV8EA/+farz85s+alSx67ImN7OaIfgyH8kEi4e1lp5shAZhYol2Rml+X38VvGtb20d6I3N7E+NTPKTpup6b3li5CHkF/evYkN6Cv12kAq04WVl9QTP3z8636RFgs9ccwkKQQ6FgGNg6d8aTQOahzUOLjI10eEcBFwlNVZCFQS1YtzRzAfZd1OVb0LM1U+syFT2eJGLZa8nK1nC2xQr+PpEOjLDiDyrKVH3U2dJbJNohsSN00+7u6miEFpMILBEuGKVVsBFFJoHqJOdTCysTcDM+4IYV+4p5YSLIDCVg/pMrsR2paD5vOJM2cqXe3JzlIaNsjKohDOfVNDq/sO7Z5edtmS/ZlsiTVglvceyaXLfczGq+SYN4hYzZb6tJ0kirU0/pnUQu3oxLM0V2ed09STxe2ZZDctzDJ30VbNMcaEwbM3XV2TLVyfK8JIL8l0PVVJPFTqevTgxaj64tQg3Xn94N6Pj61d3z0yVhvaWe5dmi4/PDWE3a9UzRys5K7qntwz23t1/+i+wgoMlYOp2uaxZcszlccnBlhfdIAFe/BPTpfRdqjUP5SZxtm1WKe1ga+lRq7sSRws9/386odpTLVeu7p3E724pg8Ya/tK/f/1wOXf1z/6/MEnjiSnCxVb/3Q7SnKFRyc2fHl6+X+4aIQJls8a2HJxpfdgpW9Zbqo7iSGX1XFwkT4xPggP7H6DNo0AfB1C4HxCQOPgabyaGgc1Dp7G2+lcVyVCeK5fwQuo/fsq+T3l/pXp6lD6QH8Kf8JqPvMK3vtTmfWJ0t28bZOC1aiWwACl41QQYHTks6bXDFMYEl0F7qa7pzEkHgFb2yZx/H7SiTdDs3ixeGkgbGa8YoAhC5aIJbBQzSUz9ZlKvsKEwBS7C5qtDx4Im4RAwqC6swXsYqv7R1l+ZkmXGS1hgJOlHvQH3lOHIvalZ9Bpvqlh/RpIJqRrKIfBLTecKbJODCwU/lZkSmGqcnGudn3XxEyle3Wm+ND0kqtzhQ+PrajVB7DFVetLcBm9c+m+u8dXXdG3f9PhSx8v9vUnqw9MrGQF1GIggdd1j+8p9KwdGNszvgLjYU+ytnNmsLue+NbhoUoldWmuuKnQd/OaA09ODi7Nz5bC7hc0pi9dYJ9G1oCZruXfMvwg0yMP1rKHiv2XdG19bPq6yWrXrYP7vz113eVdB749dTHoXNG7/Zr8VCJV3F8e+Or4ZRt6DkJ99xUG06niTMI25JioLOtGYbI63L1tMJnuT2ZOdN6geKDdmDqEwHmLgMbBdl9ajYMaB9t9j3WmfhHCzrwuatUCCDxWet23Jg6vTtbW5/b1pxITyTt7yv/a1/eJRPUQq1iW6xBCtjrgzV/H6UQAN5urm+vWvGDlrW+zDTAmqWDX1M4jpSNPTDzBjMH7xx8IBkPWprGqcSRlJVJmALLSS6HKTvTmOFosd0EF4ZIse4qbJW6ZUMR8pghLhBxC/A4VBnqyRVw0e7MFDIAplvFMVcjCDxOTHTu/l2zX+CoENJuqMbEQ99HDtexghkVMk2tyU9lApaZq6d3lrpFK7olSfn8lU5rtvzxbvLxr8q6J4Wv7Ds1ODu8v99CYhydXwQC/Mb6C9l6en5ktZwfzM7sLPSyd05NkxuCSXKL+wOjqWi05kKodKmXXdU/vn+5JpGtregoT5VxftjJW6qK1cFo25yjUs1DB3cUluXRh18zFhVr2lsGZx2bWZFIzu4uDa/PjO4qrWPToUJVuFpbnDh6uZZ47sCOdmt1RGnqo0JfNzF7Rt/XS3AjQXTswduMgy65WVqYnmSzE5MmBFJtzNNbUNXDnH5oUMR8TpQiB8xQBjYNn5cJqHNQ4eFZuvDNZqQjhmURbdT0jBA4Wu588PMyak49V+l7S99i6rsRQKl2fej/LjbKJQrVerbD0Zdj0PJFe+oxqUuFFEfBtEq9Zdi1SL1p1C+E7Eu+AJY4XJ7ZN7oBifXfsqS8c+TqmM/MgTdYgcriMYt/ryxYD92NLwwx+veOV7ulynoWCxos9y7smEUYmD8FL1vKs+WlLd2bKVbMHEkEPfDKbYD/BWljUxpqIe+qErftSm6jmsRWzTQV1YVubSdSHc4Vhqk4hkN9eyS3NlB+aXjpSzo2Ul1GQZrAjxbMHD/3b+FB3tthd6ppiGmOi/pXR1fijDqSqh8vZdb1T+wvdy7tnmMA4izyLfM50T5QyyzK1cjIxZDscpvlQI6bC6TQ+q9WZai6TKq/OTh6u9lzW9yQWz0v6nqolUmPsopGeOVxPjNXrE6Xly7KHdpUGejO5ZKry7N5Dg9mpA8WVO6eHL+s9OFpYun9myfeuvX8glVqbztmWg8dbQkY80G4BHULgwkJA42CHXG+NgxoHO+RWPF3NMLcs28VZhxDoeAQmipMffPSfNw5ffs+Ru64ZPPTGS391euIzE4X7q+kNbAhQK74XNri9/rqbVvzi0q5LOr43538DuV50csfE7tHi+MHZw/tnRz81+iAT/Hjc2CREVojJVNhRHq/RGtMLU/y3BUux8RJiG6zX0pjg2D2iLzeLETJMUMRxtDrOzhDZWdxQmaYIV7TJiuahauujulspdklqIXEGjpeqHcEyyWb39WQPe1dUclf0jj0xuWx9z/jO2YGl2eJosYtFaGhMf7oyXs5c1jN9oNC9qmfmcLErl6wR1qopSGAvxLKUvbR/+vBs17KB6elqelU/DDbZn5tl04tcqjKYne1OlXvSNDiDU+twbmzXzMp1PXu2zqyFMV7fv3XbzMUr86M7Zlax08bKrkM7ptcuz04tzY0RWZoDqGShTL0jS7Kjvani6nR1fbaSYWdI68exB2sp5W5LapGYY1F5JmfcQxTXOPhMMFTZM4aAxsEzBvVpqUjjoMbB03IjtVsJ46AGwnaDLP2nH4HdU3syqdqqnotRfWT6IRbkWN67sVDckcksyaQHT3990nj6EJgoTo0VJ7aM7WLm4SNHNu+YHXlwZg/qzbuUBU5TVSKYE0MKy5T6Ci0Y9FiqtFytpXttX0EWP8WpssoCpDy/zKc0zOXDgscpa6LCDLE0EmK+g3mWcCvNFkYKvUP52QOF3iXZ4sFCD0rYcb4rVZuqZC7tndw6OXBp3+SBmd7+bHmsmEfhTCWDvyif5bkSDqVd6ep0KbO0q3RkNt/bU+SXURbI6cpWejLlrgyEsNqfLZaq6TXd46PFfk7ZT2Km0tOfmTlS6h/IzLDvoi91Q79y6VKpmu9Jz9AFpikm2LcwO16s9O+cuGz94NbL8vsuyZSG2HFzzk6D4oFg155D42B7cJXW9iKgcbC9+LZTu8ZBjYPtvL9ORbcI4amgpjJCQAicXgRsdCwElphMfPvIpk8eftC2Mkza4jP8ZsXyoWHmoZFDWCNrrlA7TA/qyPb0/DTA9EI2A4RcmXywFpKNmRFySMjqpnicMmOQwjPVLEUKlcxAtoRt0CjirFHEqXIOIyFq4YFUsSRbnq1k8qnabDnTny2NFboGcuWx6fxkkUU+EwN9xWIiuXZwslRLLe+ehlgO5AoYMDFT0xKIbS5dYZIkzcajlUVZ8XTtzRQmSn1YAvfNLluWY4+KxHihb0m2MNx15KnDl17ad/iqgSe6U5WhVOmybGGAnSx8SR0apEViTu+ttpA2EcKFUFGaEBACZxQBjYMaB8/oDXdsZSKEx+KhMyEgBDoDAXez2T6x53Bx7FDh8IHCwS+MfR0GGEhgDU4I94Ppwe4wBsL6WKUTish/0klkv3iIIk6A2BKNPtrGD7YHPeStnmCze7ibOZTWjS6m4If4mOJcWqqmlnYVp0q5nkxlopjvy5YnCvnpcqZeS3ana1OlzJq+mbHZ/IrBqelytqeryHo5uUwlkFKIZtV4KYZJc1etL8nNzlZy/dnCWLGXBgx1TRbKXXiWThZ7ezOlZV1jo7NLV3cfGe46hJ0wk6xclJm5Jje2JJW1LVUwBqbXJ3Ibk93PTcri3f4bUoSw/RirBiEgBE4aAY2DGgdP+qY51QIihKeKnMoJASFwxhGYKE1OFCe2Tu7ATPf4xBMsbQoZ83VloYL4eeJhOstKpMnaeDlPlu18GOYZmidqwmyJblf0huNTShHWL+V0lk0xEgmsiPDJmXIW9oj/J6yxUMW4V8F4iGvoocne/nx5vJBLpepscUGc7U0G8kWWMlqSL1CkJ1OiOqOlyToepJRqLKWTm8VUSBtQXq9icpyu1yGZ7KlYXJ4bG07PXJWdvjhbZg+VZO62RO6aZPdNZxzaC7pCEcIL+vKr80LgnEJA4+A5dbnOmcaKEJ4zl0oNFQJCYA4C0/sug3QVa0senV2arXcdrmZGq9lvFruDmNE/dpPH7jddYfXQ5BiL0NTYtj5nFLGWzKerxWoamgctxJxoDqXmZQqvZJvEVC5ZZSUYJLEWsrTMZDFfrSVxIqW6rkwVvseOF9DCSj052FXATtibLVmpFLSSYlj8aniNlrEQ5grjBSyE9TV9h8YLgyt6jkyUu5fnppZl2VJidiBduTY7vq77xfm8eOCca3tGT0UIzyjcqkwICIHTh4DGwdOH5QWtSYTwgr786rwQOKcRKOy7nG3v6cLhKn6ZbN2eTuTemUyvnZj63ancL28a+9N85qbNs0/uqaYPVrMYA+FpExU2h2Bfwd58os7qo/VaiiVGmTcIY+zLVMrG9ypm68uy6QVeqXVCTIhEWNWGgix5in8pDqWTZche0SYc4qSaqlGKXKggRHGmDHtMre4bq1UzrI+Kn+qy7EwtWV+VmS7WU92pErxxSaqysfe69b0vyvfcksxdek5fhfOg8SKE58FFVBeEwIWJgMbBC/O6n/ZeixCedkilUAgIgTOEAAMh8/UyidRErTSQyqVtBuHtbEpJ9SXmBNoaMXawf8WmcjVRH1i69G93zo6T8tTkY9+YuI/FY4JDqa1NOs3WFNUMK5F22+YT+WItRRbTAvH5xM+TOYrIhE0sqjiXYhjESMi6o8w8ZKFRnEvhgWjrSkMLq6RDDlkmlbmCYSOK0pX5wztKA8/rGttZzV7VffX6wR9Y27MhJR4Yrk4nBCKEnXAV1AYhIAROAYEOHwcHMkV+A+1Jaxw8hWt7RouIEJ5RuFWZEBACpxGB0r518C4Mg2O10pJUzjRDCDmqd09m3lkvfWDA15NJJDaVbU+/dSvvbjXHTZYmkN0zvXO8fHisdGi0NPLgxD2FWgZbYrGWxWUUvjdZ7oL7YSfEWRSnULKwFvKBH+Joypqly3KFmWpmMFMareRZQmZJuvSCwU396dmvT1z1ov5tX5q++NLckZl6dnn2+puWvupZ/Zf35GUPtEvUUYcIYUddDjVGCAiBE0fgLI6D+MLwu6fGwRO/WJ0syThoDlc6hIAQEALnHAJJpv+Fzfp6kylby4VZgOn1uIzWZ+/OJxMzCegg+1XYgUtonxn5jjn6cwOcPyt3XUz9kcS7YYmT5Yltk9tZLPTxiSd3ze7/zvQuNLGlYZaFYdK2tGlvujzFbEH2w0hVZ+vJbLq8smuiUBi4vvfA5sKy8WrPWLWnnKxtqdx5x/KbNwxsWNV9WU/W6tIhBISAEBACQuA0InA2x8G6xsHTeCXPvip7SWJx9rPfELVACAgBIXAyCFT2beDJZZ6i8WhaCFtdRsncVSl2JfMrhu9qtRDGQk8bYe3v7eN7DhfGkHx0bPOnR7+F2RBv0jLzA3Oz0EKcRZlkuL7r8ou6V1/Us+zSviv7MktW9178tJol0AkIyELYCVdBbRACQuAUENA4eAqgqch8BOQyOh8TpQgBIXBuIFDbfwXT+I4hhM2GTyaZTHhXf9MquL9STCVzK4ePcRltyp7iX98hCquk89H+XP8pKlKxs42ACOHZvgKqXwgIgVNEQOPgKQKnYsciIJfRY/HQmRAQAucUAnPZYO6dieoW5hB2ZdYXq3fFrthe8XzSS2LKM48M5MUAnzmK0iAEhIAQEALPCAGNg88IPhVuItBcdaF5rr9CQAgIgXMFgalaeYGm2lqjW2zSoHuQmkRjMuECwkoSAkJACAgBIXDOIqBx8Jy9dJ3VcC0q01nXQ60RAkLghBCojiOWajhsNkuUPuCxYq1eTCT6Enf7adXWldEhBISAEBACQuA8QkDj4Hl0Mc96V2QhPOuXQA0QAkLgpBFgOZlavW6b0bceuIzyYUvA3l9nodFoIWRH+FYpxYWAEBACQkAInOsIaBw8169gR7Vf70kddTnUGCEgBE4UAeYFztYrSNsO8cceyeqebOvqowmo47ESOhMCQkAICAEhcI4joHHwHL+AHdR8OVJ10MVQU4SAEDhxBGCBOduBMFFJVNPuE9p0GU2W/iSBN2m14TIqC+GJoypJISAEhIAQOFcQ0Dh4rlypzm+nLISdf43UQiEgBBZGIJ1o7CzBxvQmwSoyYSGZVPoOO/fT9O0V26RehxAQAkJACAiB8w0BjYPn2xU9S/2RhfAsAa9qhYAQOE0IlOu13JzJhGYfxJe0YSFMOl08TdVJjRAQAkJACAiBjkJA42BHXY5zsTEihOfiVVObhcCFjgAGQChfJVFLJdIMhI394ZsMMFJBh6lSZ55FPZEevNBRU/+FgBAQAkLgfEFA4+D5ciU7oh/ypOqIy6BGCAEhcLIIwAPTSXMZnazXfV2ZZPev+SqjZJm24D5aqddm6jbVUIcQEAJCQAgIgfMJAY2D59PVPLt9ESE8u/irdiEgBE4dgXTwDM02FdRnf5Ut6TlLZ+6ssPQoBsP07UdqxVSiNl6z9Uh1CAEhIASEgBA4nxDQOHg+Xc2z2BcRwrMIvqoWAkLgFBHA4pdJpth2olCvZJL1dNhkwiyE4ahW7sqElFrlrul6rZpIYic8xZpUTAgIASEgBIRA5yGgcbDzrsk53CIRwnP44qnpQuCCRaBcGSvWq4VadX+1WGuuGVOv7kmk1x+plr9SsDVlMA+mMneMVTOcFJsyFyxi6rgQEAJCQAicTwhoHDyfruZZ74sWlTnrl0ANEAJC4FQQKNero7XK3mrvldnpUr1arqc2T3xoZTqdSL+mmLw6kXgvSkdK/zJTz9YSyaomEZ4KxiojBISAEBACnYuAxsHOvTbnWstECM+1K6b2CgEhkDB7365K7VB12eFqz1R6ujuZqNWzpe5fSib3dpc+9PzMVwGpXPn8oUo3f+thqqFgEwJCQAgIASFw3iCgcfC8uZSd0BG5jHbCVVAbhIAQODkEmBGI3W+21jWVuO27pRUHqziGTq0p/VZP+a8nKr2HS4O1ev1g8o7p7LuYQJhKaAbhycEraSEgBISAEOhwBDQOdvgFOreaJwvhuXW91FohIAQMgWx6SX+y+qzBn3985C+2Vi/ryb19V3nTdUted2T6I4N9G5K5jbvH31pJszf9pmyiyu9e2cz3CTghIASEgBAQAucNAhoHz5tL2QkdESHshKugNggBIXByCOQyg5f2/j/7ivc+u++K784Ojlcy09WuysR3qrVcrrJ9eSU5We3pqX1uaddPHy7Uk/VkLr3+5CqQtBAQAkJACAiBDkZA42AHX5xzr2m2Fl+dDbt0CAEhIATONQSmZx46VBodrWTW9qyfLB/ZMHjD44fvGe7eMNR9ydd2vac/Vbty5e9OFrfSrWW9G8+1zqm9ZwiBZNihROPgGYJb1QgBIXBaEdA4eFrhvECVMQ6KEF6g117dFgLnNwLFyjhPN35APb+7qd49cwQ0Dj5zDKVBCAiBDkRA42AHXpTObJIIYWdeF7VKCAgBISAEzhACIoRnCGhVIwSEgBAQAh2JAOOgVhntyCujRgkBISAEhIAQEAJCQAgIASEgBNqPgAhh+zFWDUJACAgBISAEhIAQEAJCQAgIgY5EQISwIy+LGiUEhIAQEAJCQAgIASEgBISAEGg/AiKE7cdYNQgBISAEhIAQEAJCQAgIASEgBDoSARHCjrwsapQQEAJCQAgIASEgBISAEBACQqD9CIgQth9j1SAEhIAQEAJCQAgIASEgBISAEOhIBEQIO/KyqFFCQAgIASEgBISAEBACQkAICIH2IyBC2H6MVYMQEAJCQAgIASEgBISAEBACQqAjERAh7MjLokYJASEgBISAEBACQkAICAEhIATaj4AIYfsxVg1CQAgIASEgBISAEBACQkAICIGORECEsCMvixolBISAEBACQkAICAEhIASEgBBoPwIihO3HWDUIASEgBISAEBACQkAICAEhIAQ6EgERwo68LGqUEBACQkAICAEhIASEgBAQAkKg/QiIELYfY9UgBISAEBACQkAICAEhIASEgBDoSARECDvysqhRQkAICAEhIASEgBAQAkJACAiB9iMgQth+jFWDEBACQkAICAEhIASEgBAQAkKgIxEQIezIy6JGCQEhIASEgBAQAkJACAgBISAE2o+ACGH7MVYNQkAICAEhIASEgBAQAkJACAiBjkRAhLAjL4saJQSEgBAQAkJACAgBISAEhIAQaD8CIoTtx1g1CAEhIASEgBAQAkJACAgBISAEOhIBEcKOvCxqlBAQAkJACAgBISAEhIAQEAJCoP0IiBC2H2PVIASEgBAQAkJACAgBISAEhIAQ6EgERAg78rKoUUJACAgBISAEhIAQEAJCQAgIgfYjIELYfoxVgxAQAkJACAgBISAEhIAQEAJCoCMRECHsyMuiRgkBISAEhIAQEAJCQAgIASEgBNqPgAhh+zFWDUJACAgBISAEhIAQEAJCQAgIgY5EQISwIy+LGiUEhIAQEAJCQAgIASEgBISAEGg/AiKE7cdYNQgBISAEhIAQEAJCQAgIASEgBDoSARHCjrwsapQQEAJCQAgIASEgBISAEBACQqD9CIgQth9j1SAEhIAQEAJCQAgIASEgBISAEOhIBEQIO/KyqFFCQAgIASEgBISAEBACQkAICIH2IyBC2H6MVYMQEAJCQAgIASEgBISAEBACQqAjERAh7MjLokYJASEgBISAEBACQkAICAEhIATaj4AIYfsxVg1CQAgIASEgBISAEBACQkAICIGORECEsCMvixolBISAEBACQkAICAEhIASEgBBoPwIihO3HWDUIASEgBISAEBACQkAICAEhIAQ6EgERwo68LGqUEBACQkAICAEhIASEgBAQAkKg/QiIELYfY9UgBISAEBACQkAICAEhIASEgBDoSARECDvysqhRQkAICAEhIASEgBAQAkJACAiB9iMgQth+jFWDEBACQkAICAEhIASEgBAQAkKgIxEQIezIy6JGCQEhIASEgBAQAkJACAgBISAE2o+ACGH7MVYNQkAICAEhIASEgBAQAkJACAiBjkRAhLAjL4saJQSEgBAQAkJACAgBISAEhIAQaD8CIoTtx1g1CAEhIASEgBAQAkJACAgBISAEOhIBEcKOvCxqlBAQAkJACAgBISAEhIAQEAJCoP0IiBC2H2PVIASEgBAQAkJACAgBISAEhIAQ6EgERAg78rKoUUJACAgBISAEhIAQEAJCQAgIgfYjIELYfoxVgxAQAkJACAgBISAEhIAQEAJCoCMRECHsyMuiRgkBISAEhIAQEAJCQAgIASEgBNqPgAhh+zFWDUJACAgBISAEhIAQEAJCQAgIgY5EQISwIy+LGiUEhIAQEAJCQAgIASEgBISAEGg/AiKE7cdYNQgBISAEhIAQEAJCQAgIASEgBDoSARHCjrwsapQQEAJCQAgIASEgBISAEBACQqD9CIgQth9j1SAEhIAQEAJCQAgIASEgBISAEOhIBEQIO/KyqFFCQAgIASEgBISAEBACQkAICIH2IyBC2H6MVYMQEAJCQAgIASEgBISAEBACQqAjERAh7MjLokYJASEgBISAEBACQkAICAEhIATaj4AIYfsxVg1CQAgIASEgBISAEBACQkAICIGORECEsCMvixolBISAEBACQkAICAEhIASEgBBoPwIihO3HWDUIASEgBISAEBACQkAICAEhIAQ6EgERwo68LGqUEBACQkAICAEhIASEgBAQAkKg/QiIELYfY9UgBISAEBACQkAICAEhIASEgBDoSARECDvysqhRQkAICAEhIASEgBAQAkJACAiB9iMgQth+jFWDEBACQkAICAEhIASEgBAQAkKgIxEQIezIy6JGCQEhIASEgBAQAkJACAgBISAE2o+ACGH7MVYNQkAICAEhIASEgBAQAkJACAiBjkRAhLAjL4saJQSEgBAQAkJACAgBISAEhIAQaD8CIoTtx1g1CAEhIASEgBAQAkJACAgBISAEOhIBEcKOvCxqlBAQAkJACAgBISAEhIAQEAJCoP0IiBC2H2PVIASEgBAQAkJACAgBISAEhIAQ6EgERAg78rKoUUJACAgBISAEhIAQEAJCQAgIgfYjIELYfoxVgxAQAkJACAgBISAEhIAQEAJCoCMRECHsyMuiRgkBISAEhIAQEAJCQAgIASEgBNqPgAhh+zFWDUJACAgBISAEhIAQEAJCQAgIgY5EQISwIy+LGiUEhIAQEAJCQAgIASEgBISAEGg/AiKE7cdYNQgBISAEhIAQEAJCQAgIASEgBDoSARHCjrwsapQQEAJCQAgIASEgBISAEBACQqD9CIgQth9j1SAEhIAQEAJCQAgIASEgBISAEOhIBEQIO/KyqFFCQAgIASEgBISAEBACQkAICIH2IyBC2H6MVYMQEAJCQAgIASEgBISAEBACQqAjERAh7MjLokYJASEgBISAEBACQkAICAEhIATaj4AIYfsxVg1CQAgIASEgBISAEBACQkAICIGORECEsCMvixolBISAEBACQkAICAEhIASEgBBoPwIihO3HWDUIASEgBISAEBACQkAICAEhIAQ6EgERwo68LGqUEBACQkAICAEhIASEgBAQAkKg/QiIELYfY9UgBISAEBACQkAICAEhIASEgBDoSARECDvysqhRQkAICAEhIASEgBAQAkJACAiB9iMgQth+jFWDEBACQkAICAEhIASEgBAQAkKgIxEQIezIy6JGCQEhIASEgBAQAkJACAgBISAE2o+ACGH7MVYNQkAICAEhIASEgBAQAkJACAiBjkRAhLAjL4saJQSEgBAQAkJACAgBISAEhIAQaD8CIoTtx1g1CAEhIASEgBAQAkJACAgBISAEOhIBEcKOvCxqlBAQAkJACAgBISAEhIAQEAJCoP0IiBC2H2PVIASEgBAQAkJACAgBISAEhIAQ6EgERAg78rKoUUJACAgBISAEhIAQEAJCQAgIgfYjIELYfoxVgxAQAkJACAgBISAEhIAQEAJCoCMRECHsyMuiRgkBISAEhIAQEAJCQAgIASEgBNqPgAhh+zFWDUJACAgBISAEhIAQEAJCQAgIgY5EQISwIy+LGiUEhIAQEAJCQAgIASEgBISAEGg/AiKE7cdYNQgBISAEhIAQEAJCQAgIASEgBDoSARHCjrwsapQQEAJCQAgIASEgBISAEBACQqD9CIgQth9j1SAEhIAQEAJCQAgIASEgBISAEOhIBEQIO/KyqFFCQAgIASEgBISAEBACQkAICIH2IyBC2H6MVYMQEAJCQAgIASEgBISAEBACQqAjERAh7MjLokYJASEgBISAEBACQkAICAEhIATaj4AIYfsxVg1CQAgIASEgBISAEBACQkAICIGORECEsCMvixolBISAEBACQkAICAEhIASEgBBoPwIihO3HWDUIASEgBISAEBACQkAICAEhIAQ6EgERwo68LGqUEBACQkAICAEhIASEgBAQAkKg/QiIELYfY9UgBISAEBACQkAICAEhIASEgBDoSARECDvysqhRQkAICAEhIASEgBAQAkJACAiB9iMgQth+jFWDEBACQkAICAEhIASEgBAQAkKgIxEQIezIy6JGCQEhIASEgBAQAkJACAgBISAE2o+ACGH7MVYNQkAICAEhIASEgBAQAkJACAiBjkRAhLAjL4saJQSEgBAQAkJACAgBISAEhIAQaD8CIoTtx1g1CAEhIASEgBAQAkJACAgBISAEOhIBEcKOvCxqlBAQAkJACAgBISAEhIAQEAJCoP0IiBC2H2PVIASEgBAQAkJACAgBISAEhIAQ6EgERAg78rKoUUJACAgBISAEhIAQEAJCQAgIgfYjIELYfoxVgxAQAmcPgS1btiSTyUceeeTsNUE1CwEhIASEgBAQAkKgcxEQIezca6OWCQEhIASEgBAQAkJACAgBISAE2oqACGFb4ZVyISAEhIAQEAJCQAgIASEgBIRA5yIgQti510YtEwJCQAgIASEgBISAEBACQkAItBUBEcK2wivlQuA8ROCNb3zj3//93+/evfvd7373TTfdNDg4+MIXvvDXf/3XC4XCifT2RIpffvnl/+t//a/f//3ff/azn/2lL30JtfV6/QMf+MDLXvay5cuXr1y58jWvec3dd989v7p/+Id/eOUrX7lq1apLLrmEihaUmV9KKUJACAgBISAEzgoCJzImnpWGqdILCoEkveVN64LqszorBITAM0EAtnbrrbfec889sLXXve510LNHH330D/7gDwYGBu66666LLrpoceUnUhyZ/v7+w4cPwznf9a53ofkVr3jF448//jM/8zM333zzzMwMtf/FX/zFz//8z//Gb/yGV1er1RhWP/OZz7znPe+57bbbyuXyF7/4RWTe8pa3/Nmf/dnDDz984403Lt4w5V6YCLDmEB3XOHhhXn31WgicdQROZEw8641UA85vBBgHNRCe35dYvRMCpx8BRq/t27f/4R/+4U//9E9H7WNjY5jvhoaG/vVf/9UfLDFrTuREiiMD63vsscewB1L8Z3/2Z//lX/7ly1/+8ooVK6I2Tl/+8pf/1V/9FTyQxN/6rd/6zd/8zQceeKCV+H3ta1976UtfiulShDDipsgcBDQOzgFEp0JACJxJBE5kTDyT7VFdFyACIoQX4EVXl4XAM0WA0Wv9+vUYA+co+ta3vvWc5zznm9/8JuGcrNbTEymOzGtf+1pcRimIrQ8L4Yc//OEf+IEfaNVDHKJIdV/5ylcw7yxduvQ//+f//Au/8AtzZP7H//gfpIsQzoFFpxEBEcIIhSJCQAiceQROZEw8861SjRcUAoyDmkN4QV1xdVYInB4E7rjjjvmK8CCFlUEL52fNSTmR4tH1FDshJr7Xv/719gvWsQd+qk8++STKN2/ePD4+fuedd86piNPv/d7vnZ+oFCEgBISAEBACHYLAiYyJHdJUNeN8RSBzvnZM/RICQqB9CBxvwlU6na5Wq09b74kUT6UaP1cNDw+j8G//9m83bNhwPM2ukGmE8wWOV9d8SaUIASEgBISAEDjzCBxvnDrBIfXMN1g1nn8IyEJ4/l1T9UgItB2BBVfvfOSRRw4dOvTc5z73aas/qeJr1qxh1dADBw58z7zjwQcf3LdvH9VdccUVS5Ys+fznPz+/auY0zk9UihAQAkJACAiBDkHgpMbEDmmzmnGeISBCeJ5dUHVHCJwJBBi92ASitabJycmf/MmfZF2ZxScQepGTLc6eFr/6q7/67W9/u7XGj3zkI6xq4/ZDPEn/03/6T//9v/93KGKrzEMPPcRKM60pigsBISAEhIAQ6CgETnZM7KjGqzHnBwJyGT0/rqN6IQTOKAI/+qM/+t/+23/DIvf93//9mO+gav/n//yf7u7uj3/843Czp23KyRaHarJyzPOe9zwY4Itf/GK8az7xiU/gRPre9773RS96kVfHcjKsHPP85z//p37qp9h2IpvN3nfffSyFStn/+3//79M2SQJCQAgIASEgBM4KAic7Jp6VRqrS8xsBWQjP7+ur3gmBtiCwcePGr3/96ywh87u/+7s/+IM/+LGPfQzexW6EcSWYxWs9heLsKPihD33oO9/5zjve8Q52JhwZGWGHif/wH/5DrIg5h3/3d3/30Y9+dNOmTaw++hM/8RNESMFyCEvs6emJkooIASEgBISAEOgcBE5hTOycxqsl5wcC2ofw/LiO6oUQOHMIsEY2ljo2hT+1Kp9h8VOrVKWEwPEQcJv28RZ1OF4ppQsBISAETgsCGhNPC4xS8kwQYByUhfCZAKiyRxHAXsT95AeWnKMZ82Ks/9EUPPqXn8eiIBvcvepVr1q9evUll1zy5je/GV/BmNXhkZNq+Ve/+tXv+77vY6d1uomRDdtXa+/Y+R1vzGXLlq1bt+6Xf/mXi8Via+6Jo91a6ozFj17XebEF23Ai3WHj+5UrV85noV/4whfYep4VZS6++OI3velNW7ZsWbCKM5N4UjeAN+l4/VpE1dN+g85MZ0+wlhO5uFHV4rc9ZuG3ve1tPBk43vKWt7DOUCyoiBDoBARO/G5/2m/xIk+ATujpIm04qZafx+MgEJ0UFCdy8xxvvNA46O8arW+Si9yiZz7rRC5ubNXZGgdFCOMlUOQZIXDttdd+LRxMJFtcEU59LhnDF7zgBfxC5qXe//73v+IVr4AIMSftf//v/53JZFinBI/ExXV2Qu5JtZz5b7feeiuz7z74wQ/+3u/9HvslsInfJz/5Se8I9I8t9WZnZ6HWv/Irv/Lnf/7nP/dzP9faxxNHu7XUGYtz7XDg/KEf+qG/+qu/oqfwtHw+T2RqamrBNpxId/7Lf/kv3AxMXGzVwI3BNoMsQ/pnf/Zn3C2HDx++4YYb2LewVeaMxU/qBoitWrBfi6ta/BsUNXdI5EQurjd18du+VCrxKGDbyfe9731MCv3ud7/Lzl3lcrlDuqlmCAEQOPG7ffFv8eJPgE6G+qRafn6PgycFxQnePAuOFxoH/WWy9U2y074jJ/5kOMvjIK4yOoTA6UKgr6+Pp/yJa9u1axcv+vyQRpEjR44MDg7yyGst/ou/+IvYf1jEsjWx0+In1XIMHcy+Y+XM1l4wHQ4LGHpI5Pne29vLTusu8Pd///dsRgQ/bJX3+MmiPV/DKaTQPOyZxyv4tFAsUvx43WGzexjmP/zDP7RWCkNYu3btO9/5zpgIr4Yz3H777THljEWettcLtmTBfp2sqtZv0IK1dEji8S5ubN7itz3fAr4UcH6X37t3L88N1haKxU854q8Op1xcBYXAfASe9m6fU6T1W3yyT4A5qs7i6Um1/FwfBxfH+aSgmDMmHu/mWXC80DjoF6L1G7T4pTm7uce7uLFVZ3ccTMR2KCIEnjkCT3u7z6nil37pl66++mpP/PSnP81LXiRCnjg6OsobG1lzCnbU6Um1/MMf/jB2VH4Hau3CxMQErO9Tn/oUiZ/97GfZZSHmsncCCGzdujWmxMjJoh0Lti9yUlDMacaC3YHmsQEh5qA5wvgaAcvmzZtb01lFpqurq1KptCaegfgp9Pp4/TpZVa3foDPQ01OuYsGL26pt8dueb8R1113XKn/ZZZf9zu/8TmvKqcVFCE8NN5VaBIGnvdvnlG39Fp/sE2COqrN4elIt1zh4vCu14M1zvPFC46DD2PoNOh6wnZC+4MVtbdhZHAe17YS/DCg8OwgUCgW2s8NW5tXzGw9z6gYGBlpbg/soZsMOny90Ui3Hp3H9+vW5XK61m/39/ZdeeilZr3nNa5hbyBFzcSXFosjrb0zp5MhJQXEiHcEdFEqMCz7zA5lRyVQBL8XGEq973euYOtiqBDbITYU9Gatya3q746fQ6+P166RUzfkGtbubbdW/+G3PtwPyv3//fhytacaePXt2796NH05bmyTlQuAMIDDnW3xST4Az0LwTr+KkWq5x8MSBRfJ444XGQcCZ8w06KWA7TfgsjoOaQ9hpN8OF1R5+I2QW0Fvf+lbvNlYyeNF8CCCEx5t+Nl/4rKScVMuZ88br7Jx2VqtVXnZZYCam84z7z//5P7/2ta/9jd/4DaYRRiIUBTozclJQPG0XDh06xL4R9J1NCDds2OCTS/k5jYLMusRjcA6v5idq5qOeYTZIY06214v066RUzfkGPS2enS9wvNue9aX4djDBGM9hnGoYNZ/znOewKWXn90gtFAKLIzDnW3xST4DFNZ/h3JNqucbBE786i4wXGgeBcc436MSB7VjJszIOihB27P1wQTTsD/7gD/7dv/t32NBjbxekPcwfiwIdGznxlrNtOm6xLCrQ2hcoH06kcZt1snARYaretm3biLPFX6twh8dPHIqn7QjLijJ5Es9A1t1i2hjGZEgy3oMLFrznnnvYrvC//tf/umBuuxNPqteL9+vEVc3/BrW7m+3Wf7zbnp0kWVqJLwKLFb3xjW/89re//e///b9vfXS0u2HSLwTahMD8b/GJPwHa1KRTVnviLdc4eOIgLz5ezNGjcXAOIOfi6VkZB+Uyei7eKudJm++//35e7+bwovOkb4t2Az+3n/qpn2KDdVbIwOkR2yAz31hKh5RWCyFvwL7u6Mc//vEf/uEfxmv0Z37mZxZVfL5lPvXUU3/zN3/z13/91z/2Yz/mfQMBbMgsJPP2t799jg8tE+7ZvePHw9HhQCzerxNv/Hn5DTrebf+Hf/iH//E//kdenVm0FohYuYrvCxYJvkonjpgkhUCnIXBefotPBGSNgyeCEjKLjxcaB8/Lb9BZGQdFCE/wKymxBgK8gbF8VoQDF74FnTyjwCIR3vDYM4DtmFpl+F2k9dTjCybOFztjKfNBoOoFG7lgIsJszDA0NMTPfu9+97uZA0AKfO+3f/u3F+zC61//esTYVqEDCeEzh2LBLnvifffdh7dwZIOeiIMxpqEHHnigdSCEDbLqDEuM/umf/ukiCtuateC1XjBx8X7RyAVLzU9c8BvU1j6eSeWtt/3MzAy/mGAfjl8Bvjvce/4zCpuanMmGqS4hMP+5p3GQlwFujPmPqeMlkq5x8ES+SouPFxoHNQ6ernHwHPDEO5EvjGTOGAK8cPMAigeOjqdWNbPPmQAWX+9cCcvJsO/qfIWMvmd+Vtj8ZsSU+SCcbMtZUBS/RxxHH3/8cX7fYlIcuxHCCakCN9E/+qM/inV55CUveQk+kywwPSf9rJ8+cygW6cLOnTvZWGKOAP7D7EveOgnzwQcfZNtG2KDvzzFH/sycntQNsHi/TlDVgt+gM9PZdtSy+G3P14SHwPd///e3Vs0pP06x0kxrouJC4AwgMP+5d2qVLvgtPsEnwKnVeBpLzQfhZFuucfBELsfi40XUoHEwQnHuRs7uOChCeO7eOWen5fww37pC7s/+7M+eWjv++I//mMU/sBC2Fr/ooouYPN1qgSQXv0qIExPQWyXPbnw+CKfWcobDq6666hd+4RfYUPUtb3mLd4qnP3sSsk5max9ZZJVdCucsoNIqcLbipwuKBdv/rGc9i5VFmVrZmsuUwh07drBViSeyACm2wZe+9KWsNeK21lbhMxY/qRtg8X6doKoFv0FnrL+nvaLFb3s3PmAnbK13enqaUyztrYmKC4EzgMD8596pVbrgt/gEnwCnVuNpLDUfhFNrucbBxS/K4uOFl9U4uDiG50ru2R8HW9/vFRcCzxCB+buswG3Yd56XuaiZ+PLly3EXiSkeYSlRirNqSGs6ljR+dySrNbHT4k/b8vkgeBdYBIWtF1khI/YISUD4lV/5lZiCYfDGG2/EVTKmxMh8tGPW2YqcMhQ0eE53WHYVy/Cc+4FT9ibhVwPkGQWhCkwdZK3as9Vfr/eker14v55WFTUe7xt0dkFYvPY5Fxfh1i/F0972bDuBq3BrFW9729twOG9NObW4vyucWlmVEgILIrD43e5FjvctPpEnwIKVnvXEp21561e+tbUaB1vRmHPzLD5eUFDjYCt6HR6fc3FpbeuX4qyPg9qYvsPvn3OsefNv94cffphXrtYNxD/4wQ8ihnfo/L6xgAp2MFYGwaGU413veherlv3+7//+fMlOS1m85fNBoP0YP4eHh+f8zkr65z73OfbT+4mf+InPfOYzH/nIR57//OdjHmRUmN/l+WjPlznzKacAhTdyfnc+9KEPQZiZRsj6OqDBjcEpCxEh/93vfpefFaDK7Mz7b8cezGPp8F4v0i9avjiACCzyDTrzHT/BGudf3DlfisVv+7vvvhsLMIZ09u3lTviRH/kR7gQST7D2RcRECBcBR1mnhsDT3u2oXeRb/LRPgFNr1RkotXjL53zlvT0aB+dcl/k3zyLjhcbBBd8k50DaOafzL+6cL8XZHQdFCDvnVjkfWvK0tzudvOGGG1gb8Hi95SWPhQSZpogz2Mtf/nIGmONJdlr6Ii2f8533lr/nPe9hjhy/Cc3vyL333ssea4DJuqNYQnAZnS9Dyny0FxQ784knC4W3cMHufPOb33z1q1/NvEFY8ate9SpOXfj973+/v8rPD9nD58x3mRpPqtfH65e3fBFVCCz+DTorfX/aSudf3PlfisVv+0ceeeQ1r3kNG9PzM8orX/lKVhJ62kpPRECE8ERQksxJIXAid/vi3+LFnwAn1ZgzLLxIy+d/5WmbxsE5F2j+zYPA8cYLjYNz0Ovw0/kXd/6X4myNg0nGQuCb/0alFCEgBISAEBAC5z0CvnOaxsHz/kKrg0JACAgBIbAgAoyDWlRmQWSUKASEgBAQAkJACAgBISAEhIAQOP8RECE8/6+xeigEhIAQEAJCQAgIASEgBISAEFgQARHCBWFRohAQAkJACAgBISAEhIAQEAJC4PxHQITw/L/G6qEQEAJCQAgIASEgBISAEBACQmBBBDILpipRCLQVgTtTbzT9yWSCWawEFiYTKVviyBZ4SIXfKQg5MZmjkiZmHyvVzPWUELpO9CDT0J+sN4q3JIaKkKk3tKHKq0ja8kok+u8kITcUTzQkLdcELLEp6UUatbBEkwnYJxQJwo1SIdFLWW6LWCwVI1TRLOWRhnxDwKo/qiFKWnJTbYyExCAcYrHqqCFIHqs/KGkRiLnIHo2jz7XFuixSnyvQzDVh4s3T0K+6YRV1tsaRDfqbAg3JRmIodDQOWi5syi0ern9MtEvRmtiop0UyXOrQkOZNYY3yuPcyxEOi3afhFogCjUudsvviaKKdJuvhVgoRSiXqMTGZqFmu3fWNUkh6cVLSnmjyJNZIMUnDvlmKxFBXSK8FzQiYJHEvjkAoGHI9bhosMY2eUDzNaaNULe11NSVDcavFBWgekVC8lramhlZRV7NVJFqTrHgiJHpLLDF0LRH0U5BcEum4XZZ0uFbE+UqnwiMgxNGBTDKVTKVWbbKrrUMInHcIaBzkGWFPhPAw4vI2TmOERySpLhAixwqQ0SyChihpyceUCvotLRQP2VZjkIkaQpG5iUF2bqJpOlq82f6QGnT60BZLtbb/aBzxFmGLR52tcY2DhrSNKRoHwx3W3oBxWYcQEAJCQAgIASEgBISAEBACQkAIXIgIiBBeiFddfRYCQkAICAEhIASEgBAQAkJACICACKFuAyEgBISAEBACQkAICAEhIASEwAWKgAjhBXrh1W0hIASEgBAQAkJACAgBISAEhIAIoe4BISAEhIAQEAJCQAgIASEgBITABYqACOEFeuHVbSEgBISAEBACQkAICAEhIASEgAih7gEhIASEgBAQAkJACAgBISAEhMAFioAI4QV64dVtISAEhIAQEAJCQAgIASEgBISACKHuASEgBISAEBACQkAICAEhIASEwAWKgAjhBXrh1W0hIASEgBAQAkJACAgBISAEhIAIoe4BISAEhIAQEAJCQAgIASEgBITABYqACOEFeuHVbSEgBISAEBACQkAICAEhIASEgAih7gEhIASEgBAQAkJACAgBISAEhMAFioAI4QV64dVtISAEhIAQEAJCQAgIASEgBISACKHuASEgBISAEBACQkAICAEhIASEwAWKgAjhBXrhz363k0lvQ7IZ4dTintySSHL4WHZLs1vjzeRYPOhqpLaWivEQaahYSNOcuhaQjHUdq6op2aK0JXpU7TGJjU43/4SGR4EQaZw14YmQmGiUbI0fmxiLHxVpVRV1xETT21Tcmhjjx+oP4o0CrXW1aGmIHP1jckE26gwJc4s3EhuSxxYPZ40CMd48j3+bEZNoxuONYGnHTWxmNEvGax4TWosvlGga4g3SrKiReLTfIcGLt1bpzW2WsvygK0rbeUjlT0iMZ57aDBvJ4fRovFl9U3/QcFRhUzF//3/2zgNAkqs6193TPXlzTlpptasIykJIQkiAREZgggl+z+nZYDDGNn5+xhnDc07YYAMGJ+AZ2yCRRFZCOecs7Wq1OefJ0z39vlv/zNlSz+xqV+pF26u/1Lp76txzz731V0+d/uvcuqWaMW+jne51lEY1OoBUt7cilHubjjbOWoR2bASpq70yfva6ClsLRuCIQmDsC5/75sffU/p7yB3s2F9EvTJnIhGDaBfGIWATciaM2kaTvL+wTMqxnbwlOu0+3dWYz5xpTgxPe8eZ3I/tPc1ybDSZcrQmbxnGIWSuRps9XRnN95rkXaUhZFsoR/fGKcPg6f7VWl7CVV45Kuf/SXaZbfjMFPXNR5WjlnsdjNrF0LOapByriH/HhGQxJo+d0Uy3T+VYxVjLOOehSC7HWeWUqS6+IGOWo8q9x50p5DPvTMMda5XqM19hnfYzLf9kytiTdqwcVWe7e+Wx7sf8Zx72OhxzzL+qGfM22uleR2lUowNIdXsrQrm36WjjrEVox0aQutor42evq7A9RELqqVarHSLvdmsEjIARMAJG4HBGQNHXcfBwPkcemxEwAkbACBw6BIiDzhAeOnjt2QgYASNgBIyAETACRsAIGAEjcFgjYEJ4WJ8eD84IGAEjYASMgBEwAkbACBgBI3DoEDAhPHTY2rMRMAJGwAgYASNgBIyAETACRuCwRsCE8LA+PR6cETACRsAIGAEjYASMgBEwAkbg0CFgQnjosLVnI2AEjIARMAJGwAgYASNgBIzAYY2ACeFhfXo8OCNgBIyAETACRsAIGAEjYASMwKFDwITw0GFrz0bACBgBI2AEjIARMAJGwAgYgcMaARPCw/r0eHBGwAgYASNgBIyAETACRsAIGIFDh4AJ4aHD1p6NgBEwAkbACBgBI2AEjIARMAKHNQImhIf16fHgjIARMAJGwAgYASNgBIyAETAChw4BE8JDh609GwEjYASMgBEwAkbACBgBI2AEDmsETAgP69PjwRkBI2AEjIARMAJGwAgYASNgBA4dAiaEhw5bezYCRsAIGAEjYASMgBEwAkbACBzWCJgQHtanx4MzAkbACBgBI2AEjIARMAJGwAgcOgRMCA8dtvZsBIyAETACRsAIGAEjYASMgBE4rBEwITysT48HZwSMgBEwAkbACBgBI2AEjIAROHQImBAeOmzt2QgYASNgBIyAETACRsAIGAEjcFgjYEJ4WJ8eD84IGAEjYASMgBEwAkbACBgBI3DoEDAhPHTY2rMRMAJGwAgYASNgBIyAETACRuCwRsCE8LA+PR6cETACRsAIGAEjYASMgBEwAkbg0CFgQnjosLVnI2AEjIARMAJGwAgYASNgBIzAYY2ACeFhfXo8OCNgBIyAETACRsAIGAEjYASMwKFDwITw0GFrz0bACBgBI2AEjIARMAJGwAgYgcMaARPCw/r0eHBGwAgYASNgBIyAETACRsAIGIFDh4AJ4aHD1p6NgBEwAkbACBgBI2AEjIARMAKHNQImhIf16fHgjIARMAJGwAgYASNgBIyAETAChw4BE8JDh609GwEjYASMgBEwAkbACBgBI2AEDmsETAgP69PjwRkBI2AEjIARMAJGwAgYASNgBA4dAiaEhw5bezYCRsAIGAEjYASMgBEwAkbACBgBI2AEjIARMAJGwAgYASNgBIyAETACRsAIGAEjYASMgBEwAkbACBgBI2AEjMBhgkCRcdRqtcNkNB6GETACRsAIGIEfJwLFouPgjxNv92UEjIARMAKHFwLEQT9DeHidEo/GCBgBI2AEjIARMAJGwAgYASPwY0PAhPDHBrU7MgJGwAgYASNgBIyAETACRsAIHF4ImBAeXufDozECRsAIGAEjYASMgBEwAkbACPzYEKgnhD/90z/NRFJtXV1dRx999Jvf/OYrr7zyUAxoxYoVdHTffffhfMmSJX/zN39zUL38ZLZFk2fhIdoerPCWt7zl7W9/e12rlStXcjj//M//XKf/3Oc+N23atGq1WqefcDd/FHl5vPG//du/nXHGGZMmTTrllFP+8R//cWRkRDb7bzXez6HT7GuE9Fh37g7dGOzZCBgBI/CsEfjWt771pje9ad68eZ2dnSeccMK73vWua6+99ll7O3QN3/rWt374wx8e7/+oo44aDee5f9atWzfecv+al7zkJfkAPX369Msvv3z/TSasfeMb3/jBD36wrupXf/VX+aVRp/zOd74D5gMDA3X6CXdjPCHUmQ0NDX30ox897rjjpkyZcuGFF950000y2Jd9XfNDt7uvgdFjHeaHbgz2bASMgBEAgXpCiIrL5a233nrzzTcTC//8z/8cWviGN7zhBz/4gfEKBF796ldfc801dRzv6quvxkBlWCJAp1/1qleVSqW88jnKxOYPfOAD/Ai47LLLKH/jN37j4x//+HP02djmh/8IG3u89mYEjMCRhMDWrVuhf2zLli37xCc+AYX45Cc/uXDhwp/4iZ945zvf2dfX1xQHe8MNNyxfvvzRRx9ltP/1X/+FzAa/fb4G/9rXvvaHP/xhXe9XXXXV6tWrGVheT9y86KKLOjo68spnLX/oQx/6whe+wO8Zej/ppJMuvvjihx566Fl7a2DDw3ZgDTxGuzICRqApECiPH+WMGTNe+tKXhv4973nPT/3UT0E5DukFlIv14sWLo9NnITx3Dwfe6Wte8xqu43fdddc555wTraCCU6dOhSiyait3Y6UncYfmT/7kT8LsuQv4h/597GMf+8hHPoK3173udeVy+U//9E9/93d/t62t7bn7f+4eDv8RPvdjtAcjYASOVAS4bjMHpLe3lwksxx9/fBwmfOZXfuVXXv/61xMTv/GNb4T+sBWOOeYYxqZ7l4sWLVq6dOnzO1QA/LVf+7Unn3zy2GOP1UhIVz7yyCMwbWgh3DuGB3N773vfG7vPRdizZ8/nP/95/HNnFj/nnnsup/Uf/uEfPvOZzzwXt8+97WE7sOd+aPZgBIxA0yEwQYZw/DEQGrnFSHQcX9UozWOPPcYd2efi7bl7OPDe+YkAfc3PpIUCQfyYtLN58+YHH3wwXEEat2/fTkYxNM9d4Gbq7t27X/GKV4Qrfp2QJ6Tr0Dy/wuE/wucXH/duBIzA4YzA3/3d3917771f+9rX8mxQA4bJfP3rX2cq5vr166Vh2uF///d///qv//qLX/xiaZgH+Md//McQDyYonnbaaVCgXbt2xfFOOE2Re4jXXXcdNtR+9atfJZdFQ541gMN885vfjLYIlUrlL//yL1/2spdx/5GnBuiXcJA3OBC5bsz7GRITFxnbnXfe+Zu/+ZsI/+f//B/5h2T+4R/+IQ8s0JZBgsmB9Mu0W2aH5kMnN1JJ2fEQAYQtPKxZswaWCHuUZnh4mDue5513Hniefvrpv/zLv6yo+ju/8zvRZD8C8YgATS9h89d//ddMXtXu/g+kv7//D/7gD84++2yezoBOA0LdD5VnHFsd1DEGhH0NbELMx/vZ/9iYc8sh8yUhy71jxw71O6EyPyTLRsAIvKAR4FoZ2//8n/+TKTGxK0HTINeuXcsudxy5mBIvzzzzzOuvvx4NN1P/6Z/+6ZWvfCWpxTlz5vDEBVf2Og/sMl+FXNbcuXMJpe94xzuw0RQR4m64zbciwHDJXrBgAdfBCy644Itf/CIXbhlwdc6fMIY0oQeawMRmzZrFnVFccccx718HwoMQzEtRmCHUcYXN2+xH/sVf/EUoWRjcf//9DAmIIIrMLwo9uUEe6otdhC996UskGGfPno3l2972tu9///sikFBuajUq2eflvAcGyQTUz372s3llyGpFLy9/+ct1XP/7f/9vJjiFAcIBnrIDGWr+mxBd7GeE+zl3eVfPOEKmNJOeJUiff/75d999t7qeUBmjsmAEjIARGI+AoknoCTRcn//iL/4iNPsXoG388ubKxuPiWML94HKQh3/5l3+5/fbbv/zlL1NFDCItJj/YM9W/zidj+NGPfoSSWigT8ZQISISFgXC1J8LKvqenB7ZADOX6f9tttxFViV+wVmgSzLDOZ+zCIfF/4403hqZuzPsZEtyMSA3X/e3f/m0E6BBOsGcMhBg4MzwWxtvS0vLpT39a/r/97W8z1TP6qhPe9773EfhCybIFJF2vuOIKAn2EeKADMdlAd/mxwS4JPQ75K1/5CpGXMMcDgQxJNjH+EMI/AvGISMHvlrwyGu7nQLZs2UIvAA7h594uvxY4LyQzwUHND3Bs+a9Hfgz7Gti+MM/72f/YuFvBwz5M7OIHBiQczOl3QmV+PJaNgBF4wSKgOFjIH/+EhJAbgVysZcaFmJuCXBP/7//9v6SkeOabgAQPZJebT9za5Nk2gsHv/d7vhVuu8gQAZjMSNrikEsN+6Zd+id1f+IVfYATjCSFkgJRXa2srz5rj8P/9v/8H+8InTJULKG4feOABnnLk0syGcM8996AUEVKn3LSDl/JI+m/91m8RVv/zP//z537u53R3Uway5/JKmCFOfPe73/2rv/orjoKwSsSVDQNjeHHpj4YSiBAcQhhDAokcVNERXYcxoev973+/donKrNBDK46LHwR4ACsO83/8j/9BRwdOCPHG9R2y9/d///dEhehLAjgQACDe3EXmjHCnGaLODxRClwwO5JQd4FDz34S6YexrhPs6d3lXzzhC7ndyUJB8vh48DcJJ5Jb8hMq6UXnXCBgBI1CHAJdftlDq4YhbbrklNPsXICFk6rgEyYxL38knn8xswGhFFTk08l3STEhaGEAQQm51DQ4ORnOSY1zPuSqigTtBNbnWRS3hlaBD84MlhPkx739I9AWnzRMq7CFpGpJGwpItBFANe/+EkNhH2ooQo4bz58+HVRKeePCBPKSUJLX4eSCZRWg4ZFKC2qXkkPmFwCEfICGkCSucwav5wQCRFqeVt/0fCHOjLrnkEn5ORNf8OHn3u99N7leaAxxbHupwJWFfA6N2POZ5P/sfG/OZ+aWh7yQ/b7gxjcMJlXXj8a4RMAIvTAS4oqYtf/B5QsjVhPn9n/rUpyAtcEKZwTe47sclFW4DEdq0aVPeCQ+yQ8a4kycl8ay7u1vEL8wIt3pefDwhhJtxcyvSPmrCqjZ1PJM0I1s4zBPCP/qjP4IvMeckahGIQxwsHElK7LkNzD2zsGH+Bv2yGoo0+yeEIMA90e9973syhgTqJhxpycmTJyuEMMmWizLRTjZclMFB9DU6Za0CbBjYQRFCiDE3//BGkCPvypyicMhxwd6feuqp0LCaK6cjwvmBnLIDHGr+mxDdSdjPCDEYf+7yrp5xhFqWQHfcH3/8ce4XcB4nVNaNyrtGwAgYgToEUhTMxUHycuzmSVedfd0upCJmhTAXg2sy+a46Gy2mrUvW/tkXtaxek2/OxBPG8/DDD0PAYE3c4szXIpOThGIdLCGMMeNh/0PCYDw5YUJHfhgxSJTcJ+UJjnxtXgZYIGI2B0pmxxBGt23bhsyMUG5fIsD3Zs6cqd8P+zpk4i9QHDghxC2zeICIu4f8noHIiZFy4Ps6kJ07d3ITuS5e44cfRXTNnc0DH1seajzUbRMODJvxmIefZxybvm98kWCw0d2Eyqi1YASMwAsZAaLMBM8QchuJ6yAbRIVMIBPombXC/T+stXGHjOs1MrSHSTJMreHX/Fhl+pcZnjwOzsULGXwxoDlJqrwN99jwnNdIxh42QhU3w/K1TLOE2Pyv//W/8soJZa6AkDrWXDnxxBPzBtxRI8eoUUnPYnHcngwbpo7wJB6PAkrDsyJMt8gbhCUCCHCLVM9CEMD4DcHdR/SU3BvmTicySqqk57hI2cFUSUJSFRszHidcLjwMJhTgxhwj6UGSn3jmNRhaYEbGnKD8Kt4cCMfF7WdqD/CUHeBQ45swfpD7H+F4+3B1ICPk6PgZwToBwMv9CARO04TK8R1ZYwSMgBHYDwKKbk888YRseKmPAmJd2d7eDiWQDYFSAjyBixJX9Tr/zIBg1iK1dfoJd+viKYuCcrnbsGED9AkawzLgda24+3nWWWfVKZ9xN8b8jJYTGpC0zOu5AkPz9EIL7lSOf/YyjOFgrFqn+3c8OUKgZ7oHtcz1YBeBe8EQHlJzyGRrJzxkzhETfMLngQicAqL/qlWrmJvDtKPf//3fV6t9HQg8jdjK8OrOO7gxJKYOHfjY9g/1vgY2/qDCzzOO7dRTT+UnEHeNmevLTWq5mlA5vhdrjIAReGEiMAEh1GsnmIrJ8w8k2cj+keIjIAVA3GOTrHuWTAetu2Kyy60p7hFixnWT+5cTrqpCait8hkAYJhjE0+ShR+B5wliaLK+vk0kZQclEw+qqUHLDL5RxIKHhuT7irnZJ9DEMcoZRWydwUApgd9xxBzNetMoLPqEoeuoSusjkH+IfDfeDgyJfnfMD2SXvx7P4sFYe3oDCcb7UCmZb15znGOMOLj9invUpqxvqeADr+t3XCOvM2A1XB/Kl4tcGXzCiO+lQ5iqTjcTDhMrxHVljBIyAEdgPAtxJhCQoT4gZs2a4etdtP/uzP8ujZZrkgk3ER3JHE3omJlIFkaCWDBW3vfJmdS+xqHMC0eLmrK5ytIJw5ts+aznGjIdnHNL4XmiSV+INDVNG88p9yQTWIIRa+RNLBJ5yJDxRBbbMc0Gpg53wkLnzuy//+9EzTm6PcpOauZoy29eBiChyb7fu1GsX+nrgY8tDva+xjR/YeMvw84xjoy2zung3MlDz0ArMUN4mVI7vyBojYARegAjspXlx8Nyu4wYeGxdlQqMmNEYtQlyVmHLJLjfbYI/jN2auU8s9NsoJr92qoja/hfO88qBkRdMJnXPpz4eWukhAL9zarIvN++mapCV3fCHMWictXu5EYAtCiI08CAH9IKjzOSE4dTb5XezzR0EVzyLSOzN1ZTb+lEHMdDP7QE7ZgQ91XyfrGUeYPxzJ4epARkgT5vxwN5r1ckgPwrq1iN+EyvF9WWMEjIAR2BcCXIt4Bp5lQnUTjdmYPMOW37jAMimUq814Dzw9SADigYi6Kt3oJEWDnmv1xo0b8wakrfK7+5LJ9hCzgqmGGTdAWfIkdp+F8KyH9Cz6oglhkeVhmPbJgjQwKzkhrQpt5hkKCGHcEeaQ4cPjD5nZMbrjfCADYDUaHkPIW3Ka8LB/+so7MEjq8iBJ/tQjE2v+9V//lXsBz31sz25gHMgzjo0QzMZt1r/9279lGhcb9yAmVOZhsWwEjMALGYEJCOGBw0HKjkACIxKBzJfM+lCqjXQZKTLdDqzzPOHL7rnScWtwQnsu3wfy6gsu2Uyh0QzJuh65AVk3Y7PO4KB2CWDkD+F+bEoPqjmEkAckuDkHXYzUKPkrRpVfbjv6onnIByKQEwP5OkuobN1d5zoD7R7IKXvuQz3UI4Tc8mXgzgXPhPDDgpwwX5gJlROCYKURMAJGYD8IsHYL4YxUUkwcDWMe9mY5K2p5Q28oQ4AnsBoKE/jzoYof4ixhzepres8ez4axuFr+ph6LiIaH/QhMUuXeH654DC/MuPXJfTHdEQvlwQrPekgTdsRNVZ6WnLBKSu41Ew3/7M/+jIs2D5hIydHxLg2eWCF6BiHkVibPibA4XN0hc+Wf8O7qhJ1C6blvqOmsMiDmwgnpcUJ7KbkvwDOKPC/DlKUwYxjMryHiwMyf+9ie3cAYzDOOjWd23vCGN2jYpBM5I0A9oTIOzYIRMAIvcASeEyEEu49//OM8H1j3aASPgxNQlerhnh9XVRgCFDGPNVM3eVYwrwmZF6zjts6eyavc7iIwhNm+BHokZDKBvi6W83AgC5wysH01rNPzZDxhIybq1NWyy31iYvy3vvUtAlh+hirkkPV4GAAxj98NasiomLYBDvk5q1TBZ3ioXTYHWNIp67vSb9gzWZQAPP7BlTDICwdyyp7jUA/1CPkBFMSem7jckuf30ITK/IFbNgJGwAgcCAJcrgkWPNzFo+9cWFiQjDXGWB4MesCVh1/Y1O7LD7P3uSLx7NkXvvAFrvas+MVtQWbQBOvjoXEiJgtrcUuUe5RkGqndl7c6PSSKm7AMjJwPWUGigN6odIAX/zpvsftchhROQrj22mtf9KIXxe54gaQfTx/wcCZLCXArMwzIFsLc0DDpI5Qsh8bjGyRXqeKQwRO6SNysW2Ug7McLLBZANo+4zHnkjIAhq9dQjres0/BDgqdU+A4wv5QvAMv5MGwGzzMasnyOY3vWA6P3/Y+NZQW4+8z6fPyGYZooERkMJ1TWHbJ3jYAReEEjwC3G2PKrjIYyL+QX85SeNa+5LUqkZBYN8ennf/7nuXnGGxGiFbdCWTIEpV47gQ0hFjalKTfjVxnlfirDwJ5pHtxJJQzDT7icQbriNQ84H79SZSykyb1Dlv0kfceLa4kf3HT80Ic+RJDm9mqMavyBUMU0IWbJymb/q4zKhsfYuFfHrwcYWnhGIGCjr3ujIxm8Sy+9lAPn7iYHxcAIwxymVso5qFVGCSTc3YSKkxnjaTretcjDhBrAMx4XZs94yg52qPljl7yfEWKwn3On5vsfIelfMGdxUSI092vhhLy4aULl+IFZYwSMgBHII6Dwn9eEzG1Ers8s2UI6iFucPPfOOl5EqDBAYAoMTCOv4YagXkxP2CKgkJmpCxA8KU0sYCEW7nISHFlmE2rEPdAJvaEkwvIuB3VBPOUeIu8AJL/Ewiqsbs0ay9wqJRjJYHypZFrdewjrxryfIeFw/IqXdc2xiUHu/7UTGt4///M/AzshTLsq9Rg80SGvRAZPbh8DEXhCNXltFfOSiOlQSlnGKQihzgOZPe4F8yw9HshDEjfrGubt40BQgjbA0oSGJBX56cIE3bzxgY8t3yrkfQ0Mg2fEfP9jY3142CzD5rcQ8VE9TqiMwVgwAkbgBYvAKA3OH/+zIIQ0h7YRKVkbjQjHDUsu63mfkmFlzGEg/mHG6/howkWKSzyzLzAYT2PgjbLnEs/tz3//93/n8pd3+4ykgocbeVaB+6n0iMAA8s3H90jtwRJCvbGKu495z8hQX8CNcJWv/Y//+A+NioGBA8yQ1VPBQS+KyI8qL+c9IEPYeCwe2snNVHrnZica2UzYKn9cMjuQU3bgQ60b3v5HSO0znjts9j9C7tBz4HB+oiZUUAOYUKkql0bACBiBCRHYDyGc0N5KI2AEjIARMAJHEgLEwSL/c0iKiC6NgBEwAkbACLygEGC6AcfrOPiCOuk+WCNgBIyAEQgEiIPP9RnC8GXBCBgBI2AEjIARMAJGwAgYASNgBJoLARPC5jpfHq0RMAJGwAgYASNgBIyAETACRqBhCJgQNgxKOzICRsAIGAEjYASMgBEwAkbACDQXAiaEzXW+PFojYASMgBEwAkbACBgBI2AEjEDDEDAhbBiUdmQEjIARMAJGwAgYASNgBIyAEWguBEwIm+t8ebRGwAgYASNgBIyAETACRsAIGIGGIWBC2DAo7cgIGAEjYASMgBEwAkbACBgBI9BcCJgQNtf58miNgBEwAkbACBgBI2AEjIARMAINQ8CEsGFQ2pERMAJGwAgYASNgBIyAETACRqC5EDAhbK7z5dEaASNgBIyAETACRsAIGAEjYAQahoAJYcOgtCMjYASMgBEwAkbACBgBI2AEjEBzIWBC2Fzny6M1AkbACBgBI2AEjIARMAJGwAg0DAETwoZBaUdGwAgYASNgBIyAETACRsAIGIHmQsCEsLnOl0drBIyAETACRsAIGAEjYASMgBFoGAImhA2D0o6MgBEwAkbACBgBI2AEjIARMALNhYAJYXOdL4/WCBgBI2AEjIARMAJGwAgYASPQMARMCBsGpR0ZASNgBIyAETACRsAIGAEjYASaCwETwuY6Xx6tETACRsAIGAEjYASMgBEwAkagYQiYEDYMSjsyAkbACBgBI2AEjIARMAJGwAg0FwImhM11vjxaI2AEjIARMAJGwAgYASNgBIxAwxAwIWwYlHZkBIyAETACRsAIGAEjYASMgBFoLgRMCJvrfHm0RsAIGAEjYASMgBEwAkbACBiBhiFgQtgwKO3ICBgBI2AEjIARMAJGwAgYASPQXAiYEDbX+fJojYARMAJGwAgYASNgBIyAETACDUPAhLBhUNqRETACRsAIGAEjYASMgBEwAkaguRAwIWyu8+XRGgEjYASMgBEwAkbACBgBI2AEGoaACWHDoLQjI2AEjIARMAJGwAgYASNgBIxAcyFgQthc58ujNQJGwAgYASNgBIyAETACRsAINAwBE8KGQWlHRsAIGAEjYASMgBEwAkbACBiB5kLAhLC5zpdHawSMgBEwAkbACBgBI2AEjIARaBgCJoQNg9KOjIARMAJGwAgYASNgBIyAETACzYWACWFznS+P1ggYASNgBIyAETACRsAIGAEj0DAETAgbBqUdGQEjYASMgBEwAkbACBgBI2AEmgsBE8LmOl8erREwAkbACBgBI2AEjIARMAJGoGEImBA2DEo7MgJGwAgYASNgBIyAETACRsAINBcCJoTNdb48WiNgBIyAETACRsAIGAEjYASMQMMQMCFsGJR2ZASMgBEwAkbACBgBI2AEjIARaC4ETAib63x5tEbACBgBI2AEjIARMAJGwAgYgYYhYELYMCjtyAgYASNgBIyAETACRsAIGAEj0FwImBA21/nyaI2AETACRsAIGAEjYASMgBEwAg1DwISwYVDakREwAkbACBgBI2AEjIARMAJGoLkQMCFsrvPl0RoBI2AEjIARMAJGwAgYASNgBBqGgAlhw6C0IyNgBIyAETACRsAIGAEjYASMQHMhYELYXOfLozUCRsAIGAEjYASMgBEwAkbACDQMARPChkFpR0bACBgBI2AEjIARMAJGwAgYgeZCwISwuc6XR2sEjIARMAJGwAgYASNgBIyAEWgYAiaEDYPSjoyAETACRsAIGAEjYASMgBEwAs2FgAlhc50vj9YIGAEjYASMgBEwAkbACBgBI9AwBEwIGwalHRkBI2AEjIARMAJGwAgYASNgBJoLARPC5jpfHq0RMAJGwAgYASNgBIyAETACRqBhCJgQNgxKOzICRsAIGAEjYASMgBEwAkbACDQXAiaEzXW+PFojYASMgBEwAkbACBgBI2AEjEDDEDAhbBiUdmQEjIARMAJGwAgYASNgBIyAEWguBEwIm+t8ebRGwAgYASNgBIyAETACRsAIGIGGIWBC2DAo7cgIGAEjYASMgBEwAkbACBgBI9BcCJgQNtf58miNgBEwAkbACBgBI2AEjIARMAINQ8CEsGFQ2pERMAJGwAgYASNgBIyAETACRqC5EDAhbK7z5dEaASNgBIyAETACRsAIGAEjYAQahoAJYcOgtCMjYASMgBEwAkbACBgBI2AEjEBzIWBC2Fzny6M1AkbACBgBI2AEjIARMAJGwAg0DAETwoZBaUdGwAgYASNgBIyAETACRsAIGIHmQsCEsLnOl0drBIyAETACRsAIGAEjYASMgBFoGAImhA2D0o6MgBEwAkbACBgBI2AEjIARMALNhYAJYXOdL4/WCBgBI2AEjIARMAJGwAgYASPQMARMCBsGpR0ZASNgBIyAETACRsAIGAEjYASaCwETwuY6Xx6tETACRsAIGAEjYASMgBEwAkagYQiYEDYMSjsyAkbACBgBI2AEjIARMAJGwAg0FwImhM11vjxaI2AEjIARMAJGwAgYASNgBIxAwxAwIWwYlHZkBIyAETACRsAIGAEjYASMgBFoLgRMCJvrfHm0RsAIGAEjYASMgBEwAkbACBiBhiFgQtgwKO3ICBgBI2AEjIARMAJGwAgYASPQXAiYEDbX+fJojYARMAJGwAgYASNgBIyAETACDUPAhLBhUNqRETACRsAIGAEjYASMgBEwAkaguRAwIWyu8+XRGgEjYASMgBEwAkbACBgBI2AEGoaACWHDoLQjI2AEjIARMAJGwAgYASNgBIxAcyFgQthc58ujNQJGwAgYASNgBIyAETACRsAINAwBE8KGQWlHRsAIGAEjYASMgBEwAkbACBiB5kLAhLC5zpdHawSMgBEwAkbACBgBI2AEjIARaBgCJoQNg9KOjIARMAJGwAgYASNgBIyAETACzYWACWFznS+P1ggYASNgBIyAETACRsAIGAEj0DAETAgbBqUdGQEjYASMgBEwAkbACBgBI2AEmgsBE8LmOl8erREwAkbACBgBI2AEjIARMAJGoGEImBA2DEo7MgJGwAgYASNgBIyAETACRsAINBcCJoTNdb48WiNgBIyAETACRsAIGAEjYASMQMMQMCFsGJR2ZASMgBEwAkbACBgBI2AEjIARaC4ETAib63x5tEbACBgBI2AEjIARMAJGwAgYgYYhYELYMCjtyAgYASNgBIyAETACRsAIGAEj0FwImBA21/nyaI2AETACRsAIGAEjYASMgBEwAg1DwISwYVDakREwAkbACBgBI2AEjIARMAJGoLkQMCFsrvPl0RoBI2AEjIARMAJGwAgYASNgBBqGgAlhw6C0IyNgBIyAETACRsAIGAEjYASMQHMhYELYXOfLozUCRsAIGAEjYASMgBEwAkbACDQMARPChkFpR0bACBgBI2AEjIARMAJGwAgYgeZCwISwuc6XR2sEjIARMAJGwAgYASNgBIyAEWgYAiaEDYPSjoyAETACRsAIGAEjYASMgBEwAs2FgAlhc50vj9YIGAEjYASMgBEwAkbACBgBI9AwBEwIGwalHRkBI2AEjIARMAJGwAgYASNgBJoLARPC5jpfHq0RMAJGwAgYASNgBIyAETACRqBhCJgQNgxKOzICRsAIGAEjYASMgBEwAkbACDQXAiaEzXW+PFojYASMgBEwAkbACBgBI2AEjEDDEDAhbBiUdmQEjIARMAJGwAgYASNgBIyAEWguBEwIm+t8ebRGwAgYASNgBIyAETACRsAIGIGGIWBC2DAo7cgIGAEjYASMgBEwAkbACBgBI9BcCJgQNtf58miNgBEwAkbACBgBI2AEjIARMAINQ8CEsGFQ2pERMAJGwAgYASNgBIyAETACRqC5EDAhbK7z5dEaASNgBIyAETACRsAIGAEjYAQahoAJYcOgtCMjYASMgBEwAkbACBgBI2AEjEBzIWBC2Fzny6M1AkbACBgBI2AEjIARMAJGwAg0DAETwoZBaUdGwAgYASNgBIyAETACRsAIGIHmQsCEsLnOl0drBIyAETACRsAIGAEjYASMgBFoGAImhA2D0o6MgBEwAkbACBgBI2AEjIARMALNhYAJYXOdL4/WCBgBI2AEjIARMAJGwAgYASPQMARMCBsGpR0ZASNgBIyAETACRsAIGAEjYASaCwETwuY6Xx6tETACRsAIGAEjYASMgBEwAkagYQiYEDYMSjsyAkbACBgBI2AEjIARMAJGwAg0FwImhM11vjxaI2AEjIARMAJGwAgYASNgBIxAwxAwIWwYlHZkBIyAETACRsAIGAEjYASMgBFoLgRMCJvrfHm0RsAIGAEjYASMgBEwAkbACBiBhiFgQtgwKO3ICBgBI2AEjIARMAJGwAgYASPQXAiYEDbX+fJojYARMAJGwAgYASNgBIyAETACDUPAhLBhUNqRETACRsAIGAEjYASMgBEwAkaguRAwIWyu8+XRGgEjYASMgBEwAkbACBgBI2AEGoaACWHDoLQjI2AEjIARMAJGwAgYASNgBIxAcyFgQthc58ujNQJGwAgYASNgBIyAETACRsAINAwBE8KGQWlHRsAIGAEjYASMgBEwAkbACBiB5kLAhLC5zpdHawSMgBEwAkbACBgBI2AEjIARaBgCJoQNg9KOjIARMAJGwAgYASNgBIyAETACzYWACWFznS+P1ggYASNgBIyAETACRsAIGAEj0DAETAgbBqUdGQEjYASMgBEwAkbACBgBI2AEmgsBE8LmOl8erREwAkbACBgBI2AEjIARMAJGoGEImBA2DEo7MgJGwAgYASNgBIyAETACRsAINBcCJoTNdb48WiNgBIyAETACRsAIGAEjYASMQMMQMCFsGJR2ZASMgBEwAkbACBgBI2AEjIARaC4ETAib63x5tEbACBgBI2AEjIARMAJGwAgYgYYhYELYMCjtyAgYASNgBIyAETACRsAIGAEj0FwImBA21/nyaI2AETACRsAIGAEjYASMgBEwAg1DwISwYVDakREwAkbACBgBI2AEjIARMAJGoLkQMCFsrvPl0RoBI2AEjIARMAJGwAgYASNgBBqGgAlhw6C0IyNgBIyAETACRsAIGAEjYASMQHMhYELYXOfLozUCRsAIGAEjYASMgBEwAkbACDQMARPChkFpR0bACBgBI2AEjIARMAJGwAgYgeZCwISwuc6XR2sEjIARMAJGwAgYASNgBIyAEWgYAiaEDYPSjoyAETACRsAIGAEjYASMgBEwAs2FgAlhc50vj9YIGAEjYASMgBEwAkbACBgBI9AwBEwIGwalHRkBI2AEjIARMAJGwAgYASNgBJoLARPC5jpfHq0RMAJGwAgYASNgBIyAETACRqBhCJgQNgxKOzICRsAIGAEjYASMgBEwAkbACDQXAiaEzXW+PFojYASMgBEwAkbACBgBI2AEjEDDEDAhbBiUdmQEjIARMAJGwAgYASNgBIyAEWguBEwIm+t8ebRGwAgYASNgBIyAETACRsAIGIGGIWBC2DAo7cgIGAEjYASMgBEwAkbACBgBI9BcCJgQNtf58miNgBEwAkbACBgBI2AEjIARMAINQ8CEsGFQ2pERMAJGwAgYASNgBIyAETACRqC5EDAhbK7z5dEaASNgBIyAETACRsAIGAEjYAQahoAJYcOgtCMjYASMgBEwAkbACBgBI2AEjEBzIWBC2Fzny6M1AkbACBgBI2AEjIARMAJGwAg0DAETwoZBaUdGwAgYASNgBIyAETACRsAIGIHmQsCEsLnOl0drBIyAETACRsAIGAEjYASMgBFoGAImhA2D0o6MgBEwAkbACBgBI2AEjIARMALNhYAJYXOdL4/WCBgBI2AEjIARMAJGwAgYASPQMARMCBsGpR0ZASNgBIyAETACRsAIGAEjYASaCwETwuY6Xx6tETACRsAIGAEjYASMgBEwAkagYQiYEDYMSjsyAkbACBgBI2AEjIARMAJGwAg0FwImhM11vjxaI2AEjIARMAJGwAgYASNgBIxAwxAwIWwYlHZkBIyAETACRsAIGAEjYASMgBFoLgRMCJvrfHm0RsAIGAEjYASMgBEwAkbACBiBhiFgQtgwKO3ICBgBI2AEjIARMAJGwAgYASPQXAiYEDbX+fJojYARMAJGwAgYASNgBIyAETACDUPAhLBhUNqRETACRsAIGAEjYASMgBEwAkaguRAwIWyu8+XRGgEjYASMgBEwAkbACBgBI2AEGoaACWHDoLQjI2AEjIARMAJGwAgYASNgBIxAcyFgQthc58ujNQJGwAgYASNgBIyAETACRsAINAwBE8KGQWlHRsAIGAEjYASMgBEwAkbACBiB5kLAhLC5zpdHawSMgBEwAkbACBgBI2AEjIARaBgCJoQNg9KOjIARMAJGwAgYASNgBIyAETACzYWACWFznS+P1ggYASNgBIyAETACRsAIGAEj0DAETAgbBqUdGQEjYASMgBEwAkbACBgBI2AEmgsBE8LmOl8erREwAkbACBgBI2AEjIARMAJGoGEImBA2DEo7MgJGwAgYASNgBIyAETACRsAINBcCJoTNdb48WiNgBIyAETACRsAIGAEjYASMQMMQMCFsGJR2ZASMgBEwAkbACBgBI2AEjIARaC4ETAib63x5tEbACBgBI2AEjIARMAJGwAgYgYYhYELYMCjtyAgYASNgBIyAETACRsAIGAEj0FwImBA21/nyaI2AETACRsAIGAEjYASMgBEwAg1DwISwYVDakREwAkbACBgBI2AEjIARMAJGoLkQMCFsrvPl0RoBI2AEjIARMAJGwAgYASNgBBqGgAlhw6C0IyNgBIyAETACRsAIGAEjYASMQHMhYELYXOfLozUCRsAIGAEjYASMgBEwAkbACDQMARPChkFpR0bACBgBI2AEjIARMAJGwAgYgeZCwISwuc6XR2sEjIARMAJGwAgYASNgBIyAEWgYAiaEDYPSjoyAETACRsAIGAEjYASMgBEwAs2FgAlhc50vj9YIGAEjYASMgBEwAkbACBgBI9AwBEwIGwalHRkBI2AEjIARMAJGwAgYASNgBJoLARPC5jpfHq0RMAJGwAgYASNgBIyAETACRqBhCJgQNgxKOzICRsAIGAEjYASMgBEwAkbACDQXAiaEzXW+PFojYASMgBEwAkbACBgBI2AEjEDDEDAhbBiUdmQEjIARMAJGwAgYASNgBIyAEWguBEwIm+t8ebRGwAgYASNgBIyAETACRsAIGIGGIVBumCc7MgKHEoHdgz1yv3Ngz46B3YVa2ts2sKtQK27u34G8vm97oVhY2belWKgt6ZqNfNqMpWoys3OqhOkdUxCmdUymnNI+SUqXRsAIGAEjYAQOfwQcBw//c+QRGoEmRaDIuGu17Md1kx6Bh92ECOyNaoO7Gf5OCN4ouytsHtgJ2dvQvz37Yhaf6ttyR9+6kZq+qNQU4XvpiIv8x4ZCQtphy5QSn1aOGkuX7VCc3bVQimMgkNk2v3Mm/87pnDajY9r2gZ1Lpy2W3hxSOLg0AkceAkUuKo6DR955PeyPyHHwsD9FHqAReKEgQBx0IHyhnOxDd5wTRjW6I323ZSDl7jb2b028rVB4qm/z3X3rarUiu9yFGC0ljI0v2cHrMBnbpNHemDzKCcdMRv8dq61T52tpiJWG8zSzTLtPPhmmZ3UtvKtvHeXRXXNW9W2mHOWQHcEhj9oxuHt6+5RpWTaShk5FBnoWjMBhiIDj4GF4UppuSI6DnLI5joNN98X1gI1AhoAJob8I9QjsHtwj1a5MSJMzs410mdgdexsGtlGu6t98b99aCBYEj92M5j2tlHIkcbdgarXsFgSavaqxuqeRtDFlstSWacboHKqcRU4csz60/+aGsbcjRpEd6rjRnNm1EBpMeUznHJmLQ87umB6tl047SrJopDlkIGPBCBxqBEwIDzXCTeffcfAATpnj4AGAZBMj0CQImBA2yYlqxDCz+5e1nYN7ICyiecHxIHir+zfRyX19a6OrRG5yeTxN2hxJmpS8GyOBkWvLSGHGhUSIslI+Mkfhd0wYNZB1pkRUhBkzeXb/ykfWNQeQ8//s3D2/rSakkTM6p27v31XHIRmnaeTze7Lce5MiYELYpCfuWQzbcfBZgPa8N3EcfN5PgQdwxCNgQniEnGLdziSnN57pcYSQvTzTQ5PYkngdibY0Y5PP2BxO5Kwq0z+No+WncQZw+yFcWdXevN9+LMPbvoQJu96X8QtWT9SMY1c2klRk5CGhkapdMnURD22SijSBDLgsvJARMCE8Ms6+4+CRcR6f41E4Dj5HAN38hYmACeHhft73fztzVd+m+/tHJ21CmRKRqxWrzMosFlqKIxmv28vCSPFVRloGh8vd7UOlFmoLaKq1FsyEAh6QEkGkOXwxU1Pg8HCAyZzw0J2FiKDnzThJveRpJM9DSunprIfuFNjz84iACeHzCP6BdO04GCg5DgYUDRccBxsOqR02EQImhM/nyaq7nTm6BEutsHFgG2yNJ/Tu61+j/FrK4GWsDEZHPCi3jPQOtreVK7A+yBt0TbXlYm3HQDu75dJIZ+uwlO0t1cGRUkep2lasDoyUdw22HzN519BIiSPvr5bboYSFYn+1xPso20vVZa1L7u5fe/akBQ8OrDpr0vwttUfeMuO+zpbBv11zSc9QW5VWGWmkLbwRxpgPTmkYSc/Gv4lWZnLaz2+q03Hl9Qcr57s+2La2f44IEDjrnoo8dfqy8FmXilw8dUFUWTAChyECJoTP40lxHHwu4DsOPhf0nmNbx8HnCKCbH1YImBAektOR3c7kVQppdZadg7sgZnpab0N/WouFF+Xd0bOhOlKEtkHtEkEao1DiUz2DbShbioXJ7YMIbS0jySCzG6qUO8rVYstI/1DbSHGk1KLWhUmlCqyPtF5PpTy9bWiwWhrOqFlHCySwJE6IEwkiiqnXbJOyvWVkoNpyVtciCCGXOTJFn1l35TvmnDapbWig9uTlG/reOO0lX9lyH/1B8whCInsv6V4g+7t614/5G/0XGzrCfuyNEchQxKRMVaPUccw4S0sGgRwVUuswGJMO4l91MprqPIh2z6tpytA+bQtURrWjR/U0m6bZ4avFXNan+jdTxso6eiSSMlKRcEj+iDyjtWnOa5MP1ITwUJxAx0FQdRx8dl8tx0Fwcxx8dl8et3p2CJgQHjRuEeS29yW+t30orb3JxWt9IntF3kPAOwmGqy0Dw61Ego7WSmup2lKsiRS1FkdIzbG7e6C9s1wttCRCqBHA+garLZQYdLZUabtzuI2UXd34YFPieDjsLlfgVfC9POuD3cG7BqqlzhKJQdJ+8MkqPsdYX8oWImcdtSBPKQ9RSjm5dWhxaRnjf/vMM3g/BMzvjt71vKkPRnJn3zqGzbsWpIGf3tm3nqPOJwHjfQx0ihNKRgsBuK13/TmdC5Fv6d1wbtcCyvO651PmD43jHeM9oz7ZRRk2kpMyo47SjzXJuK94VKYKfTQ/ECHf3YHYHzqbdJi5Yz/YjhJqe1njXjD2onmwHp9Xe75C6r9uOqtoZKys49j5vJ6l5u7chPBgz5/jYPYOW8fBg/3iHIS942AeLMfBPBqWDwUCJoQHh+p1a2/5o8cvq460QPlI1vFw3aT2IZgSP98hPLA7pnR2lCrDI6Xe4VYoWblUrWTP9UU3ImbQOShZqcgE0BrzNmF0+tnOT/ZE8EpVGJ0s1TDonHbF31SiEScMJSwx0cuM+KVZnYUaMmY4oYxWIYRPBJG6ENhVLQTvV466hAzn17bdjUbXJiYNStbsQWRpOJYzoJF0XCuQOQSTszsX3tq74aUZG5RDStFCSmlEEWly/iQ0xZt7EmNU7Xj2qCYJtL1MMunEUYM0pojydNaaqtBBt/Y1pVWun6FM3CpPhp/BvBHV+obkPGUHIhqcRjNWIzV7OtAxbjimyBpk7Dsdw5G4jQXOhMh5M07kS5tfWQfl0mmLWXtpevaWyGkdk9E4FXkkfhEO4phMCA8CrELBcdBxMPvCpBjiOHhQfzs/NmPHwR8b1EdMRyaEB3cqP/3QF57Ytf32no2wPogcjUniwTEqtZSLYyO510+Kr1Ttq5agiGi6ZFmsUcWuyBsVmMEDuaCKraXGuU1pPRRicQjB3yREGY0mIo0p/ag84eTy8FCWMIyGslcXoRQhfNusM5TqTISwCKlbe1Z3YoYiflGiedvMM7627R4ENi5A9/avPaNrEfI9fWvhuqd3LRInhEIrR0rVS7oW3N63/pwunitLWTDyh3AU2Nm5GTOEN6IkkYhlHQ8Ugfyp2acy55YqwSvGeDP5xlEmWbgpY5I0B2RcwQcz+pd+8qXdpOffNGlVTJ59LFtaEjlHSJ+M4UvIbLNmUWSjjb0DENTtjztwHsDAjlgTfTcO6vDi3getzp15kmgkL1mWkxkd00Qg2V08db6nswqWI6Y0ITyoU+k4SLBzHCRqH8zXxnHwYNBqhK3jYCNQfAH5MCE8uJP9u3f9+c07tw5VS9A8UTvaS6CUL9gga7QgI8A6ohb6x/UTVhbMDRoGTYKHqGGeoUkjnhYyQtjkm6AMG8k0DAOE2JWSMvyIKKojlMe1LUk2XOcz0qSYhwKBMqJgskmsLylJBlILh2SWKaQLKqj0IO+sp/a0zkUkCc/oXAQxu713PWzwmK45X916Lw8fUssEVIT0UFnf5jv6Um1S9qXHEZFDA4F856zT4YGJPbLVii/N2KMY43j2KOp4XtcCDgKb87pSyvHGng0vo1WteF3Pxp+bfcqKnm3X7GbAKbvLsFOJ52KtlGl4OFN0keVY0asqdZ3JiVJmNDJTZA1Vke0/rcjO7UEGznCQGv+Y779G380rPIso+OwOto5Dyok4JASSx4ZJRYZn08iA4jAUTAgP6qQ4DkZkBDfHwezL4zh4UH9Dh9zYcfCQQ3zEdUAcTGkubweIwKKO+aXiFpie7IMEShAZgw2KbuXZoIiHiFmUwcrUUD6lzGvydE7MTWXdmFHSi6rCc1hmmcyULeQywbOCcFqaYwZBVUOIx55qK4xI3C+cR+QjGchaIHk9acOjO+ewGirc6RvZVNKoFRt8y8wzv5n0rI8DhRqlXjhRLo7nEs/pXkipBw4xYgzZw4qFd8w6/atb7k3kK1GxwjndC2jFU4vnTlqwpHPOysQe18EYz580D9J410Binm2lvRzymK5Zt/evu2NgLXnIxOhKdF5jCZ8ia/DUCszU/cKWB2jS2lq4cNK8G3s3vHzyPOUVf3rOKSv7tt7Uux4qxumju2O7Z30F+poxVbKXtIrrLAaDw61Tu/ovnvUUf0W1Wsvygak91VbwZFEfphPD9nlOdCwxyaEcADHUGyDpJm3p4OnlwDbsMpgz6xcyjeSvoA6xOGV1+ue4q++tnOTl/buFRh7dNYcMPGakImUMjYRDRsMl0xayJBXTWT2XNTCxcPgg4DgY54L46DjoOBjfh8NHcBw8fM5FE41EPzrrf0I10QH8OIf6qQe+dNmm+3h3X75TremCRn+BLNcJ78qIzGgiLugZNiEjyElwNglRRhe4lc+xJkX8o8GAdUH1019VtFUr9TJeGT4l5LtGo1YImn0eVFDG2tVx3ZNNDSUZqCqVtOJeKelBsTj4z719607rWjRCjq1WhDpetvVeHsSH8qmMtqGUBgOE7JH9wjFds5/q26ImKFUlMwyghZRLumbDHs/OJrWqipLcY5ZUTEvjQMTYSDPyr7gBVC3lFbmnmZEH2OO7Z532n1vu5ynHZFos4JOk5W1ZrpK5rLRFrVmv2EDsyFXi6iXtiy6cv2x7y7XHdM7fOrS6WDieR0Z/uO32pW3H0tfDfatFATlH7S0VHuw8r3vTnNLg13ce81TPNHnLektWDBOeTCuEOuqYYa6aNDpt7CeSncwbuCWvRwyZPEQ8sIFwH7gr0ciwP2360s0DO5WNzOch/UhkQHRQgjOEBwWX46Auu46DjoMH9YfzvBg7Dj4vsDdjp84QHtxZ40+Ln8ws+kJijR/0naUK7UkPkjMcYDmZjOPBshDEzcS4JIue8QM+CJ6W+kw0slhgiVGEISYqZiXPKD6tVW6YeiYw44RVRpLvCCs0soVdDGfEVaPtLvEMISnENGc1nMmGgSFQMg5udkLzMDu6Yw7RjumdJACxR3lW9+jDgezyoKDYYPbEYO2+/rUkA7HUsWBAhpB3YJzWdVSS+9fyDGGWQikGG4TIQfZEEbER/UMQ5SNDKB4YDFANw5JdmrPLhuVLMjYo9oiGWnKPl4093MiZIsVHCM8eXEzM8N2zT3uyd0s6F4UabPA9c06DWqX1YEsjCCnr2J+4bqmYkpN39CZv6GPm6p19a98z53Qyh8dNm37LtkfuGxz+wMIT/n39qtM7e186/eRNfY++dfoZPIQ2Z/LiV84/6wsrv3vpgvO3Dm6/v+fe8+b+9FM9Xz9/8plzittu3b2RLCLfqHPTIjp0uk4UkXORbifU4nUdGUccLdKAs2Gnb1HiiNlM10Tjsm2UZGbf0hxTfBptzOnHmu39N1UecE5yb7N9SGkurhz++Elm445iHwf341Xz55NPQn5t6z3771/TWZWKPHfGaB6SJrM7ps/smLq1fzdleiqyVpjWOWlq56T9e3OtEcgj4DgY4S8ELpyOg46D+T+TMdlxcAyJ5/yv4+BzhvAZHDDZzdtBIFBuqek5QNqI7+lHr35ni56J18lpyAhJk/14F51Tc0oc8otZCbo8o8McUpeWDK22wBOwwVIEL6rQY5A8ZymmYSaFaqdQmJQWkhnNGfYyHZRtrE59UY7Zpn8hG4Q3rQqzamAzR5SfCHpv/xoeBaQ7loqBNKR8YP/aulxWqirU7s+c3Ne39r6+NeKE+nlK1k5ML98pcj4NCDOEzkEUZSOiiKauYZ5Ahg1m0QoZvTjnO2efBmnMOkoPK4oWAuZtvevI/pHxu3zbPRws9I8jYqjMKX37rDMu33oPMh5wpYThS7sXQHrv6k/8MLNPk2DPn3VicVthQ9/2S6edTRfr+7YPVMrrenc81bcVAvlfm+5HefOur+vrsavn3geHSpdOLy9un1vtbuHhyZW96cFIUMVMj01mR5pIOGycFz/+5KzEPBnz+TNP/Ls115yTrftK6nIkuyVB9pWW/D7DPwIlUqLl2dkM6siuukj60VrEZK8c6aicdGmTNwljjtlTD1Fm9Zm30RoU8pkJyXUaSNpC0O4zlWo2OgqOKNc8DTkbSd6HlHnNC1dO2NWK/K3xrbh1z0buEVy/cwvfEwS+M7z+NG4PgGu6oVAsvGLqnAvmHMdZn9M1ddn0oxZPyfLkL1wIfeTPjIDjoOOg4+BYJCLoZjGLYiwWjf3Ln5Lj4DNfTxpu4Tj47CA1ITwI3OZ1zCKrQwMRP9EqflRFZg95VJm9AFCuRfyiGyXuKPmNrjLP6JTBk0Zl4p8l3k/YwocmTEqkLOsClDlVE0RGNql1iJdeIPPiip5KRgLHOtbA2OtOL8ZIGU4llII0Ms5I/Y01SmvG6KH5t8w4a1X/pqTPrnrkADHWM4TosmcFRxud3nUUxJJydD/3j4iZKFzMIA0ih2EQOVmqKfbs5quQx7fKN8/XohevkzcSgJoCqos3uy+dlNZQ5eaTllflJzLEb3QOarHAsjfsHtM9h7QMFJGc4V19a6GOpCVjhdW7s4wiTsQnVw1sKrYUWPnmvJknfWrN1R866mKqyBmeOmNZMaUe0xb3usotUMGFrLgDt7xrzbrswEejCV2LwTJIPtJKOG/SvFt6Nv7U3FO/tOkBHn18sncrPo/tnv3FTQ/89pJX/PnK6y6YNA/NjT0bEShTl9kBp7OXfXnwhgBt4HvDNypx+6RJyowqyB7CMDqYZIfN6O6YNvtzSKapddrG/tXe3jJrHOFyzGrUdzLLGYQ2CTk2GN7CIDQvREGJGo5cqWby8PA9/qjTor6ZkE7u2OnmFLe01FrLvAg1LajLI7XcoGJx3daW6mO13U9sfgJ9uThSWjvy4q5F7178lhfNfNELEVMf8wEg4DiYQOIaVkxzYRwHHQfT4yfaxiLb2L+j6vgHu+yL83TrsdZocwahTYLjYGBYJzgO1gHyXHZNCA8OvfRy+RqvCqzAuMQDIVrieHlehwxPUxk5vegpy3mkVxGKywWjwwBvlHCzqGVXBmhoKC46mP1o17UlLRKTkUDM9gy3qReRvSCBrCbaUylDLLms7Km0oucpxIFqGYEjwmdfpZWZrnf2rIcA5J8hxP70zkX8/IcUZcwwZRju7UvZQmIhnywrmPpkQdH0DGHnotoI8lFQDZRoaH5MZ3oUUAOjFFuD5iGIHIag3fH0D73sg+lpV/Yha5fmIdBd5BuZRMobEdHweCFpN9KDZOHgV2jOm3kipE6kCFKnuXbB2TDQhoZaGcAPIZDoET541CW3bnuEVrzyDpu3LDzvvh0rWDhkw8A2SN2t2x/BDL0CJx3hgR/lHzvuHR994jLabujbxm1EGOPZafHV9Gggecg8yeQVlzTBw5JJs/lZDwhLurMZs8UCC6Jydlgvh3VxWAKnpTjyF0/96ILJ87VMTuJtGceDNH5pM9TxVObKwg9/Zu4pUMefmXPqFzY98LOzT2F4SZh7yr9velBHqjIFosQPRQNTR0kxqhHbIPGdAlxSjvHMsbZZLioX/dCn70Taxu5nZPuZ19GqMQOZTVDKeKwi1x6nY9oj7N90kNy+ocgmA6dE31i6LyN+6eYOf218WlrS2RH34zvDqSmVoHyJBLJ8LvSP7DcCJbV8YIOJHBarUMGkSbXVPYXt/7LmvvLa6jsXfOjs2RcdYWD6cBqCgOOg46DjIBFH0ZCA5jjYkAvLfpw4Du4HnEZVmRAeBJIQhkr22xcGNfrjOJvVKRd5XoccjC70Qc9a07zO7Oud3lBfhVhqkRgsxfdwCOGkpAmBRx7QcPXBkt1JpQokEBfMJt011K7f4hpG9NJGtgD/6ZdkYfdwGwuNsosTDKCCYoy8TREeiJxekpH9TMdGt7sQjumcS4aK5nf2spbaAqgdS8vc1bvuzK6j0FBSxVsKYRxpomn6Sbr3h/7dfeshlmNNstdFFEafIaQVhC2ftRN/QxNUEA20J+hfGsPYLNDgltFKyUY1lyUlmwxU0pzpl7y7An1MHEWGsBWKI7dknA3Cxpg5RpExTrF23zbzTOGgFGJynduIi3A5FDBArR4JzYt6cUh2MYN2/sOaq38FBpjZQxpRwiSpTZ2OTXlFSaShpK16BHxItSgrycl3ZEQU0kjDl02e+9/b72E6a1u5+pUd97SW0xsdb+1d+3Im/uGlxgI5M28lu9hSuGDyvJU859lSePnkuQu6pkMPnuzbwgqrX9pyPyeOg2UXejDKFcUY557yZE/KPbKxHOtPz06M8efmvnhFlpBEeUMu95hZZQEyY2bpO8kXNCUWEzNM8TJLRWKWvmGySW2SPlOmcvyWHYRMqMxuM/Bv1mT0n/SXlBTZv0mu20ZtVR92Y9qxf+saPQ+7afzZXxACZ4S/ysj1haA/WwwBUDByC6C1lIgfQro7UCgkKgjBK41wbeGEZvQvUT5kFjcSOcyYYWKMiRYWK1S1FqvtxQpnLdFImherN279v0/s/ubMtsUnTnvl4smnPQ+IuMvDEgG+eI6DjoN1300CluNgRBMupKMxznGw7ovyTLuOg8+E0CGsNyE8GHCzX7T8hGLKJc2gfJRERzG6dJM+I2+igmJ08DFqs+ReAfYFGSOdqF+1cD/0StPJOIaS0nfZGwv5XUgPlDRMtcW01ggTw3aPpExg/DrsSFm+9CAiDjEWCVR3mNERrK+/MnquRQIHyHPmniFkSGQyeWcG9OnoLKGH8+zdD+q2cGfPBvJXOOe6z0xR3i7Ih7qzoIVp5mGBJ+JgifyihcakNtnv1UxOTfgjv/PpzxAGr8uME9+bkP6J7GEjXheMMdijWuWbY8kupTgnJc3RkL4TISQxCMFj4ijpwfSLuWUE4oeQaG0ad+GXF10iishBwdOeYq5ssQAr41xzsIm5FWvsfj1etoGyH4ac0o9EAl7RgcCrMsTiNAB2iZe0umX7w9RiHzlAGtLqg4su+ce1V+n1Hm+fdTqjxWBB5wx6wYBdWvFs55ndC7+x/W7ODuebM85c03OzvCI8kOcM/3b1tSl5mBZKncVgOV48l0tVShggHjgAnlq8fe261lKBRXRu7V3/NOrYD3VM8wk5QHgjZaGlduOeDUxDvXHlehjjhZPnfnHzA/hhYzIqoFHesGfTz8578b9vfBpXlA3ly2cf/6crr4dG/tu49COjkhl+9gqJ6vDZG1DZTVQp/ZP9rxCrBhOVmf34isxBFm1SHX9ZY52G6T4aRv1eAUuNOGsSYiaMeRn7d28rJJnSO0L6ZMSPs5k+mvOZpnPzpz16laAJfjgpQJSYXkoIJyHN8GTKQPbmzMTushxgmYR/C/eNapQQvKSE3SXKBzNMuUE67MjoH3wvY30jlVpLd8tQ8lMYaStWpIQi0ik2xdp1uwYLd27+QrX610umpZnP3owA30h9Gx0HHQcdBx0HuSRmwU7BDTETMtVYVf1VU6aOg/W4PN/7JoQHdwZ4eI+fqoRDfsANp/xeIod888nacRdf3/KggjCx9IrCYkr3YaPUHP2VSzVeVYeAnl+EYmgY87MMh+gzbslf1WhCj/k52NQNFG+D/IobzSKOslMcijriLRlkU0lTRxknVFnnJ5Q7h9tEpUSxZCbNWRlTgvZx1IwNNsWw4Yd6hlDMMCsTGzyzc5H4GAQZJoOGJqNXjNzjfLIZ3xc9Yq2Sjs5Or4KoZRRu9DHCukFqlxLeFalCuRXnDObJ43zZQRVxq5mipAfTgGnbOYfVUDkcHTVTPRN7yE4ntVLmk3ho2BXxI69Ic9mcN+Mk9JA38UmUNIcKyj4xyWyTzzwDRKPbq9TjTblKlFBEShoyPL5nWvWHBzuP7px7+ZZ7STlCCAGHVhzU+v7tfDPhkMh8UN4BwYMKFmtfydZc5YBYRId1TVkm57+23McUU77MiToWRB2ZgDpC8vD8yfOeYlWhlsJ/bbuX93Pw4NnfrLmmtZwI5C29Gy6cyrKoRTww+xSSuXTSLLoArNS2ewak8dhUlfKKaEkhKotIUhFEoYXkGH93yYU3bHl8afcslIwBS9kjwB7/JGOPdZNXqWLDAxuHg2dkCdKI7VEGk8xsKRKT5OuaDVI6uVHNmFVmFjsIY0ZJh6weMwNErDNxtIidTEj1+u6MVtM7klgfMkK1OvoQrx72S/b8tTP47L4yxwW1owRSGGDigUkmd5eIX5oCOsYAJSTulzWB/tFQxA9OWJZ9IaX7sExpQxpmu1kTGCCXIW4WFFDiHJDaChWxx9Q2u9HT1lLhYNbu/KV1O0dmdX5g6cz3tZanjh6Y/3mhIuA46Dio777joOPg2FXQcXAMiab9t55mNO2B/FgGXitomT5+mms5UHggFK7CEg78aGRGVrGWOB6vf0jJt/S7lVpxM8anlB27StYFYZvSOpSmdGb8Lf3UTfwtnRexuyB1aKIJgnYpw7+EsM/vTm0dEgXFni3R1DGBocZu0LM8ywolr4bPGhXJBGLA+xhYzFAadhGwFDEbbVLTHNFFUDWxsnicT5apTJMhE//Jsog1PUSHBhIFVZNbBPJp+Ad2fjFLnygiRCvBVRNpFAVSX2qYjW10VMgxyNEH8AoFUmp4E53Tr/a0jmhvmhabdceP9PTaDGUO+VH+gUWv1gOBvJaDQCimJ3anvsQDYXHapcRMBpTQRarggcH9VCUEMGYXKOgoO9K9b/vgdSC8POAza69kALdtfzj7jqSMJcSVRW5GuV92drgk//2aa9Q7y5ayPClAkQv99aNedfO2RzleSCP2kEZ6oSSFuLJ/C/bppYsZvJ8Ya8401PMSdUx8jwVv/nbVtW9d/NLbHv36y2adcNO2x/h2Jz6Jk+7pX956Hymsl0+ZC28Exlv715/XPR/qSHnR1Dnwwy9tvv/lc4+7uW89xlAaxnBT74aXzzkeHriyd6sOhzFAHa/P5qCKH4o9/t6SC0URUdJjqsr41dOYZKb5Uc9G/fnoVAqEUWbGoY3uj/HJDGSOQhwSNETGsII/JtvUJKutTyTSIuNvYw7r/oVXoUn0L63skq3vMrbEC6k/scuM+6Uvc+J7xSzRlyX9aAmS3D3KeF2a9gk9S4QtY4BY6qm/tIucTexMAuwxm+c5pk+7miMKzRP3SzRPNoUsecjoai2tLUwbSCyRA0od8SeW7cIPGSgEkiatcMVsAGgqg3//5IZPLF3wSLk0lV1vL1AEHAfTiXccTK+nchx0HJzwMug4OCEsh7nShPAgTtCsjhm8R+HBgdUs6pCmaGZN9YAfxI8/AFE4iB9y8DSs8iSNXfG6UAYb1FDyRC4vwxtF6tQ8Tw7DVZ39UJaHhPLtGm7TBFHlA0UCxQNVRm2eSjEeiBwa8TcJe7lclpTDAI1GrnJfTagNSpZertC/Of0KR5vhqHVc2Ev0L3MoeNmFK0ovQSXMDTPkfEOt/ImSVjRBoIvcg3/p9/joARZTR7wxgt3LR99yUSTnRhPoBE2+tvVu1sshZ0hSbm/mMHv2DyeMGZKWhOwljZBGyVBHevyJmWeyywhpSFXK4NXSLjwQyifSyOB1mJRBC7++bXShGrWla3hgRkEzilIobGQA4JJ5oyFZ2UQ8shctQv/oiMznr41xPxigRiUkkaUZW2S1wHssZABvzJbYmc3rEP/k+Lf/3uOXi0AmQsymoli7b/sKsos3b38UivLfY68GKW5PGUg4NinH3zz6lXBFkofwzP8x55QFXTNu2voYaUMeX8QHOa4NA9v1KCPs8a9XX4tvONLLuufDDykvnDKHhVJZNxX2iGYh7LFYU9pzYdcMscRsPAmNf9/8gIjfRZPmXde7gfIVk+YdO2nmij3bllL2bCMDuTzJs9BcOHfZHz5+88ePP5/ylxac9Nl1o8gkbxxfRvlgYuKEaVrmWMIQTeJ1LOeUcTx6TBL9Z5jAP9NVYFSV5SHJ/JHVS+s/JT3sN+N+JO6ScxCgScr7jWYCUy27ULJs2U+QTFXsoiQTmPgYiT7yeFl6MM0UzQyyLF9aYoqnBJOcSmhkSgOS9MM+NUwELw0j44TZMAojPCXILja0TWNIvDEZ80mWNE/0j910kK0JhOyRwgQR3DmVVO3a8U8zZ/1WMvH2gkTAcZCQ5ziYvvuOg46DhArHwSMlEJgQHsyZZPnHlDsq8ZuJX+G0hJWxuAs/m1JiMJtOqWmceWIWHQSFUy36OoFdPnn7vCzeGK3UVk3CcyQbMcM+aJ4YYPBA6SlZaQaSGRlCdRcJPXaD7Cn+ocm/ElAZPzjVhE3UfJSAyfVYCeEBPtgXCr3OgXKsMumDB8qGKgmx/ie7MhP3k4FaYYMQPFALvTA7VG1JbMZBcSkj9yjPqcnMM6Cp2mWWLE8A0haOx2jxDz3jE4NEVuYQzd7MYUYg0fAKR9Xubd6fXlah5ij5CsEbMaPEf6KXWYIOA3WHgNmt6YHDIq88vm9dmjL6ze13QVP5Xa5FXEk23rLtUWinkoRyrhLuBz/8ahau4Ip1aUNswF/UEbN88y39O+CH0LDMpkYVJBOiSO+SqaXq3bNPo4T4Qf8Q4H78FaRDKNSQsSTxyCclHrel9COfUkuBVW1u61//ntmn/eeW+/7HnOSBDSc/PfdU0pU3wh67EntEySopjIHnGFf2bebBxb/M2OMt/evO717AAqovmzT/5+e9mLk6HMex3TNhgDTRUjfwOijov2x86OVzjvv8hoepuqZn49V70lOg129cnsx6mA9cEznMlx877mV/tPzGjx73shu3PE5bBpMoYkvtlp4N6XUO2dqeoqBZHjLl/7M/eYDMyFV2+lLGrzTUyoIupSqPbiZqxzxwnuHMyBW9Yy1SV82eEkwzC2oFLQQq+ketCCG7yDSBs0EFkzKb7TlqgFma1ZmtFpOl9TBoKTDRdDTLp1yfmF5bcVjzSGnCABLDVIawSIaQXph6nnHCMSpIVMCMT+aNCaWMmWdW0fDVKxQH/37H+k9OmXVTqe1ohuftBYeA46Dj4NiX3nEQJBwHs8joODj2V9G0/2b3gpt29D/mgWe3yNOvQNgg87VYu4UlYfgxx9OD/K5Cw2IwegiQgYmwaYQQNj4h54ctRid7SsxSJjCjhaoKWaxPflBqlyaylz6SjRgECcRGiUGVYoNSkjmUkl02yBtkCZonyhQkEL14HWVM+3zGJnKYmaU3JaSJnUWeBpwNF1JbOJgonEooHPoxPjaaf5NN2MMb0cQuxtEKvic9LE5OQoOeVuKc9P6O2adTQhFT7i7bZJkWbsle6i0OSeYw9Z7NIOXXPyf9J2acScnSqUyVQaDkwy7vjcAgEokwOrgckRK+R8mHTtAoc6iSXXVNScOwoQmae/vXfGv7XVnbdLC8xuPNM86iR94GmfGQwrkzTiavGFNPlRuUQ7hfRuHWi+apihVW+SiLiBlzR2VMiVk05+e+mvNajjCAAZIwZBcz5HzbRP86Z0AXb9n2CFNMP7H6GogfJdThdhKwxRoZRc2o+fMT38ruW496KV3A+phNmggHW1r/ZvZ/br13ff82VkOFx1L+zeprOUGQc2afHjtp1i196z+y5BUXTJoPb8TJR5ZchE+IIlWwQfgYC6VCAkkqUntD74Y/feo6+OSNWx+nXNm/lfJjJ5xfKtUuXLCUxyDfsewsaF7WdSovmTJnXc8OSujiqybP/dgTN12za/PHl9/4o92bR6ot1+3c/O4Zpw9Vyh85+hWVSun3jrmQx//+15xTyBn+wtwXZ4QwHUO6MjDVs6XGWq8c1lCl1DfU2jfUNpQ9+kvmrbs8NL2tb17HnoVdOxd2ps/RXduPmbSd8qjunbM7eqid2dY7pbV/Wmvf9Lbe6a19M1p7+cxu2zOrrWdmmc+ema09M8q908p900q900p905FLfVPTp39KqV9Cpumf0pI+3S2Dc9rOn1zqQ+gqDk5LygE+k1qGJhWHpyEUK5NahruLlY60xOhIZ7HWVaxNKnJJuXRyS3FyS0t3saWr2NKRPqX2Yqmt/Mb2IusSt1IOb3vFSP+9tequhKS3FxICjoOOg46DjoOOg0deHOResLeDQADux49OGgzyUy+b8ahpnBCw9Et99EfuqENxuWSc4295OToOdkdtGMQcUTR1TsIefZBAccJonieBmLEbJQJbPFioKjTigUQ7CflkYF1t2EQT0BhbRzTl00AJmETeMNbyLeg1TxJ9ED/ImMyoZWNXfCxspJcNrUIvD6pVK2lkGWYhKGeITTBGTpeaoGG4Y6vOJJdqhVtopBqyWA6HRalHHBHSg5HZoqP5yZ9EynTkhfT+xvxDg9Cb+R0zUfItgQGK+EELJWAPjYQBsmBM1vao0YcGF6anFvFHW/TzOmfiBwECRqcADidhXRmYHkoxwHzeDz2EjSpK5QyRoYtQO4RIGyJjqeZ/fPzbf//xy3lVo3YJe5poKicotZtnjFThEM1vLE5PKiJjRgqRTplBCs2GK27u34nZ6KRT5rLmJp1SC4niuLJHFmdDKWGPv/3o12GPtz32ddjj+ZNYxoa/rZR7XNm75eZtjyWEi+m+LE8q4pa81U1bH03pxP7NTLb8m5Pf8usPXsG8U5q8/eizr3vwijldU5nGOadrGhm8e7esvGTqnOs3PwEJ/Ojym3B1FflDmOHkuVft3vxLC170+Y0P/dGyl39sxY0XzDzh2l2bF3TMhApypwAqiAfK02cvqax9hMFUq8WrL3nvVx658Z0nXXDzugep3TKw4+btj9zZv65vsO03F6y6v3faVX3dfcOtfeXKpNZBPpPTWyKqrOeZZnum5wB53Us6AhKGTETnQODY6Xm/0QRgSuiluaBpVk4BfZZyHOEbpsf/8IA9tRxpa0ETR0dmdL53z8Bnsmf/eITxuzzYzB9jyjdiltKDygfilr7SRgwgAYjMhxcZlspvLFS/q4wgu+lTfl2t8oMSOcKRq4vl1xeqV6ZUIda7frJWurgw9XeKThVmSL5wCsdBRUDHQcfB/F+946DjYP770HSyCeFBnLKZHVN5pIeF2jtbqr0jvAqCDOHeTB2OgqeJm6ERPZMQBsHZpKHUpubRFqY3VpP+5ZefXkCf5HQXPy1FQ7qPX3l1DtlFT8kHY/E9VsFhtHkSSHow+c02pQ2x1LIuqaylV7oT8PjcxbvUO0dJHe9C4FG9u/pSKRoGdxJJTqm/bFYZ9A+vMJbUdiwNiCYImyiWug56BvXCQHk8ValUK0pchZPgjeNbyTK6CB4obziJquiLJkCaLU6TqCDO8wNAFjOU8vKnz26FkuFQw4Dp8VzfeTNOhrBhnB4aJH/Yvw22g76ON5JdZAkZXnKIJcK5008iG3l0x1x2eb0EraCLlJzftIJr9la6f1h9dXZ0vNp+Ef3+Sm7KKAnALFm3XoOkJDjVTRmFCoo6yiZPHdHA/TBgyqhopAghBI9nDvXYoWRNGVWqUPSP2tNmLM1TRHFCPOCQ7y1lftJpNoZiftIpSUJ4481bH4Vl8TSj2OO925+EK2YL2BR4nQYjvG31ejS8NoOS7ycPLvLFY5Gb9X07OHYmebLsTWHVbZv7djLLFA5JznDp1iex+dqqOyh/9aEreHX7ir6tP+rd+AvzXvz59Q9/7PgL/uiJGz992hs/cO9337H0rKvu/e4Zc46pbXh4TvfUV06aC+YXdS5Yt2fnK7vmXb9+xau6573vru9DmlT+waO3wIouufrzkMx/2fQQ/l85ed4Nfevperhaetcx5UuPev3WjZ+fsfF0/tDghPzx9lbaesvtU9oGJpfJzg12FwdhdIkcphcApnPC3YTBkVaeCSynyZ+1pM/mo2qXQ05cEU32Pgn4GZNIJ7deNFi5BtIoyodyePBTJPqSWTYFNOl5LpGD4QJVvrRWvQKBT5oFmng2M0JT5+ktJokQthSrPyiX31is/pDxZH/NxWL1qmLr61qqV3PLC06YSCI8sJqt2YuyZ2lxxv9Jo/f2wkDAcZAY4TjoOMifu+Og42C66h8pcdCE8OBiOL/q+JnVT8avlF4QjyzSFSwOd8HrqJIcfciY3ahKv8Jqo5yNNxZC27RKDZ7VKrxpVw2ReyqtnaWKHGLM6w3VRMJgclWJjuQEgz2VVvmRcTjPRsEPw+zJyMyC956L+LFHLeMUHws2hR4uhxKKkqdn8k8pYkYZmrxAEzlUmbcX+1LDaJ7vCHk/reSZsak7CfkmMWyNMI7o3Jkn8Zr46JHm49lmDIMqGsJ5kn0tHSz9qkexwehRTaiCHCKrO+gcoFKmNGNWJfYYWUetdHpnz3o8J+edc1jTVYvxwCGhT+SMaKIexf3ghOwq7xfcL1/FaNnFBvqkhiplQ8l3TivNwPFE/GSglWbUVjlDZD1nKFffXHsL9vm28oASS2R50DQbykTzsk2PILIgDfyQ3ODoA4ereVtGyhnS8Kscz8YAAQAASURBVMOLX8Uc1PdkDxyKNP7G0a9c35cecWSDEyJ/ecv9PzX7VOaaFlbfdkvfhltWb2CtmuLWlBkDY+gQ2cLrezf8/Ysu/dUHr3jHkrOvf/CKM2YfU9pETm/qxVPmbu7ddfGUOfdsfuqSKXMvW3HXxZPmvv+e7yYCNdJy5a7NtZHilbu2/PHx5//eY7d8/szXv/eu73/+rNe99+7vf+3l73zbTf997cXv/bPbv/n+U1551ar7Fkya/vbus+/d+uQFc46f0zmtc9pLTt6z9OSlhW+svo2fj5xrphWwzhO0cFe5Y17nnoGW1smlga7SECysvTDc3sKTfiNTSn3ZI/p0TyovPTqYhGzBTy0ew8OB/PFmfC8RwkL1B50868haNSmpmA6ZUiQw2STil/39JuJH3LpCPBAS2NH9l8N9H6FyNA2YEUK+k+zy1UwMMfvDb0m0MQU8SGBRJFC7SZttQ5+rDb3TScIxOF4Q/zoORtTgfEdAIQQo3MStRmq5ekcIGP/lULRCj41qw54qBSA5Ua1cqSPk/bSSZ42HthLyTWLY6jGOyHHQcdBx8IUZB7NgryuNywNAgN9bekpQtA1CJWYFMdMHH+Jsomoq5RhjfmNB+SBy+IH+UfJrDwEnSZm9zxCzsA8ZTf7pxDrnMqPUB2MyftEW492VthhJ6EPAntVl9PsPOWIGAqECTZQI+VAnA0qUlFjmjUNGkCUhKllkREt+pEeD56jNTFJSToI8Y4AQHcWQwqyulXYp1RyBJvpIpiTyUbLJCWwQOXpBUKfRCjPkrMVowfRReBosjp/7l225lw9JHjTsQv9YpYZdSsjbmZ3p1RHswh5hd+wmZbaoKcY0+eqW+5gCyis9YCBndC765YWvRp/acsdg9OZAAUbEb366Yy0Z3gKipz0v23aPsnkqyfuRguMDE1Ouj7FC24INwhXJ7FGlxwXJIlLFLiXfAR4XhMUxZVRHCHljl7ShnFDSlpJ8IB8Im3bF+vJt8SD6R9d4YLIopTgkTUCBdUrhgXyY+UnGjxK3P7HoXMo/O+FtlL+w7LWUbHBFDhwDGsIAKRmq2qbm/Vv486HkuUQ9r/hXJ/0EbOaC2Sfw93X6rGMpT5qxGBsyhxdNngtho7x89e2vmDz31x7+1nV9G3gpIjlDHgK8tnfjhfOXXd2z6bOnvwEmSc4QTvSTWTln0tTXTJ6zqXfXa6bM3dSzG/nhzWs4F1956Kardm2+cuX9n9/w0PqeHRBO1rb5sxU3fG317Wd85/d+88FvXf7UHczprbDgS5ayg2zxY3rPcPuUkVOnt7x409CUbcPde6od/SNtfSNtI4UWcoDdLQOTW/ontwx0tQzx+B/ypKQZ6CoOdbbwGe5AKA61ZS+QaC9WOwrVzpb0HCBtKVlElMeboYXkA1t5g0U2KbStWCwXi+3FFoS2VLaM9P0Or5bk01psyVgizC/tEgixLJdfzRzRUTao0xBsEFA6PyadytquP6v1XJXXWD6yEeBvynEwTnFEighPVClS5EuUsoyYhb3joOOg46DjYFxMnl/BGcKDxl+JQR4BipZwOSZb5meQin11soZn+iGYcobZ3Xal4FJSUVM0EWSJAYIYmgRVSa+OIrmn3XyTGAlBGjNVyXNU5e2R6/RhHLENg7hlKGOqEIIRaVel9Mh1TdCjpAkfBMnhpM4+7mUSI7FXpMRYm9oSSmWGgIYqdjWkulbR1/ghhZN/XHNVvqNo8sGjLlGVeqEcG8Xov/m+0uqLejtid3rBPSSt7tlLyAwULnvYssgc2tyqPLyncT0aOf348e/4w8cv+9BR6S2F2KeEYdfCaAvjwiyeclSrDx11MRoSg9i/M2NNd60ZpeWyh/vRStwPoS5tOOGUUQ1GBBIZ9gXr4xMyvA6ZEr5HrajgvqaMqmF4oOFtvRveNfs02N27xjgnx4ifRAXX3qrJopRQxH9Z/gPKtERNtiEzWVQyOS9mlirTSI6RDwMm63j6zKV43ty/4/xJ87Dk1Ny7LU0Z/bm7v4h847bH0iTSSTN5IyIr0/zFyus+eUp62vAdx2Y5wznHtGx6kFY8Ybi5bxfZwns2rXz1lDlfXXEXJYlBmPwPdm2hZENYNmUmpHHhpGksVHPGvKMvHtiycPK0iwbmskLp0skzL150xj/OP/6/Hv4hQz22fxb4LJt+1Iqdq3nI8NYdjzzQv/runvVvmXXmvYPreirt/W19F05/2cq+K4Zq5Umt59Sq17GsqOaLQtGZDjqr4308Fqg8IXm/dO9mLJnfUX79cOW7SgkyMHhgR+ulraVlQ4OfKFWvKKc5ot9OTlKaEECKre2/OjL4Kbhfqf2DxeqKWvUHyC2l1/BkIAwwHVtMB217X2Hoc0nDFspMqPV/VOrRMk2YubrQdpXzhE+D5YjecRzU6VUYUolGkW580FFwUW0EGkUudvP2joOOg46DjoPPS/TIfgE8Lz03c6f8JhRhE4+ihOApQGrGJukIPmKDHGhmNrqLPJ6whZ+opVWewslARI5SQkAYrUQaZSwPKuvsUWITH/kRR1WIouSTp2QRwxBkjwFhDFnGEghmQZ+wDGMJslfzKPP2uMIySgS5DRv8Y8Bu8EB29YlW6iXfl3zKCTLHVddEHcXhKFWY71T91jXEXk2wTO8SzKgCA4bhUMIA4WkwNziblt6RErooOndettQny67yQXP/9uWUpP7EHlP2r3d9MECUNKetmqsJxnLFLvaaQfqptVcz3ffybfewSg1fQlJq0E7oYqwyGmlDHGqD1MEVlb5T2jC/3gxMggmfWCpbqLwfTSacMgr3I3NIBk9UULyRtvLwlkXnndO9AHw0UxS98n4wvW+svZVS9A9+iIwNJVNGoYJ64BAZJQPgsyQ3AO6wMhiMf/uxrzHvlJcfstgMy5bCmniBIVTwb170FuS3LT6H5/3IGbIcKFM6eb+FnjYkZ0i2kJzhK6fM/eOnrh/NGfZt4GnDq3s3XjR/KavOME0USvWNi36S8gOnX8TIX730FOA9efZRF2cEEnKIktceZlNY75vROZldjDkWAGFK7d8++JWPPnHZP669iqWDTu066tKZZ31z692vmnL+ruGO9f1Tfrjttq3Dk3qqHSv7Hh0uvmJGx8/zrDIZPzKBHcVK/+A/UKZkYPaZ1Po61oxhlw98b2rHr7JUDFSwq/ymMhNdK1dUBj/BlR36V6h+u73912GDyHwQaoP/0Nb+IVKCsMFi9crEDDM2mEYL2YMEkgmkZKuuSKW2SA+GMFYz+m9ij0/U6bx7ZCPgOMj5JXZE4IiA5TgIMo6DjoOOg80VApwhPLjzxe9vuERXqdJXaVWSUOxLJbxLwuTycGTqRMbQR21eDqomJaOJ2hBQhlkMl9owlqAq9TLevq7hMxqLRMlMMiW7BD+iXXiTIIYmgyiDoWGjSClaJVkNg3NGF9TijV0+9EUp52ols2glz1KqFaU8U0pDp7TFRmb7akItASxGSJPoJRqSOQyuiFI4aGBZp6MPBMLixNNQQgWhcAgwOmibWCICGvgbZmJ66IM0ouQjjkdbDCgZGwaUkEA1Z1e9RMoxSKMEmdEELkQJM4Ecwlr3lTYkejGq8WlDlEwZhRbGhE80kBzK2OCHaGKiKa6QydrRqWz0ekN4ESSpLukH3+Mj+sdk0d957GtkC2977GuMGb2aI4s6sgv9YzItArWQQAky0/KkLDMDISRbyLOFs7umMZV0S99OXl1ItpAS5kn+8Hce/xpNijtqtw1sIKF368C63zz6FTdteewPlr3sxs1PXDB7GTNI33b02cUn78Ts1VNn85zha6bNvvLJB147dfZn7v0R5dtu/G9eZfHu27/May2KW0du7l/fsr1KK2bDXjBl7tdWXju/aybJwAtmn1jaXj1vxgmzO6e3rL+ZpYNQ/tqJ71o8dUHh/sIV2+/gxYEsGrpjsIsnDAdHylPKA6sGHt08dN/RbbzOkDfas8AMC4d+sG/gk7Mm/cXu3v/DYqEdxfTqQiBgRSl+lJMM1HV8pHpFZ/uHBwc/wZizWaApLVgd/HsWialVySKmSaFUjQx9mvTgyOA/0nVL+wdqg59t6fq/RZJ+kD3xPSUG98P98BLJQ2Q2ko09n6v1X9wy4y8KpanSuTxSEeDKwwXQcbDu/CpUgQz6KBV9FCZURpSJ5nWxhrZYKn4hOw46Duqr4jjoOBgXjUMhOEN4EKhO75iiqzwTRHnZNJRMrKzOBWQs2CBVYSZjaoOtIUgZZQg0VK2M83r5DCdUSaYMpYakVnVjCJsp5SGZqeSgJHCMOkyVKCOYISiYyZIwhgYzhPHGeW5GrQxkH7t552EvoiXmib2cq4maU8LN1FatKGmFnlLNacUHG/WLgMO67uqaME00jBHwpibREDaIPsaJNzY0dMQKoiwcAvVCIwYoUpeZpPVaJYjCQdigcykHmOOBzP+MtjKmJABQ7s3+ZclGdoPy4YHu+OS7o0mQRmUR4aiMMzUcSxuSRSRtmD7w1WKBhWGV+ouu4XjIMDoIpBKG0DyRSfSk/nhiEL6hVkyJRBkTTWmLMW3JE2qlGcnoWW6UdJ8+NEEIDxA8TZXhgUNkmmAAe4T1BYdEQxXrylDCIWn+spknoDx95rGpeV+aa8ou273bUoLrI49+ncVmWIAUikjCjBK6SBYRD7wjERlymN5sMXleWkol+/B4HdSXU/m1VbcXSyMfX3nDtX0beU3F1T0bSbBd2bP5ogVLf7h78+fOfD3+/+vcnyJP+L6TLvmZeS9+2+KXsv7nLxz/WvKQAHLitGN4BwY2GCTPnJTO2RsGtpHD0y6a31nyns+f8RFeONFfbeXBwlfPfPeOShefxV1v3VW7ZHLHL/aNtGPdM/ApVpfp6f3NtrT0KC+puKKdh4oLhc5iIZFDiF0xLR7DBhtsK18KG0w7Y1u5/aL0vWQk7R+ECiJkbDDZjAx+hqqRvj+o1tJQJ96UOVTaEAsyh7BBlPlN7LF69ciuz+XVlo88BBwHFWi4ourkcv1Ho0AQSjTUUubjBbUykH3syrLOXoHMcRBYHAcdBx0H+UM41JszhAeHMLf/9eOOCaIwKzEuUSxk7VKqKgzoI2TZqNf9W+ZtkMM470FdyzLsQ1Cn0RA9JJAFZtRKgmwoJRCiFKsUz/KeI5JFMKM2lBIUvdRKdEs20URuVdJkP/aic3Kl5pQEV3UUmTr1UmdMK3mOLmiLJbtygryvJtGjBJX0q6hP82gobxgoc4gNcQuGRgnvyh714lUWKQdIiR5LlaKFYw8NXqypntRKr9ygdsdPGUWPQ/G9aILApqgprgj3y3SpEGlEiIaypFS+EWaFPQ9AYiPuB/FjZmlqnG16OaGYm4gialiTbCAWWmWUtrALSKCeM8QSWeuLMln09scvhzFSpcwhtfQLbyT1B+UjJTjaWfYPGiaIwgmhfCgYKjL2stnQt10y2UIMUvNHv64qZAQ4GDQP4S9PfOtv8UrDxS+95ZFvnDbz2C9vuQ+lSCN/yyAP8btl+yMQqk+suTo13D5y+8AGXnmfcoaLX3Hjlsc/etzLbiBnOGsp2T+eEnzN1NlpgRmmiWZrkz68dTV9XbPunq9sv2fVwKZ7B9f8+kOfg7w9tPIpngOkqryn+tjw6o8e/d4p7ZMf2v3Y6v5NLNXyqUe/vLhz7nd33LlmYGNhI92mrVIr3bjlsdfMevd3tn51Re/mPdUntg3dP7d1Ck6KLf08K9jdumyYJwPTuwTT5E/GzMUIwCuFWrVWq5UvxQnzRYcqV0zt/utqdR3nZXjw71AO9n2kVHpDofq9yuCn0hOv2QxSrVVE2jDzkRjkCMsZtb+/CNlL8z9zvC4yh/hiG+N+2hsts2cLxRJrQ6v8MOHTwDnidhwHCQR8CAF8OL0qpWF3P3FNTbDETCWa/dhHuJE9xrRyHASNCGfICp0IbI6DgOA46DiY/TUcXGFCeHB4jd7qzxrliZZk1GJr+SqxL3UT7CsMQsBgQsvwGULeLN9cBuoCPbvaZK8y2KAaYiDL2M1HJmoV5yQo2ikgoYlgJkEl+tiirRqON5BlxLbx9kHDwqaOZI4PljF+LGmVH4wGoF7iKBCiSRjTkObsqhQVZDfGw/KkpBPhgeJyKjHmww9rnhOTK8jG3f3r0uOFWcSCxQU3QxMPDSKjF1ekjNgWxop8MoC5KQOpXdrmuR+72nDCM4rwSUosg/tRq4YYiG2yqx71zCGJysQSO+fwknfYhugfKTu4n6aMQuFwEqlCdYcZq5XqvRfwPSxlBv3TCqVaZVRJv/zbKWjOj0utEIOedB+RjPQaTE+eRQUlQx0RFOoQ4HU0VNXoZNGs+S8e95rb7v33OZ3TmSzKAjOU921bQSbwG2tuRdYrDTk1LG/DsCl/PWOeH178Sl55n0hvsaCcIQJmKtOxF2vkCVG9757vQZ3gh1f1bF42JdHOBV3TW7YXfmLxua0bKnw3Ng9smdc56/hpS27YdPu8jlltu4Z3D+2GcpEwfOP8C76/8frXzbtwRvu0dYMbPnj8T01tn/ynD3zmgyf8T/x84N4/e3TN6jfOfsUdPddNKnUyOxRumQ6wXCgO/yBRwcR1i63lN9Uq30mjIzdYKLa3f2gYXsiraAY+0dH+4YHBT/QPXgczxJJvY5XVTeF8le9oN5sTwqOH6Y0U6Y03eJv8z9U978scFIqDn2ESaaGyosYRZv4pR+CdnCYk/suIH/6KpZSir1WvqrX9Eq6qg58tlF5dq1yJy5b+m8p+VX2C94jdHAcjgkRYkTA+zCni8FWgKlqN/2ZEjBtvH3EnbAg0+X4dB8HTcRAQHAcBwXFw/OXlADUmhAcI1NPMIFfVWstgtT4lmDcKShYCteJd0oQ+r5RNnaXciumFgZT5hxXDIVWif7JRqdpwglL95o2pDfKj0KUYJrkumBGcaKvohUyIQtau9Pn4p+gVetmzG5FMzlVG2MOATcbyJg1lOBc3YzdahVA3JLXVSFTSVgNWE+yxQSOleqEMxhhhWPlJeGBdF3nnNJTb87LXG6ZXxUELu3m7Q2I10DARs6TMGCM0T2QPA2SIEAYictggUFKFcT7yYSN+GKQRBqi8H8PDWGUd91MSMt8QjXqkl7SxGk3XbAgYvA7uF/QvMofwQ7gTKUFKlDFlFMqXZ4xqCy1BSbaQ6anxqGHwRjyIKCqvSEKMfCNMT9NEoX8QP/ghZWQIacJxMeD8u+yV92PsV6+7h5IFZtKBbCswQZSGTBZNTwlufew3Fr+S9UjPZ6JpLaHKIjfJbGzDLQT15u2Pwn/+bu3VPKxY3F67uX8jOcMb+tf/7jEX3bBhOe8kvG7Divef+oqrbvqvS44+/Utb7z9pxtHn7EjUNxHIQuGa3bf8VNebbtp82427b3pFy3mrhpf/85P/hn75wMqNQ+tWVZ66egtP/xWXD62a2jF5Sttk3ib/6ce/hAGLhb5m+ktW9m6c23Li+srjrS2VtpHhtpGOcnWkvfyq1vIxAwO8C6JWqHx7UvdfVcj7wdA4V4P/yMsJK7WR1ux5QkYwWElvny/B+rK3zydOiF32K75jyr8P7fl59qqJ4qUc48juX8AMTUb6mET66SQlz1nuMMlI6X+4YrGyotD5sWrf7xcqP2R3pPLDwuBnqjJN77Kv1arfrw7WStU3Fv0kYYL0SN6IF46DXOQ5xxFH6iICeoKCSswUIGSPTFsFHcdBMHEcdBx0HOQP4XncTAgPAvxpHZNlnZ4hzH75Bb/KkzqUMgvGlSdjeVmWaPKWE/oM4ofnvHF+WdHoNw4pbym30oQ+LOWWMqIXMSzCWDAiNGqCmSKf7PN6DNiNJrKXEuOwR0Cpss4ez9KLnUawlD3G1IproYm5o8hsYSwzNBoJu8hyG2F4wtdOyLnitHqRnNennsZeW49AFxqq7OGZGhVVtJKcHjLMoQft+eCiS+BpsDLRPFE+moitER35iMihzG/olf3DEr2ooAyogg0i15FG9DJQE/VFKeooe/HDp1HHLEMIqdv/lFGa//7jl8s/RFGkEYH8W7Rlyugdj18+njfCEmlIKUFOaAhv/I0scSe6yPRRqJpqYXfIo5NFaZ5RvttWJz/iY1BNZN0rfetRL2U2KQvM/NfW+9RcJWuBwo/ghzT621XXooT7kS1c0jnn1t4Nmq2q8icWnltYc+vps5b8vFpCj1LasPbZ+6+9ZPLc99z65VdOm/8L93yho1wp7aw+OriqdVeFtN62oU3llsJJ3YvLpZHu0tBr511E644tlVOmvGjj5kdfP/9CKNb6NY89sP0+km1PDa/48rmf3z205xfv+s1TZxw3f2DmlzZ8p1Rsh63h6oRJb5xcqqzp//KMjpcPkjKsVRjsnrS6TEocwuWqhSrMkHmkPGXB6Mj+lcqXVllrNOOEZBh5pJBL0khGC3t2/yw2WbYwW5lntIkYYGqeasuv43UUjFBrzyCwoU9PHpZeW+v73UyRdqlisqoMpEyWtULvplMnLVgVGgtHEgKOg4omnFMu746Din2g4TjoOOg4GJf6ZoyDJoRx+g5UYHFRng7qZ/G/jJtFM1iW5DzpCmUY19G24GZ5S4zzZshapUZORO3UlyzVts6DPEsZTSCWPP0YbaNJGItHBZuSZVzxgxRFRJSAPjThHAE9Zb5Kcp4uBrULIezlVp3mGReWQRoxxiA/QvWoLqiKgK1W2I/3n2+CrCZ4QI5NrdSdlNEFuxqk9GKAdR3FUUQr1heBFvJDG5LGWyIgNtC2/RA5koH57F8MDD0f7SKI+1GKNFLmG2ImYwyexv0yYkmtpowe3TknPQaZbVC7Z5wyml+bFFlTRiFmtI0po8jj4yVZQegfbFCPGv6v4153+33/+qqFZyrNSP+ap3rLtkeQRRrhH+d0LyQfSAKTEsZIFpHlSVk+VMuTzumaBjPUAjOsLIrN19feSqnJoolAjr3TQtlClVDHwprbeM4QjqMDV0lbztH/efQbIyNF3i8/MtJS21S8aveW9y846TPrHv3c2W/i7fafPOWtV2y4+aXTTzh2aNa8zpmte6onTTlxRse0b63/6gmTT7q75xq5gt1Bq9pKMLW0tZeGJXRkAlUndx2FBuFFXYtfM+eiT6764nD/jNbild2lwUktXQ/t+usZpa5qYWC4ONxeHGEtGeaUlmqJ75HIpN2kKV/cs/tnWnkBfftFvbVCtXoFR8JfO3M+uTYxgRRySP5TGnKAaTAdv1oZ/CSdoqSq1P4hmgwPfqrc/iEeOBwZXs6DhWPPGWaDrXwvzRdNbXmQEd74fcpSeelIZcVI9ftZSrHW1n7hUN93h/vvae08Qwfo8shD4AUYB4kLOo+6mCtMSKPogF5C3elGjyZfJVmxQMYR/kIIe7lVp46D+XAWdwnBEL1Cm2THQcdBx0FdWw68NCE8cKySJT+heNNgNf2uS5xtiMdzakUEeRGtQhbRCrol49jNC0HV5CF2w0beQl+3q65VyoMM1Fxlvm0dsVQVZmGTD1G4IiZFMFNYCk1ej6V2RbrYJW5RalNDGagMkoaBomzYy7/KCI3yI+d4CGP0ch7DxgYlzSmRFT5lI2U0QSn/aoJem7qu60VVYam5pmTY8r8JYlThQQOIsY31MDowdqMXvk/8qqfhO2an39BP9W6GJSq8IZADVE6vLvuHHhtFPlqpSZ40UkXIpKxriDEN81NG0Yg6ImjjMUIeC1SWj0SflMr+IcPulLtT6k+1MECInOa9qAmlBPQIed7IA4fYY0zakNo037JQ+Ncnvg9FfP99/4pD0T9KTSVFg0CnUEQa3pTeZX/eN9J7LNLDhCxPmmje9hXs8uIKSqZ90hZUKVOub+ujrCxKSlCrkuKBVpjF9vU1tyH/9tjiNIlArro2bH7rmItoS4+XP3X72xadVVhx9xlzj/lAoQDtvGjSPK4Lt/euP3/2Cd/bcccbW85aMfhkeesIt402DD/21Q2PdLSM3LHj+zjfXn1kRV9lcmnwzp3f5U7ApNLgqr4H4beTywNfXP5PGwY2tLUUX77g/HU967619fJrtlZP7lr8xMDKtQPTj+rYAUtNF52MxVWLxcFiFXrZCoUr1ijLvLmCQ+37CjZQO9Yj1VOF7HJtYgHSxHFrta72D/OOCmihMoRYMnW1UHrDSOU75A8xqQz8ffbgIc8EfpL9YvU7KEevbhhqCIkx18pdf1YdvK7U9ae16rpqZXlLeVmtiuO0DfX9Nt56dlw6vXOtNC6PMAT4XjkOcvXmtKrUNT92ufLXxTVV1Zk5DipoOg46DjoOHg4xgp8E3g4CgfSjaWwLzhbsK/gVJsgyRMjLqgqNyFhYRsNoJcu8GVk+7VKlrmmFEJ+8t7wsn2qrJnW17OZZDbtcqYlhCmMqoTdqRSmCFDQJA5pTyiDiZRigp1YG4TmMZZ/3T7yUErcIch7NaYhSzhV90SBIxpiPelQXrAGDfTSRMprgVv6lZxdLDUZdsBtNlAPc/2sqwlhcDoc6HHrRCHGLki7UCyQzBpNyN9kDh5RifZHNS/quhXpHBRQOGR7IJzXIqoL7kWyE9al51NKQJm9edB4aBkat2mKpu610hHzZtnsgabA+NWTKKAJkDA0UkY/0cDySh8g8TIgcK5SiEWNEgARSkvqjjA0Poo60Im1IKRop/skLLeB+KvVGe7rmQ61yjxjThEcNs06v5lUWpBBTujW7ZUO2EIFsYZSp3wxSmJFSjjxGiE45w9HMYTJKr8HQB5lJp5RkDkefXawV7926AjeXP3Un0yLfd9f3P7320Y8+fvO1uzfxggr6BkDKuR2zIGm/fuKv/86LP3p0+7L3LPxgW7F6/szXvH7eu4/uOP78ma/taBl+3fz3sNterJw142Vnz7gAzTkzX/bmhe/gZTY9w7tX96w8vuuYdx711taW6sXTz+W4dlQ6e0Y6+mute0Y6dlS7dox0bR/p3DHSubPW1jNS3jmCvqVnpLZ76IpBVoOp1TrKlzLOcvnShEiW8Zw65UsIiQpmSqaVUsUVanDw74Yq3+YhQyhc9knpRH3SXFBtmRMKzOIz0PcRvFEODX5yuPrdwcFPRhUCzltY19TbEYqAvlc6uHwMQqP4omDEruKXhLxcp4moFE1krJgVyryZ4yCwKCpFUFNkQY8g2XHQcZDvg6IhguMgGDgOpm/CRJsJ4USoPJOOKNXWMtJeSpRPEYtAhaxSrUPOVyHHJzqZ0DK81dWiV5ZPfuhdgryxK2Fy6xB6jS2qwlJV6BHyJfaEFnEh9Ah8EKSURqxJcShCTljKmHI8rwtX8kPJFmawI/EulCHILWWMSvbsyiYsM2epUC8I2GiQ0QQWl69F5ljyTbDko1YqI77KTMeedxK9yB4ztmCA2oU3SqCkrcYTGgk0h06EE8xY85NnDrPJgLWzuxaIAaqkiVifKBykTqxPkQ86h0GeNLILdaSE+9EEg2+tvQWDUe439lZDDNjEGEnfZZMCU94Salc3ZVTpPlE4ZFrB+jDTKqOUkLfMWaKFEkj9IYs3kniUB6pQijGqiR4yVBOV2bvsCyxAyocHC2mozCGMUbwUxohMohJ75poyYJKNrEnDmMnvMY+UUi9xYsAsMIPB7X3rju2exec9c06FIkL8KCF+x3bNInPITFRKvEXOkOcV/3rVjyCBVN3Yu+Hl89Jr6//4xPNeM3nOV897N5NI33vCq0dqxVfMP+vFnYtntc9Y2nGsBg8z3Dm0tVys7h7eev/O67cP37+674G2lsqDO697cNd1naXhH23+j/9e9zvHdBx/9eb/+OKaP4QBfn75X924/Yel4sjt225mVt5dPdcu7VjyngUf7B9p3TY8aUelG064a6SLz/Zq95bK5K3V7l3V9t5aefdI6wAsjBwgDLC0rKPjwx3Z6wf57Q4/2937FaJg/+AnWG8GAaU4YarNAqSUcELxOo1ftbDCepaYTUMdrnyHquQnZxCWFbKO1V3y4/KIRIB44TjImXUcdBx0HHQcPALioKeMHkSkntI+SdbwqNbiSJ5NoSc6SoMQZpLRR23eMi/XWdKEX65q1Z693pofZ3r5Ia2o7ShVKJmzKgGlFp7pSDS11lNpRS9LSm04RJBPGrLLL0V+99Nw93AbDQdHWni7+lndC+/uSyWTBnmQLC2FUqy9fdbpX9t2z9tmngH3wEmQosxz7azuRXf1jk4uRUN4IEbC1mSWN0YO2iMz1Qb1EklTFxjIPlxl3SXGiD21dcbUQrfkKmzURH7kMJxLUJMYDILaqiElvaBEwFKyxhOLx4S9utZaNdhrN/xroil67CnZcBuHwG6AozGwK0v54XyJAWIp1iceCLVDQwnrg+PB6IIuoqQJZaQNsdSGh7opo+KxY/WJPeqBRl5hzwfCQzouCB6C+BhEDr1awaOIi/qgDD21GMd7KdjFBg+0FasMt9A/5Mg0hp7lSdUKP3zoiIYx1xSKeEyWqIR2YqbXWiAwANKMWPKUIKVYMYcJRaREwwDgeJBG7ptqoimtEoFccw0liUFq3zP7VP5K6JEpo7zPsLb6Nmxiu3LlA6+aNPfq1fcydfzfV3zv0eE1K1Z9sbs89MnH/pZ039bhRxcML2hNhHA9j/y1twzPbpu7sq9CyZ9iW7Hy0hmvqxVe9/1Nf/naOR/ZMXT2Yz13njPzNSt6Hto4uG5B18xd1acu7HrZPXuuXtk7d2HbCRuGHtlZ6YJPjhRaZredt2PoDlYlnd79zs6WkUWTL3li69uHRkqlEV5ucSnrbpMe3NX/96wM2lYcSVeE9BaK0VGL5nEvEPKGiguBKtJOdgeZfWihlKrjiUG44lhtunIg87+MKDErlt9YrXxbesqO7r8aGnyyo+sMZG9HEgJPj4M1ogZHRzDSMUakU0RTlWRsohb9hHKdJU34tsnScVBxRzhTKu7kg0hEEMdBx0HHQcfBuFYcoJAu5d4OCgFeKzcWn0aJH83zsU2hMcoQZBZtEaJfZBia2J2oGnQuZ1AjVyA2iLe8Q7icnGA89nwgEXo07spS/UbvsqchTTL2mFKOBF3FdWq1GCYlDJBSr9HTEiNJ07+WD/ZndS/gkwlQx9kI8MY0/zC9fC/ZfH3b3SgRxswWUIVMyeeuvmRDd+JaRDINjF1p3p6bmwojklJ8jBJ7BT/piX9qLu4kmRIbGUujJsHHNEuTqgi0eIsmCGol5+qIMprH4jExnjp7+Q+ihT3NMZY9Q9V45FnGOA//6LGRGe825Dc3+AtncTnQhuxRJtizKYtigBrGvqbKUCtqhAClVFv4IaSR5vm0oZxj9vHj38HExV9b/Cp++C/pmqNsHhwJxgWtEoVDDiJHE5T51B/GaCh1J1VkkrZqJT+0Qo+sfGO+OVWqpSRPyLRPSODvPX45JU6yylQoT6hZpnWTVKmFG5MJZHIp5d+vuTpK8RrNFOVpQyw16ZQTByec38XE2s2aaKqc4Z+svP7ano3Xb1j+wz0pO3rlrs1YVkdaLp3/Mhae+dDin63Wim9d+I43zX/ngvYTlkx6Eem+U6Zd9KJpF81pO5W+5redsqD7hKlts+a1v5jmu4e3zG17EQK8LGa9njz5rKO7j+MpQXKMZ0951VkzLqDqzCkX42p3taOn2nHM5LfO7Xw3ecKHer65s1q+a+u/7BghVdi5oe/zW4au7KmVe4a/O1woDtVYr4b3VKRJMiO8YSJb9YXpo2k3mzJKFSk+mKEShgxj1FhN9paoo2o0iwhFLLf/WjbRlJVPa0w9LbX/Ohc1Oa9U1u3e/ZmhwVU09HbkITAWB1tyccpx0HFwdFYRX/iIm46DoBFB03HQcfCwDQfOEB70qYEjtRT3zrSEucFveCchcVFMDI9ByeRdVdJTapdf9kruIUACZRkNQ1ArdqN5eEAYI4H1PUbzfLTGnhk+WmWUhuzGFoN/UcdilBxjSgZm60wih5nIYV5DlRKJVKHPV0lDiUFej4wyy0OugxMig4C6E9tJ9okrJnqZhCyxKQ+0UiqSqsu33pt6H8tDBqnTaCFdIloqpQxZghgdlqoNekYAU5YPASU/98XKwnOdPV1jgFJjkD278h9TRulUXVCFIOdB/6ArGMgPtXzUC0o+mDFO3feFGbKBFR3B6JSzhdEpbUgJkcMA0ggnhPuhoYTvAZoShiqVYAx7arHXjVVkucUP3u7fvpwyI1QFvhVQMvTawJ/UH+QNYoY8PvUHlSIEirYFY0RDc+wplSekhAqOX19UDekCS0gJs0P5IFPShJuglNBCGjIqJouiZ7IopdanibukKYuYLWeqZKOGCnUkJ8lSMTSBrKoviCJKQIY00pH64qghbOlhwlpxScdchnL6jKW1J+4+Y84xv8TqoLOOGllDFq1wSufirYPbR2ott2y9JS0qM/jodzY+sLjzRTds+RLsbvfwXQ/uvnNW6xnHTjm1r7L7kV3fOn3WKzb3r75v52cf3j2yaej+ma2n37/r21uGHuipnnLnzn+m692V+3jV247h1duHH94+/MCLJ7+WgT3S870fbvrzmW2n9o20lgrlB/d8G+edxSldLYM8lzipNFAc6W9rfW1/5fu8bJAJo7DKxOfS6qDpj2lX71fYgbbVbXDC0SzPWEWaRZDJ+spJTUPtUpXk0psq1W8r2dg/8ImxpuntwMg9fRfNaD86lBaOGAS4IDsO5s+m46DjoOOg4yDXhOaNgyaE+Uv6Qcjk1gZ4+9fYVJk6ypfXS5YBrfgVBSUTFRTNCzKm7sXl8q2krzNDiSZvFiQQJQlAET+1jSr1KKUcIsuPlIpqyCSjpKGEs1HyC0BcTpp9WYq5BZ/EmB/T4YFdbXVmdCeb8W6j0zxpzH6gJ8b4VP8mfrWSjcQtP9mfxjCz2a35KCVel+datIJuUaIU3UJWKk8CZTA6mdXRM5TaoqOwl2eV2GBAL5Sy3w8DxOBjx73jo09cFkOC+6EMqskThvwoJ20oV3QBYYP1kd+DB+a5nLjfhFNGacIHD8Ao5/KmUvyQEsbIB6Woo4z5GmupFXjU01vtXS0GgqcqBB4sjCmjNIElKlUYBiKHH8jWF1VDSnij9CoxFm3TZFF2RQ4R6iaLQv+YLAob1JRRSKPaYi8PaBiAHlAk00gtG7WwSsp8p2HPOqVM47xxy2Mvm3Hi9ZueWL5zB28BfN/d34dj/evW+9vKxW+sufXB4XVL+meTIYR8zWqbU+1tuXTe//7h5j9//byPML3z5q13vn7BJ6/d8P7Ve+6ju/7KrQg9w9sWtJ/yklk/s2d4+0O7vvXSWT+3vu+xXcMbLp77Px/aed3OoaemtR5z0rRXfG3tby7rvnTr0Kql3edw6nmkcFX/Y9VCJ24XtJ+0q3LPCFNCRwrVFt5yz0TRwoz2iwaG09eDub6lFtYgRVvEmLZDlSvQI/BJ+5mgsjImaxeal9Uny87uv+7v/c1kLR6Y8fPqwCf4JvCJrW63WJobVRaOPAQcB8cHLM4ysawuwKF0HNT333FQD9s7DjoOHm4RwVNGD/qMcDOePBsTLPVIA2yKDyk+rTEjd0GxIGCJjJWSQXa3vSaeJmKmtjGCUKoV+tCEQxmjR1NnFvYIkQAMDxKokqsow7McpkmhY7kpMTT1qBJKhqBoN95S9pR1DXGoTxjICbvESKryTYKIhlKd0gRjDSPKfRmPes5mq67q38RPdogipJGSs5CmrWY8UNyG+AQBCNoGZ4uPOqJKtViGGVUiaRhrCmhkGmVPKe6nCTPhClIqXqpWNFcVDBCBpVDVy307VkRf2IhkqowmWjdVTTgucT+NCldKAGriqBgd9I8EIFWaMqpSIGAw4ZRRjY0powhqi0ATGAIdwdnIqkHb8qvFZAZzUObnfELhiIKEQGgYTcQGKeVBvVCqCSXNtbRpVCGoLU1iYVKInyZzivLVTRZVMpCGslECEJZIq8gcUosZH6gjXwOV+U7ViommLEjzN6uvoerPnrpOj92B+R+fcN6rJ8391IsvhQT+wrLXMnH0RVOPJ6f3uoWvO3bScZAxFpWZ03aKHJYKI8t3XTuj7SWb+x95cvc1MzNhc/99XEme3HPNyj1f7yhW0W/pv3Gw+uSmvkc39P13azHl99b3Pcrzh3R63KSXTGmbhbeZracM1cqDI62DI+WprUdfMvt391Q6d4507a527ax29Y60L9/9JwMjbQM1Pq1MHGVIjJCEYXxI7nE/iRtUMEWVaKBzlPqkWl5EMfbZzassMplS+ihlz4su+DBfAm8q2d2666f39N2Tx9PykYGA4yBRhuDlOOg46DjoOEg0PALiYPnICE4/tqPgNzqkosoPvcTW9s7zZAAwPX4gQqu4E4+cSGAijWkdNk3sDP4m6qUxB09DyOupjd2wQSkzCfIgM0rt5g1UFZqwkWXY53eRFecoJVNOeBM0LPPJQFmK/mHAJr6nkl25zWpGq9RdVOUN8EZDqkLIN0SOrjFgN4wJ0vjJe0bW2PZ6y2bDccoSY6Rt/+j8VWQY412968XoYF8wusgZUsumiZ3QM3E2Sswimyd7sUGMNXEUgyByaqUcIHQOIoeZGGCdMU0wln10QUfE4PBW1+TutWuxJ7UDbYPmweX+8PHLYiKoSGNd2jDyhBNOGWVsTMXERuPEBo02Hmxb0p0mmv7a4otJoEGoxPTG6tO/mjI6/pX0VInFKR0HV0Qzu3M6VDA1KxSUvtNkUXbRE3cp+dAQCqrmNOSPkYZk9jRltG6yKG1hgHV5Pwgq/uUhTxc1fjpKI8g6lcAg0zhrhSd7t37i5Ldc/tQd71h69j2bnlItbyO8sHv+1WvvZaW1B3Y8weVh4aSFCwsLR1YVr9n2/7pahu7dUePlExjvGX6yp3Lb9Mox5WJhsHJjpXI0Ob5d1R9Nbl/Mp6fy5Jz2uS21J5LbkbXTy33dpeGRkeWVanVSy8DG/q/2DL9kzeBDc9pOWznw2CmTXnPHrmt5Lz0lPQ7Uyqwew9xzuB+XoM7iEIvZtBZ5PrjUWksvLeRqhRlJQnxDMmGGbeU3oRzOEoYo+YMASaopD3zDPnnMbSPJU+KZMFhcIXg7whDgCuM4GOdUgSaCEXrFGuKRQhIaBJlRsqtSHlSl2qjKG0T4CyHfEDm6xoBddYrsOOg4mL/16TjoOKhLx75KE8J9IbM/PXf0RfbgWnwwFfVC0C4CBnKhZF2YSUmpJiJpqqUMP8z5FI0Me4Q6M7UNgzzzrPMfNuEkL+SN9Qxh3p7oogAjpeJWPlzl9eMtI1aFT2wi+EmmKsxCQzxTSjDSgFEVxpEzDGH/xnhQW41fxuH2g4suuWX7I7DB0bhbTLQfxljHuBitGJoYILKeBkQPCYSnyT4ShkHqMIND7p8BhnPMMM6TTPzAA5VgDEaK/YRNGDZ63lcRj/+xm+dymv+Z2aRlaRB0oxdBG0rsVcVMVJRqss+1STMaIU6o1J9oleT8wxXI4mAqYXRifTC94I3ifnSq2TU4ESUTkUOPMSVmiacVCvmGmp4ak0VJ8bHYDPSPjpAhgZQifhEsoYvjgyX2jJAS/xoqXwcmyvLWQYQP3MdrFVpqIwno4obqj/o2L508c7hWnNcxa3hX6fr1N8OWh2uln1n88f9e+7sXzP5pZnT+cMNdZ876hes33rZk8sX9la0DlRVHT7l4oLJtqLLiqMmXrN9z1bTWY7tb50yrHju94/SpHccNVpa3l5fQ+8JJIPblhR3vOm7KKzvKM7659h8rtdKx3S9a07dx5eCKlsLIrTuvKxfbKqUWHl+stjCKQqVYYhXT9uIw61GVRqqdxcq0zg8OV5ZP6bhoR89HKumrXasOf2fGpL/sG/4OstYgFV3MiCHdJqPYkLkDmpplhHE8y0OT3SgbbQEtFFPsaj82nFg4khBwHHQc5PvsOAgIe6NndtF0HHQc1KW+ueKgCeFBB2juMYjs7YfjBbULm+By0V/YhAZBZgjBBkMjszoSqCaikXVd1O2qOcowloA+/8Bh0DBokpqoFIWjHCVLYzdBqQ0OhiwzhOBsyNFkvE9q2WgFowtiJiVu0auWktrwiXH4lHG0VZMw1q6aI/MZP9qgiLDBOj9fyx5N5F44DUUUxTxZVRWluqaEAUILgwGiEQmEH6IMUkeGDbN90cs8A8RM6ThcyQkCSk0ZlUxJE7xhAD+kVnqRRuT0JsN+Hv9LyS5xObJ8es6QEg2lVpqRASUGee6nzCH6fJqRXW10JIYZrThqZPJCiUYWUwLw71dfw2qfd2RPBtIKcgUT0yTMyMLJGxEUWsg0UaidEndiblGLgAHNaZhvK8KmhhC/Ox6/HNKoVpQicko21i1pQ63mlIr40d0Y6xttrXHmU5S39Gx4adfCY7t5aUTxfe0kDIssKvPVJ+5+/4svuva2/7z4qNO/suNucpUndxyNC34YVGulu7bfNKv1xQ/svJ7rBnzprq3/Oq31pfdv/+fByg3T2s5fvuNzlcpVXa2vWrPzs8OVH3S2vmZH7xODlR9Mnvz1aR1HT2oZ6Rn8dFv51fdu+bdZbRet7fvKGTPfltawGsvJwcEWty1bMbCykuaxk/VjodI0+PQcYZHnCTkVNd4XMdDSNr37DwYGr5vccVG5NJfVR7MfLSyNVdva85FspDQQxVNVckJ6T6q0s48Ng7byG4cq36VexjMm/cWWnt8Oc8axvfeOuVMvCY2FIwMBx0GCBacyIub4yEJtxCxZqonK+BpEkEJwHHQcdBx0HIyLw49ZMCE8aMAr2c10ZoTmF5WRl6Bb7IqP5fmbCJg0wdaCleX1NA+9PMeuGua7i6poJVfsYhxkD5lPGOczkCKfqhof1SJc4VDBDw0hLR/VpJFl3p4m4RD7fZmpSrU0kYcIpXSKUt0FD5RbNaQ2b0xMjTFgoIbYsMn+vBknsRwLTaSB2EQY1hjUBFm74VxmakhbrWSDIDNWQEVmrqmyasjK44mhQZb4oGQTeYMoBl3cFwOsI5k0oTlulYrMnKVCpFH+KfGmkqr/e/zbWZwmJT+z1zPoQcG6KaPhB5aoBCCaWJkGeV9TRqlSIlFTUmGYeFbDjHKk/CoTOMnmjSdj+ZmfEDxYHGSPjtCLmAUfQ4MBJR/lA8UVRf8gYBohtSwWShnZQvRqqFYqRf9EI/Md0bvsKdnwI0EpSmQMfuPoV63v3c5sS9jPwu7p129YURhJB/rZB6591eS5n3vkKp4h/Ohjl3WUh1s23NBWqpCIgxGtHnh8TvuiGW3zh2vlBV3n9w6vf9WiD+8cWPX4jsvOmf+/t/Teu3nPlTM6z+gdOBZfUzvP3N5TGxh8eFt1W2v1OzNb3zip4+ULJr/jgW2/PrPtVXdv+9qGAe5ElMgK3rTth2uGnhweaanVyiQMeSMFzYfSSqE8qFyGGZK9JO/HTHaENX03dtR+0NfzfSgia85odZl08xLip0z4GP1TkjBV5DbIHhrZZ+pae/kNKAcr35vacVF1IAmJgxYKvcPplZWyxICJqdsH7jEhzGF5hIiOg/lgoZMqDSW7KuNkOw46DhJTHAcdB+OacBgKo9MaD8ORHeZDihmhjFMETKUIm2SqRMOkDD6mXR2glCJj4UoNw4mEKBGiKlzllTRn+qj8Q/byNtGFSCBVGoB6pIwwFrQHpSgZAlEtT7FknLfEhg0bWWo3AmT4kV68K1+iD2+wL5khSManwipN6rqWQQw+PwA5jDHgk0ygdtUKcqgxUCXP0Cd1rQETyVRFSUPGEN2xy0dmlHy0Yk32PBapHH4S1+rSeiJ14oTqRexOWT4YoEiglBhjyYddjBVTJbOLZ4gf9mKe8hwlM1RpqEcTs3xjDb6nj9abUe/k9KBzlG9edJ66EKOD6WEsm3g5IbthL9576oxlsqGEbaqt/LMLReAhQ95rzzsMz+le8M7ZpyVikSUP1Sp4F0k8yBvxklJJOfgbH/ExSqUNI/uHGfQPJ8r+wR7RsEupNWnyK9NA/6BzKrFEpiM+kXKkYXRHX7Rllx7VqYaKhuckeYbwC5sf+NeND310+U3ol+/ZBjm8cN5xV+3a/Pajz+GRlU+e8l7mbf7U0W/+yaPeWhkpLZt08ry2k06bduGU1tnT206Xq3wJHkwNRQN507zNztZlI9WEPI8ddrUuHa6u6x/ePKX15VC1RV2nnjbtTVA8HgJ84/x3Hd2+bEnH0gumvQxSx7eN3+gwQ54hZKpqtqJMa3+tdTB9ytuGbx0YaUXPB5JWKRR5PyFlWmMmtaJt4q58Mid4eNpHyundf55WpmFa7NR/7xv+fqm0DLONe363Z/gHCLgaKrRsH/j0UKHEMjbpFYjZZ1v/ZyG9+UO2fMQg4DjIqVQ0icgVJzcfhsIs4kXeDFkxKCJReFOswQBBsuOg46DjoONgXEAaKDhDeNBgnp1WHFkHAeuvJvQQxLhUBlWLKmnCJuxDHw0lxIBiFyFaURv6sAwhquB7oYwmURtdy638Yya9whtRB4EPISqiFDbSyEa7RCmIUJ1ZWBLY8iFNDfMOsWSTGbUY19Wyq1bRNu9TXcuJxiAz2aDBoTRYaiTyP94tBvIDfVIT2cfbHahVK3UaDtFjH87lX0pm4L1t5ulUvX3W6YQxTTSF0Yn1QdgkqF/YHXQOdocgDSXG4nsYi1tG9g9jmqsJtSKuCpY0kR+5UpOXdC/A22VbJng5YeJ+a2+B7NVNGRXBiwQgHA8emNHL0acmvjXWirY4oWTMIpOamApjzD+7yDN4+BR/E4XTkUK9xOWo0iN/gQBMDArHBwNmhIZegvKEKqF8wTBVu6/Jovm8XzzTGJ4ZA3J+sihd43lJ11yI15KOORC/Be0z1/XsPGP2MfdsXAXtv3jKnM39O8/rnn/t+rtO7Trq+o13bBhed0z70i+u/aelnUu+v/E/d1bu72gZ7h2+o62l8uUn/nN2+zkDletuWsOCMVdOab14e89Xa9XvdJTf0NO7vFC9oly+dLBveXuxMDz4iULp0u3Dnx4aaR8Y6V4y5Y8f23XP9uH7F7WfcfeOGzcMPTqv9aSNQ+vOn3rBnPbZ39j8dehlJeX/yrwFscoM0lIBkgcDLxergy28tDA9Bgh51I3ANPU0rfui/N/o0XeWXztU+X5AIUEWvZX1UD40K3f+IgQW7kfrbA0bnCTXOMdSWyVZoq51li+e0r5kTO1/jxAEHAcjgnBGdf1XXIjrv840VWjyMYtwoPhSZ4m94yAgEK0cBx0HHQd1AflxliaEB4c2v6eZKMjD9EGuxKbyu+Gxrkr68co8K5MfETPsZRwCu/nm0bDOcziJ5vKQnz6KRs2jLzmhVJRSGAvl+KgmOiTjCGwyU7SjLQLBT2EyPEtPE5mpDDbFLh9qwyBCKX7CiXwqYSil/NQFVJQxzrpB5t1GhJarvCUaLCk1QvUiGSWbhipZtVqcBg1zU6mFoWEfBDtemciPZ2rfPutMfjRfvvVevloibwjyNp4uolGVjPEsDbvBMOsSkvlHMmhLEwgBAswNdoeQZ2swuvyaMTC3fJ5QPDBvT3MM0Ojlh/BGbGhFL2KGGARjRA7GSNoQqgxjTGFvYXrwb//ri9KWDUoGbySzp8Sdpn1qlqkmi4rIYYCxKFzWbnRlGuw1RzQEalPeL8MBexmLUqJnl45ogr2ERJ3Gtus3LucBvU+vThi2bBoplUYobxnYsHTKjLt61p8740RAuGzLN35q/k8+3HP3OTNeXS6+9oeb/+KC2X/02K5vnjP753cMPLy1v7Z0+nsr1bdv2fPVBdPf3z94Ifxp9tQ3bd++rFxe2NF+8o7tV7R1fJgJqidM/cCDW94xtfX81Xvu2tD3wLy2Fz818OCySS8hVcjZXD2wgoa37r6eBGMr2b70cohE/UaKxfLIyHBLCah5+UT/SFuZBWaYzsmA08TRRAWDwHWXLxmoXEkVqyTPnPRnG/f8/tiB7v13R/9naEomEFXWPBHK3DzSpBwjm2myqFruHv5Rf2Vne3nqXkeWmhwBLjiOgzqHEV/YzQcOZIUD9AiEAMWsfZnlI4vaokEIP/mAFU4cBwmajoN8HxwHHQf5GjzHzYTwoAFkstbQSIk3yytDSPsgaXmKJVYWVZiJeqFRlRpGqVp5QCl93S7KOjPt5n2GTXLx9C2fNgzP+RFirvATcSiiEVXEp2A1Mgj3Cl2UEfPGh8nwjJm8UYaZPKCJHiP+BfsKIXig/NBExjiRjZzIJzJmYYlSgwxvqsWszjIGgIEOXKOVfSjlEIMYg4gTBsxNpaxLMGKmtqKLT/Vvwq0mmj4jAxTx4+4pHthiOdM8LVQt6UGmjGLDYPK1aPidriFJz51+RkJiUAk9gqvSgFjC3+B7CHVpQyxlg4ESgLSCB4phouRDK23BGNnFTIxxrDLdDD4mm/kZD/4FkVMSj8SgMoRQMjE3Ebbgb6J/OERDrg8blUozUsIVYXTocaV+lfrDT/Qlhhn0TyQQYwQlJ9Xwlp6NPCU4XG25oJPvcMqV/fKik6BBC6dO5VX17zv5lbfc/YXTZiz75va7T5h27M6h7cuyzFiWgitkLyR88Zq+ByBRvIdwqLKeE7Fuz1Uj1eW8gmJbz5WF6vI50z5QLk3F70hlXV9lHUxuYPATnV1/vbXnyq7yq7cPXt87fAGPCHIJWtB+8tHdx7VshYIVzpv6clydOuW07239yuL249YNPcaQqsVya6HKpM2WWhnWxxsL+1ta22rDKDGeVH4l9yP6hq9NnDAjhbMm/eSTOxORHqpcU+ln8meJqpRAzLbUjfJ9NEs7o3okJQZTdcYSSRhKpoQrYoeTtXtuZo2c0Fs4AhBwHKyLL5xTaSLEoIkAh6z4QhlmapI3U5WMkREiDEXACsFxEHzYHAcBwXHQcTD7a3hOhQnhQcPHIzT8yhGpq+NjdeRKrvNKyegR+CAEMZNSTUKWTSil31cTDSaMo20IVOXbSlYryVhGQIo4FA4jkkkzoSWxSvogVxhLM6HDMAvnMqMJrqJWPcpGDim1K3aXCMnYOqVqpVr8BIsTAdMuevmkFNlDiMyefKKRMP6g1Hz8EWGPNw2A5jKLvsKbqqBGqoq4zqKgsTKNXmUBA8Q4powiiDTmOd74KaMKkJTQLWUdKWnCB4fIsTwpztk2DGyDFjIe0np13A+ah8FBTRnFCYwxGGYwRuiivOEQIUgjhIRMFwuZ/v5jl0deTtwPS7FBBDTwt+CBaOBv7Gp90fxaplSxabJo8EzRSPQ4VC8wPTinsoUQP5RB/6KXOtLITFE6rVVb1vXuYMoovKg2UmBpGQghFIlFZV7WPZ9+Oa2/9fBnOktD3DbasPmxjtLwLdtHOoqVnZUHukoVrrl7hjcMV1bCA7tbF1S5wVRd3t66cGhk+Z7eKyuDD9dGlvN4Xqm8sDJUaC1fyuqgLaVlQ9UftBW7Birr1/Vf1lk+d9vwff+99uHWYuvG4UeYOLqo/fhtw2voiJ/pDLLK04j8k60xU65VeQJwuMBTha2T0iozjLS2c/i6o6f8/p7hH2V7Ca4ndnyQZxgzClfbM3ztCDnGbMvTQmlU5okfmpGMHo9WjdFFnqUUk5zRme5NeDuSEHAc1Nnkj10X/Hw4GB8yMJZl3kweaB4hAyFvhuw4SMByHHQcdBz8MYQPE8JnAzKv9mIRBVqKa+VZFprYFddCI7N8FW1Vq+7VpM4+r1RfYaCqvMOoksOoCgF9TBkNY/mhShqVBCE0QZOQx4cr2ajEILawpEpyXfxTkzqz8INA/MMbBsh8ECSjDFcwK9kgyD6ycDKmDGMFVFxpLZlgX3hAGW5lEH2xG5ZSooFS0hGe4Z8og4XKVSCGgWQxQLVSXyo1eJVo+OAt+Cp6NmX2RAWJiHxQigcGr2MA0ihzKIPxU0ZhhhjIVXI9toVzKaAKMNJ3zD6D3ad6yculh/2UNpQBQ0I4kCmjmI2fMooGEig9brFR+pGjQE4zXrpHF/YUG8w/wpcZj87hpFZPDJ4445jCmtGpMvm1TDGG0YkBKs2YnyNKrTaFWGhhnv5RJbp4/qy0YipjW9I3Z34213SUNPbtWLF7G2zrs6sf5TX0vOnhNVPnXL9xxTW9m37pqBP/Y+t9H1l64a07Hv7tY9/T0jLy6dX//vNHvffGbVe+ft5P9gxvu3fnyIunvv7RXd9aNuVVg5VTV+/5Wmd5Tm9l3bSOCye1n9xTWddaWtjefnLf0HVt7WdyOlKqkHVHu985UCt0D7+m0lKa1Dr/xVN/bWXvrSdPesOli3/1ow+8ZX7HiesGH1vYsWDT0FqWhHmyf8XCtpPWDD2enu6DzLF4TK30noV/+o11v5mtLtNWJPmXscCVu/+URwyVA0yG2RaCdimpkE1q9PSNSaGo1JKuZUYZPkUI2eXdiU9v6r0jAQHHwbqrepxUrueSdW1HjngUeoQ6M3lDj+A4KKAcBx0HHQf1t/BjKE0IDw5kfiBy711TRvkNlKdb4SivlEwVgsrgYNKoVb6JNOJmkusaapcyz+LkQc6jrzobrSwaAwg/eQFZUSqCE7v7imoyjqgWlmobHjBjC36FLId5M1G78ICNZIQIpYqRaEIIDiZXNJFxdC0n0Z126YuG4ZZzGqRRlvnewydKghMGGGsAwUJlHw7ZxYxeENhERPHDh12qMvXoE4ZhFs6pDWOIB5SPDwlDmB6l2J08KBM4flkaaiF7kEBoJG5lLNJIyW6eNOadYw8auD2mO5nd1b/2mP7ZCHVpw3wCEHseIEwYPtOUUQz0kCEONSpaQS/zs1IhGCxJKtLISwJFNjThU++loC38TbNMP5C93lCZQ4gfgVOTRevmf2IQk0XVnJJNjHGU/o09KwjrozuSloyQAfPBcmVGGhkMf/Iv6VgEUzu2e9YvZe8hXNA1fT1Ly8xfXFh558WLT185kG5nJOZULGwd3H585zGP73kEb3dvv2F3ZRUpwRu2fGxe++n3bfvnSvWG6W3nr91zeaH6g2mdn5vefcbQ4MN7en9zoK/QXiy2t/9eS2naQN+bhoev6Okt7Bz+TkvpTQOVa1qHl24dWrVj+N6u7IX1rUXeNVg7quOEme28GpHhpfVCVw08zsTRpwZWkBtkwGQLr970Zd54MTgyMtBSLtUqeWIHA4TVjeeBOgrhj6yNrGAwQDTB/VQrTogym0SadIkZ1lLy8P5tl79i4Ydl5vIIQIC/XMfB/HmMKzZKZMURXerjgi97x0HHQcdBx8H81ePwkU0ID/pc8LuHNpAuhDy5QimGRlVQsjqD6Ez6sJfDMA4h9BLUPGrpJeQQwkZN9m+j2rzzCGYSqMpHtSBg6HVZR2BT/MszoiB4480wplb68WbqlzKondiaOlIpG+R8jxpGGMh/7OY7lZJSw6jjdWFJLTb4SWwnm48aPrGhit4pqZIfCdJQRuDXaGmiVKHcYjAhA0Qv53+07Cf/aPlXf+WoSzJmspkS5jYhA6xjdHgQG0SQrAQjZtql1JRR6CVulSekVl2IK9IEDV903m7PL79R+xkn3QJX7JqDZSQAmQiqyZ+RAER4RsaYRvb0jSmjf/j4Zcoc8gKM+7fzJgayTsWVGf2GBI5fX3T/L6MXUaSTmPYp7kcJP6RU2pASm7rJohxyHA61ZAsLW7MFZvp2kC1ct2cnPHDtrl1UXb9+BaxnRd+WYkvhp+/8Umtp5J41q3m7xisWnbt2z7qvbvpmZ3mo3FKY2TZ3XsfsO3Z+/rTJb2VW51Fdp+4YPIYbn/M6T+sdPHbPwN1bioX+wevmzPzu5K4ztm163/DgE+kqU/lOW+ul5baLSu0X7Ri4Dvo3uXXB9qGn5radsWf4qStWf5IlZHZV7mfy3k3bHxyqlYdH0pse4HcQuZdOvfDuPddwjWJpmbWDj7YWSyz7SZKwlQmvAJs9OkgXcDY9FEhvKBOF2+82bqZo6ku8MbecTJopqutktuRMccmUV+3XqyubDwHHQZ2zCArsculmNx+VuNrXxY4ww1iBA814M0WN8Iax/KhTlbJBzvfIriIIAgZ8ZKzdfKeh1zAcBwHEcZDgGLNmHAfjb6ROOFLjoAlh3Yl+5l3+SEQFWVRGxC/IWPCrOk3eTLIsJdd1qbaU6OuMpZR9GMg+71AvncdsXzZ5P+END+ouwoyqVCqu5CMNcp0lEQtj2QQdkgbLCFoKfnVm8oaxzFTLrrpQbXQXrhQjqR0fULEJ+hreJGicdbU4xwk9smGApczYJVKqIaXGoAhKVZ0THXXeDE2MFh4VbvHDJ3U2hpgYIM8xYoby/h3LMb5l+8PId/Wth6El0yxiEbQiYSiluJxKCF6eASpPqOcMKSG34Ur0Usbygxxr0iiXSEd77beT7KqfMhqTPyMBqKzaMzJGLUtDvzxkSBOmjNJExw4z1HjQ0Dsl7zAUyUzcbFuBbCH5wPy6avGsIA01XxTKJwZIeKt7sYQ4YYQ9mjwtWzhrdMkcUpfAxZA29I0+Ybiyb8uTPVuv27W5MtIyNFS6eNI8pQOXTp4Jk7pw6rE3bXvsbce85Nvrb16ze/2uoV3HdSx5xayXP9Hz8NGTjlvf99gJ3W+a0jp3ff/NRxVOpdPIy4mEUbaWEjuFKI5UN7FbKr8xsm0oeRn9zM6TdgxvgBOWiyMvnvaKJ3u/NbXtFC5HS7rPeWjP3WsGHufNhKsGltN289BaypSvq/FkYI08IVyRhwmJZOm98Xs7FwXM7nLpYBhBtqW2Y3L+32wiKF+ERAPhe+KQjCFjhnt3qWWx06SsFbvK0/MeLB8BCDgOchIVDlTGOVXg4OqNRhFBVYoLEQ4cBwWg46DmyDgOErDYKB0HMySeh8KE8KBBZ5lBbqqLO+WZVRAzPKpWrkNGkE1YSsAMIVqFmZrXlaqNhtFKJFB+NDU0qkKIZwjlMz8waWj+oo7FXKDjGi29Ap70inP5UjZhKUHBD1nxj13CpHjU+Bg53lt+DOq9zlUo1VF+l17oQqSOVoq7CNGvupswMGPGhjcJeUtkjZ9SteoCY9ljwAc5b4DlvhKMcD+oZuAzxgDTwqThExmfrEGKT15NIV6HUsRJJbsxaxT6BF3kg1JUkN5jVy8tpGrCKaNvXnjeXU8kJhYJRryNn2LKk36wxE+tHl2VVGNIx7iPKaPjGSNdRCumjN6Z0b9YYwZBr77ADJ+aj6r1S0UyGVVx3S38Hv3U2quxgZrcyfqii9P6okwWpeQ9Ft9cd4smmooWosRy4mcFxyaLYhOzUjFmY2z0yHiUDMlII9Rr9girjM4+Ye3OnQsnzVi3aycRbGH31Os3P7Fo6hRa3c+iqcXaN1dftWFoXbml9oV1n3vz7LefMO2UaW3TvvjUL67vH+BthCv2VIcr150y8xOLp128ctumKR0nz+w+Y/3g3Vu3/0kr7wysfmc4o4W16ndrtZEps/6qNPjk9p7fKhfa5006fUv/Ixv6/l97+YJrN35gQfs56wfvGayVlhTOSa+tbz9+9cBjiztOnN26AELYkhE6OJtoGxeuaksLicTsbYTQy5QnzA40scN95QaxYM4npaI19iQk1SotXZNJul2KBz6qVY/JZy2R0lu3fIMnHtXK5ZGBgOMg55GLc5T506qIoCrJcZ13HMyDA0SOg0RPx0HHwfwF5PmSTQifDfI8TM+9dloGp8rLwdZEz1QlWfbRCiGMNQ7t5vV1DSe0p61IoDzv3yY/npBjSBoGcQ4GQugSMxHPUfCTQZQYqBZNyLLM2ysGqMRSTfJm+e7CuczyoTR6UZVcUebZHcSP2vEO1URVjISji+ZyWze8fL9U0VD2QSylRF83FxSz/KYEo/qlbbiFFMknQ0Wf753m+TDJbuJ13enV80oAyr9k+B5EEbKnbB6lUn8YYxazTEULaZKcH3UJeiwZgzQo79uxAhv5FLFUL3X27NIqFoCBs0GZ+Ig7UfJRNo8SvZwwmDxjDM8IPFDBM4qUsD4mixbW3hK13DqlVXQRennWLNOfX/a6O+//F6pETii/te4WStHFO/vXvYQTlx6TKzBmGCPpvvldM9b3pWcFacVkUZKNPIKoWanwzOK29L4NqnQgCHSXWrG0TJoyOp2FZNb1bl/Rs2PFnu3IuP6nTQ+xhExpS/WW/g3lMou/FOZ1zFrYNeOyzd94zcyXbxvadMfm63sqm2e1nXrSpLN7KhuO6j51qHJ+3/DmXQOrWDi0ozK3b3AVHXV0XNTVfvLgnsKk6b/HgPds/W6p/KahwScHBx+e1vlrfX3/uLX33kp13Zy2c9vLRy/uOv+BXX/XWuyY1nr6yt7btww/yrPNpWILi46+bOarW/bU1g89xq0r3JKjYwGYSrE0VGttr1VQ4BwNVXUbqlGOmKuA3WV7ie+JG7MLG5SJ2CDeMCJ5iBIPmfO00GmWYyyePPUVMnZ5JCHgOBhnU7FA1/CQI2TkzZDzl3oZo4xYEGE3WlGFWQQORS41VFU0dxx0HBy9WDsOOg7GFeSABRPCA4YqM5zTMY1/+eHFD0fCIUKesPEDiNqBakJVWcTsFfYteYYmOVoFE5OQZ3TjW+EWg7wHNGzhTVVhQ1VkBalqaxkJ3hg2aqsyczbK1uoClcIVBuhllhfyEUu1YRNmaGRGCQVCL86GHnm8WcS/cKXmskdGCBsFUZQIMguH2qVUp/IWpbqmVgaKtWGJIHopJ9G19BF9Re0iTue7jkFGjwhqLm8Y41xmz0gX7+5bizG0Tdk/pe/kGSXDoIyJoNJjo8QgTG//U0ZFBfHAh7ZyLlYpV3WepYQsIYi51WXzGA+cKr8gjZqIaIn+kR7klfRoVMVb7BG4XRqWCJpRQ0fR5MTpo+uL0vDfln9/wl6UZmT8DENHgcDnqf4tnKlgquqaRyU1K5Uu6I4PE3g0GI3kKxlp/K+t91WqpeFKywVpwC1ppmhGCJdNmw5pOnP+4uKakfNmHnfbzoePn7pkxZ4VF007HyImD5RcHVb33TqzbXHv8OadAzfyfsBqdR3vIdzWs3xgcGl18BMtHR/m9YPFyndKpb8aqe5sKb1hpPKdvt5lzB/oHfi77tZLdw88vHvgs1M63r9neMWMztNntL2kd+B+qNhFs//nPTtueGDPDxe0n7R24LHbtv8AVsY1KrpOyTrmcLLqzOgLBkdJ3pgBlslaVzCMlTNEE++TiFpRQSzlU3QxWWZsMHhmZp/4IRxyjFKO9eZ/mxwBx8G4yOcFhYmIC3GSsQkzlDKjVKChHB87wixinLzhR83ZlYwQNviREkFm0W9dp/IWJWZqqFFpPBoeNgpYVMkJGhlLH8fL1RWD8ceCkiZhFp2qubzJocwcBx0HHQfjz+THKZgQPhu0dWdUbFAUDi/8QgpahbItuzk+UOUV9tWwyXcmSoYGQQY0D6X0sldttI1dGatV3jJPAvMri8peZXgLQW65LodGMiWXcq7R6CMk6PKtUvZU6ToercIPVXISYUbhSj6jeZ036VHKIbsyoMzHv9CrF3Y1SIJNdEcT7VKFgfqVN3YjUGGfdx7DkyVtcYJME20p6zWWRJVG/knu6ZX0sg8zbDDArcxUG8ci0oIN+hhhDJImMTZsdBeQJsrvRVqPKtE5SpFGbILg7WvKqEhjPsEoNkieUA4nnGJ67oyT7uhdjwFbOsYslVeXzYN3KfUnxihqhw1NqDomrRxTiCcG0YiexZRR3MLQ5Dzf5INZSpAqmmAMgVQvQRqzNWloUcCGcj8MkyZ19I/uILF1B6KM6EvJMc44ad2eHQu6ZqzduWvhpOlpymiBKaPTb9j8xFOrNrG0DOvutJRqv/3IZyaVB1tLlSnlwWWdS3ZuX7W7et/C9pOhSfM6T3/RzFc+uXP26j2Xd5YX9FafaCsvbSsvHK5eSm25vHBoqNaz59usSlytfrdaqDGJdKSyvKv8pl3D366Vl04qDg0M/ONIrbx8x/X9I21txc5dw3dPbvtV1qqBxfFhpuic9kWbB9dqaijDY+PqBBuEs1HWCumyT1/6FmX1yGnWaEbzpEjl2MxP6kZTf2hoRVs0ZP+SkLlJZZodmnbFDGk+VlVY0/v4cVNP2evX0hGBgOOgrs9xleas6grPVR0luyp1tuuu4VznHQdBxnHQcdBxUJeI5700ITzoU8APL2UIaRkUK9igaJWq5JrfWaO1RVKLLTKIhrIJJqld1cpJ3lJmE3rAOJhh/hnCUIZnudVuXilZQYtSuwp14jDI2qUqSJTMKMMy5DBGEw6Rg2hJH2ayUZhEyQcDbRpVcCcp1VBmktHHwETVaLivoBveMKChPGAfB6vxoA+fmEmpJnWLzYTDYINhpgGrl/AG8ZA+GGD0jj4fJmmCMSNJ1AgKytsgumZjk+dp7H7suHd89InL9jVlVH2JQOZfTI8mRgJ1rJtiKq6Yn2KKH+4EY0lZl5oTo0MJy4JTxWROmqSRj73FATM+Gg9lfn1RMTpVyb7OWFVqwvxSEoyk8sKV6F8wTNFLajUkBswwZBNNUEIXRf9QamCyZ5fxqJWeMGRRGZRf2PTgz8w+5Z82pPcQjlRaXjttNlNGXz2VMwIhSq8xhC7+7Qm/8eC2h2/ecisTR7cPbTpn1vm9ldfct/M7R3edu6eyub+yC8vu8rHTOk4mSdheXtDdfvKOwes6289sK88c7Cu0tZ1cGXq4WHoDtJB3Era0nbmn9ytdrW+cM/VdNOwbXjGtvAzjJ3d/rX/wrtZC9Yldt+4cXtXWUtk+/EB7S3HH0Ootw4+UijzRNzqxM40sEbw01TOWFSWbN8bZRlOCHEFmmbKFYoO00vOF7PJJu0mz123QP9WmbrItup7XdvI5s189pva/RwgCjoOcyLiYx0lVHIk7elzSpZGBwodkx0HFx4g+joOxzrbjoONgXFJ+nIIJ4UGjzaoJ7aWKniGkMfSMZCDrOkjOczY0wcfE4tpLTP5KtDCMg92h0YYmWqGRwzpNtJJeNmouOZyEEDZoppSHd1da1TYMaI5eTihFzBCIZyrzwUyX8oh2eRvFP/GZ8INSNmjYouF+zNSdvKlJ1jSNJ1ypljLImAYmffSiXcUblHoFPN6kkdtwjoFGRVlXhR9pNLbotM6h+lWn2MsYQW4ZoTzr/YT5nwU0iV8YdWESJ3yCguKEOcm/vOjV8Dc4m4z1EGBM7FR+j34R8muHhp4qpRYR2GhIldKJlEoSxjBkQJlvzovsP3TUJVAsJQDFo4JxTTiZU1RWAU/O8+uLqi0MDVdwM8rwqYbQP2rVhDQgtaJ/Yc8I85NF2dUGrwtGui/6Rxd8GFUcAm3pDofkQm9//PI0+7Ha8nIWFy0WLpk898K5x123bsWyyTMvWnQsDyWeMe/o+7avmN81XcyQtusGNuyori4Xq/fsuHFP5Slyfffu/PSc9tO39t84zHsIW89/cMu72ovDs2d/bVr36Xv6lm3d9dNtxUJrsbhzxxu5RvAAYaXy7b6+wtDwFXC51vKl3e1Hc8nmAzPbPnAPC8Ow9OistjOmtM5GpiMyivPaTp3eumjj0MNK2umpP+z5JFJHkjBlA9lLm2Z4pqqxiZ0yo2qUQ6aMYmoLdUz3tpKYccJMkByckF1pZKbddUMpSevtCEPAcZAT6jgICI6DjoOOg+kW6th9Ul3q053TsSiJpinioAmhzt3BlaPPEKZXfEEI95f0C7o1ytOqiTfWbSJmowbj3l+PMU6iSV4eXyVXMlbX0TAvwAajrfpVmfRto4ZBVMRn4Cp1dCh2w3OQGTREShrmW6GUJszgRePNZKNO1URm+3EVxtiouYakXdEwaSjZVQYvDlC9yECuNCoFe/UrvZzH+MPneGonn+paS86EB6rUdXiTJs8q0Yz3yUjGh14sIXXidUHVoHN8SBWSQqTEJjZxMKUWRf/E/ZDFAJVglCyHGORTkXnP0SP+oVIQLahUPqeHHFNGscEgRkKVOJ4eMkSPRlxOrpDz3AwZbqYlZ9QEszr/+RureVKHWZ4xYqYB46GuCzzTL/Z8NFQaIqQnDGecyAo08ztm3rBx+RmzlizfsQM9bJDy+o3Liy0jX3rovnKp2rVj+CWT5995xwMrh5ef3LVYUWFG21zm1+0aXnVs91tZmpP3EK7pGZnVecHk1rdt67lsz8BDpUJtYHj5jKlfai/P3L79DTNnfHd48OHhweto3tH5zmLLMnrpH/zEth3/wTOHQ9XvtRTeUC4t669cVyp27qncvnTqxx/YeUUrk0XbTtk4eP+M1qPmt5+0fvDRUpq4nm4/pQmdWaqPO1lcTdhRBi/SemPBjHfqwAAT62PT4CVnu2lRmczVaKirMwibaCLhkR33vmTOhXVK7zY7Ao6DnEEuy3XnMR8gxgc4jOPKrxCjACdleNMupZwjyEzxKHqUWcQF6bGRPr+rYBQN2XUcdBzk+0AQdBx0HNSVwYQwrpAHJMzomBYzrmBcWsQvaJg4mDibynwVHcSuCBhPGPJTLd9Kg1Bt2GsXs7zPUGImDxLCQ9hLIxt5QEPzkKNhR/b2C3brIoo8RHBS0FLIkVKlwpuMZYMsMwmqolRwmpCShWU+/uV70djCDAFNdMcuW95eIRkl3aFXGTbhDY16RAhv8hPhNs/ZsMGzLDFDwEy9yDneENhE7TAIY9nLm0BDQwqLMjDJ+8RGiMFe1Kmcf3rtlfgn2wNJQ9CUUU3vTB1nG275l6cHR/ezf2LKKEwSBW5VsvuMU0bVhB7xnLji2qtoK8aFQGihhHHVTRlFKTYY2Tk02vJTRkXPFKKwlHFwM+zxzK6aYCy2hj4hM/bei3yCUU0o9xX2RBSji3BIFzzZz6xUSr0hCpb0X1vue9fMM27o2Xjtg9/mPYQg/4PtW5kyumzazCf7tv7M3FOghYsmT7t1x8M/fcybdg7tuGrLtadMPp0po0d38x5C7uskIsZg2Jgv2ldZP7l1NowN7Z6Bhwcq3x2uXDRS2YSGZUWhcQhVsnLZVq0u72z/cEf7yXsGrpvS8aHhWqGjndfTP9kzePvMtrNX7b5zT+XOee1nbxq8t1xsOXX6hdds+o+MuRXhhBC8LJeYXJESrNZ4PeDo44IcQpLTGEZJ4Gh3T7/ZiTJCZpYvTFbKPdY1TK7G0o9ytaD9xJOmny7Z5ZGBgOMg5zGu5whcrtGo1CVdJ1o2qgp7VVHqqh7XfJlFrbxho2t+XS91u7RCoy7qPMSoJDgOBj6Og0BB6CRYOw46DvJlMCGMi8PBCXCnvhFWkRmB1Gm+qNoH7wrClqdewcFkxk8r1bIwqQQYZp19fhe5bpRRW9evzKhlZdHxDy7KT3ijLYczMFLio4aKHCpDkw82+ZugMlB8kk3IeT8KbBjn4x9+FJ/Qq1U+/k3YNT4ntJExpToNG3Y1KgkxcgzUJCxlJg/IsqSV4jFCXdZOA8YeD8j6HYCZHMZhykx6lNGvGKDGgA1COETI00XZ4EHsi131Nbp6Ta2ohN6EU0aVPIS5YYNMSVpPLBE/7AZXPPApo5EYRMjH1DomsJ8po+KNhCLGkJ8yqtTfeHpJuk9PDObXF1UOMN9Ebikhh2QmdZiU0gdRzNM/VSm1mKd/jAozHven1JsSoWYsKgPfYcpoeg/hrp2sJbOsa9bCydPW9W9/+dxlxdIIL6ZfM7RxSfec6zfdsWE4vYfwO1u+urTz2C+s/uik0mBbsTK51P/a+Z86Zspp92/ZtL3/hrWV5dXqlcd3/86UjqMfG7wOskf+kI56Bq4bqV5RLl8KJ9zT95Va5QqAGqwUdg18As3AcKlYeuO6/s8PjbROa3vV1qFbeobPn9l65pahu1sKpTltp/ZUtvOIF90R5HJELlG1bI3QlOiTPhlkST/xurwxPWqLpwHHFOlfscG8JuS6yLpm8PGosnCEIeA4qEtxnFZdw/PX87iqcwHHTNEBIQIEsuOgAMzjhoZdx0FwqAtSxCPHQcdB/ck0tjQhPGg80++nYg3uBNfKUnyjHvL8SirRP5X5Wsn5WuzZRU/Jj7b8yypCL5vRzrJ/5CGvqZMxUHP0MYC8TXjgcCI9mDfQBTo0Cmkog8VRFTYIGKgUR6JW8Q9lmIUTahVNZaO2YRbMTb1Tm3clJ3lXMsvbSJNvjvOI3xI0MPmJHnGiYSDoSDFAE6X0sZooXcgDghrKWGbhVmb5fmNsCIp8NJEy6KIcogyfeMCMXREeVjEhLQM3C54G/RMDhOyJK9JcmT1xMLFE7MMSWZZhTxM5xOZApozS8Knep+X0CGNwsFj/k9HC03R0DIMqxTkxOvRQL00ZRQ5L2YubIe9rfVFxOdzCEnHLR3SRhvLAHVA+6hH6x2BkPJr9y3rP0z/+xpkLML8rUdbRbOHC876x9tbTZy5dsXs7nO2f1j/8S/NftHzP9uW9267u3VguV9tbKy+fkvj2mxee98qjX8qiMpet/drRHctOmnz67PZXr+2/nzRdZ0t1cz8zOQu9w+tndb58dudJW3qO3TP4BDyQCDdj0jvbSzM3bb9i+tQPVCrvZCEZlFOnfGBw8KL+wevIGc6Z8oG1295YLr+hVFo6s7SMDOGWoRvKhfKm/pt6KneWi2W+Cbsrdz2ws7Zz+IG29N74NGeUNWB4+o/EIMeSzQhN80WfTtsSLaRWtBAnkS7kSKUUjPkyXkWYV4aMExY9TWWh8ND2e18698JMdHGEIOA4yBXYcRAEHAcVKPWH7TjoOJi/xDdXHDQhzJ+7A5WhgpVamupJA7E4BFGvKPdfq57UVlQNezWhSjxT7zPM1qFJP88mNK5T5oehLqQZr4++qMqnB/P0IzwEEUIjWhJV2lWrPPOBtIgjETBkrF1Zis/IHn3ehipRJimpFUeiofrCm1zFGGRDK9lEd7JXXyhjSDIIbzEMeWBXBjEqWaoX5Bih+BgaPtEpggYsDSXEY/zY8AYDJONHNJUfSpRY0iR8ouEjnyjVBA0+UYbbMzsXsUuyDubG+jF57oelduuCFrthSVs2WdYZE970sgr0PF7IIGUAS0QOrog3ZB4vO7pzDhxMUzq1AIycBz0LhqYeKWUMo+P5QMYgRqdW0MK8fd5Y64sGb9SLm/LZQoz3Q/+go6PPCj5+mfqiFKpj9O/cwrpbeN8aL5xQtvDo7Stu711/0yPfGK6Uatm6UCv2MB23eNH8pcv6eA9hbdG0qWBy4eTjtgzs2D24Z8fgjjOmnrazsmH70ObZ7el8QdBYC5TcXam/1lN5crCyYri6bvfAP7UUfql34O7Byvd6BpbtrCxvKV+6u/fK4crySuWK9vKle3qvrFSWE1f49A88XCq9qVIo8poK6FbP8JOtzAAtFM+Y+QvXb7yN1xLyM33b0F1dLdVJpQFYH5xttMxeFyEZG3E5Go7xwPp5nuqO607kDDHOkpdBFUdhk4fAcLywqP0Es8HxsBwBGsfBOIlcnLksKxA4DkbMCnwQuLpGwJIeMz6OgxF3WCDNcdBxMP9X82OWTQgPGnCi4GC1pdSS5nZG4yB1aIJrhYFom/RRG23VJDzIQE1SWU2ZQ9KGsMT8Kw1ln3eiJuENP2GDvO+VRYdYG4BW0ELWvVjWtkRsZMLwhlnoEbQbpS737LIpNCLIXsbapRTtCcZFLSSKXbWiRKOwqsHQhC0faGUjbpa3iY6wl1v6khKzzM0on5Q3qjQYOWQ3Pyrs0ef7DXuqsGSXjwYfDtULpTzH0qAY0CS8iQHqyf7xh/BHy37yj5Z/VdxSfugoHkdUF9Rq2NAwNGJoCGJxsomJoBA2VUHeFIHCEnvk8WlA2sqSEh6oxwtpzkfOEfJTRqUkEQeDQoakIUcCUDk6Tc4UVcNGxpQTMjqMsZGxRqL5pfn1RWGMNFfXSjBiCRRo1AR5PP2LJqP0L6Ojszunazz4PGbHinSkTyS6mJ8K+1cn/cSHH/zW25ecvaRj1Rlzlnz1sXvUNWzpxs2Pt5RGmDJaKlXveuCBJ4d4eeBwd2nwuK5jd2xftSe9h/CkzpbC7uGneCU9vKqrdWlneX6lfAke2ssLewaKvIQQptc3cN3C2b+1u++eLbsK/UwWLX+YCw0MbaDy7f4a75z4/lChZcfQlcOFUqVWmtL6ql3DN6zZc3V7y/BA5SYoX0dL8cVT33zL1tuYysBuhZmsteJwjRWESRKmJUPF8ZL89OzfvjJ+NBECxexpQx3vhGXOMg1Y2+qBx/cM7Z7cNmVM4X+PBAQcB7n2ciK5LNeVXK5VhT4EzHSt1rlXK13Y8xFHoUStZK94gSwlzSOCyD+uaKVOw0b+oy8ZSFlnI29UaTDUIvPJj0od5fsNe6ocBx0HHQf1t6ayeeOgCWH+PB6QPDzCbzk+aSIVbE1tguYFKwsBA3GzKNHw7njeFqhWeT1VweLkWZbyFo8aUoWsJW1kJleS6zxImV9ZdEp5aHelTXNEJexNErYlcy73USpOyEk+loRMlWQFEmQ+yCrDTwQtuQq97KlFQ8hRdJFSNtFQMQ/PMpaNOpWNPKtfaSKqyRVVEdVgC+EQfV1bGqoX6SnVnXbpFCFchc1+zNRElnjGf3SqKlKFUEd8yub+HcvRR5DOG6PHAxoOQTJNRNIShxl7dYT8iOyphNSJI0Ht4Iek/sQSlfQLakdDaeQTm/FcEQ0JQxlER7Ra1b/5zr710kDkxMTI42mWJvqgalQp+6dSDC2mjDJOLPPGooVBL2kV65fKv5KQdBHZSA0Dz3JOOX6y6PhnBWnFZFG11To9umt72oylvIdwc//OCybNu2fzyn9a/+jFu7f/YNfWK3s2tZSqr54+u6WciBNQ0/y0mcc+umcWQlu5Mqd99rGTl23oP3P38IaOYnVa2/z5XSc+tmNFd3nB9I6T9gxcP7XzjM7yzG09hUntJ1eq2/rUd1aWypfOmvKurbv/G/JWrr1p9tQPrNvJCjGFRdPezwVo9Y7PVAu8DbF/Zvv8nsH+9Or5QstwrbS652sdLcPQRXJ6fIYKrel6lR2TFhGFDSqzBzlMF7KxLSLZmGLvv6OIjJHD2N1rkZPqatf1rj6x7cW5eotNj4DjIKeQy29cmdmVjFJ6diVQyphSgSnfSrVo+CjoOA46DvJVYXMcFA6Og8LhUJcmhAeH8PSOKfH7icVaRL3yZCzkoHnictJHbf7d8YwgjCXHmNDnGwY5lBnvP5QQRFEUMXqp86aRoBQJ1BIyooIoteUDlWSFNGrZDXnMPP0begloxjMlqvioVcQ/dkVsIv6Jv8lPPnCGZw0gdvX0gjxHKw1JSuzxg1sJUqrTWKAlRiU9u6JhE5IxPOjodBTqi5Jd9YIchy+HXNPFPOvMaJL3BvmRTxlrqGqCPubVKKMoXif58rSCaPpNDxWBoWkiKCU27FImnpatJsquKByWMRE0LPHAVmeJRj4RcMUgZcAucnBFbMQtNQZ4F4QNezgYJYStjqTRnA1GB8GDBNYxOnb5iCjKkhI/lJpfOv6V9/jHng89qhT9Q89gKDUMPIzPFkYXcv6WRecW191y2vSlHOnmgZ3UkpDEZ/YewgLvIbyxZ9OSjnT/4ieXnlWo3nPRgmNZVIalZYrldK9o4/CWxL60FWsbB9dtG1qzq7Kxt7KyzAtLq7fPqZyxOb2H8KbWYm19z8fai5VZ3X/RXp761NbCqm1vZiVSHjV8aHX6BcnWVn5jZ/vRpdLCbT0faSu/YVvvlUOV77FSaEf5d3cPPAHhG6z8YHL5tZ3lObyEsLv8SnKJe4avn9q6ZM/wTfIA8eNdhaVCsVJId68YGRqxwbGJo09LgcomnxSVH+lDrhO4Ko4dM13sZZiYHdV+3InTzQbrAGvuXcfBuD7HiUSjS7oE9BEI8jbRUFd77SpSOA6CieOgYiXfGcdBx8G4dPx4BBPCg8aZH778/IVccZc9MoThRaSLEk2ey0kjZb42GkoY3xy9lAiikXIbShmEmZyzixmfMAu9EoNaRYYyhNDTlq0uvEWQUxgjgMkmzKRHiSBqJI0CXvI47n4qTkS6ZB9dyFi8SE4o8x2pIUoYUbSSDW2lkQ2yhqqSWvqSf3kOg7xScniOh+ZhZSipxRt+JIdxnihSxa6q8sxTeg6N3uu8YUwtHw0sEoaCSGGSMswk8PTgWV2LRPBEmdAH30OuS+6FZfA3McPxliQAIw2IZ5KKdVyR3eCKNFctlvKMRoENyjchSYNiaYqmqJqMYWvjjWN+KT41ZXT8+qXYyIwgCsP8/+y9B6BdR3Xuf+7pt19d6V5ZxZJsyb3KxgVsbNwoxiaAg0nCI4GXhABJKPnnQRLyXngJ6b1CSF5ID2AgJMY0N2yMbXCXwVXFllVv7/f08//NfOeMto6uKrrSkby2juauvWbNmtnf3mfP/s6aPYNPsOKDEs/URVmlCNiQiv5BF5E1tYycM1krx8ULhC6NbJ849aaPPffFy/tOrVZa1vaddO3Y6MDMuMtviW2YGt44M3h/fksqWWlPFxh6vbW4/YXi86xDmPW32EXpfgjhgvTKfKm8tO3S7nTfS5OVhVm3DuHmsZ8bmn7IrRUYi52/4qViafy5bWcu7vkn3ipEM5X786GJO6Zz92SS18O4sgwobam0Ja/fMvzb/BrEKvblWLlc/trQVDUdY3j5HQQJW2KJvtbzt838c9y9tOjImT4U9wywNr8ocnTIKAwwMDrflt0SBRJhquGVwt2y97mzOL3MhozuE6FjMtP6QU6bupVw3w53b7K4jVs/CCDWD1o/aP0gN4Rjoh80QnjQnbEeE+Fa/MoepVviadG0ITfsyibsItCIUDDoUSpLTQxmMsA+WEbl4AqBECLPeQokEkXkFUSKzPqxpgoMigTKv1K6sTm7t6iNDIJ9Qxa7ojSYSUaQRrt0olEeGOwR2MS1ECiixoRUSlJxxQOxif7mqjYoVduCQ4Q9DyoQPAXisEGguLr5aFnaTHFaFW38gZvhik1OQqVigNIH8ikkaYMChu9ffh08DY7nHcTCQFDtiqFpIGiYDwamRC6WpFDBfViSxTOffDawSnFF5tLkFTscqg2impTiC6L4EmQsDBmNLgKBjZgt79CrAVA1eCCyooVidOpHg4EEsTsM8Kwho5pfVLmkVNrwwyqe+Yj4iTRqCKjoH0UcUdzygOjio7NbVrX3odTRKb1q6YVUh/LitqWMwbxvakdl08N3TA5ymAwZhVi1JGKrOxat6VnQEq++VNz+7tVvWNW97Inhdc9MPDtafvGyha9NxCtTpRU9qcU7Sxvwow2extaavC5fGiDEp3GqxfIYUUEZMK9MkNPJNRC2dHIxQ0DJZT7SydyjudL6TPINM6Wv93e87YXR29FDLHGbTS70ITtEZ8x/gooK3Pl9N6kMgsJ6OlmqkbRhEGnQKyu6u1/ZVVFt+d7EPdcW32jvEO4XrmPLwPrBPbuMcAbJQo72Nci6e8uGXW741g/StQkoCcjWDwKC9YPWD+pGcYRTI4QHDXiYao/f11mBMLAyOYru7snT5syVGcWVGy2FHNoXZAl6D5AiMgilgoA+FEF2DLD+xmOmpQItREkKS8yV3WMkj25ECx+Z3obsfv2d3sI8GerD1L1hHwRkbWjgMKR86OTU7UnGQN0eu8jq/2SpsjKWrG4AWTxKRaJ+ZEYqZSi7b5vgLbSEiuRkb+0JlphhEzywq0rF2diNbmSxG3o1tUoHBXkLlHJPM7ypoKgasipFwEk4WClD1SJUSgOvowhPaQwBRYANRt/xEwMkV49xwRIziBxVB32DpeIA8kkp0T8ENi17KK5IcSz54A35AIeMRmN0FNHhRBldiOlRHSyOmUWV0gy6TNcIP3UNqQgbgoiliJ9SjAO3RHaWnv7JeaCIOGew6KPPf+GqJRcGSDVk9B83fA3P/7Xlge/NbFuZPQGZSWVuH/3alUvWuHUIu3q2zowsbe/Znh8iizNF2pXpHM6PbMtvGyo/f/9IJR0vTZYeJ+2I56bHv9ufeUWhdN9grDqZqxbLt0/kqlOUicWe3fYevopwv0Vd10Lmxqbhe04mWkioEINS6Y3oS6XbRqcczWP4KD9LZZNvyJd2ZpPXFUp3tSWvLsZiI7NP6+1BTw+dZ0rVCSGi2/i+i7BpV8NHJc+ZRo3nNNib8qKu1yxtX763XNMfowhYPxg9cdyorR8EkD07ODTWD9Ir0fXwS6V6omi/BmjWD1o/GL2ZHEXZCOHBgd+T7dRzFWG3ipuyoTGyJw4mnoasXdJA9qivgcWhCWZqjeylD66UJT/IvAfYkCUnUoYsBB4Ao0ND5WdXhLAKV6zsacBARD3lc4yMS0SGK+5Sei8ob1rENDCPRgyW8aB5E2+FzQ6qIn4EpUsQt5FGu5LpREWiAu9SLikGSmUpakQa3T0QG/VGWOrnWKXsqkaEMCJUlnqglwFp2GiMDORBu8jh0NDwYTd4UFmxwQYzsoI3BAqG8KNKKTcwQCldp1Kfwg6c4S2iYQrZhVChjMWv5gzuYS8bhfXE9+a0jIYBKQKBdFeCPwuSA1dkFwNaSJOomi5Q0Tw6P2SlCBA/9OJ1itGpnTLGAzak4mlRY5Rsml+0wRi9+CG1yxuCN4+8MVi/ctCLr/oAY20aVZS8fxulf7/h5xeVN0B46Lkv/sjyV/Ia4dXL1/7T9u9jf23nYr4af73lmZ9fcfqndz6VHPTrEHYvfrLw4ubcjpMHvruztHVF65K2UvGMjvMXZXvXjZfWtF+cL21du+it7anudYN/v6Lzmt7sSd/b8s0Te94LD3xx+KuLWX6wPDw+/fnZ/IvF0k5qKZTWM90oAqFCiGJ79oKdY7chp3zAMFeqZlPX54pfG88xB+ntvCg4U7qLSWWyydW6TVGQzbFB+JznhAi7hn3W1x7cpXE2ja8UupI/3GZDRn84/JqutPWD3AO50enEWD9o/aD1g9YP7vc23fz9IM8hth0cApqPAQbFC4TiYCofWBxPVC7Xk0ARMwxCrmR2pZGAMhC5wOWCgfyjD94kh7KhuJwrJVcCjUHoThVI2UQFg6Bc7ZKKYCBIDk//2g2UIwjMaAIzUe+oDpL0iwRhmNTQTXDYAo3kgwxpJJUSGQqET5EryJKoFGkDmyILM6UIbCqoVJo5bYIfqtjTmIJwLRWHicmDLCGK4qiUUkERNhkHVxQJZlBKcjGLWqKRgZxTMGoW9YZZOARaFRomG3ySK6DC64grWxeTGzohDeyUPSyRD2dEDI0ho5wRLHXKYDjsiufIklJRS+zDySXrv7c+gD1cMVwJ7GJDFjSJFOql4pTChpY8NL2Nn0JhgCJ1omqwL+1ShCx4HQKvFyIHYyxD3I9cbcFY40tlDKODN8oYuW7r3lrEgBggnkmRRS9JaQApRLSmaV9KqUdnt27PDTNW8+PP3wJrUnRUx/WX5/wMBkwtE5xDX5H//tlvkv7ik1+5Y3Ln1mmnWdbRc03n4v998qvfuficyxae9pZFa29YetmbVly3PHtCX6YvTLbSm1rJLKMvzHyRIq3J7pnShgXZk5hLht3BqTu2jX2KwZ+j07ePTn0eDdOKTufvKZa+gjyTe0rhQeRCaacGkcIM2eXFQs/gYos73hb3U0yh9LFBlyIrF8mvIci3r6bSTYw9AoOBDaLk476hh3V7aOJbh9WfOWsKBKwf5DSEW6X1gw0dXOgoQcn6QUCwflC3LesHhUNzpkYID/28iJ4p1bMUIzD1REX8MOWHZUqjXD8XaCWQOioWZ2togZQyQ0YI1A4haKRUWYaPSsBY9iqOMgjjxbRsYIDihIEKSh9SPfpHCYCy0PDoDx9AEOVAzy4fBJRKEaTRruSQIqgTFVGEMYorSiAN/A0WJJqEHzZ1MErhSGgCifL5jTYNfoIr0S38YBCU7AZvyFHmphpVnZhbKBWYm4J7OJTPYKaywQwahqbBDM8yY2QpMrlBgx5X6k1RssnVWxY6MsAW4mDIOjsCH/oXcPaGLuHckasTR4oGG9HFEDBECSmSZQgYUgqH8hm1DENG5VbXhlZEFEmjefSCaqSoGuwOquZa4zeGyvBX7A4z+Bu74o0ifpSSJamMmV80aIJnNKJ/+EcO9E/NYMIYFRG35C1BeDUn8cER9yKlnLC8BPK7Vr9BlqTPjL1AyjGSKlp4v2/MW1ZcguY/LnkH6dr+Va/r7u9v70Zm2zg99J2hZ1+YGViY7TmxaymrTXx54D83555/ZvqR7458/Zmp21iVHtr25MgX79v2J5PFb/9g8O+eG/p3phXNJJd2Zq9gQV4WIezIXgkPTCWWQctSyRuQWbw+k3xjoXQbJLCv6wYEPgNjPzWV+4ti6auMGoVJTuQehd3xOiLNaE+9ZmHrWt+iSEJ2C7HB3ehfiCLyHdTzfaRATWSZCv9xBnVZmgNN37H0ffYC4Z7AHjcadTFK1dNZP8gtkY96PU50kKUh5WP9YLS7BCXrB+m5rB+0fvAodg1GCA8FfO7mcCooGU9ZpPSCdIf6BHdRwqYsjGfLCTpLuCJF3Iwv/gf8TByWWJE9qXpW7UpuSKOeVR3DRxEwI0u5SiGK+rWf1ooEkmIpTogQ9PKDnq5Lcui6Qt+GRnLUQDIphCF0e1En4TdUWZIlNsKuqkCQjXZFFEkVUSSFJaqrCDxQjEsOSZUbtUHJE78+0gdWRiekmFsoLmPM5J9dDfJEaOix5ESp3Kqs0kAU4ZPkYiaOJ8pBbiCK2GPQQDtFF8mSmarGCZ/QNrkKAIqzAakAJAVDUrgZfmSs4F4oouCeUmwUKlS63zAgnqGLVKd6xRvFFUndxVx1ldKr4ZAUOqdgHRWxwfQCVVPbokNGFS0UrwvGKhI1DvOLRrklZvr9lRQSyG44LswILZLWuKX/PkTp3+k9q1zj/Hb39kf4K/qnaKGqlj1DRmt2sdhTQ5uRHxt44RvjA+959Ot3Tu789sDzyqWGb+14+J4tDzw6/uRb+t/CG3R9qeXXLf6J0zuuX9F26YLURSe0nndS59WdqVcv67y2J3smpbqyZ2aSi1uTb2jPuF14YCq5mAGivECYTd7Q3X4d8T3PCXeWymMYECRkGlJWoUglr4cN0oNSKZ/Z0u0QzpnS3UwqoylqvJoSfpgoZ6i+SayltRUKa3mwxCjxq5dQDfW9g/n77OQzk4XJgylhtscGAtxt6EHoa7iQSK0f5LRZP2j9oPWDfBGsH2y4iR8T/WCyodG2u18EuNB5Jm5pcb0gs7PkWTasPiKUTjHKyiSHXIRobgjQwRKZ2SUNJywneJEpOAkF0SBHN/xoV8bIUZtQr94zFBWUfaCCUW8yUJZaRa8mUoFZjad5oojMsUsTsiSIcpDLbrS4AkcolYXQILOrumQQqoB+SCOGgwxL9KX9nDc0w43628UVxaBCbwSJQhaVQojyQMXclEsaeCDOsfdVuAQz7cLcAkVULkVE24IxAsyNWigiY2Q+6IlHyQ8p1ckMPaQFjWzQ85E3mUEm0Sg36P+qHl4TYsKZFINwUqJgopeNMJQNqfiSg7ReKZawRFgQlshyghDCgMgYU5aPXllUtBDK5Er54bLQOc4I7AtjUpE06JlCfyi1kYUSOUQLg7EMsMetUnxGjfGJRvOLwv2gecqloCpSaBHmqWYgYEZFpHy5cEv79a6g6N8vPvn3lGVkLGmgf3BCooWPPPn3/dkeZz87hp8nRly08CNPf5m5PL+909G/Ze0LSD99wesfG9zEOoT3jz592aLTvjf+g3N7iElWl2VOGMgPPjHzrRv6bn5i9NsbZr56TucbRosPPz9R4cegYvnbPdnfzSa71w/FNg69mTghvxNtHb4t4dayv2Fy+vPF0q04TyVvHBz/pIsZJm9g8YnZ/D3Z1BtzRYKEV8IYecNw1kUIr+/MXjBTel2xeAdF2JhUhpQ7lQsKuuUHYZS1IaT1kaR+Rhk/njUaJwyyd3N4kp35rZ3pzsPjy7w0DQJcUdwQrB/khHCL0GlBsH7Q+kHrB60f3PM+fUz0g0YI9zxx+9J0ZTrggfSF2pAZbRWleSJmgbBhJln6hlx2lRvm/5Q3jeCCJVJcNqQqG1JVqlS1yFINa6hXyj2JH/rAS8llV31b4ANoAmeI0jw5DFkqGJgbxUNWEPCMHPpOisheVUhPilJFcCJliG6xGzy4rGrNA8oGNgWnUhiQFE4liiUbUu2GfkvHgj7qBGUwEMETCaRsIIqB2qkuigQbucI46kccL5g1VPfxNW/7+PpbIJ8huqXiNTP3ouY2cT+QEUoQGGQ++AQE4SbQSAOS5LKxC5KyIcWAT6B20ssSh+xKIznUi71CZ1iqnX/10h1crrw3CGXiAyuDicG+SAOpEyEUVVMVSjW7Gu/1Me1n0MtMdE5K8TQZI1OL9GqA732dQnrxQ1I+GCsw6LJaaldy9EgpRfSvkf7lxkCGmUVJFS3E7OGZba/sdS8r/sEZb/6l7//3z511ze33ffbM/hNft3XjwPT4p7c/9XOJ02FT3xl+9snC1u9v/OwFHUtfLLl1CCkyXNiZaon1Z872t47qtcs/0Zbsuvul92+fejjpeFlsec//K5d3wvcWdtycyz+aTizr733HwMi/zeTvactcibxh+3ugf22ZC7raL9qw7Qxms+rrvmFw/CuMGuXnKNjelrGfZmGJ1uTrp0p34lCbCxLWXgh0tYS7FpEct+8VnDsZExWUcNjTKxZde9h9msOji4D1g1H8o7cUbhrcaXXbVCrLYIMBMmnwIHt2g15OVAQnMrZ+0CFm/eCpP8qYF+sHrR8MN5DDKBghPBQwk/FKsqXCO4F7o2fSy3WQA3kLGgwkh6wgkBVYIqNM2RVh4+EPfTZRQhM1lp/gLWRJIA2Pg/IzJznMJMrqhNRpUYW6KCnpkEI3Jg0GbDJGI6VsglJCUGIf+kvspY9SzdD/hSr27cE3oTYPKpZQMgKJq1oXvzC7U+RNBqTsQq4CzZMeTZSYIWMWCB420sgs6iEURynnCjxKL03UDxqIJcaBT0a9Ia8bXU8q8qnq5MoNmvWgCRDwYVccSUM32RWMDSE+2eihJECNDaUC4LA7ysKsSOVNluwGrqjqFDwkxSC0BOEmPzcPLA4CRqpKKQIHg4zB6/g0ULXQfg0Z/W/PBqO8DvuG6B9FZCyCp9CiPCsOSb0chRqgFBuy1ACo0apZF5PUpKlKo8tLNNA/jktABbr40Lr/19e64KK2ZYMzY5d3nHDHi4/j7a33fa5SiscSFSaV8bysyqQyqfESje9v63lmctEJrYuemymu7jhzQXrB/UObVrSfUyhvpCDbVPHbLbGbcqUBlouQRmkysetJMaonGMiLhaXSGKFC9DO5jcUS7xZeTwSwEqv2d/zO9qmP5Uvf9PHA2IuTv12tunXnNcEM9jBAP3mM0+0ZBoQiRuuKymHKGSmx82SyZhL8R4s0yPcO3fHKEy5vUNrucYCA9YO6E+pUcs9kVx807CqVRrlBSZb1g/R34GD9oPWD1g/yRWiGzQjhoZyFYiVeirnAoKgaLqJ8TB4DE1NWsAn6oJEgPcYI0SwxNzQa0sn4UuRAEXmSC0QxX3G/8csPgrZQe3iMU0gw0ELMJCMw8Uzo1fbswDBQ/0eqXDSBWiCIMcpGPV/IxVJb6B3ZlQ2CSE4wkBAaQMcpn9EetMGGXewxYEoPBOZ8kwGP6WoYKQ+vEC3pd2Ncuw8uhcXJhtBisEdDvyWCp2Gc7IbYI7lYqntTERk3+EFJbmCk+w0YEp6ClsBzhA/HJXYnt6QcFClKjd5UxEzGwplcoSdw2BX3gz4FM5Rs2Ac+qSx5UxrlihiLLF3aewbxJVyFcZuwL+/MzfaJPOeQUewD9wtDRmGAMqa4HMpP1Dj4hO9hTEpoUWQSYxFF0T+VFQtFJrR469b7cQUIGiwaXV5Cg0VBias30L9dg0Xr0UK8/fpzX3S0qhy/b2rnZX5q00+vfcPPPPSN95171ae+f9f5fSe9sGVHLQzXEhvKj3xr/IFrE5dszj//wEgl01KeLK+7Z/CR9nj+y5uuTbXwTm9s6+QXUi3VUun2wamYF77K6oKV8lcyyRte2M4Uo27I6Ews9vxLv0z0D3narUz4vzTTzNj0GqYeTdWWsHe5fPnTydcWfYRwZefHNkz+rtPWv/Z1EriLDcL0xAN9kNDZhs2TxrC3m1D3V1M2WB4IP9zNne0cywhYP8gtRX1T6OnU3QQ9Amc45IazrfuzcpWSpRuvbOQWWa5IdSdHg4ClPMg42MgeA+sHQwcnJIWeYAQl6wcBwfpBQLB+EBDCNl/DhEIFx6XA2z4cV+Ba4RjF5UgDK5ONNMoV5SNtyJJeBaNZegKDs/EJwztVY3SXX/55vOMDV3RrJPpJazRdTWiq3O5ZFg2u5I27JzdNPjILqbof6UOKwGM0t1rMEEgbikspJ/KATCnkoJQQdoNzuUXPnV1KvKlgsAllQxtUCwYiS8EAAQrBs69fEqM2tSnTdfr4m3sREbIngkcET7G+wPHIEtmTgXyK+MlGHFJpoIINfoIeJ7IMAUM0+IQokroZRP0SHXp5j4ic+v7osXN0UXCQFbiTTYAI2oZDESHZkyVvIbQolKJ8kiIYY6lzJ3vM+KhegY+ZuCKkDhoGxwNeWJNGbBIiwyBsUmKApZibNFoKgjRYYiPiJ7cydmNK/cL0pJoylINCDqFFeUZDA+RTDcAYDb+/cji0llQHDlzOoV82QynRQo6LDpI0RAuxF7Dg88jslk+cehOlfubU15L2t/a4dQj9dvumdUwq84Hv3/qdqe3MRPrY7JZL+s+7afUbr+p5JacSpvXK3tde1Pv6/vS5a3veS4mfOOV7b1vzCMLyzptOWvBeIoSnL/6DJd3vY5IY1iHs6/5nshZ2v689++F08kbkExZ+1U8zc0N75sqe9j9Ew5ZKLmNqmXzpq3yYoXRhhwvBzZa+2Zp0YIrecU/g7uE+rhm6kbiYIQZ1foi42+Z6x7BQxm45B7SD2z09vGnp2w6osBkdawhYP8gZ447Bh7ul7rG6bXLrkF6nVErJZPFRQQnIQQg2FJeN3CJbPwhKIBnQA6IoOMjWD9LDcqlYP2j9oO4kB5saITxYxIK93sNxu+J7CGJcpHxE/2Qd9GE35IasBioob8oVFYzSP/FDeZM+mis9qXsinIsouglO/fMhCQVDWdxSihsuH91Zwi4C91/di5HZZIYy9HbIwUaW7AZjyUGDwB1cu8FtlMWF/k8VkUY9SJYHstRPoJSHqEP80IZoI9WkoHQrJXiiqAlOoYh8eIaucbP6IWAP/dNHHgJRZBe+J16HAYKIojieZhNVERFLuCIGcuhtHEGF5JCi5KDCEYUD0USXZPHBRoxIbC0cGgxQCMhGNYr8YCMclBXFmSL6uTR4QwhIYo+MK5QqS5OgVfziIBpGsI5PoIJqGA7RyEBcEQ+QQJmRYkaqFor1ySzan5ErKqgRNVqSXlOGhtAiNgotkkICqUI0lRRX0flF1TDBqOUlRP9EvJlahuPS8wSHiRy11NQyNIMho3//3Dcvb1/yC0/eyjqEt2xwvE5Ty/zF2Tdc1rGES+LNCy/YML55Ij9J1uldp5+YOWWsMIRcI2RI9W22NLBp9FOc8nSyO53sRRifuX1i+vPkT0zfPpv7U1afRzmbe0oLErL+RL0ok446OtqZ/UA2eX2htHVk6r5s8nXOuHRnPFZ9YfK365aOkXoW6P6yiQoGfuhDfCFLJoch9W5rfjZOPX8YPJqLJkXA+kHXXXKTDPdhZD4oOWNKdQvVruSgQQj3WBljFr0/626sshjwiXqQLA9k0QYZWD8IAuqwgI7N+kH1s9YP6nqwflA4NKQ2ZLQBkP3v8iDLsyZvJUGfNHQT2sajPHPAwLICuSILjX/fz7Es6WtPXp46igEqxUDcDyFwRQRe6puorx9IlgibKJw4odySsqs01BWUaNhUVjIpwUMMXCPrgceeVEEHcmHb0lVuzImL5/jJQpbrMZ0nWviMmAy9jlyp+yElCwNyQ3eFUrlYqttjF1m36Wgu9266PbKUBs8SKCs/FEQglZk8YCPn6gvlHIfsoleKTagCg6CUZ/xEncgDSm6dVCSWqJa8ddEFlJUrvdqnZ20Inudy1QvblmOpACCCKKIoCsjwEeWTGUQU5+Dsaqy6QY8NDQ51oRdvaThwNZtUbVafx65rdv13aASK85GN+gMdWrBEiJqplqhlcIgfjKG4UDJF+djlECRznehS0cUDSZOeXcy04VZDRoOxmCHGmJFioK4L+7B+vcpSRPOL4laVYgy82lWltWr8r6RQRNpJ2jJShdSFLM0vGsKkHLsGi4Z3CznehmghBLhaaXloZtsHl137uxvv/atz3/zeR7/KYNFvfOsWppa5ZmDxwMw4/pe29X5q+zfi8co3B+95obj+u5P3tifzw8M/WJk9daq8LjPtbgX3bP0TBogyrehsaVtrcvVk7pPrd/5BNrUM+ndi30dGJu6AE3KjYH5RzwlvhAdiDCfs674tlehl1ChOxqY/z2qEuZK7o1Ri17OCzWzpG9nk60ulOxi9wJDR9ZO/S1YgfshRkhaR+dY6J/xRH4l8WDa8qfbvTzz+xtibD4tPc9I8CFg/yLng1qEzovsnqfWD3EisH7R+0PpB3RmOrX7QIoSH0sNGH7Ogbf5Zyq0xCMXCnagagmaFIeUDE+DjRnLGK3yQmZOGgtoCCezyrEzkkDTKBsnCPx9HFD2LU3V4aCB7YRcDyaQqq+qCHHIRxoppCC1P/EyoqEd2jPWwTsrH8Rx/CLBE6BDHwocAl4KQZPFBTy4fHWzIxUZvNdBlwnPUj4pdUAsaNSyk6l/ZDUwGPsNjOrsqLgO4GTbyhtDAA2UjIiSSg42ckKVSONQnZGEg40AYFHnDDJuwUZzRp49MOz8ih9BFcpHdg7VHgxQcNEgV/fuXX4cBjVSTkOUzAIIrHRF68TFsMEYfjlqlNBxUQz3lDRuyZCkbuZJlsIE04lyHpiIYy48bGFln7Aiil6RS4o2CkFgdLKXUQi4VOj/F97hIYHRkKbKHgaigaKEsRfY0pBNjikimlHgdBjiMxgC1JD0+McaMTVXjHDNVjZLiGMAYcRt8qgiHoPlCG5aXoNQBRgtZsuJ/nvJ67Ptae4gQDkyPMWT09o1PomFqGYaM3jfwvBsyOvQMmhOyC1/Rc/Yp2ZN+pO+t7L575ccZMtqXPvfsnhtdbuv5K7vcwM6lHdcu6byOlQMXdlwHJ+NeXCyPo4cZdrW7S6U1++FkXW7PfqhYGp7JP69JZXrab2bIKNcZA02xLJQ2tCZflyt9k1tKW/Lq2dJWHPr7En9rtJDvIJtPnOAuURI/uWiw9BmHLWH4KL5syOhhA7TJHFk/yM3Q+kHrB/leWj9o/eDebs/HUD9oEcK9ncS96nk0TDHLaLwqvgdtg02RQtJI6R7E08S1JOOLXckqFXIVY1SKGdO6KAtXaoHcIkMOkRGUFfQIoRZygyw/ckIaasQAOZgFQUr9soV9EOSB3aDRQ7ZS/8TvHvvYGgzQcJeUGQ+lD0/rx9QWr3HPpfBGHikcKao5qMWpyHLcwxMhZFG4KGmEnIggiedolyIoSXn6byiOk7Apl11KifhFsyRTPBggixQ11IIBXEusKUql1LA9/aDBGG+qlGbwQYmGD7KckMtB6WDR64hUFyyuoc2qPdgIsagNHhSflEO1irpkQypmhQ2Wal6DNyHsRtVCTX0argQxNzFALgPROTnU8E6ROux1DWApMzTM2IlSjFF+ZEYqhzA69PIWnYQt6pMa+dAwfxE6liiBqw4nuOJDLjZ8K3/r1JvghIoB1lYXjLwxGMg/4Oi0iiSft2A1HhQt/Ifnv86Q0f/1zJeLpQSTi94xObi6YxG5TC3zvnW3Xd5/yur8gmUdC5Lj5b5s78LW7icmnyCX7cWp57ldDBbWbZk5kd0ds4+PjP5zb/qK9aN/Wy1/oyP1um1jnyqXb2tNXr954H9VSrfy6uDI+HomlSmWXJxweHw9paZzfxaLfQhBaxIisMEJc6Wvtmc/wASkO6Y+pgjhdOmulP9G+W+VCKAXSSpO4EeKXbTQ+5nX5LvD95/Re9a8VmHOjzwC1g/qtqbU33n8t8z6Qf8jJv2IOjiuTDoX0tAxWT9o/SDXg/WDR/6mvd8aLUK4X4gaDVriVSb9oztk1GhrskRarMaVSqkCEK3AtdAgi5JFiRl6dvWRzEr3kEM+PMLyph8fDeiC9YkBBqKIPRpSNMpFVo3RKkIbEKK5qhRNqF0yTujhol1deLBWFqmes4M+7IZSygp+VCTk+t1tHBdP7YwtISD58F4iinrjjn5FXQs9CmXVuyCwK/aiFBuyRGDIFY1E0BY6p+AkEDMMVEXot+qF3F8VJCvUIg9k0bGRS43QCRnIZ6hCjRGrkR9VRFnsQ+wOM3bFysiiIlI2jIMr8bqgj4b4KBsqQsaGXcriRymaUJ14jppEa3UIspRzeZMfNLKkA/vC4OMhFqdpYCBanFNxMBEzzrsIG0pkzq8+8ixXSsPbfbp4gpkuEtmTpaPGp8xINWeMGiCzkMoMGhmcEFpEVo0y08hbmCHIhNOHrDcGoYvhwFU1loDjLFnLpK3/e9Nb/9fK11zRcQLrEOJwbf+q13U7RuqXnYj968A6Sj02s+WC/rPPWXQGP3asm3icXNYhHCnsQOhJndCTesWVy37pDSvvYfeyEz953pK7ERZ1vK0r+wFWGiTux1wyXe03r1jyaaggEUJyWYqQlKlHF3W9vaf9umzyBjihhowWS7fxEMqKFJO5ezM+QtiavBbNouyrSWF99VGgPjro55jBFbPGKNfJfqub1fcP019FkPoyDiLbjjMErB8MN6UgcIqRdf8hRdZuuCNFc72x9YPWD7qJ1rgwuB6sH7R+8Oh2E0YIDxp/sT5m3EYoVVsS8QqrAvpP2VHERKnVLygf/IqesSs+FmVlDUqyAitTFruMRIUciiVq+lCe87qSRbICFUQQURQzxIBNciCHezZDdZHywT4Y+NKOFob+DA27oVfjthU6tiCgVEFS5FBcymguGuVCMBpyPbvgYZWNiKIjimF4KoNzGHRKcAO2wzM6D+5KeZoXh5GrwAPR6yN9YHHaJQtBqTTBCR7ErxCCAULUgCLsKg3NaKhCTrCBvAU/7AYzyANOqEUV4Udm8izmhr3YHTbI2JBiICfBFUo2eAspNjKWn2Cj6pSqSaFVokOqUc1WFjTigaFnOAW41QUgEqjdQLQ4jw1csWHIKPZsgbAhRyklwcAoqxSZpI/0hVyiIaOYhSuQo0CvesNVJK4oakrbMCa0yPUpYw0ZFdSB/uGEGCBAKQaoAaUCR5aQdnJB5sL2pef1rn573/kUYfvbH9zJkNH3PPr1b44PMLUMQ0a3zYygFyaPDnz/ySF3Lq7qc7zxNYtvuLz/TQwZRd7blk4ug9eRqy+vzEql9a2ZK5PJxZBDvhWlsqtCW5+blfSNLDvBkFGy0snVOfcO4XUzpTsYMrp58rc8BfQzyviQIKVqO3UP/J0nHhhq0Df5/N5XBI0Jxw0C1g9aP8jFrF6G1PpB6wetH5zz9n4M9YM2ZHTOM7gvJQPneNh6bGYrjEs0jAihCqDhzUD4G7vwKxGtwLuChtzAwRCCgZxoVzYiaTIOpSBFvO+nLI01ZfaafDlBDy1LXX+koaCMlQbP5PKBvsI5laoBpDxGSxZzC3oJegQPVFBKPdPLPhSny5Q+5GIccsPzvTiA/ChXzpXyiI9ASr0UIVT1ijbeVOSH2OUrW/v8+4q1FxTxEEgOXRQydEjsiDQwK9mQqjOjlARsJIh37WmgFoZaVAVp8CxX8qMOUhoMQgPQsEuumiEDUs1rQhYym5ibWiINaTgcZOqFfuAWP8EVgmwCD5QN9sGVbNBQViQwLHEhMzxzDb8wPQhhEL9SSi4b54JPkDkjOstRmzmHjELY4ITwOorvbcgobqM1il9pyKgoIh7cNeCvhPArA2aBQCKTG1qIQzkRvBoyGp1fVEQRswCdhoxiGdv6gA5TlJJ1CMvVlosyy789NfCTi8791JanP33BG1mH8G2rL7xr3W3Xrjj/hdxOlqNY27Z8MDc6XBh+PvfCysmnXAPa3UjRocK62bHvZePFf3/+4tZ4IdNSumvTaV2pq0rlbwy55QdvY75Q4n7V0q3l0o2T05/XOoRKGSfANjzOeNFbEeCH49O3Ex6EPcIJO7JXzhS38g7hVPH2llgc5aLW/7l95p89uxQxRHSvs3o3bsioBG8Q5LruMP0NL5g9PvLwsg432ZJtxxMC1g9aP6h+x/pBuhvrB60fnPP2fmz1g0YI5zyJ+1LyAPXo9FYmhtGsMHqeYjrQQiUuZsgjGNSLxzI+CBiImCmNsjI07JLumUULlCX7aCnJKiIzaoKLIkvpeKkPLWYSlXw5zvtEs76i0JJg2ZvKBz/dzGfj58XR3S08UnOnQz4QZbQIVbC7J9NDj0NSNtlrt6GWwCShBzghN9r7olRZZTlXtRcUce7mRMUerkh3FaVhYkQo1YEFciW2hhN1bwh7M2jwEMgbtVCEgmzBPxoM2IV0EXSSAWkgZsjkqhnIDfOIBlc4oQhO1DztKld1UTbYuBbUCaf8ywYlFQU/cCTCiZTSsEl5xoYiRGKBDs4g5oYlIPMhV8AiiNpFmZtkDMJ5QQiEjSLyoHPNLkNGSdlkpuKyYfFABPE66TWiBm9cA7owaIBIGh4w5hOq9l5dUJEqAvlE2RADlBkHHn1jkEo5lcoK0cKa5fS2D554zXeGn33lgtOqlTjL0P9cpWVgepwho6TYfPqpOx/Ib3/kuS9mk6VEvJz0wwQUlPvmli8tzCzE5tV9H//e8K9fuOj3OlO9jw/8j7P7PjeZf2oyV2XIaLl8xXTuHoaM5vNrsGzNXDA57WomQtiaPXP78PWQQMaOFpNriiX3emEs+WHGI0BQWYdwduqr5Wq86EKA7pcp3ZHcH6Z98hruD9yL/A0JweXrNXfNK+Ny5mGrNWMePJvLZkCA82v9YEO3pfOieyCybnfs6gY1Zy5K2cu4waH1g+CjLiz0ceqtrB/korJ+0PpB3VUOV2pDRg8aSfdo1eLe3NNkoX7WULe8BEEV/9QVg4bBskS0HDFjUQo/gJOUj3YDDUOIkj18B40EWYYUY2RlhaZLE5QIZLGwBPVlE5XWeLnNr5Ax46OXPOujUWNGihmRWIKEzGeT96HO0DPpkY7uKnRsEsKDfrCkumDWYLy3XYrgh1wKqkeUhpSNm53IgKI9ch7qJTe4VXHtkqqUnLgzUm0hooigoacctZv4tB4SxIzeBZpEykdRNVI6G7KgBw0G8CU0MpDMLpuKk2qX4soVu4B0sUsuu6oFD2gUnaMI9uhDM+REuWSJbYaAIWYyIFWDEVSROBIGc9pgRqXyo1GRwezjp7wNiFhMgt8yBCCoypvGeapGwUtKFqnw4VxwjvggYEYXRarhnRhISYrDvQ0ZlXNSqsYsvF6IRlXgTTIXgDRQRNUrz+TSHqrGQFUrqIieTevXi/oK1TBEVqcJGx2sYIlaasioe7ewfal3Ftvuh4b+/Lqv/O22p+7ZvuGbEwNbp0fJummVGxj5237x+v993s/96jnvPyW76tTOM1ZkTlnZcYrKThYHF6Sc2eaJO9tTV4zlnsr56UDzpZ1TuXt4h7BY2gnfw6DkhFsRWHOCNQlhg/KgNJm8sdtPQ9qR/WAmeT3vXXRmf0FZvEPIpDJDs//gKSChQC52/99nu53ddiPk0GcdloRKWQsxuLriBHdJ2HacIeBOsPWD/qRy8+GvUu5gB7VLQesHPYrWDwoG6wetH6xdCUf+j0UIDxpzOAaPUTxAa77Q2qyhiRIUkSzc5StxaJuCcuJmoY4onQtZWMLQSFkJEFaGsbJkHE1DVoNDdjXmk4JtifJMOSFLlDziz1Z2O8vtySIDTdVUppcgKhi8KWLw0Iybs54NG0ZmElL43vQ25lekq+MhgM/nB5+4uH0pD9PoMbuobSlovI3RobMDCO5BgeUrWvee65kJZVUcNCm7qR4GpHekInzAMUjZ0IgWBrInA7LoetUBi0uggSGoP1Zx5ZLihNzaxnH5JRb9rpvbRgsGhodY8asQ1oN3SRPCdOxC4fjIIVCI6aEPHIMsGZCi5CPiJ1fkiijCylSEWpBVFxoxNxkHP2KAKC/tdQFD2FHwhsMXpmnVtpv81Du8b4kNNHhJFlY2wKExBJT5ezinpMRRQQysFAzUCE9PiloC7Do0WBbsS8hjT406F8oVvA7AOoGXXvwKBsiubDgpUDU0pGg0ZFTGoUZs9maGXnFCiiCroOolpVXkosQVu/jnIxtSXmh8+Lk55hcV8RO2OjXRaGEojuCihdXYn790F5c384t+e2rnb6x5zb3bNvzoyRd88+GvX3fSuZ/e8QMeji9jOYrZMey7Mp2kDBndvuWZtkTxMy9+vC1RaIvHNs88MFN6aMNEJdtSyZXunU2dXCpt6G29YsXCdwxNLB6d+ny8xcXvWHOio23t9Mwv44TXCElFDsc9RWQXfjg4/kmEqdyfM2TUfeN8YDCbeu1kkXcIr0slV2+d+Rd04S1B/83V19pVMa9bg//vjzzxyhMun9cazfmRR4A7CSfa+kHu1dYP6vIDCm6noQsL12TovzDgY/2g9YPWD4ZvR1MJFiE8lNNBL0hgkJGZCCrP8FEN2iTN+jXf0QdeJ1InDXJDFruwQVK3EmAklqjnPDkJrjDDQ9Rhe6LkPHhetzCdhw3KgJagVEEMwjZVSrGMdS104Dp1Vw+p/3grv8Agh6aHSB0hS3Jj5V/Yc88BdIHIWF9Uj5zcAn9Ti6sxjL2944qqd7fcell4IAY8WGyaHsQbPqGRsE2qlow/nu/lAYHHfciACAZK5EARAw8UGQg2lIraoA/0A6qDBz6UZfYaVlkkiwGTRBRpEm1QLJGZTiFddGl0Y1AI6g2BNYWP6OFCrI9cLDEg5UMWGrE4BDZpEMhVGs11FnsPGIIGlA+QaR78h6b+5eY7kXVSaDDTYOKfw/FunA1H51iZJ4Fe6WJ3CKSBwgmTgAxFFOVTLE6lMHYozbj50NDopAAsxgFe7DEQCZQxZnywUaUURJZDhQF1LvZrRqV4xgnFQ9XySe3UJQO1jV2q0Dw0717zemRFC/ecX5SzBlw6iTQ+nIhotJAfBfDAqQfbD554Nb+DvGXFJT+1+BwdxWM7X3xtV/+n1t3NTyAf+P6t35ne/oC/Qv7yyX/50oavYPOu5T/HWbtpyQd//MTfYfeSRe9i99zen1nV+VYihEs6rj2h80fzpW2F0ji5RAiJ+7HwIOsNlvyChCgZMtrb/T4ESODC7n/pbv8jRQvb/dSjzCvDkFE/y+hf8Z3OFd06hHw9Z0obKYK8a/N3KvHDwBJ35R4micBgNDZ4mLyamyZFgHuO9YPWD1o/aP2g9YPRe/Sx2w/uFjuKHpLJe0OAZ0eewnOVGpdm5k9mcyE8SKgQqoRAN0lZ/vNAJubWQOECncMMwsYuxuKE0pAGm+h0L7Lh3SEM2JQ1XU4G4+FCBv2eJJAihAp5EPT0b1fcwLtxrfXt9X99s9V4p8VWh+M8OCpI3XwkkKo7RGAj9EQKFYTmhRRNoIUNuY4l1nNVJJRy8UY/pyiVIgApRPGi9mVE9nTzhSFQlqd/cQztomEXGdogYiAbUpFJhJAlFqHipDKIlsWDD6Yt43hFFDGjJd6DaxJP/4/6H0QDURTfU6CJuqJxPFFElGyY8Qk/lCKghHaiDAFDnMBGIKVkce62zwxf2LocWSQqepi0R2SMVEe9NxuKw3Yoq2PnYJFRwgAhckIGiiVGhA25InjYIPMJ9A9Z1XmUdoXj5EpsDQOcUCRUhx9tmKHEjFSaOc3Cq4D1cm4QKTXyCcdIllql1E1wuuUBSCD6u7c9Qlo/hOqjs255CWaRiRJ4eVa0ULJS8UPmICU86A/kaWKtZP3n5u/eMzFQKCbKpcTq9j6GjP7dK15/57rb/uLsG3/5mf9k8foP/eDTb155zVhhPDYYG8kPn5g5lVIvTT9H+r2hf1yQumjT5J3F0oZkS2zj6N+Wy9/oTF33/M6PMKlMW+p6lh8sl24t+UllsCdgmMvfM5O/BxJIkLBUuhIZPVtb9syxaScwy2h79sp4Ys1sacNM8Zt8E7lcp4r3xmIJfSsrbhy4izyiZ+Na0k3A7x22hLrqNezmkxuIhQd3Q+R42eFLYf2gvm6k1g9yw7R+MHy5rR+0fjBcDAjHSj9oEcLoWTsImV9GZQ37ciMw/Tt7dJA695x+Zv50D0l+DBg2UZKGDIXDUgKpmB5KPoz5VFZHJPRHGJAPI0vJggEymQQ1EgwkhezJOZXO+UxGEc9RRedcQddKnyKwqw0hyCFXbJBdHZoMqKhWpu4qWhafYoakvkm7AoaUgvgRbFFxRQghgXyCJniWGaRRua6gm9/VHSLjS+FOVAql0Qd+QsEoYxE9EG1QljyLwAR7iqMXNVIRDII9ZnsxaHEeqi2NEcWqgxqiSMiRWfhgO+wqjocAu1PqJuijjmpMCy0qosiTOk8VLiyJvhXO9rRyfYO3cSDhSKOHiRsMRKtgPmqtji6wsmBDrvzoAOVHxkrRYxBAw56OjSxSgcNTIEphWCdau7hicBWGjKpGVYcTBLUKgU1mqlGVqi6ytHCFgntRb6qUxkBfMZY35qGhiNrGG4NUGngm+nq08A3ImlpGDhveGOSK4pmGFLoIRccbFD1MLcMJZcgoHk5q73vX4rN/Y83l13b1L+voef/yMwZmxq/pql3Sd219FJsTu5ayDiHCusnHOdEX9V9xSd917HanVnWmTnr10l+6oP8j08V7eltfvaCV1SPWMKlMZ/YDycQaViCE+xEkJDCIPRuym0um/j6hlOxuG34jcqF0W6H0Vd4/LJQ2uGUnUq+VQX/ruxB0h4p8W/33dm/3CJU8pJQfROf0GufHslj1gZ3fPiSvVugYQMD6QZ0kdal82d2N3W/WD1o/GO25Gjo46wetH6zfKpror0UID/pkOJLTUmVOUUpy9+e+z+BMFyGs1H6Sd/G6cgIDqJpSaICMyVKYThQOJYKz90NGSSGBU+XaSUHQu4WYQQKT8UqpHpYMxfELOXTea6zP/U6vIWE+KuhylBs6Km9bTziW2tBQZyabqH1QRv3AymSDMrgN9QaNE/yB+7JV8bpA/4IA0xOBROCm+Qo/ylTBQwqiVEH4IaFCdlHq51ifS8zQBSThiowyFWlBr5svgigKAvdfuERgLHAGZKWUQsBGVBCBTRpsQhHplYbcqAeK00h3LNUWeIU/FkdTFXkjDR48oXKrLEqjmVE9ASYc6pyoLM6xUQMUK4NT0ceonei9H9dy2ZALEQogYIk+StuwDDG3YOlqibBiiggrvXqnFopfyQyNEAsRRTS0BAZFs4MNgg6ZLBzKA84RQquwQQ7I4CGY8VojbkPjg7dgo2ZgTxavCLKrjbrQKL1q6YU0SazyM+u/DnvXEFAsCdgqkBuNFooEhmghJ1E+XbSwGjt3wZovb3lwaWvvfQPPL8ksIstNKjM+EN9ejScq8RceQrO0rTc2FvvWlgfhQuy+ZtE1n93+19unXyLWx7as7dznJr486weI9rW+uyd75nS+OpEjmreW3PCdYorRVHLh9Iz7JmczFzDTjCKEvV1vH5n4nHMUu3Fx9/vECYkQppJrSsUNaP2QUTf4fGD2HyMRQprifPMNJfVfyVCVUxzyhsO9OdLhy3NPesEhV2EFmxYB6wetH+ROy/XJPRyBVP0CN3zrB4UM4CBYP2j9IFfCMdEPGiE86A437t8eJFQH64MK0i/CBvFC98ADF4FBx/FYGNAr5V1jO2FuPJXpZb/ORGmyTvwoTMF2PxlMYIMqqGk/kXn2CmxQWfjyz3cuQijCKb3YIDJKPa7VAgXeXhoVdGn9iU76kIvABwM+EqJZqsiV9rkSQr0hNyrMmctBoRffU0qRIAQqyGuKMEA3M0o9V1mOB7b1y36T54oMK/WVtpAlY7oo+idk8ZDQgaEJxAmZzoxUm/o2ZBEehOAEdsGuDEijHtCzyQBBNhgg+ByXpVokqPtEFseTjQqGIuyqDRgT+CIVX8JANmj4YBatSMcrA1IM1H61VvPHyE9oErUgh+MVk1SNgZKpLlUUVoagLj6qS4fA4VBEdUmDXKNtkacHcddAFLEMrqLeRF81ClRUUEdHa+VcrZIfUpRMV6Ms0p9f9/9IVbuO8RdPvPbBkaeZNXTlqGNQbBrmJKKo1z4JD4ZBvNBF0PBVP82brlzvv//CPa9qW/p/1993VfsJVy5ZzXfgbade8L4nbnvPGdc88Ng/UTvrEOJ2KD/CLKPPTT7NLKP3DHxlsvjCovS59w1+fHHm/Lu3/nqxdG+6pTSR+7vWlkKipbJ17GvplvKJC7/S3b42l390ZPydeODnpULp1pHxW53sZxltzaxMJZaNT/8ys4yyDmEqeQMjBHKlry7o+MdpTwgzqdcWindi76YwLXzHfcH9N9R90aotLE3hvrFuc1/cmugMaj8hKe8A0+Brd/sqt6Oo5uKuK89YcHZUY/LxgYD1g+E8uq+T30GYs6eLWMowKJxg/aD1g+qRuRisH7R+cLe7w5HdMUJ4KHhD7cTrQqTOM0AXBhBLZBBpq593VN554RBOOFXahbbYoEaHSo7mqhRUMzC9aDdDLb7jcVZO2J3USRn6J7mKprgVd5UyeJNAKiEUoYqwNWShV0vQa5OxzKREs7fceuCiXjjylz5STA96oGOPEsXAAxUzjGbhg10MRCD9Q7zjxsQPYYkIN/edpxlNVRv+AwMRuQocbE4DjBsMMBM/IQsPpFE6RG4DZQq8iyzs1RNQio80pFEneAv8RwaUgvzAu5TiED1blLyxSy5+SKlCTQq1kIsTPtJDeJBJo3QLDfY4B65XLTwdXoSADRj+10sPALIgxRVTAZG6Ab38qU+cgxyarbpoyV+8dGc08IhP7Pc0o8EyU3BP60ZwdGoSaThkT9VqbyHizR2gjxYKVbFTuRKJpToaxkBQTeoTWtvwxiAsESrIkNEQLWQQL8YntfWtTC/eODX81+e98QvrH0HDxjqEDBm9Y/MTyL/+7BcyyfIzG/99bcfSFwqbTmrv35J/7tzu8xdn+h4Z+7vLet8/Xdp+7fIP7Zh6fMvkHSs6r53O/yCTXLy0+9ptI/82k39KgcSu9j9i7cGhketbsx+GAeJ22r86OOBs7nFV+q1Y+op+Y+rInDI2u7oce9108XaGkLemVueKG/23xn3zOF7/RavfQiBs7q1CfSsdFcRIJ05u953yxayX3c0QPTeWoEJ2dXMlT37r1J1nXrr41SHLhOMGAesHw6nk0uejax6lvgn6Zkm5j1zrB7mlWz9o/aD1g+F+crQEe4fwoJFflOlNxSuM89ytpJ/7kaX/0Cs2CCfk0ZldF0WszwKqBSEY/Kmy7E5HWGLUIU9qlYqb67LsUwR91PFELZFDl4Osjkdm0pPyQaNc/EiW3qtrBjJT4+QhmqLng0ZPfnJCcZWK7tZrcxXukRsMXZZy1TCfUcv1T69gQHWYuE1PoghQEQ0oRdAHJa8jkrJBBUnFBhEwcNq6AZzQUcSqmyWVcCIpaDAyUGkgcghQC27QpJQVnyEN9AllMEAgC41YimR22XSLl0Eojl7+RRRVBUrYS7AJTvAgJ3IeiBDkBxs6UVIMROTkARv8ow82amqwUV2kbBAqcvFDGvikaoG/yQC47h96RjPB/vnmu1AKZGGOrN0lrb0EcjEglfFNC9fygS7iSueVXyIgk2gwU6vwRnX6qEY0ZIUho8jSqFUco84CSo4otBwzHWD9jcHXY6DFLRQUDcsbMgoXssdJ38cbg5RVzJBQIWYMLuX6+5Hlr+S4GBTKsXxh48PY/PozDzCpzL3b1985MbCsfQGaT5z2o6S/d8b7fnzFmzDry/Qtz5y6ILWoO7WoL30O3Kt2NfuJQFmHcMyNF61tLDk4m3tqNven7LP2YDxxo+R6fgyWyOjQtuyH0bRlLpC+I/uBHeOfy7l3CJli1LmfLTK7zC63KInmuSx/DsLvR8EtQmhVVNkg44QPlrsbExKsuHcF62wQgSChN6smYgyTqKzoOKnBle0eBwhYP2j9IJexbuOkfNRt6UYtWde59YPWD1o/eEz0g7tiVsdBF3XEDoHRm5oGhucevQGoUKHChjQDElhb8L2SSPm3iaJPUdAPbGoPaNEMfwA8tilbORr3yWOcuh9YoisbKYUf/EnhH/n04Od91RPKOg91oiZBFUWLSI6+fKiC6IMxZdV++VaRej3hb6hL7dLBSnYp/yNMz5Wq2fm/dZ91Xa3ZblfNcAUiGwiIItIniQFGeWADOYQTKlcCKaEtOIyfscYdWnTcKVSHJQrJwv/b+s4T5VCoivohJ/f7STJF8BRhU9dILs/gb1u09gvDj9FBqiBKeMgtLGvhDpNT4qpz86nOuvlF4W8cHVE48mCq5NLOm/vOpxS14xyBjbroX6lFu/S7fGghGjxISXWhGWjUNwcexW7Dz7Fqf2gnxWnPtpkRMTf5BCg+hAqZW+W83tU0ibfpwFbwBsBZmwEDUUQVRIZPYgDI0iBjdv/IMz+y7JX/5Zb44/KtcuANzca4YcgoBqHlHAXHSMoHJolefDI6vygkUDViyckKNlICaTQGyCFLDwNEDwkMeLphpdXYx577Iufr3vEBhoyu7liE5hOLT7ln24b3nnvlXQ/+R39bz2UdS6h9bduy58Y3PTax7pTsSU9OPp5qqT448vWJ0hPZRDE39r3+9NqvvviLxdI9a7p+9dRFP75tvH9w6paZ3KOV8nrOuxt67hejZ/lBTSQzm78nX19+cMfw9WphPHnj2PTnE4kbSm5Gmb9Y0PH78eKGbPK1RGwmi3cSIWyPtYwW7seYK42LSt81XXW1244c1VN3DwlfwbpSf/cyNNRlRl8UdBd04IT1xSdwe2HX1Uvb3Rha244/BKwfDOe03mcFhQTrB60frF0S1g9aP9hwd2jCXYsQHspJ8YTOPUCJDZKGydbkruQHZYlZIfMcqQ4j4X8+Zxcz/6zmhOgjGrJT1QlS6GYcMZOeh2f/+Ob3XOL81OOHeNZHuSpCWq/RE5F6Sde8ekXoqFdVyxg5FKzl1h/4FAFAGYqrLLvaECQHn/hW4FGlormSQxHtYqbG1D3VPKt4PXW2vmDNNjzU1ss6KiWugoAxG7viNsgIXucEDGSDHOKNTuMmNR1gqhuIIngBDr/2USmC4zmeSRI4Ypeolwx+cfk1sAuIisxYrgC9YpLYo1cDAmsSI8U/uXzcWfFROMzwzMl9Resy6oKeYaO6aENoCfZE59glS7WoGb95yo+ipDH4owp8PjS9lUZiI8KJQ3KxYboUUggSBfWhGTA9sAIKfUJr8fNfWx6gbZp1MzRVBgTlyIIrohekAVVIIEoWZiB1W9X5If380OO0XAdOM9RsmgSGDUNGKQS1IwVDSCwMNhpUhL+RG+KcmEWjheyyieMx7yvVRWOAGjJKPFBzzCBAAqGFTABLqvGlNy86/8f6zvvoqitXty8iHrhhcthdfLHYp568+5quxR/6wX/fP72dHwgen9nyvdEfLM+esD636ZzO87kmL+19/RsW/y+APb/n57tSJ1217BOXLP7n2dLW8dyL+dLObHJ1Z/YC/LB6BFPIMH8MC9P3934MDXJn+82sPUhUEPmUE7cj6H1C1iFkilHq57Og/fJ0cjWzjFLEPYFGNigiCt9MtDQBeXeLurGP/rkYYMOnnr/rr0KCgQ1ya3I3BBcVrIUEnQevZATsY1N3bp/esquwSccRAtYP6rvE10oCKZ/6180JkqX3Z976QTc2h9uv9YO1O8HLqR+8rPfdi1LnWj/YtJ2ARQgP5dRc1HHCvaMuGqPAYHiTUL54GOK5lickUmnCBKE8nyVbKrwzADNEJlfGQWh4XMNMrjAN3jAOj32hK1JFpDz0kctAU2mCZX233qa6w3qP5TzVvTkfwWEQQoxRAnoZqRTHC5lF9s+gaoVzI6lCRp0TUqpWEYyU5tRfN8JCeqgjP1S4Em6rtaRWxO0HkVwn1y29+QEkoojBULvQGFzTUUFs4DDsiuGIQCpFI2PMnL2nlDJGdrE+P7eNwoZoFD2DOGHMR+E1uVUWSvGlUJxcOUGjLRAqqlYVpGRRVjE69GqVmodeVTwxsgF52+wIIEG6kDkf8qBaahUEgrfZtVO1kNJaUqoIh8MuH0pRnFbJVXAiEvjrz31RGizVTg4HV8HJPzz/dWoJZBJZfkjlQSE+uKKrzgcVyaI6gqVEETUeSVUQKWVXFDE0Q0FFvTEoPsn8ojx8BKKIvZxAgJlgxqX1NwbFAHGlaCG0kE8tWjgzzJUMkvcNPsdF+7cvPVMpx785OnRdV//qzoV/t+MHn1z7JtYh5EgTY+X/seoNfK93FBx39ZdrbKI41Jc+d6K4vTu1pDXZPe6HjL4wdkul9DyTx0zm3AISxdKacnkrE8lkM39IQc0sOptfQ7UzuT8lKqhj5PvAQhTIrErPjDKp5PVTuecZMkqrZkq3V6qJwZn/V3CxxpSvXYVoeO17xJ/wDeI7676A9W2XVNeEv+7bXf+tSko0wV5kEj1HrVwEJMaLLsuc3pnuCn5MOJ4QsH5QXwGlXPAI1g/qfs6NPdzzrR/kW2/9YLj1WT8YoGg2wQjhQZ+Rk7tXrNvIb96tDSX1DMQDlusY6g9Gsin75SLUVSh4KDZILjRP5BABrqhc9Hq0EgmUTzTaTcQrcoiZGKNq0bOYGGCUB7pHM795ougWwxBJQxfpyepDUp3St75KGxzfgstR0Asyr3lzxTH0zsvuIdNZYSv/MlLVarb3oVznh7J6GCWFAeKm1hjnk4dNl+t9ez0BB29DQd8c5z4847odv+FBRZTW1e5vY9OjeV4W1UEUAXNsJMQS64XVsZGKOIVUTEwuVUreMECpDhKBDhKNnMiAlFyyQqXicuhlQJZqET1jV2RMSmwQ+JALxYJQITd4CDZqJDZqAK6itSCrnWoPfmCSpFHm1lBRcIVDiuOQbe4ho8N+yGjdBmM3ZHT4Gd7Kc0HC+kFRdfCj1sqMishaku0lkMiH6wxSB8fjQ1lSjYaF2rkW+E2rEYorQv+gjhoyGp2wVJZ7zi+KHwKJSrFRtBBqymV8YSujH93MLz97wllkrV100i3rH13bv+pnW6oDs2MMGVUE8t6dD90zcX86URwcejrdUnxwpMwUU+PFdW0sKlPc9O1tW0ulDfDA1uTSmfLzy3reu7B97QsDrjlMIZOLxUbHv4KsIaNeWA85zJW+smH7e0oll5X3KUNGvfy1fOnKfOnrNCybvI5vItHCrTP/RBZfK//N4vtVu4L9n/BVw4SDqfiV65285+a/1bt9mXRfkiV8T76CUgKloIjY8B0/vfPCjpQRwj2hPeY11g+GU2j9IPdtdT3hBi6NdskCK/UvCNYP6soBHOsHrR8Mt5FmEIwQHvRZOLFz2e+d/n7oCotQ0xPwrElae+Cai5CI2PA0qYen8HjFwxNKygZyCBsMnJAsZGwCRUSjtsIG4YHIFEQpP+xKiRAsZc8DdNj0xOb5kRiXaFytbTILLWTNxJrGCzSVY8EDewhk8T88a7Irhkbt7jnSPxEGzqZdbNRgiodGqnW72uiBkn9V4Tx70uirUKU1wowmutXbu0uH26AMgrJ31bjLvFEK3RsZQUZQDyfqRRYvAbo0wovYDR2kiJNKyQlZIVf0jF0EUpghKcXVVcD0XIgvFhM9E29kV1voYmWjX2SjHqBzsglNxYY2qDFUJCZJKoeYqT1qJynFyUKJjfyzKwHneAg2onBhyChFQmt1RJ849SbihwwZfeiJf1B1YoOhwVJSET55TVEtUXVqj/xw/THYlUGwm2YHoG28WEjK11DFNSUpQ0lhg4oWkhviitEJSykCmXzrwrVcUoH+4cS9MRiLMVhUDokWulAhc6j6a75Sii9tW7hlfJwLCzaIzXsf/2oiUb0mv+jB2e1runoZMvqqRaf/RNsNj088cXJHX1+m/4oTrt4588J9g5UVbRdtm7n/pM5rxnNLuPx6smcQElQt7LZnL0gnF+byLEy/uOTXHiSLNQkZRzox/Xnkvu73jU+vKZTWs5Z9IrFsdOoj+qGjUJKTllzp9mI1kU6sls8qIzedxIXv/vJl1FfAK1G5n2/Qwwk5tPBVxbB+l3CF2TDiuy+ZFB4oGSU/07AFe2UFe6q4d+RfXrvsrbK39HhCwPpBvhK6J3DBWz/ItW39ICBYP2j94LHbDxohPJQ++tx+9wB93uLTSD8U+4mJ/NRYbnLD2GYenZjS8EvDj0Wdqs+AIkGB9AjGsxfPUmJEtdx6AZFDR7qqsUAFYYbKd4yRGUojWSKBpBSkWwrcEns9pYmAhRQ93Vjt8c4/5IVdWoLet8f78r1dzdJXr6biiuc86qJgcOsarKdOWdZ7Sledd+ujEDW3IbypR1I1QM7lh1Ly7525iqKbKqo1w7moxTajNtHi0gcnIUuasBuKo9nvJn4SCI/sUYog1dhXS62DDEE2mQUWx64YTmA7CMpFL9KlFH2wEWdTrxOoZmgwZpRVA8QhZRNtKgfO63CiW4Ft4gEbYatWiZihxxKHtCQ0AyVyaKpqpyIEDRlVM6JkkiyNBd1zyKgarDaraow1ZBShwZXM0NNUonYaU8rFx65IoBaZ0PyiGjLKXDKKKColnMgHGknM0AX0/PkWCdQ6hEr1VmGUKFIp8clyKfGZHT8oFJJXt5/Alwzlb55y2bcHnr9p5YUPPPOf2J+fX35q90kbJjeQtSjtXt0kRDadWoCweeYBxnGuG/l7JpU5Z+Gf9necP5V/asPQm18cqjL11FSuyhc5EasOjd+KXw0ZHZ92oUIeN1l1kLUHp3N/hoArDDRkNJN8Qzq5LJN8fbIamyjdyb1lpuSq5lYAJnzYSPm+eJGkxu3c11jZ/obAXt2g9heNu6fUtygPlJqbg4SGIaN8i+UN/at7f7LuwP4ebwhYP8j1ry8X17x6Q3bDlwZZ/Ron3vpB6wf1w6v1g9YPNm1PYITwMJyarkwHnxXdS/B11YpLPhh7BxQR+YXxLcO58cHc6PaZ4S8SSKzwoOSf0ngQcz1J7cks2gL9Tq+ntzo928UM1etgz7oXJbzFWGzaBRJVClnMUA7DI2Don9DjgVRdlwQZk4r7KZVSDZBGHVtN3tXl1R40fa47KlVKLWhcWnvWjLGKMZrocelxE4twUFTKro6l/tTqG1J7tOUps/bCoTsG5SjLP7i6uqT0Qr2pNcvwJ5SVJrrrzon/SAhFgiD/e8vFDPZCGmVfKiueA5nBQE4QZIwBuciiOmJfeJBSuaQKr2ET4mYYKEanKkgDWUL+nm8JKZ+L68NK/+ylu9QnyZgXCxE4ZHxiJpaIE1E78UllNThhqGdsywOimniQjVx96MSrqSUQTpRsuIoeL7LIJI0RI8Umesjsio7qiJCho5KpS7hx+HzIAgTS2HBtSfqwyISr2E9CA/FD0LBS6CJBQqijJizVmFLihGI+c88vGotBEbl0+TC7z8nti05u61uSWbR1apQho4/tfNFV0xJ7fHgjQ0YfGH46kYj9ytOf7EjmU4kyQ0ZXZ0+a2LQ901IeKqw7t+v6ZW3nntl79caxO4dnH0vGqmOz9y7q+N32ZD+zhnZmr2RNQlalX9j9L+XSTuYXxXFX+828RkhUkGGi6eSHYIMsPxiL3ZArrue1QzjhbOlrqcQahoyWoJOxeFvq6p7sq0eL9/E98lcaPyZp/KZrpt844W4hCkimvqfRrwCaupn7G4/V30WmjKd/+NQ3y+di7DihSvGNRhAVrCurZy+4OOrQ5OMYAesHrR/U5a1btPWD1g9aP+g7x2OmHzRCOC8dNF0jfvUDqioQSySQuH70JTRM+BEebdnVU5h7vHLPV7UtKksVCJXe6+NxzXEwz4P0TIYHxxUdS3RkjI2ZwUUU3es+7gnP/eFREaXbrQcqVVz+nV194+HPFXBPl/JXy6BtLss78/kku1ouPy7X69yu54cYEeFk1+X4J0ecsIvon19rVVCID1WwX/dBU93RikXLniKuVt8wlQzGzjCCZL1xnjiys79N3oKVa0ztUJwQcqXHLBiELJUNenWQpBh8buhxGBoCaY2GzbrBoqE68TR2sVGPQq5miOGaQU8ppcpVIA6lGBqCDOQw9EmiXgo5UgW1YwDLwlhcS7tkUSQ4wUwesFHtmGmoJ8QvHEJwhYBSLQkHgpJj+e1Tb2LlBsjk9577ogxC/FOHTxVAJCoYHXqqn5bJklkYTKviUpL70LPMZ+OmWkX+xKk/+n+e/0KYg5QGELuDBGohimiokCxtS9rcq4N6YxCN4oQIaEgv7T2DdThwsnVyjGO5d+fzcKW/2fws1258Z5kho2t6Ftw/tf2jq6/YkR/640t/acvk1r/f8I+vW3LFd4ZvX91+Fpf9ROkF/ISNa4ONKUbzpa0QwqCPClTERsoIAahgd/t1g+Pr0fS035xMLBydXjOZ+wsOmQghrw5ixjditngXI731FcPSf1P898R5cq50saL3F6f7FqslPt8lZAUGicw+Bu7jv2i+oCviBF/S2e/Kcnp2vZnztm164wltJzrJtpcfAtYPcs6tH+Q+wZ2HlDszgJCya/2g9YNcDL4PsX6wWfoGI4RH7kxEf0C9etXFqljDTcUS141siD5DY+Ae4Oo3Uy/WbqziQkqlJxW7g+np/utomFdqWCnGGnoaqKDCiRBIzKgI6hgCjHKl4qS7vyDhvsN6OvSzzmiUaY3SOVe+0TTCMz0UboOeUQ2aQjkh51iJs7HbcDiuhXWWSJOcpS+LtsFSntG751CKiL76GpUICskYqW0hC0EHUjfQ31oaLYvKH9ZuBtpxKPstGISCQRMEDMkVYVPvyM+oypUSNiWWhSWCWwWxrZ+rQrlSokcZNKJwZMHQZACnQpbNniyRguJ4pNizNfA3ET/xN7mSJQVDpXM6wZXaQC6NVCm8uQhefcgoZJIskUlXd/1lSwSaIXaHvOdspc7Ub2K2DVFHcqLvKDIsh3f/NMWL5hcVCQzzi2o9QzmEKCJopLfmFw2LTzDFKCSQXFxhtnF66NvjO1/VuuyuyUGGjL5/2RlkLevuZsjo0rYFr+pYUuvfvF9O671Dd6bi1Q3TP9g0c+vSzJmbpr+0deZz60aKrS2FBenLt01uKJe/0Z66bmhqfaX81elcbNZfTHpjUJPKDI+/083Y5Anh4PgnCQ/CDFNJt0DIdO7PFSFMJxdjkE6+lrv5RPHuc/p+5e6tb9JF6L8v0DX2nBO+sNBFfQE964t+IXa1nS+Fs/Ybxvzl61X7LcntuFz07nvns2ScYAiEK+KUkvvTZ5/Ws9bpbDME6ghYP2j9INcC9wr1JtYP6ptB30eXjWz9oPWD9Zvlkf5rhPBII95QX0Pv+KHYj2u46aaxrW646ewoT6LMfuEesjyD0rOXdhtciQFKiazHPnYdRfTLgokKBtZHVlTGHrfih2ThIZMo58sJOQzGNEDPiBhgz64ao3CBM4u7yCHPoGicR5ghD4p+sKceP317PH/zBC/abF/CHWK08ezKkxv85jdVqmagcM+7vhF7RAUZF0c7XBk1EiE02DfNNaMhy/nyUZFgHAzUKqXea81bXZZLV500CKhC8VC1BOkbctU7YvDQzFb/HO/elMPPxe1LWTwwUCyn8fPQqE8VhWO4poJmpMqNUi/KQh3V5VAcITBJ7EXhUIq/scu1R3E5xBKZUhoRKoInJ6pazZABReSHXW2QNwxkE+qlIHLdpPZwIBswoaLaWFDfQQafwkel6EFpCWZEHR+qL3ex2zuKVfdr9AdXuCXpNVgUdocsBhjmodE8pWGq0u25YYUKVYumIb3AzS8au2zRaUwqc3LbwssXnsaQ0VjZnWknxKr3DT7fEq/+1eY7Usnytoe3pRKlTYVN56WWcc33phdn49dz2bTGl0OrTuu+eiz31FDu20s7b5rKndzfeV1X5qRNA9VlCz+WSva8sO3W5f1/OJvfODzu3h48YSFE8alpP3y0p/3mQT8B6cDYJ4Eo5ZedoAUDU7fQttnS7eUYI7OTsyXaU9v8JYStPigdr4Nh0hhd+eFC1S8j2g2EkO+CeCAlKaJUXzZSjJ1BjTHWZKd3k804unhKx0XtNsVo7VTYn30hYP2g9YO6PrircKOxfhA0wMH6QesH93XfPNx5RggPN6I/tD+6Rnxoxho5C/PWrB9zw015Syo8y7LrnvXcs1njhlJaGZBqQ9AznPghsroikUOUCFiiL1bcGhXIuOKFKE0GA2NEX/Plaw+Pjyhxxce7csG3dKKSKyWT8SpLZaAnCOm9udK151EeGz1fdSq31Q5ETZKKVNTI2epZ1qnc0TGIVFmypEaEgIWyxEJl4FJvIzNn7e0RcK6ifmxqfRofr8It7cFAkVIBK+tamXpZvAW9PPu0buWekl3r2Pd6GYdcl6esev4ujWih3srARsQJkiaeBrOSxhd0CbtkkcIDoV5ikujnDAaKmAV6phCcYn0UCVUgh+gcsliiKqJJr1x4BrIMGmrBmE0hR7JwKINAJtHw0VHA8fCjIaPyFh0yKhKIHg8iq2gYT4t/BRVxAkphOm/edfSVx6LzizZECxVIxAyiyOfR2S0qomjhz5947QNDTzP6lCGjvAx8cnsfF8C9O57nZLp1CLv775refm1P/+V9p3xn+Nmb+tbylblq6QVjxdE7Bytnd53FZbOyfc32mcoLMw8uyzpWOTD79Fjuvq7UybnSwGjukyf23JxOdqNPJXq4AJhLJplwu9pm62xQu5pRhiXsU8nFO8bexSKEs8WvL+5424tjblX61uQ1xeK3nhv9NDLXKvzQfwW44nHsLjz3v/71d7+V+F0ltLN2v4jMI0oWF784oW4Fzsx5qL1byG7I9R5EBfGMt+rp3ZdGajDREDgIBKwfBCzrB8ONiy6GzfpBqDJdnvWDXAzWD/rvxOFMjBAeTjTnz9duP6CuvBiKSF3EEhsCidEG6AbqGYZ/FuSx0PNGHmcxI5c/pHQ5NdbnC/Oyk2ibXIkcEgip0bZqC2xQ1BEDyCHFfcSvGuYORS+qicAAUZ4L2S3XI42dqSKWM+VENlFSA3LlpJqEfa3N/jE0KNG7LJ5q/RMr3jCjzaqXMsgYo3SNrNM2lWJXj73umLVFQoIUkQEF4zw840CH4xoCrXVHRyG349BzAgZoXClkT+94sIbmOn29DjTexv9xZcn1BeoWdUPn2Bv5g/MVqaAyXB2+Lo5BstKgJIooD/o9tYHCiVlBnGBuYoYUhESJwkGc4GPBBpkPWa5Kv4m/8Qvld4afOcmH8sjlQ1tZIP6709t/zK8UD9Uk7CaUgUtZD05vv6SdTmsZ/qkRNoWBfw1yGayJNv/p5ruJeaoiUUFq11ITkD00vG2oXGoUm+VAxA9JoXy8OclR68BFZTVkNLpehd51/PPNbnkMsR2GjEL5avOL+go0dlQUUekFbcte2XsGcUKOC67IB/+YfXd6G5duoZRgyCi/S1yxeM3q9r71k8OfXHv9Fzc9rNbSyO+OPsU6hDuKW/gd5LbBW7LxYma01B7PL8+e8cL0Fy/r+79MKvP9werC1jOWdJxfKm3dMPw7HcnVhdJXt46sYVoYooJbB/+gWFovh/nyVg0TJVV4kGuA30GYUQZI86WvZZLXy5IUTTZ5VTZ5crVwf7hU/LWni9Zdt8xiyvUJXdS3PpQly3+cwl3t/sJCIzNlISOQJRLIF8Qbu595pCH3tI4b1k//t7PHSe2yxco2Q+AwIGD9oO8q3NfZ+kFA4D7DXcb6QesHrR88DLdX5vg4LF7MyVFBgN5xzkDiaG5iJDf2xOgG3oBibgxRGe6bej6rs5FdTRbt0b7YIE9+KDUBTCBFCLWnwxZmyXfjQiGHaSKH5YSMVbbGITGm0xJT8vyNB0lY01SR6fFdE4qVBF0aQht/GfDmKCLmbsuVE+JdyDxWIvvnS93+XRH+o/Q2VYicuKizdKWdgf5oF239odZ1osgUZNZTHmHL1TgDXHVcOFFd3tg556GZil0uz9C+qTQDzxygKB8HKAE/6CniDMQSkX17KKiyoUn4pxStFIMkV4C4holbYuo2qnTHolyvcfv+oNhrzJKKXDLUPSiwFlgijAsbcSd5IxXFkg00DApHDdjA6Bgyyq7422cHn4Dg0dQPr7j6TzffdVJbnzwgwAA9CawxSWVJj42Cb6SwKT5ofqzvPLyRUgqzJa292/1Ci+Tykr3cQgIhojBJ+CQaBQPVfrJko11IIKwPA5S+wUvZDUQR0ijqiDGkVI1haGiUBFJQg0VJ+QQS+OLsgL4+DBaFKBItBB+ioEwqQ3VbJsa3To9yutZ0LvzChodbEtXf3XQvUfR1L734CyuuufnU139/+KlbXvrPi7uu5OI5veuM6RJ8+8FV7TdR3Uhu83RpW1upP18aZ7e/422tyYUD5fWsQ5hJLpzJ/VlX+3XFUm3twbbMBYwb4rRWqjf0d79v+8gbk4kbWIcwmfwgU48yxSiccMvY17PJ1xeKd8wU7ypUE73ZK7hmWOiFy1IfdxHWyRmXJRcctJCqGzaX5a4fdx3rUney13BxIvhrsmbgJiCNBA9liXLjzJehtewuSp/vizRUYruGwGFGwPpBvpjWD4arituO9YPN1g/y1HRap/WD4SJtUsEIYZOemENrVvQH1KtWXiInmrfGLZPI7KajG8KK245/+Ec2x2b22MipES1/e+WZ0jGZeipzvuTEAJFFAiGQksUMxb6CY8eXeBR1/IaaXXCBUtgQLVRBeKCqoAjkEKN8JZFuqeSrCexbE2VIo7zJgxvF6mmhPJP6XFdBOl5hnlU+PcniZCk5WmL5NxfPxD8ChTAlgEMIMIRD9cgLJaVV9bGyLWU/Mo4yOBc9RghLaNT4Kyh6kklB8UOcowGr0FrHFD0v5QHdif4BvZ7r+CrGar97WndP/85HMEAlrNT+4BkDGam6Wm5d6ZtNS+TGsUQKQpDUSCgfGe5EeJoH+8JeRA5SRNafbL5bJaGCfMT04FTI0sPrggEC5A1SR1mYnlKUwQb5d097668++yWy8CADLGWAhpb8zqnOgBcCH3z2S/Ic+CpkDCpIyoeK0AcSyHGJ5XIIyIocasgoNsQGSSGE9w8988gsAdUYbwySiv7pZUIao+o0ZFRxQnDT/KIrW2tEkSGjfDZOD2+YHL59ZLDK/L3l+Gu7+9d0L+LtwTXdvS3xytr+k9aNrp/IT+JweZZ1aErCf8P09/ozK8cKG6eKm4ZzKwuljbnShoHJL5TK3yyUr+XqLJW/NjIVK5Vuy6ZuGBj/ZKV0Kx6SyRtn8o9C/AgPwgOn809xshJ+CfuJ3KO8CphKXD9d+vpJi/7r6cEfLVaTrEqfSb3muYnfz1dTyFxs8MNiLJGvJvmm4NBfaaTucHHl/taYntPo+uH6VLgPTS0G6Gmhhom2+MGiMpAG2fujON8WVxY/CJcueldf6wrn1zZD4MgiYP2gv7VbP+guO+sHm6Ef1A3A+sEjeyM86NpcV+5iPba9nBAI89YQSBzIjfGUv4sluguCq8JxNi4LPeo5ncfHkxaJbqQoOhEVBP+sWTMXtyELPWZkOcmPxuSvY0R+xBqPpFhqF33Y0KAlVz6DZ3HC7lTBhRDrEQ+90Oj8tFTJSrg3GGO9ycJQMT1cSqGEE46VUovS+Wk3NtWte1GoJAqR1yBbE6XZchIKSlP5RJutNtAAeCOphNBOdiXzhmSxnPCvWaKqgUDVMG05JOWgZIyBZJ7L0eNEcNVzXXlMaYZqR9bRkdZc1NEOZRF8FbU65AElHur1Ot2utoVjqJ+L0AaqgJ5B7RTE05hPzCGEgd0FA7mBUEHGRCbRKBKIDbKKKw0GspGBPCgLM8KPqldOyMVMuZctPE38EKWmwwkvMRJOJMip9+8VMwzNhiJiDy0kFW90p2cXkO69QRggOhFFAonED6GLRAjftOyV/731ARacYJ7SC1uXwUhfkV1eKccZQLt+dOSKxadsnRhf1tmzdXaEc72sq5uxtalUOR4vJ5OVTYX12WSxK5lf3XpyOl4aL61bljmjPVEiNLey/ZVjuW93pk7qTi2dyt27oPXVbcnFA+PvWtLzj7Rtcvrzq5Z8enTiDgRatWLJp3eO/NvE9C/H3buFa6Zyf170UytxZiGEfNKJ16WSa6ZKG8cK92aSrxkuPHB27x89OPTrOccJoYhxmGGpklSEEIrINUCKZzZ90YIceGCN+/kbATZcSN7SpZSEE7pUuQ5JNwy1bkDs0f3gQJGzu3/xksVvl/MmT91IAPf1CF+vJm+vNe/wIGD9oPWD9Sup9kxi/aD1gy/nftAihPUbwsvpLz+gcrhzDjcNgUSNlxMqnld4McJe/GOiYyA8DfM8Je4hyhFYIrmYUZJc3WrrdMtp4GbwKBEkFXSmPohYZTwanNDXWaM6xBJL7nIdK6QZHiNvmhPVVe2IX4nA4FQpNVNKDBZTvckixn3pPJpMvDxRSuHcsccKIRa3hdlx0NMSFm/EJ674iIyhlCWpjkLhRGQ+mKEnxUyhVI6FZqCkeC1G6p4yeWp3G0P4yGWXsjpY3gTjMdo/hToN7ijMzUhy8E9Z59SzQWxwQpar2z2Uuyw+QOQVTuM39uvV1NmPBqC62p2xc+mcuRa4o/C7rjif701DkFo+N/g4LUF2zaxvooIwNCk0WBQKB38ThUMmK3A/jdIkjRpALMX65EcRRZTyIJ+MTYVkvnn5pV/e8qAcMniVxipOWG+OI5xomIFGGohfw3oVIoFECCVgtmlmgKOOkkBeKYQEakypGgzFZRWKR57/AvbARR/pXoMcalnV2rdhYpj16KvF+NapsfWTI+unh+6a3pFIVDLDpSu6+wH7hqWXX7XiEoaMPjD4YCZZAvCLF75q++wFE8UdrfFyV2pJR2rRWC62qHVte3IhhDAcC0KptFNYw7gYF1oury+Xx6vlrT0dfzSTu6cje8FULtaV/cVkYtlMadvw7N+kk68bKX6rK3Hqzvz9xWp2NP9QqZreOrNutpIu+CChixBW3FesHutztUlGcCzOKdyGTFPZRQi7MEMvu8uyP33OcPEJ5bLLdehSL1CqzgNdSJ3vWE/q4tVdr3R+bTMEmhUB6wc5M9YPcs+zfpArwfpBQHiZ94NGCJu1szri7TrAYTaBJAVaopbyNOk17mnSURq36wTlOorlvmy7qAv2sCbMMOBLyGO3k3nC9CWkd678I6vL8JbyppRpaYh10J/hArMx6KJnjOTCDCc98RsupikKX9I0p2Q57leJk8ohxZnVRg5FVt1bkX7iHJgSbqNcizZTl1pOSqmozHM2cRiUjsFGWqtDQ0OeK+M3d1yOm/Hk7zgeu3LuXlzcnV07cAioMn8PbBYnHlOB7yryqOLBaXY7Bc6x6qr9cZXXRS+4xnh40YY8LzgznbtwBln0AqXSzw0+gZn4G0ooHKleL4RKQe0CYwxBRQzmHDIanASDn17zuu8+8Rl22cQGcSIiCo2kPX60Zz/DYBg+KjPNQAOBhBzu+fIhpBFCyFhZGUMaV7b28wEuWgsD5INPOKFYosx4ddBHC938onrn0Omrsfumdtz95FeKxcQ1HYs5/2s6FrrBokQIu7vvH36W2WhUfDQ/inJHfuvS7JLHR789XdrEKqA7y9/tz1wwkquUy/csyH6kJ7ty23h1dOpXp1zMrTo6/k6uBuLbm7cs4UcLfl1gKOnIEDTxKwUGfSZvzOUf5WKYyP1lb/vvgX8m+bpitYWchdnzN0z/R66SJiTo3iGsxKcqWb4aJfdJ+CsKc3fKKS5BjWSXRkrvTrm7nv2wT3/16j1D2bjmFR93kcDAA0PkMJTyzvkSUWRBetUiGy8qlC09phCwftD6QS5YbpR6FrF+0PpBroeXST/on/zcU65thsABIRCG2bBMIgtgUIY1MGAjKsyVxNVUu4fWlT7P3WB1nQUDKJBoDM/BIlGO8xzAxpeTKng2dbTKs7LQADozXIWUXDFYaUS6HEPzD6/k4kfNwABLspxQPwS1RWWhjuxqHh2q46OCpHxEMqN6jNXCQBTxIw6sZiuXsoKFI6nj4w5Kx+hyvVY/YaIXtrtypfIjbD3OzolcUVaN9CaOu/JKpGOQ/jAdAr7Hk73T1hsgWU6C3JDLbkN1UXty3YhQHY+Gmw49oaAfWSKBzqA+plQRRQ0ZlT4QRXaVG8KJwQwbcpliTkNGw7SoMEPigZBA+KHaCQlUhBAaGcadYgAagQSGoTLwQ4iiYoYPDD9NQV3hgHlR6zKGjF6+6LQt4+PL2hdEh4y+UNjZkqg8WXgxFS+3JoutycJZbSsGyk8xZHRZdim/UEwUX+hMFE9se2VPum/r1Bf7Wy9f1Hrm9rFPsmhENrmQ2WJOXHhbuTQ8O/P57q73lQpPlfP3xJPMWfORiaE/KJXXJzJXtnfesGnb6ZXEjVzhE6VvFFhysJJKJq8uVOODhYfbEpcMFB7vTF3YlVz1g6mvM1KUD23mJKRbysmWMidIP1iAiTZ9C9DX4of+/UDxQK4FLhiyFAaMDCKV3ocH3fWD4F7NdU78Vw0NF/Zrl3+pN7uiXk+z/9W3ovZNa/bGWvuaAgHrB/nlNNrfcRPgY/0gd8J6v+cuVNed1ffpwj5r/aD1g01xA5ujEfSDFiGcAxdT7QOB6DCbq1dejOWuZRJHX2KX1QWIxrh7oH9MVBSLPUeyPH/jKVxPjY6YeZIoG3bpUSjILdXbS3CexIK82mfVC2KvR14FAOFsYoNQL+oiC2vZIDgCpl0ceg8uxbffIIHupb7dY5i7WuP14mbMo0OAEbdyruKk6h3DLjZqm59GxuUyKpXccNSh50CQpcqKKHrm5qpAcLm1ZjroRGI5Uperp3yfjQm77uMPX7lUh0AWTrymFo10SjSOcPojYb++qQHKUnX1ymu11wz8afH+XZNkI3vSB6fcpDXahblh8Ccv3o3m0vYlFCHWBx9jl7leSBVXDCQQYc8ho4pDykYsERlySMqQUaKFXHikYVZV2GDDehWuumFXHS8WQhGV8nIgTrTIhCKB0D9WpxBRfNhRytqQUWaUWZJd+O2dz+Ph09ufes8JZ/3N1meYTobxotf29CUI51Vjb1m4FnDOW7h6tDg0mB9Ylezvz/ad3LHm6fFvndR+yVDuPn9aceC28dxT+dLXC+UryuWd0LAcc8aUtnK1FApPVfL3ECTU3Tnpx04nEwQkoWfuEm3PXlEprRkvbMzEW4aLD6zq/MiO/CODhceK1dSO/PeZiklskCCh/zq48CMfd9m4C6G2cQFzdtg5o+P6Z6e/EuWBKD0PrAUMMePSkT2sj0OgRbQEPUzSXVUuJHjJZPEBWS5pe8cxxAZd820zBA4SAesHrR+0ftD6weOsHzRCeJD9gJnPhUB0mM3Vqy7+cOzHx/NTGL4wtnV4dpz1AAjLMAVInTXUyYlnjO4h0scFAzNxT6mOqu3aeJiF44WHWYSa7D2IeomhUQbmw0e7IYtd/KMXj2IXh4y4C9UgkKVSNImnZxpALZhRigdfp1FIxLvyjZNZlbF5qlfFSSmCRkJ9ztIYrJXBqDSpzuXcQWJJv4Kx9KquBhRv97lmuFwXZgytdbEMp1ct5Gqjulq7hU+d6dEU99nVbGcutEmRlaWKwoEri93gXA2gCm0uy8uyqfnHPBJfVVbUQLObRoeS4g0DqvullVdBGnlv8MFn/lNVRIeMolGQUAFDOKT78dVvzE3KX5zA8TRklKlKWZKeeCAMUMwTfghdVKqAIaE/GVD284NP3Nx3XogWoiFgqJlIz+1do0Xteany5oVrGTJa9bPdMtfoa7v6r1y6muXf4YrxVHlp+4KdhUEB9uj4kwTlWsvFkdLmTdPfmyg9zoKEXYmZ2fEH+jIXl0rfWtL3UYaMrsvdM5W7J93C+M8bFy54R6Hw4ujgK6F8ifLXE8k3xAufyg1sqJS+zkU9M/GVofFqInnjRPFrscQps8UNnLeJ0n2LW9/FeelIXjxZWCce2JNauTn3LNcYHzARCeTseE5YO53s1q4rR+qYFab2A4enhe6Xfk8I3cUhS+0GPQLFndITQsymi/czqS/KrtSrzuq9yZ8WSwyBlxcC1g9yvrkdcX9AIFUPJcH6QesHrR9s8huiEcImP0HHavO695i3RixxPDe5vh5IDIP6dJD+0dOJjoL4HqVOPXjkdrrANGRJyhNpxY8RVVboivx7gLA4V4iHVPTMx4hMzySK6J05DuYs6puylLrnXR5wPYekIuiKp2qOdTjzGrlyck2Drt7/qdkMnpFjV9xXjamzr6XuUdq73fXmIbluoQuvx4pcLXGBzJw0TuEpInq5ci30IPgJS13LvKVapcqdMy/5Znuf8owrz29r2bLWsQAXNs6bj5qqvFJ18DRABirl2luvx7Vtjs2Z7yXLWSvupxDir3g2KFmU78cWuUULa4e3RziR4no7ESYZ8zPQoNGS9EQLAwkkWsj1RhpGjbLLJ0xDSvOIFq5q7YcHQgKZfRTOyfyimoaUI7y43a1XcXnHCSe1LnxPto8VJm557lE09+58njjhg/ktqWS5LVX8jVPeduXySzhxdw/eTW5fejnrLz05XlrTfnGhvKU7tYQho+O5VeO554kBEiFcsejLnL7B4evz+Rer5ZFk8ob2zveVZ9fwwihDRpPd72kZX1MtPp9pu3lB20UMGWXamGRy2Wxh01jhvlTyyk3Tn2fgqJ9QNNmbPndr7ulV7Wc/MnEnF4zmMeIiYRTrCZmzBgtP9qXPGSg8yRkS8UPg+mEdeZYV5SQik3ru18gDXa6njjWDOg/E2NNCX9x/ZU5ovXxBdiUHbpshYAhYP2j9YORbYP2g9YORy6H5RCOEzXdOjt8W0TvyWdHN+mwxAok6UGKJUZZIINHdNesgSHC0h81zJccV4Wme2yA7TuSok0tlzBMwtKrG03y0ijyUcgml4cOuuA1KAncUZbgpGmRRLKwVx0PDI2/JDdarJuIonRtfn6uOIv4x2gu+AvIbmI920at5DORzRVwQxg3/0ywy3mutIAeCJ1ro2u7948Efr2OYrnn1qXGcT5/lAPEf5cJd9bIiSg1Y9YTN1a8DFxqOxjkc2BM64OQ2lDpA8oEGjTsEjy4GrlLMHfAuywtquStdK+ut6rso5diZH9QmoqiUgtFdkUANGR2YHYM6Kpyo5RPZpcUNJFAvFrpIdSwGUSSlOKn05/Wuhh8uae39yy13MmoUvbY3LX/lw8994VWLTv/O0LPnL1xNYHBpW+9vPHv/NRMj35wY5LeKu6Z3XtvT/z/6z3HDR5PlJ0efX9Nz4nB+ZElmyVjlxTUdZ4JRb2rFRHH7QO6Wc3s+wCL1A7OfSba8ezr3GJfW6PTt1fL6VPLG8YnPtVTWc47zs7cnyhsq5W8kMj/fkuiultybumzAzeITqbKLVmeSq9OVOCNF25OvXNnxqofG/o7ZRGGDzCKzrO1kzhe/gDh7T9jO7HhDb3rxSGHdotSKkcITnAxOE6njcnycvBsPrE8ZWtO768NbSvCluJh1Ddey5PCs3j87uecatdZSQ8AQmBMB6weBxfrBOa+NOZXRjg+D6K71g9YPznnNHJrSPai5uINthkDTIABF5LrcFBlu+sXhx2gdlyl6pWFXAk/A0kOBggZBz6n1LOgQKvYcgeHp1vMlx4f0Kjx6/ZyJBpnUESG/uX1P//xfl4WgghpNqnplTaXsIquUiijFIbn+C1fzHHK93rVKX0fKSxCzjeSqIJlOqNXrm+pimN6dq7petyN9viF4pg+ODj1VWUo4+8jxhvaThV71SUlxnOAqlPUVukQtrKW+RrWw0cBneXt3FrAPBhJqENXrjeaqMS6lVB1JB7TfLmlb+t2Z2suH7rXA2WHpoXmaXSaQQMaRYhBN//ylu2TMaFIEcv9iy52/dcqPsg4hc6xhSeSQYa6lMjPbxi9japlSnHUI79m6YU1XbyxRXdbRsz0/JELo0nh5Z2krcbmB0lPMhdsWL7TH80uzZxXLD/SlX9GbXjme+4elbe9OxarT+b9a2vk7/Cw3Nf3LCzr+qKW8FXKYzVxZzd9LfDtevqOaeW8l/9d5rtDE9aOFW6eribFKNp64bmfhwVTi1bOVBMOVT2y79L6Rf1yYOmdr3hHCK3vfefvwv+vSpQ0d8fyK1tOHCk/oAEELRoesJSUauJ/0Ioq96Ysmit/l4nG7EQLJ2Xds0H2cE1LJ7akrLlnyu9lktyo6hlL9YGH94DF0yl4OTbV+kLNs/SA32IbN+kHrBxsuicOySz9oEcLDgqQ5OZwIzDnMZsvk1vc/+vs7ZjpO6xphFbyHZzujVcIHeQLmRskwPPe06zoSNxTT8yund3dVnlt5snVkyfGQcJ8VdxLDwV5DRt37V45LOWeB/IQxpZquMxhjRmBHY0qpxjXDczPV5VrjN+oXu/IcRs2s58kgQrQCo3Otdt0imytCXb49tebD0JwOsldndGQotumP3bWEYv7jiuhYSP1DvCPD+qXWP97XnItLqD7vpFYXf1w40TsJNjooQUoqfaCCNAwNTnyNuKxD4M6C3911Htz50bGQUatSxxx2Xb5U7pBdeW9XqTC2tvrA9HZCyH/84t1kPTi9TR4wurR9KbODclpEAlnPkJXl4XiwRBFF3MADoxQRzStalzFLDctsaKUNCCHKPzjjzV/a9NBbV7zilvWPssu2rGPBvYPPbZwZfCC/5dXdi5ll9ML2JSd3LCLr3K7znpspntO1VpPK9KZPKJSX96SXdKf6yuUre7Lnt6cWvZj/y/bMmZWy465u2piWGBOKxhOLq8nVHEsLQ0Y7bs6V1gNiNbGmq/tf8lO35PL3LM5eMVRKb8k9wVIT3anzvzX8z+Vqckv+mf7UWS/ln3tq8hFeVQVz8GStC66Q4cIT9TljROFcFvi4tMXNDqrd2poTXsncobwTmGjhdUlniTc0NBJLUjmveZCfWHVl503HIhvkcGwzBJoQAesHOSnWD+rK9L2dujvfA3LzdSrrB60fPJy3LiOEhxNN8zV/CHRnOttT+el8b2cqd373xve1PV+p9LfHOoeKVz49sXW2Wn2+kL1/xrFE34Vwv1Qs0P/xXNE9v4oHehLF7dQ919YZCzJF4C36eOMafYJTQbHEBpGdZz8ZjEgOGj4F/5IhOYwpxZWyeHrmaZpd0STqYrBfw4alNM7Gt6fBAL0v6Mw4NFlrfhq12Xt3hWmkq9GvQk7q6vVNpYhqqdk72uyy/McX8a2qt9mh5GSO1ONGq2iDDoEyEc+umGue75qiS4aoGeFAVLvSoJRALTpA58ofXK2iup0/XuU42NmUivM4Y85mrT7XFEz80biGyccDUxBFt30PxuhPHytVEC3kw2BR2RBFROB9QjSkerFQuZ849SbeSOTNQ4ahDs6O3Tu146TBTXdM7Lx9bBAi6l5vTcRevfiUNbkFL+QG3rJoLZanL1j1wPADg/lB1bt5+rnR4oszpU3Fyv3p+M3F0tbp4rd2TFUy/mrsaV9bKo/PzNzIghPV8lcyyRvyM59PlL/BwSTafieeXlkBIc4gE7fkHi1VY8QAF7WesX7qu92pC7NuQtH4pQve/ezkw1vzz27JPwe935xz86CygUa6pZSOl5jexu16audT90OAwoAAUhMirwhyxvmBg3li0n6BRJVVSkEOKuoKfXvqNQuzl6+ywaJgYZshMJ8IWD8Iuq6T8yBbP2j9oPWDh/F+Y4TwMIJpruYXAV7zg8nMllOsvj1d6WyNVZZmdi7t7Dm3e3iq+LXniqk1uaWbcn0nJRme975sctG60fUvzA48OrNVzYIh0JHQlUSZCUqxCffQ7R6h3dO3NG6/Thr9Y7Cjdtx/ea5GgARCEZl6GzPPxFzqeBRFPCfRS4lU6EZpwhs8K/MP0/itsSDs8SMCgxJX/r8jY+IzrjE1ClqzUqtcRbsKuoNil2b7JosJeXtPXCniSzlntNz7dIeGEz71XKcJ9UpJ6vFxe8oiEOffz3QatTAIxJc0i4n0dVRr/tUyV2z3rX6YNbOQSXWSdRh1jGqnxh+pz3dmDnN/4P7kueOoweuVPqbpbR0b98zWhROndtBgUh4pAAFYiBaGRgQSSOSQOXJFFHlzg0UI8bS0vZf0b9e+4bEdLy7r7Nk6O7JtejSejJ3cvujFmR0v5XY8OrnuxeLzrYlSV3I2M1XKxMvtCYJ1nLMSk8okWWGiUFze+aPcfAfGbxuZuKNa3onf9vaby/k1tCGduaA0485pPndPoVotlr9ail9fKK4vJ9ZMlb5ZinXes/0945XW2XJmppJekDpnJL9zc/75Cm8PZk57Kfecx8Kd1xTxvZZKKsY16gZ2eprn9KDH8brUxwaJ+XpO6N6e9YLf9XLdFS1XQVdWHhyUZPsNNnhm30/U9+yvIWAIzCMC1g8KXO4/1g/WrjPrB60fPBy3HCOEhwNF8zH/CPDU35bMZ+KVmXJqtpoox3IdiclYrCNW+DSV0zcwvM3FQxKFeHLimhPPTqdXXrXyErWLRYTHchMbxl5i94nRDV8aekz6Ghtx/Yrz4biF8+Seep0Bj+SOW0ChHOXQT3GoPcFzZsyjLd7irV0ZskS6IIfOmyeQPKk7bzygxysU4bWuQiXhn8idgevSfK6ry1XtOIvTOAdQUBrgtASjZMcufMY1yRs4I99IHOKq5sErlUB4/GG6amgcZkw3gqVyFfZUWefQexD7cgfuDwFlzVotrJNVPFBELfHGzsrX5ZCrUTjffsrrKGpK1e1Tyvha3IFHN/lBE47IN09VulRZyq49Frijc/WSCUTem2O/4WCVpZpokTNz5WtV6/z++851yXgLM52CTIgoelfu0C5bdFq17Cq5tnPxwMz432x9+v0nnv7pHT9IpSqZVJEho8lkdWVbP/PT3D1UvKDnnIny9lf0Xr68fcWtL338rJ4bX5ostiX7O1ILZ/PXFEo7Cfplk9fLuQ4/nlxWyt8Ti13gjjq5mjSdOXNmuhpLrulIX5CoxlKF15VL373shL/7/ug9T0x+g/cGV3dctKLt1AfH7yFO+kKO6WfifqV4oKhACFnrguuNg/VMDx5Ye+WPnyG80u96WWNKndLvItCwugxEumJcqYYzRXjQ2KBOoqWGwHwjwF3L+kFA5mZu/aD1g9YPHt4bjhHCw4uneZtHBJLxcilenakkduR7x7Kto6W+dMtULH5NrHR/SzxXqCZ5wubpv8AsH+5pdtdWXx5qKSpY4odiPwFFRGbempHc2EBujAlIvjj0WJ3M+II+lojkHs29gkTUglF5dYX76ww8N+OZ21NIp3Gk0XM2HqnhgSUfoUHPLnwsyhsziTLsET9kyb9SjJ1zF7zyDj3D8R58hm9V4KiYkYUXnNSy/R80tMll1UOF7j0wzNzUqeKHbhfSyEJ0qpdj8S1xByVAIHfuWOS41lTHe13JyFYzrmlqUMhIp0NppIQTa159Ixuy2KXZzsZzPDERZHc4LCXi3ht0+Dh4vBfV5fZBw7fTWdYBQU2Wz3U+/TBTV70zJnDqdM6YiWQ5Xu1yXgjr+Vhiy7/tdKs1/MEL32JembvGB4rFxJ2TA7yoh4druxa/esmaHYVBIocPjT95Ts8puF+WXTJcGHh29varF7+xPdW1KL1i48R/xir3LO/8bV60e2n4a+PTt6ViFcZkzsz4kHLp1lmigqWv0IzihEtbSu6wpnJ/RpCwMPun/W1fbYv3liZa8tXEzplnnpu+tT99Qa7SMpgf6Ez2ETMnPOte/GOaXMoSl45VIYT8RJKMQWH99Vejhe7idGi4q8UJNSrof4YILwrqQvI00iEjvZN232CDrz7xb3bX2Z4hYAjMIwLWD3Jj5Narjb/WD1o/aP3gYbnjGCE8LDCakyOBwILkTFcmz4PvZCk7UWnrbCnPVthu701fNVq8bazYP1PJsCAb8y6K9uyjTVBEcs9bfFqwEUscY5lEH0hcN7I+LEAXaF4wRqj3R17nCUkgioFZiZm4QKKzctEq/8exEj5+j3lNXeiPIiqFgZihRpzC3NCEpZxccV81LAafLXHeWPTjV8nwm6/RsR2cUMo/+juZTBVxxX30DraDQEquUnahE7QWFuTKOh5LLNK1HRv+wJ7kluMh7Ign59cdmLNxbr1Pv+sOypE3WXgz78HhFtH5jL0noThCrbjc+qp9Rc6Z6nes1bXRPSuoCv5SyjeM1PN5dxLc8buPq9ctNOJUgQRy+IlyOBfYuFw3cShXmkMDTSJeLbJOSaqczyf/YfBJ6vj2pm3pFNHpcmcm//wL/9qWLHSm8qe3rYCPfXfoy/2Ti6eKm1a1X7Zz6o6h6YeIF6daKueveIlXB1/cdnpn283l8s5CLMZKg9XSlaXy1mzbdbPTt+dL61mHMJW5YHT8J8vxG7dO3jFTaclV4/zwMVTYma8mx3LPsAJhvpzYOrudLwUN4xg5ZM4Rc8lk4sWORK41XnABwzoOoFHb9WFAqCDGDgVnUBN0rgVdNMuhVd+ApCN1pY0UreNhfw2BI4eA9YNg7W537s7v7/DWD/q7t8PCD/9xN3PrB60fPPh7khHCg8fMShwNBDpSXfQBC9L5sVKKxfHGy63xRL4Ym+rp+KOZ/D2zjAyMz+SrHcyxUai4UOEhbPVAol8mcWVtmURiiVoAA4dMO3lLfYU69/hc4z/+3lsjR7uq9XTF7dbZhSNgKHnghrC5W7d/TPd3cLdTI1QQEH9f10SmKuse8f2rbtj4W33Nlc91Ddm11ZibqxR+gt7xIE+BXHjMcz+q9h2HjsBX5vtXjMVd4YfISqkXGVNIBSvd0RJHjRwv8j5cs2G0rjp/XM6U3ojEZfuNssihlUHYLRsDD2ZNOdcfCnpkHAK0SSTbh8UcyYvUAbX21NA1wiHOnm9UrcXsOhLoMh3WFOUHZkggKfpwFhAYV0lDoH8cXamUgBb6BqgZTAcKEC2FYjKZKgEw3BuI+Oyc6syVkmsXbX9ybODEttQ3tny/Lf7E6d0vbhzduaaj9/ZNn8qVWi/vr47nptpTsUzqxlJ5Z7m0lSMqlXbO5u9JJtbk8k9N5P6UJQfdGNFYbIawXzzWk11bKQ4OF17IV1Knd78mV44/PHHXkvTpfanl/AKyIbdRtZPSElqeiZcYL+rDg2G8KOdVsjtHnC+ODqH20aOVM9DmLGti5A+48WGFiXP6PtpjC9BHkDHREDgCCFg/yN2YmzM3J38Pd/dkNJEexp8E6wcdM7R+0PrBg7snGSE8OLzM+ugiAJXiNcKRYutEubUtXhyvpr4z+LcnJHILk0mCNjwrwwZJi6XRWGblYWkqLDEEEq9edfGHYz8ehpsOz45DEamFtc55dnaPz7WeSUzEPW4HXuQyXW6NaDnJ7/KXJ3L6ttqDeL1rC4wKt+gUJPR3eEq4QYZwGrFEnuxd7b6bRPKu3A65LvTnzB1h412yfJn341wzqdHX6lgcnp0l4a86d8WV8+l7Ewq6BrTABt27c77rZcwhxV2lNEwaVeFq8u13HAOW6GibCz+iY8Oh/7srqWlci9y2p4H0pO4JoJ5K6Wb4ROnJoWr11bgq1DikMpP5uEN1jZIH54R6eIaotCQSTJBDuA7a7DQ0FRuRQLWE0aEQRYhfIsF1V2slWWicgVdk0kVOBv55oQUjigtwYoDZZLErPdmVzHOhdqVHlmW3t8Xz7fEyk4vCtTZuuwrL7vSWfGINrZot3sqcMflKbEHbBQu7rh0tbB3M/c2JPTcv6rp23XDrbP47L+S+m6u6a7szeem9A//yUu7ppZlzi5XYw5N3F6uJRKzl0u5XPzZ5F6eyFh6M51tbCjWa6hrvKiWrhobngdSLgTAhVcsR2ByO9c3h6WQ3nSunuy316nP7PmJssA6P/TUEjjQC1g/qBsUty/pB6wetHzxcNyAjhIcLSfNzJBAgHjVTTreXUtOVzFi5tRK//uw2HuJjE8U7iZP48KBbqrvgH2DnqUHR4aZQRGqBJbKI8DjDTUf9vDUjG1jE3NXuCIvbwuM1lMORv5baxKRiKSiwqTErT5/cU7somX8Ud7u7+4G/YcAiBDzYu8QbwFIwQ09KLTWO55uAkmcI3hlL+ngX9k4NB/REkSyYD6XS8fJMKdWWLE4V0x3p/GyJAJULG5I6tz4SKOc0yJPPWjthU76B/ihclptZNdij4OhcqM2Nj3Xsy/dh9TIquc/UFXEHVWPajq8Kk5oPh2qgpr4xyvBmaipA+TY4FVstqunCm65J/sGCOiCBKfdWJzE5huO6jbAhKT6JRnoFCwaWS8VEKlUql1OuYawRCA2LV8Atmyys7M61J4ttiXx7ophuKY/muidK6QU9uRdzJ5/d8+gPxk7hnnt5/5a7dqzlur1x+ZY7Nj/ZnZruTp/zjQ1b1i4YLg1+8r6Rr7xx+feWdv7C+vGnhgupsUrbwuxPtiaXjhR20ICnp75yZd/HciOJkqO7rnoGpvJnsLCFA0nG3FwyrfFieyKHnsPEQG8JivtBC9G4j79OEKI8UAdI6kmgw5zTKEpfjsXbk5ef1PWWUxZcE8xMMAQMgSOPgPWDwtz6QesHrR88jPcfI4SHEUxzNe8I8CRLEGaWQaHVltbExcOFrzN6raX09clybz7GmwQ83MIEHC86whuLCPNZ0e2Hm/pAImMCacMLY1uHc+MDs27eGlhindi41nEr1260qTyms+uYiWgYB+w3p/fzqilX7xaS5/mgs2fDG/+AAH7jKJN/i8DnuATEwAXqUPfolC1+DlJ4ILlUXPDDJvMVR4pmy0lmLwDMTKIEx2bthMliuiudz5VSWDqWCJmgeT7GSHvom32lLtTmd111uGUXlgVlKpbciFM0rtW+yTpYnEjPLrLLdk2bc6vluHp1vP5ko3VtIfXoQeAc4fEOqrxcp5w663OZvgb+wHbILPsYID7llhaKO8HxkKFDXHJcWtjLv6s6FssXUi63ApdWkC2WLyU7IWGVxMBUW3fPjp1TCxckZieL2TZGk3r2RcGu1Myi9OREcdl5C17oyYyOF5ef2L6zNTW2ZfrEr0z0nNA+dkrP+suSTKM7+dTYbfzAMTV2ZzLZN1b+wuaJk4iBd2fHYonE3z772bV9PXhj7cEWN7eOa8+2wrOcI5Yc5L1B3h48IXPBZPEBz/d2cT8Py67dvVBB5w1cOToop/tUYx3Jy5a3XXZG7xtak93Ua5shYAgcRQS4u1k/yA3Z+kEuQt9zuYce7tskLrV+0PrBQ7o9GSE8JNis0NFAgMd3ZmXMJkrM2JGKl4aLTyxIp7rTJ+2c6s7GJ6fKXUykkXafkl4wOxpt3FVnd9bPW3PCrnlrXCAxNzWe3xVI3DQzyCLpu8r4J3vu647NeIZGFlzCGXiC418sdNmBmbgnfh/j4vHdExg38E9kqM78XGmc+N7CyQ0briiLUv8dZ0BTgQc6lgg1ghvQy7jJUeMVWCIcillnOAssnsFSezOlZHeqMA7q/pVFbChFF+X6JwQicEn3Yh7+RZwQ1IFhht4NyIy7Yat07ehdLv99YxBodPQoJCv1VeiYHFXGLhyjL+iYqvPmWLFzy+avCqdl3404JVjmQHbHSy5W3tixO0ggPqG4HI5ayy4ZLugKCSy5MFs6XcxVUow7pf2U5fnMVYGPSqK/faItlevq3Nmemjxr4YgfKcoATvdKZ0dqusBlnNwxWliwYXJFd3ryo+svefvCnZf2r/vAifGFbTt25nsH853l9MgoAfBqy0QpuyA9NZJflE5PtSZzs6XWZKWyrHvr9vx2WHo8Fs/AWGGn5Uw2mYeN8t5geyLfGZ9lWXkCgxwaB6s5Qj0pdY1k80qJtZSjA2z3cUfKYdNc9wsL6endHz6z9w1tRgV3A8x2DIGjg4D1g9YPcuVZP2j94GG/ARkhPOyQmsP5QoCHVAY0+mfZapY1vhOzi5Jj1dL6jHtfrsjTa5mXqdxAyvr7ePPVkEP3C0vkEwKJctQw3LQ2b41YChYInrd4yXEYr/Op2/NUjz8+UgePgR/w6M9DAy+MkSKrFlcgsmGJhk8tO5KFChhBG3oAHXSkzPEjp2xNFgt+UCVxQiatyVXc23Uz5SRzokIgFdV07K6F1xfjikbiR+NRi2UfIXTv/rnKHFdhTY6ko0k6KCZu4W09F6/T24+ufrJ2tQxRDXbcxh9XPXU2fjyqxqy6Q3M+vZkc4AcNBd1BYe08eytn5Eq5dra4iB8EjyzVi+BGujooVLwCDWNC0ULB3zmJhbo5daocIISWQCu/UyQTji+WSqmpYlt/ZuTZkdPypfTFi57ZPLHykr6Be3aetigz25Ecv2vnaRcs2JpITq5O5/PVlpFCz0szvcnUDGHAiWJra2EBbwYOzvTunOnKZsepdP3Iclp+8oKtpHduPvu8vm3tmUnGrGZSeY4mHZ/mYLQSfWc8x7eDQWWOtvoPRRTzdDD5jcOB4Xt03bhh4OB8OaVb4dOxwVI10Zm6+KzuH+lvXdPXuqJezv4aAobAUUaAr6f1g/7u7e/dtXu59YM1imj9oPWDh3yHMkJ4yNBZwSONAIELnll5YoYwMCiuNzHTHc8v6n7fiZW37Rj7qYn4LIPlqrF24lc8HB9D257DTXfNW5MbH5wdZbjpF9zsphy9J3ueybhX2hyj8fSGDM92PGOC3sCXACoBV+HpwbECH0V09MZH/BxLdM68ucaXqqSnPZRymb4gf+E83p8TiArCLgqMTPTEm5gkBh2p/Hghy6Q1BNZ4cW6qlGZAqWcgbroaCKTCiRBIxxiZxsUPiBXjctOEuhlr3DmFDeI2wduMBNzgXxXa0QI/dEfhR2a6F/yY0tPTM+eFNtabjQ2l2BDcH22e5knjDWosL1weHAsunBNH7TxWek/Sp/hwVNBfTQ5EZ+mijCXW9qvGcvkUpcik5eyix5jDGZxtZ8bbhQun6o2ILWkdy6bG13RvYmWUu6c6Tylk1vbFb5/snqm2vK515/9Y9UAmNQqvPv8Et5hEsdCzuHN7LJGH4LZlJte0DU8UOtNJJtfNdCQLIMxZv+LEpzWEtTPNovUuJkzrGCzKKYAN8tWACoaVA925doe420Z73aHVUwSuFtqA2/bUJad3OR7Y33bibmVsxxAwBJoAAesHOQm6XXNns34QNFwX5q9M3805CcErfGL9oL9grB/cdUnsRTJCuBdgTN18CMBGxvLZ7uxse6LQFZ9dlH5VvHp7MvdNWsrMjYydY8goT8aKdTRf8w+iRdF5a1QsLJO4YWwzmidGN3xp6DGX5cmb2Ah9gKd/7tW4WndQj4k5xlJTuRfHZEZZ/4KdI0Xa9BdvEB0XJKx3M1iyy/t4zo+3VvcDR8LtNCvwaTQpC6mXU4k4oUA316VrnQs6uYlhqZVXE+nFqYKUDwXdmMyWEqtZ4Bzjsi9CTJKNBrDMA3Wp2QyO9CzRhRMJJEILC/lUOlNkcpekX2TEt8QVrB+mqwVPPr7naByhv5BLFbVch1ic8KZim66838gFIo6CFuIQOipayABXNDG/296azxUSmSQH60btQoA5xnSiNFFIz5aZ9TO2pGMH0bblnduhaiwcP17oXty2+d3Ln3MR0sTETf0vdaRyU5XkU2Mn97eN4P+lmYXLOrcPzfaOFVpXLtg8WehgQprZSmJR+8hkvmtp5zAIzFSIQbrYL8gwNJoxojQP8kwVnBvGiy5ITLHURP04OEx3Ftg8qO4oEPQIxdekPijUxQM7khc8c53ZAABJd0lEQVSf2f2mvtY1i40HCjJLDYGmRMD6QZYLtn7Q+kHrBw/7/ckI4WGH1BzOFwKzpUy1lMi2VFoThY54rhUSGGuZLrzop3NkNCBPvf6tJz/4bb4acfT8RpdJvGrlJVBE2qJlEkdyYwM5N29NjSX6RkIVRGPYCwL0zBMjH/aCHngmBoHExpG0GnPwlJJCsvLeRDKYZkaxKXiFN3aL8/l8qJtjHexhAJtiNQbRKpcScGuJMSEN3I9xlcQPMaBSpbTTT0ITzyZLDCtN+7GvjrYwcNHzMfz7AB0LRTglKz2QwgbRszp8rXYcEl/0RM6xSF5ldLvuxT+HA5QpXuWNRxy2p4qTxVRfJrd1tg0qOFth/h5aEmtNlmZLyfZ0cbqQyiQqaiQHJCZJLVBBgpzTxURHplAuuqNzS00wTSvvtaZKTMPj53SprOoc52jTifzGsRVnLdjw/ZGToLtrF1a3zixa0Lpt28wCAqrDhT7go+msJIFnqLJ42yQvACY4rlaWl5gptSLnC1mGnnIAo/l21rFIpXKxShI9wUDoHyFxhsfgwfHDljJTm2pXmJDC/XROYe58aK1GhDo2WI13pS5uT558avdr+lpPak91hVImGAKGQNMiYP0gXaFeu7B+0PpB6wcP453KCOFhBNNczS8C7hGcl7X8jCZM1DGUf+Tk1lzOR2Z4ns6X2xn2xttWhSpzkM5vS5rHO11jWCaRVu0ZSHxxZuCRma2QIseLSOAFnhMKITQ6Fv447uR3EMiFMCgLNgKzEncklOR2fQY2nlw6Jul5pvMFCyOCFuZxwT9xtlwxybuCRHchYExkWvVTyECT3Et3icpMMdWeKrDQBT/4ef7iKBzUBbrFHEK8nge1w5ihp12p4lAuQxb1858G4JxdyJ6axNBTF83zefgiPoZ/QmcdqWKumOpP59ZPdXXEK/lyYrKQoW1LWqc3THaf2DnxwkR3G+s4VOJMhepYYskNbEVgns9ssjxbTPKGpIsWcjix6thUa7Xcki8k4wk3zNUHIWlHPFdhls8Eiz3CDPPlLPwN9nVi5yDHFY/Prlrw3FBh4el938dzodrS370JhEdKHe3tw9VEfrbYnm0dLyWKuVI21To9XmqtJgrFUqa7fZw2MEiso3WKI51Gk5rFYUc835mYZfV5DQ3lnLDqIB9/Zlyis0RKKVLHAGspLwdetKr1VYvbTj+569xgb4IhYAgcEwhYP7jnabJ+0PpBrgrrB/f8ahyUxgjhQcFlxkcTgQyBEeIwiVJrPM8z//L0FmY/SVdvZVUE3uJyj+iVRLGSzFXKpaPZzKNcd0MgUa0hkDjHMBsfFYRW8YQBO3JxNBGJGr1ypMsRSJSeXiDR6+x2eC7TT61ZJ5PO2AWjnCWEkhRqJd7YnnZ0BQ0hLaqD4BHHm53JZjL5qUKGYiWivL4lsC+EIuu/t7ggIUVoGhFFN+g0Xl3ZMfHCZNdJHRMDs+29mfxQLgsddaM0/QBRV7s7EBiac5eOV6YKqa5EeaqUxJ6GLc3ODBa7T26bfGxiQU/ShRln8hnSoVwrKdRxqphamJ0dzrW6WGKBgKXDYaqQ5m1G+G6hlOxpz41OtrZlCrSHkCmEkwAdKxBCF1OJ0uL2UYYuJ5jtMz01Xui8ddup/DzxrpPH14+tOmXBxkeGzsSyr3X0yZGVi7JTC7Ljmyf7M0nYcHHr5IJFbRO5cno417aie2Cy0D5VyO6YaT+3f0uxmGpLz4BANl6CBHYmcrBBguQaM0MDUy0lOLNH3p8wD0ItGOjD5jD5rtRFS9teuaT19JO7jQdyqm0zBI5JBKwfPJDTZv2g9YM8egCC9YMH8n2RjRHCA8fKLI8yAm6NtVSxPZXjAb1YTc62/MhI+c51hTf3JkoL4rfkq6mZSjpfSeYrrLG+O285yg0/+tU39I4KJNKsTWNbGW46mHPz1nxp+DF4GmTND751VNDt+kGlXu+Owg8LjZA/f2QQNv/X8UZv4wycIxiSj/URx0NfYB3CeLVYchPSkEsKvU+1zroiGEIOZzOpNBN4ppLJUpnYWxxKV4HnwAtxzK6fT6j64lQnZV+Y7qTg9Ewb6WXtS54tvggtJEaXb0nAJN0YVT+tTQFW6esihQRunu1IEEWGQOI4Vn10bCHFmSWV9MT2yZemOyGBDFslnkZ4cDKfIZZY8sHJbpjnVGt3tjicZ90HJhp1E+S46CVB63iZVwfxwAqEpN8bWEF62eIXx2a7VnZt70pSXWy2koIH8hohWUTqcpXkwuwULeGdQGYlTaXykMYFbZNET1Mtha5sdbacJrY3XUphz9aFMaG+aozYI7HxrsQsnLCtpeDm+/ExQCi9PwtuIK4PBjLFq5skhph5d+oVq9ouPbXrlTZJjMC01BA4phGwfvCQT5/1g9YPWj+4j6+PEcJ9gGNZzYUAD9+sQNiWKDDUkFk6BvPfZRm3szr6J4ubeHSerqRZF44ho7yolnMExLZ9IUDXSHZ0uOkHY+9oCCRquKm8QJ98qKw2Y00ggeQ6xhVJCQxiqqgjwyw1qtMzFngaS9S7d97crg/9EZ1rI0KYy3S2zcaTblBma6bglxsuMacolnjwA4DdYFcoH940HhW9q9MToftntrcwwNO7hQ36JrlBrqmWaluivDBFvA4eVdlazA4WU0Mjfbh6abady2mVDzZ2p4ktx5kcFfuR2VZkyF6+Eu9OF2aKye7WwtB00r3B3xKbnMmk4tWJ2bR7exCO6ttDdSxJ35ooggOXH7tuayn3tE5MFtsu7HuRvelKJpudyPECZHaKHyyH8p3fGVq6om16ccdYLhYvQv9YbSLfmkkVhmY6B2Zbz+7bDq+DInZkkkwx35op8oogze5I+sl1EzPZOAtflMNvn+41SKKvMccDaUMpluhMvuKcnhsW2yQx/mxYYggcNwhYP3gYT6X1g9YPHsbL6Vh3ZYTwWD+DL6P2T5YzEL/xQlt3cna6mpkt9axsfUWG96lSK18sZsfLbVBBqAjL3xEbeRnhcvgOteEHVDnWvDXDuXF2142sZwEMiJmyIiyx1ggyNF4XTgW7cysIw1I8LRQJxI4AHf8cV4RTEY1jsYps3oXdkq4oQz2JItbduflmGPaJMXTIad2y6S7+qEUsGFxKQA8lQ0Yhmqw40pYsLcrkULYnS4wC7U3n10921Sr1VS/tmNo23d6VzU1Mdo4VMhRhWCYkkCoggT2QwBIkMDc700rTCSNO5dKQvJGZLK8+trpJZZILumcmculMquTGsnooCHVinIy7GVPX9Awxzcx0mUZkOtMzwzM9xXJyaefQ8Ex3V2Z2eIbwZqwnO9Wd4kXC6kwxs2O6syuTg9rOFNNooHZcw4VSihU7eIWyLTPDGNFMS9G9N5jIdyenWViCICHIEyQnAAgnDCFB2GBn6oK13Tcublt9gk0WWruG7I8hcFwhYP3gfJ9O6wetH5zva6w5/dcfvJqzddYqQyCCQG/lnCdmRtriZXrETfm+ltglD0zc9/rl7y8Vh4cmb5kotzr24R6neVEtUszEHw4BescQSLx65cXR4abDs+NPjGzA/WcHnxA1csAz/tO/yOZPAm8MOlIHd4IcwuVEJUUOSaE9ZPFBzheTSRamr8ThV/BDSmEM+ZQBY0fd7DJuBKij+yoCkcOYGgkDQttoA/PK8LpgrpwcK6WYNbSUa+1IlrrShW0zbQtaZ8szbbOlFGbDs214GPQjTtuZ/cWNAi1iz/VDRZN5RwIHJpne063xwGDX9kxptpBkCGs2zVSoDBZFSWtdtBM/Li5djU8Wson41GS+lfGuJ3SMjrA8CsywmGlP5ScLbdA2WoWH1lQeob99igGfk8X0UD7L5DTdrdOd/oXG3vapRR2TvJfYliKD8bXl1niRqDgTijJStD2eZ6kJeKDYoAsGVhM96QtPbLvUXg7kXNhmCBz3CFg/eFROsfWD1g8elQvvSFZqhPBIom11/VAIFIrJ8ZnsC5WW0WJmpKPt7Pb45lLHf2377Lb80y2xpbyXBWFwY/Xc4toLfqiarPA+EaBrJF8s8epVFyN/OPbj47mp8dzk+lFm+ok9MbTxXwbWoYdQ8XYgdEtTnviX7nbRRZZ2YFbPNIsKMo0no30dm3fMkILikAjQLaUuqujWTeQF0dqGHYEywo/UU/LvAcIeqQuSWWL+FcZOUtyteh+fKCV5J3Akl82VE7lZNxKVS4XPgtbc6GwWdkeckqAingen2vlFgWlLC6V4R4a5SZPZNC/9cVFRVSxX4O3GOGSRtjB4ldY6lguxKyeKVOvie25NxXQqny+nulqncwxyzk5Dg/NM8hmv5phOlFlwC628TzhZZPQpnmN9rTNMRTPD8hL5TGcmN1vIzBbSqxYOZFvc5DHYwAOZpUZUkJYz+tQTwkRP6oLVTBLTdtoamySmfknYX0PguEfA+sEmOcXWD1o/2CSX4uFqhnu4Yu6Iw+XO/BgC84cAlOPTj3zjgsUrv7ztgfa22Y+d+977dt75/YnH+9LLCUk9Onk35GFB/Oz3rP6p5Z3L5q8Z5vkAEeB8YblpdNvw7MTA9Ni2mZHP7Px+rayneeJUyaQbNQrfcv99dJHzqCxSaBhLVogoKpfpSRkv6maOcRzNv6zomSeeuZ3V7mXOWYxoIfQMewrKJ6yvK1MY55XFTGGape0TlXyJ5SbcYoowRmagIRKYKyZYV9BNgdMSKxQT0NRSyRFa0rYsc94kMq3FUrmltbVIIJr4pOOTfoIciCiTjnrWGmPNielCtjMzy5uBcNTeVhc5JOhHihlBQgTWXXTMkFcHiQ06bploS+eQIYEdycKC1DTBwAXJGT+baBlWSTCQJVW6UxesaLv0tO5LbQX5A7wO92vGTxTYWD+4X6DMoBkQsH6wGc7CgbfB+kHrBw/8ajmKlvSD1hEeRfyt6kNEYPPYdhZEP7FrKeWfGvkB6Zm9Z22b3tKZ6upMuxfGbGtaBOgdx2an1o9soYWPDW7aOD10z9QO31qonAsPwu0VJCRe6G5PnuZB5+BdxPRIoYHM6UKWzOBycDbMsFGKIxigJ4FuLlP3dmKi4tYSZABpKQmZZE1CnChuCRtsYzH6fKqNeGAhBcGDCkLPiAQ6blltSSfL0D+ihxBCJheFIqbSTC4aa0lUWYSQmCesFerIC5AUwVWetVHilY50rlRJwg+Zb4YaNfwVf2yuSVXiiW69CoKLNI+GwQYnZ9q626a7MzNdydnOZL4tXoANtsVzkEBmS+pMXriyHR54ib0cKBgPY2r94GEE01wdMQSsHzxiUB/2iqwftH7wsF9UP6RDI4Q/JIBW3BAwBA4DArv1jkMb/9EHEqF39BlKqQPZ1QTTq1cYcom5QckgkG7wqP+wJ0IIcXOjQ/3YTmRIIDSSFOLHK4uZVJlRoFC+Ygn65wKODAGlilSywqqDvO8HW4OCFiGBxA+LCTghlTtC2BJj5k/cZtJwOUcaaQye3eIQUEU/oJRdlNjQEngjM+UQ/ZsuZFhIEyfsQhTbUoXR6fbObJ4JRXlZkalKe9PTEMLehKOCpWoylbjkXCYLbVu9xCaJqZ/3w/7XCOFhh9QcGgKGwMEiYP2g9YMHe80cRnsjhIcRTHNlCBgChw0BukZ81Yabzowy3PRfB91LiY74+Q3e5kKC/CFC6GODqAN7ZO4ZEUgXRYSReQMsCcRhRijPpYwUdVN6uvcA2fVEsQWKyGuNjgSSsvJhiflt3IBSCmDmZpRhQKkbMhpnpCsvFjrW5+uCBCITCVQzXFzRRSZdhBCemU0xbw2vF7qpaOCWmXQhV0i3pgvMN8MoDeKEncnCosxUT3JmQXJ6eeactuTJ/a3nrem+oCNlEW8Qnd/NCOH84mveDQFD4JAQsH7Q+sFDunAOpZARwkNBzcoYAobAUUGA3pF5a54f3QKBe3x4w2eHnqAZIn5o4Iek0DCGYjKRTNAQM8SGjyeT3si3XixRbJB4IGySWKI8kIUsisgEM3hgFGg+zyLyFRYkhAFSl3v1kWih0oSb11aTjromSa8Jcnx40HFF3wY8s4o9gUfM8NmWLPC24cLUTF9m8qTsqad1XtTfesZJXef6BlpyhBAwQniEgLZqDAFD4IdGwPrBHxpCczAHAkYI5wDFVIaAIXBMIPDxJ3+EFf9ilcRz0/29ieJq1gwsZv97qgvqBYeD27kInptaBoYYY2oZDgq6KKIIGYPyJZgypuqIGWFAch2l9OFERwWJ9kEsSwwxdW8P8tZfCCS6mB5LYnjLVIoho25YqSONnvJ55skcpLWBqUwJSI0stMgI1bZ0IV9O+rlkiul4ifUV+9LTZ3cuP7PzwhUdpxoPPFpXnRHCo4W81WsIGAI/JALWD/6QAFpxIUA/mDQsDAFDwBA4FhGAenUlpzvi+bFqamFy+nVdjy1q/aWPppaNT/zFS/GPPrTzS/HUmvvHd7xYzOYqiWy1CLVjHlHSWeZ9YdlDN3kMw0cTzCXD4TNAlOGexAOhhXEWmiePVwGTbm2JZLrM1DRpPzyV4B4EkiGjnvK5VwEhh9g4fuhjg2WWoKi2tGUK8EBKZFLFjI8iLmidqcTi2WSO4pl4+cLOEy5aePa5Cy6yyUKPxWvP2mwIGAKGQDMgYP1gM5yF46MNRgiPj/NoR2EIvPwQqMZmytmOeDHjVm8v5KuJ9sr66uynU8n4ivJvnbV08sXSvX3tXQ9MrRkqdGRbqj+78tcGpifheetG1n9h6HHmBXUszr9eyHIUBAxZdRCaVykliOa5iUz9OM8a30u4BQ/hcqBM4JFRo7G4n0VGBNLPJUNckeIsKkhBzPDPS4MED3vSOVYdXJTJjRfTazuXvHbxlau7VyxpP/Hld8LsiA0BQ8AQMAQOKwLWDx5WOF/OzowQvpzPvh27IXAMI8C0nClHBWeLjOesxreXO7bNphYnXru85c5v5N/z6vK/9qZeWpyYZNLOsVJba7y4vKPjrEVncsBXr7z4Q7GfmMj7eWvGto7kxgZzo9tnh7809DhDP1nxj1GgxAkJ+bnV6t27hYQL3chSl7LUBENDmYTGDwrlhUCM3RykfuBoJlFe3D7JkhID051L2ie3z3S2JYu5cnJt+9I3Lrn8lAXLl3UsP4YRt6YbAoaAIWAINBMCR7EfdMNnEu6FC+sHm+mKOPS2GCE8dOyspCFgCBxFBNzknImZVKzcxoLtsdhoueOs1lVLMkuzudsX+bXsC9Vsb3xqZXJ6Z0vPwuSMGyEa2boyHeydt/i0oPtg7B2wxLHc5PrRl1A+Mbxx4/Tgd6Z2wAnp+TQrjJtOhiGjWueCxQzRx6sMEC3nM73Z2YlCJl9O8KIgrPKE2OlvWX7GGb0rV3Yt78p0hlpMMAQMAUPAEDAEDgsC1g8eFhjNCQgYIbTLwBAwBI5JBPhllLlemJ0lQVQv1pKrphbFnu8prc/FYpck/l9ncnK6kmZ5iGwizzt77ZgR19vfBkvks6J7CYZXr7pY5szqtmlk2/DsBLuPDW38J79MYtmHDXnnkFcNIYFMMXpGatWq7v7lHT2n9ZzUk+la0b10f7VZviFgCBgChoAhcOgIWD946NhZyd0RMEK4Ox62ZwgYAscIAswCmo6XkzGidW45+VwlNVO8PVdNjpXjX5x6w5tb71mW2ZmpJlItM+mW8pSbFOYQt+5sx/lLT1Xha1a/4pdjNyNrhSgpCSEq3niIFVgxQ8AQMAQMAUPg4BGwfvDgMbMScyNghHBuXExrCBgCTY4AHWE2nk+2sKBElZTBnB3Z9+XKGzMtd1/ScQJzv9D+YqylI85q9NXWllIs0XMYjwiWeBi9mStDwBAwBAwBQ+BgEbB+8GARM/u9IeBW37LNEDAEDIFjDgEm83wp38+QTVperCbhhETqehLVBenXnJZ8dllmIJa4JhFrScUqbtmJeCl7yCHCYw4aa7AhYAgYAobAywAB6wdfBif5CB2iRQiPENBWjSFgCBxGBKaKE0z0kiLuxzoQsXgyVi5VE18b/eKZ6dm+ePmvR07/hZ7+ZZk7yZ2tVpniZbbiFhu0zRAwBAwBQ8AQOD4QsH7w+DiPTXIUFiFskhNhzTAEDIGDQwCa15ucYsEJBosS/OOH0os733Z6xzvb4pUPLnsba9YTISxVy9PVdLGaYAqY3eYYPbiqzNoQMAQMAUPAEGg6BKwfbLpTcsw2yAjhMXvqrOGGwMsegZcKC4sxFp1IMnCUmWP6EpXWFl4XrHRUt3YmpoGnFGvJxBJQwWS86BaVt80QMAQMAUPAEDiOELB+8Dg6mUfzUGzI6NFE3+o2BAyBQ0YgX012xHNT5Va3GGBLpS1eSBf/KlZOs3p8rPBPJLHyneVqZbCcYTQp40sPuSIraAgYAoaAIWAINCEC1g824Uk5RptkhPAYPXHWbEPg5Y4Ac4eyCGFLjDUnYowahRZurrxxJcHC6l07K6+Jle/uSL9qktGk8dZidXCmmrIhoy/3K8aO3xAwBAyB4wsB6wePr/N5NI/GCOHRRN/qNgQMgUNGIB5jPYkKrw7yGiGcMBMvtrh3CauzlURHotLRMkuEcLoYyxdXVmNLjA0eMs5W0BAwBAwBQ6A5EbB+sDnPy7HYKiOEx+JZszYbAoaAm0WGxejT5QyzjLbHCywvsTD2n62V9lYGi8a+LoDa4sWBcgrGyByjyUS3oWYIGAKGgCFgCBw3CFg/eNycyqN+IDapzFE/BdYAQ8AQOBQEytVEPF4pxBIsKZFuqS5OTLS0/lEs/Z6hUvah6eV4zMWvHiylJyvZQiUx7t4ttM0QMAQMAUPAEDh+ELB+8Pg5l0f7SIwQHu0zYPUbAobAISHAe4PFanys1B6PMSC0JV9tfX78j2eKmyqxxNLsOZPl9oni1yZbXjdbSrXEqlPlzCFVYoUMAUPAEDAEDIEmRcD6wSY9Mcdgs4wQHoMnzZpsCBgCvCzYEpsstY2XGSJaScfzs0wl2vrRTcV4qmWwUL5zorrz2UJ24+yTflGKGNTRMDMEDAFDwBAwBI4nBKwfPJ7O5tE9FntIOrr4W+2GgCFwKAhMFCZmyumZSmqilCU+OFHOzjCCtLIlmVr59YmzPr790k2ltsHY9YnExbmKW6WwZITwUGC2MoaAIWAIGAJNioD1g016Yo7NZtmkMsfmebNWGwIvewQI/U2V0tPlzILUzGSZFekTT00+eGJmtC259vL2s7eVnijEks/MbCpVOoCqErN1CF/2V4wBYAgYAobA8YWA9YPH1/k8mkdjhPBoom91GwKGwKEhUK3GxovZ2XI6V052JBMzlXSlmji986eSLS9uy3251LJusNS1o7BhothBeNC44KGBbKUMAUPAEDAEmhYB6web9tQciw0zQngsnjVrsyFgCDCPTIz1JFal1ryYXw8c7Yn8t8f/rTuZmy0uyleSnZnJnuQ5K1In3T/5bbc6IW9a2GYIGAKGgCFgCBxPCFg/eDydzaN6LPYO4VGF3yo3BAyBQ0KgK92dbim9/YQbn5zZMjTbfmb6ddnKBTcveX974pzzeq788RU/y7uF0MUdha3MQcpnRWb1IdVjhQwBQ8AQMAQMgWZEwPrBZjwrx2ybLEJ4zJ46a7gh8DJGoDPdeWXv5c9OP3Jx9+LJXGu5HJ8tph8b3jBeyE4WhmeKz0EIZ8qbXtX96s2F9cw6sySz9GWMlh26IWAIGAKGwPGGgPWDx9sZParH44ZRVRmGbJshYAgYAscaAk+Pfn84N14qJVZ3rxjNTZzbf/q9W+8/qWvliZ3LfnPdJxIt5Q+c9ksvTW3msM7sPetYOzhr7xFCoMW/ZGr94BGC26oxBAyBw4qA9YOHFc6XqTP6QSOEL9Nzb4dtCBzfCEwWJjlAfkA9vg/Tju6HR8D6wR8eQ/NgCBgCTYiA9YNNeFKas0lGCJvzvFirDAFDwBAwBI4QAkYIjxDQVo0hYAgYAoZAUyJAP2iTyjTlmbFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgwBQ8AQMAQMAUPAEDAEDAFDwBBoSgSMEDblabFGGQKGgCFgCBgChoAhYAgYAoaAITD/CBghnH+MrQZDwBAwBAwBQ8AQMAQMAUPAEDAEmhIBI4RNeVqsUYaAIWAIGAKGgCFgCBgChoAhYAjMPwJGCOcfY6vBEDAEDAFDwBAwBAwBQ8AQMAQMgaZEwAhhU54Wa5QhYAgYAoaAIWAIGAKGgCFgCBgC84+AEcL5x9hqMAQMAUPAEDAEDAFDwBAwBAwBQ6ApETBC2JSnxRplCBgChoAhYAgYAoaAIWAIGAKGwPwjYIRw/jG2GgyBI4vASSed9Md//MeqMyqj+cxnPrN27dqOjo5zzjnnr//6ryuVypxmR7K9e2sSbXib345kY5qhrg0bNrS0tDzxxBPN0BhrgyFgCBgCL3MEot1oVAaWvfVfDWZHEsC9NYk2vDy71CMJ/jFdlxHCY/r0WeMNgYNAAJb4vve97y1vecsXvvAF0l/6pV/6zd/8zYMoPw+mTdikeThKc2kIGAKGgCFwvCHQhP1XEzbpeDvrx+/xJI/fQ7MjMwQMgV0IVKtV6N///b//96Mf/Sja17/+9clk8nd+53d+7dd+LZ1O77I7glITNukIHr1VZQgYAoaAIXCsItCE/VcTNulYPbsvy3YbIXxZnnY76JcfAps3b56YmHjNa14TDv0nfuInnn766YGBgeXLlwflkRSasElH8vCtLkPAEDAEDIFjFIEm7L+asEnH6Ml9eTbbhoy+PM+7HXWzIPDmN7/5J3/yJ/fbmre//e1Rs3/913993ete19/fv3Llyptuuukb3/jGD37wA148e/bZZ/fmavHixYlE4vHHHw8Ga9as+Y//+I8oG8TtFVdc0d3dzXuGv/zLvzw7OxuMEfj18dOf/vTVV1+9cOFCvN1444133nln1EDyvtumNyv+/M///MILL1y/fv3emnTRRRdxOAxtZUOgFM6jZb/97W8fSJMeeOCBSy65pLOz87LLLnvsscfUwjmVypoz5b2Lz372s1u2bHnve997/vnng8+rXvUqwq25XC7YH0LbVPZzn/vcG97whhNOOGHFihVUNCekoRYTDAFDwBAwBPaBgHWpc/byTdWl7uP0WdZRRoDnPNsMAUPgqCDwn//5n21tbcTuQu1PPvnkpZdeun379qAZHh7OZDJ33303mlKp9KY3vYlBnh/4wAcgS9AJXgtMpVLveMc7uI8888wz2KxateqP/uiPVDwqv+c97+nq6oKMDQ4OKjekmJ122mnQvD/4gz+47bbbfu/3fq+3t/e8884LDYP8XHnllVDQ3/qt38Lglltuod5sNvuxj30sODnAtjGfzbJly/BDcHJvTQKEBx988Cq/IcDlqIVGRsvut0mjo6McxRvf+EZae80110B9C4XCnMpwCHMK1PvOd76TNoP8P/zDP3D4v/u7v7tkyRIQe+mll1TkYNtGqXK5/Na3vpVT+cEPfvCLX/winPPnfu7n2P3pn/5pTiWd+pyNMeVhR0Ad8GF3aw4NAUPgyCNgXeqcvXxTdalH/qqwGveLQI2I7tfODAwBQ2CeEICiLFq0CJoR/P/v//2/+Wb+zd/8TdD85V/+JQEoZgRF89u//dvt7e0iSMHgO9/5jt4D3DchJOLHRDIUJy7HO4T/9V//FTzAZxYsWPDCCy8EDXNdtra2BmIJ/zzllFN27twZDBAI02Hz+c9/XsoDbBuscmhoSEX20SQMftRvsiSlkdGy+23SN7/5TZDcuHEjZZ977rmf+Zmf2bZt25zKUMWcAvXih7MQzYVYEke97rrrdF4Otm244gVOzkUD8SN6Cc2mugZ9tGqTDy8CoM12eH2aN0PAEDgqCFiXurdentPRJF3qUbkwrNJ9I6B+0DrCfaNkuYbA/CJAgIiBmqGOM844A/rHyMyggXgwQJFduAfjFf/wD/8wZAVBU8XsmxDKeGZmBgrHiFO+/x/5yEekhM8Q8QveJPCS4Q033IBMFwtL+dKXvtRgwC6s7JWvfCXCgbeNIg1+5mwSNnv2XqHsgTSJAbTxePxXf/VXCV2GGudUhtw5BcAhwLhn1sMPPwyGDz30EFnYHFTbBBfx2D3dwqtxa4RwT2TmSaOOcJ6cm1tDwBA4wghYl9rkXeoRvh6sugNBQP2gEcIDwcpsDIH5QoBwH6/JKZD11FNPwWF4J5AgngZ2QgzQvPjii1RPmGtvVOH2228n60AIYTgMgpAU+e53v4sGPkPAKmRJYDgor8wh0wYs97YxLBObA2/bnCxINUabhGZPQhjKHkiT8PBXf/VXUFkGi8Ko6SNVy5xKZc2ZAg5jROfMIqz6qU99iixsDqptgqsh0qsqHnnkEaDmAOes0ZSHHQFd2IfdrTk0BAyBo4KAdakB9ubsUkPzTGgeBOgHbZbRvT3lmt4QOEIIQLrOPffcf/7nf/6N3/gN3iVjwpLXvva1BAkZ0snrZIwmJTzFjCO0hrASKfGuPVumrD31QYMBtx54ZtAQEoQmMezz4osvRqlBpyEXgeGgmjelr6+PXWaLYR6aqEFUPvC2wW9VcL9NivqXHMoeSJMo8vM///M//uM/DrZ/8id/Qlz0vvvuI8Q6p3LPuqIaoIvuBhk8eRVQuwfVNjmc86ztra5QqQmGgCFgCBgCe0PAutSATEMvH/RBOKhui1Jz9p5zKkMVJhwrCNgso8fKmbJ2Hs8IvOtd7/qnf/onmACEkIlGOFRSZAZG/tu//dv//J//Uwd/6qmn8r64goENcOx3dkomcVm6dGlDKd5hKxaLDco9dynINJi8QMiMnQ3bo48+yvw3FDmEts13k2Cz+XyeAOaHPvQhAqEE5XiBcE7lnofcoJkT3ieeeIKXIZm9rcH4QODihcyenh690NhQnPhwg8Z2DQFDwBAwBA4cAetSA1aHsZefs/ecUxlqN+EYQsAI4TF0sqypxy0CzBHKqgZwQgYKvuUtb+E4WUwCEoKGABTzaOvIGVnK++LwKIbERLGA7fzZn/1ZVLOnzByhzOr53//93yHre9/7HuNUCUgGzT4EYokEMJmpLGpDzO0XfuEXFKw7hLbNd5P+v//v/+OnYjW4o6MjmUyOj4/PqYwe1Jwy54IlN6JZk5OTP/uzP8urnq94xSuieskHAtev/MqvcCph1NHinFm9QxhVmmwIGAKGgCFw4AhYlyqsDm8vP2fvOafywM+UWTYXAsQlbDMEDIGji8CP/MiPsFzeBRdcoGYwmJA339C8//3vjzaMgB4LADK8k6gXs7wwrPTDH/4wy04oirjvdwhvvvlmlq+A1xGY+ou/+AtmN2XhOznnFbgwoWio7hOf+MTpp58edvnNlffxWJ/w1ltvhVi++93vpl4WsQgGB9s2Cu6jSeTu+Q5hQyP33aRvfetb0FQmF2UiciKucEJWiZhTGQ5hTgFweLxgnQlYOqNPQe+P//iPTzzxRIKi0WUnDqptVATV/7Ef+zEw1LITQErPypll+A09hL1DOOe5mA+l+uP58Gw+DQFD4GghYF1qQy/PiWiSLvVoXRJW7z4QqPHSfVhYliFgCBwZBCAtfCEJGYXqmLUSjSaxDEoJjCPlPUOGcbKxOB7MkFUiWL1Q60ZE2V1UhrD9/u//Pkv5MYbk7LPPZqIUNHIYNQt1NRBC9IxiZb0K1n6AHbG+H78+BuMgHHjbKLKPJpG7394Lm303icX9OFJWeiSOBxVUI+dUKmvOVOBs3rwZbok3WDoTqxIDZKhMsJ8TwH23TWW//OUvX3/99ZB/UOVUUgSSyalkgGtwbsK8ImCEcF7hNeeGwFFBwLrUhl6es9AkXepRuR6s0n0jQD/Ywn+M1CNaaggYAoaAIdCAABP8MDKW8F2D3naPDwQII3Mg1g8eH2fTjsIQMAQMAUPgYBGgH7R3CA8WNLOfGwFiU4zQYAKPk08++dd+7deYzGNuO6/lnTcuPm0ElBosmTTlqquuYsoNRuUxpJDYV4NBk+/u++iijWdmkToMu/6y6mCwOXahuOOOOwghEkhkftS3v/3t999/fzioPYV9Xzz7BoGrbhd2XuLFyz2r2LemwUN0d98F95Z74NcAHvaB1X6vkL01oHn0+z59+2jnO9/5Tk7EPffcI5vjAIp9HKxlHR8I7PtW1nCM+75LHPIXp6GWo7W776OLtmq/X+1jF4p93NujCEje98WzbxAOSz+4Z5N+SM2BXwNUtA+s9nuF/JDtPALF93369tGAI9kP2rIT+zgRlnWgCED/rrvuOgIpsLsdO3awFPjo6OgnP/nJvZU/66yzHnzwQXIhfg02X/jCF+APvFtFTIb36P72b/+WJRkYNnnmmWc2WDbt7j6OrqHNt9xySwNz5oVARoHK7NiFgqX5eBGOde2Z6oaFGRi6w+QrrFrBeJUGBNjd98WzXxB+8IMfcL3RHQbPq1evDvIBClNTUwdoeYBmB34N7BurfV8hB9iYo2i239O3t7Z97Wtfa/ip6FiHYm9HavrjBoF938r2PMx93CUO+YuzZy1HS7OPo2to0r6/2scuFPu+tzeAsO+LZ78gHJZ+sKFJP/zugV8D+8Zq31fID9/O+faw39O3twYchX5w3+NKLdcQ2C8CXO68lsYUjrLkHS1owOzs7H4LMs8HPCGYscrCsmXL3vOe9wQNnBAuwUJ8QXMMCQ1Ht9+W8/IYM2HyOxmWxy4U/BbAWn+//uu/Hj3ej370o4R8mZkzqpS8j4tnvyCw1jzLKN199917uj2MGlr4/e9//9Ac7vsaOFisolfIobXnSJba7+nbW2MmJiYYHcAbm3ST4c3PBuPDCIU64wb/tmsIHCwC+7iV7dtVw13ikL84+67laOU2HN1+mxH9ah+7UBzsvX0fF89+QTgy/eB+T9w+DPZ9DRwsVtErZB+VNknWfk/f3tp5VPpB9+6EbYbAD4PAbbfdxtyVwYMWRWBJg6DZm9Bwm3jggQd4OFu/fn3U/j/+4z+Y3JLV2KPKY0JuOLr9tplI1xlnnCGzYxcK5iDVAg/R4x0eHubMkhVVSt7HxbNfEB555BHcshDinm6bRLPva+BgsYpeIU1ygPtoxn5P397KMrMuI8+3bt3Kyd0bITyMUFAL294aY3pD4AAR2MetbN8eGu4Sh/zF2XctRyu34ej224zoV/vYheJg7+37uHj2C4L1g/u9qI6iwX5P397aduT7QRsyqocBS38oBJgmkS24YC2EBQsWMO9i0BygwBT8rLlHcCBqDxtkOkeCS4SYovrjTOYYWemOuSt1XMcuFPx6x4SZXV1d0RPEy6WEDVnaPqqUvI+LZ78gME4Gt4sXL2YKUN43a7hy9qyr2TQHhVXDFdJsx7Jne/Z7+vYsgua+++5j+NA3vvENBh3MaYDymINibwdi+uMJgX3cyg7qMA/ti3NQVTStccNX+9iF4qDu7ZyOfVw8+wXB+sGmvZ5p2H5P35yNPyr9oBHCOc+FKQ8FAW7lrJrA2uWMe2bMNw/oB+vlwgsv1FTR0YL80sbbicc3G+R4//3f/501GH7yJ39Sx37sQsE4BxZmiJ5ByTC3fbyqN+fFs18Q6Ai5zE477bTnn3+eWnj98vd+7/d+6qd+as/am1NzUFg1XCHNeUTRVu339EWNJXMZ/PRP/zRn8Nprr2UI+p4G0hxzUOztQEx//CEw563soA7zEL44B+W/mY0bvtrHLhQHdW8PZ2TOi2e/IFg/GABsQmG/p2/PNh+tftBmGd3zXJjmEBHgfT9etdq0aRPl161bd4hedi921113feYzn/k//+f/7K4+DvdYQ5al3hlds7dj+//bu38QKX4+juP3BwRrLR4VbLQQEQWxs9DmEQVBUBRLO0XQykIQC8FOsVFsRBG1ErGwEsVSeBARbWweLAQRK7GxsPDu+SSZZLKT2ezc/Z65ze6+D7nLfCfJJK9kZ8z+mwmiaH0uQB/2G9Y1xTtOngbC58+ff//+ffbs2a9fv3779k1fyaO07hqfOVBpu7pbjZwhpXUtbU9j+NIM165d0zrw5s2b6a44MgUUcXdIT5NAx1PZiro88oGzotpKzjzyoT1BFN3P7WFEOk6eBgLXwQA4EYnG8KVtHud1cNgbWIkjsDoB3Vx73bp1OrOPLJ7/dMH79+/1wqCWSSPrKTNDvndxm/Xd+rp4ZO5FPkEUeo1ux44dce9cWm8h1peOpvFGJDN5UgR9dkLvrIhruH379vr1679//x4Hx5jOz4HuViNnyBj72PHQ6fA1Cn748EGfPn369KmL//r1S1fK9DOE/3cKdz1uNIZNBP6hQOZU1qg5f5YY+cBp1FbaZr53cWtHPrQniKL7uT0WCOnM5EkRuA4Gt/IT6fA12jzG62DuOft02UoEAb0RQq/GhB99tC81OX78+Llz527dupXu6h7ROU5vG9NXjN67d697qbXM2YWiY3u0jDl8+LButtOav2SKVgQ9zZl2pDWYZhs2eVoR9u7du3///rgSvUL49+/ft2/fxsGS060saTA/Q0ruoGtb6/DFzda3RunNokeOHDl58mQcT9OTTpH2iMjECbSe9xq9GHYqa2TLb4584OSLr8HeLhQdm5F/aJdM0YqQnsbl0BpMfYZNnlYEroMpYJmR1uGLmzre6yALwngsSI8W0PJML/WEnwcPHuhtonfu3GmUPHjwoO6yqu/bbcQ7buo5Et3YUKtBdweLjqXWOFtKsboG6NPn+uTkhQsXWosXTpEi6Otk3Gs7je7okpl+ELTj5GlF+PTpkz6t2jiKPsC9adMmPWHRiJe52dEqP0PK7FrcqtbhizMofePGDX298N27dxvxxuakUzS6w+aECqTnvY6nshX1t8sDZ0UV9pE5pVjdUfIP7cIpUoSO53Zn1XHytCJwHVzdfFv7Uq3D12jG+K+Djdcr2URgRQL6umS93VH/3Y9L6a70+u7HONKabn0ziR42+pJSPUOmL1lpLTUpwdbepY2/fPny9u3b9cRhumsSKV68eKGPC/78+TPujm47oUny6tWrOKh0l8kzDOHhw4d6d2jjdpdai+roWig2DjSuzfwc6GiVmSHj6lf34w4bvkYN+/bta1waw+b169dD5j4o3IHCIUggsAqBLqeyYdW2niU6PnCG1VlOvLV3afMyD+1JpOh4bncOXSbPMASug+lcKjAybPgaTR37dZD7LzVGhM2VCehdozrjX716NRTTC4N79uzRF2a6iDLoZuu6d2rIEBLppUL3MNQtCk6cODHpq0H1Me1dSiGWDRs2tH64bkIp9FWi6viVK1fCKCuh21TqGVPtUjpGGDl5Mghac4pOt7yPD3Tx4sXNmzc3np6IM6xxOj8HRlqptZkZssZ9WcXhMsOn2uKZoG8n/s/gz+vXr7VU02uG+rogd+ieKFgQrmJkKdIQGHkqi2d7o2x6lsg/cBrFC99Me5dSZB7aE0ox8tweI4ycPBkEroOFz381LzN82hvPhPFeB7ntRHgamsQqBXS6f/bsme4fqPuD67fO7Pr04I8fP3QnMVfjly9f9GlAvR9s27Zt+WPoy7KUU3eTu3Tpkt4IEWfWu+T1ElMcmcR0SqFv2f7z58+ZM2ca3ZlcCt0+7tGjR6dPn9aNxY8dO6Z+vXz5UrdY1KLX3VkuRshPnjyCXka+f//+qVOntGDQ4fQS6+PHj3WTkufPn7fe96IhPK7NuPsjrdTIYTNkXO3vftz88OnhHFPs2rWrUbO77cTOnTu3bNnidk0uRaNrbE6fQP5Upv7Gsz3f/ZEPnHzx8vemFMMe2pNLMfLcHiPkJ08egetg4RM+P3zFXQfLX17TwvIF9GWABw4c0Hlt69atuo2YbkEe2vzx40c9YrUgDJGQUP4nT56ETd2QethjWzdmCdkmJdHonZqdUuzevfv8+fNpjyad4s2bN1qq6bOmGzduPHTokBZpoY8pwrDJ0wXh3bt3R48e1avKugmh1p+630k4UAmJLnMgY6UuDJshJfQu34aRw5fOhLjC9FtGe6Jw55z40KQRWJ3AsFOZasvM9sZZYuQDZ3VtG1epRu9aKYY9tCedInNuT+fDsMnTBYHr4Lim98jjjhy+dCbEda7lddC85KJjD/tfOHEEEEAAAQSmWMC99YDr4BQPMV1DAAEEEMgI6DrIt4xmfNiFAAIIIIAAAggggAACCEyzAAvCaR5d+oYAAggggAACCCCAAAIIZARYEGZw2IUAAggggAACCCCAAAIITLMAC8JpHl36hgACCCCAAAIIIIAAAghkBLjtRAaHXX0J/HvhpKlat5HQp1j1y/yen1swX3FkvuBhwT5Pod/aMHnqnCab+WdK+b0uYn+7OlWP8lT1zy9XxaOgPZDyLFe1qSp3iHnz9UoKuudJ7F5bfK7KafaaDCboc7oi1VH0FU0mg/lni9jMVSkbdKXM3ihbKBUSOoQv5RJV/iqDOXxdQ8hpwr7akHAHMvnt7nDoUIPNOVi/rSTKEPYqb522NZtNW4PpskksNzP4vVUGv+kyG6tQZ5xWZbZ+n8Fu+rKmTMhggj6zT9u+hqDtejX+Jlgdx5ZyOe1QmzrDpKjTBqOK26CZp3YK+MyWVnUumHlRB83m/LKdSjZhjrscgvNzS2avmfVVKeV0xRVZdEGTX8ElRUxOVR5KKWiPZeNLtmZlMDmVdsWVwRa0e13a1GCCi6rHFl/UZlVqadEdy+e0xc1RXAY1TwlbfGnRNNW2SsfyrVLQNMkUn7NB1xITtF2bs/WroPYqqI4b+0XzeDNpPaQX9LdK24Qi8wsL//qvBoYfBKZPgOugzhHmjGBPRhrfajMkdIpU1GWwicEM2uGLqIaQ04QHStn6TcwWt7vNEW2eUIMt0gzavM2gqaku7ttvo7bOqbsOVv/lsaOka5ntvblycR3U4E/VdVDXZX4QQAABBBBAAAEEEEAAAQRmUYAF4SyOOn1GAAEEEEAAAQQQQAABBCTAgpBpgAACCCCAAAIIIIAAAgjMqAALwhkdeLo9QwLL9vMVM9RhuooAAggggAACCCDQVYAFYVcp8iGAAAIIIIAAAggggAACUybAgnDKBpTuIIAAAggggAACCCCAAAJdBVgQdpUiHwIIIIAAAggggAACCCAwZQIsCKdsQOkOAggggAACCCCAAAL/XMDcWJGfWRBgQTgLo0wfEUAAAQQQQAABBBBAAIEWARaELSiEEEAAAQQQQAABBBBAAIFZEGBBOAujTB8RQAABBBBAAAEEEEAAgRYBFoQtKIQQQAABBBBAAAEEEEAAgVkQYEE4C6NMHxFAAAEEEEAAAQQQQACBFgEWhC0ohBBAAAEEEEAAAQQQQACBWRBgQTgLo0wfEUAAAQQQQAABBBBAAIEWARaELSiE1kJgft4dZd4ntGnSLhwFFbb/zO6oYXHah0NxW1cVjUuFtE1UVbTV1DhWS85wrMGqfM6o0ihZVzsQrDrt/9iGhww2UW15nkBisoaccXowGIrXWeKqQh0haOr1FcfBkB6s32avCsTHimqpstR/TD6bN9RpA83iVbDKOVjcblUFQtpvh78+YXL4dJgIJjY06Hf4kmHMQyAu3hY0NYQJ4g9UBet+24ArHh/SNdeXMvttXSG32bZR/bHBsOWi/ncVtpt12h/e129rqCv0Feuv2+Nrqw5aV2RaVTXA7Kt3hGBdtCpsS4Sob4E5VJ1WPXVVIS8JBKZKwE/4aOaHx5N5PESd9Y+IZjDK4pLKEMqFzCGhPCFtE1XeUCSuL+Q0Qb8R51TMbQ5W5euMskbJUFPdTlO93xrI6Vtjg9WeOGfIHBK2qqrYYDAUr7PEVZkm2J8QrLaSYMgwWL8r7WoJVcXBKh3/Mfls3lCnDTSLV8EqZ11BlS803e4xQb8j/PUJk8On/Yja2NCg3+FLhjEPAVNlkisKmn1hgvicVbDutw24OuPKXHN9KbPf1hVym20b1R8bDFsu6n9XYbtZp/3hff22hrpCX7H+uj2+tuqgdUWmVVUDzL56RwjWRavCtkSI+haYQ9Vp1VNXFfL2lDBHWl7mvpM98VItAggggEDRAu7qy3Ww6EGicQgggAACvQnoOsgrhL3pUjECCCCAAAIIIIAAAgggULYAC8Kyx4fWIYAAAggggAACCCCAAAK9CbAg7I2WihFAAAEEEEAAAQQQQACBsgVYEJY9PrQOAQQQQAABBBBAAAEEEOhNgAVhb7RUjAACCCCAAAIIIIAAAgiULcCCsOzxoXUIIIAAAggggAACCCCAQG8CLAh7o6ViBBBAAAEEEEAAAQQQQKBsARaEZY8PrUMAAQQQQAABBBBAAAEEehNgQdgbLRUjgAACCCCAAAIIIIAAAmULsCAse3xoHQIIIIAAAggggAACCCDQmwALwt5oqRgBBBBAAAEEEEAAAQQQKFuABWHZ40PrEEAAAQQQQAABBBBAAIHeBFgQ9kZLxQgggAACCCCAAAIIIIBA2QIsCMseH1qHAAIIIIAAAggggAACCPQmwIKwN1oqRgABBBBAAAEEEEAAAQTKFmBBWPb40DoEEEAAAQQQQAABBBBAoDcBFoS90VIxAggggAACCCCAAAIIIFC2AAvCsseH1iGAAAIIIIAAAggggAACvQmwIOyNlooRQAABBBBAAAEEEEAAgbIFWBCWPT60DgEEEEAAAQQQQAABBBDoTYAFYW+0VIwAAggggAACCCCAAAIIlC3AgrDs8aF1CCCAAAIIIIAAAggggEBvAiwIe6OlYgQQQAABBBBAAAEEEECgbAEWhGWPD61DAAEEEEAAAQQQQAABBHoTYEHYGy0VI4AAAggggAACCCCAAAJlC7AgLHt8aB0CCCCAAAIIIIAAAggg0JsAC8LeaKkYAQQQQAABBBBAAAEEEChbgAVh2eND6xBAAAEEEEAAAQQQQACB3gRYEPZGS8UIIIAAAggggAACCCCAAAIIIIAAAggggAACCCCAAAIIFCjwPxvYKe8QPqjfAAAAAElFTkSuQmCC", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import pyvista as pv\n", + "import numpy as np\n", + "\n", + "\n", + "camera_position = [(-2.0, 2.0, 0.5), (-0.5, 0.12225, 0.15775), (0, 0, 1)]\n", + "\n", + "result = pv.read(\"./results_nbk/graph_27.vtp\")\n", + "\n", + "p_min = result.point_data[\"p_pred\"].min()\n", + "p_max = result.point_data[\"p_pred\"].max()\n", + "s_min = result.point_data[\"wallShearStress_pred\"].min()\n", + "s_max = result.point_data[\"wallShearStress_pred\"].max()\n", + "\n", + "plotter = pv.Plotter(shape=(2, 2), window_size=(1200, 1200))\n", + "\n", + "plotter.subplot(0, 0)\n", + "plotter.add_mesh(result, scalars=\"p_pred\", clim=[p_min, p_max])\n", + "plotter.add_text(\"Prediction: Pressure\", position=\"upper_left\", font_size=10)\n", + "plotter.camera_position = camera_position\n", + "\n", + "plotter.subplot(0, 1)\n", + "plotter.add_mesh(result, scalars=\"p\", clim=[p_min, p_max])\n", + "plotter.add_text(\"Ground Truth: Pressure\", position=\"upper_left\", font_size=10)\n", + "plotter.camera_position = camera_position\n", + "\n", + "plotter.subplot(1, 0)\n", + "plotter.add_mesh(result, scalars=\"wallShearStress_pred\", clim=[s_min, s_max])\n", + "plotter.add_text(\"Prediction: Wall Shear Stress\", position=\"upper_left\", font_size=10)\n", + "plotter.camera_position = camera_position\n", + "\n", + "plotter.subplot(1, 1)\n", + "plotter.add_mesh(result, scalars=\"wallShearStress\", clim=[s_min, s_max])\n", + "plotter.add_text(\"Ground Truth: Wall Shear Stress\", position=\"upper_left\", font_size=10)\n", + "plotter.camera_position = camera_position\n", + "\n", + "plotter.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "As you can notice, the predictions of the model match well with the ground truth results. \n", + "\n", + "As one can notice, the current output only provides wireframe output and lacks cell data. In the subsequent steps, we will resample the wireframe data onto a mesh to get more smoother visualization. This will also enable us to compute the surface averaged quantities such as drag coefficients with more accuracy. \n", + "\n", + "**Note:** To demonstrate this, a sample mesh which contains the cell data has been provided in the package." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCASwBLADASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwDrNO0zwdovw08M6pf+D7DUJ7u1sodsOnwPNNNKigEl8ZJY8knvVu2j8DtqNrZal8PItIe6kEUEl/pFuI5JD0QOhYBjjgHGe1U9Rlkh+EfgKWGBriVJ9HZIUYKZGHlkKCxABPTJIFbOpx+I/Fs2nWM/h19IsYL6G8nubm7ikciJg4VFjZuSQBkkYGetAG3/AMIJ4P8A+hU0P/wXQ/8AxNH/AAgng/8A6FTQ/wDwXQ//ABNclpF1qFqPGPia+1bULm30e+vvs2niY+UVRM4bqT6AdFxnFZlrfzX+nRX17qfjxNXmQS+ZZ6XcLbRMRkKkXl7GQdPmyT1zzQB6B/wgng//AKFTQ/8AwXQ//E1m674e8D+H9KbUbvwjpDwrLFEVi02AtmSRYx1A4ywz7ZrKi8QeINesPDGkS/aNF1DVFuGvpzAYpVjgIB8tHHymTcpBI+UE1R8feGrzRfD0c9nr+pXNo99aLdW2o3BnDDz4yrIzcqwYDgHBGeKAOmtPDPgu71bUNOXwbpSSWPl75H0yEJJvXcNhxzjv05q//wAIJ4P/AOhU0P8A8F0P/wATXL6r4k1DRNT8dXEEhmktzp8VpHM5MUTyqEBx2G5gxxjOK1pPBWqJa/aLbxhrX9sAbhPNKGt2fuDBjYEPoOQO9ABqHh7wPpuqaVp83hHSGl1KV4YSmmwFVKxtId2RwMKemeafaeGfBd3q2oacvg3SkksfL3yPpkISTeu4bDjnHfpzWJ4v8PC98ZeEJLu/1GO4uriVJhaX0scaMts5JiAb5MkdRyQTnqaTUNfv/D9742e2mkne1GnW9mlzIzpG8qhAxz2ywY+uKAOr/wCEE8H/APQqaH/4Lof/AImj/hBPB/8A0Kmh/wDguh/+JrHvvCeq6fpM+o2ni3WH1iCJpvMuJg1vK4GSrQ42qh5HGCPXismC81Hxl4u0cxarf6Zp174cjv57e1lKks0gwAf4Tz94DJAxnBoA67/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiazNEa90PxzN4bk1K71Cwn077dbveyeZLCyyBGTf1ZTuUjOSOa6TRYNRtdGtYdXvEvNQRMT3CIEEjeoAAA/KgDjNasfBej61BpMXw9ttSvJrdrkJY6XanbGrBSTvK92H51Lomn+BNavptPbwTYafqMMYla0vtJhjkMZON64BDLnjIJwetN16+vdP+LGny2Gkz6pKdDnUwwSxxlV8+I7syMoxwBjOeah1G31+9uNW8UX9gdGWy0O6trSEXKyTs7AOZGZDhQNi4AJOcnigDpf+EE8H/wDQqaH/AOC6H/4mj/hBPB//AEKmh/8Aguh/+JripdR1jw18P9GvG1bU77Vdee1gacxm4NsGjLsYogDuYKCOcljgnNVrrVLvR4lv9Afx3e30TKXtNR0+5liulyNy/MmI2xnBXAB7YoA77/hBPB//AEKmh/8Aguh/+Jo/4QTwf/0Kmh/+C6H/AOJrj/EM+o23i28uNW1nWNGsjJB/ZV5CpexRcLuWdRxuLbhl8DBGCKXxHNqNp4vvZ9W1rV9Hs/Mg/su9gBexVcLvWdRxlm3DL4GCMEUAdf8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTVbQru5m8e+LbaW4leCD7H5MTOSse6Ik7R0GT1x1rk/GOrapb/APCwBZ6jcxPawab9m2zMBCzsd23B+XPGcdaAO1/4QTwf/wBCpof/AILof/iaP+EE8H/9Cpof/guh/wDia5bxLeah4a/sbw+mq63cyapJNPeX8Fubm5VEVdyxIqkICWGMDCjPc1UtNVvNL1vTW0YeMry3uLlIL221awuHQRsceasjplCpIJGcEZ4FAHaf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAc/8A8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXQUUAcH408F+FbXwL4huLfw1o0M8WmXLxyR2ESsjCJiCCFyCDzmofC/hfwbF8OdE1PU9A0IKNKt5ri5uLKL/nkpZmYr16kk10Xjv/AJJ54l/7BV1/6KauNuArfDT4dpd/8gtpNOF9n7mzyfk39tvmeXnPHSgCVG8EzIbi3+Gcs1h1F4mgRbGX+8qnEhHfha6LTPC/gLWNOh1DTvDvh+4tZhuSRNPiwex/h4IPBB5Bq9qetavZXzwWnhW/1CFQCLiG5tkVsjnh5Fbjp0rh/EmrQ618OvFFva6NJo0lvewxT/6o7p2mjZmBjJVmGRk5znrzQB2v/CCeD/8AoVND/wDBdD/8TR/wgng//oVND/8ABdD/APE1gapotpa6v4e8IW73FvpN4Lm5u8Tv5l20YTCNJncd28s3PIXHTNOvtHsvBviLw9PoCG0j1C9+xXVlG58qZDG7b9hOAylAdw7E5oA3f+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJrP8Ahw6x/DLS2dgAkMm4nth3z/KuJ0fSodZ034XWdy8ot20+5MqRyFDIojT5SRzgnGR3HHegD0b/AIQTwf8A9Cpof/guh/8AiaP+EE8H/wDQqaH/AOC6H/4msW10y28J/EDTrHR42t9P1SxuXls1cmMSxGMq6qThSQ5BxweKqeFPDGleL/DNr4h16Nr/AFLUVaZ5WmcfZ8k/u4sEeWE6cc5BzQBqaV4d8D6w1+tv4R0hTY3b2cnmabCMuoUkjAPHzDrj6Vh6ivhHTdZi0t/haZZp3kS3aLS7IrPsGWK5kBxjnkCtL4Xwm2sPEVu17JemLXbmP7RI2532qgyx7njBPc1e8Qf8lB8HfW9/9EigCxbeCvCNxawzN4O0eBpEVzFLp0G9CRna2ARkdDgke9S/8IJ4P/6FTQ//AAXQ/wDxNYGk6Haah8VPFGoXfmSNZyWht4/MIRHMAy+0HBboBnOOcdav/Dh1j+GWls7ABIZNxPbDvn+VACav4d8D6LHZvc+EdIYXV3FZp5emwnDyNtUnIHGev8qLLw74HvtX1PTIvCOkCbTmjWZm02Ha29N428ZPB5yBXDjQNJ1L4Z/D+5vdOtridrjT7YySRgkxM/zJn0OTxW7pXgzRNR8eeKbe7sxJYWgs4rey3FYU/cDnYDgkAADPTnHWgDrP+EE8H/8AQqaH/wCC6H/4mj/hBPB//QqaH/4Lof8A4msbw5a3aQ+LvDFlqM8C2Nx5NhcuTK9sssCOoG45YIznAJ6YGa7aJGSFEd97qoDPjG4+tAHmFtP4Mu7Q3sHwtkksQzj7Smj2jqQjFWIVXLEZU/w54rqrDwn4G1TT7e/svDWgzWtxGJIpF06LDKRkH7tcz4HvPFo8HQW+laNpbQedciK7utQcf8t5OWiWI9DnjdzjqKg0zwTZweOdM0W9mluoNO0CNmUMUWaT7Q53MAeQCSQp46elAHa/8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTXCRJN4m1bWrzVvB9/ryw6jPaW4F5AkNvHG20BY3lUhjjcWIyc8HGKQaT4g1PRJNLlsJJ7Kw1VZItKvdSiaa5tvLyYGdHblGYMA55AXNAHef8ACCeD/wDoVND/APBdD/8AE0f8IJ4P/wChU0P/AMF0P/xNcC+k22raZph07TL3VdJ02W7jvNAu7kJcW8hYY2gkBvL5Cgt0YYNbHhG4srjxnYyafdXNzaN4ajWKW7OZmC3DAh+B8wPB96AOm/4QTwf/ANCpof8A4Lof/iaP+EE8H/8AQqaH/wCC6H/4muX8SureJvGgBBK+FFDexzcH+oqhe2UnhzwD4ettKjvZrnW7i1hvpobkLPKDEzkK7sAhO3aMEAA4HOKAO3/4QTwf/wBCpof/AILof/iaP+EE8H/9Cpof/guh/wDia4abTbzR7mxvtA8F32h3SXUSyyPqVsIriNmAdJR5x3kgnBwW3AYr1mgDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJo/4QTwf/ANCpof8A4Lof/ia6CigDn/8AhBPB/wD0Kmh/+C6H/wCJrD8aeC/Ctr4F8Q3Fv4a0aGeLTLl45I7CJWRhExBBC5BB5zXeVz/jv/knniX/ALBV1/6KagDI0XQv7e+GvgqL7T5H2SHTb3Ozdv8AKVH29RjOMZ7ehrt65/wJ/wAk88Nf9gq1/wDRS10FAGJo/hyLTLfWIJ5Vu4tTvZrp0aPACyYBQjJyMDrxnPSsmHwfrdjajTdO8X3VvpKjZHE1qkk8Mf8AcSYngAcAlWI454rsaKAObvvBdjPpGm2dlPcWFxpZ3WN5E26WJsYOS2d4bJ3A9ayNU8Bar4hjiTXfFUlylvNHPbxwWawIro4bc4DEuSAR1AG4nGcV3dFAHNyeD7S7vvEcl+4ubXW0hSS32bfLEabchs8nuDgYIqi/hHxBNbf2fN41vG0zGxglqi3TJ/dM4PXHG4KD712VFAGJfeHI7rUvD91FOYY9Hld1iKlzIGiaMDcTkY3ZzznH41Wl8HWl5e+I5L+T7Ra62kKSQbNpjEabchs8nPIOBgiukooA4ybwdrl5Ztpd94xup9JdfLkjFoiXEkfTY0wPccEhQTzzzWvb+GobTxRFq9vIscEOmLp0doseAqh9wIOegHGMfjW5RQBjtoW7xjF4g+0/6vT3svI2dd0ivu3Z/wBnGMd+tWdFsrvTtGtbO+1B9Quok2yXTpsMp9SMnH51fooAx30Lf4xh8QfaceVp8ll5GzrukR927PbZjGO/WrmrWP8AamjX2n+Z5X2q3kg8zbu27lK5xxnGauUUAYM/hW1vPCllodzPLmzjhEN1D+7kjkjACyJ12nI9+pHIqh/wier37wxa94mlv7CJ1k+zQ2i2/nFSCPNZSSwyM4G0HvXW0UAcZrPga81WbUrdfEVxDo+qOHvLJoRI3QBhHIT+7DBRkYPfGM0mr+BbvU5NRtY/ENxBoupuGu7FoRI3QBljkJ/dqwUZGD3xjNdpRQBzOp+Frt9bbWNC1htKvJoUguVa3WeKdUJ2EqSCGGSAQehrIb4bPLY+IY59enuLvXPszT3M0AO1omzwoIGCMADjAA613tFAGP4g8Px65HbSJdTWV/ZyGW0vIMb4mIwRgghlI4Kng1QtPDGpTalbXuv6++pC0fzLe3ithbxK+CN7gEl2APGTgdcV09FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imqHwhZ22ofDDw/Z3kEc9tNo9skkUi5VlMK5BFTeO/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0AUv+EDiRPIg8R+IoLHoLSO++RV/uhypkA+jVpXXhPSrjwwfD0MLWmn5QhbcgEFXD5yQcksOSck5PfmtuigDM1vQbLX7WOG8EqvDIJYJ4JDHLC44DIw5BwT7HPOapaZ4RtbDVE1O5v8AUNTvokMcM1/MH8lT97YqhVBPAJxkgda6CigDlJfAGmvLcpFqGq29hdSNLPp0Fztt5GY5bjG5QxzkKwByeKt2Hg/TtN/sL7PJc40WCSC2DODuV1CnfxycKMYxXQUUAZ1zo1vda7p+ru8ouLGKaKJVI2ES7N2RjOfkGMEd+tY9x4Fs2uLmSw1XV9LiunMk9vYXISN3P3mAKnYT3KFa6migDH8OeGdN8K2U9npUbx2807TlGbdtYgA4PXHyjrk9eanvNGt77WNN1OV5RPp5lMSqRtbzF2ndxk8dMEVo0UAZ1lo1vY6vqepxPKZtRaNplYjauxNg28ZHA5yTWLL4A015blItQ1W3sLqRpZ9OgudtvIzHLcY3KGOchWAOTxXV0UAc+ng/To9B0fR0kuRbaVNDNbneNxMRyoY45HrjH4VfstGt7HV9T1OJ5TNqLRtMrEbV2JsG3jI4HOSa0aKAMSXwxaSDX9tzdxPrYHnvHIFaIiIRAxnHBwoPOefyrYijEMKRBmYIoUFjknHrT6KAM7Q9Gt9A0iLTbR5XhiZ2DSkFsu7OegA6se1A0a3HiNtc3y/amtBZlMjZsDl84xnOT6/hWjRQBzt/4PtbrUp9Qs9R1LSrm4x9pawnCLOQMAsrKy7sDG4AHHeo5fAmkHTbS0tnu7Oa0na5ivYJv9IErAh3Z2B3FgSDuBB/AV01FAHIL8O9MhWGSzv9Us7+Myl9QgnAnn81gz+YSpDZIB6cYGMVYk8CaSLHTLexlvNOl02Nora6tJtsqo33gxIIYEgE7geea6eigDlbXwDpdqdWf7VqE1xqtmbS7uJ5w8jr8w3ZIwGw2BxgADita48P6feeH49Eu4mms440jXcxDjZjawZcEMCAcjHNalFAHN2ngy1hv7a7vtT1XVWtW320d/cB0hfoGCqo3MOcFskZ4rpKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiikLBRkkAe9AC0VWl1Cyg/1t5bx/78oH9apv4n0KPrrFif92dT/I0nJLdiujVorAk8beG4hltVh/4CrN/IVVb4h+GV6X7N/uwP/UVLqQXVC54rqdTRXHP8TPDy/da6f/dh/wATVV/ipo4+5ZX7fVUH/s1J1qa6k+1h3O7orz1/ivZDPl6XcN6bpFH+NVX+LMmTs0Vcdi1z/wDY1LxFNdROtT7nplFeWSfFa+J/d6Zbr/vSMf8ACqjfFHXSTi309R2xG/8A8VUvE0+4niIHr1FeQJ8UNeXrBYN9Y2/o1W4/irqA/wBZpts3+6zD/Gj61T7gq8D1SivMk+K8o+/o6H/duCP/AGU1Zj+K9uf9ZpMq/wC7MD/QU1iaXcftodz0SiuEX4p6Ufv2N6PoFP8AWrEfxO0F/vR3sf8AvRL/AEY1Sr0+4/aw7nZ0Vyq/EXw43W6lX6wt/QVYTx34ak6amo/3onH81p+1h3Q+ePc6KisZPFmgOBjVrUZ/vPj+dTp4g0WT7mrWJPoLhP8AGq549yuZdzSoqul/Zyfcu4G/3ZAf61Orq/3WB+hqhi0UUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/wCO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtHjv/AJJ54l/7BV1/6KajwJ/yTzw1/wBgq1/9FLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFIWCjJIA96ry6hZQf628t4/wDflA/rQBZorKfxPoUfXWLE/wC7Op/kaqyeNvDcQy2qw/8AAVZv5Cp549xcy7m/RXLN8Q/DK9L9m/3YH/qKrv8AEzw8v3Wun/3Yf8TU+1h3RPtIdzsaK4R/ipo4+5ZX7fVUH/s1QP8AFeyGfL0u4b03SKP8aXt6fcXtodz0KivM3+LMmTs0Vcdi1z/9jUEnxWvif3emW6/70jH/AAqfrFLuT7en3PU6K8hb4o66ScW+nqO2I3/+KpE+KGvL1gsG+sbf0al9api+sQPX6K8rj+KuoD/WabbN/usw/wAanT4ryj7+jof924I/9lNP6zS7le3h3PTaK87j+K9uf9ZpMq/7swP9BVlfinpR+/Y3o+gU/wBaf1in3H7aHc7uiuMj+J2gv96O9j/3ol/oxqyvxF8ON1upV+sLf0FUq1N/aQ/aQ7nVUVzqeO/DUnTU1H+9E4/mtWU8WaA4GNWtRn+8+P501Ug9mPnj3Nmis1PEGiyfc1axJ9BcJ/jVpL+zk+5dwN/uyA/1qk09h3RYopFdX+6wP0NLTGFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/wCO/wDknniX/sFXX/opq6Cuf8d/8k88S/8AYKuv/RTUAHgT/knnhr/sFWv/AKKWugrn/An/ACTzw1/2CrX/ANFLXQUAFFFFABRRRQAUUUUAFFFFABRRRQBh+KPEsPhfT4rua3knEknlhUIHOCe/0ripPi+c/utFAHq1z/8AY1p/Flc+G7Q/9PY/9AavH9tcdetKMrJnLVqSjKyPQZvi5qjAiHTrRD2Llm/qKpyfFTxC/wB1LKP/AHYif5sa4rAo4rB15vqZe0n3Oom+IvieYnGoCMY6JCg/pmqkvjTxJMDu1e5GR/A23+VYXFGRUupN9WLnl3L8mvaxKMSapeuP9qdj/WqklzcTNulmkkPqzk1HkUZFTdsm77hub1P50ZbOQTRkUZpCHrM6ggEH6jNJ5j/3v0puRRkUreQ9yTzX9vypGmcdAtNyKRjmhLUTSHfaH/urS/aW/ufrUVFVZdibEv2n1Q/nS/aV/utUNFFo9gsyf7SnfI/Cg3cQ6sfyNQU0ihRiwdy2LiI/xCniRD/EPzqoFGOlG0VLjHoOzLgYHoR+dGapqNjBlOGHQinmSU9ZG/Opcew1tqWqM1VEsgBGcg+opoZx0c/nQodwfkXKKq+ZJ/eo82QDqPyo5ALVAJBBBwR0NU/tE3+z+VH2mXuq1Xsn3J5jVTUL2IYjvLhB/sysP61Ouu6wn3NVvl+lw4/rWJ9qcdUH50C7PeM/nT5KncftDpI/FviCI5XVro/7z7v51YTxz4lTpqjn6xof5iuU+2L3RqX7YncMPwp/vl1Y/avudnH8RPEafeuon/3oV/oBVhPibry9Us2+sR/oa4X7XF/e/SnC6iP8YquesurH7aXc9Bj+KerD/W2Nk3+6HX/2Y1YX4rXI+/pUR+kxH9K838+M/wAa/nS+ah/iH50e2rLqP28u56fH8WEP+s0dh/u3Gf8A2WrKfFXTj/rNOul/3WU/1FeU7ge9GRT+s1V1KWIn3PXU+KOiNgPa36++xCP/AEKrSfEjw63WadP96E/0rxjIoyKpYuoNYiZ7cnxA8Mv11Aqf9qCT/wCJq1H4y8OyjK6tbj/eyv8AMV4RkUZprGT6of1mXY9+XxNoT9NYsfxnUfzNWo9V06X/AFd/av8A7syn+tfO9FUsY+qK+svsfR6TwyY2So2f7rA1JXzWWA6kCnx30kHEVzIn+45H8qtYtv7IfWkt0fSNFfPCa3qqn91f3/8AwGVx/WrcPiDxMMeXf6gf9+dj/Mmn9bS3VhrFxfQ98orxSHxF4zONuoSD/eVW/mK0rfxB4yB/eakjD0a3j/wrOWZYaPxSNY1JS2g/uPWaK4Ox8V61GR9rjtZ177VKN+eSP0rqNP160vtqEmGY/wADnqfY96dHMsLWlyxnr9x0ezna7RqUUUV3EBRRRQAUUUUAFFFFAHP+O/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtHjv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABWH4o8Sw+F9Piu5reScSSeWFQgc4J7/StyuB+LK58N2h/6ex/6A1RUk4wbRE21FtGZJ8Xzn91ooA9Wuf/ALGqc3xc1RgRDp1oh7Fyzf1FefbaMCuB4ib6nJ7Wfc7WT4qeIX+6llH/ALsRP82NUpviL4nmJxqAjGOiQoP6Zrl+KOKl1pvqT7Sb6m7L408STA7tXuRkfwNt/lVGTXtYlGJNUvXH+1Ox/rVDIoyKhzk9xcz7kklzcTNulmkkPqzk1Hub1P50ZFGRS3EGWzkE09ZnUEAg/UZpmaMik1fcE7bDvMf+9+lO81/b8qjyKXIpBZDmmcdAtJ9of+6tNY5ptUrW1RLRL9pb+5+tH2n1Q/nUVFO0ewrE32lf7rUv2lO+R+FQUUcsQsyc3cQ6sfyNOFxEf4hVQinhRjpQ4wsCuWxIh/iH50oYHoR+dU9ooUbGDKcMOhFRyoaLmaWqpklPWRvzoEsgBGcg+oqeVlaFrNFUwzjo5/OneZJ/ep8gi1RVXzZAOo/Km/aJv9n8qapt9RN2LgJBBBwR0NWU1C9iGI7y4Qf7MrD+tZX2mXuq0v2px1QfnVezktmLnNtdd1hPuarfL9Lhx/WrMfi3xBEcrq10f9593865sXZ7xn86X7YvdGp8tVbP8R+08zq08c+JU6ao5+saH+YqzH8RPEafeuon/wB6Ff6AVxn2xO4YfhS/a4v736U71l1Y/bPud0nxN15eqWbfWI/0NWY/inqw/wBbY2Tf7odf/ZjXnwuoj/GKXz4z/Gv50/aVl1ZXt5dz0hfitcj7+lRH6TEf0qxH8WEP+s0dh/u3Gf8A2WvMPNQ/xD86duB70/rFZdR+3n3PVk+KunH/AFmnXS/7rKf6ip0+KOiNgPa36++xCP8A0KvIsijIp/Wqo/rEz2dPiR4dbrNOn+9Cf6VOnxA8Mv11Aqf9qCT/AOJrxHIpciq+t1OyK+syPd4/GXh2UZXVrcf72V/mKnXxNoT9NYsfxnUfzNeA5op/XJdh/WX2PoiPVdOl/wBXf2r/AO7Mp/rU6TwyY2So2f7rA184UhYDqQKpYx/yj+s+R9KUV83R30kHEVzIn+45H8qtJreqqf3V/f8A/AZXH9atYrvEPrcT6HorwOHxB4mGPLv9QP8Avzsf5k1ow+IvGZxt1CQf7yq38xSeOpR+J2+aLjiFLaL+49rorya38QeMgf3mpIw9Gt4/8K3bHxXrUZH2uO1nXvtUo355I/SsXm2EW8zohGc/stHeUVl6fr1pfbUJMMx/gc9T7HvWpXdSrU60eam7obi4uzCiiitBBRRRQAUUUUAFc/47/wCSeeJf+wVdf+imroK5/wAd/wDJPPEv/YKuv/RTUAHgT/knnhr/ALBVr/6KWugrn/An/JPPDX/YKtf/AEUtdBQAUUUUAFFFFABRRRQAUUUUAFFFFAHC/FZS3hSAjHy3iE/98OP6141uNe1/FFd3hAHn5blD+hH9a8Urz8Sv3hx1/iEopaKwMLCUUtFAWEopaKAsJRS0UBYSilooASilooASilooASilooASilooATNO3UlGKAF3UZFNxRSsFx2RRkU2iiwXH5FISMU2iiwXEopaKu5IlGKXFJRcLCYFG0UtFO4rDdopNop+KTFO4rDNg9KTYPSpMUmKfMwsiPyxTShHQn86loq1NkuKK5Mo6SP+dJ5syn/WN+dWMUFFNWqi6onk7EIu3XqSfqTSfbpM9ac8APSqzJtcZrWCpyIfMjRguN/3yx+hrYitrAsFkk3Z7rITWVaQBwK0o7Q9q87FVIJ2UrHRRpyava5uW2gafMoMZyfQ1oR6DbJ/AD+FYVs1xbMCjHjtXR6fqiT4jl+V/U968DFSrrWM20ethlQbtOCTJE023TpGKsLbRr0QD8Ks4FHFeW6knuz1IwjHZEIiHpThGKk4ozU3ZQ3ZS7aXdSbqQzoNH1xo2W3vHzH0WQ9V+vtXT9a83311XhzU/tERtJWzJGMoT3X/AOtX1OTZlKT+r1nfs/0/yOOvSS96JvUUUV9KcgUUUUAFFFFAHP8Ajv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLR47/wCSeeJf+wVdf+imo8Cf8k88Nf8AYKtf/RS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFcL8VlLeFICMfLeIT/3w4/rXdVxfxRXd4QB5+W5Q/oR/Ws6vwMip8DPFNxpKWivMR54lFLRQFhKKWigLCUUtFAWEopaKAsJRS0UAJRS0UAJRS0UAJRS0UAJRmlooAXdRupMUmKVgHZFGRTaKLBcdkUuRTKKLBccSMUyloprQT1EopaMVVxCYpMCloouKwm0Um0U6jFO4WGbRSbB6U/FGKpSYrIj2D0pPLFSYpKakxOKIihHQn86YTKOkj/nVikxWiqEOJX82ZT/AKxvzpwu3XqSfqTUxRTUTwA9KpThLdCcWthv26TPWrUNxv8Avlj9DWcybXGa1LSAOBTrezhG4oqUnZGrFbWBYLJJuz3WQmtm20DT5lBjOT6GsOO0OOKv2zXFswKMeO1eDXnKS/d1GmejSgou86aaN2PQbZP4Afwq0mm26dIxUen6ok+I5flf1PetXArwq1SspWm2ezRhQlG8EisttGvRAPwp4iHpU3FHFc/MzpSSIxGKdsp2aN1K4Cba6DR9caNlt7x8x9FkPVfr7Vz+6k3104XFVcNU56b/AOCTOCmrM9I60Vg+HNT+0RG0lbMkYyhPdf8A61b1ffYbERxFJVYdTzZxcXZhRRRW5IUUUUAFc/47/wCSeeJf+wVdf+imroK5/wAd/wDJPPEv/YKuv/RTUAHgT/knnhr/ALBVr/6KWugrn/An/JPPDX/YKtf/AEUtdBQAUUUUAFFFFABRRRQAUUUUAFFFFAHH/EwZ8GTH0mjP614jXuPxJGfBN1x0kj/9DFeHV5+K+P5HHiPiCiiiucwCiiigAooooAKKKKACiiigAooooAKKKKACiiigAoxRRRcAxSYpaKAEopaKAEopcUmKYCYopaKAEooxRQIKKKKYCUUtFACUUYopiCiiigAxSYpaKYhu2jbTqKLhYbg1WuF4zVuoLjpWtKXvIia0LulvnArpbdRsziuR059riursn3JivIzWny1LnpZZNNWZeEanqBQ1sp6cU5alrxbtbHruEXui3aXjIojmbcB0c/1q/vFYhpsV48D7GOV7e1YypX1Q+fkWpuGSmmSqonDDOaaZhUKBpctmSmmT3qmZxTDPVKArl0yVNY6g1jfRXC/wNkgdx3H5Vkmf3phm960pqUJKUd0J6qx7IrB1DKcqRkEd6Wsvw5cC58PWUgPSPZ/3ydv9K1K/QKc+eCl3R5rVnYKKKKsQUUUUAc/47/5J54l/7BV1/wCimo8Cf8k88Nf9gq1/9FLR47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABXH/ABMGfBkx9Joz+tdhXJfEkZ8E3XHSSP8A9DFRV+B+hM/hZ4dRRRXlHnBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAGKMUUUAJiilooASiloxTASkxS4ooASilpMUAFFFFAgpKWimAlFLSYoAKKKKYgoxRRTEJik206ii4WG7aMGnUUXCxUuF4zWjpb5wKpXHSpNOfa4rarHnw7RnB8tVM663UbM4q2I1PUCqNk+5MVorXyNRWdj6inaUUxjWynpxWlaXjIojmbcB0c/1qpSGsZLmVmVGKi7o294pDJWHFePA+xjle3tWgJwwzmsHTaLjUUtC0ZKQyVUMwphnFNQKuXDJ700yVSM9MM/vVKmK5rWOoNY30Vwv8DZIHcdx+VelqwdQynKkZBHevGzN716j4cuBc+HrKQHpHs/75O3+lfR5DNpzpPbf+vwOXELZmpRRRX0ZyhRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQByvxGGfA9+fRov8A0YteF17x8QVL+BtSA9Iz+UimvB64MV8a9DjxHxBRRRXMYBRRRQAUUUUgCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKYBRRRQAYpKWigBKTFLRTASilpKBBRRRQAYpMUtFO4CUUtFACUUtFFwEqvcdKs1Xufu1rR+NGdT4R1h94V1WnjiuV0/wC8K6uw+7XBm+52ZZ8RprUlRrUlfPs94Q1Eyg845qU0yhCauRRzMrEMTUhmqGUVXMhqlG4lpoWzN70wzVVLn1pN3vVqCC5YMvvTDLUO4Um6nyoLnrngo7vCtqf9p/8A0I10FYHgoY8JWXvvP/j7Vv19nhf4EPRfkcM/iYUUUVuSFFFFAHP+O/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtHjv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABXK/EYZ8D359Gi/8ARi11Vcz8QVL+BtSA9Iz+UimoqfA/Qmfws8HoooryjzgooooAKKKKQBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRTAKMUUUAJRS0lMBMUUtFACUUUUCCjFFFFwExRS0UwEopaKLgJRS0UXArXHSn2H3hTbn7tO0/wC8K6v+XDOd/wAQ6rTxxWotZlh92tNa+RrfGz6fD/w0SUhpaQ1gbkTKDzjmmRzMrEMTUtQSjvVJXJa1uiYzUwze9VDIaaXPrVqA7lozUwze9V93vSbhVcqFcmMteq+Cju8K2p/2n/8AQjXke6vXfBQx4Ssvfef/AB9q9bJ1+/fp+qMaz9036KKK+kOUKKKKACuf8d/8k88S/wDYKuv/AEU1dBXP+O/+SeeJf+wVdf8AopqADwJ/yTzw1/2CrX/0UtdBXP8AgT/knnhr/sFWv/opa6CgAooooAKKKKACiiigAooooAKKKKAOf8cru8F6oMZ/dA/kwNeAGvoPxkN3g7VR/wBO7Gvns9a4cV8SOTEboWikormOe4uaXNNozQA6ikzRmkAtFJmjNAC0UmaM0WAWikzRmiwC0UmaXNFgCiiigAooopAFFFFABRRRQAUUUUwCiiigAoopKACiiimAUlFFABRRRQIKKKKACiiigAooooAKr3PSrNVrn7ta0fjRFT4R+n/eFdXYfdrldP8AvCursPuVwZu/eOzLNzRWpKjWpK+fZ7ohplPPSmU0BDL0qk5xVyXpVGQ1rEiWw0tSbvemE0ma0sYuQ/dRupmaM0WFzHtHg8Y8J6f/ALhP/jxrcrH8KjHhbTv+uINbFfZUdKcfRGD3CiiitRBRRRQBz/jv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtHjv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFc/45Xd4L1QYz+6B/Jga6CsPxkN3g7VR/07samfwsmXws+fDRSHrRXlHnXFozSUUBcdmim5pc0ALRSZozSAWikzRmgBaKTNGaLALRSZozRYBaKM0UWAKKKKACiiikAUUUUAFFFFABRRRTAKKKKACkoopgFFFJQAUUUUCCiiigAooooAKKKKACiiloGVrnpT9P+8KZc/dqTT/vCuv/AJcM53/EOqsPu1pLWdYfcrRWvkKvxM+nofAiSkNLSHpWJsMqGXpU1QS9KuIim5xUZanSGoSa2MZOw/d70m6mZozTsRzD91ey+Dxjwnp/+4T/AOPGvF817Z4VGPC2nf8AXEGvXydfvJehFR3RsUUUV9AZBRRRQAVz/jv/AJJ54l/7BV1/6Kaugrn/AB3/AMk88S/9gq6/9FNQAeBP+SeeGv8AsFWv/opa6Cuf8Cf8k88Nf9gq1/8ARS10FABRRRQAUUUUAFFFFABRRRQAUUUUAY3ixd3hHVhj/l1f+VfPJWvonxQM+FNWH/TnL/6Ca+dga4sV8SOTEbobRSkU3Nc61OUXNLmkzRRYLi5opKKLBcWikopWC4tFJRRYdxaKSiiwXFozSUUWC47NGabmlzRYLi0UlFKw7i5ozSUUWC4uaM0maM0BcXNGaTNGaAFzRmkooAWikooC4tJmiigVwooooAKKKKACiiigAooooAKWkpaBhVW56Vaqrc9K1ofGjOr8JNp33hXV2P3K5XTuorqrH/V152b/ABHble5oLUlMWn14LPcEboaZT26GmU0BBL0qhL1q/L0rPl61tAznsQ0UUVqcwUUUoFIaR7h4bGPDOmf9eyfyrUrP0Fdvh3TAQQRaxdf90VoV9nS+CPoYsKKKKsAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKxvFi7vCOrDH/Lq/8AKtmsnxQM+FNWH/TnL/6CamXwsUtmfOxWm04GkIry15nmPyEozSZpc07CuLmjNJRRYLi0UlFKwXFopKKLDuLRSUUWC4tFJRRYLi5pc02jNFguOzRSZopWC4tGaSiiw7i5ozSUZosFxc0ZpM0ZoAXNGaTNFAC5opKKAFopKKAuGaKKKBBRRRQAUUUUAFFFFABRRRQAtFFFAyrc9Km077wqG56VPp3UV1P+Azn/AOXh1Vj9ytBaz7H/AFdaC18hV+Jn1FH4ESUjdDS0jdKyRqMqCXpU9QS9KuIihL1qCppetQ1ujnnuFFFFUZhXuPhsY8M6Z/17J/KvDwK900Fdvh3TAQQRaxdf90V6+UfHL0FPY0KKKK94zCiiigArn/Hf/JPPEv8A2Crr/wBFNXQVz/jv/knniX/sFXX/AKKagA8Cf8k88Nf9gq1/9FLXQVz/AIE/5J54a/7BVr/6KWugoAKKKKACiiigAooooAKKKKACiiigCpqlo2oaRe2SsEa4geIMegLKRn9a8cufhf4kt1JjS1uMdopsE/8AfQFe3UVnOlGe5nOnGe589XPg/wARWf8ArdIuiB3jTzB/47msae3mtpfLnhkifGdsilTj6Gvp6mvGki7XRWU9mGRWX1ZdGZPDJ7M+XqK+jbnwxoV4xafSLNmbqwhAJ/Ec1j3Pw08M3CsEtJbcn+KKZuPpuyKl4eXRmbw0ujPCqM165c/CGwZT9l1S5jbt5qK4/TbWNdfCPVYyPst/aTDnPmbkP8j/ADrN0ZroZuhNdDz3NGa6u6+HPia2biwEy/3opVP6Eg/pWNceHtZtCfP0q9jA7mBsfnjFQ4tbohwkt0ZuaM0YpCKnQkXNJupKSnYVx26l3UyinyiuPzRmmZozRyjuSUUzNGaXKFx9FN3UbqVh3HUUm6jNFguLRSZozRYLi0UmaM0WC4tFJmjNKwXFopM0Zp2C4tFJmjNFguLRSZozRYLi0ZpM0uaVh3FzRSUUWAWiiipGLVW56VZqrc1tQ+NEVfhLOndRXVWX+rrldO6iuqsv9XXmZt8Z35WaC0+mLT68JntCN0plPbpTDTQFeXpVCXrV+Y4BrPkdc1tDYzmrojxS7aaZBTGmA71qotmfKTcCk3AVUa5Aq7oFsdZ8QWVgMlZZRvwM4QcsfyBrSFCU5KK6j0R7xp0Zh0y0iPVIUU/goqzRRX2CVlY5AooopgFFFFAHP+O/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLR47/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFVNUtG1DSL2yVgjXEDxBj0BZSM/rVuihq+gHiNz8L/ElupMaWtxjtFNgn/voCsi58H+IrP8A1ukXRA7xp5g/8dzX0LRXO8NFnO8PHofMM9vNbS+XPDJE+M7ZFKnH0NRV9QvGki7XRWU9mGRWXc+GNCvGLT6RZszdWEIBP4jmpeH7MzeFfRnzlRXutz8NPDNwrBLSW3J/iimbj6bsisa5+ENgyn7LqlzG3bzUVx+m2odCZDw80eR5pc16FdfCPVYyPst/aTDnPmbkP8j/ADrIuvhz4mtm4sBMv96KVT+hIP6VDpzXQh0procpmjNaVx4e1m0J8/Sr2MDuYGx+eMVm4qHpuQ01uGaM0hFJQSLuo3U2inYVx+6jNMozRyhcfmlqPNLmlyjuPopmaXdSsO46im7qXdRYLi0UmaM0WC4tFJmjNFguLRSZozRYLi0UmaM0WC4tFJmjNFguLRSZozRYLi0UmaM0WC4uaXNJmilYdxaKSlpWGFLSUUhla56VY07qKrXNWdO6iumf8BnOv4p1Vl/q60FrPsv9XWgtfI1PiZ9TS+BD6RulLSN0rJGgyq8vSrBqvMcA1cdxMoS9aixUkjrmojIK3SZjKN2O20vAqFpgO9QtcgVag2Cii3uAr3rTozDplpEeqQop/BRXg+gWx1nxBZWAyVllG/AzhByx/IGvoGvcymk4qUn1MqrWiQUUUV7BiFFFFABXP+O/+SeeJf8AsFXX/opq6Cuf8d/8k88S/wDYKuv/AEU1AB4E/wCSeeGv+wVa/wDopa6Cuf8AAn/JPPDX/YKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBBcWVpdqVubWGZSMESRhgfzrGuvBHhq7H7zSIF5z+6zH/6CRXQUUnFPdCcU9zh7r4VeH5+YXvLc+iSAj/x4Gsa6+Dyk5tdYIH92WDP6g/0r1GiodKD6Gbo030PFbj4T+IIifKlsphnjbIQT+YFYtz4F8TWqkyaROwH/ADyxJ/6CTX0JRUuhEh4WDPmO6029siBd2k8BPA82Mpn86r7T6V7D8VuLbRX7LdHP/fNcnJa28kOXgjY46lRmvLxmKWGmotXuKGB572exxOKMV7L4f8DeHtX8OWd3cWbefKp3SRysMkMR0zjt6U25+EekvuNtf3kRPQPtcD9BXbGnOUVJdTF4WaPHMUuK9KuvhDeopNpqtvK3YSxmP+W6si6+GfiW3GY7eC55/wCWMw/9mxUunNdDN0JrocZg0c1s3PhbXrNiJtHvQAMllhLL+YyKy5I3ikMcqMjjqrDBH4VDut0Q4NbkVFPwKMClzCsMozT9tGyjmQ7MZmjNO2UbKLoVmNzRml2Gk2mnoGoZozRg0lOyFdi5ozTaM0+ULjqM03NGaXKFx+aXNMzS5pWHcfRTQaXNS0O46qtzVmqlwa0oL3yKj90u6d1FdTZf6oVzGmoSRXVWq7YxXk5q17Sx6eVp2uXFp9MWn14jPYGt0pppzdKz9Sv0sbcuxGeij1NVFNuyE2krsg1G7WMiJT8x5PtWW9171mS3zyOzsSWY81A07NXp08PZamLnc0nu/eqz3noapFiepppNbqkkTcsNcs1eufCfw5JBaya9dph5xstwRyE7t+J4/Cub8C/Du41qWPUdVieHTVOURuGn/DqF9+/avb440hiSKJFSNFCqqjAAHQCvUwmGs+eS9DKcug6iiivRMwooooAKKKKAOf8AHf8AyTzxL/2Crr/0U1HgT/knnhr/ALBVr/6KWjx3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+iloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAqC4srS7Urc2sMykYIkjDA/nU9FAHP3Xgjw1dj95pEC85/dZj/wDQSKx7r4VeH5+YXvLc+iSAj/x4Gu4oqHTi+hDpwe6PLrr4PKTm11ggf3ZYM/qD/Ssa4+E/iCInypbKYZ42yEE/mBXtVFS6MDN4em+h893PgXxNaqTJpE7Af88sSf8AoJNY91pt7ZEC7tJ4CeB5sZTP519OV558VuLbRX7LdHP/AHzWNWmoQck9jOWFj0Z49tPpSYrtpLW3khy8EbHHUqM11/h/wN4e1fw5Z3dxZt58qndJHKwyQxHTOO3pXBg8T9abUVaw6mAlDZnjWKMV7Hc/CPSX3G2v7yInoH2uB+grHuvhDeopNpqtvK3YSxmP+W6u10proYPDzXQ81xRg12d18M/EtuMx28Fzz/yxmH/s2Kxbnwtr1mxE2j3oAGSywll/MZFZuMluiHSkt0Y3NFSyRvFIY5UZHHVWGCPwpuBU3JsMop+BRto5kKwzNGafspNlHMgsxuaM07ZSbDRdBZiZozRtNGDT0FqGaM0lJTsFx2aKbmjNHKFx2aXNMzS5o5QuPzS0zNKDUtFXHUtNzS1Nh3K1zVvTuoqlcGr+moSRW9XTD6mUNaqOnsv9UKvrVO1XbGKuLXyE9z6mmrRSH01qdTW6VmixprK1G7WMiJT8x5PtU+pX6WNuXYjPRR6muRlvnkdnYksx5rrw9Fz97oZyn0Rpvde9V3u/es1p2aoyxPU13qijPmLr3noaga5Zqrk16B4F+HdxrUseo6rE8OmqcojcNP8Ah1C+/ftXRSoObtFEuVjpPhP4ckgtZNeu0w842W4I5Cd2/E8fhXptNjjSGJIokVI0UKqqMAAdAKdXt06apxUUYN3dwooorQQUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQB578WOdO0lf713j/AMdNcuf9R+Fdt8RdKv8AU7DTjYWj3LQ3O91TGQu08/59a5ddE1aSDaum3QbphoiO3vXzucUqk6sXGLaNqDSbud54H/5E7T/o/wD6G1dBWL4TsrnTvDFla3cZjnjVtyEg4yxI6exFbVe9SVoRT7IxYUUUVoAVHLBDOu2aJJFPGHUEfrUlFAGLdeEfD14G87R7TLZyUjCE591wc1jXPwv8NzqRFFc2x9Ypicf99Zrs6Klwi90S4Re6PNbn4QWzf8eurzR+0sIf+RFY9z8J9aib/RryzmTH8RZDn6YI/WvYqKzdCD6EOhB9Dwa48AeJ7YnOmNIo/iikVs/hnP6Vjz6Tqdopa5067hUDJMkDKB+Yr6RorN4SL2ZDw66M+YuKXANfSNzpen3oxdWNtOP+msSt/MVkXXgTw1dkF9KjQjODEzR/opArJ4OXRkug+h4JsFIYxXsl18KtDmbdBcXkH+yHDD9Rn9axrj4RTDJttZRvQSQFcfiGP8qj6vVWxDoy7HmJippjNdvc/DDxLApMa2lwR2imwT/30BWPd+EfEdmMzaNdkZx+6TzP/Qc0clVboylSa3RzpUim4xV24gntWC3NvLCx6CVCpP51B8p9KfO1ujJxIc0bqkKA0xkxVKSZDTQbqcGqE5BpDJVcl9hc1ifdxVWY7nAoaamRHdMCa1p0+X3iZSvodBpUeAK6KPoK56ymVFHNa0V6AB0r5rGxnOq3Y9zBVKcKdmzUWnkgdTWcLwkcVUvtWjs0JlbLkfKgPJ/+tXCqE27WOv6xF6R1NG7v7e0gaWV8AdAOpPoK4jUb+TUbkyycKOEQHhRWnbaN4h8VTiS00+eWP+Ftu2NQf9o4FdXpPwc1K4w+q38NqveOEeY/58AfrXrYTATjrbUmVS+55sWFW9P0vUdXm8rTrKe5bOD5SEgfU9B+Ne86R8NvDWk7X+xfa5hj95dHfyP9n7v6V1ccccMaxxIqIowqqMAD2FepDAv7TM3PseI6T8IdbvNr6jcQWEZxlc+ZJ+Q4/Wu+0P4Z+HtFljnaF725Q5ElycgH1Cjj8812VFdcMPThsiXJsOlFFFbEhRRRQAUUUUAFFFFAHP8Ajv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLR47/wCSeeJf+wVdf+imo8Cf8k88Nf8AYKtf/RS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAV578WOdO0lf713j/AMdNehVxfxF0q/1Ow042Fo9y0NzvdUxkLtPP+fWscQm6Ukt7ClscSf8AUfhXpXgf/kTtP+j/APobVwa6Jq0kG1dNug3TDREdvevQvCdlc6d4YsrW7jMc8atuQkHGWJHT2IrxcmpThOTkmjpryTtY2qKKK+gOcKKKKAI5YIZ12zRJIp4w6gj9aybrwj4evA3naPaZbOSkYQnPuuDmtqik0nuJpPc4y5+F/hudSIorm2PrFMTj/vrNZFz8ILZv+PXV5o/aWEP/ACIr0qiodKD6EOlB9Dx25+E+tRN/o15ZzJj+Ishz9MEfrWRceAPE9sTnTGkUfxRSK2fwzn9K95orN4aDIeHgfN0+k6naKWudOu4VAyTJAygfmKp8V9O1UudL0+9GLqxtpx/01iVv5isng10ZLw/ZnzdgGk2Cve7rwJ4auyC+lRoRnBiZo/0UgVj3Xwq0OZt0FxeQf7IcMP1Gf1rN4WotmS6EjxsximGKvTrj4RTDJttZRvQSQFcfiGP8qxrn4YeJYFJjW0uCO0U2Cf8AvoCl7GqjJ0ZdjiDGaYVIrorvwj4jsxmbRrsjOP3SeZ/6Dmsi4gntWC3NvLCx6CVCpP50e+t0ZShbcpYxRmpvlPpTSgNUprqZuPYj3Uu6hkxUZyDVpJk3aJg1G7ioDJTGmpqk2HOEx3OBW7pUeAK5+I7pgTXQ2Uyoo5rDMFJUeSJrhGva80joY+gqytZcV6AB0qYXhI4r5iVKfY+g+s0+5okgdTVS7v7e0gaWV8AdAOpPoKzr7Vo7NCZWy5HyoDyf/rVmW2jeIfFU4ktNPnlj/hbbtjUH/aOBW1DBTqPyD299UZmo38mo3Jlk4UcIgPCiqhYV6TpPwc1K4w+q38NqveOEeY/58AfrXd6R8NvDWk7X+xfa5hj95dHfyP8AZ+7+le7SwU7W2Rm5o8G0/S9R1ebytOsp7ls4PlISB9T0H4122k/CHW7za+o3EFhGcZXPmSfkOP1r26OOOGNY4kVEUYVVGAB7CnV1wwcF8WpDmzjdD+Gfh7RZY52he9uUORJcnIB9Qo4/PNdl0oorqjGMVaKIbuFFFFUAUUUUAFFFFABXP+O/+SeeJf8AsFXX/opq6Cuf8d/8k88S/wDYKuv/AEU1AB4E/wCSeeGv+wVa/wDopa6Cuf8AAn/JPPDX/YKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUANdFkUq6hlPUEZFZtz4a0O8YtcaRZO56sYF3H8cZrUooE0nueSfETwro2i2tjNp1n5Dz3BWTEjEEbSehJx+Fcqvh6KWPKXEinHcA16H8Vj/oOkrjrcsf8Ax2uVtv8AVfhXzuaV6lKraDsa08PSmtYlHS/AF/riXJsrq23QMqsJsrnIzxgGmXfwx8UW5+WwjnX1hmX+RINej/DsDy9UPfzUH/jtdtXrYROdCM5btHJUwlLmaR8zXfhTXLRmWfSL5NvVvJYr+YGDWSLd45SrAqwOCCMEGvq+opbaCY5lhjk/31Brqs+5hLAxezPmS3VyQACSegFdRpXhTXdRK+Tp8yp3eUbF/Xr+Fe5RWtvAcwwRRn1RAP5VNWEsMpP3mXDCKO7POLL4YzPGPt2peTnqtsuT/wB9NwPyNdHpfgTw7pLiWLT0nnHPnXJ81yfXngfgBXSUVcKFOHwo6oxUVZAAAAAMAdqKKK2GFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/47/wCSeeJf+wVdf+imo8Cf8k88Nf8AYKtf/RS0eO/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFNdFkUq6hlPUEZFOooAy7nw1od4xa40iydz1YwLuP44zXnHxE8K6NotrYzadZ+Q89wVkxIxBG0noScfhXrdee/FY/6DpK463LH/wAdrnxWlGTW9hKnGT1R54vh6KWPKXEinHcA1b0vwBf64lybK6tt0DKrCbK5yM8YBq9bf6r8K7b4dgeXqh7+ag/8drwstxFWrX5Ju6Lr4Sjy3SPOLv4Y+KLc/LYRzr6wzL/IkGsO78Ka5aMyz6RfJt6t5LFfzAwa+maK+i5LbM4ZYKD2Z8oC3eOUqwKsDggjBBq/bq5IABJPQCvpuW2gmOZYY5P99QaSK1t4DmGCKM+qIB/KpnTct2QsDZ/EeG6V4U13USvk6fMqd3lGxf16/hXY2XwxmeMfbtS8nPVbZcn/AL6bgfka9HorJYSnvLU6YUYxOb0vwJ4d0lxLFp6Tzjnzrk+a5PrzwPwArpAAAABgDtRRXTGKirJGwUUUUwCiiigAooooAKKKKACiiigAooooAK5/x3/yTzxL/wBgq6/9FNXQVz/jv/knniX/ALBV1/6KagA8Cf8AJPPDX/YKtf8A0UtdBXP+BP8Aknnhr/sFWv8A6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA86+LDhbbSFPUzuR/3z/8AXrmLf/VfgK9H8YeEx4qtrVBeG2ktnLo2zcDkYwRkVgw/D7Uol2HUbdlxjOwg14WZ4KtWqc1NXNqVVR0Ze+HWDbamR1+0KP8Ax0V2tYfhnw9/wj1rPEbjz3mk3khdoHGMVuV6uEpyp0YwlukZN3dwoooroEFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAc/47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtHjv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFedfFhwttpCnqZ3I/wC+f/r16LXNeMPCY8VW1qgvDbSWzl0bZuByMYIyKxxEHOlKK6jTtqecW/8AqvwFdv8ADrBttTI6/aFH/joqjD8PtSiXYdRt2XGM7CDXT+GfD3/CPWs8RuPPeaTeSF2gcYxXjZdga9HEc81oa1KqnFJG5RRRXvmIUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/AI7/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0eO/8AknniX/sFXX/opqPAn/JPPDX/AGCrX/0UtAHQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAVz/jv/knniX/ALBV1/6Kaugrn/Hf/JPPEv8A2Crr/wBFNQAeBP8Aknnhr/sFWv8A6KWugrn/AAJ/yTzw1/2CrX/0UtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUVl+Jb2fTfCusX1qwW4trKaaJiAcMqEg4PXkVz1laeNb7RLfVR4gt4r6WBZlsPsaG3yVBCM33yT0LAjqcDtQB2tFZnh3WE8QeHdP1eOMxLdwLL5ZOShI5Ge+DkVieKdR12PxP4f0jRbmC3W/S6M8s0QfYEEZDAdyNxAGcZPOQKAOuorkDc634b1vS4NR1X+1tO1Oc2oeS3SKWCbYzqRsADIQjDBGRxyea5ix8XeI7LwPpOr6lqaT3etTJbQBLAyLbDDs0pSIbpGKoTtGBkjsCaAPVqK830/wAV31rrWnwx6nqmt293OsE8dzoU1s0G7gSK4iVdoOMhuxznjnVs59f8VT395Za0NJ0+3u5bS2jitUleUxMUZ5C4OAWDYVQDgDnmgDqLTUrS+ub23t5d8tlKIbhdpGxyiuByOflZTxnrVuuI+H51D+0vFo1QwNerqqrI8AIR8W8IDAEnGQAcZOM4rt6ACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/AB3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+ilo8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiivPfCp8Y+KPC1lrFx4jjsJZoyYooLKORWAJAaQtyc9cLtwMc5oA9CorF8LazPrejGa8ijivbe4ltLpIiSnmxOUYrnnacZH1raoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/ACTzxL/2Crr/ANFNXQVz/jv/AJJ54l/7BV1/6KagA8Cf8k88Nf8AYKtf/RS10Fc/4E/5J54a/wCwVa/+ilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAoorL1XXrXR5I0uLbUpjICQbTT57gD6mNGA/GgCDxD4kTw4tvLcabfXFtLIkbz24jKQl3VF37nB5LDoDW3XJ/EFxJ4JdwGAa7sSAykH/j6i6g8iusoA5fVvGMujpezXHhfXGtLQO0l0n2byyi5JcZmBxgZ5APtUt14xtYGs4IdO1K8vrq2W6+xW8KmWKNuhkywVecjluSDjOKreLv+JxqWk+Fk5S8k+1XwHa1iIJB/wB9yi+4LU7R/wDkpHijP3vslhj/AHf339c0AbOi63Z67ZvcWnmqYpGhmhmQpJDIOqOp6HkH6EEcGtGuV8N4/wCEz8Z7fu/bLbOP732WPP8ASuqoAKKKKACiiigArl7rxzYwXF2sGm6re21k7R3V5a22+KJl+8OoZtvfaGxWle+IbSw1BbKW11N5G24eDTZ5Yxn1kRCo9+eO9ZXw6x/whUOcZ+1Xm/Pr9plzmgDpba6gvLSG7tpUlt5kEkcinKspGQQfTFcz/wALB0vYLs2WpjSC+waqbf8A0brjdnO7Znjft29845pngC3F18K9HtmZlSWwCBh1CkEAj8DXNahe67YeErLwDNoJN/e250mC8WaM27xiPa023dvGEG4grgHAzyMgHaav4utdJvns0sNQv5oYRcXAsoQ4gjJOGbJHXa2AMk4PFX5tYQ6JFqun2txqcUyJJDHabN8iNghhvZRjBzyRWTqOny6pol3p3hzWYrC+gxaXF19nEr5WPhGzjBw6nPOM8dam8C3UV14K0vyrUWqwRm08lX3BDCxiOGPUZQ4PcUAaOh6xb6/odlq1qkqQXcQlRZQA4B9QCRn8a0K5X4a/8k28Pf8AXkldVQAUUUUAYvjCN5vBOvRRIzyPp1wqqoyWJjbAA7msHTPC2uDw9aadH4puYNNa2RDGbVTcxoVGUWYnjHIBKlh65Ga7isvVdetdHkjS4ttSmMgJBtNPnuAPqY0YD8aAOf8A+Ess9BvZvDWleGNbvY9JhiQmwiieNFKAquWkB3Y7EZ78ggnGubqXx5rnhTVdOj1PSUjS+MNw6xsyMBEPmVWdcE71Ktg8NwMA10PglxJdeKXwwZtackMpDAGGHGQeRxineA8f2frG37n9t3+30/17Z/XNAGXMLrT/ABnox8STX+ps0pi0+a3tYoLOKZ0fJZfMLl9isM4wATjk1Nqeh6f4c+G1rZahfXBOlbHtru1jAm88NhPLQkgsS23acg7iDwaveMv+Qh4S/wCw4n/pPPWbrVhd+LfHQ05dTm0+00KKK7zAiM8txLvCt86suEVTjg8t6igCS8S60gWOr+J9aub7ypVWz060sxE0s7AgBlVm8xwM8ZCjBPbIq6Wtxda5qUGh6je+H7mc/bLnTNTsEl5c4M0RD45I5wzDd1Azzo6RdrLNqUfiWeC4uPDV3uTUWAiUq8AYOwHyhgkjKe3cAZp3h8XPiDxI/iySF7awW0az06KRdsk0bOrtMw/hDFF2g84GT1oAv+FvC48MjUidQnvpL+6+1SSzgbt5RVbpxyVJ4AAzgDAroKKKACiiigDI1vxFa6G9tC8F1d3l0WFvaWke+WTaMscEgADIySQOR60ui+IbXW2uYY4bm1vLUqLi0uo9kse4ZUkAkEHBwQSDg+lZc3/JWLLd/wBASfZn/rvFnH6UQY/4Wxfbf+gJb7sf9d5sf1oA2tb1m10DSJ9SvN5iiAwka7nkYnCoo7sxIAHqaj1bX7XRNOhu7yOcPO6xQ20aeZNJIwyEVVJy3B744JzjmuX1qwu/FvjoacupzafaaFFFd5gRGeW4l3hW+dWXCKpxweW9RWdDqN/N470TSdVnS6uNL1SeEXaoE84PZNIhZRwHAYg444yAM0AddYeLIL6a5tG03UbXUYYDcCxuYlSWZBxlDuKNzgfe4JGcZqmfGzQ39ha3nhjXLP7dcLbRSTLbld5BPO2YnAAJOAeAaXV8f8LJ8L4+99jv8/7v7j+uKba/8T74g3N396y0GM2sPo11IA0p/wCApsX/AIG1AHWUUUUAFFFFABRRRQBif8JIi+KY9Bm02+hkmjkkgunEfkyhAu7aQ5bjeByorad1jRndgqqMkk4AFctq/wDyUnwv/wBeWof+0K6HULu0sNPuLq/lSK0iQtM8n3VXuT7UAYeleNtP1W+tLdLPULeO+Vmsbm5gCRXYA3fIckj5QWG4LkDIpNS8cafpl3eRtZ6jcW9gQL68t4A8NqSAx3HIJwpBO0NgHnFZXiO1vNL8VeH9YluY7jR4ryO0g09IRH9meZfJWUMPv43EbSBgMcdKwptK1/XrHxbdaHfwWekXt1PG9lMm6Sdo/wB1MVk/5Zb9hHIbHXjNAHq6srorowZWGQQcgilrN8P39vqnhvTL+0iaK3ubWKWKNuqKyggfgOK0qAOf8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOilo8d/8k88S/8AYKuv/RTUeBP+SeeGv+wVa/8AopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACsTxD4kTw4tvLcabfXFtLIkbz24jKQl3VF37nB5LDoDU+q69a6PJGlxbalMZASDaafPcAfUxowH41j/EFxJ4JdwGAa7sSAykH/j6i6g8igDrK5fVvGMujpezXHhfXGtLQO0l0n2byyi5JcZmBxgZ5APtXUVyfi7/AInGpaT4WTlLyT7VfAdrWIgkH/fcovuC1AFm68Y2sDWcEOnaleX11bLdfYreFTLFG3QyZYKvORy3JBxnFaWi63Z67ZvcWnmqYpGhmhmQpJDIOqOp6HkH6EEcGsbR/wDkpHijP3vslhj/AHf339c0eG8f8Jn4z2/d+2W2cf3vssef6UAdVRRRQAUUUUAFFFZN74htLDUFspbXU3kbbh4NNnljGfWREKj35470AZt145sYLi7WDTdVvbaydo7q8tbbfFEy/eHUM23vtDYrora6gvLSG7tpUlt5kEkcinKspGQQfTFc18Osf8IVDnGftV5vz6/aZc5qPwBbi6+Fej2zMypLYBAw6hSCAR+BoAf/AMLB0vYLs2WpjSC+waqbf/RuuN2c7tmeN+3b3zjmrmr+LrXSb57NLDUL+aGEXFwLKEOIIyThmyR12tgDJODxXF6he67YeErLwDNoJN/e250mC8WaM27xiPa023dvGEG4grgHAzyM9bqOny6pol3p3hzWYrC+gxaXF19nEr5WPhGzjBw6nPOM8daANabWEOiRarp9rcanFMiSQx2mzfIjYIYb2UYwc8kUuh6xb6/odlq1qkqQXcQlRZQA4B9QCRn8azvAt1FdeCtL8q1FqsEZtPJV9wQwsYjhj1GUOD3FVvhr/wAk28Pf9eSUAdVRRRQAUUUUAFc9qHi62s9Un06103U9TuLYK1yLGAOINwyAxZgMkc7Rk4xxzV3VdetdHkjS4ttSmMgJBtNPnuAPqY0YD8ax/BLiS68Uvhgza05IZSGAMMOMg8jjFAHQaVqlnrWmQ6hYS+bbTAlWIIIIOCCDyCCCCD0Irybwv/buhaH4c0mC/wBUsLfUwIxJNYxTJBMwdiqs0iuvCFsMjAZ49B3fgPH9n6xt+5/bd/t9P9e2f1zR4y/5CHhL/sOJ/wCk89AFsnS/AfhNmJne2tssT/rJriV2z7bpHdvbk9hVvV9etdE0yK8vY5w0zpFFbRpvmklbpGqg4Lde+OCc45rl9asLvxb46GnLqc2n2mhRRXeYERnluJd4VvnVlwiqccHlvUVb0i7WWbUo/Es8FxceGrvcmosBEpV4AwdgPlDBJGU9u4AzQBr6P4mttWvZrB7O90+/hjErWt7GEcxk4DqVJVhnjgnB64rarkPD4ufEHiR/FkkL21gto1np0Ui7ZJo2dXaZh/CGKLtB5wMnrXX0AFFFFABWRrfiK10N7aF4Lq7vLosLe0tI98sm0ZY4JAAGRkkgcj1rXrlZv+SsWW7/AKAk+zP/AF3izj9KANTRfENrrbXMMcNza3lqVFxaXUeyWPcMqSASCDg4IJBwfSptb1m10DSJ9SvN5iiAwka7nkYnCoo7sxIAHqaxYMf8LYvtv/QEt92P+u82P61m61YXfi3x0NOXU5tPtNCiiu8wIjPLcS7wrfOrLhFU44PLeooA6jVtftdE06G7vI5w87rFDbRp5k0kjDIRVUnLcHvjgnOOaqWHiyC+mubRtN1G11GGA3AsbmJUlmQcZQ7ijc4H3uCRnGa5GHUb+bx3omk6rOl1caXqk8Iu1QJ5weyaRCyjgOAxBxxxkAZro9Xx/wALJ8L4+99jv8/7v7j+uKAEPjZob+wtbzwxrln9uuFtopJltyu8gnnbMTgAEnAPANdXXJ2v/E++INzd/estBjNrD6NdSANKf+ApsX/gbV1lABRRRQAUUUUAFYn/AAkiL4pj0GbTb6GSaOSSC6cR+TKEC7tpDluN4HKituuV1f8A5KT4X/68tQ/9oUAdS7rGjO7BVUZJJwAK5vSvG2n6rfWlulnqFvHfKzWNzcwBIrsAbvkOSR8oLDcFyBkVuahd2lhp9xdX8qRWkSFpnk+6q9yfauN8R2t5pfirw/rEtzHcaPFeR2kGnpCI/szzL5Kyhh9/G4jaQMBjjpQBq6l440/TLu8jaz1G4t7AgX15bwB4bUkBjuOQThSCdobAPOK6VWV0V0YMrDIIOQRXlE2la/r1j4tutDv4LPSL26njeymTdJO0f7qYrJ/yy37COQ2OvGa9G8P39vqnhvTL+0iaK3ubWKWKNuqKyggfgOKANKuf8d/8k88S/wDYKuv/AEU1dBXP+O/+SeeJf+wVdf8AopqADwJ/yTzw1/2CrX/0UtdBXP8AgT/knnhr/sFWv/opa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAM/WtIt9d0xrC6eVImlilJiIDZjkWQdQeMqM+2akvNPW8urGdri4iNnMZQkT7VlJRk2uMcr82ceoB7VcooAzrfRre31291jfLJdXcUcJ3kFY40zhU44BLMTnOSfpVXVvDNtql/HqMd5e6ffpF5P2mykCs0ec7WDBlYA5IyOMnHWvnL4vf8lU1n/th/6IjrjVrshhOaKdzqWGuk7n2Romh2mg2clvamZ2llaaeaeQvJNI2MszHqeAPQAACtKvitamWr+pf3vwJdC3U+zqK+NlqVaPqP978CHTt1PsSivj9akFP6j/e/D/gmbVj68rmLvwPZXE92YNS1Wxtr12kurS0uAkUrN948qSpbvtK56183CpBS+o/3vwM3Kx9RQ6LbWt5YzWzzQQ2Vq1rFaxviHYdmCV7kBAAc8An1oTRLZfEUuuO8st21uLaMORshTOWCADgscEkk52joBXzAKeKPqX978CHWt0Po3UfCNtfalNf22palpk9yoW5NhMEE+BgFgVPzAcblwcY54Fa+m6daaRptvp9jCIbW3QJGgJOB9TyT6k8mvl4U4UvqX978DN4m3Q+mdD0i30DQ7PSbV5XgtIhFG0pBYgepAAz+FaFfLIp4pfU/7xDxlvs/ifUdFfLwp4pfVfMzePt9n8f+AfT1FfMgpwqfq3mQ8yt9n8f+Ae86h4St7zUp9QtdS1LTLm5VVuWsZlQT7RgFgysMgcbhg4xzwKntfDNjYWulW1lJc28GmytKkccpxMWVgfMJyXyXLf72DXgIp4pfV/Mh5pb7H4/8A+hNT0i31WbTpZ3lVrC6F3FsIALhHTDZB4w56Y7VU1bwxbapqEeoxXl7p2oJH5P2qykCs8ec7GDBlYZ5GRxk4xmvCRTh0qXQ8yXm9vsfj/wD2G/+HemX2jRaYL/UoIxd/bJpUlR3upf70vmKwfkA4xjgcYArW0rRLrTbpppvEOq6ipQqIbzyNgOQdw2RKc8Y645PFeFinipdO3Un+2f7n4/8A+iaK+ehT1qHGwv7a/ufj/wD6CorwIVKtZuVg/tn+5+P/APYta8PWutvbTPNc2t5aljb3drJslj3DDAEggg4GQQQcD0qnb+D7O307VLf7fqL3Wpx+Xc6g84+0EBSF2sBhduTgAAAk8V5ctSrWUq1uhaze/2Px/4B6lqnha21K9iv4r2+0+/ji8n7VZyhXePOdrBgysM8jI4ycYzUH/CE6R/Y/wDZ/wDpO77T9s+2eeftH2j/AJ6+Z13Y49McYxxXnK1MtYyxlvslrNL/AGPx/wCAej6T4Yt9L1GTUpb2+1G/eLyRc3soZkjznYoUKqgkAnAycDPSreiaNb6Fp32O2eWQGWSaSWYgvJI7FmZiAASST2rzJalWsZZjb7P4/wDANVj7/Z/H/gHrNFeWLUq1hLOOX7H4/wDANo4q/Q9OorzZalWueWf8v/Lv8f8AgG0at+h6JRXALUq1hLiW3/Lr/wAm/wCAbRVzqrnSLe61yw1Z3lE9lFNFGqkbSJNm7IxnPyDHI79aSXRra4vL6e5eaeK9tltpbWVt0Owb84XHVt5B9QB6Vza1MlZ/60/9Ov8Ayb/gG8aF+pZsfBNlZ3dpNNqOqX0Nk2+ztry4DxQNjAIGAWIBIBctjtSXngexuri8aLUdUs7a+cyXdna3ASKdm+8TwWUt32Fc/WrGmf8AH9F+P8jW/XuZZj/r1F1eXls7b36J9l3M6tPkdrkdvbw2ltFbW8axQQoI441GAqgYAHsBUlFFeiZnP+O/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLR47/AOSeeJf+wVdf+imo8Cf8k88Nf9gq1/8ARS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFZ+taRb67pjWF08qRNLFKTEQGzHIsg6g8ZUZ9s1oUUAU7zT1vLqxna4uIjZzGUJE+1ZSUZNrjHK/NnHqAe1RW+jW9vrt7rG+WS6u4o4TvIKxxpnCpxwCWYnOck/StGvlf4vf8lU1n/th/wCiI61o0vaStc0pU/aO1z6N1bwzbapfx6jHeXun36ReT9pspArNHnO1gwZWAOSMjjJx1qzomh2mg2clvamZ2llaaeaeQvJNI2MszHqeAPQAACvjdamWun6l/e/A2eGt1PtSivjFalWn9R/vfgZujbqfZNFfHa1ItP6j/e/D/gkOFj7Aor5DFSCl9R/vfh/wTNux9I3fgeyuJ7swalqtjbXrtJdWlpcBIpWb7x5UlS3faVz1rVh0W2tbyxmtnmghsrVrWK1jfEOw7MEr3ICAA54BPrXy6KeKPqP978DN1LdD6fTRLZfEUuuO8st21uLaMORshTOWCADgscEkk52joBVDUfCNtfalNf22palpk9yoW5NhMEE+BgFgVPzAcblwcY54FfOQp4pfUv734EPEW6H1DpunWmkabb6fYwiG1t0CRoCTgfU8k+pPJqHQ9It9A0Oz0m1eV4LSIRRtKQWIHqQAM/hXzMKeKX1P+9+Bm8XbofU1FfLgpwpfVPMh4632fxPqGivmEU8UvqvmQ8wt9n8f+AfTdc/qHhK3vNSn1C11LUtMublVW5axmVBPtGAWDKwyBxuGDjHPArwYU4Uvq3mQ8zt9j8f+Ae/WvhmxsLXSraykubeDTZWlSOOU4mLKwPmE5L5Llv8Aewas6npFvqs2nSzvKrWF0LuLYQAXCOmGyDxhz0x2r57FPFT7DzIebW+x+P8AwD3bVvDFtqmoR6jFeXunagkfk/arKQKzx5zsYMGVhnkZHGTjGazL/wCHemX2jRaYL/UoIxd/bJpUlR3upf70vmKwfkA4xjgcYArx4dKeKl0vMl5x/c/H/gHumlaJdabdNNN4h1XUVKFRDeeRsByDuGyJTnjHXHJ4rYr52FSCpcLE/wBs/wBz8f8AgH0LRXz6tSis3oH9s/3Px/4B77WTrXh611t7aZ5rm1vLUsbe7tZNkse4YYAkEEHAyCCDgeleOrUq1k6lug1nH9z8f+Aeo2/g+zt9O1S3+36i91qcfl3OoPOPtBAUhdrAYXbk4AAAJPFTap4WttSvYr+K9vtPv44vJ+1WcoV3jznawYMrDPIyOMnGM15atSrWUsTboWs1v9j8f+Aejf8ACE6R/Y/9n/6Tu+0/bPtnnn7R9o/56+Z13Y49McYxxU+k+GLfS9Rk1KW9vtRv3i8kXN7KGZI852KFCqoJAJwMnAz0rzhalWsZY632fx/4Bqsyv9n8f+Aem6Jo1voWnfY7Z5ZAZZJpJZiC8kjsWZmIABJJPatGvJlqZawlmtvsfj/wDWONv9n8T1OivMVqVawlnfL/AMu/x/4BtHEX6HpNFedrUq1hLiLl/wCXf4/8A2jK539Z9zpFvda5Yas7yieyimijVSNpEmzdkYzn5Bjkd+tcqtSrWL4nt/y6/wDJv+Abxp36nSS6NbXF5fT3LzTxXtsttLaytuh2DfnC46tvIPqAPSsqx8E2Vnd2k02o6pfQ2Tb7O2vLgPFA2MAgYBYgEgFy2O1VkrQ0z/j+i/H+Rq8PxJ7atGl7K3M0vi7v0NHh7RbuV7zwPY3VxeNFqOqWdtfOZLuztbgJFOzfeJ4LKW77CufrXR29vDaW0VtbxrFBCgjjjUYCqBgAewFSUV9QcwVz/jv/AJJ54l/7BV1/6Kaugrn/AB3/AMk88S/9gq6/9FNQAeBP+SeeGv8AsFWv/opa6Cuf8Cf8k88Nf9gq1/8ARS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHyv8Xv+Sqaz/2w/wDREdcatfYt54Y0DULt7q90PTLm5kxvmmtI3dsDAySMngAfhUP/AAhvhb/oWtH/APAGL/4mu6GLUYpWOuOISilY+RlqZa+tP+EO8Mf9C5pH/gDF/wDE0f8ACH+Gf+hc0j/wCj/+JqvrsexLrp9D5QWpVr6r/wCER8Nf9C7pP/gFH/8AE0v/AAiXhr/oXtJ/8Ao//iaf12PYzdRM+V1qQV9Sf8In4b/6F/Sv/AKP/Cl/4RTw5/0ANK/8A4/8KPr0exm3c+XhUgr6d/4RXw7/ANADS/8AwDj/AMKP+EV8O/8AQB0v/wAA4/8ACj67HsZuNz5lFPFfTH/CLeHv+gDpf/gHH/hR/wAIv4f/AOgFpn/gJH/hS+ux7Gbot9T5qFOFfSn/AAi/h/8A6AWmf+Akf+FH/CMaB/0A9M/8BI/8KPrkexm8M31Pm4U8V9Hf8IzoH/QD03/wEj/wpf8AhGdB/wCgJpv/AICR/wCFT9bj2M3g5PqfOQp4r6K/4RrQf+gJpv8A4Cp/hR/wjehf9AXTf/AVP8KX1pdjN4GT6nzwKcK+hv8AhG9C/wCgLp3/AICp/hR/wjmh/wDQF07/AMBU/wAKX1ldiHl0n9o+exTxX0D/AMI5of8A0BtO/wDAVP8ACl/4R3Q/+gNp/wD4Cp/hU/WF2IeVzf2keACnDpXvv/CO6J/0B9P/APAVP8KP+Ee0T/oD6f8A+Ayf4VLrLsQ8pm/tI8FFPFe8f8I/ov8A0CNP/wDAZP8ACj/hH9F/6BFh/wCAyf4VLqJkPJ5/zI8LFPWvcf7A0b/oE2H/AIDJ/hS/2Do//QJsf/AdP8Kzcri/saf8yPEhUq17R/YWkf8AQKsf/AdP8KX+w9J/6Bdl/wCA6f4Vm43Gsnn/ADI8bWpVr2D+xNK/6Bll/wCA6/4Uf2NpX/QMs/8Avwv+FYyot9S1lM19pHki1Mterf2Ppf8A0DbP/vwv+FL/AGRpn/QOtP8Avwv+FYywkn1NFlk19o8tWpVr03+ydN/6B9r/AN+V/wAKX+ytO/58LX/vyv8AhXPLLpP7RrHASXU83WpVr0T+zNP/AOfG2/79L/hR/Zth/wA+Vt/36X/CueWUTf2kbRwrXU4BalWu7/s6x/587f8A79L/AIUv9n2X/Ppb/wDfsf4VzyyKpL7aN402jiFqVa7L7BZ/8+kH/fsUv2K0/wCfWH/v2K55cN1X9tHRF2ORWpkrqPsdr/z7Q/8AfApfslt/z7xf98CsnwxW/wCfi+5m8a6XQxtM/wCP6L8f5Gt+o1ghRgyRIrDuFAqSvoMqwEsDRdOTvd3/AAX+RnVqKbugooor0zI5/wAd/wDJPPEv/YKuv/RTUeBP+SeeGv8AsFWv/opaPHf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK+V/i9/yVTWf+2H/oiOvqisq88MaBqF291e6HplzcyY3zTWkbu2BgZJGTwAPwrajUVOV2a0qihK7PjpamWvrn/hDfC3/QtaP/4Axf8AxNL/AMId4Y/6FzSP/AGL/wCJrq+uR7GzxCfQ+S1qVa+r/wDhD/DP/QuaR/4BR/8AxNL/AMIj4a/6F3Sf/AKP/wCJp/XY9jN1U+h8qLUi19Uf8Il4a/6F7Sf/AACj/wDiaP8AhE/Df/Qv6V/4BR/4UfXo9jNzufLYqQV9Q/8ACKeHP+gBpX/gHH/hR/wivh3/AKAGl/8AgHH/AIUfXY9jNq58xCnivpr/AIRXw7/0AdL/APAOP/Cl/wCEW8Pf9AHS/wDwDj/wo+ux7GbptnzOKeK+lf8AhF/D/wD0AtM/8BI/8KX/AIRfw/8A9ALTP/ASP/Cl9cj2M3Qb6nzWKeK+kf8AhGNA/wCgHpn/AICR/wCFH/CM6B/0A9N/8BI/8KX1yPYzeFb6nziKcK+jf+EZ0H/oCab/AOAkf+FH/CNaD/0BNN/8BU/wpfW12M3gpPqfOop4r6H/AOEb0L/oC6b/AOAqf4Uv/CN6F/0BdO/8BU/wqfrS7EPL5PqfPIpwr6E/4RzQ/wDoC6d/4Cp/hR/wjmh/9AbTv/AVP8KTxC7Gby2b+0fPwp4r3/8A4R3Q/wDoDaf/AOAqf4Uf8I7on/QH0/8A8BU/wqfbrsQ8qm/tI8CHSnivev8AhHtE/wCgPp//AIDJ/hS/8I/ov/QI0/8A8Bk/wqXVRDyif8yPBxUgr3T/AIR/Rf8AoEWH/gMn+FH9gaN/0CbD/wABk/wqHO5P9jz/AJkeHLUor23+wdH/AOgTY/8AgOn+FH9haR/0CrH/AMB0/wAKzeof2NP+ZHi61Kteyf2HpP8A0C7L/wAB0/wpf7E0r/oGWX/gOv8AhWUqbZSyif8AMjx9alWvW/7G0r/oGWf/AH4X/Cl/sfS/+gbZ/wDfhf8ACsZYZvqaLK5r7SPKVqVa9S/sjTP+gdaf9+F/wo/snTf+gfa/9+V/wrCWBk+ppHLpL7R5ktTLXpH9lad/z4Wv/flf8KX+zNP/AOfG2/79L/hXPLK5v7SNo4OS6nna1Ktd/wD2bYf8+Vt/36X/AApf7Osf+fO3/wC/S/4VzyyWo/tI3jQa6nCLUq12/wDZ9l/z6W//AH7H+FH2Cz/59IP+/Yrnlw9Vf20bxjY41alWuu+xWn/PrD/37FH2O1/59of++BWD4Zqv/l4vuZvGokculaGmf8f0X4/yNbP2S2/594v++BTlghRgyRIrDuFArTDcOVaNaFRzXutP7mavEJxasSUUUV9acoVz/jv/AJJ54l/7BV1/6Kaugrn/AB3/AMk88S/9gq6/9FNQAeBP+SeeGv8AsFWv/opa6Cuf8Cf8k88Nf9gq1/8ARS10FABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHP+O/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0eO/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/47/wCSeeJf+wVdf+imroK5/wAd/wDJPPEv/YKuv/RTUAYfgvxp4VtfAvh63uPEujQzxaZbJJHJfxKyMIlBBBbIIPGK3P8AhO/B/wD0Neh/+DGH/wCKrjdM0rwhpHw58L6he+DrHUbm9tbOELDp8DyyyyRA5JfaDk5ySa0tPsPBN3qsWmXngG10u7nVmgS+0q3Am28sFZNykgc4znHNAHQf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE1hjTPAr6jPZR+CdPkkg1BLCRk0qAqrtEJQ544TBAJ9T070Abn/Cd+D/APoa9D/8GMP/AMVR/wAJ34P/AOhr0P8A8GMP/wAVR/wgng//AKFTQ/8AwXQ//E1n634c8D6Bo8+p3XhHR3hh27li02EscsFGMgDqR3oA0P8AhO/B/wD0Neh/+DGH/wCKo/4Tvwf/ANDXof8A4MYf/iqpjwr4NOsvpn/CGaWHW3W4886XD5RBYrtDY+98ucY6EVc/4QTwf/0Kmh/+C6H/AOJoAP8AhO/B/wD0Neh/+DGH/wCKo/4Tvwf/ANDXof8A4MYf/iqP+EE8H/8AQqaH/wCC6H/4mj/hBPB//QqaH/4Lof8A4mgA/wCE78H/APQ16H/4MYf/AIqj/hO/B/8A0Neh/wDgxh/+KqK58F+D7a0mn/4RDRpPKRn2R6dCWbAzgZA5NZFxp/w9tvBn/CVN4V0dtO+yLdgLpsG8qwBCgYxuOQMZ696ANz/hO/B//Q16H/4MYf8A4qj/AITvwf8A9DXof/gxh/8Aiqx73SPANl4XXxAfCekS2bxRyxrFpsBeQSFQgAIAySwHXvWx/wAIJ4P/AOhU0P8A8F0P/wATQAf8J34P/wChr0P/AMGMP/xVH/Cd+D/+hr0P/wAGMP8A8VWfqfhzwRpMmnxz+EdHY312tpF5emwnDlWYFsgcYQ9M9uKNJ8OeCNZjupLfwjo6i2u5bR/M02EZeNtrEYB4yOP5UAaH/Cd+D/8Aoa9D/wDBjD/8VR/wnfg//oa9D/8ABjD/APFVzeq2/gPTL+6tE8CWt81kivePZ6PC62wI3DdkAk7ecKCcY45q9ZaN4A1HVUsLTwto0pksY9QjmXTYfLeJ2IXBxnPy56dCKANb/hO/B/8A0Neh/wDgxh/+Ko/4Tvwf/wBDXof/AIMYf/iqP+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4muVtX+Hl1c2oXwTZpY3c4t7bUpNHhFvNIThQpxuwSMAlQD2PNAHVf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVZ9p4c8D3usajpkfhHRxNp/leazabDtbzF3DbxnoOcgVof8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTQAf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVH/CCeD/+hU0P/wAF0P8A8TR/wgng/wD6FTQ//BdD/wDE0AH/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTQAf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVH/CCeD/+hU0P/wAF0P8A8TR/wgng/wD6FTQ//BdD/wDE0AH/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTQAf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVH/CCeD/+hU0P/wAF0P8A8TR/wgng/wD6FTQ//BdD/wDE0AH/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTQAf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVH/CCeD/+hU0P/wAF0P8A8TR/wgng/wD6FTQ//BdD/wDE0AH/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTQAf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVH/CCeD/+hU0P/wAF0P8A8TR/wgng/wD6FTQ//BdD/wDE0AH/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTQAf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVH/CCeD/+hU0P/wAF0P8A8TR/wgng/wD6FTQ//BdD/wDE0AH/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTQAf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVH/CCeD/+hU0P/wAF0P8A8TR/wgng/wD6FTQ//BdD/wDE0AH/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUf8IJ4P/6FTQ//AAXQ/wDxNH/CCeD/APoVND/8F0P/AMTQAf8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVH/CCeD/+hU0P/wAF0P8A8TR/wgng/wD6FTQ//BdD/wDE0AYfjTxp4VuvAviG3t/EujTTy6ZcpHHHfxMzsYmAAAbJJPGKPBfjTwra+BfD1vceJdGhni0y2SSOS/iVkYRKCCC2QQeMUeNPBfhW18C+Ibi38NaNDPFply8ckdhErIwiYgghcgg85qt4b8OeCrT4baJq2r6DoaoNLtpbi5nsYmLMY1ySSuSST9STQB0v/Cd+D/8Aoa9D/wDBjD/8VR/wnfg//oa9D/8ABjD/APFVyxTwTFCbq6+G/wBl08cm9m0ODYq/3mUZkUe5QY74rpovBPgueFJofDGgSRSKGR0sISGB5BB28igB/wDwnfg//oa9D/8ABjD/APFUf8J34P8A+hr0P/wYw/8AxVH/AAgng/8A6FTQ/wDwXQ//ABNH/CCeD/8AoVND/wDBdD/8TQAf8J34P/6GvQ//AAYw/wDxVH/Cd+D/APoa9D/8GMP/AMVR/wAIJ4P/AOhU0P8A8F0P/wATR/wgng//AKFTQ/8AwXQ//E0AH/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFUf8ACCeD/wDoVND/APBdD/8AE1Q0Xw14I13SLfUrXwjo6QzglVl02EMMEjnAI7etAF//AITvwf8A9DXof/gxh/8AiqP+E78H/wDQ16H/AODGH/4quWtrfwpca7/ZB+FgiuVSOWQyadYbY43ZlDkiQ8ZRumTx06V1P/CCeD/+hU0P/wAF0P8A8TQAf8J34P8A+hr0P/wYw/8AxVH/AAnfg/8A6GvQ/wDwYw//ABVH/CCeD/8AoVND/wDBdD/8TWDb2PgC503Rb9PB+mCLV5xb24bTINysVdsv6DEZ6Z7UAb3/AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VR/wgng//oVND/8ABdD/APE0f8IJ4P8A+hU0P/wXQ/8AxNAB/wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FVjQaX4Bm8MXOvnwnpEVpbJO0ySabAHTyiyuCAMZBQ96Y2n/D9fB0fidfCOlPZSQJMka6ZB5p34CpjGN2SFxnr3oA3P+E78H/8AQ16H/wCDGH/4qj/hO/B//Q16H/4MYf8A4qj/AIQTwf8A9Cpof/guh/8Aiaoal4b8EaXNp0U/hHR2a/uhaxbNNhIDlHfLZA4wh6Z7UAX/APhO/B//AENeh/8Agxh/+Ko/4Tvwf/0Neh/+DGH/AOKqhovhvwRruiWerWvhLR0t7uISosumwhgD64BGfxrHsV+H99dWaL4IsorS+kMVnfy6RAILhsEgKfvDIBILKAe2aAOn/wCE78H/APQ16H/4MYf/AIqj/hO/B/8A0Neh/wDgxh/+KqhonhvwRr2iWeq2vhHR0gu4hLGsumwhgD6gAjP41f8A+EE8H/8AQqaH/wCC6H/4mgA/4Tvwf/0Neh/+DGH/AOKo/wCE78H/APQ16H/4MYf/AIqj/hBPB/8A0Kmh/wDguh/+Jo/4QTwf/wBCpof/AILof/iaAD/hO/B//Q16H/4MYf8A4qj/AITvwf8A9DXof/gxh/8AiqP+EE8H/wDQqaH/AOC6H/4msPVtL8E6XqA0+LwJZajeeT9okhstKgYxxkkBmLbRyQwABJODgUAbn/Cd+D/+hr0P/wAGMP8A8VR/wnfg/wD6GvQ//BjD/wDFVi2umfD6+l0hLXwto8qatDJNbyDTIQoVApYNkAg/MBjHUHOK2v8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiaAD/hO/B/8A0Neh/wDgxh/+Ko/4Tvwf/wBDXof/AIMYf/iqP+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiaAD/hO/B/8A0Neh/wDgxh/+Ko/4Tvwf/wBDXof/AIMYf/iqP+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiaAD/hO/B/8A0Neh/wDgxh/+Ko/4Tvwf/wBDXof/AIMYf/iqP+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiaAD/hO/B/8A0Neh/wDgxh/+Ko/4Tvwf/wBDXof/AIMYf/iqP+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiaAD/hO/B/8A0Neh/wDgxh/+Ko/4Tvwf/wBDXof/AIMYf/iqP+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiaAD/hO/B/8A0Neh/wDgxh/+Ko/4Tvwf/wBDXof/AIMYf/iqP+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqP+E78H/8AQ16H/wCDGH/4qj/hBPB//QqaH/4Lof8A4mj/AIQTwf8A9Cpof/guh/8AiaAD/hO/B/8A0Neh/wDgxh/+Ko/4Tvwf/wBDXof/AIMYf/iqP+EE8H/9Cpof/guh/wDiaP8AhBPB/wD0Kmh/+C6H/wCJoAP+E78H/wDQ16H/AODGH/4qj/hO/B//AENeh/8Agxh/+Ko/4QTwf/0Kmh/+C6H/AOJo/wCEE8H/APQqaH/4Lof/AImgA/4Tvwf/ANDXof8A4MYf/iqw/GnjTwrdeBfENvb+JdGmnl0y5SOOO/iZnYxMAAA2SSeMVuf8IJ4P/wChU0P/AMF0P/xNYfjTwX4VtfAviG4t/DWjQzxaZcvHJHYRKyMImIIIXIIPOaAMu4e5j+GHw8e0gjnuFl0wxxSSeWrt5XALYOB74NdHFpviDWfEel6jrNtp+n2umNJLFBbXLXDyyMhjyzFECqFZuADk4pnhnR7fV/h54N+0PKv2S0sbuPyyBl0iXAOQeOf/AK9dhQB49p+oa5p/w/0C9Gp6xqGoa7cJbO6SI7wxgSNiISELvITG5iTkk9gK1rO41nTNb0xrHTvFKW89ysN2ms3cE0TI3BZT5zMrr1wvBAIx6dbF4R0uPwpb+HJBNNZW6qI3eTbKrKdyuGXGGB5BGKZY+Eba21GC/vNR1LVJ7bP2Y30yssBIwSqqqjdgkbjk4zzyaAMjS7K58YS6lqN7rOp2sUN9PaW1pY3JgEKxOU3Pt5Z2KlvmyACBiuX037bZa5NBPqL3cw8YQxSXAwplQWXAYLgE4C57ZB4rvrzwfbTajcX1lqWp6XLdENcixmCpM2MbirKwDYAG5cE461BY/D/RtN8v7K12gTUV1L5pd5aYR+XliwJIIJJ5yWOc0AZ+m2Vz4xk1K/vdZ1O0ihvp7S2tbC5NuIVicpufbyzsVLfMSACBj1o+O9Cuh8NrlNU1e7vJ7R18ueKRrcyI0qACVUYK7AdyMZ5ABrpb3wfbT6hcXtlqWp6VNdENciwmCrMwGNxVlYBsADcACcdae/g7Sj4Xm8PxiaG0mbfLIsm6V33hy7O2dzEjknNAHN315J4T1/WpLaS6u49O8M/ao47m4eYuwlmb5mYknpjPoParE/h7UoPDb6yvirU21iO2Nz55n/0VmC7tvk/c8vt0zjvnmupbRLR9dm1Z97zTWa2TxNgxmMMzdMZySxB5xjtWJ/wr7TzbixOp6udIHH9lm6/cbf7mcb9nbbuxjjGKAMa3kvvF3i2xL6nqOn6fceHba9ltLS4aPMju/wDEOVwOCRgnA5wMVpaSNXW48T+GrfWJmmtEiewvrpRLJD5qNgNkfPtZSRnnBwc10cei2sWvNrCGRZ2tEs/LGBGEVmYYGM5yx74xjiqt14ZtbmbWphdXkEur26W80kMgVoggYBozjhvnPJz2oA2lBCAMcsByfWvK7UBobHwMQD9n8QujxnvaRf6WmfbDRJXqirtQLknAxk9aw4/CenR+M5fFKmb7fJa/ZihYeWBkfMBjO7CqM56DpQBxGlE3Efh7waxJOm6zOJgf+eFqfMiyPfzLatvS7K58YTalqN7rOp2sUN9PaW1pY3JgWFYnKbn28s7FS3zZABAxW7aeE9OsvF9/4miM3269hWGRWYeWoAUZUYyCQiZ5/hFQXfg62m1G4vrHUtT0uW6Ia5WxnCpM2MbirKwDYAG5cE460AcJaS6j/asdrqOqS6k9p4xjgjmkPRBaEgYHAPPOABuycc12ngT/AI89c/7Dl9/6NNFh8P8ARtNEYtWu0Caiup/NLvLTCPyyWLAkggknnJJzmtrSdHt9Gjuo7d5WFzdy3b+YQcPI25gMAcZPH86AOdudN1ODWNW1XwrqOn3TXEii/wBNuhuQzIirgSKcxsUCAhgR0PGa5/TY4vGHjKyu7O5vNL06Xw7BI1vZv5L/AOukATevKhTn7uM4Hauu1DwZaXmo3V7b6lqmnNeAfa47G4CJOQNuWBU4bAA3LtOAOau6f4a07S9SjvbJHh8uxjsI4FI8tIkZmXAxnPzHnNAHFX+vav4W07xRp8F3c38lhJZiynuNsksa3LBMMSQH2ncQWPcAnHNVrifXdJiS+03TvGRvI3Qyf2peWz284yAysvnEJkZwUAwccHpXeyeGtOuLzV57mNrhdVhjguYZCChVAwGBjI+8e/pjFUYPBNos9u15qmrahbWzrJBaXlyHiRl5UnADOR1G8tg89aAKXk3XirxTrVrPql9ZWGlSRW8dtZTmBpHaNZDI7r82PnAABA+UnnNcjrc2u6TceLtOh8R30xt4NM+xSySnfAJLkghsYBPYnqy7Qc16HqnhW11DUjqdve3+m37RiKS4sZQhlQdA6sGVsZOCRkZ61nJ8OdGRNQ/f37S6h5Bup5J98kjQyeYrEsDzng9sAAAYoA6Cwsk0nSvImvbi5SMM0lxey7mPcljwAPYYAFcVGt74M0Ww2z2Gt+EUlgSEsmJ7eN3URMrAlJVUsuDhTjnJr0NlV1KsAVIwQRwRXK2vgDTLV7eIXupy6bayia30yW43W8TKcrgY3EKcEKWIGBxxQByUmjJpWpfEK8tdQ1ZZ7LTw8LtqMzfMbZzlst8xB+7n7vbFTXdzrGm23hzSVvNe1GTVoJLy9ntZY/POxI/3cRkZVjXL5OPmwPUk13Fx4YsrlteLy3AOtwCC5wy/IojMeU44OCeueadqHhqx1HTbOzlaeNrLaba5hk2TQsF27lYdyMgjoc8igDlNDudasvEtrZwWWvxabdRSiUa5cwzeU6rlXjYStIRngqcjkHiqGjDxBbw3unS6pqtv4qm0yYrDqEgltp5wRiaBxlVUFgNgAwGGV4zXZ2HhKytbma6vLm81S7lha3M9/IHKxH7yKFCqoPfAycDJOKzofhvoywPb3dzqN/bi0ayt4ru43C1hbGVjwAQflXkkn5RzQBH4GmMdzf2M95rIu444nlsNXfzZISdwLpLkh0Yg9DgFe2cV2lYmieGLfRbu4vTfX2oXs6JE1zfSh3Ea5KoMAAAFiemSTyTW3QAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/jv/AJJ54l/7BV1/6KauSUrF8P8A4c3l3/yC7X7DJeE/dQfZysbt/siRoz7cHtXW+O/+SeeJf+wVdf8AopqZ4Jijn+G/hyKaNZI30i2VkcZDAwrkEHqKADVdV8QWZuZYtH0eTTo1LfaLnVmiymMksPIYD8zXJtDB4x1XwRJqWmfYbaexvpTYRynYYwYdgJAUlSNrYwOwIrql8AeGFdD/AGZuijIZLd55WgUjpiEtsH/fNbcum2k2o2uoSRZurVJI4X3EbVfbuGM4Odi9fSgDjND0HS/Fg1XUNegF7frqNzbbJXb/AERI5CqJGAfkOwK24YJ35z0rP8O3k9zr/hZri5e5SL+17S3upGy1xGksYjYn+IlEPPfBNdjqXg/QtWvJLu7sj58qhZnhnkh84DgCQIwDjHHzZqxfeHNI1HTYNPuLCL7LblTAkeYzCVGAUKkFCBxwRxQB534+2Xmu+ILZJmRhYaPE7Rthoyb9zwexwwP5Vu6jolh4T13w7eaHbfZnu71rO5jRzi5RoJXG/J+Zg0akMeevPNbcXgnw7DHMiacB5/lec5mkLyGOTzELMWyxD85JyehyOK1rzTrS/ltJLmLzHs5/tEB3EbJNrLng88OwweOaAOF8OeHdF1/wda6/qztJqt1Cbi51IzMkttL1ZUbP7sRkFdowBt5HWtf4YNu+G2iN5nmZhJ3kY3fO3NXrnwT4dvLyW6m01S0z+ZNGsrrFK/8AeeMMEc+5BzWtp+n2ulWMVlZQiG2iBCRgkhQTnv7mgDAtv+Spap/2BbP/ANHXNcP4a0m003wh4Bv0DGe71K3a4nkckn9xOqLz0A3hQB6+pr1ZdOtU1WXU1ixeSwpbvJuPMaMzKMZxwXbnGefpVT/hG9IPh6PQWsUfTI41jSByWCgdMEnOQeQc5FAGD4htrDVfiHoWm30EF1E2mXzS28yh1Kl7fblT2yp/KuO0/wAPaY/gTwLBDapbLfatG101t+6abEM+dzLg8gY+hr0rTfCWiaTfJfWlmReIjILiWaSWQq23ILOxJHyrgHp2xk1NF4d0qCz020jtdsGmyia0XzGPluFZc5zk8O3XPWgDnI9Is/DvjqxsdIQ2Vlqmn3RnghYhBJE0W2RV6BsSMCR14z0rrtPtvsWmWtr9pmufIhSPz533SS7QBuY92OMk+ppl1pVleXsF5cQ77iCKWGN9zDakm3eMA452L9McVJY2Vvpun21haR+XbW0SwwpuJ2ooAUZPJ4A60Aeb6r+51fVfB3bV9WtbmJe5glBefHtm3mz/AL9Ft8+sWvgr+G016W8ZP+nVALmP8PMliX/gNd/PoOmXOvWuty2qtqVrE0MM+45RG6jGcHv1Hc+tCaFpkfiCXXktVGpy24tnuNxyYwchcZx1xzjPAoA5PQ9B0vxYdW1HXoBe366jc22yV2/0NI5CqIgB+QlArbhgnfnPSud8PLGllo6Q3b3kSeM7lUuJH3tIoinAJb+I4HXv1r0PUvB+hareyXl3Yn7RKoWZ4ZpIvOA4AkCMA4xx82alt/C+i2kVtFbafHDFbXRvIUjLKqSlSu4AHHRiMdOelAGV8P5I4vhhockxxEmnqz8Z4A54rLtDP4XsdCew1GDVvDFzcW9vawzxfvoElIETRyD7yruHDLnaOvFdtpunWmkabb6fYxeVa26COKPcW2qOgySSfxrLsvBnh/Tr9L2105Y5Y2Z4l812jiY5yUjJ2IeTyoHWgDzzSvC+nW3wVj1+FZE1q20p72G/EjebG6IXVQc8IMBdvQjtV+SDUfFHirWBd6DpurxWvkLb219qLwpDG8KvvWMROCWYt85Oflxxt59Aj0HTIvDp0BLbGlm3a1MHmN/qiCpXdnd0J5zmodR8LaNqjwSXVo3mwR+VHNDNJDIE/u70YMV9icUAcBBJfjTrbRLy9jh0q41/7DJ9kv5J2t4vKZvsxmKo3MqhPUBtuau+JfDmiaB4g8IyaWi2Ek2rxo1rC5CTgI53FM4LKcfN1+Yg9a7j/hHtHGh/2J/Z1t/Zm3b9m2DZjOc49c8565561RtfBHh60u4rtLAvcwurxTT3EkroRnAVnYkDnoOPbigDI8AaRaR3Ou6uVZ72XV7+EO7k7I/tDfIo6AEjJ9TV3UdPa88T3V1oOtLZa3BbRR3UEsHmxSxku0e9eD1L4ZWHU5zXQWOnWmmxzR2kXlrNPJcONxOZHYsx5PcknHSqWreF9H1u5jub61ZriNDGJoppIXKE52lkYErnscigDgrWy0fxN4l8KareaDpy3VxHfLchYVZXkgZUDAkcgFSVJ7Gqr2r6X4U8U67YM66lNrdzatcNOyfZ7dr0K4VsHyxjLFgCR97nAr06LQ9MglsJIbOOI6fG0VqI8qsSMAGAUcdFHUdqfbaRYWlrc2sVsn2e6llmmjfLq7SMWfIbPBJPHTmgDzqXRr7Q77Srqz0TRdCne9hi+0Q6xLI10rN80boYB5pZd2CTkEA54p+kW1ro/ihLq9eN31Ga8a1160uwyTqQ7mOdTwPLVTtxlRs7dK7PTvB+g6Vex3dpY7ZoQVhMk0kghBGCI1diE44+UDii18IaDZao+o2+nIty5c8uzIpf75VCdqlu5AGcnNAHK+AbGHQ9YTTbi2VNRm07zhfWt0ZbfU41ZQZ2B5EmXXOc8NwSOno1Y2jeFNE0Cd59MsRDI6CPcZHfYgOdi7idi5/hXA9q2aACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+ilo8d/8k88S/wDYKuv/AEU1HgT/AJJ54a/7BVr/AOiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAorB8V+KIfCmnRXk9tJOJZfKCoQMHBPf6Vw0vxnOf3OhjHq9z/AEC0nJIynWhB2kz1eivGp/jHqrKRBplnGexcs38iKpSfFvxI/wB1LGP/AHYT/VjS50ZvF0u57lRXz/P8S/Fcx41IRDHSOBB/MZqlN458Tzg79Zuhn+4wT+WKXOiXjIdmfRtIWCjJIA96+ZZPEOtSjEmr37j0a4c/1qlLc3E7Bpp5JCOhdyf50vaEPGroj6fl1Gxg/wBdeW8f+/Ko/maoP4r8PR9db08/7twrfyNfNe4+poyfU0vaMl419EfRUvjvwxEMtrEB/wB0M38hVVviV4VXpqDt9LeT+orwFZXXoQfqM0eY/rS9pIl4yfRI91f4p+G0+612/wDuw/4kVUf4u6KPuWOoH6qg/wDZq8W81vb8qDKw7CjnkQ8XV8j2GT4v2Az5elXLem6RR/jVN/jE+T5ehrjsWuv/ALCvKfOb+6KXzz/d/WjmkS8VW7np0nxfvif3elW6j/akZv8ACqb/ABZ19idtrpyjt+7c/wDs9efef/sml89fQ0uaRLxFV9Tv1+LHiBetvp7fWJ/6NVuP4vaiP9Zpdq3+67L/AI15r56+/wCVL58f979KXNMSxNVfaPUU+MMo+/oiN/u3JH/spq1H8YLU483R5l/3Zg39BXkolQ/xCl3r/eH50c8yli6vc9jX4uaQfv6ffD6BD/7NVmL4reHpPvR30X+/Ev8ARjXimQe9Lmj2kiljKp7mnxM8MN1u5V+sDf0FWY/iD4Wl+7qqj/ehkX+a14HRT9qyvrtTsj6HTxj4ckAxrNmM/wB6QD+dWI/Eehy8R6zp7HrgXKZ/nXzhRR7V9ivr0ux9MpqFlJ9y8t2/3ZVP9asK6v8AdYN9Dmvl6lBKkEEgjkEU/a+RX17+7+J9Q0V8zx6nfxDEd9coP9mVh/WrCeIdaj+5rGoL9Llx/Wn7Vdivr0ex9IUV89R+M/EcRyus3Z/3n3fzqwnxA8Up01Zz/vRRn+a0e1RX16HZnvtFeGR/EzxOn3ruGT/egX+gFWU+KviFesdi31iP9Gp+0iP65TPaqK8ei+LmsD/XWFi/+4HX+bGrSfGC5H39HiP0nI/pT9pEpYul3PV6K8wj+MMZ/wBbojL7rc5/9lFW0+LumH/Wabdr/ulT/UUc8e5SxNJ9T0SiuCj+LWgsQHtdQT38tCP/AEKrafE/wy3We4T/AHoD/SnzLuUq9N/aR2VFcrH8RvCr4B1IqT/egk/+Jq3H428NSjK6xbD/AHiV/mKfMilVg9mjforITxV4ffpren/jcKP5mrcer6bL/q9QtH/3ZlP9aLopST2ZcoqNJ4ZMbJY2z/dYGpKYwooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCilo8d/wDJPPEv/YKuv/RTUeBP+SeeGv8AsFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiisHxX4oh8KadFeT20k4ll8oKhAwcE9/pQKUlFXZvUV5RL8Zzn9zoYx6vc/wBAtUZ/jHqrKRBplnGexcs38iKnnRg8VSXU9lorw2T4t+JH+6ljH/uwn+rGqM/xL8VzHjUhEMdI4EH8xmlzol4ymfQFFfOU3jnxPODv1m6Gf7jBP5YqjJ4h1qUYk1e/cejXDn+tL2hLxseiPposFGSQB71Xl1Gxg/115bx/78qj+Zr5glubidg008khHQu5P86j3H1NHtCHje0T6UfxX4ej663p5/3bhW/kaqy+O/DEQy2sQH/dDN/IV865Pqacsrr0IP1Gal1GT9dl2Pfm+JXhVemoO30t5P6iqz/FPw2n3Wu3/wB2H/EivCvMf1pfNb2/Kj2kifrlXsj2l/i7oo+5Y6gfqqD/ANmqvJ8X7AZ8vSrlvTdIo/xrx4ysOwpPOb+6KOeRDxdU9Wf4xPk+Xoa47Frr/wCwqCT4v3xP7vSrdR/tSM3+FeY+ef7v60vn/wCyaXNIn61W7/kegv8AFnX2J22unKO37tz/AOz0i/FjxAvW309vrE/9GrgPPX0NHnr7/lS5pk/WKv8AMelR/F7UR/rNLtW/3XZf8asJ8YZR9/REb/duSP8A2U15d58f979KUSof4hRzTH9aq9z1qP4wWpx5ujzL/uzBv6CrS/FzSD9/T74fQIf/AGavHN6/3h+dLkHvR7SRX1ur3Pa4vit4ek+9HfRf78S/0Y1ZT4meGG63cq/WBv6CvDM0U/ayKWNqeR75H8QfC0v3dVUf70Mi/wA1q2njHw5IBjWbMZ/vSAfzr54oo9qyljp9Uj6Pj8R6HLxHrOnseuBcpn+dWk1Cyk+5eW7f7sqn+tfM1FP2vkUsc+sT6hV1f7rBvoc0tfLwJUggkEcgirMep38QxHfXKD/ZlYf1o9r5FLHLrE+mKK+b08Q61H9zWNQX6XLj+tW4/GfiOI5XWbs/7z7v50/aopY6HVH0LRXgSfEDxSnTVnP+9FGf5rVqP4meJ0+9dwyf70C/0Ap+1Q1jafme50V4qnxV8Qr1jsW+sR/o1WYvi5rA/wBdYWL/AO4HX+bGj2kSljKR7DRXlCfGC5H39HiP0nI/pViP4wxn/W6Iy+63Of8A2UU/aRK+tUu56fRXnafF3TD/AKzTbtf90qf6ip4/i1oLEB7XUE9/LQj/ANCp88e5X1il/Md7RXGp8T/DLdZ7hP8AegP9KsR/Ebwq+AdSKk/3oJP/AImnzLuV7an/ADI6qisCPxt4alGV1i2H+8Sv8xVhPFXh9+mt6f8AjcKP5mi6K549zXoqnHq+my/6vULR/wDdmU/1qwk8MmNksbZ/usDTKuSUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFc/47/5J54l/7BV1/wCimroK5/x3/wAk88S/9gq6/wDRTUAHgT/knnhr/sFWv/opa6Cuf8Cf8k88Nf8AYKtf/RS10FABRRRQAUUUUAFFFFABRRRQAUUUUAeefGEZ8L2Z/wCn1f8A0B68WxXtvxeQt4QtyMfLeoT/AN8OP614lmsZ7nl4v+IFFJRUnKLRSUUALmikooAXNGaSigBc0ZFJRQAuaDSUUDCiiigQUUUUAFJilooAMUYoozQADIORwR3p29z/ABH86TNGaQxwkcAjPWk3MP4j+dJmiiwC+Y/96l8x/UflTaM0CshfOk9vyo85/QUyiq0FYk+0N3T9aX7R/sH86ioosgsS/aF/utS/aE9/yqGjFFkFifz4/X9KXzk/vCq2BRgUcqFYtean94fnS71/vD86p7RRto5UBdzRmqO2jB7E0ciAvUVn5cfxN+dHmSD+M/nT9n5iNCiqAncdSTR9pf1o9kwL9FV45d33sn6GtFIbQkBpA2fRjxWcly7hcrZA71LFe3EAxDcyxj0RyP5Vqw6VayfdPPvVyPRkH8IrmnjKUNzSNOpLWJkJrmsKf3epX4/3Z3H9auQ+JfEyY8vVdQI/252b+ZNaselIP4RVpNNQfw1yzzalHY6IYau+pnxeLvGI+7qkv/AkU/zFaFv4x8Yqfm1JXHo1vH/QVYSwUfw1YSyX0rknnrXwo64YKu95Mu2Hj3XY2Au7W1uF77QUb88kfpXY6T4osdUCoc285/5Zydz7Hoa4dbNfSpltVHasY5/VjK7V0d1PCVFuz02iuX0bW2hK214+6Posh6r9fauo619Hg8ZSxdPnp/NdhTg4OzCiiiusgKKKKACiiigDn/Hf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaPHf/JPPEv/AGCrr/0U1HgT/knnhr/sFWv/AKKWgDoKKKKACiiigAooooAKKKKACiiigArzz4wjPhezP/T6v/oD16HXAfF5C3hC3Ix8t6hP/fDj+tTLYyr/AMNniWKKM0lYnji0UlFAhaM0lFAC0ZpKKAFzRmkooAXIozSUUWHcU0lFFAgooooAKKKKAExS4oooAMUDggjgjvRmlzQMXe5/iP50okcAjPWm5ozSsguLuYfxH86PMf8AvUlFArDvMf1H5UnnSe35UmabTQND/Of0FL9obun61HRT0FYl+0f7B/Oj7Qv91qioosgsTfaE9/ypfPj9f0qDFJgUWQrFnzk/vCl81P7w/OquBSbRRyoLFzev94fnS5qltpNtHIgL2aKo4PYmm5cfxN+dP2a7iNCis/zJB/GfzpRO46kmj2TAv0VQ+0v61PHLu+9k/Q0nTaC5YoyB3qykNoSA0gbPox4rRh0q1k+6efesJ1YQV3caTbsjKivbiAYhuZYx6I5H8qtJrmsKf3epX4/3Z3H9a149GQfwirUelIP4RXLLMaMTeNGs9jKh8S+Jkx5eq6gR/tzs38yavxeLvGI+7qkv/AkU/wAxWgmmoP4asJYKP4a5Z51CPwo6YYSu/tMr2/jHxip+bUlcejW8f9BW7YePddjYC7tbW4XvtBRvzyR+lUksl9KmWzX0rknn1T7J2U8FVW8mdxpPiix1QKhzbzn/AJZydz7Hoa268yW1Udq6XRtbaErbXj7o+iyHqv19q7MFnsakvZ11a/Xp8zreHmlfc6iijrRX0JgFFFFABRRRQAVz/jv/AJJ54l/7BV1/6Kaugrn/AB3/AMk88S/9gq6/9FNQAeBP+SeeGv8AsFWv/opa6Cuf8Cf8k88Nf9gq1/8ARS10FABRRRQAUUUUAFFFFABRRRQAUUUUAcN8WE3eC88/Lcxn+Y/rXhde8/FMZ8Dzn0mjP/j1eDVlPc8zGfxPkFFFFQcoUUYoxQAUUYoxQAUUYoxQAUUYoxQAUUYoxQAUUYooAKKKKACiiigAooooAKKKKACiiigQUUUUAFFFFABikpaKAEooxRTAKKKKACiiigQUlLRTuA3FBUGnUYouBG0dRkYarFRSCtIy6CaLEEe4CriQGqtoelbEKgiuPEVXBmlOmpDIXlhIKk49K3LDUgxCvw3oe9UBCMdKQwZ7V59SdOsrTRoqU6bvE7CBkkGRVjygBkVzFlfPAQshJHZv8a2I7/nBNfPYjBzjJ8uqPYoYqDjaejNBcVIGFUhOGwQad5tcTg0d0ZroXg4o3j1qj53vSeePWjlZrzl4yD1rqvDeqfaYjaStmSMZQnuv/wBauGM4qax1NrG+huV/gbJA7juPyrvy7Eyw1dT6PR+hnVtONj1KikVg6hlOVIyCO4pa+9OAKKKKACiiigDn/Hf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWjx3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWgDoKKKKACiiigAooooAKKKKACiiigArhviwm7wXnn5bmM/wAx/Wu5ri/imM+B5z6TRn/x6lLYzrfw5eh4NRRRWB4wUUUYoAKKMUYoAKKMUYoAKKMUYoAKKMUYoAKKMUYoAKKKKACiiigAooooAKKKKACiiigQUUUUAFFFFABRiiigBKKWkxTAKKKKACiiigAoooouISkxTqKdwGlQaa0dSYopp2CxXIw1W4I9wFV5BVq0PSnUk+S6ElqWkgNW4XlhIKk49KfCoIq0IRjpXlzxNnZm/sLq6L9hqQYhX4b0PeuggZJBkVx5gz2rRsr54CFkJI7N/jXlY3DQqrmp6M68NVlSdpao6fygBkUq4rPjv+cE1OJw2CDXhzozj8SPXhWhLWJdDCnhxVHzaPO96z5WdCmXt49aQyD1qj549aQzinylc53PhvVPtMRtJWzJGMoT3X/61b9eW2OptY30Nyv8DZIHcdx+VeoqwdQynKkZBHcV9rk2KdahyT3j+XQ4a0UpXQtFFFeuZBRRRQAVz/jv/knniX/sFXX/AKKaugrn/Hf/ACTzxL/2Crr/ANFNQAeBP+SeeGv+wVa/+ilroK5/wJ/yTzw1/wBgq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQBx3xPGfAl5x0ki/9DFeCV7/8TBnwDqHs0X/oxa8ArGpuebjP4i9AoooqDkCiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKADFJilooASilopgJRS0lABRRRQAUUUUCCiiigAxSUtFACUUtJimAUUUUAFFFFABTJKfTJKqO4mSWpw1bdq/FYdt1FbNqK5cak0XQbTNiEBqsrCCKr244FX4xxXy9eTjLQ92jBSWpQuI9o4qtFctE21jx29q0Llc1myR4Oa7cJUUo2kcWKp2loacV2eOauC4BGc1ziylTzVtLj5BzWWJwqvdF4as1ozXNyPWmm596yzce9MNx71yrDHX7Y02ufeo2ufesw3PvUZufetI4Ul1z3Pwvci78NWEoPSPZ/3ydv8ASteuZ8APv8HWZ/2pP/QzXTV9hQv7KN+yBO+oUUUVqAUUUUAc/wCO/wDknniX/sFXX/opqPAn/JPPDX/YKtf/AEUtHjv/AJJ54l/7BV1/6KajwJ/yTzw1/wBgq1/9FLQB0FFFFABRRRQAUUUUAFFFFABRRRQAVx3xPGfAl5x0ki/9DFdjXJfEwZ8A6h7NF/6MWlLYzq/w5eh4BRRRXOeMFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFGKKKAExRS0UAJRS0UwEooooAKKKKBBRRRQAUYoooASilooASijFFMAooooAKKKKAGSVJanDVHJT7bqK0+wQ9zctXrVhAase1FbNuOlfOY1Weh6mF13LCwgiq9xHtHFX4xxVe5XNeZSqtT1O+rSXJdGfFctE21jx29q0Yrs8c1mSR4OajWUqea9adONWJ5UXKnI6MXAIzmkNyPWshLj5BzSm4968p4WzPTjW0NQ3PvUbXPvWYbj3qM3HvTWFH7c02ufevXPC9yLvw1YSg9I9n/fJ2/0rww3PvXsngB9/g6zP+1J/6Ga9jKaTp1X6E+05nY6aiiivfGFFFFABXP8Ajv8A5J54l/7BV1/6Kaugrn/Hf/JPPEv/AGCrr/0U1AB4E/5J54a/7BVr/wCilroK5/wJ/wAk88Nf9gq1/wDRS10FABRRRQAUUUUAFFFFABRRRQAUUUUAct8R0MngHVAMcCM8+0in+lfPlfRPj1d3gbVhjP7oH8mBr51rKe55uNXvr0CjNFFQcgtFJRQAtFJmjNIBaKM0UAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAJRRRTAKKKKACiiigAooooEFFFFABRRRQAUUUUAFRyVJUclVDcTH23UVtWtY1t1FbNr2rnxuxdDc2IOgq9G2BVGDoKuLnFfKV/iPeoOyFkAIrPmXJq82cVAyDFFGfI7jrR50Zcq4qISlRirs6gDpWa5+Y161OXtEedKPIx5mPrTTIajzSZrZU0Rzsk3mk3UzNGarkFzHunw/BHgjT8ggnzDz/10aumrnvAy7fBemDOf3ZP5sa6GvYpq0EvI9CHwoKKKKsoKKKKAOf8AHf8AyTzxL/2Crr/0U1HgT/knnhr/ALBVr/6KWjx3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+iloA6CiiigAooooAKKKKACiiigAooooAK5b4joZPAOqAY4EZ59pFP8ASuprnPHq7vA2rDGf3QP5MDSexFRXg/Q+dqSiisDxQzS0lFAC0UlGaQC0UmaXNABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABSUtJTAKKKKACiiigAooooEFFFFABRRRQAUUUUAFFFFAEclPtuopklSW3UVr9gh7mza1sQdBWPa9q2IOlfOY7c9TCl6NsCkkAIpFzikbOK8a2p6t9LFGZcmqUq4rUZBiqc6gDpXpUK3Q8+rS1uUhKVGKaZj60xz8xpma71BPU53JrQkMh9aTeajzRmqUETzsfur3H4fgjwRp+QQT5h5/66NXhea958DLt8F6YM5/dk/mxrswkbTbN8O7yZ0NFFFegdYUUUUAFc/wCO/wDknniX/sFXX/opq6Cuf8d/8k88S/8AYKuv/RTUAHgT/knnhr/sFWv/AKKWugrn/An/ACTzw1/2CrX/ANFLXQUAFFFFABRRRQAUUUUAFFFFABRRRQBg+Nhu8FauP+nZjXzhX0l4wG7wbrAxn/RJD+lfNuKyqbnnY34kFFJRUHELRSZpc0AFFFFABRRRQAUUUUAFFFFABRRRQAZpc0lFAxc0UlFAC0UlFAC0UlFIBaKSigBaSiimAUUUUAFFFFABRRRQIKKKKACiiigAooooAKKKKBhUclSVHJVQ3JZJbdRW1adqxrbrW1adBXNjdjShua8HSri9KqQdKtr0r5Wt8R71HYRqhbpUzdKibpWcTSRRuOlZcn3jWpcdDWVJ9417GE2PMrjM0lFFd5yhRRRQB9A+Dht8H6UP+ndTW3WR4VGPCek/9esf/oIrXr1I/Cj1I/CgoooqigooooA5/wAd/wDJPPEv/YKuv/RTUeBP+SeeGv8AsFWv/opaPHf/ACTzxL/2Crr/ANFNR4E/5J54a/7BVr/6KWgDoKKKKACiiigAooooAKKKKACiiigArB8bDd4K1cf9OzGt6sTxgN3g3WBjP+iSH9KT2Jn8LPm2ijFJWB4YtFJRmgBaKM0UAFFFFABRRRQAUUUUAFFFFABRmiigBc0ZpKKBi0UlFAC0UlFAC0UlFIBaKSimAUUUUAFFFFABRRRQIKKKKACiiigAooooAKKKKACiiigZHJUlt1FRyVLbda1+wZvc2bTtWxB0rItOgrYg6V83jtz1cKW16UjUq9KRuleP1PU6ELdKpXHSrzdKo3HQ11UNznrbGXJ941HmnyfeNR17kNjy5bhRRRVkhX0D4OG3wfpQ/wCndTXz9X0L4VGPCek/9esf/oIrpwvxM6sN8TNeiiiu07AooooAK5/x3/yTzxL/ANgq6/8ARTV0Fc/47/5J54l/7BV1/wCimoAPAn/JPPDX/YKtf/RS10Fc/wCBP+SeeGv+wVa/+ilroKACiiigAooooAKKKKACiiigAooooAx/FYz4Q1kf9OUv/oBr5qzX1Dq1m2o6NfWKOEa5t5IQxHCllIz+teKXXwp8TW6Fo47W5I7QzYJ/76ArOabOHF05SacUcTSVvXfgvxLZf67Rrsj1iTzP/Qc1jT289rJ5dxDJC+M7ZFKnH0NRY4JRlHdEVFFFBIUUUUAFFFFAwooooAKKKKACiiiiwBmlzSUUALRSUUALRSUUgFopKKAFopKKAFopKKAFopKKAFopKKAFopKKAFopKKAFopKM0ALRRmigAooooAKjkqSo5KqG4pbEtt1rategrGteoratO1cuO2NKG5rwdKtr0qpB0q2OlfK1viPeo7CNUTdKlaoW6VETSWxSuOhrJfqa1bk8Gsp+pr2cJseZX3GUUYortOWwUYpeBSbwKVykj6J8NqU8L6SpBBFnDkH/AHBWnVTS4jBpFlC3WOBFP4KBVuvXWiPTWiCiiimMKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACsfxWM+ENZH/TlL/6Aa2Kp6tZtqOjX1ijhGubeSEMRwpZSM/rQxSV00fL2aK7a6+FPia3QtHHa3JHaGbBP/fQFY134L8S2X+u0a7I9Yk8z/0HNYWaPGdKot0YNFSz289rJ5dxDJC+M7ZFKnH0NRUGewUUUUCCiiigAooooGFFFFABRRRRYQUZooosMXNFJRQAtFJRSAWikooAWikooAWikooAWikooAWikooAWikooAWikooAWikooAWikzS5oAKKKKACiiigZHJUtt1qKSprXqK0fwGb3Nm16CtiDpWRadq2IelfNY7c9bCFpelI1KOlI1eT1PUIm6VRuOhq63SqNyeDXVh9zmrbGU/U0ynv1NMxXux2PLluFFLijgU7isJivonw2pTwvpKkEEWcOQf9wV87FwK+kdLiMGkWULdY4EU/goFdWE1bOrDKzZbooortOsKKKKACuf8AHf8AyTzxL/2Crr/0U1dBXP8Ajv8A5J54l/7BV1/6KagA8Cf8k88Nf9gq1/8ARS10Fc/4E/5J54a/7BVr/wCilroKACiiigAooooAKKKKACiiigAooooAKKKKACmvGki7ZEVlPZhkU6igDIuvCugXrM1xo9kzt1YQqGP4jmsS6+F/ha4VglnNbs2fminbj3AYkfpXZUUrIh04PdHmd18GtOdD9k1W6ibsZUWQfptrEu/g5q8ZH2TUbOdec+YGjP8AI/zr2eilyoyeGpPofP138NPFVq2Bp6zr/ehmUj8iQf0rGufDet2hP2jSL6MDqTA2PzxivpqilyIzeCh0Z8o0V9TXFjaXiFLm1gnUjBEsYYEfjWLd+A/C94MS6Nbrzn9zmL/0EilyMyeBfRnzlRXud38JPDs5Bhe8tiOyShgf++gf51i3XwYUtm01ogf3ZYM/qD/SjlZk8JVR5LRXf3Hwi8RRE+VLYzDPG2VgSPxUfzrEufAPii0UtJo1wwH/ADy2yf8AoJNFjJ0ai3RzdGatXem31iQLyzuLck4HnRMmfzFVsGkZtNbhmjNJijFAC5paTFGKQC0UlFFgFopKKLALRSUZosAtFJmjNFgFopM0ZosAtFJmjNFgFopM0ZosAtFJmlosAUUUUhi0UlGaQC1FJ1qWopOtXDcT2J7brW1adqxrXrW1adq48dsa4fc1oelWx0qtAOlXUUEV8rWfvHv0Y3RC3SoXPFWJwFFZ80nXmqowc2TWlybkFywweay2cZp99chW8sHk9az2mr2qNPkR50nz6lkvTDL71Uaf3qJrj0rYapl1pq0fDVgdb8SWOngErLKPMwM4Qct+gNc4ZmNeyfCHwy9vaSa/dph7gbLYMOQndvxP6CtKNPnmkbRpnqVFFFeqbhRRRQAUUUUAc/47/wCSeeJf+wVdf+imo8Cf8k88Nf8AYKtf/RS0eO/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQA140kXbIisp7MMisq68K6BeszXGj2TO3VhCoY/iOa16KBNJ7nG3Xwv8AC1wrBLOa3Zs/NFO3HuAxI/SsW6+DWnOh+yardRN2MqLIP0216ZRS5UZuhTe6PGLv4OavGR9k1GznXnPmBoz/ACP86xbv4aeKrVsDT1nX+9DMpH5Eg/pX0DRS5EZPCU2fMtz4b1u0J+0aRfRgdSYGx+eMVl19XVXuLG0vEKXNrBOpGCJYwwI/GlyGbwK6M+WaSvo278B+F7wYl0a3XnP7nMX/AKCRWLd/CTw7OQYXvLYjskoYH/voH+dLlZk8FNbM8MpK9auvgwpbNprRA/uywZ/UH+lYlx8IvEURPlS2MwzxtlYEj8VH86LMyeGqrocBRXSXPgHxRaKWk0a4YD/nltk/9BJrFu9NvrEgXlncW5JwPOiZM/mKRk4TjuirmlzRg0mKCRc0ZpMUuKQC0UmKKLALRSUUWAWikoosAtFJmjNFgFopM0ZosAtFJmjNFgFopM0ZosAtFJmjNFgFooopAFLSUUDFopM0tICKTrU9t1qCTrVi161o/gI6mzadq14elZNp2rYgHSvmcc9T18IiyOlNbpUyKCKjnAUV5Kd2eo42Vyu54qhcsADzU80nXmse+uQreWDyetephqDep51ard2QxnGajMlVmm96iaf3r0kc6p3LZl96Y01UmuPSojMxpmipHR+GrA634ksdPAJWWUeZgZwg5b9Aa+k68t+EPhl7e0k1+7TD3A2WwYchO7fif0FepV6OGhywu+p0QjyoKKKK6CwooooAK5/x3/yTzxL/ANgq6/8ARTV0Fc/47/5J54l/7BV1/wCimoAPAn/JPPDX/YKtf/RS10Fc/wCBP+SeeGv+wVa/+ilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDzX4wZFhozdhdH/0GuPuIrWawLNBGWA4JUZrsfjFzpGlD1uyP/HDXGvERp5J9K8LNtKkHezM4t801a6sdp4Z8BeG9Z8LWN5c2TfaJYyXkjmYZIJHTOO3pS3Pwd0eQsbbUL2EnoH2uB+gP610PgD/AJEfTP8Adf8A9Daulr2opOKJVGnJK6PHrv4NXyITZ6vbzN2EsTRg/iC1Y138LPFFsMx20Fzz/wAsZh+fzYr3qinyIh4Sk9j5quvCfiGyYifRr4ADJZYWdfzGRWTJFJDIY5UaN16qwwR+FfVdRywQzqVmiSRTxh1BH60uQyeCXRnyrRX0ld+DvDl6H87RbPL53NHGEJz3yuDn3rFuvhV4YuEIihubYnvFOTj/AL6zS5GZPBT6M8HxRivXrr4M2rf8emsTR+00If8AkRWLdfB/W4mP2W9sp0x/EWRs/TBH60uWRm8NVXQ87xRiuqufh14qtSc6W0qj+KKRGz+Gc/pWLcaNqtmpe60y8gUDJMsDKAPxFTqjN05LdGdijFP4o4pcxFiPFFSYFG2nzBYjpM0/bSFad0Kw3NLmjBpKYhc0uaZmlzRYB9FMzS5pWHcdUUnWpN1RMctVQWomW7UZNbtmnSsizXGK3LYgAV5eYTeyOrCxV7s0oUwKtBwoqiJwB1pklx718/7Cc5Hse3hTWhPcygg1h394trE0jEZ6KPU1Nd30dvC0krYA7dz7CuTvLqS+n8x+FHCr2Ar1cJhuT4jiqTdZ3Q5rppHLsck0wyMajGFFXdN0nUtYm8rTbG4umzg+UhIH1PQfjXZ8T0RrGCSKnJ6mkyBXoukfBzXL3a+pXEGnxnGUB8yT8hx+tehaD8L/AA5oksdw0L31yhyslycgH1Cjj881rHDTlvoaJHnngL4cXGtzR6lq8TwaapykbcNOfp1C+/ftXuscaQxJFEipGihVVRgKB0AFO6UV3U6caasigooorQAooooAKKKKAOf8d/8AJPPEv/YKuv8A0U1HgT/knnhr/sFWv/opaPHf/JPPEv8A2Crr/wBFNR4E/wCSeeGv+wVa/wDopaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK81+MGRYaM3YXR/wDQa9Krzb4xc6RpQ9bsj/xw1FT4WZVv4bOOuIrWawLNBGWA4JUZrs/DPgLw3rPhaxvLmyb7RLGS8kczDJBI6Zx29K4t4iNPJPpXq3gD/kR9M/3X/wDQ2rxsn151e5U4qclzLoc9c/B3R5CxttQvYSegfa4H6A/rWJd/Bq+RCbPV7eZuwliaMH8QWr2Giva5UZvDUn0PBbv4WeKLYZjtoLnn/ljMPz+bFYl14T8Q2TET6NfAAZLLCzr+YyK+laKXIjJ4OD2Z8qSRSQyGOVGjdeqsMEfhTK+qpYIZ1KzRJIp4w6gj9ax7vwd4cvQ/naLZ5fO5o4whOe+Vwc+9LkM3gn0Z820Yr3i6+FXhi4QiKG5tie8U5OP++s1i3XwZtW/49NYmj9poQ/8AIilyMyeEqI8hxRivRLr4P63Ex+y3tlOmP4iyNn6YI/WsW5+HXiq1JzpbSqP4opEbP4Zz+lK0jN0Ki3icrikxWjcaNqtmpe60y8gUDJMsDKAPxFUeKm7M3G24zFJipOKMCjmFYjoqTbTdtVzIVhmaM04rSYNO4gzRmkpM0WAfmlpmaM0WGPopuaXdSsFyOTrVq1GTVRjlq0bNcYp1Xy0xJXka9mnSteFMCs22IAFXhOAOtfK4rnnLQ9vD8sI3ZeDhRVW5lBBqCS496oXd9HbwtJK2AO3cn0FRRwcm02VVxaa5UQ394trE0jEZ6KPU1zbXTSOXY5Jpt5dSX0/mPwo4VewFRDCivajFQjbqc9Om73ZIZGNN5PU1b03SdS1ibytNsbi6bOD5SEgfU9B+NdzpHwc1y92vqVxBp8ZxlAfMk/IcfrVxpzlsjdI86yBXongL4cXGtzR6lq8TwaapykbcNOfp1C+/ftXoeg/C/wAOaJLHcNC99cocrJcnIB9Qo4/PNdp0rppYWzvMtIbHGkMSRRIqRooVVUYCgdABTqKK7BhRRRQAUUUUAFc/47/5J54l/wCwVdf+imroK5/x3/yTzxL/ANgq6/8ARTUAHgT/AJJ54a/7BVr/AOilroK5/wACf8k88Nf9gq1/9FLXQUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHDfE3R9R1fS9PGnWj3Lw3W91TGQu0881xh0TXJrXyV0i9DdPmhI/U17ZRXLiMHTryUpdCHB3bT3MLwbYXWmeE7CzvIjFcRq29CQcZYkdPYit2iiulKysUlZWCiiimMKKKKACiiigAooooAKKKKAKd1pOnXwIu7C1nB/56wq38xWNd/D/AML3hBfSYkIzgws0f6KQK6WilZEuEXujgLr4RaBM26C4vbf/AGVkDL+oz+tYlx8GZgSbbW0bngSW5XH4hj/KvWqKXKjN4ek+h4Vc/CjxPAhMa2dyR/DFNgn/AL6ArFu/BfiayAM2i3ZGcfuk83/0DNfR9FLkRk8HB7Nnyxc2tzZsFuraaBj0EsZUn86g4NfVjosilXUMp6gjIrMuvDOhXrFrjR7GRz1YwLuP44zS5DJ4J9GfMpUU0rXp3xM8KaLoVjY3GmWf2eSe4KyYkYgjaTwCSBz6VysPhyK4tfMjuJFbGeQCK5q+Kp4d2qMz+pVW7R1OZ6Ubq6/RPAGo+IYrprK5tg1s4VlmyucjPBANLd/C7xVbH5LCO4XHWGdf5Eg10wkppSWxg6FTexxpekQ5fmti68K69Zuy3GjXybereQxX8wMGsnymjkKsCrA4IIwQa0VrGTi1ujSt3Cir0dxgCsmJWJAAJJ4AFdNpXg7xBqhHkabMqf8APSYeWv5tjP4Vx1KEZM0pym9IoqfaCarXmox2iZkOXP3UHU16PYfCaV4wdR1QxE4ylquT/wB9N/hXV6T4A8NaOwkh01Jpxz51yfNfPr83A/ACpjhTrp4actZ6Hgtj4f8AEXiicSWem3Esf8Lbdsag/wC0cCuy0f4LancbX1fUIbRO8cI8x/z4A/WvbgAAABgDtRW6w8Fvqd0YKKsjj9H+GXhfSNr/AGH7ZMMfvLs+ZyO+37v6V10cUcMaxxIqRqMKqjAA9hTqK2UUtEWFFFFMAooooAKKKKACiiigAooooA5/x3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCilo8d/wDJPPEv/YKuv/RTUeBP+SeeGv8AsFWv/opaAOgooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAK4b4m6PqOr6Xp4060e5eG63uqYyF2nnmu5opSSasyZR5lY8TOia5Na+SukXobp80JH6mvT/BthdaZ4TsLO8iMVxGrb0JBxliR09iK3aK5sNhKeHvydQUXe7dwooorqKCiiigAooooAKKKKACiiigAqndaTp18CLuwtZwf+esKt/MVcooC1zmrv4f+F7wgvpMSEZwYWaP9FIFYt18ItAmbdBcXtv/ALKyBl/UZ/Wu/opcqM3Sg90eS3HwZmBJttbRueBJblcfiGP8qxLn4UeJ4EJjWzuSP4YpsE/99AV7rRS5EZPC030PnC78F+JrIAzaLdkZx+6Tzf8A0DNY9za3NmwW6tpoGPQSxlSfzr6nprosilXUMp6gjIpciM3go9GfKfBpCor6auvDOhXrFrjR7GRz1YwLuP44zXmPxM8KaLoVjY3GmWf2eSe4KyYkYgjaTwCSBz6VMlyq7MZYOSV0zzErTeldND4ciuLXzI7iRWxnkAirOieANR8QxXTWVzbBrZwrLNlc5GeCAa56GMpVpOMHqjOeEqxtdbnIbqaXrsrv4XeKrY/JYR3C46wzr/IkGsS68K69Zuy3GjXybereQxX8wMGuxWMZUpx3RjocvzWpbuFFZvlNHIVYFWBwQRgg1aiViQACSeABUVYqSITaZrR3GAKlFwTVvSvB3iDVGHkabMqf35h5a/m3X8K7Ow+E0rxg6jqhiJxlLVcn/vpv8K4/qqb0R0whWqbI84vNRjtEzIcufuoOpqvY+H/EXiicSWem3Esf8Lbdsag/7RwK960nwB4a0dhJDpqTTjnzrk+a+fX5uB+AFdMAAAAMAdq2jhktzupYdQ1erPEdH+C2p3G19X1CG0TvHCPMf8+AP1rvtH+GXhfSNr/Yftkwx+8uz5nI77fu/pXYUVtGlCOyOiyGxxRwxrHEipGowqqMAD2FOoorQYUUUUAFFFFABRRRQAUUUUAFc/47/wCSeeJf+wVdf+imroK5/wAd/wDJPPEv/YKuv/RTUAHgT/knnhr/ALBVr/6KWugrn/An/JPPDX/YKtf/AEUtdBQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQB5t8YGxpmlLjrcsf/ABw1y2l/8eS/Sul+MbhbDSF7m4f/ANB/+vXLWM8cdmAT0Wvns5TbViqMkqjv2O6+GgG3WGHe4Qf+O13lcD8LmD2Oqv3NyP8A0EV31ezhVy0ILyM4u6uFRS20E5zLDHJ/voDUtFdBRDFaW0BzDbxRn1RAP5VNRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHP+O/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0eO/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFebfGBsaZpS463LH/AMcNek15l8Y3C2GkL3Nw/wD6D/8AXrKur02iZO0WzmtL/wCPJfpXbfDQDbrDDvcIP/Ha4WxnjjswCei13HwuYPY6q/c3I/8AQRXgZVF/WZSNKklywiv60O+ooor6QkiltoJzmWGOT/fQGkitLaA5ht4oz6ogH8qmooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/Hf/ACTzxL/2Crr/ANFNXQVz/jv/AJJ54l/7BV1/6KagA8Cf8k88Nf8AYKtf/RS10Fc/4E/5J54a/wCwVa/+ilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAOX8a+D18XWdrGLw2sltIXRtm8HIwQRkVy0fwu1JBs/tS2K467GBr1Gis50YT+JGcqUZO7Oe8JeGP8AhGLGeBrr7Q80nmMQm0DjGBzXQ0UVaSSsi4pRVkFFFFMYUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBz/jv/knniX/sFXX/AKKajwJ/yTzw1/2CrX/0UtHjv/knniX/ALBV1/6KajwJ/wAk88Nf9gq1/wDRS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABXL+NfB6+LrO1jF4bWS2kLo2zeDkYIIyK6iik1fRiaTVmeXR/C7UkGz+1LYrjrsYGuw8JeGP+EYsZ4GuvtDzSeYxCbQOMYHNdDRWcKMIO8URGlGLugooorU0CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8d/8AJPPEv/YKuv8A0U1dBXP+O/8AknniX/sFXX/opqADwJ/yTzw1/wBgq1/9FLXQVz/gT/knnhr/ALBVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWjx3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8d/8AJPPEv/YKuv8A0U1dBXP+O/8AknniX/sFXX/opqADwJ/yTzw1/wBgq1/9FLXQVz/gT/knnhr/ALBVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWjx3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8d/8AJPPEv/YKuv8A0U1dBXP+O/8AknniX/sFXX/opqADwJ/yTzw1/wBgq1/9FLXQVz/gT/knnhr/ALBVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWjx3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8d/8AJPPEv/YKuv8A0U1dBXP+O/8AknniX/sFXX/opqADwJ/yTzw1/wBgq1/9FLXQVz/gT/knnhr/ALBVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWjx3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8d/8AJPPEv/YKuv8A0U1dBXP+O/8AknniX/sFXX/opqADwJ/yTzw1/wBgq1/9FLXQVz/gT/knnhr/ALBVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWjx3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8d/8AJPPEv/YKuv8A0U1dBXP+O/8AknniX/sFXX/opqADwJ/yTzw1/wBgq1/9FLXQVz/gT/knnhr/ALBVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA5/x3/yTzxL/ANgq6/8ARTUeBP8Aknnhr/sFWv8A6KWjx3/yTzxL/wBgq6/9FNR4E/5J54a/7BVr/wCiloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACuf8d/8AJPPEv/YKuv8A0U1dBXP+O/8AknniX/sFXX/opqADwJ/yTzw1/wBgq1/9FLXQVz/gT/knnhr/ALBVr/6KWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACioL28g06wuL26k8u3t4mmlfBO1FGScDk8A9KrRa5ps12bVLpfNFot6QylQIWJAckjA5U8deKANCisPSvGOga3eizsNQEs7IZI1aJ0Eqjq0ZYAOPdcitHTNStNY02DULCXzrWdd8cm0ruH0IBH40AW6KKKACiiigAorBt/Gnh661VdNh1JGuHkaJD5biORxnKLIRsZhg8Ak8Vp2+pWl1qF5Ywzbrmz2eem0jZvGV5Iwcj0oAt0UUUAFFFFABRRRQAUUUUAFFFVLvUrSxuLOC5l2S3sxgt12k73Cs+OBx8qsecdKALdFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFAHP8Ajv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLR47/wCSeeJf+wVdf+imo8Cf8k88Nf8AYKtf/RS0AdBRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRWVZ+JNJ1D+zfst35n9pRPLafu3HmKmNx5HGNw6460AatFYNv408PXWqrpsOpI1w8jRIfLcRyOM5RZCNjMMHgEnitOw1K01NJ3s5fMWCd7eQ7SNsiHaw5HOD36UAW6KKKACiiigAorBuvGnh6z1RtOuNSRLhZFic+W5jjc4wrSAbFY5HBIPNaf8AaVp/a39l+d/pvkfaPK2n/V7tuc4x14xnNAFuiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArn/AB3/AMk88S/9gq6/9FNXQVz/AI7/AOSeeJf+wVdf+imoAPAn/JPPDX/YKtf/AEUtdBXP+BP+SeeGv+wVa/8Aopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiuf1jxba6RfyWS2Go380MIuLgWUIf7PGSQGfJHXa2AMng8Vs2d3BqFjb3trKJbe4jWWKRejKwyD+INAE9FYmpeJE0vXNP02fTb4pfSiGK8QRmESFWbafn3ZwhP3ce9JrHie30rUItOjsr7Ub+SMzfZrKMMyR5xvYsyqozwMnnBxnFAG5RWdoutWmvWH2uzMihZGilimQpJFIpwyOp6MP8A6/Q1o0AFFFFABRRRQAUUVk+I9dHhzSZdSk069vYIVZ5haCMtGiqWLne68ADtk+1AGtXP6pqV3pnizR0eXOl6gsloyFR8lwBvjbdjPzKsi4zjO2ty3mW5top0BCyIHAPXBGa4/wAbzXeo2FzpNvoetNIjRzW+pWiW7LFKhDq6h5kJwRgggZ5HSgCKx+ISSxeJri7thHb6WGmtCp5uoQzxhh6kyROBj1X15Le78T315ZeHf7TjtL6DTo7zVL4W6O++RmCxxqRtGCj5Yg8BeMkmube30S50TwPf2o1TakXlRaZHbK8+oxxsjgv8+1VDxrJuLEYbGfmrqVSPxTqb6ro95d6Jr1gv2S6hurYMfLPzqssZOGHJZWVu5weooAwPEt34k0+51bQ9R1SO+06Xw3qFzHL9nSOR3UIuHwMfLuOCuAd3I4zVuPTF1rXLrSnlaJb3whBbmReqh3lXP61oT+ALu/vr3UNS8QS3N7daXcaaT9mCRRpLtwUQNxggnkknPXgVeuPBFrfS3Avblpba40ePSpIlTacKzHzA2Tg/NwMcY6mgCja3d3Y6roWn+KdGtlmjcxadqdnKWhMvlsNpUgNGWQNx8w7Z6VgeErfxVB8MLLVrPXIYvs1o0sGnm0VopETJCu5+fLY6qRjPQ456608K6lJf6fPrfiBtTh05/NtoharCTJtKh5GBO8gM2MBRk5xVzSPDn9leDIvD32rzfLtWt/P8vbnIIztyfXpmgDltV8dSXd/p1la3dzpkE+mxajNc2+myXsuJc7I1Coyr91iWYHPAA64fpnji4sY9Z+2ve6nZWGnvfx3sumS2bts+9EwdFUt0IKgcZyOK2R4OmtbXSZNM1ZrPU9PsY7E3PkB47iJQOJIyemQSMMCMnnk1bsPDLBb+TW799VuL+H7PNuTy4liwRsSME7QdxySST68CgDJa18arorav/b9t9u8kz/2cLNPsw4z5Yf7/ALbs9ecdqyPDms+J/EOpaFAdYWG2l0C11C8b7MheWRnYMF4AXcBye2OACc1DqK6haNceFrXWtb1Owt4ljuY7LT43uIImHEbTl1BJX0Uvjnqc1ueDvD+nGxsNQs9Q+3WJ0WLSgrQlCwjZ9xbJyDklSuOCDzQBnwQXPhLSNL0TxDpUF7oVncwx22p20mGiYSAQtLGQCp3FcspIyeRzVBbfWNH1rx9qEfiK8lmsbFJgJLeDEjfZ3KFsRj7pxjGM45zUV0ItKI0K91bVL7wvo00IuGjsE22+wq8cU027LKvyE7UyBjcetbfiJLa01LxFYR/br+/8Q2IX7NY2wdraMRtF5jFnVSCTwCRnBAzQBTu/E2uaTY6JZ3+qk32sRNdSXUGmPOLSJVTKRxxglmLOBubgcnHQVb0HxRfnxDBpi3eoaza3UchE93pEtm9vIq7gGYxojKwBHQEHHXPCaneaZc+DLHXLS5v4L3RnW1ieGBRcJMSsLQvFIQDuJXKkjsQehq14btNW1XUbi78QLrEdwkDQwGWKG2hRX+8UWKaQ7+B8zHjtjmgDK0PxD4on0/UmbUI7rXY9OkmbRLmz+zy29yPuiPgeZF2zk5+X5hmt7wPql1qK3i3OujU/LEeY57T7Nc2zkHckkeBgdNpx68nrVdPAd/Oytqfii9uZLeyks7KeGMQzQhyuXZwSXf5F54B5yOasWulanoF9c65etdeItSuY4rTbZQQ24jiTewO15AD8zHJ3Z5GABmgDa8S3s+m+FdYvrVgtxbWU00TEA4ZUJBwevIrnrK08a32iW+qjxBbxX0sCzLYfY0NvkqCEZvvknoWBHU4HaneIdX1HUfC2tWbeFtYtfN065USzNbMoPlNgYjmZiScAYU8kVFpnhbXB4etNOj8U3MGmtbIhjNqpuY0KjKLMTxjkAlSw9cjNAFPTvEus+KPEenNp9+NP0qXRoNSniMCyvvMrhkUkdwuCefu8DJzWXZ+N77WrFdVXXtQ05pwZLeyh8Oz3EKp/CHk8slyRjJRlHPHrXd6d4XttK1xL6zcRW0WmxadFahOEWN2YHdnn72MY7ZzXKaVbaj/bWoeGNJ1LUtFs4E89IbqzhkKRO7L/AKPIsh2rlWwHUkDH0oAvR+KdY1y08P2VjCNL1LVIpprmS4gY/Zo4WCuVRsElmZdu7oDkjtVPV7PXbDxh4Phv9VXU7F9RkZZZIFimjkFvNx8mFZSCewIx3zxr+ItG0zSPD1lejUbnTZNFXFteqPOk+bClGU58zedoI6k4wQa5/TkuNY8Y6Hc+Ib/VbfULaSSWxt7nT1toJx5bK4ULI/z4YN8zZAU4UZNAHp9Fchf+O5NLtzcXvhLxBDAHWPzCtsRuZgqgATZOSQOB3rr6ACiiigAooooAKKKKACiuTT4g6W+Lj7FqQ0ppfKXVTbj7KTu2g5zu27uN+3b74rrKACisLU/Fmm6X4k0vQJvOe+1It5YiUFYwASC5yMA7WA65wfQ1U1bxjLo6Xs1x4X1xrS0DtJdJ9m8souSXGZgcYGeQD7UAdRRUFldC9sLe7WKSITxLII5Rh03DOGHYjPNT0Ac/47/5J54l/wCwVdf+imo8Cf8AJPPDX/YKtf8A0UtHjv8A5J54l/7BV1/6KajwJ/yTzw1/2CrX/wBFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUVz+seLbXSL+SyWw1G/mhhFxcCyhD/Z4ySAz5I67WwBk8HigDoKKgs7uDULG3vbWUS29xGssUi9GVhkH8Qay9S8SJpeuafps+m3xS+lEMV4gjMIkKs20/PuzhCfu496ANuisPWPE9vpWoRadHZX2o38kZm+zWUYZkjzjexZlVRngZPODjOKt6LrVpr1h9rszIoWRopYpkKSRSKcMjqejD/6/Q0AaNFFFABRRRQAUUUUAFFZPiPXR4c0mXUpNOvb2CFWeYWgjLRoqli53uvAA7ZPtWlbzLc20U6AhZEDgHrgjNAElFYWpeIbvT72SCPwzrN7GgB+0W32fy2yM8b5VPHTkdqpx+PNPm0PStRisdRkl1UMbOxjiVriQL1bAbaFxg7i2MEc80AdTXP8AifUrvRpNJ1COXGni8WC/jKgjy5fkV84yNshTp2JzVvRPEFrrguUihuba6tXCXFpdR7JYiRkZGSCCOQQSD68VY1jS4Na0W90u5/1N3C0LEdRuGMj3HUfSgDAtfGYl8b6no08KRWFrCWiuycB5IwjTL/wESx/k3pxl2mv+IdTttGs4LiO2vdb+0X6zTQhvsdmrLsVV43OVkj+9nBLE5wBUt18NheeFrHSZtZlN3Dcyz3V+IcPdCbeJ1I3fKHVyOpxgeldBrfhxNU+w3Fpdyadf6eSbS5hQNsDDDIynhkIAyOOgIIxQBz13P4o0LxZ4bsZtbXUNN1C6kjmeS2jjmBWGRgvygAqSAcgAjbjJBrG8HjI+HI5502+6f9sq6YeDtRu/EGla1q/iF7ufTZWeGGK1EMOGRkb5dxO47gckn7uABk07TPA8dhD4fhlvRcR6TaXFqymHaJxLtBP3jtxt6c5z2oAxoILnwlpGl6J4h0qC90KzuYY7bU7aTDRMJAIWljIBU7iuWUkZPI5pnhnTfEV9/wAJC9jry6bAmtXogjS0SXe3mHJkLc4zxhdvTrzxq2/ga8SztNIuvEEtzoFpJG0Vm1sqyssbBo43lz8ygqvRQTjk02x1W28M32q6Va2up6zcG9lvbn7Dagi285i4RizAE4PQZJGDgZoAy7r4gXk+jeHkBNheamk7XVxBZS3fk+QwR/LjQEklyMFuAOuTjNvQPFN4PEdvpr3uoavY3McjG5utHltHtnVd3zMY0RlYAgcAg4654m0Tw5Z6j4T0i40jWZVubSWea01COHaV8yRi8bxtnI52spwcoDwRxTXWk0fxDdXuuX2par/ZwFvNdWtmIrKx3hWO5d5ZmwVJb5toPbmgC7pDeKfFGkQ6/Br0emx3iedZ2S2aSosZ5TzWPzMxGCdpXGcDpWFpXiHxfrQ8O2n9pwWd1fT6ml5ItukgQQyhVCAgZIHygnrnJzireowXnhe6XQNA1jUXhmRp49OtNPjuJrSIsclJHdVRNxIUPu9BnFQaJbaR4e8Pabr1rd391baNPd27Wj24Fy0tzMg8pgzDDqxA5+9nP1ALl9ZXvhXStZt9T0qHW/C9zPPd3MkUm24iSRi8m+MjDhSScqwOB04p0Wn3s3xbjuIvEN21u+lC4VBDCVMRm4izszt987v9qodN8O6pqcGo6PG+saL4fuWd5LW8trdn2ysTJHFKkrbVJLdVJG7g12MWgJB4nTV4pgscenixW3CcAB9wO7P4Yx+NAHA2Pi7xHZeB9J1fUtTSe71qZLaAJYGRbYYdmlKRDdIxVCdowMkdgTV7T/Fd9a61p8Mep6prdvdzrBPHc6FNbNBu4EiuIlXaDjIbsc5456C38FwxeDdO0GS+l87T9r299CoR45VJIdQcjuQQcggkd6ltPD2qSalbXet6818loxeC3gthbxl8EbpMMxcgE4GQO+OmADC0jXNcm8XTWmoaolvP9ouFXR7m1EayQLnynglx+8JAUtyeC3AxUngbWtY1G/aLV9XV7z7OXu9KuLP7PLaS7h/q+PnixkbiWz8pzzirY8E3k1/bfbvEV1dadZ3ElzawNHiZHdXUZn3bmCiRtvAPTJOKn0bwlfWWsWmoarr0uqNYWz2tnvgEbKrldzSMCTI5CKM8dzjJoA6qisTTvEiX+v3ujPpt9Z3NrEs2bgR7ZY2ZlDIVdjglD1APtV7VtVs9E0ybUb+Ux28IBYhSxJJAAAHJJJAAHUmgC7RXPad4utr3UoNPutO1LTLm5Vmtlv4Qgn2jJClWYZA52nBx24qjf+O5NLtzcXvhLxBDAHWPzCtsRuZgqgATZOSQOB3oA6+iiigAooooAKKKKACiiuTT4g6W+Lj7FqQ0ppfKXVTbj7KTu2g5zu27uN+3b74oA6yiisLU/Fmm6X4k0vQJvOe+1It5YiUFYwASC5yMA7WA65wfQ0AbtFcvq3jGXR0vZrjwvrjWloHaS6T7N5ZRckuMzA4wM8gH2robK6F7YW92sUkQniWQRyjDpuGcMOxGeaAJ65/x3/yTzxL/ANgq6/8ARTV0Fc/47/5J54l/7BV1/wCimoAPAn/JPPDX/YKtf/RS10Fc/wCBP+SeeGv+wVa/+ilroKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigArL1XXrXR5I0uLbUpjICQbTT57gD6mNGA/GtSigDm9Whl8RaXfw+HNXi06/3eTcXH2UO6nZkIwOCDh1OeozT/At1FdeCtL8q1FqsEZtPJV9wQwsYjhj1GUOD3FGo+Eba+1Ka/ttS1LTJ7lQtybCYIJ8DALAqfmA43Lg4xzwKtw+HbK0g0m3smntLbTH3QwwyYV/kZcSZyWHzFuudwBzQBl+Mv+Qh4S/7Dif+k89XdavtJ8MmfWZoGa+uxHbJHCC010ylvLjRe5yzfmSeBV3U9It9Vm06Wd5VawuhdxbCAC4R0w2QeMOemO1ZGr+C49V8Rxa4uuatZ3cMJgiW3aEpGp+8VEkbYJ7kdQMUAT+EdJvdOsby61PYuoandte3EUZykJZVURg99qooJ7nJroKo6Vp82m2rQz6pe6izOWEt55e9RgDaNiKMcZ6Z5PNXqACiiigAooooAoarq8GkRRyXEF9KHbaBaWctwR9RGrED3NZPi26S9+Gmv3MaTIkmk3LBZomicfum6qwBB9iK6Wqmq6dDrGj3umXDOsN5byW8jRkBgrqVJGQRnB9KADSv+QPZf9e8f/oIrD8cXMzaTDolnIUvdamFlGy9Y4yCZZP+Axhzn1xWvdaRHdabb2IubmFIHhdXhfa7eWysFJx0O3BHcE0kmjW8viGDWpHle4gtnt4YyR5aBmBZgMZ3HaoznoPrQBVvk0Lw3aRavdJFbRadam2ik5ykRK/Io7klEAAGTgAVS8K2V/NqGp+I9St2tJ9UESRWbfeggjDbN/8AtkuxI7ZA7VN4j8Iw+I7/AE+8k1XUbOSwYvCtq0RTeeN5WRGBYDoe2Tir+kaTcaX53n61qOpeZtx9t8r93jP3fLjTrnnOeg6UAadFFFABRRRQAVl6rr1ro8kaXFtqUxkBINpp89wB9TGjAfjWpRQByfglxJdeKXwwZtackMpDAGGHGQeRxineA8f2frG37n9t3+30/wBe2f1zVrUPCVvealPqFrqWpaZc3Kqty1jMqCfaMAsGVhkDjcMHGOeBU9r4ZsbC10q2spLm3g02VpUjjlOJiysD5hOS+S5b/ewaAMLxfp83iPwndvoGpW0dm3mm8gEQ23uw4eNpBgpnYylhk/lVw69otholt4sNo63OrWtusMEY3TTkqWjhVe5+dunuTwKdeeBbC7nuyl/qdrZ3rmS7sba4CQTs33iRjcu7+Lay5yc9abqXga3vtdt9Wg1jVNPmtbf7Nbx2ph8uFO+xXjbaTwCR2AHSgDEl8MX1xo8VrfatBpms6vrK6oyKolCtGFYRIDwxVYkJJyCVJwRWzY3Gq6J4ttNEvtUk1W1v7WaaGWeKNJoXiKbgfLVVKkSDBxkEY5zVyXwnDd6T9h1HVNSv3ScXEF3M6JPbuBgFGjRQMc9j94g5BxUukeGbbSr+TUJLy91DUJI/J+1XsgZ1jznYoUKqjPJwOe+cUAbdFFFABRRRQAVysH/JV7//ALAdt/6Pnrqqz00i3TxDNrQeX7TLapaMuRs2I7uCBjOcue/pQBi+O/8Ajx0bd9z+27Hdnp/rlx+uKPGWP7Q8JY+//baY9f8AUTZ/TNaVz4asb211W1vXuLm31GUSvHLLxEQqgeXjBTBQMO4bJqtYeEbe01SDUbrUtT1O4tlZbY30wYQbhglQqqCSONxycE880AVNT/4nvjrTtKHzWmkINRu/QzNlYEP0+d/+ArXWVnaZo1vpdxqNxE8sk9/cm4mklIJztChRgD5VVQAP51o0AFFFFABRRRQAU10WSNkcZVgQR6inU2RDJE6B2QspG5eq+496AOG16ODVrT/hX+gQqsCwpDfTLzHY23HyZ7yMowq9R94+/Va7rNp4c0G81e+Yi2tIjI2Op7AD3JwPxrndM+HzaNam20/xb4ghiZ2lbm1ZndjlmZjASxPqSTXSrpkY1S5vmmnk+0QxwtBIwMShCx3KuOGO7k98D0oA8qh1bRP7Q8NahdeIdHudav8AWhc3/kX0biFfs0ypEMNwiblQHuxJ6tXb+Lv+JxqWk+Fk5S8k+1XwHa1iIJB/33KL7gtWvf8Ah7T9RutOuJI9j2Fz9piEaqAzbHTDccjDk8Y5AqS30a3t9dvdY3yyXV3FHCd5BWONM4VOOASzE5zkn6UAaNFFFAHP+O/+SeeJf+wVdf8AopqPAn/JPPDX/YKtf/RS0eO/+SeeJf8AsFXX/opqPAn/ACTzw1/2CrX/ANFLQB0FFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQBl6rr1ro8kaXFtqUxkBINpp89wB9TGjAfjWfq0MviLS7+Hw5q8WnX+7ybi4+yh3U7MhGBwQcOpz1Ga6Sue1HwjbX2pTX9tqWpaZPcqFuTYTBBPgYBYFT8wHG5cHGOeBQAeBbqK68FaX5VqLVYIzaeSr7ghhYxHDHqMocHuKreMv+Qh4S/7Dif8ApPPWpD4dsrSDSbeyae0ttMfdDDDJhX+RlxJnJYfMW653AHNTanpFvqs2nSzvKrWF0LuLYQAXCOmGyDxhz0x2oApa1faT4ZM+szQM19diO2SOEFprplLeXGi9zlm/Mk8CmeEdJvdOsby61PYuoandte3EUZykJZVURg99qooJ7nJqDV/Bceq+I4tcXXNWs7uGEwRLbtCUjU/eKiSNsE9yOoGK2dK0+bTbVoZ9UvdRZnLCW88veowBtGxFGOM9M8nmgC9RRRQAUUUUAFUNV1eDSIo5LiC+lDttAtLOW4I+ojViB7mr9FAHNeLbpL34aa/cxpMiSaTcsFmiaJx+6bqrAEH2Ira0r/kD2X/XvH/6CKNV06HWNHvdMuGdYby3kt5GjIDBXUqSMgjOD6VFdaRHdabb2IubmFIHhdXhfa7eWysFJx0O3BHcE0AZHji5mbSYdEs5Cl7rUwso2XrHGQTLJ/wGMOc+uKuXyaF4btItXukitotOtTbRSc5SIlfkUdySiAADJwAKtSaNby+IYNakeV7iC2e3hjJHloGYFmAxncdqjOeg+tZ3iPwjD4jv9PvJNV1GzksGLwratEU3njeVkRgWA6Htk4oAh8K2V/NqGp+I9St2tJ9UESRWbfeggjDbN/8AtkuxI7ZA7V1FZmkaTcaX53n61qOpeZtx9t8r93jP3fLjTrnnOeg6Vp0AFFFFABRRRQBl6rr1ro8kaXFtqUxkBINpp89wB9TGjAfjWP4JcSXXil8MGbWnJDKQwBhhxkHkcYrrK5/UPCVvealPqFrqWpaZc3Kqty1jMqCfaMAsGVhkDjcMHGOeBQBV8B4/s/WNv3P7bv8Ab6f69s/rmqfi/T5vEfhO7fQNSto7NvNN5AIhtvdhw8bSDBTOxlLDJ/Kt218M2Nha6VbWUlzbwabK0qRxynExZWB8wnJfJct/vYNULzwLYXc92Uv9TtbO9cyXdjbXASCdm+8SMbl3fxbWXOTnrQA069otholt4sNo63OrWtusMEY3TTkqWjhVe5+dunuTwKxJfDF9caPFa32rQaZrOr6yuqMiqJQrRhWESA8MVWJCScglScEVt6l4Gt77XbfVoNY1TT5rW3+zW8dqYfLhTvsV422k8AkdgB0q1L4Thu9J+w6jqmpX7pOLiC7mdEnt3AwCjRooGOex+8Qcg4oAp2NxquieLbTRL7VJNVtb+1mmhlnijSaF4im4Hy1VSpEgwcZBGOc11lYmkeGbbSr+TUJLy91DUJI/J+1XsgZ1jznYoUKqjPJwOe+cVt0AFFFFABRRRQBysH/JV7//ALAdt/6Pno8d/wDHjo277n9t2O7PT/XLj9cVtJpFuniGbWg8v2mW1S0ZcjZsR3cEDGc5c9/Sq9z4asb211W1vXuLm31GUSvHLLxEQqgeXjBTBQMO4bJoAzfGWP7Q8JY+/wD22mPX/UTZ/TNN1P8A4nvjrTtKHzWmkINRu/QzNlYEP0+d/wDgK1bsPCNvaapBqN1qWp6ncWystsb6YMINwwSoVVBJHG45OCeea0NM0a30u41G4ieWSe/uTcTSSkE52hQowB8qqoAH86ANGiiigAooooAKKKKAGuiyRsjjKsCCPUVw2vRwataf8K/0CFVgWFIb6ZeY7G24+TPeRlGFXqPvH37mRDJE6B2QspG5eq+49647TPh82jWpttP8W+IIYmdpW5tWZ3Y5ZmYwEsT6kk0AdFrus2nhzQbzV75iLa0iMjY6nsAPcnA/GvLYdW0T+0PDWoXXiHR7nWr/AFoXN/5F9G4hX7NMqRDDcIm5UB7sSerV6qumRjVLm+aaeT7RDHC0EjAxKELHcq44Y7uT3wPSoL/w9p+o3WnXEkex7C5+0xCNVAZtjphuORhyeMcgUAZHi7/icalpPhZOUvJPtV8B2tYiCQf99yi+4LV1lZ1vo1vb67e6xvlkuruKOE7yCscaZwqccAlmJznJP0rRoAK5/wAd/wDJPPEv/YKuv/RTV0Fc/wCO/wDknniX/sFXX/opqADwJ/yTzw1/2CrX/wBFLXQVz/gT/knnhr/sFWv/AKKWugoAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooA+V/i9/yVTWf+2H/oiOuNWuy+L3/JVNZ/7Yf+iI641a9il8C9D04fAvQmWplqFamWtSJEq1KtRLUq1RhIkWpBUa1IKDCRIKkFRipBQYSHinimCnipZzyHinCminCkYSHinimCnipZhIcKeKYKeKhmMh4pwpopwqWYyHCnimCnioZjIeKcOlNFOHSs2YseKeKYKeKzZmyQU9aYKetYyJJRUq1EKlWueRSJVqVaiWpVrlmaxJVqZahWplrjmbxJVqVaiWpVrjqHTAmWpVqJalWuCodcCValWolqVa4ah20yValWolqVa4ZnbTJVqZKhWpkrmZ2QNDTP+P6L8f5Gt+sDTP8Aj+i/H+Rrfr7jhn/dJf4n+SMMT8a9Aooor6I5zn/Hf/JPPEv/AGCrr/0U1HgT/knnhr/sFWv/AKKWjx3/AMk88S/9gq6/9FNR4E/5J54a/wCwVa/+iloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACvlf4vf8lU1n/th/wCiI6+qK+V/i9/yVTWf+2H/AKIjrqwnxv0OnC/G/Q41amWoVqZa9JHVImWpVqJalWqMZEq1ItRrUi0zCRIKkFRipBQc8iQU8UwU8UjCQ8U8UwU8VLOeQ4U8UwU8UmYyHinCminCoZhIeKeKYKeKlmMhwpwpopwqGYyHinimCnioZjIcOlPFMHSnis2ZMeKkFRipBWUiGPWpRUS1KKwkCJVqVaiWpVrmmWiValWolqVa5Jm0SZalWolqVa46h0QJVqZahWplriqHVAlWpVqJalWuCodlMlWpVqJalWuCodtMlWpVqJalWuOZ20yZK0NM/wCP6L8f5Gs9K0NM/wCP6L8f5GujL/8Ae6X+KP5o6ZfA/Q36KKK/TzzQrn/Hf/JPPEv/AGCrr/0U1dBXP+O/+SeeJf8AsFXX/opqADwJ/wAk88Nf9gq1/wDRS10Fc/4E/wCSeeGv+wVa/wDopa6CgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigDKvPDGgahdvdXuh6Zc3MmN801pG7tgYGSRk8AD8Kh/4Q3wt/wBC1o//AIAxf/E1t0VXM+4+Z9zF/wCEO8Mf9C5pH/gDF/8AE0f8If4Z/wChc0j/AMAo/wD4mtqijnl3C7Mb/hEfDX/Qu6T/AOAUf/xNL/wiXhr/AKF7Sf8AwCj/APia2KKOeXcLsx/+ET8N/wDQv6V/4BR/4Uv/AAinhz/oAaV/4Bx/4Vr0Uc8u4jI/4RXw7/0ANL/8A4/8KP8AhFfDv/QB0v8A8A4/8K16KOeXcDJ/4Rbw9/0AdL/8A4/8KP8AhF/D/wD0AtM/8BI/8K1qKOeXcVkZX/CL+H/+gFpn/gJH/hR/wjGgf9APTP8AwEj/AMK1aKOaXcOVdjK/4RnQP+gHpv8A4CR/4Uv/AAjOg/8AQE03/wABI/8ACtSijml3Fyx7GX/wjWg/9ATTf/AVP8KP+Eb0L/oC6b/4Cp/hWpRS5n3Dkj2Mz/hG9C/6Aunf+Aqf4Uf8I5of/QF07/wFT/CtOijmfcOSPYzP+Ec0P/oDad/4Cp/hS/8ACO6H/wBAbT//AAFT/CtKijmYvZw7Izf+Ed0T/oD6f/4Cp/hR/wAI9on/AEB9P/8AAZP8K0qKLsPZw7Izv+Ef0X/oEaf/AOAyf4Uf8I/ov/QIsP8AwGT/AArRoouw9lD+VGd/YGjf9Amw/wDAZP8ACl/sHR/+gTY/+A6f4VoUUheyh/KvuM/+wtI/6BVj/wCA6f4Uv9h6T/0C7L/wHT/Cr9FA/ZQ/lRR/sTSv+gZZf+A6/wCFH9jaV/0DLP8A78L/AIVeopWQezh2RS/sfS/+gbZ/9+F/wpf7I0z/AKB1p/34X/CrlFLlj2D2cOxT/snTf+gfa/8Aflf8KX+ytO/58LX/AL8r/hVuijkj2HyR7FX+zNP/AOfG2/79L/hR/Zth/wA+Vt/36X/CrVFL2UOyHyrsVv7Osf8Anzt/+/S/4Uv9n2X/AD6W/wD37H+FWKKXsaf8q+4LIr/YLP8A59IP+/YpfsVp/wA+sP8A37FT0UvYUv5V9wyD7Ha/8+0P/fApfslt/wA+8X/fAqail9Xo/wAq+5DuyNYIUYMkSKw7hQKkoorSMIwVoqwNt7hRRRVCOf8AHf8AyTzxL/2Crr/0U1HgT/knnhr/ALBVr/6KWjx3/wAk88S/9gq6/wDRTUeBP+SeeGv+wVa/+iloA6CiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACsq88MaBqF291e6HplzcyY3zTWkbu2BgZJGTwAPwrVopptbDTa2MT/AIQ3wt/0LWj/APgDF/8AE0v/AAh3hj/oXNI/8AYv/ia2qKfNLuHM+5i/8If4Z/6FzSP/AACj/wDiaX/hEfDX/Qu6T/4BR/8AxNbNFHPLuF2Y/wDwiXhr/oXtJ/8AAKP/AOJo/wCET8N/9C/pX/gFH/hWxRRzy7iuZH/CKeHP+gBpX/gHH/hR/wAIr4d/6AGl/wDgHH/hWvRRzy7gZH/CK+Hf+gDpf/gHH/hS/wDCLeHv+gDpf/gHH/hWtRRzy7isjJ/4Rfw//wBALTP/AAEj/wAKX/hF/D//AEAtM/8AASP/AArVoo5pdwsjK/4RjQP+gHpn/gJH/hR/wjOgf9APTf8AwEj/AMK1aKOaXcOVdjL/AOEZ0H/oCab/AOAkf+FH/CNaD/0BNN/8BU/wrUoo5n3FyR7GX/wjehf9AXTf/AVP8KX/AIRvQv8AoC6d/wCAqf4Vp0UuZ9w5I9jM/wCEc0P/AKAunf8AgKn+FH/COaH/ANAbTv8AwFT/AArToo5n3F7OHYzf+Ed0P/oDaf8A+Aqf4Uf8I7on/QH0/wD8BU/wrSoouw9nDsjN/wCEe0T/AKA+n/8AgMn+FL/wj+i/9AjT/wDwGT/CtGii7D2UOyM7/hH9F/6BFh/4DJ/hR/YGjf8AQJsP/AZP8K0aKVxeyh/KjP8A7B0f/oE2P/gOn+FH9haR/wBAqx/8B0/wrQooD2UP5V9xQ/sPSf8AoF2X/gOn+FL/AGJpX/QMsv8AwHX/AAq9RSsh+yh2RR/sbSv+gZZ/9+F/wpf7H0v/AKBtn/34X/CrtFHKuwezh2RT/sjTP+gdaf8Afhf8KP7J03/oH2v/AH5X/CrlFLkj2HyR7FT+ytO/58LX/vyv+FL/AGZp/wDz423/AH6X/CrVFL2cOyDlj2Kv9m2H/Plbf9+l/wAKX+zrH/nzt/8Av0v+FWaKXsqf8q+4dkV/7Psv+fS3/wC/Y/wo+wWf/PpB/wB+xViil7Gl/KvuGQfYrT/n1h/79ij7Ha/8+0P/AHwKnoo+r0v5V9yHdkP2S2/594v++BTlghRgyRIrDuFAqSihUKSd1FfcHMwooorUQVz/AI7/AOSeeJf+wVdf+imroK5/x3/yTzxL/wBgq6/9FNQAeBP+SeeGv+wVa/8Aopa6CuD8F+NPCtr4F8PW9x4l0aGeLTLZJI5L+JWRhEoIILZBB4xW5/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAdBRXP/8ACd+D/wDoa9D/APBjD/8AFUf8J34P/wChr0P/AMGMP/xVAHQUVz//AAnfg/8A6GvQ/wDwYw//ABVH/Cd+D/8Aoa9D/wDBjD/8VQB0FFc//wAJ34P/AOhr0P8A8GMP/wAVR/wnfg//AKGvQ/8AwYw//FUAHjv/AJJ54l/7BV1/6KajwJ/yTzw1/wBgq1/9FLWH408aeFbrwL4ht7fxLo008umXKRxx38TM7GJgAAGySTxijwX408K2vgXw9b3HiXRoZ4tMtkkjkv4lZGESgggtkEHjFAHeUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQUVz/APwnfg//AKGvQ/8AwYw//FUf8J34P/6GvQ//AAYw/wDxVAHQVz/jv/knniX/ALBV1/6Kaj/hO/B//Q16H/4MYf8A4qsPxp408K3XgXxDb2/iXRpp5dMuUjjjv4mZ2MTAAANkknjFAH//2Q==", + "image/png": "iVBORw0KGgoAAAANSUhEUgAABLAAAASwCAIAAABkQySYAAEAAElEQVR4Aey9B6AsSVn2P+Gcc+/d3Xvv5l2WDYSFJckCkgRRAQEREQSJghERMYABE6Lfp5gjwSyff5QMggGzoEQBBfYSBGUTbGTjjXvDOTPzf96q7pqa7p45PTM9Mz3n/GbvzqmurvDWr3rq7aerurvR4AMBCEAAAhCAAAQgAAEIQAAC25JAU63u9Xrbsu00GgIQgAAEtjuBZhM/uN2PAdoPAQhAYDsTkB9sbef203YIQAACEIAABCAAAQhAAALbmQCCcDv3Pm2HAAQgAAEIQAACEIAABLY1AQThtu5+Gg8BCEAAAhCAAAQgAAEIbGcCWUH4S7/0S1pIGj67du26293u9n3f932XXXbZTDEdOXJElb7qVa/ytTz60Y9+3OMeN26Nf/RHf6RCbr311pBxsnJC9mkCiyI5jc3khQAEIACBQODo0aO//uu//ohHPOLMM888+eST73vf+z7zmc/8p3/6pyW68f77v//7H/jAB4YWhcBd73rX4OhDYPfu3SHBBIGf/umfVrEho+pV7WFzgsCP/MiPBNtWV1fvdKc7PfShD/3N3/zNO+64Y4LSyAIBCEAAAsMIrBTueMtb3qL4EydOHDhw4Oqrr/6rv/qrN77xje9///sf9KAHFaYnchgBSA4jQzwEIACBOhP4t3/7txe+8IVf/vKXn/SkJz3lKU/RtdEbb7zxPe95z5Of/GS5wne+853nnXdene0fbdvrX/96yV2leeUrX3n48OFf/dVfVbjdbo/ONf+9559//p/+6Z9KgR87duzQoUO6Nv2Lv/iL//7v/67TkpWV4hOY+RtJjRCAAASWnUDxePqsZz0rbtjLX/7yRz3qUc95znP+53/+J46fXfi5z31uq5WdvZyguqrKmaBqn2XhJCe2nIwQgAAEti2Bv/mbv3nqU5/6Td/0TZKF0iSBww/90A999rOffcYznqGpqk9+8pNnnXVW2LVcga/5mq/xBktu7d+//wlPeEI97T/llFNi257//Oc//elPf+QjH/nWt771277t2+ppM1ZBAAIQWDoCpUTXaaed9qIXveh///d/b7nllvm08Ju/+Zvliaevq6pyprfElzB/klVZTjkQgAAEtgmBm2666Xu/93u/5Vu+RbIwVoO++Vo4+s///M8K61JpAOKXR/7jP/6j1MvP//zPh/iPf/zjz372sy+55JJTTz31a7/2a3/5l39ZS2/C3sJFlVqYo3WSEjw+mU/z3//93xJCmpO8y13uouuMn/nMZ0IhIfCpT31KGune97636nrsYx/7K7/yK+vr62Fv+UC+LZva+VVf9VWyWdOMWlLkF3mq4b7G//zP/5S0Pvfcc7WaVAtu9+3bV96SwpRawXuPe9zjYx/7mPbmTfVZ/uu//kvYlUyLYL/yK7/yd3/3d2PsX/jCF9RNe/bsuc997vO2t73NZymM3LThE9vgK+UbAhCAQE0IlBKEslUuUN/ySfr2N+Z97nOf05yhvE5oyTXXXKMbBjT46l6Lu9/97nKoupIa9oaAVt285CUvkf/QSC0nIQ/x6U9/Ouz1AQ3lz3ve8+JIadEf+7Ef03VBDeIXXXSRlN6//Mu/hASawJQTkmpVjG72UFg3MyicL+faa6998Ytf/JCHPES1f8VXfMW3f/u3Z2r3rbv99ttVmlp9+umn627GP/zDPwx1TRmohKQYCv4ZZ5xx4YUXvuIVr9jY2PBWDYsfdi+lrnZfcMEFPu+U3TolFrJDAAIQqAmBn/qpn5IT0U3pw+yRSpQr+eM//uM4gVYzfuu3fuva2pr8kY+XQHr4wx8uean78F/72tdKXUgQyvsob5xx0/B//Md/SAVJ5knj/cAP/ICcr4r9h3/4hzjj7/zO7zz4wQ+WHvvu7/7u3/u931Ndv/Ebv6FR/YYbboiTlQzn2zI6o2qXGJY3P+eccxTQR2JMWT784Q8/5jGPkeW/9mu/ptMDnUKoIX/3d38XStOy1de97nVf/OIXQ0yZgLy8qPqUeVN1k6FOMDTtqVsQ5bulwyXdv/qrv9o/X0AiWZebdbqiu0MlCLWM6BOf+ERhZBlLJrOhfMmkhAAEIDA/AlqdHz66nUAVh80Q+JM/+RPFX3XVVYr5uq/7uvvd737yiN7l+DS6yVASSxcvf+InfkLeSA5VmnDHjh3KGApRQFfj5BvkM+QbXvOa1/z4j/+4VJkeXaOwytdlPJ9YVXz91399yPjXf/3XEj9anCO3qpR6XouGe6V/wQteoHFcyf71X/9Vd+t9z/d8jyLlpBXWZUjFZ8r5i7/4Cxkpy3/wB39Q5fzoj/7oPe95T92qLscQ6lIWXc2V8apCF3rlxnQDiYqVq5PuCsl098L/+T//J2zmA7Mj2e12H/awh8kj/uzP/qxUt05cvCXD4mVbhkOwVhxEw28qzcTdGgokAAEIQGC5CGh41yfYrHFeTkoXBENMmcADHvAAFfKmN70pJNbdhhqc5UFCjAJaaKNrcN/4jd/oI5UrX5GUjIqSFwtptCnXGcrRI1Wk9C6++OJOp+MjP/CBD+gmi5/7uZ+TFwjJrrvuOl2fVV7VEiLzAYnY2NsqQb4tZexURvl9nQOEKnw5/9//9/+FGD09TsJM7jVYrouYslA3BIY0ceClL33pve51rzhG4dtuu03nDNLGCudNfd/73ifs8u9xriuvvFJXn3UdWZF+6lKSVWHZc+mll/7+7/9+YaQvv2QHxV2/qQ2xbYQhAAEILJyAxmH7xHYUyhjdca576DWe+pRSDsol4Rd8j9SXtJ/02PHjx0NpWqEhvSEvpak8H6kJw507d2pQ1oAeksn7amQ3O4YIQt24qClHqTJNEoZcCsj3KIvMCJF+Hi9OFgshXUfUtVvNGR48eDBk0X3qkqbyH5KUPtK3TlcWQ+sU7/Ww9GTIOJkgrISkv5NTqtsboxZ5dz4sXsliDqEJCmQEoecZGl6yW+MCCUMAAhBYLgLmeyI/6Kfv/INMyjdEykTqK06vC6ZaQRPH+PC73vUuVSfNoM0yQktpNPWXKUcOS4V88IMf9PGaddQ8WBi6Q+LPf/7zcs0qIcTkA4WCMNOWMnaq5LwgzFftV9tqSae3pLwglIbUyh1x0/VQXR3+0pe+pBJUfsZUofDCL9NSrf4VMS0I0sJaBXS2ECcojPTllxGE49oQV00YAhCAwMIJaFQsfqjMX/7lX2qf7NNDvTTs6lKlrmvGC1S0kFJP+pKOUjJ9JJ80q6YFNvEzyjTz9upXv1oX3nTZUopFyXS1T4+NfsMb3qBpOpfPvpRFXkTl/9mf/VmIjAM//MM/rOlBXX7TYtE4XjJSyy/1CDhJyjJPG5ORWmuq1SknnXRSKEfOUkZ+5CMfkW1h7ahap8JD65RYU5Fa8CNZGBaySojqSmcoZ1hgRiRVnfejCrz5zW+Oax8WH6cZFp6sW4eVRjwEIACBpSOgy5qyWfd7B8u1pFBSJGzGASULD9/2twP4vVoJKY3xB3/wB3FiH5Zc0QSX7qwLj3XJp8nEaJYsE+Pr0kpL3UYhg3W5Ux4q9lk+vfyyXzyZyb7pZtyWTROPSKA1mZm9WhOkGKluiSgFtFxIZxqZNPGmNG2mXWrRu9/97nCzQ2yqUOhJP1qkqunZuBCF5et1ZVkTg7r35PGPf7zuHNHJify+t1CrY/KRmRJGbI5rg6obURq7IAABCMyfQLEg1CVDb4pEncSYlI90lJb+B/vkZjTb5jd13U6CSk5Oqz5CghDQwCcdpbX7cpy6nUDqLlaDIZnuiygUhLrk+aEPfUhjd0YN+oxBnoVyhgUkGmWk1ojGatAnlpjUjfgve9nLpH69bVqKI+eRKer+979/7GO0bDWToHBzFiS1zFVPO5Ag150k4qYLw95fDosvNKwwcoJu1eFRWBSREIAABJaRgB7cIscnwfa0pz3N2y8JofccFLZFC0y04NDvip2LpJo8o95Ukc+lVTNaM6lHmPhdeTmkjJlcUjKZGD/w+vdG6L5B3TpRWJdyaVT/6Ec/msm+6WbcFp+4jJ35YvPuXjc7KFn5B9TppgbN1vqSJaR1VVefuKLYVKGQr9cyXX3iNCGsC6Zyl7oJRTc9Sq7r5ERnFwrrfKYwUhnLNHxcG4I9BCAAAQjUhECxIMyPgBlzY/+kS326pUGziPpkkoVNjcJarK+Pv9E8xIeAHowWwnFAhWuZZf76aJymTFh3U8h3SukVJpbLVJM1S+mvWeZ9mHJpjcoEL8OdBUmdCuh+Et0qqUUvupNETxfQ9Kl/HfCweNlfaIl8Zwxkgm5FEMYACUMAAstOQPc1aLJIDz7RRTe/9kRvnsg3SopOjsnfYpDfK3kmValb1+LnroVkWnej1ydoUzcr6kJkiPeBvFjKTJEpWfxaJt22p7pkj1+JkylNfi0TM8FmSTvzJectz8fkc8UxmddOxLvyYaFQl+lURE9kze8NMepiPXZOV4F1f6MkvWYptY6pMHKChpe0IRhDAAIQgEAdCLSmN0IrNzRk67lekhzDPlpQcec731l6Q8s/CmvUZFdhvK4OqvDp33/ojQwXZTN16WE5itFVWx9f6LFiB5zJXtVmSZKqTj5P1zW1JEnXrfUQufCapmHxWguqR5nn7QwPasvvKm9MPi8xEIAABJaXgNyZtJx/VHVhKzQjp9UZWiyqxRqFCTTjpKeVvOMd78jv1aCt+9j9xUfdQ5H3blr/mc81IkbrWXS/op7rJv+bSSaXpyeUZiIn2KzEzgnqHTeLR/H2t789n1FLhPT2SF1i1tyvV/hyl7oZRK/E0A2ZhZEqZIKGl7Ehbx4xEIAABBZLoAJBKKWk2xg0SZWZbvINk9f8mZ/5GYV1r6AWnerSXaEyKbzXQrlUuBasKpc8aJ6UPOuwejOJVY7uRJef8Gts4r1yoppY0yShrgXG8fMPlySp9T/+sQSyUOtFf+EXfkFeX6cvw+KVTGr88ssvj1/EpEgtTNKd/cOaWdKYYdmJhwAEILCkBDT1p/UXv/Vbv6UX8/hnfsYN0ZzbE5/4REkI3Q+vqbl4VxzWw6v1QLXf/u3fjiOvv/56TUnpJjc/c6iHpelOtvj+fN2sGJ6yFmccHVYtEn56BZFusggpb775Zr1Xyd8SGSInC1RlZ752eeQJXjuRLyfECIUcfeaMQjrwO77jO7RGSZeYdW+LnvIaulWeUWuCCiNV5mQN39SGYC0BCEAAAjUhUIEgVEv0ughN8ekVCLHcktD6yZ/8ST1pJqyZedWrXqXFMLphz78OyCNQMsW//vWvH0ZED31Rer0sKM6lxFrsoZLf//736zrfsLxxvO4T0EIdXRGUVwjxEkV6eYOe1ZZx2yHBsIDeSFG4jmhY+pLxZUjqVkbd3BLmVP0sn9TssHhVrVsZ5f/8I2SDJVKSAhI284EyxuRzEQMBCEBg2QnoBnWJBC0c1W0OL3zhC+WG3vnOd0rjyX/5x6JogYYuI45optbz662DcoJ6k618nC47ytcorzSbf2y18mrOSuJQ04x65pmue/r3Fvq77EaUnN+la7KqQhObCuhbikjiUPe96zYHvbE2n37cmKrszNerS71yyuNOiubLCTG6giwlr3cda+GMbu/Uc910SVqztVJ9mkRVMi3W1VmHpLLmb/VwOFX9nd/5nYWRSjxZwze1IVhLAAIQgEBNCJSSUpvaquupeiWDhvX3vve9el+fPKhuGpR707tf//zP/zwIQt0oqEuqSqb0Gmf1vBk9I/Sf/umfNE+llyjo4c6FFak0PUhTL9sNuVSsnj2tC6IqRH66MFc+Up5YVyJ1dVZm6AYD+XLdWCh/L7+u9/jpkmE+y4gYeXrNocnlj0gzwa4yJOW6dF7yrGc9S6cpugYs56c3T2hR6LB4maHH2anhuiXmYx/7mM4/5A717G/NKOpMJQjLvLVljMnnIgYCEIDAFiCgJ6XpZkI9vVNvLNCiDK1t0S3TUlm6yqnppsI7CzKt1gO0VYJeEK/5Rk1SyeVpHNZL0vVwFJ9SK0vlAXWpTk+/lHuSs9MKRr2eV5f8xpWFeoeQ7iHU7KKurmrWS/fF6Srq//2//1dVhyfAZcwrv1mhneUrnTilnh0qN6fnyuj0QFOywi7mkoj+Kejya6Kt9cDf9V3fpQcIiY9OJFRXYeTEDR9tw8RNIyMEIACBGRKQPAifwvcQhr0+MOyldhp55ZP0FiDd9adHOUvC6UlrmbzalIbRQ9t0IVMPDtWjR7X2RhN0mlpUsbpiN6wKTQ/qFnDl2rt3r1Z9KJf8aKZwjezCNOw9hD6xvLJcsjdSj+WUfJUuissZ1jo5ac3ChZSalpzsxfShBAWG1bUpST1ZW+cZskco5Hj03B1f7LB4v1fPcVUunWfogQc6V1BiXVSWL/R7JzbGZ+cbAhCAwDIS8M51GS3HZghAAAIQgMD0BOQHm/pfBXmPyDcEIAABCEBgWxHwc334wW3V6TQWAhCAAAQCAfnBau4hDCUSgAAEIAABCEAAAhCAAAQgAIFlIYAgXJaewk4IQAACEIAABCAAAQhAAAIVE0AQVgyU4iAAAQhAAAIQgAAEIAABCCwLAQThsvQUdkIAAhCAAAQgAAEIQAACEKiYAIKwYqAUBwEIQAACEIAABCAAAQhAYFkIIAiXpaewEwIQgAAEIAABCEAAAhCAQMUEEIQVA6U4CEAAAhCAAAQgAAEIQAACy0IAQbgsPYWdEIAABCAAAQhAAAIQgAAEKiaAIKwYKMVBAAIQgAAEIAABCEAAAhBYFgIIwmXpKeyEAAQgAAEIQAACEIAABCBQMQEEYcVAKQ4CEIAABCAAAQhAAAIQgMCyEEAQLktPYScEIAABCEAAAhCAAAQgAIGKCSAIKwZKcRCAAAQgAAEIQAACEIAABJaFAIJwWXoKOyEAAQhAAAIQgAAEIAABCFRMAEFYMVCKgwAEIAABCEAAAhCAAAQgsCwEEITL0lPYCQEIQAACEIAABCAAAQhAoGICCMKKgVIcBCAAAQhAAAIQgAAEIACBZSGAIFyWnsJOCEAAAhCAAAQgAAEIQAACFRNAEFYMlOIgAAEIQAACEIAABCAAAQgsCwEE4bL0FHZCAAIQgAAEIAABCEAAAhComACCsGKgFAcBCEAAAhCAAAQgAAEIQGBZCCAIl6WnsBMCEIAABCAAAQhAAAIQgEDFBBCEFQOlOAhAAAIQgAAEIAABCEAAAstCAEG4LD2FnRCAAAQgAAEIQAACEIAABComgCCsGCjFQQACEIAABCAAAQhAAAIQWBYCCMJl6SnshAAEIAABCEAAAhCAAAQgUDEBBGHFQCkOAhCAAAQgAAEIQAACEIDAshBAEC5LT2EnBCAAAQhAAAIQgAAEIACBigkgCCsGSnEQgAAEIAABCEAAAhCAAASWhQCCcFl6CjshAAEIQAACEIAABCAAAQhUTABBWDFQioMABCAAAQhAAAIQgAAEILAsBBCEy9JT2AkBCEAAAhCAAAQgAAEIQKBiAgjCioFSHAQgAAEIQAACEIAABCAAgWUhgCBclp7CTghAAAIQgAAEIAABCEAAAhUTQBBWDJTiIAABCEAAAhCAAAQgAAEILAsBBOGy9BR2QgACEIAABCAAAQhAAAIQqJgAgrBioBQHAQhAAAIQgAAEIAABCEBgWQggCJelp7ATAhCAAAQgAAEIQAACEIBAxQQQhBUDpTgIQAACEIAABCAAAQhAAALLQgBBuCw9hZ0QgAAEIAABCEAAAhCAAAQqJoAgrBgoxUEAAhCAAAQgAAEIQAACEFgWAgjCZekp7IQABCAAAQhAAAIQgAAEIFAxAQRhxUApDgIQgAAEIAABCEAAAhCAwLIQQBAuS09hJwQgAAEIQAACEIAABCAAgYoJIAgrBkpxEIAABCAAAQhAAAIQgAAEloUAgnBZego7IQABCEAAAhCAAAQgAAEIVEwAQVgxUIqDAAQgAAEIQAACEIAABCCwLAQQhMvSU9gJAQhAAAIQgAAEIAABCECgYgIIwoqBUhwEIAABCEAAAhCAAAQgAIFlIYAgXJaewk4IQAACEIAABCAAAQhAAAIVE0AQVgyU4iAAAQhAAAIQgAAEIAABCCwLAQThsvQUdkIAAhCAAAQgAAEIQAACEKiYAIKwYqAUBwEIQAACEIAABCAAAQhAYFkIIAiXpaewEwIQgAAEIAABCEAAAhCAQMUEEIQVA6U4CEAAAhCAAAQgAAEIQAACy0IAQbgsPYWdEIAABCAAAQhAAAIQgAAEKiaAIKwYKMVBAAIQgAAEIAABCEAAAhBYFgIIwmXpKeyEAAQgAAEIQAACEIAABCBQMQEEYcVAKQ4CEIAABCAAAQhAAAIQgMCyEEAQLktPYScEIAABCEAAAhCAAAQgAIGKCSAIKwZKcRCAAAQgAAEIQAACEIAABJaFAIJwWXoKOyEAAQhAAAIQgAAEIAABCFRMAEFYMVCKgwAEIAABCEAAAhCAAAQgsCwEEITL0lPYCQEIQAACEIAABCAAAQhAoGICCMKKgVIcBCAAAQhAAAIQgAAEIACBZSGAIFyWnsJOCEAAAhCAAAQgAAEIQAACFRNAEFYMlOIgAAEIQAACEIAABCAAAQgsCwEE4bL0FHZCAAIQgAAEIAABCEAAAhComACCsGKgFAcBCEAAAhCAAAQgAAEIQGBZCCAIl6WnsBMCEIAABCAAAQhAAAIQgEDFBBCEFQOlOAhAAAIQgAAEIAABCEAAAstCAEG4LD2FnRCAAAQgAAEIQAACEIAABComgCCsGCjFQQACEIAABCAAAQhAAAIQWBYCCMJl6SnshAAEIAABCEAAAhCAAAQgUDEBBGHFQCkOAhCAAAQgAAEIQAACEIDAshBAEC5LT2EnBCAAAQhAAAIQgAAEIACBigkgCCsGSnEQgAAEIAABCEAAAhCAAASWhQCCcFl6CjshAAEIQAACEIAABCAAAQhUTABBWDFQioMABCAAAQhAAAIQgAAEILAsBBCEy9JT2AkBCEAAAhCAAAQgAAEIQKBiAgjCioFSHAQgAAEIQAACEIAABCAAgWUhgCBclp7CTghAAAIQgAAEIAABCEAAAhUTQBBWDJTiIAABCEAAAhCAAAQgAAEILAsBBOGy9BR2QgACEIAABCAAAQhAAAIQqJgAgrBioBQHAQhAAAIQgAAEIAABCEBgWQggCJelp7ATAhCAAAQgAAEIQAACEIBAxQQQhBUDpTgIQAACEIAABCAAAQhAAALLQgBBuCw9hZ0QgAAEIAABCEAAAhCAAAQqJoAgrBgoxUEAAhCAAAQgAAEIQAACEFgWAgjCZekp7IQABCAAAQhAAAIQgAAEIFAxAQRhxUApDgIQgAAEIAABCEAAAhCAwLIQQBAuS09hJwQgAAEIQAACEIAABCAAgYoJIAgrBkpxEIAABCAAAQhAAAIQgAAEloUAgnBZego7IQABCEAAAhCAAAQgAAEIVEwAQVgxUIqDAAQgAAEIQAACEIAABCCwLAQQhMvSU9gJAQhAAAIQgAAEIAABCECgYgIIwoqBUhwEIAABCEAAAhCAAAQgAIFlIYAgXJaewk4IQAACEIAABCAAAQhAAAIVE0AQVgyU4iAAAQhAAAIQgAAEIAABCCwLAQThsvQUdkIAAhCAAAQgAAEIQAACEKiYAIKwYqAUBwEIQAACEIAABCAAAQhAYFkIIAiXpaewEwIQgAAEIAABCEAAAhCAQMUEEIQVA6U4CEAAAhCAAAQgAAEIQAACy0IAQbgsPYWdEIAABCAAAQhAAAIQgAAEKiaAIKwYKMVBAAIQgAAEIAABCEAAAhBYFgIIwmXpKeyEAAQgAAEIQAACEIAABCBQMQEEYcVAKQ4CEIAABCAAAQhAAAIQgMCyEEAQLktPYScEIAABCEAAAhCAAAQgAIGKCSAIKwZKcRCAAAQgAAEIQAACEIAABJaFAIJwWXoKOyEAAQhAAAIQgAAEIAABCFRMAEFYMVCKgwAEIAABCEAAAhCAAAQgsCwEEITL0lPYCQEIQAACEIAABCAAAQhAoGICCMKKgVIcBCAAAQhAAAIQgAAEIACBZSGAIFyWnsJOCEAAAhCAAAQgAAEIQAACFRNAEFYMlOIgAAEIQAACEIAABCAAAQgsCwEE4bL0FHZCAAIQgAAEIAABCEAAAhComACCsGKgFAcBCEAAAhCAAAQgAAEIQGBZCCAIl6WnsBMCEIAABCAAAQhAAAIQgEDFBBCEFQOlOAhAAAIQgAAEIAABCEAAAstCAEG4LD2FnRCAAAQgAAEIQAACEIAABComgCCsGCjFQQACEIAABCAAAQhAAAIQWBYCCMJl6SnshAAEIAABCEAAAhCAAAQgUDEBBGHFQCkOAhCAAAQgAAEIQAACEIDAshBAEC5LT2EnBCAAAQhAAAIQgAAEIACBigkgCCsGSnEQgAAEIAABCEAAAhCAAASWhQCCcFl6CjshAAEIQAACEIAABCAAAQhUTABBWDFQioMABCAAAQhAAAIQgAAEILAsBBCEy9JT2AkBCEAAAhCAAAQgAAEIQKBiAgjCioFSHAQgAAEIQAACEIAABCAAgWUhgCBclp7CTghAAAIQgAAEIAABCEAAAhUTQBBWDJTiIAABCEAAAhCAAAQgAAEILAsBBOGy9BR2QgACEIAABCAAAQhAAAIQqJgAgrBioBQHAQhAAAIQgAAEIAABCEBgWQggCJelp7ATAhCAAAQgAAEIQAACEIBAxQQQhBUDpTgIQAACEIAABCAAAQhAAALLQgBBuCw9hZ0QgAAEIAABCEAAAhCAAAQqJoAgrBgoxUEAAhCAAAQgAAEIQAACEFgWAgjCZekp7IQABCAAAQhAAAIQgAAEIFAxAQRhxUApDgIQgAAEIAABCEAAAhCAwLIQQBAuS09hJwQgAAEIQAACEIAABCAAgYoJIAgrBkpxEIAABCAAAQhAAAIQgAAEloUAgnBZego7IQABCEAAAhCAAAQgAAEIVEwAQVgxUIqDAAQgAAEIQAACEIAABCCwLAQQhMvSU9gJAQhAAAIQgAAEIAABCECgYgIIwoqBUhwEIAABCEAAAhCAAAQgAIFlIYAgXJaewk4IQAACEIAABCAAAQhAAAIVE0AQVgyU4iAAAQhAAAIQgAAEIAABCCwLAQThsvQUdkIAAhCAAAQgAAEIQAACEKiYAIKwYqAUBwEIQAACEIAABCAAAQhAYFkIIAiXpaewEwIQgAAEIAABCEAAAhCAQMUEEIQVA6U4CEAAAhCAAAQgAAEIQAACy0IAQbgsPYWdEIAABCAAAQhAAAIQgAAEKiaAIKwYKMVBAAIQgAAEIAABCEAAAhBYFgIIwmXpKeyEAAQgAAEIQAACEIAABCBQMQEEYcVAKQ4CEIAABCAAAQhAAAIQgMCyEEAQLktPYScEIAABCEAAAhCAAAQgAIGKCSAIKwZKcRCAAAQgAAEIQAACEIAABJaFAIJwWXoKOyEAAQhAAAIQgAAEIAABCFRMAEFYMVCKgwAEIAABCEAAAhCAAAQgsCwEEITL0lPYCQEIQAACEIAABCAAAQhAoGICCMKKgVIcBCAAAQhAAAIQgAAEIACBZSGAIFyWnsJOCEAAAhCAAAQgAAEIQAACFRNAEFYMlOIgAAEIQAACEIAABCAAAQgsCwEE4bL0FHZCAAIQgAAEIAABCEAAAhComACCsGKgFAcBCEAAAhCAAAQgAAEIQGBZCCAIl6WnsBMCEIAABCAAAQhAAAIQgEDFBBCEFQOlOAhAAAIQgAAEIAABCEAAAstCAEG4LD2FnRCAAAQgAAEIQAACEIAABComgCCsGCjFQQACEIAABCAAAQhAAAIQWBYCCMJl6SnshAAEIAABCEAAAhCAAAQgUDEBBGHFQCkOAhCAAAQgAAEIQAACEIDAshBAEC5LT2EnBCAAAQhAAAIQgAAEIACBigkgCCsGSnEQgAAEIAABCEAAAhCAAASWhQCCcFl6CjshAAEIQAACEIAABCAAAQhUTABBWDFQioMABCAAAQhAAAIQgAAEILAsBBCEy9JT2AkBCEAAAhCAAAQgAAEIQKBiAgjCioFSHAQgAAEIQAACEIAABCAAgWUhgCBclp7CTghAAAIQgAAEIAABCEAAAhUTQBBWDJTiIAABCEAAAhCAAAQgAAEILAsBBOGy9BR2QgACEIAABCAAAQhAAAIQqJgAgrBioBQHAQhAAAIQgAAEIAABCEBgWQggCJelp7ATAhCAAAQgAAEIQAACEIBAxQQQhBUDpTgIQAACEIAABCAAAQhAAALLQgBBuCw9hZ0QgAAEIAABCEAAAhCAAAQqJoAgrBgoxUEAAhCAAAQgAAEIQAACEFgWAgjCZekp7IQABCAAAQhAAAIQgAAEIFAxAQRhxUApDgIQgAAEIAABCEAAAhCAwLIQQBAuS09hJwQgAAEIQAACEIAABCAAgYoJIAgrBkpxEIAABCAAAQhAAAIQgAAEloUAgnBZego7IQABCEAAAhCAAAQgAAEIVEwAQVgxUIqDAAQgAAEIQAACEIAABCCwLAQQhMvSU9gJAQhAAAIQgAAEIAABCECgYgIIwoqBUhwEIAABCEAAAhCAAAQgAIFlIYAgXJaewk4IQAACEIAABCAAAQhAAAIVE0AQVgyU4iAAAQhAAAIQgAAEIAABCCwLAQThsvQUdkIAAhCAAAQgAAEIQAACEKiYAIKwYqAUBwEIQAACEIAABCAAAQhAYFkIrCyLodgJgUICR44c0FWN9V7zyJH9673G7YduP233aT7lqaecuueUPYW5iIQABCAAAQhsDQL4wa3Rj7QCAgskgCBcIHyqzhI4ePhAo9lo9BoHDu/X38OHb5fGU+CWg7fdduCmjV7zhtuvV4T+ffGWqz52wycbjeZKu9Ns9toWZyn9RzFJQClUXKNx/7Mfqu+LzriLvs899U76Pmvv2frW5/Q9Z+jby8i9p+xVeM/J9s0HAhCAAAQgMGcC+ME5A6c6CEBABOwUutdLzp4hAoFqCRw8ckAFHji0X9/7D93uC7/94K23Hrhptdk43mvcuv96ybyrbrlKeu4TN/6njseNbrvZaqx32q1mY6W90WrZMWqizqm9JOwLct9e70URFmylgjATr82mKycfn4lRsfc762E+8sIz7qrAOaeep28vI0/fbRry1HQqUjISDelZ8Q2BpSPQdIMCfnDpOm5ZDMYPLktPYScEti0B+UE7O8YRbtsjYKyGD/Nqtxy4yZfz5duvV+CG267QAs7Pf/mDpzR7R3qto41VXXBY77UlxTZ67a5S6KjrtFZXN/RXU3kZjZcReMOknTtyh5nfa7lZwWG7J4gvpyKt4Pue9XB9X+A05LlOQ54ZpiIjGYmGnKAXyAKBWRDAD86C6lYtEz9Ypmfxg2UokQYC9SGAIKxPXyzGkjuOHNhoNA4d3n/rgf2tVu/WA7dqZaafL/bq7ppbr5Rln7/pA7pu0G70Tmn1DmgGr9GQtNOfY72VTq/ZabS6jZaE3UbHViCvrHRbza6JvUZe7JkCzDR1hLRzJWTTZ7JnNssrt0zG+W96FDL4Pmd9la/9gtPvqoCmIodpSO1lKnL+PUWNW5sAgnBr9++mrcMPbopodgnwg7NjS8kQKE8AQVie1ZKlPHDgYGult9+t1Tx4+PZur/GFaz6nNjiZ17v19iv/64YPrTQ7yULMRmOt2fXS7qRm947e6rFeWzpMqk+qrtMzsbfeWWk1m213w55m7bzuimfz/FxfwDRC5rk0oybxqhV1GcOChUsdyGtINefMvWe5RjW1opXlrEvdvxg/TwIIwnnSnmdd+MFAGz/ItdRwMBCAQJ4AgjDPpO4xBw4fXG32+tN6zd6t7pY82X3j7ddLoV1zy1X/e9MHN3qN01bWN3q9/esnnbxyTJLuRKNxoruyu9E80JOc0yrO5npjdaPR6jTavWZ7rdnbsXpCSm+j1/KrNN2qS6lCHSTuHtOeTe117YEuypxINu9jYlkoM5TAfZTGBy0wWuON3puUl/szbDVpLuGCIzwI38YUzrxN8rXf201FhnlIGZFqyIa7K7LpZSRP1pl391DfQgkgCBeKf5LK8YOBWu39YHIa4PyghbV4CD8Yuo8ABGpCAEFYk45IzDhw8GCr3bv94IEDh2/TI1U0aH7hmv/WMHrDbTfsavW+dPNVt936rzdurN3RkSTs2TrNRk/TerZ6s9c93tHteS09kaXXaXVbjZ3NExJvbS3ndB8n6Sx0pLNj7+pR7VJYuk7xWvDpkjR2uPv7jndWTlnd2Nnu7GofP2dl/bR2S8mu7qzd0ls73m2vtFSvScQg+5TXrw6VwgzKUGGVbN9WkYLJHjdtWNYXmInjf1TTZpOT4xca5aj87sSo7LJBB7Zs4onTef5eQ6qQICMHNaSerHO69qIhJ+ZMxoUTQBAuvAtiA/CDMQ38YEwjDuMHYxqEITAlAQThlADLZj9w6KDu0NPIfvjw/psO7peeu+m2227bf6Nux9O0nkq58stf/OyXPyglo7WdG92VQydWd650Tmmf6HZbiuk0e2e2Ord1mse7K+1md2dr47b1kyX51nQHXzr3JtUlQbiu5Z295g49nDNWbINmKmX8yY+qlrfZOKl14pT28Vaje1b7+BUnTtVTYY521joSma5kmxtsSUr2VmxtqYlCPQ7U/jYakqfaZQFnxI5WV8rVS1Dt1wykMkjHKhwMCUpSsSU/ylsmsdJYSrfGtWSW2ABXRYo43rGc4bQXZmi9iOU1pOqLZaTXkIrkyToz7AmKLk0AQVga1VQJ8YPpCIwfnOpAmjJz2gtTFjMqO35wFB321ZIAgrCCbtErg6SfDh7ef+jQbevd5m0Hb715/03drt6Yd8Nqq3vdLVdefuOHbuusaIBYbW6s9Fo7Gr0rDu5q72istjZ2rx7v6ma66CO1pim79W5b8aesHDNlFn28cOr2mhrRpLZsSi6IqjRZEHjK6SSQpRgoJU057K8fLk1quiI6PU0Mbqw0u9JycRabizOtZXEyIzbEx0v1qRXKpcB6b0XmetEoHbkihda0l0PYWyUUcGpZYZuBdJ9QWlqLn/dzNXpF6tNV9G21uIZMUJ41dqD1E5RRxyyuC/qifXYmCqDq8ndFhnlIVeeerKM9dkukNlnOOrsu2OYlIwinPwDwg2K49H4wfTrABMcDfnACaHEW/GBMg/D8CSAIx2N+6PCBf/7I32vC67pbb/zSTVcfP9a84eAHD/bsTQrhI5cgeXCws1MP4dT02q72etilgPZKbklrKZUEoU52FZP/aGhQsiAz7KR4MJHXbIpTmoxoHExYvOWm7op3eV2Utyqvl2zhqPu4xDqWBlqiKmSkhLFWtHpNqIfTKHmaKNOgAWM0w+iS6a+dqkk32pGqt8979ejFpNSjpiLT8jwHffvJQGdt8tyaTE3aTG0YqLTkhi+/ZOJMMlUdDM7sWurNcA2iqlbYMe26Lekpt/w4LtwfbF5DKmFORvpbInlRZMyM8FACNsrY4DzNwDC08K23Az8Y96kfi9zxoy/zU/Fe/GBMw4fxg3kmhTFZP2hjlPOLaWr8YEqCvxUQsLNsFYMjLMnyuhu/+MLXPvlIZ21Dd+m5FY87WusDP1BXkNNyFvKizmuVjAYrVHRxGnmVoPTi+GBqKCG/N+xSYr83jgkl5DNqlz8RD2nygYzDCwlCaWqvSTiz38YvPbdGuDRV6GVD8JZhaAsBL6yDqSHeVxFopJtJzaYeXaFSj4qyKUfJSJu9tJWrK/acVLvJUh220tb7MFy8CTPNxGpPMuSqAN9wRVlD3DszXKpgr1WX7+vEiOF/hGuCXL48a8Lwkmu7J/TUALspzE1O1B2LzFExQam8IGsCaFs7C35wrP7FDwoXfnCsYwY/OBauwsR2luJOq7QXP1iIiMhpCMgP2ovj+JQkcODQbfpN6qe4Yg90sY9u29O3CQsTFP0TYB/yMf78OOgcl6//5TPmz/sVExfYz5CGggBLI/p/87vyMf3ULjRaBxY6v8IyY5utUbpcqsfQGLP8LE+KK9f4oCjC3Ksf/sIgGOIdVd1l6YroOiVn3+qX5HPCTcYqo1tnu6KAFq9q3lLrYG1Pq7m2ohs5e3qIjkSj7snUylihkHRUFyteDfCeLFWSZrNrlw3MCviKrHx3MKTN1560dVZN2EiscuamJg7+dSVYpYpWDWkpg4mGbxV2yvDk1eyZwM6xKg4HpzpxrIyFiT9780cU778VKEPYa0jVfcEZdz331POUixdFFrIlcjsQwA/GvVw45KaOwBLamKWBHD+YG2xHjOb4wfgY82GPS9/4wTwcYiohgCAcA+Pe3adr9WNeHemEOHYAo0ss9B+js2hvOCfeNGUlCfJtzBSbaUXcfGmnOLFSSrYpgYt1znEwQZzYS75QWjvs81kiB+LkWNhtp/W+Vl+C/za1ZmWZGvSiRROV7a6e1GM3NypSO3WDowzudPQmj+Zxs9PeumEvXNT/za6esbqqqU1rkrLY83M066iPX7x6cqu3c2V918bKDZ21s9rrt2veWApz7bhssULSyUZvZWKYTg1s2bC/lKCzBNekZELSEibtyimfqOm+vOJvgxAljYLF6X1yj06JfWBo0kXv0GHpMVZiSPnGBvX4GScmy9TuNaRSXnjGXfV9jpORZ+09W2HVewYviiwDkTS1JIAfDN2CH5TLxA+G42E+AfzgfDhvw1oQhGN0us7j5QC8XNGJuw8U5s/4icI0caTKKvOJdVruzNj0zSw+I9qSIWC6yX1kiHYl2kZhbTtemyqOuIEj2tIvOk3Ub3oqHUPtSiKz7J9koabydGenZKAJM5uIU+u8ePPplcwbr1wW6PU6pt6sXRKSSTkNe6mjst+wvqvX2/Xd53ziEWtrSnBr78Sh7q4TG6uaXTzeXDvWax1vrPon67RamlNur53YqfeK7Frp3GJvBOlqNrLd6jrLbUWrjfKuUjPYKpRojKWsJbQE7q/LZWkyH2uFy6t4pUmDmVSbb/qOGFbL5vmrTlHJNdGMUYWtm5hYpvCgIUMgk8C6MhPVaHgZmdeQSsiTdXK0iFgMAfxghnt/zHU78IP4wcwRUtUmfhA/WNWxVFgOgrAQy5DIaNlj1geUOPfeVLCVkUNBnrmliKkCcyeXGZOGtGGq6HwVsjmc14aA6lDKWJJNVWsusxdFueiBCG+qU1aJIJW868pau7XQ7gG1eUCpQS0NtaAJAZOIQ8xWW5RSZ0JK43WW0p6+46jS/8uhu+9ZOar1pSvNzm0bJ5nMa3R156TSuX+KV76N3a0jD9t1nWq5dmPjljsuOtppn2isbjRW9ShazUPq1kd9qzRZoGeuagZSalCRKkPx+mPfvZ5bxdpw3yZlvXpJpaypRy90rS3eSoWij0+viLinov1R0PVqSB/t2CRoJUeHxCapo92zcHVR8WWDgcwEbS9bh0sXKopzefU4TEPGKUM4PxXp5yGVAN8ZKBGokgB+MOfu8YP4wfgnhh+MaYwI4wdHwJn/LgTheMztND3nDEIRTqiErYLAppJvROG+OK9zFHY/pNFnrX5v8ovzYkN6RrF5qRaKzRsdrnfmd6UmxXvcQ1lchESNv8cy3j3nsOsRx8F96Ykz0oSyQeLL1mvaDKGkm/Sh2633QA6f+DVNqJwipXyuEN8WabDbN05yQtEijrrbSlOxpsWgrmTNHzZOuXXjlB2tjdPaR090V1TOWmND4lEyTphkTnLd3dRUT4+mVXW+s3wtXVe51rOqO/RiEjNaeZRWMd76nt4A2Vhtm2j1dpqlzl5pU4U9Cn80eJuSNvsKCr5d7n68JVd2FdWPy4VsnxOTyWGXS0BEtQSCegwBle/g+ysydvxpSlxvKNXLbI53Vo51Vx5x9gPc0dW6+9kX3Pn0c8857cwz9p5+1ztdtPfkPdXaRmlblcBS+cGBTvAjoR9a8YP4QR0czp+N8FfaH++15NrGDw78rha9EdxfCPhuUve6nzl+sFQPIQhLYQqJdO4tX+g3I/nnz7ZDKltZ2t8oDoUEKiaEi5OGWO/MtFk4nRKUWyok+qOYQu5XYQrBe8HQCpUma/1eV1HGnl6oK5Qf2ROCIWDvSHQbqioVQ64Ke6qn+5Rvbyg0E/BNyERqs7Bkn1gSTliUIGhCWaPIkMX3ZvLE0nzRFpN2U9KOwURWkO2wuT190s04kZ5Pe0d39faNXUFPrrX16hHRt4whMsridrguSwzrmG7UE1I7XVv1qjtatYxV4QiIiUHT4c5Ia1TLlqdKwqp8KUb1tR6fo2h7cI49fDWpQuk9CvWyAo6G4Qpt9tI2HD+RkZsHfcn+e/PURSlca4p2bO84hzRho9+dektveZHwk/w70Vk52llVWAeJPc9C/a+H7q70Vla6Hz7w31q3vLrSvfyGj619ubej3Vltddba3Xud/uALTrvrvc679GH3euz25krrNyGgcSMd5+0SWvqppx9MrbNReYF+0IZSccMP4gf7R+SYof5PbcyMWzs5frCq/kUQjkFy7+5TlTryf8lJvEb5uJRw4ptIkcG9OuHWeVt0Bj+Q14uuoMHiYuXLVHXYFZVgqVRp2KVN7Q0O2xciI73wcCeOdu4YPhk3nlEmoXW6HS40TQOTkkVl+PvsvAL0yWzskhneTqUMeUO9EwfGKkqJvQ1mcCp/zDbFuvFVDVTA7/UxhYbFrY2ThXhvVTRkx3jcEtCUmNFLjwp1Rwy8b62zKpGXpvF0MpF8lKXd7trzjbp6Fk6r1bZ2JZzt0FLKbtJSbXZtxap1hdatugJ8Sq1CdVva4fWkvfXRbPFv6XD6UALSFrJqNWy7pxsj3Xs9rBw/5aiAEmuvb7JrkYKWIK3C1xA2IzbJnqQL0q3iv46J3xX1X3Hagti4VjMu/cTxPi7em6aqy19nrT+S7VemZyHpAPCTfse7Kyck/OwawYrd56qHR7dN/K/s6J7U7u1Y2ViT5GtL8kn7baw1uzvbG1rGfNLK+o5mZ0d7/aTWxlprY1drY0fv6vbtnasPdG/4n+7pu59+r3u/+Ow7PbAu7ceO2hDAD7qxzvpDv0pzHFHXBC/s3Iv9GrVTYTck2u835I0yTRgcqyg3ipoxZnA0jnqbFD+lH1QJHoS3ympKPjEe58hkgdtl9Jy/0FZpP9hfohX7Qa2awQ+mwIv/Rj2SQk97IZNhoMMy+xa96Vphx6yOHdmJH6y2QwyvhodqC92qpR08fODZv/qodbvDK7k+6sVSGNQyDXdHrQGOf4o+jYhrr3cPYW8oR/Ea7CynzreTwdPG61C+9sZjuo8PKX3GfK96GePjffmhQB+Iqwi7YvGjyLiWTHpnlSWJk8nOETWGWmYd8H1hxjjf40zS1JmJKt3yp9v28gbkCefTjBvjUcQdkcHrC/TJFI73puRlqvW+f1zNhls46o4ly+qaYfz9eUnSattjx5J9J8eVi8lEWlzBJ8PB7sFUMf4Gx6bWqVq5upBhtz6GGchmQ5NOMl5HiJlrDbFkOsLtILdW2abvhQL0afoCawajYj6De/yW1eLrCnu9PWGzbgE3CDir3YGqfpTPSyb9uivHOivr3RV1vWYChVHNX1kR/O7qioSfTflqznmtJfnX26FJP5v3k+Tr7Gxt7Gxu6Im4O6UAW+u7bNfGDlOGHX3v0OrlZnOt1V5ttdYa6SVCuyLQ2HHyN5554TPPvujr60apWnuMox2NhUditVVthdLwg+rFdDS2Ds37QUX6YTMk06aOLx1hfgS2bIv4pON/1g/qUrMuEi2VHzR8opr6Qa2hsUb537D7th81ftAdZfhB/KA7EEZ+yQ+m7n9kOnYGAhpjpB+0mU7X2C8tTOv50dbrOlNHqWwLXiF4jqDoNGDFJ7U+pfcl/vzEh1WjsoSUfpdigmFmkh8L0yi/NzYpGSxdAh+fpk3+uqmkTFw6xLqzee1LJ4QsmWzPpE5sziSzMrIpMxnntan+MkGjh3uG8z/fiNA13hJHuw809MKUdsbA4rArNsaZ4NLxIKQupbsz0Osom/Yz3y0PKFmguyI99tQ2bZrWTTfd8aqNtAdCX7hc/ZlknyE59lxqX2x8IClNs9FRYb1OUn7HTzumldnqxORjM5BusaKK1JyitU6/FNHXhq7mKp0Tk7q+ojdAalGr/ZQsmb5lvCW3sBmSNiVYnlaR7FLKfkwacpH+J2PFDPvkesEndA0tzuOs8rYVJygbm7bMzNM4IM6a5dP1phPd1vHuqtZ8+jlAIVDDRcwmbFd6q+3GKVrw2T6h12aumhSUojPtp9tHNfWnm1Sd9uvsbK6vSQRK8tnsn4SfAl0TfpJ/rdZqU/9U3qrZ6rRfwkFh98+i1dMn/v6mK/7hwA1PPO28Z5554RaXhYaCTzkCOmRjP+h/gPjBAM+PnDaI+NHC7dBmwTgV8sw1ILMG/GD41bs1Kf0BcxF+0I3/CY3EksgPJvJbh5xgRn7Qnhg36KbxgwYRP4gfLD+0IAjLs0pS2kmtjez9sd1t2l6Nnn5IcuOXwklkkGpueLXIsMtt2Jc+Np0S+Q8l9vH9736dFpc5P9bNCd4x+/Tx3jjcLy0XKkrWN8jvHTDBbajSYGiweSBZrqL5Rzj/YfLAk5fB5gKjTsyYFPShGqKGm0SZ4yeuTnbq42WWC4q2fLmOFDvK1IRwV2QwMGR3jdXDVAc+noBFWXH+psQkgXWxq8Nvq/jQ7Lj3/VGaxLgU0YVY2/a5zESt6dC3bpxxH3/FwW9427wRfv2qn3s02m5NqjJL9arhLaleLWRV2G6DtPWrar7t1dSuM1AkUqt97bap8s1Cb4p1dxqyKvtH7EAiZ6TltaNDH9/EGIkifUFphVayyzbkK7VAuawtXvjpqra/u0+qT095kfDrdNsdt4jXqpb20/1+a3rLZW/PyoZXfXoarSSfFnz6W/60+HOnmwPc1dbKTxdumhrcJU3Y7K61bN5Pam9Hq2nzfs32ymbazxrmMQ4cAtaq9aP/cPOV/3Dgxieed6+X79pz0ZCGEr29COAH+/3thpZ4VFlmPzgwnNXPDxprP6jKUPygvA9+UNdD8YP94WiKkP34pUKmKGF7ZX3Sz95fJ7U2DEXngRqetKmTPC1k79hTLO18XVy8V9Cmwul5YRaX3+tj82nivXFOX2McM6Nw3qQZVTSfYj1Pr2d8ja1mp+084KBgCP1lyiHtBfWpS5qzVTvSNLl9Y0YEB5zP5/tC52G6T0wzb3a82UNl7HjLG+ZMypeRxMQE4kRuJOi3MTNjHLfRK0CfN463496GlKSQeDMSjS5NlNKXUDgOxYW7HPZj0yeox+SHKBmjjlRqm4TUuzqsB6UkxVOPUZE17sk6SW9ql4fp1gybzvTY/bSqtdrWZFoT9L9+xdKnFqVNuxjkAu5bkf6IsP3px9qubT3sp+Ee+eMWfEry6dGy65oA7LbFQdpPOe1dIxK3btntztWuwqtNe9aLaT/3z9Z8alpPaz5t6s+mAXe13I1/+taknyIb0n5aEapXmzRMAdoEYHul2VqRtnRWJGJVYfdPNvqYYdovSmDtSbJbMPmcdbc/O/38rTZV6IUwfjDt5M3/4gc3Z1TjFH7Y8qO0RlMNV3brhLvS5fygH7atAf5CpB/9ksFOQ5sf9XIN1I40TW7fmBH4wQywPFj8oG5/wA9mjpNpNuUHmSEcD6BO7JTBnuYxkM+GUv1iNcLat8ShGzF17hh+xiHg88W7QkmZNCE+HyifMp+3TIw/XS6TcrnSJO2Kz9+t13xnJjO6apE/e3Y+yU7tIxqD3Z423p3/Z3YN1JEmHONvVGk/l58Qk4Swh8ZoIs1mlC1hpm5lUPWFFniNF6vffum5UJha9KcOdoynn0wJ8QFpws+dWyhtEu/Q5kqzslyJCfn0R2W/oHRXrC1dpPZZi/UlBkpnjddfhfWAnSTgfneyWZZYfE+PWunrIWuLsbPa7aMlrPYMTsVKUlrfyySFTSU6oah5OW1KlSmte8SObVo+e1iPgvYwJ1PmeqZL1wL2cBeb8WtJSVrv6L+WPeVFhdtDPls93e+nR7zoKa8Se3pUjz3uxStAu9nPVnhqnadWuUjpKWwOz39rXahb9rmqEmzqT3f7efm3umJS2IiYOS5gYR9j327LDMl+4mQhfSZRksblVvi2q7/72IEfPO++P5FJxua2IoAfXOruTpyLHyWsJTYMJ27QxtL+UKERVMOgGzz0Hcb/EBjAoGw2GA98+kUNRJfeiCrt55m9H3RNSSvMea5+G6fzg74CQ+RKxA+6K6H4wfTAm/9fBOF4zHWqpwwapOxdcC4wYsCz89UhH7+rcLAbkqOC6EJ75mxDBc2YsAgNuUl3ZJrsz+s1JId1Pu4dD7o3z2oKwiEZtEMpg2YorfBGJfd9xmDCgS3vBhTlM/ZNjFJ55xdFTBWMmtMvx6tEbbuz/0LLs6bJR8bnDR6OLyfcvpgWlGA3bea7wBxfEqlKQ3RaZpIvTWOq19tqJZgHtk2fyO9SjC9cSS2BOyuxlzs2m133kFW1OlGbaQZ1dig2eTWlTn30XJyWzRiftLqhVpy8tq7lmp2mPc/zjvVVPcfljp6e6CppaA8w0Dsh9XoPlXyi15JI27FjY8eOzsl6cMuKvcJB31KPttpTT2e12yYNYLOlY8r0mVtCIJ1o+lAicNWt9tSjX/w0oJv60+NAba/Wf2r2T8P0Dl29a2rx56ru/bNuckisqDTgus9tudM3Tyx8x8lcyrCnH7CSLK/764rt74tCR/e/9kufuOLse/zMzt0XRdEEtxEB/OAW6OxBb2X3MFtM7Ac1RkoRunTp5VG1OxmfC0cIRWo0Hix5c1SuRDfsuMJtrMxlmq8fTNqYsyJrWuqzQkLLWNoPpihdbvwgfjAcRosK2O9OP/pFVb909T76p79SI6NOHHWi6Y33Z1oOo0XYtEDBaKYoIHtgy/GtXg49m7kKmGmA85SZuHiz3+95Jxeni8PB+dnUlh05SVYfH5aMylt7cVJ4yMUFjhtWlX27izIHNZXfmd8VBKdPnLHWtbFfTLw3tiEUGxL4cctnV6TZnEzRu4DObNymcOnN7KliTCrShJ5CafmiLJ1mWlKETRa6gFaZalMlqxw379fvQG+MKc6Wrdrdu7u7ttprrzU22mt6zsGulXXd0nD6jmOnrd5x2sqx3e3j+meP+tQLHiTz3AihaV6tI9CTWPXOQJWjGleb9mhQab+1hhZ/6smf9thPPUO0barTTrL8AZl+J4dn8qfPz0Jpmn54cL87nlLtF9Jn0mTi8xWd9xUf2Bqa0DS6HTzp4ZAHQcwgAfzgII8tuKUfg05aNPSkP/y+Qyxs7Vb1g2rsiHEheKU8k/yu2vtB8wv4wXxXpj8B2xOHfcqt5AeZIcz3fqkYjRHJiWlutED7lSJY+0Th/NCP7PmBwLcgvGg4NGhQQPZVhBIUec10/ivkt+msJJc7xvol6L7BKFUcjqKrCKro3HE9UO5gG5Ndjpgpq4GkavWA2dqZJBh9JVUQ4hZGZaQ9o1+a3TCoTbNXf/zlGJWuGOWWClQhLc0T6gKOq9P/ZpXL4nRJVorRyrV6XEDPwLEdmjN0dbfd4nBLo3J0DcinSRpgz7yxCWS19+ixxonjPT25ZWf7yO4dnd2nNM5faf1PZ8/+xik39dbPaB09p3noTisH97Q7p7a6p7TbO+1Ov7AQyervNDqas9TNj34C0Lc8/U44JH8G4aZpLLYwQYjfdOovZA+BwaqiLdcx13/mUefdb4towqhtBMcg4H9rliH5Tffz4gf7LJYw5H7i+grXSWyc1WfY4LBV/WBGxWV6Mu/slMCBWkY/aI2b2g/ao7Dxg5njZFk2/ZlQbixfFvPnbuePvuYH913/YT9DGFYYzt0KKpwHgSLxZqc9zlOWMmCY7wyZ8+4kU6kpnEgJhYwK9PWPS+ClTpxgyvBoRzi68GE2h1zDEuQrzbdrUC1bkX78Upmp3kvmBhXv5waVRbtMASbaz0nH9CZDK8Epw0x3aNPHqxwXSJylVSk1KNFojwN1L4SwGxHt8ae6J1Cv0Gg3et9x9ifP3rHjxMkrn++e85nu2SutjT0rx85aPXLe6v7zVw6d0e6d2t5pxeTm/VzZxWdd/nCKv82SwY8dnOnU34jDL94VhwcLS7dGHvFbQBMyQ5j2dNm/+MGypJY+nbtJJmqFH2wVMXJUiDK4UW5gO7eRGXi1Hz+YgbQQPyiPKe+AH0z6YuQRvzX8IDOEmd8dmxBICAwT/MEjbk4qTRoulmeGFMmMuBANvvnrrE7GJErGJ1aetOB+7lBFiMq7kLBrREAl5wsfkb5wl3chhbt8ZHwGEEOwxZHRR/ow3y73zE+fyCYAFXJA7IqsKT6z34Wd3rOZQLk0WzKqekzhWU4r1E36patJrSIn/yy7qUf3cSpRIb8+3LI6YelN6kgKegWqA0UWdPUSDE0w6jmidvfpiV7v1uPHG+vHP32ovXft9qMnrxw/efWG5qm39nZf173jLmu33qV98OzWyi69DtBakP34yPg7m8K1Wg2x/9ynsJzMrhFpkvKLjEl2DfnDPOEQMERDYAsQsGdnFX6SQbJwXyYyTRoG88wwE7sAZdUwhR+MES7QD5rHNO9pPYYfjDslE94afhBBmOnWUpt+OPMXsYbJhlIF1TiRxu7MqF1jY/um1bM7gjZLPWPf4IGQ250hH1pUtpC0xOB9fUTInu4v/qtO179hJwHFedJYp7bGPmqCEkuL6f8N+jC2J25XEId+RajqtmZq7s4CicaTIrSHy1izTBMqu9LIz7mlpJKFtppUzxnVPgVsn1OG5ged1nKNsnjlsP7x8ZohVBG2tFQvR3biUnccqkZbtqpsvT+/+aFtvUNCTxNtNQ73Nlb1vvn9R9fWGrvPWL395FOOnthxffe0u6zeerfVQ+e0d8mSINVCoE/B9YizRV+GtzCNTx/visNxaf2wFVbBZ2v4wgpAbLMi/OGDH6xhtwevUSvb/Igqk/CD+X6psR80Y0v5QfOezgPiB/MdvAwxCMIJe0nHvf+FTJi/RLbMqokSOUiSXWoSE6mnj4wt9GENqnl/GVxpPr0cSZJxpISPddSI0nxRXomp3LwleQOimLEVRubacFTUQFD2FBrj1Z2S2mSddKzd+qeAM6MvC/VTFRqlULwCltISuvTKon/aZd/uLkS7V9CVaxdEnThUyuTjjbBirBLdpKhHido/vfBQS0btfYB6k4QWWdm7JezNh/amCidmVV2z1el0j9x64sShE709rcaek/53Y/W2xu67Nm6568rG7qatIPUfZ5oVr00fTvcM/I13xeGBRGHDCqv6YyCTz01f+OULH/RH6RZ/twsBHXj4wRp2dv7kwY1YZil+MPQXfrBiP+hvncAPhiNs2QIIwml7bOLro/khe1pTyD+SQCFwd4Y/i5PlkabMZmdQhqH4YXLLi6J+Mqc9wmYIiEuMJp6mC2niwIhrnHGyEB5mXkgQAptW7VNKFqaaUBGmEY2J++N6XyGLsXY5pefVoM6QZEmQhV1bZmo3H5pgVFrl1KaJO1nhJgkdFKlBu43Qbh3srrRN+Ol9FfrWqyZMIkrP6SGl7mYYJXeHmdmoZ5Cur3cP7e+cOLKxvrvd3Hvysd7qbb1Dd13Zf+Hqmr1QfsgnSL4QGJLQRTsLRyUYd5+Ry35iJbB+9B+u/+9fP+8+vJ8wS2mbbPvRdQKxUTgsbxNo82xmGBIKgbsBKiSZp13V12Vj/uBnmKPBD+IHB4+UzbayR5al30p+EEG42REwuF9DZuEJWeEgO5iVrToSsA51U0axcROc1sTZZxH2Rm56RTNTdcY1TukX/bShr2KYQgs1DqsrY2GZzWF1jc7riPU1YfjZmoqz53kaSy8CNciHZ8+o6xVWfKcrUWcvmtfr5u0dFa1EE2p5qT9vUoH2pkGpQXutvD1Oxt47rxcPtk0Qap7Q16J4PYxUtxvKWsHRQGGHXK+xvt44dPvG+pHGidNWmrv33NHbcbB3+11WjulhM75837pg9tDGxqmHJhpzx3C3pz3DKrzj9tceuuVxu8984JiVkXz5COgYKDwy8YPL15fOYutQ/OAggWHeFj9oDhM/OOSnvtR+EEE4pFeHRF901t323fBhf645JAnRS08gf1pTE4noffYwR7Up96DWlHKEYMucGRRWt6lTLFNXSDPCmE0blUngJwlDZKJe3Hygr06tSzSh1nNqRtGm7zSdpzsJTSO6765EoN5HLzUoaagb/Dpu4lExNk+ouxGl6ew/y6cS7J8eLtre0Pvo1/QiQbdgptXW2+T1GkP9M93YUiJnk15nr7/Sh/4Y63W6B2/eOHKgddreHd1TTz/SOHxh49CFK7us+MLPkOjCtGUjh8u/ESUkmdz8a0h24+efsushn17ZsTfEENiSBPCDW7JbM43CD8ZA8IP4wfh48OEt5gcRhPkuJgYCWQKxa1ygOCz0SVlby20HMabko/XYpvowFocqLT+ht2ldPsEwM3z5+WJ9Q2VeBotK80XFu3wVXk8pXmX6GLdG043qltpu+1Nppv16XXsZvZOFcoR6P6GmRDY0UeiWiMSmSg3uaG/onfL6J0G41u6sNju2jtTmA2160PSnm5E0O3WvoaKdHSrR7Wqo9IO3a0pypXv6nk6jfbR36IJ28+TW2jBV6Bs+4XfiwQZyh0Uv2lkoOWO3FxfQz5jG3vCFP7rgfiwcHcDLBgS2BgH8YOjHjNNRPH4w7wftHgq7qR4/GA6cWgcQhBN2jz/p1DliOCWasCCyLRuB2Cl62+cjEfMeqCpyXhqptFjnDCvcFFP6caGsgoj9Yl7Fhbry1Y3YpcS+2HyB2uV+hX0zFFJRmuvzu/SdkTkiaW+BN43Xb4sl0lyf03var5sApdY6vd6GTRJqqjBZIidN2OkkdfnfvmrXIbGqucFmZ2drY2drfUdrQ/OEesroSqPTNkFoc5Eq3/BKEKpagygbWpowlAUKa63psQOta4+0D+45qXu6dh26qHnilObagH3WkDE/RflHDFm+YUmmwdk/VRwyhkDemjtue+3Rg8/ateei/C5ith4B/WR0VOMHt17Pbtoi/GBA5AbMxCmESPwgfnDp/CCCMPx+xw4kZ4hj5yPDViOQcY0z0ocZ2TMLiF4glZGFvnbnA4N8yHpEpYmdojYzci7WY5lKh+2KC4xLM4XlPkE2O6lldwkqOuz1afy3ZXd79TcuVlWvSPvYaa6Un1Z7dnUP4bom81wp0kLuATNWhtLYmyqcotOm/N9ac31Hc+Ok1gnNE2rJqAShv4dQjyGVrlSapq0d1ULStqYcu3qxvRtEpAwV0NRht9M4eqDzhWM7Nu7U7jQOXLyyrhcVWk1lPgmAbNIR4k1J+5kmkn9WgivCcqclXPu5X7rHw/44awfbW5QAfnCLduzYzcIPCllwQDG+2L8oPvZc2hzm7EbsiguMSwueLpixaD+4IWeKH4wPhjqHEYTj9c65p98pk8FOE/tnVZmdbG5HArPzi364D2N9reAGVySrhlk4zI0pyzROcYRH9O5Q5XtlaBU1/OSh9F7yuzVdlxqgvfY4mPSjV0dofedGq7Fidw42V3RTYK/XafU0SagCVYBaaktAG3pNvVmhib61lpaPrq81bOHoml5FaLcUao7RKtZHulJf2nZXDSzQ6bXXm21tmizUd6+1cax79XXto2fsXT1l/z1XdRdijC21rGjMKTMQpW2W5QOfkDcEBnanG35vLP/8HsUnu3qNzuF/2H/TJ089+4FpJv5uNQL4wa3WozNoD37QjbFu3B/EGw/osfNSquCVFB73ImlcVHDH3hfjBwd7IHV/6UXMsDe4vxAIu+JA4uxUzGAJik92LacfRBDGvTxheDJNuKgLq6MP9AkRkG04gdgvVjJ5mBnrh9c89p6MBxo7f5qhjIVTOkV3GCe+trCoYIOMklMMHlGb2uX1YWivPLcvyySZcxYqU/+cR7dnh/o5RCfYmib+JBu7JjI7Xb1n0JRlMgEoTWgTgz29fEKCcKcEYbO3o6UVoYZGP3mZIUkoldkxm+ymxI3mRkeV6yUVjdZ6o72hf3rMaad58KbGRw+cevL5t1ywepJldqZYIAS9cS4mc9KhPYoJOfI/+RATAq6Y7Jff651eKM0M8F7QGxDvSAu46Yo/OPVsJglTHNvjL35we/TzhK3cln5QrGx81GA/jFqh8/KJxxWHhUUt1g/aY7fxg8P6vn7xCMJq+sSru2FnV4vSfoVtq5UxhRZOEzmsC6Yps8K81TrFMNaP8DcVGj9BUd7CTc0r9GS+utgpKiZIOHcY97VIiC8sKg/K68NQeKIPEx1lzrtjYR1NiSNXx+kppFKCeheF3VJor6Kw//XMUWnCDb3L3uX1BeruwdXmxo5md4fUYLO5JkFo03xaLtrU00fNcjNIZfc6kqb6Z4HmiV5zQwJTE5J614Vttk8cb3zwi3u+8txD9zx5lz+tMAKei/t2JfVjkl1OrcXJwo8iBKIyBoJKkP7Xj/e54u/+viiUJrCa7zj097d/+ZOnnfPAaD/BrU/AOxd/JORbWyvXUytj8qymjBnWBVMWW1V2/GAhyULn5VMGV+U3g7/TZrwrxBcWNX8/uIYfXCo/iCAs/GFOGLm1fcyEUOabbXQX1MpNBqc4/bRhfqCfjLpcS/Aok5WQzxU0VX5XYUyhJwspY+enyGBtYXxclBLbutC+erIig22JPrS4/ifs9VFKo+udelyangujsnp69qitF9WaT1s1qk60iT7nnlWLqrZ7J5qNtWZrR6O96pZ96uA0PWjzhFakHasmBe0+xI2GqceOvf6wsd5snGi0VnvdXXqiTbd51Y29zqlHLjn9ZG+Gy2pBCS/TXguSf1azqna/qPhnFcK3XvsvCELrp+33GT0Ibz8eC2jx6C4IP9IFWJarEj+YQ2IRsfOK14L6xIX+TrsK4+OilGYBfrCBH/T9VvdvBGHdewj7KiQwzE0u1kHWyiMGjxK0VoX8xy0q9mR5p6jSgrW+5GBzYXxcmtLn/WLGvCAI/fyhq84kpX/3oKYQ9ZEks4WjWmja1UNhtOzTaUJz5/bo0RU9PrShdxBmanY53ZdNNjp5qLfRSxnq33qzs97rrdmTa6QN7aOD87b9jSs6d9z9LFs7qk0f73baZiaQbOf+uIwudVSCzx5/5/IlEWkaK8GH/Y40vp9PKRS5/8uvPrz/WaecelF/ByEIQKAGBPCDm3ZC8CDBp2yaZXYJYv+xXH7QXRvVhVH84HL4QQTheL/is04922fQ2Z6fEBgvP6lrSSDvIONT3rmZXLkyDJJmgiZ4jzilO5zGgIzN3ikWusOQMnhxHxOML4yPvWwoIQ74mwkVY3f9pU+a0UJRdx+gXi1o60p162CyalQzhDZPKC3kNaF0o80Htgcu9cbFZ8PSjV5A7mz21jVZaFOJ7tPThGH30JHe548fu+TOOxWlebn0+AzzhdnSEp3mJhBNw6UfxSe74th0b/ibprFEPux3hXA/YClMoyrGf4dCbrnugwjCQGMrBfCDW6k3Q1vSESdEDPz2+7EzDuEHRwBeLj+orsQPLpEfRBCO+Omxa/sSyLjGcPo7HyJVeUS/QnIaVTalLJzegAzwMu4wZIl1YBCH2hvHh8T5gFaGKq3+lzJ0mtCmBxVjglDvJVRY/3e1yrOpewilBjt6RWFvRc+JcctH7dX2OoqCrAvlD9dwZpkupirlqoRk+GjdqR5t2ui2N1pXXbd+9mntnTv94dkvyR+fpsicKgtZk3hryCbndibnXGqfxZcQwv2A26FN+2eFum+fOvr26W+48ifvct9vi6IJQgACy0QAP+h7Cz8oDvjBLe8HEYTLNDpj66IIxH4xDApzMKYSZehVmaydWBnWyh06dTM2+4wIjPVhviyf2HW6VJKJM+t0zRO69aIb9gYKU5XuJfN6eb10YHtDj5bRGyPszYS22WisFxSbjxqMiQ+zeI/egdhqtGVCZ6Nx/U3d885p7dyhXjWj4qPRh+PvuJA4nKYxlsNK8OlDFUoWwnFRSTIryYrKqMpDt39x92kX5dMTAwEILB2BeICKx41ZNwQ/mCfsRtx89CYx+MEYkD+GMz5LCdL4ftrg+8zHpQn6u9NQyJgpc1n8IIIw7Un+QqAcgYU4xQo94rLLQs2L+amx0ctHR3dmxi+OSBxS6hky6gU9EdTUoE3m2SVTLSXVC+a1atQEYbe9rpcKSha6J4Vq6aimEWfxue7L3Tud6TShrWWVdxoQdfkaUy9lpxA+7NOEcD/gdmjT/g0mzhSbpHE53Vey34fD95e/9EEEYQYdmxDYAgTwg5N1or8+O7EXDpWaD3Ib+EG7VosfDEfGFAEE4RTw7KRQFwJmc9I3lV1knhOB+TvF6ZXhlBOGXiBNfNin7lAdNO0PJ9wEOI1HHHaghMKVQOWr1VpBqueJ2nNF9df/a+nZL9YOu5NQUrDbWpcmtBcJrnQa65KFnVgqDaupXHwoSYecwjfc0t17am91hyRnU+tLVwafW6MEmSuUqiSUEAIWmcZbljSct8hnsTQu5Dd9MhdpwRDpA/bd613zvy+7+FJWjeaJbqkY/OCW6s7xG4MfHJfZ8vtB+b1q/GD+rooRMIOXwQ+OoDTxLgTheOjO2HvGeBm2ZmqbJBn3M7GEGLeiRaXPMAkj14zs8cpwmldWeJ8k8ya4Wjm1LLRqJ6i3EGYQb+4qYWGS8SJDgT6bNp0mdG+ZcItF1dfNli0PtRsK7fX09vLA5J/NE7Y0VaiHTnV6Xd1JqOeIjld9idQnuhvX3Nw55dTeSSe3dvY6EoQ7G6vKFx91IdwPuJK1af8GE2fq9GlcgUrYL9YXFX/7vT5G6UKxIc2xowd27tqbKZ/NpSaAH3Tdhx8sOIrxgwVQhkS50yiNlOOfThUVGLuZSi6SxgWqwkE/qIPfjv9SflAXTvGDy+AHEYRFP6xli/Nn5zW3uloj6y8vY7+YnC7PoIdsEePU7mRiZRj6dLLuSK+SVuMORVcF5csa1zVmvKDvNEX690z4KmS5YqyL3QyhPLpeT29vqLclo+0TvRXNE57otU40u3rBoN48Udjz8RFSmGBEpKSm3oN44/7eSb3m3p3d1WZ3Z7N7UnPVHmpqCi75xPIshNOd/b9BvBVPACqh05A+Q0gcyz9L4urVdzeShYrff8uV517wQJ+XbwjMjkAYjmZXxfQlV2vkZAPv9K0oX0I8yvkhonze8inxgxlWeZezSD/YG+UHM5aPtYkfHAvXpokRhJsiqlGCan1JjRo2viljoVi415ypU5x+qjDgn1ihxd0xLu2J5Wgwe3Qg7xqVvtA7FqYcXbid4jgNqslCzRDaqlHNDTpZuG5PHNVLBTs79Y76qj96vI2ebaNJyNv2947uWZEmlPLcaJ04qbmyoupKTADKIhlfrABTaeet9qdx9j2o9Fz2rPzLNFRpjh65NRPJJgSmIRCPNtOUswXyjoXC/ZDdaLWglsd+UCY4eyozBT84GmWhd8MP+uMQP+gPnurPVEYflFtorztpmnl7JlmUMnOjlq2C0V5zXAEzZetn5BTr4A49mUB7XLCRMlRJsz1xKfSOmZ6N7DFjrF26hTBN1HelWjpqb5G3f3riqMlCmyRMHjAjQdhpdPSMUFdCplGZzbToEn/dUh0V2jveaB480NN6nOaO3rruWWxu7G5nR3V/4qXvQren2kICX7NL6YKRAgxpMhOAGWN93vCtvUcOfTmThk0IjEkAPzgmsKLkzvWEASybYtzhOpt//O3YFfrhZfwysjm2kh9U26paTZrFlG7Xww+m1pT7Gx8q+MFyzMqmyp46lM1HuhkTCGfVM66H4t2Jfg7D3Lxj7BRlRTzY5YzaPKJadzilNwrH8LgwnU4aOHGZ0pLNweVSBDWY22NLMr09vrMSVecm5ewNhCYI7c0TWjUqZXi80TnR6Kw6QZgvymJU3PjCUDcNrjXt5YQr9rbD1uFDvZ3Nxo7VxtFGb2dvwyYJ3bFUqAC92eFI629G8s9n998jFGDIq4D/pyw+Vwh8+dr33e8hPFfG8+B7PAJhDBkvG6nHJ1CIetyhe/xqkxz4wUJ0GTeEH8xQwg9mgEy5iSCcEmD12QvH5eqrocSRBAp7YQ7eMfaL/mx7pJnFO70s1L4pby+UN6rEAwWYEwPM+EU1zenF8YVUMbB87IAc9bsLjI/qVwb3rFGbJ/R3EkoNOk2oOcPehj33xSYJCz5RIQV7h0RJYW40pQYbK/agY1s7uv9gb+fulebq+tFeZ62hF9zbxV9/CMXfA5E5BehTTqkAY5NV4MHb/qbR+OM4kjAENiUQBo1NU5JgdgQKe6FgJKzaAvxgIVH8YAYLfjADZMpNBOF4AE/dc7rPkP9ljlfQkNSF4++QtETPm0DcO+7UeaJz+dJWx05RmfzJeunclnB6ZeiP80pkoezxACs5n3Doi2TbBNNtBUzd8tCC+MGoqH7Zoy31UafR6jb0KkK/drQlqWaC0B4tM0QQDhZZfktvm1jX9KB79veK/vSahw92d5/U6Ozs3dFb15NsdjTtuaP6+CPHvosU4Aj5F+dVdv/PFZmUGRL4SLfp6kgrtZhG4+gdB3adtDekIbDsBPCDy96D09g/6Adn6wRlJ35wRGfhB/GDIw6PcXchCMclFtLPfBwMNRGoIQHnpXSua59KFI4vasR37Bf9Kf6IxJldQRkqfoJpw8pl4eyIeVMzzdfmWJo2X0jS066vZbwTV34E6I8DCoVXKimN42zfegCM/uVNCh0aAvk0cUym0yX5VrVw1G5QlKbT479Nce5qto+c6LbbTd27qLWka412yOUD+p5GAYbSDGm6EUr21qqpFuPW0KZJ4nYQhgAEtgiBOYtDUYtHy3GHF/yg80N9nzX6KIz9oHdg/lu5nBMs6wflnSr0gzoA4n5fiB90BAI8eUIhLbgSunR+EEEYOpUABCYkEJyiGxcmLGSsbMEpxiNjyRKCUxxXGQb3MJa4KrSqwqnCwvLzkcH4/K60OTaoD/uM2JXPErxmIeHQd/mM48bo4uhqr6vpQZXZtq/m8Y3uoeO9k0+2N100mx3dXuhel+gcUzBrsBrtC/+0JxxRPpBuKrMtTg0J0ngrS7G2WaQA9f4pJTh6ZD8zhEaKDwS2LgH84Lh9uwg/KBttTM5/RvtBUzzyAflsw2NCNf0HsEWJl9cPqhE5V2ht1f/L7gcRhNERWjqo8zw93690chJuFwILGN/TETo+QS+JOyhDpS+ULsPK8eIq9R/DUm0eP/8TiEKbRmhFn14JNm9ss6F31Pve0LfCyivCHnJrPE9aaGYSGXe0Lo7udLcmqsZOo7fe7d2hB9qsNQ4f7a3q3RO6n7HVWcvdu6gS4n+hMl9yXL52uU1zf8n1T5fabVrIAi4mfIWUCvhPPk1ITGCpCeAHl7r7Zmc8fnAstkvrB9MhPm5tsR9M7l6pxA96zxL8iyqfpx80p5e2W38tXHQlVPGxaJSRFhODqmUYQThet5y6e6/PoDOw8DMerwhSb3UC4cCY24ShiPrrbWGoGpdxEIfllWGsozbXSyMNmv8JxEhzkp0Zq2xTT7+X6uu/e6KgGEfSRv4ooLCXigXpp4za0bQx3N430eieaGyomhMnGm3ND2resKvVoZqia5nR5p+Sf6FGf7QMHjPyYioj8Xlhl3k9d5nXCknzp45xYOZQOy2BS2aJfeqQJ83L36UmgB9c6u6bj/FhnMQPlgTuic0TVxnDUquU1ryYbSZ+cFTuyP3Jo5gDkFPaCn5QTUm9W9x+5+ysmYnLW04/iCCM+3SM8IzOcDQWhGF0DGtIWksC6Ug6KzGQb7RkYRiP8nvLxEygDFVsEIfTKMP4yF+4UwzGKLCJMbnuDTT8Hq3kLEN+mjRts9KeK6qapATX9erDI73Wim4slPFaU5o8z8YfG+7bD2BmmN/Ut9+bOjK3mfN8Ks4nSxOHAsznR4W4UwDFFN49OU1TyVsnAvjBOvVGTW3xY+kmo2iltuMHq8IZ+cHgIIaUHXk55zf6ZwV+T7V+sLCLS/tBeTFvlAawxG7n0cy7pa7Nxjbv1PIK0NygG/vSxMbEZ7ASnCv0mFyqAc/o4+v5jSCsXb/oSA0/wtoZh0HjE5izOywcKMe32ia4ys8WhvKDFlLM8orD/A/QtSvyeKHBITC4M+hqRQ/uCRkqDsjdtput1UZ3vdeT/ms1G51OQw84dX5K762XWkw+/r4+bSQ+zUVbutwcoM/g0w94PuflvOdz5fiEiSP0KUP61HUmafgDgTIE8INlKC1RGvzgBJ0Ve6JUw0xQzIRZ4tp9EVvFD9pV3tRDhYA1se8HzT8OcBvqB9OUIb0C9s/lDpG+8IES67eBIJyqT2b0E8UXTtUrtcwcxtYZHTNxoyvUhCp2AlnojZmFOFTJcwAYw4zDxdLOFpEmqUIva1ua0BPQ3tktlQnm6eKobqXYMBXf9HcwKtjpNVrtbqdnE4WyccA5pZdCzQVGns8cZbqZCWjT9qSuzoIu7JOFxN6kfinBRAJblMCMfpIqNv5BbVF426tZoUNndMzENPGDMY0Kw6m7Gywy8oPeAfnddfeD3v1Fjiy0yryb82rBtfmAxSuR/3aptWmR9sftCkVYvKVdlg+CsKY9hS+sacdMbZb3iLN2h1X5wqmbmxQwKA4VWexTylQXTinixBXyLCzf1zXCaDUwkX92g0ViWgjEps4urElCaUJnibmljv7plYiaqNWbEdM7Cc1jFTgt81ux5/JOzFIOOrkkxvu/XEsGisjtJQIC4xLAD45LbFnS4wfdGJu6ivG7rdBP4QcFcnM/OGQtjPNuA05sXD84kHn8Pl14DgTheF2w55S942WYIjW+cAp4dc+q0bzCsbuwtVVpwsnWjhaa5COdD/QjreTJ5B4xrqLQO8YJFC4DfLNyErNdaQPFm3DKfdTHYe2o7aykrSqkqC4VLzVoVeht9XqgjFvwq1cOShP2tNHUOtKkelnlrY1tDjG+7LDLNp2AVEyItLa4T1JQujns70knnzpsF/HLSAA/uIy9VkOb8YO+U7aMH0yPsQFXJzdRuR8c4gOt/owf9I5rDD+oS6qpswsub5v4QQRhegDX8q8/hd3sJLWWpmPUZgR8t5ZRKZuVNHR/VZpwaAVT7/Czar6YqpziMKMm/h2Ne9Yy4AyHWTObeN1GqCWjegWFXQM1J2Y3Pihs04apDgw1m9tTKvvfpXU7Ys9nZdh2/1NSAfYzEILA1ATwg1MjrG8B+EH1DX6w2gN0wA82euYQR/tB7wG3vR9EEFZ7HM6kNNzhTLDWo9Bxxca4VleiCSufJCxshbswt0AxVWjUZpHJw8rSZDUwX2+ZsOWrdhOjnKC9IqNrilDXaJMLn0HyeaOTzZxHTPZmFGHaUP5CYM4E8INzBj7P6vCDgTZ+MKCYJtD3g3ofr/Num/hBd+nTp1S9Gb+3Ta6EIginOeTmmjdMJfkranOtm8pmSWDWl0jRhJP13liOWZDjj/LGm/MP67Gietao/KBmBm3RaLexbo+bMUO8q8t7vsp9nvlh9znp5L0+wDcEpiSAH5wSYG2z4wdD14zlekKuGQWmMaYOftCuhG7qB/310Ihghd5wifwggjA6BMYMpmc7Y2abOjkXSqdGWMcCZnqJdIk0YU36xnsy/615NvVOq9d096KPYaCyKbUe87KQj2qXL/TrRXUToWzQfRTmG3PWTO/8gs9T2aF8t0gnVxkREKiCAH6wCoq1KwM/WKsumYkf1Klz5gJq1Obhe6JE4wQ1VSiXlPWD6YXRuCT8YP/lVDEXwiMIXHqnR4zYO7ddduU//Te3SqkIArMmsPBrijJgMhskubS2NjxxW4GBO+lnDa6ofPlC3WHf1j89abRrNxR65SbPF3+Ksg6NUwn+X0dPrEn/WdnpP+nAjvsnH6x/un9jaFnsWFoC+MGl7ToMXwICk/mgChu2xfygnqkW/KCtjnFOKXaCCo9Fb0v6QWYIxzoGksSmxBZ01b/QXH+t1O9yV0MKUxFZdwJcHE2O4VRCTP+Ymfm4VXXcsOuafpzQg7DVtEWNGZKFegVFcHdlPJ/XjbI55Ao/nnjeT3ovfPJTj1qqGvYS2HoEnB9c1EFdgBM/WABlCaPwg77TgvPaan5wmLOc6bHas/Uy3g+W8YDelgn8oDJmXOES+UEEYcXHYHyVYTGHfSRVEYcV9+6SF6cDMj4+J2vNfB4w420LHlGbEzjFOPtkjQ1m+NrtFx00ThweXnr9f4Pe54Vm+abEqk8x4wq/+DA7/cxvHY6HPVuTQHwA4Ae3Zh8vbavwg5N1nTt9sAtAS+cH4+FoWNvz2s/pugHHuOX9IIJw2OGxabzdHxQfZ3HYZ87EzN8vxldMvUn1Pz3dlDsJtieBvLrLS8R8mjmz8j/5sGp0rNqnHB8yo40fmxSZ/MuZIv/nfV3QfrG3U/JNL3Nma3RV5K+GDnjUnBlEbDEC+aMiEzPlcT4BLvzgBNDIUk8CeR9XQz/o0U3mB6fEnhlt4nN0eaK8M8IPxsARhDGNCcPZQ3BIMXGy+TtFb1TeNfp4hOKQTiO6gMA8JwkLqk+j8q4x3TPPvwMTheV/1+VTTt6YvPdLy/JeUFLQi8Cg/ZzLHMgWj1ppbk0YDqQJ8YWxhZEhC4EtQ6DwUMm3Lk42j19B3gKdF0braOL9+MGYBuHRBPCDEZ8BPxjFbxKcxwgw3APhBzPdgyDMANl886Kz7rbvhg8r3TCnsnkRlrefah4/iX5txaGx2oLXLIY4XexYXTBdVeSeOQG9A1B1+F+KLbIZ/hm9tyBfNHQU7B0cWwoTyAu6B8DYTknBoO7iQSlEZkoYXXlhrszEY6ZANpeUAH4QPziLQxc/OAuqiypzKfygvx4anNd29oMIwql+KTp0ppdz8fE3fWlTtadc5omHbDzoMMATIx1WIPHTE/DPCNXb+4YUlY3X4a2ocJD77JlE1fzAR8uyIeaGaOWWSAtS0I8/wR0q2bDi4zShtDhQqP2GlRZnJLzUBPCDY3VfGCLGyrUdEuMHa9jL29MPjnBbW9gPIgin/QHGcm7asgav7ldz7ji9TdWVMGK4384+cgSW6thXXFJNVstU3KqoOO8FFeFbmlmeGt+24Q/dYQewF4T+KaMqLT9cyPFkRGNkxXjBET4sFKQ7B7VS1KtBb4x3byHvMG9XKPZCsflAKPC00y7O7yVmixHIH9jTNDAuDT84DcklyosfrGFnzcgP5luKH8wzmX8MgnBC5vEZ4YRFbJZtazvFTOvzzmDYGXYm47Jv5hs+uxbFR9TsatkCJQcvOFlbgnrUgpm6nc7K73ppp4MhloJBB44WfkHmTUaGXFuMAH6w2g7NuwP8YLWEVRp+sCRS/OAwUFvVDyIIh/X4qHgNKENuSh+Va5p9mSGsbmeZ0zRtWN4t7xrzDRyGoobxwxdS1tDYWZk0ogfjX2irsinAChriPZnknw/o20vBWAeO8nba5yY0NclY8lM6YcnySFYXAvjBOfREfpDZYhIx38A5UK2qCvygSI7owWXwg4krHM8PpgfQFvODCMK0Y0v/Pff084alLRwdprzKUljXNtSH4pAZd5baL2baUtjL1UZmjplqC9+GpRXqHB2T4ffuj083W9jUq+GHIRq6Y1iG6eJ1Lcv+6X8LmBoMUjBu0Sg/F6ebzhhyLy8B/OCi+i7jO/CDY3UEfnAsXJsmLvQGS+UHrYnFfjC8kWlTClslAYKwgp4s1IGh3PzecMoY0kwZyI9xw88/p6yqRtmX1y9mLJ8D0/wRMk2l+UN6mtLqlrfsz9N5QhNW6afMmdmifpg6APrvHXQGy+7gBX0b9F14nPRbmLZ0rL97T73zWOlJvKQERg8L+b1lf2ilceSP3kX93EqbXEHCjDcpMwpVUGsVRWQsr6LITcrIHyGbZBi5O39Ij0y+ZDvH+nnGPmJ2R+D03acSNveDsVOPOi1uYxRdNrgUfhBBWLY7C9NpRBjrZ+MLyYwjE5RQaEwcmf/lbHnXmPEusxuVYs7jhjNGjpt9svT5g2GycnyuzNE7TVHLldfP9fVt1tRe5CIKJ/pa/dRzCpXpa6cD+6b7UJgS7O+Yk8lUs/QE8IP16cKMi8EPhq4pMzaGxJsG8IMDiFL/l/4d2IkfHMBR4w0E4YSdE4bd6ceFfAnzkYhxy7eeXAwd5Ju5cL+YsSeGP9MwXnAsvJX89FRIKEcO0v+45vwTG6brQryfHozVYNjliWlzDo8MGat3SFw3AmFYy3uxcU3NlxB+ROMWNSL96PFwzj/SEXZWtSt0kC8QP1gJ2PyxWkmx9Smkkp8efrA+HVrSEgRhSVBzTZYfbir5fY5ow2g3OSLjuLsW5XEzflFmz8E15isdF1et0ucPy1qZN70xk/3KMod0uEQ6WWnTt2J0CSbzBpfE+LnBoAYr7+U9e84ZbRJ7IVBIIH8ozvo3hR8s7IgpI/GDUwKcc/bJfmX4wdHdtBR+EEE4uhML9p516llyVO2wFDlNsunV9PAw+jTHGH/n7xrHMG6cpKM9bmZMGafgsdMO81KTCcVhpY1tVqUZRtMeq6r8EThW9qVOLAcZNz9oJ2uUHbJJhEKjXOlAtsXwSG+fsNr1LBlZFKvBuI3Bvm655ym3mjVoXjCawOwJ4AenYTx6ZMYPTsM2n3c07Xz6ETGFg+SI9Ftp1yg/GLVzdn5Qv4vQlfI3gxc2IwuGBPt5Qwg/OMgKQTjIo9RWmANIUm8qBX26fLJqJaJqGXU+WqppC04U/U77lszTO6rWekq7Po7SoUKYpXMPJNwOXnCM305OIw37IUsmZQeLAa5uY/MUA3kmcIRxfu9EY1fqlVzcxSVFYFxsJgv6MIazRcPZAzfv4Aobnk827OdTmD0TGR+3YdcYv+WQp06BwqEbPzhZFxXCnKyowoNtsqJqm2uM307OuQ3LOws/OBnAcDDgBwsBIggLsZSNlG+bxplV6xpl9LABa9gPtWw7F5ou/IaDFXN2jaHeJQrkoS2R8XM2dexfR9Fvvh83eJ6srVq9h1BsdfdgIgL9n2jcyOi6iTvCl4MsnBjgcmXED86hv/JDOn5wU+x5aJtm2bYJxvaDYW1MhKzvB6NIBRfuB+PVMd40/OBgF9kWgjDPpFRMGGjyoq5U/iGJ8qUN+4ENKaA4ephQVOoJRoHiOuYYG+D7OvGLMfsMnHjXZOERB89kBdYq1yyO/2az22p2i3+5g3JxPijCIZEKQKvWh+OYvBqM98amlmxEvsC4EMJbgEB0aJU8KEo1Gj9YBlOA7xPjB2NoGTjxrsnC+MFNufUGD8G6+cFgf+zX8IMBiwIIwpjGtOHC1YaT3ZMWTJmRawzljzvMzeIEOhgzWSA/9A+OS5OVuny58hymb8O4h8f0Nc6zhOkOZn8GbA4lnAurwKADk8iwb4qGVdWzsSMM5vgujsVbYbKQXoE4wabt27P7zDgv4S1PAD+4kC7OjxL4wao6Aj9YQHLAwzk/mB6C5gfTu8oHUqkUv63kGc+Rj8lVmRaf2zFmROy/Qlb8oFAgCMPxUDZw+t4z8kkLXaBPVrgr/FryRW0ak5eIyhJOQzfNPmWCaUbG6c6/xzA8M3Bsbb+YaewYmDZLOk1fb1b2su4v/Dl7B5Mc3pGfU0x/vWgUP//Gh4PE3zuhJ8rEn7BXkYXOMk6cCfv0C21cxiI250EAPzgxZfzgxOhGZIwHsRHJJtiFH8xDm9wP5suafUz+2MAPFlJHEBZiGRWZl175o21Ufrcv83OaRh/6uharEjdtr08wemCdnZss7KAtoBIL21WyL0YnG91To/Nuw70j5JB29TXhSDTzOSCl34LkC+EweoRdIy0t2OkzjuBQkIeoZSaAH5y490aPrvjBccHiB8clNqP0I8b/BfrBwsMj+D6hCGH8IIJwkp+Ghmx/9tb0x5ptJKdSGaVXsvR8ruklojvQh/5C8+68pKmzS1boJufsHX3r5nNqPhnJwtFtsqIKcxX2QmFKIvsEmsnb5/sxCg398Vmq5rwOsmEHTPrCib6VyRA20IYKNnafcnoFpVBE/QjgB2fRJ4UjMH4wg3rYsJZJNvFmYS9MXNp2yVjoB+vdePxg3D8IwphGuXCzsdKyo8idSbWUJzy/yAYRKS33NN5wsUEJJlBfM5KIoYWxeSGywsAETS6svXBcnp139DbM2tkUtnThkYWoF25VrQzwB0b4bWYVVDIiaJmo/eRnfZRWRSa+e7CqMiln6xPAD5boY/xgCUj1SoIf3LQ/xvKDVf0ECqzKOuCBJOEsTpde3dVXuedRGfCDwocgHDiGymzosNIh7r4VcjfjNJP3Peg2WonDTrMVThl9gXn1NcGPJFOmSq5kFrFMkydIk2/yiELGpZEfspfl5HsEhAXuyvNcoDFLVrW5mv7Bvum0n1LP+hMc4VgV5V1l5qjgJzYWzy2fGD9Ypov7Q0OJ1PjBEpBmmCQz4s2wpq1X9HA/OAeXVyHObe4HEYRjH0un7j5N79fS5YT40EnOlpq9Trn3OOT9xLjOQHbnJaIi66wSh7HO0/ApyzPJDOWcvA5DnYnPcMvsrXBzWBeXqaL8YVCmtErSBNEVe7v8T0/HYUgQApUYULKQYGeaPh600rjcXyXK95cOFX5WOVTbNwI/WHnf5390voryA2BmPOcHW7KPMtxK5pog2bAuLlNU+cOgTGmVpAn+JfZu+MFK2C6kEAThhNj9D9uPI27Y1UlU/0ehveE5fraotMSncKSYYAjYMipRzCZmUji+4x3DYVjIJ+ydPlDYcRMX60ub4LcwcY2FGSdrVH9QKCx07pFy4U7vjaq4qpbyYvpRlLfEPn+o4Adn2pmFv8cy42HhOI8fDJ1VyCfsnT5Q2HETF+tLK9PvE1dRJuNkjcIPlmG78DQIwkm6oNNr+aHEqy+9glpqcNg4G5RhqKmkRFT6/G9vsuGgUCUGe/JXdMKuGgYmZlI4+g/rtRo2PDYpXH4obFSccm7hfL9UW3Uof7KfQEXGFPg1//MZ8SMq+ZTRMSxU95f4hMu3Ibl/1rb/LlHAGEnyXIIaPOWUU8coiKTLQwA/uMC+CuNhsKHkwFjoMvCDAeOUgXy/TFlgJnsov2R3Z7LPbnMzPxi9gckWss3OkGzJqR9M/9p5tblE/GCWFPcQ5omUidGQ2nUqME2sRwb2/DjbP+jSffm/dZCIsVWj5WKcctbhEWfVI6oOQ2SUpr9UL4osCBZ6R6VbrIMcZlXcgHCWH0fOP1wEf7ZW+Brr5g5Dm3XkSAvpu6SFc/SMwcYkEA6hEMimiLZH/CLyTQhqMCqA4FYjgB+cUY9W5QdLDkFqxTCPM+JXP6O2x8UOsypOU2bsitPPKIwfzIDt+8GBm6syqeanDEccJ2FXCGStjLZH/CK2gB9khjDq6tJBKagwVOn4cCLQNKEK8PfW+pKC0Np0fM9LRJVQciKxcCQq7wlKN3oeCQOxTGWbAsykVz/kf9tjMQn9myvZIlRU/sdfmLJ85Ogay5czn5SFR918qva11EoWhuMzBLyRlR8kYxDO/wBymct34sRecETGnDlELBkB/OCMOqwqP1j4A8cPVthrhYQrLH/TovCDmyDCD24CaGA3gnAAR5mNvbtPjZP583g9V9TEoN5LaD9QPzmQSMSQOB7lMyeOIU0cyKvEkhJRhRSOU2N5gtiShYdjdDKmDL28zRUyUVElxpm8CbWOKeRTa4vdcT7/o3pY1xfKv3mujamqs9QQ30Y3lA0tNd/ezMSgV4Mnn7x3aBHsWFoC+MH5dx1+cA7M8YMlIY/nB6NHbJQsf+HJtqEfRBBOctT5X0IYOHTaJFmYasKGAnoEadsl8mkUzPx4MiO7jCgjcvISURmnVIlx+yc+tw4o4tI2DU9eXW5+rgy9QntGWD6xeYUVLTByRBsXaFVVVfvWza2zhsDMi6O0fZlffho97t8ya9E3LTO2JQ5nMgrm8PYkaTMJCqWgkmaSZSpic6kJ+EMo/CJ02OAHJ+jQiceuyc4iCi0MnZjfO7F5+aIWGzOijYs1rJLafevm1llD3McE432SpeSV00394BDDJmEsmJu2J5Ng2f0ggnCSA2W12T3W66PTT1GHTuoLrcB2Q/cYJoeK9ppA1G2HNn+YOX76tRfuKqNz8ipRdQytpl9hQciPKQU7ZhM1urqxhrYMvTLcNm3TaPNC9rHsDLnmEChp/xwsmUMVauwEHWG/2ey1mk2NNY9jx9vgbyxsaWHAQBGDWwO75rIR3zov0+PNUL9sLO9H8w2KvWDgmU8WqiOwNQgEP2jrYuznZ1/18YPlL5VmumPOI+fo6sYa1vCDma7U5mi8+fRLHeN/g+M2YSI/aJUsyg9uKguLCaROTufMabAfUBb8YF/VFBMkNkdAB82O5vodvdWBMx634bWZPyUKJ0Y69iQO9UPt6kkT/idUbj4wJM6YsKnacad9mUy2ObGDLChr9lFF4/hwPT1oT8Yvauem0AYLGGOryM4xspttA4PSeHkzqac3JlPgEm2q7RWSLGy4+4EP/O4Lk8WRrfGSx1knDwdvN7SIzVMUZM00JdaBSh1GvJBsIW0vsJuoGRBQL8d+MDlFc31fEz/ozcg3ffn9YFmXgR/M9/6Wj5mXHxwP5EJ8QRkvF7xV+fZksmwlP4ggLH8Y9FPqJ+fPvMMJqEZeSQ4fGfxQOENSoDM4oTDNSJ3PK8vKCJ5gWL8laagSHzlMiKaVFP8dp+rsnXuBf3HRUWweWhliUQEzDPrDZoYVTFp0HtqkJVm+OQAXyfKHhG/L+BdHEy8T99qgh8i6oexkoqtYhi7kExsnVqEVMifeFduWtzR2gWGUU5aQ0rt/bZ555jPjoghvJQI6eNLjJ7lUpxEDP6jf0bCf0ojeH8cPqvzwU7Miyw96+SF9DsPyiFbHuzKNinctNpyHNo09cwAukuUPCd+W8f1gwiDutYGDMvcjqJkf7P9G8YPheEYQBhRlA7tP2auksefzv73+qJH+LCTA4rMln2tYNf3sUYryY8eU2UdoxcicmQRHVz3aTcaDkTeu/DhYSCwpJCs8Z9LwOhc6As7EZo8os/xxvmntM/KF6ZEmL5L+vAdMSbyLfu86AvWtf+3Boyi7lHQge9mNCZfK2Hg1WIWcs4uJo9WwzOZgHtsqlIKBSLgM7GP04w278kURs9QEYj+oftaRgx+cpkPxg9PQm1HeET5r4hpHlLk8flCtLxzaEwcywg9muNltVONfHDW9G/uqqFCZNWSPG6GK9mWy5Dej4pPgVvWDCMJ8X28eo6NBx5U/tNzRM3A9xv/g9dvWSaTXhLr6snmhRSkKx47yo8aU2Yssmndc3k3OTiKGthVyC3vrEyh/JJSxeYGtzlQ9Zbsm0IRl+Lg0GWfhz4ST3EEOhdLkF0PYB8b3fZkCJtrMWjFQiEazbroWe9g4Ffs/ZQ7tCunDlKDba4Ojd/PtgarY2FIExveDEzY/Mz74UsqPElNmn9DoSrPhB0fgLH8kjCgk7Co8WsLemQYyVU/ZLvxgtrPwg1ki2W0EYZZI+e1wMqQs+u3py/7PXTmZWA0OsyQzavhk5ceOwuz5ugoLLJk3X9q4MYW1+0IW4hrHtX8+6efWHfNpTqgl064RB0PIkgmM6wvHWjDjfuxJhWEQyEimjD013xThDHNvcKZR5XWg6UM9vkaj4UgfXHMsmFeGQPgJKPGI353zg3Y0DLu0X6auOE3hEVt+rCjMHpfvw4UFlsybL23cmMLafSH4wQBzbt0RapxPYLBdmkkbu9oRv8fCsubgBydoRd7UGXkV/CCCMH+wbR6j+ercbzMfs3k5FaYYHDus4BG+pEy9+QLL5KoqTWHtw1o0vWs0XJy6VtR5sV4aXWRJ5vHBMOwYyFc0M1+Y++m7oydvwGJjSrpMaTydBKhJI8CO1oH63djSUOfnpQO77sTfItT+XuO00y5eLAdqnx2BQj+4aXU6MqrShPm64rHC7x1xYOez52PyBebTzC6msPZhLcIPzq4jJii5aj/olqWldgw7BtL9/b/4wT6L4SH8oGczev3dcH7be8/zvvpn/Q/SeURJQVODirF/aWDhhORLMv9mYVKmigo389bmC8+n8TFyjfG/YcnieI2bI/7FKecZHmHSLHZN3LTYmPKFxLlCeHT2cAyMTjbLvW6WI6rASZ9oezBouij6THsqnCkuKtkHJyhfw1euGLs+Ih/p/9mY5v5J7+lf291/2JIO7On9Os12U6/ukBBsNDr61bkhsNtsdJrdDX23LJ7PFiUwwg+qxd4hDjZ9s8N3MHUlW2HECIFKis0UEgqvPJCpSJv5KvJpfEzsBEv+EMM4XBgYVtGs4wuNmV3kxM2JTSpfSJwrhEdnD8fA6GSz3DuxHxwYBCa4gbBMo/CDZSjl0zBDmGeyeczTn/Bc/Tt4+MBV111924FblOFTV+77m31vsJx2amTSIlOK/wFnIue8KRvmXKOqK+mH8lcmCq3VSUbchHyaTAKfOG9Dvrq42Hw436H5NFsgpg7NHLRh6CEbur6wx31f+KIKBc/EnRXqtRJ0MOZ+UrmIiauaYUYZabN57qWEChciCg1RSn3sy00GWljQBVfN9//S+UC9YEfxpg/Vb1qF2muef9GDLDOfrUgAP1i+V/M+qDBv3jENDDhpnsygl0+TSeDz5W3IV5fWUPx3cHAuTrMFYuvQzEEb8IMzOazk1PCDMVkEYUxjvPCeU/ZeesmlPs+jH/rYlzz7RyURtXn19VfdduDWT115mcJ/+6k3aLDWSZIfizMj8qbDsZ136ZQrf9apuDp9Mu2awLTRJQRQec+numLnl08Q7w2GFVYXagnJCCyaQHIBoFCxeNtCjxd2tNLIs47IHjdw2B0UySpIV1ScPhOWASUrymSc7aYNIP2Pe/Z3P8ppPdurqBD2qRMdqB05HWjLTC29jWza67vAtd50oB5R0zLkttcNX/3aCW09AvjB0KeFbiXsLRMYXULwUGHQi8uMB8B8gnhvyFVYXaglJCOwaAL4wal7oO/0rCj8YCFQOwfwmqNwN5HTE5BKPHDowOXXfEEH4b4rLnvHvrf4wVrf/hDVtzu7GlpV4VCeSZ13AJkElW8W+pJMLYNXuTI7N98cfXo9wm+NJjZ6b6FZI+oqTF+ryDI9VZXBcwA1+qgY0bmjM3oC4X65GIif8+/2Wse77eOdlWPd1aPrK+u91c6GJsNaK6vdtdXOyWvr+rd75fje1eO7V46d0jp26srRva3jp6/2Tm6ttputtkSS1lumj91sa+Wl+3jdpWBrOLugrEx/SWVpai/3sXhNyHddAlNldnegNjud3oZ92zh/otfZ6HUV3LC07n9lSQsLlqgob0tqkUk7jVbOW9j0YNceS2qST0U4WajKbNN5WavIanf2fPuLPn72uRfljK1XhF+2hB+caa/gByfGO3rgSn+kBcWPGAyVevTeguLs8cJL/MEP+s4bfTj5NNvND3pvlbTdeWbF6LPd/KBz8b7pHgbfsyfgJxLdctNbb9p/0w23Xi+VqGqDqEsUoxOKip9g4J6gEaH2TfPql5KeQxak3VQEhmfcF2S2GfwRZSc5ho1oI9zVpgw3TVBoLZETEBjRTeVLG3YMhBKGdejojCMdYfN4d0WC8Ghn9dhGKghbrZWVAkG4p310b/vY3tax01YbNRSEko3h9Mj/3rzYEz3fO/IJpvQ0+KQ6UMGuxgiv+rTDFKA+JgKdJkx0oEqwnVoT4aTpT73SVtTX/IMgXEgH4QdHYMcPjoCzNXbhB9WPc7gwqiugcka6iJm/MJrxg3JboVMU8MJoW/lBlowuYGzRGhvVGpabKvzS59hy0/2HDlxxzeXa/NQVl11981Ufv/6j/ZM2nX6N/bGTtfIyLz6Bdr+EoTVqh8758p9CKTha/uULGZ3eu8l8Rf4sP+AKxYafd55D3F6lzydQZCZNKHZ7BgoReRRjgcp3kwoJPVWSrT8GRqg7b23esE0zljQgJMtXEXYNBIb+niyVfqrTfLzrypeQj3fqTbTTH3H2l9yfDDQ56PZqus9Un9d+1nl+pahklDXdTQZaPfrf/snHurDU49l3elreJGIg4AnU3w/KzlGDXvL7yPZn3j0pxWi/li1is/T4wTyxucWMOiRs0Cz7wQ/mSc3ID+YrUoxc4Ag/KA3pz0m2lR9EEBYeKguIlHfUvwvvdKHqfvRDH+MtyKjEd+57c3A2haOJP//zOtD/tEafrQ4b2lzeZGgblibDKBgW4gtdoCt07JPfcN5fWKa8Y752maFceUpBeBS2K4OrME1oYCZxiF/qwOgmD2ta+VzDoGV6KnTTsBp9vPo9HBuFKb1h+UrDAZPJPuw2wsLCFZkv2fxM9PE/wyhikUFvmtSdTmXdt/uROCHnz2T0QxJ5v4hU6JQuuS/S0lhuF6nfmz7WEH0p4CcDFXD/0vWivcapZ1y8yNZS9xISqJUfFL/wA/fH/6ZEw8ASUhb6LPxg4FPPQMnuzhhfPlc4rjIl4AczQGaxaX7NJv76ftA7MtUV+8HEx20nP+h9vBudZgGeMqsm4JfZXH3dVbceuO1mLTe97fp3XvbmUEneG2lXZogJiYcFhg1q+fhMyZnaM44wszdTe3wI2kFZ7pM5mw+Zhq23GZo+5CwKDBu7i9KWiqu8wFK1pony/ZjuWczfTWlsKg6HdWvcntG1ZErILBz19xBu9Jrr/SWjq+u9Fd1DqLcuaMno6kq3fw/h2vHd7WNhyejpa42Tmu4eQr2fwVbIJEZllsqYfxp+3HutpZzeb+XvIUxcl37sbqGm5JpEXX6pzEbP7iW0dTJK4Kby/Bygvn0JyqWLpgrLWdq3VWkBd39gWrsTgYqMJwO1qQL10cocxXtJ+QM//slzz7vIYuv9YclovfunwLqt6gfV1OAKh48HWSCZ4Svsxg8GFPkAfjDPJHMgbTc/aE7Neb1t6weZIcz/KGod45fZ3P+SBwQr/dNN3XJTPbfGXoDxrn1vCnsVcBdC+hFBmGXkXEiROXUO46aPD5tKb5MJIZsqSn5NUVQaDE4ujbC/hZEj4uO83lmGtoRdfkTLaFHtnXiNjdkzfL1uhlUwY3RgRIGjMy7XXhtbS5zUZGjkkcbHWKE49IdBxpllWKmWfMkhTaaEeJ5QYSVTWwY+mc1037AzsHT/DP8OsWjwV2bqzs3g2Y/VGmb6LR0irL96ulFQ37LTEviu8W1XL3ilp830nyvKbaocLwh9C5Xg3POfvhRqcIZdQtEzI4AfFNowuPrhK4aNH4xpLDaswRA/OLcumNIP6jdl/bVd/SCCcG4H6gwrGlxmE70AwyYSb715/82aSPyrfW/0FgTn4c+tw+m4j49Pvn36+DQ6Htp8Rl9IyCU/1C+/qfPFxGfZzywCEIej6DGCmRLm4Bq9cRlBEuhtanqMcdPENU9QvtU6GkZ/8lgyhWcShCMt0xGqxR94/kyosFKVnCmtMJmP9DowSqBDTGLJ/kWRSdDm9+z6yGatzeccN0Y1lKwkSiappmeNqi9EwPB6Cef6pmkvn7cfbVJsutdEootzuZLZP4WlH32MhaPJQLXD7U1Eo2o87UzWi47bu6SfisB284PRTzzhFsam4IUD0Kokoi8wM/xmBu1QaT5QfgTO561bTPlWa2wc/cljyRSeSYAfxA+OPqIm24sgnIxb3XNtegH101fuu/rmKz95/X9kBho30LuzRjsdNP/ihx43otnisXCtK2TUyOXdg08Za8I4HCNTuYUjZGYQjLMMht2y7igqU1rwi0qScY3eLyo+c2av5vhcmfRKGbJYLv0//JNxk3HC0k2LMw2EA/CB2NlsTG9tSbsyFeXbGCeI94aOyDD33Rd3WWyJSosLiXcp7I7uzKGUSWKbiUkDytDlig+7XL6RO3OpK43Qz9a/dqKjeTyTtVa6BJ/ibT7QH+Ah7H6bUnTapQTpv1GTgUpp6d1qVVOMLvCVD3l8pY2gMAiMTWBr+8H8UJYZvOIxJ+PXwgiZ9YN+dXjObwp9yKJwGH4LuyQzJsdp4vE8ji8fzre6fN5xU05vbckaMxXl2xgniPeGjsgwxw/myeMH80wyMQjCDJCtvJm5gOqbmj63xpabSiX+9b43aIiJRxz3ICY7VzZn0zRPoN+VhcKZsULuoyHJD0/ecyiLmzlJ5gkVmfFJab4J/po0TT9Zcaj4/k6XqIxf1ICbyRWW/w0zO3aQ3pgwOqe2jfqbGcFHJXX7YpewaeIlTZBpY3wcqkVhbxwfmMc8fZflO8gXEmcvA8qXFh1yIZMd4ZmPDptWo6t5t1CLmz7MpJrfpqTg8RNdr9bsIDfF5sSgTQbqd9Tr6L5BZ6sa6KWdjHPhUZOB+rH4FaROaiq9uykxEZC98+/yjHvd54HzayQ1QaA0gar8oCr0I1IYl7wJc/ODg/VW5Afl6AcdIX6w9JFVWcLBnh04HwtHnQLBxSiMHxxNHz84mo/fiyAsQ2krp8l4xx9+tr0AQw12z63RclM9t+YGLTfV0JOcZOs8Mj3R7tr5pNLqj50XD/qRPjR5FH8ZUvmCuFKGfHo/wGVGw35BxaHCkgaS+orMxNwn2KM9abOSRNlLp+kUYigjzhsiQyBTWogPgTCCh5iSgVj5lMwy52T5pgl+If/RhoUjIfZ8ylIY7yuN4SRHbO5AU/ZMgaPNSPdKR9k/baYN1JFlm3qnvAq0NkZXFXSst1v2jJY0e/9vkJEF+/qpKgjJBjnCDb2HPp3rc4e5frbJaZ81p9fQC5pCApN2aeL8nYG2SwtQ0/S2ajQkdppQRivNfe7/tRVYTxEQmBeBSfygBiL3A95efjCdQgw944fEsJkJ4AdjIOYj4u1y4UJ/p6yF8bP3g/IYY/hB2dlu4gfL9fSiUyEIF90D9atfrlFGlXlujc6B3WXJnvOINvL7yZD0dLmgbd49eBfiR0av1uKk4aw6jHfx3jgcUsaRw8K+ohHDcXBshT7MzqEHi1aycGY/uMe2Qmn5XYUxhZUWphyBtzB9iIzFUogcNzBZ7UKXoZepd7Rt4UjI9Hg+PpgXCkwPtoH6fcZMaRmTJttMjwqbJ3R92tT822RFjZtL2sx/0r+NjU7vyNFOp2Pyzys9JdBer+I0bWibNjEY6TptpI8J9Xv1bStCnbZ0YUsc5/XTjz6Bwne9+BkPeNCjlJIPBJaXAH6w0CVpONXPP/6kI14c1w/jB/ss/Ngbb+fCwW3l9lhE3t/5ZPn4mvhBeVj8YGFX1jMSQVjPfqmdVZkLqHq0qUzUXOJV1119mz235qbrb73+nfveYjMHuqXQXTh1swh2Kiz3EZadhKlC30J5i8zJcuxrKjxfz9SyKV+ZsWmWvGv0zSyTN2/AuI4zlFDotsPeOBCcRBxZk3DGtmF+Me/5vP35+FCgLyrgjXH5XPFh5g7I+BhM8ITs2o7D2owLTFIrUpdGFrtUtNFY7/TuWN84esKeBuyb5JWeV4Pue4zJQP2wvfCLM4YJQ+0K04lf8YCvPf/8iwIKAhDYMgRm4QfdbxM/mBwjmdG1/JFTOA4XZg+uoXDvYiMzti27H/QwNz2Vmilz/GB5vAjC8qxImSUg73jpJZeG2Jc+x5ab6gUYtx+8XU83/dQVl+nE+O2XvcWfdvuRrh1Ns7mh35+pav4hGTT0J+8SkkShpuGBSYeetPq05MnKCa3Y1OCgkH2FSj9Zjcqex5U2YpK/5d3qJKWXzhP7xUKnmNdyvux8vC8qFOJxxc1UllgTlraxIGH+QaPxiV4cLshcaZTm+Y5tbNx+qNPotnV0SbDpIy0XTwbaZsc0YfhYTDwZaCrQcvl/NnfoNpUlCUQ60MpvNO5+z2d+5UOYHgxECWx9AlX4Qf22zAOEwVwbIRwIRr/UEFccmNSb4AcTnrGDKEY8l1j84JSY8YNjAUQQjoWLxJsQiC+gPuahj1Hqlz7nR7xKvPyay7X5qSv2SSL6UnSW2T8Rdx7MO0U/CGqn/F9yfp9Wm/eR6Z5Rfysf3IOcUxNshihyo9oVZOEIm/Jpyjv7wmKDSYV7y0dORrh8+SFl+U4JTjGIulBIcnhEVxn8rnx8ppDQTG9GnF67ytum6nTk2j9bPT1lH4ZmTRuQPcc7nQNHuseO91bdHY+m7IKuM1FoGs9/FAgTfT6+cDLQkrm7B50UtDfch49KCqU94EFMDwYwBLYpgUn8YDR6+OEIPzjB0YMfjKHFfs3Hbw0/mPquuK0FYfxgAZSRUSImX14S78iS2AmB0gT8c2u03NS9JvGmG2694R373hxyx+f9OjS9dtLQ1neQOp2ddEatKocRrJ0gkFeDExRS2yzTEC6pxOIjJOZQKMkKIzMlxPWG9IlWdEdat9fq9JrHuyvHOyvHOqvHNtqd3qruzWutNFfbnR07uietnjhlbX3PyrHdqyd2t4/taR89rX10b+vEqWutk5prMnKlbY8k1eNn9HEBG3v1v5821LcP2O7cx+s0RftAfsS2+P7exnq3e+DI+rU3dXqdxmpjxecqmAz084HJpF/BZKBqDFpROjAsCvUG+kpjYy++5Jk//4t/EscsRdg/KzZPdSmMx8jlJYAfXN6+29TyresHG8e7q4v1g/JKcnj5ERs/uOlhOSKB/CCCcAQfds2VwOBE4mVfvPmqj1//0fhM3caA6PFWkoj2z8V6fWi7y5k8WGy5POOkmkCvWkOimcZxaptH2tEtGpdnSWc5utiMqAsUgqIbEeN3ZUoI1eVLyAnClU5vpdNptVak9Do7hwnC9olTV+ctCDd63cPHNq65cePw0V6721pttMx9uo/8pRd4igivi7BId63FHhnqFpf6ZGmCAR1oJTnxmZQ4+OdXfvtTy3j3IH5wsBvZWiSByfygLNaPGD84657DDw5eGF2MHywjCPGD4/4W5AdZMjouNNLPikB+mY1q8t7xCrfc9G8/9M5P3PQRPbRKg7KdyrurGSYRTRam84dOJWpnWBqhU9hClZgRA1W3Svdebf6JvYuMzOsQFaGz8zjZ5oWOmcJO8YsJjVHQWBZKdxXOkeZVYig2SLXYpkA405VeV8cw8zG+HJUQ51V1vqI4vQ8rpQJiZf8cstiSwnAwr3DvZJHOgFHVbzS6unXw+ps39h/SM041vjftUadOB7opPnc4mQK0q6vpv3g1qb1SYvSi0GGWf+tzfm8Z1eCw5hAPgYUQmMgPmqVugMIPTthpblQtPFMYo8DgsMrkWVo/6JzgQv2gHe1mxdAPfnAompE7EIQj8bBz0QRi73jzl65/977/fNxF1z7gtOaRxq7jvfbhjfah9danD612G822jRH2YEfvF92ps4V9C0Ig36BYOeT3Thwje/QZUa/2arGeS1XwFfyTLSYcmUwtLNRXBYUWR43hBYeaW1zysNiCGvPeMdaHwdEWosjoOl9rIB/6Nx+jlJm8QRNqV0gfmhHFFDQhJPMBSc3NE2XyTLEpiadrohvdzi37O4eO2GSfapfB6+F1EdKAuSfEqEIpQB1fXgduuii00EDVpcWiT/mW5xXuJRICEJiSAH4QP6hDqB5+0C7Bp8dzCKQRub/4wRyS+kYgCOvbN1iWIaCxZ727ckevvbN9/PRdG6urTb34e2Wl+Q29xsGj7cPrK9ceXNt/R2P1tK//rxv/Q3mlEu1BH27+0CYO0+Ki4SyNSv8G8ZBGDP1rZ9tDd/Z3xAXm6/WiUanzu1wRVsNmE002IsfayWUc+BotF73ECp5mIGe0sWljhzQhKsIFrUOycX4761rCQke/27cxtbPvkfxeTyme7guVeMPyHRFiRuT1hfhKvdkqTZtqQ78Zg4brQaP+n8+7MjAHGYyqOKALolKDUoH7j3Rvuq17xzE9PlT3DjZP2KFV8LoIGS+8WhTqJwPdd2KSNXP01dfU9kBAavDHfuI302j+QgACMySg8QY/6PlqsEpHX/uLHxzhy/CD3s053x0tisEPRmMVgjCCQbDeBDTk72ieONjZqWcoZiw9aXVjR3vjpMbRm9Y3zjz1wl97ye/5+/Wvvv4qe03i7TfdcNsN79r3RuWys/nk3+aKrlBguKrNE6WuKGPL8E1bcph8MjIviBPtziir0aIxFiZp2f2/XskUukmZElcUz7z5XP1SXEiN7Vsf7YsLiaKHBoeld0AzNQwAjvVhKzI3CibiubDXVG8MWfZ5S0KkesRnVPPjMn1LCs32ikhWKr2+pQMt3Oy29K/RW2l2Nby2ZywIOz09OKZzwom+E93Gjbd2Dh7uKkq2yOYNyUHTfvbtAlqtO/mi0BTFQOdKDb7gRT+7e8/egVg2IACB2RDQIIMfzKDFD8Y+a7QsDC7PM8QPDntiduYYy2x67x8it4YfRBCGDiWwBAQ0eK33Sh20WmOj9tz/ng8IrfrhZ9trEg8cOnD5tV+QuPn0lfuuvvnKT1z/ESXwY2IshEyfhJy5gNuVUS+5RCMjgszzqeKqTa5GnyAdi8fxKGVS1MDkWVJUKCRKnhVI2uXnEoNrGbDKARmwzJUlqzy9qGQLupQBobYU9rnjQD+TKs2UY3kGUITSbHYrfCQOw8dbPswd+vIHyxyQhSFjaLhZlR4IqjOErcak3sQUPyuowqUq23r3X6O72uiuNZttf7trZGSwtmQg43gyuTYanWM9CcLeRqdx7bWdG2/qdjdkgLVSqtDfKyhc/uKovidbFKpK82aow7wX5NbBTKewCYGZEtBQhh/0hJNRPYd70JskozR+UJzwg/jB3M8liSh1bj0sM/EQmCcBnfrv2XHsmB7939NkoN6vVnCSOtoefyfGBXe6UMke/dDH+sTpc2u+oM1PXbnvXfvepBEznMAPaIC0dEUG1ZTGDfzNqI6BfUUbmdLcyXdiQkY6+txm1YBYsmjv7TJV+9E/FBK5SfORGQcZiyvbm7hRK9xziLBYZPpRurAnjbO/Pr/fFYe1y0UmasvCbneuEJ/JirIEoWmOv4uNjNREqDdSO8QzNC0zYWiJ8vVE84c+Y5gt9Jvi5gvXtywxu5xtVpQtTDbbWi2bFWxrYrDZXW12bIbQ7myd4We90Vnvddd7veO95k239W7d31tXyExtaKnq+vDJQBnvzN/ENnccZtPYbKP7SA3++E/+FnODWUBsQ2CWBPCDga6NybXxg3IBkXsNNirgB0zvdeKwdrnIxHFZ2O32KaMSkhHXYhTED0ZoLIgfzACZeBNBODE6Ms6bgIZNvd7tju7eWzrd04ZU3il1ojuQOb5fXyrxJW4iUSn0mkRbbrpfr0m8/p3RaxJTYWBjuQ/7UVpZgq7LO4YwiA/UPbgRsmvIb2bFWpLU19hPGZUgDZOvd2C+UPJp0H1KPPgC0ozmefpSyhInCTLi0Md6P+X0Vd4XOjyReQqmUiIpMwKmnSHS8rgyA16LsY+vz4fThgiIz6nHYQcsQdkqJhZ4LqsJR30yPSICcUyAoJTapU1NnyqjsvpvV4SFVbu0qPKmy0Q7UoM7mht6+eCOGT/G+USvc7zX1Y2Cxzea11+/cfj2bqun6UED1es2s5OBznIze7OP5xOncjEJfb/3Hvd65o+hBmNGhCEwFwL6AeIHa+gH1fnyAqknDYfCgF/zsekAG3YFx2VlhJy24bZ8Y/vx+ME+CwsV+kGdDukcAT84iGqTLQThJoDYXSsCa82NldaJ27q7ItkyYODxXiceLQf2ld7wy00vveTSkOMlz7HlpvsPHfAvwNh3xWXv3PeWqKJkEM8O3CG/AlFqHx3kR+wNkl1RasmMuJiCgqLd3UQcRVEuGAxzolFRqdcxjePCzf6UmleJzmlpl9UuOdR2AswnjsRhaHhfX3mPqEaFSpPqUjeYNa5oO5/Wqb3UbFmVSlnLnYrD0K5gYVCG4YGufupPmTIKsMgKi/PNUVv0zzShAqraxKGMMQ2pKtysYG+lJSm4sbO1vqOpf52dLe0JBrviB7eG1VgyXg+SESV1j6y68cbubTd3JAUVo7a3pAbTw8anKVNmnnmYDFR2v9d/P/N5v/8tT+OZomWgkgYC1RPAD2pAGoG1jB9Udj+2Wzn4wRE03S78oCe0tf0ggnCz3wH7a0PgrLPOWWt19vdWD9tzZe4otOtYr3HDdZcX7poycnAi8TEvfc6PqkCpRE0k3ppOJL5j31uG19JXA16T+BE2Fk4ZiagMhU5viOizmtvFObTHSkrO+FND9BRWy2P7LOC0oovx3tGCiUpMJaKirAxvp8sVv1HQZQh70/k61wRfrNWU+fi7FjOR8WZ4Ik5ivNvnxKFPZSYFcRjb5qpshGnDQmWYdEGqJ1VFIdvUFxooEVEtFkgsUTv1qr+G1OCaRGBzY0drY6cJwt4Ot6o5bku1Ybv6aYY0rr9m4wuf3XBBCVTds6ietC7MwS6oP6Yadg/TgUp8j3s/8/FPeNajvvZxITEBCEBgngTwg5524Vjtd5Xyg0rqnESFftB5Iyu074nwg/hBf1AuwzeCcBl6CRtTAnZrVqN7vLui24ILPyd69i6KuX2kEuOJRKlEP5F4+8HbpRI/dcU+WfL2y7IqUU/7CJ/g1YIeCxLR667Q0NxUoZURNGQosDBZsnfQ//mUTu0k2sH/iR1kYlUiERN9mAgklzojDoNH9IpJKV2dJrWctakoDeZuFsgoRq8PvQE+q1vomdhvtYeZw1TmebcflKE2fSF+Ds3McllcQ6yd+gSqoTRvhnZaQ2yG0Fdu1amFKy27b1BS0NRgQ2qwuyN9nEySbtgfR8fVOCyFxRfKtpYa0miu39H4/Gc2/CGvwkycBuuHF1lYYNCBYa8C9s8ZoMBzv/33H/u4J+/hgaLDwbIHAnMggB/MQM6PefjBBBF+MHOsRJvB00VxcnmJd0//Oie4PfwggjA+EgjXnYDUoJbndYY8yl+vYnNqapGtiCcSH/PQx8iUlz7nR7xKvPwam7rcd/m+t1/2Vm+i3JgUhVcdEn6pV3MaR1N2Xqik4sOWKyqNNrUm0EVKijhBkoxfsQvsSxZfU/StB2BaVaZt3MdpEi/YsldMnQE+mVeJUiDKo8SW3q0yTXWsNcTbr12ST658i0lqcX/0ZTuGKBxnjdvr7Ui347bk9aGrwBWq9OlfX0pSwKBHVGS6oNSUoZnkSHjaVobrlCSvmWo3EOqjgP1zC2gV8An0IBlh13smtKTWnizqniWzpueL6qkuM/7IkhMneu/59+PqFE9Zc4Mt91DTYTWnndHf72JUkn3CXgXsn4vx4Uvu88wX/cArLrjgIp+SbwhAYIEE8IM2aOMH00PQHJmN4qn/S//a/mR0185+yOfDD3oOBsnYJHxc2PZ436fYJNBrbHk/OPOzFuPKBwIVETjeaWkqRndQxOV5jaGYlWZLS/XiXTUJZ1TijzzXJKJsS5abutckxhOJXm9sSFrZyF6gDxNx6IYuPUfEJdIDYywQy0Lbdo888QF9p54iTykMg+6pJO7ONJ/aT/J5QegN85kVk9eHLoGKSkWUE4eq10+qyYsnUso1TOmSybpUX1nKYGNqdD7GlWe7Y32YOkWLV/Eulyz1cs8ih3lEv5pU07Y+aVCGypFIQWuODiz3T0WaMrTUko6qtK3pwZak4Ia7k9BuI2zbXYU2aReOTKs9+qhHo63xgsrp+R873vvwRzckSHVmpMhWsli0oLR+16Y7/emD3/J7w7e1zwtC+b97P/MrH/J1D3nYo5CCKTn+QmDxBPCDfqze5n4w+Lb5+kHzreYv8IOLHwkqtgBBWDFQipspgfVe+3ivfXIr+2L6UOmO2T7nP9QzbUASUUUMLjdNJhLDc2u+ePNVH7/+o6EmL05M1khHuZWKWafYkwqR4DKx4SfrtKmnlcpbeN2VUSGt/vNEQyX9gOmL5GODv+eayEJXhWkkJ+S0Vx+vD714kw2yVnslPNo+7GSeLPHSzxSHa4K5ldQ879XUKF9IUnnxnwHd7zkEpxgpQ9dis1b1BO9pCNNSLUFYTRqUYbrX/prNrqUdPaHV7HaPllEXCI8XhNKE9jJ6+xZtRYtVBnVc4JRho9Ns3HGs+76PHDt8KFGe+TKtz3KfUTpQTbO+sH/3vPczH/yQr7vHPe/7Ffd/YK4MIiAAgQUTwA/a4Lvt/WDGUc7LD7acm1isHzRXhh+sfBhCEFaOlAJnSOBEd2WXHuTYXB9Wh5+zGra35vHxROKj3XJTGeyXm15xjV6T2PzUlZfpNYkmbbyucu3p2rvQE7ljgksjpS03tX2muZx480rAOwxJFydXsooho2HcylJXiH0ln1QlWh2SPQX60GxLxKG3U+O2F4cqIgm7acNghLdDu/zcpgSYK2PAnKytzpyQQnlTA63OoAwVmYpDl8BQqCSPwXLYhhOHKkH7pAwVGXb7YpXGi0A1ViU7GawYm5drt8L0oGat7W0Tq017FaFN1vXltFU05ccTsUJkilOD7/3I0UMHZbRrV1S66/xo2wWDDgx7fUDf9s9KtcsGl9znWejALDu2IVA/AvhB/GA4KoMPWGI/6NxyaNGwQOQHEzWIHxzGarJ4BOFk3Mi1GALHuqs6Dz/JBKG9mD7/0XsIl2SOMG97cUxGJYbXJF593VV6bs2nr9ynsfSv9r1RmVMBYw5iw82/uSmt7LSh5IHTPKYdldK7E80W2ka66YLJV/A32vYq0ae0xInsMX0R68NEKLoKFNYumaNq9Z3MHLpqLdxfjalCzCCtt7TiUl3nGyUbQqWJWbkYb6dP79OopAFxaGVYKqVJlZT0UNI+25eKw34Vbq/Sq5xEFnZbUuDOQFObK63eqp4v2tpwr6HXnYQdX1xSaCjIBQojB5Nkt9Z7evV8x0Gzd82LzKFjGx/4yIkjh5t+VarP4IgN5PUMM3sVaf8U695OoYB04L3u8yytC30o60IH+LEBgfoSwA/iB8PRaeN59PFeZkZ+UNdM8YMR7K0WRBButR7dwu3RwHdwfWeju3JGS2fIBR+d5W80mnoh+Nb++OWm97/kAWrmox/6WH177+hek6iJxIZU4l/ve4NXZdq0gPkHE2R+U49CUSCeNuw02pI3igyiJSwoDTHa6z8hJqRJ4p0+lOIwcai6rMqkRlODLpGMScShSUHZoHgXSJWhi7N4k5C2PtOyKY0K8iX4upzgCYaYlov3+jRRaywiLA1VOL3TMrFKMb46S2cfVacCrXxfrsxWizZ6LS0c7dqji7RktNdu6+7BjgSh3dfaWl9t6E2A3RUnMZ3QjMxzhY711dEb53sbx+2ZurLcDG41OoePNj7ykfUTh+122cIGh8nAYLkPKLH42Lf7pzcYXnLvZ33DNz7rrne/54U8KmasjiExBBZKQD9k/KB6AD/oD0N5yXA8ztoPmrPGDwbcWy6AINxyXbp1G6RhT+flu1eOn9waKvpO9BonbV0CI1o2OJH42B9+tr0AQ+n9ROLN+2+68bYbpBIV47SOuZCOEzxyIRYppeD8imkd97EbEtNP3+GYlCpW40qrXV1Nm7nSnBJUWa7wnDh09VlvZsWhe6Snk4gmFKUWdbekBZz6sWk6hVOr4r+xUwzxIaVvo4u3pqiNplfdx8tgbfZb63ekrlUlC1Sn2+qYGtT0oM9oc4z2tgl7nIzJQr35ypShg6OizGCPwJc25DsxItrbaUgHdqQG9WJBHczHe029SUVCVML42LHGJz96rHe4p0eYOh79bHkd6BPoWxDsGx3Yp0UIAktMQIMGfnBY/+EH82Qq9IMbboFM5X4wbzN+MM9kDjEIwjlAporKCJzaPnRKeMV4Uakneu30YcpFu7dTXHwB1bfbq0SbSLzWTSRese+LN1+574YP294gBBVyHz8r5cPhW6k0lxg2FfCCyscEeaPzlaI05phsltLq0gNYpMFMhXp3NUwc6hUjJgKVxeYMLYMKMWVocss2vBRUmrjGTNinsdwywCq0kmS54lOB56Y0DYMlUKTToKY/FZZtUoPr0oQd+2etsMfJ2KsmVvSeCdOEWjWqxaJqk5mnB5Z29X4UCeTNPjLIy0a9MWW9tyERuN7r6RG6WhKt+cwT7u5QSc2djc76eve/PrbePaSbFfuD9igdGElBPSTm/Avu8aiveTwPidmsQ9gPgSUggB8s30lb3A/KgbjVMdX6QflVnWfl/aBul1jvNiv3g94Jqk/xg+UP7Fmk7J9bzKJ0yoRAtQRObh87uTUgSOLydbfVscZa/Nr3eC9hERi4gPoQW26qT/LcmlQl/s2nbCJRHz+F6MNDv03SmAqKP+mCTCf94h1p2PSYD5vasv+DOHRhU2Ius3Z1u7oqKQelZE4Hmp5TwASPvp1WNCFn6k5FWkZTiUlN7m+iFZ3YcymcwU6MWR4rxCW3Ai2zpZcZLl5zg8kimQ2pQflCPUa012iv9LRk1M8QmiZMrLCiZOpGr7cuudhT0lGasNvT4lPpwK7mA6UmddxqSnC90V53U4Iyxr1hpaO1oRuHuu//93VdNF3T/ZhmbdK8fjPdBKDZnupAHhLjupQvCGxBAvjBKTt1a/hBt3zGXR6t2g/Ko+qKsD7z8YOdptbF4AenPKgryI4grAAiRcyNwMmtE3vt/rdiTXikt3Git1Ov7+MzFoGMdxy23FS+IRTrdVeyaRsF0BXpM4Trf0qfU4/9MrVLKb2YNDGWPq1Uy0ZVQ3LDoVvAaV7KCjIRqF2WMvWIMmTANrfX26l4VRZMkuJUxsxHBSomlGnTg6rR3Tq40Wnb3YNaFNvu6aKEXj+of5onVKF2q71Ka+j9mK1129k92u12Wus7FbbFqJJ2UoayTGm6eu6RzN1odNdNP9pdr3oCUKepham2KnXdvdFekSpEu4/3Vq696sSNn9F7VmylqPKb6EvNVsCH9e0nxr0O1ENi7slLIzJdyyYEtgoB/OAsenIp/aCupcqPVe8HBdg5NedbB/1gq1o/qHUxKl/rYvCDsziqxyoTQTgWLhIvkoBOeY90dxzVokWdlbtPemKcWHXM3sBgJ+h8piQg16gS/HNrfFH95abXXK6YT11x2V/ue4vf5TtDikeBMCM2IBFdP3kpGERYmEVUIfku8zrHpZH3sVXCKsMt5DR9Zf7PzcUpRiJKskr/e6elsFNJiTNz+SzsTVWxbiWMleXr9RUle10yJ65SXygh6mcInRpM3vDhXj8oN6wWyQwrxzmz9e7KCenWXmujsbGmp8t0u8c0T+iKbtsrs6yZLrktx9Hs64aua9jEYFsyUvdmKGCS2/7Tv97u1vGjxxtfvr4jNag4LUCVYf6At4ALJwHz2Tws1Pch3xDY4gTwg3Pr4Dr5wZ7ObOQatp4flOMzh2h+Ez84t0O7uCIEYTEXYmtI4LTTz7z66BmNE50LO9dd3NiRsVADpb+70AuPzF42pycwcAH1oY95yXN+VGVquelV112tF2DcdPtNN9x6w1sue5sipWe8/HH6UMLJKpfGUh+F3tEu026m8szLubAly30sczw5qUWTThKqPMuve0Zt8tArQwW8MrQZRRN1UokqWrFWg/soJqlQ0cmFBdvhEyRCy1WglNrUglXVIlkoh+xL0LeK0MEmsaeAMjpn1j5ubyfsWbjXkjLUmtJVq97mPJXM/eeMcbJZKlfJVMBGs72hx7K6F3JoU63TRQ2pzfXjvc//x9EDB3qr9iaOZAIwMc8LQqcDeVho6BQCENgOBPCDi+3lBflBa/SW9YNeDeIHF3tkN9zypEXbQP0QKEVA5+W3H9vVO7Jz1yntE2s3XbRy7Mz2Tp9TZ8h6NqMfLqOT/FLFkmgaAvKOl15yaSjhR577IwcOHzxwcP9tB2+/9cBtl12+76qbrv7QdR/3es+rQRNIXgdKAJmms3lFv0uPaXG77NuX6XWaNoOiU3wi3lwKZZQws93SZU4H6r3wpuXs0aC6purnEhWT5nIB7VUWV5T/tmS26fe6XV4KmsKUGnQrZ1KjzGwVqPq0Vw+bOdbU7YQ6+mxBqW4C1LNG9YZ6pXHqzsrXHKeteTVzrBLpP1vnIwfftGWiEpImLG32Urcd2pNLj+7f+Mj7j2reT2pQGcx4M8x99xqX3PuZiuGlEUaWDwS2GQGNA/jBuvU5fnAaP6i78nWXBH5w4Ue1Pwdypz8LtwUDILAZgX99/3sbq83PXLPvwNEvnLlrY+Xw2+901tN1Xn5io3HO3otXdp1/xln3ufc9HrhZMeyfN4EDhw7uP7T/C1+6QhVfdvmn/uzj75JS0+gjHeTXXlrYxShBEmlzZdJIlsCtBE6GKWk5pUn0mylAC5tsc20yTWhhi/GqTzf4OX1ob4xwssq+XYxJRJe3rxV9yS5vUkWqBmWCxcjI1kpvba27Y3Vj52pnx0pn18r6rpXOWntjZ6uzo72hV9Wv6IWEzuwVU34y1y161VNVTRBaKyzSTRYqWUcbvcZa225PVMPa6yf233jsxn13KFLPpAk6kIfEuO6dyVci08WaDwSWgQB+cBl6qcBG/KA8X4Ef1Dt2ze3jBwuOmblF2emJKus/o2BuNVMRBCCwvQnINQrAVddefYstN735+ltu+H+f+CsvBRUfhKKH5DYtsm1zaPZAF6kozaf5vYn2s6WbJvAUq2k3aULd8qd5O4WVU2GvBr0s9IIwRAYtYNmTqUKnHp2G1HSmpfdLRrVSVO+jX+uurUgTdlfa3Z0r62vt3lpbmtD+aW5QRurRo7Zk1JaCmtBtewXoRlzJjraeSWM3u+rVhXpwqF5fYW+tkEzsnNj43w/d3jmoJ5TaOlVZhQ70XTzTb/zgTPFSOAQgMIwAfhA/OOzYmHM8gnDOwKkOAhAYRSBzAfXKm67+wLWfCCoxmcExwebuyjPp5VZmOpXoXwyoXapAatBenqsJQHuGp+YG3d3qepine7O8mz9Uuv70oBeEQQ2qBBOZbkVo8t2/h7DZWumurPZWVrqra10putV2V/N7qyt6IaEEnr2LIhWEjVU9Y1QWSkO29E8V2kpR2SxBuNbUu+w7SmBvrdCjR090Dn/5jusvO+BfOs9DYkYdJVXvQxBWTZTyIACByQlsIT/o7qXHD05+LMwvJ4JwfqypCQIQmIyAbkrcf3D/5dfYctN9l+97wyffEZej10BoGtBrRT19xb0HIlmMKmEm7SgNZqLO3V643m1rFtHmD/2bJGwKMZlX9GU6ZeiEYqoJlSCdIVQS92gYqb5VibruyorNVa60e6sr0oGaHrSwDNDbIaQSTRa6GUIZoHjJQilGmy20vZKOiSDsHu8cuPHowU/dvOoWiOohMQ9+6Nc95KGPuvDCi+JmEp4dAQTh7NhSMgQgUAmB5fKD9lqmBn6wkp6fUyEIwjmBphoIQKBCAgcOH1Bp//Bvf//mv/vFtZNWG2srX+quuVWgdouePQxGr3NwTweVhJO8k3NSerdAXjfmJWH/mBmJw3W9UsItCvWvE+zoJfGWyT1Z1M0i+tsIJQttUtEpTD0MVLN8miRstezbNKFXhlrL2nJzgFKDmjxcaWjTVroqmSYDpSTtLfa23FSzggofvvn4gc/f2jqwfsl9nqWHxNzlbve8CB1Y4YFSrigEYTlOpIIABGpEYCw/qEUvckmyfq5+UG7RFtHgB2t02AwzRX6Q104Mg0M8BCBQUwJ73WsSzz75zLMPXXf6Snv37uY37G332ju67dWbju7csdq45o4dHz+wFt9P2Om21zt6pKfdDWh3TqcPsPEt1HpO3dS3ancp9jS1p8eduYeI2l2IelSpZv30rTzKa1klGDUXafIwca6uImXp2v2Kdq+h1eEWrPqZS6tE5Wuf1KBmDleb3SPrOy88fOl9T7v7fb/3QQ980EP37NnrLeEbAhCAAAQgsCmBJfCDdueFPCV+cNPOrEUCBGEtugEjIACBiQlIvOmza+XEzh3rZ+y8Y9fO5v3bzW/sNQ4da+tVJNcdXvufwyvn7nnsGWdd/LbL3qpbCvVP84f6dnODur3QKT1bbmp6zr7tn23qZRESgVKD9u0fS+M1YUtPSrOH1Zj207etTbWZQEWaXHQtcfc3Wkhl2n2Dze5OrVc90b1o9yMeeP4j737n+3zF/R/oEvIFAQhAAAIQmIpALf2gOVPvEPGDU/XuXDIjCOeCmUogAIGZEdgh9Vb02bWyIUF41z0bB9bX73vGRc96zo++5Dk/evDwgf2HDlxxzRc01bfvisv0msSPXvex9Z7mD+0F8RsdW1XjtZ9bYGPrPCX+JAhXeiYOTfPphRD2lghbNSoxqEWhumlQ0UofnJ/MkT60mJYydvQImXuf+VUPOP+R9zjnXve9NzqwqLeIgwAEIACBSQngByclR76EAIKQQwECEFhiAnqyy6YfPUsmSEa9QVj/LrzThcr16Ic+xueVSjxwaP8V116u9S37rtj3xk+83cSh3SFoT5np6A4ITfzZY2bcJKCitEfrRm2KsNu2B8koYDOKenuEwvYwm2ZvR2tjtdF78LkPeeDdHvmAix9557Mv2tROEkAAAhCAAATGJbAgP6hrovYMbfzguP1Vz/QlTqbqaThWQQAC25uAJNrKiuk0yTabkBv+SVdxDk3hVeIFdzLNJpX40uf8iCSiwlddd/Ut+/WaxFuuu+WG1/3XXynGLQ2V/9OUoOYDm3pYqH+CqHbYU0btDRPdB57z4Idd/FWXXHDvU3efdsE5pjz5QAACEIAABConsFA/aJc+8YOV9+miCkQQLoo89UIAAtUQ0AVK9/rBRBNqOi9T7ord4DfeRxJRGS695NKQ7cef9xL/eqjLv3SFl5837b/Z7hh0T5c5+9Szzth7hhTgheeiAAMzAhCAAAQgMA8C+MF5UN7SdSAIt3T30jgIbBcCupdvRFPH1YPFRe3dvUf/LjoPyVfMh1gIQAACEFgcAfzg4tgvf82jzqGWv3W0AAIQ2CYERq0a1QsiqlGE24QlzYQABCAAgeUjgB9cvj6rj8UIwvr0BZZAAALTEJAvLPj0urrFsHfaGecX7CMKAhCAAAQgsHUI4Ae3Tl/OuSUIwjkDpzoIQKAaAjfd/OWyBRW7yLK5SQcBCEAAAhCoIQH8YA07ZUlNQhAuacdhNgQgkBDQY6/dU0YHloXa+yLSTzcN8BcCEIAABCCw9QjgB7den865RQjCOQOnOghAoGICtiY0+Vdc8kYkDotTEAsBCEAAAhBYWgL4waXturoYjiCsS09gBwQgMBYB/7L51RW99kGfgenBTDknOvb+XD4QgAAEIACBrUQAP7iVenOxbUEQLpY/tUMAAtMScIJQX4kmzD9RlAnCaRGTHwIQgAAEakwAP1jjzlkO0xCEy9FPWAkBCAwj4PXeiPcQcg/hMHTEQwACEIDAFiCAH9wCnbjYJiAIF8uf2iEAgWkJdPRmiaJPeK7MSqNz2ulnFyUhDgIQgAAEILD0BPCDS9+Fi24AgnDRPUD9EIDAdATW7dLo0HsIuz0Thl1/+XS6isgNAQhAAAIQqCEB/GANO2W5TEIQLld/YS0EIJAl0GlqhlCCr1jzHe4el1wMs4XZzGxDAAIQgAAElpwAfnDJO3Dx5iMIF98HWAABCExA4IYbrlOuTqPbKpoedPOCVuqhXqPd2CheVDpBrWSBAAQgAAEI1IMAfrAe/bAVrEAQboVepA0Q2IYENCG40ejs72w0/YsnihAozYmejXLFs4dFWYiDAAQgAAEILAUB/OBSdNNSGIkgXIpuwkgIQKCAwJFu93BvtefkXk/rQnNJOr3OiV670ynYlUtLBAQgAAEIQGDJCOAHl6zD6mougrCuPYNdEIDASAJaBXqwt+Nw46QjPf/eedOD8UsIJRRv6q53eq0RU4gja2AnBCAAAQhAoL4E8IP17ZtlswxBuGw9hr0QgIAjoBcPSuwd6e68pXPyjZ11PU0086TRA90Tx3orim2yYpRjBgIQgAAEthwB/OCW69KFNWhlYTVTMQQgAIEpCOhqVrvR2Wg0blrfc+z4KUd2HTmpe2Jve/2MTkO7jnQad3RbnV5zpdFt89aJKTiTFQIQgAAE6kkAP1jPfllGqxCEy9hr2AwBCDSe+tRn33LbFz67/x8P9XYd67a/ePz0k1fWT+qduHm9t9bo7GicWGustxo93T54wblPv8uF9wQZBCAAAQhAYCsRwA9upd5cbFtsjVV4PvtiTaF2CEAAAuMSOHz4wFXXXXX74Vv333GT7hW8/eD1a2u9lVbjzFPOW+k1zjvnPnol/cV3feC4xZJ++xBouhtP8YPbp8dpKQS2GAH84Bbr0Pk3R34QQTh/7NQIAQhAAAJ1IYAfrEtPYAcEIAABCCyCgPwgD5VZBHjqhAAEIAABCEAAAhCAAAQgUAMCCMIadAImQAACEIAABCAAAQhAAAIQWAQBBOEiqFMnBCAAAQhAAAIQgAAEIACBGhBAENagEzABAhCAAAQgAAEIQAACEIDAIgggCBdBnTohAAEIQAACEIAABCAAAQjUgACCsAadgAkQgAAEIAABCEAAAhCAAAQWQQBBuAjq1AkBCEAAAhCAAAQgAAEIQKAGBBCENegETIAABCAAAQhAAAIQgAAEILAIAgjCRVCnTghAAAIQgAAEIAABCEAAAjUggCCsQSdgAgQgAAEIQAACEIAABCAAgUUQQBAugjp1QgACEIAABCAAAQhAAAIQqAEBBGENOgETIAABCEAAAhCAAAQgAAEILIIAgnAR1KkTAhCAAAQgAAEIQAACEIBADQggCGvQCZgAAQhAAAIQgAAEIAABCEBgEQQQhIugTp0QgAAEIAABCEAAAhCAAARqQABBWINOwAQIQAACEIAABCAAAQhAAAKLIIAgXAR16oQABCAAAQhAAAIQgAAEIFADAgjCGnQCJkAAAhCAAAQgAAEIQAACEFgEAQThIqhTJwQgAAEIQAACEIAABCAAgRoQQBDWoBMwAQIQgAAEIAABCEAAAhCAwCIIIAgXQZ06IQABCEAAAhCAAAQgAAEI1IAAgrAGnYAJEIAABCAAAQhAAAIQgAAEFkEAQbgI6tQJAQhAAAIQgAAEIAABCECgBgQQhDXoBEyAAAQgAAEIQAACEIAABCCwCAIIwkVQp04IQAACEIAABCAAAQhAAAI1IIAgrEEnYAIEIAABCEAAAhCAAAQgAIFFEEAQLoI6dUIAAhCAAAQgAAEIQAACEKgBAQRhDToBEyAAAQhAAAIQgAAEIAABCCyCAIJwEdSpEwIQgAAEIAABCEAAAhCAQA0IIAhr0AmYAAEIQAACEIAABCAAAQhAYBEEEISLoE6dEIAABCAAAQhAAAIQgAAEakAAQViDTsAECEAAAhCAAAQgAAEIQAACiyCAIFwEdeqEAAQgAAEIQAACEIAABCBQAwIIwhp0AiZAAAIQgAAEIAABCEAAAhBYBAEE4SKoUycEIAABCEAAAhCAAAQgAIEaEEAQ1qATMAECEIAABCAAAQhAAAIQgMAiCCAIF0GdOiEAAQhAAAIQgAAEIAABCNSAAIKwBp2ACRCAAAQgAAEIQAACEIAABBZBAEG4COrUCQEIQAACEIAABCAAAQhAoAYEEIQ16ARMgAAEIAABCEAAAhCAAAQgsAgCCMJFUKdOCEAAAhCAAAQgAAEIQAACNSCAIKxBJ2ACBCAAAQhAAAIQgAAEIACBRRBAEC6COnVCAAIQgAAEIAABCEAAAhCoAQEEYQ06ARMgAAEIQAACEIAABCAAAQgsggCCcBHUqRMCEIAABCAAAQhAAAIQgEANCCAIa9AJmAABCEAAAhCAAAQgAAEIQGARBBCEi6BOnRCAAAQgAAEIQAACEIAABGpAAEFYg07ABAhAAAIQgAAEIAABCEAAAosggCBcBHXqhAAEIAABCEAAAhCAAAQgUAMCCMIadAImQAACEIAABCAAAQhAAAIQWAQBBOEiqFMnBCAAAQhAAAIQgAAEIACBGhBAENagEzABAhCAAAQgAAEIQAACEIDAIgggCBdBnTohAAEIQAACEIAABCAAAQjUgACCsAadgAkQgAAEIAABCEAAAhCAAAQWQQBBuAjq1AkBCEAAAhCAAAQgAAEIQKAGBBCENegETIAABCAAAQhAAAIQgAAEILAIAgjCRVCnTghAAAIQgAAEIAABCEAAAjUggCCsQSdgAgQgAAEIQAACEIAABCAAgUUQQBAugjp1QgACEIAABCAAAQhAAAIQqAEBBGENOgETIAABCEAAAhCAAAQgAAEILIIAgnAR1KkTAhCAAAQgAAEIQAACEIBADQggCGvQCZgAAQhAAAIQgAAEIAABCEBgEQQQhIugTp0QgAAEIAABCEAAAhCAAARqQABBWINOwAQIQAACEIAABCAAAQhAAAKLIIAgXAR16oQABCAAAQhAAAIQgAAEIFADAgjCGnQCJkAAAhCAAAQgAAEIQAACEFgEAQThIqhTJwQgAAEIQAACEIAABCAAgRoQQBDWoBMwAQIQgAAEIAABCEAAAhCAwCIIIAgXQZ06IQABCEAAAhCAAAQgAAEI1IAAgrAGnYAJEIAABCAAAQhAAAIQgAAEFkEAQbgI6tQJAQhAAAIQgAAEIAABCECgBgQQhDXoBEyAAAQgAAEIQAACEIAABCCwCAIIwkVQp04IQAACEIAABCAAAQhAAAI1IIAgrEEnYAIEIAABCEAAAhCAAAQgAIFFEEAQLoI6dUIAAhCAAAQgAAEIQAACEKgBAQRhDToBEyAAAQhAAAIQgAAEIAABCCyCAIJwEdSpEwIQgAAEIAABCEAAAhCAQA0IIAhr0AmYAAEIQAACEIAABCAAAQhAYBEEEISLoE6dEIAABCAAAQhAAAIQgAAEakAAQViDTsAECEAAAhCAAAQgAAEIQAACiyCAIFwEdeqEAAQgAAEIQAACEIAABCBQAwIIwhp0AiZAAAIQgAAEIAABCEAAAhBYBAEE4SKoUycEIAABCEAAAhCAAAQgAIEaEEAQ1qATMAECEIAABCAAAQhAAAIQgMAiCCAIF0GdOiEAAQhAAAIQgAAEIAABCNSAAIKwBp2ACRCAAAQgAAEIQAACEIAABBZBAEG4COrUCQEIQAACEIAABCAAAQhAoAYEEIQ16ARMgAAEIAABCEAAAhCAAAQgsAgCCMJFUKdOCEAAAhCAAAQgAAEIQAACNSCAIKxBJ2ACBCAAAQhAAAIQgAAEIACBRRBAEC6COnVCAAIQgAAEIAABCEAAAhCoAQEEYQ06ARMgAAEIQAACEIAABCAAAQgsggCCcBHUqRMCEIAABCAAAQhAAAIQgEANCCAIa9AJmAABCEAAAhCAAAQgAAEIQGARBBCEi6BOnRCAAAQgAAEIQAACEIAABGpAAEFYg07ABAhAAAIQgAAEIAABCEAAAosggCBcBHXqhAAEIAABCEAAAhCAAAQgUAMCCMIadAImQAACEIAABCAAAQhAAAIQWAQBBOEiqFMnBCAAAQhAAAIQgAAEIACBGhBAENagEzABAhCAAAQgAAEIQAACEIDAIgggCBdBnTohAAEIQAACEIAABCAAAQjUgACCsAadgAkQgAAEIAABCEAAAhCAAAQWQQBBuAjq1AkBCEAAAhCAAAQgAAEIQKAGBBCENegETIAABCAAAQhAAAIQgAAEILAIAgjCRVCnTghAAAIQgAAEIAABCEAAAjUggCCsQSdgAgQgAAEIQAACEIAABCAAgUUQQBAugjp1QgACEIAABCAAAQhAAAIQqAEBBGENOgETIAABCEAAAhCAAAQgAAEILIIAgnAR1KkTAhCAAAQgAAEIQAACEIBADQggCGvQCZgAAQhAAAIQgAAEIAABCEBgEQQQhIugTp0QgAAEIAABCEAAAhCAAARqQABBWINOwAQIQAACEIAABCAAAQhAAAKLIIAgXAR16oQABCAAAQhAAAIQgAAEIFADAgjCGnQCJkAAAhCAAAQgAAEIQAACEFgEAQThIqhTJwQgAAEIQAACEIAABCAAgRoQQBDWoBMwAQIQgAAEIAABCEAAAhCAwCIIIAgXQZ06IQABCEAAAhCAAAQgAAEI1IAAgrAGnYAJEIAABCAAAQhAAAIQgAAEFkEAQbgI6tQJAQhAAAIQgAAEIAABCECgBgQQhDXoBEyAAAQgAAEIQAACEIAABCCwCAIIwkVQp04IQGBeBK644opms7lv3755VUg9EIAABCAAAQhAYJkIIAiXqbewFQIQgAAEIAABCEAAAhCAQIUEEIQVwqQoCEAAAhCAAAQgAAEIQAACy0QAQbhMvYWtEIAABCAAAQhAAAIQgAAEKiSAIKwQJkVBYFsQeMYznvGWt7zl2muvfdGLXvSABzxg7969j3jEI37hF37h2LFjZdpfJvtd73rX3/qt33rVq171lV/5lR/4wAdUbK/X++M//uPHPOYxZ5xxxjnnnPPkJz/5Pe95T766t771rU984hPPPffcCy+8UBUVpsnnIgYCEIAABCCwEAJlfOJCDKPSbUWgqdbqTGtbtZnGQgAC0xCQWnvUox713ve+V2rtqU99quTZpz71qVe/+tV79uz513/91/PPP3904WWyK83u3btvu+02ac7v+77vU8lPeMITPve5z/3QD/3Qgx70oDvuuEO1/9mf/dmP/diPvfKVr/TVdbtdudV3v/vd3//93/81X/M16+vr//Zv/6Y0z3/+81/3utdddtlll1566WjD2Ls9CeiZQ2o4fnB79j6thsDCCZTxiQs3EgO2NgH5QRzh1u5iWgeB6gnIe1199dWvec1rfvAHfzCUvn//fk3fnXnmmf/0T//kB5awKxMok11ppPr++7//W/OByv6Sl7zkH/7hHz74wQ+effbZoTRtPv7xj3/9618vHajIX/mVX/mlX/qlD33oQ7Hw+8hHPvLoRz9aU5cIwsCNQIYAfjADhE0IQGCeBMr4xHnaQ13bkACCcBt2Ok2GwLQE5L3ufve7azIwU9DHP/7xBz/4wf/5n/+p78yueLNMdqX55m/+Zi0ZVUbN9WmG8E1vetO3fMu3xOUoLKGo6j784Q9reue00057+ctf/rKXvSyT5pd/+ZcVjyDMYGEzEEAQBhQEIACB+RMo4xPnbxU1bisC8oPcQ7itepzGQqAaAl//9V+fL0grSKXKJAvzuzIxZbKHpaeaJ9QU39Oe9jS7gjX40TrV//mf/1Hhl19++YEDBx73uMdlKtLmN3zDN+QjiYEABCAAAQjUhEAZn1gTUzFjqxJY2aoNo10QgMDsCAy74ardbnc6nU3rLZO91UouV5111lkq8A1veMPFF188rGRfoG4jzCcYVlc+JTEQgAAEIACB+RMY5qdKutT5G0yNW48AM4Rbr09pEQRmTqDw6Z379u275ZZbHvKQh2xa/VjZzzvvPD019Mtf/vLDcp9PfOITN9xwg6q7xz3uceqpp/7zP/9zvmrd05iPJAYCEIAABCBQEwJj+cSa2IwZW4wAgnCLdSjNgcA8CMh76SUQcU2HDh363u/9Xj1XZvQNhD7LuNn1Touf//mf//SnPx3X+La3vU1PtfHzh1pJ+lM/9VO/+Iu/KIkYp/nkJz+pJ83EMYQhAAEIQAACtSIwrk+slfEYszUIsGR0a/QjrYDAXAl827d92//5P/9HM3JPecpTNH0nqfa7v/u7u3bteuc73ylttqkp42aX1NSTYx760IdKAX7t136tVte8613v0iLS3/md33nkIx/pq9PjZPTkmIc//OEvfvGL9dqJ1dXV973vfXoUqvL+3u/93qYmkQACEIAABCCwEALj+sSFGEmlW5sAM4Rbu39pHQRmQuCBD3zgRz/6UT1C5td//def/vSnv+Md75Du0tsIw5NgRtc6QXa9UfCNb3zjZz7zme/5nu/RmwlvuukmvWHih3/4h0NFuufwzW9+89vf/vYvfOELevroC17wAgUUo5lDqcSTTjoppCQAAQhAAAIQqA+BCXxifYzHkq1BgPcQbo1+pBUQmB8BPSNbM3V6KfxkVU6ZfbJKyQWBYQT8nPawhzoMy0U8BCAAgUoI4BMrwUgh0xCQH2SGcBqA5O0T0HyRjif/0UxOf0cupOd/pAn7f3V5LCTUC+6e9KQn3elOd7rwwguf9axnaa1g2FXzwFiW/8d//Mc3fuM36k3raqYm2TT3FbdOb37XaszTTz/9bne728/8zM8cP3483luedpxrbuF+v+ZChTaUaY5efH/OOefkVei//Mu/6NXzeqLMBRdc8MxnPvOKK64orGI+kWMdAN6kYe0aUdSmv6D5NLZkLWU6NxQ1+rDXtPB3fMd3aGTQ5/nPf76eMxQyEoBAHQiUP9o3/RWPGAHq0NIRNoxl+Rb2g0I0FooyB88wf4Ef9Oca8ZnkiEN0/rvKdG6walF+EEEYuoDAVATue9/7fsR9dCPZ6IK0qM+nDN9f9VVfpStkPtcf/uEfPuEJT5AQ0j1pv/3bv72ysqLnlGhF4ugy67B3LMt1/9ujHvUo3X33p3/6p7/5m7+p9yXoJX5//dd/7Rsi+adX6h09elTS+hWveMX/+3//76UvfWncxvK041xzC6vvtIDz2c9+9utf/3q1VDptx44dChw+fLjQhjLN+dmf/VkdDLpxMS5BB4ZeM6jHkL7uda/T0XLbbbfd//7313sL4zRzC491AASrCts1uqjRv6BQck0CZTrXmzr6sD9x4oSGAr128g/+4A90U+hnP/tZvblrfX29Js3EDAiIQPmjffSvePQIUGfUY1m+tf3gWChKHjyF/gI/6E8m4zPJuv1Gyo8MC/aDWirDBwJVETjllFM0ypcv7ZprrtGJvi6kKcvtt9++d+9eDXlx9p/8yZ/U/I8eYhlH1i08luWa6NDdd3pyZtwK3Q6nGTCVo0iN7yeffLLetO4TvOUtb9HLiKQP4/Q+PC7tfAkTxMg8zWcOy7gpihHZhzVHL7uXwnzrW98aVyqFcOc73/mFL3xhiJSulmZ47GMfG2LmFti01YWWFLZr3KLiX1BhLTWJHNa5wbzRh71+BfpRSPP79Ndff73GDT1bKGSfOOBPHSbOTkYI5AlserRnssS/4nFHgExRC9wcy/Jl94OjOY+FIuMThx08hf4CP+g7Iv4Fje6axe4d1rnBqsX6wUawgwAEpiew6eGeqeKnf/qn733ve/vIv/3bv9VJXhBCPvLWW2/VGZt2ZTLWanMsy9/0pjdpHlXXgeImHDx4UKrvb/7mbxT5d3/3d3rLQtirdyeIwJVXXhliQmBc2iHj7AJjociYUdgcyTy9gFDTQZnEWmskLJdffnkcr6fI7Ny5c2NjI46cQ3iCVg9r17hFxb+gObR04ioKOzcubfRhr1/E/e53vzj9Xe5yl1/7tV+LYyYLIwgn40auEQQ2PdozeeNf8bgjQKaoBW6OZTl+cFhPFR48w/wFftBjjH9Bw8DWIb6wc2PDFugHee2EPxngezEEjh07ptfZaa7MV69rPLqnbs+ePbE1Wj6qacOa3y80luVa03j3u999bW0tbubu3bsvuugi7Xryk5+sewv1CXu1lFQzijr9DTF1DoyFokxDtBxUklhL8HV/oO6o1K0CPpdeLPHUpz5Vtw7GhUgN6qDSfLJmleP4WYcnaPWwdo1VVOYXNOtmzrT80Ye9fh0S/zfeeKMWWsuM66677tprr9U6nJmaROEQmAOBzK94rBFgDuaVr2Isy/GD5cEq5TB/gR8UnMwvaCywdUu8QD/IPYR1Oxi2lz26Rqi7gL7927/dN1uzZNJFeQQShMNuP8snXkjMWJbrnjedzmbs7HQ6OtnVA2ZCvMa4l7/85d/8zd/8yle+UrcRBiEUEtQzMBaKTZtwyy236L0RarteQnjxxRf7m0t1OU0ZddelVgxmdLUuUet+1DmrQRkzbqtHtGusojK/oE151j/BsMNez5fSr0M3GGvlsBbVyGs++MEP1ksp698iLITAaAKZX/FYI8Dokue8dyzL8YPle2eEv8APCmPmF1QebG1TLsQPIghrezxsC8Ne/epXf9d3fZfm0ENrC2WP7h8LCWobKG+5XpuuZbF6qEDcFkk+LSINr1nXLi0R0a16V111lcJ6xV+cuObh8ig2bYgeK6qbJ7UyUM/d0m1jmkyWSNbqwcKM733ve/W6wp/7uZ8r3DvryLFaPbpd5YvK/4Jm3cxZlz/ssNebJPVoJf0Q9LCiZzzjGZ/+9Kd/4Ad+IB46Zm0Y5UNgRgTyv+LyI8CMTJq42PKW4wfLQx7tLzLl4AczQJZxcyF+kCWjy3iobBGb3//+9+v0LqOLtkjbRjZD69xe/OIX6wXrekKGFj1qblB3vulROoqJZwh1BuyfO/rOd77zOc95jlaN/tAP/dDIgrfazv/93//9i7/4iz//8z9/3vOe59smAppD1oNkvvu7vzuzhlY33OvtHd/pPjUHMbpd5Y3fkr+gYYf9a17zmp/4iZ/QqbMeWitEenKVfi+akdBPqTwxUkKgbgS25K+4DGT8YBlKSjPaX+AHt+QvaCF+EEFY8idJsoSAzsD0+KyAQ0v4Chd5hgQjAjrD0zsD9DqmOI2ui8SbPlwYmU82t5g8BFVdaGRhpBLrxQxnnnmmLvu96EUv0j0AipHe+9Vf/dXCJjztaU9TMr1WoYaCcHoUhU32ke973/u0WjioQR+pBcaaGvrQhz4UO0KpQT11Ro8Y/ZM/+ZMRBc50V2FfF0aObpeMLMyVjyz8Bc20jfMsPD7s77jjDl0x0fxw+Anot6Njz19G0UtN5mkYdUEgP+7hB3UyoAMjP0wNi1Q8frDMT2m0v8AP4ger8oNLsBKvzA+GNHMjoBNuDUDho4WOk1Wtu891A1g4vfOF6HEyeu9qvkB53/nfFZY3I8TkIYxruR4oqnWPWjj6uc99Tte3dFOc3kYoTagqtEz0ta99bajLB77u675Oayb1gOlM/MI3p0cxoglf+tKX9GKJTAKtH9Z7yeObMD/xiU/otY1Sg/79HJn089kc6wAY3a6SRRX+gubT2FnUMvqw189Eg8BTnvKUuGpt6uKUnjQTRxKGwBwI5Me9ySot/BWXHAEmq7HCXHkI41qOHyzTHaP9RSgBPxhQLG9gsX4QQbi8R85iLNeF+fgJuS95yUsms+P3f//39fAPzRDG2c8//3zdPB3PQGqv1lVKOOkG9DjlYsN5CJNZLnd4ySWXvOxlL9MLVZ///Of7Rmn01zsJ9ZzMuI16yKreUph5gEqcYFHhqlAU2n+ve91LTxbVrZXxXt1S+MUvflGvKvGRegCp5gYf/ehH61kjfq41Tjy38FgHwOh2lSyq8Bc0t/ZWXtHow95PPmieMK73yJEj2tRMexxJGAJzIJAf9yartPBXXHIEmKzGCnPlIUxmOX5wdKeM9hc+L35wNMNl2bt4Pxif3xOGwJQE8m9ZkbbRe+d1MhdKVviMM87QcpEQ4wN6lKiy66khcbxm0nTdUbviyLqFN7U8D8E3QQ9B0asX9YSM0CKlFIRXvOIVIUYTg5deeqmWSoaYEMjTDrsWFZgYhQzONEePXdXMcOZ40KbeTaKrBkovLyipoFsH9azaRbXX1ztWq0e3a9OiVOOwX9BiIYyuPdO5Shz/KDY97PXaCS0Vjqv4ju/4Di04j2MmC/tzhcnykgsChQRGH+0+y7BfcZkRoLDShUduann8k4+txQ/GNDIHz2h/oYz4wZhezcOZzpW18Y9i4X6QF9PX/PhZMvPyh/tll12mU674BeJ/+qd/qmRaHZpvmx6gonkwPRlEC0r1+b7v+z49texVr3pVPmXdYkZbnocg+zX5edZZZ2Wusyr+H//xH/U+vRe84AXvfve73/a2tz384Q/X9KC8Qr7Jedr5NPOPmQCFNzLfnDe+8Y0SzLqNUM/XEQ0dGNrUg4iU/rOf/awuK0gq6828/zX40X0sNW/1iHbJ8tEAlWDEL2j+DS9ZY75zMz+K0Yf9e97zHs0AayJd7+3VkfDc5z5XR4IiS9Y+IhmCcAQcdk1GYNOjXcWO+BVvOgJMZtUcco22PPOT9/bgBzP9kj94RvgL/GDhmWQGaX02852b+VEs1g8iCOtzqGwFSzY93NXI+9///no24LDW6iRPDxLUbYpaDPb4xz9eDmZYyrrFj7A885v3ln//93+/7pHTNaF8Q/793/9d71gTTD13VDMhWjKaT6OYPO3CZPOPHBeFt7CwOf/5n//5Td/0TbpvUKr4SU96kjZ94j/8wz/0p/L5b73DZ/5NVo1jtXpYu7zlI4pSgtG/oIW0fdNK852b/1GMPuz37dv35Cc/WS+m12WUJz7xiXqS0KaVlkmAICxDiTRjEShztI/+FY8eAcYyZs6JR1ie/8nLNvxgpoPyB48SDPMX+MEMvZpv5js3/6NYlB9syhcKX/6MihgIQAACEIDAlifg35yGH9zyHU0DIQABCECgkID8IA+VKSRDJAQgAAEIQAACEIAABCAAga1PAEG49fuYFkIAAhCAAAQgAAEIQAACECgkgCAsxEIkBCAAAQhAAAIQgAAEIACBrU8AQbj1+5gWQgACEIAABCAAAQhAAAIQKCSwUhhLJARmSuBxrWdY+c1mQ3ex6su+m42WPeLIHvDQctcp9K0NS9NPacnsn+VK9/oY9+3LVDlKk5Tf7CXZo0hXkdL0ktJUlK+iaY9XUqS/TuL2uuyNJKXttQQWmab0WZJa9IgmS2D/XBaXOMnlIn0u2xslC7lCQFWkuXwgSZ8ksOr7JYSUFp0WGwIu0iV2oVB1KMGlHCzfFRIlCHuVth9Web60UJcFetkE6V5LrHC66drVM1ahzDistK78NEGSMol0mfph0fKJrXALu/4PkdYVcWRST5TSdbUzJD0ozCgf9q10YRdpx6k7BEKCpKtbdlz0I22z2XOHkgsoV6MXIpuNru21oz7JpZQ+u2LaPtLSK7KrGEtp7NNcinR1ufiuK1kJLKXCPrsSuIxurw9bCRbZVjkue1ubSa5u29eVpnTZrRafQOYp4LJ322aqs0p1pVYp0kyy7A0X6S2xSNe0hitfGbVXkWq4dUvb9ZXC+km33BDgwipDaZqtZqt17hest/lAYMsRwA9qjLARwQ1G6t5kMwQ0RCrWJ3CBwQTakWZRCSGlRQ/kcuVbnMvudluNLk0owWXJRrq02UgrqZ89td/FujK9awu5Yvv7YSWPEls4lBmH8YNG2nwKftAdYbP9kl/mAwEIQAACEIAABCAAAQhAAALbkQCCcDv2Om2GAAQgAAEIQAACEIAABCAgAghCDgMIQAACEIAABCAAAQhAAALblACCcJt2PM2GAAQgAAEIQAACEIAABCCAIOQYgAAEIAABCEAAAhCAAAQgsE0JIAi3acfTbAhAAAIQgAAEIAABCEAAAghCjgEIQAACEIAABCAAAQhAAALblACCcJt2PM2GAAQgAAEIQAACEIAABCCAIOQYgAAEIAABCEAAAhCAAAQgsE0JIAi3acfTbAhAAAIQgAAEIAABCEAAAghCjgEIQAACEIAABCAAAQhAAALblACCcJt2PM2GAAQgAAEIQAACEIAABCCAIOQYgAAEIAABCEAAAhCAAAQgsE0JIAi3acfTbAhAAAIQgAAEIAABCEAAAghCjgEIQAACEIAABCAAAQhAAALblACCcJt2/OKb3Wx6G5ppQJsW9tFRpKLdP9sdmR2H0+iQ3ZWVxMa5QtgFkiKKSsrUVZAy1DVYVJoyKjQK9osdiEwanf5xhocELpBspXgCEksaUsbhwciQvZ8kLiqUESKt3LTgODKEB8t3yZMMcV1RKUmS/h9L59KGMl1ENnsSmaQczO62kgwhnG6Hv2nAUqThcCBY3NDIdEeaM/R5iIizF0VaCeEASStKIvvtdhE+e1ylNzfNZftdWSG1bbtY/XGRYcvHpt9JtNvsh9Pq0/JdCf0C04L11+9JS0sq7RdkViUG2L7+jhDZz5pkdjlCbGqBVdUPq5x+USEtAQhsKQLpAR8d+eH3ZL+HqLHpLyIbGSXxQSUI+ULiEFCaEHaBJG3IEpcXUlpkuhGnVJzfHCwqLTNKGgVDSX07rfh0ayBlao2LTPbEKUPiEHBFJdkGI0P2fpK4KDPBfUJkspWLDAkGy/e5fSmhqDgyCcd/LJ1LG8p0EdnsSWSSsl9Aki6Y7vZYZLoj/E0DliINpz3q4oZGpjvSnKHPQ4QVmUsVRdq+cICkKZPIfrtdhC8zLsybm+ay/a6skNq2Xaz+uMiw5WPT7yTabfbDafVp+a6EfoFpwfrr96SlJZX2CzKrEgNsX39HiOxnTTK7HCE2tcCq6odVTr+okHZGAaup1+vNqHSKhQAEIAABCNSZgPe++ME69xG2QQACEIDA7AjIDzJDODu8lAwBCEAAAhCAAAQgAAEIQKDWBBCEte4ejIMABCAAAQhAAAIQgAAEIDA7AgjC2bGlZAhAAAIQgAAEIAABCEAAArUmgCCsdfdgHAQgAAEIQAACEIAABCAAgdkRQBDOji0lQwACEIAABCAAAQhAAAIQqDUBBGGtuwfjIAABCEAAAhCAAAQgAAEIzI4AgnB2bCkZAhCAAAQgAAEIQAACEIBArQkgCGvdPRgHAQhAAAIQgAAEIAABCEBgdgQQhLNjS8kQgAAEIAABCEAAAhCAAARqTQBBWOvuwTgIQAACEIAABCAAAQhAAAKzI4AgnB1bSoYABCAAAQhAAAIQgAAEIFBrAgjCWncPxkEAAhCAAAQgAAEIQAACEJgdAQTh7NhSMgQgAAEIQAACEIAABCAAgVoTQBDWunswDgIQgAAEIAABCEAAAhCAwOwIIAhnx5aSIQABCEAAAhCAAAQgAAEI1JoAgrDW3YNxEIAABCAAAQhAAAIQgAAEZkcAQTg7tpQMAQhAAAIQgAAEIAABCECg1gQQhLXuHoyDAAQgAAEIQAACEIAABCAwOwIIwtmxpWQIQAACEIAABCAAAQhAAAK1JoAgrHX3YBwEIAABCEAAAhCAAAQgAIHZEUAQzo4tJUMAAhCAAAQgAAEIQAACEKg1AQRhrbsH4yAAAQhAAAIQgAAEIAABCMyOAIJwdmwpGQIQgAAEIAABCEAAAhCAQK0JIAhr3T0YBwEIQAACEIAABCAAAQhAYHYEEISzY0vJEIAABCAAAQhAAAIQgAAEak0AQVjr7sE4CEAAAhCAAAQgAAEIQAACsyOAIJwdW0qGAAQgAAEIQAACEIAABCBQawIIwlp3D8ZBAAIQgAAEIAABCEAAAhCYHQEE4ezYUjIEIAABCEAAAhCAAAQgAAEIQAACEIAABCAAAQhAAAIQgAAEIAABCEAAAhCAAAQgAAEIQAACEKgJgabs6PV6NbEGMyAAAQhAAALzJNBs4gfnyZu6IAABCECgXgTkB7mHsF5dgjUQgAAEIAABCEAAAhCAAATmRgBBODfUVAQBCEAAAhCAAAQgAAEIQKBeBBCE9eoPrIEABCAAAQhAAAIQgAAEIDA3AllB+PznP18LSf3npJNOuuiii775m7/5X/7lX2Zh0BVXXKGK9u3bp8Lvete7/tZv/dZYtTzDfUKWCUoIeccNPOUpT3n605+eyXXVVVepOX/6p3+aif/jP/7jU089tdPpZOILN+NWxOF84j/7sz974AMfeMopp3zFV3zF7/3e73W7XZ9mdK58ObOLGWahasz03exsoGQIQAACExP4m7/5m2/6pm8699xzd+3adckllzzrWc/6t3/7t4lLm13Gb/mWb/mRH/mRfPkXXHBB4s6jP9ddd10+5eiYhzzkIbGDPu200/7yL/9ydJbCvU960pN+4Ad+ILPrh3/4h3WmkYn8u7/7OzE/duxYJr5wM9gTAplkJ06c+Pmf//l73OMee/bs+Zqv+ZoPfehDPsGw9Jnss9scZphqzDCfnQ2UDAEIQEAEsoJQURouP/KRj3z4wx+WL/zVX/1VycJv/MZv/Kd/+id4BQKPe9zj3vve92Y03nve8x4l8N8hpQKS0495zGPa7XYcOWVYvvn7v//7dRLwjne8Q98/+qM/+gu/8AtTlllt9vpbWG17KQ0CENhKBG655RbJP30uvvji3/md35GEePWrX33nO9/5qU996jOf+cw77rhjKRr7gQ984PLLL//85z8va9/ylrcorI/07aKMf8ITnvDP//zPmdr/9V//9Utf+pIMi+PlN7/2a792586dceTE4R/6oR96/etfr/MZ1X7ve9/7sY997Gc/+9mJS6swY20Nq7CNFAUBCCwFgZW8laeffvrDHvawEP+c5zznuc99riTHTAdQDdYXXnhhqHSCwPQllK/08Y9/vMbxj3/84w996ENDLknBvXv3Sijqqa26GuvjNXGnmF/6pV8KyaYPqHzJv//7f//vT/7kT6q0b/iGb1hZWfnlX/7ln/mZn1lbW5u+/OlLqL+F07eREiAAga1KQOO21oAcOXJEC1juec97hmZKz/zgD/7gE5/4RPnEv/qrvwrxtQ3c5S53kW3+2uX5559/97vffbGmCuBLXvKSK6+88m53u5u3RNOVn/vc56S0JQulvYN5Um7f+73fGzanCRw6dOhP/uRPVL6uzKqchz/84erW1772tX/wB38wTbHT562tYdM3jRIgAIGlI1AwQ5hvg1yjLjHKO+Z3VRXzP//zP7oiO01p05dQvnadIki+xitpJYEk/LRo56abbvrMZz4TipJovO222zSjGGKmD+hi6sGDB7/u674uFKWzE80TquoQs9hA/S1cLB9qhwAE6kzgd3/3dy+77LJ3vvOdsRr0BkvJvOtd79JSzOuvv97HaNnhW9/61pe+9KX3u9/9fIzWAb7yla+U8NACxUsvvVQS6MCBA6G9hcsUdQ3xfe97n9Jo79vf/nbNZSmj7jWQhvnrv/7rkFeBjY2NX//1X3/kIx+p64+6a0D1yh3ECcqEMzaPMEkLF2Xbf/3Xf/34j/+4Ai972ct8+RKZP/dzP6cbFpRXRopJmXq17FarQ2PXqQupmrLTTQQSbKGEa665RipR6tHHrK+v64rnV33VV4nnAx7wgBe/+MXeq/70T/90yDIiIH8kB61aQprf/M3f1OJVvzm6IUePHn3FK17x4Ac/WHdnSE4LQuZEZVPbMqiDDQoMM6yQeb6c0bZpza2arINEs9y33367r7cwMjaJMAQgsK0JaKwMn+c973laEhM2fcAvg7z22mu1qSuOGkzlLx/0oAe9//3vV4wupv7RH/3Rox/9aE0tnn322brjQiN7pgRtar2K5rLOOeccudJv/dZvVRq/RER+NxQb55KD0ZB93nnnaRz86q/+6j//8z/XwO0TaHSOO0wmFZagLFJiZ555pq6MqihdcYzL9w3RjRBal+LdjFydRtg4zYjwC17wAkmykOBTn/qUTBIiCUWtLwrxmhvUTX1hU4G/+Iu/0ATjWWedpZRPe9rT/vEf/9ELSElu7fVW+fRxOC5BRmoB6h/+4R/GkSHsc6mWRz3qUb5dP/ZjP6YFTiGBAiW7rIyp8ZEQqhhh4Yi+i4va1EItadb0rJz0Ix7xiE984hO+6sLIYBUBCEAAAnkC3puEeDkajc+/9mu/FmJGByTbdOatkU23iyultJ+0nMTD6173uo997GNvetObtEs+SNNivhyl11L/TJmy4d///d8Vqb2STPKn8oDysFIgGu3lYX36w4cPSy3Ih2r8/+hHPyqvKv8l1SqZJGWYKTNsSkOq/A9+8IMhJmPzCJOkzeSppXV/6qd+SgHJIRWi9LJBLkaaWTpWirfVav3+7/++L//d7363lnqGujKBF77whXJ8IVKPLdCk69/+7d/K0QcXL3Qi5tNI7upkQ5ua0FOT3/a2t8nzys3phkCZ5NME+0MglK+A/JE8hc5b4siQcURDbr75ZtUi4BL8urarswX1iyYzxcFnL2lbfHjENgwzbBjzuJzRtulqhW720cIunWBIhIu56i2MjO0hDAEIbFsC3g824vYXCkJdCNRg7ZNpINZFQY2Jv/iLv6gpKd3zLYckHahNXXzSpU3d2yZn8PKXvzwUq1FeDkCrGeU2NKTKh33f932fNr/ne75HFuQFocSAprxWV1d1r7kKfMMb3iD1pTKlVDWAqthPf/rTustRQ7M+Cnzyk59UpBdCvlJdtJMu1S3pP/ETPyG3+uY3v/k7v/M7/dVNn8Cn1/AqNyM/8fd///e/8Ru/oVbIrcrj+jQyTOaFoT9k9AF5CDUhJJYIlOfQLlWkqkNiua4XvehFflNeWU/oUS61SycEKkGs1Mxv+7ZvU0XlBaFK0/gusfeqV71KXiHU5QPiIAcg4a2ryOoRXWmWUNcJilyXT1Cmy0qaGh8JGTOGWTis7+KiNrVQ1zvVKIl8HR66G0SdqEvyhZEZq9iEAAQgkCGg4VefEOlvjviP//iPEDM6IBGimToNQT6Zhr773Oc+Wg0YcmmX5tA03+VjCkWLDAiCUJe6jh8/HrJrckzjuUZFxUg7SWpqrAt75V7ldJR9XEEY2zzaJNUlTRsLKqWXSPMmeUv0yBY5UG/2aEEo36dpK7kYn/FOd7qTVKXck2580Dykj9Sklk4PfFgPoVGTNSXoN/WtJusMQU0uKQiVRU84k67WCYOEtNe0vrTRDdHaqK//+q/X6USoWicnz372szX362NK2hajDkX5wDDDtDfPPC5ntG1az6wzDX9M6vRGF6ZVYGFkxh42IQCB7UlAI6p94sbHglCjidb3v+Y1r5FokSb0yaQ3NO6HIVXaRkLoy1/+clyIbmSXGNOVPB8pf3byySd74ReSyd36+8XzglDaTBe3wrTP/9/efwDMklz12fiE933v3dXu3k3SSiu0CwoorxIoAJ8lgYQJIkgYCZs/2SbZyAkDJtgGY4NN+IyRiTbJYGzAGIT5YyRAYMBglBEI0CrtStoc7t27u/e+YWa+36nqrqnpND0zPaFnnt6781ZXV5069VR1nT5d1d0+i95qk/EzNc2oLQiMHcJ/8S/+hfwlrTkJRxWQHVJl5SP5SKXXbWDdMwtptH5D5eptKD6m2iEUAd0T/Y3f+A2fWE6gvwmnacnLL7/cmxAtstWgLGvn02hQFgfvvoZC9a4CpZFiMzmEcox180/SZOQ076o1RUGg6iXv/YMf/GCI0dtc1RzBnNdpspqqxj0hFOcDFRoqQb7tYlFTNfSvJfB33N/znvfofoHasTAyoxW7EIAABDIEzApGdlDzctqNna5M+syunIqwKkRrMTQma74rk8a/TNsPWdXel47q7TVxdi08kT7vfve75YDJa9ItzviowpqTlIs1q0MYdJaEapWUIO+caEFHrEZQUpG6T6onOOKjcVhghUirORSp1TEyo/fee6/CWhGq25cKyN+75ppr/PVDWZVlf4WivkMosVrFI0S6e6jrGTly3iNVxcsqcvbsWd1EzthrydFFkYrWnc36usWoJSGzFSqmNHnmQc5U3Xx/U0eSBxuKK4wMRwlAAAK7TEBWpuAZQt1G0jioTY6KZgK1gF6rVnT/T6n9pjtkGq8VltujRTJaWqOr+fSg/dUKTz0OrsFLYfFVAmXXJFWcRvfYJDmO8WGllzeiQ7oZFh/VMks5Nl/+5V8eRxaGNQLKqdM7V57ylKfECXRHTXOMXisfr5fF6fZkSKOlI3oST48C+hg9K6LlFnGCkFIBEdAtUv8shAyYriF091Hx+tW9Yd3pVFiROuTjVS9N2clT1SSkDoVNKx4LXxceEhQG5Burjpoe1OSnJOszGP4FMz6xGih+i7cqonrp9rOO1myymqqGnpBXslrDfPogqo6Gqp0uI/SeAOHV/QgF1EyFkfmCiIEABCBQQcBbt5tvvtmn0Ud9vEHM/J46dUougU8jQ+kD8hM0KGlUz8jXCgitWtTRTHzhbsae6qWgGu5uv/12uU9yY/Qa8Ewu3f183vOel4mcuht0npqyMIEmLeN4jcBy8/wHLXSnMv/sZUgsH0xvrfP37/TkiAy9lnvoqNZ6aFcB3QuWw6OpOYU1W1tYZbWRFvgEmXUCagJZ/1tuuUVrc7Ts6Fu/9Vt9rrKKyE+TbZV6mXYXN6mkpUP1datGXaZYvlJBzlTdbrrpJl0C6a6x1vrqJrUXVRiZL4UYCEBgNwkUOIT+sxNaiqnnHzTJptk/TfHJIAVAusfmw/6epZaDZkZM7erWlO4RKpnGTd2/LHyriqa2gswQkBmWMQhPk4d4BfQ8YXg1WRyfCWvKSC6Zd8MyhxSpG34hMlQkxOi5Ptldv6uJPqmhOcNwNBNQpbwBe/Ob36wVL/4tL5IpF8U/dSl3UYt/ZP+UsYKDt3wZ4XV2Ne+nZ/HlterhDblwai+fS55tJrueYwx3cHURM3eTZVTNA8yUW6ZhJpl2g6g6nUpXG+pgsu6aDtVaZc1GSkJhZL4gYiAAAQhUENCdRDkJfp5QybRqRqN3ZvuSL/kSPVrmF7koTbCPmjsqlCybqENyJHRUM1S67RUny3zEIiNEjpZuzvpRTrnkcMZ55w4HnSVhqkr5UpQljpQ0xWjJaBxZFpZhDQ6hf/OnUiqgpxxlnnRIbLXORZG+soVV1p3fMvkV8dJTt0d1k1prNX2ysop4R1H3djNN73flvtbXLUZdpltesXzKIGeqbsqrVV36NrJQ66EVeYZeWmFkviBiIACBHSQwdvNC5XW7TjfwtGlQlmn0CxrDUQXCqKQll9rVzTZ5j/lNK9d1VPfY9Fs4dvtDOhpvQXgcOVPYW9NC4Rr6Y9OSsQQqRbc2M7a5omhNWuqOrxxm/5608HEnGbbgECqNl+AJ+AuCjMxCOJk08a7Sx7XQIT2LqNK1UtcnyzeZHDN/M7tOk9VXtayxpmoYV8eHg6g6GiqL1vzobrTel6PpQXnd/iV+hZH5soiBAAQgUEZAY5GegddrQv1NNK3G1DNs8aYBVotCNdrkJejpQRkgPRCROeRvdGqKRvEaq++44444gaat4t2ysGZ7ZLOCpxqS6QaoXnkSducIzK3SHGUpi8yiXg+jZZ96IY08Ky9E06pym/UMhRzCcEdYVZY/nK+yVsf4O851FNDbaPQYQpxSzSQJ1e6rvoGhSV09SBI3vcKyNT/xEz+hewGL6zafYqrIVN1kgrXpNuv3f//3axmXNt2DKIyMsRCGAAR2mUCBQ1gfh6bsZEjkEXkHMv7Vqg8/1abpMk2R+duBGcmFH7vXSKdbg4XpNXzX+fSFhmwtofErJDMl6gZkZsVmJsFMuzJgmj+U76fNTw/67HII9YCEbs7JXQxTo5q/klbx67ZDWcoewnUCmhMT+UxKubKZu86ZBH63TpMtruqyNZRzq86gOxd6JkQXFpoTVocpjCyEQCQEIACBCgJ6d4vMmaaSwsLRkFgPe+t1VjqqL/SGyBCQn6C3oWgBf2yqdCGuV1jr7Wv+O3t6NkwvV4tv6uklokFCRUCLVHXvT6L0GF5Iplufui/m74iFyFkDc6tUWJBuquppycJDPlL3mmUNv+u7vkuDth4w8ZGqnb6loSdWZD2DQ6hbmXpORC+Hy1RZI3/h3dXCQuXS676hX87qE8jmyidUiYXpfaTuC+gZRT0voyVLIZnU0PoaWRx55ovrNp9iUmaqbnpm5zM+4zO82ppOVIsIdWFkqBoBCEBgxwks5BCK3Xd8x3fo+cDMoxF6HFwG1U/16J6fRlV5CHIRY9ZauqlnBeOYENYH1iU2k16LV3W7S4YhJCsLqESZTC2gz9hyPRyoF5xKsbKMmXg9GS+zERbqZI5qV/eJZeNf//rXy4DFK1TlHOp9PFJANk/XDT6jtNKyDXGI16zqkPwZPdTu09T8VaF6v6vKDem1WFQGOP/gSkgQB+o02YKqLltDXQAFx143cXVLXtdDhZFxxQlDAAIQqENAw7WMhR7u0qPvGlj0QjK9Y0yvB5N7oJFHV9g6WiZHq/c1IunZs5/+6Z/WaK83fum2oFbQBK9PD43LYurFWrolqnuUmmnU0TJpmXg5UboJK8U056NZQVkB/0WlmoN/RlrYXUSlICQE3vSmNz396U8Pu/mAJv309IEeztSrBHQrMyTQbKE8N8Vo0UeI1OvQ9PiGJld1SFUWT7mLspuZtwyE9PmAXhag2TzZZbWjWkQM9fYa/eZTZmJ0IaGnVNQHtL5UHUCv85HaUl7PaPiUC+o2t2IqvVo3vVZAd5/1fj5dw2iZqCyyGBZGZqrMLgQgsNMEdIsxbPFbRkNkHIhf5unj9c5r3RaVpdQqGtmnL/uyL9PNM30RIeTSrVC9MkSR/rMTSiMTK2/KL7nJv2VU91OlhtJrmYfupMoMyz/RcCanK3zmQcLzb6oML9LUvUO99lPTd/pwreyHbjp+3dd9nYy0bq8GrfIV0SEtE9IqWZ+m+i2jPo0eY9O9Ol09yEMLkhWQwVZ85ouOmsH7rM/6LFVcdzdVKSkmM6xq+jflzPSWURkS3d2UK66ZMT1Np28t6mFCr8DUeinZ1CabVdW47j5coaESVLSdz16toaZ/xVwvF5WF1v1a+YT6cFNhZF4xYiAAAQjEBLz5j2NCWLcRNT7rlS2aDtItTj33rvd4yUKFBApoCYw8jThGNwT9h+lltmRQNDOTMRB6Ulq2QC9i0V1OGUe9ZlOuke6BFkpTpCysvuXgi5A91T1EfQNQ80t6sYrebq13LOtWqYyRT5D/9ZNpme8QZnSuUEkC82+8zGRXmqBk9WcnvHr/8T/+R2GXCfO7/tc/Bi/rEEcqLJ66fSxE4ilXU5+t0rok2XS5lD5laIIQyEjQzJ7uBetZeknQPKTsZiZjnD5URJGiLbDKooyaVNSlixboxonr6xbnCuEyxZRgKvNq3fR+eHmzUlvXQrKPvsTCyKAMAQhAYGcJJG5wXP85HEJll9smS6l3o8nC6YalhvVYpg/LK9MaBtk/JdPn+JRFg5SGeK2+UIK8GyO/0afXEK/bnz/1Uz+l4S8WO9Wp0MONelZB91NVogJSIM6eL1FHZ3UI/RerdPcxlqywXF/BDeYqPvpzP/dzXispJg7yDPX2VHHwH4qItYrDsQSF5bDpsXi5nbqZqtJ1s1MxPk1hrrhePlmdJquvaka9ag11dGrbKU21hrpDr4rL55fVlCvoFSiM9If4hQAEIFBIoMIhLExPJAQgAAEIQGCbCMgOdvW/quQtIr8QgAAEIACBnSKg5QaqL3ZwpxqdykIAAhCAQCAgO7joM4RBFgEIQAACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAaTC9cgAAzMZJREFUAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOAQNoYSQRCAAAQgAAEIQAACEIAABNpFAIewXe2FthCAAAQgAAEIQAACEIAABBojgEPYGEoEQQACEIAABCAAAQhAAAIQaBcBHMJ2tRfaQgACEIAABCAAAQhAAAIQaIwADmFjKBEEAQhAAAIQgAAEIAABCECgXQRwCNvVXmgLAQhAAAIQgAAEIAABCECgMQI4hI2hRBAEIAABCEAAAhCAAAQgAIF2EcAhbFd7oS0EIAABCEAAAhCAAAQgAIHGCOw1JglBEFgHgQcePNftdoaj7rnzZ0ed7v0PnL3qiiu9IldefubMZVesQynKhAAEIAABCKyIAHZwRaApBgLbSwCHcHvbtoU1O/fgA17rs+fPKnDu/P3y9BS459x9d99/96jT+ci9d3Q6FvO+u299w4f/XK5gt6fokQUUq2P+jwvHP5/8mGdq9wmPvEG/11/9aP0+6qprfYJrz1ytwFWXX6Vf+ZD6xY30ZPiFAAQgAIEVE8AOrhg4xUEAAiJgl8+jkS6p2SDQPAFv2Lx3p7k7X8C95+696+zdvc5oMOreft8d6nzvu+tWHfqtj7xLv6NRr6NjQ/Pw5Oz19Fvi43lp+d9u1zmR+QP1YkJpL3E+pDI9PnIjr7vykYq5JvIhtctUZD20pILAJhLouiEGO7iJbbMVOmEHt6IZqQQEtpmA7CAO4TY3cLN1K7Vq99/tC/rIfbcr8IG7bxmNur9329v3e6OTUU//dMNh1NFEnjpb4qvJ3+vv2bRe9TaTazdNWHVRjR01N7Lbefy1NhX5WD8V6XxI7cZuJD5kY8QRBIHFCGAHF+O3W7mxg3XaGztYhxJpILA5BHAIN6ct1qOJHjyQn6an7+47d1ZujJ+7k/emzXt377/rFoX/921v1688rlO94YVhX4Gh5u26nYGcPfl4dkSZukOb1uv2em4Bp8Vlt5oOXlHWrKgF96XJghIazP7ix9zkpQU38lElPqSSsZy1QfKIgoAI4BDueDfYSTsoC6iOjx3c8b5P9SGQEMAh3NqucO6BB7TwMl2reb+G/r/44F+qtt7Nu/WeD/72h94ZG4NeVwumzLXb6w5PRv2Bc/PM01MiLcDUW1uG1lvcA3uKK+AmCQWxuaiirLlElRGx2pUJt/lg3odUbWM30j8SqUimIre5H1C3JghoYJMYlow2wXKzZGAHN6s9mtYGO9g0UeTtLgG7wlftMYQt6gLnzj9gPlpmWs8t2rQXrnRHmtP737e9YzjqnOoP5MsdnvT3+wM1sx7Yk1P3qP7wzpOem9LrDrs9e2WLe1Cv3+n09zTB56b+HA7zu+weYsLGvD33aJ4CUXQ4vnSE+IGLI/bmM8xDSmDsQ2qXN+ssDhkJrSOAHWxdk2EHW9dkm6MwdnBz2gJNNocADuHmtIVpIiMnB+vsA+fuP3+/d7v+4hY3rXfP7f2eeXq33v1/P3B46mSol67ooTxLIt9MTp2e0hsMu1rGqbD8vGGno4k+Vzcdn9iOh/1T/ROb+lNud9CHtduXQOc37veH/f5orze8dG/wUQcPX9EdvuvwygeH+3Ipe0pjvcbuISSuYuoxKq+LT/edbt6hDEqkx6ygWTccwlmJNZI+fwsWH7IRsAjZHAI4hJvTFtIEO1jRHNjBCjjLO4QdXB5bJG8IARzCFTWEt3Byh84+ePa+s2c1pt99//1333fnqN+9TdN6o85777j1Tbe9XQnkaA1HveOTrjxAuWSalFOMHKpLe4OHBzahpzndfnd4aC6b/k/0D66dJga16c2cZQs4fcp8tSP/LnHfVLr+dTujM/tH9xxeoqLliOqYd+psarmnn5G5iNpsotGO2P+agHSHunbIHinUn0S+aueS2DJUi0w222liw1g2QXFRGXnbKYmxG8ly1kURk79RAjiEjeIsFYYdxA6Wdo6tO4Ad3Lom3fIK4RA20MD2zrGRPa1n03rDrr6Yd5e+mDfqyNOTf6Jpvd//8Nsvak7PHCh7B8tBZ3juoYPRgVyp0b5cvmiL/DrzA/f6toYz3oI7J7siZyw+FMLe5JQdDcmmBiQ9tV7qJ/r8w0g+YZyrThGSIB9VLq4CEuidRulu/qpbuCqnUULNdbTHFb33qAhVdFyWSg87IZBqYlLT8Ax/N9l1zOoWqqiA3+IaRzOx6eFaf9V8Tl4sq1bGJSXy5pPlrEvCi9gKAjiEFXBqHsIOVoDCDlbAKTuEHax+S7m48Ya5ss5D/BwEcAhngyab9+u//7/kJ33knjved8eto6PR++972y0D+6xCZtPKTMXoVSx7bn4sPqq0fn6sbBIvTjxfOPKmxq7UfKIyuWxisN7m/A1za+VGqsryCZUvD6pYmCtGtXCOormjfjbSTzymnqStX/XqxPiVOOgYAlEpSlsUHaXIBLNmKXN4mbvmcueVnbkGkYrmPeYlRglaFay4BRt/4UN14s06rWrYVSuLQzgTcewgdnCmDrNgYuxgNUDsYDUfjtYkgENYE1SS7Nbbb/3Uf/OlWjnpV2bKtdDqzTIRuvKW/xP7J5mU7tJ87L1kjm7O7kzGL6jtc4mAqilcmTWiIZmDEPaSZGG/2oFM8qb6uRWqahPnRjoRivErWnt9c4Nc2D6J4V0qr5Ivy8XYobBFwRBXK7BGB7KWftueKG7WwrraB7I6ncc/8gb9Vt+C5f5rIcDti8QhnKlNsYPVuJyFSZJgB6tZcXRJBLCDSwK7xWJlB/e2uHqNV+2+c/dLprwU97icD1Y5Ds4Y2PyY32I7oRjthkNpkk35G7tGNXWKa5fJboxcbfOw4lxW0JiW7WXc5Zz3aGmUyv7Xz8Dt+X37tdIUFyhriLR/8ths0tLp0uv2+uYH9nvDnoU1+2hH5Dr6rylKgkvnFEm1N5192Ep2Gkczb1MH4kTL9A8OZEqigb914P/u7e9SSf63ZpHeh1RiuZHXX/1oBa7jQ5E12ZFs6whgByuaNLZo2MEKUJlD2MEMkEV2sYOL0NvlvDiEM7T+1VdcJe8iM8rXyR8biTh96mLEcfOHpVhwfupLmaM6XnhZpeKiY5XqFxRqkeGTLTHyHmPXMXbXbGRMsSjov7CiBIo3b85ezCrfz2Z85TqmHqXVQAt6pbC5i3JLvbHys41WjWQScr/X6feHl49G9w0OLukNDvWkqLLs2Uc+FEi0jevgC429R/NXfXIrtOaG7SwDlSFTxy6WiYrjg/cYAvHRwnDsQypB3o3kzTqF3IjcfALYwbiNknE+jsqFsYMyh+MNOzhmsZQQdnApWHdAKA7hDI1sLkT5VscwlOeedsQcjOlbfadruqyiFFPrGBRQSu+YhUCRvOK4IKT4cBob/EZFxHAmlPQTeOnhxBWc+GqGOX75LRlSh1aIeXD6Hdjy4OBjKOrhk/6o07/u6juvOTjZ6wweGF16PNgbDGwB2qCjj3t0B92+8qtQTSnrmx2XDTonvf6l/dFtQ4vQ/35C0pVvaow7mIVMdVdHOz4+ZDtFGiu+3paxFvUykWo2AsF1DIE6+ePlrHkfUhL4UGQdjKRZKoGJsShXUjyM5Q4uHIEdzCFcux186FiPZGAHcw1DRLQSBzvYiu6AQzhbM+ni3hu8EJgtv80ujXNoKI93xwfmDS3XGJdoVVYFKaP6+UxxuETMzNFl5WYEjZWwA/K75NPpf9vRXykmFb0oH2kHijfLE7wpVe1Ak4Hd0a0PXqUvN0qOJhUPB325eQr7D4RYeleiyj3oHV97yX2X7x1+8OLVowuXnwz1Slq5i27Nqtw9LVhVTkvsFiRb2DLr/+QxSLmHaW/xOlhql8TUctXRcVUhMDcJRVvwaYsOzhwXgMycc8kZmq3mkpVNxHurOYftVH7/VKTcSJazrqaxdrkUNxT5gUcjlht9ZsQRj942RLsBb0YZpcnnU6lUXL0DZVWQMmFMjsP1pE5PVVZuJudYCTvQpB08tX+CHcQOZvrbIrvYwUXoLZIXh3BOemUmp+bo7EstTFwmeU5FG81WqHCmhLz+yiVrtK7N6+wVSJ2oxEBLL8Uogdc5Sen8sLy24YrFLKvt6E9ySXQ0sJlAv53kc7qYi529vzq5TstTT/UHA/c9kd5oqCWrI1uoOuoMRv71RJIc3lMUZFoKV6R/z6rNLXaTd9va5GOqmcJa4+ovz5Rc0V4pO557+WqqrymXCrDwrNuCfleDdnRBTWat+IakD95jCGQUU0PbP81X658+JTrqfur1z3A9o/vE6x732Guuu+7qa/VS1sdffwMv0cmgY3cqgfxo77O44Wpq7iRBYeIyyXWFLjNdocKZAvP6K1c8pGfSL3vX6+wV8KbBNHRDvwL6qwRe5yRliVkIxkKiXMpEiP5gB+duROzg3Oh8xmD+QiAjEDuYAVK4i0NYiKVupAbEdFisnSW5Sq+bPpMuGazjy/koRfXRkDAZwsO+uQQlEqM0NYN54TUzLjWZbykVEWxh0DNT98xuXisPORMfIoPYTALtnujbG4Pe0WAvJO477ImRDpY2zRl0VkQypzlwac171LJXvb5V0eOGG3tEoQD7wKP5jJp21I9/Eat/56o8Ss09hoRWhPMcLJlzdFMt5v8rUVO3sc5Tk5KgnIBQ+/sI6hLuy5/67Q2GfddD1Jy61rMHX3t9/Y7ecM97+nujfq/zvx/+i/4H7RVKavF+f/RJj3zq46953LNueMYn3/Ti8qI4AoEsgXgYyR4r2Z86zJbkS6J9idHgN5G8+mhImhurzS9qassJb0rwQnJEJjE3yWivvaTSmRbJ7OZLLWz0EBnE5jNiB/NMsIN5JnPEYAfngBZnsbEgvqaMjxHOEDh3/oHnf+vn5ke6qUNnRk4ju0GNwtLD0bgspQzxhaY0jOZxLoULi8gkLhSovL5EHQ1FZ4Svctcr6V0o/ar361/yQUg/pTZ2r1ap12xleZL2MQ+9E2dol/tl8DNyi61OaMhkstEvUrUpyeS7HVrFKp+h566WxEvl2bBhm/2N5h5dXJM/aTlNymy7LDGR76d+qtaUv6dJP00469VIbte+9mnOv5pMvl/f5pLl6cn309Sx/fZGfRejM7rfG+0psqvfoUX29NFU+26qTgf9PvWqZ736uV/69Mc9q+246uhvvIxnG07+OvVZchrsYAw4DIY+sqwT+UFbR30glrD6sFcy7u46AVpoB0XO7n8JqbeD2i/jH0PGDsY0WhrGDjbecLKDGMIZqBYawkJnqY7Q2DDkhcRHg7SQzB+Nx76MWVKW+GiQEAIZ+ZIs82C9oWTLy/cJMyoV5vaaZEosTLnsyFSTpBzVV/VSFfK1y9CohrlstSU/o6H0EU/3G11hRFcbscIx+Tg+VrvYRsYpnAdiEV4Ve6Oq8z18QAz10Q7vj9iMU+JAKqASY+WtwyinXYyYgPiiRLt+81OUdnjebZG885bZcD5VIXL8Om7qrzeU+zfqCdpIs81u/O66+Wb5e4LfVxN0R8H3kzcoD3BPfmB3aL6fhce+n5xA5wqO9rr65orzDM0b1KtyOxbTsciPPvOCT3zCK5//xE9uuG4bJg47OFODbLEdHA9MJUTioSxOsq12MK6jwmXmI5NsebsZ/tLHDYa2pmVs5rCDaQNgB2XFsINpd5jyV3aQJaNTGGUO23iUGxT9IJWJHg9PkyL8FbKPC1nsmnny3A2H4twZmRO7uQwTR2MpRWGfuPDq3CV3425JxqB5ToUkg8ROVq5I0EripKqUtDbMFZex6BmFQx1z+RaNiOHE4UK5QUnzsawmPiKtTaT0pMJJAvP6ojQqYtxk5l2Oj43jIz3kpzm/0UlLVigaSfkqaSrthbCJT1apShtNJGpglt9oXqN5L0po0frnPJlx1UyWCfFKe3E+bEfqbWHVa6zN1KwzJZ4qbaYEqqD985N++vWOny0JTmZmRanbt09lyq8zx8+m+8zf04dPxNLMnvf93CyfTfRpVwtB5QFavAv4aUCbG7S33O71BpZRvqLz/fa7A0Va4o5NFer9t/u94fHhb/3vv3zjX334pc+68fNu+piXzVQjEm8xARtecmOEReai41ElBuLHLx8TJNmINnkShkNx3ozMid1chomjsZTisBVfPtosZAeLC1xHrCCLk7VhrvQwDhdyy7ROLvf8EdIktHwcLpI4tjeJBUxqktYmCJLMKByqix0sojqOm2A2jl5FSE1o/7CDq4BdUAYOYQGU6qjJIWYibTyGesOUH9om0ozHQDsJJmTZiTHDiTlT4kxBNXYLniobW46s4mPNfRpVY8nq1ahBZZK4mbzOUZ1maoVZ2sypFEuPw3l9A3AdsrD5Z5bK4U2Sx2mSqPhProDQmTUET+RNU2Z6ZWSKxx02eF+uKO80JqWO/LtyXLbRwNT1l1TenXS7SUr7E7Rxix5VMUXYPz8JKc/IwuZRqkSnoPtVZCTDB83kuy1qx1yioggvNoHqdvLClU9iC+OLRI4bSFmUUUjVeO5JP30BU5N+WvCpprSn/iypLon9gk/5yfLrzPdz9zhteac5cs4PlM+mgOb9Ooq0ZL2OHDk5h1oXqvk98wDlDbr05t1p2lCRbvZPkW5KcLivXB3zFfXFlDRy1O/Y2lFNDyq7jrp/w8GFX3/He/7nrR/59E945rdcfebGwjoSuWsE1EvLtni0zwwgIctEmvhkymWIU4bsZYGZEpcJKY9fyA5K7JLVK1d89iPBHEhnhWfSfKbh0asW96Y4nFc8PmpKYgfdO4F0DsVkPDfsIHYwfwZVxFgX4tmJCkCZQ0/5B5+cidGuHy79dV5szmYaQ/NidyNGtqNgIKuou2WwLT/6+fjpv3Eb2cV3gZX3BaRFFYlcXuMGS1xUbGKcdUgK5Lucz1Kld6FQk6atFGmwK3HuGONEfCSnNE3qsCljLHwGzcMFqfNHzY3ShJc22UXVw16io9lIU0VrWS08WTm/6+MtjesDnrxHIYctbJY1zZ7+9RFKpIgJ4a46ltkayNXOrfbUp0h6A2sye9jPOo9z/OTfmpJ+waeF5bzZq2K1nlNpvddnHqA96Weumk366ahNsTqPzuLN9zPn0D0WaC6cIs0VNO/RzfuN9n0Ci5fvZ0cti/3aDGHG/VMRkmAJ3FyiXERtXqzKfdqTfu5jbni5i9ueH3PBraWiJt+eyi2lJtjBprFa3wtjSx3haWedKdOE4Li/6wxI7aAET8j0o+JEzmgHOxhjjMDY+B92S9NgB7GDoZesO6CmYIZwtkbwJ3bm4jI+8yUusztbATuTutrMVGBIR9nUIFYkjQ7FjZJtviJJSqMsFUpWHIqKzQaDGlmrO05YpE161GtualWligxRmtH/rcjkqBYc9wp7f2lSmPO7nMOTjY8urb3Caelp0+mKI3XCVZEQjuVMeImm1zhvomVA4D0w87KSzR/xEkIqc2q0yUlSOgUTlPqrSyE7YgfkjWk3lWRF6mDPPhLir5akaqhRUFtFmI/n/5nL59bQatdEObWdj6e5PDe/57486d7sol33rlc5YDYNqMk9v8JTu3Lz5IPpGT/zzdxUoXlxNlVoc3rel7MJwNT9s4zOIdSvPSgod1ELQZ3jZ7vOgZRrl3H/JMEdch5m6v5JpuNkkWJiTqmyGyGtMu186AN/8+Fz/+Tpz/xGF8HPjhLwp1U4FzyFMLgV7u4oqWnVduPRtERFx9MBMQx7RYlycXEbZZsvkZQKdnmVxkZoNybkhFlExaHC9D4yqKEyJ8ob56mql9fc1KpKVSa5vDKJMgVCvcJhzB+rqSyuAt7cTMQ3bwcnUCVaBgROAzNDqRL+iFcspHLGzyyBUcAOmmmzpTTYQd9rcAjTs6fe32Qgs5vJ4eRMbvIoJjrr6onb1VTzWZFFaPkS07Ey23ZBshL4Y9bAlsf9hMMLB0LFgwZ1RHqTUydldZrCQlMmxVmDwpnDof8X2siU4kSmpAkmy0us6fhsSrJIbDibXJrJbJEjmljisR2UBKtoKsHCTpQ+1dHV9x6tgLBi1W7fuM2du1aQ8+P29uTN2eSbrMVISzrtI35a3qky3AtdTL55jpqYU0gKKNzfd3N38r60yFOzdrak09xI/5EP53PKIVRp8gIts4XMITQvy1w795jfnnMR961cc//cPJ7zEr1vZpN+bm7QJvG0NNSm+9y0oc342cpPiXIPBJovZ0Yu8QA16Wdzhno40Io246eizc3zaSwwdv+cH2i62Sylu2bw3qB9scK07xye/b6/eMt7b3jytz7i8hs9PH53jUBy+kd2MAwUOhTO3F3DMmt9A7RZM86dPpQYBnAvKsT7XTdOWRA7WMjHRxofGxFtcyO6D8a/ydHJKNH1JmkcrfHWIlNp4UBqxSyisIjE/JlAJwI7iB0MvWf2gPWh8ZXX7Pl3LceTXvspqrLO3jKblz+ldw1RdX0zhqc68bKPBrPnCwptF5T0w3RNNUKumumrkxX2scRsmEEx46FO6H+rRc1x1JUwc74AsCxnsF75BPVPqNKUziL6Irz+oTgDldy3sZInJEzaURGW3fW/SilPSRklzX4noVhl5djJk+t0Di7p9ve73X15aeZK2qRcf3SqPzy1N9C//f7goG9vajH/zc34mRLqLnLOOuZqSowK9UdVopw37cph81nk+8ljNI+up8WfcgLNzbOFnebgORfRu3POu3MpzVF0Nz69S+lufzr3T8Wal2j3RO2fRn8LOP/QhZ3f6pzAxP0zv9GWtfqM+rX5QrcpwY03/dElW+ETir7qhB30LVvnFztYh1JFmmaNRUVBcxwKw3hQ0p0fdSWFXHUzVKZT0ZmBV8m9Pj7eW0D/WylpnoOTQ35dCQFgWYbYGGXS5CvrE+RllqbEDppdk3HEDmY615Rd2UFmCKcwyhzWaZkZ7+KY/Embyb47uxlKm1lxuwys3MrG3DhTMJb1W78OnFC0xMosSVWzjKmBql9WrOqyw/l6ZfQsvMcprWQgA8askqHO6QF3Bk40nU/ihevXpDmZjpzPZh5XJMmyJ7sTt1R9pPf9XBo98Zc6k1H2RKYNBi77yeFocKyJQdmh4/2Dzv6l/SsOOvd3LzkajS4ORqd7g0s6x5f2jw/6J6f7Jwe9gV7gKYsllUxPe8FMd2Bzj25NaXe4b25eZ7/n7JkMm3v7i9gqUk6gfywwzAFqhtBNJ9pLZWz+0Dt4mmBUercYRg6c+YdJvCYkrQ/5aUDzLV1V3FyilS595AGqWuYHukPeA5T7pz1F64/9s73O7X/2/1z/jN8/vRU+oasrP3UJqK9mTvY4RuG6grY9XYbS5lc3tTATmuaGvomjfsePCQrXb/06cELREosdjLm7M3DiRPOssIPYwbifzBTGIZwJV+dlj3vWb3/oHT5PGPhCYDZZ7UldZ+BuT21KNZ2vmsFilcqdPJAfxyeP62IqMcrujxvxbSYwMbTe7vpCl9TxkuIzas21G+oSchfq7G1YSKNAuIcarjPio6kzl8aliWKLqGNiJzmqTio/MZ/adSmT3UwLxruDgecvIc7zM82cXPdjxWrXPCdV1BaX2pccjjsnDwyfsn/Hl131/j/vPelDJ9e8b3Dm6GT/wmhweu/k0tHxFfuH8g9P905Odc05lIcm59C1uoowd02OnCYD9eve/2kzgd6dc96YvQnGze+lkenRvpbKWEa9XcYmNqVacAL9NKAH4p1AJ8Sq5nXXa3fkASa7FmmftXfp7bgS+01veE1C6Z/b8AlTFDv1Fzu4rc3tznUNQjNv8bBZJ7MGmEJbEPIG2+G0SfRS2OfyQ74vtFpOEDhrYB4KJWWEuoTjhTqndiqkwg5iB8edYQUhHML5IfuTvPDEnk9oftSYTw65NpnAdMPpbZ2rQ5xYPU09pC0msKwJQievPnEqTKMkR4QmMBkg59iIUpCgS4lJt9CrlswZ2nSi3Bxn/GPaLsb8QKW2g+5KxOdMEltj6KUwdshlVEorU68n1Uzbg8O919/3BHmRH3xoeGr/3ODSU4NL9x8enj48OfXw6PTl+0dneheHvaNu97jXO5FPeNA5ces/7elGiZUI5/XZ6lAXGLuFSmBTgm5RaHLU1nxqoanZTpsV1K5T1PuBCrtnCy3K+4HJ0UIn0OYGzesLnl/eCXSyxz/4hGMWOxnyZ7Q/UxoBEIaIRqQhZDMJTAy2hSpGo3yc2IZ2jZF+xHbjc2HuRSKd7EUETM8bOnn1iROsWJBo1iHdIkIuKsWEHcQOpn1ktr84hLPxyqe2sWn2USkMB3mBxLSIgLMc4wF6Mc2TkT4d1cfCQgcLgfEx5x3ls8QJ6od9TVZgDqXSrCdObBpjo2iinN4eghOrGIuyMzOd2NP+RC4Xb4YzubCwo7EcO6fTQ7E3mAgxl83P25oIE+LWW1qBLvVf3f8oFW1rV/qdE31r4uT45IHD3l731NWnDy/dHxzvPTQ8OLN/eGb/4uW9w0uGmiXUu2iOT3U1cziQZ+ibrOscPzl4ehxC/p6rqXUSm1HUr/MDbVYw9QPlxdmraJzelix94E8rQjXjpxiXyyYDFXZynO/njszqBzphyQ8+YUxjN8PuTPJdbAYA7mSZIT1JN5ZAoW2aS1s30LnRO5M9FBECcQIbed2IHUfOF/b9uCFhU1SY9cQJFk1yJywadhA7OKWvTT+MQzid0dQUwarlx6lwaKoQErSRgLMcieHIt/6MNUp9lxmzKXlyse8yLm4Uw2XdMiyikxlKmL2qLoeMYsYWBgiqvm8U/fVYLCb1puLyJMGEZDxD088EGEYtvHS7TmfLmi1UCfUooHwpey+MPeNnWWzzLxdNZSmjZMo7HHUO7z/sP3g0umy/c/nB/Sf9C6ODh0eHVx1cOBkeHXX6xz09CXj4iP7RJb1jOYGmiqYKnUPo3T+VIPdP8fq1sNPPPQHo3h+THFK8TfF5P9CE6IsRrlLSzD7KaNndrwMUu4JT5wOVd7wFLp3OXTf/6xue+6PjQ4R2j0AwdskpGBEIh6I4gltFIDRxvvVnrKcNTdHQMkPuFtlBV0er6SIbdhA7uEj/yefFIcwzmT8mjInziyAnBBYjEBtFL8l8m7m22F7NKyNfcCw1f3TRGFXf19efjOYMpjFBtEfkrKlVy3mG5nppxyd2Evx1SfoMovb0oF/qiFpa5+Ipvbwv+5yguYLmodmve3GLEiuTdv3mnEmLGcr5e+Bo8NDx3mV7nTMHw+P+xdH+VQcXr+heHI6O5V/qSUK5f/qAxOnesfmBzvFz3xVUZOIEhroooKWh+nXzhIkr6Id1eal+5aecQKNgLqJND1pitxu7gi7SJ1cwtxU1f9yvji/8xm3v/rfXP+0bcjmJ2DkC/tTbuWpT4U0iEAbeoFQ8XoXIOgEbMdOtaCBMj83yd2G3eUphqj52EDs4pZfkDuMQ5pAQAYHtIpAxjfPZxeaMorldCwLOztSVi3Nm0XtDSSJ/teqZeOsul8sxsT1zDvWbWtOhPkqvCB0e2WtevE8YSpMQ+X42Q+hdQXMLO/rghHLbKzvNG9RbXswnTDwzF+OyjwaDzkhu4YWTwZmDzmX7dx/2Dwd7WkF60u8fj/aOenvX7D0kJ/Cga88W2uOFzuVLHh0MGlgZzo91TF2a5FjGG5Q2UiT2BmMZCk/MDXoukyl8t9GRssZ7+P7Xnb/n5Zdf+5zJfOxBAAIQWD+BjbKDzsKUDaV1WWEHx6Swg2MW84dwCGdj94TrbvRvGdXgMt+F9Wzlkbo9BBoZ4ldQ3dguzteHM3asyH0orYfLm+RY0l3SzLmZbxclkKsmFT0Kg5Cezs4VtEOytXo9TE/+nNxAuYWdoSb87HMQ/vk+SUiraFNzzjO0Z/bcwlGT7+cMlca5i2OHUPmsdOV2EIbDw/suHp/vnb5iv3PF6eNu/6izd6Z70Sy9XnBqYh/ud4+UWqXocxT60qEU0jxh2LSuVBq5GiRxbhYx0U4eoD/kClNKebWqlxQO6ifjWCxBgqZ2DC9QBScBV/gdf/k5l3z8u/ZOnQnqEdhKAtjBrWzWRiqVH28bEdu4kGgInD7cFZY+HkPd4XgkLEyfiRQoH4MdlInFDma6x1p2cQjXgp1Ct5NAGOJVvSWN8s2CW9woSp/YLs5kFAOuWVmFpZtlNFSv2KVxe6amAqHKoXTvHMoF836hUiivdt3U4FAfCZRXaDHyD4f2kQez4/qXVlvJFWdvc7H5QPd2GDdDqHglNY9SotwiUqumPfaXqiAJKkyJBsOLZw9Ho4PRmQO5gsej3hV7h0o2POl0962sfvdw3z4ln3iDfvbPPw2oiUY5hHpvjXcLVQepNhgpsennPUAFwmYNZP/b5v/qVzoNTIDNE6bVcinSnySHc//S3HYsQA6B22/+0cc9g4WjKTj+QmD3CIShVVWfdWxfC610RLbCw1A2qybxyBkPklPlBFyzsjJL5exWWRFmXCJV3J6pqUCocihd0nRoNXZQdmZcWWmEHSxrwtXG4xCuljel7QyBMM6OB77NrnuwELEJmVXl2Cgqb2SMqiQFVi5LRkZxxsR6lZvDvC3MCPLt4oset5FNy0nrJLecJLl5Q70FVLH6pIRcJvmEgVQqURHm/iXfivCeocmRl2g+oP2ms3r6UKCkm9slY653vPjKyi0cnZy7eP7hvaNH9IdXnlYG/VPkPScjvRBGW7d36N8+Km9QsqxAbS6gGT+F5BP62UIllhMqn9CSKKDqKNjpnthziRKrqnSV2CVzTqD7+KGVoc25hb7V9NrTTPOFjhECyuE9SZfZfs7f++8ffuA1l15xY4ghAAEI7CwBP8Cq+jYItWGzQdNt8Sg3q+KpjFRUvfyBlZLXxNVKOyg63Z7MqEONHazXOVaSCodwJZgpZIcJxKO8MNQc6NcIrBGL6PWP7WLGuyirYIxrKitvDo1qkWdYWBGZecX79MqeL8IUcHormfwnN0NoPpd5gz35P4pUtG3xFYNy7PX0eUD7cPx+b6CUtpRUMekMoRIra68/tI/QuxlDK8ZNyFnAVOmOBidH50/OHvWH153Wuk69V6a/N7znpGsvCU1KPXE9qOsHbv9M4F7XnD1NB6qoY+cWakdOo3xCeY/S1bmCTmFXL72oxqmvH+XQZm6hB2GapNvAHEzbQjWlZRIzlpAcNYmaZnSiPvjuf/W0F/6YT8kvBCAAARGIB3btxkPNZvIpNB/zqToeVa3itbYY11RWNsq7bfPtoNSUBZRlxA7W6gerTYRDOBvvx177aGWw68n02mi2/KTeeQIzDfTbRGupRrGmRczwzJtPyYlPbe3KZ5KxNW/Q/+qr8FpGqnfMaBwwL9Qv2rEZRAlXes0oyi20eTz3TXl7qtBePJMYbPMplck9UyhXU1OFA/M69c/iXaAzOhw+cNvo5Or93mWX7OtGaqdz3+DSRHOlGp3YCla5naaBlaq32Lipv97xSFOZpqo8Of2Twn3nzEkZt3ZUu85/c1c6OiKHU2JTt887h/ZNRb/cNMPK7dpRbebb2n8+bL+GRH9c5PGDr7/vzq+5+rrn2AG2bSSAHdzGVl1pnbCDwu2H0Knc67Nqhx30FhA7OLXhV54Ah3DlyCkQAimB+gN9mmN1f/2l/5LKky0IW7NGMVhEyc/7e6HQsoCyxBJ8MpsqtJDcHb/Kxc8x2uJLK0I+pJtI9PZdcUrvvUH3a08dKmACbPmoyTIf0TIqaW9gvqXEDuQfKudg1NfLbEbD48N7jm8/v79//SV7BwPvNcoPtP96nePOYH80OtL3Cc0FtRWk8hNdIHFR9QCiL87ei2McTEO5eVo+albY9rQe1D50GDXEwD+UqIxapprkt+TaXAYfcr9OEf34Gvuo5NdU7HQ+/L4fuvq6H584wA4EIACBIgIaCEN0fD8uRK4xgB0M8Bexgz23dmbCDsqe6UYpdjDw3YAADuFCjaAzZKnjxULKkblVBIJR3ASLuMpeHfkk0XVBZdsFVkpVgSt27czw1NsKU1qJNp9nz9jJ9XKekAkUKKU3L8iJN5Np/5J5wmQRqd4046bfzBm0RZwyhLZexsyhvfLTAto3z1BLSp3zpl35h4PD4w99qDe47pJHP+KCpToedvZGSvYIe8eMLRDV7569P0ZLSuUcOh3NxzMze6xZO3eVJXdRy0yVWB6n00LLVJ3mDrabrwxcEscvbf3kKs3vakf6haSZgCU1Conv+OD5X73njq+99tHPySRjdysJWJdOOstW1o9KrY6AjWJuqxjYV6bNKnt1PLbWPJkCKwGpwLV2O9jvDvZkE7GDK+u48xaEQzgzOXc1NXMuMkCgJoEwylcM8TVFzZ1sXVd4CxpF1bcM2nxGMQaodhEWm8WzKTZbWuon21zYFJcL59P7P0qv5aXyIe1RQPeCUC0U9a+ZcfN28hvl3PVsYrAzHHT6emeMfZXefES99MWeuZebp0cC779zOLhy/7FXjR7sn9obaAXp6LjbO9U72R/p+4SDPQnvdA66A91vVUD/pKN+j+3FoaaOfuSI+hWk+tUhv7rT3FZz4RInUCm9u+ivgcKv9x59vez9NNFm2X0RUXQS0xnd/uE34BBGtLYtiB3cthbdsPpgB32DRINrVQsFXGmu2JaOM67FDtr7tzVDKAuFHRw3xSaGcAg3sVXQCQIiEIb4MidnqZT8BZ+/vl9qQWXCg0GraRG9nADN7xaii41ipnT5Y5mYzK7y2hyfT2WauQWYbslocK6kg57akyuogBbJ2M1Rt5bTe4aW11aQmhDLoqTy/TT36D1DzRba1KFL4MrW7sVzwzsGvUdee6D5QGU63T1+eLjf78gbHMknPNU56Xf7p7WO1BldiVI+NZyeTvTvj7GFo/IJnRtqL6VxAXl3mh50K1kVY9X2DqSvb5g5TLxHl8sOCajzJH0y/Wri0aLjRnKO4t13fM+5+7/gzFU3hpQEIAABCMxKwIZJtxUO5rNKmzU9drCQ2Kx2UPcrc3ZQFhA7WEh3bZE4hLOhf9TVjwwZln2LdNnyQ0VmDUxc/M2amfSzE1ijRQydcI2NPna+FkDns069pMj7it5FFIdAwGyhc4zsQsVNFboFoFZCLF9plMsm7pyfZlOFdqPUvVjUYVVirRLVQk55X5Ikt1AZNEWo8ECTiQl6eVcm9vDh0R23dbuPOdDzhZoN9BOUtmS0OzzoHZ/unpx0Tva7gwPtdob7SmQzhPaUoMICKMPr5wkl3WYkbf2nfpMrLR1KQuk8oX21InXzfMW9i+ijwyFfZ5dUxengeL5Rh2778B/gEBqirduwg2pSf15sXdtuboXCKBUPs6tRNxmM19roG2YHZaxmsIP+XWu6QTmvHTT02MFl93YcwkUJa6RoxDCEEWdRhZafv1lVG6G3/EpvRAlrt4hrbKxFzGFovAAwxBReW7hz2hdoKyuVOLiFCpqjJ/dI7pQTJyY6qnk8rfi0lTEKmL9mzwcqoxxCt3BUAS0ctaf49KspQB2yocM+x2QTePq6oV4noyIle9DV5+gtb7J1tWNi9f6Zu27rDq7eH56WAvb2UXmPcgj3h3sXuieX9I5PdU8ORif2q0lHfe7e1LC7sEomUa4eKj3x/SzKbW4RqQ1ivsLxPKFvbou3+UaXw/+knp9P4OVEGidD4gfe/9qnPvML/VF+t5sAdnDB9o1PpQVFbX32dAzTkOQHrdXV2F/8rLGxfIXD6D1fzQPAkL2QpJKF+JwdNKOAHfR3QOP+0Go7iEMYzoj5A/PZQj+yzF/qtuSclUN87m0Lg5nr4Qf0MFjPnH/eDOs1hwtawcJK502jTxbHi7Mzh2YfA3MXY/6WPD496SfddMh+bc5QVtsra79K6VzB5GFChbVK1FKaW6i1oS6XfaZC+czc71tmBbzp15pMeybQ8ihqOHrg7lHnkQePOG1vJZXL53zCvYPe4OJg76Br3uCl3SP9kyMqUUog8ZoktJLc7J/39/yuaqMyggdopcjTUzb/q4BlMTUs3p147scdmPxRfXzekEyTk9ruv++Wq66+cTIte9tJQJ23rHtUVHjW8b9CVKsPzcphDtSt5lOovB+lw5hcmGYZkb6x1tUENkQ3vXmSealxvDh7O+gG+cQ8YQdjaG23gziEcWvOEHa2LzkllK2mLZx10J9BoS1NqrFvTNnVsYxhJuW6ButVtkMYrFdsEddlDkNPWIZFrGi4hLOd86aC/ncxFpYm+qfZQH0CcKjFnHoZjHOdZBjM50um5Uy2smi1jCb0bHLQDRg28+deKmNvGA1zhvYkoR3W5qb4evochDZNOZ6odPPhuufuPjq+5sC8QDl6miTUC2l6A80aXtrrnshDtJJGWj4qGSZKvqS9b8aMuKU2rRSwTeFkwtB5gHI4/VH9JsnCxKBLH/9IjEvmvl7vsvujFhnJueXWP8AhjLltWdh6pJ0QyaZ+63tOGlH8t2wML05NrE4oO5EntjKGmZR1mmNCbgt31Om81nFXXEE9fBOsnnDoCUm1V1BVV0TC2e4xOiPljIvvmNJE/xq1gxJoT+trww6uqoX1unK25RMoG7uXX/JsJeiUDmPNbDmXljqjT4WGmZTVzFc/iC+NkAlei0UU4XVhDG2t/rCkLRQh+aEUVVmV9ndJ3amiST1nHd1TeZqy0657ck8P6CnoZw4lSf9Mhp+106pSE2PThvqjTPY1C2/7rNO6gvU8oQ4oiwvYRyn0r9/pDfQhe3mGw97Zewanr+h3Tu3pRTUn3f6eHFGT0tMkofmdg85B50QFqdhjWdXuSItITSEHywWSOpnXmrpwfobQJUl+4tUvaTLlnmh37VuUPm4RpfYxEv7nf/l1z342q0ZjqLsbrh6TN4eLurM7CzdHo6w+FRpmNK9m7s/TDarnYqpgBxfjV5A77k5umLc089lB566bjJwdVKTsG3awgP+Ko3AIZwN+7ZlryjJUj7xluTYqPj75N0qxoExTGlY0Vqtt5IotYsC4LmhxfwjmKvSWuQOxWAnRroSn/m9yMEHtjmnaziYPR+5Noc4tlHNkE27OkTOfUUddPiWSVyZ3T16bHEL7CoW9dcZspcykTKLXWX/kWLrsmgbULdK+Fo66zw32/VJM7Z699+TUmf7BpfaNw8HQ/kn0Uad70uucyAPs9d0XKU72O13NH6oUyZfVlXw1lldeATf7mHCSwknxlsiqbH+VKHX5LMbF++b2vy4u8QlDvJWX5O88/PC5Sy89Y7LYtoUAdnC9LTk+TxfTIwzgeTHh7M4f2vyYZHC24aopVFWVDhjXBS2upB93q9StfSwWq0zalXCzcyYhOZigdseq7WBiR1w+5YrsoO5hyiZiB2s3zNIS4hAuDS2C20kgDO6x+usa6GMdZgq7UTsZsmfKOF9iD229lPK1bdY0mi20iT0j5Kygs4HeLtoCUZvTs3/uE/NDTchpos4eLxwqRv+cd5fo6M2hvhih14QqrFj/kGGogib1bMWnvErNB9qHKAZHvf5wYKl79roaCbTfB88N9kd7p07Z5+4PNHk41AzhscpT3tO9E/eVwu6x3MmR8ul9pGbJ/d1Z3aX1ATmBNrPoKqUfa0EpGjWkgn4vivMvuVG8ZQvxCtispl8r636d1M5d97z/o294jlWGDQIQaAkB7OAcDYUdrLCDg07PFqGk9yWFN7WDw317Rxp2cI4e13AWHMJ5gOqSyS58dmlTbcPV6i7VO6lrG62j76KruUXqMcWUgp+wxt5S2GOrz9vCLOVVMNcpqbU5Tub1Ob9IU3J6ntA5h1rnOezJ3/OrQC2BLTqVY6bln+Y/aR5vz6331B3Tnk0Yms/mW007MqJaA3rc1QcIO0fuVwkUtlJsSWrv4rnjk8tOnTpt3zGUHzjU1+xtPnKkCcM9W0Gqb9bLrdw7JRn2EQrn1JpVNglelNRROAxoXnhyyJLYNk6chl2kZfJPS/oELq25iz7g05x/8J6wS2CbCKjXhG6zTfWqqItOmnHnrki3pYfiET5UMT73Q+TmBHwX9SPqarSKKW0CnMIe64b/Uh6FWUpTO6uX1LrMDuqhBvuHHSynuAFHcAg3oBHmUmHFlmnGAWKuKrUtUzzuB903wQAEZRSQOVylLQxFBzibByTo2FjAE5Yp1JcnZPbMJuoVoCOt9jRf68TujLq5QvvqoCK8LdZrZOy2qFs7qvUz9rIZfWjeYhKvzdQ76QyPOn3L091z3zO0SJvTM79Rpamk/uH549Fwv3vaShz1tRhV/+mO654+ZaFE2pT6wuhEaaWcdk0n2/w57cKmk45brJs5TJL4hJlf1cDXy1JHWwLBKt85kSgX0Azk/efvilIRhECTBNTRVmmbVllWk5iWKSsM9XEhyRgTR601bCPlSntKUtsAZ/OANN8eVXbQbJk9Bm9rZvSFJmclnQaRHTTbhx1svl3qS8QhrM9qs1JimTarPVJtggFII8wQrXfzV/prMYeqeAxk7SgWb4jU5mXPP1XNt7MCWjYj32xP7uGofzIaap2MXkBqS0nd9wZtXWW6+ZB+zRu079ebP+ZcxCSFf/RQeU2OvaHUvkSvBlVY04N69MKv/Dx66Hivr00LRvtafnM07Pe1TlXphn19HFEzgw/19rr9gX3e0HxSK1aG2ZXhwm7eL10C6pVK+q3VyyVMklueJIFCer5RB5M09kIdJ1L19e9SdQluvuV3/9oLeK9MQoY/zRIY98Vm5SJtMQLxsO8lRQPIYqLnzY0dnJdcQb5pdtBuKebsoNme1A4qPD53fUgNhB0sYL3CKBzCFcKmqCYI6JpzPJA0IXDZMjKmcV12cb3m0EMOKNYFYbltbUtGzQrKWDqzZ2s0NUknv1DPAcrN05YcleHTXdJxR7bPQsgW6leLS+UWpgZSIaGyZaV62lAThfID1Y79zqDvPi8haeYSuu3iAyd7j9jv7nePbamq+YonKl3Z7csW/f5o/2B0bHq5e7TS08/y6VH+0Ba20tReM2N6KkGID9BMYe/3mpdoidw/U8AnlvSQy89Oave2e/9nkEAAAhDYNQLeZIfB31c/DBQrpoEdXDpw8/RSS2e3QRM7qC/n6m1nzg7KJDoriR1cemPMVgAO4Wy8rjpz5WwZSN00gXZ5g94Wxgxiu7h6o+jNofRZ14ShR+EhrL76cUPMEfZzaxUZfY306/7Z8lE3N6hFpLZIRms45VOpS+jX+UuJTxgaRe6ZvCt75ZpNFdoml1JHNQ2oX31hQtZUE4/9riYAtV7UXgxjHp+lNwN89ODxwaXD4SlzSs3jtA9O6NsTfS3PuTjoH9orRSVHk4R2DimvV1J/ld1r7krU0egkG3uAvlLKb5tHYa6vk+bi7Ed+phcWxz/48LnLeNFoYNT+wBbbwfyI3f7mWnMNotFkrAl2UCy21g66ds7ZwcQJdOavwA6GzoEdDChWHMAhXDHwHSqu7Za1Ef0LbWHoBGs0irps92qs0TNskTlMYBm0pElz3BSvCTv7FdjEU3LLZuRS2dyaOVEWn8uYdAd5aN4b1K9Sy83TJlfSxZtPKN9OLw491rcqbO2o/cp/lItok3z6JIVzC/V7dNTVS2zsdaSKNimWSo9vXNDLa/TRCneLVnp4j07vK1U+qeRKM9W1YwfdbKG36E7zccVjT8+7kXoy0ddZKd1Rm2lUzDilHWbbNgLjPrFFNfMnQoMVWjaltssX6mT0cdD9kNIg/2pR2MFqPpmj6UA+7nQ5c6YTKE01aQclKrGDzuLkMibdYKod1D1NrZHxFnD1dtDdrZ20bs6yW+3abwdxCDMdfvru+FSYnnanUzRuWVdMcwX6x30pGMX1WUQBXkGls83oK77iWmeVmLY/YeXSxGO7p5gichMJ3HWPrj9CQysg42dTdnZonFaSvDeoeHMR3VG/plTp3atlhn29u1TvEXXrTl0vsnlC+XHDk97Fo5H/OKHEu2f07d2j8isv9PTlip5Wn3pd5UlaPUZaiGrszX00R9DilECW24XdMRejn/SoQn4m0A74Tc9M2nGX3Vt9xcshlCusuAceOnvZI86kafm7DQTUrEVdfhuq1mwdlk0pyF9SiwT5zWKJpcWah+FxPNrESZcWDp6hSsj7Kksrdix4Z+2gEMxqB+3BCnvkfj120HqIuqz9qW0HlX7UAjuIQ+jadZYfjY/x+DVLVtLuLoHCPlNoa9dlGJwyfpxbg0Vc13VARY+USpmLElMyIZTNZ/TsMxLjzebuZOrGnl5xTkvjMiml+X6WJYlRtGL0z54q1FSevarFJvjs1/7p64XDPRXiitAaVF3HHOnrEvv9owuD/p4+VCiPTKn1XKGtHT0e7Z3Yc4Xm6qlevmruV+tLrXtGMaEWLtLeBJC+gyY9osQTTwxaXfXPxPqJwTiQZuLv9hBQY1un2Z4KtbUmoRVa0RZB2xh3oebrsoNSLDiHq/cMsYNT7aC7yynz56zhyu2guoc3lL4D17eDcYff2DAO4WxNc+XlVypD4fg1myBS7xiBWfvMGs2h6+Ey3LatyyLGY67XZMW/nn91K6hNPaZ844ZLCq+2d+TGYTNjZs8075dGVtVP0rz7J7fMP0loee1jh4k3qcUq0uHYUnWGx3q6sNvRe2TsY4Ty6OxRxuPO6GjYO0o/POEKSzxAidJz/6F4T95P9qWtkKS0GT+lS1S2HCpWaXyygRafui2dXbR4/cs7kz4Zv+0lgB3cnLYbn7qbo1O5JrNqWz0Cl5fTzJEwjGMHBdSP83myatPV20FvUrGD+eZYJAaHcGZ6Oiv8IDVzTjJEBDSCzGoboty7ElRPKxuFV4NgXRYxnGJrqX4ovRpy4hm5fizfxyV2v6Fnu4B7sNAOmvWK3SmXwXy4dPNzgz63HvtTtL9dKj/Q7J/+2SN/I/tAhRZ1qqieZu7sdTX61KEtPE220VCu4gX7mr0mCI+dte51u4e93oHK986dlpDaG0dts9fAuC2D2nYTv9Aft195fT6Z/w2rQ7Vr10z2icKkQu6Thy79ODeh7SGgFq95mmxPnZdQE51kyem3BOFbIxI76MfbFTdozRPc2bykH6d20GkaerYLlNlBX8rMdtBexz2nHdwf9U7pm4hO7/nsoGyumTzXKvqZbgdlZBNMK27D2YrDIZyNF6mbIhDGiqYEbquctdtCDzaMZyu+V5qxSWuxi3W6Vtyf47DyypFLGaahSYneNE1YxPQqUZGSpn/yDGX/9ASg/esODrrdE+ciyhV0RdiCUa0sNcuk1aXujTHDk5Gsnib1Bvacon2O4rinTU6i6WNfsU9NlNdQC0O1hUiFvZ3Tw4pxvDqAN4SqjL6s6I7Z8/TaksnA9MkKn93eT2MH2SAwhYD6SebcmZKh/Yd3rb5ztxh2MEYXj9Jx/NrDcX+Ow1KsQTvoTeG+HqQfzWMHT3r6ZK/dWpVWJ1qpk26z2kENV9YQug1axw6aKd70DYdw0RbaQRu2KDLylxAo60veKdoQG+CH0RW7hQFYxj9UfONY8kWE0mcNyIvLZJGNFEA/0Zc5FHb93GByP9XWiJq1ka3Sr+YG9zr6vrxZs163d6JH6x0A9yJRN53oXjrqLLG9J1RThYpVnJJpck9rSjVrqLZLLJ/TzgEc225z9ny8FHJB5wE67dyu8w8tvfw9D19fwFCO5NFBd0lvrqDdQE3FZjGEuhLYEgJq4bSx56/R4hLmL5ucG0OgrC9hB0MT5Y3UTtlBfZX3pDvQ2pZ57WC3b26gVuQkxk47M9lBpR+omzq7Fp6J2AI7iEMYTrE5A9iwOcHtUrYyC5dhUN2X7KLeDUCZXGvZdbpU67sivfKmMVPwTNCmScs1QC4iU7r3Cc3w5LY67emcQ//CFnuvml4Uc9C1VZma8zuxr0no84P7+tUhK0AfoHfvHdVX60fOSZOXpurrFd0a6OWk6VnBY7shmnyKMGhkiMx/8xHjZnXxWgU6jrEHBV0yH+nvC/iZQCtfR11B4dci9T/bthMYd5Ftryn1m5uAhoI6/aQ6DXawkP80yxWG98Lc2chp0nKDei4iI3EZdlB3SfUo/nx20CYGzbxN9LX6dtAvGZUEf1d0a+wgDmGm307ZPXP5FdUp0osqSzXtpKqWxNEJAtlzd+Jgwzu+rAZLnBh1FlB2o2zhAvVYadbq07DOCTsLdjW1Ok6yJUsr091J62OxhR1Drpn/2FHIZ++IcdODWrypWTn5ZBZhDuVgr6tvC/YGbjLOjK4rXH+ca2YmT48YKoveLqpaHOuzTXrfqOU02fYzVtZiNPuXTPSl8WF6UB6nZzV2/0wR8zMtZ+oH+nz6tcce0/grLrvSh/ndDgLYwXW1o86swkFjGfr4shossSnNZxmQlwGmlTJXawfVa8bbUuyglsyY1ZvPDnb7XfsmE3Zw3EguhEOYATL/bnxl6aX4mOrzcP7ydixnU7akDjZf1ipLrKOVT+O7U76z1ZfQVErzR1Z3cdKU1lk5yzs9tVIzW9jkvvlakzF+z0d6I+pNlrqivCs9anhgn6S3paGyhOaMOaumPTdDqL96XFAV0hyiWw1j9y/NM7Pe0rWXxwyGvWOtNDVhSe+2Q74kV6p7WtAk2549Ipi6jlaeRbqCx06gy2Ra646p/XFbcBHD0fQIf7efgPWoyc3HLO9Emyxty/eS83YltfRlrbLE+tXCDtZnVSfl8k5P7GBuRKzTIOtJg0PYDPe8FQxyw6HlnXKhrBUE1Lk300KsoO5lRayeifpS6FdlWhG/BgK1zw3vS/vk8rvk7MkZ0yyctazNtpkR8Q8TanpQDp5mCPXghO6K6vlDPRLY7w703KDeLtMf9fVgofw0mwu0vPZ4oGywEkms7ybaMc9OctyThDbV59a6pKeyaaHuJA2sVPt/7PL5CUOLSo6YeO8WKib4fuGopXNb7CKmcfzdcgIVg1I4hB3c1k6gkaP2+NcMA+xgMxzXJ8UsU9ptsIPra4ekZBzCBpogmLpqWT5Z283hikf8aqR1jq7ASq2FySbYQudvrKX2dVp+tWncS2DqsJAJNGcp/eqDn5RLpuacyva0uvs2vRw0552ZN6jNfrtDTfTt6SvznZH8Qz1NKE8yPENo+dwkoUmTg2f+oW32O5RXOBra22ysdBevgDvkdrxhtsQ+0vLpaB3fT3Im6u0LdQKSnysuOxPvEt5KAtjBTW5WnZUTZ+kSdF22/EKVsYOFWNYWGT4uMU0D7OA0Qms4jkO4aujBarbaM1zEuiySd47WmsNKzaqhTz9rrjp1qZaJLazDsJE0U1Gnzpet56woMXNMLwGVm5f4hPbONOeNadGnPZtnyz79Zm9Dczkt0jlx+pVnZw9R6JuEmvlzmyUxPex7EjavqNefmUNnLx7Vk4Q6qEcET6wm5unZrqV28UULPpXAlam/yRb7hz4qk0CRGecwzcpfCEwQwA7q3EnP7wkyS9qZo6xZNfTpZ81Vp77VMqcOznWKWDCNG0nnYLxgsavOPhW1WRd1bD2prvuE4SSfVNMWqEyiWr8ddBraGpt0ixe/KM5qlW5bbAdxCNNGrv33Ux737N/58DtqJy9NGE4WnWOt2xZReZG8qwE1q4Y+/ay56tRlqsypA3SdUhZMsyu2cNIqVEMLvlx1snBNKAspY6R3w+giUU/jeRdxwgo5Qb4/eJ9QESKvxJoklP+or0v0zdCO9twKUUkIiRUpr1BvC5WRG5hmehWpHYxt3oTBS6YQx7rHR0Nsoe/nXc2QhsC2EsAOqmX9KTZfEy+Sd74SZ801q4Y+/ay56mg1VSZ2sA7GRtKoLQrNQV64cwvz0YUxSQuv1w5m6qX3omVjinTfJjuIQ1jUwrXjglNXO0dBwiBkBZ6h+vfUsbVAxZZEbXftNrkRdsQnLG6CemeUT5VP6z2o9Hk/e0xQS0blr3m3UL/q1X7JqAL6p02/wSfUjVgdtXg3qWjfJexqTank2OpQxbtLJfd2Gbtl6x40dK6gF5V50s9HukKSH2Up9PEKI32e2M+MRRHeVgLBhC1SwSAEO7gIRuW1s35BEWSfiwB2cCo29Uw5ivn+ueF2UPWa1fFrox3EIZzagVeXYAUWMX8erq56yy+p8dq1wrK6K/7lw51WwvbZQtVIla7wfKYhGa/5jFN6sXGMK8LKkgkxT8+lcKtDk6lChXU4/ZWbZ/6el6D0chr1ay+c0TTgaChvUO8aldk9cHIG+mSh5h7tjTK2KaOJcmtErch0K7F2Olx6VgUdUhnZv35h0DMf/aLsAfYhUE4AO1jOptaR0jO2Vu6CRBooGpdZUMxiUdjBxfiV5l7ADiYWpnC9TIvsoNBUXAZskx3EISw9DdZ4IFhE6VBwL2WNmu1Y0ZtvBTeqQTTEV4ybG6XqVGWCuYorVbM/jPNOZnAzdtmSHTGlS2ynHDN76M+lkgfoo70rqBSyPfZP60vd6170tQm9V2bPfR5CKeUQyuvTk4NyCC3Q03cm9CxHxz5CaH6gvXfUhKQrYWInsKzhalq7bK3Yh8DCBLCDCyNsRsDkMNaMzC2WEpuMtldzbMsi416zP4RkmYvY1dvBE9nBDnZwemfEIZzOaL0pMIrr5U/pMxHYJltYr+LB6pUmL37NTC6fXDItzzQHTB+Z0FyfzfulbqKTLXfuZKRv0MuwmUNofuNI36uwpwc1Q9jv6Wh/XytOdbTbPTa3UC5isuldMuYljrS81HxCbfIGXWCsR4Xv5+f6XL4pP174lEQchsCMBLCDMwIj+ToJYAfz9NdpB2U6ZQtlOrvubil2MN88LkaXHGyzEXjCdTfGxmm2zIulVrnxv8WEkRsCECgmEG6LFh9WbMbvSZysguRjf8sdzN8cje+eBqn6toSSa2Iw/DvWdwVtYtAcObdMdLTX0erQwUHv5FT3+FT35HT3+NLu0WnbPdnvDQ70q68U2ofstZRU+ZJFX35W0HmDKsIcy/BPaeylNEX/nO7Jj5Ss+BenJLzFBLCDW9y4VA0CIrAldrCT2kFntr1Fxg4W9nBmCAuxTI8sW2E1PWdzKTJ+aXxl2VwhSColoMvizOV+adIaB5qVVqPAZSXZvZujs5Cs3WO8TxhEu/lD27MHBYddeYMKy21UvGybvYrG5gntE/buI/bK3VcK7zr6N9aot6qPaUu9QYkwbQpn/3xKl7zuT9kE4xyi6hZJunUTwA6uuwXWX36zlqtZaWukgx2sgr9KO2grZfTQvK2dcS4udrC0ZXAIS9G07kDGP5T+u+wirsCu1B7TanUlL20OtfPtXqs8EpUQcDaj5Ngs0XH3GK+WUQObU6cvDrqQdhSIk1YW4RaR2jck3BtiUsNmu7aAVM8K6oP3eomM/c1tifuXxme8wVSb9PDkXx2dQcvJvDdc8zGTEexBYIkEwokVysAOBhTLCNQevWoV7qVpwJlVbL7da5VHohICM9rB0haL23FsB12hZgdD6djBgGJ9ARzC9bFffsllQ+QuGMjxQLN8zmssoayJ16hSq4ue0Qpm62puWb2eF1KFEhUocOPSEkIyRcj2upT2wKHChVvP3EL/uQrLELtzCsfeYEaCP1Qoc9ZIvyxn1lykh0CzBMoGyZqnarPKrFhaGGdWXO6Kiytr4hWrsTXFxeZmSqWKPMFZ7GBif0KJS7SDk7c1sYP5lsUhzDPZ/pg6o6c3lkUn+/bzWWMNa9pv3y512nGNdWld0cEmZTRXfOxQZY6G3bjtxqLSWOeh6bZ3Yv98Lq1jUcDN6SVifIxeKuP3VXDeQ5M36L8hkeTJ/fHLYxSdFp6k0PtFg5PmV4oGbfKl5KTaWtN8JDEQaCmBOuMndrCscTV0LG84qCnZ61CnHctqQXyeQMZIhQSN2sEg1QLLtoPuJuq4ROzgmMVkCIdwkkeNvcde++gaqVqfJAyy4ZKxfpW28s7r4vZvcQmhCWQvQwOFSAKLECizgnVlFl3CZOK82YsFKkE4WUIgeINxyopw8A8lP/brtFtdKX9qx1l8KTh+FbQ55AlgB6f2hHBGT03ZogTBimUGt/pVCBLqZylLKR2wg2Vw5ouvNhnTZRZ1i0yc2UFve1JxShBOlhBoyg6qRnqSIi2q4C92MEDBIQwo6geSvsVIVIasDplw2gchDdqJILPBQNWIUq+YxSWEcuoQDolXHKgzmbZilfLFLWr2JNH312mNap+FT6f7lEndPnmOQl8GjKxiPwpntM24ef6onEB7q4zdpFdf0Mcnxq6g5Oc9OvtivcsgreUB2qtEXfbYG8znckkW/Xn0VdcvKoL8m0gAOzilVeqM0tjBKRArD9chXClgiQd3xQ76YSDfjyfRZu2gvfvam6Cl28HYRKvDhKUx2MHJJrI9HMI8k3li6pz88fXfPGVsV57CodyusePTd/lVthKXX0qzJRSia7aIuaXVORHmFt5IxsZOw/J+U9GHfSa5cyG3wnNv8gP1Ohmf3f9JbGyxxFBmcjg0VoUrWPgO0mLxxO48gdCjKkg0dgJWlNGeQ4WD+ertYHuAjTUtRDc+vNZQnRNhrQpOuGFL0qTCDvoSY9sXh2fVZyY7mO82obGwgziEs/a9gvShPxUci6LyyTCNEZ4kmD9ddWDqyJKXUzMmc428+f5hIZ+alV12snwPX3aJs8pf5xk32dXC/dFQBVlETQaGeG8g/bOCig+OX0ifCdg8pPtEoZ/0K+wnzl0c6+G9x4wVxAPMgGW3JoGap38+2TrPypp1W3mywvN3eXYwUz/sYAbITLv5Hj5T9hUk3pwzLti7UGvsYECx+gAO4ULMFzzz89k350RdiEvTmfPWcUmmcXyx3HQVGpGX59CI2MWF5Hvy4jIbl7Cskytd+pJR2J6USLey259+NWn+aIhRwC8N1QMVJ/q2oPtahf9EoQ6NC0gL0t/CyOi4BUOa2Btckiv4yCsflSmd3S0jsODpn8++rFO15dzz4z92cKOaNN+TN0o9r8yyTq4SOxgXN7aJk1da2MEN6Sc4hDM3xKOuvlZ58kPzzIKKMuQHlPh0Ksqxo3F5/jVNoy6FJ8eiAoB10hRkW3JUvspLLrCu+HynrZtzheladB4FbzCDJzxkr7uqmc/WZ1IW7Y57vXf84qcHlX5J3mCRJsRtAwHs4Ca0Yt4oYAfX1S7YwULy3vDYI/O2JX/H1qgwj09ach22mB0cl4cdHLNIQziEKYlN/ZsfZVp0abtKqDVNY62RaJV6TysrX69pOVZ6PN8/V1p8vcJWecqUlSUvTof0Kd5CleMXz2QShElCH6/+0HiXqPAGp7ZvWX0ztWAXAosQyPdDOl4hz/zgUOgiYgcL6c0dme+fc4taXsZVnjJlZWXsYKYfYgeX1/p1JOMQ1qG0WWkKh56y02+zVF+tNjVN42qVmqG0vP4zZF5J0sKuuJKSN6cQWbRkAaaMWbj28uejfjMGT3qHZTMhsK7K+PWieW9wpmaNEzMKraspd7DcuOOF6tMDA4oQyNuRMEyFNJscyOu/adoWdsVNU3LJ+kyxg8FKBjWC+QuBcGjFAexgAI5DGFDMEND5v2mGp2xI2jQ9Z6C8hKSFpmWjrGOhhksg0YDIsi7XgOjWiUgXxOQVL+xd/iP1PnHZPVHvSSa+Zl5uFGMSnOMpy1r47hnfqZKXzUQZ88EF2zRkZ9jJs92+GDX3pjV06IEZ2pumZ0a9Fe8WWpnCkWrFioXiCjUMRzcqUNblNkrJFSmziB1M76tmVMUOZoAsdReHcKl41y+8YrTCRvrmqbA9K7ORFTqsvw8VaVDRr4qSb1VcWvdo8q/Sb6u+AxpOQyXTcprCBwgzq0ZFUyn9fU3/xm0l8O8jzYPWFwjDY7MKpconCeOOlzmUF1U/RqJCva654pr6GUkJgWUQqOjboaMuo9wWyYyHgoza2MEMkLBb0a9Cmm0NpHWP7GBlVafYwfSdNMEO5ntdA3bQ9XL9eOUL7XZar8rK1DsoUWF4aYUdxCGs17BRqmvPFF/fVIynIXe+i4dDqw/M2u9Dz169qusqsU6bBt3qNO5MAoPkTQvM2nM2Tf+59Sm0H1lpJfYxEx1bx8whCQwxeROYLa5oP5yqzhu0FMNJ1aM3yiRFNd6msS0s0pG4dhPADra7/WbRfiazhR2cBW0r09YyFsGGTVYxE71qOzg2raZWqEh4biLETGo9/1677CAO4fwtHXLWHy7zKeuMnqGg9QYWPFXCRep6a7G80vONu7yy1iV5wT6wLrUbL7eaQ34VaNm7ZPza0dgozqSqjOuko2e5g8XNd8hg9pQs/uDETIWSGAKFBPL9rTCZIvMpsYNlrFoXn2/c1lVhqsLV4//U7FuToJrD5tpBZyS96VyBHVRBwShveNPjEM7TQN63kQ3zY19szGYdDTPpY1HzaLbBearHDim+9R7jBjfOdNWmNt90EetLIeXX0LuqjUD50fIjtQi6mlbJyPuQS2pcj/3KK66qpTeJ2kbAn1PYwZnabeq5toaRaqYK7Hbiqc23yXj8gLxqDatsUZWrVJ1vai2m2sG8hKU2bivsIA5hvldMi9GDs0lX1TM/uZuduiuSv+CaJjIcz+dNywpJtjZQeDZiHdfe3oXtsnatNlGBSQs2uTflHqGWhlZvkqZxxc8lFr45Jj9XqKGjq21KydXFLnSUk3chfBueeWwH7YnWrOXCDs7bfIXjLafSvDgby1fYLo1J3yZBk5Zvcm+KNWrSDqbX4RtgB1vTujiE8zRVNDrbi+Zdx0u7vQzjYrYwo1DW0OqWSlpUJuVW7haOwhH/raz0plSqEP6mKLcZeiQnY3RKRsEJFdVp5c6VHZ1IOrkTHiPMrSzVwBPJU9CbwNQQRmKiZFFsvII0iiYIgVoE4nEYO1gL2byJCofimP+8gsk3nUAh/OnZdilF/qK02OS4hWArsoMF/IuVwg4GVDiEAUXdwFVXXGnGL7rqcr3M9lczcMRFB6XzJ2Q4tH2BPGdMY+OtnIfceBHbITA2MtYP07elqXaaOYnrGKdUvF4QGh+tDitvJrWWJ2ReFVMgweXJZCxItoQoTsklQN0gkdjBtTdGfojmpGu8UfKQGy9iOwTG1m0mO5i70VnFI9jBUBx2sIrXjMdwCGcE5pLLJXPDhC60km7pB2L9huEjvggLfXeewurl2XEvMWAPtDCNAcVMgTzJmbLvdmJ/0pvNCq5g3tqVvWCmPrp4nKmfKx6R4lzLaHHOvpjwtoZTOziun2/3uH/GvQ47OCa1nFD+XOZMnI90nuR8cnYy12x2cL5hQWWYoZ2db1mWZbR4684+HMLZO9Q4hz0u6KfmqjtTvgvOdw6MS64XKvQSlXUXphMLW6R152e9dm4gVSGuBuRum4jSEzd6o9rE6R6fa+GA9xLDbiGkSms3ViMvJDTl+PQfhyaWNhSWu3hk5iy78rIzi8tEwiYTwA5ubOuE0SDWMHOGxod2PFyIa8eZzFT9VdpBm5dJp2TySoamHFu/cSifvPmYzFnWCjuIQzhPP1BXC13LB+LLvjoS85dxyjW+yqsjYoE0QfkFZDSWdVZ0ixQcxohYSOa8jQ+1NFxYzZbWZZvULjzBCyNVa8UXjhLukM7gsnwGLBwd2p5tIWbZw8z2nU0eIL95AupUwZRgB/N8ZorBDs6Eq07iaNCrk5w06yRQZs8q7GB4ur5M79AB8nZw2c8NttQO4hCW9aVa8epwvuHtAq2sR9eSZIny138Li6xd9voShkuKjAqL88wILNsNo0YuwTyvAMkJaTiiXNuGC0JcfQLVfTXYhnCCK6BT25/d8TkuCxe/Zm2qwSvUMJTijsbiC0aYQglzR4aazi2BjG0kgB1cvNWwgzMxxA7OhGs1iWvawaDMuuxgUGBJgfbaQRzCmbvElZdfqTxhPAq2cGZBNTJMXtslGSYu8WoIaWmSQgNZPeI0XdPJV4IUSV/NmR86W5EKxNUisNTztJYGqQeoxO4Uznbw8NhhTWkzJUsLW9bgUXYiaAXRFSwZnampWpIYO7iahkrP3InSsIMTONipTWAT7GBQdl12cHkXVG23gziEoXPOH/DnWBi4rU9owI6cuSg4fykhZ17asq7yQpEbEwiQvUartYsFFBYfWUJHWVxUgX5ENUwg9721IvmhTTMHw5nrA42ctrMImSVtRvXK3cL6Rg+TVGbm4LYQwA6urCWxgytDTUGTBGS7zI5keuBkmmSv0C7EKRu0g7HYaWHsYCkhHMJSNKUH7EvPyVkR0vir+XACWCB2VlzH970/kzJIWCTgJWckLKvXZ4pZ625+VIqpr1W1uoX7/lA3NekWIODOycJzZTahiYjJmz5BROGH4O1ktG/EJ1sIxAtE04PFf2NPdKY+U/3kfXFhtWPDiJfJgTeYAbKFu9jBjWlU7ODGNEULFFmzHcyulDFis9pB3+HDk4F1oGMH61DCIaxDKZum3xudnIwjw4VefKZNXCq5FOrE4UouBLyUicRjwQuFCq98w5XoQqI3OHPGNIam2WCVUW11BOIzdNmldrszGawCdcryF57acX6fYOhu5Q7TJ2FDIE45d7hwyMr7gUtdBzu38mRshAB2sBGMyxCCHVwG1a2RuR12cGpzYAenIsokwCHMAKm1mxlt/a73PYKnF18wKRziCwsoPBpLKMw1R+TUS8lZPcapAmsqOWu5NcVmWkq5cBFrossky5PMJCjc3XHaeXfIn9Q6a6aSKTuzwplS/z1pw8o7sFNHp7hlywaljCsYKv6sR78wzk54mwhkxgS/ix1cpInD2b2IkHzeTEspwdTxJy+EGBHIk6yDZcdpB3OQstJNSrNvi9jBVJRapO5Jgx0M0MoCOIRlZErjz1x2hU5v68qTSWJzqCPexyu7fprMWry3Mi8xLr7sMjROs4xwRbkZzguWXjig7/h4XYG0EFdF+syhiuzrYq7TaoGzMumMFfUK44JfIhp17FHGa4oOZbDZbuFRlRviQyBk9lOCftePHiFNfCiknylQBi1Tqdj263OL09/LNJMSJN4YAtjBZTRFOGHzwrGDeSYri6ka8GsoUZG9bXbQd8Okn1bUq8wOqr6ZKk/2+cm9EjuYIjdlshk6ndjYbYId1Md52mIHcQjTrjXjX98L82N0OEN8p1/s6rNAJ9+/MwfKrtUyyVq6mz/h89gXrFpotQo5mVGsImXFoToFZbI3Um5G5tTdOfScKjOToLCI1VS22bMy9MaeaZ+9XRmOxiepjwyHPBkt6Yy9qYCrbOFoSKBAdZr8GRTnVVi6FQ4s/lAmsd/N+IHaDdWRH+jFGY8QWyiFyJYTwA6urAHzZ3Hj51bhmJypYCNDdJ2CllFuRubU3Tn0nCozk6CwiEYgZwrK7y5gBwu6XoiqtoOJRXA19FlCRqdhdzjqbIcdDB9p7amGk5XMt8WGxOAQzt8Q9Zs4P5TPX2pRzvzFXHz1WZSj3XF5nvXbYu6aFw7cc0urn3Fd5dbXsMGUmcouzy4uYAsnqxt63tR7gCHlpIB4T36dUhUmLFTYnwgVa2ZkXCs2eXEhb80RI+MHSnhsvL0JVJkygdaUo6H7U6ECh1pPoLC7FtaqsjMW5pgtEjtYvy1mIxulzgzR0ZHlBtdV7nJrVSI9U9mtsoMlVY6j/f3NwjfNrNIOarwqO6Hq2EHVyKxq8t+yB7+Y3/xhHMJ52KmZyzqKF5c5gZV4xd0hbxoTxZavSFnR1aBrXo+WCcnjrW6gMjnEbxSB2C5mzqnF9Sw0LdVig/uUTTZjb8t316zA3L6yeIXj9TC5VLZaRjONShmjyyebNabK/umGbur2OT/Qpgf9O1F1Ut9wzcfMWhbp20IAO1jRUtjBCjgcmolAPJhjB1dmB/NWfTY7aPd3zdR3h8O22MFCJ3ymvrqLib/lZV9T1FdsbbT/t7FQ7Eqx8l9e8+r0+aN5CXVi8nJCTJ3s+TTuAtrcX/8vn4CYdhFwboZ5HrF13NgqjC1HkQHPjx6hImZAZtkG6U1M/xJRDycWovPI5HkXbRbJSqta+H8hn+YD7Z/OKjs/5Xrq1piVZg7ocDQa2pRgbzSUCewMhqOBdkNWAttGADsYjFQ+MF9j5+WEmHkF2ikY/s0nhFybQ8BbQP+7OVqVaVLHDjZiIurawTJFK+PnsIMyhb3RoOPsoCxjW+wgM4SVHaHk4N/6zNfo37nzD3zgwx+859y9SvX297zrx97yqz65RvBMPhvTG+n1GblL2M0rv4RCZhNZqNKsM4qF+LPtNJtepF4bgXA2FXlbM2ilrjVrR5pBehNJMw8HVk8PZgpMXESL1bRh8b0/b7Djmc+xCY/EmQfoNzsbXdgFLKT//XLZ5FLFR+m5Rh0YPfNjnu3z8bt9BLCDq2xT7OAqabeirO2wg3Uuw5qyg3YPs6hp69rBxNKZ1auyg970KpHZRJXXGjuIQ1jUO+rFnbn8imc/9Saf9mUvfOk/+eLXykXU7gc+8sF7zt779pvfpfCPv+VX1SXUdcJorh5ZeMlVr8z2pYqvNadqX59M4BlkznFlXzw0BIkENp7A4hZRHWmOnpMBU8ekZbJU7KpbZgRmzKHyJhanXEqcIFCKk2uiL6QpPO+EJajhEbknDhWX3u+UdZSuku4eVez5SUjtuvLMdBYWHCtBuP0EsIN12hA7WIfS2tPkx961q1RHgTDQzn2HFDsozoV2MHlIXj3D7ny6PwrZ33I7KFNoV6jOgCplyGUyNnrDIWyyeWQaJe7ZTzEvUS6ifv/JF5mXePb82ZtvfZ/6j7zEH37z632ReZfGx1f/hovX+bJXC1/86ExmL19cRfbCczWWkAcSWMXJpobTM74gYbg+Lji2YVEVtViBpqsH5Yfcuc3h8pgk5iQUMKmiPD3964ejmcCMrTieP0wz2vN95VvsE/pUsfFTjM6g4OWpTV2zSrS5fPrjn1a0jHItvf3zJjLdVcInPO6JXjK/u0MAO1hhyOp0g4rs2ME6AEOadCAMEdMDc2QpE4odDGSydjAccIE57GCFk9WMHQz9wJk2U9OsWzzDWGoHZR+t6Z3xtKvQ9tjBRG3XKPysiEAykajlpmfvvev+ez5yz+3BS7RelJ0eyGo1k58zVVpW+uz77sxc/dBnik61jpnazIQuk5fdBgksu7tMul21FK/uGzqP1M8Ho95g2LXfQW/U6duw3+vJ1PX2Rv294X5/tNcfntof7vUGp/onB/3h6f7JJfuDve7gVG9weu/kdO/kVH9w0Ds56AxO7Q32OkMd2u8N9zujftf+mSjr1aawG5qT70nIXnr7p1//z0/uyWbp2QmpcdKRVt2TUe9Y6g17ChwO+sdD/9s9OulfOO4fjfqmuR7xczzGVlO7wfi5Q96GmfGbOOTsm3uFjNk4I2J+oFn6aNdOSUmztBb4zf/3L53Ijf5JvhgphdlWSAA72BRs7GBTJFcsBzu4Gjt4NOwfDcwOXjzpHw7L7aBv/tQO7KAdZIZwxSOAFZfcQE2XmyrGLze1icRb3qddTSS+785b3njrO9Oeabnm26ovc2OZ87mO7iKq7rA23xVXxcV94Z3UCutYWMf6iGJchBchUNix63ajGgX7nlbRc2rIKE9SqH2U3L47lNtiZbyAolTjbErjk8kbLNxiLULY3DRN3SXuZEE+57Apjb0aVIcFSn/GmphrlwhzkW7fRWhFqNKbE2jZOn6BqJxRi7DzyuXTj0tm7x0djW76qE8u0IAoCDgC2MGZOkI8gGQyYgczQNqymwy1k+qOR+PJ+Dn2/Fhe0XPmkDnOUqj9+LB9fCi/xcp4AUWpxvmUxidbwA6OV3oGudjBgCIO4BDGNNYZlnXUvxuvv0FKvOxFL/WqjJebuvfW/Ei63FRHp52MM9cl4xcV+k55oTq90wvI/EGLqT5anGcytkxCPLLEOfLWscJFNA2j6+EgJ0MjxBNYHoFMl662E3XUmMkc+m7QaLtnKpRVWYfnq2N49s+/Ws1byokZP1uJap6hitTpkDmkSB1wjp35bS7NWBOXJ/X6dHLouP2zyUCX0WLM23P7dva7Xf21o25Poi3/aHTDI59gedggUJvAttpBAfBnUG0S2YRl2bGDWVIt33cD7bgO89mIcf6045X1kzilwttoB62KteygmbCEx27aQVfrspEmIcOfDSIwXmZz7t677rPlprGXWKho2sMLD06PdNmnDEp5HyzIXWXnqjnkSbdqFzEonw806jDkxRNTTGBK/yvONI6t3zGUp7CJZSbl5OhdnbZkdKjfaMlob9jrd/a0ZHRvtN8f6PdAv251qBaIhiWjl+yfnOomS0b3O1pBaktG97uDvZIlo9LEnzve3/Nh/eqfdLCj6XpRJZBnqGWichSPnG7H6ZLR40HvaGALRw+Pu37JqK+IP2FNpju95cf5WvtSbPbPDimt6mwY7Rari0vutfpdu3bw+d3nJYKL6dxB5fpP3/Kmxz32Rsu/2RtLRje7fQq0W70dlBLW3ys37GAlHg4uSmBK/5smHjt4dNIzE2nPTdh9Up2wZuvM2AV2/n6n7e6gHWSGMPSDdgSqltnovTVuIvFH35K8t8ZXSYOI9fjcVhSXS2TXx9qStGUWUf5VhS0sEJpGlQlMj0/8LbxSj1Pkq1k2Aua1rekiVis8VcNY2/aGqyHUrNdMrEJfnc8iqmOU9YS8ti5tKNCO16jvRHovc2qJhXXJ5PLeYF7JghjzV7Mio7lBJ9gZP9f5ncL6cdZQNVTQFoUmCZLqyAm247Znv12VYFnMDzSkbtowsag6ruLd/VU9BqmjN9348lZ4gwUkidp4Aqu3g0IShqyyAQE76DtOALXx/WghBcu6wUxCZ2KVjMvJVdlM5VhiG7SzJqJUiEsbCnTZ/cVgaQ4rIX9waomFGmVyzWoHMzKDHTT75dSUAXM3MN2+fsygJervsh3EIXT9o+U/E8ts4g9guM8k3nXf3R+5546Ml6gah3MmOUkcBH++lPEIg1d+KJzDFuaFlJXr4wvTB5UK88ZV8wkyA03IlXcRdaimlzgWMoYa4moGctfyNfMtJ1kh6maLyhdR3ZS+9NA/Q++tqZV6QlnT5yW4tElRXs9sR9LBaRoEVfPylTURmzsmyyeHKr/pnTF664zivW1Tsrh89V5fnJwyJTgZ9S1gkbY7dB/IHZlRtTui3aF+rASToID9M3toyd2u9/QshfatvGi9qIQ64ykRilUZMqtjP1A7o9HAxT/uUawXdQj5WRUB7GAZ6ezwZaatOK3O3fwB7GCeSVMx3hDE0rCDorFkO6g1Ntapndkrs4N22AyibbtiB3EIfXtv22+dG6h6b81vf+gdVvPIOARrEExIckZMEvJjljsUciTXhRmLItlB1KSMZvZmHU8zykRVL9AnUxefYlbrWCC3ICq5oC84MldUoVHJs5pL9rIyxeoV6h8XHLrluP/Fh4vCavrq5i7KNBGX6TwTx9KdoHl9xZTVOXiWVV5f3haqsv613eaduVWcFjAzJX+vo8WiWjI67PROOj3Fn3T67rd3PNo7GXX1rtGTYfdk2BnadJ/NIsqVM2Ut4AyeJKkAd8wk2iHbVJKvr/1x/2x3JDH2AKJ+JvxA8wydaPMYdbzzkhe83Ephg8D6CKzKDtpJEzZ/XmRsB3Yw8KkMYAeTMdhTCtakDJrG4TRlGpr2V2P31tvBgVnDUjso82RWzIyYfp2dM3tmYYvUscQOmp3bKTuIQzjt7Nmi45kbqL5mmffW/NhbfjWpcTpmeFvnzx0dcueQJfGnTJI4+iNzuGJbGBVuwVix6eNpqI+TklY6I3JiN1O7iWPpVXImcvW7MYTVl754ibH+1Y0YGnDioqxEAxvf66QryV4QXXkB44uq1t/L1Kyg3szpPzvhY5Q3aKo6DvTPHC4LyN9TJ9RNTgXsSxLuAULNCurRQbl/ijn0YfvtHg86h8fupTDOrTPr5x5BNNNnLwt1vp1bL6pdd+rqryvaPEgzh/affEhnJqWbneBO25FU0QGLt03JFHDJh896/CtuevpzCnARBYF1E8AO5lvATuloqzNIYgcjYMsKbpUdrLSVnmBTdlA3SU/01aVSOygjNrRH8WVizXwV20GpZCZ4l+wgDuGyzuS2yM1YR/8BDCn/gcnlpmMLkVoOXSmmQZ1T4dq1uN7KPk6dJnFxUzKmaef86xWzi9p6W0bJca3rZVeqajOZEePvJWci2c0QiHtXRVOGNq7uUr6Jp7aspFXLiZV0byUJ5duROK/K6tkUoG1+ok+H/QnRt0PjTWGf0dlF+w6gvlkvuQNDoJ41ksunJAr5Dw+6+6DyAM0V1KeW5B8qIM/Q2UIFbHrweDA6ORkYNzlt9vIXd8dGQj3W4MtZOa4wuYgSZKkti6yhV8n7gSrb+YHmBjqBIZcSD4dWS8v4gme9eFwrQhDYeALYwbiJsIMxjQ0Jt94O2m1Ebwft1qdRndcOHplR0rN+pXZQDuFxuR3s6KEJs4Nm6bCDcffGIYxpEDYCMo36fXbhZxIz761JrwbtcjM9sxRwl5zxJbHiFKlLRQvEWyZjfKjBsJ38/np3RqEZhad6ETOKF5AspZoSRLhmyi1LFoxiRYN6NNVk1bLTWrMUcdBhzNbdagyNUqibTQC6TmjOnn2A3oyiJvnSicHRnvsqvXcFXZrEOVR15BAe28SgZVSfSYyqnx4c2RfnzSE0+9d3htDmCfVPrtzx8ejwojd7yiiXTVTMYfMd24SZUU5ugjrT6FZ+yrNLK+nrYm+XMdvpPD79KJuk2X9uAauXqF+X7Lkf+9kvfN4njeEQgkALCWAHQ6P54SLsThs5Q8K6gXntoN5iVbeILUuXDs92566sav5ANSG17DSGy7CDpvLcdlALU/SFwxNneSRHNbBlMrPYwXDj0nds7GDoQtZbkquDEEcAAvUIaLlpOpFoH8D44fQziWG0KhvoMwYmLi3kjSPnDlcMl/PLrB5i55a7kozBaVlJaUsvpE77VjfXNHOYVEE9VvZVX5uQlyW/aziQmdwzi9vvyknr73fs6xH7w/2+/XOfnbAPSxz0BpccnOjbEvrghL4/od/Te9odnuqd6NBB92SvOzxQ9q68QX2CQsLMTKa/3n80BU5Gw+Ph6MQCHb012/w9TQZ2+kfDPd0HPeqYK6gZQnmDh8O+rKO+NmHPDZ50Di+ODh887o/kdpoX6VZ3Ol/OzkD1hfFkoEU4O5niMrPg/EB5rc5EJIndjvl+di0iB9DSuF2bTFRkd/R3v/AHXv2qLzS9W7Lx2YmWNNSGqrn5dlDnfHpeN8aw5sjZWHmNCsIOZnDWbM2G7OCxjOMidvB42NE90Ao76G6PYgczjTxlV3aQGcIpjDhcQUD3UAsnEu8/d/aes/e+/eZ3Ke8PvfnXgoTgHxaOPnZJavMV7k/IkwsUeYyWpXGDlys5ibAL5ZUVVqbEvPGhCeYVsLx889zu9Z2hus/4/lTWYkmvKzu8WHXtKT+7FZp0aXUblSPfLCisQ7o0sUlCd9dTiYM3aDHmHkqE3Dm5ne6zEObNmVBV3D09aG+Rsa8j2rtkbLGoIuUW6lfJjk9GFx4cKJ25eq6ectyU04IGzjw5/fhDKQCb5dMUpP3KX9SvRZjaSqaSwr1V25cki3HOobuPLHWf8+RXvvDjmR5crN+Qu1UENsQOhlFlNfB0/mMHl4B6B+ygu1pbxA7KHlXbQT00UWEHrdVktrCDue6LQ5hDQsQCBGQa9e/G62+QjJe96KX69Q8lnj1/9uZb3qddeYmaSHRXmONivCWLB8LYb7HzNtqWavbaa+EiQi0NyvUYaz7THVzfnao7RpCdej7jshSqf3ET98wJEVU7SZneP/QJ+2bRks1WkLokzgmUF9j1/zT76By4Xrc7tI8jOT3NA3Tun7w+H/CWT28Z9TGiYfEno4sPDweaN5QbZ28UNQfOeYPeELpdfxvFCTbfzxJaIXbMH3feYGI7bcGozQHaFKE28zNtPlC/5rY69dQEL/q4l9zwuBsVwQaBnSWAHdzZpl+44jtgB6NVqPPZQXc7Ezu4cF/LCbDLEBn3XDwREFgiAa2xkXRbbnr23rvun1hu6ksNV+2ha/qL/vzl+NTO21IfT/XKa56vbD7NEpttTaJncg6r3cJQg9DBQowPlPEUeXVFLRm1f/LEypeMHuwN93sD/dOqmFO949MHQ60U1RpR7Z7uHZ/SgtKurRc9rTTdwZ5Wk/Zsvehet6N/8qzcwtGeHMI9e1BCltPOgOORPLvh0Wh0Ydi9MOzr5WlaEnM42reFMfYA4Z4WiB7ab/ro4LB38eHR+bsvyrfb6wz29WpSm8sbj/SqvutL5tfZBYj+k6unonwPU0I5hza7qTh7v6j3A+UPKqzIpEUsvYUD8+c85fO+/1//uGVr1caS0VY11/Yoix2c2pYakPJjsh+l4rz5NPHR7Qgvww6KTKEpLOPpbMX67eDFUffhAXaw4X4tO2idYXyZ0LB8xEFgBgKyjvFEoj6T+MZb31mW3y5L3eWo/Tb0fMRMA65dQ5sKhcOpHWh8y1vBxotYnsAyAzNTiTM1UPBSKoooa7y8toIv/2iYOoSDgXy3vt2k6NuLXsqeITzVPz69P4NDKG9wT9c/tpS/p2nBnpw16+R6uag5hBfNIew9LIdwuH9xtKd/RyM9Q6iPDdrymOOBPUM4GNpnCY+PRufuPDq+eCJnb79z0u/oLaO2mTl3/5sjl/iHNhWoZPr1h1VP69qK0MP7tuOWiQY/MJx47i6vhxww/tQPvL2N04PYQdc7+NkIAgvYQekfzsX56zLTMKuhQmU2ZYLrKG0jVWu3vGWZoyozNRB2EDtYs4+5C4+aaUkGgSUTyC+zUYGJdXRvN/2l3/2137rtT70Wqd0z42DXr/Zrm7u4Te4pZqzUTMOoE1b5454HS4utTJkezOiTRtf9W2hL2mIdK/QsrFchFA+wZjuat+auVQpF+UjfZ9K+NE4obWOttBt61zjRLCF7Fi+6VvMlepkVYkIC80XTf/6FonIF9U+eoaYrvUPoniG0RaSDwej8vcd6l4wka71or2fv2DYPb2Ben5aeWlvYo3/uQs7CtuPT+MlAmxDU/3ow0dU7AW4phcWUSn6d6uGC46v+f69rozdYwZ9DEFg9gQXsoJS1k9v+2E2eFdlBFVdzTHaqScX8iOuP1PqNR+aQIal22N/UQIWehfUqrAd2UGYOO1jYNxaM5BnCBQGSfbkEYut454du+5U/fevjHnP0rEeefXR/bzTcPzo+uP947/fuPyUl7PrWtmARx1YqPeSOl/yE69qS47NFF5ZYx2rOaizLrEiF4ZmtJkWp3TXHzEa9jHBe1bJKeV0aN4dFVUy70cSxqMoK+iuviQRlO/Ym0lB9/44Z5ZYMfWawepMjZzN5cs/cjJ1z//ry/WzJ6HBfy0S9QyiHTms/7dHBUe+hc0eHD8n5M3fO3gszHHbN+1ONtGtLQRXvPED9WuGKcjEu1kqyBIqXzibEcrpTye6AWHyg4HdNhL5S85S/0a43i3q1+YVAKwhgB6ubqcxk5I1LtZyZjtpoOB4O62aNh804T17Vskr5XBtlB4NRiGuUCzdoB2XpsIM5wAtH4BAujBABqyKgQUcXt7rk7feOr7/ijjMH3TN7p/b3un9j1PnQQ/s3P3zpn9135uxDvec86gVv+LBNJNqlr10H24WsG7tXpahZiboFeiWDZtVOY313scyWLAgio21Qu06gIm8GV0bJwroEFNXEvGK+6EwpQWffVPVMmsukDC6P9NTK0czmtfVen2TaG0Q1TdcZ6el59/IYc8KkSQhbgvTlohlR3hvUM3zyBvXNCfP97HUy9tygwvoqvfcGLWxvhLE5AbmCD9+vxaIDO1VMSy0WHZhLqUKdV2n+nts1F9MO2GOB9utqpCyekvcDpaR09TFO7UTBNMblMW/w8//5N31vRnl2IQCBZRDADobBfyreQtuhXDYgLrBV2LKpUivypuNqIiOjZGFdAor12sG89cQOTu0JG5gAh3ADGwWVSgn0OoOjQf/CaF+X1RqDwjB05uCha0YXH3OF3r+x9/RrH/MfvuH/TZ7X/4h7b819d3/knjt+7C2/6sfiMM5ONQpBfqlCcx2Iy83YgEJ5wYTUGfQlIRiJvLS8UQk08olXFhMqGEqMscQaFulvrVSHTCglFh5KVKPUae4MW59LGaWAdPOSw6+P0VF5Vlq9ud8Z7umfvULGAuYlyle0yUMrWh+c0K/fnCtobqd5g+liUS3hlO8nh878QHML/WJROYc2PSjFtFj0ofuPDh8ejAb20hudLPakoMkwj0/dQn/NLbQFoeYH2jyhKy8BktbNYLoDaUUSrfyu673uR9GjzrOf+vl//2u+9YorziSJ+AMBCCyZAHawDuB0PCtIW2RHCpKtOCpYqFBuMjK7/U22g15hGQ3sYGi7NgZwCNvYajutsy5iD4d6eqp3nCx5Mxq6gNY0i/MKkktVrbFR/LOfclOANf4Ahnsi8e3veZfeW/NbH3qnZU+vb0NiRfhr5RDTYGAmyU6TnH5F2gRz4jgUpUhqOlH+TKZR9imUUlxAGptHmh4p/ptRI1NKsIux2DhLsP0VdQ8Fe+FBZhRvwQk64ZihkzMV7btgPrEUsNm/3lCfqvf+nl4lqknCvt4pah+jt5eLHnTsk/SaM7Rpw2h6MEjTW0GtU9v/9ujfsa0Xtblx/0+PTyhgE4M2SWjx5igORg/cfvHBew67JzZZ6D5TYdODzvOTGHmD8izNCdTmC5JH6prTCvIxnp4nE5Txu5bIqWTx6Y73Bnl00NiwQWCFBLCDZbCD7aiwBekQOJYRWxMfmx/tQ2oNiaGUEFkYqBBSmD6jRqaUYLNisXGWUK+Kuody57WDVvmpG3ZwKqINTIBDuIGNgkqlBA72NAuy9/DwlCYJL3SOjoZ6j6Iua/VmRbtitkt299BTWf74SYyXvfClPln83hp5iT/6ltfXHexLiokH6JIktaL9sBuuy2vkmTJOC47TrSBZMCQqJdY/P/SPbVKp61RD01ySfEGxJoV2Mc4SdA4VmWYRCyB4pfyBgN3vZhSI1XcFWXKRUdj/2u2Jjl4ZapOBcgv9lKC+AGFuYUdfmzBvcM9mCG3RaVyWmxuUEPVkq5/8Of076XaP3ecukvWinfEH6M0bdGtH9ejghXPHwyO9QkaLRZXZTSWOju17gclkoFVFh6w4w2S7Kl2bZxVaNujjPT+ls0opneXwPxbQStF/8U3fy9ygsWCDwAoJYAcrYbtxqjyFs4PFaYL5UO5gUxSObY0XHEbLCtNQrkLpkXxBSho0yZTldYizjFO6AdvlLa5pqkHpUX8g2AK/6xQIcakM93cOOyjLmNpBXcYF3c3EFNvBTufI3f3EDk6gb24Hh7A5lkhaMgFdI+vzbvrY2sODg/OjU6eHw0f0jt0US+eoM9Jb+P2VcekIV6Jexkv0E4lKa59JPHfvXffZZxJ/5M2vL8ldEB0P0P7weKgrSF4aNWsuP07nSw8FyH4Uwik0kN40ZnSI8yfWaOzOhHIaC8TFSWhQJraLZUYxmHbv7eR0Um8yPycXn0TkD3hlFO9LtwRpfivC5NmnA211qHl6thzUuYKDfrRMVC6iizdv0CYSXRZfpGbwlF3felBBZg7dStETW/FpVvDicO+oY58fdN8e1JLRxA/U9+g1VzAYdB68/eHDc4duYlDzfjK0eozwRKtFvY7e8fPtrxjfVQIZnyay8+Pa+1nR8b7pZgie/bTP/xf/FG/QNx2/EFgdAexgNWvsoOfjzeVq7GBa4mx2cK/r7aCenjDb57dKO6hXqfWxgwmpJfzBIVwCVEQujUBfj1D1hro4dpOEJxdGJ1oCp1c16vts8gaP3cs2Ml7EHLoky02fWrTc9Jb3SeDbb37XElxEXWmng6JTOnU3ZqtBYgYmRQURhcbSFzS+6PepJ3SRK2H7Y5csTR2UjJ00n3Lxhghq+0AscKxJVNOkIkE3V4UKiyidg/6ZsuJds3LpDVfFOzUSOg6KuMi7M1dQS0M1+6f1ot4VtIlB908+oc0NKuweI7QXybipbX1s0Bckr0+bZvOku8L+183+dXQHxPzATv/icN/1cH2Jvi99hloW6txQhc/feeHCvRe79voZ8wD1WtFuz2YjrXYJDVXAQr487wqGuvtIHQ4xSquwYtLcFvAQFPu1X/K613zeFzqV+YEABFZNYKvtYBbmeFDKHqna9yOrDWNFWzLiJaNbksIXNBmXjpipED8GpsO2twV2LCiZKdEMTFZiKmvev7HAsSZRTZOKpOUmKFITFu4DhvKlc9A/ROYDSuar7w85NcZ4rZQ57WByXSF9p9jBYU8v1j4cYQfzjdNMDA5hMxyRsgICj3rUdbqSPh52j076F4YHF4bHD2lPL+cYdS52ehfdMKFr6Pd/2Hy2xreJicQXvVQTiSpCy01tIvHsvXfdbxOJP1xvIlEjZzyme1X9as6gdp0BOs0YMo0DIXu+ICUqsSKWPaRPjEoqUlZk0tSNLYHPkqYfx6d2KBXR6N+gp6Tmq5MoU8Mi+koFXNU6BnNoAVdR/aT/RrpK08SgPL193fg0f0yPEYbpwYHeKCOG+ifXUcWZT5gWpmlACfHm0F4Cao8LJt+Q0K66tPvkoL1l+1Dv2nbvldFvqNwDtz18/3vOuicDrXQvXAtTnfW2VL52Tl+xmojxQvSrNJbA7ftI087cSWt4CzrX8rlP+xuf/emv/uQXv9yi2CAAgZUT2HY7OAHUj10TUSU7wQrEx0P22F6EBCGLtwI+3o+TIb2XEIZE7KAoFdpBxQudcM1hBz1z3Q8NN0ML7aC8QVsggx0MPbjpAA5h00SRt0wCusjWde7JsHs46MsD1DJRTYZo1kXDxMC/i1+XxOFCe5maeNnyEp9dMJF47v5z999z9r633/ynSpb3EoN1iRWUBfLDoo/UXjBXigmGLWSJj4bIfEAeQoicLMGig9ix/XNKhJTaGx8aB82vCGmcIUgLCanNmzBZXs8oOk3Z3N8gPDCJdLM6hgRen2DPvHekyERV/c1tQVQI5FPZdF8vWSbqXUFNCer9MckzhG7JqFlK92ZRX4IMnopV4/giZQi18lNflfDviTnWS0TtyUD9dgdDWyCqFynpzTF6klCHFJAQCdS/vYsX7vur+yTLWJtARcoRPZFD2HP3W1184gf6XJbQbSpaEkJ1xpV3cZKkGKFTQMt49PvaL/v3n/6przjDC0VTgPyFwFoIYAcD9jDmh5jCAHZQQ/1y7aBfIzOnHUwuU6rtoH2MHjtY2L8bisQhbAgkYlZCQKO/NwC6nrYLaJst0Sfa3FsW7XWKdpE9vq5diUqZQtKJxMcp/mUveol+07ebnrs5WW76pz/05l/L5PK7yQV6OBbVRN5IsHx2EZ/WM0SGTP5o2I0DVYe83yCxzrMIKUUzFBGCHrHSeAULnUPLmLoaltH7K1GNYsWaCicKOXGx2l6+1yekUQLznpLEDqn3qryb5OKtEj6Qau6zaC/NaZN+mkXzu+YZ2ltDbcmoxPsvEMox82tElctufNpcnrlYXoQ6rbstaq6g+rM6s+YDtSjUf1hC6Qf27U1NDNrXJlSS9Xk3Saj8J0fD9/7hvVLQ0XZrVu3xxaFeZKpCvUre6R3XPdE7mfhLq2WVNDKJK2jdQLv6J/X0+/xnvOrrv+5beZuo6wv8QGDNBDR2+fENO2ijlobCdFgLDeNHvLAbB6oOpXL8yB9SqpRQRAgmRbuB0+lgqdKCkoCLSkZZy+iifcY0ZfN/Y/mx2r6ksS1w+0rQjB10RsNXe0l2UPdDZR/9sxLYweb7jb0Gjw0C7SEw0IyJW3SXPIXl7iv50Ucr8dyFeBgAN6hWGS9x4r015ctN5SdoCxYmda9s3B1XMhr7faQzV/F1vhMyluKElv/krYXSejsXDkUFmaBg81RqcA6VJpgZbzFjtSOty1VZ7EgoIrDytZDUUBELJxY6AeTdp5BSCUItJNCH7ahVNWkaSZATqE6ovPqnVaP6dTH2CjUfH6oiCQNrPXVc/ecLlyuooN3UsIlu9+YYW/88tElC88fsw4N2+8P903OGfdcxzBv88Fvv04ng+FtFFNYpcNA92bd301hb2K9L4JrGVTblYse0mR5K4YIWNk9VSpq/Oux8/DNe9YnPf8mLXvhJNz7uRpeaHwhAYP0EsINqAw1WfgC19kiHNQV9pB21sW1iG6efiC7YiW2EDgeZFnZibdT1tiP5ayokhyy923MZgwXxSsZqR1oX6NBIVCgi1N3UdFuoiPaSupTcIVWCUAsJ9OGkuqu1g/bFXWeksIO+EZv9xSFslifSlkvAromH3f3e8KB3oiV5pzsnp92nvY/MG7QvvO319Sb/FmxyEaVl0XLTs+lEon0m8Y23vjNUZsK4pTuxdfH2xtKnY78f8YOETKDiqAQ4s6FiDGeB5bAyTAmze454+lcxLt60sIOWXUnN+fFHnBY+z4Qdd/GTP8EIxdHez4ljqsNOHUuSlumr5mIilfzRfIk+u69IQj35Y1ykTPhnNykcE9VUc4OOipJ4v16OnC0BVbwOqhvLc5MGI036WYqe3gVjU4L2YiR7TEILY/QWGd0EVQLz0NyCUmlifprt9o4Ph7e89ezxQ8cqRZq7XytdZ4EcQkNkKU1R/W9VU9jFWbW1+bAdd3OA/rjzA/V6U80HfsLzX/LUJz/t2Tc9x1KwQQACm0QAO+hbQ6OaxjZtZn18yHbcuKZAGuVNlaUr2iqOSkBaguX0Kb14L9vKVRJXZhRjiXfKDuqJiVXYQb9ixWyWNTl20PpZoxsOYaM4EbZkAro/tGeL1IenuseXdI9O904uddMgui4+0ENTLpxagyWrsgTx6UTiDZL9she91JeQ/0yijw9WbxxIjaJiEiuVi1Feb7e8kIpfL8HG3XRzpk9WVzGFVtBSWtHOMHs7mjGKls3fgwxK+9QuY1pO/NfKivddEdkY839qbLkyVZNElOlVIiOtSKJ5ksUldibQLdTs2UygmwxU9fUlwFQbgyinbmgrP+2frfm0ycORnoPtK5VZUblhWvPctUN6RkIOof2TyzdUQAl11DvWJlRKytE8ujj8wFvPHj147Du8ClE15FeK5enusTmlUWXiqiXRZkpNpiTq9oqpMbQpwRc845X4gWnL8RcCm0sAO6jPBfvmCUPdOKAx2I3rirEBWONbLkaRqeWZ0speQpxIGU2wSXZBNywnJRbFmAJOOZfBlLJs22UHVUezgNjBuKO0MIxD2MJG22GV9Qp+3Yo66A0u6R1f2j26xDxDG/sP7eX+em6q9LK+vcwmvMQX2ttN5SKqOv4ziW9/z7sUDtYxVNNZLLcX7KQ3WYpLY4KZVFze7AVRcSBNZlbWtlSmj3e2zqJNsmsLf9wrkxhF77S7AzKKPlkQlapmEdqSVN52+qii38S4ukN1nMO4lFjDItkW59PrVwXpV7fnDZg7ouLMELq5QecQTuiaPOHTVadVGntnjPcVtaxUTCTHfju2BFQfT/FPR2hiUHOJ7qFBW0Qqn1C/Xgd7Scywc3g4+tBb7z1+6ETvUxI9/bNynTZ7nRP7/KDb0usg0900T1vY5huTivh1oaPnP9PmAz/hBZ904w03+rz8QgACm0wAO4gdzPdP7CB2MN8rZorBIZwJF4nXSUCXtnqwqjfsnu7JFTx2S0a7p+xie3TgLo7lX9jqO7sW3uYtXm76she+VFVN31tz9uZb36ddeYk/9pZfzSAIWOQbeC/IEqSxE5EupzyNCQkaaycjdDSXxg/IFi/BwdfyyXxu80ysWNszh8h+fYyCSQlpGovRFny82OD5Q/nffJqQPZ9YMSmA7EEpFR8yV9DPqpkXZomltGb1knui6Z1R7xn6EpVKubTsU4n1Qu2LQ30IwtaRal8zciZNc4Yqpmvun4XdGlH5kMqS+oT2IhlbHuY8Z72t5vji8CNvu3vw4LG8Qfdtw1QbE6vPXZxM6mwKqHRFSpl4MlAxmg985Ste/cTHfyx+oDUnGwRaQkDnMnZQbYUdrOiw2MEAx8wfdjDgqAzgEFbi4eCGEdC5fdAfnuqfaMnoqZ5mBW0BnpbK2byHe/2Un3XZMK1XoU71ROJd9939kXvuCF6iMGozn8e5Z4l+PjZVVoecpzbe96HEH0qj47/OhzIPRFvIa5FRcT67i0hKzziHGrt9lqBe0CtxtJL5MF+O/UbGT4JNgdgJjI5a4viQ7ZdsoVAdl4QwsZYkNx2Gycfo7THA5KUy8sqCPPcuUC0WtacHj0cdvWBmqJePejxduYJuOajdwtCLZNyv/EC5heYcdtzTg7bKVFMB9hi9eZUjPTd4x9vu7jx0ZB+4d0/MhgcURUyfuPCFp5or18hOB2cOvf76fSF+YGghAhBoJwGd1NjBwqbDDjos2EFZbd9BsIOFJ0pxJA5hMRdiN5CArqXlB+719nRt7dXTH11w24Wvu4Z26+vcQrgN1H7lKsU3UH3h+YnE9911y+986B3+6IRz6KL8kOpdOJ9Gv/lkwfcLaTKBkMB7ejrqnb3g+AWx5hyaC2NuXRqyxvaFJvq4eUXLkt72s+OJZ+v9reSQ0mQ2eYPKVdMnVF4l9qWkYZtnM5luhk9yvCg91OrWbZovaynNwbOXwWjqTyL05Uz5hPpASrdnx02mUtunI7RSVL99uW3yD+X4+e9J2KygPW3o16bqbUmSNxhcHNz9trtHDx16b1CfGXSrQz0qmx701IyVeYDKq5B7IY19NOKVH3PDkz7lxS/nJTHWPGwQaDMBjSHYwfoNuCN2cBKIs0PO1kzGJ3ubZgd1N9/eGoodLGytFUbiEK4QNkUtTGC/P9jvDXT9rUvto9HgUFfa9n22zoXR3uFoz5686ux552HhorZTQOYGqq/kxHtrbn7Xj08uN9UwnWERHDwf7/20bBpz0qo272cGD6rCOXRuljlyXqLa15cY+XXOzzI9lSS1hYWFO29QoqK8E+nCIQV0wBdqv3KxtM7TveVF8b2+vhNhk43y7Ow7EyrV/XrF5PrJqbMi9BVdxY98GpvNVjo3a2cvZbMnDN1KUZOvOUPz3yygWUHFWyn21lCr9PDBi7f+nzvl46nn73dODvSsoC0ZVec3iZZA/9wEoWIkTXLcZODn6qMRvCxUfNggsE0EsIMLtub22UGN/IkxbaEd9F9d8uYPO7hg314kOw7hIvTIu2oCe73hqd5A0yO2mm7UPdIl+7Bz1OleHPX16bajob2Qw1+Ur1qzNpeXsY7/5Ism3lsTLzf1zvZUwubdpY5ZACNzNXbXQqwL+PjgZwbn0Ns4P/klX0ki9b+V7gqIinCOkY6Y8zUpenLPVhY7b1ApJ4+M9/wcoBfukzlX0DmHcryc76cfmxL0n2By9bKiEwdPCzl7cv4UM3TzgDre67kXjMql9A6h1ou6gN47Kr9P7p/fNUfOJh9t03pUzRmejDrnP/TAg395p2I0N6gPq+irEj1ND+pWiKEwx0/pLaA3hTq38EXPxA90BPmBwJYSwA4uo2GxgzHVldlBGa9Bt28mDDsYN8A6wjiE66BOmXMR0LX+8UBv4Og9PDp4aHSwrzeODi7u9ewlHA8NTx3bR73dtMxcwskUE5Bp1G71ZxJ/5M2v91m8AyMHSwHvZkWu2lhqeshinA81PqSQFzIRlSZznmTiLcrnM89L6TUB5/LoJ3Xe3DSdcwtNjk+XSlQ2m7Jz7pbCivb6+ONOUppUWVN3UUVYecmvOWCWyH1+0P0dqy0dBkN5ifYBwkFnpBsWfZvis3+WUof1qKucPHmL0sR5k0pvGruDSm+eoXl3Vl39f9A91qfnH7rrgrxBcyltbnCguUF9stDfEBkM3bJS98UIafiiZ3zuJ76Al4U63PxAYHsJ6GTHDq6mebGDnnNqAb01bNgOyiDq/qq3fdjB1XTsslJwCMvIEL9xBK655trzepXM8alzx517joeD/pG+5X1gK+e6Dw/3HhycNjdDg4u/Bt849Vuv0MQN1BfZBzBUJS03tQ9gnL3vzvvuvu2eO/7Dm38t1NM5YLbn3T8ZlWhLHDw7GntmaWKfUt5RlMWCltjJkczgGZoH53bS9GlJiU+o3YI1oopNHLxMGRafHEp0Np/QVyJNaq6c/pmDaIltjaj29M5RTdKZOydnzzmEKQOVbzcrlMfU1x8rXU6iCdAMocJJrw00hkd6aPD2o/P6rqBNDMoJ1L+uPS3bObSZcMurol/0jM951Ste88THP4mXhaZtw18IbDMB7OB6Wxc7OOY/rx00uxfZQS0ZlYXFDo7BrimEQ7gm8BQ7F4Gj473zF0/dfXF/eHDqXPf48sHhgbscPzruHunJK/eKfxtn2FZFQNZxYiLxS77Onkh84Nx9D9x/79n73vZX73rvnbf+xi1/Gvwc71eFXakZPK3kkDlvpdonHlp0XLnkWTnHS/nkGZqrph85WYo0RzFxC5M85tpNbkFmfCiJtLzmvElifNSkm0unQ3LR9NSfKaB7nMd6KajuSNgKUL0y1Pl/rja2gtTSOPspF9ApKamK8901NZAur14rf/7iPX/8QZWgfPYCic5AE4MqUSui5QrKCZQyr/rMVz/xCfiBk23JHgR2gAB2cNMaecV2MF/9VttB95Y1s8vYwXzLrjLGtUG4IFplyZQFgdkJvPH3fqez13vHB//slgdu2T81es+5P7np0c9T/x0OOh991Q2PvOz6j33MU5715GfNLpgcyyXgvcSbb7HPJL7tPX/2A3/8686Fs0k2X7B5RrFzaK6THZKdc0fMzXObIv2o5XflqdnmPMDEZ/PjmTlyiWyLt39JjrFrpwTm5kleeixk8cUm3qBNy41z2ecH9zr9vVG/P+r3RhbodfR9zH7fft23KJzacgrt+4KmrV8sGhxCF3Bp5PC56ughQ6VTlbVQ9MJdDz70F7frgL4rKG9QnqbNBw47PByYNmDDf605rH3TTtCweMRBoGEC2MGGga5KXHN2MNE4DFp+8PLWxIyaDWiWxkxYOrBtqB2U1ZPVdJtGYuygR7H6X9lBDOHqsVMiBCBga01F4f0f+qAmEu+87x59JjHvJXpMbpRy7pMbrpzNcI/fJUYksXxKLMMXLKKsoFuQ6Y96h1DH/Yjnf3Vo7A0Gq2lSbFYwdSCdUEtp8ea06XmHXl8vGvX/5BYO5RBqtaj/9a6dJglVVL/v5Guizz096N1Ck6Hlo272TzOH9ml799YZCR4endz7lg+Nzl/QOmg9Q6j3x2jp6cc/45WfxMtCHfsl/WAHlwQWsRCAQDWBee2gu3uoJSZmkJISgglzJiuxdFtjB/XxJF6aXd2XFjyqnuQ7jb/SWVAa2SEAAQgsRCBzA/W9d9zy/7/lXWEWMSPaeYb2wJ42m1tL5xK1W2gRh+4tLM5q+nHPOYSp0GBNJxxCJ8gEBodQRfX0gKB+NTdoPqEFvF9nvzpsypgDqLWjeo7QRlkp6ZxBfx/UHEWXzPmBmlQ0+UeDo7sfOP/uO7Q69HT36FTn+HnP+LxPeD4viUmbZ5l/sYPLpItsCEBgNgKbYQedlfQ3RrGDszVgK1PbpYoUZ6lMK1sPpSGwGwQS63irX276rtf9yf+M6x0cOQWcM2YHzTlMJuf8VJ/N7ymBffDdLRO1xTPOzfO7XqDC480SuElCZwvtkHxCbSZ35P3AnrmFcgjl+JkHaHN95gi6sL1hRvGmhLLZlKB9acLSaz5Qu5pC1D95koOjk4t3PXj4lx/Wy2Me0T36hGd+9ovwA8fNsIoQdnAVlCkDAhBYgEC77KDZO62CwQ4u0OIrzmrXJyoSh3DF3CkOAhBYhIBMo7L/z9/+je/+le/rnT4YHuyf10cZbCiT3+fHNIUtoDj978Y5t2cmykW5Pe8NpitLJzImzmFwC91jhIkI+Xjm4Jkr6LxBNzFo3mDikao4GUI7JMfPvT3UDtmu+9Xn5m2G0PQ7uvehC++949SD5//aTZ/xyle8+omP/1heFppAXuEf7OAKYVMUBCDQDIEW2EGZRWcusYPNNPkypcgO8pbRZQJGNgQgsAQCeqWbpF53+dWXP/SRS/f2B4848+yrLz7t9IX9/iXnHr780oPOXz1w+n/ff9o7h/ZZeU3buUcK5SUOSvTRh/7sSOIu2jOH2tOrXBRhzqE8PD9h6LI7F8LEe1fTJVYyi5ZPqpzmhqabfD/zSV1ie1zQrSDVl8RedHzjEz7qhud8xnM+7rnPP3PFmTQ5fyEAAQhAAAJTCCzTDspomRFb1A7K7OoRCVcP7OCU5tyAwziEG9AIqAABCMxFQP7XQe/4YTcreHr/gatOP/gxl5275HT3uY/tfs7w+M/PX/aRo0e879yZ+x7ce8G1z/no6x7/w29+vb5N4r/gJ1PnAvYCT3ME/SpTcwPd9J15cXrYT8fcC2nk7mm20OIshR4jNG/QhZP0+mNbMi+pUPJhQYszGbamVDKPhy+8+mmf9PiPf+oNT372Tc+xHGwQgAAEIACBeQlgB+clR74JAjiEEzjYgQAE2kWg777bfjTcO+70Lo2WPFzS3b/u1Nlznd61l5+6eHzqSY/8qH/wxa/9J1/8WnsS4/zZm/VE4qjz9pv1mcRb3nDrnw70vhndy3S/bjrReX1+Zk84bE7P1qNqcz+28tPcQfdrS2Kca+heHjMBTx+lkF/oZgU7n3TdMz7poz/uaR/1pGc97dkTidiBAAQgAAEILEAAO7gAPLImBHAI6QoQgEBbCdid0b3OoXPTzg5OXxgdXmZeWrJd0etd0bt4T+9kODhls3Nu0zIb/bvx+hu097IXvdRHZrzE1/3f/+kfLPRH5SgqIH/PpvokR3/cIxFyBeUWuqBWhPo5QD+7aPH26tHO6CWPfcYnPen5n/DUF9zw6Md5afxCAAIQgAAEmiKAHWyK5I7LwSHc8Q5A9SHQYgJ7e/K8RnvdwV5v9PBo/7hzIa7MI7oHl3cPL+0ddjqPGLuJcYo0nPES/USiDr7/w/4ziXd/5O47/90f/3qS3OYDbWmoJgb9A4EKOA9Qk4t63UznU65/+v/zlOc/7aM/9qorrroRPzCFzF8IQAACEGicAHawcaS7KRCHcDfbnVpDoP0EnD/W7w73esPD4f6F0YF/eN2+/ZBuV/YGmiR0blsyQ5gemfLXP6//nKfeFNJ9w5f+PZtIfODczbe8T+6ftjvvv0cB8wY7nUddde21V15tHuBjmAk0IGwQgAAEILB0AtjBpSPelQJwCHelpaknBLaPgB6c6Ou9nXr7i9Zxui9BZOp4WW//kt6RIqtnCDO5ynaTicTH4vKVESIeAhCAAARWSgA7uFLc21tYI5dJ24uHmkEAAptNwB7qc5t9BSK37Xf1zb/kLaK5g0RAAAIQgAAEWk8AO9j6JtyACuAQbkAjoAIEINAEAfd5wJwg93rQ6659bO4AERCAAAQgAIGtIoAd3KrmXGFlcAhXCJuiIACB5gjcedddEhYeDey5mcAi8f77gUVHiIMABCAAAQi0lgB2sLVNt3GK4xBuXJOgEAQgUJ+AfV8+XS4TD2fJZwNHo5OOvgbIBgEIQAACENhOAtjB7WzX1dYqvoJabcmUBgEIQGAxAoNOb9Dpnox6e3rXqL1dJjugHXUGF4f7Mpb4hIuRJjcEIAABCGwiAezgJrZKC3XiLaMtbDRUhgAE3GcA9fl4vTVmOOoNRqP93kmeyoXR4OJobzAYryzNpyEGAhCAAAQg0EYC+u4RdrCNDbeBOmdvqG+giqgEAQhAoIzAIJ3967mPxSuZ/zCgT38yGukThRZmitAT4RcCEIAABLaLAHZwu9pzPbXBIVwPd0qFAASaIqAPTujz9P3eaJh7r8yg0zka7RV+kaKp0pEDAQhAAAIQWC8B7OB6+W9B6TiEW9CIVAECu0vAHp/QktFOb797ciSXMNr0XpnDUe9wtNftjK655rroCEEIQAACEIDAlhDADm5JQ661GjiEa8VP4RCAwAIEdE/0ZNQfdnoH3ZMruoenu9mHos8NT10YWOSkq7hAkWSFAAQgAAEIbAwB7ODGNEW7FcEhbHf7oT0EdpmAvMGHhwfDUaffGXY7neORloiOt3uGF/WK0SO9f5QNAhCAAAQgsI0EsIPb2KprqBOXSmuATpEQgMDiBD5y+0f0zYnjUV+3tfpd+YLds6Ph8eBwOOxq5ejRYHTn4NQdJ1fKIex1TniMcHHgSIAABCAAgY0igB3cqOZotTI4hK1uPpSHwE4TkLM3GOw/otfVtyUeGJ4+GZ483Bk8fNLVx+gHw+69J5c9NDyQo7jTjKg8BCAAAQhsL4FiOzjo6un6E+zg9rZ74zXDIWwcKQIhAIEVEdD04HDU1xcl9G2JC/L9Bnv6Qv2xrOCodzTo3T+8VIHBUP/47MSKWoRiIAABCEBglQSK7eAQO7jKRtiGsnAIt6EVqQMEdpCAvUR0uLff3T8Zdk6Gg/PDU3vDzkFvcDjo61MTJ4Peg4NTD52cGtqHCfkK4Q52EKoMAQhAYMsJYAe3vIFXWD0cwhXCpigIQKA5Al19in7UPR70Hx70Hhj0+oP9vc7ooDfsn9iHB48H3YuDPc0QqkDWjDZHHUkQgAAEILApBLCDm9IS7dcDh7D9bUgNILC7BGzq7+LJ/r1Hp46POvs9zRIOTu/35CpqyahmDocjmUu9gFR3UdkgAAEIQAAC20cAO7h9bbqGGuEQrgE6RUIAAosTePWrXvPBu9//u3e99VhfIxz0zh/t7/eHF4ajg71+rzvaG530OoNudyRn8OM/6oVP+ugnLl4iEiAAAQhAAAKbQwA7uDlt0XZNbC3ViLvnbW9G9IfArhI4d/7cBz78wXvP33fX+Xvl/t1+9o7ufleB6888WmPbk69/sp4fvOmJz9pVPNR7OgF1FyXCDk4nRQoIQGAjCWAHN7JZ2qSU7CCGsE0Nhq4QgAAEINAsAexgszyRBgEIQAAC7SIgO2hvXGCDAAQgAAEIQAACEIAABCAAgR0kgEO4g41OlSEAAQhAAAIQgAAEIAABCBgBHEL6AQQgAAEIQAACEIAABCAAgR0lgEO4ow1PtSEAAQhAAAIQgAAEIAABCOAQ0gcgAAEIQAACEIAABCAAAQjsKAEcwh1teKoNAQhAAAIQgAAEIAABCEAAh5A+AAEIQAACEIAABCAAAQhAYEcJ4BDuaMNTbQhAAAIQgAAEIAABCEAAAjiE9AEIQAACEIAABCAAAQhAAAI7SgCHcEcbnmpDAAIQgAAEIAABCEAAAhDAIaQPQAACEIAABCAAAQhAAAIQ2FECOIQ72vBUGwIQgAAEIAABCEAAAhCAAA4hfQACEIAABCAAAQhAAAIQgMCOEsAh3NGGp9oQgAAEIAABCEAAAhCAAARwCOkDEIAABCAAAQhAAAIQgAAEdpQADuGONjzVhgAEIAABCEAAAhCAAAQggENIH4AABCAAAQhAAAIQgAAEILCjBHAId7ThqTYEIAABCEAAAhCAAAQgAAEcQvoABCAAAQhAAAIQgAAEIACBHSWAQ7ijDU+1IQABCEAAAhCAAAQgAAEI4BDSByAAAQhAAAIQgAAEIAABCOwoARzCHW14qg0BCEAAAhCAAAQgAAEIQACHkD4AAQhAAAIQgAAEIAABCEBgRwngEO5ow1NtCEAAAhCAAAQgAAEIQAACOIT0AQhAAAIQgAAEIAABCEAAAjtKAIdwRxueakMAAhCAAAQgAAEIQAACEMAhpA9AAAIQgAAEIAABCEAAAhDYUQI4hDva8FQbAhCAAAQgAAEIQAACEIAADiF9AAIQgAAEIAABCEAAAhCAwI4SwCHc0Yan2hCAAAQgAAEIQAACEIAABHAI6QMQgAAEIAABCEAAAhCAAAR2lAAO4Y42PNWGAAQgAAEIQAACEIAABCCAQ0gfgAAEIAABCEAAAhCAAAQgsKMEcAh3tOGpNgQgAAEIQAACEIAABCAAARxC+gAEIAABCEAAAhCAAAQgAIEdJYBDuKMNT7UhAAEIQAACEIAABCAAAQjgENIHIAABCEAAAhCAAAQgAAEI7CgBHMIdbXiqDQEIQAACEIAABCAAAQhAAIeQPgABCEAAAhCAAAQgAAEIQGBHCeAQ7mjDU20IQAACEIAABCAAAQhAAAI4hPQBCEAAAhCAAAQgAAEIQAACO0oAh3BHG55qQwACEIAABCAAAQhAAAIQwCGkD0AAAhCAAAQgAAEIQAACENhRAjiEO9rwVBsCEIAABCAAAQhAAAIQgAAOIX0AAhCAAAQgAAEIQAACEIDAjhLAIdzRhqfaEIAABCAAAQhAAAIQgAAEcAjpAxCAAAQgAAEIQAACEIAABHaUAA7hjjY81YYABCAAAQhAAAIQgAAEIIBDSB+AAAQgAAEIQAACEIAABCCwowRwCHe04ak2BCAAAQhAAAIQgAAEIAABHEL6AAQgAAEIQAACEIAABCAAgR0lgEO4ow1PtSEAAQhAAAIQgAAEIAABCOAQ0gcgAAEIQAACEIAABCAAAQjsKAEcwh1teKoNAQhAAAIQgAAEIAABCEAAh5A+AAEIQAACEIAABCAAAQhAYEcJ4BDuaMNTbQhAAAIQgAAEIAABCEAAAjiE9AEIQAACEIAABCAAAQhAAAI7SgCHcEcbnmpDAAIQgAAEIAABCEAAAhDAIaQPQAACEIAABCAAAQhAAAIQ2FECOIQ72vBUGwIQgAAEIAABCEAAAhCAAA4hfQACEIAABCAAAQhAAAIQgMCOEsAh3NGGp9oQgAAEIAABCEAAAhCAAARwCOkDEIAABCAAAQhAAAIQgAAEdpQADuGONjzVhgAEIAABCEAAAhCAAAQggENIH4AABCAAAQhAAAIQgAAEILCjBHAId7ThqTYEIAABCEAAAhCAAAQgAAEcQvoABCAAAQhAAAIQgAAEIACBHSWAQ7ijDU+1IQABCEAAAhCAAAQgAAEI4BDSByAAAQhAAAIQgAAEIAABCOwoARzCHW14qg0BCEAAAhCAAAQgAAEIQACHkD4AAQhAAAIQgAAEIAABCEBgRwngEO5ow1NtCEAAAhCAAAQgAAEIQAACOIT0AQhAAAIQgAAEIAABCEAAAjtKAIdwRxueakMAAhCAAAQgAAEIQAACEMAhpA9AAAIQgAAEIAABCEAAAhDYUQI4hDva8FQbAhCAAAQgAAEIQAACEIAADiF9AAIQgAAEIAABCEAAAhCAwI4SwCHc0Yan2hCAAAQgAAEIQAACEIAABHAI6QMQgAAEIAABCEAAAhCAAAR2lAAO4Y42PNWGAAQgAAEIQAACEIAABCCAQ0gfgAAEIAABCEAAAhCAAAQgsKMEcAh3tOGpNgQgAAEIQAACEIAABCAAARxC+gAEIAABCEAAAhCAAAQgAIEdJYBDuKMNT7UhAAEIQAACEIAABCAAAQjgENIHIAABCEAAAhCAAAQgAAEI7CgBHMIdbXiqDQEIQAACEIAABCAAAQhAAIeQPgABCEAAAhCAAAQgAAEIQGBHCeAQ7mjDU20IQAACEIAABCAAAQhAAAI4hPQBCEAAAhCAAAQgAAEIQAACO0oAh3BHG55qQwACEIAABCAAAQhAAAIQwCGkD0AAAhCAAAQgAAEIQAACENhRAjiEO9rwVBsCEIAABCAAAQhAAAIQgAAOIX0AAhCAAAQgAAEIQAACEIDAjhLAIdzRhqfaEIAABCAAAQhAAAIQgAAEcAjpAxCAAAQgAAEIQAACEIAABHaUAA7hjjY81YYABCAAAQhAAAIQgAAEIIBDSB+AAAQgAAEIQAACEIAABCCwowRwCHe04ak2BCAAAQhAAAIQgAAEIAABHEL6AAQgAAEIQAACEIAABCAAgR0lgEO4ow1PtSEAAQhAAAIQgAAEIAABCOAQ0gcgAAEIQAACEIAABCAAAQjsKAEcwh1teKoNAQhAAAIQgAAEIAABCEAAh5A+AAEIQAACEIAABCAAAQhAYEcJ4BDuaMNTbQhAAAIQgAAEIAABCEAAAjiE9AEIQAACEIAABCAAAQhAAAI7SgCHcEcbnmpDAAIQgAAEIAABCEAAAhDAIaQPQAACEIAABCAAAQhAAAIQ2FECOIQ72vBUGwIQgAAEIAABCEAAAhCAAA4hfQACEIAABCAAAQhAAAIQgMCOEsAh3NGGp9oQgAAEIAABCEAAAhCAAARwCOkDEIAABCAAAQhAAAIQgAAEdpQADuGONjzVhgAEIAABCEAAAhCAAAQggENIH4AABCAAAQhAAAIQgAAEILCjBHAId7ThqTYEIAABCEAAAhCAAAQgAAEcQvoABCAAAQhAAAIQgAAEIACBHSWAQ7ijDU+1IQABCEAAAhCAAAQgAAEI4BDSByAAAQhAAAIQgAAEIAABCOwoARzCHW14qg0BCEAAAhCAAAQgAAEIQACHkD4AAQhAAAIQgAAEIAABCEBgRwngEO5ow1NtCEAAAhCAAAQgAAEIQAACOIT0AQhsG4GP+ZiP+b7v+z5fqzismJ/8yZ98znOec9lllz3zmc/8D//hPwyHw8JkqyRSppJ0+Hy3rVKZTSjrfe97X7fbfec737kJyqADBCAAgR0nEJvROCwsZfYrk2yVAMtUkg67aVJXCb/VZeEQtrr5UB4CMxCQl/g1X/M1r3zlK3/pl35Jv//oH/2j7/iO75gh/xKSbqBKS6glIiEAAQhAYNsIbKD92kCVtq3Vt7c+e9tbNWoGAQiMCYxGI7l/3/7t3/6N3/iNiv20T/u0vb29f/2v//U3f/M3HxwcjNOtMLSBKq2w9hQFAQhAAAJtJbCB9msDVWpr6+6k3jiEO9nsVHr3CNx6660PPPDAS17yklD1v/W3/tZf/MVf3HXXXR/1UR8VIlcZ2ECVVll9yoIABCAAgZYS2ED7tYEqtbRxd1NtlozuZrtT600h8Lmf+7lf/MVfPFWb17zmNXGyn/3Zn/3rf/2vP+pRj7rxxhs/7/M+7zd/8zf//M//XA+e/dVf/VWZqOuuu67f77/jHe8ICZ74xCf+/M//fOwNSuxf+2t/7cyZM3rO8Ou//usvXLgQEiugu48/9mM/9smf/MnXXHONpH3WZ33Wb//2b8cJfLhaN/9kxQ/8wA8873nPe+9731um0sd//MerOlraqk0B5ZLwOO/v//7v11Hpj/7oj17wghdcfvnln/iJn/j2t7/da1gY6Q8V/uq5i//6X//rhz/84a/+6q9+9rOfLT6f8AmfoOnWixcvhvRz6Obz/rf/9t8+/dM//dGPfvQNN9ygggqRhlIIQAACEIBABQFMaqGV3yiTWtF8HFozAV3nsUEAAmsh8D/+x/+49NJLNXcXSn/Xu971whe+8Pbbbw8x995776lTp970pjcp5uTk5LM/+7O1yPO1r32tnCW5E3oscH9//wu/8As1jvzlX/6l0nz0R3/0937v9/rscfgrv/Irr7jiCjljd999tz8afpXsyU9+sty8f/tv/+2v//qvf/d3f/fVV1/9rGc9Kygm5+fFL36xXNB/+S//pRL84i/+oso9ffr0t3zLtwQhNXXT+2we+9jHSo4mJ8tUEoQ//uM/fqnbFJAvp1KkZJx3qkr333+/avGZn/mZ0vZTPuVT5PoeHR0VRoYqFAZU7hd90RdJZ5H/iZ/4CVX/u77rux7zmMeI2Ic+9CGfZVbdlGswGLzqVa9SU/79v//3//t//+/yOb/qq75Ku1/xFV+hppRRL1SGyMYJeAPcuFgEQgACqyeASS208htlUlffKyhxKoHEEZ2ajgQQgMCSCMhFufbaa+VmBPnf9m3fpjPzh37oh0LMD/7gD2oCSm8EVcy/+lf/6hGPeIR3kEKCP/zDP/TPAVY7hJrx04tklF3zcnqG8Fd/9VeDBPkzV1111Qc/+MEQo3ddXnLJJcGxlP/5pCc96c477wwJFNA0ndL8wi/8go+sqZu8ynvuucdnqVBJCf6G23xK/UrJOO9Uld7whjeI5Pvf/37lfc973vO3//bfvu222wojQxGFAZUrOWqF+KgcS82jvvzlL/ftMqtuEqUHONUWGcdPs5dys1VcJj4umnCzBERbW7MykQYBCKyFACa1zMqrOTbEpK6lY1BoNQFvBzGE1ZQ4CoHlEtAEkRZqhjKe+tSnyv3TyswQI8dDCxS1K99D6xW/53u+JxwKAf+qmGqH0Cd++OGH5cJpxanO/2/4hm/wkfJnNOMXpPmAHjJ8xSteobBMrLyUX/7lX84k0K68she96EUK1NdNWTJyClVSmrz1CnnrqKQFtL1e75/+03+qqctQYmFkOFoYEBxNMOYPveUtbxHDN7/5zTqkNDPp5nFpPjYvVn61xOIQ5sksKcYbwiUJRywEILBiApjUDTepK+4PFFeHgLeDOIR1WJEGAssioOk+PSbnJ7Le/e53y4fRM4GaxPMLO+UYKOaWW25R8ZrmKnMV3vjGN+pQHYcwVEOTkMryf//v/1WM/BlNWIVDPqDloHpkTmHpoJRlm5ZlKk193Qq9IF9irJJi8g5hyFtHJUl43eteJ1dWi0XlUctG+lIKI/2hwl/B0RrRwkOaVv2RH/kRHVKamXTzuDIzvb6It771rUKtChaWSGTjBHzHblwsAiEAgbUQwKQG7JtpUoN6BDaHgOwgbxktu8olHgIrIiCn66abbvqZn/mZf/7P/7meJdMLSz71Uz9Vk4Ra0qnHybSaVNNTeuOItNG0kn4135XXzB/Kx4cYJdDQIz8zxGhKUG6Sln0+//nPV6RfdBqOKqDloP69KY985CO1q7fF6D00cYI4XF83+bc+41SVYvk+HPLWUUlZ/u7f/bt/82/+TbH9/u//fs2L/sEf/IGmWAsj82XFMUIX74aweOpRQL87k25eYGGrlZUVCiUAAQhAAAJlBDCpgUzGyof4EJjJbClXofUsjAxFEGgLAd4y2paWQs9tJvClX/qlP/3TPy1PQA6hXjSiqupXYS2M/Lmf+7kv//Iv95X/2I/9WD0v7icDMzimvp1SL3G5/vrrM7n0DNvx8XEmMr+rjHoNph4g1Bs7M9vb3vY2vf9GWebQbdkqyZs9PDzUBOY/+Af/QBOhmpTTA4SFkfkqZ2IK8b7zne/Uw5B6e1smcR1ceiDzyiuv9A80ZrJrfjgTwy4EIAABCNQngEkNrBq08oXWszAylE6gRQRwCFvUWKi6tQT0jlB91UA+oRYKvvKVr1Q99TEJOSGK0QSU3qPta66VpXpeXH6UlsTELOTt/Lt/9+/imHxY7wjVWz1f//rXh0N/8id/onWqmpAMMRUBzSVqAlNvKovTaM7t7/29v+cn6+bQbdkq/eN//I91q9grfNlll+3t7Z07d64wMq5UYVhtoU9uxIfOnz//d/7O39Gjnh/3cR8Xx/twHVzf9E3fpKaURx1nV8v6ZwjjSMIQgAAEIFCfACbVs2rWyhdaz8LI+i1Fys0ioHkJNghAYL0EPudzPkefy3vuc5/r1dBiQj35ppiv/dqvjRXThJ4+AKjlnZr10ltetKz0H/7Df6jPTvhZxOpnCF/96lfr8xXy6zQx9e///b/X20314TsvXI/AhReKhuK+8zu/8ylPeUrY1T1XPY+n7xP+2q/9mhzLL/uyL1O5+ohFSDCrbspYoZKO5p8hzChZrdLv/u7vyk3Vy0X1InLNuMon1FciCiNDFQoDgqPLC31nQl66Vp+K3vd93/c97nGP06Ro/NmJmXRTQXL1v+ALvkAM/WcnhFSWVS2r5TeyEDxDWNgWy4j09ngZkpEJAQisiwAmNWPl1RAbYlLX1SUot4JA4pdWpOAQBCCwGgJyWnRCasooFKe3VirGv8QyRPqA1pHqOUMt49Smj+PJM9RXIvT1Qv/diNi7i8Ny2P7Nv/k3+pSf1pA84xnP0ItSFOMFxslCWRmHUPFaxarvVejbD/KO9H0/3X0MiUOgvm7KUqGSjk61XkpTrZI+7qea6kuPmseTK+iVLIz0hwp/PZxbb71VvqWkyUvXi1U1B6ilMiF9IcBq3XzeX/mVX/mMz/gMOf+iqqZUFjmZakotcA3CCSyVAA7hUvEiHAJrIYBJzVh5tcKGmNS19AcKrSYgO9jV/0rkLSK/EIAABCCQIaAX/GhlrKbvMvHsbgcBTSOrItjB7WhNagEBCEAAArMSkB3kGcJZoZG+mIDmprRCQy/wePzjH//N3/zNeplHcToXq2fe1Pn8pgmlTEq9NOWlL32pXrmhVXlaUqi5r0yCDd+trl2svN4skmIY/9VXB0Oa9qL4rd/6LU0haiJR70d9zWte83/+z/8JlcoHqjtPNQT1ujE7F9KDl/kiqmMyEuLd6oxlR+v3AUmoYDW1h5QpsDnx1c1XoecXfdEXqSF+7/d+z6fZAhQVleXQdhCoHsoydaweJeY+cTKlrGu3unaxVlNP7faiqBjbYwI+XN15qiE0YgfzKi0YU78PqKAKVlN7yIJ6riB7dfNVKLBKO8hnJyoagkN1Ccj9e/nLX66JFHl3d9xxhz4Ffv/99//wD/9wWf6nP/3pf/zHf6yjcvwyaX7pl35J/oOerdKcjJ6j+9Ef/VF9kkHLJp/2tKdlUm7sbkXtMjr/4i/+YsZz1gOBWgXqk7UXhT7Npwfh9F17vepGH2bQ0h29fEVfrdB6lQwB7VZ3nqkQ/vzP/1z9TeYwSH7CE54QwjUDDz74YM2UNZPV7wPVrKp7SE1l1phsavOV6fYbv/EbmVtFbUdRVlPit4ZA9VCWr2bFKDH3iZMvZV0xFbXLqFR9arcXRfXYnoFQ3XmmQmjEDmZUWny3fh+oZlXdQxbXc9kSpjZfmQJrsIPV60o5CoGpBNTd9ViaXuHoU+oZLbkBFy5cmJpR7/mQnxCS6SsLj33sY7/yK78yxMgnlC+hD/GFmBYFMrWbqrkeHtObMHWfTCnbi0L3AvStv2/91m+N6/uN3/iNmvLVmznjSB+u6DxTIehb8/qM0pve9Ka82AZjpOGf/dmfzSewug/MyiruIfPps8pcU5uvTJkHHnhAqwP0xKbMZHjyM5O4QRTeGGfkswuBWQlUDGXVojKjxNwnTnUp6zqaqd1UNeJTu70oZh3bKzrPVAirsYNTG64iQXUfmJVV3EMqCt2QQ1Obr0zPtdhBe3aCDQKLEPj1X/91vbsySPAfRdAnDUJMWSAzTPzRH/2RLs7e+973xul//ud/Xi+31NfY48hWhDO1m6qzZrqe+tSn+mTtRaF3kPoPPMT1vffee9WyOhRH+nBF55kK4a1vfavE6kOIebEbElPdB2ZlFfeQDalghRpTm68sr96sq5XnH/nIR9S4ZQ5hgyhUirYyZYiHQE0CFUNZtYTMKDH3iVNdyrqOZmo3VY341G4vilnH9orOMxUCdnBqp1pjgqnNV6bb6u0gS0b9xQC/CxHQaxK1BRH6FsJVV12l9y6GmJoBvYJf39zT5ECcXt6gXueoySVNMcXxWxZWHfWlO7270tervSh0904vzLziiiviBtLDpZo21Kft40gfrug8UyFonYzEXnfddXoFqJ43y/ScfFmbFjMTq0wP2bS65PWZ2nz5LIr5gz/4Ay0f+s3f/E0tOihMoMjWoSirCPHbRKBiKJupmvOdODMVsbGJM6d2e1HMNLarOSo6z1QI2MGN7c9SbGrzFSq/FjuIQ1jYFkTOQ0BDub6aoG+Xa92z1nzrAn1WKc973vP8q6LjjLrTpqcTt9sbVH3/y3/5L/oGwxd/8Rf7urcXhdY56MMMcQv6sDy3ikf1CjvPVAgyhOpmT37yk2+++WaVoscvv/u7v/tLvuRL8qVvZsxMrDI9ZDNrFGs1tfnixD6sbvAVX/EVasGXvexlWoKeT+BjWoeirCLEbx+BwqFspmrOceLMJH+TE2dO7faimGlsDy1S2HmmQsAOBoAbGJjafHmd12UHectovi2ImZOAnvfTo1Yf+MAHlP9P//RP55Qyme13fud3fvInf/Kf/bN/Nhm9hXv6hqw+9a7VNWV1axGKwnsBetivrGqKr9l5MhDe/e53P/TQQ1/1VV91yy23fPjDH9YreRTWV+MrCtq0Q/VZTe0hm1a1vD6Z5ssn+PZv/3b5gd/7vd+bPxTHbAGKuDqEt4lAzaFspipPPXFmkrbJiaee2i1CUX9sDy1Ss/NkIGAHA8BWBDLNl9d5nXawbAEr8RCYj4A+rn1wcKCRfWr26qcL3vKWt2hiUG7SVDmbmaC6drHOere+jEfFt8hbhEJzdE95ylPi2vmwlhDrpaP5+ExMRefJQ9CzE1pZEUv4wR/8wUsuueS2226LI9cYru4D9VlN7SFrrGPNovPNl8n4tre9TU+f/sIv/IKPP3v2rCxl/hnCxlF4e5xRhl0ILEigYijLSK4eJaaeOBlpm7ZbXbtY26mndotQ1B/bYwIhXNF58hCwg4Hb5gfyzZfReY12sOqefd5tJQYCWgih2Ziw6dG+PJNXvepVX/3VX/393//9+UP1YzTGadmYXjH64z/+4/VzrTJlHRQ19ZEb82mf9mn62E5h+k1GUQhBtznzFSmMzCcr6zyFEJ773Od+4id+YixEM4SDweAP//AP48hNDhdiyUdW95BNrqDXrbD5YrX11igtFv30T//0z//8z4/j8+G2o8jXiJjWESgc9zK1KBvKMsmqd6eeONXZV3C0DoqaalSf2puMohBCfhgXh8LIPJ+yzlMIATuYB7iZMYXNF6u6XjuIQxi3BeHpBOSeaaonbD/xEz+hZaKve93rMjlf8pKX6Curet9uJr7mru6R6MOG8gb9Fyxq5lpxsjyK+RTQ0+d6cvLrvu7rCrNvOIo8BL1Oxs/tZKojk5l/ELRm5ymE8M53vlNPq2ZK0QPcj3nMY3TDIhO/mbs1WVX3kM2sWqxVYfPFCRT+nu/5Hr1e+Id+6Icy8ZndtqPIVIfdlhLIj3s1h7KZ6lvnxJlJ4DIS51HMV0r1qb3hKPIQao7tnlXNzlMIATs4X39bfa7C5suosX47mJmvZBcCMxHQ65K13FGX+3EufZVe736MYwrDhYtJdNroJaW6Q6aXrBTmaktkYe3yyn/TN33TE5/4RN04zB9qI4rXv/71elzwvvvui6ujz06ok7zhDW+IIxWu03nKIPzUT/2UVodmPncpX1Sly1HMFLSu3eo+UJNVRQ9ZV73ql1vWfBkJH/dxH5cxjWH3O7/zO0PiZaDwBYUiCEBgDgJ1hrIysYWjRM0Tp0zm5sQX1i6vXsWp3UYUNcd2z6FO5ymDgB3M96UNjClrvoyqa7eDfH8p0yLszkZAq0Y14n/bt31byKaJwWc961l6YaaPUQJ9bF3fTg0JQiBvKvQNQ32i4PM+7/Pa7g2qjvna5VEIyzXXXFP4cF1LUehVoqr4t3zLt4RWVkCfqdQdUx1SOIYwtfNUQJDPKXT65H1c0Gtf+9rrr78+c3siTrDicHUfmMpK2lb0kBXXZY7iKppP0uKeoLcT//Hk9sY3vlGumuYM9bogX/SSUOAQztGyZMkQmDqUxb09kzc/SlSfOJnsG76br10eRcWp3VIUU8f2GMLUzlMBATu44f1f6lU0n47GPWG9dpDPToTb0ATmJKDh/pd+6Zf0/UB9H1y/Gtn19OAdd9yhL4l5ie973/v0NKDWgz3hCU+oLkMvy1JKfU3u67/+67UQIk6sVfKaYopj2hjOo9Bbtg8PD7/0S780U532otDn437mZ37mC77gC/Rh8c/5nM9Rvf7X//pf+sSinF7/ZbkYQnXnqYagaeT/9J/+06tf/Wo5DCpOU6z/+T//Z32k5Jd/+ZcLv3uRIbyu3bj6U1lJybIesi7965db3Xw6nWMUz3jGMzKS/Wcnnva0pz32sY/1h9qLIlM1drePQPVQpvrGvb26+lNPnOrsm380j6Ls1G4viqljewyhuvNUQ8AObniHr26+jbODm+9eo+HmE9DLAF/84hdrXLvhhhv0GTF9gjzo/I53vENnrBzCEBMCSv+zP/uzYVcfpC47t/VhlpCsLYFM7aR2HsVNN930tV/7tfkatR3Fb//2b8tV07Om11577ad+6qfKSQt1zEMo6zx1IPzJn/zJK17xCs0q6yOE8j/1vZNQ0CYE6vSBClaqQlkP2YTaVeswtfnyPSEWmH/L6JJQ+DEnLpowBOYjUDaUSVpFb8+MElNPnPl0W1euTO0KUZSd2m1HUTG25/tDWeepAwE7uK7uPbXcqc2X7wmxzFXaQZtyUdllV+HEQwACEIAABLaYgF96gB3c4iamahCAAAQgUEFAdpC3jFbw4RAEIAABCEAAAhCAAAQgAIFtJoBDuM2tS90gAAEIQAACEIAABCAAAQhUEMAhrIDDIQhAAAIQgAAEIAABCEAAAttMAIdwm1uXukEAAhCAAAQgAAEIQAACEKggwGcnKuBwaFkEXt77fBOtz0joKVb92G+307NXHNkLHnruPoV+tWNpxiktmf2zXOlRH+N+vUzJUZpEfneUZI8iXUFKM0qkSZQvomuvV1Kkv0/ijrrsnSSlHbUEFpmm9FmSUvSKJktg/1wWlzjJ5SJ9LjsaJQu5QkBFpLl8IEmfJLDixxJCSotOxYaAi3SJXSgUHSS4lJPynZAoQTiqtOOw5HlpoSwLjLIJ0qOWWOF019VrZKyCzDistE5+miBJmUS6TOOwaPnEJtzCrv1DpDVFHJmUE6V0Te0USTuFKeXDvpYu7CKtn7ouEBIkTd2zfjGOtN3uyHUlF1CuzihEdjtDO2q9PsmllD67Yvo+0tIrcqgYS2ns01yKdGW5+KGTrASWUmGfXQlcRnfUh02CRfYlx2XvazfJNez7stKULruV4hNIPQVc9mHfVHVaqaxUK0WaSpa94yK9JhbpqtZx8pVRRxWpiluz9F1bKaxTuueGABeWDKXp9rq93qNvttZmg8DWEcAOaoywEcENRmreZDcENEQq1idwgckEOpBmkYSQ0qIncjn5Fueyu8NWoksTJLgs2UiXNhtpksbZU/1drJPpTVvIFes/Dit5lNjCQWYcxg4aabMp2EHXw5b7I7vMBgEIQAACEIAABCAAAQhAAAK7SACHcBdbnTpDAAIQgAAEIAABCEAAAhAQARxCugEEIAABCEAAAhCAAAQgAIEdJYBDuKMNT7UhAAEIQAACEIAABCAAAQjgENIHIAABCEAAAhCAAAQgAAEI7CgBHMIdbXiqDQEIQAACEIAABCAAAQhAAIeQPgABCEAAAhCAAAQgAAEIQGBHCeAQ7mjDU20IQAACEIAABCAAAQhAAAI4hPQBCEAAAhCAAAQgAAEIQAACO0oAh3BHG55qQwACEIAABCAAAQhAAAIQwCGkD0AAAhCAAAQgAAEIQAACENhRAjiEO9rwVBsCEIAABCAAAQhAAAIQgAAOIX0AAhCAAAQgAAEIQAACEIDAjhLAIdzRhqfaEIAABCAAAQhAAAIQgAAEcAjpAxCAAAQgAAEIQAACEIAABHaUAA7hjjb8+qvd7XodumlAuxb20VGkot0/OxypHYfT6JDdyUpi41wh7AKJiCJJmbIKUoayJkWlKSOhUXAsdiIyqXT6xykeErhAspfiCUgsaUgZhycjQ/ZxklhUkBEiTW4qOI4M4Un5LnmSIS4rkpIkGf+xdC5tkOkistmTyCTlZHa3l2QI4XQ//E0DliINh45gcaWR6YE0Z2jzEBFnL4o0CaGDpAUlkeN6uwifPS7Sq5vmsuNOVkht+y5Wf1xk2POx6W8S7XbH4bT4VL6TMBaYCtZffySVlhQ6FmRaJQrYsfGBEDnOmmR2OUJsqoEVNQ5LzlhUSEsAAltFIO3wUc8P55OdD1Fl0zMiGxkl8UElCPlC4hBQmhB2gSRtyBLLCyktMt2JUyrO706KSmVGSaNgkDTW08SnexMpU21cZHIkThkSh4ATlWSbjAzZx0liUaaC20JkspeLDAkm5fvcXkoQFUcm4fiPpXNpg0wXkc2eRCYpxwKSdEF1d8Qi0wPhbxqwFGk4bVEXVxqZHkhzhjYPESYylyqKtGOhg6Qpk8hxvV2ElxkL8+qmuey4kxVS276L1R8XGfZ8bPqbRLvdcTgtPpXvJIwFpoL11x9JpSWFjgWZVokCdmx8IESOsyaZXY4Qm2pgRY3DkjMWFdIuKWAljUajJUlHLAQgAAEIQGCTCXjrix3c5DZCNwhAAAIQWB4B2UFmCJeHF8kQgAAEIAABCEAAAhCAAAQ2mgAO4UY3D8pBAAIQgAAEIAABCEAAAhBYHgEcwuWxRTIEIAABCEAAAhCAAAQgAIGNJoBDuNHNg3IQgAAEIAABCEAAAhCAAASWRwCHcHlskQwBCEAAAhCAAAQgAAEIQGCjCeAQbnTzoBwEIAABCEAAAhCAAAQgAIHlEcAhXB5bJEMAAhCAAAQgAAEIQAACENhoAjiEG908KAcBCEAAAhCAAAQgAAEIQGB5BHAIl8cWyRCAAAQgAAEIQAACEIAABDaaAA7hRjcPykEAAhCAAAQgAAEIQAACEFgeARzC5bFFMgQgAAEIQAACEIAABCAAgY0mgEO40c2DchCAAAQgAAEIQAACEIAABJZHAIdweWyRDAEIQAACEIAABCAAAQhAYKMJ4BBudPOgHAQgAAEIQAACEIAABCAAgeURwCFcHlskQwACEIAABCAAAQhAAAIQ2GgCOIQb3TwoBwEIQAACEIAABCAAAQhAYHkEcAiXxxbJEIAABCAAAQhAAAIQgAAENpoADuFGNw/KQQACEIAABCAAAQhAAAIQWB4BHMLlsUUyBCAAAQhAAAIQgAAEIACBjSaAQ7jRzYNyEIAABCAAAQhAAAIQgAAElkcAh3B5bJEMAQhAAAIQgAAEIAABCEBgowngEG5086AcBCAAAQhAAAIQgAAEIACB5RHAIVweWyRDAAIQgAAEIAABCEAAAhDYaAI4hBvdPCgHAQhAAAIQgAAEIAABCEBgeQRwCJfHFskQgAAEIAABCEAAAhCAAAQ2mgAO4UY3D8pBAAIQgAAEIAABCEAAAhBYHgEcwuWxRTIEIAABCEAAAhCAAAQgAIGNJoBDuNHNg3IQgAAEIAABCEAAAhCAAASWRwCHcHlskQwBCEAAAhCAAAQgAAEIQAACEIAABCAAAQhAAAIQgAAEIACBzSPw/wHsm7CihxXgBAAAAABJRU5ErkJggg==", + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import pyvista as pv\n", + "import numpy as np\n", + "\n", + "\n", + "mesh = pv.read(\"./case27_surface_mesh.vtp\")\n", + "camera_position = [(-2.0, 2.0, 0.5), (-0.5, 0.12225, 0.15775), (0, 0, 1)]\n", + "\n", + "data_to_probe = pv.read(\"./results_nbk/graph_27.vtp\")\n", + "result = mesh.sample(data_to_probe)\n", + "\n", + "p_min = result.point_data[\"p_pred\"].min()\n", + "p_max = result.point_data[\"p_pred\"].max()\n", + "s_min = result.point_data[\"wallShearStress_pred\"].min()\n", + "s_max = result.point_data[\"wallShearStress_pred\"].max()\n", + "\n", + "plotter = pv.Plotter(shape=(2, 2), window_size=(1200, 1200))\n", + "\n", + "plotter.subplot(0, 0)\n", + "plotter.add_mesh(result, scalars=\"p_pred\", clim=[p_min, p_max])\n", + "plotter.add_text(\"Prediction: Pressure\", position=\"upper_left\", font_size=10)\n", + "plotter.camera_position = camera_position\n", + "\n", + "plotter.subplot(0, 1)\n", + "plotter.add_mesh(result, scalars=\"p\", clim=[p_min, p_max])\n", + "plotter.add_text(\"Ground Truth: Pressure\", position=\"upper_left\", font_size=10)\n", + "plotter.camera_position = camera_position\n", + "\n", + "plotter.subplot(1, 0)\n", + "plotter.add_mesh(result, scalars=\"wallShearStress_pred\", clim=[s_min, s_max])\n", + "plotter.add_text(\"Prediction: Wall Shear Stress\", position=\"upper_left\", font_size=10)\n", + "plotter.camera_position = camera_position\n", + "\n", + "plotter.subplot(1, 1)\n", + "plotter.add_mesh(result, scalars=\"wallShearStress\", clim=[s_min, s_max])\n", + "plotter.add_text(\"Ground Truth: Wall Shear Stress\", position=\"upper_left\", font_size=10)\n", + "plotter.camera_position = camera_position\n", + "\n", + "plotter.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Computing Drag Coefficient\n", + "\n", + "An important metric in aerodynamic analysis is drag coefficient. Prediction of accurate drag coefficient has implications on vehicle performance and efficiency. In the subsequent sections, we will compute the drag coefficient for this Ahmed body configuration. To compute that, we would need to compute the surface area at each cell, compute the cell normals and project the point-data to cells. This can be done using below" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
HeaderData Arrays
\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
PolyDataInformation
N Cells70805
N Points71174
N Strips0
X Bounds-1.019e+00, -9.082e-17
Y Bounds0.000e+00, 2.445e-01
Z Bounds0.000e+00, 3.155e-01
N Arrays15
\n", + "\n", + "		\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
NameFieldTypeN CompMinMax
vtkGhostTypePointsuint810.000e+000.000e+00
NormalsPointsfloat323-1.000e+001.000e+00
vtkGhostTypeCellsuint810.000e+000.000e+00
pCellsfloat321-1.526e+001.362e+00
p_predCellsfloat321-1.423e+001.326e+00
wallShearStressCellsfloat323-3.344e+002.999e+00
wallShearStress_predCellsfloat323-3.018e+002.415e+00
vtkValidPointMaskCellsfloat3211.000e+001.000e+00
wallShearStress_pred-normedCellsfloat3211.243e-013.042e+00
wallShearStress-normedCellsfloat3217.489e-023.360e+00
NormalsCellsfloat323-1.000e+001.000e+00
LengthCellsfloat6410.000e+000.000e+00
AreaCellsfloat6412.335e-062.950e-05
VolumeCellsfloat6410.000e+000.000e+00
TimeValueFieldsfloat3215.000e+035.000e+03
\n", + "\n", + "
" + ], + "text/plain": [ + "PolyData (0x7fc6348cd180)\n", + " N Cells: 70805\n", + " N Points: 71174\n", + " N Strips: 0\n", + " X Bounds: -1.019e+00, -9.082e-17\n", + " Y Bounds: 0.000e+00, 2.445e-01\n", + " Z Bounds: 0.000e+00, 3.155e-01\n", + " N Arrays: 15" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "result = result.point_data_to_cell_data()\n", + "result = result.compute_normals()\n", + "result = result.compute_cell_sizes()\n", + "result" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "0.23073415190620303 0.20531973401498171\n" + ] + } + ], + "source": [ + "import json\n", + "\n", + "from physicsnemo.metrics.cae.cfd import compute_force_coefficients, compute_frontal_area\n", + "\n", + "# Load the stats to denormalize the data\n", + "f = open(\"node_stats.json\")\n", + "stats = json.load(f)\n", + "\n", + "# Load case info to read velocity and relevant info\n", + "data = np.loadtxt(\"./dataset/test_info/case27_info.txt\", usecols=(2), max_rows=7)\n", + "velocity = data[-1]\n", + "\n", + "p_true = result.cell_data[\"p\"] * stats[\"p_std\"] + stats[\"p_mean\"]\n", + "p_pred = result.cell_data[\"p_pred\"] * stats[\"p_std\"] + stats[\"p_mean\"]\n", + "wss_true = (\n", + " result.cell_data[\"wallShearStress\"] * stats[\"wallShearStress_std\"]\n", + " + stats[\"wallShearStress_mean\"]\n", + ")\n", + "wss_pred = (\n", + " result.cell_data[\"wallShearStress_pred\"] * stats[\"wallShearStress_std\"]\n", + " + stats[\"wallShearStress_mean\"]\n", + ")\n", + "\n", + "frontal_area = compute_frontal_area(result, direction=\"x\")\n", + "normals = result.cell_data[\"Normals\"]\n", + "areas = result.cell_data[\"Area\"]\n", + "\n", + "coeff = 2 / (frontal_area * 1.225 * velocity**2)\n", + "\n", + "cd_true, _, _ = compute_force_coefficients(\n", + " normals, areas, coeff, p_true, wss_true, np.array([1, 0, 0])\n", + ")\n", + "cd_pred, _, _ = compute_force_coefficients(\n", + " normals, areas, coeff, p_pred, wss_pred, np.array([1, 0, 0])\n", + ")\n", + "\n", + "print(cd_true, cd_pred)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This completes the inference analysis for the Ahmed body checkpoint. " + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + }, + "orig_nbformat": 4, + "vscode": { + "interpreter": { + "hash": "767d51c1340bd893661ea55ea3124f6de3c7a262a8b4abca0554b478b1e2ff90" + } + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/loggers.py b/examples/cfd/external_aerodynamics/aero_graph_net/loggers.py new file mode 100644 index 0000000000..d7e67faaea --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/loggers.py @@ -0,0 +1,108 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from abc import ABC, abstractmethod +import functools +import logging + +from hydra.utils import instantiate +from omegaconf import DictConfig, OmegaConf + +import wandb + +from physicsnemo.distributed import DistributedManager + + +def init_python_logging(config: DictConfig, rank: int = 0) -> None: + """Initializes Python logging.""" + + pylog_cfg = OmegaConf.select(config, "logging.python") + if pylog_cfg is None: + return + + # Set up Python loggers. + pylog_cfg.output = config.output + pylog_cfg.rank = rank + # Enable logging only on rank 0, if requested. + if pylog_cfg.rank0_only and pylog_cfg.rank != 0: + pylog_cfg.handlers = {} + pylog_cfg.loggers.agnet.handlers = [] + # Configure logging. + logging.config.dictConfig(OmegaConf.to_container(pylog_cfg, resolve=True)) + + +def rank0(func): + """Decorator that allows the function to be executed only in rank 0 process.""" + + @functools.wraps(func) + def rank0_only(*args, **kwargs): + if DistributedManager().rank == 0: + func(*args, **kwargs) + + return rank0_only + + +class ExperimentLogger(ABC): + """Provides unified interface to a logger.""" + + @abstractmethod + def log_scalar(self, tag: str, value: float, step: int) -> None: + pass + + @abstractmethod + def log_image(self, tag: str, value, step: int) -> None: + pass + + +class WandBLogger(ExperimentLogger): + """Wrapper for Weights & Biases logger.""" + + def __init__(self, **kwargs) -> None: + if DistributedManager().rank != 0: + return + wandb.init(**kwargs) + + @rank0 + def log_scalar(self, tag: str, value: float, step: int) -> None: + wandb.log({tag: value}, step=step) + + @rank0 + def log_image(self, tag: str, value, step: int) -> None: + wandb.log({tag: wandb.Image(value)}, step=step) + + +class CompositeLogger(ExperimentLogger): + """Wraps a list of loggers providing unified interface.""" + + loggers: dict[str, ExperimentLogger] = None + + def __init__(self, config: DictConfig) -> None: + if DistributedManager().rank != 0: + self.loggers = {} + return + # Instantiate loggers only when running on rank 0. + self.loggers = instantiate(config.loggers) + + @rank0 + def log_scalar(self, tag: str, value: float, step: int) -> None: + for logger in self.loggers.values(): + logger.log_scalar(tag, value, step) + + @rank0 + def log_image(self, tag: str, value: float, step: int) -> None: + for logger in self.loggers.values(): + logger.log_image(tag, value, step) diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/models.py b/examples/cfd/external_aerodynamics/aero_graph_net/models.py new file mode 100644 index 0000000000..363157c223 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/models.py @@ -0,0 +1,94 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from dataclasses import dataclass + +import torch +from torch import Tensor + +import physicsnemo.models.meshgraphnet.meshgraphnet as mgn + +from physicsnemo.core import ModelMetaData +from physicsnemo.nn.module.gnn_layers.utils import GraphType +from physicsnemo.nn import get_activation + + +@dataclass +class MetaData(ModelMetaData): + jit: bool = False + cuda_graphs: bool = False + amp_cpu: bool = False + amp_gpu: bool = True + torch_fx: bool = False + # Inference + onnx: bool = False + # Physics informed + func_torch: bool = True + auto_grad: bool = True + + +class AeroGraphNet(mgn.MeshGraphNet): + """A variant of MeshGraphNet model that also predicts a drag coefficient. + + This model is based on a standard PhysicsNeMo `MeshGraphNet` model + with additional output, C_d (drag coefficient). + """ + + def __init__( + self, + *args, + hidden_dim_processor: int = 128, + hidden_dim_node_decoder: int = 128, + num_layers_node_decoder: int | None = 2, + mlp_activation_fn: str | list[str] = "relu", + recompute_activation: bool = False, + **kwargs, + ): + super().__init__( + *args, + hidden_dim_processor=hidden_dim_processor, + hidden_dim_node_decoder=hidden_dim_node_decoder, + num_layers_node_decoder=num_layers_node_decoder, + mlp_activation_fn=mlp_activation_fn, + recompute_activation=recompute_activation, + **kwargs, + ) + # Update meta. + self.meta = MetaData() + + self.c_d_decoder = mgn.MeshGraphMLP( + hidden_dim_processor, + output_dim=1, + hidden_dim=hidden_dim_node_decoder, + hidden_layers=num_layers_node_decoder, + activation_fn=get_activation(mlp_activation_fn), + norm_type=None, + recompute_activation=recompute_activation, + ) + + def forward( + self, + node_features: Tensor, + edge_features: Tensor, + graph: GraphType, + **kwargs, + ) -> Tensor: + edge_features = self.edge_encoder(edge_features) + node_features = self.node_encoder(node_features) + x = self.processor(node_features, edge_features, graph) + c_d = torch.relu(self.c_d_decoder(x.mean(dim=0))) + x = self.node_decoder(x) + return {"graph": x, "c_d": c_d} diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/requirements.txt b/examples/cfd/external_aerodynamics/aero_graph_net/requirements.txt new file mode 100644 index 0000000000..87b1da6db5 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/requirements.txt @@ -0,0 +1,5 @@ +hydra-core>=1.3.0 +omegaconf>=2.3.0 +wandb>=0.13.7 +torch_geometric>=2.6.1 +torch_scatter>=2.1.2 diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/train.py b/examples/cfd/external_aerodynamics/aero_graph_net/train.py new file mode 100644 index 0000000000..0e4c1f5ce8 --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/train.py @@ -0,0 +1,273 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from collections import defaultdict +from functools import partial +import logging +import time +from typing import Mapping + +import hydra +from hydra.utils import instantiate, to_absolute_path +from omegaconf import DictConfig, OmegaConf + +import torch +from torch import Tensor +from torch.nn.parallel import DistributedDataParallel +from torch.utils.data.distributed import DistributedSampler + +from torch_geometric.loader import DataLoader as PyGDataLoader + +from physicsnemo.distributed.manager import DistributedManager +from physicsnemo.utils import load_checkpoint, save_checkpoint + +from loggers import CompositeLogger, ExperimentLogger, init_python_logging +from utils import batch_as_dict + + +logger = logging.getLogger("agnet") + +# Experiment logger will be set later during initialization. +elogger: ExperimentLogger = None + + +class MGNTrainer: + def __init__(self, cfg: DictConfig): + assert DistributedManager.is_initialized() + self.dist = DistributedManager() + + # instantiate training dataset + logger.info("Loading the training dataset...") + self.dataset = instantiate(cfg.data.train) + logger.info(f"Using {len(self.dataset)} training samples.") + + # instantiate validation dataset + logger.info("Loading the validation dataset...") + self.validation_dataset = instantiate(cfg.data.val) + logger.info(f"Using {len(self.validation_dataset)} validation samples.") + + logger.info("Creating the dataloaders...") + # instantiate training dataloader + train_dataloader_cfg = dict(cfg.train.dataloader) + sampler = DistributedSampler( + self.dataset, + shuffle=train_dataloader_cfg.pop("shuffle", True), + drop_last=train_dataloader_cfg.pop("drop_last", True), + num_replicas=self.dist.world_size, + rank=self.dist.rank, + ) + + self.dataloader = PyGDataLoader( + self.dataset, + sampler=sampler, + **train_dataloader_cfg, + ) + + # instantiate validation dataloader + self.validation_dataloader = PyGDataLoader( + self.validation_dataset, + **cfg.val.dataloader, + ) + + logger.info("Creating the model...") + # instantiate the model + self.model = instantiate(cfg.model) + + if cfg.compile.enabled: + self.model = torch.compile(self.model, **cfg.compile.args).to( + self.dist.device + ) + else: + self.model = self.model.to(self.dist.device) + + # distributed data parallel for multi-GPU/multi-node training + if self.dist.distributed: + self.model = DistributedDataParallel( + self.model, + device_ids=[self.dist.local_rank], + output_device=self.dist.device, + broadcast_buffers=self.dist.broadcast_buffers, + find_unused_parameters=self.dist.find_unused_parameters, + ) + # Set the original model getter to simplify access. + assert not hasattr(self.model, "model") + type(self.model).model = ( + (lambda m: m.module) if self.dist.distributed else (lambda m: m) + ) + + # enable train mode + self.model.train() + + # instantiate losses. + self.loss = instantiate(cfg.loss) + + # instantiate optimizer, and scheduler + self.optimizer = instantiate(cfg.optimizer, self.model.parameters()) + self.scheduler = instantiate(cfg.lr_scheduler, self.optimizer) + + self.scaler = instantiate(cfg.amp.scaler) + self.autocast = partial( + torch.amp.autocast, + "cuda", + enabled=cfg.amp.enabled, + dtype=hydra.utils.get_object(cfg.amp.autocast.dtype), + ) + + # load checkpoint + self.epoch_init = load_checkpoint( + to_absolute_path(cfg.resume_dir), + models=self.model.model(), + optimizer=self.optimizer, + scheduler=self.scheduler, + scaler=self.scaler, + device=self.dist.device, + ) + if self.dist.world_size > 1: + torch.distributed.barrier() + + self.visualizers = instantiate(cfg.visualizers) + + def train(self, batch: Mapping[str, Tensor]): + self.optimizer.zero_grad() + losses = self.forward(batch) + self.backward(losses["total"]) + self.scheduler.step() + return losses + + def forward(self, batch): + # forward pass + batch = dict(batch) + graph = batch.pop("graph") + with self.autocast(): + pred = batch_as_dict(self.model(graph.x, graph.edge_attr, graph, **batch)) + # Graph data (e.g. p and WSS) loss. + graph_loss = self.loss.graph(pred["graph"], graph.y) + losses = {"graph": graph_loss} + # Compute C_d loss, if requested. + if (pred_c_d := pred.get("c_d")) is not None: + c_d_loss = self.loss.c_d(pred_c_d, batch["c_d"]) + losses["c_d"] = c_d_loss + # Get total loss and detach intermediate losses. + total_loss = sum(losses.values()) + losses = {k: v.detach() for k, v in losses.items()} + losses["total"] = total_loss + + return losses + + def backward(self, loss): + # backward pass. + # If AMP is disabled, the scaler will fall back to the default behavior. + self.scaler.scale(loss).backward() + self.scaler.step(self.optimizer) + self.scaler.update() + + @torch.no_grad() + def validation(self, epoch: int): + losses_agg = defaultdict(float) + for batch in self.validation_dataloader: + batch = batch_as_dict(batch, self.dist.device) + graph = batch.pop("graph") + pred = batch_as_dict(self.model(graph.x, graph.edge_attr, graph, **batch)) + pred_g, gt_g = self.dataset.denormalize( + pred["graph"], graph.y, self.dist.device + ) + losses_agg["graph"] += self.loss.graph(pred_g, gt_g) + if (pred_c_d := pred.get("c_d")) is not None: + losses_agg["c_d"] += self.loss.c_d(pred_c_d, batch["c_d"]) + + losses_agg["total"] = sum(losses_agg.values()) + + # Visualize last batch. + for vis in self.visualizers.values(): + vis(graph, pred_g, gt_g, epoch, elogger) + + # Log losses. + num_batches = len(self.validation_dataloader) + loss_str = [] + for k, v in losses_agg.items(): + loss = v / num_batches + elogger.log_scalar(f"val/loss/{k}", loss, epoch) + loss_str.append(f"{k}: {loss:6.4f}") + + logger.info(f"Validation loss: {', '.join(loss_str)}") + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + init_python_logging(cfg, dist.rank) + logger.info(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + torch.set_float32_matmul_precision("high") + torch.backends.cudnn.benchmark = True + torch.backends.cudnn.allow_tf32 = True + + # initialize loggers + global elogger + elogger = CompositeLogger(cfg) + + trainer = MGNTrainer(cfg) + start = time.time() + logger.info("Training started...") + + for epoch in range(trainer.epoch_init + 1, cfg.train.epochs + 1): + losses_agg = defaultdict(float) + for batch in trainer.dataloader: + batch = batch_as_dict(batch, dist.device) + losses = trainer.train(batch) + for k, v in losses.items(): + losses_agg[k] += v.detach().cpu().numpy() + num_batches = len(trainer.dataloader) + for k, v in losses_agg.items(): + losses_agg[k] /= num_batches + + cur_lr = trainer.scheduler.get_last_lr()[0] + logger.info( + f"epoch: {epoch:5,}, loss: {losses_agg['total']:.5f}, " + f"lr: {cur_lr:.7f}, " + f"time per epoch: {(time.time() - start):5.2f}" + ) + for k, v in losses_agg.items(): + elogger.log_scalar(f"train/loss/{k}", v, epoch) + elogger.log_scalar("lr", cur_lr, epoch) + + # validation + # TODO(akamenev): redundant restriction, val should run on all ranks. + if dist.rank == 0: + trainer.validation(epoch) + + # save checkpoint + if dist.world_size > 1: + torch.distributed.barrier() + if dist.rank == 0 and epoch % cfg.train.checkpoint_save_freq == 0: + save_checkpoint( + cfg.output, + models=trainer.model.model(), + optimizer=trainer.optimizer, + scheduler=trainer.scheduler, + scaler=trainer.scaler, + epoch=epoch, + ) + logger.info(f"Saved model on rank {dist.rank}") + start = time.time() + logger.info("Training completed!") + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/utils.py b/examples/cfd/external_aerodynamics/aero_graph_net/utils.py new file mode 100644 index 0000000000..692edf437c --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/utils.py @@ -0,0 +1,71 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from typing import Any, Mapping, Optional + +import torch +from torch import Tensor + +from torch_geometric.data import Data as PyGData + + +class RRMSELoss(torch.nn.Module): + """Relative RMSE loss.""" + + def forward(self, pred: Tensor, target: Tensor): + return ( + torch.linalg.vector_norm(pred - target) / torch.linalg.vector_norm(target) + ).mean() + + +def batch_as_dict( + batch, device: Optional[torch.device | str] = None +) -> Mapping[str, Any]: + """Wraps provided batch in a dictionary, if needed. + + If `device` is not None, moves all Tensor items to the device. + """ + + batch = batch if isinstance(batch, Mapping) else {"graph": batch} + if device is None: + return batch + return { + k: v.to(device) if isinstance(v, (Tensor, PyGData)) else v + for k, v in batch.items() + } + + +def relative_lp_error(pred, y, p=2): + """ + Calculate relative L2 error norm + Parameters: + ----------- + pred: torch.Tensor + Prediction + y: torch.Tensor + Ground truth + Returns: + -------- + error: float + Calculated relative L2 error norm (percentage) on cpu + """ + + error = ( + torch.mean(torch.linalg.norm(pred - y, ord=p) / torch.linalg.norm(y, ord=p)) + .cpu() + .numpy() + ) + return error * 100 diff --git a/examples/cfd/external_aerodynamics/aero_graph_net/visualizers.py b/examples/cfd/external_aerodynamics/aero_graph_net/visualizers.py new file mode 100644 index 0000000000..fcffc530db --- /dev/null +++ b/examples/cfd/external_aerodynamics/aero_graph_net/visualizers.py @@ -0,0 +1,117 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import pyvista as pv + +import numpy as np + +from torch import Tensor +from torch_geometric.data import Data as PyGData + +from loggers import ExperimentLogger + + +class MeshVisualizer: + """Mesh visualizer. + + Visualizes mesh in 3x3 grid where each row contains 3 images, + (GT, prediction, abs error), using different camera positions. + """ + + def __init__(self, scalar: str, tag: str, camera_positions: list) -> None: + if scalar not in (supported_scalars := ["p", "wallShearStress"]): + raise ValueError( + f"Scalar {scalar} is not supported, must be from {supported_scalars}" + ) + self.scalar = scalar + self.tag = tag + self.camera_positions = camera_positions + + def __call__( + self, + graph: PyGData, + pred: Tensor, + gt: Tensor, + step: int, + elogger: ExperimentLogger, + ) -> None: + vertices = graph.pos[:, :3].cpu().numpy() + + assert self.scalar in ["p", "wallShearStress"] + if self.scalar == "p": + gt = gt[:, :1] + pred = pred[:, :1] + cmap = "jet" + scalar_clim = (-600, 400) + err_clim = (-10, 100) + else: + # For vector quantity, pyvista plotter will use vector magnitude. + gt = gt[:, 1:4] + pred = pred[:, 1:4] + cmap = "coolwarm" + scalar_clim = (0, 10) + err_clim = (0, 1) + + gt = gt.cpu().numpy() + pred = pred.cpu().numpy() + abs_err = np.abs(pred - gt) + + plotter = pv.Plotter(shape=(3, 3), off_screen=True) + + # TODO(akamenev): this is currently plotting point clouds as + # opposed to meshes. This limitation is due to PyG graph not storing faces. + def plot_point_cloud( + scalar, pc, cam_pos, cmap, clim, show_bar=False, text=None + ): + data = pv.PolyData(vertices) + data[scalar] = pc + plotter.add_points( + data, + scalars=scalar, + cmap=cmap, + clim=clim, + point_size=5, + show_scalar_bar=show_bar, + ) + plotter.camera_position = cam_pos + if text is not None: + plotter.add_text(text, position="upper_left") + + def plot_column(col, scalar, data, text, cmap, clim): + num_rows = 3 + for row in range(num_rows): + plotter.subplot(row, col) + text = text if row == 0 else None + show_bar = row == (num_rows - 1) + plot_point_cloud( + scalar, + data, + self.camera_positions[row], + cmap, + clim, + show_bar=show_bar, + text=text, + ) + + text = "Pressure" if self.scalar == "p" else "Wall Shear Stress" + plot_column(0, f"{self.scalar}_gt", gt, f"GT {text}", cmap, scalar_clim) + plot_column( + 1, f"{self.scalar}_pred", pred, f"Predicted {text}", cmap, scalar_clim + ) + plot_column(2, "abs_err", abs_err, "Abs Error", "jet", err_clim) + + img = plotter.screenshot() + elogger.log_image(self.tag, img, step) diff --git a/examples/cfd/external_aerodynamics/domino/README.md b/examples/cfd/external_aerodynamics/domino/README.md new file mode 100644 index 0000000000..f728c94a25 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/README.md @@ -0,0 +1,660 @@ +# DoMINO: Decomposable Multi-scale Iterative Neural Operator for External Aerodynamics + +DoMINO is a local, multi-scale, point-cloud based model architecture to model large-scale +physics problems such as external aerodynamics. The DoMINO model architecture takes STL +geometries as input and evaluates flow quantities such as pressure and +wall shear stress on the surface of the car as well as velocity fields and pressure +in the volume around it. The DoMINO architecture is designed to be a fast, accurate +and scalable surrogate model for large-scale industrial simulations. + +DoMINO uses local geometric information to predict solutions on discrete points. First, +a global geometry encoding is learnt from point clouds using a multi-scale, iterative +approach. The geometry representation takes into account both short- and long-range +depdencies that are typically encountered in elliptic PDEs. Additional information +as signed distance field (SDF), positional encoding are used to enrich the global encoding. +Next, discrete points are randomly sampled, a sub-region is constructed around each point +and the local geometry encoding is extracted in this region from the global encoding. +The local geometry information is learnt using dynamic point convolution kernels. +Finally, a computational stencil is constructed dynamically around each discrete point +by sampling random neighboring points within the same sub-region. The local-geometry +encoding and the computational stencil are aggregrated to predict the solutions on the +discrete points. + +A preprint describing additional details about the model architecture can be found here +[paper](https://arxiv.org/abs/2501.13350). + +## Prerequisites + +Install the required dependencies by running below: + +```bash +pip install -r requirements.txt +``` + +## Getting started with the DrivAerML example + +### Configuration basics + +DoMINO training and testing is managed through YAML configuration files +powered by Hydra. The base configuration file, `config.yaml` is located in `src/conf` +directory. + +To select a specific configuration, use the `--config-name` option when running +the scripts. +You can modify configuration options in two ways: + +1. **Direct Editing:** Modify the YAML files directly +2. **Command Line Override:** Use Hydra's `++` syntax to override settings at runtime + +For example, to change the training epochs (controlled by `train.epochs`): + +```bash +python train.py ++train.epochs=200 # Sets number of epochs to 200 +``` + +This modular configuration system allows for flexible experimentation while +maintaining reproducibility. + +#### Project logs + +Save and track project logs, experiments, tensorboard files etc. by specifying a +project directory with `project.name`. Tag experiments with `expt`. + +### Data + +#### Dataset details + +In this example, the DoMINO model is trained using DrivAerML dataset from the +[CAE ML Dataset collection](https://caemldatasets.org/drivaerml/). +This high-fidelity, open-source (CC-BY-SA) public dataset is specifically +designed for automotive aerodynamics research. It comprises 500 parametrically +morphed variants of the widely utilized DrivAer notchback generic vehicle. +Mesh generation and scale-resolving computational fluid dynamics (CFD) simulations +were executed using consistent and validated automatic workflows that represent +the industrial state-of-the-art. Geometries and comprehensive aerodynamic data +are published in open-source formats. For more technical details about this dataset, +please refer to their [paper](https://arxiv.org/pdf/2408.11969). + +#### Data Preprocessing + +`PhysicsNeMo` has a related project to help with data processing, called +[PhysicsNeMo-Curator](https://github.com/NVIDIA/physicsnemo-curator). +Using `PhysicsNeMo-Curator`, the data needed to train a DoMINO model can be setup easily. +Please refer to +[these instructions on getting started](https://github.com/NVIDIA/physicsnemo-curator?tab=readme-ov-file#what-is-physicsnemo-curator) +with `PhysicsNeMo-Curator`. + +Download the DrivAer ML dataset using the +[provided instructions in PhysicsNeMo-Curator](https://github.com/NVIDIA/physicsnemo-curator/blob/main/examples/external_aerodynamics/README.md#download-drivaerml-dataset). +The first step for running the DoMINO pipeline requires processing the raw data +(vtp, vtu and stl) into either Zarr or NumPy format for training. +Each of the raw simulations files are downloaded in `vtp`, `vtu` and `stl` formats. +For instructions on running data processing to produce a DoMINO training ready dataset, +please refer to +[How-to Curate data for DoMINO Model](https://github.com/NVIDIA/physicsnemo-curator/blob/main/examples/external_aerodynamics/README.md). + +Caching is implemented in +[`CachedDoMINODataset`](https://github.com/NVIDIA/physicsnemo/blob/main/physicsnemo/datapipes/cae/domino_datapipe.py#L1056). +Optionally, users can run `src/cache_data.py` to save outputs +of DoMINO datapipe in the `.npy` files. The DoMINO datapipe is set up to calculate +Signed Distance Field and Nearest Neighbor interpolations on-the-fly during +training. Caching will save these as a preprocessing step and can be used in +cases where the **STL surface meshes are upwards of 30 million cells**. +Data processing is parallelized and takes a couple of hours to write all the +processed files. + +The final processed dataset should be divided and saved into 2 directories, +for training and validation. + +#### Data Scaling factors + +DoMINO has several data-specific configuration tools that rely on some +knowledge of the dataset: + +- The output fields (the labels) are normalized during training to a mean + of zero and a standard deviation of one, averaged over the dataset. + The scaling is controlled by passing the `volume_factors` and + `surface_factors` values to the datapipe. +- The input locations are scaled by, and optionally cropped to, used defined + bounding boxes for both surface and volume. Whether cropping occurs, or not, + is controlled by the `sample_in_bbox` value of the datapipe. Normalization + to the bounding box is enabled with `normalize_coordinates`. By default, + both are set to true. The value of the boxes are configured in the + `config.yaml` file, and are configured separately for surface and volume. + +> Note: The datapipe module has a helper function `create_domino_dataset` +> with sensible defaults to help create a Domino Datapipe. + +To facilitate setting reasonable values of these, you can use the +`compute_statistics.py` script. This will load the core dataset as defined +in your `config.yaml` file, loop over several events (200, by default), and +both print and store the surface/volume field statistics as well as the +coordinate statistics. + +> Note that, for volumetric fields especially, the min/max found may be +> significantly outside the surface region. Many simulations extend volumetric +> sampling to far field, and you may instead want to crop significant amounts +> of volumetric distance. + +#### Training + +Specify the training and validation data paths, bounding box sizes etc. in the +`data` tab and the training configs such as epochs, batch size etc. +in the `train` tab. + +#### Testing + +The testing is directly carried out on raw files. +Specify the testing configs in the `test` tab. + +### Training the DoMINO model + +To train and test the DoMINO model on AWS dataset, follow these steps: + +1. Specify the configuration settings in `conf/config.yaml`. + +2. Run `train.py` to start the training. Modify data, train and model keys in config file. + If using cached data then use `conf/cached.yaml` instead of `conf/config.yaml`. + +3. Run `test.py` to test on `.vtp` / `.vtu`. Predictions are written to the same file. + Modify eval key in config file to specify checkpoint, input and output directory. + Important to note that the data used for testing is in the raw simulation format and + should not be processed to `.npy`. + +4. Download the validation results (saved in form of point clouds in `.vtp` / `.vtu` format), + and visualize in Paraview. + +**Training Guidelines:** + +- Duration: A couple of days on a single node of H100 GPU +- Checkpointing: Automatically resumes from latest checkpoint if interrupted +- Multi-GPU Support: Compatible with `torchrun` or MPI for distributed training +- If the training crashes because of OOO, modify the points sampled in volume + `model.volume_points_sample` and surface `model.volume_points_sample` + to manage memory requirements for your GPU +- The DoMINO model allows for training both volume and surface fields using a + single model but currently the recommendation is to train the volume and + surface models separately. This can be controlled through the `conf/config.yaml`. +- MSE loss for both volume and surface model gives the best results. +- Bounding box is configurable and will depend on the usecase. + The presets are suitable for the DriveAer-ML dataset. + +### Training with Domain Parallelism + +DoMINO has support for training and inference using domain parallelism in PhysicsNeMo, +via the `ShardTensor` mechanisms and pytorch's FSDP tools. `ShardTensor`, built on +PyTorch's `DTensor` object, is a domain-parallel-aware tensor that can live on multiple +GPUs and perform operations in a numerically consistent way. For more information +about the techniques of domain parallelism and `ShardTensor`, refer to PhysicsNeMo +tutorials such as [`ShardTensor`](https://docs.nvidia.com/deeplearning/physicsnemo/physicsnemo-core/api/physicsnemo.distributed.shardtensor.html). + +In DoMINO specifically, domain parallelism has been enabled in two ways, which +can be used concurrently or separately. First, the input sampled volumetric +and surface points can be sharded to accommodate higher resolution point sampling +Second, the latent space of the model - typically a regularlized grid - can be +sharded to reduce computational complexity of the latent processing. When training +with sharded models in DoMINO, the primary objective is to enable higher +resolution inputs and larger latent spaces without sacrificing +substantial compute time. + +When configuring DoMINO for sharded training, adjust the following parameters +from `src/conf/config.yaml`: + +```yaml +domain_parallelism: + domain_size: 2 + shard_grid: True + shard_points: True +``` + +The `domain_size` represents the number of GPUs used for each batch - setting +`domain_size: 1` is the standard training regime, and domain_parallelism +will be ignored. `shard_grid` and `shard_points` will enable domain +parallelism over the latent space and input/output points, respectively. + +Setting domain_size > 1 without specifying `shard_points=True` or `shard_grid=True` +will result in a runtime error during configuration - if you do not want to use +domain_parallelism, leave `domain_size=1`. + +### Performance Optimizations + +The training and inference scripts for DoMINO contain several performance +enhancements to accelerate the training and usage of the model. In this +section we'll highlight several of them, as well as how to customize them +if needed. + +#### Memory Pool Optimizations + +The preprocessor of DoMINO requires a computation of k Nearest Neighbors, +which is accelerated via the `cuml` Neighbors tool. By default, `cuml` and +`torch` both use memory allocation pools to speed up allocating tensors, but +they do not use the same pool. This means that during preprocessing, it's +possible for the kNN operation to spend a significant amount of time in +memory allocations - and further, it limits the available memory to `torch`. + +To mitigate this, by default in DoMINO we use the Rapids Memory Manager +([`rmm`](https://github.com/rapidsai/rmm)). If, for some reason, you wish +to disable this you can do so with an environment variable: + +```bash +export PHYSICSNEMO_DISABLE_RMM=True +``` + +Or remove this line from the training script: + +```python +from physicsnemo.utils.memory import unified_gpu_memory +``` + +> Note - why not make it configurable? We have to set up the shared memory +> pool allocation very early in the program, before the config has even +> been read. So, we enable by default and the opt-out path is via the +> environment. + +#### Reduced Volume Reads + +The dataset size for volumetric data can be quite substantial - DrivAerML, for +example, has mesh sizes of 160M points per example. Even though the models +do not process all 160M points, in order to down sample dynamically they all +must be read from disk - which can exceed bandwidth and CPU decoding capacity +on nodes with multiple GPUs. + +As a performance enhancement, DoMINO's data pipeline offers a mitigation: instead +of reading an entire volumetric mesh, during preprocessing we _shuffle_ the +volumetric inputs and outputs (in tandem) and subsequent reads choose random +slices of the volumetric data. By default, DoMINO will read about 100x more data +than necessary for the sampling size. This allows the pipeline to still apply +cuts for data inside of the bounding box, and further random sampling to improve +training stability. To enable/disable this parameter, set +`data.volume_sample_from_disk=True` (enable) or `False` (disable) + +> Note - if you volumetric data is not larger than a few million mesh points, +> pre-shuffling and sampling from disk is likely not necessary for you. + +`physicsnemo-curator` supports shuffling the volumetric data during preprocessing. +If, however, you've already preprocessed your data and just want to apply +shuffling, use the script at `src/shuffle_volumetric_curator_output.py` + +The shuffling script will also apply sharding to the output files, which +improves IO performance. So, `zarr>=3.0` is required to use the outputs from +curator. `src/shuffle_volumetric_curator_output.py` is meant to be an example of how +to apply shuffling, so modify and update as you need for your dataset. + +> If you have tensorstore installed (it's in `requirements.txt`), the data reader +> will work equally well with Zarr 2 or Zarr 3 files. + +#### Overall Performance + +DoMINO is a computationally complex and challenging workload. Over the course +of several releases, we have chipped away at performance bottlenecks to speed +up the training and inference time (with `inference_on_stl.py`). Overall +training performance has decreased from about 5 days to just over 4 hours, with +eight H100 GPUs. We hope these optimizations enable you to explore more +parameters and surrogate models; if there is a performance issue you see, +please open an issue on GitHub. + +![Results from DoMINO for RTWT SC demo](../../../../docs/img/domino_perf.png) + +### Example Training Results + +To provide an example of what a successful training should look like, we include here +some example results. Training curves may look similar to this: + +![Combined Training Curve](../../../../docs/img/domino/combined-training-curve.png) + +And, when evaluating the results on the validation dataset, this particular +run had the following L2 and R2 Metrics: + +| Metric | Surface Only | Combined | +|--------------------:|:------------:|:--------:| +| X Velocity | N/A | 0.086 | +| Y Velocity | N/A | 0.185 | +| Z Velocity | N/A | 0.197 | +| Volumetric Pressure | N/A | 0.106 | +| Turb. V | N/A | 0.134 | +| Surface Pressure | 0.101 | 0.105 | +| X-Tau (Shear) | 0.138 | 0.145 | +| Y-Tau (Shear) | 0.174 | 0.185 | +| Z-Tau (Shear) | 0.198 | 0.207 | +| Drag R2 | 0.983 | 0.975 | +| Lift R2 | 0.971 | 0.968 | + +With the PhysicsNeMo CFD tool, you can create plots of the lift and drag +forces computed by domino vs. the CFD Solver. For example, here is the drag force: + +![Draf Force R^2](../../../../docs/img/domino/drag-r2.jpg) + +### Training with Physics Losses + +DoMINO supports enforcing of PDE residuals as soft constraints. This can be used +to improve the model predictions' adherence to the governing laws of the problem +which include Continuity and Navier Stokes equations. + +Note, if you wish to modify the PDEs used for DoMINO, please edit the +`compute_physics_loss` function from `train.py` appropriately. + +#### Prerequisites for PDE residuals + +The computation of Physics residuals is supported using the PhysicsNeMo-Sym +library. Install it using + +```bash +pip install "Cython" +pip install "nvidia-physicsnemo.sym>2.1.0" --no-build-isolation +``` + +To execute the training using physics losses, run the `train.py` with the +configuration below + +```bash +torchrun --nproc_per_node= train.py \ + ++train.add_physics_loss=True ++model.num_neighbors_volume=8 +``` + +Note, the `num_neighbors_volume` is set to 8 to reduce the memory requirement. +Also, when the Physics losses are applied, it will automatically sample +`num_neighbors_volume // 2` additional points, for each point in +`num_neighbors_volume`. These are considered as "2-hop" neighbors, which are +required to compute the higher order gradients required for Navier-Stokes +equations. Hence, even if `num_neighbors_volume` is set to 8, for the fields, +it will sample `num_neighbors_volume (num_neighbors_volume // 2 ) + 1` (in this +case 40) total points. + +The results of physics addition can be found below (using the DrivAerML +dataset). The results are computed on the design ID 419 and 439 from the +validation set and averaged. + +We observe that, addition of physics losses improves the model +predictions' ability to respect the governing laws better. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
L2 Errors
TypeVariableBaseline (full dataset)Baseline + Physics (full dataset)
Volumep0.154130.17203
U_x0.155660.16397
U_y0.322290.34383
U_z0.310270.32450
nut0.210490.21883
Surfacep0.160030.14298
wss_x0.214760.20519
wss_y0.316970.30335
wss_z0.350560.32095
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Residual L2 Error (Computed w.r.t true Residuals)
VariableBaseline (full dataset)Baseline + Physics (full dataset)% Improvement
continuity30.3520722.1126293.04%
momentum_x19.1092782.3380087.77%
momentum_y99.366623.1845296.80%
momentum_z45.738622.69172594.11%
+ + +*Addition of physics constraints to the DoMINO training is under active +development and might introduce breaking changes in the future* + +### Retraining recipe for DoMINO model + +To enable retraining the DoMINO model from a pre-trained checkpoint, follow the steps: + +1. Add the pre-trained checkpoints in the resume_dir defined in `conf/config.yaml`. + +2. Add the volume and surface scaling factors to the output dir defined in `conf/config.yaml`. + +3. Run `retraining.py` for specified number of epochs to retrain model at a small + learning rate starting from checkpoint. + +4. Run `test.py` to test on `.vtp` / `.vtu`. Predictions are written to the same file. + Modify eval key in config file to specify checkpoint, input and output directory. + +5. Download the validation results (saved in form of point clouds in `.vtp` / `.vtu` format), + and visualize in Paraview. + +### DoMINO model pipeline for inference on test samples + +After training is completed, `test.py` script can be used to run inference on +test samples. Follow the below steps to run the `test.py` + +1. Update the config in the `conf/config.yaml` under the `Testing data Configs` + tab. + +2. The test script is designed to run inference on the raw `.stl`, `.vtp` and + `.vtu` files for each test sample. Use the same scaling parameters that + were generated during the training. Typically this is `outputs//`, + where `project.name` is as defined in the `config.yaml`. Update the + `eval.scaling_param_path` accordingly. + +3. Run the `test.py`. The test script can be run in parallel as well. Refer to + the training guidelines for Multi-GPU. Note, for running `test.py` in parallel, + the number of GPUs chosen must be <= the number of test samples. + +### DoMINO model pipeline for inference on STLs + +The DoMINO model can be evaluated directly on unknown STLs using the pre-trained + checkpoint. Follow the steps outlined below: + +1. Run the `inference_on_stl.py` script to perform inference on an STL. + +2. Specify the STL paths, velocity inlets, stencil size and model checkpoint + path in the script. + +3. The volume predictions are carried out on points sampled in a bounding box around STL. + +4. The surface predictions are carried out on the STL surface. The drag and lift + accuracy will depend on the resolution of the STL. + +### Incorporating multiple global simulation parameters for training/inference + +DoMINO supports incorporating multiple global simulation parameters (such as inlet +velocity, air density, etc.) that can vary across different simulations. + +1. Define global parameters in the `variables.global_parameters` section of + `conf/config.yaml`. Each parameter must specify its type (`vector` or `scalar`) + and reference values for non-dimensionalization. + +2. For `vector` type parameters: + - If values are single-direction vectors (e.g., [30, 0, 0]), define reference as [30] + - If values are two-direction vectors (e.g., [30, 30, 0]), define reference as [30, 30] + +3. Enable parameter encoding in the model configuration by setting + `model.encode_parameters: true`. This will: + - Create a dedicated parameter encoding network (`ParameterModel`) + - Non-dimensionalize parameters using reference values from `config.yaml` + - Integrate parameter encodings into both surface and volume predictions + +4. Ensure your simulation data includes global parameter values. The DoMINO + datapipe expects these parameters in the pre-processed `.npy`/`.npz` files: + - Examine `openfoam_datapipe.py` and `process_data.py` for examples of how global + parameter values are incorporated for external aerodynamics + - For the automotive example, `air_density` and `inlet_velocity` remain constant + across simulations + - Adapt these files for your specific case to correctly calculate + `global_params_values` and `global_params_reference` during data preprocessing + +5. During training, the model automatically handles global parameter encoding when + `model.encode_parameters: true` is set + - You may need to adapt `train.py` if you plan to use global parameters in loss + functions or de-non-dimensionalization + +6. During testing with `test.py`, define `global_params_values` for each test sample: + - Global parameters must match those defined in `config.yaml` + - For each parameter (e.g., "inlet_velocity", "air_density"), provide appropriate + values for each simulation + - See the `main()` function in `test.py` for implementation examples + - If using global parameters for de-non-dimensionalization, modify `test_step()` + +7. When inferencing on unseen geometries with `inference_on_stl.py`: + - Define `global_params_values` and `global_params_reference` in both + `compute_solution_in_volume()` and `compute_solution_on_surface()` methods + - Adjust these parameters based on your specific use case and parameters defined + in `config.yaml` + +## Extending DoMINO to a custom dataset + +This repository includes examples of **DoMINO** training on the DrivAerML dataset. +However, many use cases require training **DoMINO** on a **custom dataset**. +The steps below outline the process. + +1. Reorganize that dataset to have the same directory structure as DrivAerML. The + raw data directory should contain a sepearte directory for each simulation. + Each simulation directory needs to contain mainly 3 files, `stl`, `vtp` and `vtu`, + correspoinding to the geometry, surface and volume fields information. + Additional details such as boundary condition information, for example inlet velocity, + may be added in a separate `.csv` file, in case these vary from one case to the next. +2. Modify the following parameters in `conf/config.yaml` + - `project.name`: Specify a name for your project. + - `expt`: This is the experiment tag. + - `data_processor.input_dir`: Input directory where the raw simulation dataset is stored. + - `data_processor.output_dir`: Output directory to save the processed dataset (`.npy`). + - `data_processor.num_processors`: Number of parallel processors for data processing. + - `variables.surface`: Variable names of surface fields and fields type (vector or scalar). + - `variables.volume`: Variable names of volume fields and fields type (vector or scalar). + - `data.input_dir`: Processed files used for training. + - `data.input_dir_val`: Processed files used for validation. + - `data.bounding_box`: Dimensions of computational domain where most prominent solution + field variations. Volume fields are modeled inside this bounding box. + - `data.bounding_box_surface`: Dimensions of bounding box enclosing the biggest geometry + in dataset. Surface fields are modeled inside this bounding box. + - `train.epochs`: Set the number of training epochs. + - `model.volume_points_sample`: Number of points to sample in the volume mesh per epoch + per batch. + Tune based on GPU memory. + - `model.surface_points_sample`: Number of points to sample on the surface mesh per epoch + per batch. + Tune based on GPU memory. + - `model.geom_points_sample`: Number of points to sample on STL mesh per epoch per batch. + Ensure point sampled is lesser than number of points on STL (for coarser STLs). + - `eval.test_path`: Path of directory of raw simulations files for testing and verification. + - `eval.save_path`: Path of directory where the AI predicted simulations files are saved. + - `eval.checkpoint_name`: Checkpoint name `outputs/{project.name}/models` to evaluate + model. + - `eval.scaling_param_path`: Scaling parameters populated in `outputs/{project.name}`. +3. Before running `process_data.py` to process the data, be sure to modify `openfoam_datapipe.py`. + This is the entry point for the user to modify the datapipe for dataprocessing. + A couple of things that might need to be changed are non-dimensionalizing schemes + based on the order of your variables and the `DrivAerAwsPaths` class with the + internal directory structure of your dataset. + For example, here is the custom class written for a different dataset. + + ```python + class DriveSimPaths: + # Specify the name of the STL in your dataset + @staticmethod + def geometry_path(car_dir: Path) -> Path: + return car_dir / "body.stl" + + # Specify the name of the VTU and directory structure in your dataset + @staticmethod + def volume_path(car_dir: Path) -> Path: + return car_dir / "VTK/simpleFoam_steady_3000/internal.vtu" + + # Specify the name of the VTP and directory structure in your dataset + @staticmethod + def surface_path(car_dir: Path) -> Path: + return car_dir / "VTK/simpleFoam_steady_3000/boundary/aero_suv.vtp" + ``` + +4. Before running `train.py`, modify the loss functions. The surface loss functions + currently, specifically `integral_loss_fn`, `loss_fn_surface` and `loss_fn_area`, + assume the variables to be in a specific order, Pressure followed by Wall-Shear-Stress + vector. + Please modify these formulations if your variables are in a different order + or don't require these losses. +5. Run `test.py` to validate the trained model. +6. Use `inference_on_stl.py` script to deploy the model in applications where inference is + needed only from STL inputs and the volume mesh is not calculated. + +The DoMINO model architecture is used to support the +[Real Time Digital Twin Blueprint](https://github.com/NVIDIA-Omniverse-blueprints/digital-twins-for-fluid-simulation) +and the +[DoMINO-Automotive-Aero NIM](https://catalog.ngc.nvidia.com/orgs/nim/teams/nvidia/containers/domino-automotive-aero). + +Some of the results are shown below. + +![Results from DoMINO for RTWT SC demo](../../../../docs/img/domino_result_rtwt.jpg) + +## References + +1. [DoMINO: A Decomposable Multi-scale Iterative Neural Operator for Modeling Large Scale Engineering Simulations](https://arxiv.org/abs/2501.13350) diff --git a/examples/cfd/external_aerodynamics/domino/requirements.txt b/examples/cfd/external_aerodynamics/domino/requirements.txt new file mode 100644 index 0000000000..d1970aef76 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/requirements.txt @@ -0,0 +1,6 @@ +torchinfo +warp-lang +tensorboard +cuml-cu13==25.10.* +einops +tensorstore \ No newline at end of file diff --git a/examples/cfd/external_aerodynamics/domino/src/benchmark_dataloader.py b/examples/cfd/external_aerodynamics/domino/src/benchmark_dataloader.py new file mode 100644 index 0000000000..e4fa13f175 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/benchmark_dataloader.py @@ -0,0 +1,184 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code defines a distributed pipeline for training the DoMINO model on +CFD datasets. It includes the computation of scaling factors, instantiating +the DoMINO model and datapipe, automatically loading the most recent checkpoint, +training the model in parallel using DistributedDataParallel across multiple +GPUs, calculating the loss and updating model parameters using mixed precision. +This is a common recipe that enables training of combined models for surface and +volume as well either of them separately. Validation is also conducted every epoch, +where predictions are compared against ground truth values. The code logs training +and validation metrics to TensorBoard. The train tab in config.yaml can be used to +specify batch size, number of epochs and other training parameters. +""" + +import time +import os +import re +import torch +import torchinfo + +from typing import Literal, Any + + +import hydra +from hydra.utils import to_absolute_path +from omegaconf import DictConfig, OmegaConf + +# This will set up the cupy-ecosystem and pytorch to share memory pools +from physicsnemo.utils.memory import unified_gpu_memory + + +import torch.distributed as dist +from torch.cuda.amp import GradScaler, autocast +from torch.nn.parallel import DistributedDataParallel +from torch.utils.data import DataLoader +from torch.utils.data.distributed import DistributedSampler +from torch.utils.tensorboard import SummaryWriter +from nvtx import annotate as nvtx_annotate +import torch.cuda.nvtx as nvtx + + +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, RankZeroLoggingWrapper + +from physicsnemo.datapipes.cae.domino_datapipe import ( + DoMINODataPipe, + compute_scaling_factors, + create_domino_dataset, +) +from physicsnemo.models.domino.model import DoMINO +from physicsnemo.models.domino.utils import * + +# This is included for GPU memory tracking: +from pynvml import nvmlInit, nvmlDeviceGetHandleByIndex, nvmlDeviceGetMemoryInfo +import time + +from utils import ( + ScalingFactors, + get_keys_to_read, + coordinate_distributed_environment, + load_scaling_factors, +) + + +from physicsnemo.utils.profiling import profile, Profiler + + +def benchmark_io_epoch( + dataloader, + logger, + gpu_handle, + epoch_index, + device, +): + dist = DistributedManager() + + # If you tell the dataloader the indices in advance, it will preload + # and pre-preprocess data + # dataloader.set_indices(indices) + + gpu_start_info = nvmlDeviceGetMemoryInfo(gpu_handle) + start_time = time.perf_counter() + for i_batch, sample_batched in enumerate(dataloader): + # Gather data and report + elapsed_time = time.perf_counter() - start_time + start_time = time.perf_counter() + gpu_end_info = nvmlDeviceGetMemoryInfo(gpu_handle) + gpu_memory_used = gpu_end_info.used / (1024**3) + gpu_memory_delta = (gpu_end_info.used - gpu_start_info.used) / (1024**3) + + logging_string = f"Device {device}, batch processed: {i_batch + 1}\n" + logging_string += f" GPU memory used: {gpu_memory_used:.3f} Gb\n" + logging_string += f" GPU memory delta: {gpu_memory_delta:.3f} Gb\n" + logging_string += f" Time taken: {elapsed_time:.2f} seconds\n" + logger.info(logging_string) + gpu_start_info = nvmlDeviceGetMemoryInfo(gpu_handle) + + return + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + # Initialize NVML + nvmlInit() + + gpu_handle = nvmlDeviceGetHandleByIndex(dist.device.index) + + model_type = cfg.model.model_type + + logger = PythonLogger("Train") + logger = RankZeroLoggingWrapper(logger, dist) + + logger.info(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + ################################ + # Get scaling factors + ################################ + vol_factors, surf_factors = load_scaling_factors(cfg) + + keys_to_read, keys_to_read_if_available = get_keys_to_read( + cfg, model_type, get_ground_truth=True + ) + + domain_mesh, data_mesh, placements = coordinate_distributed_environment(cfg) + + train_dataset = create_domino_dataset( + cfg, + phase="train", + keys_to_read=keys_to_read, + keys_to_read_if_available=keys_to_read_if_available, + vol_factors=vol_factors, + surf_factors=surf_factors, + device_mesh=domain_mesh, + placements=placements, + ) + train_sampler = DistributedSampler( + train_dataset, num_replicas=data_mesh.size(), rank=data_mesh.get_local_rank() + ) + + for epoch in range(0, cfg.train.epochs): + start_time = time.perf_counter() + logger.info(f"Device {dist.device}, epoch {epoch}:") + + train_sampler.set_epoch(epoch) + + train_dataset.dataset.set_indices(list(train_sampler)) + + epoch_start_time = time.perf_counter() + with Profiler(): + benchmark_io_epoch( + dataloader=train_dataset, + logger=logger, + gpu_handle=gpu_handle, + epoch_index=epoch, + device=dist.device, + ) + epoch_end_time = time.perf_counter() + logger.info( + f"Device {dist.device}, Epoch {epoch} took {epoch_end_time - epoch_start_time:.3f} seconds" + ) + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino/src/cache_data.py b/examples/cfd/external_aerodynamics/domino/src/cache_data.py new file mode 100644 index 0000000000..026a7985c6 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/cache_data.py @@ -0,0 +1,162 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This script processes DoMINODataPipe format files into cached versions +for faster loading during training. It processes files in parallel and can be +configured through config.yaml in the data_processing tab. +""" + +from physicsnemo.datapipes.cae.domino_datapipe import ( + DoMINODataPipe, + compute_scaling_factors, +) +import hydra +import time +import numpy as np +import os +from pathlib import Path +from omegaconf import DictConfig +import torch +from torch.utils.data import DataLoader +from torch.utils.data.distributed import DistributedSampler +from physicsnemo.distributed import DistributedManager + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + compute_scaling_factors(cfg, cfg.data_processor.output_dir, use_cache=False) + assert cfg.data_processor.use_cache, "Cache must be enabled for cache processing!" + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + vol_save_path = os.path.join(cfg.project_dir, "volume_scaling_factors.npy") + + surf_save_path = os.path.join(cfg.project_dir, "surface_scaling_factors.npy") + if os.path.exists(vol_save_path): + vol_factors = np.load(vol_save_path) + else: + vol_factors = None + if os.path.exists(surf_save_path): + surf_factors = np.load(surf_save_path) + else: + surf_factors = None + + # Set up variables based on model type + model_type = cfg.model.model_type + volume_variable_names = [] + surface_variable_names = [] + + if model_type in ["volume", "combined"]: + volume_variable_names = list(cfg.variables.volume.solution.keys()) + if model_type in ["surface", "combined"]: + surface_variable_names = list(cfg.variables.surface.solution.keys()) + + # Create dataset once + dataset = DoMINODataPipe( + data_path=cfg.data_processor.output_dir, # Caching comes after data processing + phase="train", # Phase doesn't matter for caching + grid_resolution=cfg.model.interp_res, + volume_variables=volume_variable_names, + surface_variables=surface_variable_names, + normalize_coordinates=True, + sampling=False, + sample_in_bbox=True, + volume_points_sample=cfg.model.volume_points_sample, + surface_points_sample=cfg.model.surface_points_sample, + geom_points_sample=cfg.model.geom_points_sample, + positional_encoding=cfg.model.positional_encoding, + volume_factors=vol_factors, + surface_factors=surf_factors, + scaling_type=cfg.model.normalization, + model_type=cfg.model.model_type, + bounding_box_dims=cfg.data.bounding_box, + bounding_box_dims_surf=cfg.data.bounding_box_surface, + num_surface_neighbors=cfg.model.num_surface_neighbors, + resample_surfaces=False, + for_caching=True, + deterministic_seed=True, + surface_sampling_algorithm=cfg.model.surface_sampling_algorithm, + ) + + # Create output directory on rank 0 + output_dir = Path(cfg.data_processor.cached_dir) + if dist.rank == 0: + output_dir.mkdir(parents=True, exist_ok=True) + + # Wait for directory creation + if dist.world_size > 1: + torch.distributed.barrier() + + # Create distributed sampler + sampler = DistributedSampler( + dataset, + num_replicas=dist.world_size, + rank=dist.rank, + shuffle=False, # No need to shuffle for preprocessing + ) + + # Create dataloader with distributed sampler + dataloader = DataLoader( + dataset, + sampler=sampler, + batch_size=1, # Process one at a time for caching + num_workers=0, # Must be 0 due to GPU operations in dataset + ) + + # Process and cache files + for _, sample in enumerate(dataloader): + filename = sample["filename"][ + 0 + ] # batch size 1, we can just pull out the filename + output_file = output_dir / f"{filename}_cached.npy" + + if output_file.exists(): + print(f"Rank {dist.rank}: Skipping {filename} - cache exists") + continue + + print(f"Rank {dist.rank}: Processing {filename}") + start_time = time.time() + + try: + # Remove batch dimension since we're processing one at a time + processed_data = {k: v[0] for k, v in sample.items()} + if cfg.model.model_type == "volume" or cfg.model.model_type == "combined": + print( + f"{filename}: volume min/max: {torch.amin(processed_data['volume_fields'], 0)}, {torch.amax(processed_data['volume_fields'], 0)}" + ) + if cfg.model.model_type == "surface" or cfg.model.model_type == "combined": + print( + f"{filename}: surface min/max: {torch.amin(processed_data['surface_fields'], 0)}, {torch.amax(processed_data['surface_fields'], 0)}" + ) + np.save(output_file, processed_data) + print( + f"Rank {dist.rank}: Completed {filename} in {time.time() - start_time:.2f}s" + ) + except Exception as e: + print(f"Rank {dist.rank}: Error processing {filename}: {str(e)}") + + # Wait for all processes to complete + if dist.world_size > 1: + torch.distributed.barrier() + + if dist.rank == 0: + print("All processing complete!") + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino/src/compute_statistics.py b/examples/cfd/external_aerodynamics/domino/src/compute_statistics.py new file mode 100644 index 0000000000..520fcfb9e9 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/compute_statistics.py @@ -0,0 +1,164 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +Compute and save scaling factors for DoMINO datasets. + +This script computes mean, standard deviation, minimum, and maximum values +for all field variables in a DoMINO dataset. The computed statistics are +saved in a structured format that can be easily loaded and used for +normalization during training and inference. + +The script uses the same configuration system as the training script, +ensuring consistency in dataset handling and processing parameters. +""" + +import os +import time +from pathlib import Path + +import hydra +import torch +from omegaconf import DictConfig, OmegaConf + +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils.logging import PythonLogger, RankZeroLoggingWrapper + +from physicsnemo.datapipes.cae.domino_datapipe import compute_scaling_factors +from utils import ScalingFactors + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + """ + Main function to compute and save scaling factors. + + Args: + cfg: Hydra configuration object containing all parameters + """ + ################################ + # Initialize distributed manager + ################################ + DistributedManager.initialize() + dist = DistributedManager() + + ################################ + # Initialize logger + ################################ + logger = PythonLogger("ComputeStatistics") + logger = RankZeroLoggingWrapper(logger, dist) + + logger.info("Starting scaling factors computation") + logger.info(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + ################################ + # Create output directory + ################################ + output_dir = os.path.dirname(cfg.data.scaling_factors) + os.makedirs(output_dir, exist_ok=True) + + if dist.world_size > 1: + torch.distributed.barrier() + + ################################ + # Check if scaling exists + ################################ + pickle_path = output_dir + "/scaling_factors.pkl" + + try: + scaling_factors = ScalingFactors.load(pickle_path) + logger.info(f"Scaling factors loaded from: {pickle_path}") + except FileNotFoundError: + logger.info(f"Scaling factors not found at: {pickle_path}; recomputing.") + scaling_factors = None + + ################################ + # Compute scaling factors + ################################ + if scaling_factors is None: + logger.info("Computing scaling factors from dataset...") + start_time = time.perf_counter() + + target_keys = [ + "volume_fields", + "surface_fields", + "stl_centers", + "volume_mesh_centers", + "surface_mesh_centers", + ] + + mean, std, min_val, max_val = compute_scaling_factors( + cfg=cfg, + input_path=cfg.data.input_dir, + target_keys=target_keys, + max_samples=cfg.data.max_samples_for_statistics, + ) + mean = {k: m.cpu().numpy() for k, m in mean.items()} + std = {k: s.cpu().numpy() for k, s in std.items()} + min_val = {k: m.cpu().numpy() for k, m in min_val.items()} + max_val = {k: m.cpu().numpy() for k, m in max_val.items()} + + compute_time = time.perf_counter() - start_time + logger.info( + f"Scaling factors computation completed in {compute_time:.2f} seconds" + ) + + ################################ + # Create structured data object + ################################ + dataset_info = { + "input_path": cfg.data.input_dir, + "model_type": cfg.model.model_type, + "normalization": cfg.model.normalization, + "compute_time": compute_time, + "timestamp": time.strftime("%Y-%m-%d %H:%M:%S"), + "config_name": cfg.project.name, + } + + scaling_factors = ScalingFactors( + mean=mean, + std=std, + min_val=min_val, + max_val=max_val, + field_keys=target_keys, + ) + + ################################ + # Save scaling factors + ################################ + if dist.rank == 0: + # Save as structured pickle file + pickle_path = output_dir + "/scaling_factors.pkl" + scaling_factors.save(pickle_path) + logger.info(f"Scaling factors saved to: {pickle_path}") + + # Save summary report + summary_path = output_dir + "/scaling_factors_summary.txt" + with open(summary_path, "w") as f: + f.write(scaling_factors.summary()) + logger.info(f"Summary report saved to: {summary_path}") + + ################################ + # Display summary + ################################ + logger.info("Scaling factors computation summary:") + logger.info(f"Field keys processed: {scaling_factors.field_keys}") + + logger.info("Scaling factors computation completed successfully!") + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino/src/conf/cached.yaml b/examples/cfd/external_aerodynamics/domino/src/conf/cached.yaml new file mode 100644 index 0000000000..f526644b25 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/conf/cached.yaml @@ -0,0 +1,35 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - config + - _self_ + +exp_tag: cached + +data: # Input directory for training and validation data + input_dir: /lustre/cached/drivaer_aws/drivaer_data_full/train/ + input_dir_val: /lustre/cached/drivaer_aws/drivaer_data_full/val/ +data_processor: + use_cache: true + +train: # Training configurable parameters + dataloader: + num_workers: 12 + +val: # Validation configurable parameters + dataloader: + num_workers: 6 \ No newline at end of file diff --git a/examples/cfd/external_aerodynamics/domino/src/conf/config.yaml b/examples/cfd/external_aerodynamics/domino/src/conf/config.yaml new file mode 100644 index 0000000000..e8921fd17f --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/conf/config.yaml @@ -0,0 +1,235 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ┌───────────────────────────────────────────┐ +# │ Project Details │ +# └───────────────────────────────────────────┘ +project: # Project name + name: DrivAerML_Dataset + +exp_tag: 1 # Experiment tag +# Main output directory. +project_dir: outputs/${project.name}/ +output: outputs/${project.name}/${exp_tag} + +hydra: # Hydra config + run: + dir: ${output} + output_subdir: hydra # Default is .hydra which causes files not being uploaded in W&B. + +# The directory to search for checkpoints to continue training. +resume_dir: ${output}/models + +# ┌───────────────────────────────────────────┐ +# │ Data Preprocessing │ +# └───────────────────────────────────────────┘ +data_processor: # Data processor configurable parameters + kind: drivaer_aws # must be either drivesim or drivaer_aws + output_dir: /user/aws_data_all/ + input_dir: /data/drivaer_aws/drivaer_data_full/ + cached_dir: /user/cached/drivaer_aws/drivaer_data_full/ + use_cache: false + num_processors: 12 + +# ┌───────────────────────────────────────────┐ +# │ Solution variables │ +# └───────────────────────────────────────────┘ +variables: + surface: + solution: + # The following is for AWS DrivAer dataset. + pMeanTrim: scalar + wallShearStressMeanTrim: vector + volume: + solution: + # The following is for AWS DrivAer dataset. + UMeanTrim: vector + pMeanTrim: scalar + nutMeanTrim: scalar + global_parameters: + inlet_velocity: + type: vector + reference: [30.00] # vector [30, 0, 0] should be specified as [30], while [30, 30, 0] should be [30, 30]. + air_density: + type: scalar + reference: 1.205 + +# ┌───────────────────────────────────────────┐ +# │ Data Configs │ +# └───────────────────────────────────────────┘ +data: # Input directory for training and validation data + input_dir: /user/data/aws_data_all/ + input_dir_val: /user/data/aws_data_all_val/ + bounding_box: # Bounding box dimensions for computational domain + min: [-3.5, -2.25, -0.32] + max: [8.5, 2.25, 3.00] + bounding_box_surface: # Bounding box dimensions for car surface + min: [-1.5, -1.4, -0.32] + max: [5.0, 1.4, 1.4] + gpu_preprocessing: true + gpu_output: true + normalize_coordinates: true + sample_in_bbox: true + sampling: true + scaling_factors: ${project_dir}/scaling_factors/scaling_factors.pkl + volume_sample_from_disk: true + max_samples_for_statistics: 200 + +# ┌───────────────────────────────────────────┐ +# │ Domain Parallelism Settings │ +# └───────────────────────────────────────────┘ +domain_parallelism: + domain_size: 1 + shard_grid: false + shard_points: false + +# ┌───────────────────────────────────────────┐ +# │ Model Parameters │ +# └───────────────────────────────────────────┘ +model: + model_type: combined # train which model? surface, volume, combined + activation: "gelu" # "relu" or "gelu" + loss_function: + loss_type: "mse" # mse or rmse + area_weighing_factor: 10000 # Generally inverse of maximum area + interp_res: [128, 64, 64] # resolution of latent space 128, 64, 48 + use_sdf_in_basis_func: true # SDF in basis function network + volume_points_sample: 8192 # Number of points to sample in volume per epoch + surface_points_sample: 8192 # Number of points to sample on surface per epoch + surface_sampling_algorithm: area_weighted #random or area_weighted + geom_points_sample: 300_000 # Number of points to sample on STL per epoch + num_neighbors_surface: 7 # How many neighbors on surface? + num_neighbors_volume: 10 # How many neighbors on volume? + combine_volume_surface: false # combine volume and surface encodings + return_volume_neighbors: false # Whether to return volume neighbors or not + use_surface_normals: true # Use surface normals and surface areas for surface computation? + use_surface_area: true # Use only surface normals and not surface area + integral_loss_scaling_factor: 100 # Scale integral loss by this factor + normalization: min_max_scaling # or mean_std_scaling + encode_parameters: false # encode inlet velocity and air density in the model + surf_loss_scaling: 5.0 # scale surface loss with this factor in combined mode + vol_loss_scaling: 1.0 # scale volume loss with this factor in combined mode + geometry_encoding_type: both # geometry encoder type, sdf, stl, both + solution_calculation_mode: two-loop # one-loop is better for sharded, two-loop is lower memory but more overhead. Physics losses are not supported via one-loop presently. + geometry_rep: # Hyperparameters for geometry representation network + geo_conv: + base_neurons: 32 # 256 or 64 + base_neurons_in: 1 + base_neurons_out: 1 + volume_radii: [0.1, 0.5, 1.0, 2.5] # radii for volume + surface_radii: [0.01, 0.05, 1.0] # radii for surface + surface_hops: 1 # Number of surface iterations + volume_hops: 1 # Number of volume iterations + volume_neighbors_in_radius: [32, 64, 128, 256] # Number of neighbors in radius for volume + surface_neighbors_in_radius: [8, 16, 128] # Number of neighbors in radius for surface + fourier_features: false + num_modes: 5 + activation: ${model.activation} + geo_processor: + base_filters: 8 + activation: ${model.activation} + processor_type: conv # conv or unet (conv is better; fno, fignet to be added) + self_attention: false # can be used only with unet + cross_attention: false # can be used only with unet + surface_sdf_scaling_factor: [0.01, 0.02, 0.04] # Scaling factor for SDF, smaller is more emphasis on surface + volume_sdf_scaling_factor: [0.04] # Scaling factor for SDF, smaller is more emphasis on surface + nn_basis_functions: # Hyperparameters for basis function network + base_layer: 512 + fourier_features: true + num_modes: 5 + activation: ${model.activation} + local_point_conv: + activation: ${model.activation} + aggregation_model: # Hyperparameters for aggregation network + base_layer: 512 + activation: ${model.activation} + position_encoder: # Hyperparameters for position encoding network + base_neurons: 512 + activation: ${model.activation} + fourier_features: true + num_modes: 5 + geometry_local: # Hyperparameters for local geometry extraction + volume_neighbors_in_radius: [64, 128] # Number of radius points + surface_neighbors_in_radius: [32, 128] # Number of radius points + volume_radii: [0.1, 0.25] # Volume radii + surface_radii: [0.05, 0.25] # Surface radii + base_layer: 512 + parameter_model: + base_layer: 512 + fourier_features: false + num_modes: 5 + activation: ${model.activation} + +# ┌───────────────────────────────────────────┐ +# │ Training Configs │ +# └───────────────────────────────────────────┘ +train: # Training configurable parameters + epochs: 1000 + checkpoint_interval: 2 + dataloader: + batch_size: 1 + preload_depth: 1 + pin_memory: True # if the preprocessing is outputting GPU data, set this to false + sampler: + shuffle: true + drop_last: false + checkpoint_dir: /user/models/ # deprecated: Use only for retraining + add_physics_loss: false + lr_scheduler: + name: MultiStepLR # Also supports CosineAnnealingLR + milestones: [50, 200, 400, 500, 600, 700, 800, 900] # only used if lr_scheduler is MultiStepLR + gamma: 0.5 # only used if lr_scheduler is MultiStepLR + T_max: ${train.epochs} # only used if lr_scheduler is CosineAnnealingLR + eta_min: 1e-6 # only used if lr_scheduler is CosineAnnealingLR + optimizer: + name: Adam # or AdamW + lr: 0.001 + weight_decay: 0.0 + amp: + enabled: true + autocast: + dtype: torch.float16 + scaler: + _target_: torch.cuda.amp.GradScaler + enabled: ${..enabled} + clip_grad: true + grad_max_norm: 2.0 + + +# ┌───────────────────────────────────────────┐ +# │ Validation Configs │ +# └───────────────────────────────────────────┘ +val: # Validation configurable parameters + dataloader: + batch_size: 1 + preload_depth: 1 + pin_memory: true # if the preprocessing is outputting GPU data, set this to false + sampler: + shuffle: true + drop_last: false + +# ┌───────────────────────────────────────────┐ +# │ Testing data Configs │ +# └───────────────────────────────────────────┘ +eval: # Testing configurable parameters + test_path: /user/testing_data # Dir for testing data in raw format (vtp, vtu ,stls) + save_path: /user/predicted_data # Dir to save predicted results in raw format (vtp, vtu) + checkpoint_name: DoMINO.0.455.pt # Name of checkpoint to select from saved checkpoints + scaling_param_path: /user/scaling_params + refine_stl: False # Automatically refine STL during inference + #TODO - This was hardcoded anyways, remove it. + # stencil_size: 7 # Stencil size for evaluating surface and volume model + num_points: 1_240_000 # Number of points to sample on surface and volume per batch diff --git a/examples/cfd/external_aerodynamics/domino/src/inference_on_stl.py b/examples/cfd/external_aerodynamics/domino/src/inference_on_stl.py new file mode 100644 index 0000000000..781e0e03ef --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/inference_on_stl.py @@ -0,0 +1,646 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code shows how to use a trained DoMINO model, with it's corresponding +preprocessing pipeline, to infer values on and around an STL mesh file. + +This script uses the meshes from the DrivaerML dataset, however, the logic +is largely the same. As an overview: +- Load the model +- Set up the preprocessor +- Loop over meshes +- In each mesh, sample random points on the surface, volume, or both +- Preprocess the points and run them through the model +- Process the STL mesh centers, too +- Collect the results and return +- Save the results to file. +""" + +import time +from typing import Literal, Any + +import hydra +from hydra.utils import to_absolute_path +from omegaconf import DictConfig, OmegaConf +import torch + +# This will set up the cupy-ecosystem and pytorch to share memory pools +from physicsnemo.utils.memory import unified_gpu_memory + +import torchinfo +from torch.utils.data.distributed import DistributedSampler + +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils import load_checkpoint +from physicsnemo.utils.logging import PythonLogger, RankZeroLoggingWrapper + +from physicsnemo.datapipes.cae.domino_datapipe import ( + DoMINODataPipe, + create_domino_dataset, +) + + +from physicsnemo.models.domino.model import DoMINO +from physicsnemo.models.domino.utils import sample_points_on_mesh + +from utils import ScalingFactors, get_keys_to_read, coordinate_distributed_environment + +# This is included for GPU memory tracking: +from pynvml import nvmlInit, nvmlDeviceGetHandleByIndex, nvmlDeviceGetMemoryInfo +import time + + +# Initialize NVML +nvmlInit() + + +from physicsnemo.utils.profiling import profile, Profiler + + +from loss import compute_loss_dict +from utils import get_num_vars + + +def reject_interior_volume_points( + preprocessed_data: dict[str, torch.Tensor], +) -> dict[str, torch.Tensor]: + """ + Reject volume points that are inside the STL mesh. + """ + ###################################################### + # Use the sign of the volume SDF to filter out points + # That are inside the STL mesh + ###################################################### + sdf_nodes = preprocessed_data["sdf_nodes"] + # The sfd_nodes tensor typically has shape (n_vol_points, 1) + valid_volume_idx = sdf_nodes > 0 + # So remove it if it's there: + valid_volume_idx = valid_volume_idx.squeeze(-1) + # Apply this selection to all the volume points: + for key in [ + "volume_mesh_centers", + "sdf_nodes", + "pos_volume_closest", + "pos_volume_center_of_mass", + ]: + preprocessed_data[key] = preprocessed_data[key][valid_volume_idx] + + return preprocessed_data + + +def sample_volume_points( + c_min: torch.Tensor, + c_max: torch.Tensor, + n_points: int, + device: torch.device, + eps: float = 1e-7, +) -> torch.Tensor: + """ + Generate a set of random points interior to the specified bounding box. + + Args: + c_min: The minimum coordinate of the bounding box. + c_max: The maximum coordinate of the bounding box. + n_points: The number of points to sample. + device: The device to sample the points on. + eps: The small edge factor to shift away from the lower bound. + """ + # We use a small edge factor to shift away from the lower bound, + # which can, in some cases, be exactly on the border. + uniform_points = ( + torch.rand(n_points, 3, device=device, dtype=torch.float32) * (1 - 2 * eps) + + eps + ) + sampled_volume_points = (c_max - c_min) * uniform_points + c_min + return sampled_volume_points + + +def inference_on_single_stl( + stl_coordinates: torch.Tensor, + stl_faces: torch.Tensor, + global_params_values: torch.Tensor, + global_params_reference: torch.Tensor, + model: DoMINO, + datapipe: DoMINODataPipe, + batch_size: int, + total_points: int, + gpu_handle: int | None = None, + logger: PythonLogger | None = None, +): + """ + Perform model inference on a single STL mesh. + + This function will take the input mesh + faces and + then sample the surface and volume to produce the model outputs + at `total_points` locations in batches of `batch_size`. + + + + Args: + stl_coordinates: The coordinates of the STL mesh. + stl_faces: The faces of the STL mesh. + global_params_values: The values of the global parameters. + global_params_reference: The reference values of the global parameters. + model: The model to use for inference. + datapipe: The datapipe to use for preprocessing. + batch_size: The batch size to use for inference. + total_points: The total number of points to process. + gpu_handle: The GPU handle to use for inference. + logger: The logger to use for logging. + """ + device = stl_coordinates.device + batch_start_time = time.perf_counter() + ###################################################### + # The IO only reads in "stl_faces" and "stl_coordinates". + # "stl_areas" and "stl_centers" would be computed by + # pyvista on CPU - instead, we do it on the GPU + # right here. + ###################################################### + + # Center is a mean of the 3 vertices + triangle_vertices = stl_coordinates[stl_faces.reshape((-1, 3))] + stl_centers = triangle_vertices.mean(dim=-1) + ###################################################### + # Area we compute from the cross product of two sides: + ###################################################### + d1 = triangle_vertices[:, 1] - triangle_vertices[:, 0] + d2 = triangle_vertices[:, 2] - triangle_vertices[:, 0] + stl_mesh_normals = torch.linalg.cross(d1, d2, dim=1) + normals_norm = torch.linalg.norm(stl_mesh_normals, dim=1) + stl_mesh_normals = stl_mesh_normals / normals_norm.unsqueeze(1) + stl_areas = 0.5 * normals_norm + + ###################################################### + # For computing the points, we take those stl objects, + # sample in chunks of `batch_size` until we've + # accumulated `total_points` predictions. + ###################################################### + + batch_output_dict = {} + N = 2 + total_points_processed = 0 + + # Use these lists to build up the output tensors: + surface_results = [] + volume_results = [] + + while total_points_processed < total_points: + inner_loop_start_time = time.perf_counter() + + ###################################################### + # Create the dictionary as the preprocessing expects: + ###################################################### + inference_dict = { + "stl_coordinates": stl_coordinates, + "stl_faces": stl_faces, + "stl_centers": stl_centers, + "stl_areas": stl_areas, + "global_params_values": global_params_values, + "global_params_reference": global_params_reference, + } + + # If the surface data is part of the model, sample the surface: + + if datapipe.model_type == "surface" or datapipe.model_type == "combined": + ###################################################### + # This function will sample points on the STL surface + ###################################################### + sampled_points, sampled_faces, sampled_areas, sampled_normals = ( + sample_points_on_mesh( + stl_coordinates, + stl_faces, + batch_size, + mesh_normals=stl_mesh_normals, + mesh_areas=stl_areas, + ) + ) + + inference_dict["surface_mesh_centers"] = sampled_points + inference_dict["surface_normals"] = sampled_normals + inference_dict["surface_areas"] = sampled_areas + inference_dict["surface_faces"] = sampled_faces + + # If the volume data is part of the model, sample the volume: + if datapipe.model_type == "volume" or datapipe.model_type == "combined": + ###################################################### + # Build up volume points too with uniform sampling + ###################################################### + c_min = datapipe.config.bounding_box_dims[1] + c_max = datapipe.config.bounding_box_dims[0] + inference_dict["volume_mesh_centers"] = sample_volume_points( + c_min, + c_max, + batch_size, + device, + ) + + ###################################################### + # Pre-process the data with the datapipe: + ###################################################### + preprocessed_data = datapipe.process_data(inference_dict) + + if datapipe.model_type == "volume" or datapipe.model_type == "combined": + preprocessed_data = reject_interior_volume_points(preprocessed_data) + + ###################################################### + # Add a batch dimension to the data_dict + # (normally this is added in __getitem__ of the datapipe) + ###################################################### + preprocessed_data = {k: v.unsqueeze(0) for k, v in preprocessed_data.items()} + + ###################################################### + # Forward pass through the model: + ###################################################### + with torch.no_grad(): + output_vol, output_surf = model(preprocessed_data) + + ###################################################### + # unnormalize the outputs with the datapipe + # Whatever settings are configured for normalizing the + # output fields - even though we don't have ground + # truth here - are reused to undo that for the predictions + ###################################################### + output_vol, output_surf = datapipe.unscale_model_outputs( + output_vol, output_surf + ) + + surface_results.append(output_surf) + volume_results.append(output_vol) + + total_points_processed += batch_size + + current_loop_time = time.perf_counter() + + logging_string = f"Device {device} processed {total_points_processed} points of {total_points}\n" + if gpu_handle is not None: + gpu_info = nvmlDeviceGetMemoryInfo(gpu_handle) + gpu_memory_used = gpu_info.used / (1024**3) + logging_string += f" GPU memory used: {gpu_memory_used:.3f} Gb\n" + + logging_string += f" Time taken since batch start: {current_loop_time - batch_start_time:.2f} seconds\n" + logging_string += f" iteration throughput: {batch_size / (current_loop_time - inner_loop_start_time):.1f} points per second\n" + logging_string += f" Batch mean throughput: {total_points_processed / (current_loop_time - batch_start_time):.1f} points per second.\n" + + if logger is not None: + logger.info(logging_string) + else: + print(logging_string) + + ###################################################### + # Here at the end, get the values for the stl centers + # by updating the previous inference dict + # Only do this if the surface is part of the computation + # Comments are shorter here - it's a condensed version + # of the above logic. + ###################################################### + if datapipe.model_type == "surface" or datapipe.model_type == "combined": + inference_dict = { + "stl_coordinates": stl_coordinates, + "stl_faces": stl_faces, + "stl_centers": stl_centers, + "stl_areas": stl_areas, + "global_params_values": global_params_values, + "global_params_reference": global_params_reference, + } + inference_dict["surface_mesh_centers"] = stl_centers + inference_dict["surface_normals"] = stl_mesh_normals + inference_dict["surface_areas"] = stl_areas + inference_dict["surface_faces"] = stl_faces + + if datapipe.model_type == "combined" or datapipe.model_type == "volume": + c_min = datapipe.config.bounding_box_dims[1] + c_max = datapipe.config.bounding_box_dims[0] + inference_dict["volume_mesh_centers"] = sample_volume_points( + c_min, + c_max, + stl_centers.shape[0], + device, + ) + + # Preprocess: + preprocessed_data = datapipe.process_data(inference_dict) + + # Pull out the invalid volume points again, if needed: + if datapipe.model_type == "combined" or datapipe.model_type == "volume": + preprocessed_data = reject_interior_volume_points(preprocessed_data) + + # Run the model forward: + with torch.no_grad(): + preprocessed_data = { + k: v.unsqueeze(0) for k, v in preprocessed_data.items() + } + _, output_surf = model(preprocessed_data) + + # Unnormalize the outputs: + _, stl_center_results = datapipe.unscale_model_outputs(None, output_surf) + + else: + stl_center_results = None + + # Stack up the results into one big tensor for surface and volume: + if len(surface_results) > 0 and all([s is not None for s in surface_results]): + surface_results = torch.cat(surface_results, dim=1) + else: + surface_results = None + if len(volume_results) > 0 and all([v is not None for v in volume_results]): + volume_results = torch.cat(volume_results, dim=1) + else: + volume_results = None + + return stl_center_results, surface_results, volume_results + + +def inference_epoch( + dataloader: DoMINODataPipe, + sampler: DistributedSampler, + model: DoMINO, + gpu_handle: int, + logger: PythonLogger, + batch_size: int = 24_000, + total_points: int = 1_024_000, +): + ###################################################### + # Inference can run in a distributed way by coordinating + # the indices for each rank, which the sampler does + ###################################################### + + batch_start_time = time.perf_counter() + + # N.B. - iterating over the dataset directly here. + # That's because we need to sample on the STL and volume and + # that means we'll preprocess after that. + for i_batch, sample_batched in enumerate(dataloader.dataset): + dataloading_time = time.perf_counter() - batch_start_time + + logger.info( + f"Batch {i_batch} data loading time: {dataloading_time:.3f} seconds" + ) + + procesing_time_start = time.perf_counter() + stl_center_results, surface_results, volume_results = inference_on_single_stl( + sample_batched["stl_coordinates"], + sample_batched["stl_faces"], + sample_batched["global_params_values"], + sample_batched["global_params_reference"], + model, + dataloader, + batch_size, + total_points, + gpu_handle, + logger, + ) + + ###################################################### + # Peel off pressure, velocity, nut, shear, etc. + # Also compute drag, lift forces. + ###################################################### + # TODO + # TODO + # TODO + # TODO + # TODO + # TODO + # TODO + + procesing_time_end = time.perf_counter() + logger.info( + f"Batch {i_batch} GPU processing time: {procesing_time_end - procesing_time_start:.3f} seconds" + ) + logger.info( + f"Batch {i_batch} stl points: {stl_center_results.shape[1] if stl_center_results is not None else 0}" + ) + + output_start_time = time.perf_counter() + ###################################################### + # Save the outputs to file: + ###################################################### + # TODO + # TODO + # TODO + # TODO + # TODO + # TODO + output_end_time = time.perf_counter() + logger.info( + f"Batch {i_batch} output time: {output_end_time - output_start_time:.3f} seconds" + ) + + batch_start_time = time.perf_counter() + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + ###################################################### + # initialize distributed manager + ###################################################### + DistributedManager.initialize() + dist = DistributedManager() + + # DoMINO supports domain parallel training and inference. This function helps coordinate + # how to set that up, if needed. + domain_mesh, data_mesh, placements = coordinate_distributed_environment(cfg) + + ###################################################### + # Initialize NVML + ###################################################### + nvmlInit() + gpu_handle = nvmlDeviceGetHandleByIndex(dist.device.index) + + ###################################################### + # Initialize logger + ###################################################### + + logger = PythonLogger("Inference") + logger = RankZeroLoggingWrapper(logger, dist) + + logger.info(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + ###################################################### + # Get scaling factors + # Likely, you want to reuse the scaling factors from training. + ###################################################### + vol_factors, surf_factors = load_scaling_factors(cfg) + + ###################################################### + # Configure the model + ###################################################### + model_type = cfg.model.model_type + num_vol_vars, num_surf_vars, num_global_features = get_num_vars(cfg, model_type) + + if model_type == "combined" or model_type == "surface": + surface_variable_names = list(cfg.variables.surface.solution.keys()) + else: + surface_variable_names = [] + + if model_type == "combined" or model_type == "volume": + volume_variable_names = list(cfg.variables.volume.solution.keys()) + else: + volume_variable_names = [] + + ###################################################### + # Check that the sample size is equal. + # unequal samples could be done but they aren't, here.s + ###################################################### + if cfg.model.model_type == "combined": + if cfg.model.volume_points_sample != cfg.model.surface_points_sample: + raise ValueError( + "Volume and surface points sample must be equal for combined model" + ) + + # Get the number of sample points: + sample_points = ( + cfg.model.surface_points_sample + if cfg.model.model_type == "surface" + else cfg.model.volume_points_sample + ) + + ###################################################### + # If the batch size doesn't evenly divide + # the num points, that's ok. But print a warning + # that the total points will get tweaked. + ###################################################### + if cfg.eval.num_points % sample_points != 0: + logger.warning( + f"Batch size {sample_points} doesn't evenly divide num points {cfg.eval.num_points}." + ) + logger.warning( + f"Total points will be rounded up to {((cfg.eval.num_points // sample_points) + 1) * sample_points}." + ) + + ###################################################### + # Configure the dataset + # We are applying preprocessing in a separate step + # for this - so the dataset and datapipe are separate + ###################################################### + + # This helper function is to determine which keys to read from the data + # (and which to use default values for, if they aren't present - like + # air_density, for example) + keys_to_read, keys_to_read_if_available = get_keys_to_read( + cfg, model_type, get_ground_truth=True + ) + # Override the model type + # For the inference pipeline, we adjust the tooling a little for the data. + # We use only a bare STL dataset that will read the mesh coordinates + # and triangle definitions. We'll compute the centers and normals + # on the GPU (instead of on the CPU, as pyvista would do) and + # then we can sample from that mesh on the GPU. + # test_dataset = DrivaerMLDataset( + # data_dir=cfg.eval.test_path, + # keys_to_read=[ + # "stl_coordinates", + # "stl_faces", + # ], + # keys_to_read_if_available=keys_to_read_if_available, + # output_device=dist.device, + # ) + + # Volumetric data will be generated on the fly on the GPU. + + ###################################################### + # Configure the datapipe + # We _won't_ iterate over the datapipe, however, we can use the + # datapipe processing tools on the sampled surface and + # volume points with the same preprocessing. + # It also is used to un-normalize the model outputs. + ###################################################### + overrides = {} + if hasattr(cfg.data, "gpu_preprocessing"): + overrides["gpu_preprocessing"] = cfg.data.gpu_preprocessing + + if hasattr(cfg.data, "gpu_output"): + overrides["gpu_output"] = cfg.data.gpu_output + + test_dataloader = create_domino_dataset( + cfg, + phase="test", + keys_to_read=["stl_coordinates", "stl_faces"], + keys_to_read_if_available=keys_to_read_if_available, + vol_factors=vol_factors, + surf_factors=surf_factors, + normalize_coordinates=cfg.data.normalize_coordinates, + sample_in_bbox=cfg.data.sample_in_bbox, + sampling=cfg.data.sampling, + device_mesh=domain_mesh, + placements=placements, + ) + + ###################################################### + # The sampler is used in multi-gpu inference to + # coordinate the batches used for each rank. + ###################################################### + test_sampler = DistributedSampler( + test_dataloader, + num_replicas=data_mesh.size(), + rank=data_mesh.get_local_rank(), + **cfg.train.sampler, + ) + + ###################################################### + # Configure the model + # and move it to the device. + ###################################################### + model = DoMINO( + input_features=3, + output_features_vol=num_vol_vars, + output_features_surf=num_surf_vars, + global_features=num_global_features, + model_parameters=cfg.model, + ).to(dist.device) + + # Print model summary (structure and parmeter count). + logger.info(f"Model summary:\n{torchinfo.summary(model, verbose=0, depth=2)}\n") + + if dist.world_size > 1: + torch.distributed.barrier() + + load_checkpoint( + to_absolute_path(cfg.resume_dir), + models=model, + device=dist.device, + ) + + start_time = time.perf_counter() + + # This controls what indices to use for each epoch. + test_sampler.set_epoch(0) + + prof = Profiler() + + model.eval() + epoch_start_time = time.perf_counter() + with prof: + inference_epoch( + dataloader=test_dataloader, + sampler=test_sampler, + model=model, + logger=logger, + gpu_handle=gpu_handle, + batch_size=sample_points, + total_points=cfg.eval.num_points, + ) + epoch_end_time = time.perf_counter() + logger.info( + f"Device {dist.device}, Epoch took {epoch_end_time - epoch_start_time:.3f} seconds" + ) + + +if __name__ == "__main__": + # Profiler().enable("torch") + # Profiler().initialize() + main() + # Profiler().finalize() diff --git a/examples/cfd/external_aerodynamics/domino/src/loss.py b/examples/cfd/external_aerodynamics/domino/src/loss.py new file mode 100644 index 0000000000..e328af4efc --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/loss.py @@ -0,0 +1,553 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import torch +from typing import Literal, Any + +from physicsnemo.models.domino.utils import unnormalize + +from typing import Literal, Any + +import torch.cuda.nvtx as nvtx + +from physicsnemo.models.domino.utils import * + + +def compute_physics_loss( + output: torch.Tensor, + target: torch.Tensor, + mask: torch.Tensor, + loss_type: Literal["mse", "rmse"], + dims: tuple[int, ...] | None, + first_deriv: torch.nn.Module, + eqn: Any, + bounding_box: torch.Tensor, + vol_factors: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Compute physics-based loss terms for Navier-Stokes equations. + + Args: + output: Model output containing (output, coords_neighbors, output_neighbors, neighbors_list) + target: Ground truth values + mask: Mask for valid values + loss_type: Type of loss to calculate ("mse" or "rmse") + dims: Dimensions for loss calculation + first_deriv: First derivative calculator + eqn: Equations + bounding_box: Bounding box for normalization + vol_factors: Volume factors for normalization + + Returns: + Tuple of (data_loss, continuity_loss, momentum_x_loss, momentum_y_loss, momentum_z_loss) + """ + # Physics loss enabled + output, coords_neighbors, output_neighbors, neighbors_list = output + batch_size = output.shape[1] + fields, num_neighbors = output_neighbors.shape[3], output_neighbors.shape[2] + coords_total = coords_neighbors[0, :] + output_total = output_neighbors[0, :] + output_total_unnormalized = unnormalize( + output_total, vol_factors[0], vol_factors[1] + ) + coords_total_unnormalized = unnormalize( + coords_total, bounding_box[0], bounding_box[1] + ) + + # compute first order gradients on all the nodes from the neighbors_list + grad_list = {} + for parent_id, neighbor_ids in neighbors_list.items(): + neighbor_ids_tensor = torch.tensor(neighbor_ids).to( + output_total_unnormalized.device + ) + du = ( + output_total_unnormalized[:, [parent_id]] + - output_total_unnormalized[:, neighbor_ids_tensor] + ) + dv = ( + coords_total_unnormalized[:, [parent_id]] + - coords_total_unnormalized[:, neighbor_ids_tensor] + ) + grads = first_deriv.forward( + coords=None, connectivity_tensor=None, y=None, du=du, dv=dv + ) + grad = torch.cat(grads, dim=1) + grad_list[parent_id] = grad + + # compute second order gradients on only the center node + neighbor_ids_tensor = torch.tensor(neighbors_list[0]).to( + output_total_unnormalized.device + ) + grad_neighbors_center = torch.stack([v for v in grad_list.values()], dim=1) + grad_neighbors_center = grad_neighbors_center.reshape( + batch_size, len(neighbors_list[0]) + 1, -1 + ) + + du = grad_neighbors_center[:, [0]] - grad_neighbors_center[:, neighbor_ids_tensor] + dv = ( + coords_total_unnormalized[:, [0]] + - coords_total_unnormalized[:, neighbor_ids_tensor] + ) + + # second order gradients + ggrads_center = first_deriv.forward( + coords=None, connectivity_tensor=None, y=None, du=du, dv=dv + ) + ggrad_center = torch.cat(ggrads_center, dim=1) + grad_neighbors_center = grad_neighbors_center.reshape( + batch_size, len(neighbors_list[0]) + 1, 3, -1 + ) + + # Get the outputs on the original nodes + fields_center_unnormalized = output_total_unnormalized[:, 0, :] + grad_center = grad_neighbors_center[:, 0, :, :] + grad_grad_uvw_center = ggrad_center[:, :, :9] + + nu = 1.507 * 1e-5 + + dict_mapping = { + "u": fields_center_unnormalized[:, [0]], + "v": fields_center_unnormalized[:, [1]], + "w": fields_center_unnormalized[:, [2]], + "p": fields_center_unnormalized[:, [3]], + "nu": nu + fields_center_unnormalized[:, [4]], + "u__x": grad_center[:, 0, [0]], + "u__y": grad_center[:, 1, [0]], + "u__z": grad_center[:, 2, [0]], + "v__x": grad_center[:, 0, [1]], + "v__y": grad_center[:, 1, [1]], + "v__z": grad_center[:, 2, [1]], + "w__x": grad_center[:, 0, [2]], + "w__y": grad_center[:, 1, [2]], + "w__z": grad_center[:, 2, [2]], + "p__x": grad_center[:, 0, [3]], + "p__y": grad_center[:, 1, [3]], + "p__z": grad_center[:, 2, [3]], + "nu__x": grad_center[:, 0, [4]], + "nu__y": grad_center[:, 1, [4]], + "nu__z": grad_center[:, 2, [4]], + "u__x__x": grad_grad_uvw_center[:, 0, [0]], + "u__x__y": grad_grad_uvw_center[:, 1, [0]], + "u__x__z": grad_grad_uvw_center[:, 2, [0]], + "u__y__x": grad_grad_uvw_center[:, 1, [0]], # same as __x__y + "u__y__y": grad_grad_uvw_center[:, 1, [1]], + "u__y__z": grad_grad_uvw_center[:, 2, [1]], + "u__z__x": grad_grad_uvw_center[:, 2, [0]], # same as __x__z + "u__z__y": grad_grad_uvw_center[:, 2, [1]], # same as __y__z + "u__z__z": grad_grad_uvw_center[:, 2, [2]], + "v__x__x": grad_grad_uvw_center[:, 0, [3]], + "v__x__y": grad_grad_uvw_center[:, 1, [3]], + "v__x__z": grad_grad_uvw_center[:, 2, [3]], + "v__y__x": grad_grad_uvw_center[:, 1, [3]], # same as __x__y + "v__y__y": grad_grad_uvw_center[:, 1, [4]], + "v__y__z": grad_grad_uvw_center[:, 2, [4]], + "v__z__x": grad_grad_uvw_center[:, 2, [3]], # same as __x__z + "v__z__y": grad_grad_uvw_center[:, 2, [4]], # same as __y__z + "v__z__z": grad_grad_uvw_center[:, 2, [5]], + "w__x__x": grad_grad_uvw_center[:, 0, [6]], + "w__x__y": grad_grad_uvw_center[:, 1, [6]], + "w__x__z": grad_grad_uvw_center[:, 2, [6]], + "w__y__x": grad_grad_uvw_center[:, 1, [6]], # same as __x__y + "w__y__y": grad_grad_uvw_center[:, 1, [7]], + "w__y__z": grad_grad_uvw_center[:, 2, [7]], + "w__z__x": grad_grad_uvw_center[:, 2, [6]], # same as __x__z + "w__z__y": grad_grad_uvw_center[:, 2, [7]], # same as __y__z + "w__z__z": grad_grad_uvw_center[:, 2, [8]], + } + continuity = eqn["continuity"].evaluate(dict_mapping)["continuity"] + momentum_x = eqn["momentum_x"].evaluate(dict_mapping)["momentum_x"] + momentum_y = eqn["momentum_y"].evaluate(dict_mapping)["momentum_y"] + momentum_z = eqn["momentum_z"].evaluate(dict_mapping)["momentum_z"] + + # Compute the weights for the equation residuals + weight_continuity = torch.sigmoid(0.5 * (torch.abs(continuity) - 10)) + weight_momentum_x = torch.sigmoid(0.5 * (torch.abs(momentum_x) - 10)) + weight_momentum_y = torch.sigmoid(0.5 * (torch.abs(momentum_y) - 10)) + weight_momentum_z = torch.sigmoid(0.5 * (torch.abs(momentum_z) - 10)) + + weighted_continuity = weight_continuity * torch.abs(continuity) + weighted_momentum_x = weight_momentum_x * torch.abs(momentum_x) + weighted_momentum_y = weight_momentum_y * torch.abs(momentum_y) + weighted_momentum_z = weight_momentum_z * torch.abs(momentum_z) + + # Compute data loss + num = torch.sum(mask * (output - target) ** 2.0, dims) + if loss_type == "rmse": + denom = torch.sum(mask * target**2.0, dims) + else: + denom = torch.sum(mask) + + del coords_total, output_total + torch.cuda.empty_cache() + + return ( + torch.mean(num / denom), + torch.mean(torch.abs(weighted_continuity)), + torch.mean(torch.abs(weighted_momentum_x)), + torch.mean(torch.abs(weighted_momentum_y)), + torch.mean(torch.abs(weighted_momentum_z)), + ) + + +def loss_fn( + output: torch.Tensor, + target: torch.Tensor, + loss_type: Literal["mse", "rmse"], + padded_value: float = -10, +) -> torch.Tensor: + """Calculate mean squared error or root mean squared error with masking for padded values. + + Args: + output: Predicted values from the model + target: Ground truth values + loss_type: Type of loss to calculate ("mse" or "rmse") + padded_value: Value used for padding in the tensor + + Returns: + Calculated loss as a scalar tensor + """ + mask = abs(target - padded_value) > 1e-3 + + if loss_type == "rmse": + dims = (0, 1) + else: + dims = None + + num = torch.sum(mask * (output - target) ** 2.0, dims) + if loss_type == "rmse": + denom = torch.sum(mask * (target - torch.mean(target, (0, 1))) ** 2.0, dims) + loss = torch.mean(num / denom) + elif loss_type == "mse": + denom = torch.sum(mask) + loss = torch.mean(num / denom) + else: + raise ValueError(f"Invalid loss type: {loss_type}") + return loss + + +def loss_fn_with_physics( + output: torch.Tensor, + target: torch.Tensor, + loss_type: Literal["mse", "rmse"], + padded_value: float = -10, + first_deriv: torch.nn.Module = None, + eqn: Any = None, + bounding_box: torch.Tensor = None, + vol_factors: torch.Tensor = None, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Calculate loss with physics-based terms for appropriate equations. + + Args: + output: Predicted values from the model (with neighbor data when physics enabled) + target: Ground truth values + loss_type: Type of loss to calculate ("mse" or "rmse") + padded_value: Value used for padding in the tensor + first_deriv: First derivative calculator + eqn: Equations + bounding_box: Bounding box for normalization + vol_factors: Volume factors for normalization + + Returns: + Tuple of (data_loss, continuity_loss, momentum_x_loss, momentum_y_loss, momentum_z_loss) + """ + mask = abs(target - padded_value) > 1e-3 + + if loss_type == "rmse": + dims = (0, 1) + else: + dims = None + + # Call the physics loss computation function + return compute_physics_loss( + output=output, + target=target, + mask=mask, + loss_type=loss_type, + dims=dims, + first_deriv=first_deriv, + eqn=eqn, + bounding_box=bounding_box, + vol_factors=vol_factors, + ) + + +def loss_fn_surface( + output: torch.Tensor, target: torch.Tensor, loss_type: Literal["mse", "rmse"] +) -> torch.Tensor: + """Calculate loss for surface data by handling scalar and vector components separately. + + Args: + output: Predicted surface values from the model + target: Ground truth surface values + loss_type: Type of loss to calculate ("mse" or "rmse") + + Returns: + Combined scalar and vector loss as a scalar tensor + """ + # Separate the scalar and vector components: + output_scalar, output_vector = torch.split(output, [1, 3], dim=2) + target_scalar, target_vector = torch.split(target, [1, 3], dim=2) + + numerator = torch.mean((output_scalar - target_scalar) ** 2.0) + vector_diff_sq = torch.mean((target_vector - output_vector) ** 2.0, (0, 1)) + if loss_type == "mse": + masked_loss_pres = numerator + masked_loss_ws = torch.sum(vector_diff_sq) + else: + denom = torch.mean((target_scalar - torch.mean(target_scalar, (0, 1))) ** 2.0) + masked_loss_pres = numerator / denom + + # Compute the mean diff**2 of the vector component, leave the last dimension: + masked_loss_ws_num = vector_diff_sq + masked_loss_ws_denom = torch.mean( + (target_vector - torch.mean(target_vector, (0, 1))) ** 2.0, (0, 1) + ) + masked_loss_ws = torch.sum(masked_loss_ws_num / masked_loss_ws_denom) + + loss = masked_loss_pres + masked_loss_ws + + return loss / 4.0 + + +def loss_fn_area( + output: torch.Tensor, + target: torch.Tensor, + normals: torch.Tensor, + area: torch.Tensor, + area_scaling_factor: float, + loss_type: Literal["mse", "rmse"], +) -> torch.Tensor: + """Calculate area-weighted loss for surface data considering normal vectors. + + Args: + output: Predicted surface values from the model + target: Ground truth surface values + normals: Normal vectors for the surface + area: Area values for surface elements + area_scaling_factor: Scaling factor for area weighting + loss_type: Type of loss to calculate ("mse" or "rmse") + + Returns: + Area-weighted loss as a scalar tensor + """ + area = area * area_scaling_factor + area_scale_factor = area + + # Separate the scalar and vector components. + target_scalar, target_vector = torch.split( + target * area_scale_factor, [1, 3], dim=2 + ) + output_scalar, output_vector = torch.split( + output * area_scale_factor, [1, 3], dim=2 + ) + + # Apply the normals to the scalar components (only [:,:,0]): + normals, _ = torch.split(normals, [1, normals.shape[-1] - 1], dim=2) + target_scalar = target_scalar * normals + output_scalar = output_scalar * normals + + # Compute the mean diff**2 of the scalar component: + masked_loss_pres = torch.mean(((output_scalar - target_scalar) ** 2.0), dim=(0, 1)) + if loss_type == "rmse": + masked_loss_pres /= torch.mean( + (target_scalar - torch.mean(target_scalar, (0, 1))) ** 2.0, dim=(0, 1) + ) + + # Compute the mean diff**2 of the vector component, leave the last dimension: + masked_loss_ws = torch.mean((target_vector - output_vector) ** 2.0, (0, 1)) + if loss_type == "rmse": + masked_loss_ws /= torch.mean( + (target_vector - torch.mean(target_vector, (0, 1))) ** 2.0, (0, 1) + ) + + # Combine the scalar and vector components: + loss = 0.25 * (masked_loss_pres + torch.sum(masked_loss_ws)) + + return loss + + +def integral_loss_fn( + output, target, area, normals, stream_velocity=None, padded_value=-10 +): + drag_loss = drag_loss_fn( + output, target, area, normals, stream_velocity=stream_velocity, padded_value=-10 + ) + lift_loss = lift_loss_fn( + output, target, area, normals, stream_velocity=stream_velocity, padded_value=-10 + ) + return lift_loss + drag_loss + + +def lift_loss_fn(output, target, area, normals, stream_velocity=None, padded_value=-10): + vel_inlet = stream_velocity # Get this from the dataset + mask = abs(target - padded_value) > 1e-3 + + output_true = target * mask * area * (vel_inlet) ** 2.0 + output_pred = output * mask * area * (vel_inlet) ** 2.0 + + normals = torch.select(normals, 2, 2) + # output_true_0 = output_true[:, :, 0] + output_true_0 = output_true.select(2, 0) + output_pred_0 = output_pred.select(2, 0) + + pres_true = output_true_0 * normals + pres_pred = output_pred_0 * normals + + wz_true = output_true[:, :, -1] + wz_pred = output_pred[:, :, -1] + + masked_pred = torch.mean(pres_pred + wz_pred, (1)) + masked_truth = torch.mean(pres_true + wz_true, (1)) + + loss = (masked_pred - masked_truth) ** 2.0 + loss = torch.mean(loss) + return loss + + +def drag_loss_fn(output, target, area, normals, stream_velocity=None, padded_value=-10): + vel_inlet = stream_velocity # Get this from the dataset + mask = abs(target - padded_value) > 1e-3 + output_true = target * mask * area * (vel_inlet) ** 2.0 + output_pred = output * mask * area * (vel_inlet) ** 2.0 + + pres_true = output_true[:, :, 0] * normals[:, :, 0] + pres_pred = output_pred[:, :, 0] * normals[:, :, 0] + + wx_true = output_true[:, :, 1] + wx_pred = output_pred[:, :, 1] + + masked_pred = torch.mean(pres_pred + wx_pred, (1)) + masked_truth = torch.mean(pres_true + wx_true, (1)) + + loss = (masked_pred - masked_truth) ** 2.0 + loss = torch.mean(loss) + return loss + + +def compute_loss_dict( + prediction_vol: torch.Tensor, + prediction_surf: torch.Tensor, + batch_inputs: dict, + loss_fn_type: dict, + integral_scaling_factor: float, + surf_loss_scaling: float, + vol_loss_scaling: float, + first_deriv: torch.nn.Module | None = None, + eqn: Any = None, + bounding_box: torch.Tensor | None = None, + vol_factors: torch.Tensor | None = None, + add_physics_loss: bool = False, +) -> tuple[torch.Tensor, dict]: + """ + Compute the loss terms in a single function call. + + Computes: + - Volume loss if prediction_vol is not None + - Surface loss if prediction_surf is not None + - Integral loss if prediction_surf is not None + - Total loss as a weighted sum of the above + + Returns: + - Total loss as a scalar tensor + - Dictionary of loss terms (for logging, etc) + """ + nvtx.range_push("Loss Calculation") + total_loss_terms = [] + loss_dict = {} + + if prediction_vol is not None: + target_vol = batch_inputs["volume_fields"] + + if add_physics_loss: + loss_vol = loss_fn_with_physics( + prediction_vol, + target_vol, + loss_fn_type.loss_type, + padded_value=-10, + first_deriv=first_deriv, + eqn=eqn, + bounding_box=bounding_box, + vol_factors=vol_factors, + ) + loss_dict["loss_vol"] = loss_vol[0] + loss_dict["loss_continuity"] = loss_vol[1] + loss_dict["loss_momentum_x"] = loss_vol[2] + loss_dict["loss_momentum_y"] = loss_vol[3] + loss_dict["loss_momentum_z"] = loss_vol[4] + total_loss_terms.append(loss_vol[0]) + total_loss_terms.append(loss_vol[1]) + total_loss_terms.append(loss_vol[2]) + total_loss_terms.append(loss_vol[3]) + total_loss_terms.append(loss_vol[4]) + else: + loss_vol = loss_fn( + prediction_vol, + target_vol, + loss_fn_type.loss_type, + padded_value=-10, + ) + loss_dict["loss_vol"] = loss_vol + total_loss_terms.append(loss_vol) + + if prediction_surf is not None: + target_surf = batch_inputs["surface_fields"] + surface_areas = batch_inputs["surface_areas"] + surface_areas = torch.unsqueeze(surface_areas, -1) + surface_normals = batch_inputs["surface_normals"] + + # Needs to be taken from the dataset + stream_velocity = batch_inputs["global_params_values"][:, 0, :] + + loss_surf = loss_fn_surface( + prediction_surf, + target_surf, + loss_fn_type.loss_type, + ) + + loss_surf_area = loss_fn_area( + prediction_surf, + target_surf, + surface_normals, + surface_areas, + area_scaling_factor=loss_fn_type.area_weighing_factor, + loss_type=loss_fn_type.loss_type, + ) + + if loss_fn_type.loss_type == "mse": + loss_surf = loss_surf * surf_loss_scaling + loss_surf_area = loss_surf_area * surf_loss_scaling + + total_loss_terms.append(loss_surf) + loss_dict["loss_surf"] = loss_surf + total_loss_terms.append(loss_surf_area) + loss_dict["loss_surf_area"] = loss_surf_area + loss_integral = ( + integral_loss_fn( + prediction_surf, + target_surf, + surface_areas, + surface_normals, + stream_velocity, + padded_value=-10, + ) + ) * integral_scaling_factor + loss_dict["loss_integral"] = loss_integral + total_loss_terms.append(loss_integral) + + total_loss = sum(total_loss_terms) + loss_dict["total_loss"] = total_loss + nvtx.range_pop() + + return total_loss, loss_dict diff --git a/examples/cfd/external_aerodynamics/domino/src/shuffle_volumetric_curator_output.py b/examples/cfd/external_aerodynamics/domino/src/shuffle_volumetric_curator_output.py new file mode 100644 index 0000000000..00703d7d0c --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/shuffle_volumetric_curator_output.py @@ -0,0 +1,189 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import multiprocessing as mp +from functools import partial + +import numpy as np +import shutil + +import zarr +from numcodecs import Blosc + +""" +This script reads each zarr file from a specified directory, and copies the +data to the output directory. For the keys "volume_fields" and "volume_mesh_centers", +the script will apply a permutation (aka shuffle) of those fields in tandem. + +Since the datasets used are often very large, this script also applies +sharding to the output files which is a Zarr3 feature. + +Therefore, zarr >= 3.0 is required. +""" + + +def check_file_completeness(input_file: str, output_file: str) -> bool: + """ + Check if the output file exists and contains all required data from input file. + """ + if not os.path.exists(output_file): + return False + + in_file = zarr.open(input_file, mode="r") + try: + out_file = zarr.open(output_file, mode="r") + except zarr.errors.PathNotFoundError: + print(f"No output, returning False") + return False + + # Check if all keys except 'filename' exist and have same shapes + for key in in_file.keys(): + if key == "filename": + continue + if key not in out_file and key not in out_file.attrs: + print(f"Key {key} not in output, returning False") + return False + if isinstance(in_file[key], zarr.Array): + if key in out_file.attrs: + continue + if in_file[key].shape != out_file[key].shape: + print(f"Key {key} shape mismatch, returning False") + return False + return True + + +def store_array(store, name: str, data: np.ndarray): + # By default, chunk size is 10k points: + chunk_size = (10_000,) + data.shape[1:] + # By default, shard size is 2 million points: + shard_size = (2_000_000,) + data.shape[1:] + + zarr.create_array( + store=store, + name=name, + data=data, + chunks=chunk_size, + shards=shard_size, + compressors="auto", + ) + + +def copy_file_with_shuffled_volume_data( + input_file: str, output_file: str, random_seed: int | None = None +): + """ + Copy a file with shuffled volume data, using Zarr v3 sharding for efficient storage. + Only processes if the output file doesn't exist or is incomplete. + """ + file_is_complete = check_file_completeness(input_file, output_file) + if file_is_complete: + print(f"Skipping {output_file} - already complete") + return True + + print(f"Processing {input_file} -> {output_file}") + + # return False + + # if the output folder exists but isn't complete, purge it. + # It's probably an interrupted conversion. + if os.path.exists(output_file): + shutil.rmtree(output_file) + + # return file_is_complete + volume_keys = ["volume_fields", "volume_mesh_centers"] + + in_file = zarr.open(input_file, mode="r") + + # Create store with sharding configuration + store = zarr.storage.LocalStore(output_file) + root = zarr.group(store=store) + + # First copy all non-volume data + for key in in_file.keys(): + if key not in volume_keys: + if key == "filename": + continue + in_data = in_file[key] + if in_data.shape != (): + # For array data, use the same chunks as input but with sharding + store_array(store, key, in_data[:]) + else: + # Store scalar values as attributes + root.attrs[key] = in_data[()] + + # Open and shuffle the volume data + volume_fields = in_file["volume_fields"][:] + volume_mesh_centers = in_file["volume_mesh_centers"][:] + + if random_seed is not None: + np.random.seed(random_seed) + + # Generate a permutation + permutation = np.random.permutation(volume_fields.shape[0]) + + # Shuffle the volume data + shuffled_volume_fields = volume_fields[permutation] + shuffled_volume_mesh_centers = volume_mesh_centers[permutation] + + store_array(store, "volume_fields", shuffled_volume_fields) + store_array(store, "volume_mesh_centers", shuffled_volume_mesh_centers) + + print(f"Processed {output_file} - COMPLETE") + return True + + +def process_file(file: str, top_dir: str, out_dir: str): + """ + Process a single file, creating output directory if needed. + """ + os.makedirs(out_dir, exist_ok=True) + input_path = os.path.join(top_dir, file) + output_path = os.path.join(out_dir, file) + return copy_file_with_shuffled_volume_data(input_path, output_path) + + +def main(): + import argparse + + parser = argparse.ArgumentParser(description="Shuffle volumetric curator output") + parser.add_argument("--input-dir", required=True, help="Input directory path") + parser.add_argument("--output-dir", required=True, help="Output directory path") + parser.add_argument( + "--num-cores", type=int, default=64, help="Number of cores to use" + ) + args = parser.parse_args() + + # Get list of files to process + files = os.listdir(args.input_dir) + + # Create a partial function with fixed directories + process_func = partial( + process_file, top_dir=args.input_dir, out_dir=args.output_dir + ) + + # Use multiprocessing to process files in parallel + num_cores = max(1, args.num_cores) # Leave one core free + print(f"Processing {len(files)} files using {num_cores} cores") + + with mp.Pool(num_cores) as pool: + results = pool.map(process_func, files) + print(f"Results: {results}") + print(f"Total conversions: {sum(results)}") + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino/src/test.py b/examples/cfd/external_aerodynamics/domino/src/test.py new file mode 100644 index 0000000000..1317728d57 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/test.py @@ -0,0 +1,923 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code defines a distributed pipeline for testing the DoMINO model on +CFD datasets. It includes the instantiating the DoMINO model and datapipe, +automatically loading the most recent checkpoint, reading the VTP/VTU/STL +testing files, calculation of parameters required for DoMINO model and +evaluating the model in parallel using DistributedDataParallel across multiple +GPUs. This is a common recipe that enables training of combined models for surface +and volume as well either of them separately. The model predictions are loaded in +the the VTP/VTU files and saved in the specified directory. The eval tab in +config.yaml can be used to specify the input and output directories. +""" + +import os, re +import time + +import hydra +from hydra.utils import to_absolute_path +from omegaconf import DictConfig, OmegaConf + +# This will set up the cupy-ecosystem and pytorch to share memory pools +from physicsnemo.utils.memory import unified_gpu_memory + +import numpy as np +import cupy as cp + +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable, List, Literal, Mapping, Optional, Union, Callable + +import pandas as pd +import pyvista as pv + +import torch +from torch.nn.parallel import DistributedDataParallel +from torch.utils.data import DataLoader, Dataset + +import vtk +from vtk.util import numpy_support + +from physicsnemo.distributed import DistributedManager +from physicsnemo.datapipes.cae.domino_datapipe import DoMINODataPipe +from physicsnemo.models.domino.model import DoMINO +from physicsnemo.models.domino.geometry_rep import scale_sdf +from physicsnemo.models.domino.utils import * +from physicsnemo.models.domino.utils.vtk_file_utils import * +from physicsnemo.nn.functional import knn, signed_distance_field +from utils import ScalingFactors, load_scaling_factors + +# AIR_DENSITY = 1.205 +# STREAM_VELOCITY = 30.00 + + +def loss_fn(output, target): + masked_loss = torch.mean(((output - target) ** 2.0), (0, 1, 2)) + loss = torch.mean(masked_loss) + return loss + + +def test_step(data_dict, model, device, cfg, vol_factors, surf_factors): + avg_tloss_vol = 0.0 + avg_tloss_surf = 0.0 + running_tloss_vol = 0.0 + running_tloss_surf = 0.0 + + if cfg.model.model_type == "volume" or cfg.model.model_type == "combined": + output_features_vol = True + else: + output_features_vol = None + + if cfg.model.model_type == "surface" or cfg.model.model_type == "combined": + output_features_surf = True + else: + output_features_surf = None + + with torch.no_grad(): + point_batch_size = 256000 + # data_dict = dict_to_device(data_dict, device) + + # Non-dimensionalization factors + length_scale = data_dict["length_scale"] + + global_params_values = data_dict["global_params_values"] + global_params_reference = data_dict["global_params_reference"] + stream_velocity = global_params_reference[:, 0, :] + air_density = global_params_reference[:, 1, :] + + # STL nodes + geo_centers = data_dict["geometry_coordinates"] + + # Bounding box grid + s_grid = data_dict["surf_grid"] + sdf_surf_grid = data_dict["sdf_surf_grid"] + # Scaling factors + surf_max = data_dict["surface_min_max"][:, 1] + surf_min = data_dict["surface_min_max"][:, 0] + + if output_features_vol is not None: + # Represent geometry on computational grid + # Computational domain grid + p_grid = data_dict["grid"] + sdf_grid = data_dict["sdf_grid"] + # Scaling factors + if "volume_min_max" in data_dict.keys(): + vol_max = data_dict["volume_min_max"][:, 1] + vol_min = data_dict["volume_min_max"][:, 0] + geo_centers_vol = ( + 2.0 * (geo_centers - vol_min) / (vol_max - vol_min) - 1 + ) + else: + geo_centers_vol = geo_centers + + # Normalize based on computational domain + encoding_g_vol = model.geo_rep_volume(geo_centers_vol, p_grid, sdf_grid) + + if output_features_surf is not None: + # Represent geometry on bounding box + geo_centers_surf = ( + 2.0 * (geo_centers - surf_min) / (surf_max - surf_min) - 1 + ) + encoding_g_surf = model.geo_rep_surface( + geo_centers_surf, s_grid, sdf_surf_grid + ) + + if ( + output_features_vol is not None + and output_features_surf is not None + and cfg.model.combine_volume_surface + ): + encoding_g = torch.cat((encoding_g_vol, encoding_g_surf), axis=1) + encoding_g_surf = model.combined_unet_surf(encoding_g) + encoding_g_vol = model.combined_unet_vol(encoding_g) + + if output_features_vol is not None: + # First calculate volume predictions if required + volume_mesh_centers = data_dict["volume_mesh_centers"] + target_vol = data_dict["volume_fields"] + # SDF on volume mesh nodes + sdf_nodes = data_dict["sdf_nodes"] + # Positional encoding based on closest point on surface to a volume node + pos_volume_closest = data_dict["pos_volume_closest"] + # Positional encoding based on center of mass of geometry to volume node + pos_volume_center_of_mass = data_dict["pos_volume_center_of_mass"] + p_grid = data_dict["grid"] + + prediction_vol = torch.zeros_like(target_vol) + num_points = volume_mesh_centers.shape[1] + subdomain_points = int(np.floor(num_points / point_batch_size)) + sdf_scaling_factor = ( + cfg.model.geometry_rep.geo_processor.volume_sdf_scaling_factor + ) + start_time = time.time() + + for p in range(subdomain_points + 1): + start_idx = p * point_batch_size + end_idx = (p + 1) * point_batch_size + with torch.no_grad(): + target_batch = target_vol[:, start_idx:end_idx] + volume_mesh_centers_batch = volume_mesh_centers[ + :, start_idx:end_idx + ] + sdf_nodes_batch = sdf_nodes[:, start_idx:end_idx] + scaled_sdf_nodes_batch = [] + for p in range(len(sdf_scaling_factor)): + scaled_sdf_nodes_batch.append( + scale_sdf(sdf_nodes_batch, sdf_scaling_factor[p]) + ) + scaled_sdf_nodes_batch = torch.cat(scaled_sdf_nodes_batch, dim=-1) + + pos_volume_closest_batch = pos_volume_closest[:, start_idx:end_idx] + pos_normals_com_batch = pos_volume_center_of_mass[ + :, start_idx:end_idx + ] + geo_encoding_local = model.volume_local_geo_encodings( + 0.5 * encoding_g_vol, + volume_mesh_centers_batch, + p_grid, + ) + if cfg.model.use_sdf_in_basis_func: + pos_encoding_all = torch.cat( + ( + sdf_nodes_batch, + scaled_sdf_nodes_batch, + pos_volume_closest_batch, + pos_normals_com_batch, + ), + axis=-1, + ) + else: + pos_encoding_all = pos_normals_com_batch + + pos_encoding = model.fc_p_vol(pos_encoding_all) + tpredictions_batch = model.solution_calculator_vol( + volume_mesh_centers_batch, + geo_encoding_local, + pos_encoding, + global_params_values, + global_params_reference, + ) + running_tloss_vol += loss_fn(tpredictions_batch, target_batch) + prediction_vol[:, start_idx:end_idx] = tpredictions_batch + + if cfg.model.normalization == "min_max_scaling": + prediction_vol = unnormalize( + prediction_vol, vol_factors[0], vol_factors[1] + ) + elif cfg.model.normalization == "mean_std_scaling": + prediction_vol = unstandardize( + prediction_vol, vol_factors[0], vol_factors[1] + ) + # print(np.amax(prediction_vol, axis=(0, 1)), np.amin(prediction_vol, axis=(0, 1))) + + prediction_vol[:, :, :3] = prediction_vol[:, :, :3] * stream_velocity[0, 0] + prediction_vol[:, :, 3] = ( + prediction_vol[:, :, 3] + * stream_velocity[0, 0] ** 2.0 + * air_density[0, 0] + ) + prediction_vol[:, :, 4] = ( + prediction_vol[:, :, 4] * stream_velocity[0, 0] * length_scale[0] + ) + else: + prediction_vol = None + + if output_features_surf is not None: + # Next calculate surface predictions + # Sampled points on surface + surface_mesh_centers = data_dict["surface_mesh_centers"] + surface_normals = data_dict["surface_normals"] + surface_areas = data_dict["surface_areas"] + + # Neighbors of sampled points on surface + surface_mesh_neighbors = data_dict["surface_mesh_neighbors"] + surface_neighbors_normals = data_dict["surface_neighbors_normals"] + surface_neighbors_areas = data_dict["surface_neighbors_areas"] + surface_areas = torch.unsqueeze(surface_areas, -1) + surface_neighbors_areas = torch.unsqueeze(surface_neighbors_areas, -1) + pos_surface_center_of_mass = data_dict["pos_surface_center_of_mass"] + num_points = surface_mesh_centers.shape[1] + subdomain_points = int(np.floor(num_points / point_batch_size)) + + target_surf = data_dict["surface_fields"] + prediction_surf = torch.zeros_like(target_surf) + + start_time = time.time() + + for p in range(subdomain_points + 1): + start_idx = p * point_batch_size + end_idx = (p + 1) * point_batch_size + with torch.no_grad(): + target_batch = target_surf[:, start_idx:end_idx] + surface_mesh_centers_batch = surface_mesh_centers[ + :, start_idx:end_idx + ] + surface_mesh_neighbors_batch = surface_mesh_neighbors[ + :, start_idx:end_idx + ] + surface_normals_batch = surface_normals[:, start_idx:end_idx] + surface_neighbors_normals_batch = surface_neighbors_normals[ + :, start_idx:end_idx + ] + surface_areas_batch = surface_areas[:, start_idx:end_idx] + surface_neighbors_areas_batch = surface_neighbors_areas[ + :, start_idx:end_idx + ] + pos_surface_center_of_mass_batch = pos_surface_center_of_mass[ + :, start_idx:end_idx + ] + geo_encoding_local = model.surface_local_geo_encodings( + 0.5 * encoding_g_surf, + surface_mesh_centers_batch, + s_grid, + ) + pos_encoding = model.fc_p_surf(pos_surface_center_of_mass_batch) + + tpredictions_batch = model.solution_calculator_surf( + surface_mesh_centers_batch, + geo_encoding_local, + pos_encoding, + surface_mesh_neighbors_batch, + surface_normals_batch, + surface_neighbors_normals_batch, + surface_areas_batch, + surface_neighbors_areas_batch, + global_params_values, + global_params_reference, + ) + + running_tloss_surf += loss_fn(tpredictions_batch, target_batch) + prediction_surf[:, start_idx:end_idx] = tpredictions_batch + + if cfg.model.normalization == "min_max_scaling": + prediction_surf = unnormalize( + prediction_surf, surf_factors[0], surf_factors[1] + ) + elif cfg.model.normalization == "mean_std_scaling": + prediction_surf = unstandardize( + prediction_surf, surf_factors[0], surf_factors[1] + ) + prediction_surf = ( + prediction_surf * stream_velocity[0, 0] ** 2.0 * air_density[0, 0] + ) + else: + prediction_surf = None + + return prediction_vol, prediction_surf + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig): + print(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + input_path = cfg.eval.test_path + + model_type = cfg.model.model_type + + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + if model_type == "volume" or model_type == "combined": + volume_variable_names = list(cfg.variables.volume.solution.keys()) + num_vol_vars = 0 + for j in volume_variable_names: + if cfg.variables.volume.solution[j] == "vector": + num_vol_vars += 3 + else: + num_vol_vars += 1 + else: + num_vol_vars = None + + if model_type == "surface" or model_type == "combined": + surface_variable_names = list(cfg.variables.surface.solution.keys()) + num_surf_vars = 0 + for j in surface_variable_names: + if cfg.variables.surface.solution[j] == "vector": + num_surf_vars += 3 + else: + num_surf_vars += 1 + else: + num_surf_vars = None + + global_features = 0 + global_params_names = list(cfg.variables.global_parameters.keys()) + for param in global_params_names: + if cfg.variables.global_parameters[param].type == "vector": + global_features += len(cfg.variables.global_parameters[param].reference) + else: + global_features += 1 + + ###################################################### + # Get scaling factors - precompute them if this fails! + ###################################################### + pickle_path = os.path.join(cfg.data.scaling_factors) + + vol_factors, surf_factors = load_scaling_factors(cfg) + print("Vol factors:", vol_factors) + print("Surf factors:", surf_factors) + + model = DoMINO( + input_features=3, + output_features_vol=num_vol_vars, + output_features_surf=num_surf_vars, + global_features=global_features, + model_parameters=cfg.model, + ).to(dist.device) + + model = torch.compile(model, disable=True) + + checkpoint = torch.load( + to_absolute_path(os.path.join(cfg.resume_dir, cfg.eval.checkpoint_name)), + map_location=dist.device, + ) + + model.load_state_dict(checkpoint) + + print("Model loaded") + + if dist.world_size > 1: + model = DistributedDataParallel( + model, + device_ids=[dist.local_rank], + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + gradient_as_bucket_view=True, + static_graph=True, + ) + model = model.module + + dirnames = get_filenames(input_path) + dev_id = torch.cuda.current_device() + num_files = int(len(dirnames) / dist.world_size) + dirnames_per_gpu = dirnames[int(num_files * dev_id) : int(num_files * (dev_id + 1))] + + pred_save_path = cfg.eval.save_path + + if dist.rank == 0: + create_directory(pred_save_path) + + l2_surface_all = [] + l2_volume_all = [] + aero_forces_all = [] + for count, dirname in enumerate(dirnames_per_gpu): + filepath = os.path.join(input_path, dirname) + tag = int(re.findall(r"(\w+?)(\d+)", dirname)[0][1]) + stl_path = os.path.join(filepath, f"drivaer_{tag}.stl") + vtp_path = os.path.join(filepath, f"boundary_{tag}.vtp") + vtu_path = os.path.join(filepath, f"volume_{tag}.vtu") + + vtp_pred_save_path = os.path.join( + pred_save_path, f"boundary_{tag}_predicted.vtp" + ) + vtu_pred_save_path = os.path.join(pred_save_path, f"volume_{tag}_predicted.vtu") + + # Read STL + reader = pv.get_reader(stl_path) + mesh_stl = reader.read() + stl_vertices = mesh_stl.points + stl_faces = np.array(mesh_stl.faces).reshape((-1, 4))[ + :, 1: + ] # Assuming triangular elements + mesh_indices_flattened = stl_faces.flatten() + length_scale = np.array( + np.amax(np.amax(stl_vertices, 0) - np.amin(stl_vertices, 0)), + dtype=np.float32, + ) + length_scale = torch.from_numpy(length_scale).to(torch.float32).to(dist.device) + stl_sizes = mesh_stl.compute_cell_sizes(length=False, area=True, volume=False) + stl_sizes = np.array(stl_sizes.cell_data["Area"], dtype=np.float32) + stl_centers = np.array(mesh_stl.cell_centers().points, dtype=np.float32) + + # Convert to torch tensors and load on device + stl_vertices = torch.from_numpy(stl_vertices).to(torch.float32).to(dist.device) + stl_sizes = torch.from_numpy(stl_sizes).to(torch.float32).to(dist.device) + stl_centers = torch.from_numpy(stl_centers).to(torch.float32).to(dist.device) + mesh_indices_flattened = ( + torch.from_numpy(mesh_indices_flattened).to(torch.int32).to(dist.device) + ) + + # Center of mass calculation + center_of_mass = calculate_center_of_mass(stl_centers, stl_sizes) + + s_max = ( + torch.from_numpy(np.asarray(cfg.data.bounding_box_surface.max)) + .to(torch.float32) + .to(dist.device) + ) + s_min = ( + torch.from_numpy(np.asarray(cfg.data.bounding_box_surface.min)) + .to(torch.float32) + .to(dist.device) + ) + + nx, ny, nz = cfg.model.interp_res + + surf_grid = create_grid( + s_max, s_min, torch.from_numpy(np.asarray([nx, ny, nz])).to(dist.device) + ) + + normed_stl_vertices_cp = normalize(stl_vertices, s_max, s_min) + surf_grid_normed = normalize(surf_grid, s_max, s_min) + + # SDF calculation on the grid using WARP + time_start = time.time() + sdf_surf_grid, _ = signed_distance_field( + normed_stl_vertices_cp, + mesh_indices_flattened, + surf_grid_normed, + use_sign_winding_number=True, + ) + + surf_grid_max_min = torch.stack([s_min, s_max]) + + # Get global parameters and global parameters scaling from config.yaml + global_params_names = list(cfg.variables.global_parameters.keys()) + global_params_reference = { + name: cfg.variables.global_parameters[name]["reference"] + for name in global_params_names + } + global_params_types = { + name: cfg.variables.global_parameters[name]["type"] + for name in global_params_names + } + stream_velocity = global_params_reference["inlet_velocity"][0] + air_density = global_params_reference["air_density"] + + # Arrange global parameters reference in a list, ensuring it is flat + global_params_reference_list = [] + for name, type in global_params_types.items(): + if type == "vector": + global_params_reference_list.extend(global_params_reference[name]) + elif type == "scalar": + global_params_reference_list.append(global_params_reference[name]) + else: + raise ValueError( + f"Global parameter {name} not supported for this dataset" + ) + global_params_reference = np.array( + global_params_reference_list, dtype=np.float32 + ) + global_params_reference = torch.from_numpy(global_params_reference).to( + dist.device + ) + + # Define the list of global parameter values for each simulation. + # Note: The user must ensure that the values provided here correspond to the + # `global_parameters` specified in `config.yaml` and that these parameters + # exist within each simulation file. + global_params_values_list = [] + for key in global_params_types.keys(): + if key == "inlet_velocity": + global_params_values_list.append(stream_velocity) + elif key == "air_density": + global_params_values_list.append(air_density) + else: + raise ValueError( + f"Global parameter {key} not supported for this dataset" + ) + global_params_values_list = np.array( + global_params_values_list, dtype=np.float32 + ) + global_params_values = torch.from_numpy(global_params_values_list).to( + dist.device + ) + + # Read VTP + if model_type == "surface" or model_type == "combined": + reader = vtk.vtkXMLPolyDataReader() + reader.SetFileName(vtp_path) + reader.Update() + polydata_surf = reader.GetOutput() + + celldata_all = get_node_to_elem(polydata_surf) + + celldata = celldata_all.GetCellData() + surface_fields = get_fields(celldata, surface_variable_names) + surface_fields = np.concatenate(surface_fields, axis=-1) + + mesh = pv.PolyData(polydata_surf) + surface_coordinates = np.array(mesh.cell_centers().points, dtype=np.float32) + + surface_normals = np.array(mesh.cell_normals, dtype=np.float32) + surface_sizes = mesh.compute_cell_sizes( + length=False, area=True, volume=False + ) + surface_sizes = np.array(surface_sizes.cell_data["Area"], dtype=np.float32) + + # Normalize cell normals + surface_normals = ( + surface_normals / np.linalg.norm(surface_normals, axis=1)[:, np.newaxis] + ) + surface_coordinates = ( + torch.from_numpy(surface_coordinates).to(torch.float32).to(dist.device) + ) + surface_normals = ( + torch.from_numpy(surface_normals).to(torch.float32).to(dist.device) + ) + surface_sizes = ( + torch.from_numpy(surface_sizes).to(torch.float32).to(dist.device) + ) + surface_fields = ( + torch.from_numpy(surface_fields).to(torch.float32).to(dist.device) + ) + + if cfg.model.num_neighbors_surface > 1: + time_start = time.time() + # print(f"file: {dirname}, surface coordinates shape: {surface_coordinates.shape}") + # try: + ii, dd = knn( + points=surface_coordinates, + queries=surface_coordinates, + k=cfg.model.num_neighbors_surface, + ) + + surface_neighbors = surface_coordinates[ii] + surface_neighbors = surface_neighbors[:, 1:] + + surface_neighbors_normals = surface_normals[ii] + surface_neighbors_normals = surface_neighbors_normals[:, 1:] + surface_neighbors_sizes = surface_sizes[ii] + surface_neighbors_sizes = surface_neighbors_sizes[:, 1:] + # except: + # print(f"file: {dirname}, memory error in knn") + # print("setting surface neighbors to 0") + # surface_neighbors = surface_coordinates + # surface_neighbors_normals = surface_normals + # surface_neighbors_sizes = surface_sizes + # cfg.model.num_neighbors_surface = 1 + else: + surface_neighbors = surface_coordinates + surface_neighbors_normals = surface_normals + surface_neighbors_sizes = surface_sizes + + if cfg.data.normalize_coordinates: + surface_coordinates = normalize(surface_coordinates, s_max, s_min) + surf_grid = normalize(surf_grid, s_max, s_min) + center_of_mass_normalized = normalize(center_of_mass, s_max, s_min) + surface_neighbors = normalize(surface_neighbors, s_max, s_min) + else: + center_of_mass_normalized = center_of_mass + pos_surface_center_of_mass = surface_coordinates - center_of_mass_normalized + + else: + surface_coordinates = None + surface_fields = None + surface_sizes = None + surface_normals = None + surface_neighbors = None + surface_neighbors_normals = None + surface_neighbors_sizes = None + pos_surface_center_of_mass = None + + # Read VTU + if model_type == "volume" or model_type == "combined": + reader = vtk.vtkXMLUnstructuredGridReader() + reader.SetFileName(vtu_path) + reader.Update() + polydata_vol = reader.GetOutput() + volume_coordinates, volume_fields = get_volume_data( + polydata_vol, volume_variable_names + ) + volume_fields = np.concatenate(volume_fields, axis=-1) + volume_coordinates = ( + torch.from_numpy(volume_coordinates).to(torch.float32).to(dist.device) + ) + volume_fields = ( + torch.from_numpy(volume_fields).to(torch.float32).to(dist.device) + ) + + c_max = ( + torch.from_numpy(np.asarray(cfg.data.bounding_box.max)) + .to(torch.float32) + .to(dist.device) + ) + c_min = ( + torch.from_numpy(np.asarray(cfg.data.bounding_box.min)) + .to(torch.float32) + .to(dist.device) + ) + + # Generate a grid of specified resolution to map the bounding box + # The grid is used for capturing structured geometry features and SDF representation of geometry + grid = create_grid( + c_max, c_min, torch.from_numpy(np.asarray([nx, ny, nz])).to(dist.device) + ) + + if cfg.data.normalize_coordinates: + volume_coordinates = normalize(volume_coordinates, c_max, c_min) + grid = normalize(grid, c_max, c_min) + center_of_mass_normalized = normalize(center_of_mass, c_max, c_min) + normed_stl_vertices_vol = normalize(stl_vertices, c_max, c_min) + else: + center_of_mass_normalized = center_of_mass + + # SDF calculation on the grid using WARP + time_start = time.time() + sdf_grid, _ = signed_distance_field( + normed_stl_vertices_vol, + mesh_indices_flattened, + grid, + use_sign_winding_number=True, + ) + + # SDF calculation + time_start = time.time() + sdf_nodes, sdf_node_closest_point = signed_distance_field( + normed_stl_vertices_vol, + mesh_indices_flattened, + volume_coordinates, + use_sign_winding_number=True, + ) + sdf_nodes = sdf_nodes.reshape(-1, 1) + vol_grid_max_min = torch.stack([c_min, c_max]) + + pos_volume_closest = volume_coordinates - sdf_node_closest_point + pos_volume_center_of_mass = volume_coordinates - center_of_mass_normalized + + else: + volume_coordinates = None + volume_fields = None + pos_volume_closest = None + pos_volume_center_of_mass = None + + # print(f"Processed sdf and normalized") + + geom_centers = stl_vertices + # print(f"Geom centers max: {np.amax(geom_centers, axis=0)}, min: {np.amin(geom_centers, axis=0)}") + + if model_type == "combined": + # Add the parameters to the dictionary + data_dict = { + "pos_volume_closest": pos_volume_closest, + "pos_volume_center_of_mass": pos_volume_center_of_mass, + "pos_surface_center_of_mass": pos_surface_center_of_mass, + "geometry_coordinates": geom_centers, + "grid": grid, + "surf_grid": surf_grid, + "sdf_grid": sdf_grid, + "sdf_surf_grid": sdf_surf_grid, + "sdf_nodes": sdf_nodes, + "surface_mesh_centers": surface_coordinates, + "surface_mesh_neighbors": surface_neighbors, + "surface_normals": surface_normals, + "surface_neighbors_normals": surface_neighbors_normals, + "surface_areas": surface_sizes, + "surface_neighbors_areas": surface_neighbors_sizes, + "volume_fields": volume_fields, + "volume_mesh_centers": volume_coordinates, + "surface_fields": surface_fields, + "volume_min_max": vol_grid_max_min, + "surface_min_max": surf_grid_max_min, + "length_scale": length_scale, + "global_params_values": torch.unsqueeze(global_params_values, -1), + "global_params_reference": torch.unsqueeze(global_params_reference, -1), + } + elif model_type == "surface": + data_dict = { + "pos_surface_center_of_mass": pos_surface_center_of_mass, + "geometry_coordinates": geom_centers, + "surf_grid": surf_grid, + "sdf_surf_grid": sdf_surf_grid, + "surface_mesh_centers": surface_coordinates, + "surface_mesh_neighbors": surface_neighbors, + "surface_normals": surface_normals, + "surface_neighbors_normals": surface_neighbors_normals, + "surface_areas": surface_sizes, + "surface_neighbors_areas": surface_neighbors_sizes, + "surface_fields": surface_fields, + "surface_min_max": surf_grid_max_min, + "length_scale": length_scale, + "global_params_values": torch.unsqueeze(global_params_values, -1), + "global_params_reference": torch.unsqueeze(global_params_reference, -1), + } + elif model_type == "volume": + data_dict = { + "pos_volume_closest": pos_volume_closest, + "pos_volume_center_of_mass": pos_volume_center_of_mass, + "geometry_coordinates": geom_centers, + "grid": grid, + "surf_grid": surf_grid, + "sdf_grid": sdf_grid, + "sdf_surf_grid": sdf_surf_grid, + "sdf_nodes": sdf_nodes, + "volume_fields": volume_fields, + "volume_mesh_centers": volume_coordinates, + "volume_min_max": vol_grid_max_min, + "surface_min_max": surf_grid_max_min, + "length_scale": length_scale, + "global_params_values": torch.unsqueeze(global_params_values, -1), + "global_params_reference": torch.unsqueeze(global_params_reference, -1), + } + + data_dict = {key: torch.unsqueeze(value, 0) for key, value in data_dict.items()} + + prediction_vol, prediction_surf = test_step( + data_dict, model, dist.device, cfg, vol_factors, surf_factors + ) + + if prediction_surf is not None: + surface_sizes = torch.unsqueeze(surface_sizes, -1) + + pres_x_pred = torch.sum( + prediction_surf[0, :, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + ) + shear_x_pred = torch.sum(prediction_surf[0, :, 1] * surface_sizes[:, 0]) + + pres_x_true = torch.sum( + surface_fields[:, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + ) + shear_x_true = torch.sum(surface_fields[:, 1] * surface_sizes[:, 0]) + + force_x_pred = torch.sum( + prediction_surf[0, :, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + - prediction_surf[0, :, 1] * surface_sizes[:, 0] + ) + force_x_true = torch.sum( + surface_fields[:, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + - surface_fields[:, 1] * surface_sizes[:, 0] + ) + + force_y_pred = torch.sum( + prediction_surf[0, :, 0] * surface_normals[:, 1] * surface_sizes[:, 0] + - prediction_surf[0, :, 2] * surface_sizes[:, 0] + ) + force_y_true = torch.sum( + surface_fields[:, 0] * surface_normals[:, 1] * surface_sizes[:, 0] + - surface_fields[:, 2] * surface_sizes[:, 0] + ) + + force_z_pred = torch.sum( + prediction_surf[0, :, 0] * surface_normals[:, 2] * surface_sizes[:, 0] + - prediction_surf[0, :, 3] * surface_sizes[:, 0] + ) + force_z_true = torch.sum( + surface_fields[:, 0] * surface_normals[:, 2] * surface_sizes[:, 0] + - surface_fields[:, 3] * surface_sizes[:, 0] + ) + print( + "Drag=", dirname, force_x_pred.cpu().numpy(), force_x_true.cpu().numpy() + ) + print( + "Lift=", dirname, force_z_pred.cpu().numpy(), force_z_true.cpu().numpy() + ) + print( + "Side=", dirname, force_y_pred.cpu().numpy(), force_y_true.cpu().numpy() + ) + aero_forces_all.append( + [ + dirname, + force_x_pred, + force_x_true, + force_z_pred, + force_z_true, + force_y_pred, + force_y_true, + ] + ) + + l2_gt = torch.mean(torch.square(surface_fields), (0)) + l2_error = torch.mean( + torch.square(prediction_surf[0] - surface_fields), (0) + ) + l2_surface_all.append( + np.sqrt(l2_error.cpu().numpy()) / np.sqrt(l2_gt.cpu().numpy()) + ) + + print( + "Surface L-2 norm:", + dirname, + np.sqrt(l2_error.cpu().numpy()) / np.sqrt(l2_gt.cpu().numpy()), + ) + + if prediction_vol is not None: + target_vol = volume_fields + prediction_vol = prediction_vol[0] + c_min = vol_grid_max_min[0] + c_max = vol_grid_max_min[1] + volume_coordinates = unnormalize(volume_coordinates, c_max, c_min) + ids_in_bbox = torch.where( + (volume_coordinates[:, 0] < c_min[0]) + | (volume_coordinates[:, 0] > c_max[0]) + | (volume_coordinates[:, 1] < c_min[1]) + | (volume_coordinates[:, 1] > c_max[1]) + | (volume_coordinates[:, 2] < c_min[2]) + | (volume_coordinates[:, 2] > c_max[2]) + ) + target_vol[ids_in_bbox] = 0.0 + prediction_vol[ids_in_bbox] = 0.0 + l2_gt = torch.mean(torch.square(target_vol), (0)) + l2_error = torch.mean(torch.square(prediction_vol - target_vol), (0)) + print( + "Volume L-2 norm:", + dirname, + np.sqrt(l2_error.cpu().numpy()) / np.sqrt(l2_gt.cpu().numpy()), + ) + l2_volume_all.append( + np.sqrt(l2_error.cpu().numpy()) / np.sqrt(l2_gt.cpu().numpy()) + ) + + # import pdb; pdb.set_trace() + if prediction_surf is not None: + surfParam_vtk = numpy_support.numpy_to_vtk( + prediction_surf[0, :, 0:1].cpu().numpy() + ) + surfParam_vtk.SetName(f"{surface_variable_names[0]}Pred") + celldata_all.GetCellData().AddArray(surfParam_vtk) + + surfParam_vtk = numpy_support.numpy_to_vtk( + prediction_surf[0, :, 1:].cpu().numpy() + ) + surfParam_vtk.SetName(f"{surface_variable_names[1]}Pred") + celldata_all.GetCellData().AddArray(surfParam_vtk) + + write_to_vtp(celldata_all, vtp_pred_save_path) + + if prediction_vol is not None: + volParam_vtk = numpy_support.numpy_to_vtk( + prediction_vol[:, 0:3].cpu().numpy() + ) + volParam_vtk.SetName(f"{volume_variable_names[0]}Pred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk( + prediction_vol[:, 3:4].cpu().numpy() + ) + volParam_vtk.SetName(f"{volume_variable_names[1]}Pred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk( + prediction_vol[:, 4:5].cpu().numpy() + ) + volParam_vtk.SetName(f"{volume_variable_names[2]}Pred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + write_to_vtu(polydata_vol, vtu_pred_save_path) + + l2_surface_all = np.asarray(l2_surface_all) # num_files, 4 + l2_volume_all = np.asarray(l2_volume_all) # num_files, 4 + l2_surface_mean = np.mean(l2_surface_all, 0) + l2_volume_mean = np.mean(l2_volume_all, 0) + print( + f"Mean over all samples, surface={l2_surface_mean} and volume={l2_volume_mean}" + ) + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino/src/train.py b/examples/cfd/external_aerodynamics/domino/src/train.py new file mode 100644 index 0000000000..613662c279 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/train.py @@ -0,0 +1,712 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code defines a distributed pipeline for training the DoMINO model on +CFD datasets. It includes the computation of scaling factors, instantiating +the DoMINO model and datapipe, automatically loading the most recent checkpoint, +training the model in parallel using DistributedDataParallel across multiple +GPUs, calculating the loss and updating model parameters using mixed precision. +This is a common recipe that enables training of combined models for surface and +volume as well either of them separately. Validation is also conducted every epoch, +where predictions are compared against ground truth values. The code logs training +and validation metrics to TensorBoard. The train tab in config.yaml can be used to +specify batch size, number of epochs and other training parameters. +""" + +import time +import os +import re +from typing import Literal, Any +from tabulate import tabulate + +import numpy as np +import hydra +from hydra.utils import to_absolute_path +from omegaconf import DictConfig, OmegaConf + +# This will set up the cupy-ecosystem and pytorch to share memory pools +from physicsnemo.utils.memory import unified_gpu_memory + +import torchinfo +import torch +import torch.distributed as dist +from torch.distributed.fsdp import fully_shard +from torch.distributed.tensor import distribute_module + +from torch.amp import GradScaler, autocast +from torch.nn.parallel import DistributedDataParallel +from torch.utils.data import DataLoader +from torch.utils.data.distributed import DistributedSampler +from torch.utils.tensorboard import SummaryWriter +from nvtx import annotate as nvtx_annotate +import torch.cuda.nvtx as nvtx + + +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, RankZeroLoggingWrapper + +from physicsnemo.datapipes.cae.domino_datapipe import ( + DoMINODataPipe, + create_domino_dataset, +) +from physicsnemo.models.domino.model import DoMINO +from physicsnemo.models.domino.utils import create_directory + +from utils import ScalingFactors, get_keys_to_read, coordinate_distributed_environment + +# This is included for GPU memory tracking: +from pynvml import nvmlInit, nvmlDeviceGetHandleByIndex, nvmlDeviceGetMemoryInfo +import time + + +# Initialize NVML +nvmlInit() + + +from physicsnemo.utils.profiling import profile, Profiler + + +from loss import compute_loss_dict +from utils import get_num_vars, load_scaling_factors, compute_l2, all_reduce_dict + + +def validation_step( + dataloader, + model, + device, + logger, + tb_writer, + epoch_index, + use_sdf_basis=False, + use_surface_normals=False, + integral_scaling_factor=1.0, + loss_fn_type=None, + vol_loss_scaling=None, + surf_loss_scaling=None, + first_deriv: torch.nn.Module | None = None, + eqn: Any = None, + bounding_box: torch.Tensor | None = None, + vol_factors: torch.Tensor | None = None, + add_physics_loss=False, + autocast_enabled=None, +): + dm = DistributedManager() + running_vloss = 0.0 + with torch.no_grad(): + metrics = None + + for i_batch, sampled_batched in enumerate(dataloader): + with autocast("cuda", enabled=autocast_enabled, cache_enabled=False): + if add_physics_loss: + prediction_vol, prediction_surf = model( + sampled_batched, return_volume_neighbors=True + ) + else: + prediction_vol, prediction_surf = model(sampled_batched) + + loss, loss_dict = compute_loss_dict( + prediction_vol, + prediction_surf, + sampled_batched, + loss_fn_type, + integral_scaling_factor, + surf_loss_scaling, + vol_loss_scaling, + first_deriv, + eqn, + bounding_box, + vol_factors, + add_physics_loss, + ) + + running_vloss += loss.item() + local_metrics = compute_l2( + prediction_surf, prediction_vol, sampled_batched, dataloader + ) + if metrics is None: + metrics = local_metrics + else: + metrics = { + key: metrics[key] + local_metrics[key] for key in metrics.keys() + } + + avg_vloss = running_vloss / (i_batch + 1) + metrics = {key: metrics[key] / (i_batch + 1) for key in metrics.keys()} + + metrics = all_reduce_dict(metrics, dm) + + if dm.rank == 0: + logger.info( + f" Device {device}, batch: {i_batch + 1}, VAL loss norm: {loss.detach().item():.5f}" + ) + tb_x = epoch_index + for key in metrics.keys(): + tb_writer.add_scalar(f"L2 Metrics/val/{key}", metrics[key], tb_x) + + metrics_table = tabulate( + [[k, v] for k, v in metrics.items()], + headers=["Metric", "Average Value"], + tablefmt="pretty", + ) + logger.info( + f"\nEpoch {epoch_index} VALIDATION Average Metrics:\n{metrics_table}\n" + ) + + return avg_vloss + + +@profile +def train_epoch( + dataloader, + model, + optimizer, + scaler, + tb_writer, + logger, + gpu_handle, + epoch_index, + device, + integral_scaling_factor, + loss_fn_type, + vol_loss_scaling=None, + surf_loss_scaling=None, + first_deriv: torch.nn.Module | None = None, + eqn: Any = None, + bounding_box: torch.Tensor | None = None, + vol_factors: torch.Tensor | None = None, + surf_factors: torch.Tensor | None = None, + add_physics_loss=False, + autocast_enabled=None, + grad_clip_enabled=None, + grad_max_norm=None, +): + dm = DistributedManager() + + running_loss = 0.0 + last_loss = 0.0 + loss_interval = 1 + + gpu_start_info = nvmlDeviceGetMemoryInfo(gpu_handle) + start_time = time.perf_counter() + with Profiler(): + io_start_time = time.perf_counter() + metrics = None + for i_batch, sampled_batched in enumerate(dataloader): + io_end_time = time.perf_counter() + if add_physics_loss: + autocast_enabled = False + + with autocast("cuda", enabled=autocast_enabled, cache_enabled=False): + with nvtx.range("Model Forward Pass"): + if add_physics_loss: + prediction_vol, prediction_surf = model( + sampled_batched, return_volume_neighbors=True + ) + else: + prediction_vol, prediction_surf = model(sampled_batched) + + loss, loss_dict = compute_loss_dict( + prediction_vol, + prediction_surf, + sampled_batched, + loss_fn_type, + integral_scaling_factor, + surf_loss_scaling, + vol_loss_scaling, + first_deriv, + eqn, + bounding_box, + vol_factors, + add_physics_loss, + ) + + # Compute metrics: + if isinstance(prediction_vol, tuple): + # This is if return_neighbors is on for volume: + prediction_vol = prediction_vol[0] + + local_metrics = compute_l2( + prediction_surf, prediction_vol, sampled_batched, dataloader + ) + if metrics is None: + metrics = local_metrics + else: + # Sum the running total: + metrics = { + key: metrics[key] + local_metrics[key] for key in metrics.keys() + } + + loss = loss / loss_interval + scaler.scale(loss).backward() + + if ((i_batch + 1) % loss_interval == 0) or (i_batch + 1 == len(dataloader)): + if grad_clip_enabled: + # Unscales the gradients of optimizer's assigned params in-place. + scaler.unscale_(optimizer) + + # Since the gradients of optimizer's assigned params are unscaled, clips as usual. + torch.nn.utils.clip_grad_norm_(model.parameters(), grad_max_norm) + scaler.step(optimizer) + scaler.update() + optimizer.zero_grad() + + # Gather data and report + running_loss += loss.detach().item() + elapsed_time = time.perf_counter() - start_time + io_time = io_end_time - io_start_time + start_time = time.perf_counter() + gpu_end_info = nvmlDeviceGetMemoryInfo(gpu_handle) + gpu_memory_used = gpu_end_info.used / (1024**3) + gpu_memory_delta = (gpu_end_info.used - gpu_start_info.used) / (1024**3) + + logging_string = f"Device {device}, batch processed: {i_batch + 1}\n" + # Format the loss dict into a string: + loss_string = ( + " " + + "\t".join( + [f"{key.replace('loss_', ''):<10}" for key in loss_dict.keys()] + ) + + "\n" + ) + loss_string += ( + " " + + f"\t".join( + [f"{l.detach().item():<10.3e}" for l in loss_dict.values()] + ) + + "\n" + ) + + logging_string += loss_string + logging_string += f" GPU memory used: {gpu_memory_used:.3f} Gb (delta: {gpu_memory_delta:.3f})\n" + logging_string += f" Timings: (IO: {io_time:.2f}, Model: {elapsed_time - io_time:.2f}, Total: {elapsed_time:.2f})s\n" + logger.info(logging_string) + gpu_start_info = nvmlDeviceGetMemoryInfo(gpu_handle) + io_start_time = time.perf_counter() + + last_loss = running_loss / (i_batch + 1) # loss per batch + # Normalize metrics: + metrics = {key: metrics[key] / (i_batch + 1) for key in metrics.keys()} + # reduce metrics across batch: + metrics = all_reduce_dict(metrics, dm) + if dm.rank == 0: + logger.info( + f" Device {device}, batch: {i_batch + 1}, loss norm: {loss.detach().item():.5f}" + ) + tb_x = epoch_index * len(dataloader) + i_batch + 1 + tb_writer.add_scalar("Loss/train", last_loss, tb_x) + for key in metrics.keys(): + tb_writer.add_scalar(f"L2 Metrics/train/{key}", metrics[key], epoch_index) + + metrics_table = tabulate( + [[k, v] for k, v in metrics.items()], + headers=["Metric", "Average Value"], + tablefmt="pretty", + ) + logger.info(f"\nEpoch {epoch_index} Average Metrics:\n{metrics_table}\n") + + return last_loss + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + ###################################################### + # initialize distributed manager + ###################################################### + DistributedManager.initialize() + dist = DistributedManager() + + # DoMINO supports domain parallel training. This function helps coordinate + # how to set that up, if needed. + domain_mesh, data_mesh, placements = coordinate_distributed_environment(cfg) + + if data_mesh is not None: + data_replica_size = data_mesh.size() + data_rank = data_mesh.get_local_rank() + else: + data_replica_size = dist.world_size + data_rank = dist.rank + + ################################ + # Initialize NVML + ################################ + nvmlInit() + gpu_handle = nvmlDeviceGetHandleByIndex(dist.device.index) + + ###################################################### + # Initialize logger + ###################################################### + + logger = PythonLogger("Train") + logger = RankZeroLoggingWrapper(logger, dist) + + logger.info(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + ###################################################### + # Get scaling factors - precompute them if this fails! + ###################################################### + vol_factors, surf_factors = load_scaling_factors(cfg) + + ###################################################### + # Configure the model + ###################################################### + model_type = cfg.model.model_type + num_vol_vars, num_surf_vars, num_global_features = get_num_vars(cfg, model_type) + + if model_type == "combined" or model_type == "surface": + surface_variable_names = list(cfg.variables.surface.solution.keys()) + else: + surface_variable_names = [] + + if model_type == "combined" or model_type == "volume": + volume_variable_names = list(cfg.variables.volume.solution.keys()) + else: + volume_variable_names = [] + + ###################################################### + # Configure physics loss + # Unless enabled, these are null-ops + ###################################################### + add_physics_loss = getattr(cfg.train, "add_physics_loss", False) + + if add_physics_loss: + from physicsnemo.sym.eq.pde import PDE + from physicsnemo.sym.eq.ls.grads import FirstDeriv + from physicsnemo.sym.eq.pdes.navier_stokes import IncompressibleNavierStokes + else: + PDE = FirstDeriv = IncompressibleNavierStokes = None + + # Initialize physics components conditionally + first_deriv = None + eqn = None + if add_physics_loss: + first_deriv = FirstDeriv(dim=3, direct_input=True) + eqn = IncompressibleNavierStokes(rho=1.226, nu="nu", dim=3, time=False) + eqn = eqn.make_nodes(return_as_dict=True) + + # The bounding box is used in calculating the physics loss: + bounding_box = None + if add_physics_loss: + bounding_box = cfg.data.bounding_box + bounding_box = ( + torch.from_numpy( + np.stack([bounding_box["max"], bounding_box["min"]], axis=0) + ) + .to(vol_factors.dtype) + .to(dist.device) + ) + + ###################################################### + # Configure the dataset + ###################################################### + + # This helper function is to determine which keys to read from the data + # (and which to use default values for, if they aren't present - like + # air_density, for example) + keys_to_read, keys_to_read_if_available = get_keys_to_read( + cfg, model_type, get_ground_truth=True + ) + + # The dataset actually works in two pieces + # The core dataset just reads data from disk, and puts it on the GPU if needed. + # The data processesing pipeline will preprocess that data and prepare it for the model. + # Obviously, you need both, so this function will return the datapipeline in + # a way that can be iterated over. + # + # To properly shuffle the data, we use a distributed sampler too. + # It's configured properly for optional domain parallelism, and you have + # to make sure to call set_epoch below. + + train_dataloader = create_domino_dataset( + cfg, + phase="train", + keys_to_read=keys_to_read, + keys_to_read_if_available=keys_to_read_if_available, + vol_factors=vol_factors, + surf_factors=surf_factors, + device_mesh=domain_mesh, + placements=placements, + normalize_coordinates=cfg.data.normalize_coordinates, + sample_in_bbox=cfg.data.sample_in_bbox, + sampling=cfg.data.sampling, + ) + train_sampler = DistributedSampler( + train_dataloader, + num_replicas=data_replica_size, + rank=data_rank, + **cfg.train.sampler, + ) + + val_dataloader = create_domino_dataset( + cfg, + phase="val", + keys_to_read=keys_to_read, + keys_to_read_if_available=keys_to_read_if_available, + vol_factors=vol_factors, + surf_factors=surf_factors, + device_mesh=domain_mesh, + placements=placements, + normalize_coordinates=cfg.data.normalize_coordinates, + sample_in_bbox=cfg.data.sample_in_bbox, + sampling=cfg.data.sampling, + ) + val_sampler = DistributedSampler( + val_dataloader, + num_replicas=data_replica_size, + rank=data_rank, + **cfg.val.sampler, + ) + + ###################################################### + # Configure the model + ###################################################### + model = DoMINO( + input_features=3, + output_features_vol=num_vol_vars, + output_features_surf=num_surf_vars, + global_features=num_global_features, + model_parameters=cfg.model, + ).to(dist.device) + + # Print model summary (structure and parmeter count). + logger.info(f"Model summary:\n{torchinfo.summary(model, verbose=0, depth=2)}\n") + + if dist.world_size > 1: + if domain_mesh is None: + model = DistributedDataParallel( + model, + device_ids=[dist.local_rank], + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + gradient_as_bucket_view=True, + static_graph=True, + ) + else: + model = distribute_module( + model, + device_mesh=domain_mesh, + ) + model = fully_shard(model, mesh=data_mesh) + + ###################################################### + # Initialize optimzer and gradient scaler + ###################################################### + + optimizer_class = None + if cfg.train.optimizer.name == "Adam": + optimizer_class = torch.optim.Adam + elif cfg.train.optimizer.name == "AdamW": + optimizer_class = torch.optim.AdamW + else: + raise ValueError(f"Unsupported optimizer: {cfg.train.optimizer.name}") + optimizer = optimizer_class( + model.parameters(), + lr=cfg.train.optimizer.lr, + weight_decay=cfg.train.optimizer.weight_decay, + ) + if cfg.train.lr_scheduler.name == "MultiStepLR": + scheduler = torch.optim.lr_scheduler.MultiStepLR( + optimizer, + milestones=cfg.train.lr_scheduler.milestones, + gamma=cfg.train.lr_scheduler.gamma, + ) + elif cfg.train.lr_scheduler.name == "CosineAnnealingLR": + scheduler = torch.optim.lr_scheduler.CosineAnnealingLR( + optimizer, + T_max=cfg.train.lr_scheduler.T_max, + eta_min=cfg.train.lr_scheduler.eta_min, + ) + else: + raise ValueError(f"Unsupported scheduler: {cfg.train.lr_scheduler.name}") + + # Initialize the scaler for mixed precision + scaler = GradScaler() + + ###################################################### + # Initialize output tools + ###################################################### + + # Tensorboard Writer to track training. + writer = SummaryWriter(os.path.join(cfg.output, "tensorboard")) + + epoch_number = 0 + + model_save_path = os.path.join(cfg.output, "models") + param_save_path = os.path.join(cfg.output, "param") + best_model_path = os.path.join(model_save_path, "best_model") + if dist.rank == 0: + create_directory(model_save_path) + create_directory(param_save_path) + create_directory(best_model_path) + + if dist.world_size > 1: + torch.distributed.barrier() + + ###################################################### + # Load checkpoint if available + ###################################################### + init_epoch = load_checkpoint( + to_absolute_path(cfg.resume_dir), + models=model, + optimizer=optimizer, + scheduler=scheduler, + scaler=scaler, + device=dist.device, + ) + + if init_epoch != 0: + init_epoch += 1 # Start with the next epoch + epoch_number = init_epoch + + # retrive the smallest validation loss if available + numbers = [] + for filename in os.listdir(best_model_path): + match = re.search(r"\d+\.\d*[1-9]\d*", filename) + if match: + number = float(match.group(0)) + numbers.append(number) + + best_vloss = min(numbers) if numbers else 1_000_000.0 + + initial_integral_factor_orig = cfg.model.integral_loss_scaling_factor + + ###################################################### + # Begin Training loop over epochs + ###################################################### + + for epoch in range(init_epoch, cfg.train.epochs): + start_time = time.perf_counter() + logger.info(f"Device {dist.device}, epoch {epoch_number}:") + + if epoch == init_epoch and add_physics_loss: + logger.info( + "Physics loss enabled - mixed precision (autocast) will be disabled as physics loss computation is not supported with mixed precision" + ) + + # This controls what indices to use for each epoch. + train_sampler.set_epoch(epoch) + val_sampler.set_epoch(epoch) + train_dataloader.dataset.set_indices(list(train_sampler)) + val_dataloader.dataset.set_indices(list(val_sampler)) + + initial_integral_factor = initial_integral_factor_orig + + if epoch > 250: + surface_scaling_loss = 1.0 * cfg.model.surf_loss_scaling + else: + surface_scaling_loss = cfg.model.surf_loss_scaling + + model.train(True) + epoch_start_time = time.perf_counter() + avg_loss = train_epoch( + dataloader=train_dataloader, + model=model, + optimizer=optimizer, + scaler=scaler, + tb_writer=writer, + logger=logger, + gpu_handle=gpu_handle, + epoch_index=epoch, + device=dist.device, + integral_scaling_factor=initial_integral_factor, + loss_fn_type=cfg.model.loss_function, + vol_loss_scaling=cfg.model.vol_loss_scaling, + surf_loss_scaling=surface_scaling_loss, + first_deriv=first_deriv, + eqn=eqn, + bounding_box=bounding_box, + vol_factors=vol_factors, + add_physics_loss=add_physics_loss, + autocast_enabled=cfg.train.amp.enabled, + grad_clip_enabled=cfg.train.amp.clip_grad, + grad_max_norm=cfg.train.amp.grad_max_norm, + ) + epoch_end_time = time.perf_counter() + logger.info( + f"Device {dist.device}, Epoch {epoch_number} took {epoch_end_time - epoch_start_time:.3f} seconds" + ) + epoch_end_time = time.perf_counter() + + model.eval() + avg_vloss = validation_step( + dataloader=val_dataloader, + model=model, + device=dist.device, + logger=logger, + tb_writer=writer, + epoch_index=epoch, + use_sdf_basis=cfg.model.use_sdf_in_basis_func, + use_surface_normals=cfg.model.use_surface_normals, + integral_scaling_factor=initial_integral_factor, + loss_fn_type=cfg.model.loss_function, + vol_loss_scaling=cfg.model.vol_loss_scaling, + surf_loss_scaling=surface_scaling_loss, + first_deriv=first_deriv, + eqn=eqn, + bounding_box=bounding_box, + vol_factors=vol_factors, + add_physics_loss=add_physics_loss, + autocast_enabled=cfg.train.amp.enabled, + ) + + scheduler.step() + logger.info( + f"Device {dist.device} " + f"LOSS train {avg_loss:.5f} " + f"valid {avg_vloss:.5f} " + f"Current lr {scheduler.get_last_lr()[0]} " + f"Integral factor {initial_integral_factor}" + ) + + if dist.rank == 0: + writer.add_scalars( + "Training vs. Validation Loss", + {"Training": avg_loss, "Validation": avg_vloss}, + epoch_number, + ) + writer.flush() + + # Track best performance, and save the model's state + if dist.world_size > 1: + torch.distributed.barrier() + + if avg_vloss < best_vloss: # This only considers GPU: 0, is that okay? + best_vloss = avg_vloss + + if dist.rank == 0: + print(f"Device {dist.device}, Best val loss {best_vloss}") + + if dist.rank == 0 and (epoch + 1) % cfg.train.checkpoint_interval == 0.0: + save_checkpoint( + to_absolute_path(model_save_path), + models=model, + optimizer=optimizer, + scheduler=scheduler, + scaler=scaler, + epoch=epoch, + ) + + epoch_number += 1 + + if scheduler.get_last_lr()[0] == 1e-6: + print("Training ended") + exit() + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino/src/utils.py b/examples/cfd/external_aerodynamics/domino/src/utils.py new file mode 100644 index 0000000000..d34989a359 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/utils.py @@ -0,0 +1,495 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +from dataclasses import dataclass +from typing import Dict, Optional, Any +import numpy as np +import torch +import torch.distributed as dist +import pickle +from pathlib import Path +from typing import Literal +from omegaconf import DictConfig +from physicsnemo.distributed import DistributedManager + +from torch.distributed.tensor.placement_types import ( + Shard, + Replicate, +) + + +def get_num_vars(cfg: dict, model_type: Literal["volume", "surface", "combined"]): + """Calculate the number of variables for volume, surface, and global features. + + This function analyzes the configuration to determine how many variables are needed + for different mesh data types based on the model type. Vector variables contribute + 3 components (x, y, z) while scalar variables contribute 1 component each. + + Args: + cfg: Configuration object containing variable definitions for volume, surface, + and global parameters with their types (scalar/vector). + model_type (str): Type of model - can be "volume", "surface", or "combined". + Determines which variable types are included in the count. + + Returns: + tuple: A 3-tuple containing: + - num_vol_vars (int or None): Number of volume variables. None if model_type + is not "volume" or "combined". + - num_surf_vars (int or None): Number of surface variables. None if model_type + is not "surface" or "combined". + - num_global_features (int): Number of global parameter features. + """ + num_vol_vars = 0 + volume_variable_names = [] + if model_type == "volume" or model_type == "combined": + volume_variable_names = list(cfg.variables.volume.solution.keys()) + for j in volume_variable_names: + if cfg.variables.volume.solution[j] == "vector": + num_vol_vars += 3 + else: + num_vol_vars += 1 + else: + num_vol_vars = None + + num_surf_vars = 0 + surface_variable_names = [] + if model_type == "surface" or model_type == "combined": + surface_variable_names = list(cfg.variables.surface.solution.keys()) + num_surf_vars = 0 + for j in surface_variable_names: + if cfg.variables.surface.solution[j] == "vector": + num_surf_vars += 3 + else: + num_surf_vars += 1 + else: + num_surf_vars = None + + num_global_features = 0 + global_params_names = list(cfg.variables.global_parameters.keys()) + for param in global_params_names: + if cfg.variables.global_parameters[param].type == "vector": + num_global_features += len(cfg.variables.global_parameters[param].reference) + elif cfg.variables.global_parameters[param].type == "scalar": + num_global_features += 1 + else: + raise ValueError(f"Unknown global parameter type") + + return num_vol_vars, num_surf_vars, num_global_features + + +def get_keys_to_read( + cfg: dict, + model_type: Literal["volume", "surface", "combined"], + get_ground_truth: bool = True, +): + """ + This function helps configure the keys to read from the dataset. + + And, if some global parameter values are provided in the config, + they are also read here and passed to the dataset. + + """ + + # Always read these keys: + keys_to_read = ["stl_coordinates", "stl_centers", "stl_faces", "stl_areas"] + + # If these keys are in the config, use them, else provide defaults in + # case they aren't in the dataset: + cfg_params_vec = [] + for key in cfg.variables.global_parameters: + if cfg.variables.global_parameters[key].type == "vector": + cfg_params_vec.extend(cfg.variables.global_parameters[key].reference) + else: + cfg_params_vec.append(cfg.variables.global_parameters[key].reference) + keys_to_read_if_available = { + "global_params_values": torch.tensor(cfg_params_vec).reshape(-1, 1), + "global_params_reference": torch.tensor(cfg_params_vec).reshape(-1, 1), + } + + # Volume keys: + volume_keys = [ + "volume_mesh_centers", + ] + if get_ground_truth: + volume_keys.append("volume_fields") + + # Surface keys: + surface_keys = [ + "surface_mesh_centers", + "surface_normals", + "surface_areas", + ] + if get_ground_truth: + surface_keys.append("surface_fields") + + if model_type == "volume" or model_type == "combined": + keys_to_read.extend(volume_keys) + if model_type == "surface" or model_type == "combined": + keys_to_read.extend(surface_keys) + + return keys_to_read, keys_to_read_if_available + + +def coordinate_distributed_environment(cfg: DictConfig): + """ + Initialize the distributed env for DoMINO. This is actually always a 2D Mesh: + one dimension is the data-parallel dimension (DDP), and the other is the + domain dimension. + + For the training scripts, we need to know the rank, size of each dimension, + and return the domain_mesh and placements for the loader. + + Args: + cfg: Configuration object containing the domain parallelism configuration. + + Returns: + domain_mesh: torch.distributed.DeviceMesh: The domain mesh for the domain parallel dimension. + data_mesh: torch.distributed.DeviceMesh: The data mesh for the data parallel dimension. + placements: dict[str, torch.distributed.tensor.Placement]: The placements for the data set + """ + + if not DistributedManager.is_initialized(): + DistributedManager.initialize() + dist = DistributedManager() + + # Default to no domain parallelism: + domain_size = cfg.get("domain_parallelism", {}).get("domain_size", 1) + + if dist.world_size == 1: + domain_mesh = None + data_mesh = None + placements = None + else: + # Initialize the device mesh: + mesh = dist.initialize_mesh( + mesh_shape=(-1, domain_size), mesh_dim_names=("ddp", "domain") + ) + domain_mesh = mesh["domain"] + data_mesh = mesh["ddp"] + + if domain_size > 1: + # Define the default placements for each tensor that might show up in + # the data. Note that we'll define placements for all keys, even if + # they aren't actually used. + + # Note that placements are defined for pre-batched data, no batch index! + + shard_grid = cfg.get("domain_parallelism", {}).get("shard_grid", False) + shard_points = cfg.get("domain_parallelism", {}).get("shard_points", False) + + if not shard_grid and not shard_points: + raise ValueError( + "Either shard_grid or shard_points must be True if domain_size > 1" + ) + + # Not supported with physics loss: + if cfg.train.add_physics_loss: + raise ValueError( + "Domain parallelism is not supported with physics loss" + ) + + if shard_grid: + grid_like_placement = [ + Shard(0), + ] + else: + grid_like_placement = [ + Replicate(), + ] + + if shard_points: + point_like_placement = [ + Shard(0), + ] + else: + point_like_placement = [ + Replicate(), + ] + + placements = { + "stl_coordinates": point_like_placement, + "stl_centers": point_like_placement, + "stl_faces": point_like_placement, + "stl_areas": point_like_placement, + "surface_fields": point_like_placement, + "volume_mesh_centers": point_like_placement, + "volume_fields": point_like_placement, + "surface_mesh_centers": point_like_placement, + "surface_normals": point_like_placement, + "surface_areas": point_like_placement, + } + else: + domain_mesh = None + placements = None + + return domain_mesh, data_mesh, placements + + +@dataclass +class ScalingFactors: + """ + Data structure for storing scaling factors computed for DoMINO datasets. + + This class provides a clean, easily serializable format for storing + mean, std, min, and max values for different array keys in the dataset. + Uses numpy arrays for easy serialization and cross-platform compatibility. + + Attributes: + mean: Dictionary mapping keys to mean numpy arrays + std: Dictionary mapping keys to standard deviation numpy arrays + min_val: Dictionary mapping keys to minimum value numpy arrays + max_val: Dictionary mapping keys to maximum value numpy arrays + field_keys: List of field keys for which statistics were computed + """ + + mean: Dict[str, np.ndarray] + std: Dict[str, np.ndarray] + min_val: Dict[str, np.ndarray] + max_val: Dict[str, np.ndarray] + field_keys: list[str] + + def to_torch( + self, device: Optional[torch.device] = None + ) -> Dict[str, Dict[str, torch.Tensor]]: + """Convert numpy arrays to torch tensors for use in training/inference.""" + device = device or torch.device("cpu") + + return { + "mean": {k: torch.from_numpy(v).to(device) for k, v in self.mean.items()}, + "std": {k: torch.from_numpy(v).to(device) for k, v in self.std.items()}, + "min_val": { + k: torch.from_numpy(v).to(device) for k, v in self.min_val.items() + }, + "max_val": { + k: torch.from_numpy(v).to(device) for k, v in self.max_val.items() + }, + } + + def save(self, filepath: str | Path) -> None: + """Save scaling factors to pickle file.""" + filepath = Path(filepath) + filepath.parent.mkdir(parents=True, exist_ok=True) + + with open(filepath, "wb") as f: + pickle.dump(self, f) + + @classmethod + def load(cls, filepath: str | Path) -> "ScalingFactors": + """Load scaling factors from pickle file.""" + with open(filepath, "rb") as f: + factors = pickle.load(f) + return factors + + def get_field_shapes(self) -> Dict[str, tuple]: + """Get the shape of each field's statistics.""" + return {key: self.mean[key].shape for key in self.field_keys} + + def summary(self) -> str: + """Generate a human-readable summary of the scaling factors.""" + summary = ["Scaling Factors Summary:"] + summary.append(f"Field Keys: {self.field_keys}") + + for key in self.field_keys: + mean_val = self.mean[key] + std_val = self.std[key] + min_val = self.min_val[key] + max_val = self.max_val[key] + + summary.append(f"\n{key}:") + summary.append(f" Shape: {mean_val.shape}") + summary.append(f" Mean: {mean_val}") + summary.append(f" Std: {std_val}") + summary.append(f" Min: {min_val}") + summary.append(f" Max: {max_val}") + + return "\n".join(summary) + + +def load_scaling_factors( + cfg: DictConfig, logger=None +) -> tuple[torch.Tensor, torch.Tensor]: + """Load scaling factors from the configuration.""" + pickle_path = os.path.join(cfg.data.scaling_factors) + + try: + scaling_factors = ScalingFactors.load(pickle_path) + if logger is not None: + logger.info(f"Scaling factors loaded from: {pickle_path}") + except FileNotFoundError: + raise FileNotFoundError( + f"Scaling factors not found at: {pickle_path}; please run compute_statistics.py to compute them." + ) + + if cfg.model.normalization == "min_max_scaling": + vol_factors = np.asarray( + [ + scaling_factors.max_val["volume_fields"], + scaling_factors.min_val["volume_fields"], + ] + ) + surf_factors = np.asarray( + [ + scaling_factors.max_val["surface_fields"], + scaling_factors.min_val["surface_fields"], + ] + ) + elif cfg.model.normalization == "mean_std_scaling": + vol_factors = np.asarray( + [ + scaling_factors.mean["volume_fields"], + scaling_factors.std["volume_fields"], + ] + ) + surf_factors = np.asarray( + [ + scaling_factors.mean["surface_fields"], + scaling_factors.std["surface_fields"], + ] + ) + else: + raise ValueError(f"Invalid normalization mode: {cfg.model.normalization}") + + vol_factors_tensor = torch.from_numpy(vol_factors) + surf_factors_tensor = torch.from_numpy(surf_factors) + + dm = DistributedManager() + vol_factors_tensor = vol_factors_tensor.to(dm.device, dtype=torch.float32) + surf_factors_tensor = surf_factors_tensor.to(dm.device, dtype=torch.float32) + + return vol_factors_tensor, surf_factors_tensor + + +def compute_l2( + pred_surface: torch.Tensor | None, + pred_volume: torch.Tensor | None, + batch, + dataloader, +) -> dict[str, torch.Tensor]: + """ + Compute the L2 norm between prediction and target. + + Requires the dataloader to unscale back to original values + """ + + l2_dict = {} + + if pred_surface is not None: + _, target_surface = dataloader.unscale_model_outputs( + surface_fields=batch["surface_fields"] + ) + _, pred_surface = dataloader.unscale_model_outputs(surface_fields=pred_surface) + l2_surface = metrics_fn_surface(pred_surface, target_surface) + l2_dict.update(l2_surface) + if pred_volume is not None: + target_volume, _ = dataloader.unscale_model_outputs( + volume_fields=batch["volume_fields"] + ) + pred_volume, _ = dataloader.unscale_model_outputs(volume_fields=pred_volume) + l2_volume = metrics_fn_volume(pred_volume, target_volume) + l2_dict.update(l2_volume) + + return l2_dict + + +def metrics_fn_surface( + pred: torch.Tensor, + target: torch.Tensor, +) -> dict[str, torch.Tensor]: + """ + Computes L2 surface metrics between prediction and target. + + Args: + pred: Predicted values (normalized). + target: Target values (normalized). + + Returns: + Dictionary of L2 surface metrics for pressure and shear components. + """ + + l2_num = (pred - target) ** 2 + l2_num = torch.sum(l2_num, dim=1) + l2_num = torch.sqrt(l2_num) + + l2_denom = target**2 + l2_denom = torch.sum(l2_denom, dim=1) + l2_denom = torch.sqrt(l2_denom) + + l2 = l2_num / l2_denom + + metrics = { + "l2_surf_pressure": torch.mean(l2[:, 0]), + "l2_shear_x": torch.mean(l2[:, 1]), + "l2_shear_y": torch.mean(l2[:, 2]), + "l2_shear_z": torch.mean(l2[:, 3]), + } + + return metrics + + +def metrics_fn_volume( + pred: torch.Tensor, + target: torch.Tensor, +) -> dict[str, torch.Tensor]: + """ + Computes L2 volume metrics between prediction and target. + """ + l2_num = (pred - target) ** 2 + l2_num = torch.sum(l2_num, dim=1) + l2_num = torch.sqrt(l2_num) + + l2_denom = target**2 + l2_denom = torch.sum(l2_denom, dim=1) + l2_denom = torch.sqrt(l2_denom) + + l2 = l2_num / l2_denom + + metrics = { + "l2_vol_pressure": torch.mean(l2[:, 3]), + "l2_velocity_x": torch.mean(l2[:, 0]), + "l2_velocity_y": torch.mean(l2[:, 1]), + "l2_velocity_z": torch.mean(l2[:, 2]), + "l2_nut": torch.mean(l2[:, 4]), + } + + return metrics + + +def all_reduce_dict( + metrics: dict[str, torch.Tensor], dm: DistributedManager +) -> dict[str, torch.Tensor]: + """ + Reduces a dictionary of metrics across all distributed processes. + + Args: + metrics: Dictionary of metric names to torch.Tensor values. + dm: DistributedManager instance for distributed context. + + Returns: + Dictionary of reduced metrics. + """ + # TODO - update this to use domains and not the full world + + if dm.world_size == 1: + return metrics + + for key, value in metrics.items(): + dist.all_reduce(value) + value = value / dm.world_size + metrics[key] = value + + return metrics diff --git a/examples/cfd/external_aerodynamics/domino/src/validate_cache.py b/examples/cfd/external_aerodynamics/domino/src/validate_cache.py new file mode 100644 index 0000000000..e6789737e3 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino/src/validate_cache.py @@ -0,0 +1,172 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This script processes DoMINODataPipe format files into cached versions +for faster loading during training. It processes files in parallel and can be +configured through config.yaml in the data_processing tab. +""" + +from physicsnemo.datapipes.cae.domino_datapipe import ( + CachedDoMINODataset, + DoMINODataPipe, +) +import hydra +import numpy as np +import os +from omegaconf import DictConfig +import torch +from torch.utils.data import DataLoader +from torch.utils.data.distributed import DistributedSampler +from physicsnemo.distributed import DistributedManager + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + assert cfg.data_processor.use_cache, "Cache must be enabled to be validated!" + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + vol_save_path = os.path.join(cfg.project_dir, "volume_scaling_factors.npy") + surf_save_path = os.path.join(cfg.project_dir, "surface_scaling_factors.npy") + if os.path.exists(vol_save_path): + vol_factors = np.load(vol_save_path) + else: + vol_factors = None + + if os.path.exists(surf_save_path): + surf_factors = np.load(surf_save_path) + else: + surf_factors = None + + # Set up variables based on model type + model_type = cfg.model.model_type + volume_variable_names = [] + surface_variable_names = [] + + if model_type in ["volume", "combined"]: + volume_variable_names = list(cfg.variables.volume.solution.keys()) + if model_type in ["surface", "combined"]: + surface_variable_names = list(cfg.variables.surface.solution.keys()) + + # Create dataset once + dataset_orig = DoMINODataPipe( + data_path=cfg.data_processor.output_dir, # Caching comes after data processing + phase="train", # Phase doesn't matter for caching + grid_resolution=cfg.model.interp_res, + volume_variables=volume_variable_names, + surface_variables=surface_variable_names, + normalize_coordinates=True, + sampling=True, + sample_in_bbox=True, + volume_points_sample=cfg.model.volume_points_sample, + surface_points_sample=cfg.model.surface_points_sample, + geom_points_sample=cfg.model.geom_points_sample, + positional_encoding=cfg.model.positional_encoding, + volume_factors=vol_factors, + surface_factors=surf_factors, + scaling_type=cfg.model.normalization, + model_type=cfg.model.model_type, + bounding_box_dims=cfg.data.bounding_box, + bounding_box_dims_surf=cfg.data.bounding_box_surface, + num_surface_neighbors=cfg.model.num_surface_neighbors, + for_caching=False, + deterministic_seed=True, + ) + + dataset_cached = CachedDoMINODataset( + data_path=cfg.data_processor.cached_dir, + phase="train", + sampling=True, + volume_points_sample=cfg.model.volume_points_sample, + surface_points_sample=cfg.model.surface_points_sample, + geom_points_sample=cfg.model.geom_points_sample, + model_type=cfg.model.model_type, + deterministic_seed=True, + ) + + # Wait for directory creation + if dist.world_size > 1: + torch.distributed.barrier() + + def get_dataloader(dataset, world_size, rank): + sampler = DistributedSampler( + dataset, num_replicas=world_size, rank=rank, shuffle=False + ) + + return DataLoader( + dataset, + sampler=sampler, + batch_size=1, # Process one at a time for caching + num_workers=0, # Must be 0 due to GPU operations in dataset + ) + + dataloader_orig = get_dataloader(dataset_orig, dist.world_size, dist.rank) + dataloader_cached = get_dataloader(dataset_cached, dist.world_size, dist.rank) + + # Process and cache files + for _, (sample_orig, sample_cached) in enumerate( + zip(dataloader_orig, dataloader_cached) + ): + filename_orig = sample_orig["filename"][0] + filename_cached = sample_cached["filename"][0] + mismatched = False + if filename_orig != filename_cached: + print( + f"Rank {dist.rank}: Mismatched filenames: {filename_orig} != {filename_cached}" + ) + mismatched = True + for k, v in sample_orig.items(): + if k in ["filename"]: + continue + if k not in sample_cached: + print(f"Rank {dist.rank}: Key {k} missing from cached sample") + mismatched = True + elif not torch.allclose(v, sample_cached[k]): + print(f"Rank {dist.rank}: Mismatched values for key {k}") + # Get boolean mask of mismatches + mismatches = v != sample_cached[k] + # Get indices where values mismatch + mismatch_indices = torch.nonzero(mismatches, as_tuple=False) + print( + f" Found {len(mismatch_indices)} mismatches, of {v.numel()} total values" + ) + print(f" Tensor shape: {v.shape}, vs {sample_cached[k].shape}") + # Get the actual values at those positions + for idx in mismatch_indices[:5]: # Show only first 5 mismatches + idx_tuple = tuple( + idx.tolist() + ) # Convert index tensor to tuple for indexing + val1 = v[idx_tuple].item() + val2 = sample_cached[k][idx_tuple].item() + print(f" Index {idx_tuple}: {val1} vs {val2}") + mismatched = True + if mismatched: + print(f"FAILED Rank {dist.rank}: {filename_orig}") + else: + print(f"Rank {dist.rank}: {filename_orig} validated") + + # Wait for all processes to complete + if dist.world_size > 1: + torch.distributed.barrier() + + if dist.rank == 0: + print("All processing complete!") + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/README.md b/examples/cfd/external_aerodynamics/domino_nim_finetuning/README.md new file mode 100644 index 0000000000..5993282048 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/README.md @@ -0,0 +1,325 @@ +# DoMINO-Automotive-Aero NIM Fine-tuning + +## Overview + +This example showcases a **fine-tuning recipe** for the **DoMINO-Automotive-Aero NIM**, +featuring an innovative **predictor-corrector approach** specifically designed for +automotive CFD simulations. + +**Accelerated Training**: Dramatically reduce training time by leveraging pre-trained +models instead of starting from scratch + +**Smart Transfer Learning**: Efficiently adapt powerful base models to new vehicle +configurations and boundary conditions + +**Predictor-Corrector Approach**: An approach that combines the strengths of +pre-trained models with AI model based corrections + +The predictor-corrector methodology is described below: + +```bash +Y_finetuned = Y_predictor + Y_corrector +``` + +**The Components:** + +- **Y_predictor**: Output from the pre-trained DoMINO-Automotive-Aero NIM (frozen weights) +- **Y_corrector**: A lightweight, trainable network that learns to correct prediction errors +- **Y_finetuned**: The final enhanced prediction combining both components + +> **💡 Core Insight**: The predictor leverages extensive pre-training to provide robust +baseline predictions, while the corrector focuses on learning dataset-specific refinements. +This division of labor leads to faster convergence and superior performance compared to +training from scratch. + +The finetuning example validated on the OSS DrivAerML dataset with only 16 training +and 8 testing samples. The results presented are preliminary and show encouraging +results. A thorough investigation is underway to provide more concrete datapoints +in terms of accuracy improvement and convergence acceleration. + +### Key Features + +- **Predictor-Corrector Approach**: Combines pre-trained models with learnable corrections +- **Transfer Learning**: Efficient adaptation to new vehicle configurations and boundary +conditions +- **DrivAerML Integration**: Seamless integration with the DrivAerML dataset +- **Modular Design**: Easy customization of both predictor and corrector models +- **High Performance**: Optimized for multi-GPU training and inference + +### Architecture Components + +| Component | Description | Training Mode | +|-----------|-------------|---------------| +| **Predictor** | Pre-trained DoMINO-Automotive-Aero NIM | Frozen (Evaluation Only) | +| **Corrector** | Custom DoMINO architecture | Trainable | +| **Combined** | Predictor + Corrector outputs | End-to-End Inference | + +## Code Structure + +```bash +domino_automotive_aero_nim_finetuning/ +├── src/ # Core Implementation +│ ├── conf/ # Configuration Management +│ │ ├── config.yaml # Main training configuration +│ │ └── config_base_pred.yaml # Base prediction settings +│ ├── model_base_predictor.py # DoMINO predictor architecture +│ ├── train.py # Training pipeline +│ ├── test.py # Testing & inference pipeline +│ ├── generate_base_predictions.py # Base model predictions +│ ├── process_data.py # Data preprocessing utilities +│ └── openfoam_datapipe.py # VTK → NPY conversion +├── nim_checkpoint/ # Pre-trained Models +│ └── domino-drivesim-recent.pt # Pretrained model weights +├── download_dataset_huggingface.sh # Automated dataset download +└── README.md # This documentation +``` + +## Dataset & Model Setup + +### DrivAerML Dataset + +The **DrivAerML** dataset provides comprehensive automotive CFD simulations with +multiple vehicle configurations. +The dataset maybe found here: [DrivAerML Dataset](https://caemldatasets.org/drivaerml/) + +| File Type | Description | Extension | Use Case | +|-----------|-------------|-----------|----------| +| **Geometry** | Vehicle STL meshes | `.stl` | 3D vehicle structure | +| **Volume Fields** | 3D flow field data | `.vtu` | Velocity, pressure, turbulence | +| **Surface Fields** | Vehicle surface data | `.vtp` | Wall pressure, shear stress | + +### Dataset Download + +```bash +# Download specific runs (e.g., runs 1-32) +./download_dataset_huggingface.sh -d ./drivaer_data -s 1 -e 32 +``` + +### DoMINO-Automotive-Aero NIM Checkpoint + +Download the DoMINO-Automotive-Aero NIM checkpoint from NGC and add it to the +checkpoint directory + +**Source**: [Domino Checkpoint](https://catalog.ngc.nvidia.com/orgs/nim/teams/nvidia/models/domino-drivsim) + +**Note**: Requires NGC API key for access. See [NGC documentation](https://docs.nvidia.com/ngc/) +for setup. + +## Usage Guide + +### Complete Fine-tuning Workflow + + +
+```mermaid +graph TD + A[Download Dataset and pre-trained DoMINO NIM] --> B[Generate Base Predictions] + B --> C[Process Data VTP → NPY] + C --> D[Configure Training] + D --> E[Train Corrector Model] + E --> F[Test & Evaluate] + F --> G[Deploy Fine-tuned Model] +``` + +
+ +### Step-by-Step Instructions + +#### **Step 1: Generate Base Predictions** + +Generate initial predictions using the pre-trained checkpoint. Modify the eval tab in +`config_base_pred.yaml` to specify the path to the downloaded checkpoint. + +```bash +# Run predictor model on dataset +python src/generate_base_predictions.py + +# Output: Predictions saved as VTP files with base model outputs +``` + +#### **Step 2: Data Processing (VTP → NPY)** + +Convert VTP prediction files to efficient NPY format for training: + +```bash +# Convert and preprocess data +python src/process_data.py + +# Output: Training-ready NPY files with predictor outputs + ground truth +``` + +#### **Step 3: Train Corrector Model** + +Train the corrector network to learn prediction refinements: + +```bash +# Start training with default configuration +python src/train.py exp_tag=combined + +# Custom configuration example +python src/train.py \ + exp_tag=1 \ + project.name=Dataset_Finetune \ + model.volume_points_sample=16384 \ + model.surface_points_sample=16384 \ + train.epochs=500 +``` + +#### **Step 4: Test Fine-tuned Model** + +Evaluate the combined predictor-corrector model: + +```bash +# Run inference on test dataset +python src/test.py \ + exp_tag=1 \ + eval.checkpoint_name=DoMINO.0.500.pt \ + eval.save_path=/path/to/results \ + eval.test_path=/path/to/test_data +``` + +Output of the test script are final predictions combining predictor + corrector +written to a VTP/VTU file. + +## Benchmarking results on DrivAerML dataset + +The finetuning recipe is benchmarked for a subset of the DrivAerML dataset. +The finetuning is carried out on the first 24 samples from this dataset and +compared against training from scratch with the DoMINO model on the same dataset. +The DoMINO-Automotive-Aero NIM is trained on a dataset consisting of RANS +simulations, while this DrivAerML dataset consists of high-fidelity, time-averaged +LES simulations. The goal of this recipe is to demonstrate the finetuning of an +existing model checkpoint to a new design space and physics and compare it against +training from scratch. + +Both models are evaluated at 50, 100, 200, 300, and 400 epochs to demonstrate +faster convergence of the finetuned model to an acceptable accuracy as compared +to training from scratch. 18 samples are used for training and 6 for validation. +The results averaged over the validation set are presented in the table below +and demonstrate that finetuning results in faster convergence (in fewer epochs) +of results as compared to training from scratch. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
EpochsBaseline Model L2 ErrorFine-tuned Model L2 Error
VelocityVol. PressureSurf. PressureWall ShearVelocityVol. PressureSurf. PressureWall Shear
500.5210.5570.5460.6830.3420.3160.3740.563
1000.4440.4740.4360.6130.3320.3070.3330.473
2000.4050.3880.3860.5710.3130.3030.3120.416
3000.3900.3650.3690.5630.3100.3010.3080.406
4000.3800.3620.3650.5520.3090.3000.3070.403
+ +It must be noted that the training and validation accuracy for training from +scratch can be improved as more samples are added and the same is the case +with finetuning. The goal of this analysis is to demonstrate the benefits of +finetuning from a pretrained model checkpoint as compared to training from +scratch. A more comprehensive analysis correlating the training from scratch +and finetuning accuracy with the dataset size will be carried out in future. + +## Customization & Extensions + +### Custom Model Architectures + +The recipe is designed for easy customization: + +| Component | File | Customization Level | +|-----------|------|-------------------| +| **Predictor** | `model_base_predictor.py` | **Pretrained Custom +Model (or DoMINO NIM)** | +| **Corrector** | Built-in DoMINO | **Fully Customizable Models** | +| **Training** | `train.py` | **Configuration-driven** | +| **Testing** | `test.py` | **Workflow Adaptable** | + +### Integration Guidelines + +The predictor-corrector approach is model-agnostic. + +**To use custom architectures:** + +1. **Custom Predictor**: Replace `model_base_predictor.py` with your pretrained model +2. **Custom Corrector**: Modify the corrector architecture in training configuration +3. **Maintain Interface**: Ensure input/output compatibility between components +4. **Update Testing**: Adapt `test.py` for new model combinations + +--- + +## Additional Resources + +### Quick Links + +- [DoMINO-Automotive-Aero NIM Docs](https://docs.nvidia.com/nim/physicsnemo/domino-automotive-aero/latest/overview.html) +- [DrivAerML Dataset](https://caemldatasets.org/drivaerml/) diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/download_dataset_huggingface.sh b/examples/cfd/external_aerodynamics/domino_nim_finetuning/download_dataset_huggingface.sh new file mode 100644 index 0000000000..37a07dceb1 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/download_dataset_huggingface.sh @@ -0,0 +1,154 @@ +#!/bin/bash + +# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This Bash script downloads the DrivAer files from the Hugging Face dataset to a local directory. +# Only the volume files (.vtu), STL files (.stl), VTP files (.vtp), and force_mom files (force_mom_i.csv) are downloaded. +# It uses a function, download_run_files, to check for the existence of four specific files (".vtu", ".stl", ".vtp", "force_mom_i.csv") in a run directory. +# If a file doesn't exist, it's downloaded from the Hugging Face dataset. If it does exist, the download is skipped. +# The script runs multiple downloads in parallel, both within a single run and across multiple runs. +# It also includes checks to prevent overloading the system by limiting the number of parallel downloads. + +# Function to display usage information +usage() { + echo "Usage: $0 [OPTIONS]" + echo "Options:" + echo " -d, --local-dir DIR Local directory to download files (default: ./drivaer_data)" + echo " -s, --run-start NUM Starting run number (default: 1)" + echo " -e, --run-end NUM Ending run number (default: 5, max: 500)" + echo " -h, --help Display this help message" + echo "" + echo "Example:" + echo " $0 -d ./my_data -s 10 -e 100" + exit 1 +} + +# Default values +LOCAL_DIR="./drivaer_data" # Directory to save the downloaded files +RUN_START=1 +RUN_END=32 + +# Parse command-line arguments +while [[ $# -gt 0 ]]; do + case $1 in + -d|--local-dir) + LOCAL_DIR="$2" + shift 2 + ;; + -s|--run-start) + RUN_START="$2" + shift 2 + ;; + -e|--run-end) + RUN_END="$2" + shift 2 + ;; + -h|--help) + usage + ;; + *) + echo "Unknown option: $1" + usage + ;; + esac +done + +# Validate arguments +if ! [[ "$RUN_START" =~ ^[0-9]+$ ]] || ! [[ "$RUN_END" =~ ^[0-9]+$ ]]; then + echo "Error: run_start and run_end must be positive integers" + exit 1 +fi + +if [ "$RUN_START" -gt "$RUN_END" ]; then + echo "Error: run_start cannot be greater than run_end" + exit 1 +fi + +if [ "$RUN_END" -gt 500 ]; then + echo "Error: run_end cannot be greater than 500 (maximum available runs)" + exit 1 +fi + +# Set the path and prefix +HF_OWNER="neashton" +HF_PREFIX="drivaerml" + +# Create the local directory if it doesn't exist +mkdir -p "$LOCAL_DIR" + +# Function to download files for a specific run +download_run_files() { + local i=$1 + RUN_DIR="run_$i" + RUN_LOCAL_DIR="$LOCAL_DIR/$RUN_DIR" + + # Create the run directory if it doesn't exist + mkdir -p "$RUN_LOCAL_DIR" + + # Download the drivaer_i.stl file + if [ ! -f "$RUN_LOCAL_DIR/drivaer_$i.stl" ]; then + wget "https://huggingface.co/datasets/${HF_OWNER}/${HF_PREFIX}/resolve/main/$RUN_DIR/drivaer_$i.stl" -O "$RUN_LOCAL_DIR/drivaer_$i.stl" + else + echo "File drivaer_$i.stl already exists, skipping download." + fi + + # Download the boundary_i.vtp file + if [ ! -f "$RUN_LOCAL_DIR/boundary_$i.vtp" ]; then + wget "https://huggingface.co/datasets/${HF_OWNER}/${HF_PREFIX}/resolve/main/$RUN_DIR/boundary_$i.vtp" -O "$RUN_LOCAL_DIR/boundary_$i.vtp" + else + echo "File boundary_$i.vtp already exists, skipping download." + fi + + # Download the volume_i.vtu files + # Check if the .vtu file exists before downloading + if [ ! -f "$RUN_LOCAL_DIR/volume_$i.vtu" ]; then + wget "https://huggingface.co/datasets/${HF_OWNER}/${HF_PREFIX}/resolve/main/$RUN_DIR/volume_$i.vtu.00.part" -O "$RUN_LOCAL_DIR/volume_$i.vtu.00.part" + wget "https://huggingface.co/datasets/${HF_OWNER}/${HF_PREFIX}/resolve/main/$RUN_DIR/volume_$i.vtu.01.part" -O "$RUN_LOCAL_DIR/volume_$i.vtu.01.part" + # Concatenate the volume files + cat "$RUN_LOCAL_DIR/volume_$i.vtu.00.part" "$RUN_LOCAL_DIR/volume_$i.vtu.01.part" > "$RUN_LOCAL_DIR/volume_$i.vtu" + # Remove the part files + rm "$RUN_LOCAL_DIR/volume_$i.vtu.00.part" "$RUN_LOCAL_DIR/volume_$i.vtu.01.part" + else + echo "File volume_$i.vtu already exists, skipping download." + fi + + # Download the force_mom_i.csv file + if [ ! -f "$RUN_LOCAL_DIR/force_mom_$i.csv" ]; then + wget "https://huggingface.co/datasets/${HF_OWNER}/${HF_PREFIX}/resolve/main/$RUN_DIR/force_mom_$i.csv" -O "$RUN_LOCAL_DIR/force_mom_$i.csv" + else + echo "File force_mom_$i.csv already exists, skipping download." + fi + + wait # Ensure that all files for this run are downloaded before moving to the next run +} + +echo "Starting download from run $RUN_START to run $RUN_END to directory: $LOCAL_DIR" + +# Loop through the run folders and download the files +for i in $(seq "$RUN_START" "$RUN_END"); do + download_run_files "$i" & + + # Limit the number of parallel jobs to avoid overloading the system + if (( $(jobs -r | wc -l) >= 8 )); then + wait -n # Wait for the next background job to finish before starting a new one + fi +done + +# Wait for all remaining background jobs to finish +wait + +echo "Download completed for runs $RUN_START to $RUN_END" diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config.yaml b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config.yaml new file mode 100644 index 0000000000..7ed2dc9c40 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config.yaml @@ -0,0 +1,210 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This is the config for the finetuned model. + +# ┌───────────────────────────────────────────┐ +# │ Project Details │ +# └───────────────────────────────────────────┘ +project: # Project name + name: AWS_Dataset_Finetune + +exp_tag: 1 # Experiment tag +# Main output directory. +project_dir: outputs/${project.name}/ +output: outputs/${project.name}/${exp_tag} + +hydra: # Hydra config + run: + dir: ${output} + output_subdir: hydra # Default is .hydra which causes files not being uploaded in W&B. + +# The directory to search for checkpoints to continue training. +resume_dir: ${output}/models + +# ┌───────────────────────────────────────────┐ +# │ Data Preprocessing │ +# └───────────────────────────────────────────┘ +data_processor: # Data processor configurable parameters + kind: drivaer_aws # must be either drivesim or drivaer_aws + output_dir: /user/data/drivaer_data_finetune_processed_1 + input_dir: /user/datasets/drivaer_aws/drivaer_data_finetune + cached_dir: /user/cached/drivaer_aws/drivaer_data_full/ + use_cache: false + num_processors: 6 + +# ┌───────────────────────────────────────────┐ +# │ Solution variables │ +# └───────────────────────────────────────────┘ +variables: + surface: + solution: + # The following is for AWS DrivAer dataset. + pMeanTrimDelta: scalar + wallShearStressMeanTrimDelta: vector + volume: + solution: + # The following is for AWS DrivAer dataset. + UMeanTrimDelta: vector + pMeanTrimDelta: scalar + nutMeanTrimDelta: scalar + global_parameters: + inlet_velocity: + type: vector + reference: [38.89] # vector [30, 0, 0] should be specified as [30], while [30, 30, 0] should be [30, 30]. + air_density: + type: scalar + reference: 1.226 + +# ┌───────────────────────────────────────────┐ +# │ Training Data Configs │ +# └───────────────────────────────────────────┘ +data: # Input directory for training and validation data + input_dir: /user/data/drivaer_data_finetune_processed + input_dir_val: /user/data/drivaer_data_finetune_processed_val + bounding_box: # Bounding box dimensions for computational domain + min: [-3.5, -2.25 , -0.32] + max: [8.5 , 2.25 , 3.00] + bounding_box_surface: # Bounding box dimensions for car surface + min: [-1.1, -1.2 , -0.32] + max: [4.2 , 1.2 , 1.3] + gpu_preprocessing: true + gpu_output: true + +# ┌───────────────────────────────────────────┐ +# │ Domain Parallelism Settings │ +# └───────────────────────────────────────────┘ +domain_parallelism: + domain_size: 1 + shard_grid: false + shard_points: false + +# ┌───────────────────────────────────────────┐ +# │ Model Parameters │ +# └───────────────────────────────────────────┘ +model: + model_type: combined # train which model? surface, volume, combined + activation: "relu" # "relu" or "gelu" + loss_function: + loss_type: "mse" # mse or rmse + area_weighing_factor: 10000 # Generally inverse of maximum area + interp_res: [128, 64, 64] # resolution of latent space 128, 64, 48 + use_sdf_in_basis_func: true # SDF in basis function network + positional_encoding: false # calculate positional encoding? + volume_points_sample: 8192 # Number of points to sample in volume per epoch + surface_points_sample: 8192 # Number of points to sample on surface per epoch + surface_sampling_algorithm: solution_weighted # random or area_weighted + geom_points_sample: 300_000 # Number of points to sample on STL per epoch + num_neighbors_surface: 7 # How many neighbors on surface? + num_neighbors_volume: 10 # How many neighbors on volume? + combine_volume_surface: false # combine volume and surface encodings + return_volume_neighbors: True # Whether to return volume neighbors or not + use_surface_normals: true # Use surface normals and surface areas for surface computation? + use_surface_area: true # Use only surface normals and not surface area + integral_loss_scaling_factor: 100 # Scale integral loss by this factor + normalization: min_max_scaling # or mean_std_scaling + encode_parameters: false # encode inlet velocity and air density in the model + surf_loss_scaling: 1.0 # scale surface loss with this factor in combined mode + vol_loss_scaling: 1.0 # scale volume loss with this factor in combined mode + geometry_encoding_type: both # geometry encoder type, sdf, stl, both + solution_calculation_mode: two-loop # one-loop is better for sharded, two-loop is lower memory but more overhead. Physics losses are not supported via one-loop presently. + resampling_surface_mesh: # resampling of surface mesh before constructing kd tree + resample: false #false or true + points: 1_000_000 # number of points + geometry_rep: # Hyperparameters for geometry representation network + geo_conv: + base_neurons: 32 # 256 or 64 + base_neurons_in: 1 + base_neurons_out: 1 + volume_radii: [0.05, 0.25, 1.0, 2.5] # radii for volume + surface_radii: [0.01, 0.05, 0.1] # radii for surface + surface_hops: 1 # Number of surface iterations + volume_hops: 1 # Number of volume iterations + volume_neighbors_in_radius: [10, 10, 10, 10] # Number of neighbors in radius for volume + surface_neighbors_in_radius: [10, 10, 10] # Number of neighbors in radius for surface + fourier_features: false + num_modes: 5 + activation: ${model.activation} + geo_processor: + base_filters: 8 + activation: ${model.activation} + processor_type: conv # conv or unet + self_attention: false + cross_attention: false + nn_basis_functions: # Hyperparameters for basis function network + base_layer: 512 + fourier_features: true + num_modes: 5 + activation: ${model.activation} + local_point_conv: + activation: ${model.activation} + aggregation_model: # Hyperparameters for aggregation network + base_layer: 512 + activation: ${model.activation} + position_encoder: # Hyperparameters for position encoding network + base_neurons: 512 + activation: ${model.activation} + fourier_features: false + num_modes: 5 + geometry_local: # Hyperparameters for local geometry extraction + volume_neighbors_in_radius: [32, 128] # Number of radius points + surface_neighbors_in_radius: [32, 128] # Number of radius points + volume_radii: [0.05, 0.25] # Volume radii + surface_radii: [0.05, 0.25] # Surface radii + base_layer: 512 + parameter_model: + base_layer: 512 + fourier_features: false + num_modes: 5 + activation: ${model.activation} + +# ┌───────────────────────────────────────────┐ +# │ Training Configs │ +# └───────────────────────────────────────────┘ +train: # Training configurable parameters + epochs: 1000 + checkpoint_interval: 1 + dataloader: + batch_size: 1 + pin_memory: false # if the preprocessing is outputing GPU data, set this to false + sampler: + shuffle: true + drop_last: false + checkpoint_dir: /user/models/ # Use only for retraining + add_physics_loss: false + + +# ┌───────────────────────────────────────────┐ +# │ Validation Configs │ +# └───────────────────────────────────────────┘ +val: # Validation configurable parameters + dataloader: + batch_size: 1 + pin_memory: false # if the preprocessing is outputing GPU data, set this to false + sampler: + shuffle: true + drop_last: false + +# ┌───────────────────────────────────────────┐ +# │ Testing data Configs │ +# └───────────────────────────────────────────┘ +eval: # Testing configurable parameters + test_path: /user/datasets/drivaer_aws/drivaer_data_finetune_test # Dir for testing data in raw format (vtp, vtu ,stls) + save_path: /user/data/drivaer_data_finetune_test_predicted # Dir to save predicted results in raw format (vtp, vtu) + checkpoint_name: DoMINO.0.455.pt # Name of checkpoint to select from saved checkpoints + scaling_param_path: /user/modulus/physicsnemo/examples/cfd/external_aerodynamics/domino_nim_finetuning/outputs/AWS_Dataset_Finetune/ + refine_stl: False # Automatically refine STL during inference + stencil_size: 7 # Stencil size for evaluating surface and volume model diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config_base_pred.yaml b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config_base_pred.yaml new file mode 100644 index 0000000000..45121cf2de --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config_base_pred.yaml @@ -0,0 +1,180 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This is the config for generating base model predictions. + +# ┌───────────────────────────────────────────┐ +# │ Project Details │ +# └───────────────────────────────────────────┘ +project: # Project name + name: domino_base_pred + +exp_tag: 1 # Experiment tag +# Main output directory. +project_dir: outputs/${project.name}/ +output: outputs/${project.name}/${exp_tag} + +hydra: # Hydra config + run: + dir: ${output} + output_subdir: hydra # Default is .hydra which causes files not being uploaded in W&B. + +# The directory to search for checkpoints to continue training. +resume_dir: ${output}/models + +# ┌───────────────────────────────────────────┐ +# │ Data Preprocessing │ +# └───────────────────────────────────────────┘ +data_processor: # Data processor configurable parameters + kind: drivesim # must be either drivesim or drivaer_aws + output_dir: /user/data/drivaer_data_finetune_processed + input_dir: /user/datasets/drivaer_aws/drivaer_data_finetune + cached_dir: /user/cached/drivaer_aws/drivaer_data_full/ + use_cache: false + num_processors: 12 + +# ┌───────────────────────────────────────────┐ +# │ Solution variables │ +# └───────────────────────────────────────────┘ +variables: + surface: + solution: + # The following is for AWS DrivAer dataset. + pMeanTrim: scalar + wallShearStressMeanTrim: vector + volume: + solution: + # The following is for AWS DrivAer dataset. + UMeanTrim: vector + pMeanTrim: scalar + nutMeanTrim: scalar + +# ┌───────────────────────────────────────────┐ +# │ Training Data Configs │ +# └───────────────────────────────────────────┘ +data: # Input directory for training and validation data + input_dir: /user/data/drivaer_data_finetune_processed + input_dir_val: /user/data/drivaer_data_finetune_processed_val + bounding_box: # Bounding box dimensions for computational domain + min: [-3, -2.5 , -0.35] + max: [8.5 , 2.5 , 3.5] + bounding_box_surface: # Bounding box dimensions for car surface + min: [-1.4, -1.5 , -0.35] + max: [5.25 , 1.5 , 2.0] + +# ┌───────────────────────────────────────────┐ +# │ Domain Parallelism Settings │ +# └───────────────────────────────────────────┘ +domain_parallelism: + domain_size: 1 + shard_grid: false + shard_points: false + +# ┌───────────────────────────────────────────┐ +# │ Model Parameters │ +# └───────────────────────────────────────────┘ +model: + model_type: combined # train which model? surface, volume, combined + loss_function: + loss_type: "mse" # mse or rmse + area_weighing_factor: 1000 # Generally inverse of maximum area + interp_res: [128, 64, 64] # resolution of latent space 128, 64, 48 + use_sdf_in_basis_func: true # SDF in basis function network + positional_encoding: false # calculate positional encoding? + volume_points_sample: 8192 # Number of points to sample in volume per epoch + surface_points_sample: 8192 # Number of points to sample on surface per epoch + surface_sampling_algorithm: area_weighted # random or area_weighted + geom_points_sample: 300_000 # Number of points to sample on STL per epoch + surface_neighbors: true # Pre-compute surface neighborhood from input data + num_surface_neighbors: 1 # How many neighbors? + use_surface_normals: true # Use surface normals and surface areas for surface computation? + use_surface_area: true # Use only surface normals and not surface area + integral_loss_scaling_factor: 100 # Scale integral loss by this factor + normalization: min_max_scaling # or mean_std_scaling + encode_parameters: true # encode inlet velocity and air density in the model + surf_loss_scaling: 5.0 # scale surface loss with this factor in combined mode + vol_loss_scaling: 1.0 # scale volume loss with this factor in combined mode + geometry_encoding_type: both # geometry encoder type, sdf, stl, both + solution_calculation_mode: two-loop # one-loop is better for sharded, two-loop is lower memory but more overhead + resampling_surface_mesh: # resampling of surface mesh before constructing kd tree + resample: false #false or true + points: 1_000_000 # number of points + geometry_rep: # Hyperparameters for geometry representation network + geo_conv: + base_neurons: 32 # 256 or 64 + base_neurons_out: 1 + volume_radii: [0.1, 0.5, 2.5, 5.0] + surface_radii: [0.01, 0.05, 0.1] # 0.05 + hops: 1 + geo_processor: + base_filters: 8 + geo_processor_sdf: + base_filters: 8 + nn_basis_functions: # Hyperparameters for basis function network + base_layer: 512 + fourier_features: false + num_modes: 5 + aggregation_model: # Hyperparameters for aggregation network + base_layer: 512 + position_encoder: # Hyperparameters for position encoding network + base_neurons: 512 + geometry_local: # Hyperparameters for local geometry extraction + volume_neighbors_in_radius: [64] # [64, 128] + surface_neighbors_in_radius: [64] # [64] + volume_radii: [0.05] # [0.05. 0.1] + surface_radii: [0.05] # [0.05] + base_layer: 512 + parameter_model: + base_layer: 512 + scaling_params: [30.0, 1.226] # [inlet_velocity, air_density] + fourier_features: false + num_modes: 5 + +# ┌───────────────────────────────────────────┐ +# │ Training Configs │ +# └───────────────────────────────────────────┘ +train: # Training configurable parameters + epochs: 1000 + checkpoint_interval: 1 + dataloader: + batch_size: 1 + pin_memory: false # if the preprocessing is outputing GPU data, set this to false + sampler: + shuffle: true + drop_last: false + checkpoint_dir: /code/user/finetuning # Use only for retraining + +# ┌───────────────────────────────────────────┐ +# │ Validation Configs │ +# └───────────────────────────────────────────┘ +val: # Validation configurable parameters + dataloader: + batch_size: 1 # Set to 1 + pin_memory: false # if the preprocessing is outputing GPU data, set this to false + sampler: + shuffle: true + drop_last: false + +# ┌───────────────────────────────────────────┐ +# │ Testing data Configs │ +# └───────────────────────────────────────────┘ +eval: # Testing configurable parameters + test_path: /user/datasets/drivaer_aws/drivaer_data_finetune + save_path: /user/nim_predictions # Dir to save predicted results in raw format (vtp, vtu) + checkpoint_dir: /user/nim_checkpoint + checkpoint_name: domino-drivesim-recent.pt # Name of checkpoint to select from saved checkpoints + refine_stl: False # Automatically refine STL during inference + stencil_size: 1 # Stencil size for evaluating surface and volume model diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config_baseline.yaml b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config_baseline.yaml new file mode 100644 index 0000000000..768f5c4e84 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/conf/config_baseline.yaml @@ -0,0 +1,210 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This is the config for the baseline model. + +# ┌───────────────────────────────────────────┐ +# │ Project Details │ +# └───────────────────────────────────────────┘ +project: # Project name + name: AWS_Dataset_Baseline + +exp_tag: 1 # Experiment tag +# Main output directory. +project_dir: outputs/${project.name}/ +output: outputs/${project.name}/${exp_tag} + +hydra: # Hydra config + run: + dir: ${output} + output_subdir: hydra # Default is .hydra which causes files not being uploaded in W&B. + +# The directory to search for checkpoints to continue training. +resume_dir: ${output}/models + +# ┌───────────────────────────────────────────┐ +# │ Data Preprocessing │ +# └───────────────────────────────────────────┘ +data_processor: # Data processor configurable parameters + kind: drivaer_aws # must be either drivesim or drivaer_aws + output_dir: /user/data/drivaer_data_baseline_processed + input_dir: /user/data/drivaer_data_baseline + cached_dir: /user/cached/drivaer_aws/drivaer_data_full/ + use_cache: false + num_processors: 6 + +# ┌───────────────────────────────────────────┐ +# │ Solution variables │ +# └───────────────────────────────────────────┘ +variables: + surface: + solution: + # The following is for AWS DrivAer dataset. + pMeanTrim: scalar + wallShearStressMeanTrim: vector + volume: + solution: + # The following is for AWS DrivAer dataset. + UMeanTrim: vector + pMeanTrim: scalar + nutMeanTrim: scalar + global_parameters: + inlet_velocity: + type: vector + reference: [38.89] # vector [30, 0, 0] should be specified as [30], while [30, 30, 0] should be [30, 30]. + air_density: + type: scalar + reference: 1.226 + +# ┌───────────────────────────────────────────┐ +# │ Training Data Configs │ +# └───────────────────────────────────────────┘ +data: # Input directory for training and validation data + input_dir: /user/data/drivaer_data_baseline_processed + input_dir_val: /user/data/drivaer_data_baseline_processed_val + bounding_box: # Bounding box dimensions for computational domain + min: [-3.5, -2.25 , -0.32] + max: [8.5 , 2.25 , 3.00] + bounding_box_surface: # Bounding box dimensions for car surface + min: [-1.1, -1.2 , -0.32] + max: [4.2 , 1.2 , 1.3] + gpu_preprocessing: true + gpu_output: true + +# ┌───────────────────────────────────────────┐ +# │ Domain Parallelism Settings │ +# └───────────────────────────────────────────┘ +domain_parallelism: + domain_size: 1 + shard_grid: false + shard_points: false + +# ┌───────────────────────────────────────────┐ +# │ Model Parameters │ +# └───────────────────────────────────────────┘ +model: + model_type: combined # train which model? surface, volume, combined + activation: "relu" # "relu" or "gelu" + loss_function: + loss_type: "mse" # mse or rmse + area_weighing_factor: 10000 # Generally inverse of maximum area + interp_res: [128, 64, 64] # resolution of latent space 128, 64, 48 + use_sdf_in_basis_func: true # SDF in basis function network + positional_encoding: false # calculate positional encoding? + volume_points_sample: 8192 # Number of points to sample in volume per epoch + surface_points_sample: 8192 # Number of points to sample on surface per epoch + surface_sampling_algorithm: area_weighted # random or area_weighted + geom_points_sample: 300_000 # Number of points to sample on STL per epoch + num_neighbors_surface: 7 # How many neighbors on surface? + num_neighbors_volume: 10 # How many neighbors on volume? + combine_volume_surface: false # combine volume and surface encodings + return_volume_neighbors: True # Whether to return volume neighbors or not + use_surface_normals: true # Use surface normals and surface areas for surface computation? + use_surface_area: true # Use only surface normals and not surface area + integral_loss_scaling_factor: 100 # Scale integral loss by this factor + normalization: min_max_scaling # or mean_std_scaling + encode_parameters: false # encode inlet velocity and air density in the model + surf_loss_scaling: 1.0 # scale surface loss with this factor in combined mode + vol_loss_scaling: 1.0 # scale volume loss with this factor in combined mode + geometry_encoding_type: both # geometry encoder type, sdf, stl, both + solution_calculation_mode: two-loop # one-loop is better for sharded, two-loop is lower memory but more overhead. Physics losses are not supported via one-loop presently. + resampling_surface_mesh: # resampling of surface mesh before constructing kd tree + resample: false #false or true + points: 1_000_000 # number of points + geometry_rep: # Hyperparameters for geometry representation network + geo_conv: + base_neurons: 32 # 256 or 64 + base_neurons_in: 1 + base_neurons_out: 1 + volume_radii: [0.05, 0.25, 1.0, 2.5] # radii for volume + surface_radii: [0.01, 0.05, 0.1] # radii for surface + surface_hops: 1 # Number of surface iterations + volume_hops: 1 # Number of volume iterations + volume_neighbors_in_radius: [10, 10, 10, 10] # Number of neighbors in radius for volume + surface_neighbors_in_radius: [10, 10, 10] # Number of neighbors in radius for surface + fourier_features: false + num_modes: 5 + activation: ${model.activation} + geo_processor: + base_filters: 8 + activation: ${model.activation} + processor_type: conv # conv or unet + self_attention: false + cross_attention: false + nn_basis_functions: # Hyperparameters for basis function network + base_layer: 512 + fourier_features: true + num_modes: 5 + activation: ${model.activation} + local_point_conv: + activation: ${model.activation} + aggregation_model: # Hyperparameters for aggregation network + base_layer: 512 + activation: ${model.activation} + position_encoder: # Hyperparameters for position encoding network + base_neurons: 512 + activation: ${model.activation} + fourier_features: false + num_modes: 5 + geometry_local: # Hyperparameters for local geometry extraction + volume_neighbors_in_radius: [32, 128] # Number of radius points + surface_neighbors_in_radius: [32, 128] # Number of radius points + volume_radii: [0.05, 0.25] # Volume radii + surface_radii: [0.05, 0.25] # Surface radii + base_layer: 512 + parameter_model: + base_layer: 512 + fourier_features: false + num_modes: 5 + activation: ${model.activation} + +# ┌───────────────────────────────────────────┐ +# │ Training Configs │ +# └───────────────────────────────────────────┘ +train: # Training configurable parameters + epochs: 1000 + checkpoint_interval: 1 + dataloader: + batch_size: 1 + pin_memory: false # if the preprocessing is outputing GPU data, set this to false + sampler: + shuffle: true + drop_last: false + checkpoint_dir: /user/models/ # Use only for retraining + add_physics_loss: false + + +# ┌───────────────────────────────────────────┐ +# │ Validation Configs │ +# └───────────────────────────────────────────┘ +val: # Validation configurable parameters + dataloader: + batch_size: 1 + pin_memory: false # if the preprocessing is outputing GPU data, set this to false + sampler: + shuffle: true + drop_last: false + +# ┌───────────────────────────────────────────┐ +# │ Testing data Configs │ +# └───────────────────────────────────────────┘ +eval: # Testing configurable parameters + test_path: /user/data/drivaer_data_baseline_test # Dir for testing data in raw format (vtp, vtu ,stls) + save_path: /user/data/drivaer_data_baseline_test_predicted # Dir to save predicted results in raw format (vtp, vtu) + checkpoint_name: DoMINO.0.455.pt # Name of checkpoint to select from saved checkpoints + scaling_param_path: /user/modulus/physicsnemo/examples/cfd/external_aerodynamics/domino_nim_finetuning/outputs/AWS_Dataset_Baseline/ + refine_stl: False # Automatically refine STL during inference + stencil_size: 7 # Stencil size for evaluating surface and volume model diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/generate_base_predictions.py b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/generate_base_predictions.py new file mode 100644 index 0000000000..6369d0d461 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/generate_base_predictions.py @@ -0,0 +1,851 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code defines a distributed pipeline for running the +DoMINO-Automotive-Aero NIM model. The model predictions +are loaded in the the VTP/VTU files and saved in the +specified directory. The eval tab in config.yaml can be +used to specify the input and output directories. +""" + +import os, re +import time + +import hydra +from hydra.utils import to_absolute_path +from omegaconf import DictConfig, OmegaConf + +import numpy as np + +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable, List, Literal, Mapping, Optional, Union, Callable + +import pandas as pd +import pyvista as pv + +import torch +from torch.nn.parallel import DistributedDataParallel +from torch.utils.data import DataLoader, Dataset + +import vtk +from vtk.util import numpy_support + +from physicsnemo.distributed import DistributedManager +from physicsnemo.datapipes.cae.domino_datapipe import DoMINODataPipe +from model_base_predictor import DoMINO +from physicsnemo.models.domino.utils import * +from physicsnemo.nn.functional import signed_distance_field + +AIR_DENSITY = 1.205 +STREAM_VELOCITY = 38.89 + + +def loss_fn(output, target): + masked_loss = torch.mean(((output - target) ** 2.0), (0, 1, 2)) + loss = torch.mean(masked_loss) + return loss + + +def test_step(data_dict, model, device, cfg, vol_factors, surf_factors): + avg_tloss_vol = 0.0 + avg_tloss_surf = 0.0 + running_tloss_vol = 0.0 + running_tloss_surf = 0.0 + + if cfg.model.model_type == "volume" or cfg.model.model_type == "combined": + output_features_vol = True + else: + output_features_vol = None + + if cfg.model.model_type == "surface" or cfg.model.model_type == "combined": + output_features_surf = True + else: + output_features_surf = None + + with torch.no_grad(): + point_batch_size = 256000 + data_dict = dict_to_device(data_dict, device) + + # Non-dimensionalization factors + air_density = data_dict["air_density"] + stream_velocity = data_dict["stream_velocity"] + length_scale = data_dict["length_scale"] + + # STL nodes + geo_centers = data_dict["geometry_coordinates"] + + # Bounding box grid + s_grid = data_dict["surf_grid"] + sdf_surf_grid = data_dict["sdf_surf_grid"] + # Scaling factors + surf_max = data_dict["surface_min_max"][:, 1] + surf_min = data_dict["surface_min_max"][:, 0] + + if output_features_vol is not None: + # Represent geometry on computational grid + # Computational domain grid + p_grid = data_dict["grid"] + sdf_grid = data_dict["sdf_grid"] + # Scaling factors + vol_max = data_dict["volume_min_max"][:, 1] + vol_min = data_dict["volume_min_max"][:, 0] + + # Normalize based on computational domain + geo_centers_vol = 2.0 * (geo_centers - vol_min) / (vol_max - vol_min) - 1 + encoding_g_vol = model.geo_rep_volume(geo_centers_vol, p_grid, sdf_grid) + + if output_features_surf is not None: + # Represent geometry on bounding box + geo_centers_surf = ( + 2.0 * (geo_centers - surf_min) / (surf_max - surf_min) - 1 + ) + encoding_g_surf = model.geo_rep_surface( + geo_centers_surf, s_grid, sdf_surf_grid + ) + + if output_features_vol is not None: + # First calculate volume predictions if required + volume_mesh_centers = data_dict["volume_mesh_centers"] + target_vol = data_dict["volume_fields"] + # SDF on volume mesh nodes + sdf_nodes = data_dict["sdf_nodes"] + # Positional encoding based on closest point on surface to a volume node + pos_volume_closest = data_dict["pos_volume_closest"] + # Positional encoding based on center of mass of geometry to volume node + pos_volume_center_of_mass = data_dict["pos_volume_center_of_mass"] + p_grid = data_dict["grid"] + + prediction_vol = np.zeros_like(target_vol.cpu().numpy()) + num_points = volume_mesh_centers.shape[1] + subdomain_points = int(np.floor(num_points / point_batch_size)) + + start_time = time.time() + + for p in range(subdomain_points + 1): + start_idx = p * point_batch_size + end_idx = (p + 1) * point_batch_size + with torch.no_grad(): + target_batch = target_vol[:, start_idx:end_idx] + volume_mesh_centers_batch = volume_mesh_centers[ + :, start_idx:end_idx + ] + sdf_nodes_batch = sdf_nodes[:, start_idx:end_idx] + pos_volume_closest_batch = pos_volume_closest[:, start_idx:end_idx] + pos_normals_com_batch = pos_volume_center_of_mass[ + :, start_idx:end_idx + ] + geo_encoding_local = model.geo_encoding_local( + 0.5 * encoding_g_vol, + volume_mesh_centers_batch, + p_grid, + mode="volume", + ) + if cfg.model.use_sdf_in_basis_func: + pos_encoding = torch.cat( + ( + sdf_nodes_batch, + pos_volume_closest_batch, + pos_normals_com_batch, + ), + axis=-1, + ) + else: + pos_encoding = pos_normals_com_batch + pos_encoding = model.position_encoder( + pos_encoding, eval_mode="volume" + ) + tpredictions_batch = model.calculate_solution( + volume_mesh_centers_batch, + geo_encoding_local, + pos_encoding, + stream_velocity, + air_density, + num_sample_points=cfg.eval.stencil_size, + eval_mode="volume", + ) + tpredictions_batch = tpredictions_batch[ + :, :, [0, 1, 2, 3, 5] + ] # drop the second to last one + running_tloss_vol += loss_fn(tpredictions_batch, target_batch) + prediction_vol[:, start_idx:end_idx] = ( + tpredictions_batch.cpu().numpy() + ) + + prediction_vol = unnormalize(prediction_vol, vol_factors[0], vol_factors[1]) + + prediction_vol[:, :, :3] = ( + prediction_vol[:, :, :3] * stream_velocity[0, 0].cpu().numpy() + ) + prediction_vol[:, :, 3] = ( + prediction_vol[:, :, 3] + * stream_velocity[0, 0].cpu().numpy() ** 2.0 + * air_density[0, 0].cpu().numpy() + ) + prediction_vol[:, :, 4] = ( + prediction_vol[:, :, 4] + * stream_velocity[0, 0].cpu().numpy() + * length_scale[0].cpu().numpy() + ) + else: + prediction_vol = None + + if output_features_surf is not None: + # Next calculate surface predictions + # Sampled points on surface + surface_mesh_centers = data_dict["surface_mesh_centers"] + surface_normals = data_dict["surface_normals"] + surface_areas = data_dict["surface_areas"] + + # Neighbors of sampled points on surface + surface_mesh_neighbors = data_dict["surface_mesh_neighbors"] + surface_neighbors_normals = data_dict["surface_neighbors_normals"] + surface_neighbors_areas = data_dict["surface_neighbors_areas"] + surface_areas = torch.unsqueeze(surface_areas, -1) + surface_neighbors_areas = torch.unsqueeze(surface_neighbors_areas, -1) + pos_surface_center_of_mass = data_dict["pos_surface_center_of_mass"] + num_points = surface_mesh_centers.shape[1] + subdomain_points = int(np.floor(num_points / point_batch_size)) + + target_surf = data_dict["surface_fields"] + prediction_surf = np.zeros_like(target_surf.cpu().numpy()) + + start_time = time.time() + + for p in range(subdomain_points + 1): + start_idx = p * point_batch_size + end_idx = (p + 1) * point_batch_size + with torch.no_grad(): + target_batch = target_surf[:, start_idx:end_idx] + surface_mesh_centers_batch = surface_mesh_centers[ + :, start_idx:end_idx + ] + surface_mesh_neighbors_batch = surface_mesh_neighbors[ + :, start_idx:end_idx + ] + surface_normals_batch = surface_normals[:, start_idx:end_idx] + surface_neighbors_normals_batch = surface_neighbors_normals[ + :, start_idx:end_idx + ] + surface_areas_batch = surface_areas[:, start_idx:end_idx] + surface_neighbors_areas_batch = surface_neighbors_areas[ + :, start_idx:end_idx + ] + pos_surface_center_of_mass_batch = pos_surface_center_of_mass[ + :, start_idx:end_idx + ] + geo_encoding_local = model.geo_encoding_local( + 0.5 * encoding_g_surf, + surface_mesh_centers_batch, + s_grid, + mode="surface", + ) + pos_encoding = pos_surface_center_of_mass_batch + pos_encoding = model.position_encoder( + pos_encoding, eval_mode="surface" + ) + + if cfg.model.surface_neighbors: + tpredictions_batch = model.calculate_solution_with_neighbors( + surface_mesh_centers_batch, + geo_encoding_local, + pos_encoding, + surface_mesh_neighbors_batch, + surface_normals_batch, + surface_neighbors_normals_batch, + surface_areas_batch, + surface_neighbors_areas_batch, + stream_velocity, + air_density, + num_sample_points=cfg.model.num_surface_neighbors, + ) + else: + tpredictions_batch = model.calculate_solution( + surface_mesh_centers_batch, + geo_encoding_local, + pos_encoding, + stream_velocity, + air_density, + num_sample_points=1, + eval_mode="surface", + ) + running_tloss_surf += loss_fn(tpredictions_batch, target_batch) + prediction_surf[:, start_idx:end_idx] = ( + tpredictions_batch.cpu().numpy() + ) + + prediction_surf = ( + unnormalize(prediction_surf, surf_factors[0], surf_factors[1]) + * stream_velocity[0, 0].cpu().numpy() ** 2.0 + * air_density[0, 0].cpu().numpy() + ) + + else: + prediction_surf = None + + return prediction_vol, prediction_surf + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config_base_pred") +def main(cfg: DictConfig): + print(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + input_path = cfg.eval.test_path + + model_type = cfg.model.model_type + + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + if model_type == "volume" or model_type == "combined": + volume_variable_names = list(cfg.variables.volume.solution.keys()) + num_vol_vars = 0 + for j in volume_variable_names: + if cfg.variables.volume.solution[j] == "vector": + num_vol_vars += 3 + else: + num_vol_vars += 1 + else: + num_vol_vars = None + + if model_type == "surface" or model_type == "combined": + surface_variable_names = list(cfg.variables.surface.solution.keys()) + num_surf_vars = 0 + for j in surface_variable_names: + if cfg.variables.surface.solution[j] == "vector": + num_surf_vars += 3 + else: + num_surf_vars += 1 + else: + num_surf_vars = None + + vol_factors = np.array( + [ + [2.1508515, 1.0027921, 1.0663894, 1.1288369, 0.05063211], + [-1.9028450e00, -1.0032533e00, -1.0505041e00, -1.4412953e00, 1.5563720e-18], + ] + ) + surf_factors = np.array( + [ + [0.98881036, 0.00550783, 0.00854675, 0.00452144], + [-2.4203062, -0.00740275, -0.00848471, -0.00448634], + ] + ) + + print("Vol factors:", vol_factors) + print("Surf factors:", surf_factors) + + model = DoMINO( + input_features=3, + output_features_vol=num_vol_vars + 1, # +1 for nutmean + output_features_surf=num_surf_vars, + model_parameters=cfg.model, + ).to(dist.device) + + model = torch.compile(model, disable=True) + + checkpoint = torch.load( + to_absolute_path( + os.path.join(cfg.eval.checkpoint_dir, cfg.eval.checkpoint_name) + ), + map_location=dist.device, + ) + + model.load_state_dict(checkpoint) + + print("Model loaded") + + if dist.world_size > 1: + model = DistributedDataParallel( + model, + device_ids=[dist.local_rank], + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + gradient_as_bucket_view=True, + static_graph=True, + ) + model = model.module + dirnames = get_filenames(input_path) + + dirnames = get_filenames(input_path) + dev_id = torch.cuda.current_device() + num_files = int(len(dirnames) / dist.world_size) + dirnames_per_gpu = dirnames[int(num_files * dev_id) : int(num_files * (dev_id + 1))] + + pred_save_path = cfg.eval.save_path + create_directory(pred_save_path) + + # ... existing code ... + for count, dirname in enumerate(dirnames_per_gpu): + filepath = os.path.join(input_path, dirname) + tag = int(re.findall(r"(\w+?)(\d+)", dirname)[0][1]) + stl_path = os.path.join(filepath, f"drivaer_{tag}.stl") + vtp_path = os.path.join(filepath, f"boundary_{tag}.vtp") + vtu_path = os.path.join(filepath, f"volume_{tag}.vtu") + + vtp_pred_save_path = os.path.join(filepath, f"boundary_{tag}_predicted1.vtp") + vtu_pred_save_path = os.path.join(filepath, f"volume_{tag}_predicted1.vtu") + + # Skip if required input files are missing + missing = False + if not os.path.exists(stl_path): + print(f"Skipping {dirname}: missing {stl_path}") + missing = True + if (model_type in ["surface", "combined"]) and not os.path.exists(vtp_path): + print(f"Skipping {dirname}: missing {vtp_path}") + missing = True + if (model_type in ["volume", "combined"]) and not os.path.exists(vtu_path): + print(f"Skipping {dirname}: missing {vtu_path}") + missing = True + if missing: + continue + + # Skip if output files already exist + skip_surface = (model_type in ["surface", "combined"]) and os.path.exists( + vtp_pred_save_path + ) + skip_volume = (model_type in ["volume", "combined"]) and os.path.exists( + vtu_pred_save_path + ) + if skip_surface and skip_volume: + print(f"Skipping {dirname}: output files already exist.") + continue + + # Read STL + reader = pv.get_reader(stl_path) + mesh_stl = reader.read() + stl_vertices = mesh_stl.points + stl_faces = np.array(mesh_stl.faces).reshape((-1, 4))[ + :, 1: + ] # Assuming triangular elements + mesh_indices_flattened = stl_faces.flatten() + length_scale = np.amax(np.amax(stl_vertices, 0) - np.amin(stl_vertices, 0)) + stl_sizes = mesh_stl.compute_cell_sizes(length=False, area=True, volume=False) + stl_sizes = np.array(stl_sizes.cell_data["Area"], dtype=np.float32) + stl_centers = np.array(mesh_stl.cell_centers().points, dtype=np.float32) + + # Center of mass calculation + center_of_mass = calculate_center_of_mass(stl_centers, stl_sizes) + + if cfg.data.bounding_box_surface is None: + s_max = np.amax(stl_vertices, 0) + s_min = np.amin(stl_vertices, 0) + else: + bounding_box_dims_surf = [] + bounding_box_dims_surf.append(np.asarray(cfg.data.bounding_box_surface.max)) + bounding_box_dims_surf.append(np.asarray(cfg.data.bounding_box_surface.min)) + s_max = np.float32(bounding_box_dims_surf[0]) + s_min = np.float32(bounding_box_dims_surf[1]) + + nx, ny, nz = cfg.model.interp_res + + surf_grid = create_grid(s_max, s_min, [nx, ny, nz]) + surf_grid_reshaped = surf_grid.reshape(nx * ny * nz, 3) + + # SDF calculation on the grid using WARP + sdf_surf_grid = signed_distance_field( + stl_vertices, + mesh_indices_flattened, + surf_grid_reshaped, + use_sign_winding_number=True, + ).reshape(nx, ny, nz) + surf_grid = np.float32(surf_grid) + sdf_surf_grid = np.float32(sdf_surf_grid) + surf_grid_max_min = np.float32(np.asarray([s_min, s_max])) + + # Read VTP + if model_type == "surface" or model_type == "combined": + # Use pyvista directly for Geko dataset + reader = vtk.vtkXMLPolyDataReader() + reader.SetFileName(vtp_path) + reader.Update() + polydata_surf = reader.GetOutput() + + celldata_all = get_node_to_elem(polydata_surf) + + celldata = celldata_all.GetCellData() + surface_fields = get_fields(celldata, surface_variable_names) + surface_fields = np.concatenate(surface_fields, axis=-1) + + mesh = pv.PolyData(polydata_surf) + surface_coordinates = np.array(mesh.cell_centers().points, dtype=np.float32) + + surface_normals = np.array(mesh.cell_normals, dtype=np.float32) + surface_sizes = mesh.compute_cell_sizes( + length=False, area=True, volume=False + ) + surface_sizes = np.array(surface_sizes.cell_data["Area"], dtype=np.float32) + + # Normalize cell normals + surface_normals = ( + surface_normals / np.linalg.norm(surface_normals, axis=1)[:, np.newaxis] + ) + + if cfg.model.num_surface_neighbors > 1: + interp_func = KDTree(surface_coordinates) + dd, ii = interp_func.query( + surface_coordinates, k=cfg.model.num_surface_neighbors + ) + + surface_neighbors = surface_coordinates[ii] + surface_neighbors = surface_neighbors[:, 1:] + + surface_neighbors_normals = surface_normals[ii] + surface_neighbors_normals = surface_neighbors_normals[:, 1:] + surface_neighbors_sizes = surface_sizes[ii] + surface_neighbors_sizes = surface_neighbors_sizes[:, 1:] + else: + surface_neighbors = surface_coordinates + surface_neighbors_normals = surface_normals + surface_neighbors_sizes = surface_sizes + + dx, dy, dz = ( + (s_max[0] - s_min[0]) / nx, + (s_max[1] - s_min[1]) / ny, + (s_max[2] - s_min[2]) / nz, + ) + + if cfg.model.positional_encoding: + pos_surface_center_of_mass = calculate_normal_positional_encoding( + surface_coordinates, center_of_mass, cell_length=[dx, dy, dz] + ) + else: + pos_surface_center_of_mass = surface_coordinates - center_of_mass + + surface_coordinates = normalize(surface_coordinates, s_max, s_min) + surface_neighbors = normalize(surface_neighbors, s_max, s_min) + surf_grid = normalize(surf_grid, s_max, s_min) + + else: + surface_coordinates = None + surface_fields = None + surface_sizes = None + surface_normals = None + surface_neighbors = None + surface_neighbors_normals = None + surface_neighbors_sizes = None + pos_surface_center_of_mass = None + + # Read VTU + if model_type == "volume" or model_type == "combined": + reader = vtk.vtkXMLUnstructuredGridReader() + reader.SetFileName(vtu_path) + reader.Update() + polydata_vol = reader.GetOutput() + volume_coordinates, volume_fields = get_volume_data( + polydata_vol, volume_variable_names + ) + volume_fields = np.concatenate(volume_fields, axis=-1) + # print(f"Processed vtu {vtu_path}") + + bounding_box_dims = [] + bounding_box_dims.append(np.asarray(cfg.data.bounding_box.max)) + bounding_box_dims.append(np.asarray(cfg.data.bounding_box.min)) + + v_max = np.amax(volume_coordinates, 0) + v_min = np.amin(volume_coordinates, 0) + if bounding_box_dims is None: + c_max = s_max + (s_max - s_min) / 2 + c_min = s_min - (s_max - s_min) / 2 + c_min[2] = s_min[2] + else: + c_max = np.float32(bounding_box_dims[0]) + c_min = np.float32(bounding_box_dims[1]) + + dx, dy, dz = ( + (c_max[0] - c_min[0]) / nx, + (c_max[1] - c_min[1]) / ny, + (c_max[2] - c_min[2]) / nz, + ) + # Generate a grid of specified resolution to map the bounding box + # The grid is used for capturing structured geometry features and SDF representation of geometry + grid = create_grid(c_max, c_min, [nx, ny, nz]) + grid_reshaped = grid.reshape(nx * ny * nz, 3) + + # SDF calculation on the grid using WARP + sdf_grid = signed_distance_field( + stl_vertices, + mesh_indices_flattened, + grid_reshaped, + use_sign_winding_number=True, + ).reshape(nx, ny, nz) + + # SDF calculation + sdf_nodes, sdf_node_closest_point = signed_distance_field( + stl_vertices, + mesh_indices_flattened, + volume_coordinates, + include_hit_points=True, + use_sign_winding_number=True, + ) + sdf_nodes = sdf_nodes.reshape(-1, 1) + + if cfg.model.positional_encoding: + pos_volume_closest = calculate_normal_positional_encoding( + volume_coordinates, sdf_node_closest_point, cell_length=[dx, dy, dz] + ) + pos_volume_center_of_mass = calculate_normal_positional_encoding( + volume_coordinates, center_of_mass, cell_length=[dx, dy, dz] + ) + else: + pos_volume_closest = volume_coordinates - sdf_node_closest_point + pos_volume_center_of_mass = volume_coordinates - center_of_mass + + volume_coordinates = normalize(volume_coordinates, c_max, c_min) + grid = normalize(grid, c_max, c_min) + vol_grid_max_min = np.asarray([c_min, c_max]) + + else: + volume_coordinates = None + volume_fields = None + pos_volume_closest = None + pos_volume_center_of_mass = None + + # print(f"Processed sdf and normalized") + + geom_centers = np.float32(stl_vertices) + + if model_type == "combined": + # Add the parameters to the dictionary + data_dict = { + "pos_volume_closest": pos_volume_closest, + "pos_volume_center_of_mass": pos_volume_center_of_mass, + "pos_surface_center_of_mass": pos_surface_center_of_mass, + "geometry_coordinates": geom_centers, + "grid": grid, + "surf_grid": surf_grid, + "sdf_grid": sdf_grid, + "sdf_surf_grid": sdf_surf_grid, + "sdf_nodes": sdf_nodes, + "surface_mesh_centers": surface_coordinates, + "surface_mesh_neighbors": surface_neighbors, + "surface_normals": surface_normals, + "surface_neighbors_normals": surface_neighbors_normals, + "surface_areas": surface_sizes, + "surface_neighbors_areas": surface_neighbors_sizes, + "volume_fields": volume_fields, + "volume_mesh_centers": volume_coordinates, + "surface_fields": surface_fields, + "volume_min_max": vol_grid_max_min, + "surface_min_max": surf_grid_max_min, + "length_scale": np.array(length_scale, dtype=np.float32), + "stream_velocity": np.expand_dims( + np.array(STREAM_VELOCITY, dtype=np.float32), axis=-1 + ), + "air_density": np.expand_dims( + np.array(AIR_DENSITY, dtype=np.float32), axis=-1 + ), + } + elif model_type == "surface": + data_dict = { + "pos_surface_center_of_mass": np.float32(pos_surface_center_of_mass), + "geometry_coordinates": np.float32(geom_centers), + "surf_grid": np.float32(surf_grid), + "sdf_surf_grid": np.float32(sdf_surf_grid), + "surface_mesh_centers": np.float32(surface_coordinates), + "surface_mesh_neighbors": np.float32(surface_neighbors), + "surface_normals": np.float32(surface_normals), + "surface_neighbors_normals": np.float32(surface_neighbors_normals), + "surface_areas": np.float32(surface_sizes), + "surface_neighbors_areas": np.float32(surface_neighbors_sizes), + "surface_fields": np.float32(surface_fields), + "surface_min_max": np.float32(surf_grid_max_min), + "length_scale": np.array(length_scale, dtype=np.float32), + "stream_velocity": np.expand_dims( + np.array(STREAM_VELOCITY, dtype=np.float32), axis=-1 + ), + "air_density": np.expand_dims( + np.array(AIR_DENSITY, dtype=np.float32), axis=-1 + ), + } + elif model_type == "volume": + data_dict = { + "pos_volume_closest": pos_volume_closest, + "pos_volume_center_of_mass": pos_volume_center_of_mass, + "geometry_coordinates": geom_centers, + "grid": grid, + "surf_grid": surf_grid, + "sdf_grid": sdf_grid, + "sdf_surf_grid": sdf_surf_grid, + "sdf_nodes": sdf_nodes, + "volume_fields": volume_fields, + "volume_mesh_centers": volume_coordinates, + "volume_min_max": vol_grid_max_min, + "surface_min_max": surf_grid_max_min, + "length_scale": np.array(length_scale, dtype=np.float32), + "stream_velocity": np.expand_dims( + np.array(STREAM_VELOCITY, dtype=np.float32), axis=-1 + ), + "air_density": np.expand_dims( + np.array(AIR_DENSITY, dtype=np.float32), axis=-1 + ), + } + + data_dict = { + key: torch.from_numpy(np.expand_dims(np.float32(value), 0)) + for key, value in data_dict.items() + } + + prediction_vol, prediction_surf = test_step( + data_dict, model, dist.device, cfg, vol_factors, surf_factors + ) + + if prediction_surf is not None: + surface_sizes = np.expand_dims(surface_sizes, -1) + + pres_x_pred = np.sum( + prediction_surf[0, :, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + ) + shear_x_pred = np.sum(prediction_surf[0, :, 1] * surface_sizes[:, 0]) + + pres_x_true = np.sum( + surface_fields[:, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + ) + shear_x_true = np.sum(surface_fields[:, 1] * surface_sizes[:, 0]) + + force_x_pred = np.sum( + prediction_surf[0, :, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + - prediction_surf[0, :, 1] * surface_sizes[:, 0] + ) + force_x_true = np.sum( + surface_fields[:, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + - surface_fields[:, 1] * surface_sizes[:, 0] + ) + + force_y_pred = np.sum( + prediction_surf[0, :, 0] * surface_normals[:, 1] * surface_sizes[:, 0] + - prediction_surf[0, :, 2] * surface_sizes[:, 0] + ) + force_y_true = np.sum( + surface_fields[:, 0] * surface_normals[:, 1] * surface_sizes[:, 0] + - surface_fields[:, 2] * surface_sizes[:, 0] + ) + + force_z_pred = np.sum( + prediction_surf[0, :, 0] * surface_normals[:, 2] * surface_sizes[:, 0] + - prediction_surf[0, :, 3] * surface_sizes[:, 0] + ) + force_z_true = np.sum( + surface_fields[:, 0] * surface_normals[:, 2] * surface_sizes[:, 0] + - surface_fields[:, 3] * surface_sizes[:, 0] + ) + print("Drag=", dirname, force_x_pred, force_x_true) + print("Lift=", dirname, force_z_pred, force_z_true) + print("Side=", dirname, force_y_pred, force_y_true) + + l2_gt = np.sum(np.square(surface_fields), (0)) + l2_error = np.sum(np.square(prediction_surf[0] - surface_fields), (0)) + + print( + "Surface L-2 norm:", + dirname, + np.sqrt(l2_error) / np.sqrt(l2_gt), + ) + + if prediction_vol is not None: + target_vol = volume_fields + prediction_vol = prediction_vol[0] + c_min = vol_grid_max_min[0] + c_max = vol_grid_max_min[1] + volume_coordinates = unnormalize(volume_coordinates, c_max, c_min) + ids_in_bbox = np.where( + (volume_coordinates[:, 0] < c_min[0]) + | (volume_coordinates[:, 0] > c_max[0]) + | (volume_coordinates[:, 1] < c_min[1]) + | (volume_coordinates[:, 1] > c_max[1]) + | (volume_coordinates[:, 2] < c_min[2]) + | (volume_coordinates[:, 2] > c_max[2]) + ) + target_vol[ids_in_bbox] = 0.0 + prediction_vol[ids_in_bbox] = 0.0 + l2_gt = np.sum(np.square(target_vol), (0)) + l2_error = np.sum(np.square(prediction_vol - target_vol), (0)) + print( + "Volume L-2 norm:", + dirname, + np.sqrt(l2_error) / np.sqrt(l2_gt), + ) + + if prediction_surf is not None: + surfParam_vtk = numpy_support.numpy_to_vtk(prediction_surf[0, :, 0:1]) + surfParam_vtk.SetName(f"{surface_variable_names[0]}BasePred") + celldata_all.GetCellData().AddArray(surfParam_vtk) + + surfParam_vtk = numpy_support.numpy_to_vtk(prediction_surf[0, :, 1:]) + surfParam_vtk.SetName(f"{surface_variable_names[1]}BasePred") + celldata_all.GetCellData().AddArray(surfParam_vtk) + + surfParam_vtk = numpy_support.numpy_to_vtk( + surface_fields[:, 0:1] - prediction_surf[0, :, 0:1] + ) + surfParam_vtk.SetName(f"{surface_variable_names[0]}Delta") + celldata_all.GetCellData().AddArray(surfParam_vtk) + + surfParam_vtk = numpy_support.numpy_to_vtk( + surface_fields[:, 1:] - prediction_surf[0, :, 1:] + ) + surfParam_vtk.SetName(f"{surface_variable_names[1]}Delta") + celldata_all.GetCellData().AddArray(surfParam_vtk) + + write_to_vtp(celldata_all, vtp_pred_save_path) + + if prediction_vol is not None: + try: + volParam_vtk = numpy_support.numpy_to_vtk(prediction_vol[:, 0:3]) + volParam_vtk.SetName(f"{volume_variable_names[0]}BasePred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk(prediction_vol[:, 3:4]) + volParam_vtk.SetName(f"{volume_variable_names[1]}BasePred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk(prediction_vol[:, 4:5]) + volParam_vtk.SetName(f"{volume_variable_names[2]}BasePred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk( + volume_fields[:, 0:3] - prediction_vol[:, 0:3] + ) + volParam_vtk.SetName(f"{volume_variable_names[0]}Delta") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk( + volume_fields[:, 3:4] - prediction_vol[:, 3:4] + ) + volParam_vtk.SetName(f"{volume_variable_names[1]}Delta") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk( + volume_fields[:, 4:5] - prediction_vol[:, 4:5] + ) + volParam_vtk.SetName(f"{volume_variable_names[2]}Delta") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + write_to_vtu(polydata_vol, vtu_pred_save_path) + + except Exception as e: + print("Error occurred:", str(e)) + print("Error type:", type(e)) + import traceback + + traceback.print_exc() + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/model_base_predictor.py b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/model_base_predictor.py new file mode 100644 index 0000000000..189b4dec4c --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/model_base_predictor.py @@ -0,0 +1,1566 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code contains the DoMINO model architecture. +The DoMINO class contains an architecture to model both surface and +volume quantities together as well as separately (controlled using +the config.yaml file). This is the model architecture for the NIM model. +""" + +import math +from typing import Literal + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from physicsnemo.nn import BQWarp +from physicsnemo.utils.profiling import profile + + +def fourier_encode(coords, num_freqs): + """Function to caluculate fourier features""" + # Create a range of frequencies + freqs = torch.exp(torch.linspace(0, math.pi, num_freqs, device=coords.device)) + # Generate sine and cosine features + features = [torch.sin(coords * f) for f in freqs] + [ + torch.cos(coords * f) for f in freqs + ] + ret = torch.cat(features, dim=-1) + return ret + + +def fourier_encode_vectorized(coords, freqs): + """Vectorized Fourier feature encoding""" + D = coords.shape[-1] + F = freqs.shape[0] + + # freqs = torch.exp(torch.linspace(0, math.pi, num_freqs, device=coords.device)) # [F] + freqs = freqs[None, None, :, None] # reshape to [*, F, 1] for broadcasting + + coords = coords.unsqueeze(-2) # [*, 1, D] + scaled = (coords * freqs).reshape(*coords.shape[:-2], D * F) # [*, D, F] + features = torch.cat([torch.sin(scaled), torch.cos(scaled)], dim=-1) # [*, D, 2F] + + return features.reshape(*coords.shape[:-2], D * 2 * F) # [*, D * 2F] + + +def calculate_pos_encoding(nx, d=8): + """Function to caluculate positional encoding""" + vec = [] + for k in range(int(d / 2)): + vec.append(torch.sin(nx / 10000 ** (2 * (k) / d))) + vec.append(torch.cos(nx / 10000 ** (2 * (k) / d))) + return vec + + +def scale_sdf(sdf: torch.Tensor) -> torch.Tensor: + """ + Scale a signed distance function (SDF) to emphasize surface regions. + + This function applies a non-linear scaling to the SDF values that compresses + the range while preserving the sign, effectively giving more weight to points + near surfaces where |SDF| is small. + + Args: + sdf: Tensor containing signed distance function values + + Returns: + Tensor with scaled SDF values in range [-1, 1] + """ + return sdf / (0.4 + torch.abs(sdf)) + + +class BQWarp(nn.Module): + """ + Warp-based ball-query layer for finding neighboring points within a specified radius. + + This layer uses an accelerated ball query implementation to efficiently find points + within a specified radius of query points. + """ + + def __init__( + self, + grid_resolution=None, + radius: float = 0.25, + neighbors_in_radius: int = 10, + ): + """ + Initialize the BQWarp layer. + + Args: + grid_resolution: Resolution of the grid in each dimension [nx, ny, nz] + radius: Radius for ball query operation + neighbors_in_radius: Maximum number of neighbors to return within radius + """ + super().__init__() + if grid_resolution is None: + grid_resolution = [256, 96, 64] + self.ball_query_layer = BallQueryLayer(neighbors_in_radius, radius) + self.grid_resolution = grid_resolution + + def forward( + self, x: torch.Tensor, p_grid: torch.Tensor, reverse_mapping: bool = True + ) -> tuple[torch.Tensor, torch.Tensor]: + """ + Performs ball query operation to find neighboring points and their features. + + This method uses the Warp-accelerated ball query implementation to find points + within a specified radius. It can operate in two modes: + - Forward mapping: Find points from x that are near p_grid points (reverse_mapping=False) + - Reverse mapping: Find points from p_grid that are near x points (reverse_mapping=True) + + Args: + x: Tensor of shape (batch_size, num_points, 3+features) containing point coordinates + and their features + p_grid: Tensor of shape (batch_size, grid_x, grid_y, grid_z, 3) containing grid point + coordinates + reverse_mapping: Boolean flag to control the direction of the mapping: + - True: Find p_grid points near x points + - False: Find x points near p_grid points + + Returns: + tuple containing: + - mapping: Tensor containing indices of neighboring points + - outputs: Tensor containing coordinates of the neighboring points + """ + batch_size = x.shape[0] + nx, ny, nz = self.grid_resolution + + p_grid = torch.reshape(p_grid, (batch_size, nx * ny * nz, 3)) + + if reverse_mapping: + mapping, num_neighbors, outputs = self.ball_query_layer( + p_grid, + x, + ) + else: + mapping, num_neighbors, outputs = self.ball_query_layer( + x, + p_grid, + ) + + return mapping, outputs + + +class GeoConvOut(nn.Module): + """ + Geometry layer to project STL geometry data onto regular grids. + """ + + def __init__( + self, + input_features: int, + model_parameters, + grid_resolution=None, + ): + """ + Initialize the GeoConvOut layer. + + Args: + input_features: Number of input feature dimensions + model_parameters: Configuration parameters for the model + grid_resolution: Resolution of the output grid [nx, ny, nz] + """ + super().__init__() + if grid_resolution is None: + grid_resolution = [256, 96, 64] + base_neurons = model_parameters.base_neurons + + self.fc1 = nn.Linear(input_features, base_neurons) + self.fc2 = nn.Linear(base_neurons, int(base_neurons / 2)) + self.fc3 = nn.Linear(int(base_neurons / 2), model_parameters.base_neurons_out) + + self.grid_resolution = grid_resolution + + self.activation = F.relu + + def forward( + self, x: torch.Tensor, radius: float = 0.025, neighbors_in_radius: int = 10 + ) -> torch.Tensor: + """ + Process and project geometric features onto a 3D grid. + """ + batch_size = x.shape[0] + nx, ny, nz = ( + self.grid_resolution[0], + self.grid_resolution[1], + self.grid_resolution[2], + ) + + mask = abs(x - 0) > 1e-6 + x = self.activation(self.fc1(x)) + x = self.activation(self.fc2(x)) + x = F.tanh(self.fc3(x)) + mask = mask[:, :, :, 0:1].expand( + mask.shape[0], mask.shape[1], mask.shape[2], x.shape[-1] + ) + + x = torch.sum(x * mask, 2) + + x = torch.reshape(x, (batch_size, x.shape[-1], nx, ny, nz)) + return x + + +class GeoProcessor(nn.Module): + """Geometry processing layer using CNNs""" + + def __init__(self, input_filters: int, model_parameters): + """ + Initialize the GeoProcessor network. + + Args: + input_filters: Number of input channels + model_parameters: Configuration parameters for the model + """ + super().__init__() + base_filters = model_parameters.base_filters + self.conv1 = nn.Conv3d( + input_filters, base_filters, kernel_size=3, padding="same" + ) + self.conv2 = nn.Conv3d( + base_filters, 2 * base_filters, kernel_size=3, padding="same" + ) + self.conv3 = nn.Conv3d( + 2 * base_filters, 4 * base_filters, kernel_size=3, padding="same" + ) + self.conv3_1 = nn.Conv3d( + 4 * base_filters, 4 * base_filters, kernel_size=3, padding="same" + ) + self.conv4 = nn.Conv3d( + 4 * base_filters, 2 * base_filters, kernel_size=3, padding="same" + ) + self.conv5 = nn.Conv3d( + 4 * base_filters, base_filters, kernel_size=3, padding="same" + ) + self.conv6 = nn.Conv3d( + 2 * base_filters, input_filters, kernel_size=3, padding="same" + ) + self.conv7 = nn.Conv3d( + 2 * input_filters, input_filters, kernel_size=3, padding="same" + ) + self.conv8 = nn.Conv3d(input_filters, 1, kernel_size=3, padding="same") + self.avg_pool = torch.nn.AvgPool3d((2, 2, 2)) + self.max_pool = nn.MaxPool3d(2) + self.upsample = nn.Upsample(scale_factor=2, mode="nearest") + self.activation = F.relu + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Process geometry information through the 3D CNN network. + + The network follows an encoder-decoder architecture with skip connections: + 1. Downsampling path (encoder) with three levels of max pooling + 2. Processing loop in the bottleneck + 3. Upsampling path (decoder) with skip connections from the encoder + + Args: + x: Input tensor containing grid-represented geometry of shape + (batch_size, input_filters, nx, ny, nz) + + Returns: + Processed geometry features of shape (batch_size, 1, nx, ny, nz) + """ + # Encoder + x0 = x + x = self.conv1(x) + x = self.activation(x) + x = self.max_pool(x) + + x1 = x + x = self.conv2(x) + x = self.activation(x) + x = self.max_pool(x) + + x2 = x + x = self.conv3(x) + x = self.activation(x) + x = self.max_pool(x) + + # Processor loop + x = F.relu(self.conv3_1(x)) + + # Decoder + x = self.conv4(x) + x = self.activation(x) + x = self.upsample(x) + x = torch.cat((x, x2), axis=1) + + x = self.conv5(x) + x = self.activation(x) + x = self.upsample(x) + x = torch.cat((x, x1), axis=1) + + x = self.conv6(x) + x = self.activation(x) + x = self.upsample(x) + x = torch.cat((x, x0), axis=1) + + x = self.activation(self.conv7(x)) + x = self.conv8(x) + + return x + + +class GeometryRep(nn.Module): + """ + Geometry representation module that processes STL geometry data. + + This module constructs a multiscale representation of geometry by: + 1. Computing short-range geometry encoding for local features + 2. Computing long-range geometry encoding for global context + 3. Processing signed distance field (SDF) data for surface information + + The combined encoding enables the model to reason about both local and global + geometric properties. + """ + + def __init__(self, input_features: int, radii, model_parameters=None): + """ + Initialize the GeometryRep module. + + Args: + input_features: Number of input feature dimensions + model_parameters: Configuration parameters for the model + """ + super().__init__() + geometry_rep = model_parameters.geometry_rep + self.geo_encoding_type = model_parameters.geometry_encoding_type + + self.bq_warp = nn.ModuleList() + self.geo_processors = nn.ModuleList() + for j, p in enumerate(radii): + self.bq_warp.append( + BQWarp( + grid_resolution=model_parameters.interp_res, + radius=radii[j], + ) + ) + self.geo_processors.append( + GeoProcessor( + input_filters=geometry_rep.geo_conv.base_neurons_out, + model_parameters=geometry_rep.geo_processor, + ) + ) + + self.geo_conv_out = nn.ModuleList() + for j, p in enumerate(radii): + self.geo_conv_out.append( + GeoConvOut( + input_features=input_features, + model_parameters=geometry_rep.geo_conv, + grid_resolution=model_parameters.interp_res, + ) + ) + + self.geo_processor_sdf = GeoProcessor( + input_filters=6, model_parameters=geometry_rep.geo_processor + ) + self.activation = F.relu + self.radii = radii + self.hops = geometry_rep.geo_conv.hops + + def forward( + self, x: torch.Tensor, p_grid: torch.Tensor, sdf: torch.Tensor + ) -> torch.Tensor: + """ + Process geometry data to create a comprehensive representation. + + This method combines short-range, long-range, and SDF-based geometry + encodings to create a rich representation of the geometry. + + Args: + x: Input tensor containing geometric point data + p_grid: Grid points for sampling + sdf: Signed distance field tensor + + Returns: + Comprehensive geometry encoding that concatenates short-range, + SDF-based, and long-range features + """ + if self.geo_encoding_type == "both" or self.geo_encoding_type == "stl": + # Calculate multi-scale geoemtry dependency + x_encoding = [] + for j, p in enumerate(self.radii): + mapping, k_short = self.bq_warp[j](x, p_grid) + x_encoding_inter = self.geo_conv_out[j](k_short) + # Propagate information in the geometry enclosed BBox + for _ in range(self.hops): + dx = self.geo_processors[j](x_encoding_inter) / self.hops + x_encoding_inter = x_encoding_inter + dx + x_encoding.append(x_encoding_inter) + x_encoding = torch.cat(x_encoding, axis=1) + + if self.geo_encoding_type == "both" or self.geo_encoding_type == "sdf": + # Expand SDF + sdf = torch.unsqueeze(sdf, 1) + # Scaled sdf to emphasize near surface + scaled_sdf = scale_sdf(sdf) + # Binary sdf + binary_sdf = torch.where(sdf >= 0, 0.0, 1.0) + # Gradients of SDF + sdf_x, sdf_y, sdf_z = torch.gradient(sdf, dim=[2, 3, 4]) + + # Process SDF and its computed features + sdf = torch.cat((sdf, scaled_sdf, binary_sdf, sdf_x, sdf_y, sdf_z), 1) + sdf_encoding = self.geo_processor_sdf(sdf) + + if self.geo_encoding_type == "both": + # Geometry encoding comprised of short-range, long-range and SDF features + encoding_g = torch.cat((x_encoding, sdf_encoding), 1) + elif self.geo_encoding_type == "sdf": + encoding_g = sdf_encoding + elif self.geo_encoding_type == "stl": + encoding_g = x_encoding + + return encoding_g + + +class NNBasisFunctions(nn.Module): + """Basis function layer for point clouds""" + + def __init__(self, input_features: int, model_parameters=None): + super(NNBasisFunctions, self).__init__() + base_layer = model_parameters.base_layer + self.fourier_features = model_parameters.fourier_features + self.num_modes = model_parameters.num_modes + + if self.fourier_features: + input_features_calculated = ( + input_features + input_features * self.num_modes * 2 + ) + else: + input_features_calculated = input_features + + self.fc1 = nn.Linear(input_features_calculated, base_layer) + self.fc2 = nn.Linear(base_layer, int(base_layer)) + self.fc3 = nn.Linear(int(base_layer), int(base_layer)) + self.bn1 = nn.BatchNorm1d(base_layer) + self.bn2 = nn.BatchNorm1d(int(base_layer)) + self.bn3 = nn.BatchNorm1d(int(base_layer)) + + self.activation = F.relu + + if self.fourier_features: + self.register_buffer( + "freqs", torch.exp(torch.linspace(0, math.pi, self.num_modes)) + ) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Transform point features into a basis function representation. + + Args: + x: Input tensor containing point features + + Returns: + Tensor containing basis function coefficients + """ + if self.fourier_features: + facets = torch.cat((x, fourier_encode_vectorized(x, self.freqs)), axis=-1) + else: + facets = x + facets = self.activation(self.fc1(facets)) + facets = self.activation(self.fc2(facets)) + facets = self.fc3(facets) + + return facets + + +class ParameterModel(nn.Module): + """ + Neural network module to encode simulation parameters. + + This module encodes physical parameters such as inlet velocity and air density + into a learned latent representation that can be incorporated into the model's + prediction process. + """ + + def __init__(self, input_features: int, model_parameters=None): + """ + Initialize the parameter encoding network. + + Args: + input_features: Number of input parameters to encode + model_parameters: Configuration parameters for the model + """ + super(ParameterModel, self).__init__() + self.fourier_features = model_parameters.fourier_features + self.num_modes = model_parameters.num_modes + + if self.fourier_features: + input_features_calculated = ( + input_features + input_features * self.num_modes * 2 + ) + self.register_buffer( + "freqs", torch.exp(torch.linspace(0, math.pi, self.num_modes)) + ) + else: + input_features_calculated = input_features + + base_layer = model_parameters.base_layer + self.fc1 = nn.Linear(input_features_calculated, base_layer) + self.fc2 = nn.Linear(base_layer, int(base_layer)) + self.fc3 = nn.Linear(int(base_layer), int(base_layer)) + self.bn1 = nn.BatchNorm1d(base_layer) + self.bn2 = nn.BatchNorm1d(int(base_layer)) + self.bn3 = nn.BatchNorm1d(int(base_layer)) + + self.activation = F.relu + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Encode physical parameters into a latent representation. + + Args: + x: Input tensor containing physical parameters (e.g., inlet velocity, air density) + + Returns: + Tensor containing encoded parameter representation + """ + if self.fourier_features: + params = torch.cat((x, fourier_encode_vectorized(x, self.freqs)), axis=-1) + else: + params = x + params = self.activation(self.fc1(params)) + params = self.activation(self.fc2(params)) + params = self.fc3(params) + + return params + + +class AggregationModel(nn.Module): + """ + Neural network module to aggregate local geometry encoding with basis functions. + + This module combines basis function representations with geometry encodings + to predict the final output quantities. It serves as the final prediction layer + that integrates all available information sources. + """ + + def __init__( + self, + input_features: int, + output_features: int, + model_parameters=None, + new_change: bool = True, + ): + """ + Initialize the aggregation model. + + Args: + input_features: Number of input feature dimensions + output_features: Number of output feature dimensions + model_parameters: Configuration parameters for the model + new_change: Flag to enable newer implementation (default: True) + """ + super(AggregationModel, self).__init__() + self.input_features = input_features + self.output_features = output_features + self.new_change = new_change + base_layer = model_parameters.base_layer + self.fc1 = nn.Linear(self.input_features, base_layer) + self.fc2 = nn.Linear(base_layer, int(base_layer)) + self.fc3 = nn.Linear(int(base_layer), int(base_layer)) + self.fc4 = nn.Linear(int(base_layer), int(base_layer)) + self.fc5 = nn.Linear(int(base_layer), self.output_features) + self.bn1 = nn.BatchNorm1d(base_layer) + self.bn2 = nn.BatchNorm1d(int(base_layer)) + self.bn3 = nn.BatchNorm1d(int(base_layer)) + self.bn4 = nn.BatchNorm1d(int(base_layer)) + + self.activation = F.relu + + def forward(self, x: torch.Tensor) -> torch.Tensor: + """ + Process the combined input features to predict output quantities. + + This method applies a series of fully connected layers to the input, + which typically contains a combination of basis functions, geometry + encodings, and potentially parameter encodings. + + Args: + x: Input tensor containing combined features + + Returns: + Tensor containing predicted output quantities + """ + out = self.activation(self.fc1(x)) + out = self.activation(self.fc2(out)) + out = self.activation(self.fc3(out)) + out = self.activation(self.fc4(out)) + + out = self.fc5(out) + + return out + + +class LocalPointConv(nn.Module): + """Layer for local geometry point kernel""" + + def __init__( + self, + input_features, + base_layer, + output_features, + model_parameters=None, + new_change=True, + ): + super(LocalPointConv, self).__init__() + self.input_features = input_features + self.output_features = output_features + self.fc1 = nn.Linear(self.input_features, base_layer) + self.fc2 = nn.Linear(base_layer, self.output_features) + self.activation = F.relu + + def forward(self, x): + out = self.activation(self.fc1(x)) + out = self.fc2(out) + + return out + + +# @dataclass +# class MetaData(ModelMetaData): +# name: str = "DoMINO" +# # Optimization +# jit: bool = False +# cuda_graphs: bool = True +# amp: bool = True +# # Inference +# onnx_cpu: bool = True +# onnx_gpu: bool = True +# onnx_runtime: bool = True +# # Physics informed +# var_dim: int = 1 +# func_torch: bool = False +# auto_grad: bool = False + + +class DoMINO(nn.Module): + """ + DoMINO model architecture for predicting both surface and volume quantities. + + The DoMINO (Deep Operational Modal Identification and Nonlinear Optimization) model + is designed to model both surface and volume physical quantities in aerodynamic + simulations. It can operate in three modes: + 1. Surface-only: Predicting only surface quantities + 2. Volume-only: Predicting only volume quantities + 3. Combined: Predicting both surface and volume quantities + + The model uses a combination of: + - Geometry representation modules + - Neural network basis functions + - Parameter encoding + - Local and global geometry processing + - Aggregation models for final prediction + + Parameters + ---------- + input_features : int + Number of point input features + output_features_vol : int, optional + Number of output features in volume + output_features_surf : int, optional + Number of output features on surface + model_parameters + Model parameters controlled by config.yaml + + Example + ------- + >>> from physicsnemo.models.domino.model import DoMINO + >>> import torch, os + >>> from hydra import compose, initialize + >>> from omegaconf import OmegaConf + >>> device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") + >>> cfg = OmegaConf.register_new_resolver("eval", eval) + >>> with initialize(version_base="1.3", config_path="examples/cfd/external_aerodynamics/domino/src/conf"): + ... cfg = compose(config_name="config") + >>> cfg.model.model_type = "combined" + >>> model = DoMINO( + ... input_features=3, + ... output_features_vol=5, + ... output_features_surf=4, + ... model_parameters=cfg.model + ... ).to(device) + + Warp ... + >>> bsize = 1 + >>> nx, ny, nz = cfg.model.interp_res + >>> num_neigh = 7 + >>> pos_normals_closest_vol = torch.randn(bsize, 100, 3).to(device) + >>> pos_normals_com_vol = torch.randn(bsize, 100, 3).to(device) + >>> pos_normals_com_surface = torch.randn(bsize, 100, 3).to(device) + >>> geom_centers = torch.randn(bsize, 100, 3).to(device) + >>> grid = torch.randn(bsize, nx, ny, nz, 3).to(device) + >>> surf_grid = torch.randn(bsize, nx, ny, nz, 3).to(device) + >>> sdf_grid = torch.randn(bsize, nx, ny, nz).to(device) + >>> sdf_surf_grid = torch.randn(bsize, nx, ny, nz).to(device) + >>> sdf_nodes = torch.randn(bsize, 100, 1).to(device) + >>> surface_coordinates = torch.randn(bsize, 100, 3).to(device) + >>> surface_neighbors = torch.randn(bsize, 100, num_neigh, 3).to(device) + >>> surface_normals = torch.randn(bsize, 100, 3).to(device) + >>> surface_neighbors_normals = torch.randn(bsize, 100, num_neigh, 3).to(device) + >>> surface_sizes = torch.rand(bsize, 100).to(device) + 1e-6 # Note this needs to be > 0.0 + >>> surface_neighbors_areas = torch.rand(bsize, 100, num_neigh).to(device) + 1e-6 + >>> volume_coordinates = torch.randn(bsize, 100, 3).to(device) + >>> vol_grid_max_min = torch.randn(bsize, 2, 3).to(device) + >>> surf_grid_max_min = torch.randn(bsize, 2, 3).to(device) + >>> stream_velocity = torch.randn(bsize, 1).to(device) + >>> air_density = torch.randn(bsize, 1).to(device) + >>> input_dict = { + ... "pos_volume_closest": pos_normals_closest_vol, + ... "pos_volume_center_of_mass": pos_normals_com_vol, + ... "pos_surface_center_of_mass": pos_normals_com_surface, + ... "geometry_coordinates": geom_centers, + ... "grid": grid, + ... "surf_grid": surf_grid, + ... "sdf_grid": sdf_grid, + ... "sdf_surf_grid": sdf_surf_grid, + ... "sdf_nodes": sdf_nodes, + ... "surface_mesh_centers": surface_coordinates, + ... "surface_mesh_neighbors": surface_neighbors, + ... "surface_normals": surface_normals, + ... "surface_neighbors_normals": surface_neighbors_normals, + ... "surface_areas": surface_sizes, + ... "surface_neighbors_areas": surface_neighbors_areas, + ... "volume_mesh_centers": volume_coordinates, + ... "volume_min_max": vol_grid_max_min, + ... "surface_min_max": surf_grid_max_min, + ... "stream_velocity": stream_velocity, + ... "air_density": air_density, + ... } + >>> output = model(input_dict) + >>> print(f"{output[0].shape}, {output[1].shape}") + torch.Size([1, 100, 5]), torch.Size([1, 100, 4]) + """ + + def __init__( + self, + input_features: int, + output_features_vol: int | None = None, + output_features_surf: int | None = None, + model_parameters=None, + ): + """ + Initialize the DoMINO model. + + Args: + input_features: Number of input feature dimensions for point data + output_features_vol: Number of output features for volume quantities (None for surface-only mode) + output_features_surf: Number of output features for surface quantities (None for volume-only mode) + model_parameters: Configuration parameters for the model + + Raises: + ValueError: If both output_features_vol and output_features_surf are None + """ + super().__init__() + self.input_features = input_features + self.output_features_vol = output_features_vol + self.output_features_surf = output_features_surf + + if self.output_features_vol is None and self.output_features_surf is None: + raise ValueError( + "At least one of `output_features_vol` or `output_features_surf` must be specified" + ) + if hasattr(model_parameters, "solution_calculation_mode"): + if model_parameters.solution_calculation_mode not in [ + "one-loop", + "two-loop", + ]: + raise ValueError( + f"Invalid solution_calculation_mode: {model_parameters.solution_calculation_mode}, select 'one-loop' or 'two-loop'." + ) + self.solution_calculation_mode = model_parameters.solution_calculation_mode + else: + self.solution_calculation_mode = "two-loop" + self.num_variables_vol = output_features_vol + self.num_variables_surf = output_features_surf + self.grid_resolution = model_parameters.interp_res + self.surface_neighbors = model_parameters.surface_neighbors + self.use_surface_normals = model_parameters.use_surface_normals + self.use_surface_area = model_parameters.use_surface_area + self.encode_parameters = model_parameters.encode_parameters + self.param_scaling_factors = model_parameters.parameter_model.scaling_params + self.geo_encoding_type = model_parameters.geometry_encoding_type + + if self.use_surface_normals: + if not self.use_surface_area: + input_features_surface = input_features + 3 + else: + input_features_surface = input_features + 4 + else: + input_features_surface = input_features + + if self.encode_parameters: + # Defining the parameter model + base_layer_p = model_parameters.parameter_model.base_layer + self.parameter_model = ParameterModel( + input_features=2, model_parameters=model_parameters.parameter_model + ) + else: + base_layer_p = 0 + + self.geo_rep_volume = GeometryRep( + input_features=input_features, + radii=model_parameters.geometry_rep.geo_conv.volume_radii, + model_parameters=model_parameters, + ) + + self.geo_rep_surface = GeometryRep( + input_features=input_features, + radii=model_parameters.geometry_rep.geo_conv.surface_radii, + model_parameters=model_parameters, + ) + + self.geo_rep_surface1 = GeometryRep( + input_features=input_features, + radii=model_parameters.geometry_rep.geo_conv.volume_radii, + model_parameters=model_parameters, + ) + + # Basis functions for surface and volume + base_layer_nn = model_parameters.nn_basis_functions.base_layer + if self.output_features_surf is not None: + self.nn_basis_surf = nn.ModuleList() + for _ in range(self.num_variables_surf): + self.nn_basis_surf.append( + NNBasisFunctions( + input_features=input_features_surface, + model_parameters=model_parameters.nn_basis_functions, + ) + ) + + if self.output_features_vol is not None: + self.nn_basis_vol = nn.ModuleList() + for _ in range(self.num_variables_vol): + self.nn_basis_vol.append( + NNBasisFunctions( + input_features=input_features, + model_parameters=model_parameters.nn_basis_functions, + ) + ) + + # Positional encoding + position_encoder_base_neurons = model_parameters.position_encoder.base_neurons + self.activation = F.relu + self.use_sdf_in_basis_func = model_parameters.use_sdf_in_basis_func + if self.output_features_vol is not None: + if model_parameters.positional_encoding: + inp_pos_vol = 25 if model_parameters.use_sdf_in_basis_func else 12 + else: + inp_pos_vol = 7 if model_parameters.use_sdf_in_basis_func else 3 + + self.fc_p_vol = nn.Linear(inp_pos_vol, position_encoder_base_neurons) + + if self.output_features_surf is not None: + if model_parameters.positional_encoding: + inp_pos_surf = 12 + else: + inp_pos_surf = 3 + + self.fc_p_surf = nn.Linear(inp_pos_surf, position_encoder_base_neurons) + + # Positional encoding hidden layers + self.fc_p1 = nn.Linear( + position_encoder_base_neurons, position_encoder_base_neurons + ) + self.fc_p2 = nn.Linear( + position_encoder_base_neurons, position_encoder_base_neurons + ) + + # base_layer_geo = model_parameters.geometry_local.base_layer + + # BQ for surface + self.surface_neighbors_in_radius = ( + model_parameters.geometry_local.surface_neighbors_in_radius + ) + self.surface_radius = model_parameters.geometry_local.surface_radii + self.surface_bq_warp = nn.ModuleList() + self.surface_local_point_conv = nn.ModuleList() + + for ct, j in enumerate(self.surface_radius): + if self.geo_encoding_type == "both": + total_neighbors_in_radius = self.surface_neighbors_in_radius[ct] * ( + len(model_parameters.geometry_rep.geo_conv.surface_radii) + 1 + ) + elif self.geo_encoding_type == "stl": + total_neighbors_in_radius = self.surface_neighbors_in_radius[ct] * ( + len(model_parameters.geometry_rep.geo_conv.surface_radii) + ) + elif self.geo_encoding_type == "sdf": + total_neighbors_in_radius = self.surface_neighbors_in_radius[ct] + + self.surface_bq_warp.append( + BQWarp( + grid_resolution=model_parameters.interp_res, + radius=self.surface_radius[ct], + neighbors_in_radius=self.surface_neighbors_in_radius[ct], + ) + ) + self.surface_local_point_conv.append( + LocalPointConv( + input_features=total_neighbors_in_radius, + base_layer=512, + output_features=self.surface_neighbors_in_radius[ct], + ) + ) + + # BQ for volume + self.volume_neighbors_in_radius = ( + model_parameters.geometry_local.volume_neighbors_in_radius + ) + self.volume_radius = model_parameters.geometry_local.volume_radii + self.volume_bq_warp = nn.ModuleList() + self.volume_local_point_conv = nn.ModuleList() + + for ct, j in enumerate(self.volume_radius): + if self.geo_encoding_type == "both": + total_neighbors_in_radius = self.volume_neighbors_in_radius[ct] * ( + len(model_parameters.geometry_rep.geo_conv.volume_radii) + 1 + ) + elif self.geo_encoding_type == "stl": + total_neighbors_in_radius = self.volume_neighbors_in_radius[ct] * ( + len(model_parameters.geometry_rep.geo_conv.volume_radii) + ) + elif self.geo_encoding_type == "sdf": + total_neighbors_in_radius = self.volume_neighbors_in_radius[ct] + + self.volume_bq_warp.append( + BQWarp( + grid_resolution=model_parameters.interp_res, + radius=self.volume_radius[ct], + neighbors_in_radius=self.volume_neighbors_in_radius[ct], + ) + ) + self.volume_local_point_conv.append( + LocalPointConv( + input_features=total_neighbors_in_radius, + base_layer=512, + output_features=self.volume_neighbors_in_radius[ct], + ) + ) + + # Transmitting surface to volume + self.surf_to_vol_conv1 = nn.Conv3d( + len(model_parameters.geometry_rep.geo_conv.volume_radii) + 1, + 16, + kernel_size=3, + padding="same", + ) + self.surf_to_vol_conv2 = nn.Conv3d( + 16, + len(model_parameters.geometry_rep.geo_conv.volume_radii) + 1, + kernel_size=3, + padding="same", + ) + + # Aggregation model + if self.output_features_surf is not None: + # Surface + base_layer_geo_surf = 0 + for j in self.surface_neighbors_in_radius: + base_layer_geo_surf += j + + self.agg_model_surf = nn.ModuleList() + for _ in range(self.num_variables_surf): + self.agg_model_surf.append( + AggregationModel( + input_features=position_encoder_base_neurons + + base_layer_nn + + base_layer_geo_surf + + base_layer_p, + output_features=1, + model_parameters=model_parameters.aggregation_model, + ) + ) + + if self.output_features_vol is not None: + # Volume + base_layer_geo_vol = 0 + for j in self.volume_neighbors_in_radius: + base_layer_geo_vol += j + + self.agg_model_vol = nn.ModuleList() + for _ in range(self.num_variables_vol): + self.agg_model_vol.append( + AggregationModel( + input_features=position_encoder_base_neurons + + base_layer_nn + + base_layer_geo_vol + + base_layer_p, + output_features=1, + model_parameters=model_parameters.aggregation_model, + ) + ) + + def position_encoder( + self, + encoding_node: torch.Tensor, + eval_mode: Literal["surface", "volume"] = "volume", + ) -> torch.Tensor: + """ + Compute positional encoding for input points. + + Args: + encoding_node: Tensor containing node position information + eval_mode: Mode of evaluation, either "volume" or "surface" + + Returns: + Tensor containing positional encoding features + """ + if eval_mode == "volume": + x = self.activation(self.fc_p_vol(encoding_node)) + elif eval_mode == "surface": + x = self.activation(self.fc_p_surf(encoding_node)) + else: + raise ValueError( + f"`eval_mode` must be 'surface' or 'volume', got {eval_mode=}" + ) + x = self.activation(self.fc_p1(x)) + x = self.fc_p2(x) + return x + + def geo_encoding_local( + self, encoding_g, volume_mesh_centers, p_grid, mode="volume" + ): + """Function to calculate local geometry encoding from global encoding""" + + if mode == "volume": + radius = self.volume_radius + bq_warp = self.volume_bq_warp + point_conv = self.volume_local_point_conv + elif mode == "surface": + radius = self.surface_radius + bq_warp = self.surface_bq_warp + point_conv = self.surface_local_point_conv + + batch_size = volume_mesh_centers.shape[0] + nx, ny, nz = ( + self.grid_resolution[0], + self.grid_resolution[1], + self.grid_resolution[2], + ) + + encoding_outer = [] + for p, q in enumerate(radius): + p_grid = torch.reshape(p_grid, (batch_size, nx * ny * nz, 3)) + mapping, outputs = bq_warp[p]( + volume_mesh_centers, p_grid, reverse_mapping=False + ) + mapping = mapping.type(torch.int64) + mask = mapping != 0 + + encoding_g_inner = [] + for j in range(encoding_g.shape[1]): + geo_encoding = torch.reshape( + encoding_g[:, j], (batch_size, 1, nx * ny * nz) + ) + geo_encoding_sampled = torch.index_select( + geo_encoding, 2, mapping.flatten() + ) + geo_encoding_sampled = torch.reshape(geo_encoding_sampled, mask.shape) + geo_encoding_sampled = geo_encoding_sampled * mask + + encoding_g_inner.append(geo_encoding_sampled) + encoding_g_inner = torch.cat(encoding_g_inner, axis=2) + encoding_g_inner = point_conv[p](encoding_g_inner) + encoding_outer.append(encoding_g_inner) + + encoding_g = torch.cat(encoding_outer, axis=-1) + + return encoding_g + + def calculate_solution_with_neighbors( + self, + surface_mesh_centers, + encoding_g, + encoding_node, + surface_mesh_neighbors, + surface_normals, + surface_neighbors_normals, + surface_areas, + surface_neighbors_areas, + inlet_velocity, + air_density, + num_sample_points=7, + ): + """Function to approximate solution given the neighborhood information""" + num_variables = self.num_variables_surf + nn_basis = self.nn_basis_surf + agg_model = self.agg_model_surf + # num_sample_points = surface_mesh_neighbors.shape[2] + 1 + + if self.encode_parameters: + inlet_velocity = torch.unsqueeze(inlet_velocity, 1) + inlet_velocity = inlet_velocity.expand( + inlet_velocity.shape[0], + surface_mesh_centers.shape[1], + inlet_velocity.shape[2], + ) + inlet_velocity = inlet_velocity / self.param_scaling_factors[0] + + air_density = torch.unsqueeze(air_density, 1) + air_density = air_density.expand( + air_density.shape[0], + surface_mesh_centers.shape[1], + air_density.shape[2], + ) + air_density = air_density / self.param_scaling_factors[1] + + params = torch.cat((inlet_velocity, air_density), axis=-1) + param_encoding = self.parameter_model(params) + + if self.use_surface_normals: + if not self.use_surface_area: + surface_mesh_centers = torch.cat( + (surface_mesh_centers, surface_normals), + dim=-1, + ) + if num_sample_points > 1: + surface_mesh_neighbors = torch.cat( + ( + surface_mesh_neighbors, + surface_neighbors_normals, + ), + dim=-1, + ) + + else: + surface_mesh_centers = torch.cat( + ( + surface_mesh_centers, + surface_normals, + torch.log(surface_areas) / 10, + ), + dim=-1, + ) + if num_sample_points > 1: + surface_mesh_neighbors = torch.cat( + ( + surface_mesh_neighbors, + surface_neighbors_normals, + torch.log(surface_neighbors_areas) / 10, + ), + dim=-1, + ) + if self.solution_calculation_mode == "one-loop": + encoding_list = [ + encoding_node.unsqueeze(2).expand(-1, -1, num_sample_points, -1), + encoding_g.unsqueeze(2).expand(-1, -1, num_sample_points, -1), + ] + + for f in range(num_variables): + one_loop_centers_expanded = surface_mesh_centers.unsqueeze(2) + + one_loop_noise = one_loop_centers_expanded - ( + surface_mesh_neighbors + 1e-6 + ) + one_loop_noise = torch.norm(one_loop_noise, dim=-1, keepdim=True) + + # Doing it this way prevents the intermediate one_loop_basis_f from being stored in memory for the rest of the function. + agg_output = agg_model[f]( + torch.cat( + ( + nn_basis[f]( + torch.cat( + ( + one_loop_centers_expanded, + surface_mesh_neighbors + 1e-6, + ), + axis=2, + ) + ), + *encoding_list, + ), + axis=-1, + ) + ) + + one_loop_output_center, one_loop_output_neighbor = torch.split( + agg_output, [1, num_sample_points - 1], dim=2 + ) + one_loop_output_neighbor = one_loop_output_neighbor * ( + 1.0 / one_loop_noise + ) + + one_loop_output_center = one_loop_output_center.squeeze(2) + one_loop_output_neighbor = one_loop_output_neighbor.sum(2) + one_loop_dist_sum = torch.sum(1.0 / one_loop_noise, dim=2) + + # Stop here + if num_sample_points > 1: + one_loop_output_res = ( + 0.5 * one_loop_output_center + + 0.5 * one_loop_output_neighbor / one_loop_dist_sum + ) + else: + one_loop_output_res = one_loop_output_center + if f == 0: + one_loop_output_all = one_loop_output_res + else: + one_loop_output_all = torch.cat( + (one_loop_output_all, one_loop_output_res), axis=-1 + ) + + return one_loop_output_all + + if self.solution_calculation_mode == "two-loop": + for f in range(num_variables): + for p in range(num_sample_points): + if p == 0: + volume_m_c = surface_mesh_centers + else: + volume_m_c = surface_mesh_neighbors[:, :, p - 1] + 1e-6 + noise = surface_mesh_centers - volume_m_c + dist = torch.norm(noise, dim=-1, keepdim=True) + + basis_f = nn_basis[f](volume_m_c) + output = torch.cat((basis_f, encoding_node, encoding_g), axis=-1) + if self.encode_parameters: + output = torch.cat((output, param_encoding), axis=-1) + if p == 0: + output_center = agg_model[f](output) + else: + if p == 1: + output_neighbor = agg_model[f](output) * (1.0 / dist) + dist_sum = 1.0 / dist + else: + output_neighbor += agg_model[f](output) * (1.0 / dist) + dist_sum += 1.0 / dist + if num_sample_points > 1: + output_res = 0.5 * output_center + 0.5 * output_neighbor / dist_sum + else: + output_res = output_center + if f == 0: + output_all = output_res + else: + output_all = torch.cat((output_all, output_res), axis=-1) + + return output_all + + def calculate_solution( + self, + volume_mesh_centers, + encoding_g, + encoding_node, + inlet_velocity, + air_density, + eval_mode, + num_sample_points=20, + noise_intensity=50, + ): + """Function to approximate solution sampling the neighborhood information""" + if eval_mode == "volume": + num_variables = self.num_variables_vol + nn_basis = self.nn_basis_vol + agg_model = self.agg_model_vol + elif eval_mode == "surface": + num_variables = self.num_variables_surf + nn_basis = self.nn_basis_surf + agg_model = self.agg_model_surf + + if self.encode_parameters: + inlet_velocity = torch.unsqueeze(inlet_velocity, 1) + inlet_velocity = inlet_velocity.expand( + inlet_velocity.shape[0], + volume_mesh_centers.shape[1], + inlet_velocity.shape[2], + ) + inlet_velocity = inlet_velocity / self.param_scaling_factors[0] + + air_density = torch.unsqueeze(air_density, 1) + air_density = air_density.expand( + air_density.shape[0], volume_mesh_centers.shape[1], air_density.shape[2] + ) + air_density = air_density / self.param_scaling_factors[1] + + params = torch.cat((inlet_velocity, air_density), axis=-1) + param_encoding = self.parameter_model(params) + + if self.solution_calculation_mode == "one-loop": + # Stretch these out to num_sample_points + one_loop_encoding_node = encoding_node.unsqueeze(0).expand( + num_sample_points, -1, -1, -1 + ) + one_loop_encoding_g = encoding_g.unsqueeze(0).expand( + num_sample_points, -1, -1, -1 + ) + + if self.encode_parameters: + one_loop_other_terms = ( + one_loop_encoding_node, + one_loop_encoding_g, + param_encoding, + ) + else: + one_loop_other_terms = (one_loop_encoding_node, one_loop_encoding_g) + + for f in range(num_variables): + one_loop_volume_mesh_centers_expanded = volume_mesh_centers.unsqueeze( + 0 + ).expand(num_sample_points, -1, -1, -1) + # Bulk_random_noise has shape (num_sample_points, batch_size, num_points, 3) + one_loop_bulk_random_noise = torch.rand_like( + one_loop_volume_mesh_centers_expanded + ) + + one_loop_bulk_random_noise = 2 * (one_loop_bulk_random_noise - 0.5) + one_loop_bulk_random_noise = ( + one_loop_bulk_random_noise / noise_intensity + ) + one_loop_bulk_dist = torch.norm( + one_loop_bulk_random_noise, dim=-1, keepdim=True + ) + + _, one_loop_bulk_dist = torch.split( + one_loop_bulk_dist, [1, num_sample_points - 1], dim=0 + ) + + # Set the first sample point to 0.0: + one_loop_bulk_random_noise[0] = torch.zeros_like( + one_loop_bulk_random_noise[0] + ) + + # Add the noise to the expanded volume_mesh_centers: + one_loop_volume_m_c = volume_mesh_centers + one_loop_bulk_random_noise + # If this looks overly complicated - it is. + # But, this makes sure that the memory used to store the output of both nn_basis[f] + # as well as the output of torch.cat can be deallocated immediately. + # Apply the aggregation model and distance scaling: + one_loop_output = agg_model[f]( + torch.cat( + (nn_basis[f](one_loop_volume_m_c), *one_loop_other_terms), + axis=-1, + ) + ) + + # select off the first, unperturbed term: + one_loop_output_center, one_loop_output_neighbor = torch.split( + one_loop_output, [1, num_sample_points - 1], dim=0 + ) + + # Scale the neighbor terms by the distance: + one_loop_output_neighbor = one_loop_output_neighbor / one_loop_bulk_dist + + one_loop_dist_sum = torch.sum(1.0 / one_loop_bulk_dist, dim=0) + + # Adjust shapes: + one_loop_output_center = one_loop_output_center.squeeze(1) + one_loop_output_neighbor = one_loop_output_neighbor.sum(0) + + # Compare: + if num_sample_points > 1: + one_loop_output_res = ( + 0.5 * one_loop_output_center + + 0.5 * one_loop_output_neighbor / one_loop_dist_sum + ) + else: + one_loop_output_res = one_loop_output_center + if f == 0: + one_loop_output_all = one_loop_output_res + else: + one_loop_output_all = torch.cat( + (one_loop_output_all, one_loop_output_res), axis=-1 + ) + + return one_loop_output_all + + if self.solution_calculation_mode == "two-loop": + for f in range(num_variables): + for p in range(num_sample_points): + if p == 0: + volume_m_c = volume_mesh_centers + else: + noise = torch.rand_like(volume_mesh_centers) + noise = 2 * (noise - 0.5) + noise = noise / noise_intensity + dist = torch.norm(noise, dim=-1, keepdim=True) + + volume_m_c = volume_mesh_centers + noise + basis_f = nn_basis[f](volume_m_c) + output = torch.cat((basis_f, encoding_node, encoding_g), axis=-1) + if self.encode_parameters: + output = torch.cat((output, param_encoding), axis=-1) + if p == 0: + output_center = agg_model[f](output) + else: + if p == 1: + output_neighbor = agg_model[f](output) * (1.0 / dist) + dist_sum = 1.0 / dist + else: + output_neighbor += agg_model[f](output) * (1.0 / dist) + dist_sum += 1.0 / dist + if num_sample_points > 1: + output_res = 0.5 * output_center + 0.5 * output_neighbor / dist_sum + else: + output_res = output_center + if f == 0: + output_all = output_res + else: + output_all = torch.cat((output_all, output_res), axis=-1) + + return output_all + + @profile + def forward( + self, + data_dict, + ): + # Loading STL inputs, bounding box grids, precomputed SDF and scaling factors + + # STL nodes + geo_centers = data_dict["geometry_coordinates"] + + # Bounding box grid + s_grid = data_dict["surf_grid"] + sdf_surf_grid = data_dict["sdf_surf_grid"] + # Scaling factors + surf_max = data_dict["surface_min_max"][:, 1] + surf_min = data_dict["surface_min_max"][:, 0] + + # Parameters + stream_velocity = data_dict["stream_velocity"] + air_density = data_dict["air_density"] + + if self.output_features_vol is not None: + # Represent geometry on computational grid + # Computational domain grid + p_grid = data_dict["grid"] + sdf_grid = data_dict["sdf_grid"] + # Scaling factors + vol_max = data_dict["volume_min_max"][:, 1] + vol_min = data_dict["volume_min_max"][:, 0] + + # Normalize based on computational domain + geo_centers_vol = 2.0 * (geo_centers - vol_min) / (vol_max - vol_min) - 1 + encoding_g_vol = self.geo_rep_volume(geo_centers_vol, p_grid, sdf_grid) + + # Normalize based on BBox around surface (car) + # geo_centers_surf = ( + # 2.0 * (geo_centers - surf_min) / (surf_max - surf_min) - 1 + # ) + + # encoding_g_surf = self.geo_rep_surface1( + # geo_centers_surf, s_grid, sdf_surf_grid + # ) + + # encoding_g_vol += encoding_g_surf + + # SDF on volume mesh nodes + sdf_nodes = data_dict["sdf_nodes"] + # Positional encoding based on closest point on surface to a volume node + pos_volume_closest = data_dict["pos_volume_closest"] + # Positional encoding based on center of mass of geometry to volume node + pos_volume_center_of_mass = data_dict["pos_volume_center_of_mass"] + if self.use_sdf_in_basis_func: + encoding_node_vol = torch.cat( + (sdf_nodes, pos_volume_closest, pos_volume_center_of_mass), axis=-1 + ) + else: + encoding_node_vol = pos_volume_center_of_mass + + # Calculate positional encoding on volume nodes + encoding_node_vol = self.position_encoder( + encoding_node_vol, eval_mode="volume" + ) + + if self.output_features_surf is not None: + # Represent geometry on bounding box + geo_centers_surf = ( + 2.0 * (geo_centers - surf_min) / (surf_max - surf_min) - 1 + ) + encoding_g_surf = self.geo_rep_surface( + geo_centers_surf, s_grid, sdf_surf_grid + ) + + # Positional encoding based on center of mass of geometry to surface node + pos_surface_center_of_mass = data_dict["pos_surface_center_of_mass"] + encoding_node_surf = pos_surface_center_of_mass + + # Calculate positional encoding on surface centers + encoding_node_surf = self.position_encoder( + encoding_node_surf, eval_mode="surface" + ) + + if self.output_features_vol is not None: + # Calculate local geometry encoding for volume + # Sampled points on volume + volume_mesh_centers = data_dict["volume_mesh_centers"] + encoding_g_vol = self.geo_encoding_local( + 0.5 * encoding_g_vol, volume_mesh_centers, p_grid, mode="volume" + ) + + # Approximate solution on volume node + output_vol = self.calculate_solution( + volume_mesh_centers, + encoding_g_vol, + encoding_node_vol, + stream_velocity, + air_density, + eval_mode="volume", + ) + else: + output_vol = None + + if self.output_features_surf is not None: + # Sampled points on surface + surface_mesh_centers = data_dict["surface_mesh_centers"] + surface_normals = data_dict["surface_normals"] + surface_areas = data_dict["surface_areas"] + + # Neighbors of sampled points on surface + surface_mesh_neighbors = data_dict["surface_mesh_neighbors"] + surface_neighbors_normals = data_dict["surface_neighbors_normals"] + surface_neighbors_areas = data_dict["surface_neighbors_areas"] + surface_areas = torch.unsqueeze(surface_areas, -1) + surface_neighbors_areas = torch.unsqueeze(surface_neighbors_areas, -1) + # Calculate local geometry encoding for surface + encoding_g_surf = self.geo_encoding_local( + 0.5 * encoding_g_surf, surface_mesh_centers, s_grid, mode="surface" + ) + + # Approximate solution on surface cell center + if not self.surface_neighbors: + output_surf = self.calculate_solution( + surface_mesh_centers, + encoding_g_surf, + encoding_node_surf, + stream_velocity, + air_density, + eval_mode="surface", + num_sample_points=1, + noise_intensity=500, + ) + else: + output_surf = self.calculate_solution_with_neighbors( + surface_mesh_centers, + encoding_g_surf, + encoding_node_surf, + surface_mesh_neighbors, + surface_normals, + surface_neighbors_normals, + surface_areas, + surface_neighbors_areas, + stream_velocity, + air_density, + num_sample_points=self.num_surface_neighbors, + ) + else: + output_surf = None + + return output_vol, output_surf diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/openfoam_datapipe.py b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/openfoam_datapipe.py new file mode 100644 index 0000000000..1772579731 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/openfoam_datapipe.py @@ -0,0 +1,278 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This is the datapipe to read OpenFoam files (vtp/vtu/stl) and save them as point clouds +in npy format. + +""" + +import time, random +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable, List, Literal, Mapping, Optional, Union, Callable + +import numpy as np +import pyvista as pv +import vtk +from physicsnemo.models.domino.utils import * +from torch.utils.data import Dataset + +# AIR_DENSITY = 1.205 +# STREAM_VELOCITY = 30.00 + + +class DriveSimPaths: + @staticmethod + def geometry_path(car_dir: Path) -> Path: + return car_dir / "body.stl" + + @staticmethod + def volume_path(car_dir: Path) -> Path: + return car_dir / "VTK/simpleFoam_steady_3000/internal.vtu" + + @staticmethod + def surface_path(car_dir: Path) -> Path: + return car_dir / "VTK/simpleFoam_steady_3000/boundary/aero_suv.vtp" + + +class DrivAerAwsPaths: + @staticmethod + def _get_index(car_dir: Path) -> str: + return car_dir.name.removeprefix("run_") + + @staticmethod + def geometry_path(car_dir: Path) -> Path: + return car_dir / f"drivaer_{DrivAerAwsPaths._get_index(car_dir)}.stl" + + @staticmethod + def volume_path(car_dir: Path) -> Path: + return car_dir / f"volume_{DrivAerAwsPaths._get_index(car_dir)}_predicted.vtu" + + @staticmethod + def surface_path(car_dir: Path) -> Path: + return car_dir / f"boundary_{DrivAerAwsPaths._get_index(car_dir)}_predicted.vtp" + + +class OpenFoamDataset(Dataset): + """ + Datapipe for converting openfoam dataset to npy + + """ + + def __init__( + self, + data_path: Union[str, Path], + kind: Literal["drivesim", "drivaer_aws"] = "drivesim", + surface_variables: Optional[list] = [ + "pMean", + "wallShearStress", + ], + volume_variables: Optional[list] = ["UMean", "pMean"], + global_params_types: Optional[dict] = { + "inlet_velocity": "vector", + "air_density": "scalar", + }, + global_params_reference: Optional[dict] = { + "inlet_velocity": [38.89], + "air_density": 1.226, + }, + device: int = 0, + model_type=None, + ): + if isinstance(data_path, str): + data_path = Path(data_path) + data_path = data_path.expanduser() + + self.data_path = data_path + + supported_kinds = ["drivesim", "drivaer_aws"] + assert kind in supported_kinds, ( + f"kind should be one of {supported_kinds}, got {kind}" + ) + self.path_getter = DriveSimPaths if kind == "drivesim" else DrivAerAwsPaths + + assert self.data_path.exists(), f"Path {self.data_path} does not exist" + + assert self.data_path.is_dir(), f"Path {self.data_path} is not a directory" + + self.filenames = get_filenames(self.data_path) + random.shuffle(self.filenames) + self.indices = np.array(len(self.filenames)) + + self.surface_variables = surface_variables + self.volume_variables = volume_variables + + self.global_params_types = global_params_types + self.global_params_reference = global_params_reference + + self.stream_velocity = 0.0 + for vel_component in self.global_params_reference["inlet_velocity"]: + self.stream_velocity += vel_component**2 + self.stream_velocity = np.sqrt(self.stream_velocity) + self.air_density = self.global_params_reference["air_density"] + + self.device = device + self.model_type = model_type + + def __len__(self): + return len(self.filenames) + + def __getitem__(self, idx): + cfd_filename = self.filenames[idx] + car_dir = self.data_path / cfd_filename + + stl_path = self.path_getter.geometry_path(car_dir) + reader = pv.get_reader(stl_path) + mesh_stl = reader.read() + stl_vertices = mesh_stl.points + stl_faces = np.array(mesh_stl.faces).reshape((-1, 4))[ + :, 1: + ] # Assuming triangular elements + mesh_indices_flattened = stl_faces.flatten() + stl_sizes = mesh_stl.compute_cell_sizes(length=False, area=True, volume=False) + stl_sizes = np.array(stl_sizes.cell_data["Area"]) + stl_centers = np.array(mesh_stl.cell_centers().points) + + length_scale = np.amax(np.amax(stl_vertices, 0) - np.amin(stl_vertices, 0)) + + if self.model_type == "volume" or self.model_type == "combined": + filepath = self.path_getter.volume_path(car_dir) + reader = vtk.vtkXMLUnstructuredGridReader() + reader.SetFileName(filepath) + reader.Update() + + # Get the unstructured grid data + polydata = reader.GetOutput() + volume_coordinates, volume_fields = get_volume_data( + polydata, self.volume_variables + ) + volume_fields = np.concatenate(volume_fields, axis=-1) + + # Non-dimensionalize volume fields + volume_fields[:, :3] = volume_fields[:, :3] / self.stream_velocity + volume_fields[:, 3:4] = volume_fields[:, 3:4] / ( + self.air_density * self.stream_velocity**2.0 + ) + + volume_fields[:, 4:] = volume_fields[:, 4:] / ( + self.stream_velocity * length_scale + ) + else: + volume_fields = None + volume_coordinates = None + + if self.model_type == "surface" or self.model_type == "combined": + surface_filepath = self.path_getter.surface_path(car_dir) + reader = vtk.vtkXMLPolyDataReader() + reader.SetFileName(surface_filepath) + reader.Update() + polydata = reader.GetOutput() + + celldata_all = get_node_to_elem(polydata) + celldata = celldata_all.GetCellData() + surface_fields = get_fields(celldata, self.surface_variables) + surface_fields = np.concatenate(surface_fields, axis=-1) + + mesh = pv.PolyData(polydata) + surface_coordinates = np.array(mesh.cell_centers().points) + + surface_normals = np.array(mesh.cell_normals) + surface_sizes = mesh.compute_cell_sizes( + length=False, area=True, volume=False + ) + surface_sizes = np.array(surface_sizes.cell_data["Area"]) + + # Normalize cell normals + surface_normals = ( + surface_normals / np.linalg.norm(surface_normals, axis=1)[:, np.newaxis] + ) + + # Non-dimensionalize surface fields + surface_fields = surface_fields / ( + self.air_density * self.stream_velocity**2.0 + ) + else: + surface_fields = None + surface_coordinates = None + surface_normals = None + surface_sizes = None + + # Arrange global parameters reference in a list based on the type of the parameter + global_params_reference_list = [] + for name, type in self.global_params_types.items(): + if type == "vector": + global_params_reference_list.extend(self.global_params_reference[name]) + elif type == "scalar": + global_params_reference_list.append(self.global_params_reference[name]) + else: + raise ValueError( + f"Global parameter {name} not supported for this dataset" + ) + global_params_reference = np.array( + global_params_reference_list, dtype=np.float32 + ) + + # Prepare the list of global parameter values for each simulation file + # Note: The user must ensure that the values provided here correspond to the + # `global_parameters` specified in `config.yaml` and that these parameters + # exist within each simulation file. + global_params_values_list = [] + for key in self.global_params_types.keys(): + if key == "inlet_velocity": + global_params_values_list.extend( + self.global_params_reference["inlet_velocity"] + ) + elif key == "air_density": + global_params_values_list.append( + self.global_params_reference["air_density"] + ) + else: + raise ValueError( + f"Global parameter {key} not supported for this dataset" + ) + global_params_values = np.array(global_params_values_list, dtype=np.float32) + + # Add the parameters to the dictionary + return { + "stl_coordinates": np.float32(stl_vertices), + "stl_centers": np.float32(stl_centers), + "stl_faces": np.float32(mesh_indices_flattened), + "stl_areas": np.float32(stl_sizes), + "surface_mesh_centers": np.float32(surface_coordinates), + "surface_normals": np.float32(surface_normals), + "surface_areas": np.float32(surface_sizes), + "volume_fields": np.float32(volume_fields), + "volume_mesh_centers": np.float32(volume_coordinates), + "surface_fields": np.float32(surface_fields), + "filename": cfd_filename, + "global_params_values": global_params_values, + "global_params_reference": global_params_reference, + } + + +if __name__ == "__main__": + fm_data = OpenFoamDataset( + data_path="/code/aerofoundationdata/", + phase="train", + volume_variables=["UMean", "pMean", "nutMean"], + surface_variables=["pMean", "wallShearStress", "nutMean"], + global_params_types={"inlet_velocity": "vector", "air_density": "scalar"}, + global_params_reference={"inlet_velocity": [30.0], "air_density": 1.226}, + sampling=False, + sample_in_bbox=False, + ) + d_dict = fm_data[1] diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/process_data.py b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/process_data.py new file mode 100644 index 0000000000..c053594894 --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/process_data.py @@ -0,0 +1,106 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code runs the data processing in parallel to load OpenFoam files, process them +and save in the npy format for faster processing in the DoMINO datapipes. Several +parameters such as number of processors, input and output paths, etc. can be +configured in config.yaml in the data_processing tab. +""" + +from openfoam_datapipe import OpenFoamDataset +from physicsnemo.models.domino.utils import * +import multiprocessing +import hydra, time, os +from hydra.utils import to_absolute_path +from omegaconf import DictConfig, OmegaConf + + +def process_files(*args_list): + ids = args_list[0] + processor_id = args_list[1] + fm_data = args_list[2] + output_dir = args_list[3] + for j in ids: + fname = fm_data.filenames[j] + if len(os.listdir(os.path.join(fm_data.data_path, fname))) == 0: + print(f"Skipping {fname} - empty.") + continue + outname = os.path.join(output_dir, fname) + print("Filename:%s on processor: %d" % (outname, processor_id)) + filename = f"{outname}.npy" + if os.path.exists(filename): + print(f"Skipping {filename} - already exists.") + continue + start_time = time.time() + data_dict = fm_data[j] + np.save(filename, data_dict) + print("Time taken for %d = %f" % (j, time.time() - start_time)) + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig): + print(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + phase = "train" + volume_variable_names = list(cfg.variables.volume.solution.keys()) + num_vol_vars = 0 + for j in volume_variable_names: + if cfg.variables.volume.solution[j] == "vector": + num_vol_vars += 3 + else: + num_vol_vars += 1 + + surface_variable_names = list(cfg.variables.surface.solution.keys()) + num_surf_vars = 0 + for j in surface_variable_names: + if cfg.variables.surface.solution[j] == "vector": + num_surf_vars += 3 + else: + num_surf_vars += 1 + + fm_data = OpenFoamDataset( + cfg.data_processor.input_dir, + kind=cfg.data_processor.kind, + volume_variables=volume_variable_names, + surface_variables=surface_variable_names, + model_type=cfg.model.model_type, + ) + output_dir = cfg.data_processor.output_dir + create_directory(output_dir) + n_processors = cfg.data_processor.num_processors + + num_files = len(fm_data) + ids = np.arange(num_files) + num_elements = int(num_files / n_processors) + 1 + process_list = [] + ctx = multiprocessing.get_context("spawn") + for i in range(n_processors): + if i != n_processors - 1: + sf = ids[i * num_elements : i * num_elements + num_elements] + else: + sf = ids[i * num_elements :] + # print(sf) + process = ctx.Process(target=process_files, args=(sf, i, fm_data, output_dir)) + + process.start() + process_list.append(process) + + for process in process_list: + process.join() + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/test.py b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/test.py new file mode 100644 index 0000000000..fd6417166d --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/test.py @@ -0,0 +1,892 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code defines a distributed pipeline for testing the finetuned DoMINO model on +CFD datasets. The finetuned model is combined with the base predictions to generate +the final predictions. The model predictions are loaded in the VTP/VTU files and +saved in the specified directory. The eval tab in config.yaml can be used to +specify the input and output directories. +""" + +import os, re +import time + +import hydra +from hydra.utils import to_absolute_path +from omegaconf import DictConfig, OmegaConf + +import numpy as np + +from collections import defaultdict +from pathlib import Path +from typing import Any, Iterable, List, Literal, Mapping, Optional, Union, Callable + +import pandas as pd +import pyvista as pv + +import torch +from torch.nn.parallel import DistributedDataParallel +from torch.utils.data import DataLoader, Dataset + +import vtk +from vtk.util import numpy_support + +from physicsnemo.distributed import DistributedManager +from physicsnemo.datapipes.cae.domino_datapipe import DoMINODataPipe +from physicsnemo.models.domino.model import DoMINO +from physicsnemo.models.domino.utils import * +from physicsnemo.nn.functional import signed_distance_field + +# AIR_DENSITY = 1.205 +# STREAM_VELOCITY = 30.00 + + +def loss_fn(output, target): + masked_loss = torch.mean(((output - target) ** 2.0), (0, 1, 2)) + loss = torch.mean(masked_loss) + return loss + + +def test_step(data_dict, model, device, cfg, vol_factors, surf_factors): + avg_tloss_vol = 0.0 + avg_tloss_surf = 0.0 + running_tloss_vol = 0.0 + running_tloss_surf = 0.0 + + if cfg.model.model_type == "volume" or cfg.model.model_type == "combined": + output_features_vol = True + else: + output_features_vol = None + + if cfg.model.model_type == "surface" or cfg.model.model_type == "combined": + output_features_surf = True + else: + output_features_surf = None + + with torch.no_grad(): + point_batch_size = 256000 + data_dict = dict_to_device(data_dict, device) + + # Non-dimensionalization factors + length_scale = data_dict["length_scale"] + + global_params_values = data_dict["global_params_values"] + global_params_reference = data_dict["global_params_reference"] + stream_velocity = global_params_reference[:, 0, :] + air_density = global_params_reference[:, 1, :] + + # STL nodes + geo_centers = data_dict["geometry_coordinates"] + + # Bounding box grid + s_grid = data_dict["surf_grid"] + sdf_surf_grid = data_dict["sdf_surf_grid"] + # Scaling factors + surf_max = data_dict["surface_min_max"][:, 1] + surf_min = data_dict["surface_min_max"][:, 0] + + if output_features_vol is not None: + # Represent geometry on computational grid + # Computational domain grid + p_grid = data_dict["grid"] + sdf_grid = data_dict["sdf_grid"] + # Scaling factors + vol_max = data_dict["volume_min_max"][:, 1] + vol_min = data_dict["volume_min_max"][:, 0] + + # Normalize based on computational domain + geo_centers_vol = 2.0 * (geo_centers - vol_min) / (vol_max - vol_min) - 1 + encoding_g_vol = model.geo_rep_volume(geo_centers_vol, p_grid, sdf_grid) + + if output_features_surf is not None: + # Represent geometry on bounding box + geo_centers_surf = ( + 2.0 * (geo_centers - surf_min) / (surf_max - surf_min) - 1 + ) + encoding_g_surf = model.geo_rep_surface( + geo_centers_surf, s_grid, sdf_surf_grid + ) + + if ( + output_features_vol is not None + and output_features_surf is not None + and cfg.model.combine_volume_surface + ): + encoding_g = torch.cat((encoding_g_vol, encoding_g_surf), axis=1) + encoding_g_surf = model.combined_unet_surf(encoding_g) + encoding_g_vol = model.combined_unet_vol(encoding_g) + + if output_features_vol is not None: + # First calculate volume predictions if required + volume_mesh_centers = data_dict["volume_mesh_centers"] + target_vol = data_dict["volume_fields"] + # SDF on volume mesh nodes + sdf_nodes = data_dict["sdf_nodes"] + # Positional encoding based on closest point on surface to a volume node + pos_volume_closest = data_dict["pos_volume_closest"] + # Positional encoding based on center of mass of geometry to volume node + pos_volume_center_of_mass = data_dict["pos_volume_center_of_mass"] + p_grid = data_dict["grid"] + + prediction_vol = np.zeros_like(target_vol.cpu().numpy()) + num_points = volume_mesh_centers.shape[1] + subdomain_points = int(np.floor(num_points / point_batch_size)) + + start_time = time.time() + + for p in range(subdomain_points + 1): + start_idx = p * point_batch_size + end_idx = (p + 1) * point_batch_size + with torch.no_grad(): + target_batch = target_vol[:, start_idx:end_idx] + volume_mesh_centers_batch = volume_mesh_centers[ + :, start_idx:end_idx + ] + sdf_nodes_batch = sdf_nodes[:, start_idx:end_idx] + pos_volume_closest_batch = pos_volume_closest[:, start_idx:end_idx] + pos_normals_com_batch = pos_volume_center_of_mass[ + :, start_idx:end_idx + ] + geo_encoding_local = model.geo_encoding_local( + 0.5 * encoding_g_vol, + volume_mesh_centers_batch, + p_grid, + mode="volume", + ) + if cfg.model.use_sdf_in_basis_func: + pos_encoding = torch.cat( + ( + sdf_nodes_batch, + pos_volume_closest_batch, + pos_normals_com_batch, + ), + axis=-1, + ) + else: + pos_encoding = pos_normals_com_batch + pos_encoding = model.position_encoder( + pos_encoding, eval_mode="volume" + ) + tpredictions_batch = model.calculate_solution( + volume_mesh_centers_batch, + geo_encoding_local, + pos_encoding, + global_params_values, + global_params_reference, + num_sample_points=cfg.model.num_neighbors_volume, + eval_mode="volume", + ) + running_tloss_vol += loss_fn(tpredictions_batch, target_batch) + prediction_vol[:, start_idx:end_idx] = ( + tpredictions_batch.cpu().numpy() + ) + + prediction_vol = unnormalize(prediction_vol, vol_factors[0], vol_factors[1]) + + prediction_vol[:, :, :3] = ( + prediction_vol[:, :, :3] * stream_velocity[0, 0].cpu().numpy() + ) + prediction_vol[:, :, 3] = ( + prediction_vol[:, :, 3] + * stream_velocity[0, 0].cpu().numpy() ** 2.0 + * air_density[0, 0].cpu().numpy() + ) + prediction_vol[:, :, 4] = ( + prediction_vol[:, :, 4] + * stream_velocity[0, 0].cpu().numpy() + * length_scale[0].cpu().numpy() + ) + else: + prediction_vol = None + + if output_features_surf is not None: + # Next calculate surface predictions + # Sampled points on surface + surface_mesh_centers = data_dict["surface_mesh_centers"] + surface_normals = data_dict["surface_normals"] + surface_areas = data_dict["surface_areas"] + + # Neighbors of sampled points on surface + surface_mesh_neighbors = data_dict["surface_mesh_neighbors"] + surface_neighbors_normals = data_dict["surface_neighbors_normals"] + surface_neighbors_areas = data_dict["surface_neighbors_areas"] + surface_areas = torch.unsqueeze(surface_areas, -1) + surface_neighbors_areas = torch.unsqueeze(surface_neighbors_areas, -1) + pos_surface_center_of_mass = data_dict["pos_surface_center_of_mass"] + num_points = surface_mesh_centers.shape[1] + subdomain_points = int(np.floor(num_points / point_batch_size)) + + target_surf = data_dict["surface_fields"] + prediction_surf = np.zeros_like(target_surf.cpu().numpy()) + + start_time = time.time() + + for p in range(subdomain_points + 1): + start_idx = p * point_batch_size + end_idx = (p + 1) * point_batch_size + with torch.no_grad(): + target_batch = target_surf[:, start_idx:end_idx] + surface_mesh_centers_batch = surface_mesh_centers[ + :, start_idx:end_idx + ] + surface_mesh_neighbors_batch = surface_mesh_neighbors[ + :, start_idx:end_idx + ] + surface_normals_batch = surface_normals[:, start_idx:end_idx] + surface_neighbors_normals_batch = surface_neighbors_normals[ + :, start_idx:end_idx + ] + surface_areas_batch = surface_areas[:, start_idx:end_idx] + surface_neighbors_areas_batch = surface_neighbors_areas[ + :, start_idx:end_idx + ] + pos_surface_center_of_mass_batch = pos_surface_center_of_mass[ + :, start_idx:end_idx + ] + geo_encoding_local = model.geo_encoding_local( + 0.5 * encoding_g_surf, + surface_mesh_centers_batch, + s_grid, + mode="surface", + ) + pos_encoding = pos_surface_center_of_mass_batch + pos_encoding = model.position_encoder( + pos_encoding, eval_mode="surface" + ) + + tpredictions_batch = model.calculate_solution_with_neighbors( + surface_mesh_centers_batch, + geo_encoding_local, + pos_encoding, + surface_mesh_neighbors_batch, + surface_normals_batch, + surface_neighbors_normals_batch, + surface_areas_batch, + surface_neighbors_areas_batch, + global_params_values, + global_params_reference, + num_sample_points=cfg.model.num_neighbors_surface, + ) + + running_tloss_surf += loss_fn(tpredictions_batch, target_batch) + prediction_surf[:, start_idx:end_idx] = ( + tpredictions_batch.cpu().numpy() + ) + + prediction_surf = ( + unnormalize(prediction_surf, surf_factors[0], surf_factors[1]) + * stream_velocity[0, 0].cpu().numpy() ** 2.0 + * air_density[0, 0].cpu().numpy() + ) + + else: + prediction_surf = None + + return prediction_vol, prediction_surf + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig): + print(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + input_path = cfg.eval.test_path + + model_type = cfg.model.model_type + + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + if model_type == "volume" or model_type == "combined": + volume_variable_names = list(cfg.variables.volume.solution.keys()) + volume_variable_names_gt = ["UMeanTrim", "pMeanTrim", "nutMeanTrim"] + volume_variable_names_base = [ + "UMeanTrimBasePred", + "pMeanTrimBasePred", + "nutMeanTrimBasePred", + ] + num_vol_vars = 0 + for j in volume_variable_names: + if cfg.variables.volume.solution[j] == "vector": + num_vol_vars += 3 + else: + num_vol_vars += 1 + else: + num_vol_vars = None + + if model_type == "surface" or model_type == "combined": + surface_variable_names = list(cfg.variables.surface.solution.keys()) + surface_variable_names_gt = ["pMeanTrim", "wallShearStressMeanTrim"] + surface_variable_names_base = [ + "pMeanTrimBasePred", + "wallShearStressMeanTrimBasePred", + ] + num_surf_vars = 0 + for j in surface_variable_names: + if cfg.variables.surface.solution[j] == "vector": + num_surf_vars += 3 + else: + num_surf_vars += 1 + else: + num_surf_vars = None + + global_features = 0 + global_params_names = list(cfg.variables.global_parameters.keys()) + for param in global_params_names: + if cfg.variables.global_parameters[param].type == "vector": + global_features += len(cfg.variables.global_parameters[param].reference) + else: + global_features += 1 + + vol_save_path = os.path.join( + cfg.eval.scaling_param_path, "volume_scaling_factors.npy" + ) + surf_save_path = os.path.join( + cfg.eval.scaling_param_path, "surface_scaling_factors.npy" + ) + if os.path.exists(vol_save_path): + vol_factors = np.load(vol_save_path) + else: + vol_factors = None + + if os.path.exists(surf_save_path): + surf_factors = np.load(surf_save_path) + else: + surf_factors = None + + print("Vol factors:", vol_factors) + print("Surf factors:", surf_factors) + + model = DoMINO( + input_features=3, + output_features_vol=num_vol_vars, + output_features_surf=num_surf_vars, + global_features=global_features, + model_parameters=cfg.model, + ).to(dist.device) + + model = torch.compile(model, disable=True) + + checkpoint = torch.load( + to_absolute_path(os.path.join(cfg.resume_dir, cfg.eval.checkpoint_name)), + map_location=dist.device, + ) + + model.load_state_dict(checkpoint) + + print("Model loaded") + + if dist.world_size > 1: + model = DistributedDataParallel( + model, + device_ids=[dist.local_rank], + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + gradient_as_bucket_view=True, + static_graph=True, + ) + model = model.module + + dirnames = get_filenames(input_path) + dev_id = torch.cuda.current_device() + num_files = int(len(dirnames) / dist.world_size) + 1 + dirnames_per_gpu = dirnames[int(num_files * dev_id) : int(num_files * (dev_id + 1))] + + pred_save_path = cfg.eval.save_path + + if dist.rank == 0: + create_directory(pred_save_path) + + l2_surface_all = [] + l2_volume_all = [] + aero_forces_all = [] + for count, dirname in enumerate(dirnames_per_gpu): + filepath = os.path.join(input_path, dirname) + tag = int(re.findall(r"(\w+?)(\d+)", dirname)[0][1]) + stl_path = os.path.join(filepath, f"drivaer_{tag}.stl") + vtp_path = os.path.join(filepath, f"boundary_{tag}_predicted.vtp") + vtu_path = os.path.join(filepath, f"volume_{tag}_predicted.vtu") + + vtp_pred_save_path = os.path.join( + pred_save_path, f"boundary_{tag}_predicted.vtp" + ) + vtu_pred_save_path = os.path.join(pred_save_path, f"volume_{tag}_predicted.vtu") + + # Read STL + reader = pv.get_reader(stl_path) + mesh_stl = reader.read() + stl_vertices = mesh_stl.points + stl_faces = np.array(mesh_stl.faces).reshape((-1, 4))[ + :, 1: + ] # Assuming triangular elements + mesh_indices_flattened = stl_faces.flatten() + length_scale = np.amax(np.amax(stl_vertices, 0) - np.amin(stl_vertices, 0)) + stl_sizes = mesh_stl.compute_cell_sizes(length=False, area=True, volume=False) + stl_sizes = np.array(stl_sizes.cell_data["Area"], dtype=np.float32) + stl_centers = np.array(mesh_stl.cell_centers().points, dtype=np.float32) + + # Center of mass calculation + center_of_mass = calculate_center_of_mass(stl_centers, stl_sizes) + + if cfg.data.bounding_box_surface is None: + s_max = np.amax(stl_vertices, 0) + s_min = np.amin(stl_vertices, 0) + else: + bounding_box_dims_surf = [] + bounding_box_dims_surf.append(np.asarray(cfg.data.bounding_box_surface.max)) + bounding_box_dims_surf.append(np.asarray(cfg.data.bounding_box_surface.min)) + s_max = np.float32(bounding_box_dims_surf[0]) + s_min = np.float32(bounding_box_dims_surf[1]) + + nx, ny, nz = cfg.model.interp_res + + surf_grid = create_grid(s_max, s_min, [nx, ny, nz]) + surf_grid_reshaped = surf_grid.reshape(nx * ny * nz, 3) + + # SDF calculation on the grid using WARP + sdf_surf_grid = signed_distance_field( + stl_vertices, + mesh_indices_flattened, + surf_grid_reshaped, + use_sign_winding_number=True, + ).reshape(nx, ny, nz) + surf_grid = np.float32(surf_grid) + sdf_surf_grid = np.float32(sdf_surf_grid) + surf_grid_max_min = np.float32(np.asarray([s_min, s_max])) + + # Get global parameters and global parameters scaling from config.yaml + global_params_names = list(cfg.variables.global_parameters.keys()) + global_params_reference = { + name: cfg.variables.global_parameters[name]["reference"] + for name in global_params_names + } + global_params_types = { + name: cfg.variables.global_parameters[name]["type"] + for name in global_params_names + } + stream_velocity = global_params_reference["inlet_velocity"][0] + air_density = global_params_reference["air_density"] + + # Arrange global parameters reference in a list, ensuring it is flat + global_params_reference_list = [] + for name, type in global_params_types.items(): + if type == "vector": + global_params_reference_list.extend(global_params_reference[name]) + elif type == "scalar": + global_params_reference_list.append(global_params_reference[name]) + else: + raise ValueError( + f"Global parameter {name} not supported for this dataset" + ) + global_params_reference = np.array( + global_params_reference_list, dtype=np.float32 + ) + + # Define the list of global parameter values for each simulation. + # Note: The user must ensure that the values provided here correspond to the + # `global_parameters` specified in `config.yaml` and that these parameters + # exist within each simulation file. + global_params_values_list = [] + for key in global_params_types.keys(): + if key == "inlet_velocity": + global_params_values_list.append(stream_velocity) + elif key == "air_density": + global_params_values_list.append(air_density) + else: + raise ValueError( + f"Global parameter {key} not supported for this dataset" + ) + global_params_values = np.array(global_params_values_list, dtype=np.float32) + + # Read VTP + if model_type == "surface" or model_type == "combined": + reader = vtk.vtkXMLPolyDataReader() + reader.SetFileName(vtp_path) + reader.Update() + polydata_surf = reader.GetOutput() + + celldata_all = get_node_to_elem(polydata_surf) + + celldata = celldata_all.GetCellData() + surface_fields = get_fields(celldata, surface_variable_names_gt) + surface_fields = np.concatenate(surface_fields, axis=-1) + surface_fields_base = get_fields(celldata, surface_variable_names_base) + surface_fields_base = np.concatenate(surface_fields_base, axis=-1) + + mesh = pv.PolyData(polydata_surf) + surface_coordinates = np.array(mesh.cell_centers().points, dtype=np.float32) + + surface_normals = np.array(mesh.cell_normals, dtype=np.float32) + surface_sizes = mesh.compute_cell_sizes( + length=False, area=True, volume=False + ) + surface_sizes = np.array(surface_sizes.cell_data["Area"], dtype=np.float32) + + # Normalize cell normals + surface_normals = ( + surface_normals / np.linalg.norm(surface_normals, axis=1)[:, np.newaxis] + ) + + if cfg.model.num_neighbors_surface > 1: + interp_func = KDTree(surface_coordinates) + dd, ii = interp_func.query( + surface_coordinates, k=cfg.model.num_neighbors_surface + ) + + surface_neighbors = surface_coordinates[ii] + surface_neighbors = surface_neighbors[:, 1:] + + surface_neighbors_normals = surface_normals[ii] + surface_neighbors_normals = surface_neighbors_normals[:, 1:] + surface_neighbors_sizes = surface_sizes[ii] + surface_neighbors_sizes = surface_neighbors_sizes[:, 1:] + else: + surface_neighbors = surface_coordinates + surface_neighbors_normals = surface_normals + surface_neighbors_sizes = surface_sizes + + dx, dy, dz = ( + (s_max[0] - s_min[0]) / nx, + (s_max[1] - s_min[1]) / ny, + (s_max[2] - s_min[2]) / nz, + ) + + if cfg.model.positional_encoding: + pos_surface_center_of_mass = calculate_normal_positional_encoding( + surface_coordinates, center_of_mass, cell_length=[dx, dy, dz] + ) + else: + pos_surface_center_of_mass = surface_coordinates - center_of_mass + + surface_coordinates = normalize(surface_coordinates, s_max, s_min) + surface_neighbors = normalize(surface_neighbors, s_max, s_min) + surf_grid = normalize(surf_grid, s_max, s_min) + + else: + surface_coordinates = None + surface_fields = None + surface_sizes = None + surface_normals = None + surface_neighbors = None + surface_neighbors_normals = None + surface_neighbors_sizes = None + pos_surface_center_of_mass = None + + # Read VTU + if model_type == "volume" or model_type == "combined": + reader = vtk.vtkXMLUnstructuredGridReader() + reader.SetFileName(vtu_path) + reader.Update() + polydata_vol = reader.GetOutput() + volume_coordinates, volume_fields = get_volume_data( + polydata_vol, volume_variable_names_gt + ) + volume_fields = np.concatenate(volume_fields, axis=-1) + volume_coordinates_base, volume_fields_base = get_volume_data( + polydata_vol, volume_variable_names_base + ) + volume_fields_base = np.concatenate(volume_fields_base, axis=-1) + # print(f"Processed vtu {vtu_path}") + + bounding_box_dims = [] + bounding_box_dims.append(np.asarray(cfg.data.bounding_box.max)) + bounding_box_dims.append(np.asarray(cfg.data.bounding_box.min)) + + v_max = np.amax(volume_coordinates, 0) + v_min = np.amin(volume_coordinates, 0) + if bounding_box_dims is None: + c_max = s_max + (s_max - s_min) / 2 + c_min = s_min - (s_max - s_min) / 2 + c_min[2] = s_min[2] + else: + c_max = np.float32(bounding_box_dims[0]) + c_min = np.float32(bounding_box_dims[1]) + + dx, dy, dz = ( + (c_max[0] - c_min[0]) / nx, + (c_max[1] - c_min[1]) / ny, + (c_max[2] - c_min[2]) / nz, + ) + # Generate a grid of specified resolution to map the bounding box + # The grid is used for capturing structured geometry features and SDF representation of geometry + grid = create_grid(c_max, c_min, [nx, ny, nz]) + grid_reshaped = grid.reshape(nx * ny * nz, 3) + + # SDF calculation on the grid using WARP + sdf_grid = signed_distance_field( + stl_vertices, + mesh_indices_flattened, + grid_reshaped, + use_sign_winding_number=True, + ).reshape(nx, ny, nz) + + # SDF calculation + sdf_nodes, sdf_node_closest_point = signed_distance_field( + stl_vertices, + mesh_indices_flattened, + volume_coordinates, + include_hit_points=True, + use_sign_winding_number=True, + ) + sdf_nodes = sdf_nodes.reshape(-1, 1) + + if cfg.model.positional_encoding: + pos_volume_closest = calculate_normal_positional_encoding( + volume_coordinates, sdf_node_closest_point, cell_length=[dx, dy, dz] + ) + pos_volume_center_of_mass = calculate_normal_positional_encoding( + volume_coordinates, center_of_mass, cell_length=[dx, dy, dz] + ) + else: + pos_volume_closest = volume_coordinates - sdf_node_closest_point + pos_volume_center_of_mass = volume_coordinates - center_of_mass + + volume_coordinates = normalize(volume_coordinates, c_max, c_min) + grid = normalize(grid, c_max, c_min) + vol_grid_max_min = np.asarray([c_min, c_max]) + + else: + volume_coordinates = None + volume_fields = None + pos_volume_closest = None + pos_volume_center_of_mass = None + + # print(f"Processed sdf and normalized") + + geom_centers = np.float32(stl_vertices) + + if model_type == "combined": + # Add the parameters to the dictionary + data_dict = { + "pos_volume_closest": pos_volume_closest, + "pos_volume_center_of_mass": pos_volume_center_of_mass, + "pos_surface_center_of_mass": pos_surface_center_of_mass, + "geometry_coordinates": geom_centers, + "grid": grid, + "surf_grid": surf_grid, + "sdf_grid": sdf_grid, + "sdf_surf_grid": sdf_surf_grid, + "sdf_nodes": sdf_nodes, + "surface_mesh_centers": surface_coordinates, + "surface_mesh_neighbors": surface_neighbors, + "surface_normals": surface_normals, + "surface_neighbors_normals": surface_neighbors_normals, + "surface_areas": surface_sizes, + "surface_neighbors_areas": surface_neighbors_sizes, + "volume_fields": volume_fields, + "volume_mesh_centers": volume_coordinates, + "surface_fields": surface_fields, + "volume_min_max": vol_grid_max_min, + "surface_min_max": surf_grid_max_min, + "length_scale": np.array(length_scale, dtype=np.float32), + "global_params_values": np.expand_dims( + np.array(global_params_values, dtype=np.float32), -1 + ), + "global_params_reference": np.expand_dims( + np.array(global_params_reference, dtype=np.float32), -1 + ), + } + elif model_type == "surface": + data_dict = { + "pos_surface_center_of_mass": np.float32(pos_surface_center_of_mass), + "geometry_coordinates": np.float32(geom_centers), + "surf_grid": np.float32(surf_grid), + "sdf_surf_grid": np.float32(sdf_surf_grid), + "surface_mesh_centers": np.float32(surface_coordinates), + "surface_mesh_neighbors": np.float32(surface_neighbors), + "surface_normals": np.float32(surface_normals), + "surface_neighbors_normals": np.float32(surface_neighbors_normals), + "surface_areas": np.float32(surface_sizes), + "surface_neighbors_areas": np.float32(surface_neighbors_sizes), + "surface_fields": np.float32(surface_fields), + "surface_min_max": np.float32(surf_grid_max_min), + "length_scale": np.array(length_scale, dtype=np.float32), + "global_params_values": np.expand_dims( + np.array(global_params_values, dtype=np.float32), -1 + ), + "global_params_reference": np.expand_dims( + np.array(global_params_reference, dtype=np.float32), -1 + ), + } + elif model_type == "volume": + data_dict = { + "pos_volume_closest": pos_volume_closest, + "pos_volume_center_of_mass": pos_volume_center_of_mass, + "geometry_coordinates": geom_centers, + "grid": grid, + "surf_grid": surf_grid, + "sdf_grid": sdf_grid, + "sdf_surf_grid": sdf_surf_grid, + "sdf_nodes": sdf_nodes, + "volume_fields": volume_fields, + "volume_mesh_centers": volume_coordinates, + "volume_min_max": vol_grid_max_min, + "surface_min_max": surf_grid_max_min, + "length_scale": np.array(length_scale, dtype=np.float32), + "global_params_values": np.expand_dims( + np.array(global_params_values, dtype=np.float32), -1 + ), + "global_params_reference": np.expand_dims( + np.array(global_params_reference, dtype=np.float32), -1 + ), + } + + data_dict = { + key: torch.from_numpy(np.expand_dims(np.float32(value), 0)) + for key, value in data_dict.items() + } + + prediction_vol, prediction_surf = test_step( + data_dict, model, dist.device, cfg, vol_factors, surf_factors + ) + + if prediction_surf is not None: + surface_sizes = np.expand_dims(surface_sizes, -1) + prediction_surf[0, :, 0] = ( + prediction_surf[0, :, 0] + surface_fields_base[:, 0] + ) + prediction_surf[0, :, 1:] = ( + prediction_surf[0, :, 1:] + surface_fields_base[:, 1:] + ) + + pres_x_pred = np.sum( + prediction_surf[0, :, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + ) + shear_x_pred = np.sum(prediction_surf[0, :, 1] * surface_sizes[:, 0]) + + pres_x_true = np.sum( + surface_fields[:, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + ) + shear_x_true = np.sum(surface_fields[:, 1] * surface_sizes[:, 0]) + + force_x_pred = np.sum( + prediction_surf[0, :, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + - prediction_surf[0, :, 1] * surface_sizes[:, 0] + ) + force_x_true = np.sum( + surface_fields[:, 0] * surface_normals[:, 0] * surface_sizes[:, 0] + - surface_fields[:, 1] * surface_sizes[:, 0] + ) + + force_y_pred = np.sum( + prediction_surf[0, :, 0] * surface_normals[:, 1] * surface_sizes[:, 0] + - prediction_surf[0, :, 2] * surface_sizes[:, 0] + ) + force_y_true = np.sum( + surface_fields[:, 0] * surface_normals[:, 1] * surface_sizes[:, 0] + - surface_fields[:, 2] * surface_sizes[:, 0] + ) + + force_z_pred = np.sum( + prediction_surf[0, :, 0] * surface_normals[:, 2] * surface_sizes[:, 0] + - prediction_surf[0, :, 3] * surface_sizes[:, 0] + ) + force_z_true = np.sum( + surface_fields[:, 0] * surface_normals[:, 2] * surface_sizes[:, 0] + - surface_fields[:, 3] * surface_sizes[:, 0] + ) + print("Drag=", dirname, force_x_pred, force_x_true) + print("Lift=", dirname, force_z_pred, force_z_true) + print("Side=", dirname, force_y_pred, force_y_true) + aero_forces_all.append( + [ + dirname, + force_x_pred, + force_x_true, + force_z_pred, + force_z_true, + force_y_pred, + force_y_true, + ] + ) + + l2_gt = np.mean(np.square(surface_fields), (0)) + l2_error = np.mean(np.square(prediction_surf[0] - surface_fields), (0)) + l2_surface_all.append(np.sqrt(l2_error / l2_gt)) + + print( + "Surface L-2 norm:", + dirname, + np.sqrt(l2_error) / np.sqrt(l2_gt), + ) + + if prediction_vol is not None: + target_vol = volume_fields + prediction_vol = prediction_vol[0] + prediction_vol[:, :3] = prediction_vol[:, :3] + volume_fields_base[:, :3] + prediction_vol[:, 3:4] = prediction_vol[:, 3:4] + volume_fields_base[:, 3:4] + prediction_vol[:, 4:5] = prediction_vol[:, 4:5] + volume_fields_base[:, 4:5] + c_min = vol_grid_max_min[0] + c_max = vol_grid_max_min[1] + volume_coordinates = unnormalize(volume_coordinates, c_max, c_min) + ids_in_bbox = np.where( + (volume_coordinates[:, 0] < c_min[0]) + | (volume_coordinates[:, 0] > c_max[0]) + | (volume_coordinates[:, 1] < c_min[1]) + | (volume_coordinates[:, 1] > c_max[1]) + | (volume_coordinates[:, 2] < c_min[2]) + | (volume_coordinates[:, 2] > c_max[2]) + ) + target_vol[ids_in_bbox] = 0.0 + prediction_vol[ids_in_bbox] = 0.0 + l2_gt = np.mean(np.square(target_vol), (0)) + l2_error = np.mean(np.square(prediction_vol - target_vol), (0)) + print( + "Volume L-2 norm:", + dirname, + np.sqrt(l2_error) / np.sqrt(l2_gt), + ) + l2_volume_all.append(np.sqrt(l2_error) / np.sqrt(l2_gt)) + + if prediction_surf is not None: + surfParam_vtk = numpy_support.numpy_to_vtk(prediction_surf[0, :, 0:1]) + surfParam_vtk.SetName(f"{surface_variable_names_gt[0]}Pred") + celldata_all.GetCellData().AddArray(surfParam_vtk) + + surfParam_vtk = numpy_support.numpy_to_vtk(prediction_surf[0, :, 1:]) + surfParam_vtk.SetName(f"{surface_variable_names_gt[1]}Pred") + celldata_all.GetCellData().AddArray(surfParam_vtk) + + write_to_vtp(celldata_all, vtp_pred_save_path) + + if prediction_vol is not None: + volParam_vtk = numpy_support.numpy_to_vtk(prediction_vol[:, 0:3]) + volParam_vtk.SetName(f"{volume_variable_names_gt[0]}Pred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk(prediction_vol[:, 3:4]) + volParam_vtk.SetName(f"{volume_variable_names_gt[1]}Pred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + volParam_vtk = numpy_support.numpy_to_vtk(prediction_vol[:, 4:5]) + volParam_vtk.SetName(f"{volume_variable_names_gt[2]}Pred") + polydata_vol.GetPointData().AddArray(volParam_vtk) + + write_to_vtu(polydata_vol, vtu_pred_save_path) + + l2_surface_all = np.asarray(l2_surface_all) # num_files, 4 + l2_volume_all = np.asarray(l2_volume_all) # num_files, 4 + l2_surface_mean = np.mean(l2_surface_all, 0) + l2_volume_mean = np.mean(l2_volume_all, 0) + print( + f"Mean over all samples, surface={l2_surface_mean} and volume={l2_volume_mean}" + ) + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/train.py b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/train.py new file mode 100644 index 0000000000..278b3de8bf --- /dev/null +++ b/examples/cfd/external_aerodynamics/domino_nim_finetuning/src/train.py @@ -0,0 +1,1091 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +""" +This code defines a distributed pipeline for training the DoMINO model on +CFD datasets. It includes the computation of scaling factors, instantiating +the DoMINO model and datapipe, automatically loading the most recent checkpoint, +training the model in parallel using DistributedDataParallel across multiple +GPUs, calculating the loss and updating model parameters using mixed precision. +This is a common recipe that enables training of combined models for surface and +volume as well either of them separately. Validation is also conducted every epoch, +where predictions are compared against ground truth values. The code logs training +and validation metrics to TensorBoard. The train tab in config.yaml can be used to +specify batch size, number of epochs and other training parameters. +""" + +import time +import os +import re +import torch +import torchinfo + +from typing import Literal, Any + +import apex +import numpy as np +import hydra +from hydra.utils import to_absolute_path +from omegaconf import DictConfig, OmegaConf +import torch.distributed as dist +from torch.cuda.amp import GradScaler, autocast +from torch.nn.parallel import DistributedDataParallel +from torch.utils.data import DataLoader +from torch.utils.data.distributed import DistributedSampler +from torch.utils.tensorboard import SummaryWriter +from nvtx import annotate as nvtx_annotate +import torch.cuda.nvtx as nvtx + + +from physicsnemo.distributed import DistributedManager +from physicsnemo.utils import load_checkpoint, save_checkpoint +from physicsnemo.utils.logging import PythonLogger, RankZeroLoggingWrapper + +from physicsnemo.datapipes.cae.domino_datapipe import ( + DoMINODataPipe, + compute_scaling_factors, + create_domino_dataset, +) +from physicsnemo.models.domino.model import DoMINO +from physicsnemo.models.domino.utils import * + +# This is included for GPU memory tracking: +from pynvml import nvmlInit, nvmlDeviceGetHandleByIndex, nvmlDeviceGetMemoryInfo +import time + +# Initialize NVML +nvmlInit() + + +from physicsnemo.utils.profiling import profile, Profiler + + +# Profiler().enable("line_profiler") +# Profiler().initialize() + + +def compute_physics_loss( + output: torch.Tensor, + target: torch.Tensor, + mask: torch.Tensor, + loss_type: Literal["mse", "rmse"], + dims: tuple[int, ...] | None, + first_deriv: torch.nn.Module, + eqn: Any, + bounding_box: torch.Tensor, + vol_factors: torch.Tensor, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Compute physics-based loss terms for Navier-Stokes equations. + + Args: + output: Model output containing (output, coords_neighbors, output_neighbors, neighbors_list) + target: Ground truth values + mask: Mask for valid values + loss_type: Type of loss to calculate ("mse" or "rmse") + dims: Dimensions for loss calculation + first_deriv: First derivative calculator + eqn: Equations + bounding_box: Bounding box for normalization + vol_factors: Volume factors for normalization + + Returns: + Tuple of (data_loss, continuity_loss, momentum_x_loss, momentum_y_loss, momentum_z_loss) + """ + # Physics loss enabled + output, coords_neighbors, output_neighbors, neighbors_list = output + batch_size = output.shape[1] + fields, num_neighbors = output_neighbors.shape[3], output_neighbors.shape[2] + coords_total = coords_neighbors[0, :] + output_total = output_neighbors[0, :] + output_total_unnormalized = unnormalize( + output_total, vol_factors[0], vol_factors[1] + ) + coords_total_unnormalized = unnormalize( + coords_total, bounding_box[0], bounding_box[1] + ) + + # compute first order gradients on all the nodes from the neighbors_list + grad_list = {} + for parent_id, neighbor_ids in neighbors_list.items(): + neighbor_ids_tensor = torch.tensor(neighbor_ids).to( + output_total_unnormalized.device + ) + du = ( + output_total_unnormalized[:, [parent_id]] + - output_total_unnormalized[:, neighbor_ids_tensor] + ) + dv = ( + coords_total_unnormalized[:, [parent_id]] + - coords_total_unnormalized[:, neighbor_ids_tensor] + ) + grads = first_deriv.forward( + coords=None, connectivity_tensor=None, y=None, du=du, dv=dv + ) + grad = torch.cat(grads, dim=1) + grad_list[parent_id] = grad + + # compute second order gradients on only the center node + neighbor_ids_tensor = torch.tensor(neighbors_list[0]).to( + output_total_unnormalized.device + ) + grad_neighbors_center = torch.stack([v for v in grad_list.values()], dim=1) + grad_neighbors_center = grad_neighbors_center.reshape( + batch_size, len(neighbors_list[0]) + 1, -1 + ) + + du = grad_neighbors_center[:, [0]] - grad_neighbors_center[:, neighbor_ids_tensor] + dv = ( + coords_total_unnormalized[:, [0]] + - coords_total_unnormalized[:, neighbor_ids_tensor] + ) + + # second order gradients + ggrads_center = first_deriv.forward( + coords=None, connectivity_tensor=None, y=None, du=du, dv=dv + ) + ggrad_center = torch.cat(ggrads_center, dim=1) + grad_neighbors_center = grad_neighbors_center.reshape( + batch_size, len(neighbors_list[0]) + 1, 3, -1 + ) + + # Get the outputs on the original nodes + fields_center_unnormalized = output_total_unnormalized[:, 0, :] + grad_center = grad_neighbors_center[:, 0, :, :] + grad_grad_uvw_center = ggrad_center[:, :, :9] + + nu = 1.507 * 1e-5 + + dict_mapping = { + "u": fields_center_unnormalized[:, [0]], + "v": fields_center_unnormalized[:, [1]], + "w": fields_center_unnormalized[:, [2]], + "p": fields_center_unnormalized[:, [3]], + "nu": nu + fields_center_unnormalized[:, [4]], + "u__x": grad_center[:, 0, [0]], + "u__y": grad_center[:, 1, [0]], + "u__z": grad_center[:, 2, [0]], + "v__x": grad_center[:, 0, [1]], + "v__y": grad_center[:, 1, [1]], + "v__z": grad_center[:, 2, [1]], + "w__x": grad_center[:, 0, [2]], + "w__y": grad_center[:, 1, [2]], + "w__z": grad_center[:, 2, [2]], + "p__x": grad_center[:, 0, [3]], + "p__y": grad_center[:, 1, [3]], + "p__z": grad_center[:, 2, [3]], + "nu__x": grad_center[:, 0, [4]], + "nu__y": grad_center[:, 1, [4]], + "nu__z": grad_center[:, 2, [4]], + "u__x__x": grad_grad_uvw_center[:, 0, [0]], + "u__x__y": grad_grad_uvw_center[:, 1, [0]], + "u__x__z": grad_grad_uvw_center[:, 2, [0]], + "u__y__x": grad_grad_uvw_center[:, 1, [0]], # same as __x__y + "u__y__y": grad_grad_uvw_center[:, 1, [1]], + "u__y__z": grad_grad_uvw_center[:, 2, [1]], + "u__z__x": grad_grad_uvw_center[:, 2, [0]], # same as __x__z + "u__z__y": grad_grad_uvw_center[:, 2, [1]], # same as __y__z + "u__z__z": grad_grad_uvw_center[:, 2, [2]], + "v__x__x": grad_grad_uvw_center[:, 0, [3]], + "v__x__y": grad_grad_uvw_center[:, 1, [3]], + "v__x__z": grad_grad_uvw_center[:, 2, [3]], + "v__y__x": grad_grad_uvw_center[:, 1, [3]], # same as __x__y + "v__y__y": grad_grad_uvw_center[:, 1, [4]], + "v__y__z": grad_grad_uvw_center[:, 2, [4]], + "v__z__x": grad_grad_uvw_center[:, 2, [3]], # same as __x__z + "v__z__y": grad_grad_uvw_center[:, 2, [4]], # same as __y__z + "v__z__z": grad_grad_uvw_center[:, 2, [5]], + "w__x__x": grad_grad_uvw_center[:, 0, [6]], + "w__x__y": grad_grad_uvw_center[:, 1, [6]], + "w__x__z": grad_grad_uvw_center[:, 2, [6]], + "w__y__x": grad_grad_uvw_center[:, 1, [6]], # same as __x__y + "w__y__y": grad_grad_uvw_center[:, 1, [7]], + "w__y__z": grad_grad_uvw_center[:, 2, [7]], + "w__z__x": grad_grad_uvw_center[:, 2, [6]], # same as __x__z + "w__z__y": grad_grad_uvw_center[:, 2, [7]], # same as __y__z + "w__z__z": grad_grad_uvw_center[:, 2, [8]], + } + continuity = eqn["continuity"].evaluate(dict_mapping)["continuity"] + momentum_x = eqn["momentum_x"].evaluate(dict_mapping)["momentum_x"] + momentum_y = eqn["momentum_y"].evaluate(dict_mapping)["momentum_y"] + momentum_z = eqn["momentum_z"].evaluate(dict_mapping)["momentum_z"] + + # Compute the weights for the equation residuals + weight_continuity = torch.sigmoid(0.5 * (torch.abs(continuity) - 10)) + weight_momentum_x = torch.sigmoid(0.5 * (torch.abs(momentum_x) - 10)) + weight_momentum_y = torch.sigmoid(0.5 * (torch.abs(momentum_y) - 10)) + weight_momentum_z = torch.sigmoid(0.5 * (torch.abs(momentum_z) - 10)) + + weighted_continuity = weight_continuity * torch.abs(continuity) + weighted_momentum_x = weight_momentum_x * torch.abs(momentum_x) + weighted_momentum_y = weight_momentum_y * torch.abs(momentum_y) + weighted_momentum_z = weight_momentum_z * torch.abs(momentum_z) + + # Compute data loss + num = torch.sum(mask * (output - target) ** 2.0, dims) + if loss_type == "rmse": + denom = torch.sum(mask * target**2.0, dims) + else: + denom = torch.sum(mask) + + del coords_total, output_total + torch.cuda.empty_cache() + + return ( + torch.mean(num / denom), + torch.mean(torch.abs(weighted_continuity)), + torch.mean(torch.abs(weighted_momentum_x)), + torch.mean(torch.abs(weighted_momentum_y)), + torch.mean(torch.abs(weighted_momentum_z)), + ) + + +def loss_fn( + output: torch.Tensor, + target: torch.Tensor, + loss_type: Literal["mse", "rmse"], + padded_value: float = -10, +) -> torch.Tensor: + """Calculate mean squared error or root mean squared error with masking for padded values. + + Args: + output: Predicted values from the model + target: Ground truth values + loss_type: Type of loss to calculate ("mse" or "rmse") + padded_value: Value used for padding in the tensor + + Returns: + Calculated loss as a scalar tensor + """ + mask = abs(target - padded_value) > 1e-3 + + if loss_type == "rmse": + dims = (0, 1) + else: + dims = None + + num = torch.sum(mask * (output - target) ** 2.0, dims) + if loss_type == "rmse": + denom = torch.sum(mask * target**2.0, dims) + loss = torch.mean(torch.sqrt(num / denom)) + elif loss_type == "mse": + denom = torch.sum(mask) + loss = torch.mean(num / denom) + else: + raise ValueError(f"Invalid loss type: {loss_type}") + return loss + + +def loss_fn_with_physics( + output: torch.Tensor, + target: torch.Tensor, + loss_type: Literal["mse", "rmse"], + padded_value: float = -10, + first_deriv: torch.nn.Module = None, + eqn: Any = None, + bounding_box: torch.Tensor = None, + vol_factors: torch.Tensor = None, +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: + """Calculate loss with physics-based terms for appropriate equations. + + Args: + output: Predicted values from the model (with neighbor data when physics enabled) + target: Ground truth values + loss_type: Type of loss to calculate ("mse" or "rmse") + padded_value: Value used for padding in the tensor + first_deriv: First derivative calculator + eqn: Equations + bounding_box: Bounding box for normalization + vol_factors: Volume factors for normalization + + Returns: + Tuple of (data_loss, continuity_loss, momentum_x_loss, momentum_y_loss, momentum_z_loss) + """ + mask = abs(target - padded_value) > 1e-3 + + if loss_type == "rmse": + dims = (0, 1) + else: + dims = None + + # Call the physics loss computation function + return compute_physics_loss( + output=output, + target=target, + mask=mask, + loss_type=loss_type, + dims=dims, + first_deriv=first_deriv, + eqn=eqn, + bounding_box=bounding_box, + vol_factors=vol_factors, + ) + + +def loss_fn_surface( + output: torch.Tensor, target: torch.Tensor, loss_type: Literal["mse", "rmse"] +) -> torch.Tensor: + """Calculate loss for surface data by handling scalar and vector components separately. + + Args: + output: Predicted surface values from the model + target: Ground truth surface values + loss_type: Type of loss to calculate ("mse" or "rmse") + + Returns: + Combined scalar and vector loss as a scalar tensor + """ + # Separate the scalar and vector components: + output_scalar, output_vector = torch.split(output, [1, 3], dim=2) + target_scalar, target_vector = torch.split(target, [1, 3], dim=2) + + numerator = torch.mean((output_scalar - target_scalar) ** 2.0) + vector_diff_sq = torch.mean((target_vector - output_vector) ** 2.0, (0, 1)) + if loss_type == "mse": + masked_loss_pres = numerator + masked_loss_ws = torch.sum(vector_diff_sq) + else: + denom = torch.mean((target_scalar) ** 2.0) + masked_loss_pres = numerator / denom + + # Compute the mean diff**2 of the vector component, leave the last dimension: + masked_loss_ws_num = vector_diff_sq + masked_loss_ws_denom = torch.mean((target_vector) ** 2.0, (0, 1)) + masked_loss_ws = torch.sum(masked_loss_ws_num / masked_loss_ws_denom) + + loss = masked_loss_pres + masked_loss_ws + + return loss / 4.0 + + +def loss_fn_area( + output: torch.Tensor, + target: torch.Tensor, + normals: torch.Tensor, + area: torch.Tensor, + area_scaling_factor: float, + loss_type: Literal["mse", "rmse"], +) -> torch.Tensor: + """Calculate area-weighted loss for surface data considering normal vectors. + + Args: + output: Predicted surface values from the model + target: Ground truth surface values + normals: Normal vectors for the surface + area: Area values for surface elements + area_scaling_factor: Scaling factor for area weighting + loss_type: Type of loss to calculate ("mse" or "rmse") + + Returns: + Area-weighted loss as a scalar tensor + """ + area = area * area_scaling_factor + area_scale_factor = area + + # Separate the scalar and vector components. + target_scalar, target_vector = torch.split( + target * area_scale_factor, [1, 3], dim=2 + ) + output_scalar, output_vector = torch.split( + output * area_scale_factor, [1, 3], dim=2 + ) + + # Apply the normals to the scalar components (only [:,:,0]): + normals, _ = torch.split(normals, [1, normals.shape[-1] - 1], dim=2) + target_scalar = target_scalar * normals + output_scalar = output_scalar * normals + + # Compute the mean diff**2 of the scalar component: + masked_loss_pres = torch.mean(((output_scalar - target_scalar) ** 2.0), dim=(0, 1)) + if loss_type == "rmse": + masked_loss_pres /= torch.mean(target_scalar**2.0, dim=(0, 1)) + + # Compute the mean diff**2 of the vector component, leave the last dimension: + masked_loss_ws = torch.mean((target_vector - output_vector) ** 2.0, (0, 1)) + + if loss_type == "rmse": + masked_loss_ws /= torch.mean((target_vector) ** 2.0, (0, 1)) + + # Combine the scalar and vector components: + loss = 0.25 * (masked_loss_pres + torch.sum(masked_loss_ws)) + + return loss + + +def integral_loss_fn( + output, target, area, normals, stream_velocity=None, padded_value=-10 +): + drag_loss = drag_loss_fn( + output, target, area, normals, stream_velocity=stream_velocity, padded_value=-10 + ) + lift_loss = lift_loss_fn( + output, target, area, normals, stream_velocity=stream_velocity, padded_value=-10 + ) + return lift_loss + drag_loss + + +def lift_loss_fn(output, target, area, normals, stream_velocity=None, padded_value=-10): + vel_inlet = stream_velocity # Get this from the dataset + mask = abs(target - padded_value) > 1e-3 + + output_true = target * mask * area * (vel_inlet) ** 2.0 + output_pred = output * mask * area * (vel_inlet) ** 2.0 + + normals = torch.select(normals, 2, 2) + # output_true_0 = output_true[:, :, 0] + output_true_0 = output_true.select(2, 0) + output_pred_0 = output_pred.select(2, 0) + + pres_true = output_true_0 * normals + pres_pred = output_pred_0 * normals + + wz_true = output_true[:, :, -1] + wz_pred = output_pred[:, :, -1] + + masked_pred = torch.mean(pres_pred + wz_pred, (1)) + masked_truth = torch.mean(pres_true + wz_true, (1)) + + loss = (masked_pred - masked_truth) ** 2.0 + loss = torch.mean(loss) + return loss + + +def drag_loss_fn(output, target, area, normals, stream_velocity=None, padded_value=-10): + vel_inlet = stream_velocity # Get this from the dataset + mask = abs(target - padded_value) > 1e-3 + output_true = target * mask * area * (vel_inlet) ** 2.0 + output_pred = output * mask * area * (vel_inlet) ** 2.0 + + pres_true = output_true[:, :, 0] * normals[:, :, 0] + pres_pred = output_pred[:, :, 0] * normals[:, :, 0] + + wx_true = output_true[:, :, 1] + wx_pred = output_pred[:, :, 1] + + masked_pred = torch.mean(pres_pred + wx_pred, (1)) + masked_truth = torch.mean(pres_true + wx_true, (1)) + + loss = (masked_pred - masked_truth) ** 2.0 + loss = torch.mean(loss) + return loss + + +def compute_loss_dict( + prediction_vol: torch.Tensor, + prediction_surf: torch.Tensor, + batch_inputs: dict, + loss_fn_type: dict, + integral_scaling_factor: float, + surf_loss_scaling: float, + vol_loss_scaling: float, + first_deriv: torch.nn.Module | None = None, + eqn: Any = None, + bounding_box: torch.Tensor | None = None, + vol_factors: torch.Tensor | None = None, + add_physics_loss: bool = False, +) -> tuple[torch.Tensor, dict]: + """ + Compute the loss terms in a single function call. + + Computes: + - Volume loss if prediction_vol is not None + - Surface loss if prediction_surf is not None + - Integral loss if prediction_surf is not None + - Total loss as a weighted sum of the above + + Returns: + - Total loss as a scalar tensor + - Dictionary of loss terms (for logging, etc) + """ + nvtx.range_push("Loss Calculation") + total_loss_terms = [] + loss_dict = {} + + if prediction_vol is not None: + target_vol = batch_inputs["volume_fields"] + + if add_physics_loss: + loss_vol = loss_fn_with_physics( + prediction_vol, + target_vol, + loss_fn_type.loss_type, + padded_value=-10, + first_deriv=first_deriv, + eqn=eqn, + bounding_box=bounding_box, + vol_factors=vol_factors, + ) + loss_dict["loss_vol"] = loss_vol[0] + loss_dict["loss_continuity"] = loss_vol[1] + loss_dict["loss_momentum_x"] = loss_vol[2] + loss_dict["loss_momentum_y"] = loss_vol[3] + loss_dict["loss_momentum_z"] = loss_vol[4] + total_loss_terms.append(loss_vol[0]) + total_loss_terms.append(loss_vol[1]) + total_loss_terms.append(loss_vol[2]) + total_loss_terms.append(loss_vol[3]) + total_loss_terms.append(loss_vol[4]) + else: + loss_vol = loss_fn( + prediction_vol, + target_vol, + loss_fn_type.loss_type, + padded_value=-10, + ) + loss_dict["loss_vol"] = loss_vol + total_loss_terms.append(loss_vol) + + if prediction_surf is not None: + target_surf = batch_inputs["surface_fields"] + surface_areas = batch_inputs["surface_areas"] + surface_areas = torch.unsqueeze(surface_areas, -1) + surface_normals = batch_inputs["surface_normals"] + + # Needs to be taken from the dataset + stream_velocity = batch_inputs["global_params_values"][:, 0, :] + + loss_surf = loss_fn_surface( + prediction_surf, + target_surf, + loss_fn_type.loss_type, + ) + + loss_surf_area = loss_fn_area( + prediction_surf, + target_surf, + surface_normals, + surface_areas, + area_scaling_factor=loss_fn_type.area_weighing_factor, + loss_type=loss_fn_type.loss_type, + ) + + if loss_fn_type.loss_type == "mse": + loss_surf = loss_surf * surf_loss_scaling + loss_surf_area = loss_surf_area * surf_loss_scaling + + total_loss_terms.append(loss_surf) + loss_dict["loss_surf"] = loss_surf + total_loss_terms.append(loss_surf_area) + loss_dict["loss_surf_area"] = loss_surf_area + loss_integral = ( + integral_loss_fn( + prediction_surf, + target_surf, + surface_areas, + surface_normals, + stream_velocity, + padded_value=-10, + ) + ) * integral_scaling_factor + loss_dict["loss_integral"] = loss_integral + total_loss_terms.append(loss_integral) + + total_loss = sum(total_loss_terms) + loss_dict["total_loss"] = total_loss + nvtx.range_pop() + + return total_loss, loss_dict + + +def validation_step( + dataloader, + model, + device, + logger, + use_sdf_basis=False, + use_surface_normals=False, + integral_scaling_factor=1.0, + loss_fn_type=None, + vol_loss_scaling=None, + surf_loss_scaling=None, + first_deriv: torch.nn.Module | None = None, + eqn: Any = None, + bounding_box: torch.Tensor | None = None, + vol_factors: torch.Tensor | None = None, + add_physics_loss=False, +): + running_vloss = 0.0 + with torch.no_grad(): + for i_batch, sample_batched in enumerate(dataloader): + sampled_batched = dict_to_device(sample_batched, device) + + with autocast(enabled=False): + if add_physics_loss: + prediction_vol, prediction_surf = model( + sampled_batched, return_volume_neighbors=True + ) + else: + prediction_vol, prediction_surf = model(sampled_batched) + + loss, loss_dict = compute_loss_dict( + prediction_vol, + prediction_surf, + sampled_batched, + loss_fn_type, + integral_scaling_factor, + surf_loss_scaling, + vol_loss_scaling, + first_deriv, + eqn, + bounding_box, + vol_factors, + add_physics_loss, + ) + + running_vloss += loss.item() + + avg_vloss = running_vloss / (i_batch + 1) + + return avg_vloss + + +@profile +def train_epoch( + dataloader, + model, + optimizer, + scaler, + tb_writer, + logger, + gpu_handle, + epoch_index, + device, + integral_scaling_factor, + loss_fn_type, + vol_loss_scaling=None, + surf_loss_scaling=None, + first_deriv: torch.nn.Module | None = None, + eqn: Any = None, + bounding_box: torch.Tensor | None = None, + vol_factors: torch.Tensor | None = None, + add_physics_loss=False, +): + dist = DistributedManager() + + running_loss = 0.0 + last_loss = 0.0 + loss_interval = 1 + + gpu_start_info = nvmlDeviceGetMemoryInfo(gpu_handle) + start_time = time.perf_counter() + for i_batch, sample_batched in enumerate(dataloader): + sampled_batched = dict_to_device(sample_batched, device) + + if add_physics_loss: + autocast_enabled = False + else: + autocast_enabled = True + with autocast(enabled=autocast_enabled): + with nvtx.range("Model Forward Pass"): + if add_physics_loss: + prediction_vol, prediction_surf = model( + sampled_batched, return_volume_neighbors=True + ) + else: + prediction_vol, prediction_surf = model(sampled_batched) + + loss, loss_dict = compute_loss_dict( + prediction_vol, + prediction_surf, + sampled_batched, + loss_fn_type, + integral_scaling_factor, + surf_loss_scaling, + vol_loss_scaling, + first_deriv, + eqn, + bounding_box, + vol_factors, + add_physics_loss, + ) + + loss = loss / loss_interval + scaler.scale(loss).backward() + + if ((i_batch + 1) % loss_interval == 0) or (i_batch + 1 == len(dataloader)): + scaler.step(optimizer) + scaler.update() + optimizer.zero_grad() + + # Gather data and report + running_loss += loss.item() + elapsed_time = time.perf_counter() - start_time + start_time = time.perf_counter() + gpu_end_info = nvmlDeviceGetMemoryInfo(gpu_handle) + gpu_memory_used = gpu_end_info.used / (1024**3) + gpu_memory_delta = (gpu_end_info.used - gpu_start_info.used) / (1024**3) + + logging_string = f"Device {device}, batch processed: {i_batch + 1}\n" + # Format the loss dict into a string: + loss_string = ( + " " + + "\t".join([f"{key.replace('loss_', ''):<10}" for key in loss_dict.keys()]) + + "\n" + ) + loss_string += ( + " " + f"\t".join([f"{l.item():<10.3e}" for l in loss_dict.values()]) + "\n" + ) + + logging_string += loss_string + logging_string += f" GPU memory used: {gpu_memory_used:.3f} Gb\n" + logging_string += f" GPU memory delta: {gpu_memory_delta:.3f} Gb\n" + logging_string += f" Time taken: {elapsed_time:.2f} seconds\n" + logger.info(logging_string) + gpu_start_info = nvmlDeviceGetMemoryInfo(gpu_handle) + + last_loss = running_loss / (i_batch + 1) # loss per batch + if dist.rank == 0: + logger.info( + f" Device {device}, batch: {i_batch + 1}, loss norm: {loss.item():.5f}" + ) + tb_x = epoch_index * len(dataloader) + i_batch + 1 + tb_writer.add_scalar("Loss/train", last_loss, tb_x) + + return last_loss + + +@hydra.main(version_base="1.3", config_path="conf", config_name="config") +def main(cfg: DictConfig) -> None: + # initialize distributed manager + DistributedManager.initialize() + dist = DistributedManager() + + # Initialize NVML + nvmlInit() + + gpu_handle = nvmlDeviceGetHandleByIndex(dist.device.index) + + compute_scaling_factors( + cfg=cfg, + input_path=cfg.data.input_dir, + use_cache=cfg.data_processor.use_cache, + ) + model_type = cfg.model.model_type + + logger = PythonLogger("Train") + logger = RankZeroLoggingWrapper(logger, dist) + + logger.info(f"Config summary:\n{OmegaConf.to_yaml(cfg, sort_keys=True)}") + + # Get physics imports conditionally + add_physics_loss = getattr(cfg.train, "add_physics_loss", False) + + if add_physics_loss: + from physicsnemo.sym.eq.pde import PDE + from physicsnemo.sym.eq.ls.grads import FirstDeriv + from physicsnemo.sym.eq.pdes.navier_stokes import IncompressibleNavierStokes + else: + PDE = FirstDeriv = IncompressibleNavierStokes = None + + num_vol_vars = 0 + volume_variable_names = [] + if model_type == "volume" or model_type == "combined": + volume_variable_names = list(cfg.variables.volume.solution.keys()) + for j in volume_variable_names: + if cfg.variables.volume.solution[j] == "vector": + num_vol_vars += 3 + else: + num_vol_vars += 1 + else: + num_vol_vars = None + + num_surf_vars = 0 + surface_variable_names = [] + if model_type == "surface" or model_type == "combined": + surface_variable_names = list(cfg.variables.surface.solution.keys()) + num_surf_vars = 0 + for j in surface_variable_names: + if cfg.variables.surface.solution[j] == "vector": + num_surf_vars += 3 + else: + num_surf_vars += 1 + else: + num_surf_vars = None + + num_global_features = 0 + global_params_names = list(cfg.variables.global_parameters.keys()) + for param in global_params_names: + if cfg.variables.global_parameters[param].type == "vector": + num_global_features += len(cfg.variables.global_parameters[param].reference) + elif cfg.variables.global_parameters[param].type == "scalar": + num_global_features += 1 + else: + raise ValueError(f"Unknown global parameter type") + + vol_save_path = os.path.join( + "outputs", cfg.project.name, "volume_scaling_factors.npy" + ) + surf_save_path = os.path.join( + "outputs", cfg.project.name, "surface_scaling_factors.npy" + ) + if os.path.exists(vol_save_path): + vol_factors = np.load(vol_save_path) + vol_factors_tensor = ( + torch.from_numpy(vol_factors).to(dist.device) if add_physics_loss else None + ) + else: + vol_factors = None + vol_factors_tensor = None + + bounding_box = None + if add_physics_loss: + bounding_box = cfg.data.bounding_box + bounding_box = ( + torch.from_numpy( + np.stack([bounding_box["max"], bounding_box["min"]], axis=0) + ) + .to(vol_factors_tensor.dtype) + .to(dist.device) + ) + + if os.path.exists(surf_save_path): + surf_factors = np.load(surf_save_path) + else: + surf_factors = None + + train_dataset = create_domino_dataset( + cfg, + phase="train", + volume_variable_names=volume_variable_names, + surface_variable_names=surface_variable_names, + vol_factors=vol_factors, + surf_factors=surf_factors, + ) + val_dataset = create_domino_dataset( + cfg, + phase="val", + volume_variable_names=volume_variable_names, + surface_variable_names=surface_variable_names, + vol_factors=vol_factors, + surf_factors=surf_factors, + ) + + train_sampler = DistributedSampler( + train_dataset, + num_replicas=dist.world_size, + rank=dist.rank, + **cfg.train.sampler, + ) + + val_sampler = DistributedSampler( + val_dataset, + num_replicas=dist.world_size, + rank=dist.rank, + **cfg.val.sampler, + ) + + train_dataloader = DataLoader( + train_dataset, + sampler=train_sampler, + **cfg.train.dataloader, + ) + val_dataloader = DataLoader( + val_dataset, + sampler=val_sampler, + **cfg.val.dataloader, + ) + + model = DoMINO( + input_features=3, + output_features_vol=num_vol_vars, + output_features_surf=num_surf_vars, + global_features=num_global_features, + model_parameters=cfg.model, + ).to(dist.device) + model = torch.compile(model, disable=True) # TODO make this configurable + + # Print model summary (structure and parmeter count). + logger.info(f"Model summary:\n{torchinfo.summary(model, verbose=0, depth=2)}\n") + + if dist.world_size > 1: + model = DistributedDataParallel( + model, + device_ids=[dist.local_rank], + output_device=dist.device, + broadcast_buffers=dist.broadcast_buffers, + find_unused_parameters=dist.find_unused_parameters, + gradient_as_bucket_view=True, + static_graph=True, + ) + + # optimizer = apex.optimizers.FusedAdam(model.parameters(), lr=0.001) + optimizer = torch.optim.Adam(model.parameters(), lr=0.001) + scheduler = torch.optim.lr_scheduler.MultiStepLR( + optimizer, milestones=[50, 100, 200, 250, 300, 350, 400, 450], gamma=0.5 + ) + + # Initialize physics components conditionally + first_deriv = None + eqn = None + if add_physics_loss: + first_deriv = FirstDeriv(dim=3, direct_input=True) + eqn = IncompressibleNavierStokes(rho=1.226, nu="nu", dim=3, time=False) + eqn = eqn.make_nodes(return_as_dict=True) + + # Initialize the scaler for mixed precision + scaler = GradScaler() + + writer = SummaryWriter(os.path.join(cfg.output, "tensorboard")) + + epoch_number = 0 + + model_save_path = os.path.join(cfg.output, "models") + param_save_path = os.path.join(cfg.output, "param") + best_model_path = os.path.join(model_save_path, "best_model") + if dist.rank == 0: + create_directory(model_save_path) + create_directory(param_save_path) + create_directory(best_model_path) + + if dist.world_size > 1: + torch.distributed.barrier() + + init_epoch = load_checkpoint( + to_absolute_path(cfg.resume_dir), + models=model, + optimizer=optimizer, + scheduler=scheduler, + scaler=scaler, + device=dist.device, + ) + + if init_epoch != 0: + init_epoch += 1 # Start with the next epoch + epoch_number = init_epoch + + # retrive the smallest validation loss if available + numbers = [] + for filename in os.listdir(best_model_path): + match = re.search(r"\d+\.\d*[1-9]\d*", filename) + if match: + number = float(match.group(0)) + numbers.append(number) + + best_vloss = min(numbers) if numbers else 1_000_000.0 + + initial_integral_factor_orig = cfg.model.integral_loss_scaling_factor + + for epoch in range(init_epoch, cfg.train.epochs): + start_time = time.perf_counter() + logger.info(f"Device {dist.device}, epoch {epoch_number}:") + + if epoch == init_epoch and add_physics_loss: + logger.info( + "Physics loss enabled - mixed precision (autocast) will be disabled as physics loss computation is not supported with mixed precision" + ) + + train_sampler.set_epoch(epoch) + val_sampler.set_epoch(epoch) + + initial_integral_factor = initial_integral_factor_orig + + if epoch > 250: + surface_scaling_loss = 1.0 * cfg.model.surf_loss_scaling + else: + surface_scaling_loss = cfg.model.surf_loss_scaling + + model.train(True) + epoch_start_time = time.perf_counter() + avg_loss = train_epoch( + dataloader=train_dataloader, + model=model, + optimizer=optimizer, + scaler=scaler, + tb_writer=writer, + logger=logger, + gpu_handle=gpu_handle, + epoch_index=epoch, + device=dist.device, + integral_scaling_factor=initial_integral_factor, + loss_fn_type=cfg.model.loss_function, + vol_loss_scaling=cfg.model.vol_loss_scaling, + surf_loss_scaling=surface_scaling_loss, + first_deriv=first_deriv, + eqn=eqn, + bounding_box=bounding_box, + vol_factors=vol_factors_tensor, + add_physics_loss=add_physics_loss, + ) + epoch_end_time = time.perf_counter() + logger.info( + f"Device {dist.device}, Epoch {epoch_number} took {epoch_end_time - epoch_start_time:.3f} seconds" + ) + epoch_end_time = time.perf_counter() + + model.eval() + avg_vloss = validation_step( + dataloader=val_dataloader, + model=model, + device=dist.device, + logger=logger, + use_sdf_basis=cfg.model.use_sdf_in_basis_func, + use_surface_normals=cfg.model.use_surface_normals, + integral_scaling_factor=initial_integral_factor, + loss_fn_type=cfg.model.loss_function, + vol_loss_scaling=cfg.model.vol_loss_scaling, + surf_loss_scaling=surface_scaling_loss, + first_deriv=first_deriv, + eqn=eqn, + bounding_box=bounding_box, + vol_factors=vol_factors_tensor, + add_physics_loss=add_physics_loss, + ) + + scheduler.step() + logger.info( + f"Device {dist.device} " + f"LOSS train {avg_loss:.5f} " + f"valid {avg_vloss:.5f} " + f"Current lr {scheduler.get_last_lr()[0]} " + f"Integral factor {initial_integral_factor}" + ) + + if dist.rank == 0: + writer.add_scalars( + "Training vs. Validation Loss", + {"Training": avg_loss, "Validation": avg_vloss}, + epoch_number, + ) + writer.flush() + + # Track best performance, and save the model's state + if dist.world_size > 1: + torch.distributed.barrier() + + if avg_vloss < best_vloss: # This only considers GPU: 0, is that okay? + best_vloss = avg_vloss + + if dist.rank == 0: + print(f"Device {dist.device}, Best val loss {best_vloss}") + + if dist.rank == 0 and (epoch + 1) % cfg.train.checkpoint_interval == 0.0: + save_checkpoint( + to_absolute_path(model_save_path), + models=model, + optimizer=optimizer, + scheduler=scheduler, + scaler=scaler, + epoch=epoch, + ) + + epoch_number += 1 + + if scheduler.get_last_lr()[0] == 1e-6: + print("Training ended") + exit() + + +if __name__ == "__main__": + main() diff --git a/examples/cfd/external_aerodynamics/figconvnet/README.md b/examples/cfd/external_aerodynamics/figconvnet/README.md new file mode 100644 index 0000000000..f1c1744558 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/README.md @@ -0,0 +1,156 @@ +# Factorized Implicit Global Convolution Network + +Computational Fluid Dynamics (CFD) is central to automotive vehicle design, which involves +studying how the car geometry affects the pressure field. +This requires very fine shapes, represented by large 3D point clouds and high accuracy, +which are currently out of reach for deep learning based methods. +As a result, the problem is typically solved with slow numerical solvers. + +We propose **FIGConvUNet** [[1](#references)], a novel architecture that can efficiently +solve CFD problems for large 3D meshes and arbitrary input and output geometries. +FIGConvUNet efficiently combines U-shaped architecture, graph information gathering, +and integration, learning efficient latent representation through the representation +graph voxel layer. +We empirically validate our approach on the industry benchmark +Ahmed body [[2, 3](#references)] and the real-world DrivAerNet dataset [[4](#references)] +based on Volkswagen DrivAer [[5](#references)] dataset, with geometries composed +of 100 thousand and 1 million points, respectively. +We demonstrate a 140k× speed-up compared to GPU-accelerated +computational fluid dynamics (CFD) simulators and over 2× improvement in pressure prediction +over prior deep learning arts. + +## Supported datasets + +The current version of the code supports the following datasets: + +### DrivAerNet + +Both DrivAerNet and DrivAerNet++ datasets [[4](#references)] are supported. +Please follow the instructions on the [dataset GitHub](https://github.com/Mohamedelrefaie/DrivAerNet) +page to download the dataset. + +The corresponding experiment configuration file can be found at: `./configs/experiment/drivaernet/figconv_unet.yaml`. +For more details, refer to the [Training section](#training). + +### DrivAerML + +DrivAerML dataset [[6](#references)] is supported but requires +conversion of the dataset to a more efficient binary format. +This format is supported by models like XAeroNet and FIGConvNet +and represents efficient storage of the original meshes as +partitioned graphs. +For more details on how to convert the original DrivAerML dataset +to partitioned dataset, refer to +[XAeroNet example README](https://github.com/NVIDIA/physicsnemo/tree/main/examples/cfd/external_aerodynamics/xaeronet#training-the-xaeronet-s-model), +steps 1 to 5. + +The binary dataset should have the following structure: + +```text +├─ partitions +│ ├─ graph_partitions_1.bin +│ ├─ graph_partitions_2.bin +│ ├─ ... +├─ test_partitions +│ ├─ graph_partitions_100.bin +│ ├─ graph_partitions_101.bin +│ ├─ ... +├─ validation_partitions +│ ├─ graph_partitions_200.bin +│ ├─ graph_partitions_201.bin +│ ├─ ... +└─ global_stats.json +``` + +The corresponding experiment configuration file can be found at: +`./configs/experiment/drivaerml/figconv_unet.yaml`. + +## Installation + +FIGConvUNet dependencies can be installed with `pip install`, for example: + +```bash +pip install -e .[figconv] +``` + +It is recommended to install these dependencies in a PhysicsNeMo Docker container, +which provides a simple way to run PhysicsNeMo. + +## Training + +FIGConvUNet uses [Hydra](https://hydra.cc/docs/intro/) for experiment configuration. +The following command launches the experiment defined in `drivaernet/figconv_unet` config +using default parameters with the exception of `data.every_n_data` which enables +dataset sampling. + +```bash +python train.py \ + +experiment=drivaernet/figconv_unet \ + data.data_path=./datasets/drivaer/ +``` + +A bit more interesting example demonstrates how other experiment parameters +can be overridden from the command line: + +```bash +python train.py \ + +experiment=drivaernet/figconv_unet \ + data.data_path=./dataset/drivaer/ \ + 'model.hidden_channels=[16, 16, 16, 16]' \ + optimizer=adamw \ + optimizer.lr=0.1 \ + seed=1 \ + train.num_epochs=10 \ + ~loggers.wandb +``` + +Note that for `zsh`, you need to escape `~` in front of `loggers.wandb` with `\~loggers.wandb`. + +In this scenario: + +* some additional dataset and model parameters are overridden. +* optimizer is changed to `AdamW` and its learning rate is set to 0.1. +* number of epochs is set to 10 and `wandb` (Weights & Biases) logger is removed. + +See [Hydra documentation](https://hydra.cc/docs/intro) for more details. + +For the full set of training script options, run the following command: + +```bash +python train.py --help +``` + +In case of issues with Hydra config, you may get a Hydra error message +that is not particularly useful. In such case, use `HYDRA_FULL_ERROR=1` +environment variable: + +```bash +HYDRA_FULL_ERROR=1 python train.py ... +``` + +### Multi-GPU Training + +FIGConvUNet supports training and evaluation on multiple GPUs. +This can be done using `mpirun` or [torchrun](https://pytorch.org/docs/2.0/elastic/run.html) +utilities. For example, to train the previous example on 2 GPUs using MPI: + +```bash +mpirun -np 2 python train.py \ + +experiment=drivaernet/figconv_unet \ + data.data_path=./dataset/drivaer/ \ + 'model.hidden_channels=[16, 16, 16, 16]' \ + optimizer=adamw \ + optimizer.lr=0.1 \ + seed=1 \ + train.num_epochs=10 \ + ~loggers.wandb +``` + +## References + +1. [Factorized Implicit Global Convolution for Automotive Computational Fluid Dynamics Prediction](https://arxiv.org/abs/2502.04317) +2. [Some Salient Features Of The Time-Averaged Ground Vehicle Wake](https://doi.org/10.4271/840300) +3. [Ahmed body wiki](https://www.cfd-online.com/Wiki/Ahmed_body) +4. [DrivAerNet: A Parametric Car Dataset for Data-Driven Aerodynamic Design and Graph-Based Drag Prediction](https://arxiv.org/abs/2403.08055) +5. [Deep Learning for Real-Time Aerodynamic Evaluations of Arbitrary Vehicle Shapes](https://arxiv.org/abs/2108.05798) +6. [DrivAerML: High-Fidelity Computational Fluid Dynamics Dataset for Road-Car External Aerodynamics](https://arxiv.org/abs/2408.11969) diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/base.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/base.yaml new file mode 100644 index 0000000000..9cf9e37703 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/base.yaml @@ -0,0 +1,93 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - /logging/python: default + - override hydra/job_logging: disabled # We use rank-aware logger configuration instead. + - _self_ + +hydra: + run: + dir: ${output} + output_subdir: hydra # Default is .hydra which causes files not being uploaded in W&B. + +train: + num_epochs: 100 + batch_size: 4 + save_interval: 1 # save model for every N'th epoch + num_checkpoints: 2 # number of checkpoints to save + time_limit: null + resume: false + print_interval: 200 # save image for every N'th train data + dataloader: + shuffle: true # can also specify the shuffle buffer size, e.g. shuffle_buffer_size: 100 + num_workers: 0 + pin_memory: true + lr_scheduler_mode: epoch # epoch or iteration. + +eval: + loss: null + run_eval_first: false + interval: 5 # epoch + batch_size: 1 + plot_interval: 20 # save image for every N'th test data + print_interval: 20 # print every N'th test data + dataloader: + shuffle: false + num_workers: 0 + pin_memory: true + +device: cuda + +seed: null + +# The loss should be set in the experiment. +loss: ??? + +# The optimizer should be set in the experiment. +optimizer: ??? + +# Scheduler should be set in the experiment. +lr_scheduler: ??? + +amp: + enabled: false + autocast: + dtype: torch.float16 + scaler: + _target_: torch.cuda.amp.GradScaler + enabled: ${..enabled} + clip_grad: false + grad_max_norm: 2.0 + +output: outputs/${now:%Y-%m-%d}/${now:%H-%M-%S} + +# Loggers. +log_dir: null +loggers: + tensorboard: {} + wandb: + project_name: car-cfd + run_name: default + entity: physicsnemo # nvr-ai-algo + group_name: + mode: online + +log_pointcloud: false # save pointclouds + +# Signal handler +signal_handler: + status_path: ${output}/status.txt diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/data/drivaerml.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/data/drivaerml.yaml new file mode 100644 index 0000000000..07955953da --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/data/drivaerml.yaml @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: src.data.DrivAerMLDataModule +_convert_: all + +data_path: ??? +num_points: 100_000 diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/data/drivaernet.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/data/drivaernet.yaml new file mode 100644 index 0000000000..9505c7f8ae --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/data/drivaernet.yaml @@ -0,0 +1,25 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: src.data.DrivAerNetDataModule +_convert_: all + +data_path: ??? +num_points: 32768 # 2^15 + +preprocessors: + - _target_: src.data.drivaernet_datamodule.DrivAerNetPreprocessor + num_points: ${...num_points} diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/experiment/drivaerml/figconv_unet.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/experiment/drivaerml/figconv_unet.yaml new file mode 100644 index 0000000000..c8cb6da487 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/experiment/drivaerml/figconv_unet.yaml @@ -0,0 +1,58 @@ +# @package _global_ + +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - /data: drivaerml + - /model: figconv_unet_drivaerml + - /loss: mseloss + - /optimizer: adam + - /lr_scheduler: steplr + +train: + batch_size: 8 + num_epochs: 200 + +model: + aabb_max: [ 2.0, 1.8, 2.6] + aabb_min: [-2.0, -1.8, -1.5] + hidden_channels: [16, 16, 16] + kernel_size: 5 + # mlp_channels: [2048, 2048] + neighbor_search_type: radius + num_down_blocks: 1 + num_levels: 2 + pooling_layers: [2] + pooling_type: max + reductions: [mean] + resolution_memory_format_pairs: + - ${res_mem_pair:b_xc_y_z, [ 5, 150, 100]} + - ${res_mem_pair:b_yc_x_z, [250, 3, 100]} + - ${res_mem_pair:b_zc_x_y, [250, 150, 2]} + use_rel_pos_encode: true + +lr_scheduler: + step_size: 50 + +loggers: + wandb: + entity: physicsnemo + project_name: car-cfd + group_name: fignet-drivaerml + run_name: FIGConvNet-level2-16,16,16-res250-150-100-pool-max-2-aabb-20x18x26-ks5-np32768-b8x2 + +seed: 0 diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/experiment/drivaernet/figconv_unet.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/experiment/drivaernet/figconv_unet.yaml new file mode 100644 index 0000000000..1f23c18438 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/experiment/drivaernet/figconv_unet.yaml @@ -0,0 +1,59 @@ +# @package _global_ + +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - /data: drivaernet + - /model: figconv_unet_drivaer + - /loss: mseloss + - /optimizer: adam + - /lr_scheduler: steplr + +train: + batch_size: 8 + num_epochs: 200 + +model: + aabb_max: [ 2.75, 1.5, 1.0] + aabb_min: [-2.75, -1.5, -1.0] + hidden_channels: [16, 16, 16] + in_channels: 1 + kernel_size: 5 + neighbor_search_type: radius + num_down_blocks: 1 + num_levels: 2 + out_channels: 1 + pooling_layers: [2] + pooling_type: max + reductions: [mean] + resolution_memory_format_pairs: + - ${res_mem_pair:b_xc_y_z, [ 5, 150, 100]} + - ${res_mem_pair:b_yc_x_z, [250, 3, 100]} + - ${res_mem_pair:b_zc_x_y, [250, 150, 2]} + use_rel_pos_encode: true + +lr_scheduler: + step_size: 50 + +loggers: + wandb: + entity: physicsnemo + project_name: car-cfd + group_name: fignet + run_name: FIGConvNet-level2-16,16,16-res250-150-100-pool-max-2-aabb-275x15x1-ks5-np32768-b8x2 + +seed: 0 diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/logging/python/default.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/logging/python/default.yaml new file mode 100644 index 0000000000..042855b8a8 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/logging/python/default.yaml @@ -0,0 +1,47 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Standard Python logging configuration, as described here: +# https://docs.python.org/3.10/library/logging.config.html +version: 1 +disable_existing_loggers: false + +output: ??? +rank: ??? +rank0_only: true + +formatters: + default: + format: "[%(asctime)s - %(name)s - %(levelname)s] %(message)s" + datefmt: "%H:%M:%S" + +handlers: + console: + class: logging.StreamHandler + level: ${...loggers.figconv.level} + formatter: default + + file: + class: logging.FileHandler + filename: ${...output}/train_${...rank}.log + level: ${...loggers.figconv.level} + formatter: default + +loggers: + figconv: + handlers: [console, file] + level: INFO + propagate: false diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/loss/celoss.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/celoss.yaml new file mode 100644 index 0000000000..63353e935d --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/celoss.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.nn.CrossEntropyLoss +reduction: mean diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/loss/huberloss.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/huberloss.yaml new file mode 100644 index 0000000000..3efc30497e --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/huberloss.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.nn.HuberLoss +reduction: mean diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/loss/lploss.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/lploss.yaml new file mode 100644 index 0000000000..8950c2c31c --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/lploss.yaml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +lploss: + _target_: src.losses.LpLoss + size_average: true diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/loss/mseloss.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/mseloss.yaml new file mode 100644 index 0000000000..7dcb51b65d --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/mseloss.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.nn.MSELoss +reduction: mean diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/loss/truncatedmseloss.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/truncatedmseloss.yaml new file mode 100644 index 0000000000..91d1ae7dd0 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/loss/truncatedmseloss.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: src.losses.TruncatedMSELoss +reduction: mean diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/cosineannealinglr.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/cosineannealinglr.yaml new file mode 100644 index 0000000000..3c4d8b86b3 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/cosineannealinglr.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.lr_scheduler.CosineAnnealingLR +T_max: ??? diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/onecyclelr.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/onecyclelr.yaml new file mode 100644 index 0000000000..05d97021ed --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/onecyclelr.yaml @@ -0,0 +1,21 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.lr_scheduler.OneCycleLR +max_lr: 0.001 +epochs: ${..train.num_epochs} +steps_per_epoch: 59 +pct_start: 0.2 diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/reducelronplateau.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/reducelronplateau.yaml new file mode 100644 index 0000000000..da7937ab60 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/reducelronplateau.yaml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.lr_scheduler.ReduceLROnPlateau + +factor: 0.1 +patience: 10 diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/steplr.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/steplr.yaml new file mode 100644 index 0000000000..1290358941 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/lr_scheduler/steplr.yaml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.lr_scheduler.StepLR +step_size: 50 +gamma: 0.5 diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/model/figconv_unet_drivaer.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/model/figconv_unet_drivaer.yaml new file mode 100644 index 0000000000..1de3073733 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/model/figconv_unet_drivaer.yaml @@ -0,0 +1,39 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: src.networks.FIGConvUNetDrivAerNet +_convert_: all + +in_channels: 1 +out_channels: 1 +kernel_size: 3 +hidden_channels: + - 86 + - 86 + - 86 + - 86 +num_levels: 3 +num_down_blocks: 1 +use_rel_pos_encode: true +neighbor_search_type: radius + +resolution_memory_format_pairs: + - ${res_mem_pair:b_xc_y_z, [ 4, 120, 80]} + - ${res_mem_pair:b_yc_x_z, [200, 3, 80]} + - ${res_mem_pair:b_zc_x_y, [200, 120, 2]} + +reductions: + - mean diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/model/figconv_unet_drivaerml.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/model/figconv_unet_drivaerml.yaml new file mode 100644 index 0000000000..d6be832a76 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/model/figconv_unet_drivaerml.yaml @@ -0,0 +1,22 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +defaults: + - figconv_unet_drivaer + +_target_: src.networks.FIGConvUNetDrivAerML + +out_channels: 4 diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/adam.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/adam.yaml new file mode 100644 index 0000000000..bd62b49570 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/adam.yaml @@ -0,0 +1,19 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.Adam +lr: 0.001 +weight_decay: 1e-4 diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/adamw.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/adamw.yaml new file mode 100644 index 0000000000..4d7f7650df --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/adamw.yaml @@ -0,0 +1,20 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.AdamW +lr: 0.001 +weight_decay: 1e-4 +fused: true diff --git a/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/sgd.yaml b/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/sgd.yaml new file mode 100644 index 0000000000..762d6f6a52 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/configs/optimizer/sgd.yaml @@ -0,0 +1,18 @@ +# SPDX-FileCopyrightText: Copyright (c) 2023 - 2026 NVIDIA CORPORATION & AFFILIATES. +# SPDX-FileCopyrightText: All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +_target_: torch.optim.SGD +lr: 0.1 diff --git a/examples/cfd/external_aerodynamics/figconvnet/notebooks/figconvnet_vis.ipynb b/examples/cfd/external_aerodynamics/figconvnet/notebooks/figconvnet_vis.ipynb new file mode 100644 index 0000000000..755b0b6672 --- /dev/null +++ b/examples/cfd/external_aerodynamics/figconvnet/notebooks/figconvnet_vis.ipynb @@ -0,0 +1,701 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# FIGConvNet inference and visualization notebook\n", + "\n", + "This notebook demonstrates how to use pre-trained FIGConvNet model to perform inference\n", + "on DrivAerNet dataset.\n", + "\n", + "The following items are required and need to be downloaded separately:\n", + "\n", + "* Pre-trained [FIGConvNet checkpoint](to_be_provided).\n", + "* [DrivAerNet](https://github.com/Mohamedelrefaie/DrivAerNet/tree/main/DrivAerNet_v1) dataset.\n", + " The dataset needs to be converted to a Webdataset format. For simplicity, the small subset\n", + " of the dataset has been already converted to Webdataset format and can be used in this\n", + " example as-is.\n", + "\n", + "The inputs to the model are:\n", + "* Point cloud representing the surface of the car.\n", + "\n", + "The outputs of the model are:\n", + "* Pressure at each surface point.\n", + "* Wall shear stresses at each surface point.\n", + "\n", + "Before we begin, let's import some common packages." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Warp 1.0.2 initialized:\n", + " CUDA Toolkit 11.5, Driver 12.2\n", + " Devices:\n", + " \"cpu\" : \"x86_64\"\n", + " \"cuda:0\" : \"NVIDIA GeForce RTX 3090\" (24 GiB, sm_86, mempool enabled)\n", + " \"cuda:1\" : \"NVIDIA TITAN RTX\" (24 GiB, sm_75, mempool enabled)\n", + " CUDA peer access:\n", + " Not supported\n", + " Kernel cache:\n", + " /home/du/.cache/warp/1.0.2\n" + ] + } + ], + "source": [ + "from pathlib import Path\n", + "import sys\n", + "\n", + "import numpy as np\n", + "import pyvista as pv\n", + "import torch\n", + "import warp as wp\n", + "\n", + "if sys.path[0] != \"..\":\n", + " sys.path.insert(0, \"..\")\n", + "\n", + "device = torch.device(\"cuda:0\")\n", + "torch.cuda.device(device)\n", + "wp.init()\n", + "wp.set_device(str(device))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Dataset visualization\n", + "\n", + "This section provides visualizations of the original, mesh-based dataset." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "# Path to the dataset and pressure VTK files.\n", + "# Note: update `drivaer_orig_path` as needed.\n", + "drivaer_orig_path = Path(\"/data/src/physicsnemo/data/DrivAerNet/mini\")\n", + "vtk_path = drivaer_orig_path / \"SurfacePressureVTK\"\n", + "\n", + "output_dir = drivaer_orig_path.parent / \"vis\"" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "0e670e1bc4d24aae87943a08fe91197d", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + "Widget(value='