From 604ad8791812b028a1db3cfb04b1db99a0e6eee2 Mon Sep 17 00:00:00 2001 From: Ti Leggett Date: Mon, 10 Aug 2026 11:55:32 -0500 Subject: [PATCH 1/4] Simplify by moving information to specific skills to reduce size and impact to context consumption --- AGENTS.md | 334 +++++++----------------------------------------------- 1 file changed, 44 insertions(+), 290 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b04e0960e9..fac28123a2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # AI Agent Context for Ramble -**Objective:** This document guides AI agents in assisting with questions and tasks related to the Ramble experimentation framework. +**Objective:** This document guides AI agents in assisting with questions about, usage of, and core development tasks related to the Ramble experimentation framework. ## What is Ramble? @@ -18,309 +18,63 @@ Ramble works on Linux, macOS, and many supercomputers. Ramble is primarily controlled through the `ramble` command. Key aspects of the CLI include: -* You should always examine the command line interface before you execute any ramble commands, as the arguments might change over time. -* **Getting Help:** Users can get help on any command by using `ramble help` or `ramble help --all` for more detailed information on all commands. Help is also available for subcommands (e.g., `ramble workspace --help`). -* **Discovering Commands and Depth:** To discover the full command hierarchy and determine its maximum depth, start with `ramble help --all`. For any command that shows `...` in its help text (indicating it has subcommands), run that command with `--help` or `-h` to explore its subcommands (e.g., `ramble workspace --help`). This process can be repeated recursively until no more subcommands are found. The longest chain of commands reveals the maximum depth of the CLI. +* You should always examine the command line interface before executing any `ramble` commands, as arguments might change over time. +* **Getting Help:** You can get help on any command by using `ramble help` or `ramble help --all` for detailed information on all commands. Help is also available for subcommands (e.g., `ramble workspace --help`). +* **Discovering Commands and Depth:** To discover the full command hierarchy, start with `ramble help --all`. For any command that shows `...` in its help text (indicating subcommands), run that command with `--help` or `-h` to explore its subcommands (e.g., `ramble workspace --help`). * **Key Commands:** - * `ramble workspace create`: To set up a new experiment workspace. - * `ramble workspace config`: To manage workspace configurations. - * `ramble on`: To execute the experiments defined in the workspace. - * `ramble list`: To list available applications, modifiers, etc. - * `ramble config`: To manage Ramble's configuration settings. - * `ramble repo`: To manage Ramble repositories. -* **Command Reference:** A full list of commands and their options is available in the [Command Reference](https://ramble.readthedocs.io/en/latest/command_index.html) section of the documentation. + * `ramble workspace create`: Set up a new experiment workspace. + * `ramble workspace config`: Manage workspace configurations. + * `ramble on`: Execute experiments defined in the workspace. + * `ramble list`: List available applications, modifiers, etc. + * `ramble config`: Manage Ramble's configuration settings. + * `ramble repo`: Manage Ramble repositories. +* **Command Reference:** For local ground-truth documentation on CLI commands, refer to [docs/command_index.rst](docs/command_index.rst). -## Ramble Configuration Files +## Ramble Configuration Files Overview Ramble uses YAML files for configuration, drawing inspiration from Spack's configuration system. Configurations are applied in scopes, with higher precedence scopes overriding lower ones (e.g., user settings override system defaults, workspace settings override user settings). -### Workspace Configuration - -* **Main File:** Each workspace has a primary configuration file located at `$workspace/configs/ramble.yaml`. This file defines the experiments, software, and variables for the workspace. +* **Main File:** Each workspace has a primary YAML formatted configuration file located at `$workspace/configs/ramble.yaml`. * **Structure:** All content within `ramble.yaml` lives under the top-level `ramble:` dictionary. -* **Detailed Documentation:** [Workspace Configuration File](https://ramble.readthedocs.io/en/latest/workspace_config.html) - -### Configuration Sections - -Ramble supports various sections within its configuration files. These can exist in the workspace `ramble.yaml`, in separate files within `$workspace/configs/`, or in other configuration scopes (user, site, system). Key sections include: - -* `applications`: Defines the experiments to be generated, including application, workload, and experiment scopes, variables, matrices, etc. See the [Application Section](https://ramble.readthedocs.io/en/latest/configuration_files.html#application-section). -* `config`: Controls internal Ramble behavior, shell settings, Spack command flags, and upload configurations. See the [Config Section](https://ramble.readthedocs.io/en/latest/configuration_files.html#config-section). -* `env_vars`: Manages environment variable modifications (set, append, prepend, unset) for experiments. See the [Environment Variables Section](https://ramble.readthedocs.io/en/latest/configuration_files.html#environment-variables-section). -* `software`: Defines software packages and environments, specifying package manager specs (e.g., Spack specs), compilers, and dependencies. See the [Software Section](https://ramble.readthedocs.io/en/latest/configuration_files.html#software-section). -* `variables`: Defines key-value pairs used for parameterization and expansion within other configuration sections and templates. See the [Variables Section](https://ramble.readthedocs.io/en/latest/configuration_files.html#variables-section). -* `variants`: Customizes variants for experiment creation, such as selecting the `package_manager`. See the [Variants Section](https://ramble.readthedocs.io/en/latest/configuration_files.html#variants-section). -* `modifiers`: Specifies experiment modifiers to be applied to experiments, including mode and target executables. See the [Modifiers Section](https://ramble.readthedocs.io/en/latest/configuration_files.html#modifiers-section). -* `repos`: Lists paths to repositories containing Application definitions. -* `modifier_repos`: Lists paths to repositories containing Modifier definitions. -* **Other Sections:** - * `base_application_repos` - * `base_class_repos` - * `base_modifier_repos` - * `base_package_manager_repos` - * `base_workflow_manager_repos` - * `formatted_executables` - * `internals` - * `licenses` - * `mirrors` - * `package_manager_repos` - * `success_criteria` - * `tables` - * `workflow_manager_repos` - * `zips` - -* **Full Details:** See the [Configuration Files Documentation](https://ramble.readthedocs.io/en/latest/configuration_files.html) for a complete description of all sections and scopes. - -## Package Managers - -Ramble can leverage various package managers to install the software required for experiments. The configuration of package managers is detailed in the [Package Managers Documentation](https://ramble.readthedocs.io/en/latest/package_managers.html). - -### Spack - -A key feature of Ramble is its ability to manage software stacks, often using **Spack**. - -* **Spack:** Spack is a flexible package manager for supercomputers, Linux, and macOS, supporting multiple versions, configurations, platforms, and compilers. -* **Ramble's Usage:** Ramble can be configured to use an existing Spack instance. Experiment configuration files in Ramble define the software stacks required, which Ramble can then realize using Spack. -* **Spack Documentation:** To understand how to use Spack, specify packages, versions, compilers, and variants, refer to the official Spack Documentation: - * **Spack Homepage:** [https://spack.io/](https://spack.io/) - * **Spack Documentation:** [https://spack.readthedocs.io/en/latest/](https://spack.readthedocs.io/en/latest/) - * [Getting Started](https://spack.readthedocs.io/en/latest/getting_started.html) - * [Spec Syntax](https://spack.readthedocs.io/en/latest/spec_syntax.html) - * [Spack Environments](https://spack.readthedocs.io/en/latest/replace_conda_homebrew.html) - * [Command Reference](https://spack.readthedocs.io/en/latest/command_index.html) -* **Key Concepts:** When assisting with Ramble and Spack, be aware of Spack concepts like `spack.yaml`, environments, compilers, and specs. - -## Developing New Object Definitions +* **Configuration Sections:** Ramble supports numerous configuration sections across workspace and system scopes: + * `applications`, `config`, `env_vars`, `software`, `variables`, `variants`, `modifiers` + * `repos`, `modifier_repos`, `package_manager_repos`, `workflow_manager_repos` + * `base_application_repos`, `base_class_repos`, `base_modifier_repos`, `base_package_manager_repos`, `base_workflow_manager_repos` + * `formatted_executables`, `internals`, `licenses`, `mirrors`, `success_criteria`, `tables`, `zips` +* **Detailed Documentation:** For complete syntax specifications, refer to [docs/workspace_config.rst](docs/workspace_config.rst) and [docs/configuration_files.rst](docs/configuration_files.rst). -Ramble is extensible through user-defined objects. These objects are Python classes that follow specific conventions and structures. Ramble uses a specialized language structure, defined in Python, to parse and manage these definitions. +## Package Managers Overview -* **Definition Files:** Each object definition resides in a Python file (e.g., `application.py` for an application) within a directory named after the object inside a repository. -* **Object Types:** The primary object types that users can define include: - * **Applications:** Define the core software and execution logic for an experiment. - * **Modifiers:** Allow for systematic alterations to application configurations or execution parameters. - * **Package Managers:** Interfaces to software installation tools like Spack, EESSI, etc. - * **Workflow Managers:** Interfaces to batch systems or workflow tools (e.g., Slurm, LSF). - These types, and their base classes, are enumerated in `lib/ramble/ramble/repository.py` within the `ObjectTypes` Enum. -* **Ramble Definition Language:** Ramble uses a set of Python classes and decorators to define the structure and attributes of each object type. These are implemented in the `lib/ramble/ramble/language` directory. This includes files like: - * `application_language.py` - * `modifier_language.py` - * `package_manager_language.py` - * `workflow_manager_language.py` - * `shared_language.py` - These modules define the valid keywords, sections, and expected types for the object definition classes. -* **Developer Guides:** Ramble provides guides for creating new definitions: - * [Application Definition Developers Guide](https://ramble.readthedocs.io/en/latest/dev_guides/application_dev_guide.html) - * [Modifier Definition Developers Guide](https://ramble.readthedocs.io/en/latest/dev_guides/modifier_dev_guide.html) - * [Package Manager Definition Developers Guide](https://ramble.readthedocs.io/en/latest/dev_guides/package_manager_dev_guide.html) - * [Advanced Topics for Definition Developers](https://ramble.readthedocs.io/en/latest/dev_guides/advanced_topics.html) - -New definitions are typically placed in a user-created repository and added to the Ramble configuration. - -### How to Write an Application Definition - -This section provides a practical guide for creating a new Ramble application definition by focusing on the key patterns and concepts. - -1. **Repository Structure**: To determine the correct directory structure for a custom definition, inspect the `ObjectTypes` Enum in `lib/ramble/ramble/repository.py`. This enum defines the valid object types (e.g., `APPLICATIONS`, `MODIFIERS`). The value of each enum member (e.g., `'applications'`) is the name of the required subdirectory within a repository. Each specific definition should then be placed in its own directory inside that subdirectory. For example, a new application would be located at `my-repo/applications/my-app-name/application.py`. - -2. **The Definition File**: The definition file (e.g., `application.py`) contains a Python class that inherits from a base class. There are two categories of base definitions to be aware of: - * **Fundamental Base Classes**: These are the abstract building blocks for new definitions (e.g., `executable-application`, `basic-modifier`). This is the most common starting point for creating a new definition from scratch. You can discover them by running: - ```bash - ramble list --type base_classes - ``` - * **Inheritable Concrete Definitions**: These are fully-formed definitions that are designed to be inherited by other definitions to promote code reuse (e.g., a `base_application` like `hpcg`). This is a more advanced pattern. You can discover them by running `ramble list --type base_`, for example: - ```bash - ramble list --type base_applications - ramble list --type base_modifiers - ramble list --type base_package_managers - ramble list --type base_workflow_managers - ``` - -3. **Key Concepts and Patterns**: - * **Declarative Directives**: The application's behavior is defined by calling special functions (directives) within the class body. Instead of writing imperative code, you declare the application's properties. - * **Logical Grouping of Directives**: Directives can be understood in logical groups based on their purpose: - * **Metadata**: Directives that set the application's `name`, `maintainers`, and `tags`. - * **Software Dependencies**: Directives for specifying required software packages (`software_spec`) and compilers (`define_compiler`). These often contain package-manager-specific syntax. - * **Execution & Workloads**: Directives for defining `executable` commands, `input_file` data sources, and `workload`s, which are combinations of executables and inputs that represent a specific test case. - * **Parameterization**: The `workload_variable` directive is used to define parameters that can be set in the Ramble configuration, allowing for flexible and reusable workload definitions. - * **Results & Validation**: Directives for defining how to parse results (`figure_of_merit`) and determine a successful run (`success_criteria`) from output files, typically using regular expressions. - * **Templating**: A `register_template` directive exists to generate complex input or configuration files from a template file. - * **Conditional Logic**: A core pattern in Ramble is the use of a `with when(...)` context manager. This allows directives to be applied conditionally, based on factors like the chosen package manager, system architecture, or other variants. This is the standard way to create a single, portable definition that works in multiple environments. - -4. **Best Practices**: - * **Study Existing Definitions**: The most effective way to understand the current, concrete syntax is to study the built-in application definitions. These provide real-world examples of the patterns described above. Good starting points are `hostname` (simple), `gromacs` (complex), and `hpcg` (inheritance). - * **Write Informative Docstrings**: The docstring for the application class should clearly describe the application and include links to its official website, documentation, and source code. - * **Check for Software Conflicts**: Before adding a new `software_spec`, check for existing definitions of the same package to ensure consistency. - 1. Get a summary of all existing software definitions: `ramble software-definitions --summary` - 2. Search the output for the package you intend to add. - 3. Use a consistent version and spec to avoid conflicts and encourage software reuse. - 4. After adding your `software_spec`, confirm that no conflicts were introduced: `ramble software-definitions --conflicts` +Ramble leverages package managers like **Spack** to manage software stacks required for experiments. +* **Spack:** Spack is a flexible package manager supporting multiple versions, configurations, platforms, and compilers. +* **Local Spack Documentation:** Refer to [docs/package_managers.rst](docs/package_managers.rst) for Ramble's Spack integration details. ## Key Ramble Resources -* **GitHub Repository:** [https://github.com/GoogleCloudPlatform/ramble](https://github.com/GoogleCloudPlatform/ramble) - * Source code, issue tracker, and discussions. - * The `develop` branch has the latest contributions. -* **Documentation:** [https://ramble.readthedocs.io/en/latest/](https://ramble.readthedocs.io/en/latest/) - * [Getting Started Guide](https://ramble.readthedocs.io/en/latest/getting_started.html) - * [Tutorials](https://ramble.readthedocs.io/en/latest/tutorials.html) - * [Configuration Files](https://ramble.readthedocs.io/en/latest/configuration_files.html) - * [Ramble Workspace](https://ramble.readthedocs.io/en/latest/workspace.html) - * [Package Managers](https://ramble.readthedocs.io/en/latest/package_managers.html) - * [Developer Guides](https://ramble.readthedocs.io/en/latest/dev_guides.html) - * [Command Reference](https://ramble.readthedocs.io/en/latest/command_index.html) -* **Examples:** The [examples directory](https://github.com/GoogleCloudPlatform/ramble/tree/develop/examples) in the GitHub repo contains many example configuration files. - -## Common Tasks & Questions - -* **Using the CLI:** How to use `ramble` commands to perform tasks. -* **Writing Configs:** Understanding the YAML syntax and available sections for `ramble.yaml` and other config files. -* **Setting up a workspace:** Users often start by creating a Ramble workspace using `ramble workspace create`. -* **Defining software:** Software requirements are specified in YAML configuration files, often leveraging Spack specs. -* **Running experiments:** How to launch, monitor, and manage experiment sets. -* **Creating new Components:** How to write new Application, Modifier, or other object definitions using Ramble's Python-based definition structure. -* **Troubleshooting:** Issues related to software builds with Spack, configuration errors, or execution problems. - -### Running a Simple Test Experiment - -This workflow details how to create a workspace, configure it for a single experiment with a specific workload, and then run and analyze the experiment. This is a common task for quickly testing the Ramble environment. - -1. **Create an empty workspace:** - ```bash - ramble workspace create -d - ``` - * The `-d` flag is important, as it creates an empty workspace directory. - -2. **Add the experiment with a specific workload:** - ```bash - ramble -D workspace manage experiments --workload-filter - ``` - * Replace `` with the application you want to test (e.g., `hostname`). - * Replace `` with the specific workload for that application (e.g., `local`). - * The `-D ` flag directs the command to the correct workspace without needing to activate it globally. - -3. **Verify the experiment configuration:** - ```bash - ramble -D workspace info - ``` - * This command should show that only the `..generated` experiment is configured. - -4. **Set up the workspace:** - ```bash - ramble -D workspace setup - ``` - * This step generates the necessary scripts and files for the experiment to run. - -5. **Run the experiment:** - ```bash - ramble -D on - ``` - -6. **Analyze the results:** - ```bash - ramble -D workspace analyze - ``` - -7. **View the results:** - ```bash - cat /results.latest.txt - ``` - - -## Guidance for AI Agents - -* When Ramble and software are mentioned together, a package manager like Spack is likely involved in the software installation process. -* Refer to the official Ramble documentation on Read the Docs as the primary source of truth for Ramble-specific questions. -* Refer to the official Spack documentation for questions about Spack usage, syntax, and concepts. -* Direct users to the **Developer Guides** when they ask about creating new object types, and explain that definitions are Python classes using a structure defined in `ramble.language`. -* Point users to the **Command Reference** for CLI usage questions. -* For configuration questions, guide users to the **Configuration Files** and **Workspace Configuration File** sections of the Ramble documentation. -* Use the examples in the GitHub repository to understand common configuration patterns. -* Encourage users to provide their Ramble configuration files and any error messages for debugging. -* When making Python code changes, consult `bin/ramble` to determine the officially supported Python versions. -* Ensure all Python code is compatible with the full range of supported versions. Avoid using APIs that have been deprecated or removed in newer Python versions. When necessary, use feature detection (`hasattr`) or version checks (`sys.version_info`) to maintain broad compatibility. -* **Implementing Directives & Mock Tests**: When adding new directives (which are processed by `DirectiveMeta` in `ramble.language`), keep in mind that directives are processed lazily based on the module namespace. If you create a mock class inside a unit test to test a directive, you **must** explicitly set its `__module__` attribute to a valid Ramble namespace (e.g. `__module__ = "ramble.app"` or `__module__ = "ramble.mod"`). Without this, `DirectiveMeta` will silently skip processing the directives for your test class. -* **Mock Objects and Style Checks**: When creating mock applications or modifiers (e.g., in `var/ramble/repos/builtin.mock/`), ensure these files contain valid Python syntax and conform to Ramble's style guide (including copyright headers). The `ramble style` command runs on the entire repository, and poorly formatted or syntactically invalid mock files will cause the style checker to fail. - - -## Running Unit Tests - -Ramble uses `pytest` for its unit tests. Tests **must** be run using the `ramble unit-test` wrapper command, not by invoking `pytest` directly, as the wrapper handles necessary test environment setup. - -* **Running all tests in parallel:** The `-n auto` option enables parallel test execution: - ```bash - ramble unit-test -n auto - ``` - -* **Running all tests serially:** For troubleshooting or if `pytest-xdist` is unavailable: - ```bash - ramble unit-test - ``` - -* **Passing Pytest Arguments:** You can pass any `pytest` arguments to the command. For example, to only run tests with "gromacs" in their name: - ```bash - ramble unit-test -k gromacs - ``` - -Using `-k` is particularly useful for running only newly added tests. - -* **Writing Unit Tests & Using Test Fixtures**: - * **Workspace Creation**: **ALWAYS** use the `make_workspace_from_config` fixture from `conftest.py` when creating and configuring workspaces in unit tests. Avoid manually creating workspace directories via `tmpdir` or writing YAML files manually. - * **`make_workspace_from_config(config_str=None, name=None, activate=False)`**: - * Accepts a raw YAML configuration string (`config_str`) defining the `ramble:` dictionary. - * Automatically isolates workspace files under `mutable_mock_workspace_path` and mocks configuration scopes (`mutable_config`). - * Returns `(ws, ws_name)` where `ws` is the `ramble.workspace.Workspace` object and `ws_name` is the string name of the workspace (auto-generated from test function name if `name` is omitted). - * Pass `activate=True` if the test requires an activated workspace environment (`ramble.workspace.activate(ws)`). - * *Example*: - ```python - def test_my_workspace_feature(make_workspace_from_config): - test_config = """ - ramble: - variables: - mpi_command: 'mpirun -n {n_ranks}' - applications: - hostname: - workloads: - local: - experiments: - test_exp: {} - """ - ws, ws_name = make_workspace_from_config(test_config, activate=True) - # Test logic using ws or ws_name - ``` - * **Other Fixtures**: Look up available fixtures in `conftest.py` for other testing needs (such as configuration overrides, mock executables, or mock repositories). - -* **Getting Help:** - * For help with the `ramble unit-test` command itself: `ramble unit-test --help` - * For a full list of all available `pytest` options: `ramble unit-test --pytest-help` +* **Repository Examples:** Sample configuration files are located in the local [examples/](examples/) directory. +* **Local Documentation**: Complete Sphinx documentation source files are in [docs/](docs/). -## Running Style Checks +## Available Agent Skills -Ramble uses `isort`, `black`, `flake8`, `mypy`, and `ruff` to enforce a consistent code style and type safety. You can check and fix style issues using the `ramble style` command. +For specialized workflows, consult the relevant **Agent Skill** under `.agents/skills/`: -* **Checking for Style Errors:** To check for any style violations in the files you've changed in your current branch: - ```bash - ramble style - ``` - To check all files in the project, use the `--all` flag: - ```bash - ramble style --all - ``` +| Skill | Description | Location | +| :--- | :--- | :--- | +| **Workspace Wizard** | Interactive setup of experiment workspaces, scaling matrices, and YAML configurations | [.agents/skills/ramble-workspace-wizard/SKILL.md](.agents/skills/ramble-workspace-wizard/SKILL.md) | +| **Definition Author** | Authoring Ramble object definitions (e.g., Applications, Modifiers, Package/Workflow Managers, etc.) | [.agents/skills/ramble-definition-author/SKILL.md](.agents/skills/ramble-definition-author/SKILL.md) | +| **Experiment Runner** | Using Ramble to perform experiments - execution lifecycle (`ramble on`), dry-run validation, and execution debugging | [.agents/skills/ramble-experiment-runner/SKILL.md](.agents/skills/ramble-experiment-runner/SKILL.md) | +| **Results Analyzer** | FOM extraction, speedup & scaling efficiency metrics, and benchmark report generation | [.agents/skills/ramble-results-analyzer/SKILL.md](.agents/skills/ramble-results-analyzer/SKILL.md) | +| **Documentation Author** | Writing Sphinx/reST documentation in `docs/`, building HTML docs, and link checking | [.agents/skills/ramble-documentation-author/SKILL.md](.agents/skills/ramble-documentation-author/SKILL.md) | +| **Spack Integration** | Spack specs (`pkg@ver %compiler`), environment mapping, and concretization troubleshooting | [.agents/skills/ramble-spack-integration/SKILL.md](.agents/skills/ramble-spack-integration/SKILL.md) | +| **Workflow Managers** | Slurm batch directives (`#SBATCH`), partition settings, and launcher overrides | [.agents/skills/ramble-workflow-managers/SKILL.md](.agents/skills/ramble-workflow-managers/SKILL.md) | +| **GCP Cluster Toolkit** | Provisioning Google Cloud HPC/AI clusters (`ghpc`) to host Ramble experiment sweeps | [.agents/skills/gcp-cluster-toolkit/SKILL.md](.agents/skills/gcp-cluster-toolkit/SKILL.md) | +| **Developer Guide** | Guidelines for codebase contributors: pytest fixtures (`make_workspace_from_config`), directive lazy loading, and style checks | [.agents/skills/ramble-developer/SKILL.md](.agents/skills/ramble-developer/SKILL.md) | -* **Fixing Style Errors:** To automatically fix any style errors in your changed files: - ```bash - ramble style --fix - ``` - To fix all files in the project, combine `--all` and `--fix`: - ```bash - ramble style --all --fix - ``` +## Developer Guidelines & Code Contributions -* **Advanced Usage:** You can also specify which styling tools to run or skip. For example, to only run `isort` and `black`: - ```bash - ramble style -t isort -t black - ``` - To skip `flake8` and `mypy`: - ```bash - ramble style -s flake8 -s mypy - ``` +When modifying the Ramble codebase, writing unit tests, or checking style compliance: +* Consult [.agents/skills/ramble-developer/SKILL.md](.agents/skills/ramble-developer/SKILL.md) for detailed guidelines on: + * Running tests via `ramble unit-test` and using the `make_workspace_from_config` fixture. + * Correctly setting `__module__` on mock test classes for directive lazy-loading. + * Running `ramble style` checks (`isort`, `black`, `flake8`, `mypy`, `ruff`). From 60b7825cf6f41aac1e3ca47836da639e405c2acf Mon Sep 17 00:00:00 2001 From: Ti Leggett Date: Mon, 10 Aug 2026 11:56:19 -0500 Subject: [PATCH 2/4] Updates to reflect workflow manager changes and additional instructions for guidance --- .../skills/ramble-workspace-wizard/SKILL.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/.agents/skills/ramble-workspace-wizard/SKILL.md b/.agents/skills/ramble-workspace-wizard/SKILL.md index 2c7a4236ac..171275c63c 100644 --- a/.agents/skills/ramble-workspace-wizard/SKILL.md +++ b/.agents/skills/ramble-workspace-wizard/SKILL.md @@ -50,7 +50,7 @@ This skill transforms you into a specialized assistant for managing Ramble works - **Modular Configs:** Use `ramble workspace manage includes -a ` to include external YAML files for modularity. ### 5. Workflow Managers -- **Explicit Declaration:** Always explicitly define the workflow manager in the `config:` section: `config: workflow_manager: `. (Options: `user-managed`, `slurm`, `gke-mpi`, etc.) +- **Explicit Declaration:** Always explicitly define the workflow manager in the `variants:` section: `variants: workflow_manager: `. (Options: `user-managed`, `slurm`, `gke-mpi`, etc.) - **Letting WM Take the Lead:** If using a workflow manager (like `slurm`), **do not** manually set `batch_submit` or `mpi_command` unless explicitly overriding the WM's optimized defaults. - **Dynamic MPI Commands:** If comparing MPI types and overriding the WM default is necessary, use variable indirection to set specific commands and flags (e.g., `-ppn` vs `-npernode`) for each type: ```yaml @@ -72,6 +72,8 @@ This skill transforms you into a specialized assistant for managing Ramble works - **Application Namespace:** Access variables defined within an application's Python definition using the `{application::::}` syntax. This is particularly useful for version defaults. - **External Environments:** You can bypass Ramble generating a Spack environment by providing an external one: `software: environments: : external_env: path/to/spack.yaml`. - **Concretization:** Use `ramble workspace concretize` to automatically populate the `software:` section based on application definitions. **Crucial:** If the `software` block in `ramble.yaml` contains empty dictionaries (e.g., `packages: {}`, `environments: {}`), `concretize` will respect them and *will not* populate the defaults, often leading to missing spec errors. Either remove these empty dictionaries completely before running `concretize`, or use the force flag (`ramble workspace concretize -f`) to overwrite them. Do not manually guess software dependencies (like MPI or compilers) unless specifically requested by the user. +- **Package Key Consistency**: Always use the exact package keys exposed by `ramble info -v `. Do not simplify or rename templatized keys (e.g., keep `{app}-{version}`). +- **Compiler Association**: Configure compilers strictly via the `compiler:` directive within the `software:packages:` entry. Do not add compiler packages directly to `environments: packages:`. - **Dry-run Validation:** To pass `workspace setup --dry-run` without a local software installation, ensure the workspace is concretized OR provide mock paths for required software (e.g., `hpl_path: /tmp/mock-hpl`) in the top-level `variables:` section. ### 7. Advanced YAML Features @@ -105,9 +107,8 @@ This skill transforms you into a specialized assistant for managing Ramble works - **Prioritize Interactivity:** ALWAYS use the `ask_user` tool to present options to the user at any decision point or transition (e.g., choosing applications, workloads, next steps, or confirming actions). Avoid asking open-ended text questions when a multiple-choice menu can be provided. ### **B. Creating a New Workspace** -1. Ask the user for a workspace name or directory if not provided using `ask_user`. -2. Run `ramble workspace create `. -3. Note that the workspace lives at `./var/ramble/workspaces/`. +1. Ask the user for a workspace path using `ask_user`, if not provided. +2. Run `ramble workspace create -d /path/to/my_workspace`. ### **C. Building the Configuration** 1. Ask the user which package manager (e.g., Spack, EESSI, user-managed) and workflow manager (e.g., Slurm, GKE-MPI, user-managed) they intend to use using `ask_user`. @@ -115,7 +116,7 @@ This skill transforms you into a specialized assistant for managing Ramble works 3. Determine node counts, ranks, PPN, and if you need any specific workflow manager configurations (like partition or account). 4. Check if any modifiers are requested (e.g., profilers). Identify their dependencies. 5. **Draft the `ramble.yaml` directly** applying scaling variables, matrixing, modifiers, workflow managers, chaining/exclusions, and explicit software environments. - - **Explicit Declaration:** Set `variants: package_manager: ` and `config: workflow_manager: `. + - **Explicit Declaration:** Set `variants: package_manager: ` and `variants: workflow_manager: `. - **Environment Linking:** Use `env_name` in the application's `variables:` section to link it to a specific environment in the `software:` block. - **Parameterizing Environments:** If parameterizing over software choices (e.g., `mpi_type: [intel-mpi, openmpi]`), explicitly define the parameterized environment (e.g., `env_name: -{mpi_type}`) and its required packages in the `software:` section before running concretize. 6. Use `ramble workspace concretize` (with `-f` if empty dictionaries exist or if new applications were added) to pull in default software specs if using a package manager (like Spack) and the user doesn't have specific versions in mind. @@ -133,7 +134,7 @@ This skill transforms you into a specialized assistant for managing Ramble works 6. **Celebration:** Once the workspace is successfully created and validated, inject a random Led Zeppelin song quote to celebrate. ## Reference Documentation -- [Workspace Configuration File](https://ramble.readthedocs.io/en/latest/workspace_config.html) -- [Configuration Sections](https://ramble.readthedocs.io/en/latest/configuration_files.html) -- [Success Criteria](https://ramble.readthedocs.io/en/latest/success_criteria.html) -- [Workspace Internals](https://ramble.readthedocs.io/en/latest/workspace_config.html#internals-section) +- [Workspace Configuration File](docs/workspace_config.rst) +- [Configuration Sections](docs/configuration_files.rst) +- [Success Criteria](docs/success_criteria.rst) + From ac01fdfd3d7ccf10578dc32004428d8868eb4e5c Mon Sep 17 00:00:00 2001 From: Ti Leggett Date: Mon, 10 Aug 2026 11:56:54 -0500 Subject: [PATCH 3/4] Initial commits of new skills that still need refinement and review --- .agents/skills/gcp-cluster-toolkit/SKILL.md | 137 +++++++++++++++ .../skills/ramble-definition-author/SKILL.md | 140 +++++++++++++++ .agents/skills/ramble-developer/SKILL.md | 125 ++++++++++++++ .../ramble-documentation-author/SKILL.md | 104 ++++++++++++ .../skills/ramble-experiment-runner/SKILL.md | 112 ++++++++++++ .../skills/ramble-results-analyzer/SKILL.md | 82 +++++++++ .../skills/ramble-spack-integration/SKILL.md | 160 ++++++++++++++++++ .../skills/ramble-workflow-managers/SKILL.md | 105 ++++++++++++ 8 files changed, 965 insertions(+) create mode 100644 .agents/skills/gcp-cluster-toolkit/SKILL.md create mode 100644 .agents/skills/ramble-definition-author/SKILL.md create mode 100644 .agents/skills/ramble-developer/SKILL.md create mode 100644 .agents/skills/ramble-documentation-author/SKILL.md create mode 100644 .agents/skills/ramble-experiment-runner/SKILL.md create mode 100644 .agents/skills/ramble-results-analyzer/SKILL.md create mode 100644 .agents/skills/ramble-spack-integration/SKILL.md create mode 100644 .agents/skills/ramble-workflow-managers/SKILL.md diff --git a/.agents/skills/gcp-cluster-toolkit/SKILL.md b/.agents/skills/gcp-cluster-toolkit/SKILL.md new file mode 100644 index 0000000000..010a4be82f --- /dev/null +++ b/.agents/skills/gcp-cluster-toolkit/SKILL.md @@ -0,0 +1,137 @@ +--- +name: gcp-cluster-toolkit +description: "Guide for provisioning Google Cloud HPC and AI clusters using Cluster Toolkit (ghpc) to host Ramble experiments." +--- + +# Google Cloud Cluster Toolkit Integration Guide + +This skill provides guidelines for provisioning HPC and AI clusters on Google Cloud Platform using **Google Cloud Cluster Toolkit (`ghpc`)** and integrating them with Ramble for benchmark experimentation. + +--- + +## 1. Documentation & Agent Context References + +When working with Cluster Toolkit and `gcloud`, refer to official documentation resources and check for environment agent context files: + +- **Cluster Toolkit Documentation**: [Google Cloud Cluster Toolkit Docs](https://cloud.google.com/cluster-toolkit/docs) and [GitHub Repository](https://github.com/GoogleCloudPlatform/cluster-toolkit). +- **Google Cloud CLI Documentation**: [gcloud CLI Overview](https://cloud.google.com/sdk/gcloud). +- **Environment Agent Instructions**: Check if there are local `AGENTS.md` or skill definitions for Cluster Toolkit or `gcloud` in your environment or workspace before executing provisioning tasks. + +--- + +## 2. Overview of Cluster Toolkit (`ghpc`) + +Google Cloud Cluster Toolkit is an open-source tool that automates the deployment of high-performance computing (HPC) environments on GCP using Terraform. It provides modular blueprints for: +- Slurm HPC Clusters (with compute partitions, auto-scaling, and Slurm accounting). +- AI/ML Training Clusters (NVIDIA H100/A3, TPU v5p, GKE MPI operator). +- High-Performance Storage (Parallelstore, Filestore, Cloud Storage FUSE). + +--- + +## 3. Cluster Provisioning Workflow + +### Step 1: Create Blueprint YAML +Define cluster topology in a Cluster Toolkit blueprint (e.g., `hpc-cluster.yaml`): + +```yaml +blueprint_name: ramble-hpc-cluster + +vars: + project_id: my-gcp-project + deployment_name: ramble-slurm + region: us-central1 + zone: us-central1-a + +deployment_groups: +- group: primary + modules: + - id: network + source: modules/network/vpc + + - id: slurm_login + source: community/modules/scheduler/slurm-gcp-v6-login + use: [network] + + - id: compute_partition + source: community/modules/scheduler/slurm-gcp-v6-nodeset-bucket + settings: + node_count_dynamic_max: 16 + machine_type: c2-standard-60 + + - id: slurm_controller + source: community/modules/scheduler/slurm-gcp-v6-controller + use: [network, slurm_login, compute_partition] +``` + +### Step 2: Build Deployment & Apply Terraform +```bash +# Build Terraform files from blueprint +ghpc create hpc-cluster.yaml + +# Deploy cluster infrastructure +cd ramble-slurm/primary +terraform init +terraform apply -auto-approve +``` + +--- + +## 4. Preparing the Cluster Login Node for Ramble + +### SSH to Login Node +```bash +gcloud compute ssh --zone "us-central1-a" "ramble-slurm-login-0" --project "my-gcp-project" +``` + +### Install Ramble & Spack on Shared Filesystem +Clone Ramble into a shared directory (e.g., `/home` or `/nfs`): + +```bash +cd /home/$USER +git clone https://github.com/GoogleCloudPlatform/ramble.git +source ramble/share/ramble/setup-env.sh +``` + +--- + +## 5. Configuring Ramble Workspaces for Cloud Scaling + +When running experiments on a GCP Cluster Toolkit provisioned Slurm cluster: + +1. Set `config: workflow_manager: slurm` in `ramble.yaml`. +2. Use dynamic cloud instance PPN patterns for A/B machine comparisons: + +```yaml +ramble: + config: + workflow_manager: slurm + + variables: + machine_type: [c2_ppn, c3_ppn] + c2_ppn: 60 + c3_ppn: 176 + processes_per_node: '{{{machine_type}}}' + n_nodes: [1, 2, 4, 8] + n_ranks: '{n_nodes} * {processes_per_node}' + + applications: + hostname: + workloads: + local: + experiments: + cloud_scaling_{n_nodes}nodes: + matrix: + - n_nodes + - machine_type +``` + +--- + +## 6. Teardown & Resource Cleanup + +To prevent unnecessary cloud billing after experiment runs finish, tear down cluster infrastructure: + +```bash +cd ramble-slurm/primary +terraform destroy -auto-approve +``` diff --git a/.agents/skills/ramble-definition-author/SKILL.md b/.agents/skills/ramble-definition-author/SKILL.md new file mode 100644 index 0000000000..ae476b472f --- /dev/null +++ b/.agents/skills/ramble-definition-author/SKILL.md @@ -0,0 +1,140 @@ +--- +name: ramble-definition-author +description: "Guide for creating and editing Ramble Object Definitions (Applications, Modifiers, Package Managers, Workflow Managers, Systems, Platforms, Utilities) using Ramble's Python directive language." +--- + +# Ramble Definition Author Guide + +This skill provides step-by-step guidance for authoring and updating Ramble **Object Definitions** in Python. + +*Note*: For general codebase contribution rules, running unit tests, pytest fixtures (`make_workspace_from_config`), and style linters (`ramble style`), consult the [.agents/skills/ramble-developer/SKILL.md](../ramble-developer/SKILL.md) skill. + +--- + +## 1. Repository Structure & Complete Object Types + +Ramble object definitions live in Python files inside dedicated subdirectories of a Ramble repository (such as `var/ramble/repos/builtin/` or custom user repositories). + +Valid object types and their structure are enumerated in `lib/ramble/ramble/repository.py` (`ObjectTypes` Enum): + +| Object Type | Repository Directory | Definition File | Base Class / Interface | +| :--- | :--- | :--- | :--- | +| **Applications** | `applications//` | `application.py` | `ExecutableApplication` or `Application` | +| **Modifiers** | `modifiers//` | `modifier.py` | `BasicModifier` or `Modifier` | +| **Package Managers** | `package_managers//` | `package_manager.py` | `PackageManager` | +| **Workflow Managers** | `workflow_managers//` | `workflow_manager.py` | `WorkflowManager` | +| **Systems** | `systems//` | `system.py` | `System` | +| **Platforms** | `platforms//` | `platform.py` | `Platform` | +| **Utilities** | `utilities//` | `utility.py` | `Utility` | + +--- + +## 2. Base Classes and Inheritance + +When creating a new definition, determine whether to build from a fundamental base class or inherit from a concrete definition: + +1. **Fundamental Base Classes**: + Discover available base classes via CLI: + ```bash + ramble list --type base_classes + ``` + *Common examples*: `executable-application` (for CLI-driven apps), `basic-modifier` (for simple modifiers). + +2. **Inheritable Concrete Definitions**: + Discover inheritable definitions via CLI: + ```bash + ramble list --type base_ + ``` + *Examples*: + ```bash + ramble list --type base_applications + ramble list --type base_modifiers + ramble list --type base_package_managers + ramble list --type base_workflow_managers + ramble list --type base_systems + ramble list --type base_platforms + ramble list --type base_utilities + ``` + +--- + +## 3. Declarative Directives + +Ramble uses Python class directives defined in `lib/ramble/ramble/language/` (e.g., `application_language.py`, `modifier_language.py`, `shared_language.py`). Directives declare application behavior inside the class body. + +### Directive Categories + +#### A. Metadata +- `name(...)`: Human-readable name. +- `maintainers(...)`: GitHub handles of maintainers (e.g., `maintainers = ["github_user"]`). +- `tags(...)`: List of tags for categorizing workloads/applications. + +#### B. Software Dependencies +- `software_spec(...)`: Define package specs (typically Spack specs). + ```python + software_spec('gromacs_spec', spack_name='gromacs', default_spec='gromacs@2023') + ``` +- `define_compiler(...)`: Define compiler specifications. + +#### C. Executables & Workloads +- `executable(...)`: Declare named command templates. + ```python + executable('run_sim', 'gmx mdrun -s {tpr_file} -deffnm {output_prefix}', implicit=False) + ``` +- `input_file(...)`: Declare data files to download or copy. +- `workload(...)`: Combine executables and input files into named test cases. + ```python + workload('bench50', executables=['run_sim']) + ``` + +#### D. Parameterization & Variables +- `workload_variable(...)`: Define default variables for workloads. + ```python + workload_variable('n_threads', default='1', description='Number of OpenMP threads', workloads=['bench50']) + ``` + +#### E. Results & FOMs (Figures of Merit) +- `figure_of_merit(...)`: Extract performance data from log files using regex. + ```python + figure_of_merit('Performance', regexp=r'Performance:\s+(?P[0-9.]+)\s+ns/day', units='ns/day') + ``` +- `success_criteria(...)`: Define rules to check if an experiment succeeded. + +#### F. Templating +- `register_template(...)`: Register template files to generate complex input/config files for executables. + +--- + +## 4. Conditional Logic with `with when(...)` + +Apply directives conditionally based on variants, package managers, or target environments using the `with when(...)` context manager: + +```python +with when('package_manager=spack'): + software_spec('mpi', spack_name='openmpi') + +with when('package_manager=user-managed'): + workload_variable('mpi_command', default='mpirun', description='User MPI launcher') +``` + +--- + +## 5. Software Conflict Checks + +Before adding new `software_spec` definitions to an application: +1. Summarize existing software definitions: + ```bash + ramble software-definitions --summary + ``` +2. Check for conflicts across definitions: + ```bash + ramble software-definitions --conflicts + ``` +3. Use consistent specs and versions across applications to encourage software reuse. + +--- + +## 6. Development Best Practices & Developer Skill Link + +1. **Docstrings**: Provide informative docstrings on the class detailing what the application does, with links to source code and documentation. +2. **Developer Guidelines**: For unit testing mock classes (setting `__module__`) and running style checks, refer to [.agents/skills/ramble-developer/SKILL.md](../ramble-developer/SKILL.md). diff --git a/.agents/skills/ramble-developer/SKILL.md b/.agents/skills/ramble-developer/SKILL.md new file mode 100644 index 0000000000..59d07bd981 --- /dev/null +++ b/.agents/skills/ramble-developer/SKILL.md @@ -0,0 +1,125 @@ +--- +name: ramble-developer +description: "Guide for Ramble codebase contributors on writing Python code, running unit tests (pytest), fixture usage (make_workspace_from_config), directive lazy-loading rules, and running style checks (ramble style)." +--- + +# Ramble Developer Guide + +This skill provides guidelines for AI agents contributing code, bug fixes, unit tests, or directives to the Ramble Python codebase. + +--- + +## 1. Python Version Compatibility + +- **Supported Versions**: When making Python code changes, consult `bin/ramble` to determine officially supported Python versions. +- **Compatibility Guardrails**: Ensure code works across all supported Python versions. Use feature detection (`hasattr`) or version checks (`sys.version_info`) when necessary to maintain backward compatibility. + +--- + +## 2. Running Unit Tests + +Ramble uses `pytest` for unit testing. Tests **must** be run using the `ramble unit-test` wrapper command (not `pytest` directly) to ensure correct environment setup. + +### Test Execution Commands +- **Run all tests in parallel**: + ```bash + ramble unit-test -n auto + ``` +- **Run tests serially**: + ```bash + ramble unit-test + ``` +- **Filter tests by name or pattern**: + ```bash + ramble unit-test -k gromacs + ``` +- **Get help on test options**: + ```bash + ramble unit-test --help + ramble unit-test --pytest-help + ``` + +--- + +## 3. Writing Unit Tests & Fixtures + +### `make_workspace_from_config` Fixture +When creating and configuring workspaces in unit tests, **always** use the `make_workspace_from_config` fixture defined in `conftest.py`. Avoid creating workspace directories manually via `tmpdir` or writing raw YAML files to disk. + +#### Signature +```python +make_workspace_from_config(config_str=None, name=None, activate=False) +``` + +#### Behavior & Features +- Accepts a raw YAML configuration string (`config_str`) defining the `ramble:` dictionary. +- Automatically isolates workspace files under `mutable_mock_workspace_path` and mocks configuration scopes (`mutable_config`). +- Returns `(ws, ws_name)` where `ws` is the `ramble.workspace.Workspace` object and `ws_name` is the string name of the workspace. +- Pass `activate=True` if the test requires an activated workspace environment (`ramble.workspace.activate(ws)`). + +#### Example Unit Test +```python +def test_my_workspace_feature(make_workspace_from_config): + test_config = """ +ramble: + variants: + package_manager: spack + workflow_manager: slurm + variables: + n_nodes: 1 + slurm_partition: standard + mpi_command: 'mpirun -n {n_ranks}' + applications: + hostname: + workloads: + local: + experiments: + test_exp: {} +""" + ws, ws_name = make_workspace_from_config(test_config, activate=True) + # Test logic using ws or ws_name +``` + +--- + +## 4. Implementing Directives & Mock Test Classes + +- **Lazy Directive Processing**: Directives in Ramble are processed lazily based on class module namespace (via `DirectiveMeta` in `lib/ramble/ramble/language/`). +- **Crucial Rule for Mock Classes**: If creating a mock application or modifier class inside a unit test file to test a directive, you **must** explicitly set its `__module__` attribute to a valid Ramble namespace: + ```python + class MockApp(ExecutableApplication): + __module__ = "ramble.app" # Required for DirectiveMeta to process directives! + ``` + Without this explicit `__module__` assignment, `DirectiveMeta` will silently skip processing directives for your mock test class. + +--- + +## 5. Running Style Checks + +Ramble enforces code formatting and type safety using `isort`, `black`, `flake8`, `mypy`, and `ruff`. + +### Commands +- **Check changed files**: + ```bash + ramble style + ``` +- **Check all files in repository**: + ```bash + ramble style --all + ``` +- **Automatically fix style errors**: + ```bash + ramble style --fix + ramble style --all --fix + ``` +- **Filter specific tools**: + ```bash + # Run only isort and black + ramble style -t isort -t black + + # Skip flake8 and mypy + ramble style -s flake8 -s mypy + ``` + +### Mock Files and Style Checks +When adding mock application or modifier files (e.g., in `var/ramble/repos/builtin.mock/`), ensure these files contain valid Python syntax, appropriate docstrings, and standard copyright headers. `ramble style` runs on the entire repository and will fail if mock files have syntax or formatting errors. diff --git a/.agents/skills/ramble-documentation-author/SKILL.md b/.agents/skills/ramble-documentation-author/SKILL.md new file mode 100644 index 0000000000..f8c6b26bf3 --- /dev/null +++ b/.agents/skills/ramble-documentation-author/SKILL.md @@ -0,0 +1,104 @@ +--- +name: ramble-documentation-author +description: "Guide for authoring, updating, and building Ramble documentation in Sphinx/reStructuredText (reST) format under docs/." +--- + +# Ramble Documentation Author Guide + +This skill provides guidelines for writing, updating, and verifying Ramble documentation located in the `docs/` directory, which is published to Read The Docs. + +--- + +## 1. Documentation Structure (`docs/`) + +Ramble documentation is written in **reStructuredText (`.rst`)** and managed by Sphinx. Key files and directories: + +- `docs/index.rst`: Main table of contents and introduction. +- `docs/getting_started.rst`: Beginner tutorials and workspace quickstart. +- `docs/workspace_config.rst`: Workspace configuration file specifications. +- `docs/configuration_files.rst`: Detailed section syntax descriptions. +- `docs/package_managers.rst`: Package manager integration guides (Spack, EESSI). +- `docs/dev_guides/`: Developer guides for authoring applications, modifiers, workflow managers, etc. +- `docs/command_index.rst`: Ramble CLI command reference. + +--- + +## 2. reStructuredText (reST) Syntax Guidelines + +### Headings +Use consistent underline characters for document hierarchy: + +```rst +Document Title +============== + +Section Title +------------- + +Subsection Title +~~~~~~~~~~~~~~~~ + +Sub-subsection Title +^^^^^^^^^^^^^^^^^^^^ +``` + +### Directives & Alerts + +```rst +.. note:: + This is a helpful note regarding workspace variables. + +.. warning:: + Overriding Spack compiler specs directly can cause concretization conflicts. + +.. code-block:: yaml + + ramble: + variables: + n_nodes: 2 +``` + +### Cross-Referencing & Links + +- **Document Links**: `:doc:\`workspace_config\`` or `:doc:\`Application Guide \`` +- **Section References**: Use explicit targets: + ```rst + .. _my-custom-section: + + My Custom Section + ----------------- + Refer to :ref:`my-custom-section`. + ``` +- **External Links**: `` `Ramble Docs `_ `` + +--- + +## 3. Building Documentation Locally + +Before submitting documentation changes, build the HTML documentation locally to verify formatting and check for syntax errors or broken links. + +### Build Steps +1. Navigate to the `docs/` directory: + ```bash + cd docs + ``` +2. Build HTML output: + ```bash + make html + ``` + *(Or using `sphinx-build`: `sphinx-build -b html . _build/html`)* +3. Verify output in `docs/_build/html/index.html`. + +### Checking Links +Run Sphinx linkcheck to ensure no external or internal links are broken: +```bash +make linkcheck +``` + +--- + +## 4. Documentation Best Practices + +1. **Keep Examples Runnable**: Ensure all YAML configuration snippets in documentation reflect current Ramble schema and pass validation. +2. **Document New CLI Commands & Directives**: When adding new directives in `lib/ramble/ramble/language/`, ensure corresponding documentation is added to the Developer Guides (`docs/dev_guides/`). +3. **Check Build Warnings**: Treat Sphinx build warnings as errors—resolve any missing cross-reference target warnings during `make html`. diff --git a/.agents/skills/ramble-experiment-runner/SKILL.md b/.agents/skills/ramble-experiment-runner/SKILL.md new file mode 100644 index 0000000000..e74754fd9a --- /dev/null +++ b/.agents/skills/ramble-experiment-runner/SKILL.md @@ -0,0 +1,112 @@ +--- +name: ramble-experiment-runner +description: "Guide for setting up, executing, monitoring, and debugging experiment runs in Ramble workspaces." +--- + +# Ramble Experiment Runner Guide + +This skill provides procedural guidelines for setting up, executing, monitoring, and troubleshooting experiments in Ramble workspaces. + +--- + +## 1. Standard Experiment Execution Lifecycle + +To run a test experiment or benchmark study from start to finish, follow these standard steps: + +### Step 1: Create an Empty Workspace +```bash +ramble workspace create -d +``` +*Note*: The `-d` flag creates a workspace directory under the specified path or current working directory. + +### Step 2: Configure Experiments using CLI (Preferred) +**Prioritize using the CLI** `ramble workspace manage experiments` command to add and configure experiments in the workspace: + +```bash +ramble -D workspace manage experiments --workload-filter +``` + +#### Direct YAML Editing as Secondary Fallback +For configuration features that do not have dedicated `workspace manage experiments` CLI subcommands (such as adding custom `success_criteria`, advanced `zips:`, complex matrix indirection, or custom `internals:`), edit `$workspace/configs/ramble.yaml` directly. + +### Step 3: Verify Configuration & Experiment Expansion +```bash +ramble -D workspace info +``` +Verify that all expected experiment instances (e.g., `..`) are generated and matrix variables expand properly. + +### Step 4: Generate Workspace Setup & Execution Scripts +```bash +ramble -D workspace setup +``` +This generates required script files, directory trees, and environment configurations for all experiments. + +### Step 5: Execute Experiments +```bash +ramble -D on +``` +Launches experiment execution sequentially or via the configured workflow manager. + +### Step 6: Analyze Results +```bash +ramble -D workspace analyze +``` +Parses output logs, extracts Figures of Merit (FOMs), and evaluates success criteria. + +### Step 7: Inspect Summary Results +```bash +cat /results.latest.txt +``` + +--- + +## 2. Targeting Workspaces without Global Activation + +Always use the `-D ` flag with `ramble` to target a specific workspace directory without activating it globally in the shell environment: + +```bash +ramble -D ./my-workspace workspace info +ramble -D ./my-workspace workspace setup +ramble -D ./my-workspace on +``` + +--- + +## 3. Targeted Experiment Filtering (`--where`) + +To run, set up, or analyze specific subsets of experiments within a large workspace matrix, use the `--where` flag: + +```bash +# Target experiments based on node counts +ramble -D workspace setup --where '{n_nodes} >= 4' + +# Target experiments with a specific tag +ramble -D on --where '{tag_name}' + +# Target specific variable combinations +ramble -D workspace analyze --where '{compiler} == "gcc"' +``` + +--- + +## 4. Dry-Run Validation + +Before performing full execution, run a setup dry-run to validate scripts and templates without building software or executing long jobs: + +```bash +ramble -D workspace setup --dry-run +``` + +*Tip*: If software dependencies are not locally installed, ensure the workspace is concretized (`ramble workspace concretize -f`) or mock variable paths (e.g., `app_path: /tmp/mock-bin`) are defined in the workspace `variables:` section. + +--- + +## 5. Troubleshooting Execution Failures + +1. **Missing Environments / Concretization Errors**: + - Run `ramble -D workspace concretize -f` to populate default software specs. + - Check if empty `packages: {}` or `environments: {}` blocks exist in `ramble.yaml`. +2. **Missing Software Specs**: + - Ensure the application `env_name` variable maps cleanly to a defined environment in the `software:` block. +3. **Execution Failure Logs**: + - Inspect individual experiment log files located inside `/experiments////` for standard output and error logs (`*.out`, `*.err`). diff --git a/.agents/skills/ramble-results-analyzer/SKILL.md b/.agents/skills/ramble-results-analyzer/SKILL.md new file mode 100644 index 0000000000..49a327b0a0 --- /dev/null +++ b/.agents/skills/ramble-results-analyzer/SKILL.md @@ -0,0 +1,82 @@ +--- +name: ramble-results-analyzer +description: "Guide for extracting Figures of Merit (FOMs), evaluating experiment success criteria, comparing performance metrics across matrix dimensions, and generating benchmark summary reports." +--- + +# Ramble Results Analyzer Guide + +This skill provides guidelines for analyzing Ramble experiment outputs, extracting Figures of Merit (FOMs), calculating performance metrics (e.g., speedup, scaling efficiency), and producing structured benchmark summary reports. + +--- + +## 1. Running Workspace Analysis + +After experiment execution finishes, run the analysis command: + +```bash +ramble -D workspace analyze +``` + +This command parses output files across all experiment instances in `/experiments/`, evaluates `success_criteria`, extracts `figure_of_merit` directives, and writes output files: +- `/results.latest.txt` (Human-readable plain text summary) +- `/results.latest.json` (Structured JSON representation) +- `/results.latest.yaml` (Structured YAML representation) + +--- + +## 2. Understanding Results Structure + +The results file organizes data hierarchically: +- **Experiment Scope**: Application name, workload name, experiment name. +- **Variables**: Final evaluated key-value pairs for the run (e.g., `n_nodes`, `n_ranks`, `processes_per_node`, `compiler`). +- **Success Criteria**: Evaluation results (`PASSED` or `FAILED`). +- **Figures of Merit (FOMs)**: Parsed numerical or string metrics (e.g., execution time, GFLOPS, throughput) with units and context. + +--- + +## 3. Performance & Scaling Analysis + +When analyzing experiment sweeps, evaluate key high-performance computing metrics: + +### A. Speedup ($S_N$) +$$S_N = \frac{T_1}{T_N}$$ +Where $T_1$ is runtime on 1 node (or baseline configuration) and $T_N$ is runtime on $N$ nodes. + +### B. Parallel Scaling Efficiency ($E_N$) +$$E_N = \frac{S_N}{N} = \frac{T_1}{N \cdot T_N}$$ + +### C. Matrix Sweep Comparisons +When performing A/B testing across compilers, MPI versions, or cloud machine types: +1. Group experiments by primary variant (e.g., `machine_type` or `compiler`). +2. Compare FOM values (e.g., `Time (s)` or `FOM/sec`) for identical workloads and rank counts. +3. Identify performance deltas percentage: $\frac{\text{FOM}_A - \text{FOM}_B}{\text{FOM}_B} \times 100\%$. + +--- + +## 4. Generating Benchmark Summary Reports + +When asked to summarize results, present findings in a structured Markdown report: + +### Report Template + +```markdown +# Benchmark Analysis Report: + +## Executive Summary +- **Primary Metric**: () +- **Best Configuration**: +- **Peak Performance**: + +## Scaling & Performance Summary + +| Nodes | Ranks | Machine / Compiler | FOM () | Speedup | Scaling Efficiency | Status | +| :--- | :--- | :--- | :--- | :--- | :--- | :--- | +| 1 | 16 | gcc-12 | 124.5 | 1.00x | 100.0% | PASSED | +| 2 | 32 | gcc-12 | 68.2 | 1.83x | 91.3% | PASSED | +| 4 | 64 | gcc-12 | 37.1 | 3.36x | 83.9% | PASSED | + +## Key Insights & Observations +1. **Scaling Trend**: Scaling remains above 80% up to 4 nodes (64 ranks). +2. **Bottlenecks**: Performance leveling observed beyond 8 nodes due to interconnect communication overhead. +3. **Recommendations**: Recommended production allocation is 4 nodes per job. +``` diff --git a/.agents/skills/ramble-spack-integration/SKILL.md b/.agents/skills/ramble-spack-integration/SKILL.md new file mode 100644 index 0000000000..3cd0fb19e2 --- /dev/null +++ b/.agents/skills/ramble-spack-integration/SKILL.md @@ -0,0 +1,160 @@ +--- +name: ramble-spack-integration +description: "Guide for configuring Spack package manager specs, environments, concretization, and resolving software build issues in Ramble." +--- + +# Ramble Spack Integration Guide + +This skill provides guidelines for configuring the Spack package manager within Ramble workspaces, writing Spack specs, managing environments, and resolving concretization conflicts. + +--- + +## 1. Overview of Spack in Ramble + +Ramble leverages **Spack** to build, install, and manage software stacks required by experiment workloads. When `variants: package_manager: spack` is set: +- Ramble generates Spack environments (`spack.yaml`) for each configured software environment. +- Software specifications in application definitions or workspace configs are realized via Spack specs. + +--- + +## 2. Inspecting Workspace Software Stack + +To view the evaluated software stack and environment mappings for a workspace, run: + +```bash +ramble -D workspace info --software +``` + +This command outputs: +- Configured software packages and Spack specs. +- Environment names mapped to applications and workloads. +- Concretization status of packages and compilers. + +--- + +## 3. Spack Spec Syntax Quick Reference + +Spack specs define package configuration using a flexible syntax: + +| Syntax Element | Description | Example | +| :--- | :--- | :--- | +| **Package Name** | Target package | `gromacs`, `openmpi`, `intel-oneapi-mkl` | +| **Version (`@`)** | Version string or constraint | `gromacs@2023.2`, `mpich@4.1:` | +| **Compiler (`%`)** | Compiler spec | `%gcc@12.2.0`, `%oneapi@2023.1.0` | +| **Variants (`+`/`-`/`~`)**| Enable or disable features | `+mpi +cuda -double` | +| **Dependencies (`^`)** | Specify dependency specs | `gromacs ^openmpi@4.1.5 %gcc@12` | +| **Target Architecture (`target=`)** | CPU architecture | `target=zen3`, `target=x86_64_v4` | + +--- + +## 4. Configuring the `software:` Block in `ramble.yaml` + +A workspace utilizing Spack must include a `software:` section defining `packages` and `environments`. This would be populated with application defaults during workspace concretization (see Section 6 of this skill). For example: + +```yaml +ramble: + variants: + package_manager: spack + + software: + packages: + gcc12: + spack_spec: gcc@12.2.0 + openmpi4: + spack_spec: openmpi@4.1.5 %gcc12 +cuda + gromacs_pkg: + spack_spec: gromacs@2023.2 %gcc12 ^openmpi4 + + environments: + gromacs_env: + packages: + - openmpi4 + - gromacs_pkg +``` + +### External Spack Environments +If an existing Spack environment already exists on disk, link it directly without generating a new one: + +```yaml +ramble: + software: + environments: + custom_env: + external_env: /path/to/existing/spack/environment/dir +``` + +--- + +## 5. Linking Applications to Spack Environments + +By default, every application expects an environment named after it (e.g., `gromacs`). + +To map an application to a specific environment, set `env_name` in the application's `variables:` section: + +```yaml +ramble: + applications: + gromacs: + variables: + env_name: gromacs_env + workloads: + water_bare: + experiments: + test_run: {} +``` + +### Parameterized Environment Sweeps (A/B Testing) +To compare software stacks (e.g., GCC vs Intel OneAPI, OpenMPI vs MPICH): + +```yaml +ramble: + variables: + mpi_type: [openmpi, mpich] + + software: + environments: + 'gromacs-{mpi_type}': + packages: + - 'gromacs-{mpi_type}' + + applications: + gromacs: + variables: + env_name: 'gromacs-{mpi_type}' + matrix: + - mpi_type +``` + +--- + +## 6. Concretization & Troubleshooting + +### Automatic Concretization +To populate default software specs for application workloads automatically: + +```bash +ramble -D workspace concretize -f +``` + +*Crucial Warning*: If `ramble.yaml` contains empty dictionaries (e.g., `packages: {}`, `environments: {}`), `concretize` will respect them as user overrides and **will not** populate defaults. **Always delete empty dictionary blocks** or run `concretize -f` (force) to overwrite them. + +### Resolving Spec Conflicts +1. Check existing defined specs across builtin applications: + ```bash + ramble software-definitions --summary + ``` +2. Search for spec conflicts: + ```bash + ramble software-definitions --conflicts + ``` +3. Ensure compiler and MPI dependency versions are compatible with the host operating system and GPU drivers. + +## 7. Concretization Diagnostic Loop +If `concretize` fails, follow this loop *before* making any manual edits: + +1. **Inspect:** Read the log file and the `ramble.yaml` simultaneously. +2. **Compare:** Compare the error message against the keys in `software:packages:`. +3. **Minimal Edit:** Apply a fix that *only* addresses the naming mismatch found. +4. **Re-validate:** Run `concretize`. + +**Constraint:** If the fix involves adding a new package definition, you MUST confirm it by running `ramble info -v` again, or by referencing Section 4 of this skill to ensure adherence to external spec definition best practices. diff --git a/.agents/skills/ramble-workflow-managers/SKILL.md b/.agents/skills/ramble-workflow-managers/SKILL.md new file mode 100644 index 0000000000..ffae09596c --- /dev/null +++ b/.agents/skills/ramble-workflow-managers/SKILL.md @@ -0,0 +1,105 @@ +--- +name: ramble-workflow-managers +description: "Guide for configuring workflow managers (Slurm, GKE-MPI, user-managed) and batch schedulers in Ramble workspaces." +--- + +# Ramble Workflow Managers Guide + +This skill provides guidelines for integrating workload managers and batch schedulers (starting with **Slurm**) in Ramble workspaces. + +--- + +## 1. Overview & Workflow Manager Selection + +Ramble abstracts job submission and batch scheduling via **Workflow Managers**. Configure the active workflow manager in the workspace `variants:` block: + +```yaml +ramble: + variants: + workflow_manager: slurm +``` + +Available Workflow Managers can be listed via CLI: +```bash +ramble list --type workflow_managers +``` +Common workflow managers include: +- `user-managed`: Direct local command execution (default). +- `slurm`: Slurm Workload Manager batch script generation (`#SBATCH`) and submission (`sbatch`). +- `gke-mpi`: Kubernetes/GKE MPI operator job generation. + +--- + +## 2. Slurm Workflow Manager Configuration + +When `workflow_manager: slurm` is active, Ramble generates executable batch submission scripts containing appropriate `#SBATCH` directives. + +### Key Slurm Variables + +Set batch parameters in the top-level `variables:` or experiment scope: + +```yaml +ramble: + variants: + workflow_manager: slurm + + variables: + n_nodes: 4 + processes_per_node: 32 + n_ranks: '{n_nodes} * {processes_per_node}' + slurm_partition: 'hpc-partition' # Maps to #SBATCH --partition + time: '02:00:00' # Maps to #SBATCH --time + account: 'my-project-account' # Maps to #SBATCH --account + reservation: 'hpc-res' # Maps to #SBATCH --reservation (optional) +``` + +### Best Practice: Let Workflow Manager Control Launchers +When using a workflow manager like Slurm, **do not manually set `batch_submit` or `mpi_command`** unless explicitly overriding defaults. The workflow manager automatically generates optimized `sbatch` headers and `srun`/`mpirun` invocation flags based on cluster configurations. + +--- + +## 3. Dynamic Launcher Overrides (Advanced) + +If comparing different MPI implementations (e.g., OpenMPI vs Intel MPI) under Slurm where launcher flags differ: + +```yaml +ramble: + variants: + workflow_manager: slurm + + variables: + slurm_partition: standard + mpi_type: [intel-mpi, openmpi] + intel-mpi_command: 'mpiexec -f {hostfile} -ppn {processes_per_node} -n {n_ranks}' + openmpi_command: 'mpirun -hostfile {hostfile} -npernode {processes_per_node} -n {n_ranks}' + mpi_command: '{{{mpi_type}_command}}' + + applications: + hostname: + workloads: + local: + experiments: + exp_{mpi_type}: + matrix: + - mpi_type +``` + +--- + +## 4. Batch Execution Lifecycle & Executor Options + +Launch batch execution using `ramble on`: + +```bash +ramble -D on +``` + +For advanced batch interaction, pass `--executor` flags to interact with cluster batch query or cancellation commands: + +```bash +# Query active jobs +ramble -D on --executor "{batch_query}" + +# Cancel running workspace jobs +ramble -D on --executor "{batch_cancel}" +``` From 28076dcf05ed9b976e9cbec965874692c050cfe1 Mon Sep 17 00:00:00 2001 From: Ti Leggett Date: Fri, 14 Aug 2026 10:14:36 -0500 Subject: [PATCH 4/4] Removing CTK skill --- .agents/skills/gcp-cluster-toolkit/SKILL.md | 137 -------------------- 1 file changed, 137 deletions(-) delete mode 100644 .agents/skills/gcp-cluster-toolkit/SKILL.md diff --git a/.agents/skills/gcp-cluster-toolkit/SKILL.md b/.agents/skills/gcp-cluster-toolkit/SKILL.md deleted file mode 100644 index 010a4be82f..0000000000 --- a/.agents/skills/gcp-cluster-toolkit/SKILL.md +++ /dev/null @@ -1,137 +0,0 @@ ---- -name: gcp-cluster-toolkit -description: "Guide for provisioning Google Cloud HPC and AI clusters using Cluster Toolkit (ghpc) to host Ramble experiments." ---- - -# Google Cloud Cluster Toolkit Integration Guide - -This skill provides guidelines for provisioning HPC and AI clusters on Google Cloud Platform using **Google Cloud Cluster Toolkit (`ghpc`)** and integrating them with Ramble for benchmark experimentation. - ---- - -## 1. Documentation & Agent Context References - -When working with Cluster Toolkit and `gcloud`, refer to official documentation resources and check for environment agent context files: - -- **Cluster Toolkit Documentation**: [Google Cloud Cluster Toolkit Docs](https://cloud.google.com/cluster-toolkit/docs) and [GitHub Repository](https://github.com/GoogleCloudPlatform/cluster-toolkit). -- **Google Cloud CLI Documentation**: [gcloud CLI Overview](https://cloud.google.com/sdk/gcloud). -- **Environment Agent Instructions**: Check if there are local `AGENTS.md` or skill definitions for Cluster Toolkit or `gcloud` in your environment or workspace before executing provisioning tasks. - ---- - -## 2. Overview of Cluster Toolkit (`ghpc`) - -Google Cloud Cluster Toolkit is an open-source tool that automates the deployment of high-performance computing (HPC) environments on GCP using Terraform. It provides modular blueprints for: -- Slurm HPC Clusters (with compute partitions, auto-scaling, and Slurm accounting). -- AI/ML Training Clusters (NVIDIA H100/A3, TPU v5p, GKE MPI operator). -- High-Performance Storage (Parallelstore, Filestore, Cloud Storage FUSE). - ---- - -## 3. Cluster Provisioning Workflow - -### Step 1: Create Blueprint YAML -Define cluster topology in a Cluster Toolkit blueprint (e.g., `hpc-cluster.yaml`): - -```yaml -blueprint_name: ramble-hpc-cluster - -vars: - project_id: my-gcp-project - deployment_name: ramble-slurm - region: us-central1 - zone: us-central1-a - -deployment_groups: -- group: primary - modules: - - id: network - source: modules/network/vpc - - - id: slurm_login - source: community/modules/scheduler/slurm-gcp-v6-login - use: [network] - - - id: compute_partition - source: community/modules/scheduler/slurm-gcp-v6-nodeset-bucket - settings: - node_count_dynamic_max: 16 - machine_type: c2-standard-60 - - - id: slurm_controller - source: community/modules/scheduler/slurm-gcp-v6-controller - use: [network, slurm_login, compute_partition] -``` - -### Step 2: Build Deployment & Apply Terraform -```bash -# Build Terraform files from blueprint -ghpc create hpc-cluster.yaml - -# Deploy cluster infrastructure -cd ramble-slurm/primary -terraform init -terraform apply -auto-approve -``` - ---- - -## 4. Preparing the Cluster Login Node for Ramble - -### SSH to Login Node -```bash -gcloud compute ssh --zone "us-central1-a" "ramble-slurm-login-0" --project "my-gcp-project" -``` - -### Install Ramble & Spack on Shared Filesystem -Clone Ramble into a shared directory (e.g., `/home` or `/nfs`): - -```bash -cd /home/$USER -git clone https://github.com/GoogleCloudPlatform/ramble.git -source ramble/share/ramble/setup-env.sh -``` - ---- - -## 5. Configuring Ramble Workspaces for Cloud Scaling - -When running experiments on a GCP Cluster Toolkit provisioned Slurm cluster: - -1. Set `config: workflow_manager: slurm` in `ramble.yaml`. -2. Use dynamic cloud instance PPN patterns for A/B machine comparisons: - -```yaml -ramble: - config: - workflow_manager: slurm - - variables: - machine_type: [c2_ppn, c3_ppn] - c2_ppn: 60 - c3_ppn: 176 - processes_per_node: '{{{machine_type}}}' - n_nodes: [1, 2, 4, 8] - n_ranks: '{n_nodes} * {processes_per_node}' - - applications: - hostname: - workloads: - local: - experiments: - cloud_scaling_{n_nodes}nodes: - matrix: - - n_nodes - - machine_type -``` - ---- - -## 6. Teardown & Resource Cleanup - -To prevent unnecessary cloud billing after experiment runs finish, tear down cluster infrastructure: - -```bash -cd ramble-slurm/primary -terraform destroy -auto-approve -```