From a6876d145efc79fa3cefb2f804d3f9af2f024576 Mon Sep 17 00:00:00 2001 From: MichaelTansiniSeqera Date: Fri, 12 Jun 2026 16:27:18 +0100 Subject: [PATCH 1/5] docs: add Intelligent Compute page for AWS Cloud (preview) Adds a new standalone page covering the Seqera Intelligent Compute preview feature for AWS Cloud compute environments, including IAM permissions, setup steps, and configuration options. Updates the Cloud sidebar to include the new page after aws-cloud. Co-Authored-By: Claude Sonnet 4.6 --- platform-cloud/cloud-sidebar.json | 1 + .../docs/compute-envs/intelligent-compute.mdx | 106 ++++++++++++++++++ 2 files changed, 107 insertions(+) create mode 100644 platform-cloud/docs/compute-envs/intelligent-compute.mdx diff --git a/platform-cloud/cloud-sidebar.json b/platform-cloud/cloud-sidebar.json index 8eef893f4..b2238ce14 100644 --- a/platform-cloud/cloud-sidebar.json +++ b/platform-cloud/cloud-sidebar.json @@ -69,6 +69,7 @@ "compute-envs/seqera-compute", "compute-envs/aws-batch", "compute-envs/aws-cloud", + "compute-envs/intelligent-compute", "compute-envs/azure-batch", "compute-envs/azure-cloud", "compute-envs/google-cloud-batch", diff --git a/platform-cloud/docs/compute-envs/intelligent-compute.mdx b/platform-cloud/docs/compute-envs/intelligent-compute.mdx new file mode 100644 index 000000000..cf3737cac --- /dev/null +++ b/platform-cloud/docs/compute-envs/intelligent-compute.mdx @@ -0,0 +1,106 @@ +--- +title: "Intelligent Compute" +description: "Set up Seqera Intelligent Compute on an AWS Cloud compute environment" +date created: "2026-06-12" +last updated: "2026-06-12" +tags: [intelligent compute, aws, ecs, compute environments] +toc_min_heading_level: 2 +toc_max_heading_level: 4 +--- + +import CodeBlock from '@theme/CodeBlock'; +import AwsCloudIntelligentComputePolicy from './_policies/aws-cloud-intelligent-compute-policy.json?raw'; + +:::info[Preview] +Seqera Intelligent Compute is in preview and must be enabled for your organization by Seqera before you can use it. Contact your account manager to request access. +::: + +:::caution +Intelligent Compute may assign different CPU and memory values to tasks than those specified in your pipeline's `process` directives. The scheduler selects the most cost-effective instance that meets each task's resource request rather than provisioning exactly what the directive specifies. +::: + +Intelligent Compute is supported on **AWS Cloud compute environments only**. + +## What is Intelligent Compute + +Intelligent Compute is a scheduling service that runs Nextflow pipelines on a Seqera-managed Amazon ECS cluster. It allocates compute resources based on what each task actually needs rather than what the pipeline requests, reducing cost and improving utilization across a run. + +Unlike the standard AWS Cloud compute environment, which runs a pipeline on a single EC2 instance with a local executor, Intelligent Compute provisions and manages a multi-node ECS cluster. This allows pipelines to scale beyond a single instance while preserving fast startup times. + +When Intelligent Compute is enabled on an AWS Cloud compute environment, Seqera provisions and manages the following resources in your AWS account on first use: + +- An Amazon ECS cluster per compute environment configuration +- ECS capacity providers (Managed Instances or Auto Scaling Groups) +- ECS task definitions per container image and resource shape +- IAM roles for ECS task execution, EC2 instance profiles, and infrastructure management +- CloudWatch log groups under `/seqera/sched` + +All managed resources use the `seqera-sched-` prefix. Seqera creates them on first use and removes them automatically when no longer needed. + +## IAM permissions + +In addition to the [standard AWS Cloud IAM permissions](./aws-cloud#required-platform-iam-permissions), Intelligent Compute requires an additional policy attached to the same IAM user or role that Seqera uses to access your AWS account. + +
+Intelligent Compute IAM policy + +{AwsCloudIntelligentComputePolicy} + +
+ +[Download aws-cloud-intelligent-compute-policy.json](./_policies/aws-cloud-intelligent-compute-policy.json) + +### What each permission group does + +| Group | Purpose | +|-------|---------| +| `ECSScopedOperations` | Create, delete, describe, and tag ECS clusters, capacity providers, and tasks. Scoped to `seqera-sched-*` resources. | +| `ECSUnscopedOperations` | Register, deregister, list, and describe ECS task definitions. ECS task definition APIs do not support resource-level permissions. | +| `IAMRoleManagement` | Create, update, and delete IAM roles and instance profiles scoped to `seqera-sched-*`. Seqera creates four role types on first use: execution role, infrastructure role, per-cluster instance role, and per-cluster task role. | +| `PassRoleToECS` | Pass `seqera-sched-*` and `TowerForge-*` roles to ECS, ECS tasks, and EC2. Required to attach roles to ECS infrastructure and task definitions. | +| `ServiceLinkedRoles` | Create service-linked roles for ECS, autoscaling, and Spot. Required only if these roles do not already exist in your account. | +| `CloudWatchLogs` | Create and manage log groups under `/seqera/sched`, and read log events. Task stdout and stderr are written to CloudWatch. | +| `EC2NetworkDiscovery` | Describe VPCs, subnets, security groups, and route tables. Create security groups and VPC endpoints. Used for VPC auto-discovery and network setup. | +| `ECRAccess` | Authorize ECR and pull container images. ECS tasks pull images from ECR. | +| `S3Access` | Read objects and list buckets. Used to read Fusion trace files and pipeline work directory content. | +| `ASGEC2Operations` | Describe instance types and create or delete EC2 launch templates. Required only for Auto Scaling Group-backed clusters. | +| `ASGManagement` | Create, update, and delete Auto Scaling Groups scoped to `seqera-sched-*`. Required only for Auto Scaling Group-backed clusters. | +| `ASGDescribe` | Describe Auto Scaling Groups. Required only for Auto Scaling Group-backed clusters. | +| `SSMECSOptimizedAmi` | Read the ECS-optimized AMI ID from SSM Parameter Store. Used to look up the latest Amazon Linux 2023 ECS-optimized AMI. | +| `CostExplorer` | Query `ce:GetCostAndUsage`. Used to display cost predictions at pipeline launch. If this permission is absent, cost predictions do not appear. No error is surfaced to users. | + +**Conditional statements:** +- `ASGEC2Operations`, `ASGManagement`, and `ASGDescribe` are required only if Auto Scaling Group-backed clusters are enabled. You can omit them for Managed Instances deployments. +- `ServiceLinkedRoles` is required only if the listed service-linked roles do not already exist in your AWS account. +- `CostExplorer` is required only if you want cost predictions shown at pipeline launch. + +### Create the additional IAM policy + +1. Open the [AWS IAM console](https://console.aws.amazon.com/iam). +1. Select **Policies** under **Access management**, then select **Create policy**. +1. Select the **JSON** tab and paste the Intelligent Compute policy. +1. Select **Next**, enter a name (for example, `SeqeraIntelligentComputePolicy`), then select **Create policy**. +1. Attach the policy to the same IAM user or role that Seqera uses for your AWS Cloud compute environment. + +## Set up an AWS Cloud compute environment with Intelligent Compute + +Confirm with your account manager that Intelligent Compute is enabled for your organization before proceeding. + +1. In your Seqera workspace, select **Compute Environments**, then **Add compute environment**. +1. Enter a name and select **AWS Cloud** as the platform. +1. Select your AWS credentials. The credential must have both the standard AWS Cloud permissions and the Intelligent Compute permissions attached. +1. Select the **Region** where the ECS cluster will be provisioned. +1. Enter a **Work directory** (S3 URI, for example `s3://my-bucket/work`). +1. Under **Intelligent Compute**, enable the **Seqera Intelligent Compute** toggle. +1. Configure the [Intelligent Compute options](#configuration-options) below as needed. +1. Select **Add**. + +Seqera validates credentials and configuration on save. On first use, it provisions the required IAM roles and ECS cluster in your account. Provisioning is automatic and does not require additional steps. + +## Configuration options + +| Option | Values | Default | Description | +|--------|--------|---------|-------------| +| **Seqera Intelligent Compute** | Enabled / Disabled | Disabled | Enables the Intelligent Compute scheduler for this compute environment. This option only appears if Intelligent Compute is enabled for your organization. | +| **Provisioning model** | `spotFirst`, `spot`, `ondemand` | `spotFirst` | Instance procurement strategy. `spotFirst` uses Spot instances and falls back to On-Demand if Spot capacity is unavailable. `spot` uses Spot instances only. `ondemand` uses On-Demand instances only. | +| **Instance types** | Comma-separated EC2 instance type identifiers (for example, `m5.xlarge, c5.2xlarge`) | Empty | Restricts which instance types the scheduler can select. When empty, the scheduler selects the most cost-effective type for each task automatically. Specifying types here overrides automatic selection. | From 26cd6d29aaa77ca1e152204c0c0d05e50c653a4b Mon Sep 17 00:00:00 2001 From: MichaelTansiniSeqera Date: Fri, 12 Jun 2026 16:32:03 +0100 Subject: [PATCH 2/5] Update intelligent-compute.mdx Signed-off-by: MichaelTansiniSeqera --- platform-cloud/docs/compute-envs/intelligent-compute.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/platform-cloud/docs/compute-envs/intelligent-compute.mdx b/platform-cloud/docs/compute-envs/intelligent-compute.mdx index cf3737cac..16308d35d 100644 --- a/platform-cloud/docs/compute-envs/intelligent-compute.mdx +++ b/platform-cloud/docs/compute-envs/intelligent-compute.mdx @@ -25,7 +25,7 @@ Intelligent Compute is supported on **AWS Cloud compute environments only**. Intelligent Compute is a scheduling service that runs Nextflow pipelines on a Seqera-managed Amazon ECS cluster. It allocates compute resources based on what each task actually needs rather than what the pipeline requests, reducing cost and improving utilization across a run. -Unlike the standard AWS Cloud compute environment, which runs a pipeline on a single EC2 instance with a local executor, Intelligent Compute provisions and manages a multi-node ECS cluster. This allows pipelines to scale beyond a single instance while preserving fast startup times. +Unlike the standard AWS Cloud compute environment, which runs a pipeline on a single EC2 instance with a local executor, Intelligent Compute provisions and manages multi-node clusters. This allows pipelines to scale beyond a single instance while preserving fast startup times. When Intelligent Compute is enabled on an AWS Cloud compute environment, Seqera provisions and manages the following resources in your AWS account on first use: From d9cdab0f2ba9d88e42b1d04e6542a912edfbe591 Mon Sep 17 00:00:00 2001 From: MichaelTansiniSeqera Date: Fri, 12 Jun 2026 16:39:55 +0100 Subject: [PATCH 3/5] docs: add IAM policy inline, resource metrics, and task status reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove
collapse from IAM policy — policy is now always visible and copy-pasteable - Add Resource metrics section explaining Requested/Allocated/Used and how to interpret the gap between them - Add Task and run statuses reference table for troubleshooting Co-Authored-By: Claude Sonnet 4.6 --- .../docs/compute-envs/intelligent-compute.mdx | 46 +++++++++++++++++-- 1 file changed, 41 insertions(+), 5 deletions(-) diff --git a/platform-cloud/docs/compute-envs/intelligent-compute.mdx b/platform-cloud/docs/compute-envs/intelligent-compute.mdx index 16308d35d..b6ec943ff 100644 --- a/platform-cloud/docs/compute-envs/intelligent-compute.mdx +++ b/platform-cloud/docs/compute-envs/intelligent-compute.mdx @@ -41,13 +41,8 @@ All managed resources use the `seqera-sched-` prefix. Seqera creates them on fir In addition to the [standard AWS Cloud IAM permissions](./aws-cloud#required-platform-iam-permissions), Intelligent Compute requires an additional policy attached to the same IAM user or role that Seqera uses to access your AWS account. -
-Intelligent Compute IAM policy - {AwsCloudIntelligentComputePolicy} -
- [Download aws-cloud-intelligent-compute-policy.json](./_policies/aws-cloud-intelligent-compute-policy.json) ### What each permission group does @@ -97,6 +92,22 @@ Confirm with your account manager that Intelligent Compute is enabled for your o Seqera validates credentials and configuration on save. On first use, it provisions the required IAM roles and ECS cluster in your account. Provisioning is automatic and does not require additional steps. +## Resource metrics + +The **Metrics** tab for a workflow run on Intelligent Compute shows three resource values for CPU and memory: **Requested**, **Allocated**, and **Used**. + +| Metric | Source | What it represents | +|--------|--------|-------------------| +| **Requested** | Pipeline `process` directives | The CPU and memory your pipeline asked for, as written in your `process` directives (for example, `cpus = 4`, `memory = 8 GB`). | +| **Allocated** | Scheduler decision | The CPU and memory the scheduler actually assigned to the task container. Intelligent Compute may assign values different from what was requested — it selects the most cost-effective instance shape that satisfies the task's requirements. | +| **Used** | Nextflow trace data | The CPU and memory the task actually consumed, measured from Nextflow's trace metrics (`pcpu` × `realtime` for CPU, `peakRss` for memory). Requires Fusion to be enabled. Absent for tasks that did not produce trace data. | + +**How to read the numbers:** + +- If **Requested** is much higher than **Allocated**, the scheduler found a more efficient instance shape than your directives implied. +- If **Allocated** is much higher than **Used**, the task ran with significant idle headroom. You may be able to reduce your process resource directives on future runs to lower cost. +- If **Used** is close to **Allocated**, resource utilization is near-optimal for that task. + ## Configuration options | Option | Values | Default | Description | @@ -104,3 +115,28 @@ Seqera validates credentials and configuration on save. On first use, it provisi | **Seqera Intelligent Compute** | Enabled / Disabled | Disabled | Enables the Intelligent Compute scheduler for this compute environment. This option only appears if Intelligent Compute is enabled for your organization. | | **Provisioning model** | `spotFirst`, `spot`, `ondemand` | `spotFirst` | Instance procurement strategy. `spotFirst` uses Spot instances and falls back to On-Demand if Spot capacity is unavailable. `spot` uses Spot instances only. `ondemand` uses On-Demand instances only. | | **Instance types** | Comma-separated EC2 instance type identifiers (for example, `m5.xlarge, c5.2xlarge`) | Empty | Restricts which instance types the scheduler can select. When empty, the scheduler selects the most cost-effective type for each task automatically. Specifying types here overrides automatic selection. | + +## Task and run statuses + +### Task statuses + +| Status | Description | +|--------|-------------| +| `SUBMITTED` | Task is queued or submitted to the compute backend. | +| `RUNNING` | Task is actively executing on a compute instance. | +| `SUCCEEDED` | Task completed with exit code 0. | +| `FAILED` | Task failed. This covers non-retriable execution failures (non-zero exit code, container startup errors) and spot quota exhaustion after retries are exhausted. | +| `CANCELLED` | Task was cancelled by the user. | +| `PREEMPTED` | The Spot instance running this task was reclaimed by AWS. The scheduler retries the task automatically. If the retry limit is reached, the task transitions to `FAILED`. | +| `UNSCHEDULABLE` | No instance type could satisfy the task's placement constraints. This occurs when the requested resources exceed what any available instance type can provide, or when specified instance types are unavailable in the region. Check your **Instance types** configuration and the `cpus` and `memory` directives in the failing process. | +| `UNKNOWN` | Task status could not be determined, typically due to a transient backend failure. | + +### Run statuses + +| Status | Description | +|--------|-------------| +| `ACTIVE` | The run is in progress. | +| `TERMINATING` | The run is shutting down — final tasks are completing or being cancelled. | +| `TERMINATED` | The run ended normally. | +| `FAILED` | The run failed. | +| `DANGLING` | The Nextflow process stopped sending heartbeats. This typically means the launcher process crashed or lost connectivity. Tasks already dispatched to ECS may still be running. Check CloudWatch logs under `/seqera/sched` for details. | From 9c443b47b5b36f5ff9fe73a45b920a2373e03182 Mon Sep 17 00:00:00 2001 From: MichaelTansiniSeqera Date: Fri, 12 Jun 2026 17:02:34 +0100 Subject: [PATCH 4/5] docs: add serviceAccountTokenCreator requirement for WIF + Data Explorer on GCP When WIF credentials are used for Data Explorer, Platform has no embedded private key and must call the GCP IAM signBlob API to generate presigned URLs. roles/iam.serviceAccountTokenCreator on the SA itself is required for this to succeed. Without it, file viewing and download fail silently with a signing error. Running pipelines is unaffected. Updates both Cloud and Enterprise Google Cloud Batch docs. Co-Authored-By: Claude Sonnet 4.6 --- platform-cloud/docs/compute-envs/google-cloud-batch.md | 9 +++++++++ .../compute-envs/google-cloud-batch.md | 9 +++++++++ 2 files changed, 18 insertions(+) diff --git a/platform-cloud/docs/compute-envs/google-cloud-batch.md b/platform-cloud/docs/compute-envs/google-cloud-batch.md index 13a2aa078..5a008f10c 100644 --- a/platform-cloud/docs/compute-envs/google-cloud-batch.md +++ b/platform-cloud/docs/compute-envs/google-cloud-batch.md @@ -112,6 +112,15 @@ Setting up WIF requires the following steps in the GCP Console: tityPools/{POOL}/providers/{PROVIDER}`. If you specify a custom value, it must match exactly what you enter in the Token audience field when creating the Google WIF credential in Seqera. 4. Define an attribute mapping and condition. At a minimum set `google.subject=assertion.sub`. This maps the subject claim from Seqera's JWT to GCP's identity space. For more information see [here](https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-other-providers#mappings-and-conditions). You may see a pop-up asking to configure your application and provide an OIDC ID token path. This pop-up can be dismissed. 5. Grant `roles/iam.workloadIdentityUser` on the service account that WIF will impersonate to the Workload Identity Pool principal. This can be set for all pool identities or for a specific workspace. If you have not yet created a service account do so following the guidelines above. +6. If you use the same WIF credential for Data Explorer, grant `roles/iam.serviceAccountTokenCreator` on the service account to itself: + + ```bash + gcloud iam service-accounts add-iam-policy-binding SA_EMAIL \ + --member="serviceAccount:SA_EMAIL" \ + --role="roles/iam.serviceAccountTokenCreator" + ``` + + Replace `SA_EMAIL` with the service account email. Without this role, viewing or downloading file contents in Data Explorer fails with a signing error. Running pipelines is not affected. After setting up WIF in the GCP Console, you need the following information to create a credential in Seqera Platform: diff --git a/platform-enterprise_docs/compute-envs/google-cloud-batch.md b/platform-enterprise_docs/compute-envs/google-cloud-batch.md index 9f89ce681..f58311f98 100644 --- a/platform-enterprise_docs/compute-envs/google-cloud-batch.md +++ b/platform-enterprise_docs/compute-envs/google-cloud-batch.md @@ -107,6 +107,15 @@ Workload Identity Federation (WIF) is the recommended authentication method for 3. Set the Allowed audiences. If left empty, GCP derives a default audience from the provider resource path in the format `//iam.googleapis.com/projects/{PROJECT}/locations/global/workloadIdentityPools/{POOL}/providers/{PROVIDER}`. If you specify a custom value, it must match exactly what you enter in the Token audience field when creating the Google WIF credential in Seqera. 4. Define an attribute mapping and condition. At a minimum set `google.subject=assertion.sub`. This maps the subject claim from Seqera's JWT to GCP's identity space. For more information see [here](https://docs.cloud.google.com/iam/docs/workload-identity-federation-with-other-providers#mappings-and-conditions) 5. Grant `roles/iam.workloadIdentityUser` on the service account created above to the Workload Identity Pool principal. This can be set for all pool identities or for a specific workspace. +6. If you use the same WIF credential for Data Explorer, grant `roles/iam.serviceAccountTokenCreator` on the service account to itself: + + ```bash + gcloud iam service-accounts add-iam-policy-binding SA_EMAIL \ + --member="serviceAccount:SA_EMAIL" \ + --role="roles/iam.serviceAccountTokenCreator" + ``` + + Replace `SA_EMAIL` with the service account email. Without this role, viewing or downloading file contents in Data Explorer fails with a signing error. Running pipelines is not affected. WIF requires an OIDC signing key and for Seqera Platform's OIDC provider to be configured. See [Cryptographic options](https://docs.seqera.io/platform-enterprise/enterprise/configuration/overview#cryptographic-options). From 99e38703f9075c176b16cd8d3069fba51140a2b1 Mon Sep 17 00:00:00 2001 From: Chris Hakkaart Date: Tue, 16 Jun 2026 16:05:39 +1200 Subject: [PATCH 5/5] docs: Docs/intelligent compute AWS preview (#1567) * Deslop and tidy tables * Remove bold * Update intelligent-compute.mdx Signed-off-by: MichaelTansiniSeqera * [automated] Fix code formatting * chore: add 'intelligent compute' to doc-tags-allowed.txt --------- Signed-off-by: MichaelTansiniSeqera Co-authored-by: MichaelTansiniSeqera Co-authored-by: Seqera Docs Bot Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> --- .github/doc-tags-allowed.txt | 1 + .../docs/compute-envs/intelligent-compute.mdx | 70 +++++++++++-------- 2 files changed, 40 insertions(+), 31 deletions(-) diff --git a/.github/doc-tags-allowed.txt b/.github/doc-tags-allowed.txt index b3f773621..a1d040716 100644 --- a/.github/doc-tags-allowed.txt +++ b/.github/doc-tags-allowed.txt @@ -141,6 +141,7 @@ image input installation integration +intelligent compute interactive jupyter k8s diff --git a/platform-cloud/docs/compute-envs/intelligent-compute.mdx b/platform-cloud/docs/compute-envs/intelligent-compute.mdx index b6ec943ff..bef759639 100644 --- a/platform-cloud/docs/compute-envs/intelligent-compute.mdx +++ b/platform-cloud/docs/compute-envs/intelligent-compute.mdx @@ -12,20 +12,16 @@ import CodeBlock from '@theme/CodeBlock'; import AwsCloudIntelligentComputePolicy from './_policies/aws-cloud-intelligent-compute-policy.json?raw'; :::info[Preview] -Seqera Intelligent Compute is in preview and must be enabled for your organization by Seqera before you can use it. Contact your account manager to request access. +Seqera Intelligent Compute is in preview. Seqera must enable it for your organization before you can use it. Contact your account manager to request access. ::: :::caution Intelligent Compute may assign different CPU and memory values to tasks than those specified in your pipeline's `process` directives. The scheduler selects the most cost-effective instance that meets each task's resource request rather than provisioning exactly what the directive specifies. ::: -Intelligent Compute is supported on **AWS Cloud compute environments only**. +Intelligent Compute is a scheduling service that runs Nextflow pipelines on a Seqera-managed Amazon ECS cluster. It allocates compute resources based on what each task needs rather than what the pipeline requests, which reduces cost and improves utilization across a run. Intelligent Compute is supported on AWS Cloud compute environments only. -## What is Intelligent Compute - -Intelligent Compute is a scheduling service that runs Nextflow pipelines on a Seqera-managed Amazon ECS cluster. It allocates compute resources based on what each task actually needs rather than what the pipeline requests, reducing cost and improving utilization across a run. - -Unlike the standard AWS Cloud compute environment, which runs a pipeline on a single EC2 instance with a local executor, Intelligent Compute provisions and manages multi-node clusters. This allows pipelines to scale beyond a single instance while preserving fast startup times. +Unlike the standard AWS Cloud compute environment, which runs a pipeline on a single EC2 instance with a local executor, Intelligent Compute provisions and manages multi-node clusters. Pipelines can scale beyond a single instance while keeping startup times short. When Intelligent Compute is enabled on an AWS Cloud compute environment, Seqera provisions and manages the following resources in your AWS account on first use: @@ -41,11 +37,14 @@ All managed resources use the `seqera-sched-` prefix. Seqera creates them on fir In addition to the [standard AWS Cloud IAM permissions](./aws-cloud#required-platform-iam-permissions), Intelligent Compute requires an additional policy attached to the same IAM user or role that Seqera uses to access your AWS account. -{AwsCloudIntelligentComputePolicy} +
+ IAM policy JSON + {AwsCloudIntelligentComputePolicy} +
[Download aws-cloud-intelligent-compute-policy.json](./_policies/aws-cloud-intelligent-compute-policy.json) -### What each permission group does +### Permission groups | Group | Purpose | |-------|---------| @@ -62,7 +61,7 @@ In addition to the [standard AWS Cloud IAM permissions](./aws-cloud#required-pla | `ASGManagement` | Create, update, and delete Auto Scaling Groups scoped to `seqera-sched-*`. Required only for Auto Scaling Group-backed clusters. | | `ASGDescribe` | Describe Auto Scaling Groups. Required only for Auto Scaling Group-backed clusters. | | `SSMECSOptimizedAmi` | Read the ECS-optimized AMI ID from SSM Parameter Store. Used to look up the latest Amazon Linux 2023 ECS-optimized AMI. | -| `CostExplorer` | Query `ce:GetCostAndUsage`. Used to display cost predictions at pipeline launch. If this permission is absent, cost predictions do not appear. No error is surfaced to users. | +| `CostExplorer` | Query `ce:GetCostAndUsage`. Used to display cost predictions at pipeline launch. If this permission is absent, cost predictions do not appear and Seqera shows no error. | **Conditional statements:** - `ASGEC2Operations`, `ASGManagement`, and `ASGDescribe` are required only if Auto Scaling Group-backed clusters are enabled. You can omit them for Managed Instances deployments. @@ -79,33 +78,40 @@ In addition to the [standard AWS Cloud IAM permissions](./aws-cloud#required-pla ## Set up an AWS Cloud compute environment with Intelligent Compute -Confirm with your account manager that Intelligent Compute is enabled for your organization before proceeding. +:::info[**Prerequisites**] + +You need the following: + +- Intelligent Compute enabled for your organization by Seqera. Contact your account manager to request access. +- AWS credentials with both the standard AWS Cloud permissions and the Intelligent Compute permissions attached. + +::: 1. In your Seqera workspace, select **Compute Environments**, then **Add compute environment**. 1. Enter a name and select **AWS Cloud** as the platform. -1. Select your AWS credentials. The credential must have both the standard AWS Cloud permissions and the Intelligent Compute permissions attached. -1. Select the **Region** where the ECS cluster will be provisioned. +1. Select your AWS credentials. +1. Select the **Region** where Seqera provisions the ECS cluster. 1. Enter a **Work directory** (S3 URI, for example `s3://my-bucket/work`). 1. Under **Intelligent Compute**, enable the **Seqera Intelligent Compute** toggle. -1. Configure the [Intelligent Compute options](#configuration-options) below as needed. +1. Configure the [Intelligent Compute options](#configuration-options) as needed. 1. Select **Add**. -Seqera validates credentials and configuration on save. On first use, it provisions the required IAM roles and ECS cluster in your account. Provisioning is automatic and does not require additional steps. +Seqera validates credentials and configuration on save. On first use, it provisions the required IAM roles and ECS cluster in your account. This provisioning is automatic and requires no further steps. ## Resource metrics -The **Metrics** tab for a workflow run on Intelligent Compute shows three resource values for CPU and memory: **Requested**, **Allocated**, and **Used**. +The **Metrics** tab for a run on Intelligent Compute shows three resource values for CPU and memory: **Requested**, **Allocated**, and **Used**. | Metric | Source | What it represents | |--------|--------|-------------------| | **Requested** | Pipeline `process` directives | The CPU and memory your pipeline asked for, as written in your `process` directives (for example, `cpus = 4`, `memory = 8 GB`). | -| **Allocated** | Scheduler decision | The CPU and memory the scheduler actually assigned to the task container. Intelligent Compute may assign values different from what was requested — it selects the most cost-effective instance shape that satisfies the task's requirements. | +| **Allocated** | Scheduler decision | The CPU and memory the scheduler assigned to the task container. Intelligent Compute may assign values different from what was requested. It selects the most cost-effective instance shape that satisfies the task's requirements. | | **Used** | Nextflow trace data | The CPU and memory the task actually consumed, measured from Nextflow's trace metrics (`pcpu` × `realtime` for CPU, `peakRss` for memory). Requires Fusion to be enabled. Absent for tasks that did not produce trace data. | **How to read the numbers:** - If **Requested** is much higher than **Allocated**, the scheduler found a more efficient instance shape than your directives implied. -- If **Allocated** is much higher than **Used**, the task ran with significant idle headroom. You may be able to reduce your process resource directives on future runs to lower cost. +- If **Allocated** is much higher than **Used**, the task ran with idle headroom. - If **Used** is close to **Allocated**, resource utilization is near-optimal for that task. ## Configuration options @@ -118,25 +124,27 @@ The **Metrics** tab for a workflow run on Intelligent Compute shows three resour ## Task and run statuses +Intelligent Compute reports a status for each task and for the run as a whole. Use these statuses to track progress and diagnose failures. + ### Task statuses | Status | Description | |--------|-------------| -| `SUBMITTED` | Task is queued or submitted to the compute backend. | -| `RUNNING` | Task is actively executing on a compute instance. | -| `SUCCEEDED` | Task completed with exit code 0. | -| `FAILED` | Task failed. This covers non-retriable execution failures (non-zero exit code, container startup errors) and spot quota exhaustion after retries are exhausted. | -| `CANCELLED` | Task was cancelled by the user. | -| `PREEMPTED` | The Spot instance running this task was reclaimed by AWS. The scheduler retries the task automatically. If the retry limit is reached, the task transitions to `FAILED`. | -| `UNSCHEDULABLE` | No instance type could satisfy the task's placement constraints. This occurs when the requested resources exceed what any available instance type can provide, or when specified instance types are unavailable in the region. Check your **Instance types** configuration and the `cpus` and `memory` directives in the failing process. | -| `UNKNOWN` | Task status could not be determined, typically due to a transient backend failure. | +| SUBMITTED | Task is queued or submitted to the compute backend. | +| RUNNING | Task is actively executing on a compute instance. | +| SUCCEEDED | Task completed with exit code 0. | +| FAILED | Task failed. This covers non-retriable execution failures (non-zero exit code, container startup errors) and spot quota exhaustion after retries are exhausted. | +| CANCELLED | Task was cancelled by the user. | +| PREEMPTED | The Spot instance running this task was reclaimed by AWS. The scheduler retries the task automatically. If the retry limit is reached, the task transitions to `FAILED`. | +| UNSCHEDULABLE | No instance type could satisfy the task's placement constraints. This occurs when the requested resources exceed what any available instance type can provide, or when specified instance types are unavailable in the region. Check your **Instance types** configuration and the `cpus` and `memory` directives in the failing process. | +| UNKNOWN | Task status could not be determined, typically due to a transient backend failure. | ### Run statuses | Status | Description | |--------|-------------| -| `ACTIVE` | The run is in progress. | -| `TERMINATING` | The run is shutting down — final tasks are completing or being cancelled. | -| `TERMINATED` | The run ended normally. | -| `FAILED` | The run failed. | -| `DANGLING` | The Nextflow process stopped sending heartbeats. This typically means the launcher process crashed or lost connectivity. Tasks already dispatched to ECS may still be running. Check CloudWatch logs under `/seqera/sched` for details. | +| ACTIVE | The run is in progress. | +| TERMINATING | The run is shutting down. Final tasks are completing or being cancelled. | +| TERMINATED | The run ended normally. | +| FAILED | The run failed. | +| DANGLING | The Nextflow process stopped sending heartbeats. This typically means the launcher process crashed or lost connectivity. Tasks already dispatched to ECS may still be running. Check CloudWatch logs under `/seqera/sched` for details. |