Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions macros/data-orchestrator/run-workflow.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
macro: run-workflow
---

1. Click **Data Orchestrator** in the **Data & Analytics** section of the [console](https://console.scaleway.com/) side menu. The list of your workflows displays.
2. Click the name of the workflow you want to run. The workflow **Overview** tab displays.
3. Click the **Actions** button, then click **Run** in the drop-down menu. Your workflow starts, and a new entry appears in the **Runs** tab.

<Message type="tip">
On the **Data Orchestrator** page, you can also click the run icon <Icon name="play" /> next to the workflow you want to run.
</Message>
9 changes: 9 additions & 0 deletions macros/data-orchestrator/upload-workflow.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
macro: upload-workflow
---

The workflow upload window is displayed during the creation of a workflow or when clicking **Edit** from a workflow in Code view.

1. Click **Upload**.
2. Select the YAML file to use as the workflow definition.
3. Click **Save changes** to finish, or click **Save and run** if you want to save and run the workflow.
2 changes: 2 additions & 0 deletions menu/navigation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import { cockpitMenu } from "../pages/cockpit/menu"
import { containerRegistryMenu } from "../pages/container-registry/menu"
import { cpanelHostingMenu } from "../pages/cpanel-hosting/menu"
import { dataLabMenu } from "../pages/data-lab/menu"
import { dataOrchestratorMenu } from "../pages/data-orchestrator/menu"
import { dataWarehouseMenu } from "../pages/data-warehouse/menu"
import { dediboxMenu } from "../pages/dedibox/menu"
import { dediboxAccountMenu } from "../pages/dedibox-account/menu"
Expand Down Expand Up @@ -155,6 +156,7 @@ export default [
{
icon: 'DataAndAnalyticsCategoryIcon',
items: [
dataOrchestratorMenu,
dataWarehouseMenu,
dataLabMenu,
clustersForKafkaMenu,
Expand Down
61 changes: 61 additions & 0 deletions pages/data-orchestrator/concepts.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: Data Orchestrator - Concepts
description: Learn the fundamental concepts of Scaleway Data Orchestrator.
tags: data-orchestrator
dates:
validation: 2026-04-01
---

## Orchestration

Orchestration is the automated coordination of tasks and workflows that keeps data operations reliable, scalable, and maintainable. In the context of Scaleway Data Orchestrator, it enables users to define, schedule, and manage complex data pipelines. It also handles dependencies, error recovery, and execution order seamlessly. Instead of manually triggering scripts or monitoring jobs, orchestration brings structure and intelligence, turning fragmented processes into unified, business-aligned workflows.

## Tasks

### Action task

An action task represents the executable unit within a workflow that performs concrete work. An action task can be:
- **Serverless Jobs**: Long-running batch processes that scale automatically without infrastructure management.
- **Serverless Functions**: Lightweight, event-driven code execution for quick transformations or API calls.
- **Spark Jobs**: Distributed data processing tasks for large-scale ETL or analytics using Apache Spark.
- Other compute-intensive or service-specific jobs (e.g., data validation, model inference).

These tasks are orchestrated in sequence or in parallel, forming the backbone of data processing pipelines.

### Logic task

A Logic task controls the flow and decision-making within a workflow, enabling dynamic behavior beyond simple linear execution. A logic task can be:
- **Switch**: Direct flow based on runtime conditions (e.g., file size, data quality).
- **Fork**: Split execution into parallel branches to process data concurrently.
- **Try catch**: Implement error-handling blocks to manage failures and enable retries or fallback logic.

These tasks allow users to embed business logic directly into pipelines, making them resilient and adaptable.

## Trigger

A trigger is the event that initiates a workflow execution. A trigger can be:
- **Manual**: User starts the run via the Scaleway Console or CLI (ideal for testing).
- **Schedule**: Automatic execution based on time (e.g., daily at 8:00 AM), set with a built-in scheduler.
- **Event**: Triggered by external signals (e.g., new file in object storage, message in a queue), enabling reactive, real-time data processing.

## Views

### Code view

Every workflow can be visualized as code, showing tasks and their dependencies.

### Graph view

Every workflow can be visualized as a Directed Acyclic Graph (DAG), showing the tasks and their dependencies.

## Workflow

A workflow is a structured sequence of action tasks and logical tasks that define an end-to-end data process.

### Workflow definition

The declarative blueprint of a workflow, typically described in code (e.g., YAML or Python) or designed visually. It specifies tasks, dependencies, conditions, and execution parameters. This definition is version-controlled, reusable, and portable across environments.

### Workflow execution / run

The runtime instance of a workflow definition. Each execution (or run) tracks the state, logs, and results of every task, providing full observability and auditability. Runs can succeed, fail, or be paused, with detailed insights for debugging.
88 changes: 88 additions & 0 deletions pages/data-orchestrator/how-to/create-workflow.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
---
title: How to create a workflow using Data Orchestrator
description: This page explains how to create a Data Orchestrator workflow on the Scaleway console.
tags: data-orchestrator workflow create upload
dates:
posted: 2026-04-01
---
import Requirements from '@macros/iam/requirements.mdx'
import UploadWorkflow from '@macros/data-orchestrator/upload-workflow.mdx'
import RunWorkflow from '@macros/data-orchestrator/run-workflow.mdx'


<Message type="note">
Data Orchestrator is currently in Private Beta. During this Beta period:
- **Code mode** is the only supported mode.
- The workflows only run Scaleway Serverless Jobs.
</Message>

<Requirements />

- A Scaleway account logged into the [console](https://console.scaleway.com)
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
- A [Container Registry namespace](/container-registry/how-to/create-namespace/)
- An image [pushed](/container-registry/how-to/push-images/) to your namespace (for task execution)
- Created at least three [Serverless Jobs](/serverless-jobs/how-to/create-job/)
- Basic familiarity with YAML syntax
- Basic familiarity with [Serverless workflow.io](https://serverlessworkflow.io/) language

## Create a workflow definition YAML file

Data Orchestrator requires the user to upload a YAML file to use as a workflow definition.

<Message type="tip">
This is a basic example of a sequential workflow, more examples will be available soon.
</Message>

1. Create a YAML file, copy/paste the following content into the file.
```yaml
document:
dsl: 1.0.0
namespace: <namespace_name>
name: <workflow_name>
version: 1.0.0
do:
- task1:
call: "serverless_job"
metadata:
description: "The first job"
with:
id: "<Serverless_job1_ID>"
region: "fr-par"
- task2:
call: "serverless_job"
metadata:
description: "The second job"
with:
id: "<Serverless_job2_ID>"
region: "fr-par"
- task3:
call: "serverless_job"
metadata:
description: "The third job"
with:
id: "<Serverless_job3_ID>"
region: "fr-par"
```
2. Replace the ID placeholders with the IDs of your Serverless Jobs. You can copy them from the [Serverless Jobs page](https://console.scaleway.com/serverless-jobs/).
<Message type="note">
The fields `namespace` and `name` are required. They can be used as metadata to help organize your YAML files.
</Message>
3. Save the YAML file.

## Create a Data Orchestrator workflow

1. Click **Data Orchestrator** in the **Data & Analytics** section of the [console](https://console.scaleway.com/) side menu. The list of your workflows displays.
2. Click **+ Create workflow**. The Data Orchestrator workflow creation wizard displays.
3. Select the region in which your workflow will be created.
4. Enter a name or use the automatically generated one.
5. Enter a description (optional)
6. Click **Create workflow only** to finish, or click **Create workflow and add tasks** if you want to upload a workflow next.

## Upload the workflow definition

<UploadWorkflow />

## Run a workflow

<RunWorkflow />
4 changes: 4 additions & 0 deletions pages/data-orchestrator/how-to/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
---
title: Data Orchestrator - How Tos
description: Practical guides for using Scaleway Data Orchestrator.
---
25 changes: 25 additions & 0 deletions pages/data-orchestrator/how-to/run-workflow.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
title: How to run a workflow using Data Orchestrator
description: This page explains how to run a Data Orchestrator workflow on the Scaleway console.
tags: data-orchestrator workflow run
dates:
posted: 2026-04-20
---
import Requirements from '@macros/iam/requirements.mdx'
import RunWorkflow from '@macros/data-orchestrator/run-workflow.mdx'

<Message type="note">
Data Orchestrator is currently in Private Beta. During this Beta period:
- **Code mode** is the only supported mode.
- The workflows only run Scaleway Serverless Jobs.
</Message>

<Requirements />

- A Scaleway account logged into the [console](https://console.scaleway.com)
- [Owner](/iam/concepts/#owner) status or [IAM permissions](/iam/concepts/#permission) allowing you to perform actions in the intended Organization
- Created a [Data Orchestrator workflow](/data-orchestrator/how-to/create-workflow/)

## Run a workflow

<RunWorkflow />
42 changes: 42 additions & 0 deletions pages/data-orchestrator/index.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: Data Orchestrator Documentation
description: Comprehensive documentation on Scaleway Data Orchestrator.
---

<Message type="note">
Data Orchestrator is currently in Private Beta.
</Message>

<ProductHeader
productName="Data Orchestrator"
productLogo="data-orchestrator"
description="Scaleway Data Orchestrator is designed to help the user automate, schedule, and manage data workflows across various systems and environments."
url="/data-orchestrator/quickstart/"
label="Data Orchestrator Quickstart"
/>

## Getting Started

<Grid>
<SummaryCard
title="Concepts"
icon="info"
description="Core concepts that give you a better understanding of Scaleway Data Orchestrator."
label="View Concepts"
url="/data-orchestrator/concepts/"
/>
<SummaryCard
title="How-Tos"
icon="help-circle-outline"
description="Learn how to use Data Orchestrator and create your first workflow."
label="View How-Tos"
url="/data-orchestrator/how-to/"
/>
</Grid>

## Changelog

<ChangelogList
productName="data-orchestrator"
numberOfChanges={0}
/>
28 changes: 28 additions & 0 deletions pages/data-orchestrator/menu.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
export const dataOrchestratorMenu = {
items: [
{
label: 'Overview',
slug: '../data-orchestrator',
},
{
label: 'Concepts',
slug: 'concepts',
},
{
items: [
{
label: 'Create a workflow',
slug: 'create-workflow',
},
{
label: 'Run a workflow',
slug: 'run-workflow',
},
],
label: 'How to',
slug: 'how-to',
},
],
label: 'Data Orchestrator',
slug: 'data-orchestrator',
}
Loading