Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,8 @@ databases
.env.*
.git
.github
docs
docs/*
!docs/serviceTemplates
src/test
*.md
*.log
Expand Down
15 changes: 8 additions & 7 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,14 @@ export P2P_BOOTSTRAP_NODES=
export P2P_FILTER_ANNOUNCED_ADDRESSES=

## compute
# Each environment defines its own resources (CPU, RAM, disk, GPUs) with full configuration.
# CPU, RAM, and disk are per-env exclusive: inUse tracked only within the environment where the job runs.
# A global check ensures the aggregate usage across all environments does not exceed physical capacity.
# GPUs are shared-exclusive: if a job on envA uses gpu0, it shows as in-use on envB too.
# CPU cores are automatically partitioned across environments based on each env's cpu.total.
# CPU and RAM defaults are auto-detected from the system when not configured.
# export DOCKER_COMPUTE_ENVIRONMENTS='[{"socketPath":"/var/run/docker.sock","environments":[{"id":"envA","storageExpiry":604800,"maxJobDuration":3600,"minJobDuration":60,"resources":[{"id":"cpu","total":4,"max":4,"min":1,"type":"cpu"},{"id":"ram","total":16,"max":16,"min":1,"type":"ram"},{"id":"disk","total":500,"max":500,"min":10,"type":"disk"},{"id":"gpu0","total":1,"max":1,"min":0,"type":"gpu","init":{"deviceRequests":{"Driver":"nvidia","DeviceIDs":["0"],"Capabilities":[["gpu"]]}}}],"fees":{"1":[{"feeToken":"0x123","prices":[{"id":"cpu","price":1},{"id":"ram","price":0.1},{"id":"disk","price":0.01},{"id":"gpu0","price":5}]}]}}]}]'
# Resources are defined at the Docker-connection level (socketPath) and shared across all environments.
# cpu, ram, and disk are auto-detected from the host — omit them to use all available capacity,
# or include them to cap/reserve (e.g. limit an 8-core host to 6 cores for compute).
# GPUs and other hardware go in the connection-level "resources" array with kind:"discrete".
# Each environment references pool resources by id using lightweight refs {id, total?, min?, max?}.
# Dual-gate tracking for fungible resources: per-env ceiling (Gate 1) + engine-wide pool (Gate 2).
# Discrete resources (GPUs) are tracked globally — a GPU in use on envA shows as in-use on envB too.
# export DOCKER_COMPUTE_ENVIRONMENTS='[{"socketPath":"/var/run/docker.sock","resources":[{"id":"disk","total":500},{"id":"gpu0","kind":"discrete","type":"gpu","total":1,"description":"NVIDIA A100","platform":"nvidia","driverVersion":"570.195.03","init":{"deviceRequests":{"Driver":"nvidia","DeviceIDs":["GPU-uuid-a"],"Capabilities":[["gpu"]]}}}],"environments":[{"id":"envA","storageExpiry":604800,"maxJobDuration":3600,"minJobDuration":60,"resources":[{"id":"cpu"},{"id":"ram"},{"id":"disk","max":500},{"id":"gpu0"}],"fees":{"1":[{"feeToken":"0x123","prices":[{"id":"cpu","price":1},{"id":"ram","price":0.1},{"id":"disk","price":0.01},{"id":"gpu0","price":5}]}]}}]}]'
export DOCKER_COMPUTE_ENVIRONMENTS=


3 changes: 2 additions & 1 deletion .eslintrc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
"jest": true
},
"globals": {
"NodeJS": true
"NodeJS": true,
"RequestInit": true
}
}
15 changes: 10 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 'v22.15.0'
node-version: 'v22.22.2'
- name: Cache node_modules
uses: actions/cache@v3
env:
Expand All @@ -42,7 +42,7 @@ jobs:
fail-fast: false
matrix:
os: [ubuntu-latest]
node: ['18.20.4', 'v20.19.0', 'v22.15.0']
node: ['v22.22.2']

steps:
- uses: actions/checkout@v4
Expand All @@ -66,7 +66,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 'v22.15.0'
node-version: 'v22.22.2'
- name: Cache node_modules
uses: actions/cache@v3
env:
Expand Down Expand Up @@ -102,7 +102,7 @@ jobs:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 'v22.15.0'
node-version: 'v22.22.2'
- name: Cache node_modules
uses: actions/cache@v3
env:
Expand All @@ -118,6 +118,7 @@ jobs:
with:
repository: 'oceanprotocol/barge'
path: 'barge'
ref: '43cfdfd21154a2bae00770b779e7c39390ff5043'
- name: Login to Docker Hub
if: ${{ env.DOCKERHUB_PASSWORDNONO && env.DOCKERHUB_USERNAMENONO }}
run: |
Expand All @@ -129,6 +130,8 @@ jobs:
working-directory: ${{ github.workspace }}/barge
run: |
bash -x start_ocean.sh --no-node --with-typesense 2>&1 > start_ocean.log &
env:
CONTRACTS_VERSION: '2.9.0'
- run: npm ci
- run: npm run build
- run: docker image ls
Expand Down Expand Up @@ -194,7 +197,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: 'v22.15.0'
node-version: 'v22.22.2'

- name: Cache node_modules
uses: actions/cache@v3
Expand Down Expand Up @@ -226,6 +229,8 @@ jobs:
working-directory: ${{ github.workspace }}/barge
run: |
bash -x start_ocean.sh --no-node --with-typesense 2>&1 > start_ocean.log &
env:
CONTRACTS_VERSION: '2.9.0'
- run: docker image ls
- name: Delete default runner images
run: |
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ on:
pull_request:
branches:
- 'main'
- 'next-4'

env:
DOCKERHUB_IMAGE: ${{ 'oceanprotocol/ocean-node' }}
Expand Down
2 changes: 1 addition & 1 deletion .nvmrc
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
22
22.22.2

22 changes: 12 additions & 10 deletions CLAUDE.md
100755 → 100644
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.


Ocean Node is the all-in-one backend for the Ocean Protocol stack. A single Node process
replaces three legacy components: **Provider** (data access / encryption / compute),
**Aquarius** (metadata cache) and the **subgraph** (on-chain event indexing). It is a
Expand All @@ -13,12 +12,12 @@ of which dispatch to the same set of command handlers.

## 1. Environment & tooling prerequisites

- **Node.js 22 is required** (`.nvmrc` pins `22`). Always run `nvm use` (or
`source ~/.nvm/nvm.sh && nvm use`) before any `npm`, build, or test command. The wrong
Node version fails with errors like `Unexpected token 'with'` or missing `GLIBC_2.38`.
- **Node.js ≥ 22.13 is required** (`.nvmrc` pins `22.22.2`, matching the Dockerfile and CI;
`package.json` `engines` requires `>=22.13.0`). Always run `nvm use` (or `source ~/.nvm/nvm.sh && nvm use`) before
any `npm`, build, or test command. The wrong Node version fails with errors like
`Unexpected token 'with'`, missing `GLIBC_2.38`, or — since the SQLite layer uses the
built-in `node:sqlite` module — `ERR_UNKNOWN_BUILTIN_MODULE: node:sqlite` on Node < 22.13.
This is enforced by `.cursor/rules/tests-nvm.mdc` and the in-repo `CLAUDE.md`.
- If `sqlite3` native bindings break after switching to Node 22, rebuild from source:
`npm_config_build_from_source=true npm rebuild sqlite3`.
- **`postinstall` runs `scripts/fix-libp2p-http-utils.js`** — a patch applied to a libp2p
dependency. Expect it to run on every `npm install`; don't remove it.
- **Docker + docker-compose** are needed for the metadata database (Typesense or
Expand Down Expand Up @@ -77,6 +76,7 @@ no semicolons, single quotes, `printWidth: 90`, no trailing commas, 2-space tabs

**Tests run against compiled JS in `dist/test/`, not the TypeScript source.** The
`test:*` scripts all call `npm run build-tests` first, which:

- compiles `src/` (incl. `src/test`) into `dist/`,
- copies `src/test/.env.test` and `.env.test2` into `dist/test`,
- copies `src/test/config.json` to `$HOME/config.json`.
Expand Down Expand Up @@ -189,7 +189,7 @@ the command, looks up the handler by `task.command`, and calls `handler.handle(t
Two front doors, one dispatcher:

- **HTTP `POST /directCommand`** (`src/components/httpRoutes/commands.ts`): validates the
body, then decides *local vs remote*. If the command targets this node (or no P2P), it
body, then decides _local vs remote_. If the command targets this node (or no P2P), it
calls `oceanNode.handleDirectProtocolCommand(...)`. If it targets another peer and P2P is
enabled, it forwards via `oceanNode.getP2PNode().sendTo(node, msg, multiAddrs)`. Responses
are streamed back to the client (binary or text).
Expand Down Expand Up @@ -225,6 +225,7 @@ register it in the `CoreHandlersRegistry` constructor; add param validation; opt
REST route in `src/components/httpRoutes/` and mount it in `httpRoutes/index.ts`.

Handler source is grouped under `src/components/core/`:

- `handler/` — general handlers (ddo, download, encrypt, fees, nonce, query, status, p2p,
auth, accessList, escrow, fileInfo, persistentStorage, policyServer, getJobs).
- `compute/` — C2D command handlers: `initialize`, `startCompute` (paid), `freeStartCompute`,
Expand Down Expand Up @@ -263,13 +264,14 @@ Handler source is grouped under `src/components/core/`:
`getComputeResult` (+ `getComputeStreamableLogs`) → `stopCompute`. Paid compute settles via
the `Escrow` component; `serviceResourceMatching.ts` maps requested cpu/ram/disk/gpu against
environment pools (dual-gate: per-env ceiling + engine-wide pool; GPUs tracked globally).
See `docs/compute-pricing.md`, `docs/GPU.md`.
See `docs/compute.md`.
- **database/** — `Database.init()` factory (`index.ts`, `DatabaseFactory.ts`). The metadata
DB backend is pluggable: **Typesense or Elasticsearch** (chosen by `DB_TYPE`) for DDOs,
indexer state, logs, orders, ddoState, access lists, escrow events — behind the
`Abstract*Database` interfaces in `BaseDatabase.ts`. **SQLite** is always used for the
nonce DB, config DB, C2D job DB, and auth-token DB (works even with no metadata DB
configured). See `docs/database.md`.
configured) — via Node's built-in `node:sqlite` module (no native addon), wrapped by
`SqliteClient` in `src/components/database/sqliteClient.ts`. See `docs/database.md`.
- **KeyManager/** — provider-abstraction over the node key (`docs/KeyManager.md`). Currently
`RawPrivateKeyProvider` (from `PRIVATE_KEY`); derives the libp2p peerId/keys and the EVM
address, and caches the ethers signer. Designed to add KMS providers (GCP/AWS) later.
Expand Down Expand Up @@ -316,5 +318,5 @@ can talk to `/var/run/docker.sock`. Deployment options (Docker, local Docker bui
`Arhitecture.md` (note the spelling), `API.md` (full HTTP API reference — very large, plus a
Postman collection), `env.md` (authoritative env-var reference), `database.md`,
`Storage.md` / `persistentStorage.md`, `KeyManager.md`, `PolicyServer.md`, `services.md`
(Service-on-Demand), `compute-pricing.md` / `GPU.md` (C2D), `networking.md`, `Logs.md`,
(Service-on-Demand), `compute.md` (C2D configuration: resources, GPUs, constraints, pricing), `networking.md`, `Logs.md`,
`Publishing.md`, `testing.md`, `dockerDeployment.md`.
3 changes: 3 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ COPY --chown=node:node --from=builder /usr/src/app/node_modules ./node_modules
COPY --chown=node:node --from=builder /usr/src/app/schemas ./schemas
COPY --chown=node:node --from=builder /usr/src/app/package.json ./
COPY --chown=node:node --from=builder /usr/src/app/config.json ./
# Ship the operator service-on-demand templates so SERVICE_TEMPLATES_PATH=docs/serviceTemplates/
# resolves inside the image (the rest of docs/ stays excluded via .dockerignore).
COPY --chown=node:node --from=builder /usr/src/app/docs/serviceTemplates ./docs/serviceTemplates

RUN mkdir -p databases c2d_storage logs

Expand Down
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ This command will run you through the process of setting up the environmental va

> [!NOTE]
> The quickstart script attempts to automatically detect GPUs (NVIDIA via `nvidia-smi`, others via `lspci`) and appends them to your `DOCKER_COMPUTE_ENVIRONMENTS`.
> Detected GPUs are added to `DOCKER_COMPUTE_ENVIRONMENTS[0].resources` (the connection-level resource pool), and a lightweight ref is added to `DOCKER_COMPUTE_ENVIRONMENTS[0].environments[0].resources`.
> If you choose to manually configure `DOCKER_COMPUTE_ENVIRONMENTS` before running the script (e.g. via environment variable), be aware that auto-detected GPUs will be **merged** into your configuration, which could lead to duplication if you already manually defined them.
> For most users, it is recommended to let the script handle GPU detection automatically.

Expand Down Expand Up @@ -153,5 +154,5 @@ Your node is now running. To start additional nodes, repeat these steps in a new
- [Network Configuration](docs/networking.md)
- [Logging & accessing logs](docs/networking.md)
- [Docker Deployment Guide](docs/dockerDeployment.md)
- [C2D GPU Guide](docs/GPU.md)
- [Compute pricing](docs/compute-pricing.md)
- [Compute (C2D) Configuration — resources, GPUs, constraints, pricing](docs/compute.md)
- [Services (Service-on-Demand)](docs/services.md)
15 changes: 14 additions & 1 deletion config.json
Original file line number Diff line number Diff line change
Expand Up @@ -94,18 +94,31 @@
"validateUnsignedDDO": true,
"jwtSecret": "ocean-node-secret",
"enableBenchmark": false,
"serviceTemplatesPath": "docs/serviceTemplates/",
"dockerComputeEnvironments": [
{
"socketPath": "/var/run/docker.sock",
"resources": [
{
"id": "disk",
"total": 1
}
],
"environments": [
{
"storageExpiry": 604800,
"maxJobDuration": 3600,
"minJobDuration": 60,
"resources": [
{
"id": "cpu"
},
{
"id": "ram"
},
{
"id": "disk",
"total": 1
"max": 1
}
],
"access": {
Expand Down
Loading
Loading