Feature/docker stats

docs/compute.md

@@ -215,6 +215,16 @@ The environment references it by `id`.
 ]
 ```
 
+> **GPU runtime metrics (NVIDIA).** When `GPU_METRICS` is `auto` (the default), the node
+> records per-GPU utilization and memory for running jobs/services alongside their container
+> metrics, sampled every `C2D_METRICS_INTERVAL_SECONDS`. This uses NVML via the optional
+> `koffi` dependency and requires `libnvidia-ml.so.1` to be reachable by the node process (the
+> NVIDIA driver provides it; in a containerized node, mount it or use the NVIDIA container
+> toolkit). A job holding several GPUs gets one metrics entry per device, keyed by the
+> resource id it requested. If NVML is unavailable, GPU metrics are silently skipped while
+> container metrics continue. AMD and Intel GPU metrics are not yet collected. These metrics
+> are node-internal (never returned in API responses). Set `GPU_METRICS=off` to disable.
+
 ### AMD Radeon (ROCm)
 
 Install [ROCm](https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/wsl/install-radeon.html),

docs/database.md

@@ -41,3 +41,19 @@ To run Ocean Node with the appropriate database, you need to start Barge with sp
 ```
 
 By specifying these flags, you can configure Ocean Node to work with either Typesense or Elasticsearch databases, depending on your requirements.
+
+## Runtime metrics on C2D job records
+
+While a compute job or Service-on-Demand container runs, the node periodically samples live
+Docker (and, on NVIDIA hosts, NVML) metrics — CPU, RAM, disk usage vs quota, network, block
+I/O, PID count, CPU throttling, memory peak, exit/OOM info, and per-GPU utilization/memory —
+and stores the latest snapshot on the job record (inside the existing JSON `body` blob of the
+SQLite `c2djobs` / `service_jobs` tables). A **final** snapshot is written at termination
+(publishing results, quota kill, service stop/restart, or unexpected container death) so
+peak/exit metrics remain queryable after the container is gone. No schema migration is needed;
+pre-upgrade records simply lack the field.
+
+These snapshots are **node-internal**: they are stripped from every API response and from the
+escrow claim proof, so responses are byte-identical to before. Sampling cadence is controlled
+by `C2D_METRICS_INTERVAL_SECONDS` (`0` disables it) and GPU collection by `GPU_METRICS` — see
+[env.md](env.md).

docs/env.md

@@ -129,6 +129,10 @@ Environmental variables are also tracked in `ENVIRONMENT_VARIABLES` within `src/
 
 - `C2D_DOWNLOAD_TIMEOUT`: Timeout (in seconds) for pulling the algorithm docker image during a C2D job. If the pull exceeds this timeout, the job fails with `PullImageFailed` instead of getting stuck. Defaults to `900` (15 minutes). Example: `900`
 
+- `C2D_METRICS_INTERVAL_SECONDS`: How often (in seconds) the node samples live Docker runtime metrics (CPU, RAM, disk, network, block I/O, PIDs, exit info — plus NVIDIA GPU utilization/memory) for running compute jobs and services, persisting a snapshot onto the job record in the C2D database. These metrics are **node-internal only**: they are stripped from every API response and from the escrow claim proof, so responses are unchanged. Set to `0` to disable collection entirely. Metrics are best-effort (up to one interval of staleness). Defaults to `10`. Example: `10`
+
+- `GPU_METRICS`: Controls the GPU metrics collector. `auto` (default) detects and enables the NVIDIA (NVML) backend when a GPU host is available; `off` disables GPU collection. Requires the optional `koffi` dependency and `libnvidia-ml.so.1` reachable on the host — if either is missing, GPU metrics are silently skipped (no `gpu` field) while container-level metrics continue. AMD and Intel backends are not yet implemented. Cadence reuses `C2D_METRICS_INTERVAL_SECONDS`. Defaults to `auto`. Example: `auto`
+
 - `SERVICE_TEMPLATES_PATH`: Path to a folder of operator-published Service-on-Demand template files (`*.json`, validated against the template schema). The folder is re-read on every `serviceTemplates` request, so templates can be added, edited, or removed without restarting the node. Maps to the `serviceTemplatesPath` config field. Defaults to `databases/serviceTemplates/`. See the [Services guide](services.md). Example: `docs/serviceTemplates/`
 
 The `DOCKER_COMPUTE_ENVIRONMENTS` environment variable is used to configure Docker-based compute environments in Ocean Node. For the full guide — resources, GPU setup, constraints and pricing — see [Compute Configuration](compute.md).

package.json

@@ -109,6 +109,9 @@
     "winston-transport": "^4.6.0",
     "zod": "^3.25.76"
   },
+  "optionalDependencies": {
+    "koffi": "^2.9.0"
+  },
   "devDependencies": {
     "@types/chai": "^4.3.10",
     "@types/cors": "^2.8.17",

src/@types/C2D/C2D.ts

@@ -238,6 +238,73 @@ export type DBComputeJobMetadata = {
   [key: string]: string | number | boolean
 }
 
+// Per-GPU runtime metrics captured alongside the container snapshot. ONE entry per GPU
+// resource the job/service holds — a single job may hold several GPUs simultaneously, and
+// each is sampled independently. Only NVIDIA is emitted today (NVML backend); AMD/Intel
+// land once their backends exist. `resourceId` maps the entry back to the requested
+// resource id ('gpu0' / 'gpu1' …) so consumers can attribute each device.
+export interface GpuMetricsSnapshot {
+  resourceId: string
+  vendor: 'nvidia' | 'amd' | 'intel'
+  utilizationPercent: number | null
+  memoryUsedBytes: number | null
+  memoryTotalBytes: number | null
+  temperatureC?: number
+  powerWatts?: number
+  shared?: boolean // true → device-level number, may include other jobs' load (shareable GPUs)
+}
+
+// A best-effort, node-internal snapshot of everything the Docker engine (and, for NVIDIA,
+// NVML) can cheaply tell us about a running container. Persisted onto the C2D job record
+// (JSON body blob) and STRIPPED from every public response + the escrow claim proof — see
+// omitDBComputeFieldsFromComputeJob / toPublicServiceJob. Covers the full `docker stats`
+// column set plus what the CLI does not show (throttling, peaks, disk-vs-quota, exit info,
+// GPU). Every field defaults defensively (cgroup v1/v2 drift, missing network, daemon
+// hiccups) rather than throwing into the state-machine loop.
+export interface ContainerMetricsSnapshot {
+  collectedAt: string // ISO timestamp of the sample
+  containerState: {
+    status: string // 'running' | 'exited' | ...
+    startedAt?: string
+    finishedAt?: string
+    exitCode?: number
+    oomKilled: boolean
+    error?: string
+    restartCount: number
+    health?: string // Docker HEALTHCHECK status, when the image defines one (services)
+  }
+  cpu: {
+    usagePercent: number // % of host CPU (docker CLI formula)
+    allocated: number // cores requested (job.resources 'cpu'); 0 when unconstrained
+    usagePercentOfAllocated: number // usagePercent / allocated (0 when allocated is 0)
+    cumulativeSeconds: number // total_usage ns → s (monotonic; billing-grade)
+    throttledPeriods: number // quota throttling — signals an undersized cpu request
+    throttledSeconds: number
+  }
+  memory: {
+    usageBytes: number // usage − inactive_file (cgroup v2 convention)
+    limitBytes: number // container limit (= allocated RAM)
+    usagePercent: number
+    peakUsageBytes: number // max across samples (we track it; cgroup v2 has no max_usage)
+  }
+  disk: {
+    usedBytes: number // jobs: du(/) − base image; services: SizeRw
+    quotaBytes?: number // jobs with a 'disk' resource
+    usagePercent?: number
+  }
+  network?: { rxBytes: number; txBytes: number } // absent when NetworkMode 'none'
+  blockIO: { readBytes: number; writeBytes: number }
+  pids: { current: number; limit: number } // vs PidsLimit 512 — fork-bomb signal
+  gpu?: GpuMetricsSnapshot[]
+  // Internal accumulator used to compute one-shot CPU deltas across samples. DB-only —
+  // stripped from every public response together with the snapshot itself.
+  prev?: {
+    cpuTotal: number // cpu_stats.cpu_usage.total_usage (ns) at the previous sample
+    systemCpu: number // cpu_stats.system_cpu_usage (ns) at the previous sample
+    sampledAt: string
+  }
+}
+
 export interface ComputeJobTerminationDetails {
   OOMKilled: boolean
   exitCode: number
@@ -343,6 +410,11 @@ export interface DBComputeJob extends ComputeJob {
   jobIdHash: string
   buildStartTimestamp?: string
   buildStopTimestamp?: string
+  // Best-effort Docker/NVML runtime metrics sampled while the container runs. DB-only:
+  // it lives on DBComputeJob (never on the public ComputeJob) and is additionally stripped
+  // at runtime by omitDBComputeFieldsFromComputeJob so it can never leak into a status
+  // response or the escrow claim proof.
+  runtimeMetrics?: ContainerMetricsSnapshot
 }
 
 // make sure we keep them both in sync

src/@types/C2D/ServiceOnDemand.ts

@@ -1,4 +1,8 @@
-import type { DBComputeJobPayment, ComputeResourceRequestWithPrice } from './C2D.js'
+import type {
+  DBComputeJobPayment,
+  ComputeResourceRequestWithPrice,
+  ContainerMetricsSnapshot
+} from './C2D.js'
 
 // ── Resource requirements ─────────────────────────────────────────────
 
@@ -144,4 +148,7 @@ export interface ServiceJob {
   resources: ComputeResourceRequestWithPrice[]
   payment: DBComputeJobPayment // initial start payment
   extendPayments?: DBComputeJobPayment[] // one entry per successful SERVICE_EXTEND
+  // Best-effort Docker/NVML runtime metrics sampled while the service container runs.
+  // DB-only: stripped from every public response by toPublicServiceJob / toListedServiceJob.
+  runtimeMetrics?: ContainerMetricsSnapshot
 }