services updates

docs/services.md

@@ -29,8 +29,8 @@ and `signature` as query parameters (or an auth-token `Authorization` header).
 | `SERVICE_START` | `/api/services/serviceStart` | POST | Validate, persist a `Starting` record, and return the `serviceId` immediately (escrow + image + container happen in the background) |
 | `SERVICE_GET_STATUS` | `/api/services/serviceStatus` | GET | Read job status / endpoints — authenticated, owner-scoped (see notice below); poll this to follow a starting service |
 | `SERVICE_EXTEND` | `/api/services/serviceExtend` | POST | Pay to push the expiry further out |
-| `SERVICE_RESTART` | `/api/services/serviceRestart` | POST | Recreate the container (no extra charge) |
-| `SERVICE_STOP` | `/api/services/serviceStop` | POST | Tear down the container and release resources |
+| `SERVICE_RESTART` | `/api/services/serviceRestart` | POST | Recreate the container (no extra charge); asynchronous like start — returns once the job is `Restarting`, poll `serviceStatus` |
+| `SERVICE_STOP` | `/api/services/serviceStop` | POST | Tear down the container; the paid resource reservation (cpu/ram/gpu + host ports) is kept until `expiresAt`, so the service can be restarted anytime on the same endpoints |
 | `SERVICE_GET_TEMPLATES` | `/api/services/serviceTemplates` | GET | List operator-published service templates |
 | `SERVICE_GET_STREAMABLE_LOGS` | `/api/services/serviceStreamableLogs` | GET | Stream the container's live stdout/stderr logs — authenticated, owner-scoped; available while `Running` or `Error`; optional `since` to skip history |
 
@@ -57,16 +57,33 @@ container creation fails before the claim, the lock is **cancelled (refunded)**
 in a `*Failed` / `Error` status. This is a change from the previous synchronous flow, which
 locked-then-claimed up front.
 
+**Restart is asynchronous too.** `serviceRestart` performs only the fast validations
+(ownership, environment/access, not expired, payment not refunded), persists the job as
+`Restarting (45)` and responds immediately — the teardown, image re-pull/rebuild and new
+container happen in the background under the same per-service lock. Poll `serviceStatus`
+and watch `Restarting` → `PullImage`/`BuildImage` → `Running` (or `Error` with the failure
+reason in `statusText`). A service whose start payment was **refunded** (lock cancelled
+before it was claimed) cannot be restarted — it was never paid for; start a new service.
+
+**The reservation lasts the whole paid window — only `Expired` releases it.** The consumer
+paid for the resources for a time interval and may use them as they please within it:
+running the service, stopping it, restarting it. An explicit `SERVICE_STOP` therefore tears
+down the container/network but **keeps** the resource amounts (cpu/ram/gpu) counted and the
+host ports reserved — another consumer cannot take them, and a restart resumes on the same
+endpoints. Once `expiresAt` passes, the expiry sweep tears down whatever is left, marks the
+job `Expired`, and only then releases everything. The sweep refuses to mark `Expired` while
+teardown fails (e.g. Docker unreachable) — the job stays `Error` and is retried every tick,
+so a resource release is never silently skipped.
+
 **`Running` is monitored too.** The same background loop that advances a starting service also
 checks every `Running` service's container on each tick (~every few seconds). If the container
 exits on its own — crash, OOM, or the Docker daemon itself becoming unreachable — the job is
 moved to `Error` immediately instead of waiting for `expiresAt`. This health check does **not**
 release the service's reserved host ports/network/container record, since the consumer already
 paid for them; use `SERVICE_RESTART` to bring the service back on the same endpoints. `Error`
-counts as an active/resource-reserving status just like `Running` does — it still occupies its
-cpu/ram/gpu allocation and keeps its host ports held — until it is restarted, explicitly
-stopped, or swept by the same expiry check once `expiresAt` passes (which then fully releases
-everything, same as a normal expiry).
+counts as an active/resource-reserving status just like `Running` and `Stopped` do — it still
+occupies its cpu/ram/gpu allocation and keeps its host ports held — until it is restarted or
+swept by the expiry check once `expiresAt` passes (which then fully releases everything).
 
 **Restart is self-healing with respect to leftover Docker state.** Each service gets a Docker
 network with the deterministic name `ocean-svc-<serviceId>`. Teardown (restart, stop, expiry
@@ -84,6 +101,24 @@ the network always reflects the current code's configuration instead of silently
 whatever options a previous node version created it with, and it matches restart's overall
 tear-down-and-rebuild semantics (the container is never reused either).
 
+**Lifecycle operations are exclusive per service.** At most one lifecycle operation — the
+background start pipeline, `SERVICE_RESTART`, `SERVICE_STOP`, or the expiry sweep — runs per
+service at a time. A restart or stop issued while another operation is in flight (e.g. a
+restart still pulling the image) is rejected with
+`Service <id> has a start/stop/restart operation in progress — retry shortly`; simply retry
+once the in-flight operation settles. Without this exclusivity, the background loop's
+crash-orphan recovery could tear down the `ocean-svc-<serviceId>` network in the middle of a
+restart that had just created it, failing the restart with
+`network ocean-svc-<id> not found`. If a service expires while such an operation is in
+flight, the expiry sweep simply retries on a later tick.
+
+Exclusivity holds **across node processes** too, not just within one: each operation also
+takes a lease row in the SQLite `service_locks` table, so two processes sharing the same
+`databases/` directory and Docker daemon (e.g. an old container still running during a
+redeploy) cannot run conflicting operations on the same service. Leases are heartbeated
+every 30 s while the operation runs; a lease not refreshed for 2 minutes belongs to a
+crashed process and is stolen automatically, so no manual cleanup is ever needed.
+
 ## Configuration
 
 Service-on-demand is configured per Docker connection under `serviceOnDemand`:

src/@types/C2D/ServiceOnDemand.ts

@@ -78,6 +78,7 @@ export enum ServiceStatusNumber {
   Locking = 20, // escrow createLock in progress (funds locked, not yet claimed)
   Claiming = 30, // payment phase: claimLock on success, or cancelLock if the image step failed
   Running = 40,
+  Restarting = 45, // SERVICE_RESTART accepted; teardown + re-pull/build + new container in progress
   Stopping = 50,
   Stopped = 70,
   Expired = 75,
@@ -95,12 +96,27 @@ export const ServiceStatusText: Record<ServiceStatusNumber, string> = {
   [ServiceStatusNumber.Locking]: 'Locking',
   [ServiceStatusNumber.Claiming]: 'Claiming',
   [ServiceStatusNumber.Running]: 'Running',
+  [ServiceStatusNumber.Restarting]: 'Restarting',
   [ServiceStatusNumber.Stopping]: 'Stopping',
   [ServiceStatusNumber.Stopped]: 'Stopped',
   [ServiceStatusNumber.Expired]: 'Expired',
   [ServiceStatusNumber.Error]: 'Error'
 }
 
+// Statuses of a service job that is mid-start/restart and owned by an exclusive
+// lifecycle operation. Single source of truth for getPendingServiceStarts (DB query) and
+// the pipeline's staleness guard — the two MUST agree or a job can be picked up and then
+// ignored (or vice versa). Restarting is included so a job orphaned by a crash
+// mid-restart is recovered at boot exactly like a crash mid-start.
+export const SERVICE_START_PENDING_STATUSES: readonly ServiceStatusNumber[] = [
+  ServiceStatusNumber.Starting,
+  ServiceStatusNumber.Locking,
+  ServiceStatusNumber.PullImage,
+  ServiceStatusNumber.BuildImage,
+  ServiceStatusNumber.Claiming,
+  ServiceStatusNumber.Restarting
+]
+
 export interface ServiceJob {
   serviceId: string // unique id for a running service — distinct from a compute jobId
   clusterHash: string