Co-located special satellites race on shared loop_device_mapping.tmp — device-manager cycles abort with NoSuchFileException, EBS resources stay Unknown

[mateo-moon]

Summary

All special satellites spawned by the controller (e.g. EBS_TARGET nodes) run inside the controller's container and therefore share /var/lib/linstor.d. Their device-manager cycles race on the shared loop_device_mapping.tmp: every cycle ends in FileProvider.clearCache(), which writes the temp file and atomically renames it to loop_device_mapping — even when the satellite has zero FILE volumes (the file content is empty). When two co-located satellites' cycles overlap, one renames the file away and the other's rename throws:

StorageException: Failed to move /var/lib/linstor.d/loop_device_mapping.tmp to /var/lib/linstor.d/loop_device_mapping
Caused by: java.nio.file.NoSuchFileException: /var/lib/linstor.d/loop_device_mapping.tmp -> /var/lib/linstor.d/loop_device_mapping
    at ...
    devMgrLoop ... com.linbit.linstor.core.devmgr.DeviceManagerImpl:768

which aborts that satellite's entire device-manager cycle. For EBS targets this leaves freshly dispatched resources undeployed (stuck Unknown) — sometimes after the EBS volume itself was already created — until a later cycle happens to win the race. With three per-AZ EBS targets (the documented multi-AZ layout) the race fires regularly.

We believe this is a major contributor to the intermittent "resources on EBS targets stay Unknown" behavior we reported in #506 — see the correction comment there: with v1.34.1 under close observation, dispatch does happen and Volume ... [EBS-Target] created appears in the target's log, after which the cycle dies on this rename and the resource never reaches UpToDate.

Live evidence (v1.34.1, Kubernetes / piraeus-operator v2.10.7, 3 per-AZ EBS targets)

Three special satellites sharing the controller container:

java ... com.linbit.linstor.core.Satellite -s --port 10000 ... --override-node-name ebs-target-eu-central-1a ... --ebs
java ... com.linbit.linstor.core.Satellite -s --port 10001 ... --override-node-name ebs-target-eu-central-1b ... --ebs
java ... com.linbit.linstor.core.Satellite -s --port 10002 ... --override-node-name ebs-target-eu-central-1c ... --ebs

Shared state file (note: empty — these satellites have no loop devices at all):

-rw-r--r-- 1 root root 0 Jul 20 21:27 /var/lib/linstor.d/loop_device_mapping

Target satellite log for an affected resource — volume created, then the cycle aborts:

21:06:55 INFO  - Volume number 0 of resource 'pvc-4d925505-...' [EBS-Target] created
...
21:27:09 ERROR - Failed to move /var/lib/linstor.d/loop_device_mapping.tmp to /var/lib/linstor.d/loop_device_mapping [Report 6A5E8DE3-EC327-000000]

Controller keeps re-queuing (RetryTask: Failed resource ... added for retry) and the resource stays Unknown.

Suggested fix

Make the temporary file name unique per process (keeping the atomic-rename pattern), so co-located satellites' cycles are independent. PR incoming. An alternative/additional improvement would be skipping the persistence entirely when the satellite has no loop devices.

Related

#505, #506, #507, #508 (our native-EBS series); this issue is the mechanism behind at least part of #506.