feat: simplify pipeline, add direct TLS transport, and improve Ceph workers#24
Merged
Merged
Conversation
… compression Remove hash-based deduplication and LZ4 compression stages from the pipeline, reducing it from 6 stages to 3 (Read → SendData → gRPC). Remove corresponding hash RPCs from the protobuf API. Add transfer stats tracking and comprehensive pipeline test suite. Increase max chunk size to 16MB. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Rakshith R <rar@redhat.com>
Add direct TLS listener and client as an alternative to stunnel for gRPC transport. Add TLS certificate generation helpers, CopyUID/CopyGID utilities, and refactor base source/destination implementations. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Rakshith R <rar@redhat.com>
…nhancements Add fallocate-based zero-range support for CephFS. Improve block iterator with large block splitting (>16MB), small file threshold bypass, and buffered boundary emission to prevent deadlocks. Remove hash server (superseded by simplified pipeline). Add destination tests. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Rakshith R <rar@redhat.com>
Enable whole-object mode in RBD block diff iterator for better performance. Add fallocate-based zero-range support for RBD. Remove hash server (superseded by simplified pipeline). Add source and zero-range tests. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Rakshith R <rar@redhat.com>
…ration Refactor mover to fetch ceph-csi-config ConfigMap via Kubernetes API instead of volume mount. Extract ReadClusterInfoFromData for in-memory config parsing. Add direct-TLS port to services. Improve pod security with FSGroupChangeOnRootMismatch and spc_t SELinux context. Fix RBD snapshot volume mode annotation. Add tests for secrets, snapshot method, and CSI config parsing. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Rakshith R <rar@redhat.com>
Fix race-free premature-exit detection in rsync daemon using channel-based select instead of signal probing. Increase max connections from 4 to 8. Add munge symlinks and SELinux label/numeric ID sync for rsync passes. Add SCC rules for mover pods. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Rakshith R <rar@redhat.com>
Add ceph-csi-config name/namespace env var substitution to Makefile deploy, build-installer, and bundle targets. Update manager deployment with ConfigMap env vars. Add retry-on-conflict to e2e spec update helpers. Assisted-by: Claude <noreply@anthropic.com> Signed-off-by: Rakshith R <rar@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactor the data transfer pipeline and Ceph worker implementations for better
performance, reliability, and maintainability.
Pipeline simplification
from 6 stages to 3 (Read → SendData → gRPC)
Direct TLS transport
transport, eliminating the stunnel process dependency
verification
direct-tlsport to all mover servicesCephFS worker improvements
bypass, buffered boundary emission (fixes deadlock)
RBD worker improvements
Mover configuration
ceph-csi-configConfigMap via Kubernetes API instead of volume mountReadClusterInfoFromDatafor in-memory config parsingFSGroupChangeOnRootMismatchandspc_tSELinux context to mover podsRsync tunnel
munge symlinks = no