Skip to content

Introduce pipectl transfer command#6692

Open
khanhtc1202 wants to merge 3 commits intomasterfrom
add-transfer-command
Open

Introduce pipectl transfer command#6692
khanhtc1202 wants to merge 3 commits intomasterfrom
add-transfer-command

Conversation

@khanhtc1202
Copy link
Copy Markdown
Member

What this PR does:

Introduce pipectl transfer command, this command will be used to migrate pipecd applications and pipeds from a source controlplane to a target controlplane. Useful for migrating control plane without manually recreating all applications/pipelines, everything will keep working as is after migration.

This transfer command contains 2 subcommands

  • pipectl transfer backup: export all current piped and applications of the current project to JSON file
  • pipectl transfer restore {piped,application}: add exported piped and applications to the target controlplane

Note:

  • The implementation relies on the current cli API of the pipecd controlplane to reduce the complexity of supporting all current pipecd controlplane datastores (MySQL and Firestore)
  • Since the current cli API does not support adding deployments, all previous applications' deployments are not backed up (reason we should do it as follows)
  • After migrating the piped, it will trigger SYNC for all applications it's managing, since there are no previous deployments, triggered deployments are all QUICK_SYNC, thus basically no change/no failure as workload unchanged in users' cluster.
  • Docs for transfer flow will be prepared by a separate PR.

Why we need it:

Which issue(s) this PR fixes:

Part of #6691

Does this PR introduce a user-facing change?:

  • How are users affected by this change:
  • Is this breaking change:
  • How to migrate (if breaking change):

Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com>
@khanhtc1202 khanhtc1202 requested a review from a team as a code owner April 17, 2026 09:34
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 17, 2026

Codecov Report

❌ Patch coverage is 0% with 384 lines in your changes missing coverage. Please review.
✅ Project coverage is 29.33%. Comparing base (78fdbeb) to head (1183722).

Files with missing lines Patch % Lines
...kg/app/pipectl/cmd/transfer/restore_application.go 0.00% 127 Missing ⚠️
pkg/app/pipectl/cmd/transfer/backup.go 0.00% 113 Missing ⚠️
pkg/app/pipectl/cmd/transfer/restore_piped.go 0.00% 104 Missing ⚠️
pkg/app/pipectl/cmd/transfer/restore.go 0.00% 24 Missing ⚠️
pkg/app/pipectl/cmd/transfer/transfer.go 0.00% 15 Missing ⚠️
cmd/pipectl/main.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6692      +/-   ##
==========================================
- Coverage   29.50%   29.33%   -0.18%     
==========================================
  Files         593      598       +5     
  Lines       63390    63774     +384     
==========================================
+ Hits        18702    18706       +4     
- Misses      43243    43624     +381     
+ Partials     1445     1444       -1     
Flag Coverage Δ
. 23.10% <0.00%> (-0.19%) ⬇️
.-pkg-app-pipedv1-plugin-analysis 32.43% <ø> (ø)
.-pkg-app-pipedv1-plugin-ecs 31.82% <ø> (ø)
.-pkg-app-pipedv1-plugin-kubernetes 58.37% <ø> (ø)
.-pkg-app-pipedv1-plugin-kubernetes_multicluster 62.66% <ø> (ø)
.-pkg-app-pipedv1-plugin-scriptrun 54.83% <ø> (ø)
.-pkg-app-pipedv1-plugin-terraform 37.95% <ø> (ø)
.-pkg-app-pipedv1-plugin-wait 33.04% <ø> (ø)
.-pkg-app-pipedv1-plugin-waitapproval 52.71% <ø> (ø)
.-pkg-plugin-sdk 50.34% <ø> (+0.19%) ⬆️
.-tool-actions-gh-release 19.23% <ø> (ø)
.-tool-actions-plan-preview 25.51% <ø> (ø)
.-tool-codegen-protoc-gen-auth 0.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Signed-off-by: khanhtc1202 <khanhtc1202@gmail.com>
Comment on lines +120 to +122
if err != nil {
return nil, nil, fmt.Errorf("failed to register piped %q (%s): %w", p.Name, p.Id, err)
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not bug but why registering pipeds exit when an error occured
while registering applications keeps adding and return number of success and fail restoration

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example if there are 10 pipeds need to be restored, if it fail early then we need to add remain ones manually
P1
P2
P3: fail
P4: need manual adding
....
P10: need manual adding

I know this problem can be easily solved by deleting P1, P2, P3 from backup file and reapply the restore command but I still think that continuing even fail is better

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@armistcxy The number of piped is always smaller compare to the number of applications, thus I prefer to retry whole things (after delete P1->P3 on UI)

Another reason is the output of the pipectl transfer restore piped command is a piped mapping file, which will be used by the restore application step. I don't want to force users to manually update/add piped mapping to that file in order to make restore application step work. So, fail fast and delete P1->P3, then re-do from start is recommended.

Comment thread pkg/app/pipectl/cmd/transfer/restore_application.go
Comment thread pkg/app/pipectl/cmd/transfer/restore_application.go
Copy link
Copy Markdown
Contributor

@armistcxy armistcxy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM but there're some concerns

Comment on lines +124 to +129
mappings = append(mappings, PipedMapping{
OldPipedID: p.Id,
NewPipedID: resp.Id,
NewKey: resp.Key,
PipedName: p.Name,
})
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, I almost forget to mention that should we return Base64 encoded key here for easy copy paste

when testing this feature, I need to run "echo | base64" and copy, I would love to return the base64 encode as the UI return

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, let me address that 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants