[V2] Fix CRD update conflict error

[popojk]

Tracking issue

Why are the changes needed?

We sometimes see this error while running a workflow, which caused by k8s optimistic lock when 2 reconcile loops trying to update the same CRD:

What changes were proposed in this pull request?

Actually the error is gonna hurt, as the controller will reconcile again when an error returned. But a more elegant way to handle this is to handle conflict by using the k8s library retry.RetryOnConflict

How was this patch tested?

I ran the workflow for 10 times, the conflict error was not showing again.

Labels

Please add one or more of the following labels to categorize your PR:

• added: For new features.
• changed: For changes in existing functionality.
• deprecated: For soon-to-be-removed features.
• removed: For features being removed.
• fixed: For any bug fixed.
• security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

• main
  • Flyte 2 #6583
    • [V2] Fix CRD update conflict error 👈