tiproxy: add annotation tiproxy-graceful-shutdown-delete-delay-seconds to remove label before deleting pods

[YangKeao]

Background

I'm designing the graceful restarting of TiProxy in cloud environment. The expected process is just to:

1. Use a big maxSurge.
2. Use a big terminationGracePeriodSeconds (e.g. 24h).
3. Restart.

However, we cannot modify the terminationGracePeriodSeconds without restarting pods. To be able to restart the existing pods gracefully, we'd better to design another workaround for it.

Design

1. Add a new annotation core.pingcap.com/tiproxy-graceful-shutdown-delete-delay-seconds. With this annotation, the deletion of TiProxy object will have two extra steps before removing the pod: remove the label, wait for several seconds.
2. The annotation will be propagated from TiProxyGroup to TiProxy without rolling restart, so we can modify it for existing resources.

I understand it's not elegant (so I didn't use a spec field to describe it) as it doesn't know when the TiProxy actually has no connection and can exit earlier. IMO, it'd be still better to use a bigger terminationGracePeriodSeconds to gracefully shutdown.

Usage

Patch an existing TiProxyGroup, so it'll wait for a while before deleting the pods:

kubectl --context "$CONTEXT" -n "$NS" patch tiproxygroup pg --type merge -p '{
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "core.pingcap.com/tiproxy-graceful-shutdown-delete-delay-seconds": "20"
        }
      }
    }
  }
}'

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jlerche for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov-commenter]

Codecov Report

❌ Patch coverage is 65.38462% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 37.23%. Comparing base (f1c9aea) to head (de28487).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6829      +/-   ##
==========================================
+ Coverage   37.16%   37.23%   +0.06%     
==========================================
  Files         390      391       +1     
  Lines       22360    22410      +50     
==========================================
+ Hits         8310     8344      +34     
- Misses      14050    14066      +16     

Flag	Coverage Δ
unittest	37.23% <65.38%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[liubog2008]

Add a new task and do not run another task in this task. If task is not fail, next task will be run.

[liubog2008]

move common.TaskContextPod[scope.TiProxy](state, r.Client) before this task and no need to get pod again.

[liubog2008]

How to notify the tiproxy that it is terminating?