Repository navigation
Conversation
Each blue/green job created a parameter group named after its run and deleted it at teardown, but the delete always failed: RDS keeps the just-deleted clusters attached to the group for several minutes, and the error was swallowed. One group leaked per job until the region hit its 50 parameter group quota, failing every blue/green job since Oct 20. The parameters are the same every run, so create one group per parameter group family (odbc-wrapper-bg-aurora-postgresql18), reuse it across runs, and never delete it. Concurrent creates fall back to reuse, and parameters are only modified when they differ so clusters already using the group are not flagged for reboot. The family is now read from RDS, which also fixes the invalid MySQL family name (aurora-mysql-<full version> instead of aurora-mysql8.4).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the Blue/Green integration jobs failing with
DBParameterGroupQuotaExceededby reusing one shared cluster parameter group instead of creating one per run.Description
Each Blue/Green job (Windows, macOS, Linux) created a cluster parameter group named after its run (
parameter-<cluster id>) and deleted it at teardown. The delete always failed withInvalidDBParameterGroupState, because RDS keeps the just-deleted clusters attached to the group for several minutes.destroyonly logged a warning, so every job leaked one group. Every push tomainleaked three, until ap-northeast-3 hit its quota of 50 and every Blue/Green job since Oct 20 failed at the create step.The group's parameters are the same on every run, so this PR shares one group instead:
get_or_create_parameter_group(replacescreate_parameter_group/delete_parameter_groupinscripts/db_resources.py):<prefix>-<parameter group family>, e.g.odbc-wrapper-bg-aurora-postgresql18, so a new major version gets its own group automatically.DBParameterGroupAlreadyExists), reuses it.describe_db_engine_versions. This also fixes the MySQL family name, which used to come out asaurora-mysql-<full version>instead ofaurora-mysql8.4.destroyno longer deletes the group, and its--parameter-groupoption is removed. Otherwise one job's teardown could delete the group while another job is attaching it.--parameter-group odbc-wrapper-bg, and the three destroy steps no longer pass the option.The parameter values are unchanged (
rds.logical_replication=1,max_wal_senders=20,rds.allowed_extensions=rds_toolsfor PostgreSQL;binlog_format=ROWfor MySQL).Testing
get_or_create_parameter_groupagainst ap-northeast-3 twice. The first call createdodbc-wrapper-bg-aurora-postgresql18and set its three parameters; the second reused it and made no changes. Confirmed the values withaws rds describe-db-cluster-parameters --source user.aurora-mysql8.4.mainor a manual dispatch, so they weren't run on this PR. A manual dispatch of the Integration Tests workflow on this branch would run them.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.