[garthgoodson]

Summary

A full compaction can wait behind another compaction while new writes keep creating SST files (sorted data files). Today, that wait also blocks automatic compactions for the same column family. The file count can reach sst_files_hard_limit and cause write rejection (#16798).

This change lets automatic compactions continue while a non-exclusive manual compaction waits. They can merge new files without touching files already being compacted. Exclusive manual compactions still block new automatic work.

The runtime flag nonexclusive_manual_compaction_blocks_auto_compactions defaults to false on master, enabling the fix. Setting it to true restores the old behavior. It replaces the earlier file-count threshold.

This matches the behavior in the linked RocksDB fix. YB applies the check before queueing automatic work, rather than in the background worker.

The change does not cap file counts or guarantee write availability. Flushes can still outpace compaction, and sustained automatic work can delay the manual compaction without a time bound. The tests verify scheduling and eventual completion after writes stop; the production incident has not been replayed.

Upgrade/Rollback safety

Master changes scheduling behavior by default. The flag is local to each tserver; older nodes keep their existing behavior in mixed-version clusters. No on-disk, wire-format, or catalog change.

Set the flag to true at runtime to restore the old behavior for future scheduling decisions; no restart is needed. Already queued work is not cancelled.

Backports should default to true to preserve older behavior. Those deployments must set it to false to enable the fix. Retire the flag after 2026.1 and 2025.2 leave support. This PR does not change release branches.

Test plan

Each test passed once on the current code in a debug build:

• ManualCompactionWaitTest.FlagOnBlocksAutomaticCompactions
• ManualCompactionWaitTest.FlagOffRunsAutomaticCompactions
• ManualCompactionWaitTest.FlagOffRunsAutomaticCompactionsWithPriorityThreadPool
• ManualCompactionWaitTest.ExclusiveBlocksAutomaticCompactions
• ColumnFamilyTest.{ManualAndAutomaticCompactions, SameCFManualAutomaticCompactions, SameCFManualAutomaticCompactionsLevel, SameCFManualAutomaticConflict, SameCFAutomaticManualCompactions}
• DBCompactionTest.AbortManualCompactionOnShutdown
• CompactionTest.BackgroundCompactionDuringPostSplitCompaction
• ScheduledFullCompactionsTest.WillWaitForPreviousToFinishBeforeScheduling
• build-support/lint.sh --rev upstream/master (local build symlink excluded)

The four new tests use debug-only synchronization points. Release, sanitizer, sustained-write, and CI results are not claimed here.

[yugabyte-ci]

This change is 

[garthgoodson]

Tracking PR in our fork: Shopify#35

[iSignal]

Let's use a single boolean flag to gate this, mainly for backwards compat on older versions.

FLAGS_nonexclusive_manual_compaction_blocks_auto_compactions defaults to false on master, we can set it to true on backports as to not disturb older behavior. We can retire this flag once 2026.1/2025.2 fall out of support.

That way it looks like SunguckLee/rocksdb@21c5a9d but just with a boolean flag gating it for safety on older versions.

[garthgoodson]

Made the suggested changes. More details in comment below. There is some divergence from YB design vs RocksDB, so not exact matching of behavior but outcome should be similar. Let me know if you want something different.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[garthgoodson]

Updated in d11f5ea per the review: replaced the file-count threshold with nonexclusive_manual_compaction_blocks_auto_compactions. It defaults to false on master, so automatic compactions can run while a non-exclusive full compaction waits. Backports should default to true to keep the old behavior.

This matches the linked RocksDB fix. RocksDB changes the check in the background worker. YB has this check before work enters the queue, so the change goes there.

YB also keeps its existing ordering for exclusive compactions: finish already queued work, then run the exclusive compaction alone. New automatic work cannot enter the queue during that wait. The linked RocksDB code checks exclusivity in the worker instead. This PR leaves YB's ordering unchanged.

All 12 debug tests listed in the updated description passed, plus lint. This removes one cause of file buildup; it does not prevent write rejection if compaction cannot keep up with writes.