[garthgoodson]

Summary

A full compaction can wait behind another compaction while new writes keep creating SST files (sorted data files). Today, that wait also blocks automatic compactions for the same column family. The file count can reach sst_files_hard_limit and cause write rejection (yugabyte#16798).

This change lets automatic compactions continue while a non-exclusive manual compaction waits. They can merge new files without touching files already being compacted. Exclusive manual compactions still block new automatic work.

The runtime flag nonexclusive_manual_compaction_blocks_auto_compactions defaults to false on master, enabling the fix. Setting it to true restores the old behavior. It replaces the earlier file-count threshold.

This matches the behavior in the linked RocksDB fix. YB applies the check before queueing automatic work, rather than in the background worker.

The change does not cap file counts or guarantee write availability. Flushes can still outpace compaction, and sustained automatic work can delay the manual compaction without a time bound. The tests verify scheduling and eventual completion after writes stop; the production incident has not been replayed.

Upgrade/Rollback safety

Master changes scheduling behavior by default. The flag is local to each tserver; older nodes keep their existing behavior in mixed-version clusters. No on-disk, wire-format, or catalog change.

Set the flag to true at runtime to restore the old behavior for future scheduling decisions; no restart is needed. Already queued work is not cancelled.

Backports should default to true to preserve older behavior. Those deployments must set it to false to enable the fix. Retire the flag after 2026.1 and 2025.2 leave support. This PR does not change release branches.

Test plan

Each test passed once on the current code in a debug build:

• ManualCompactionWaitTest.FlagOnBlocksAutomaticCompactions
• ManualCompactionWaitTest.FlagOffRunsAutomaticCompactions
• ManualCompactionWaitTest.FlagOffRunsAutomaticCompactionsWithPriorityThreadPool
• ManualCompactionWaitTest.ExclusiveBlocksAutomaticCompactions
• ColumnFamilyTest.{ManualAndAutomaticCompactions, SameCFManualAutomaticCompactions, SameCFManualAutomaticCompactionsLevel, SameCFManualAutomaticConflict, SameCFAutomaticManualCompactions}
• DBCompactionTest.AbortManualCompactionOnShutdown
• CompactionTest.BackgroundCompactionDuringPostSplitCompaction
• ScheduledFullCompactionsTest.WillWaitForPreviousToFinishBeforeScheduling
• build-support/lint.sh --rev upstream/master (local build symlink excluded)

The four new tests use debug-only synchronization points. Release, sanitizer, sustained-write, and CI results are not claimed here.

[jordanwheeler]

I think this set to 12 would have prevented the recent issue we saw in production 👍

We probably still want to add some metrics/monitoring around this as well.

[garthgoodson]

Updated in d11f5ea per the upstream review: replaced the file-count threshold with nonexclusive_manual_compaction_blocks_auto_compactions. It defaults to false on master, so automatic compactions can run while a non-exclusive full compaction waits. Backports should default to true to keep the old behavior.

This matches the linked RocksDB fix. RocksDB changes the check in the background worker. YB has this check before work enters the queue, so the change goes there.

YB also keeps its existing ordering for exclusive compactions: finish already queued work, then run the exclusive compaction alone. New automatic work cannot enter the queue during that wait. The linked RocksDB code checks exclusivity in the worker instead. This PR leaves YB's ordering unchanged.

All 12 debug tests listed in the updated description passed, plus lint. This removes one cause of file buildup; it does not prevent write rejection if compaction cannot keep up with writes.

---

Generated with Pi.