Skip to content

[DiffSinger/BUGFIX]Implement DiffSinger variance retake with hard compose - #2287

Open
KakaruHayate wants to merge 3 commits into
openutau:masterfrom
KakaruHayate:variance-retake-hard-compose
Open

[DiffSinger/BUGFIX]Implement DiffSinger variance retake with hard compose#2287
KakaruHayate wants to merge 3 commits into
openutau:masterfrom
KakaruHayate:variance-retake-hard-compose

Conversation

@KakaruHayate

Copy link
Copy Markdown
Contributor

Merge order and dependencies

  1. Merge or rebase Support DiffSinger pitch local retaking #2183 first. It provides the DiffSinger pitch retake frame-mask and non-contiguous retake write-back foundation.
  2. Merge this variance retake change next. It replaces the variance crossfade and cumulative baseline behavior introduced by Add DiffSinger variance local pitch patch #2176 with tensor-cache-based partial retake and hard composition.
  3. Rebase or merge Refresh real curves after phrase rendering #2175 after this change so the real-curve scheduler consumes the final hard-composed variance result.
  4. Process Add prioritized render scheduling #2174 last. Its render-priority changes are ordering-only and should be reconciled with Refresh real curves after phrase rendering #2175 changes in RenderEngine.cs and Notifications.cs.

This PR is intentionally based on current master, where #2176 is already merged. It does not include the code from #2183, #2175, or #2174. The order above describes the intended final integration order; if this PR is reviewed before #2183 is merged, it can remain independent on master and be rebased when the stack is integrated.

Retake scope requirements

Partial variance retake is allowed only when the tensor-cache scope remains compatible:

  • Pitch and frame-level speaker embedding changes are retake-eligible.
  • Linguistic encoder inputs or outputs, phoneme durations, word durations/divisions, phrase frame layout, model identity, variance channel layout, and diffusion steps/speedup remain part of the scope key. A change to any of them uses a different scope and performs full generation.
  • Tensor cache must be enabled and a compatible previous result must exist.
  • The first version does not expand the frame mask.
  • No crossfade is used.
  • Model output outside the mask is not trusted. Hard composition restores every unmasked frame from the previous cached result.

What changed

  • Split complete variance cache lookup from the partial-retake scope.
  • Reuse cached variance curves as predictor inputs during partial retake.
  • Build a frame mask from pitch and speaker-embedding changes, then expand it to the model's [1, T, V] retake input.
  • Hard-compose predicted frames with the previous result after inference.
  • Save the final composed result to the complete tensor cache so undo/redo can reuse exact historical states.
  • Remove the fixed 50 ms crossfade, weighted blending, and cumulative merged baseline.
  • Add focused tests for frame masks, frame-grouped speaker embeddings, channel expansion, hard composition, leakage prevention, and incompatible metadata.

Validation

  • dotnet restore OpenUtau.Test/OpenUtau.Test.csproj
  • dotnet test OpenUtau.Test/OpenUtau.Test.csproj --no-restore --filter "FullyQualifiedName~DiffSingerVariancePatchTest"

@KakaruHayate KakaruHayate changed the title Implement DiffSinger variance retake with hard compose [DiffSinger/BUGFIX]Implement DiffSinger variance retake with hard compose Aug 4, 2026
keirokeer added a commit to keirokeer/OpenUtau-lunai that referenced this pull request Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants