Skip to content

Fix cross-process signal delivery to EL0-preempted guests#76

Open
Max042004 wants to merge 1 commit into
sysprog21:mainfrom
Max042004:fix-cross-process-signal-el0
Open

Fix cross-process signal delivery to EL0-preempted guests#76
Max042004 wants to merge 1 commit into
sysprog21:mainfrom
Max042004:fix-cross-process-signal-el0

Conversation

@Max042004

@Max042004 Max042004 commented Jun 6, 2026

Copy link
Copy Markdown
Collaborator

To fix #67. test-fork's phase-2 signal child spins in while (!got_usr1) usleep() waiting for a SIGUSR1 sent cross-process by its parent. The signal was delivered only ~35% of the time and lost the rest, so make check hung at test-fork until the 60s per-test timeout -- often longer, as leaked elfuse --fork-child orphans kept the driver's stdout pipe open.

Two complementary defects:

  1. vcpu_run_loop treated HV_EXIT_REASON_UNKNOWN as fatal. A host SIGUSR2 (the cross-process guest-signal transport) that interrupts hv_vcpu_run mid-execution aborts the run with UNKNOWN rather than the clean CANCELED that hv_vcpus_exit() produces for a vCPU caught between runs. Route UNKNOWN through the same cancellation handling so the already-queued guest signal is delivered instead of crashing the child.

  2. signal_deliver redirected to the handler only via ELR_EL1, which takes effect solely on an ERET from EL1 (the syscall-return path, gated by the shim's X8==2 exec_drop_frame marker). When the signal is delivered from the cancellation branch -- i.e. the vCPU was preempted while running EL0 code (cross-process SIGUSR2, or SIGALRM in a tight loop) -- there is no pending ERET, the resume uses HV_REG_PC, and the ELR_EL1 write is a no-op: the handler never runs and only the X0=signum clobber lands, re-running the interrupted nanosleep with a bogus arg and spinning forever. Detect EL0 preemption from the live PSTATE (CPSR M[3:0]==0), save the interrupted PC from HV_REG_PC instead of the stale ELR_EL1, and redirect HV_REG_PC/CPSR directly; skip the X8==2 marker since there is no shim frame to drop.

test-fork now passes 20/20 (was ~7/20); make check is green with no hang.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 2 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread src/syscall/proc.c Outdated
Comment thread src/syscall/proc.c Outdated
Comment on lines +1944 to +1945
} else if (vexit->reason == HV_EXIT_REASON_CANCELED ||
vexit->reason == HV_EXIT_REASON_UNKNOWN) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add proper comments.

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Core fix looks right -- the UNKNOWN/CANCELED unification and the EL0-preempt redirect via HV_REG_PC/CPSR both match the bug description. One follow-up worth folding in:

src/syscall/proc.c:1995 -- rseq IP fixup has the same root cause this PR fixes in signal.c. The rseq_try_abort call in the CANCELED branch reads ELR_EL1 unconditionally, so for an EL0-preempted vCPU it sees the stale syscall-return PC and won't fire the IP fixup for a critical section actually interrupted at EL0. Partial cover: when signal_pending() is also true on the same exit, signal_deliver re-runs rseq_try_abort with the el0_preempt-corrected PC, so the abort still happens. The gap is exits with no queued signal (fork-barrier wakeups, future ptrace wake paths). Mirror the CPSR-based PC selection here for symmetry:

uint64_t cur_pc, cur_cpsr = 0;
hv_vcpu_get_reg(vcpu, HV_REG_CPSR, &cur_cpsr);
bool el0 = (cur_cpsr & 0xfULL) == 0;
if (el0)
    hv_vcpu_get_reg(vcpu, HV_REG_PC, &cur_pc);
else
    hv_vcpu_get_sys_reg(vcpu, HV_SYS_REG_ELR_EL1, &cur_pc);
int rseq_rc = rseq_try_abort(g, current_thread->rseq_gva,
                             current_thread->rseq_signature, &cur_pc);
if (rseq_rc == 1) {
    if (el0)
        hv_vcpu_set_reg(vcpu, HV_REG_PC, cur_pc);
    else
        hv_vcpu_set_sys_reg(vcpu, HV_SYS_REG_ELR_EL1, cur_pc);
}

See inline note on the UNKNOWN routing.

Comment thread src/syscall/proc.c Outdated
}
} else if (vexit->reason == HV_EXIT_REASON_CANCELED) {
} else if (vexit->reason == HV_EXIT_REASON_CANCELED ||
vexit->reason == HV_EXIT_REASON_UNKNOWN) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Routing UNKNOWN through the same fall-through as CANCELED is broad: if HVF ever returns UNKNOWN for a genuine fault (not the SIGUSR2 race), the loop will silently retry instead of taking the unexpected exit reason crash path at the end of this switch. Consider gating it on something actionable being present after the drain, e.g.:

} else if (vexit->reason == HV_EXIT_REASON_CANCELED ||
           (vexit->reason == HV_EXIT_REASON_UNKNOWN &&
            (signal_pending() ||
             proc_exit_group_requested() ||
             (is_main && g_timed_out)))) {

Per the PR description the queued guest signal is already drained before we reach the switch, so signal_pending() is true for the cross-process SIGUSR2 race this is targeting -- but a genuine HVF error with no signal queued would still crash visibly instead of looping.

@jserv jserv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase latest main branch and resolve conflicts.

test-fork's phase-2 signal child spins in `while (!got_usr1) usleep()`
waiting for a SIGUSR1 sent cross-process by its parent. The signal was
delivered only ~35% of the time and lost the rest, so `make check` hung at
test-fork until the 60s per-test timeout -- often longer, as leaked
`elfuse --fork-child` orphans kept the driver's stdout pipe open.

Two complementary defects:

1. vcpu_run_loop treated HV_EXIT_REASON_UNKNOWN as fatal. A host SIGUSR2
   (the cross-process guest-signal transport) that interrupts hv_vcpu_run
   mid-execution aborts the run with UNKNOWN rather than the clean CANCELED
   that hv_vcpus_exit() produces for a vCPU caught between runs. Route
   UNKNOWN through the same cancellation handling so the already-queued
   guest signal is delivered instead of crashing the child. Gate UNKNOWN on
   an actionable event actually being present (a queued signal, a pending
   exit_group, or a fired timeout) so a genuine HVF fault with nothing
   pending still falls through to the "unexpected exit reason" crash path
   instead of silently retrying the run.

2. signal_deliver redirected to the handler only via ELR_EL1, which takes
   effect solely on an ERET from EL1 (the syscall-return path, gated by the
   shim's X8==2 exec_drop_frame marker). When the signal is delivered from
   the cancellation branch -- i.e. the vCPU was preempted while running EL0
   code (cross-process SIGUSR2, or SIGALRM in a tight loop) -- there is no
   pending ERET, the resume uses HV_REG_PC, and the ELR_EL1 write is a
   no-op: the handler never runs and only the X0=signum clobber lands,
   re-running the interrupted nanosleep with a bogus arg and spinning
   forever. Detect EL0 preemption from the live PSTATE (CPSR M[3:0]==0),
   save the interrupted PC from HV_REG_PC instead of the stale ELR_EL1, and
   redirect HV_REG_PC/CPSR directly; skip the X8==2 marker since there is no
   shim frame to drop.

   The rseq IP fixup in the same cancellation branch had the identical
   stale-ELR_EL1 hazard: rseq_try_abort read and wrote ELR_EL1
   unconditionally, so a critical section interrupted at EL0 with no queued
   signal (fork-barrier/ptrace wakeups) would never abort. Select HV_REG_PC
   vs ELR_EL1 from the live PSTATE there too, mirroring signal_deliver.

test-fork now passes 20/20; `make check` is green with no hang.
@Max042004 Max042004 force-pushed the fix-cross-process-signal-el0 branch from c21d882 to 3bf61cd Compare June 27, 2026 15:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test-fork intermittently fails with HVF exit reason 0x3 inside make check

2 participants