arm64: entry: fix non-NMI kernel<->kernel transitions
authorMark Rutland <mark.rutland@arm.com>
Mon, 30 Nov 2020 11:59:48 +0000 (11:59 +0000)
committerWill Deacon <will@kernel.org>
Mon, 30 Nov 2020 12:11:38 +0000 (12:11 +0000)
commit7cd1ea1010acbede7eb87b6abb6198921fb36957
treec7dff8e6f618239ab852898ea1ff6d862ba9563c
parent1ec2f2c05b2ab845d068bff29bd32dbfc6a6ad4c
arm64: entry: fix non-NMI kernel<->kernel transitions

There are periods in kernel mode when RCU is not watching and/or the
scheduler tick is disabled, but we can still take exceptions such as
interrupts. The arm64 exception handlers do not account for this, and
it's possible that RCU is not watching while an exception handler runs.

The x86/generic entry code handles this by ensuring that all (non-NMI)
kernel exception handlers call irqentry_enter() and irqentry_exit(),
which handle RCU, lockdep, and IRQ flag tracing. We can't yet move to
the generic entry code, and already hadnle the user<->kernel transitions
elsewhere, so we add new kernel<->kernel transition helpers alog the
lines of the generic entry code.

Since we now track interrupts becoming masked when an exception is
taken, local_daif_inherit() is modified to track interrupts becoming
re-enabled when the original context is inherited. To balance the
entry/exit paths, each handler masks all DAIF exceptions before
exit_to_kernel_mode().

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: James Morse <james.morse@arm.com>
Cc: Will Deacon <will@kernel.org>
Link: https://lore.kernel.org/r/20201130115950.22492-10-mark.rutland@arm.com
Signed-off-by: Will Deacon <will@kernel.org>
arch/arm64/include/asm/daifflags.h
arch/arm64/kernel/entry-common.c