• Peter Zijlstra's avatar
    mm/tlb, x86/mm: Support invalidating TLB caches for RCU_TABLE_FREE · e43594da
    Peter Zijlstra authored
    commit d86564a2 upstream.
    
    Jann reported that x86 was missing required TLB invalidates when he
    hit the !*batch slow path in tlb_remove_table().
    
    This is indeed the case; RCU_TABLE_FREE does not provide TLB (cache)
    invalidates, the PowerPC-hash where this code originated and the
    Sparc-hash where this was subsequently used did not need that. ARM
    which later used this put an explicit TLB invalidate in their
    __p*_free_tlb() functions, and PowerPC-radix followed that example.
    
    But when we hooked up x86 we failed to consider this. Fix this by
    (optionally) hooking tlb_remove_table() into the TLB invalidate code.
    
    NOTE: s390 was also needing something like this and might now
          be able to use the generic code again.
    
    [ Modified to be on top of Nick's cleanups, which simplified this patch
      now that tlb_flush_mmu_tlbonly() really only flushes the TLB - Linus ]
    
    Fixes: 9e52fc2b
    
     ("x86/mm: Enable RCU based page table freeing (CONFIG_HAVE_RCU_TABLE_FREE=y)")
    Reported-by: default avatarJann Horn <jannh@google.com>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Acked-by: default avatarRik van Riel <riel@surriel.com>
    Cc: Nicholas Piggin <npiggin@gmail.com>
    Cc: David Miller <davem@davemloft.net>
    Cc: Will Deacon <will.deacon@arm.com>
    Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
    Cc: Michael Ellerman <mpe@ellerman.id.au>
    Cc: stable@kernel.org
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    Signed-off-by: default avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
    e43594da