• Andrea Arcangeli's avatar
    mmu-notifiers: core · cddb8a5c
    Andrea Arcangeli authored
    With KVM/GFP/XPMEM there isn't just the primary CPU MMU pointing to pages.
     There are secondary MMUs (with secondary sptes and secondary tlbs) too.
    sptes in the kvm case are shadow pagetables, but when I say spte in
    mmu-notifier context, I mean "secondary pte".  In GRU case there's no
    actual secondary pte and there's only a secondary tlb because the GRU
    secondary MMU has no knowledge about sptes and every secondary tlb miss
    event in the MMU always generates a page fault that has to be resolved by
    the CPU (this is not the case of KVM where the a secondary tlb miss will
    walk sptes in hardware and it will refill the secondary tlb transparently
    to software if the corresponding spte is present).  The same way
    zap_page_range has to invalidate the pte before freeing the page, the spte
    (and secondary tlb) must also be invalidated before any page is freed and
    Currently we take a page_count pin on every page mapped by sptes, but that
    means the pages can't be swapped whenever they're mapped by any spte
    because they're part of the guest working set.  Furthermore a spte unmap
    event can immediately lead to a page to be freed when the pin is released
    (so requiring the same complex and relatively slow tlb_gather smp safe
    logic we have in zap_page_range and that can be avoided completely if the
    spte unmap event doesn't require an unpin of the page previously mapped in
    the secondary MMU).
    The mmu notifiers allow kvm/GRU/XPMEM to attach to the tsk->mm and know
    when the VM is swapping or freeing or doing anything on the primary MMU so
    that the secondary MMU code can drop sptes before the pages are freed,
    avoiding all page pinning and allowing 100% reliable swapping of guest
    physical address space.  Furthermore it avoids the code that teardown the
    mappings of the secondary MMU, to implement a logic like tlb_gather in
    zap_page_range that would require many IPI to flush other cpu tlbs, for
    each fixed number of spte unmapped.
    To make an example: if what happens on the primary MMU is a protection
    downgrade (from writeable to wrprotect) the secondary MMU mappings will be
    invalidated, and the next secondary-mmu-page-fault will call
    get_user_pages and trigger a do_wp_page through get_user_pages if it
    called get_user_pages with write=1, and it'll re-establishing an updated
    spte or secondary-tlb-mapping on the copied page.  Or it will setup a
    readonly spte or readonly tlb mapping if it's a guest-read, if it calls
    get_user_pages with write=0.  This is just an example.
    This allows to map any page pointed by any pte (and in turn visible in the
    primary CPU MMU), into a secondary MMU (be it a pure tlb like GRU, or an
    full MMU with both sptes and secondary-tlb like the shadow-pagetable layer
    with kvm), or a remote DMA in software like XPMEM (hence needing of
    schedule in XPMEM code to send the invalidate to the remote node, while no
    need to schedule in kvm/gru as it's an immediate event like invalidating
    primary-mmu pte).
    At least for KVM without this patch it's impossible to swap guests
    reliably.  And having this feature and removing the page pin allows
    several other optimizations that simplify life considerably.
    1) mm_take_all_locks() to register the mmu notifier when the whole VM
       isn't doing anything with "mm".  This allows mmu notifier users to keep
       track if the VM is in the middle of the invalidate_range_begin/end
       critical section with an atomic counter incraese in range_begin and
       decreased in range_end.  No secondary MMU page fault is allowed to map
       any spte or secondary tlb reference, while the VM is in the middle of
       range_begin/end as any page returned by get_user_pages in that critical
       section could later immediately be freed without any further
       ->invalidate_page notification (invalidate_range_begin/end works on
       ranges and ->invalidate_page isn't called immediately before freeing
       the page).  To stop all page freeing and pagetable overwrites the
       mmap_sem must be taken in write mode and all other anon_vma/i_mmap
       locks must be taken too.
    2) It'd be a waste to add branches in the VM if nobody could possibly
       run KVM/GRU/XPMEM on the kernel, so mmu notifiers will only enabled if
       CONFIG_KVM=m/y.  In the current kernel kvm won't yet take advantage of
       mmu notifiers, but this already allows to compile a KVM external module
       against a kernel with mmu notifiers enabled and from the next pull from
       kvm.git we'll start using them.  And GRU/XPMEM will also be able to
       continue the development by enabling KVM=m in their config, until they
       submit all GRU/XPMEM GPLv2 code to the mainline kernel.  Then they can
       also enable MMU_NOTIFIERS in the same way KVM does it (even if KVM=n).
       This guarantees nobody selects MMU_NOTIFIER=y if KVM and GRU and XPMEM
       are all =n.
    The mmu_notifier_register call can fail because mm_take_all_locks may be
    interrupted by a signal and return -EINTR.  Because mmu_notifier_reigster
    is used when a driver startup, a failure can be gracefully handled.  Here
    an example of the change applied to kvm to register the mmu notifiers.
    Usually when a driver startups other allocations are required anyway and
    -ENOMEM failure paths exists already.
     struct  kvm *kvm_arch_create_vm(void)
            struct kvm *kvm = kzalloc(sizeof(struct kvm), GFP_KERNEL);
    +       int err;
            if (!kvm)
                    return ERR_PTR(-ENOMEM);
    +       kvm->arch.mmu_notifier.ops = &kvm_mmu_notifier_ops;
    +       err = mmu_notifier_register(&kvm->arch.mmu_notifier, current->mm);
    +       if (err) {
    +               kfree(kvm);
    +               return ERR_PTR(err);
    +       }
            return kvm;
    mmu_notifier_unregister returns void and it's reliable.
    The patch also adds a few needed but missing includes that would prevent
    kernel to compile after these changes on non-x86 archs (x86 didn't need
    them by luck).
    [akpm@linux-foundation.org: coding-style fixes]
    [akpm@linux-foundation.org: fix mm/filemap_xip.c build]
    [akpm@linux-foundation.org: fix mm/mmu_notifier.c build]
    Signed-off-by: default avatarAndrea Arcangeli <andrea@qumranet.com>
    Signed-off-by: default avatarNick Piggin <npiggin@suse.de>
    Signed-off-by: default avatarChristoph Lameter <cl@linux-foundation.org>
    Cc: Jack Steiner <steiner@sgi.com>
    Cc: Robin Holt <holt@sgi.com>
    Cc: Nick Piggin <npiggin@suse.de>
    Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
    Cc: Kanoj Sarcar <kanojsarcar@yahoo.com>
    Cc: Roland Dreier <rdreier@cisco.com>
    Cc: Steve Wise <swise@opengridcomputing.com>
    Cc: Avi Kivity <avi@qumranet.com>
    Cc: Hugh Dickins <hugh@veritas.com>
    Cc: Rusty Russell <rusty@rustcorp.com.au>
    Cc: Anthony Liguori <aliguori@us.ibm.com>
    Cc: Chris Wright <chrisw@redhat.com>
    Cc: Marcelo Tosatti <marcelo@kvack.org>
    Cc: Eric Dumazet <dada1@cosmosbay.com>
    Cc: "Paul E. McKenney" <paulmck@us.ibm.com>
    Cc: Izik Eidus <izike@qumranet.com>
    Cc: Anthony Liguori <aliguori@us.ibm.com>
    Cc: Rik van Riel <riel@redhat.com>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
fork.c 40.6 KB