Skip to content
  • Vlastimil Babka's avatar
    mm: munlock: batch non-THP page isolation and munlock+putback using pagevec · 7225522b
    Vlastimil Babka authored
    
    
    Currently, munlock_vma_range() calls munlock_vma_page on each page in a
    loop, which results in repeated taking and releasing of the lru_lock
    spinlock for isolating pages one by one.  This patch batches the munlock
    operations using an on-stack pagevec, so that isolation is done under
    single lru_lock.  For THP pages, the old behavior is preserved as they
    might be split while putting them into the pagevec.  After this patch, a
    9% speedup was measured for munlocking a 56GB large memory area with THP
    disabled.
    
    A new function __munlock_pagevec() is introduced that takes a pagevec and:
    1) It clears PageMlocked and isolates all pages under lru_lock.  Zone page
    stats can be also updated using the variant which assumes disabled
    interrupts.  2) It finishes the munlock and lru putback on all pages under
    their lock_page.  Note that previously, lock_page covered also the
    PageMlocked clearing and page isolation, but it is not needed for those
    operations.
    
    Signed-off-by: default avatarVlastimil Babka <vbabka@suse.cz>
    Reviewed-by: default avatarJörn Engel <joern@logfs.org>
    Acked-by: default avatarMel Gorman <mgorman@suse.de>
    Cc: Michel Lespinasse <walken@google.com>
    Cc: Hugh Dickins <hughd@google.com>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: Johannes Weiner <hannes@cmpxchg.org>
    Cc: Michal Hocko <mhocko@suse.cz>
    Cc: Vlastimil Babka <vbabka@suse.cz>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    7225522b