• Kiran Kumar Modukuri's avatar
    fscache: Fix reference overput in fscache_attach_object() error handling · f29507ce
    Kiran Kumar Modukuri authored
    When a cookie is allocated that causes fscache_object structs to be
    allocated, those objects are initialised with the cookie pointer, but
    aren't blessed with a ref on that cookie unless the attachment is
    successfully completed in fscache_attach_object().
    
    If attachment fails because the parent object was dying or there was a
    collision, fscache_attach_object() returns without incrementing the cookie
    counter - but upon failure of this function, the object is released which
    then puts the cookie, whether or not a ref was taken on the cookie.
    
    Fix this by taking a ref on the cookie when it is assigned in
    fscache_object_init(), even when we're creating a root object.
    
    
    Analysis from Kiran Kumar:
    
    This bug has been seen in 4.4.0-124-generic #148-Ubuntu kernel
    
    BugLink: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1776277
    
    fscache cookie ref count updated incorrectly during fscache object
    allocation resulting in following Oops.
    
    kernel BUG at /build/linux-Y09MKI/linux-4.4.0/fs/fscache/internal.h:321!
    kernel BUG at /build/linux-Y09MKI/linux-4.4.0/fs/fscache/cookie.c:639!
    
    [Cause]
    Two threads are trying to do operate on a cookie and two objects.
    
    (1) One thread tries to unmount the filesystem and in process goes over a
        huge list of objects marking them dead and deleting the objects.
        cookie->usage is also decremented in following path:
    
          nfs_fscache_release_super_cookie
           -> __fscache_relinquish_cookie
            ->__fscache_cookie_put
            ->BUG_ON(atomic_read(&cookie->usage) <= 0);
    
    (2) A second thread tries to lookup an object for reading data in following
        path:
    
        fscache_alloc_object
        1) cachefiles_alloc_object
            -> fscache_object_init
               -> assign cookie, but usage not bumped.
        2) fscache_attach_object -> fails in cant_attach_object because the
             cookie's backing object or cookie's->parent object are going away
        3) fscache_put_object
            -> cachefiles_put_object
              ->fscache_object_destroy
                ->fscache_cookie_put
                   ->BUG_ON(atomic_read(&cookie->usage) <= 0);
    
    [NOTE from dhowells] It's unclear as to the circumstances in which (2) can
    take place, given that thread (1) is in nfs_kill_super(), however a
    conflicting NFS mount with slightly different parameters that creates a
    different superblock would do it.  A backtrace from Kiran seems to show
    that this is a possibility:
    
        kernel BUG at/build/linux-Y09MKI/linux-4.4.0/fs/fscache/cookie.c:639!
        ...
        RIP: __fscache_cookie_put+0x3a/0x40 [fscache]
        Call Trace:
         __fscache_relinquish_cookie+0x87/0x120 [fscache]
         nfs_fscache_release_super_cookie+0x2d/0xb0 [nfs]
         nfs_kill_super+0x29/0x40 [nfs]
         deactivate_locked_super+0x48/0x80
         deactivate_super+0x5c/0x60
         cleanup_mnt+0x3f/0x90
         __cleanup_mnt+0x12/0x20
         task_work_run+0x86/0xb0
         exit_to_usermode_loop+0xc2/0xd0
         syscall_return_slowpath+0x4e/0x60
         int_ret_from_sys_call+0x25/0x9f
    
    [Fix] Bump up the cookie usage in fscache_object_init, when it is first
    being assigned a cookie atomically such that the cookie is added and bumped
    up if its refcount is not zero.  Remove the assignment in
    fscache_attach_object().
    
    [Testcase]
    I have run ~100 hours of NFS stress tests and not seen this bug recur.
    
    [Regression Potential]
     - Limited to fscache/cachefiles.
    
    Fixes: ccc4fc3d ("FS-Cache: Implement the cookie management part of the netfs API")
    Signed-off-by: 's avatarKiran Kumar Modukuri <kiran.modukuri@gmail.com>
    Signed-off-by: 's avatarDavid Howells <dhowells@redhat.com>
    f29507ce
Name
Last commit
Last update
..
Kconfig Loading commit data...
Makefile Loading commit data...
cache.c Loading commit data...
cookie.c Loading commit data...
fsdef.c Loading commit data...
histogram.c Loading commit data...
internal.h Loading commit data...
main.c Loading commit data...
netfs.c Loading commit data...
object-list.c Loading commit data...
object.c Loading commit data...
operation.c Loading commit data...
page.c Loading commit data...
proc.c Loading commit data...
stats.c Loading commit data...