Skip to content
  • Tetsuo Handa's avatar
    kernel/hung_task.c: show all hung tasks before panic · 401c636a
    Tetsuo Handa authored
    When we get a hung task it can often be valuable to see _all_ the hung
    tasks on the system before calling panic().
    
    Quoting from https://syzkaller.appspot.com/text?tag=CrashReport&id=5316056503549952
    ----------------------------------------
    INFO: task syz-executor0:6540 blocked for more than 120 seconds.
          Not tainted 4.16.0+ #13
    "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
    syz-executor0   D23560  6540   4521 0x80000004
    Call Trace:
     context_switch kernel/sched/core.c:2848 [inline]
     __schedule+0x8fb/0x1ef0 kernel/sched/core.c:3490
     schedule+0xf5/0x430 kernel/sched/core.c:3549
     schedule_preempt_disabled+0x10/0x20 kernel/sched/core.c:3607
     __mutex_lock_common kernel/locking/mutex.c:833 [inline]
     __mutex_lock+0xb7f/0x1810 kernel/locking/mutex.c:893
     mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:908
     lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355
     __blkdev_driver_ioctl block/ioctl.c:303 [inline]
     blkdev_ioctl+0x1759/0x1e00 block/ioctl.c:601
     ioctl_by_bdev+0xa5/0x110 fs/block_dev.c:2060
     isofs_get_last_session fs/isofs/inode.c:567 [inline]
     isofs_fill_super+0x2ba9/0x3bc0 fs/isofs/inode.c:660
     mount_bdev+0x2b7/0x370 fs/super.c:1119
     isofs_mount+0x34/0x40 fs/isofs/inode.c:1560
     mount_fs+0x66/0x2d0 fs/super.c:1222
     vfs_kern_mount.part.26+0xc6/0x4a0 fs/namespace.c:1037
     vfs_kern_mount fs/namespace.c:2514 [inline]
     do_new_mount fs/namespace.c:2517 [inline]
     do_mount+0xea4/0x2b90 fs/namespace.c:2847
     ksys_mount+0xab/0x120 fs/namespace.c:3063
     SYSC_mount fs/namespace.c:3077 [inline]
     SyS_mount+0x39/0x50 fs/namespace.c:3074
     do_syscall_64+0x281/0x940 arch/x86/entry/common.c:287
     entry_SYSCALL_64_after_hwframe+0x42/0xb7
    (...snipped...)
    Showing all locks held in the system:
    (...snipped...)
    2 locks held by syz-executor0/6540:
     #0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: alloc_super fs/super.c:211 [inline]
     #0: 00000000566d4c39 (&type->s_umount_key#49/1){+.+.}, at: sget_userns+0x3b2/0xe60 fs/super.c:502 /* down_write_nested(&s->s_umount, SINGLE_DEPTH_NESTING); */
     #1: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
    (...snipped...)
    3 locks held by syz-executor7/6541:
     #0: 0000000043ca8836 (&lo->lo_ctl_mutex/1){+.+.}, at: lo_ioctl+0x8b/0x1b70 drivers/block/loop.c:1355 /* mutex_lock_nested(&lo->lo_ctl_mutex, 1); */
     #1: 000000007bf3d3f9 (&bdev->bd_mutex){+.+.}, at: blkdev_reread_part+0x1e/0x40 block/ioctl.c:192
     #2: 00000000566d4c39 (&type->s_umount_key#50){.+.+}, at: __get_super.part.10+0x1d3/0x280 fs/super.c:663 /* down_read(&sb->s_umount); */
    ----------------------------------------
    
    When reporting an AB-BA deadlock like shown above, it would be nice if
    trace of PID=6541 is printed as well as trace of PID=6540 before calling
    panic().
    
    Showing hung tasks up to /proc/sys/kernel/hung_task_warnings could delay
    calling panic() but normally there should not be so many hung tasks.
    
    Link: http://lkml.kernel.org/r/201804050705.BHE57833.HVFOFtSOMQJFOL@I-love.SAKURA.ne.jp
    
    
    Signed-off-by: default avatarTetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
    Acked-by: default avatarPaul E. McKenney <paulmck@linux.vnet.ibm.com>
    Acked-by: default avatarDmitry Vyukov <dvyukov@google.com>
    Cc: Vegard Nossum <vegard.nossum@oracle.com>
    Cc: Mandeep Singh Baines <msb@chromium.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Ingo Molnar <mingo@kernel.org>
    Signed-off-by: default avatarAndrew Morton <akpm@linux-foundation.org>
    
    Signed-off-by: default avatarLinus Torvalds <torvalds@linux-foundation.org>
    401c636a