1. 17 Dec, 2018 1 commit
    • Ming Lei's avatar
      blk-mq: export hctx->type in debugfs instead of sysfs · 346fc108
      Ming Lei authored
      Now we only export hctx->type via sysfs, and there isn't such info
      in hctx entry under debugfs. We often use debugfs only to diagnose
      queue mapping issue, so add the support in debugfs.
      Queue mapping becomes a bit more complicated after multiple queue
      mapping is supported, we may write blktest to verify if queue mapping
      is valid based on blk-mq-debugfs.
      Given not necessary to export hctx->type twice, so remove the export
      from sysfs.
      Cc: Jeff Moyer <jmoyer@redhat.com>
      Cc: Mike Snitzer <snitzer@redhat.com>
      Reviewed-by: 's avatarChristoph Hellwig <hch@lst.de>
      Signed-off-by: 's avatarMing Lei <ming.lei@redhat.com>
      Signed-off-by: 's avatarJens Axboe <axboe@kernel.dk>
  2. 04 Dec, 2018 1 commit
  3. 21 Nov, 2018 1 commit
  4. 15 Nov, 2018 1 commit
  5. 07 Nov, 2018 1 commit
  6. 24 May, 2018 1 commit
  7. 15 Jan, 2018 1 commit
    • Mike Snitzer's avatar
      block: properly protect the 'queue' kobj in blk_unregister_queue · 667257e8
      Mike Snitzer authored
      The original commit e9a823fb (block: fix warning when I/O elevator
      is changed as request_queue is being removed) is pretty conflated.
      "conflated" because the resource being protected by q->sysfs_lock isn't
      the queue_flags (it is the 'queue' kobj).
      q->sysfs_lock serializes __elevator_change() (via elv_iosched_store)
      from racing with blk_unregister_queue():
      1) By holding q->sysfs_lock first, __elevator_change() can complete
      before a racing blk_unregister_queue().
      2) Conversely, __elevator_change() is testing for QUEUE_FLAG_REGISTERED
      in case elv_iosched_store() loses the race with blk_unregister_queue(),
      it needs a way to know the 'queue' kobj isn't there.
      Expand the scope of blk_unregister_queue()'s q->sysfs_lock use so it is
      held until after the 'queue' kobj is removed.
      To do so blk_mq_unregister_dev() must not also take q->sysfs_lock.  So
      rename __blk_mq_unregister_dev() to blk_mq_unregister_dev().
      Also, blk_unregister_queue() should use q->queue_lock to protect against
      any concurrent writes to q->queue_flags -- even though chances are the
      queue is being cleaned up so no concurrent writes are likely.
      Fixes: e9a823fb ("block: fix warning when I/O elevator is changed as request_queue is being removed")
      Signed-off-by: 's avatarMike Snitzer <snitzer@redhat.com>
      Reviewed-by: 's avatarMing Lei <ming.lei@redhat.com>
      Signed-off-by: 's avatarJens Axboe <axboe@kernel.dk>
  8. 04 May, 2017 2 commits
  9. 26 Apr, 2017 4 commits
  10. 08 Mar, 2017 4 commits
  11. 02 Feb, 2017 1 commit
  12. 27 Jan, 2017 5 commits
  13. 17 Jan, 2017 1 commit
  14. 10 Nov, 2016 1 commit
    • Jens Axboe's avatar
      block: add scalable completion tracking of requests · cf43e6be
      Jens Axboe authored
      For legacy block, we simply track them in the request queue. For
      blk-mq, we track them on a per-sw queue basis, which we can then
      sum up through the hardware queues and finally to a per device
      The stats are tracked in, roughly, 0.1s interval windows.
      Add sysfs files to display the stats.
      The feature is off by default, to avoid any extra overhead. In-kernel
      users of it can turn it on by setting QUEUE_FLAG_STATS in the queue
      flags. We currently don't turn it on if someone just reads any of
      the stats files, that is something we could add as well.
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
  15. 21 Sep, 2016 1 commit
  16. 16 Sep, 2016 1 commit
    • Jens Axboe's avatar
      blk-mq: account higher order dispatch · 703fd1c0
      Jens Axboe authored
      We currently account a '0' dispatch, and anything above that still falls
      below the range set by BLK_MQ_MAX_DISPATCH_ORDER. If we dispatch more,
      we don't account it.
      Change the last bucket to be inclusive of anything above the range we
      track, and have the sysfs file reflect that by including a '+' in the
      $ cat /sys/block/nvme0n1/mq/0/dispatched
              0	1006
              1	20229
              2	1
              4	0
              8	0
             16	0
             32+	0
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
      Reviewed-by: 's avatarOmar Sandoval <osandov@fb.com>
  17. 14 Sep, 2016 2 commits
  18. 04 Aug, 2016 1 commit
  19. 20 Mar, 2016 1 commit
  20. 09 Feb, 2016 1 commit
    • Keith Busch's avatar
      blk-mq: dynamic h/w context count · 868f2f0b
      Keith Busch authored
      The hardware's provided queue count may change at runtime with resource
      provisioning. This patch allows a block driver to alter the number of
      h/w queues available when its resource count changes.
      The main part is a new blk-mq API to request a new number of h/w queues
      for a given live tag set. The new API freezes all queues using that set,
      then adjusts the allocated count prior to remapping these to CPUs.
      The bulk of the rest just shifts where h/w contexts and all their
      artifacts are allocated and freed.
      The number of max h/w contexts is capped to the number of possible cpus
      since there is no use for more than that. As such, all pre-allocated
      memory for pointers need to account for the max possible rather than
      the initial number of queues.
      A side effect of this is that the blk-mq will proceed successfully as
      long as it can allocate at least one h/w context. Previously it would
      fail request queue initialization if less than the requested number
      was allocated.
      Signed-off-by: 's avatarKeith Busch <keith.busch@intel.com>
      Reviewed-by: 's avatarChristoph Hellwig <hch@lst.de>
      Tested-by: 's avatarJon Derrick <jonathan.derrick@intel.com>
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
  21. 07 Nov, 2015 1 commit
    • Jens Axboe's avatar
      block: add block polling support · 05229bee
      Jens Axboe authored
      Add basic support for polling for specific IO to complete. This uses
      the cookie that blk-mq passes back, which enables the block layer
      to pass this cookie to the driver to spin for a specific request.
      This will be combined with request latency tracking, so we can make
      qualified decisions about when to poll and when not to. For now, for
      benchmark purposes, we add a sysfs file that controls whether polling
      is enabled or not.
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
      Acked-by: 's avatarChristoph Hellwig <hch@lst.de>
      Acked-by: 's avatarKeith Busch <keith.busch@intel.com>
  22. 21 Oct, 2015 1 commit
    • Dan Williams's avatar
      block: generic request_queue reference counting · 3ef28e83
      Dan Williams authored
      Allow pmem, and other synchronous/bio-based block drivers, to fallback
      on a per-cpu reference count managed by the core for tracking queue
      live/dead state.
      The existing per-cpu reference count for the blk_mq case is promoted to
      be used in all block i/o scenarios.  This involves initializing it by
      default, waiting for it to drop to zero at exit, and holding a live
      reference over the invocation of q->make_request_fn() in
      generic_make_request().  The blk_mq code continues to take its own
      reference per blk_mq request and retains the ability to freeze the
      queue, but the check that the queue is frozen is moved to
      This fixes crash signatures like the following:
       BUG: unable to handle kernel paging request at ffff880140000000
       Call Trace:
        [<ffffffff8145e8bf>] ? copy_user_handle_tail+0x5f/0x70
        [<ffffffffa004e1e0>] pmem_do_bvec.isra.11+0x70/0xf0 [nd_pmem]
        [<ffffffffa004e331>] pmem_make_request+0xd1/0x200 [nd_pmem]
        [<ffffffff811c3162>] ? mempool_alloc+0x72/0x1a0
        [<ffffffff8141f8b6>] generic_make_request+0xd6/0x110
        [<ffffffff8141f966>] submit_bio+0x76/0x170
        [<ffffffff81286dff>] submit_bh_wbc+0x12f/0x160
        [<ffffffff81286e62>] submit_bh+0x12/0x20
        [<ffffffff813395bd>] jbd2_write_superblock+0x8d/0x170
        [<ffffffff8133974d>] jbd2_mark_journal_empty+0x5d/0x90
        [<ffffffff813399cb>] jbd2_journal_destroy+0x24b/0x270
        [<ffffffff810bc4ca>] ? put_pwq_unlocked+0x2a/0x30
        [<ffffffff810bc6f5>] ? destroy_workqueue+0x225/0x250
        [<ffffffff81303494>] ext4_put_super+0x64/0x360
        [<ffffffff8124ab1a>] generic_shutdown_super+0x6a/0xf0
      Cc: Jens Axboe <axboe@kernel.dk>
      Cc: Keith Busch <keith.busch@intel.com>
      Cc: Ross Zwisler <ross.zwisler@linux.intel.com>
      Suggested-by: 's avatarChristoph Hellwig <hch@lst.de>
      Reviewed-by: 's avatarChristoph Hellwig <hch@lst.de>
      Tested-by: 's avatarRoss Zwisler <ross.zwisler@linux.intel.com>
      Signed-off-by: 's avatarDan Williams <dan.j.williams@intel.com>
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
  23. 29 Sep, 2015 2 commits
    • Akinobu Mita's avatar
      blk-mq: fix deadlock when reading cpu_list · 60de074b
      Akinobu Mita authored
      CPU hotplug handling for blk-mq (blk_mq_queue_reinit) acquires
      all_q_mutex in blk_mq_queue_reinit_notify() and then removes sysfs
      entries by blk_mq_sysfs_unregister().  Removing sysfs entry needs to
      be blocked until the active reference of the kernfs_node to be zero.
      On the other hand, reading blk_mq_hw_sysfs_cpu sysfs entry (e.g.
      /sys/block/nullb0/mq/0/cpu_list) acquires all_q_mutex in
      If these happen at the same time, a deadlock can happen.  Because one
      can wait for the active reference to be zero with holding all_q_mutex,
      and the other tries to acquire all_q_mutex with holding the active
      The reason that all_q_mutex is acquired in blk_mq_hw_sysfs_cpus_show()
      is to avoid reading an imcomplete hctx->cpumask.  Since reading sysfs
      entry for blk-mq needs to acquire q->sysfs_lock, we can avoid deadlock
      and reading an imcomplete hctx->cpumask by protecting q->sysfs_lock
      while hctx->cpumask is being updated.
      Signed-off-by: 's avatarAkinobu Mita <akinobu.mita@gmail.com>
      Reviewed-by: 's avatarMing Lei <tom.leiming@gmail.com>
      Cc: Ming Lei <tom.leiming@gmail.com>
      Cc: Wanpeng Li <wanpeng.li@hotmail.com>
      Reviewed-by: 's avatarChristoph Hellwig <hch@lst.de>
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
    • Akinobu Mita's avatar
      blk-mq: fix sysfs registration/unregistration race · 4593fdbe
      Akinobu Mita authored
      There is a race between cpu hotplug handling and adding/deleting
      gendisk for blk-mq, where both are trying to register and unregister
      the same sysfs entries.
          --> blk_mq_init_queue
              --> blk_mq_init_allocated_queue
                  --> add to 'all_q_list' (*)
          --> add_disk
              --> blk_register_queue
                  --> blk_mq_register_disk (++)
          --> del_gendisk
              --> blk_unregister_queue
                  --> blk_mq_unregister_disk (--)
          --> blk_cleanup_queue
              --> blk_mq_free_queue
                  --> del from 'all_q_list' (*)
          --> blk_mq_sysfs_unregister (-)
          --> blk_mq_sysfs_register (+)
      While the request queue is added to 'all_q_list' (*),
      blk_mq_queue_reinit() can be called for the queue anytime by CPU
      hotplug callback.  But blk_mq_sysfs_unregister (-) and
      blk_mq_sysfs_register (+) in blk_mq_queue_reinit must not be called
      before blk_mq_register_disk (++) and after blk_mq_unregister_disk (--)
      is finished.  Because '/sys/block/*/mq/' is not exists.
      There has already been BLK_MQ_F_SYSFS_UP flag in hctx->flags which can
      be used to track these sysfs stuff, but it is only fixing this issue
      In order to fix it completely, we just need per-queue flag instead of
      per-hctx flag with appropriate locking.  So this introduces
      q->mq_sysfs_init_done which is properly protected with all_q_mutex.
      Also, we need to ensure that blk_mq_map_swqueue() is called with
      all_q_mutex is held.  Since hctx->nr_ctx is reset temporarily and
      updated in blk_mq_map_swqueue(), so we should avoid
      blk_mq_register_hctx() seeing the temporary hctx->nr_ctx value
      in CPU hotplug handling or adding/deleting gendisk .
      Signed-off-by: 's avatarAkinobu Mita <akinobu.mita@gmail.com>
      Reviewed-by: 's avatarMing Lei <tom.leiming@gmail.com>
      Cc: Ming Lei <tom.leiming@gmail.com>
      Reviewed-by: 's avatarChristoph Hellwig <hch@lst.de>
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
  24. 15 Aug, 2015 1 commit
    • Ming Lei's avatar
      blk-mq: fix buffer overflow when reading sysfs file of 'pending' · 596f5aad
      Ming Lei authored
      There may be lots of pending requests so that the buffer of PAGE_SIZE
      can't hold them at all.
      One typical example is scsi-mq, the queue depth(.can_queue) of
      scsi_host and blk-mq is quite big but scsi_device's queue_depth
      is a bit small(.cmd_per_lun), then it is quite easy to have lots
      of pending requests in hw queue.
      This patch fixes the following warning and the related memory
      [  359.025101] fill_read_buffer: blk_mq_hw_sysfs_show+0x0/0x7d returned bad count^M
      [  359.055595] irq event stamp: 15537^M
      [  359.055606] general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC ^M
      [  359.055614] Dumping ftrace buffer:^M
      [  359.055660]    (ftrace buffer empty)^M
      [  359.055672] Modules linked in: nbd ipv6 kvm_intel kvm serio_raw^M
      [  359.055678] CPU: 4 PID: 21631 Comm: stress-ng-sysfs Not tainted 4.2.0-rc5-next-20150805 #434^M
      [  359.055679] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011^M
      [  359.055682] task: ffff8802161cc000 ti: ffff88021b4a8000 task.ti: ffff88021b4a8000^M
      [  359.055693] RIP: 0010:[<ffffffff811541c5>]  [<ffffffff811541c5>] __kmalloc+0xe8/0x152^M
      Cc: <stable@vger.kernel.org>
      Signed-off-by: 's avatarMing Lei <ming.lei@canonical.com>
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
  25. 13 Mar, 2015 1 commit
    • Mike Snitzer's avatar
      blk-mq: add blk_mq_init_allocated_queue and export blk_mq_register_disk · b62c21b7
      Mike Snitzer authored
      Add a variant of blk_mq_init_queue that allows a previously allocated
      queue to be initialized.  blk_mq_init_allocated_queue models
      blk_init_allocated_queue -- which was also created for DM's use.
      DM's approach to device creation requires a placeholder request_queue be
      allocated for use with alloc_dev() but the decision about what type of
      request_queue will be ultimately created is deferred until all component
      devices referenced in the DM table are processed to determine the table
      type (request-based, blk-mq request-based, or bio-based).
      Also, because of DM's late finalization of the request_queue type
      the call to blk_mq_register_disk() doesn't happen during alloc_dev().
      Must export blk_mq_register_disk() so that DM can backfill the 'mq' dir
      once the blk-mq queue is fully allocated.
      Signed-off-by: 's avatarMike Snitzer <snitzer@redhat.com>
      Reviewed-by: 's avatarMing Lei <ming.lei@canonical.com>
      Signed-off-by: 's avatarJens Axboe <axboe@fb.com>
  26. 29 Jan, 2015 1 commit
  27. 20 Jan, 2015 1 commit