• Jens Axboe's avatar
    sbitmap: ammortize cost of clearing bits · ea86ea2c
    Jens Axboe authored
    sbitmap maintains a set of words that we use to set and clear bits, with
    each bit representing a tag for blk-mq. Even though we spread the bits
    out and maintain a hint cache, one particular bit allocated will end up
    being cleared in the exact same spot.
    This introduces batched clearing of bits. Instead of clearing a given
    bit, the same bit is set in a cleared/free mask instead. If we fail
    allocating a bit from a given word, then we check the free mask, and
    batch move those cleared bits at that time. This trades 64 atomic bitops
    for 2 cmpxchg().
    In a threaded poll test case, half the overhead of getting and clearing
    tags is removed with this change. On another poll test case with a
    single thread, performance is unchanged.
    Reviewed-by: default avatarOmar Sandoval <osandov@fb.com>
    Signed-off-by: default avatarJens Axboe <axboe@kernel.dk>
sbitmap.c 15.4 KB