1. 26 Mar, 2018 1 commit
  2. 19 Mar, 2018 2 commits
  3. 23 Feb, 2018 1 commit
  4. 18 Feb, 2018 3 commits
  5. 27 Nov, 2017 1 commit
    • Nicolas Pitre's avatar
      percpu: hack to let the CRIS architecture to boot until they clean up · abee2105
      Nicolas Pitre authored
      Commit 438a5061 ("percpu: don't forget to free the temporary struct
      pcpu_alloc_info") uncovered a problem on the CRIS architecture where
      the bootmem allocator is initialized with virtual addresses. Given it
      has:
      
          #define __va(x) ((void *)((unsigned long)(x) | 0x80000000))
      
      then things just work out because the end result is the same whether you
      give this a physical or a virtual address.
      
      Untill you call memblock_free_early(__pa(address)) that is, because
      values from __pa() don't match with the virtual addresses stuffed in the
      bootmem allocator anymore.
      
      Avoid freeing the temporary pcpu_alloc_info memory on that architecture
      until they fix things up to let the kernel boot like it did before.
      Signed-off-by: 's avatarNicolas Pitre <nico@linaro.org>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      Fixes: 438a5061 ("percpu: don't forget to free the temporary struct pcpu_alloc_info")
      abee2105
  6. 19 Oct, 2017 1 commit
    • Daniel Borkmann's avatar
      mm, percpu: add support for __GFP_NOWARN flag · 0ea7eeec
      Daniel Borkmann authored
      Add an option for pcpu_alloc() to support __GFP_NOWARN flag.
      Currently, we always throw a warning when size or alignment
      is unsupported (and also dump stack on failed allocation
      requests). The warning itself is harmless since we return
      NULL anyway for any failed request, which callers are
      required to handle anyway. However, it becomes harmful when
      panic_on_warn is set.
      
      The rationale for the WARN() in pcpu_alloc() is that it can
      be tracked when larger than supported allocation requests are
      made such that allocations limits can be tweaked if warranted.
      This makes sense for in-kernel users, however, there are users
      of pcpu allocator where allocation size is derived from user
      space requests, e.g. when creating BPF maps. In these cases,
      the requests should fail gracefully without throwing a splat.
      
      The current work-around was to check allocation size against
      the upper limit of PCPU_MIN_UNIT_SIZE from call-sites for
      bailing out prior to a call to pcpu_alloc() in order to
      avoid throwing the WARN(). This is bad in multiple ways since
      PCPU_MIN_UNIT_SIZE is an implementation detail, and having
      the checks on call-sites only complicates the code for no
      good reason. Thus, lets fix it generically by supporting the
      __GFP_NOWARN flag that users can then use with calling the
      __alloc_percpu_gfp() helper instead.
      Signed-off-by: 's avatarDaniel Borkmann <daniel@iogearbox.net>
      Cc: Tejun Heo <tj@kernel.org>
      Cc: Mark Rutland <mark.rutland@arm.com>
      Acked-by: 's avatarAlexei Starovoitov <ast@kernel.org>
      Signed-off-by: 's avatarDavid S. Miller <davem@davemloft.net>
      0ea7eeec
  7. 04 Oct, 2017 1 commit
  8. 28 Sep, 2017 1 commit
    • Dennis Zhou's avatar
      percpu: fix iteration to prevent skipping over block · 1fa4df3e
      Dennis Zhou authored
      The iterator functions pcpu_next_md_free_region and
      pcpu_next_fit_region use the block offset to determine if they have
      checked the area in the prior iteration. However, this causes an issue
      when the block offset is greater than subsequent block contig hints. If
      within the iterator it moves to check subsequent blocks, it may fail in
      the second predicate due to the block offset not being cleared. Thus,
      this causes the allocator to skip over blocks leading to false failures
      when allocating from the reserved chunk. While this happens in the
      general case as well, it will only fail if it cannot allocate a new
      chunk.
      
      This patch resets the block offset to 0 to pass the second predicate
      when checking subseqent blocks within the iterator function.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reported-and-tested-by: 's avatarLuis Henriques <lhenriques@suse.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      1fa4df3e
  9. 26 Jul, 2017 23 commits
    • Dennis Zhou (Facebook)'s avatar
      percpu: update header to contain bitmap allocator explanation. · 5e81ee3e
      Dennis Zhou (Facebook) authored
      The other patches contain a lot of information, so adding this
      information in a separate patch. It adds my copyright and a brief
      explanation of how the bitmap allocator works. There is a minor typo as
      well in the prior explanation so that is fixed.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      5e81ee3e
    • Dennis Zhou (Facebook)'s avatar
      percpu: update pcpu_find_block_fit to use an iterator · b4c2116c
      Dennis Zhou (Facebook) authored
      The simple, and expensive, way to find a free area is to iterate over
      the entire bitmap until an area is found that fits the allocation size
      and alignment. This patch makes use of an iterate that find an area to
      check by using the block level contig hints. It will only return an area
      that can fit the size and alignment request. If the request can fit
      inside a block, it returns the first_free bit to start checking from to
      see if it can be fulfilled prior to the contig hint. The pcpu_alloc_area
      check has a bound of a block size added in case it is wrong.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      b4c2116c
    • Dennis Zhou (Facebook)'s avatar
      percpu: use metadata blocks to update the chunk contig hint · 525ca84d
      Dennis Zhou (Facebook) authored
      The largest free region will either be a block level contig hint or an
      aggregate over the left_free and right_free areas of blocks. This is a
      much smaller set of free areas that need to be checked than a full
      traverse.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      525ca84d
    • Dennis Zhou (Facebook)'s avatar
      percpu: update free path to take advantage of contig hints · b185cd0d
      Dennis Zhou (Facebook) authored
      The bitmap allocator must keep metadata consistent. The easiest way is
      to scan after every allocation for each affected block and the entire
      chunk. This is rather expensive.
      
      The free path can take advantage of current contig hints to prevent
      scanning within the start and end block.  If a scan is needed, it can
      be done by scanning backwards from the start and forwards from the end
      to identify the entire free area this can be combined with. The blocks
      can then be updated by some basic checks rather than complete block
      scans.
      
      A chunk scan happens when the freed area makes a page free, a block
      free, or spans across blocks. This is necessary as the contig hint at
      this point could span across blocks. The check uses the minimum of page
      size and the block size to allow for variable sized blocks. There is a
      tradeoff here with not updating after every free. It is possible a
      contig hint in one block can be merged with the contig hint in the next
      block. This means the contig hint can be off by up to a page. However,
      if the chunk's contig hint is contained in one block, the contig hint
      will be accurate.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      b185cd0d
    • Dennis Zhou (Facebook)'s avatar
      percpu: update alloc path to only scan if contig hints are broken · fc304334
      Dennis Zhou (Facebook) authored
      Metadata is kept per block to keep track of where the contig hints are.
      Scanning can be avoided when the contig hints are not broken. In that
      case, left and right contigs have to be managed manually.
      
      This patch changes the allocation path hint updating to only scan when
      contig hints are broken.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      fc304334
    • Dennis Zhou (Facebook)'s avatar
      percpu: keep track of the best offset for contig hints · 268625a6
      Dennis Zhou (Facebook) authored
      This patch makes the contig hint starting offset optimization from the
      previous patch as honest as it can be. For both chunk and block starting
      offsets, make sure it keeps the starting offset with the best alignment.
      
      The block skip optimization is added in a later patch when the
      pcpu_find_block_fit iterator is swapped in.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      268625a6
    • Dennis Zhou (Facebook)'s avatar
      percpu: skip chunks if the alloc does not fit in the contig hint · 13f96637
      Dennis Zhou (Facebook) authored
      This patch adds chunk->contig_bits_start to keep track of the contig
      hint's offset and the check to skip the chunk if it does not fit. If
      the chunk's contig hint starting offset cannot satisfy an allocation,
      the allocator assumes there is enough memory pressure in this chunk to
      either use a different chunk or create a new one. This accepts a less
      tight packing for a smoother latency curve.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      13f96637
    • Dennis Zhou (Facebook)'s avatar
      percpu: add first_bit to keep track of the first free in the bitmap · 86b442fb
      Dennis Zhou (Facebook) authored
      This patch adds first_bit to keep track of the first free bit in the
      bitmap. This hint helps prevent scanning of fully allocated blocks.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      86b442fb
    • Dennis Zhou (Facebook)'s avatar
      percpu: introduce bitmap metadata blocks · ca460b3c
      Dennis Zhou (Facebook) authored
      This patch introduces the bitmap metadata blocks and adds the skeleton
      of the code that will be used to maintain these blocks.  Each chunk's
      bitmap is made up of full metadata blocks. These blocks maintain basic
      metadata to help prevent scanning unnecssarily to update hints. Full
      scanning methods are used for the skeleton and will be replaced in the
      coming patches. A number of helper functions are added as well to do
      conversion of pages to blocks and manage offsets. Comments will be
      updated as the final version of each function is added.
      
      There exists a relationship between PAGE_SIZE, PCPU_BITMAP_BLOCK_SIZE,
      the region size, and unit_size. Every chunk's region (including offsets)
      is page aligned at the beginning to preserve alignment. The end is
      aligned to LCM(PAGE_SIZE, PCPU_BITMAP_BLOCK_SIZE) to ensure that the end
      can fit with the populated page map which is by page and every metadata
      block is fully accounted for. The unit_size is already page aligned, but
      must also be aligned with PCPU_BITMAP_BLOCK_SIZE to ensure full metadata
      blocks.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      ca460b3c
    • Dennis Zhou (Facebook)'s avatar
      percpu: replace area map allocator with bitmap · 40064aec
      Dennis Zhou (Facebook) authored
      The percpu memory allocator is experiencing scalability issues when
      allocating and freeing large numbers of counters as in BPF.
      Additionally, there is a corner case where iteration is triggered over
      all chunks if the contig_hint is the right size, but wrong alignment.
      
      This patch replaces the area map allocator with a basic bitmap allocator
      implementation. Each subsequent patch will introduce new features and
      replace full scanning functions with faster non-scanning options when
      possible.
      
      Implementation:
      This patchset removes the area map allocator in favor of a bitmap
      allocator backed by metadata blocks. The primary goal is to provide
      consistency in performance and memory footprint with a focus on small
      allocations (< 64 bytes). The bitmap removes the heavy memmove from the
      freeing critical path and provides a consistent memory footprint. The
      metadata blocks provide a bound on the amount of scanning required by
      maintaining a set of hints.
      
      In an effort to make freeing fast, the metadata is updated on the free
      path if the new free area makes a page free, a block free, or spans
      across blocks. This causes the chunk's contig hint to potentially be
      smaller than what it could allocate by up to the smaller of a page or a
      block. If the chunk's contig hint is contained within a block, a check
      occurs and the hint is kept accurate. Metadata is always kept accurate
      on allocation, so there will not be a situation where a chunk has a
      later contig hint than available.
      
      Evaluation:
      I have primarily done testing against a simple workload of allocation of
      1 million objects (2^20) of varying size. Deallocation was done by in
      order, alternating, and in reverse. These numbers were collected after
      rebasing ontop of a80099a1. I present the worst-case numbers here:
      
        Area Map Allocator:
      
              Object Size | Alloc Time (ms) | Free Time (ms)
              ----------------------------------------------
                    4B    |        310      |     4770
                   16B    |        557      |     1325
                   64B    |        436      |      273
                  256B    |        776      |      131
                 1024B    |       3280      |      122
      
        Bitmap Allocator:
      
              Object Size | Alloc Time (ms) | Free Time (ms)
              ----------------------------------------------
                    4B    |        490      |       70
                   16B    |        515      |       75
                   64B    |        610      |       80
                  256B    |        950      |      100
                 1024B    |       3520      |      200
      
      This data demonstrates the inability for the area map allocator to
      handle less than ideal situations. In the best case of reverse
      deallocation, the area map allocator was able to perform within range
      of the bitmap allocator. In the worst case situation, freeing took
      nearly 5 seconds for 1 million 4-byte objects. The bitmap allocator
      dramatically improves the consistency of the free path. The small
      allocations performed nearly identical regardless of the freeing
      pattern.
      
      While it does add to the allocation latency, the allocation scenario
      here is optimal for the area map allocator. The area map allocator runs
      into trouble when it is allocating in chunks where the latter half is
      full. It is difficult to replicate this, so I present a variant where
      the pages are second half filled. Freeing was done sequentially. Below
      are the numbers for this scenario:
      
        Area Map Allocator:
      
              Object Size | Alloc Time (ms) | Free Time (ms)
              ----------------------------------------------
                    4B    |       4118      |     4892
                   16B    |       1651      |     1163
                   64B    |        598      |      285
                  256B    |        771      |      158
                 1024B    |       3034      |      160
      
        Bitmap Allocator:
      
              Object Size | Alloc Time (ms) | Free Time (ms)
              ----------------------------------------------
                    4B    |        481      |       67
                   16B    |        506      |       69
                   64B    |        636      |       75
                  256B    |        892      |       90
                 1024B    |       3262      |      147
      
      The data shows a parabolic curve of performance for the area map
      allocator. This is due to the memmove operation being the dominant cost
      with the lower object sizes as more objects are packed in a chunk and at
      higher object sizes, the traversal of the chunk slots is the dominating
      cost. The bitmap allocator suffers this problem as well. The above data
      shows the inability to scale for the allocation path with the area map
      allocator and that the bitmap allocator demonstrates consistent
      performance in general.
      
      The second problem of additional scanning can result in the area map
      allocator completing in 52 minutes when trying to allocate 1 million
      4-byte objects with 8-byte alignment. The same workload takes
      approximately 16 seconds to complete for the bitmap allocator.
      
      V2:
      Fixed a bug in pcpu_alloc_first_chunk end_offset was setting the bitmap
      using bytes instead of bits.
      
      Added a comment to pcpu_cnt_pop_pages to explain bitmap_weight.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      40064aec
    • Dennis Zhou (Facebook)'s avatar
      percpu: generalize bitmap (un)populated iterators · 91e914c5
      Dennis Zhou (Facebook) authored
      The area map allocator only used a bitmap for the backing page state.
      The new bitmap allocator will use bitmaps to manage the allocation
      region in addition to this.
      
      This patch generalizes the bitmap iterators so they can be reused with
      the bitmap allocator.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      91e914c5
    • Dennis Zhou (Facebook)'s avatar
      percpu: increase minimum percpu allocation size and align first regions · d2f3c384
      Dennis Zhou (Facebook) authored
      This patch increases the minimum allocation size of percpu memory to
      4-bytes. This change will help minimize the metadata overhead
      associated with the bitmap allocator. The assumption is that most
      allocations will be of objects or structs greater than 2 bytes with
      integers or longs being used rather than shorts.
      
      The first chunk regions are now aligned with the minimum allocation
      size. The reserved region is expected to be set as a multiple of the
      minimum allocation size. The static region is aligned up and the delta
      is removed from the dynamic size. This works because the dynamic size is
      increased to be page aligned. If the static size is not minimum
      allocation size aligned, then there must be a gap that is added to the
      dynamic size. The dynamic size will never be smaller than the set value.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      d2f3c384
    • Dennis Zhou (Facebook)'s avatar
      percpu: introduce nr_empty_pop_pages to help empty page accounting · 0cecf50c
      Dennis Zhou (Facebook) authored
      pcpu_nr_empty_pop_pages is used to ensure there are a handful of free
      pages around to serve atomic allocations. A new field, nr_empty_pop_pages,
      is added to the pcpu_chunk struct to keep track of the number of empty
      pages. This field is needed as the number of empty populated pages is
      globally tracked and deltas are used to update in the bitmap allocator.
      Pages that contain a hidden area are not considered to be empty. This
      new field is exposed in percpu_stats.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      0cecf50c
    • Dennis Zhou (Facebook)'s avatar
      percpu: change the number of pages marked in the first_chunk pop bitmap · 8ab16c43
      Dennis Zhou (Facebook) authored
      The populated bitmap represents the state of the pages the chunk serves.
      Prior, the bitmap was marked completely used as the first chunk was
      allocated and immutable. This is misleading because the first chunk may
      not be completely filled. Additionally, with moving the base_addr up in
      the previous patch, the population check no longer corresponds to what
      was being checked.
      
      This patch modifies the population map to be only the number of pages
      the region serves and to make what it was checking correspond correctly
      again. The change is to remove any misunderstanding between the size of
      the populated bitmap and the actual size of it. The work function page
      iterators now use nr_pages for the check rather than pcpu_unit_pages
      because nr_populated is now chunk specific. Without this, the work
      function would try to populate the remainder of these chunks despite it
      not serving any more than nr_pages when nr_pages is set less than
      pcpu_unit_pages.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      8ab16c43
    • Dennis Zhou (Facebook)'s avatar
      percpu: combine percpu address checks · 560f2c23
      Dennis Zhou (Facebook) authored
      The percpu address checks for the reserved and dynamic region chunks are
      now specific to each region. The address checking logic can be combined
      taking advantage of the global references to the dynamic and static
      region chunks.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      560f2c23
    • Dennis Zhou (Facebook)'s avatar
      percpu: modify base_addr to be region specific · c0ebfdc3
      Dennis Zhou (Facebook) authored
      Originally, the first chunk was served by one or two chunks, each
      given a region they are responsible for. Despite this, the arithmetic
      was based off of the true base_addr of the chunk making it be overly
      inclusive.
      
      This patch moves the base_addr of chunks that are responsible for the
      first chunk. The base_addr must remain page aligned to keep the
      address alignment correct, so it is the beginning of the region served
      page aligned down. start_offset holds where the region served begins
      from this new base_addr.
      
      The corresponding percpu address checks are modified to be more specific
      as a result. The first chunk considers only the dynamic region and both
      first chunk and reserved chunk checks ignore the static region. The
      static region addresses should never be passed into the allocator. There
      is no impact here besides distinguishing the first chunk and making the
      checks specific.
      
      The percpu pointer to physical address is left intact as addresses are
      not given out in the non-allocated portion of percpu memory.
      
      nr_pages is added to pcpu_chunk to keep track of the size of the entire
      region served containing both start_offset and end_offset. This variable
      will be used to manage the bitmap allocator.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      c0ebfdc3
    • Dennis Zhou (Facebook)'s avatar
      percpu: setup_first_chunk rename schunk/dchunk to chunk · 0c4169c3
      Dennis Zhou (Facebook) authored
      There is no need to have the static chunk and dynamic chunk be named
      separately as the allocations are sequential. This preemptively solves
      the misnomer problem with the base_addrs being moved up in the following
      patch. It also removes a ternary operation deciding the first chunk.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      0c4169c3
    • Dennis Zhou (Facebook)'s avatar
      percpu: end chunk area maps page aligned for the populated bitmap · 6b9d7c8e
      Dennis Zhou (Facebook) authored
      The area map allocator manages the first chunk area by hiding all but
      the region it is responsible for serving in the area map. To align this
      with the populated page bitmap, end_offset is introduced to keep track
      of the delta to end page aligned. The area map is appended with the
      page aligned end when necessary to be in line with how the bitmap
      allocator requires the ending to be aligned with the LCM of PAGE_SIZE
      and the size of each bitmap block. percpu_stats is updated to ignore
      this region when present.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      6b9d7c8e
    • Dennis Zhou (Facebook)'s avatar
      percpu: unify allocation of schunk and dchunk · 10edf5b0
      Dennis Zhou (Facebook) authored
      Create a common allocator for first chunk initialization,
      pcpu_alloc_first_chunk. Comments for this function will be added in a
      later patch once the bitmap allocator is added.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      10edf5b0
    • Dennis Zhou (Facebook)'s avatar
      percpu: setup_first_chunk remove dyn_size and consolidate logic · b9c39442
      Dennis Zhou (Facebook) authored
      There is logic for setting variables in the static chunk init code that
      could be consolidated with the dynamic chunk init code. This combines
      this logic to setup for combining the allocation paths. reserved_size is
      used as the conditional as a dynamic region will always exist.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      b9c39442
    • Dennis Zhou (Facebook)'s avatar
      percpu: remove has_reserved from pcpu_chunk · 4af1e6fb
      Dennis Zhou (Facebook) authored
      Prior this variable was used to manage statistics when the first chunk
      had a reserved region. The previous patch introduced start_offset to
      keep track of the offset by value rather than boolean. Therefore,
      has_reserved can be removed.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      4af1e6fb
    • Dennis Zhou (Facebook)'s avatar
      percpu: introduce start_offset to pcpu_chunk · e2266705
      Dennis Zhou (Facebook) authored
      The reserved chunk arithmetic uses a global variable
      pcpu_reserved_chunk_limit that is set in the first chunk init code to
      hide a portion of the area map. The bitmap allocator to come will
      eventually move the base_addr up and require both the reserved chunk
      and static chunk to maintain this offset. pcpu_reserved_chunk_limit is
      removed and start_offset is added.
      
      The first chunk that is circulated and is pcpu_first_chunk serves the
      dynamic region, the region following the reserved region. The reserved
      chunk address check will temporarily use the first chunk to identify its
      address range. A following patch will increase the base_addr and remove
      this. If there is no reserved chunk, this will check the static region
      and return false because those values should never be passed into the
      allocator.
      
      Lastly, when linking in the first chunk, make sure to count the right
      free region for the number of empty populated pages.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      e2266705
    • Dennis Zhou (Facebook)'s avatar
      percpu: setup_first_chunk enforce dynamic region must exist · fb29a2cc
      Dennis Zhou (Facebook) authored
      The first chunk is handled as a special case as it is composed of the
      static, reserved, and dynamic regions. The code handles each case
      individually. The next several patches will merge these code paths and
      lay the foundation for the bitmap allocator.
      
      This patch modifies logic to enforce that a dynamic region exists and
      changes the area map to account for that. This brings the logic closer
      to the dynamic chunk's init logic.
      Signed-off-by: 's avatarDennis Zhou <dennisszhou@gmail.com>
      Reviewed-by: 's avatarJosef Bacik <jbacik@fb.com>
      Signed-off-by: 's avatarTejun Heo <tj@kernel.org>
      fb29a2cc
  10. 17 Jul, 2017 2 commits
  11. 21 Jun, 2017 1 commit
  12. 20 Jun, 2017 3 commits