Skip to content
  • David Gibson's avatar
    spapr: Improved placement of PCI host bridges in guest memory map · 357d1e3b
    David Gibson authored
    
    
    Currently, the MMIO space for accessing PCI on pseries guests begins at
    1 TiB in guest address space.  Each PCI host bridge (PHB) has a 64 GiB
    chunk of address space in which it places its outbound PIO and 32-bit and
    64-bit MMIO windows.
    
    This scheme as several problems:
      - It limits guest RAM to 1 TiB (though we have a limited fix for this
        now)
      - It limits the total MMIO window to 64 GiB.  This is not always enough
        for some of the large nVidia GPGPU cards
      - Putting all the windows into a single 64 GiB area means that naturally
        aligning things within there will waste more address space.
    In addition there was a miscalculation in some of the defaults, which meant
    that the MMIO windows for each PHB actually slightly overran the 64 GiB
    region for that PHB.  We got away without nasty consequences because
    the overrun fit within an unused area at the beginning of the next PHB's
    region, but it's not pretty.
    
    This patch implements a new scheme which addresses those problems, and is
    also closer to what bare metal hardware and pHyp guests generally use.
    
    Because some guest versions (including most current distro kernels) can't
    access PCI MMIO above 64 TiB, we put all the PCI windows between 32 TiB and
    64 TiB.  This is broken into 1 TiB chunks.  The first 1 TiB contains the
    PIO (64 kiB) and 32-bit MMIO (2 GiB) windows for all of the PHBs.  Each
    subsequent TiB chunk contains a naturally aligned 64-bit MMIO window for
    one PHB each.
    
    This reduces the number of allowed PHBs (without full manual configuration
    of all the windows) from 256 to 31, but this should still be plenty in
    practice.
    
    We also change some of the default window sizes for manually configured
    PHBs to saner values.
    
    Finally we adjust some tests and libqos so that it correctly uses the new
    default locations.  Ideally it would parse the device tree given to the
    guest, but that's a more complex problem for another time.
    
    Signed-off-by: default avatarDavid Gibson <david@gibson.dropbear.id.au>
    Reviewed-by: default avatarLaurent Vivier <lvivier@redhat.com>
    357d1e3b