rpi3.rst 15.9 KB
Newer Older
1 2
Trusted Firmware-A for Raspberry Pi 3
=====================================
3 4 5 6 7 8 9

.. section-numbering::
    :suffix: .

.. contents::

The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four
10 11
Arm Cortex-A53 cores, which makes it possible to have a port of Trusted
Firmware-A (TF-A).
12

13 14 15 16 17 18 19
The following instructions explain how to use this port of the TF-A with the
default distribution of `Raspbian`_ because that's the distribution officially
supported by the Raspberry Pi Foundation. At the moment of writing this, the
officially supported kernel is a AArch32 kernel. This doesn't mean that this
port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by
the Foundation can be compiled for AArch64 by following the steps in
`AArch64 kernel build instructions`_.
20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

**IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM,
which is available from both the Non-secure and Secure worlds. This port
shouldn't be considered more than a prototype to play with and implement
elements like PSCI to support the Linux kernel.

Design
------

The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a
VideoCore IV that acts as primary processor (and loads everything from the SD
card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi
3 documentation`_ for more information.

This explains why it is possible to change the execution state (AArch64/AArch32)
depending on a few files on the SD card. We only care about the cases in which
the cores boot in AArch64 mode.

The rules are simple:

- If a file called ``kernel8.img`` is located on the ``boot`` partition of the
  SD card, it will load it and execute in EL2 in AArch64. Basically, it executes
  a `default AArch64 stub`_ at address **0x0** that jumps to the kernel.

- If there is also a file called ``armstub8.bin``, it will load it at address
  **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All
  the cores are powered on at the same time and start at address **0x0**.

This means that we can use the default AArch32 kernel provided in the official
49 50 51 52
`Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and
anything else we need is in ``armstub8.bin``. This way we can forget about the
default bootstrap code. When using a AArch64 kernel, it is only needed to make
sure that the name on the SD card is ``kernel8.img``.
53 54 55 56

Ideally, we want to load the kernel and have all cores available, which means
that we need to make the secondary cores work in the way the kernel expects, as
explained in `Secondary cores`_. In practice, a small bootstrap is needed
57
between TF-A and the kernel.
58 59 60 61 62 63 64 65 66 67 68

To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode
in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The
architecture specifies that AArch32 Hypervisor mode isn't present when AArch64
is used for EL2. When using a AArch64 kernel, it should simply start in EL2.

Placement of images
~~~~~~~~~~~~~~~~~~~

The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding
between them so that the addresses they are loaded to match the ones specified
69
when compiling TF-A.
70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89

The device tree block is loaded by the VideoCore loader from an appropriate
file, but we can specify the address it is loaded to in ``config.txt``.

The file ``kernel8.img`` contains a kernel image that is loaded to the address
specified in ``config.txt``. The `Linux kernel tree`_ has information about how
a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``:

::

    The zImage may also be placed in system RAM and called there.  The
    kernel should be placed in the first 128MiB of RAM.  It is recommended
    that it is loaded above 32MiB in order to avoid the need to relocate
    prior to decompression, which will make the boot process slightly
    faster.

There are no similar restrictions for AArch64 kernels, as specified in the file
``Documentation/arm64/booting.txt``.

This means that we need to avoid the first 128 MiB of RAM when placing the
90 91
TF-A images (and specially the first 32 MiB, as they are directly used to
place the uncompressed AArch32 kernel image. This way, both AArch32 and
92 93 94 95 96 97 98 99 100 101 102 103
AArch64 kernels can be placed at the same address.

In the end, the images look like the following diagram when placed in memory.
All addresses are Physical Addresses from the point of view of the Arm cores.
Again, note that this is all just part of the same DRAM that goes from
**0x00000000** to **0x3F000000**, it just has different names to simulate a real
secure platform!

::

    0x00000000 +-----------------+
               |       ROM       | BL1
104
    0x00020000 +-----------------+
105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124
               |       FIP       |
    0x00200000 +-----------------+
               |                 |
               |       ...       |
               |                 |
    0x01000000 +-----------------+
               |     Kernel      |
               +-----------------+
               |                 |
               |       ...       |
               |                 |
    0x02000000 +-----------------+
               |       DTB       |
               +-----------------+
               |                 |
               |       ...       |
               |                 |
    0x10000000 +-----------------+
               |   Secure SRAM   | BL2, BL31
    0x10100000 +-----------------+
125
               |   Secure DRAM   | BL32 (Secure payload)
126
    0x10C00000 +-----------------+
127 128 129 130 131 132 133 134 135 136
               | Non-secure DRAM | BL33
    0x11000000 +-----------------+
               |                 |
               |       ...       |
               |                 |
    0x3F000000 +-----------------+
               |       I/O       |
    0x40000000 +-----------------+

The area between **0x10000000** and **0x11000000** has to be protected so that
137
the kernel doesn't use it. That is done by adding ``memmap=16M$256M`` to the
138 139 140 141 142 143 144 145
command line passed to the kernel. See the `Setup SD card`_ instructions to see
how to do it.

The last 16 MiB of DRAM can only be accessed by the VideoCore, that has
different mappings than the Arm cores in which the I/O addresses don't overlap
the DRAM. The memory reserved to be used by the VideoCore is always placed at
the end of the DRAM, so this space isn't wasted.

146 147
Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for
TF-A, there are 880 MiB available for Linux.
148 149 150 151

Boot sequence
~~~~~~~~~~~~~

152 153 154 155 156
The boot sequence of TF-A is the usual one except when booting an AArch32
kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it
can jump to the kernel in the same mode and let it take over that privilege
level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of
TF-A) it could only jump to the kernel in AArch32 in Supervisor mode.
157 158 159 160 161 162 163 164 165 166 167 168 169

The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel
in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The
bootstrap should take care of this.

Secondary cores
~~~~~~~~~~~~~~~

The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to
use mailboxes to trap the secondary cores until they are ready to jump to the
kernel. This mailbox is located at a different address in the AArch32 default
kernel than in the AArch64 kernel.

170 171 172
Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During
cold boot, all secondary cores wait in a loop until they are given given an
address to jump to in this Mailbox (``bl31_warm_entrypoint``).
173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189

Once BL31 has finished and the primary core has jumped to the BL33 payload, it
has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop.
The payload then makes them wait in another waitloop listening from messages
from the kernel. When the primary CPU jumps into the kernel, it will send an
address to the mailbox so that the secondary CPUs jump to it and are recognised
by the kernel.

Build Instructions
------------------

To boot a AArch64 kernel, only the AArch64 toolchain is required.

To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The
AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit
kernel.

190 191
First, clone and compile `Raspberry Pi 3 TF-A bootstrap`_. Choose the one
needed for the architecture of your kernel.
192

193
Then compile TF-A. For a AArch32 kernel, use the following command line:
194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215

.. code:: shell

    CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3             \
    RPI3_BL33_IN_AARCH32=1                                      \
    BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin     \
    all fip

For a AArch64 kernel, use this other command line:

.. code:: shell

    CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3             \
    BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin     \
    all fip

Then, join BL1 and the FIP with the following instructions (replace ``release``
by ``debug`` if you set the build option ``DEBUG=1``):

.. code:: shell

    cp build/rpi3/release/bl1.bin bl1.pad.bin
216
    truncate --size=131072 bl1.pad.bin
217 218 219
    cat bl1.pad.bin build/rpi3/release/fip.bin > armstub8.bin

The resulting file, ``armstub8.bin``, contains BL1 and the FIP in the place they
220 221
need to be for TF-A to boot correctly. Now, follow the instructions in
`Setup SD card`_.
222 223 224 225 226 227 228 229 230 231 232 233 234 235

The following build options are supported:

- ``PRELOADED_BL33_BASE``: Specially useful because the file ``kernel8.img`` can
  be loaded anywhere by modifying the file ``config.txt``. It doesn't have to
  contain a kernel, it could have any arbitrary payload.

- ``RESET_TO_BL31``: Set to 1 by default. If using a 32-bit kernel like
  `Raspbian`_, the space used by BL1 can overwritten by the kernel when it is
  being loaded. Even when using a AArch64 kernel the region used by
  BL1 isn't protected and the kernel could overwrite it. The space used by BL31
  is reserved by the command line passed to the kernel.

- ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image.
236 237 238
  By default this option is 0, which means that TF-A will jump to BL33 in EL2
  in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32
  mode.
239

240 241 242 243 244 245
- ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional.
  Please use the code from `here <https://github.com/OP-TEE/optee_os>`__.
  Build the Trusted Firmware with option ``BL32=tee-header_v2.bin
  BL32_EXTRA1=tee-pager_v2.bin  BL32_EXTRA2=tee-pageable_v2.bin``
  to put the binaries into the FIP.

246 247 248 249 250 251 252 253 254 255
- ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option
  ``TRUSTED_BOARD_BOOT=1`` to enable it. In order to use TBB, you might
  want to set ``GENERATE_COT=1`` to let the contents of the FIP automatically
  signed by the build process. The ROT key will be generated and output to
  ``rot_key.pem`` in the build directory. It is able to set ROT_KEY to
  your own key in PEM format.
  Also in order to build, you need to clone mbedtls from
  `here <https://github.com/ARMmbed/mbedtls>`__.
  And set MBEDTLS_DIR to mbedtls source directory.

256 257
The following is not currently supported:

258
- AArch32 for TF-A itself.
259 260 261

- ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any
  address by changing the file ``armstub8.bin``, so there's no point in using
262
  TF-A in this case.
263 264 265

- ``LOAD_IMAGE_V2=0``: Only version 2 is supported.

266 267 268 269 270
- ``MULTI_CONSOLE_API=0``: The multi console API must be enabled. Note that the
  crash console uses the internal 16550 driver functions directly in order to be
  able to print error messages during early crashes before setting up the
  multi console API.

271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323
AArch64 kernel build instructions
---------------------------------

The following instructions show how to install and run a AArch64 kernel by
using a SD card with the default `Raspbian`_ install as base. Skip them if you
want to use the default 32-bit kernel.

Note that this system won't be fully 64-bit because all the tools in the
filesystem are 32-bit binaries, but it's a quick way to get it working, and it
allows the user to run 64-bit binaries in addition to 32-bit binaries.

1. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To
   speed things up, do a shallow clone of the desired branch.

.. code:: shell

    git clone --depth=1 -b rpi-4.14.y https://github.com/raspberrypi/linux
    cd linux

2. Configure and compile the kernel. Adapt the number after ``-j`` so that it is
   1.5 times the number of CPUs in your computer. This may take some time to
   finish.

.. code:: shell

    make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig
    make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-

3. Copy the kernel image and the device tree to the SD card. Replace the path
   by the corresponding path in your computers to the ``boot`` partition of the
   SD card.

.. code:: shell

    cp arch/arm64/boot/Image /path/to/boot/kernel8.img
    cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/

4. Install the kernel modules. Replace the path by the corresponding path to the
   filesystem partition of the SD card on your computer.

.. code:: shell

    make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
    INSTALL_MOD_PATH=/path/to/filesystem modules_install

5. Follow the instructions in `Setup SD card`_ except for the step of renaming
   the existing ``kernel7.img`` (we have already copied a AArch64 kernel).

Setup SD card
-------------

The instructions assume that you have an SD card with a fresh install of
`Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly
324
untouched). They have been tested with the image available in 2018-03-13.
325 326 327 328

1. Insert the SD card and open the ``boot`` partition.

2. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore
329 330
   bootloader into booting the Arm cores in AArch64 mode, like TF-A needs,
   even though the kernel is not compiled for AArch64.
331 332 333 334 335

3. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore
   bootloader will look for a file called ``armstub8.bin`` and load it at
   address **0x0** instead of a predefined one.

336
4. Open ``cmdline.txt`` and add ``memmap=16M$256M`` to prevent the kernel from
337 338
   using the memory needed by TF-A. If you want to enable the serial port
   "Mini UART", make sure that this file also contains
339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380
   ``console=serial0,115200 console=tty1``.

   Note that the 16 MiB reserved this way won't be available for Linux, the same
   way as the memory reserved in DRAM for the GPU isn't available.

5. Open ``config.txt`` and add the following lines at the end (``enable_uart=1``
   is only needed to enable debugging through the Mini UART):

::

    enable_uart=1
    kernel_address=0x01000000
    device_tree_address=0x02000000

If you connect a serial cable to the Mini UART and your computer, and connect
to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some
text. In the case of an AArch32 kernel, you should see something like this:

::

    NOTICE:  Booting Trusted Firmware
    NOTICE:  BL1: v1.4(release):v1.4-329-g61e94684-dirty
    NOTICE:  BL1: Built : 00:09:25, Nov  6 2017
    NOTICE:  BL1: Booting BL2
    NOTICE:  BL2: v1.4(release):v1.4-329-g61e94684-dirty
    NOTICE:  BL2: Built : 00:09:25, Nov  6 2017
    NOTICE:  BL1: Booting BL31
    NOTICE:  BL31: v1.4(release):v1.4-329-g61e94684-dirty
    NOTICE:  BL31: Built : 00:09:25, Nov  6 2017
    [    0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517

    Raspbian GNU/Linux 9 raspberrypi ttyS0
    raspberrypi login:

Just enter your credentials, everything should work as expected. Note that the
HDMI output won't show any text during boot.

.. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S
.. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S
.. _Linux kernel tree: https://github.com/torvalds/linux
.. _Linux tree fork: https://github.com/raspberrypi/linux
.. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/
381
.. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap
382 383
.. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/
.. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/