Commit 022c6747 authored by Alessandro Rubini's avatar Alessandro Rubini Committed by Greg Kroah-Hartman

FMC: add documentation for the core

This is selected sections of the current manual for fmc-bus, as
developed outside of the kernel before submission.

Like the other patches in this set, it corresponds to commit ab23167f of
the repository at ohwr.org
Signed-off-by: 's avatarAlessandro Rubini <rubini@gnudd.com>
Acked-by: 's avatarJuan David Gonzalez Cobas <dcobas@cern.ch>
Acked-by: 's avatarEmilio G. Cota <cota@braap.org>
Acked-by: 's avatarSamuel Iglesias Gonsalvez <siglesias@igalia.com>
Acked-by: 's avatarRob Landley <rob@landley.net>
Signed-off-by: 's avatarGreg Kroah-Hartman <gregkh@linuxfoundation.org>
parent 77864f2e
......@@ -187,6 +187,8 @@ firmware_class/
- request_firmware() hotplug interface info.
flexible-arrays.txt
- how to make use of flexible sized arrays in linux
fmc/
- information about the FMC bus abstraction
frv/
- Fujitsu FR-V Linux documentation.
futex-requeue-pi.txt
......
Documentation in this directory comes from sections of the manual we
wrote for the externally-developed fmc-bus package. The complete
manual as of today (2013-02) is available in PDF format at
http://www.ohwr.org/projects/fmc-bus/files
00-INDEX
- this file.
FMC-and-SDB.txt
- What are FMC and SDB, basic concepts for this framework
API.txt
- The functions that are exported by the bus driver
parameters.txt
- The module parameters
carrier.txt
- writing a carrier (a device)
mezzanine.txt
- writing code for your mezzanine (a driver)
identifiers.txt
- how identification and matching works
Functions Exported by fmc.ko
****************************
The FMC core exports the usual 4 functions that are needed for a bus to
work, and a few more:
int fmc_driver_register(struct fmc_driver *drv);
void fmc_driver_unregister(struct fmc_driver *drv);
int fmc_device_register(struct fmc_device *fmc);
void fmc_device_unregister(struct fmc_device *fmc);
int fmc_device_register_n(struct fmc_device **fmc, int n);
void fmc_device_unregister_n(struct fmc_device **fmc, int n);
uint32_t fmc_readl(struct fmc_device *fmc, int offset);
void fmc_writel(struct fmc_device *fmc, uint32_t val, int off);
void *fmc_get_drvdata(struct fmc_device *fmc);
void fmc_set_drvdata(struct fmc_device *fmc, void *data);
int fmc_reprogram(struct fmc_device *f, struct fmc_driver *d, char *gw,
int sdb_entry);
The data structure that describe a device is detailed in *note FMC
Device::, the one that describes a driver is detailed in *note FMC
Driver::. Please note that structures of type fmc_device must be
allocated by the caller, but must not be released after unregistering.
The fmc-bus itself takes care of releasing the structure when their use
count reaches zero - actually, the device model does that in lieu of us.
The functions to register and unregister n devices are meant to be used
by carriers that host more than one mezzanine. The devices must all be
registered at the same time because if the FPGA is reprogrammed, all
devices in the array are affected. Usually, the driver matching the
first device will reprogram the FPGA, so other devices must know they
are already driven by a reprogrammed FPGA.
If a carrier hosts slots that are driven by different FPGA devices, it
should register as a group only mezzanines that are driven by the same
FPGA, for the reason outlined above.
Finally, the fmc_reprogram function calls the reprogram method (see
*note The API Offered by Carriers:: and also scans the memory area for
an SDB tree. You can pass -1 as sdb_entry to disable such scan.
Otherwise, the function fails if no tree is found at the specified
entry point. The function is meant to factorize common code, and by
the time you read this it is already used by the spec-sw and fine-delay
modules.
FMC (FPGA Mezzanine Card) is the standard we use for our I/O devices,
in the context of White Rabbit and related hardware.
In our I/O environments we need to write drivers for each mezzanine
card, and such drivers must work regardless of the carrier being used.
To achieve this, we abstract the FMC interface.
We have a carrier for PCI-E called SPEC and one for VME called SVEC,
but more are planned. Also, we support stand-alone devices (usually
plugged on a SPEC card), controlled through Etherbone, developed by GSI.
Code and documentation for the FMC bus was born as part of the spec-sw
project, but now it lives in its own project. Other projects, i.e.
software support for the various carriers, should include this as a
submodule.
The most up to date version of code and documentation is always
available from the repository you can clone from:
git://ohwr.org/fmc-projects/fmc-bus.git (read-only)
git@ohwr.org:fmc-projects/fmc-bus.git (read-write for developers)
Selected versions of the documentation, as well as complete tar
archives for selected revisions are placed to the Files section of the
project: `http://www.ohwr.org/projects/fmc-bus/files'
What is FMC
***********
FMC, as said, stands for "FPGA Mezzanine Card". It is a standard
developed by the VME consortium called VITA (VMEbus International Trade
Association and ratified by ANSI, the American National Standard
Institute. The official documentation is called "ANSI-VITA 57.1".
The FMC card is an almost square PCB, around 70x75 millimeters, that is
called mezzanine in this document. It usually lives plugged into
another PCB for power supply and control; such bigger circuit board is
called carrier from now on, and a single carrier may host more than one
mezzanine.
In the typical application the mezzanine is mostly analog while the
carrier is mostly digital, and hosts an FPGA that must be configured to
match the specific mezzanine and the desired application. Thus, you may
need to load different FPGA images to drive different instances of the
same mezzanine.
FMC, as such, is not a bus in the usual meaning of the term, because
most carriers have only one connector, and carriers with several
connectors have completely separate electrical connections to them.
This package, however, implements a bus as a software abstraction.
What is SDB
***********
SDB (Self Describing Bus) is a set of data structures that we use for
enumerating the internal structure of an FPGA image. We also use it as
a filesystem inside the FMC EEPROM.
SDB is not mandatory for use of this FMC kernel bus, but if you have SDB
this package can make good use of it. SDB itself is developed in the
fpga-config-space OHWR project. The link to the repository is
`git://ohwr.org/hdl-core-lib/fpga-config-space.git' and what is used in
this project lives in the sdbfs subdirectory in there.
SDB support for FMC is described in *note FMC Identification:: and
*note SDB Support::
SDB Support
***********
The fmc.ko bus driver exports a few functions to help drivers taking
advantage of the SDB information that may be present in your own FPGA
memory image.
The module exports the following functions, in the special header
<linux/fmc-sdb.h>. The linux/ prefix in the name is there because we
plan to submit it upstream in the future, and don't want to force
changes on our drivers if that happens.
int fmc_scan_sdb_tree(struct fmc_device *fmc, unsigned long address);
void fmc_show_sdb_tree(struct fmc_device *fmc);
signed long fmc_find_sdb_device(struct sdb_array *tree, uint64_t vendor,
uint32_t device, unsigned long *sz);
int fmc_free_sdb_tree(struct fmc_device *fmc);
This diff is collapsed.
FMC Identification
******************
The FMC standard requires every compliant mezzanine to carry
identification information in an I2C EEPROM. The information must be
laid out according to the "IPMI Platform Management FRU Information",
where IPMI is a lie I'd better not expand, and FRU means "Field
Replaceable Unit".
The FRU information is an intricate unreadable binary blob that must
live at offset 0 of the EEPROM, and typically extends for a few hundred
bytes. The standard allows the application to use all the remaining
storage area of the EEPROM as it wants.
This chapter explains how to create your own EEPROM image and how to
write it in your mezzanine, as well as how devices and drivers are
paired at run time. EEPROM programming uses tools that are part of this
package and SDB (part of the fpga-config-space package).
The first sections are only interesting for manufacturers who need to
write the EEPROM. If you are just a software developer writing an FMC
device or driver, you may jump straight to *note SDB Support::.
Building the FRU Structure
==========================
If you want to know the internals of the FRU structure and despair, you
can retrieve the document from
`http://download.intel.com/design/servers/ipmi/FRU1011.pdf' . The
standard is awful and difficult without reason, so we only support the
minimum mandatory subset - we create a simple structure and parse it
back at run time, but we are not able to either generate or parse more
arcane features like non-english languages and 6-bit text. If you need
more items of the FRU standard for your boards, please submit patches.
This package includes the Python script that Matthieu Cattin wrote to
generate the FRU binary blob, based on an helper libipmi by Manohar
Vanga and Matthieu himself. I changed the test script to receive
parameters from the command line or from the environment (the command
line takes precedence)
To make a long story short, in order to build a standard-compliant
binary file to be burned in your EEPROM, you need the following items:
Environment Opt Official Name Default
---------------------------------------------------------------------
FRU_VENDOR -v "Board Manufacturer" fmc-example
FRU_NAME -n "Board Product Name" mezzanine
FRU_SERIAL -s `Board Serial Number" 0001
FRU_PART -p "Board Part Number" sample-part
FRU_OUTPUT -o not applicable /dev/stdout
The "Official Name" above is what you find in the FRU official
documentation, chapter 11, page 7 ("Board Info Area Format"). The
output option is used to save the generated binary to a specific file
name instead of stdout.
You can pass the items to the FRU generator either in the environment
or on the command line. This package has currently no support for
specifying power consumption or such stuff, but I plan to add it as
soon as I find some time for that.
FIXME: consumption etc for FRU are here or in PTS?
The following example creates a binary image for a specific board:
./tools/fru-generator -v CERN -n FmcAdc100m14b4cha \
-s HCCFFIA___-CR000003 -p EDA-02063-V5-0 > eeprom.bin
The following example shows a script that builds several binary EEPROM
images for a series of boards, changing the serial number for each of
them. The script uses a mix of environment variables and command line
options, and uses the same string patterns shown above.
#!/bin/sh
export FRU_VENDOR="CERN"
export FRU_NAME="FmcAdc100m14b4cha"
export FRU_PART="EDA-02063-V5-0"
serial="HCCFFIA___-CR"
for number in $(seq 1 50); do
# build number-string "ns"
ns="$(printf %06d $number)"
./fru-generator -s "${serial}${ns}" > eeprom-${ns}.bin
done
Using SDB-FS in the EEPROM
==========================
If you want to use SDB as a filesystem in the EEPROM device within the
mezzanine, you should create one such filesystem using gensdbfs, from
the fpga-config-space package on OHWR.
By using an SBD filesystem you can cluster several files in a single
EEPROM, so both the host system and a soft-core running in the FPGA (if
any) can access extra production-time information.
We chose to use SDB as a storage filesystem because the format is very
simple, and both the host system and the soft-core will likely already
include support code for such format. The SDB library offered by the
fpga-config-space is less than 1kB under LM32, so it proves quite up to
the task.
The SDB entry point (which acts as a directory listing) cannot live at
offset zero in the flash device, because the FRU information must live
there. To avoid wasting precious storage space while still allowing
for more-than-minimal FRU structures, the fmc.ko will look for the SDB
record at address 256, 512 and 1024.
In order to generate the complete EEPROM image you'll need a
configuration file for gensdbfs: you tell the program where to place
the sdb entry point, and you must force the FRU data file to be placed
at the beginning of the storage device. If needed, you can also place
other files at a special offset (we sometimes do it for backward
compatibility with drivers we wrote before implementing SDB for flash
memory).
The directory tools/sdbfs of this package includes a well-commented
example that you may want to use as a starting point (the comments are
in the file called -SDB-CONFIG-). Reading documentation for gensdbfs
is a suggested first step anyways.
This package (generic FMC bus support) only accesses two files in the
EEPROM: the FRU information, at offset zero, with a suggested filename
of IPMI-FRU and the short name for the mezzanine, in a file called
name. The IPMI-FRU name is not mandatory, but a strongly suggested
choice; the name filename is mandatory, because this is the preferred
short name used by the FMC core. For example, a name of "fdelay" may
supplement a Product Name like "FmcDelay1ns4cha" - exactly as
demonstrated in `tools/sdbfs'.
Note: SDB access to flash memory is not yet supported, so the short
name currently in use is just the "Product Name" FRU string.
The example in tools/sdbfs includes an extra file, that is needed by
the fine-delay driver, and must live at a known address of 0x1800. By
running gensdbfs on that directory you can output your binary EEPROM
image (here below spusa$ is the shell prompt):
spusa$ ../fru-generator -v CERN -n FmcDelay1ns4cha -s proto-0 \
-p EDA-02267-V3 > IPMI-FRU
spusa$ ls -l
total 16
-rw-rw-r-- 1 rubini staff 975 Nov 19 18:08 --SDB-CONFIG--
-rw-rw-r-- 1 rubini staff 216 Nov 19 18:13 IPMI-FRU
-rw-rw-r-- 1 rubini staff 11 Nov 19 18:04 fd-calib
-rw-rw-r-- 1 rubini staff 7 Nov 19 18:04 name
spusa$ sudo gensdbfs . /lib/firmware/fdelay-eeprom.bin
spusa$ sdb-read -l -e 0x100 /lib/firmware/fdelay-eeprom.bin
/home/rubini/wip/sdbfs/userspace/sdb-read: listing format is to be defined
46696c6544617461:2e202020 00000100-000018ff .
46696c6544617461:6e616d65 00000200-00000206 name
46696c6544617461:66642d63 00001800-000018ff fd-calib
46696c6544617461:49504d49 00000000-000000d7 IPMI-FRU
spusa$ ../fru-dump /lib/firmware/fdelay-eeprom.bin
/lib/firmware/fdelay-eeprom.bin: manufacturer: CERN
/lib/firmware/fdelay-eeprom.bin: product-name: FmcDelay1ns4cha
/lib/firmware/fdelay-eeprom.bin: serial-number: proto-0
/lib/firmware/fdelay-eeprom.bin: part-number: EDA-02267-V3
As expected, the output file is both a proper sdbfs object and an IPMI
FRU information blob. The fd-calib file lives at offset 0x1800 and is
over-allocated to 256 bytes, according to the configuration file for
gensdbfs.
FMC Driver
**********
An FMC driver is concerned with the specific mezzanine and associated
gateware. As such, it is expected to be independent of the carrier
being used: it will perform I/O accesses only by means of
carrier-provided functions.
The matching between device and driver is based on the content of the
EEPROM (as mandated by the FMC standard) or by the actual cores
configured in the FPGA; the latter technique is used when the FPGA is
already programmed when the device is registered to the bus core.
In some special cases it is possible for a driver to directly access
FPGA registers, by means of the `fpga_base' field of the device
structure. This may be needed for high-bandwidth peripherals like fast
ADC cards. If the device module registered a remote device (for example
by means of Etherbone), the `fpga_base' pointer will be NULL.
Therefore, drivers must be ready to deal with NULL base pointers, and
fail gracefully. Most driver, however, are not expected to access the
pointer directly but run fmc_readl and fmc_writel instead, which will
work in any case.
In even more special cases, the driver may access carrier-specific
functionality: the `carrier_name' string allows the driver to check
which is the current carrier and make use of the `carrier_data'
pointer. We chose to use carrier names rather than numeric identifiers
for greater flexibility, but also to avoid a central registry within
the `fmc.h' file - we hope other users will exploit our framework with
their own carriers. An example use of carrier names is in GPIO setup
(see *note The GPIO Abstraction::), although the name match is not
expected to be performed by the driver. If you depend on specific
carriers, please check the carrier name and fail gracefully if your
driver finds it is running in a yet-unknown-to-it environment.
ID Table
========
Like most other Linux drivers, and FMC driver must list all the devices
which it is able to drive. This is usually done by means of a device
table, but in FMC we can match hardware based either on the contents of
their EEPROM or on the actual FPGA cores that can be enumerated.
Therefore, we have two tables of identifiers.
Matching of FRU information depends on two names, the manufacturer (or
vendor) and the device (see *note FMC Identification::); for
flexibility during production (i.e. before writing to the EEPROM) the
bus supports a catch-all driver that specifies NULL strings. For this
reason, the table is specified as pointer-and-length, not a a
null-terminated array - the entry with NULL names can be a valid entry.
Matching on FPGA cores depends on two numeric fields: the 64-bit vendor
number and the 32-bit device number. Support for matching based on
class is not yet implemented. Each device is expected to be uniquely
identified by an array of cores (it matches if all of the cores are
instantiated), and for consistency the list is passed as
pointer-and-length. Several similar devices can be driven by the same
driver, and thus the driver specifies and array of such arrays.
The complete set of involved data structures is thus the following:
struct fmc_fru_id { char *manufacturer; char *product_name; };
struct fmc_sdb_one_id { uint64_t vendor; uint32_t device; };
struct fmc_sdb_id { struct fmc_sdb_one_id *cores; int cores_nr; };
struct fmc_device_id {
struct fmc_fru_id *fru_id; int fru_id_nr;
struct fmc_sdb_id *sdb_id; int sdb_id_nr;
};
A better reference, with full explanation, is the <linux/fmc.h> header.
Module Parameters
=================
Most of the FMC drivers need the same set of kernel parameters. This
package includes support to implement common parameters by means of
fields in the `fmc_driver' structure and simple macro definitions.
The parameters are carrier-specific, in that they rely on the busid
concept, that varies among carriers. For the SPEC, the identifier is a
PCI bus and devfn number, 16 bits wide in total; drivers for other
carriers will most likely offer something similar but not identical,
and some code duplication is unavoidable.
This is the list of parameters that are common to several modules to
see how they are actually used, please look at spec-trivial.c.
`busid='
This is an array of integers, listing carrier-specific
identification numbers. For PIC, for example, `0x0400' represents
bus 4, slot 0. If any such ID is specified, the driver will only
accept to drive cards that appear in the list (even if the FMC ID
matches). This is accomplished by the validate carrier method.
`gateware='
The argument is an array of strings. If no busid= is specified,
the first string of gateware= is used for all cards; otherwise the
identifiers and gateware names are paired one by one, in the order
specified.
`show_sdb='
For modules supporting it, this parameter asks to show the SDB
internal structure by means of kernel messages. It is disabled by
default because those lines tend to hide more important messages,
if you look at the system console while loading the drivers.
Note: the parameter is being obsoleted, because fmc.ko itself now
supports dump_sdb= that applies to every client driver.
For example, if you are using the trivial driver to load two different
gateware files to two different cards, you can use the following
parameters to load different binaries to the cards, after looking up
the PCI identifiers. This has been tested with a SPEC carrier.
insmod fmc-trivial.ko \
busid=0x0200,0x0400 \
gateware=fmc/fine-delay.bin,fmc/simple-dio.bin
Please note that not all sub-modules support all of those parameters.
You can use modinfo to check what is supported by each module.
Module Parameters in fmc.ko
***************************
The core driver receives two module parameters, meant to help debugging
client modules. Both parameters can be modified by writing to
/sys/module/fmc/parameters/, because they are used when client drivers
are devices are registered, not when fmc.ko is loaded.
`dump_eeprom='
If not zero, the parameter asks the bus controller to dump the
EEPROM of any device that is registered, using printk.
`dump_sdb='
If not zero, the parameter prints the SDB tree of every FPGA it is
loaded by fmc_reprogram(). If greater than one, it asks to dump
the binary content of SDB records. This currently only dumps the
top-level SDB array, though.
EEPROM dumping avoids repeating lines, since most of the contents is
usually empty and all bits are one or zero. This is an example of the
output:
[ 6625.850480] spec 0000:02:00.0: FPGA programming successful
[ 6626.139949] spec 0000:02:00.0: Manufacturer: CERN
[ 6626.144666] spec 0000:02:00.0: Product name: FmcDelay1ns4cha
[ 6626.150370] FMC: mezzanine 0: 0000:02:00.0 on SPEC
[ 6626.155179] FMC: dumping eeprom 0x2000 (8192) bytes
[ 6626.160087] 0000: 01 00 00 01 00 0b 00 f3 01 0a 00 a5 85 87 c4 43
[ 6626.167069] 0010: 45 52 4e cf 46 6d 63 44 65 6c 61 79 31 6e 73 34
[ 6626.174019] 0020: 63 68 61 c7 70 72 6f 74 6f 2d 30 cc 45 44 41 2d
[ 6626.180975] 0030: 30 32 32 36 37 2d 56 33 da 32 30 31 32 2d 31 31
[...]
[ 6626.371366] 0200: 66 64 65 6c 61 79 0a 00 00 00 00 00 00 00 00 00
[ 6626.378359] 0210: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 6626.385361] [...]
[ 6626.387308] 1800: 70 6c 61 63 65 68 6f 6c 64 65 72 ff ff ff ff ff
[ 6626.394259] 1810: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
[ 6626.401250] [...]
The dump of SDB looks like the following; the example shows the simple
golden gateware for the SPEC card, removing the leading timestamps to
fit the page:
spec 0000:02:00.0: SDB: 00000651:e6a542c9 WB4-Crossbar-GSI
spec 0000:02:00.0: SDB: 0000ce42:ff07fc47 WR-Periph-Syscon (00000000-000000ff)
FMC: mezzanine 0: 0000:02:00.0 on SPEC
FMC: poor dump of sdb first level:
0000: 53 44 42 2d 00 02 01 00 00 00 00 00 00 00 00 00
0010: 00 00 00 00 00 00 01 ff 00 00 00 00 00 00 06 51
0020: e6 a5 42 c9 00 00 00 02 20 12 05 11 57 42 34 2d
0030: 43 72 6f 73 73 62 61 72 2d 47 53 49 20 20 20 00
0040: 00 00 01 01 00 00 00 07 00 00 00 00 00 00 00 00
0050: 00 00 00 00 00 00 00 ff 00 00 00 00 00 00 ce 42
0060: ff 07 fc 47 00 00 00 01 20 12 03 05 57 52 2d 50
0070: 65 72 69 70 68 2d 53 79 73 63 6f 6e 20 20 20 01
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment