Commit 334a994a authored by Heiko Schocher's avatar Heiko Schocher Committed by Tom Rini

test, tools: introduce tbot README

introduce a README how to use tbot for testing U-Boot
and/or linux kernels.
Signed-off-by: default avatarHeiko Schocher <hs@denx.de>
parent b17b7ea0
# Copyright (c) 2016 DENX Software Engineering GmbH
# Heiko Schocher <hs@denx.de>
#
# SPDX-License-Identifier: GPL-2.0+
#
What is tbot ?
==============
tbot is a tool for executing testcases on boards.
Source code found on [1]
Based on DUTS [2]
written in python
Basic Ideas of tbot
===================
(see also the figure:
https://github.com/hsdenx/tbot/blob/master/doc/tbot_structure.png )
- Virtual laboratory (VL)
VL is the basic environment that groups:
- [a number of] boards - target devices on which tbot executes testcases.
- one Lab PC
- Test case (TC):
A piece of python code, which uses the tbot class from [1].
Tbot provides functions for sending shell commands and parsing the
shell commands output.
Tbot waits endless for a shell commands end (detected through reading
the consoles prompt).
A TC can also call other TC-es.
remark:
Tbot not really waits endless, for a shell commands end, instead
tbot starts a watchdog in the background, and if it triggers, tbot
ends the TC as failed. In the tbot beginning there was a lot of
timeouts / retry cases, but it turned out, that waiting endless
is robust and easy ...
- Host PC (where tbot runs, currently only linux host tested)
must not a powerful machine (For example [3], I use a
raspberry pi for running tbot and buildbot)
- Lab PC:
- Host PC connects through ssh to the Lab PC
-> so it is possible to test boards, which
are not at the same place as the Host PC.
(Lab PC and Host PC can be the same of course)
-> maybe we can setup a Testsystem, which does nightly
U-Boot/Linux builds and test from current mainline U-Boot
on boards wherever they are accessible.
- necessary tasks a Lab PC must deliver:
- connect to boards console through a shell command.
- power on/off boards through a shell command
- detect the current power state of a board through
a shell command
- optional tasks:
- tftp server (for example loading images)
- nfs server (used as rootfs for linux kernels)
- Internet access for example for downloading
U-Boot source with git.
- toolchains installed for compiling source code
-> a linux machine is preffered.
- currently only Lab PC with an installed linux supported/tested.
- Boards(s):
the boards on which shell commands are executed.
- Board state:
equals to the software, the board is currently running.
Currently tbot supports 2 board states:
- "u-boot", if the board is running U-Boot
- "linux", if the board is running a linux kernel
It should be easy to add other board states to tbot, see
https://github.com/hsdenx/tbot/tree/master/src/lab_api/state_[u-boot/linux].py
A board state is detected through analysing the boards
shell prompt. In linux, tbot sets a special tbot prompt,
in U-Boot the prompt is static, and configurable in tbot through
a board config file.
A TC can say in which board state it want to send shell commands.
Tbot tries to detect the current board state, if board is not in
the requested board state, tbot tries to switch into the correct
state. If this fails, the TC fails.
It is possible to switch in a single TC between board states.
- tbot cmdline parameters:
$ python2.7 src/common/tbot.py --help
Usage: tbot.py [options]
Options:
-h, --help show this help message and exit
-c CFGFILE, --cfgfile=CFGFILE
the tbot common configfilename
-l LOGFILE, --logfile=LOGFILE
the tbot logfilename, if default, tbot creates a
defaultnamelogfile
-t TC, --testcase=TC the testcase which should be run
-v, --verbose be verbose, print all read/write to stdout
-w WORKDIR, --workdir=WORKDIR
set workdir, default os.getcwd()
$
tbot needs the following files for proper execution:
- tbot board configuration file (option -c):
A board configuration file contains settings tbot needs to
connect to the Lab PC and board specific variable settings
for testcases.
- name of the logfile tbot creates (option -l)
defaultname: 'log/' + now.strftime("%Y-%m-%d-%H-%M") + '.log'
- tbots working directory (option -w)
- the testcasename tbot executes (option -t)
You are interested and want to use tbot?
If so, please read on the file:
tools/tbot/README.install
If not read [3] ;-)
Heiko Schocher <hs@denx.de>
v1 2016.01.22
--------------
[1] https://github.com/hsdenx/tbot
[2] http://www.denx.de/wiki/DUTS/DUTSDocs
[3] automated Testsetup with buildbot and tbot doing cyclic tests
(buildbot used for starting tbot TC and web presentation of the
results, all testing done through tbot):
http://xeidos.ddns.net/buildbot/tgrid
Host PC in Letkes/hungary
VL in munich/germany
Fancy things are done here, for example:
- http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/43/steps/shell/logs/tbotlog
(I try to cleanup the logfile soon, so it is not so filled with crap ;-)
A first step see here:
http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/45/steps/shell/logs/tbotlog
(same TC now with the new loglevel = 'CON' ... not yet perfect)
Executed steps:
- clone u-boot.git
- set toolchain
- get a list of patchwork patches from my U-Boots ToDo list
- download all of them, and check them with checkpatch
and apply them to u-boot.git
- compile U-Boot for the smartweb board
- install the resulting images on the smartweb board
- boot U-boot
- test DFU
- more TC should be added here for testing U-Boot
- automatic "git bisect"
https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog
If a current U-Boot image not works on the tqm5200 board
this TC can be started. It starts a "git bisect" session,
and compiles for each step U-Boot, install it on the tqm5200
board, and tests if U-Boot works !
At the end, it detects the commit, which breaks the board
This TC is not dependend on U-Boot nor on a special board. It
needs only 3 variables:
tb.board_git_bisect_get_source_tc: TC which gets the source tree, in which
"git bisect" should be executed
tb.board_git_bisect_call_tc: TC which gets called every "git bisect" step,
which executes commands for detecting if current source code is OK or not.
This could be a TC which compiles U-Boot, install it on the board and
executes TC on the new booted U-Boot image. ! Board maybe gets borken,
as not all U-Boot images work, so you must have a TC which install U-Boot
image for example through a debugger.
tb.board_git_bisect_good_commit: last nown good commit id
# Copyright (c) 2016 DENX Software Engineering GmbH
# Heiko Schocher <hs@denx.de>
#
# SPDX-License-Identifier: GPL-2.0+
#
ToDo list for tbot
==================
please look also into the tbot ToDo list.
https://github.com/hsdenx/tbot/blob/master/ToDo
- cleanup tbot code:
- remove all retry / timeout pieces of code
- clean up tbot function names, as I am not good in
giving function a understandable name ;-)
- as I am not a python programmer, cleanup whole tbot code
- introduce a "layering" like yocto do, so U-Boot TC can integrated
into U-Boot source code.
Proposal:
introduce subdirs in "src/tc"
lab: all lab specific stuff
lab/common: common lab stuff (for example ssh handling)
lab/ssh_std: ssh_std specific stuff
u-boot: all u-boot tests
u-boot/common: common u-boot tc
u-boot/duts: DUTS tc
u-boot-dxr2: all u-boot dxr2 board specific tc
board: board tc
board/common: common board tc
board/dxr2: all tc for dxr2 board
linux: all linux tc
linux/common: common linux tc
linux/dxr2
- move U-Boot special TC to U-Boot source
-> need a mechanism in tbot, how it gets automatically for example
U-Boot TC from U-Boot source...
-> add a consistency checker
- simplify tbot log output (seperate a lot of output which is currently
in INFO logging level, to another logging level)
started (new loglevel "CON", whih prints read/write from console only), see:
https://github.com/hsdenx/tbot/commit/b4ab2567ad8c19ad53f785203159d3c8465a21c6
- make the timestamp configurable
- Open more than 2 filehandles ?
Do we need for more complex TC more than 2 filehandles?
- Find a way to document all TC and document all variables they use in an
automated way.
- write a lot of more TC
- get U-Boot configuration settings from current U-Boot code and use
them in U-Boot TC-es
# Copyright (c) 2016 DENX Software Engineering GmbH
# Heiko Schocher <hs@denx.de>
#
# SPDX-License-Identifier: GPL-2.0+
#
write a new testcase
=====================
A TC is written in python, so you can use python as usual. For accessing
the boards console, use functions from the tbotlib, therefore
First import the tbotlib with the line:
from tbotlib import tbot
If your TC uses variables, please add a line which adds them to
the log file (for debugging purposes):
logging.info("args: %s ...", tb.varname, ...)
Say tbot, for which board state your TC is valid with:
tb.set_board_state("u-boot")
Then you are ready ... and you can use the tbotlib funtions
for writting/reading to the boards console.
Big fat warning:
A TC must worry about to end only if a board has finished the shell
command!
Not following this rule, will end in unpredictable behaviour.
(hopefully) useful tbotlib functions
====================================
- set the board state, you want to test
tb.set_board_state(state)
states are: "u-boot" or "linux"
If tbot could not set the board state, tbot ends with failure.
- write a command to the boards console:
tb.eof_write_con(command):
write the command to the boards console. If this
fails, tbot ends with failure
- write a command to boards console and wait for prompt:
tb.eof_write_cmd(fd, command):
fd: filedescriptor which is used, use tb.channel_con for boards console
command: command which is written to fd
Wait endless for board prompt
- write a list of commands to boards console:
tb.eof_write_cmd_list(fd, cmdlist):
fd: filedescriptor which is used, use tb.channel_con for boards console
cmdlist: python list of commandstrings which is written to fd
- wait for boards prompt:
tb.eof_read_end_state_con(retry):
retry: deprecated, not used anymore, cleanup needed here...
tbot waits endless for the boards prompt
- write a command, wait for prompt and check, if a string is read
tb.write_cmd_check(fd, cmd, string):
fd: filedescriptor which is used, use tb.channel_con for boards console
cmd: command, which is send to fd
string: string which should be read from fd
return value:
True, if string is read and tbot got back boards prompt
False, else
tb.eof_write_cmd_check(fd, cmd, string):
same as tb.write_cmd_check(fd, cmd, string) except, that tbot
ends immediately with Failure, if string is not read.
- read until prompt and search strings:
tb.readline_and_search_strings(fd, strings):
fd: filedescriptor which is used, use tb.channel_con for boards console
strings: python list of strings, which can be read
If one of this strings is read, this function return the index, which
string is read. This function shoud be called in a while loop,
until this function returns 'prompt'
- read a line from filedescriptor:
not recommended to use, as the TC must check, if tprompt is read for every
readen line. Also TC must ensure, that it ends only, if prompt is read.
tb.read_line(fd, retry)
fd: filedescriptor which is used, use tb.channel_con for boards console
retry: retry of trying to reead a line
return values:
True, if a line is read. Readen line in tb.buf[fd]
False, if something read, but not a complete line
None, if nothing is read
check if string contains prompt with:
tb.is_end_fd(fd, string)
fd: filedescriptor which is used, use tb.channel_con for boards console
string: buffer, in which a prompt gets searched.
- calling other TC:
eof_call_tc(name):
call another TC from "src/tc"
if the called TC fails with failure, tbot ends with failure
call_tc(name):
call another TC from "src/tc"
if the TC which call_tc calls fails, call_tc() returns False, else True
There are more functions, but for writting TC this should be enough. But
its software, so new useful functions can always pop up.
Heiko Schocher <hs@denx.de>
v1 2016.01.23
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment