Commit d201506c authored by Stephen Warren's avatar Stephen Warren Committed by Simon Glass

test/py: Implement pytest infrastructure

This tool aims to test U-Boot by executing U-Boot shell commands using the
console interface. A single top-level script exists to execute or attach
to the U-Boot console, run the entire script of tests against it, and
summarize the results. Advantages of this approach are:

- Testing is performed in the same way a user or script would interact
  with U-Boot; there can be no disconnect.
- There is no need to write or embed test-related code into U-Boot itself.
  It is asserted that writing test-related code in Python is simpler and
  more flexible that writing it all in C.
- It is reasonably simple to interact with U-Boot in this way.

A few simple tests are provided as examples. Soon, we should convert as
many as possible of the other tests in test/* and test/cmd_ut.c too.

The hook scripts, relay control utilities, and udev rules I use for my
own HW setup are published at https://github.com/swarren/uboot-test-hooks.

See README.md for more details!
Signed-off-by: default avatarStephen Warren <swarren@wwwdotorg.org>
Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
Tested-by: default avatarMichal Simek <michal.simek@xilinx.com>
Tested-by: default avatarSimon Glass <sjg@chromium.org>
Acked-by: Simon Glass <sjg@chromium.org> #v3
parent 27067a46
This diff is collapsed.
This diff is collapsed.
/*
* Copyright (c) 2015 Stephen Warren
* Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
*
* SPDX-License-Identifier: GPL-2.0
*/
/*
* This provides pretty formatting of the HTML log file, e.g.
* - colored bars beside/above log sections for easily parsed delineation.
* - color highlighting of various messages.
*/
body {
background-color: black;
color: #ffffff;
}
pre {
margin-top: 0px;
margin-bottom: 0px;
}
.implicit {
color: #808080;
}
.section {
border-style: solid;
border-color: #303030;
border-width: 0px 0px 0px 5px;
padding-left: 5px
}
.section-header {
background-color: #303030;
margin-left: -5px;
margin-top: 5px;
}
.section-trailer {
display: none;
}
.stream {
border-style: solid;
border-color: #303030;
border-width: 0px 0px 0px 5px;
padding-left: 5px
}
.stream-header {
background-color: #303030;
margin-left: -5px;
margin-top: 5px;
}
.stream-trailer {
display: none;
}
.error {
color: #ff0000
}
.warning {
color: #ffff00
}
.info {
color: #808080
}
.action {
color: #8080ff
}
.status-pass {
color: #00ff00
}
.status-skipped {
color: #ffff00
}
.status-fail {
color: #ff0000
}
This diff is collapsed.
# Copyright (c) 2015 Stephen Warren
# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
#
# SPDX-License-Identifier: GPL-2.0
# Static configuration data for pytest. pytest reads this at startup time.
[pytest]
markers =
boardspec: U-Boot: Describes the set of boards a test can/can't run on.
buildconfigspec: U-Boot: Describes Kconfig/config-header constraints.
#!/usr/bin/env python
# Copyright (c) 2015 Stephen Warren
# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
#
# SPDX-License-Identifier: GPL-2.0
# Wrapper script to invoke pytest with the directory name that contains the
# U-Boot tests.
import os
import os.path
import sys
# Get rid of argv[0]
sys.argv.pop(0)
# argv; py.test test_directory_name user-supplied-arguments
args = ["py.test", os.path.dirname(__file__) + "/tests"]
args.extend(sys.argv)
try:
os.execvp("py.test", args)
except:
# Log full details of any exception for detailed analysis
import traceback
traceback.print_exc()
# Hint to the user that they likely simply haven't installed the required
# dependencies.
print >>sys.stderr, """
exec(py.test) failed; perhaps you are missing some dependencies?
See test/py/README.md for the list."""
# Copyright (c) 2015 Stephen Warren
# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
#
# SPDX-License-Identifier: GPL-2.0
# pytest runs tests the order of their module path, which is related to the
# filename containing the test. This file is named such that it is sorted
# first, simply as a very basic sanity check of the functionality of the U-Boot
# command prompt.
def test_version(u_boot_console):
'''Test that the "version" command prints the U-Boot version.'''
# "version" prints the U-Boot sign-on message. This is usually considered
# an error, so that any unexpected reboot causes an error. Here, this
# error detection is disabled since the sign-on message is expected.
with u_boot_console.disable_check('main_signon'):
response = u_boot_console.run_command('version')
# Ensure "version" printed what we expected.
u_boot_console.validate_version_string_in_text(response)
# Copyright (c) 2015 Stephen Warren
# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
#
# SPDX-License-Identifier: GPL-2.0
def test_help(u_boot_console):
'''Test that the "help" command can be executed.'''
u_boot_console.run_command('help')
# Copyright (c) 2015 Stephen Warren
# Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved.
#
# SPDX-License-Identifier: GPL-2.0
def test_unknown_command(u_boot_console):
'''Test that executing an unknown command causes U-Boot to print an
error.'''
# The "unknown command" error is actively expected here,
# so error detection for it is disabled.
with u_boot_console.disable_check('unknown_command'):
response = u_boot_console.run_command('non_existent_cmd')
assert('Unknown command \'non_existent_cmd\' - try \'help\'' in response)
This diff is collapsed.
# Copyright (c) 2015 Stephen Warren
# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
#
# SPDX-License-Identifier: GPL-2.0
# Logic to interact with U-Boot running on real hardware, typically via a
# physical serial port.
import sys
from u_boot_spawn import Spawn
from u_boot_console_base import ConsoleBase
class ConsoleExecAttach(ConsoleBase):
'''Represents a physical connection to a U-Boot console, typically via a
serial port. This implementation executes a sub-process to attach to the
console, expecting that the stdin/out of the sub-process will be forwarded
to/from the physical hardware. This approach isolates the test infra-
structure from the user-/installation-specific details of how to
communicate with, and the identity of, serial ports etc.'''
def __init__(self, log, config):
'''Initialize a U-Boot console connection.
Args:
log: A multiplexed_log.Logfile instance.
config: A "configuration" object as defined in conftest.py.
Returns:
Nothing.
'''
# The max_fifo_fill value might need tweaking per-board/-SoC?
# 1 would be safe anywhere, but is very slow (a pexpect issue?).
# 16 is a common FIFO size.
# HW flow control would mean this could be infinite.
super(ConsoleExecAttach, self).__init__(log, config, max_fifo_fill=16)
self.log.action('Flashing U-Boot')
cmd = ['u-boot-test-flash', config.board_type, config.board_identity]
runner = self.log.get_runner(cmd[0], sys.stdout)
runner.run(cmd)
runner.close()
def get_spawn(self):
'''Connect to a fresh U-Boot instance.
The target board is reset, so that U-Boot begins running from scratch.
Args:
None.
Returns:
A u_boot_spawn.Spawn object that is attached to U-Boot.
'''
args = [self.config.board_type, self.config.board_identity]
s = Spawn(['u-boot-test-console'] + args)
self.log.action('Resetting board')
cmd = ['u-boot-test-reset'] + args
runner = self.log.get_runner(cmd[0], sys.stdout)
runner.run(cmd)
runner.close()
return s
# Copyright (c) 2015 Stephen Warren
# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
#
# SPDX-License-Identifier: GPL-2.0
# Logic to interact with the sandbox port of U-Boot, running as a sub-process.
import time
from u_boot_spawn import Spawn
from u_boot_console_base import ConsoleBase
class ConsoleSandbox(ConsoleBase):
'''Represents a connection to a sandbox U-Boot console, executed as a sub-
process.'''
def __init__(self, log, config):
'''Initialize a U-Boot console connection.
Args:
log: A multiplexed_log.Logfile instance.
config: A "configuration" object as defined in conftest.py.
Returns:
Nothing.
'''
super(ConsoleSandbox, self).__init__(log, config, max_fifo_fill=1024)
def get_spawn(self):
'''Connect to a fresh U-Boot instance.
A new sandbox process is created, so that U-Boot begins running from
scratch.
Args:
None.
Returns:
A u_boot_spawn.Spawn object that is attached to U-Boot.
'''
return Spawn([self.config.build_dir + '/u-boot'])
def kill(self, sig):
'''Send a specific Unix signal to the sandbox process.
Args:
sig: The Unix signal to send to the process.
Returns:
Nothing.
'''
self.ensure_spawned()
self.log.action('kill %d' % sig)
self.p.kill(sig)
def validate_exited(self):
'''Determine whether the sandbox process has exited.
If required, this function waits a reasonable time for the process to
exit.
Args:
None.
Returns:
Boolean indicating whether the process has exited.
'''
p = self.p
self.p = None
for i in xrange(100):
ret = not p.isalive()
if ret:
break
time.sleep(0.1)
p.close()
return ret
# Copyright (c) 2015-2016, NVIDIA CORPORATION. All rights reserved.
#
# SPDX-License-Identifier: GPL-2.0
# Logic to spawn a sub-process and interact with its stdio.
import os
import re
import pty
import signal
import select
import time
class Timeout(Exception):
'''An exception sub-class that indicates that a timeout occurred.'''
pass
class Spawn(object):
'''Represents the stdio of a freshly created sub-process. Commands may be
sent to the process, and responses waited for.
'''
def __init__(self, args):
'''Spawn (fork/exec) the sub-process.
Args:
args: array of processs arguments. argv[0] is the command to execute.
Returns:
Nothing.
'''
self.waited = False
self.buf = ''
self.logfile_read = None
self.before = ''
self.after = ''
self.timeout = None
(self.pid, self.fd) = pty.fork()
if self.pid == 0:
try:
# For some reason, SIGHUP is set to SIG_IGN at this point when
# run under "go" (www.go.cd). Perhaps this happens under any
# background (non-interactive) system?
signal.signal(signal.SIGHUP, signal.SIG_DFL)
os.execvp(args[0], args)
except:
print 'CHILD EXECEPTION:'
import traceback
traceback.print_exc()
finally:
os._exit(255)
self.poll = select.poll()
self.poll.register(self.fd, select.POLLIN | select.POLLPRI | select.POLLERR | select.POLLHUP | select.POLLNVAL)
def kill(self, sig):
'''Send unix signal "sig" to the child process.
Args:
sig: The signal number to send.
Returns:
Nothing.
'''
os.kill(self.pid, sig)
def isalive(self):
'''Determine whether the child process is still running.
Args:
None.
Returns:
Boolean indicating whether process is alive.
'''
if self.waited:
return False
w = os.waitpid(self.pid, os.WNOHANG)
if w[0] == 0:
return True
self.waited = True
return False
def send(self, data):
'''Send data to the sub-process's stdin.
Args:
data: The data to send to the process.
Returns:
Nothing.
'''
os.write(self.fd, data)
def expect(self, patterns):
'''Wait for the sub-process to emit specific data.
This function waits for the process to emit one pattern from the
supplied list of patterns, or for a timeout to occur.
Args:
patterns: A list of strings or regex objects that we expect to
see in the sub-process' stdout.
Returns:
The index within the patterns array of the pattern the process
emitted.
Notable exceptions:
Timeout, if the process did not emit any of the patterns within
the expected time.
'''
for pi in xrange(len(patterns)):
if type(patterns[pi]) == type(''):
patterns[pi] = re.compile(patterns[pi])
try:
while True:
earliest_m = None
earliest_pi = None
for pi in xrange(len(patterns)):
pattern = patterns[pi]
m = pattern.search(self.buf)
if not m:
continue
if earliest_m and m.start() > earliest_m.start():
continue
earliest_m = m
earliest_pi = pi
if earliest_m:
pos = earliest_m.start()
posafter = earliest_m.end() + 1
self.before = self.buf[:pos]
self.after = self.buf[pos:posafter]
self.buf = self.buf[posafter:]
return earliest_pi
events = self.poll.poll(self.timeout)
if not events:
raise Timeout()
c = os.read(self.fd, 1024)
if not c:
raise EOFError()
if self.logfile_read:
self.logfile_read.write(c)
self.buf += c
finally:
if self.logfile_read:
self.logfile_read.flush()
def close(self):
'''Close the stdio connection to the sub-process.
This also waits a reasonable time for the sub-process to stop running.
Args:
None.
Returns:
Nothing.
'''
os.close(self.fd)
for i in xrange(100):
if not self.isalive():
break
time.sleep(0.1)
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment