1. 24 Feb, 2016 1 commit
  2. 15 Feb, 2016 7 commits
    • Stephen Warren's avatar
      test/py: put "Starting U-Boot" into separate log section · 97255438
      Stephen Warren authored
      The initial boot of U-Boot happens within the context of the first test
      that needs to access the U-Boot console when there is no existing
      connection. This keeps all activity nestled within test execution, which
      fits well into the pytest model. However, this mingles the U-Boot startup
      logs with the execution of some test(s), which hides find the boundary
      between the two.
      To solve this, wrap the "Starting U-Boot" logic into a separate log
      section. If the user wishes, they can simply collapse this log section
      when viewing the HTML log, to concentrate purely on the test's own
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
    • Stephen Warren's avatar
      test/py: handle exceptions in console creation · 93134e18
      Stephen Warren authored
      u_boot_console.exec_attach.get_spawn() performs two steps:
      1) Spawn a process to communicate with the serial console.
      2) Reset the board so that U-Boot starts running from scratch.
      Currently, if an exception happens in step (2), no cleanup is performed on
      the process created in step (1). That process stays running and may e.g.
      hold serial port locks, or simply continue to read data from the serial
      port, thus preventing it from reaching any other process that attempts to
      read from the same serial port later. While there is error cleanup code in
      u_boot_console_base.ensure_spawned(), this is not triggered since the
      exception prevents assignment to self.p there, and hence the exception
      handler has no object to operate upon in cleanup_spawn().
      Solve this by enhancing u_boot_console.exec_attach.get_spawn() to clean
      up any objects it has created.
      In theory, u_boot_spawn.Spawn's constructor has a similar issue, so fix
      this too.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
    • Stephen Warren's avatar
      test/py: print summary in test order · 1326022c
      Stephen Warren authored
      Use lists rather than sets to record the status of tests. This causes
      the test summary in the HTML file to be generated in the same order as
      the tests are (or would have been) run. This makes it easier to locate
      the first failed test. The log for this test might have interesting
      first clues re: interaction with the environment (e.g. hardware flashing,
      serial console, ...) and may help tracking down external issues.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
    • Stephen Warren's avatar
      test/py: fix CONFIG_SPL test · 1235c791
      Stephen Warren authored
      The Python ini file parser that's used to parse .config converts all keys
      to lower-case. Hence, all queries against the results must use lower-case.
      Fix u_boot_console.ensure_spawned() to test CONFIG_SPL correctly, or the
      connection will fail for boards that have SPL.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
    • Stephen Warren's avatar
      test/py: don't import pexpect · 06088b04
      Stephen Warren authored
      The code replaced pexpect with custom code long ago. Don't import the
      unused module so it doesn't need to be installed.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
    • Stephen Warren's avatar
      test/py: add docs for gdbserver and pytest options · d70facf8
      Stephen Warren authored
      Add documentation describing the new --gdbserver feature, and some common
      pytest options.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
    • Stephen Warren's avatar
      test/py: run all "ut" subtests · 1cd85f57
      Stephen Warren authored
      Invoke each "ut"-based unit test as a separate pytest.
      Now that the DM unit test runs under test/py, remove the manual shell
      script that invokes it.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
      Tested-by: Simon Glass <sjg@chromium.org> # v2, on sandbox
  3. 09 Feb, 2016 4 commits
    • Stephen Warren's avatar
      test/py: capture the entire U-Boot version at boot · c82ce04a
      Stephen Warren authored
      The existing regex simply ensures that the captured version string doesn't
      go past the end of a line. We really want to grab as much as possible. Do
      this by explicitly including a ) character at the end of the regex to
      match the last character of the version test.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
    • Stephen Warren's avatar
      test/py: fix off-by-one error in spawn matching code · d8926811
      Stephen Warren authored
      A regex match object's .end() value is already the index after the match,
      not the index of the last character in the match, so there's no need to
      add 1 to point past the match.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
    • Stephen Warren's avatar
      test/py: HTML awesome! · 83357fd5
      Stephen Warren authored
      Implement three improvements to the HTML log file:
      - Ability to expand/contract sections. All passing sections are contracted
        at file load time so the user can concentrate on issues requiring
      - The overall status report is copied to the top of the log for easy
      - Add links from the status report to the test logs, for easy navigation.
      This all relies on Javascript and the jquery library. If the user doesn't
      have Javascript enabled, or jquery can't be downloaded, the log should
      look and behave identically to how it did before this patch.
      A few notes on the diff:
      - A few more 'with log.section("xxx")' were added, so that all stream
        blocks are kept within a section block for consistent HTML entity
        nesting structure. This changed indentation in a few places, making
        the diff look slightly larger.
      - HTML entity IDs are cleaned up. We assign simple incrementing integer
        IDs now, rather than using mangled test names which were possibly
      - Sections and streams now use common CSS class names (in addition to the
        current separate class names) to more easily share the new behaviour.
        This also reduces the CSS file size since rules don't need to be
      - An "OK" status is logged after some external command executions so that
        make and flash steps are auto-contracted at log file load time, assuming
        they passed.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
    • Stephen Warren's avatar
      test/py: exit(1) if there are problems running py.test · ac99831b
      Stephen Warren authored
      The test/py/test.py wrapper script catches exceptions thrown when
      exec()ing py.test in order to print a helpful error message. However,
      the exception handling code squashes the exception and so the script
      exits with a non-zero exit code, leading callers to believe that it
      passed. Fix this.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
  4. 08 Feb, 2016 1 commit
    • Stephen Warren's avatar
      test/py: support running sandbox under gdbserver · 89ab8410
      Stephen Warren authored
      Implement command--line option --gdbserver COMM, which does two things:
      a) Run the sandbox process under gdbserver, using COMM as gdbserver's
         communication channel.
      b) Disables all timeouts, so that if U-Boot is halted under the debugger,
         tests don't fail. If the user gives up in the middle of a debugging
         session, they can simply CTRL-C the test script to abort it.
      This allows easy debugging of test failures without having to manually
      re-create the failure conditions. Usage is:
      Window 1:
      ./test/py/test.py --bd sandbox --gdbserver localhost:1234
      Window 2:
      gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
      When using this option, it likely makes sense to use pytest's -k option
      to limit the set of tests that are executed.
      Simply running U-Boot directly under gdb (rather than gdbserver) was
      also considered. However, this was rejected because:
      a) gdb's output would then be processed by the test script, and likely
         confuse it causing false failures.
      b) pytest by default hides stdout from tests, which would prevent the
         user from interacting with gdb.
         While gdb can be told to redirect the debugee's stdio to a separate
         PTY, this would appear to leave gdb's stdio directed at the test
         scripts and the debugee's stdio directed elsewhere, which is the
         opposite of the desired effect. Perhaps some complicated PTY muxing
         and process hierarchy could invert this. However, the current scheme
         is simple to implement and use, so it doesn't seem worth complicating
      c) Using gdbserver allows arbitrary debuggers to be used, even those with
         a GUI. If the test scripts invoked the debugger themselves, they'd have
         to know how to execute arbitary applications. While the user could hide
         this all in a wrapper script, this feels like extra complication.
      An interesting future idea might be a --gdb-screen option, which could
      spawn both U-Boot and gdb separately, and spawn the screen into a newly
      created window under screen. Similar options could be envisaged for
      creating a new xterm/... too.
      --gdbserver  currently only supports sandbox, and not real hardware.
      That's primarily because the test hooks are responsible for all aspects of
      hardware control, so there's nothing for the test scripts themselves can
      do to enable gdbserver on real hardware. We might consider introducing a
      separate --disable-timeouts option to support use of debuggers on real
      hardware, and having --gdbserver imply that option.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
  5. 06 Feb, 2016 1 commit
  6. 30 Jan, 2016 4 commits
  7. 29 Jan, 2016 22 commits