1. 25 Apr, 2016 1 commit
  2. 12 Apr, 2016 1 commit
  3. 26 Feb, 2016 1 commit
    • Stephen Warren's avatar
      test/py: skip tests that require large CONFIG_SYS_MAXARGS · 24862c64
      Stephen Warren authored
      test_hush_if_test.py executes commands that require large values of
      CONFIG_SYS_MAXARGS. Detect cases where the configured value is too low
      and skip those tests.
      
      Ideally, this would be implemented inside console.run_command(). However,
      the command passed to that function is already a completely formed string,
      and determining its argument count usage would require splitting commands
      at ;, handling quoting to deal with arguments containing spaces, etc. Even
      passing the command as a list wouldn't solve all these issues, since we'd
      still need to split commands on ; and deal with cases like "if test ..."
      which consumes 0 of the argument count.
      Signed-off-by: default avatarStephen Warren <swarren@wwwdotorg.org>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
      24862c64
  4. 15 Feb, 2016 1 commit
  5. 08 Feb, 2016 1 commit
    • Stephen Warren's avatar
      test/py: support running sandbox under gdbserver · 89ab8410
      Stephen Warren authored
      Implement command--line option --gdbserver COMM, which does two things:
      
      a) Run the sandbox process under gdbserver, using COMM as gdbserver's
         communication channel.
      
      b) Disables all timeouts, so that if U-Boot is halted under the debugger,
         tests don't fail. If the user gives up in the middle of a debugging
         session, they can simply CTRL-C the test script to abort it.
      
      This allows easy debugging of test failures without having to manually
      re-create the failure conditions. Usage is:
      
      Window 1:
      ./test/py/test.py --bd sandbox --gdbserver localhost:1234
      
      Window 2:
      gdb ./build-sandbox/u-boot -ex 'target remote localhost:1234'
      
      When using this option, it likely makes sense to use pytest's -k option
      to limit the set of tests that are executed.
      
      Simply running U-Boot directly under gdb (rather than gdbserver) was
      also considered. However, this was rejected because:
      
      a) gdb's output would then be processed by the test script, and likely
         confuse it causing false failures.
      
      b) pytest by default hides stdout from tests, which would prevent the
         user from interacting with gdb.
      
         While gdb can be told to redirect the debugee's stdio to a separate
         PTY, this would appear to leave gdb's stdio directed at the test
         scripts and the debugee's stdio directed elsewhere, which is the
         opposite of the desired effect. Perhaps some complicated PTY muxing
         and process hierarchy could invert this. However, the current scheme
         is simple to implement and use, so it doesn't seem worth complicating
         matters.
      
      c) Using gdbserver allows arbitrary debuggers to be used, even those with
         a GUI. If the test scripts invoked the debugger themselves, they'd have
         to know how to execute arbitary applications. While the user could hide
         this all in a wrapper script, this feels like extra complication.
      
      An interesting future idea might be a --gdb-screen option, which could
      spawn both U-Boot and gdb separately, and spawn the screen into a newly
      created window under screen. Similar options could be envisaged for
      creating a new xterm/... too.
      
      --gdbserver  currently only supports sandbox, and not real hardware.
      That's primarily because the test hooks are responsible for all aspects of
      hardware control, so there's nothing for the test scripts themselves can
      do to enable gdbserver on real hardware. We might consider introducing a
      separate --disable-timeouts option to support use of debuggers on real
      hardware, and having --gdbserver imply that option.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      89ab8410
  6. 29 Jan, 2016 12 commits
    • Stephen Warren's avatar
      test/py: dfu: allow boardenv to specify test sizes · 26db3a61
      Stephen Warren authored
      Allow the env__dfu_configs boardenv data to specify the set of DFU
      transfer sizes to test. Manually specifying test sizes is useful if you
      wish to test multiple DFU configurations (e.g. SD card ext4 filesystem, SD
      card whole raw partition, RAM, etc.), but don't want to test every
      single transfer size on each, to avoid bloating the overall time taken by
      testing. If the boardenv doesn't specify a set of sizes, the built-in list
      is used as a default, preserving backwards-compatibility.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
      26db3a61
    • Stephen Warren's avatar
      test/py: fix a couple typos in comments · db261f00
      Stephen Warren authored
      s/updata/update/.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Reviewed-by: default avatarSimon Glass <sjg@chromium.org>
      db261f00
    • Stephen Warren's avatar
      test/py: Provide custom IDs when parametrizing tests · d20e5e97
      Stephen Warren authored
      When pytest generates the name for parametrized tests, simple parameter
      values (ints, strings) get used directly, but more complex values such
      as dicts are not handled. This yields test names such as:
      
          dfu[env__usb_dev_port0-env__dfu_config0]
          dfu[env__usb_dev_port0-env__dfu_config1]
      
      Add some code to extract a custom fixture ID from the fixture values, so
      that we end up with meaningful names such as:
      
          dfu[micro_b-emmc]
          dfu[devport2-ram]
      
      If the boardenv file doesn't define custom names, the code falls back to
      the old algorithm.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Reviewed-by: default avatarSimon Glass <sjg@chromium.org>
      d20e5e97
    • Stephen Warren's avatar
      test/py: Quote consistency · a2ec5606
      Stephen Warren authored
      When converting test/py from " to ', I missed a few places (or added a
      few inconsistencies later). Fix these.
      
      Note that only quotes in code are converted; double-quotes in comments
      and HTML are left as-is, since English and HTML use " not '.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Reviewed-by: default avatarSimon Glass <sjg@chromium.org>
      a2ec5606
    • Stephen Warren's avatar
      test/py: use " for docstrings · e8debf39
      Stephen Warren authored
      Python's coding style docs indicate to use " not ' for docstrings.
      
      test/py has other violations of the coding style docs, since the docs
      specify a stranger style than I would expect, but nobody has complained
      about those yet:-)
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Reviewed-by: default avatarSimon Glass <sjg@chromium.org>
      e8debf39
    • Stephen Warren's avatar
      test/py: make net test aware of USB and PCI enumeration · 56382a81
      Stephen Warren authored
      The existing net test executes a list of commands supplied by boardenv
      variable env__net_pre_commands. The idea was that boardenv would know
      whether the Ethernet device was attached to USB, PCI, ... and hence was
      the best place to put any commands required to probe the device.
      
      However, this approach doesn't scale well when attempting to use a single
      boardenv across multiple branches of U-Boot, some of which require "pci
      enum" to enumerate PCI and others of which don't, or don't /yet/ simply
      because various upstream changes haven't been merged down.
      
      This patch updates the test to require that the boardenv state which HW
      features are required for Ethernet to work, and lets the test itself map
      that knowledge to the set of commands to execute. Since this mapping is
      part of the test script, which is part of the U-Boot code/branch, this
      approach is more scalable. It also feels cleaner, since again boardenv
      is only providing data, rather than test logic.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Reviewed-by: default avatarSimon Glass <sjg@chromium.org>
      56382a81
    • Stephen Warren's avatar
      test/py: dfu: error out if USB device already exists · be1df826
      Stephen Warren authored
      The DFU test requests U-Boot configure its USB controller in device mode,
      then waits for the host machine to enumerate the USB device and create a
      device node for it. However, this wait can be fooled if the USB device
      node already exists before the test starts, e.g. if some previous software
      stack already configured the USB controller into device mode and never
      de-configured it. This "previous software stack" could even be another
      test/py test, if U-Boot's own USB teardown does not operate correctly. If
      this happens, dfu-util may be run before U-Boot is ready to serve DFU
      commands, which may cause false test failures.
      
      Enhance the dfu test to fail if the device node exists before it is
      expected to.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Reviewed-by: default avatarSimon Glass <sjg@chromium.org>
      be1df826
    • Stephen Warren's avatar
      test/py: add a networking test · e5bb279f
      Stephen Warren authored
      This tests:
      - dhcp (if indicated by boardenv file).
      - Static IP network setup (if provided by boardenv file).
      - Ping.
      - TFTP get.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
      e5bb279f
    • Stephen Warren's avatar
      test/py: move find_ram_base() into u_boot_utils · 05266103
      Stephen Warren authored
      find_ram_base() is a shared utility function, not a core part of the
      U-Boot console interaction.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
      05266103
    • Stephen Warren's avatar
      test/py: add DFU test · f5d196d0
      Stephen Warren authored
      Add a test of DFU functionality to the Python test suite. The test
      starts DFU in U-Boot, waits for USB device enumeration on the host,
      executes dfu-util multiple times to test various transfer sizes, many
      of which trigger USB driver edge cases, and finally aborts the DFU
      command in U-Boot.
      
      This test mirrors the functionality previously available via the shell
      scripts in test/dfu, and hence those are removed too.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarLukasz Majewski <l.majewski@samsung.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
      f5d196d0
    • Stephen Warren's avatar
      test/py: ums: add filesystem-based testing · d054f4c2
      Stephen Warren authored
      Enhance the UMS test to optionally mount a partition and read/write a file
      to it, validating that the content written and read back are identical.
      
      This enhancement is backwards-compatible; old boardenv contents that don't
      define the new configuration data will cause the test code to perform as
      before.
      
      test/ums/ is deleted since the Python test now performs the same testing
      that it did.
      
      The code is also re-written to make use of the recently added utility
      module, and split it up into nested functions so the overall logic of
      the test process can be followed more easily without the details
      cluttering the code.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarLukasz Majewski <l.majewski@samsung.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
      d054f4c2
    • Stephen Warren's avatar
      test/py: move U-Boot respawn trigger to the test core · 636f38d8
      Stephen Warren authored
      Prior to this change, U-Boot was lazilly (re-)spawned if/when a test
      attempted to interact with it, and no active connection existed. This
      approach was simple, yet had the disadvantage that U-Boot might be
      spawned in the middle of a test function, e.g. after the test had already
      performed actions such as creating data files, etc. In that case, this
      could cause the log to contain the sequence (1) some test logs, (2)
      U-Boot's boot process, (3) the rest of that test's logs. This isn't
      optimally readable. This issue will affect the upcoming DFU and enhanced
      UMS tests.
      
      This change converts u_boot_console to be a function-scoped fixture, so
      that pytest attempts to re-create the object for each test invocation.
      This allows the fixture factory function to ensure that U-Boot is spawned
      prior to every test. In practice, the same object is returned each time
      so there is essentially no additional overhead due to this change.
      
      This allows us to remove:
      
      - The explicit ensure_spawned() call from test_sleep, since the core now
      ensures that the spawn happens before the test code is executed.
      
      - The laxy calls to ensure_spawned() in the u_boot_console_*
      implementations.
      
      The one downside is that test_env's "state_ttest_env" fixture must be
      converted to a function-scoped fixture too, since a module-scoped fixture
      cannot use a function-scoped fixture. To avoid overhead, we use the same
      trick of returning the same object each time.
      Signed-off-by: default avatarStephen Warren <swarren@nvidia.com>
      Acked-by: default avatarSimon Glass <sjg@chromium.org>
      636f38d8
  7. 21 Jan, 2016 8 commits