ANGLE Test Harness

The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the Chromium test harness. It features:

• splitting a test set into shards

• catching and reporting crashes and timeouts

• outputting to the Chromium JSON test results format

• multi-process execution

Command-Line Arguments

The ANGLE test harness accepts all standard GoogleTest arguments. The harness also accepts the following additional command-line arguments:

• --batch-size limits the number of tests to run in each batch

• --batch-timeout limits the amount of time spent in each batch

• --bot-mode enables multi-process execution and test batching

• --debug-test-groups dumps the test config categories when using bot-mode

• --filter-file allows passing a larger gtest_filter via a file

• --histogram-json-file outputs a formatted JSON file for perf dashboards

• --max-processes limits the number of simuntaneous processes

• --results-directory specifies a directory to write test results to

• --results-file specifies a location for the JSON test result output

• --shard-count and --shard-index control the test sharding

• --test-timeout limits the amount of time spent in each test

• --flaky-retries allows for tests to fail a fixed number of times and still pass

• --disable-crash-handler forces off OS-level crash handling

• --isolated-outdir specifies a test artifacts directory

• --max-failures specifies a count of failures after which the harness early exits.

--isolated-script-test-output and --isolated-script-perf-test-output mirror --results-file and --histogram-json-file respectively.

As well as the custom command-line arguments we support a few standard GoogleTest arguments:

• gtest_filter works as it normally does with GoogleTest

• gtest_also_run_disabled_tests works as it normally does as well

Other GoogleTest arguments are not supported although they may work.

Implementation Notes

• The test harness only requires angle_common and angle_util.

• It does not depend on any Chromium browser code. This allows us to compile on other non-Clang platforms.

• It uses rapidjson to read and write JSON files.

• Test execution is not currently deterministic in multi-process mode.

Normal Mode vs Bot Mode

The test runner has two main modes of operation: normal and bot mode.

During normal mode:

• Tests are executed single-process and single-thread.

• The test runner executes the GoogleTest Run function.

• We use a TestEventListener to record test results for our output JSON file.

• A watchdog thread will force a fast exit if no test results get recorded after a timeout.

• Crashes are handled via ANGLE's test crash handling code.

During bot mode:

• Tests are run in multiple processes depending on the system processor count.

• A server process records the child processes' stdout and stderr.

• The server terminates a child process if there's no progress after a timeout.

• The server sorts work into batches according to the back-end configuration.

• This prevents driver errors from using multiple back-ends in the same process.

• Batches are striped to help split up slow groups of tests.

• The server passes test batches to child processes via a gtest_filter file.

• Bot mode does not work on Android or Fuchsia.

See the source code for more details: TestSuite.h and TestSuite.cpp.

Potential Areas of Improvement

• Deterministic test execution.

• Using sockets to communicate with test children. Similar to dEQP's test harness.

• Closer integration with ANGLE's test expectations and system config libraries.

• Supporting a GoogleTest-free integration.