golden_tests.md

Golden Testing

TFF includes a small library called golden that makes it easy to write and maintain golden tests.

What are golden tests? When should I use them?

Golden tests are used when you want a developer to know that their code altered the output of a function. They violate many characteristics of good unit tests in that they make promises about the exact outputs of functions, rather than testing a specific set of clear, documented properties. It's sometimes not clear when a change to a golden output is "expected" or whether it is violating some property that the golden test saught to enforce. As such, a well-factored unit test is usually preferable to a golden test.

However, golden tests can be extremely useful for validating the exact contents of error messages, diagnostics, or generated code. In these cases, golden tests can be a helpful confidence check that any changes to the generated output "look right."

How should I write tests using golden?

golden.check_string(filename, value) is the primary entrypoint into the golden library. It will check the value string against the contents of a file whose last path element is filename. The full path to filename must be provided via a commandline --golden <path_to_file> argument. Similarly, these files must be made available to tests using the data argument to the py_test BUILD rule. Use the location function to generate a correct appropriate relative path:

py_string_test(
  ...
  args = [
    "--golden",
    "$(location path/to/first_test_output.expected)",
    ...
    "--golden",
    "$(location path/to/last_test_output.expected)",
  ],
  data = [
    "path/to/first_test_output.expected",
    ...
    "path/to/last_test_output.expected",
  ],
  ...
)

By convention, golden files should be placed in a sibling directory with the same name as their test target, suffixed with _goldens:

path/
  to/
    some_test.py
    some_test_goldens/
      test_case_one.expected
      ...
      test_case_last.expected

How do I update .expected files?

.expected files can be updated by running the affected test target with the arguments --test_arg=--update_goldens --test_strategy=local. The resulting diff should be checked for unanticipated changes.