emacs/doc/misc/ert.texi

\input texinfo
@c %**start of header
@setfilename ../../info/ert.info
@settitle Emacs Lisp Regression Testing
@include docstyle.texi
@syncodeindex fn cp
@syncodeindex vr cp
@syncodeindex pg cp
@syncodeindex ky cp
@c %**end of header

@dircategory Emacs misc features
@direntry
* ERT: (ert).                   Emacs Lisp regression testing tool.
@end direntry

@copying
Copyright @copyright{} 2008, 2010--2024 Free Software Foundation, Inc.

@quotation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
Invariant Sections, with the Front-Cover Texts being ``A GNU Manual,''
and with the Back-Cover Texts as in (a) below.  A copy of the license
is included in the section entitled ``GNU Free Documentation License''.

(a) The FSF's Back-Cover Text is: ``You have the freedom to copy and
modify this GNU manual.''
@end quotation
@end copying

@titlepage
@title Emacs Lisp Regression Testing
@page
@vskip 0pt plus 1filll
@insertcopying
@end titlepage

@contents

@ifnottex
@node Top
@top ERT: Emacs Lisp Regression Testing

@insertcopying

ERT is a tool for automated testing in Emacs Lisp.  Its main features
are facilities for defining tests, running them and reporting the
results, and for debugging test failures interactively.

ERT is similar to tools for other environments such as JUnit, but has
unique features that take advantage of the dynamic and interactive
nature of Emacs.  Despite its name, it works well both for test-driven
development (see
@url{https://en.wikipedia.org/wiki/Test-driven_development}) and for
traditional software development methods.

@menu
* Introduction::                A simple example of an ERT test.
* How to Run Tests::            Run tests in Emacs or from the command line.
* How to Write Tests::          How to add tests to your Emacs Lisp code.
* How to Debug Tests::          What to do if a test fails.
* Extending ERT::               ERT is extensible in several ways.
* Other Testing Concepts::      Features not in ERT.
* Index::                       Concept, Function and Variable Index
* GNU Free Documentation License::  The license for this documentation.

@detailmenu
 --- The Detailed Node Listing ---

How to Run Tests

* Running Tests Interactively::  Run tests in your current Emacs.
* Running Tests in Batch Mode::  Run tests in emacs -Q.
* Test Selectors::               Choose which tests to run.

How to Write Tests

* The @code{should} Macro::          A powerful way to express assertions.
* Expected Failures::           Tests for known bugs.
* Tests and Their Environment:: Don't depend on customizations; no side effects.
* Useful Techniques::           Some examples.

How to Debug Tests

* Understanding Explanations::  How ERT gives details on why an assertion failed.
* Interactive Debugging::       Tools available in the ERT results buffer.

Extending ERT

* Defining Explanation Functions::  Teach ERT about more predicates.
* Low-Level Functions for Working with Tests::  Use ERT's data for your purposes.

Other Testing Concepts

* Mocks and Stubs::           Stubbing out code that is irrelevant to the test.
* Fixtures and Test Suites::  How ERT differs from tools for other languages.

Index

* Index::                       Concept, Function and Variable Index

Appendix

* GNU Free Documentation License:: The license for this documentation.

@end detailmenu
@end menu
@end ifnottex


@node Introduction
@chapter Introduction
@cindex introduction to ERT

ERT allows you to define @emph{tests} in addition to functions,
macros, variables, and the other usual Lisp constructs.  Tests are
simply Lisp code: code that invokes other code and checks whether
it behaves as expected.

ERT keeps track of the tests that are defined and provides convenient
commands to run them to verify whether the definitions that are
currently loaded in Emacs pass the tests.

Some Lisp files have comments like the following (adapted from the
package @file{pp.el}):

@lisp
;; (pp-to-string '(quote quote))          ; expected: "'quote"
;; (pp-to-string '((quote a) (quote b)))  ; expected: "('a 'b)\n"
;; (pp-to-string '('a 'b))                ; same as above
@end lisp

The code contained in these comments can be evaluated from time to
time to compare the output with the expected output.  ERT formalizes
this and introduces a common convention, which simplifies Emacs
development, since programmers no longer have to manually find and
evaluate such comments.

An ERT test definition equivalent to the above comments is this:

@lisp
(ert-deftest pp-test-quote ()
  "Tests the rendering of `quote' symbols in `pp-to-string'."
  (should (equal (pp-to-string '(quote quote)) "'quote"))
  (should (equal (pp-to-string '((quote a) (quote b))) "('a 'b)\n"))
  (should (equal (pp-to-string '('a 'b)) "('a 'b)\n")))
@end lisp

If you know @code{defun}, the syntax of @code{ert-deftest} should look
familiar: This example defines a test named @code{pp-test-quote} that
will pass if the three calls to @code{equal} all return non-@code{nil}.

@code{should} is a macro with the same meaning as @code{cl-assert} but
better error reporting.  @xref{The @code{should} Macro}.

Each test should have a name that describes what functionality it tests.
Test names can be chosen arbitrarily---they are in a
namespace separate from functions and variables---but should follow
the usual Emacs Lisp convention of having a prefix that indicates
which package they belong to.  Test names are displayed by ERT when
reporting failures and can be used when selecting which tests to run.

The empty parentheses @code{()} in the first line don't currently have
any meaning and are reserved for future extension.  They also make
the syntax of @code{ert-deftest} more similar to that of @code{defun}.

The docstring describes what feature this test tests.  When running
tests interactively, the first line of the docstring is displayed for
tests that fail, so it is good if the first line makes sense on its
own.

The body of a test can be arbitrary Lisp code.  It should have as few
side effects as possible; each test should be written to clean up
after itself, leaving Emacs in the same state as it was before the
test.  Tests should clean up even if they fail.  @xref{Tests and Their
Environment}.


@node  How to Run Tests
@chapter How to Run Tests
@cindex how to run ert tests

You can run tests either in the Emacs you are working in, or on the
command line in a separate Emacs process in batch mode (i.e., with no
user interface).  The former mode is convenient during interactive
development, the latter is useful to make sure that tests pass
independently of your customizations; and it allows you to invoke
tests from makefiles, and to write scripts that run tests in several
different Emacs versions.

@menu
* Running Tests Interactively::  Run tests in your current Emacs.
* Running Tests in Batch Mode::  Run tests in emacs -Q.
* Test Selectors::               Choose which tests to run.
@end menu


@node Running Tests Interactively
@section Running Tests Interactively
@cindex running tests interactively
@cindex interactive testing

@findex ert
You can run the tests that are currently defined in your Emacs with
the command @kbd{M-x ert @key{RET} t @key{RET}}.  (For an
explanation of the @code{t} argument, @pxref{Test Selectors}.) ERT will pop
up a new buffer, the ERT results buffer, showing the results of the
tests run.  It looks like this:

@example
Selector: t
Passed:  31
Skipped: 0
Failed:  2 (2 unexpected)
Total:   33/33

Started at:   2008-09-11 08:39:25-0700
Finished.
Finished at:  2008-09-11 08:39:27-0700

FF...............................

F addition-test
    (ert-test-failed
     ((should
       (=
        (+ 1 2)
        4))
      :form
      (= 3 4)
      :value nil))

F list-test
    (ert-test-failed
     ((should
       (equal
        (list 'a 'b 'c)
        '(a b d)))
      :form
      (equal
       (a b c)
       (a b d))
      :value nil :explanation
      (list-elt 2
                (different-atoms c d))))
@end example

@cindex test results buffer
At the top, there is a summary of the results: we ran all tests defined
in the current Emacs (@code{Selector: t}), 31 of them passed, and 2
failed unexpectedly.  @xref{Expected Failures}, for an explanation of
the term @emph{unexpected} in this context.

The line of dots and @code{F}s is a progress bar where each character
represents one test; it fills while the tests are running.  A dot
means that the test passed, an @code{F} means that it failed.  Below
the progress bar, ERT shows details about each test that had an
unexpected result.  In the example above, there are two failures, both
due to failed @code{should} forms.  @xref{Understanding Explanations},
for more details.

The following key bindings are available in the ERT results buffer:

@table @kbd

@item @key{RET}
@kindex RET@r{, in ert results buffer}
Each name of a function or macro in this buffer is a button; moving
point to it and typing @kbd{@key{RET}} jumps to its definition.

@item @key{TAB}
@itemx S-@key{TAB}
@kindex TAB@r{, in ert results buffer}
@kindex S-TAB@r{, in ert results buffer}
Cycle between buttons forward (@code{forward-button}) and backward
(@code{backward-button}).

@item r
@kindex r@r{, in ert results buffer}
@findex ert-results-rerun-test-at-point
Re-run the test near point on its own
(@code{ert-results-rerun-test-at-point}).

@item d
@kindex d@r{, in ert results buffer}
@findex ert-results-rerun-test-at-point-debugging-errors
Re-run the test near point on its own with the debugger enabled
(@code{ert-results-rerun-test-at-point-debugging-errors}).

@item R
@kindex R@r{, in ert results buffer}
@findex ert-results-rerun-all-tests
Re-run all tests (@code{ert-results-rerun-all-tests}).

@item .
@kindex .@r{, in ert results buffer}
@findex ert-results-find-test-at-point-other-window
Jump to the definition of the test near point
(@code{ert-results-find-test-at-point-other-window}).  This has the
same effect as @kbd{@key{RET}}, but does not require point to be on
the name of the test.

@item b
@kindex b@r{, in ert results buffer}
@findex ert-results-pop-to-backtrace-for-test-at-point
@cindex backtrace of a failed test
Show the backtrace of a failed test
(@code{ert-results-pop-to-backtrace-for-test-at-point}).
@xref{Debugging,, Backtraces, elisp, GNU Emacs Lisp Reference Manual},
for more information about backtraces.

@item l
@kindex l@r{, in ert results buffer}
@findex ert-results-pop-to-should-forms-for-test-at-point
Show the list of @code{should} forms executed in the test
(@code{ert-results-pop-to-should-forms-for-test-at-point}).

@item m
@kindex m@r{, in ert results buffer}
@findex ert-results-pop-to-messages-for-test-at-point
Show any messages that were generated (with the Lisp function
@code{message}) in a test or any of the code that it invoked
(@code{ert-results-pop-to-messages-for-test-at-point}).

@item L
@kindex L@r{, in ert results buffer}
@findex ert-results-toggle-printer-limits-for-test-at-point
By default, long expressions in the failure details are abbreviated
using @code{print-length} and @code{print-level}.  Increase the limits
to show more of the expression by moving point to a test failure with
this command (@code{ert-results-toggle-printer-limits-for-test-at-point}).

@item D
@kindex D@r{, in ert results buffer}
@findex ert-delete-test
@cindex delete test
Delete a test from the running Emacs session (@code{ert-delete-test}).

@item h
@kindex h@r{, in ert results buffer}
@findex ert-describe-test
Show the documentation of a test (@code{ert-describe-test}).

@item T
@kindex T@r{, in ert results buffer}
@findex ert-results-pop-to-timings
Display test timings for the last run (@code{ert-results-pop-to-timings}).

@item M-x ert-delete-all-tests
@findex ert-delete-all-tests
@cindex delete all tests
Delete all tests from the running session.

@item M-x ert-describe-test
@findex ert-results-describe-test-at-point
Prompt for a test and then show its documentation.

@end table


@node Running Tests in Batch Mode
@section Running Tests in Batch Mode
@cindex running tests in batch mode
@cindex batch-mode testing

@findex ert-run-tests-batch
@findex ert-run-tests-batch-and-exit
ERT supports automated invocations from the command line or from
scripts or makefiles.  There are two functions for this purpose,
@code{ert-run-tests-batch} and @code{ert-run-tests-batch-and-exit}.
They can be used like this:

@example
emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit
@end example

This command will start up Emacs in batch mode, load ERT, load
@file{my-tests.el}, and run all tests defined in it.  It will exit
with a zero exit status if all tests passed, or nonzero if any tests
failed or if anything else went wrong.  It will also print progress
messages and error diagnostics to standard output.

@findex ert-summarize-tests-batch-and-exit
You can also redirect the above output to a log file, say
@file{output.log}, and use the
@code{ert-summarize-tests-batch-and-exit} function to produce a neat
summary as shown below:

@example
emacs -batch -l ert -f ert-summarize-tests-batch-and-exit output.log
@end example

@vindex ert-batch-print-level
@vindex ert-batch-print-length
ERT attempts to limit the output size for failed tests by choosing
conservative values for @code{print-level} and @code{print-length}
when printing Lisp values.  This can in some cases make it difficult
to see which portions of those values are incorrect.  Use
@code{ert-batch-print-level} and @code{ert-batch-print-length}
to customize that:

@example
emacs -batch -l ert -l my-tests.el \
      --eval "(let ((ert-batch-print-level 10) \
                    (ert-batch-print-length 120)) \
                (ert-run-tests-batch-and-exit))"
@end example

@vindex ert-batch-backtrace-line-length
Even modest settings for @code{print-level} and @code{print-length}
can produce extremely long lines in backtraces, however, with
attendant pauses in execution progress.  Set
@code{ert-batch-backtrace-line-length} to @code{t} to use the value of
@code{backtrace-line-length}, @code{nil} to stop any limitations on
backtrace line lengths (that is, to get full backtraces), or a
positive integer to limit backtrace line length to that number.

@vindex ert-quiet
By default, ERT in batch mode is quite verbose, printing a line with
result after each test.  This gives you progress information: how many
tests have been executed and how many there are.  However, in some
cases this much output may be undesirable.  In this case, set
@code{ert-quiet} variable to a non-@code{nil} value:

@example
emacs -batch -l ert -l my-tests.el \
      --eval "(let ((ert-quiet t)) (ert-run-tests-batch-and-exit))"
@end example

In quiet mode ERT prints only unexpected results and summary.

You can specify selectors to only run a subset of your tests
(@pxref{Test Selectors}).  For example, the following would run all
tests where the name of the test matches the regular expression
``to-match''.

@example
emacs -batch -l ert -l my-tests.el \
      -eval '(ert-run-tests-batch-and-exit "to-match")'
@end example

@vindex EMACS_TEST_VERBOSE@r{, environment variable}
By default, ERT test failure summaries are quite brief in batch
mode---only the names of the failed tests are listed.  If the
@env{EMACS_TEST_VERBOSE} environment variable is set and is non-empty,
the failure summaries will also include the data from the failing
test.

@vindex EMACS_TEST_JUNIT_REPORT@r{, environment variable}
ERT can produce JUnit test reports in batch mode.  If the environment
variable @env{EMACS_TEST_JUNIT_REPORT} is set, ERT will produce for
every test package @file{my-tests.el} a corresponding JUnit test
report @file{my-tests.xml}.  The function
@code{ert-summarize-tests-batch-and-exit} collects all these package
test reports into a new JUnit test report, with the respective name of
that environment variable.


@node Test Selectors
@section Test Selectors
@cindex test selector
@cindex selecting tests

Functions like @code{ert} accept a @emph{test selector}, a Lisp
expression specifying a set of tests.  Test selector syntax is similar
to Common Lisp's type specifier syntax:

@itemize
@item @code{nil} selects no tests.
@item @code{t} selects all tests.
@item @code{:new} selects all tests that have not been run yet.
@item @code{:failed} and @code{:passed} select tests according to their most recent result.
@item @code{:expected}, @code{:unexpected} select tests according to their most recent result.
@item A string is a regular expression that selects all tests with matching names.
@item A test (i.e., an object of @code{ert-test} data type) selects that test.
@item A symbol selects the test that the symbol names.
@item @code{(member @var{tests}...)} selects the elements of
@var{tests}, a list of tests or symbols naming tests.
@item @code{(eql @var{test})} selects @var{test}, a test or a symbol
naming a test.
@item @code{(and @var{selectors}@dots{})} selects the tests that match
all @var{selectors}.
@item @code{(or @var{selectors}@dots{})} selects the tests that match
any of the @var{selectors}.
@item @code{(not @var{selector})} selects all tests that do not match
@var{selector}.
@item @code{(tag @var{tag})} selects all tests that have @var{tag} on
their tags list.
(Tags are optional labels you can apply to tests when you define them.)
@item @code{(satisfies @var{predicate})} selects all tests that
satisfy @var{predicate}, a function that takes a test as argument and
returns non-@code{nil} if it is selected.
@end itemize

Selectors that are frequently useful when selecting tests to run
include @code{t} to run all tests that are currently defined in Emacs,
@code{"^foo-"} to run all tests in package @code{foo} (this assumes
that package @code{foo} uses the prefix @code{foo-} for its test names),
result-based selectors such as @code{(or :new :unexpected)} to
run all tests that have either not run yet or that had an unexpected
result in the last run, and tag-based selectors such as @code{(not
(tag :causes-redisplay))} to run all tests that are not tagged
@code{:causes-redisplay}.


@node How to Write Tests
@chapter How to Write Tests
@cindex how to write tests

@findex ert-deftest
ERT lets you define tests in the same way you define functions.  You
can type @code{ert-deftest} forms in a buffer and evaluate them there
with @code{eval-defun} or @code{compile-defun}, or you can save the
file and load it, optionally byte-compiling it first.

Just like @code{find-function} is only able to find where a function
was defined if the function was loaded from a file, ERT is only able
to find where a test was defined if the test was loaded from a file.


@menu
* The @code{should} Macro::          A powerful way to express assertions.
* Expected Failures::           Tests for known bugs.
* Tests and Their Environment:: Don't depend on customizations; no side effects.
* Useful Techniques::           Some examples.
* erts files::                  Files containing many buffer tests.
* Syntax Highlighting Tests::   Tests for face assignment.
@end menu


@node The @code{should} Macro
@section The @code{should} Macro

@findex should@r{, ert macro}
Test bodies can include arbitrary code; but to be useful, they need to
check whether the code being tested (or @emph{code under test})
does what it is supposed to do.  The macro @code{should} is similar to
@code{cl-assert} from the cl package
(@pxref{Assertions,,, cl, Common Lisp Extensions}),
but analyzes its argument form and records information that ERT can
display to help debugging.

This test definition

@lisp
(ert-deftest addition-test ()
  (should (= (+ 1 2) 4)))
@end lisp

will produce this output when run via @kbd{M-x ert}:

@example
F addition-test
    (ert-test-failed
     ((should
       (=
        (+ 1 2)
        4))
      :form
      (= 3 4)
      :value nil))
@end example

In this example, @code{should} recorded the fact that (= (+ 1 2) 4)
reduced to (= 3 4) before it reduced to @code{nil}.  When debugging why the
test failed, it helps to know that the function @code{+} returned 3
here.  ERT records the return value for any predicate called directly
within @code{should}.

@findex should-not@r{, ert macro}
@findex should-error@r{, ert macro}
In addition to @code{should}, ERT provides @code{should-not}, which
checks that the predicate returns @code{nil}, and @code{should-error}, which
checks that the form called within it signals an error.  An example
use of @code{should-error}:

@lisp
(ert-deftest test-divide-by-zero ()
  (should-error (/ 1 0)
                :type 'arith-error))
@end lisp

This checks that dividing one by zero signals an error of type
@code{arith-error}.  The @code{:type} argument to @code{should-error}
is optional; if absent, any type of error is accepted.
@code{should-error} returns an error description of the error that was
signaled, to allow additional checks to be made.  The error
description has the format @code{(ERROR-SYMBOL . DATA)}.

There is no @code{should-not-error} macro since tests that signal an
error fail anyway, so @code{should-not-error} is effectively the
default.

@xref{Understanding Explanations}, for more details on what
@code{should} reports.


@node Expected Failures
@section Expected Failures
@cindex expected failures
@cindex known bugs

@vindex :expected-result
Some bugs are complicated to fix, or not very important, and are left as
@emph{known bugs}.  If there is a test case that triggers the bug and
fails, ERT will alert you of this failure every time you run all
tests.  For known bugs, this alert is a distraction.  The way to
suppress it is to add @code{:expected-result :failed} to the test
definition:

@lisp
(ert-deftest future-bug ()
  "Test `time-forward' with negative arguments.
Since this functionality isn't implemented, the test is known to fail."
  :expected-result :failed
  (time-forward -1))
@end lisp

ERT will still display a small @code{f} in the progress bar as a
reminder that there is a known bug, and will count the test as failed,
but it will be quiet about it otherwise.

An alternative to marking the test as a known failure this way is to
delete the test.  This is a good idea if there is no intent to fix it,
i.e., if the behavior that was formerly considered a bug has become an
accepted feature.

In general, however, it can be useful to keep tests that are known to
fail.  If someone wants to fix the bug, they will have a very good
starting point: an automated test case that reproduces the bug.  This
makes it much easier to fix the bug, demonstrate that it is fixed, and
prevent future regressions.

ERT displays the same kind of alerts for tests that pass unexpectedly
as it displays for unexpected failures.  This way, if you make code
changes that happen to fix a bug that you weren't aware of, you will
know to remove the @code{:expected-result} clause of that test and
close the corresponding bug report, if any.

Since @code{:expected-result} evaluates its argument when the test is
loaded, tests can be marked as known failures only on certain Emacs
versions, specific architectures, etc.:

@lisp
(ert-deftest foo ()
  "A test that is expected to fail on Emacs 23 but succeed elsewhere."
  :expected-result (if (string-match "GNU Emacs 23[.]" (emacs-version))
                       :failed
                     :passed)
  ...)
@end lisp


@node Tests and Their Environment
@section Tests and Their Environment

@cindex skipping tests
@cindex test preconditions
@cindex preconditions of a test
@findex skip-when
@findex skip-unless
Sometimes, it doesn't make sense to run a test due to missing
preconditions.  A required Emacs feature might not be compiled in, the
function to be tested could call an external binary which might not be
available on the test machine, you name it.  In this case, the macros
@code{skip-when} or @code{skip-unless} could be used to skip the
test.@footnote{The @code{skip-when} macro was added in Emacs 30.1.  If
you need your tests to be compatible with older versions of Emacs, use
@code{skip-unless} instead.}

@noindent
For example, this test is skipped on MS-Windows and macOS:

@lisp
(ert-deftest test-gnu-linux ()
  "A test that is not relevant on MS-Windows and macOS."
  (skip-when (memq system-type '(windows-nt ns))
  ...))
@end lisp

@noindent
This test is skipped if the feature @samp{dbusbind} is not present in
the running Emacs:

@lisp
(ert-deftest test-dbus ()
  "A test that checks D-BUS functionality."
  (skip-unless (featurep 'dbusbind))
  ...)
@end lisp

@cindex tests and their environment
The outcome of running a test should not depend on the current state
of the environment, and each test should leave its environment in the
same state it found it in.  In particular, a test should not depend on
any Emacs customization variables or hooks, and if it has to make any
changes to Emacs's state or state external to Emacs (such as the file
system), it should undo these changes before it returns, regardless of
whether it passed or failed.

Tests should not depend on the environment because any such
dependencies can make the test brittle or lead to failures that occur
only under certain circumstances and are hard to reproduce.  Of
course, the code under test may have settings that affect its
behavior.  In that case, it is best to make the test @code{let}-bind
all such setting variables to set up a specific configuration for the
duration of the test.  The test can also set up a number of different
configurations and run the code under test with each.

Tests that have side effects on their environment should restore it to
its original state because any side effects that persist after the
test can disrupt the workflow of the programmer running the tests.  If
the code under test has side effects on Emacs's current state, such as
on the current buffer or window configuration, the test should create
a temporary buffer for the code to manipulate (using
@code{with-temp-buffer}), or save and restore the window configuration
(using @code{save-window-excursion}), respectively.  For aspects of
the state that can not be preserved with such macros, cleanup should
be performed with @code{unwind-protect}, to ensure that the cleanup
occurs even if the test fails.

An exception to this are messages that the code under test prints with
@code{message} and similar logging; tests should not bother restoring
the @file{*Message*} buffer to its original state.

The above guidelines imply that tests should avoid calling highly
customizable commands such as @code{find-file}, except, of course, if
such commands are what they want to test.  The exact behavior of
@code{find-file} depends on many settings such as
@code{find-file-wildcards}, @code{enable-local-variables}, and
@code{auto-mode-alist}.  It is difficult to write a meaningful test if
its behavior can be affected by so many external factors.  Also,
@code{find-file} has side effects that are hard to predict and thus
hard to undo: It may create a new buffer or reuse an existing
buffer if one is already visiting the requested file; and it runs
@code{find-file-hook}, which can have arbitrary side effects.

Instead, it is better to use lower-level mechanisms with simple and
predictable semantics like @code{with-temp-buffer}, @code{insert} or
@code{insert-file-contents-literally}, and to activate any desired mode
by calling the corresponding function directly, after binding the
hook variables to @code{nil}.  This avoids the above problems.


@node Useful Techniques
@section Useful Techniques when Writing Tests
@cindex useful techniques
@cindex tips and tricks

Testing simple functions that have no side effects and no dependencies
on their environment is easy.  Such tests often look like this:

@lisp
(ert-deftest ert-test-mismatch ()
  (should (eql (cl-mismatch "" "") nil))
  (should (eql (cl-mismatch "" "a") 0))
  (should (eql (cl-mismatch "a" "a") nil))
  (should (eql (cl-mismatch "ab" "a") 1))
  (should (eql (cl-mismatch "Aa" "aA") 0))
  (should (eql (cl-mismatch '(a b c) '(a b d)) 2)))
@end lisp

This test calls the function @code{cl-mismatch} several times with
various combinations of arguments and compares the return value to the
expected return value.  (Some programmers prefer @code{(should (eql
EXPECTED ACTUAL))} over the @code{(should (eql ACTUAL EXPECTED))}
shown here.  ERT works either way.)

Here's a more complicated test:

@lisp
(ert-deftest ert-test-record-backtrace ()
  (let ((test (make-ert-test :body (lambda () (ert-fail "foo")))))
    (let ((result (ert-run-test test)))
      (should (ert-test-failed-p result))
      (with-temp-buffer
        (ert--print-backtrace (ert-test-failed-backtrace result))
        (goto-char (point-min))
        (end-of-line)
        (let ((first-line (buffer-substring-no-properties
                           (point-min) (point))))
          (should (equal first-line
                         "  signal(ert-test-failed (\"foo\"))")))))))
@end lisp

@findex make-ert-test
This test creates a test object using @code{make-ert-test} whose body
will immediately signal failure.  It then runs that test and asserts
that it fails.  Then, it creates a temporary buffer and invokes
@code{ert--print-backtrace} to print the backtrace of the failed test
to the current buffer.  Finally, it extracts the first line from the
buffer and asserts that it matches what we expect.  It uses
@code{buffer-substring-no-properties} and @code{equal} to ignore text
properties; for a test that takes properties into account,
@code{buffer-substring} and @code{equal-including-properties}
could be used instead.

The reason why this test only checks the first line of the backtrace
is that the remainder of the backtrace is dependent on ERT's internals
as well as whether the code is running interpreted or compiled.  By
looking only at the first line, the test checks a useful property---that
the backtrace correctly captures the call to @code{signal} that
results from the call to @code{ert-fail}---without being brittle.

This example also shows that writing tests is much easier if the code
under test was structured with testing in mind.

For example, if @code{ert-run-test} accepted only symbols that name
tests rather than test objects, the test would need a name for the
failing test, which would have to be a temporary symbol generated with
@code{make-symbol}, to avoid side effects on Emacs's state.  Choosing
the right interface for @code{ert-run-tests} allows the test to be
simpler.

Similarly, if @code{ert--print-backtrace} printed the backtrace to a
buffer with a fixed name rather than the current buffer, it would be
much harder for the test to undo the side effect.  Of course, some
code somewhere needs to pick the buffer name.  But that logic is
independent of the logic that prints backtraces, and keeping them in
separate functions allows us to test them independently.

A lot of code that you will encounter in Emacs was not written with
testing in mind.  Sometimes, the easiest way to write tests for such
code is to restructure the code slightly to provide better interfaces
for testing.  Usually, this makes the interfaces easier to use as
well.


@node erts files
@section erts files

@findex ert-test-erts-file
Many relevant Emacs tests depend on comparing the contents of a buffer
before and after executing a particular function.  These tests can be
written the normal way---making a temporary buffer, inserting the
``before'' text, running the function, and then comparing with the
expected ``after'' text.  However, this often leads to test code
that's pretty difficult to read and write, especially when the text in
question is multi-line.

So ert provides a function called @code{ert-test-erts-file} that takes
two parameters: the name of a specially-formatted @dfn{erts} file, and
(optionally) a function that performs the transform.

@findex erts-mode
These erts files can be edited with the @code{erts-mode} major mode.

An erts file is divided into sections by the (@samp{=-=}) separator.

Here's an example file containing two tests:

@example
Name: flet

=-=
(cl-flet ((bla (x)
(* x x)))
(bla 42))
=-=
(cl-flet ((bla (x)
            (* x x)))
  (bla 42))
=-=-=

Name: defun

=-=
(defun x ()
  (print (quote ( thingy great
                  stuff))))
=-=-=
@end example

A test starts with a line containing just @samp{=-=} and ends with a
line containing just @samp{=-=-=}.  The test may be preceded by
freeform text (for instance, comments), and also name/value pairs (see
below for a list of them).

If there is a line with @samp{=-=} inside the test, that designates
the start of the ``after'' text.  Otherwise, the ``before'' and
``after'' texts are assumed to be identical, which you typically see
when writing indentation tests.

@code{ert-test-erts-file} puts the ``before'' section into a temporary
buffer, calls the transform function, and then compares with the
``after'' section.

Here's an example usage:

@lisp
(ert-test-erts-file "elisp.erts"
                    (lambda ()
                      (emacs-lisp-mode)
                      (indent-region (point-min) (point-max))))
@end lisp

A list of the name/value specifications that can appear before a test
follows.  The general syntax is @samp{Name: Value}, but continuation
lines can be used (along the same lines as in mail---subsequent lines
that start with a space are part of the value).

@example
Name: foo
Code: (indent-region
        (point-min) (point-max))
@end example

@table @samp
@item Name
All tests should have a name.  This name will appear in ERT output if
the test fails, and helps to identify the failing test.

@item Code
This is the code that will be run to do the transform.  This can also
be passed in via the @code{ert-test-erts-file} call, but @samp{Code}
overrides that.  It's used not only in the following test, but in all
subsequent tests in the file (until overridden by another @samp{Code}
specification).

@item No-Before-Newline
@itemx No-After-Newline
These specifications say whether the ``before'' or ``after'' portions
have a newline at the end.  (This would otherwise be impossible to
specify.)

@item Point-Char
Sometimes it's useful to be able to put point at a specific place
before executing the transform function.  @samp{Point-Char: |} will
make @code{ert-test-erts-file} place point where @samp{|} is in the
``before'' form (and remove that character), and will check that it's
where the @samp{|} character is in the ``after'' form (and issue a
test failure if that isn't the case).  (This is used in all subsequent
tests, unless overridden by a new @samp{Point-Char} spec.)

@item Skip
If this is present and value is a form that evaluates to a
non-@code{nil} value, the test will be skipped.
@end table

If you need to use the literal line single line @samp{=-=} in a test
section, you can quote it with a @samp{\} character.

@node Syntax Highlighting Tests
@section Syntax Highlighting Tests

Syntax highlighting is normally provided by the Font Lock minor mode
that assigns face properties to parts of the buffer.  The
@code{ert-font-lock} package makes it possible to introduce unit tests
checking face assignment.  Test assertions are included in code-level
comments directly and can be read either from inline strings or files.
The parser expects the input string to contain at least one assertion.

Test assertion parser extracts tests from comment-only lines.  Every
comment assertion line starts either with a caret (@samp{^}) or an arrow
(@samp{<-}).  A single caret/arrow or carets should be followed
immediately by the name of a face or a list of faces to be checked
against the @code{:face} property at point.

The test then checks if the first non-assertion column above the caret
contains a face expected by the assertion:

@example
var variable = 11;
//   ^ font-lock-variable-name-face
//             ^ font-lock-literal-face
//               ^ font-lock-punctuation-face
// this is not an assertion, it's just a comment
//   ^ font-lock-comment-face

// multiple carets per line
// ^^^^     ^    ^ font-lock-comment-face
@end example

Both symbol-only @code{:face} property values and assertion face values
are normalized to single element lists so assertions below are
equivalent:

@example
// single
// ^ font-lock-comment-face
// single
// ^ (font-lock-comment-face)
@end example

Assertions can be negated:

@example
var variable = 11;
//  ^ !font-lock-comment-face
@end example

It is possible to specify face lists in assertions:

@example
// TODO
// ^^^^ (font-lock-comment-face hl-todo)
     var test = 1;
// ^    ()
// ^    nil
//   negation works as expected
//   ^  !nil
@end example

The arrow (@samp{<-}) means that the first non-empty column of the
assertion line will be used for the check:

@example
var variable = 1;
// <- font-lock-keyword-face
  11;
   // <- font-lock-literal-face
@end example

@findex ert-font-lock-test-string

The @code{ert-font-lock-test-string} function extracts ERT assertions
from an inline string.  The @code{javascript-mode} symbol below
specifies the major mode used for comments and font locking:

@lisp
(ert-deftest test-font-lock-test-string--correct ()
  (ert-font-lock-test-string
   "
var abc = function(d) @{
// <- font-lock-keyword-face
//   ^ font-lock-variable-name-face
    //        ^ font-lock-keyword-face
    //             ^ font-lock-variable-name-face
@};
"
   'javascript-mode))
@end lisp

@findex ert-font-lock-test-file

It is also possible to extract test assertions from a file:

@lisp
(ert-deftest test-font-lock-test-file--correct ()
  (ert-font-lock-test-file
   (ert-resource-file "correct.js")
   'javascript-mode))
@end lisp

@findex ert-font-lock-deftest

The @code{ert-font-lock-deftest} macro simplifies inline test
definition:

@lisp
(ert-font-lock-deftest test-macro-test--inline
    emacs-lisp-mode
  "
(defun fun ())
;; ^ font-lock-keyword-face
;;      ^ font-lock-function-name-face")
@end lisp

@findex ert-font-lock-deftest-file

The @code{ert-font-lock-deftest-file} macro reads assertions from a
file:

@lisp
(ert-font-lock-deftest-file test-macro-test--file
    "Test reading correct assertions from a file"
  javascript-mode
  "correct.js")
@end lisp

The @code{ert-font-lock-deftest} and @code{ert-font-lock-deftest-file}
macros accept the same keyword parameters as @code{ert-deftest} i.e.,
@code{:tag} and @code{:expected-result}.

@node How to Debug Tests
@chapter How to Debug Tests

This section describes how to use ERT's features to understand why
a test failed.


@menu
* Understanding Explanations::  How ERT gives details on why an assertion failed.
* Interactive Debugging::       Tools available in the ERT results buffer.
@end menu


@node Understanding Explanations
@section Understanding Explanations
@cindex understanding explanations
@cindex explanations, understanding

Failed @code{should} forms are reported like this:

@example
F addition-test
    (ert-test-failed
     ((should
       (=
        (+ 1 2)
        4))
      :form
      (= 3 4)
      :value nil))
@end example

ERT shows what the @code{should} expression looked like and what
values its subexpressions had: The source code of the assertion was
@code{(should (= (+ 1 2) 4))}, which applied the function @code{=} to
the arguments @code{3} and @code{4}, resulting in the value
@code{nil}.  In this case, the test is wrong; it should expect 3
rather than 4.

If a predicate like @code{equal} is used with @code{should}, ERT
provides a so-called @emph{explanation}:

@example
F list-test
    (ert-test-failed
     ((should
       (equal
        (list 'a 'b 'c)
        '(a b d)))
      :form
      (equal
       (a b c)
       (a b d))
      :value nil :explanation
      (list-elt 2
                (different-atoms c d))))
@end example

In this case, the function @code{equal} was applied to the arguments
@code{(a b c)} and @code{(a b d)}.  ERT's explanation shows that
the item at index 2 differs between the two lists; in one list, it is
the atom c, in the other, it is the atom d.

In simple examples like the above, the explanation is unnecessary.
But in cases where the difference is not immediately apparent, it can
save time:

@example
F test1
    (ert-test-failed
     ((should
       (equal x y))
      :form
      (equal a a)
      :value nil :explanation
      (different-symbols-with-the-same-name a a)))
@end example

ERT only provides explanations for predicates that have an explanation
function registered.  @xref{Defining Explanation Functions}.


@node Interactive Debugging
@section Interactive Debugging
@cindex interactive debugging
@cindex debugging failed tests

Debugging failed tests essentially works the same way as debugging any
other problems with Lisp code.  Here are a few tricks specific to
tests:

@itemize
@cindex re-running a failed test
@item
Re-run the failed test a few times to see if it fails in the same way
each time.  It's good to find out whether the behavior is
deterministic before spending any time looking for a cause.  In the
ERT results buffer, @kbd{r} re-runs the selected test.

@cindex jump to the test source code
@item
Use @kbd{.} to jump to the source code of the test to find out exactly
what it does.  Perhaps the test is broken rather than the code
under test.

@item
If the test contains a series of @code{should} forms and you can't
tell which one failed, use @kbd{l}, which shows you the list of all
@code{should} forms executed during the test before it failed.

@cindex show backtrace of failed test
@item
Use @kbd{b} to view the backtrace.  You can also use @kbd{d} to re-run
the test with debugging enabled, this will enter the debugger and show
the backtrace as well; but the top few frames shown there will not be
relevant to you since they are ERT's own debugger hook.  @kbd{b}
strips them out, so it is more convenient.

@item
If the test or the code under testing prints messages using
@code{message}, use @kbd{m} to see what messages it printed before it
failed.  This can be useful to figure out how far it got.

@cindex instrumenting test for Edebug
@item
You can instrument tests for debugging the same way you instrument
@code{defun}s for debugging: go to the source code of the test and
type @kbd{C-u C-M-x}.  Then, go back to the ERT buffer and
re-run the test with @kbd{r} or @kbd{d}.

@cindex discard obsolete test results
@item
If you have been editing and rearranging tests, it is possible that
ERT remembers an old test that you have since renamed or removed:
renamings or removals of definitions in the source code leave around a
stray definition under the old name in the running process (this is a
common problem in Lisp).  In such a situation, hit @kbd{D} to let ERT
forget about the obsolete test.
@end itemize


@node Extending ERT
@chapter Extending ERT
@cindex extending ert

There are several ways to add functionality to ERT.

@menu
* Defining Explanation Functions::  Teach ERT about more predicates.
* Low-Level Functions for Working with Tests::  Use ERT's data for your purposes.
@end menu


@node Defining Explanation Functions
@section Defining Explanation Functions
@cindex defining explanation functions

The explanation function for a predicate is a function that takes the
same arguments as the predicate and returns an @emph{explanation}.
The explanation should explain why the predicate, when invoked with
the arguments given to the explanation function, returns the value
that it returns.  The explanation can be any object but should have a
comprehensible printed representation.  If the return value of the
predicate needs no explanation for a given list of arguments, the
explanation function should return @code{nil}.

@vindex ert-explainer@r{, property}
To associate an explanation function with a predicate, add the
property @code{ert-explainer} to the symbol that names the predicate.
The value of the property should be the symbol that names the
explanation function.


@node Low-Level Functions for Working with Tests
@section Low-Level Functions for Working with Tests
@cindex low-level functions

Both @code{ert-run-tests-interactively} and @code{ert-run-tests-batch}
are implemented on top of the lower-level test handling code in the
sections of @file{ert.el} labeled ``Facilities for running a single test'',
``Test selectors'', and ``Facilities for running a whole set of tests''.

If you want to write code that works with ERT tests, you should take a
look at this lower-level code.  Symbols that start with @code{ert--}
are internal to ERT, whereas those that start with @code{ert-} are
meant to be usable by other code.  But there is no mature API yet.

Contributions to ERT are welcome.


@node Other Testing Concepts
@chapter Other Testing Concepts

For information on mocks, stubs, fixtures, or test suites, see below.


@menu
* Mocks and Stubs::           Stubbing out code that is irrelevant to the test.
* Fixtures and Test Suites::  How ERT differs from tools for other languages.
@end menu


@node Mocks and Stubs
@section Other Tools for Emacs Lisp
@cindex mocks and stubs

Stubbing out functions or using so-called @emph{mocks} can make it
easier to write tests.  See
@url{https://en.wikipedia.org/wiki/Mock_object} for an explanation of
the corresponding concepts in object-oriented languages.

ERT does not have built-in support for mocks or stubs.  The package
@code{el-mock} (see @url{https://www.emacswiki.org/emacs/el-mock.el})
offers mocks for Emacs Lisp and can be used in conjunction with ERT.


@node Fixtures and Test Suites
@section Fixtures and Test Suites
@cindex fixtures

In many ways, ERT is similar to frameworks for other languages like
SUnit or JUnit.  However, two features commonly found in such
frameworks are notably absent from ERT: fixtures and test suites.

Fixtures are mainly used (e.g., in SUnit or JUnit) to provide an
environment for a set of tests, and consist of set-up and tear-down
functions.

While fixtures are a useful syntactic simplification in other
languages, this does not apply to Lisp, where higher-order functions
and @code{unwind-protect} are available.  One way to implement and use a
fixture in ERT is

@lisp
(defun my-fixture (body)
  (unwind-protect
      (progn [set up]
             (funcall body))
    [tear down]))

(ert-deftest my-test ()
  (my-fixture
   (lambda ()
     [test code])))
@end lisp

(Another way would be a @code{with-my-fixture} macro.)  This solves
the set-up and tear-down part, and additionally allows any test
to use any combination of fixtures, so it is more flexible than what
other tools typically allow.

If the test needs access to the environment the fixture sets up, the
fixture can be modified to pass arguments to the body.

These are well-known Lisp techniques.  Special syntax for them could
be added but would provide only a minor simplification.

(If you are interested in such syntax, note that splitting set-up and
tear-down into separate functions, like *Unit tools usually do, makes
it impossible to establish dynamic @code{let} bindings as part of the
fixture.  So, blindly imitating the way fixtures are implemented in
other languages would be counter-productive in Lisp.)

The purpose of test suites is to group related tests together.

The most common use of this is to run just the tests for one
particular module.  Since symbol prefixes are the usual way of
separating module namespaces in Emacs Lisp, test selectors already
solve this by allowing regexp matching on test names; e.g., the
selector @code{"^ert-"} selects ERT's self-tests.

Other uses include grouping tests by their expected execution time,
e.g., to run quick tests during interactive development and slow tests less
often.  This can be achieved with the @code{:tag} argument to
@code{ert-deftest} and @code{tag} test selectors.


@node Index
@unnumbered Index

@printindex cp


@node GNU Free Documentation License
@appendix GNU Free Documentation License
@include doclicense.texi

@bye

@c  LocalWords:  ERT JUnit namespace docstring ERT's
@c  LocalWords:  backtrace makefiles workflow backtraces API SUnit
@c  LocalWords:  subexpressions