QB64-PE/docs/testing.md

QB64-PE Testing Framework
=========================

QB64-PE features two separate testing frameworks, one for QB64-based tests
(mainly for the compiler and language functionality), and one for C++-based
tests (for `libqb`).

QB64-based Tests
----------------

The QB64-based tests live in `./tests/compile_tests`. Each folder in that
directory represents a different category of tests, and within each of those
category folders are `*.bas` files. Accompanying each `*.bas` file is either an
`*.output` or `*.err` file with the same base name as a coresponding `.bas`
file.  For test with an `*.output`, the `*.output` file contains the text that
the program compiled from the `*.bas` file should produce. For `*.err` tests,
the `*.err` file contains the text of the error the QB64-PE compiler should
produce when attempting to compile that code.

Tests can also have an optional `*.flags` file. The contents of this file will
be provide as command line arguments to QB64-PE when compiling the test source.

Tests are compiled from the directory the `.bas` file is in. That is, they are
compiled as though the user `cd`'d into the directory the `.bas` file is in,
and then ran the compiler from that starting location to compile the `.bas`. If
necessary this behavior can be changed by including a `*.compile-from-base`
file next to the associated `*.bas` test file. This will case the test script
to compile the `*.bas` file from the directory the compiler is located in
instead, and the full path to the `*.bas` file will be provided to `-x`.

These tests are run via the `./tests/compile_tests.sh` script. The script
searches the `./tests/compile_tests` directory for any `*.bas` files, and then
uses the provide QB64-PE compiler to test them.

C++-based Tests
---------------

The C++-based testing lives in `./tests/c/`. The tests themselves exist as
`.cpp` files that compile into individual programs (each has its own `main()`).
The tests make use of `test.h` to run their tests and give proper output, and
failure is reported via the exit code.

The C++ tests are compiled using the `./tests/build.mk` make build logic. It's
fairly straight forward, each test executable gets its own name and then its
own combination of source/object files necessary to produce it. If you're
testing functionality from `libqb`, then you need to link the proper object
files into your test.  Additionally, any extra compiler flags can be defined
individually for each test.

`./tests/run_c_tests.sh` is a script that runs all of the C++ tests during the
build. Internally it has a list of the names of all the test executables (to
avoid having to use `make` to discover them). It expects the tests to have
already been built via a `make build-tests` call, and runs the executables one
at a time and checks their exit code for errors.

QBasic Testcases
----------------

The QBasic testcases are a selection of QBasic programs that live in
`./tests/qb64_testcases`. During the build process all of these programs are
tested to verify that QB64-PE is still capable of compiling them with no
errors. The behavior of the compiled programs is not verified.