| 1 | # CLI tests |
| 2 | |
| 3 | The CLI tests are focused on testing the zstd CLI. |
| 4 | They are intended to be simple tests that the CLI and arguments work as advertised. |
| 5 | They are not intended to test the library, only the code in `programs/`. |
| 6 | The library will get incidental coverage, but if you find yourself trying to trigger a specific condition in the library, this is the wrong tool. |
| 7 | |
| 8 | ## Test runner usage |
| 9 | |
| 10 | The test runner `run.py` will run tests against the in-tree build of `zstd` and `datagen` by default. Which means that `zstd` and `datagen` must be built. |
| 11 | |
| 12 | The `zstd` binary used can be passed with `--zstd /path/to/zstd`. |
| 13 | Additionally, to run `zstd` through a tool like `valgrind` or `qemu`, set the `--exec-prefix 'valgrind -q'` flag. |
| 14 | |
| 15 | Similarly, the `--datagen`, and `--zstdgrep` flags can be set to specify |
| 16 | the paths to their respective binaries. However, these tools do not use |
| 17 | the `EXEC_PREFIX`. |
| 18 | |
| 19 | Each test executes in its own scratch directory under `scratch/test/name`. E.g. `scratch/basic/help.sh/`. Normally these directories are removed after the test executes. However, the `--preserve` flag will preserve these directories after execution, and save the tests exit code, stdout, and stderr in the scratch directory to `exit`, `stderr`, and `stdout` respectively. This can be useful for debugging/editing a test and updating the expected output. |
| 20 | |
| 21 | ### Running all the tests |
| 22 | |
| 23 | By default the test runner `run.py` will run all the tests, and report the results. |
| 24 | |
| 25 | Examples: |
| 26 | |
| 27 | ``` |
| 28 | ./run.py |
| 29 | ./run.py --preserve |
| 30 | ./run.py --zstd ../../build/programs/zstd --datagen ../../build/tests/datagen |
| 31 | ``` |
| 32 | |
| 33 | ### Running specific tests |
| 34 | |
| 35 | A set of test names can be passed to the test runner `run.py` to only execute those tests. |
| 36 | This can be useful for writing or debugging a test, especially with `--preserve`. |
| 37 | |
| 38 | The test name can either be the path to the test file, or the test name, which is the path relative to the test directory. |
| 39 | |
| 40 | Examples: |
| 41 | |
| 42 | ``` |
| 43 | ./run.py basic/help.sh |
| 44 | ./run.py --preserve basic/help.sh basic/version.sh |
| 45 | ./run.py --preserve --verbose basic/help.sh |
| 46 | ``` |
| 47 | |
| 48 | ### Updating exact output |
| 49 | |
| 50 | If a test is failing because a `.stderr.exact` or `.stdout.exact` no longer matches, you can re-run the tests with `--set-exact-output` and the correct output will be written. |
| 51 | |
| 52 | Example: |
| 53 | ``` |
| 54 | ./run.py --set-exact-output |
| 55 | ./run.py basic/help.sh --set-exact-output |
| 56 | ``` |
| 57 | |
| 58 | ## Writing a test |
| 59 | |
| 60 | Test cases are arbitrary executables, and can be written in any language, but are generally shell scripts. |
| 61 | After the script executes, the exit code, stderr, and stdout are compared against the expectations. |
| 62 | |
| 63 | Each test is run in a clean directory that the test can use for intermediate files. This directory will be cleaned up at the end of the test, unless `--preserve` is passed to the test runner. Additionally, the `setup` script can prepare the directory before the test runs. |
| 64 | |
| 65 | ### Calling zstd, utilities, and environment variables |
| 66 | |
| 67 | The `$PATH` for tests is prepended with the `bin/` sub-directory, which contains helper scripts for ease of testing. |
| 68 | The `zstd` binary will call the zstd binary specified by `run.py` with the correct `$EXEC_PREFIX`. |
| 69 | Similarly, `datagen`, `unzstd`, `zstdgrep`, `zstdcat`, etc, are provided. |
| 70 | |
| 71 | Helper utilities like `cmp_size`, `println`, and `die` are provided here too. See their scripts for details. |
| 72 | |
| 73 | Common shell script libraries are provided under `common/`, with helper variables and functions. They can be sourced with `source "$COMMON/library.sh`. |
| 74 | |
| 75 | Lastly, environment variables are provided for testing, which can be listed when calling `run.py` with `--verbose`. |
| 76 | They are generally used by the helper scripts in `bin/` to coordinate everything. |
| 77 | |
| 78 | ### Basic test case |
| 79 | |
| 80 | When executing your `$TEST` executable, by default the exit code is expected to be `0`. However, you can provide an alternate expected exit code in a `$TEST.exit` file. |
| 81 | |
| 82 | When executing your `$TEST` executable, by default the expected stderr and stdout are empty. However, you can override the default by providing one of three files: |
| 83 | |
| 84 | * `$TEST.{stdout,stderr}.exact` |
| 85 | * `$TEST.{stdout,stderr}.glob` |
| 86 | * `$TEST.{stdout,stderr}.ignore` |
| 87 | |
| 88 | If you provide a `.exact` file, the output is expected to exactly match, byte-for-byte. |
| 89 | |
| 90 | If you provide a `.glob` file, the output is expected to match the expected file, where each line is interpreted as a glob syntax. Additionally, a line containing only `...` matches all lines until the next expected line matches. |
| 91 | |
| 92 | If you provide a `.ignore` file, the output is ignored. |
| 93 | |
| 94 | #### Passing examples |
| 95 | |
| 96 | All these examples pass. |
| 97 | |
| 98 | Exit 1, and change the expectation to be 1. |
| 99 | |
| 100 | ``` |
| 101 | exit-1.sh |
| 102 | --- |
| 103 | #!/bin/sh |
| 104 | exit 1 |
| 105 | --- |
| 106 | |
| 107 | exit-1.sh.exit |
| 108 | --- |
| 109 | 1 |
| 110 | --- |
| 111 | ``` |
| 112 | |
| 113 | Check the stdout output exactly matches. |
| 114 | |
| 115 | ``` |
| 116 | echo.sh |
| 117 | --- |
| 118 | #!/bin/sh |
| 119 | echo "hello world" |
| 120 | --- |
| 121 | |
| 122 | echo.sh.stdout.exact |
| 123 | --- |
| 124 | hello world |
| 125 | --- |
| 126 | ``` |
| 127 | |
| 128 | Check the stderr output using a glob. |
| 129 | |
| 130 | ``` |
| 131 | random.sh |
| 132 | --- |
| 133 | #!/bin/sh |
| 134 | head -c 10 < /dev/urandom | xxd >&2 |
| 135 | --- |
| 136 | |
| 137 | random.sh.stderr.glob |
| 138 | --- |
| 139 | 00000000: * * * * * * |
| 140 | ``` |
| 141 | |
| 142 | Multiple lines can be matched with ... |
| 143 | |
| 144 | ``` |
| 145 | random-num-lines.sh |
| 146 | --- |
| 147 | #!/bin/sh |
| 148 | echo hello |
| 149 | seq 0 $RANDOM |
| 150 | echo world |
| 151 | --- |
| 152 | |
| 153 | random-num-lines.sh.stdout.glob |
| 154 | --- |
| 155 | hello |
| 156 | 0 |
| 157 | ... |
| 158 | world |
| 159 | --- |
| 160 | ``` |
| 161 | |
| 162 | #### Failing examples |
| 163 | |
| 164 | Exit code is expected to be 0, but is 1. |
| 165 | |
| 166 | ``` |
| 167 | exit-1.sh |
| 168 | --- |
| 169 | #!/bin/sh |
| 170 | exit 1 |
| 171 | --- |
| 172 | ``` |
| 173 | |
| 174 | Stdout is expected to be empty, but isn't. |
| 175 | |
| 176 | ``` |
| 177 | echo.sh |
| 178 | --- |
| 179 | #!/bin/sh |
| 180 | echo hello world |
| 181 | ``` |
| 182 | |
| 183 | Stderr is expected to be hello but is world. |
| 184 | |
| 185 | ``` |
| 186 | hello.sh |
| 187 | --- |
| 188 | #!/bin/sh |
| 189 | echo world >&2 |
| 190 | --- |
| 191 | |
| 192 | hello.sh.stderr.exact |
| 193 | --- |
| 194 | hello |
| 195 | --- |
| 196 | ``` |
| 197 | |
| 198 | ### Setup & teardown scripts |
| 199 | |
| 200 | Finally, test writing can be eased with setup and teardown scripts. |
| 201 | Each directory in the test directory is a test-suite consisting of all tests within that directory (but not sub-directories). |
| 202 | This test suite can come with 4 scripts to help test writing: |
| 203 | |
| 204 | * `setup_once` |
| 205 | * `teardown_once` |
| 206 | * `setup` |
| 207 | * `teardown` |
| 208 | |
| 209 | The `setup_once` and `teardown_once` are run once before and after all the tests in the suite respectively. |
| 210 | They operate in the scratch directory for the test suite, which is the parent directory of each scratch directory for each test case. |
| 211 | They can do work that is shared between tests to improve test efficiency. |
| 212 | For example, the `dictionaries/setup_once` script builds several dictionaries, for use in the `dictionaries` tests. |
| 213 | |
| 214 | The `setup` and `teardown` scripts run before and after each test case respectively, in the test case's scratch directory. |
| 215 | These scripts can do work that is shared between test cases to make tests more succinct. |
| 216 | For example, the `dictionaries/setup` script copies the dictionaries built by the `dictionaries/setup_once` script into the test's scratch directory, to make them easier to use, and make sure they aren't accidentally modified. |
| 217 | |
| 218 | #### Examples |
| 219 | |
| 220 | ``` |
| 221 | basic/setup |
| 222 | --- |
| 223 | #!/bin/sh |
| 224 | # Create some files for testing with |
| 225 | datagen > file |
| 226 | datagen > file0 |
| 227 | datagen > file1 |
| 228 | --- |
| 229 | |
| 230 | basic/test.sh |
| 231 | --- |
| 232 | #!/bin/sh |
| 233 | zstd file file0 file1 |
| 234 | --- |
| 235 | |
| 236 | dictionaries/setup_once |
| 237 | --- |
| 238 | #!/bin/sh |
| 239 | set -e |
| 240 | |
| 241 | mkdir files/ dicts/ |
| 242 | for i in $(seq 10); do |
| 243 | datagen -g1000 > files/$i |
| 244 | done |
| 245 | |
| 246 | zstd --train -r files/ -o dicts/0 |
| 247 | --- |
| 248 | |
| 249 | dictionaries/setup |
| 250 | --- |
| 251 | #!/bin/sh |
| 252 | |
| 253 | # Runs in the test case's scratch directory. |
| 254 | # The test suite's scratch directory that |
| 255 | # `setup_once` operates in is the parent directory. |
| 256 | cp -r ../files ../dicts . |
| 257 | --- |
| 258 | ``` |