648db22b |
1 | # CLI tests |
2 | |
3 | The CLI tests are focused on testing the zstd CLI. |
4 | They are intended to be simple tests that the CLI and arguments work as advertised. |
5 | They are not intended to test the library, only the code in `programs/`. |
6 | The library will get incidental coverage, but if you find yourself trying to trigger a specific condition in the library, this is the wrong tool. |
7 | |
8 | ## Test runner usage |
9 | |
10 | The test runner `run.py` will run tests against the in-tree build of `zstd` and `datagen` by default. Which means that `zstd` and `datagen` must be built. |
11 | |
12 | The `zstd` binary used can be passed with `--zstd /path/to/zstd`. |
13 | Additionally, to run `zstd` through a tool like `valgrind` or `qemu`, set the `--exec-prefix 'valgrind -q'` flag. |
14 | |
15 | Similarly, the `--datagen`, and `--zstdgrep` flags can be set to specify |
16 | the paths to their respective binaries. However, these tools do not use |
17 | the `EXEC_PREFIX`. |
18 | |
19 | Each test executes in its own scratch directory under `scratch/test/name`. E.g. `scratch/basic/help.sh/`. Normally these directories are removed after the test executes. However, the `--preserve` flag will preserve these directories after execution, and save the tests exit code, stdout, and stderr in the scratch directory to `exit`, `stderr`, and `stdout` respectively. This can be useful for debugging/editing a test and updating the expected output. |
20 | |
21 | ### Running all the tests |
22 | |
23 | By default the test runner `run.py` will run all the tests, and report the results. |
24 | |
25 | Examples: |
26 | |
27 | ``` |
28 | ./run.py |
29 | ./run.py --preserve |
30 | ./run.py --zstd ../../build/programs/zstd --datagen ../../build/tests/datagen |
31 | ``` |
32 | |
33 | ### Running specific tests |
34 | |
35 | A set of test names can be passed to the test runner `run.py` to only execute those tests. |
36 | This can be useful for writing or debugging a test, especially with `--preserve`. |
37 | |
38 | The test name can either be the path to the test file, or the test name, which is the path relative to the test directory. |
39 | |
40 | Examples: |
41 | |
42 | ``` |
43 | ./run.py basic/help.sh |
44 | ./run.py --preserve basic/help.sh basic/version.sh |
45 | ./run.py --preserve --verbose basic/help.sh |
46 | ``` |
47 | |
48 | ### Updating exact output |
49 | |
50 | If a test is failing because a `.stderr.exact` or `.stdout.exact` no longer matches, you can re-run the tests with `--set-exact-output` and the correct output will be written. |
51 | |
52 | Example: |
53 | ``` |
54 | ./run.py --set-exact-output |
55 | ./run.py basic/help.sh --set-exact-output |
56 | ``` |
57 | |
58 | ## Writing a test |
59 | |
60 | Test cases are arbitrary executables, and can be written in any language, but are generally shell scripts. |
61 | After the script executes, the exit code, stderr, and stdout are compared against the expectations. |
62 | |
63 | Each test is run in a clean directory that the test can use for intermediate files. This directory will be cleaned up at the end of the test, unless `--preserve` is passed to the test runner. Additionally, the `setup` script can prepare the directory before the test runs. |
64 | |
65 | ### Calling zstd, utilities, and environment variables |
66 | |
67 | The `$PATH` for tests is prepended with the `bin/` sub-directory, which contains helper scripts for ease of testing. |
68 | The `zstd` binary will call the zstd binary specified by `run.py` with the correct `$EXEC_PREFIX`. |
69 | Similarly, `datagen`, `unzstd`, `zstdgrep`, `zstdcat`, etc, are provided. |
70 | |
71 | Helper utilities like `cmp_size`, `println`, and `die` are provided here too. See their scripts for details. |
72 | |
73 | Common shell script libraries are provided under `common/`, with helper variables and functions. They can be sourced with `source "$COMMON/library.sh`. |
74 | |
75 | Lastly, environment variables are provided for testing, which can be listed when calling `run.py` with `--verbose`. |
76 | They are generally used by the helper scripts in `bin/` to coordinate everything. |
77 | |
78 | ### Basic test case |
79 | |
80 | When executing your `$TEST` executable, by default the exit code is expected to be `0`. However, you can provide an alternate expected exit code in a `$TEST.exit` file. |
81 | |
82 | When executing your `$TEST` executable, by default the expected stderr and stdout are empty. However, you can override the default by providing one of three files: |
83 | |
84 | * `$TEST.{stdout,stderr}.exact` |
85 | * `$TEST.{stdout,stderr}.glob` |
86 | * `$TEST.{stdout,stderr}.ignore` |
87 | |
88 | If you provide a `.exact` file, the output is expected to exactly match, byte-for-byte. |
89 | |
90 | If you provide a `.glob` file, the output is expected to match the expected file, where each line is interpreted as a glob syntax. Additionally, a line containing only `...` matches all lines until the next expected line matches. |
91 | |
92 | If you provide a `.ignore` file, the output is ignored. |
93 | |
94 | #### Passing examples |
95 | |
96 | All these examples pass. |
97 | |
98 | Exit 1, and change the expectation to be 1. |
99 | |
100 | ``` |
101 | exit-1.sh |
102 | --- |
103 | #!/bin/sh |
104 | exit 1 |
105 | --- |
106 | |
107 | exit-1.sh.exit |
108 | --- |
109 | 1 |
110 | --- |
111 | ``` |
112 | |
113 | Check the stdout output exactly matches. |
114 | |
115 | ``` |
116 | echo.sh |
117 | --- |
118 | #!/bin/sh |
119 | echo "hello world" |
120 | --- |
121 | |
122 | echo.sh.stdout.exact |
123 | --- |
124 | hello world |
125 | --- |
126 | ``` |
127 | |
128 | Check the stderr output using a glob. |
129 | |
130 | ``` |
131 | random.sh |
132 | --- |
133 | #!/bin/sh |
134 | head -c 10 < /dev/urandom | xxd >&2 |
135 | --- |
136 | |
137 | random.sh.stderr.glob |
138 | --- |
139 | 00000000: * * * * * * |
140 | ``` |
141 | |
142 | Multiple lines can be matched with ... |
143 | |
144 | ``` |
145 | random-num-lines.sh |
146 | --- |
147 | #!/bin/sh |
148 | echo hello |
149 | seq 0 $RANDOM |
150 | echo world |
151 | --- |
152 | |
153 | random-num-lines.sh.stdout.glob |
154 | --- |
155 | hello |
156 | 0 |
157 | ... |
158 | world |
159 | --- |
160 | ``` |
161 | |
162 | #### Failing examples |
163 | |
164 | Exit code is expected to be 0, but is 1. |
165 | |
166 | ``` |
167 | exit-1.sh |
168 | --- |
169 | #!/bin/sh |
170 | exit 1 |
171 | --- |
172 | ``` |
173 | |
174 | Stdout is expected to be empty, but isn't. |
175 | |
176 | ``` |
177 | echo.sh |
178 | --- |
179 | #!/bin/sh |
180 | echo hello world |
181 | ``` |
182 | |
183 | Stderr is expected to be hello but is world. |
184 | |
185 | ``` |
186 | hello.sh |
187 | --- |
188 | #!/bin/sh |
189 | echo world >&2 |
190 | --- |
191 | |
192 | hello.sh.stderr.exact |
193 | --- |
194 | hello |
195 | --- |
196 | ``` |
197 | |
198 | ### Setup & teardown scripts |
199 | |
200 | Finally, test writing can be eased with setup and teardown scripts. |
201 | Each directory in the test directory is a test-suite consisting of all tests within that directory (but not sub-directories). |
202 | This test suite can come with 4 scripts to help test writing: |
203 | |
204 | * `setup_once` |
205 | * `teardown_once` |
206 | * `setup` |
207 | * `teardown` |
208 | |
209 | The `setup_once` and `teardown_once` are run once before and after all the tests in the suite respectively. |
210 | They operate in the scratch directory for the test suite, which is the parent directory of each scratch directory for each test case. |
211 | They can do work that is shared between tests to improve test efficiency. |
212 | For example, the `dictionaries/setup_once` script builds several dictionaries, for use in the `dictionaries` tests. |
213 | |
214 | The `setup` and `teardown` scripts run before and after each test case respectively, in the test case's scratch directory. |
215 | These scripts can do work that is shared between test cases to make tests more succinct. |
216 | For example, the `dictionaries/setup` script copies the dictionaries built by the `dictionaries/setup_once` script into the test's scratch directory, to make them easier to use, and make sure they aren't accidentally modified. |
217 | |
218 | #### Examples |
219 | |
220 | ``` |
221 | basic/setup |
222 | --- |
223 | #!/bin/sh |
224 | # Create some files for testing with |
225 | datagen > file |
226 | datagen > file0 |
227 | datagen > file1 |
228 | --- |
229 | |
230 | basic/test.sh |
231 | --- |
232 | #!/bin/sh |
233 | zstd file file0 file1 |
234 | --- |
235 | |
236 | dictionaries/setup_once |
237 | --- |
238 | #!/bin/sh |
239 | set -e |
240 | |
241 | mkdir files/ dicts/ |
242 | for i in $(seq 10); do |
243 | datagen -g1000 > files/$i |
244 | done |
245 | |
246 | zstd --train -r files/ -o dicts/0 |
247 | --- |
248 | |
249 | dictionaries/setup |
250 | --- |
251 | #!/bin/sh |
252 | |
253 | # Runs in the test case's scratch directory. |
254 | # The test suite's scratch directory that |
255 | # `setup_once` operates in is the parent directory. |
256 | cp -r ../files ../dicts . |
257 | --- |
258 | ``` |