648db22b |
1 | Zstandard library files |
2 | ================================ |
3 | |
4 | The __lib__ directory is split into several sub-directories, |
5 | in order to make it easier to select or exclude features. |
6 | |
7 | |
8 | #### Building |
9 | |
10 | `Makefile` script is provided, supporting [Makefile conventions](https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html#Makefile-Conventions), |
11 | including commands variables, staged install, directory variables and standard targets. |
12 | - `make` : generates both static and dynamic libraries |
13 | - `make install` : install libraries and headers in target system directories |
14 | |
15 | `libzstd` default scope is pretty large, including compression, decompression, dictionary builder, |
16 | and support for decoding legacy formats >= v0.5.0. |
17 | The scope can be reduced on demand (see paragraph _modular build_). |
18 | |
19 | |
20 | #### Multithreading support |
21 | |
22 | When building with `make`, by default the dynamic library is multithreaded and static library is single-threaded (for compatibility reasons). |
23 | |
24 | Enabling multithreading requires 2 conditions : |
25 | - set build macro `ZSTD_MULTITHREAD` (`-DZSTD_MULTITHREAD` for `gcc`) |
26 | - for POSIX systems : compile with pthread (`-pthread` compilation flag for `gcc`) |
27 | |
28 | For convenience, we provide a build target to generate multi and single threaded libraries: |
29 | - Force enable multithreading on both dynamic and static libraries by appending `-mt` to the target, e.g. `make lib-mt`. |
30 | - Force disable multithreading on both dynamic and static libraries by appending `-nomt` to the target, e.g. `make lib-nomt`. |
31 | - By default, as mentioned before, dynamic library is multithreaded, and static library is single-threaded, e.g. `make lib`. |
32 | |
33 | When linking a POSIX program with a multithreaded version of `libzstd`, |
34 | note that it's necessary to invoke the `-pthread` flag during link stage. |
35 | |
36 | Multithreading capabilities are exposed |
37 | via the [advanced API defined in `lib/zstd.h`](https://github.com/facebook/zstd/blob/v1.4.3/lib/zstd.h#L351). |
38 | |
39 | |
40 | #### API |
41 | |
42 | Zstandard's stable API is exposed within [lib/zstd.h](zstd.h). |
43 | |
44 | |
45 | #### Advanced API |
46 | |
47 | Optional advanced features are exposed via : |
48 | |
49 | - `lib/zstd_errors.h` : translates `size_t` function results |
50 | into a `ZSTD_ErrorCode`, for accurate error handling. |
51 | |
52 | - `ZSTD_STATIC_LINKING_ONLY` : if this macro is defined _before_ including `zstd.h`, |
53 | it unlocks access to the experimental API, |
54 | exposed in the second part of `zstd.h`. |
55 | All definitions in the experimental APIs are unstable, |
56 | they may still change in the future, or even be removed. |
57 | As a consequence, experimental definitions shall ___never be used with dynamic library___ ! |
58 | Only static linking is allowed. |
59 | |
60 | |
61 | #### Modular build |
62 | |
63 | It's possible to compile only a limited set of features within `libzstd`. |
64 | The file structure is designed to make this selection manually achievable for any build system : |
65 | |
66 | - Directory `lib/common` is always required, for all variants. |
67 | |
68 | - Compression source code lies in `lib/compress` |
69 | |
70 | - Decompression source code lies in `lib/decompress` |
71 | |
72 | - It's possible to include only `compress` or only `decompress`, they don't depend on each other. |
73 | |
74 | - `lib/dictBuilder` : makes it possible to generate dictionaries from a set of samples. |
75 | The API is exposed in `lib/dictBuilder/zdict.h`. |
76 | This module depends on both `lib/common` and `lib/compress` . |
77 | |
78 | - `lib/legacy` : makes it possible to decompress legacy zstd formats, starting from `v0.1.0`. |
79 | This module depends on `lib/common` and `lib/decompress`. |
80 | To enable this feature, define `ZSTD_LEGACY_SUPPORT` during compilation. |
81 | Specifying a number limits versions supported to that version onward. |
82 | For example, `ZSTD_LEGACY_SUPPORT=2` means : "support legacy formats >= v0.2.0". |
83 | Conversely, `ZSTD_LEGACY_SUPPORT=0` means "do __not__ support legacy formats". |
84 | By default, this build macro is set as `ZSTD_LEGACY_SUPPORT=5`. |
85 | Decoding supported legacy format is a transparent capability triggered within decompression functions. |
86 | It's also allowed to invoke legacy API directly, exposed in `lib/legacy/zstd_legacy.h`. |
87 | Each version does also provide its own set of advanced API. |
88 | For example, advanced API for version `v0.4` is exposed in `lib/legacy/zstd_v04.h` . |
89 | |
90 | - While invoking `make libzstd`, it's possible to define build macros |
f535537f |
91 | `ZSTD_LIB_COMPRESSION`, `ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`, |
648db22b |
92 | and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the |
93 | corresponding features. This will also disable compilation of all |
94 | dependencies (e.g. `ZSTD_LIB_COMPRESSION=0` will also disable |
95 | dictBuilder). |
96 | |
97 | - There are a number of options that can help minimize the binary size of |
98 | `libzstd`. |
99 | |
100 | The first step is to select the components needed (using the above-described |
101 | `ZSTD_LIB_COMPRESSION` etc.). |
102 | |
103 | The next step is to set `ZSTD_LIB_MINIFY` to `1` when invoking `make`. This |
104 | disables various optional components and changes the compilation flags to |
105 | prioritize space-saving. |
106 | |
107 | Detailed options: Zstandard's code and build environment is set up by default |
108 | to optimize above all else for performance. In pursuit of this goal, Zstandard |
109 | makes significant trade-offs in code size. For example, Zstandard often has |
110 | more than one implementation of a particular component, with each |
111 | implementation optimized for different scenarios. For example, the Huffman |
112 | decoder has complementary implementations that decode the stream one symbol at |
113 | a time or two symbols at a time. Zstd normally includes both (and dispatches |
114 | between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1` or |
115 | `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding |
116 | compilation of the other. Similarly, `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` |
117 | and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG` force the compilation and use of |
118 | only one or the other of two decompression implementations. The smallest |
119 | binary is achieved by using `HUF_FORCE_DECOMPRESS_X1` and |
120 | `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` (implied by `ZSTD_LIB_MINIFY`). |
121 | |
f535537f |
122 | On the compressor side, Zstd's compression levels map to several internal |
123 | strategies. In environments where the higher compression levels aren't used, |
124 | it is possible to exclude all but the fastest strategy with |
125 | `ZSTD_LIB_EXCLUDE_COMPRESSORS_DFAST_AND_UP=1`. (Note that this will change |
126 | the behavior of the default compression level.) Or if you want to retain the |
127 | default compressor as well, you can set |
128 | `ZSTD_LIB_EXCLUDE_COMPRESSORS_GREEDY_AND_UP=1`, at the cost of an additional |
129 | ~20KB or so. |
130 | |
648db22b |
131 | For squeezing the last ounce of size out, you can also define |
132 | `ZSTD_NO_INLINE`, which disables inlining, and `ZSTD_STRIP_ERROR_STRINGS`, |
133 | which removes the error messages that are otherwise returned by |
134 | `ZSTD_getErrorName` (implied by `ZSTD_LIB_MINIFY`). |
135 | |
136 | Finally, when integrating into your application, make sure you're doing link- |
137 | time optimization and unused symbol garbage collection (via some combination of, |
138 | e.g., `-flto`, `-ffat-lto-objects`, `-fuse-linker-plugin`, |
139 | `-ffunction-sections`, `-fdata-sections`, `-fmerge-all-constants`, |
140 | `-Wl,--gc-sections`, `-Wl,-z,norelro`, and an archiver that understands |
141 | the compiler's intermediate representation, e.g., `AR=gcc-ar`). Consult your |
142 | compiler's documentation. |
143 | |
144 | - While invoking `make libzstd`, the build macro `ZSTD_LEGACY_MULTITHREADED_API=1` |
145 | will expose the deprecated `ZSTDMT` API exposed by `zstdmt_compress.h` in |
146 | the shared library, which is now hidden by default. |
147 | |
148 | - The build macro `DYNAMIC_BMI2` can be set to 1 or 0 in order to generate binaries |
149 | which can detect at runtime the presence of BMI2 instructions, and use them only if present. |
150 | These instructions contribute to better performance, notably on the decoder side. |
151 | By default, this feature is automatically enabled on detecting |
152 | the right instruction set (x64) and compiler (clang or gcc >= 5). |
153 | It's obviously disabled for different cpus, |
154 | or when BMI2 instruction set is _required_ by the compiler command line |
155 | (in this case, only the BMI2 code path is generated). |
156 | Setting this macro will either force to generate the BMI2 dispatcher (1) |
157 | or prevent it (0). It overrides automatic detection. |
158 | |
159 | - The build macro `ZSTD_NO_UNUSED_FUNCTIONS` can be defined to hide the definitions of functions |
160 | that zstd does not use. Not all unused functions are hidden, but they can be if needed. |
161 | Currently, this macro will hide function definitions in FSE and HUF that use an excessive |
162 | amount of stack space. |
163 | |
164 | - The build macro `ZSTD_NO_INTRINSICS` can be defined to disable all explicit intrinsics. |
165 | Compiler builtins are still used. |
166 | |
167 | - The build macro `ZSTD_DECODER_INTERNAL_BUFFER` can be set to control |
168 | the amount of extra memory used during decompression to store literals. |
169 | This defaults to 64kB. Reducing this value reduces the memory footprint of |
170 | `ZSTD_DCtx` decompression contexts, |
171 | but might also result in a small decompression speed cost. |
172 | |
173 | - The C compiler macros `ZSTDLIB_VISIBLE`, `ZSTDERRORLIB_VISIBLE` and `ZDICTLIB_VISIBLE` |
174 | can be overridden to control the visibility of zstd's API. Additionally, |
175 | `ZSTDLIB_STATIC_API` and `ZDICTLIB_STATIC_API` can be overridden to control the visibility |
176 | of zstd's static API. Specifically, it can be set to `ZSTDLIB_HIDDEN` to hide the symbols |
177 | from the shared library. These macros default to `ZSTDLIB_VISIBILITY`, |
178 | `ZSTDERRORLIB_VSIBILITY`, and `ZDICTLIB_VISIBILITY` if unset, for backwards compatibility |
179 | with the old macro names. |
180 | |
f535537f |
181 | - The C compiler macro `HUF_DISABLE_FAST_DECODE` disables the newer Huffman fast C |
182 | and assembly decoding loops. You may want to use this macro if these loops are |
183 | slower on your platform. |
184 | |
648db22b |
185 | #### Windows : using MinGW+MSYS to create DLL |
186 | |
187 | DLL can be created using MinGW+MSYS with the `make libzstd` command. |
188 | This command creates `dll\libzstd.dll` and the import library `dll\libzstd.lib`. |
189 | The import library is only required with Visual C++. |
190 | The header file `zstd.h` and the dynamic library `dll\libzstd.dll` are required to |
191 | compile a project using gcc/MinGW. |
192 | The dynamic library has to be added to linking options. |
193 | It means that if a project that uses ZSTD consists of a single `test-dll.c` |
194 | file it should be linked with `dll\libzstd.dll`. For example: |
195 | ``` |
196 | gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll |
197 | ``` |
198 | The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`. |
199 | |
200 | |
201 | #### Advanced Build options |
202 | |
203 | The build system requires a hash function in order to |
204 | separate object files created with different compilation flags. |
205 | By default, it tries to use `md5sum` or equivalent. |
206 | The hash function can be manually switched by setting the `HASH` variable. |
207 | For example : `make HASH=xxhsum` |
208 | The hash function needs to generate at least 64-bit using hexadecimal format. |
209 | When no hash function is found, |
210 | the Makefile just generates all object files into the same default directory, |
211 | irrespective of compilation flags. |
212 | This functionality only matters if `libzstd` is compiled multiple times |
213 | with different build flags. |
214 | |
215 | The build directory, where object files are stored |
216 | can also be manually controlled using variable `BUILD_DIR`, |
217 | for example `make BUILD_DIR=objectDir/v1`. |
218 | In which case, the hash function doesn't matter. |
219 | |
220 | |
221 | #### Deprecated API |
222 | |
223 | Obsolete API on their way out are stored in directory `lib/deprecated`. |
224 | At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`. |
225 | These prototypes will be removed in some future version. |
226 | Consider migrating code towards supported streaming API exposed in `zstd.h`. |
227 | |
228 | |
229 | #### Miscellaneous |
230 | |
231 | The other files are not source code. There are : |
232 | |
233 | - `BUCK` : support for `buck` build system (https://buckbuild.com/) |
234 | - `Makefile` : `make` script to build and install zstd library (static and dynamic) |
235 | - `README.md` : this file |
236 | - `dll/` : resources directory for Windows compilation |
237 | - `libzstd.pc.in` : script for `pkg-config` (used in `make install`) |