| 1 | Zstandard library files |
| 2 | ================================ |
| 3 | |
| 4 | The __lib__ directory is split into several sub-directories, |
| 5 | in order to make it easier to select or exclude features. |
| 6 | |
| 7 | |
| 8 | #### Building |
| 9 | |
| 10 | `Makefile` script is provided, supporting [Makefile conventions](https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html#Makefile-Conventions), |
| 11 | including commands variables, staged install, directory variables and standard targets. |
| 12 | - `make` : generates both static and dynamic libraries |
| 13 | - `make install` : install libraries and headers in target system directories |
| 14 | |
| 15 | `libzstd` default scope is pretty large, including compression, decompression, dictionary builder, |
| 16 | and support for decoding legacy formats >= v0.5.0. |
| 17 | The scope can be reduced on demand (see paragraph _modular build_). |
| 18 | |
| 19 | |
| 20 | #### Multithreading support |
| 21 | |
| 22 | When building with `make`, by default the dynamic library is multithreaded and static library is single-threaded (for compatibility reasons). |
| 23 | |
| 24 | Enabling multithreading requires 2 conditions : |
| 25 | - set build macro `ZSTD_MULTITHREAD` (`-DZSTD_MULTITHREAD` for `gcc`) |
| 26 | - for POSIX systems : compile with pthread (`-pthread` compilation flag for `gcc`) |
| 27 | |
| 28 | For convenience, we provide a build target to generate multi and single threaded libraries: |
| 29 | - Force enable multithreading on both dynamic and static libraries by appending `-mt` to the target, e.g. `make lib-mt`. |
| 30 | - Force disable multithreading on both dynamic and static libraries by appending `-nomt` to the target, e.g. `make lib-nomt`. |
| 31 | - By default, as mentioned before, dynamic library is multithreaded, and static library is single-threaded, e.g. `make lib`. |
| 32 | |
| 33 | When linking a POSIX program with a multithreaded version of `libzstd`, |
| 34 | note that it's necessary to invoke the `-pthread` flag during link stage. |
| 35 | |
| 36 | Multithreading capabilities are exposed |
| 37 | via the [advanced API defined in `lib/zstd.h`](https://github.com/facebook/zstd/blob/v1.4.3/lib/zstd.h#L351). |
| 38 | |
| 39 | |
| 40 | #### API |
| 41 | |
| 42 | Zstandard's stable API is exposed within [lib/zstd.h](zstd.h). |
| 43 | |
| 44 | |
| 45 | #### Advanced API |
| 46 | |
| 47 | Optional advanced features are exposed via : |
| 48 | |
| 49 | - `lib/zstd_errors.h` : translates `size_t` function results |
| 50 | into a `ZSTD_ErrorCode`, for accurate error handling. |
| 51 | |
| 52 | - `ZSTD_STATIC_LINKING_ONLY` : if this macro is defined _before_ including `zstd.h`, |
| 53 | it unlocks access to the experimental API, |
| 54 | exposed in the second part of `zstd.h`. |
| 55 | All definitions in the experimental APIs are unstable, |
| 56 | they may still change in the future, or even be removed. |
| 57 | As a consequence, experimental definitions shall ___never be used with dynamic library___ ! |
| 58 | Only static linking is allowed. |
| 59 | |
| 60 | |
| 61 | #### Modular build |
| 62 | |
| 63 | It's possible to compile only a limited set of features within `libzstd`. |
| 64 | The file structure is designed to make this selection manually achievable for any build system : |
| 65 | |
| 66 | - Directory `lib/common` is always required, for all variants. |
| 67 | |
| 68 | - Compression source code lies in `lib/compress` |
| 69 | |
| 70 | - Decompression source code lies in `lib/decompress` |
| 71 | |
| 72 | - It's possible to include only `compress` or only `decompress`, they don't depend on each other. |
| 73 | |
| 74 | - `lib/dictBuilder` : makes it possible to generate dictionaries from a set of samples. |
| 75 | The API is exposed in `lib/dictBuilder/zdict.h`. |
| 76 | This module depends on both `lib/common` and `lib/compress` . |
| 77 | |
| 78 | - `lib/legacy` : makes it possible to decompress legacy zstd formats, starting from `v0.1.0`. |
| 79 | This module depends on `lib/common` and `lib/decompress`. |
| 80 | To enable this feature, define `ZSTD_LEGACY_SUPPORT` during compilation. |
| 81 | Specifying a number limits versions supported to that version onward. |
| 82 | For example, `ZSTD_LEGACY_SUPPORT=2` means : "support legacy formats >= v0.2.0". |
| 83 | Conversely, `ZSTD_LEGACY_SUPPORT=0` means "do __not__ support legacy formats". |
| 84 | By default, this build macro is set as `ZSTD_LEGACY_SUPPORT=5`. |
| 85 | Decoding supported legacy format is a transparent capability triggered within decompression functions. |
| 86 | It's also allowed to invoke legacy API directly, exposed in `lib/legacy/zstd_legacy.h`. |
| 87 | Each version does also provide its own set of advanced API. |
| 88 | For example, advanced API for version `v0.4` is exposed in `lib/legacy/zstd_v04.h` . |
| 89 | |
| 90 | - While invoking `make libzstd`, it's possible to define build macros |
| 91 | `ZSTD_LIB_COMPRESSION, ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`, |
| 92 | and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the |
| 93 | corresponding features. This will also disable compilation of all |
| 94 | dependencies (e.g. `ZSTD_LIB_COMPRESSION=0` will also disable |
| 95 | dictBuilder). |
| 96 | |
| 97 | - There are a number of options that can help minimize the binary size of |
| 98 | `libzstd`. |
| 99 | |
| 100 | The first step is to select the components needed (using the above-described |
| 101 | `ZSTD_LIB_COMPRESSION` etc.). |
| 102 | |
| 103 | The next step is to set `ZSTD_LIB_MINIFY` to `1` when invoking `make`. This |
| 104 | disables various optional components and changes the compilation flags to |
| 105 | prioritize space-saving. |
| 106 | |
| 107 | Detailed options: Zstandard's code and build environment is set up by default |
| 108 | to optimize above all else for performance. In pursuit of this goal, Zstandard |
| 109 | makes significant trade-offs in code size. For example, Zstandard often has |
| 110 | more than one implementation of a particular component, with each |
| 111 | implementation optimized for different scenarios. For example, the Huffman |
| 112 | decoder has complementary implementations that decode the stream one symbol at |
| 113 | a time or two symbols at a time. Zstd normally includes both (and dispatches |
| 114 | between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1` or |
| 115 | `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding |
| 116 | compilation of the other. Similarly, `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` |
| 117 | and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG` force the compilation and use of |
| 118 | only one or the other of two decompression implementations. The smallest |
| 119 | binary is achieved by using `HUF_FORCE_DECOMPRESS_X1` and |
| 120 | `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` (implied by `ZSTD_LIB_MINIFY`). |
| 121 | |
| 122 | For squeezing the last ounce of size out, you can also define |
| 123 | `ZSTD_NO_INLINE`, which disables inlining, and `ZSTD_STRIP_ERROR_STRINGS`, |
| 124 | which removes the error messages that are otherwise returned by |
| 125 | `ZSTD_getErrorName` (implied by `ZSTD_LIB_MINIFY`). |
| 126 | |
| 127 | Finally, when integrating into your application, make sure you're doing link- |
| 128 | time optimization and unused symbol garbage collection (via some combination of, |
| 129 | e.g., `-flto`, `-ffat-lto-objects`, `-fuse-linker-plugin`, |
| 130 | `-ffunction-sections`, `-fdata-sections`, `-fmerge-all-constants`, |
| 131 | `-Wl,--gc-sections`, `-Wl,-z,norelro`, and an archiver that understands |
| 132 | the compiler's intermediate representation, e.g., `AR=gcc-ar`). Consult your |
| 133 | compiler's documentation. |
| 134 | |
| 135 | - While invoking `make libzstd`, the build macro `ZSTD_LEGACY_MULTITHREADED_API=1` |
| 136 | will expose the deprecated `ZSTDMT` API exposed by `zstdmt_compress.h` in |
| 137 | the shared library, which is now hidden by default. |
| 138 | |
| 139 | - The build macro `DYNAMIC_BMI2` can be set to 1 or 0 in order to generate binaries |
| 140 | which can detect at runtime the presence of BMI2 instructions, and use them only if present. |
| 141 | These instructions contribute to better performance, notably on the decoder side. |
| 142 | By default, this feature is automatically enabled on detecting |
| 143 | the right instruction set (x64) and compiler (clang or gcc >= 5). |
| 144 | It's obviously disabled for different cpus, |
| 145 | or when BMI2 instruction set is _required_ by the compiler command line |
| 146 | (in this case, only the BMI2 code path is generated). |
| 147 | Setting this macro will either force to generate the BMI2 dispatcher (1) |
| 148 | or prevent it (0). It overrides automatic detection. |
| 149 | |
| 150 | - The build macro `ZSTD_NO_UNUSED_FUNCTIONS` can be defined to hide the definitions of functions |
| 151 | that zstd does not use. Not all unused functions are hidden, but they can be if needed. |
| 152 | Currently, this macro will hide function definitions in FSE and HUF that use an excessive |
| 153 | amount of stack space. |
| 154 | |
| 155 | - The build macro `ZSTD_NO_INTRINSICS` can be defined to disable all explicit intrinsics. |
| 156 | Compiler builtins are still used. |
| 157 | |
| 158 | - The build macro `ZSTD_DECODER_INTERNAL_BUFFER` can be set to control |
| 159 | the amount of extra memory used during decompression to store literals. |
| 160 | This defaults to 64kB. Reducing this value reduces the memory footprint of |
| 161 | `ZSTD_DCtx` decompression contexts, |
| 162 | but might also result in a small decompression speed cost. |
| 163 | |
| 164 | - The C compiler macros `ZSTDLIB_VISIBLE`, `ZSTDERRORLIB_VISIBLE` and `ZDICTLIB_VISIBLE` |
| 165 | can be overridden to control the visibility of zstd's API. Additionally, |
| 166 | `ZSTDLIB_STATIC_API` and `ZDICTLIB_STATIC_API` can be overridden to control the visibility |
| 167 | of zstd's static API. Specifically, it can be set to `ZSTDLIB_HIDDEN` to hide the symbols |
| 168 | from the shared library. These macros default to `ZSTDLIB_VISIBILITY`, |
| 169 | `ZSTDERRORLIB_VSIBILITY`, and `ZDICTLIB_VISIBILITY` if unset, for backwards compatibility |
| 170 | with the old macro names. |
| 171 | |
| 172 | #### Windows : using MinGW+MSYS to create DLL |
| 173 | |
| 174 | DLL can be created using MinGW+MSYS with the `make libzstd` command. |
| 175 | This command creates `dll\libzstd.dll` and the import library `dll\libzstd.lib`. |
| 176 | The import library is only required with Visual C++. |
| 177 | The header file `zstd.h` and the dynamic library `dll\libzstd.dll` are required to |
| 178 | compile a project using gcc/MinGW. |
| 179 | The dynamic library has to be added to linking options. |
| 180 | It means that if a project that uses ZSTD consists of a single `test-dll.c` |
| 181 | file it should be linked with `dll\libzstd.dll`. For example: |
| 182 | ``` |
| 183 | gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll |
| 184 | ``` |
| 185 | The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`. |
| 186 | |
| 187 | |
| 188 | #### Advanced Build options |
| 189 | |
| 190 | The build system requires a hash function in order to |
| 191 | separate object files created with different compilation flags. |
| 192 | By default, it tries to use `md5sum` or equivalent. |
| 193 | The hash function can be manually switched by setting the `HASH` variable. |
| 194 | For example : `make HASH=xxhsum` |
| 195 | The hash function needs to generate at least 64-bit using hexadecimal format. |
| 196 | When no hash function is found, |
| 197 | the Makefile just generates all object files into the same default directory, |
| 198 | irrespective of compilation flags. |
| 199 | This functionality only matters if `libzstd` is compiled multiple times |
| 200 | with different build flags. |
| 201 | |
| 202 | The build directory, where object files are stored |
| 203 | can also be manually controlled using variable `BUILD_DIR`, |
| 204 | for example `make BUILD_DIR=objectDir/v1`. |
| 205 | In which case, the hash function doesn't matter. |
| 206 | |
| 207 | |
| 208 | #### Deprecated API |
| 209 | |
| 210 | Obsolete API on their way out are stored in directory `lib/deprecated`. |
| 211 | At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`. |
| 212 | These prototypes will be removed in some future version. |
| 213 | Consider migrating code towards supported streaming API exposed in `zstd.h`. |
| 214 | |
| 215 | |
| 216 | #### Miscellaneous |
| 217 | |
| 218 | The other files are not source code. There are : |
| 219 | |
| 220 | - `BUCK` : support for `buck` build system (https://buckbuild.com/) |
| 221 | - `Makefile` : `make` script to build and install zstd library (static and dynamic) |
| 222 | - `README.md` : this file |
| 223 | - `dll/` : resources directory for Windows compilation |
| 224 | - `libzstd.pc.in` : script for `pkg-config` (used in `make install`) |