| 1 | Decompressor Errata |
| 2 | =================== |
| 3 | |
| 4 | This document captures known decompressor bugs, where the decompressor rejects a valid zstd frame. |
| 5 | Each entry will contain: |
| 6 | 1. The last affected decompressor versions. |
| 7 | 2. The decompressor components affected. |
| 8 | 2. Whether the compressed frame could ever be produced by the reference compressor. |
| 9 | 3. An example frame. |
| 10 | 4. A description of the bug. |
| 11 | |
| 12 | The document is in reverse chronological order, with the bugs that affect the most recent zstd decompressor versions listed first. |
| 13 | |
| 14 | |
| 15 | Compressed block with 0 literals and 0 sequences |
| 16 | ------------------------------------------------ |
| 17 | |
| 18 | **Last affected version**: v1.5.2 |
| 19 | |
| 20 | **Affected decompressor component(s)**: Library & CLI |
| 21 | |
| 22 | **Produced by the reference compressor**: No |
| 23 | |
| 24 | **Example Frame**: `28b5 2ffd 2000 1500 0000 00` |
| 25 | |
| 26 | The zstd decoder incorrectly rejected blocks of type `Compressed_Block` that encodes literals as `Raw_Literals_Block` with no literals, and has no sequences. |
| 27 | |
| 28 | This type of block was never generated by the reference compressor. |
| 29 | |
| 30 | Additionally, these blocks were disallowed by the spec up until spec version 0.3.2 when the restriction was lifted by [PR#1689](https://github.com/facebook/zstd/pull/1689). |
| 31 | |
| 32 | > A Compressed_Block has the extra restriction that Block_Size is always strictly less than the decompressed size. If this condition cannot be respected, the block must be sent uncompressed instead (Raw_Block). |
| 33 | |
| 34 | First block is RLE block |
| 35 | ------------------------ |
| 36 | |
| 37 | **Last affected version**: v1.4.3 |
| 38 | |
| 39 | **Affected decompressor component(s)**: CLI only |
| 40 | |
| 41 | **Produced by the reference compressor**: No |
| 42 | |
| 43 | **Example Frame**: `28b5 2ffd a001 0002 0002 0010 000b 0000 00` |
| 44 | |
| 45 | The zstd CLI decompressor rejected cases where the first block was an RLE block whose `Block_Size` is 131072, and the frame contains more than one block. |
| 46 | This only affected the zstd CLI, and not the library. |
| 47 | |
| 48 | The example is an RLE block with 131072 bytes, followed by a second RLE block with 1 byte. |
| 49 | |
| 50 | The compressor currently works around this limitation by explicitly avoiding producing RLE blocks as the first |
| 51 | block. |
| 52 | |
| 53 | https://github.com/facebook/zstd/blob/8814aa5bfa74f05a86e55e9d508da177a893ceeb/lib/compress/zstd_compress.c#L3527-L3535 |
| 54 | |
| 55 | Tiny FSE Table & Block |
| 56 | ---------------------- |
| 57 | |
| 58 | **Last affected version**: v1.3.4 |
| 59 | |
| 60 | **Affected decompressor component(s)**: Library & CLI |
| 61 | |
| 62 | **Produced by the reference compressor**: Possibly until version v1.3.4, but probably never |
| 63 | |
| 64 | **Example Frame**: `28b5 2ffd 2027 c500 0080 f3f1 f0ec ebc6 c5c7 f09d 4300 0000 e0e0 0658 0100 603e 52` |
| 65 | |
| 66 | The zstd library rejected blocks of type `Compressed_Block` whose offset of the last table with type `FSE_Compressed_Mode` was less than 4 bytes from the end of the block. |
| 67 | |
| 68 | In more depth, let `Last_Table_Offset` be the offset in the compressed block (excluding the header) that |
| 69 | the last table with type `FSE_Compressed_Mode` started. If `Block_Content - Last_Table_Offset < 4` then |
| 70 | the buggy zstd decompressor would reject the block. This occurs when the last serialized table is 2 bytes |
| 71 | and the bitstream size is 1 byte. |
| 72 | |
| 73 | For example: |
| 74 | * There is 1 sequence in the block |
| 75 | * `Literals_Lengths_Mode` is `FSE_Compressed_Mode` & the serialized table size is 2 bytes |
| 76 | * `Offsets_Mode` is `Predefined_Mode` |
| 77 | * `Match_Lengths_Mode` is `Predefined_Mode` |
| 78 | * The bitstream is 1 byte. E.g. there is only one sequence and it fits in 1 byte. |
| 79 | |
| 80 | The total `Block_Content` is `5` bytes, and `Last_Table_Offset` is `2`. |
| 81 | |
| 82 | See the compressor workaround code: |
| 83 | |
| 84 | https://github.com/facebook/zstd/blob/8814aa5bfa74f05a86e55e9d508da177a893ceeb/lib/compress/zstd_compress.c#L2667-L2682 |