e14743d1 |
1 | UTF-8 decoder capability and stress test |
2 | ---------------------------------------- |
3 | |
4 | Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19 |
5 | |
6 | This test file can help you examine, how your UTF-8 decoder handles |
7 | various types of correct, malformed, or otherwise interesting UTF-8 |
8 | sequences. This file is not meant to be a conformance test. It does |
9 | not prescribes any particular outcome and therefore there is no way to |
10 | "pass" or "fail" this test file, even though the texts suggests a |
11 | preferable decoder behaviour at some places. The aim is instead to |
12 | help you think about and test the behaviour of your UTF-8 on a |
13 | systematic collection of unusual inputs. Experience so far suggests |
14 | that most first-time authors of UTF-8 decoders find at least one |
15 | serious problem in their decoder by using this file. |
16 | |
17 | The test lines below cover boundary conditions, malformed UTF-8 |
18 | sequences as well as correctly encoded UTF-8 sequences of Unicode code |
19 | points that should never occur in a correct UTF-8 file. |
20 | |
21 | According to ISO 10646-1:2000, sections D.7 and 2.3c, a device |
22 | receiving UTF-8 shall interpret a "malformed sequence in the same way |
23 | that it interprets a character that is outside the adopted subset" and |
24 | "characters that are not within the adopted subset shall be indicated |
25 | to the user" by a receiving device. A quite commonly used approach in |
26 | UTF-8 decoders is to replace any malformed UTF-8 sequence by a |
27 | replacement character (U+FFFD), which looks a bit like an inverted |
28 | question mark, or a similar symbol. It might be a good idea to |
29 | visually distinguish a malformed UTF-8 sequence from a correctly |
30 | encoded Unicode character that is just not available in the current |
31 | font but otherwise fully legal, even though ISO 10646-1 doesn't |
32 | mandate this. In any case, just ignoring malformed sequences or |
33 | unavailable characters does not conform to ISO 10646, will make |
34 | debugging more difficult, and can lead to user confusion. |
35 | |
36 | Please check, whether a malformed UTF-8 sequence is (1) represented at |
37 | all, (2) represented by exactly one single replacement character (or |
38 | equivalent signal), and (3) the following quotation mark after an |
39 | illegal UTF-8 sequence is correctly displayed, i.e. proper |
40 | resynchronization takes place immageately after any malformed |
41 | sequence. This file says "THE END" in the last line, so if you don't |
42 | see that, your decoder crashed somehow before, which should always be |
43 | cause for concern. |
44 | |
45 | All lines in this file are exactly 79 characters long (plus the line |
46 | feed). In addition, all lines end with "|", except for the two test |
47 | lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls |
48 | U+0000 and U+007F. If you display this file with a fixed-width font, |
49 | these "|" characters should all line up in column 79 (right margin). |
50 | This allows you to test quickly, whether your UTF-8 decoder finds the |
51 | correct number of characters in every line, that is whether each |
52 | malformed sequences is replaced by a single replacement character. |
53 | |
54 | Note that as an alternative to the notion of malformed sequence used |
55 | here, it is also a perfectly acceptable (and in some situations even |
56 | preferable) solution to represent each individual byte of a malformed |
57 | sequence by a replacement character. If you follow this strategy in |
58 | your decoder, then please ignore the "|" column. |
59 | |
60 | |
61 | Here come the tests: | |
62 | | |
63 | 1 Some correct UTF-8 text | |
64 | | |
65 | (The codepoints for this test are: | |
66 | U+03BA U+1F79 U+03C3 U+03BC U+03B5 --ryan.) | |
67 | | |
68 | You should see the Greek word 'kosme': "κόσμε" | |
69 | | |
70 | | |
71 | 2 Boundary condition test cases | |
72 | | |
73 | 2.1 First possible sequence of a certain length | |
74 | | |
75 | (byte zero skipped...there's a null added at the end of the test. --ryan.) | |
76 | | |
77 | 2.1.2 2 bytes (U-00000080): "\80" | |
78 | 2.1.3 3 bytes (U-00000800): "ࠀ" | |
79 | 2.1.4 4 bytes (U-00010000): "𐀀" | |
80 | | |
81 | (5 and 6 byte sequences were made illegal in rfc3629. --ryan.) | |
82 | 2.1.5 5 bytes (U-00200000): "" | |
83 | 2.1.6 6 bytes (U-04000000): "" | |
84 | | |
85 | 2.2 Last possible sequence of a certain length | |
86 | | |
87 | 2.2.1 1 byte (U-0000007F): "\7f" | |
88 | 2.2.2 2 bytes (U-000007FF): "߿" | |
89 | | |
90 | (Section 5.3.2 below calls this illegal. --ryan.) | |
91 | 2.2.3 3 bytes (U-0000FFFF): "" | |
92 | | |
93 | (5 and 6 bytes sequences, and 4 bytes sequences > 0x10FFFF were made illegal | |
94 | in rfc3629, so these next three should be replaced with a invalid | |
95 | character codepoint. --ryan.) | |
96 | 2.2.4 4 bytes (U-001FFFFF): "" | |
97 | 2.2.5 5 bytes (U-03FFFFFF): "" | |
98 | 2.2.6 6 bytes (U-7FFFFFFF): "" | |
99 | | |
100 | 2.3 Other boundary conditions | |
101 | | |
102 | 2.3.1 U-0000D7FF = ed 9f bf = "" | |
103 | 2.3.2 U-0000E000 = ee 80 80 = "" | |
104 | 2.3.3 U-0000FFFD = ef bf bd = "�" | |
105 | 2.3.4 U-0010FFFF = f4 8f bf bf = "" | |
106 | | |
107 | (This one is bogus in rfc3629. --ryan.) | |
108 | 2.3.5 U-00110000 = f4 90 80 80 = "" | |
109 | | |
110 | 3 Malformed sequences | |
111 | | |
112 | 3.1 Unexpected continuation bytes | |
113 | | |
114 | Each unexpected continuation byte should be separately signalled as a | |
115 | malformed sequence of its own. | |
116 | | |
117 |