[sdl_omap.git] / test / utf8.txt

UTF-8 decoder capability and stress test
----------------------------------------

Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19

This test file can help you examine, how your UTF-8 decoder handles
various types of correct, malformed, or otherwise interesting UTF-8
sequences. This file is not meant to be a conformance test. It does
not prescribes any particular outcome and therefore there is no way to
"pass" or "fail" this test file, even though the texts suggests a
preferable decoder behaviour at some places. The aim is instead to
help you think about and test the behaviour of your UTF-8 on a
systematic collection of unusual inputs. Experience so far suggests
that most first-time authors of UTF-8 decoders find at least one
serious problem in their decoder by using this file.

The test lines below cover boundary conditions, malformed UTF-8
sequences as well as correctly encoded UTF-8 sequences of Unicode code
points that should never occur in a correct UTF-8 file.

According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
receiving UTF-8 shall interpret a "malformed sequence in the same way
that it interprets a character that is outside the adopted subset" and
"characters that are not within the adopted subset shall be indicated
to the user" by a receiving device. A quite commonly used approach in
UTF-8 decoders is to replace any malformed UTF-8 sequence by a
replacement character (U+FFFD), which looks a bit like an inverted
question mark, or a similar symbol. It might be a good idea to
visually distinguish a malformed UTF-8 sequence from a correctly
encoded Unicode character that is just not available in the current
font but otherwise fully legal, even though ISO 10646-1 doesn't
mandate this. In any case, just ignoring malformed sequences or
unavailable characters does not conform to ISO 10646, will make
debugging more difficult, and can lead to user confusion.

Please check, whether a malformed UTF-8 sequence is (1) represented at
all, (2) represented by exactly one single replacement character (or
equivalent signal), and (3) the following quotation mark after an
illegal UTF-8 sequence is correctly displayed, i.e. proper
resynchronization takes place immageately after any malformed
sequence. This file says "THE END" in the last line, so if you don't
see that, your decoder crashed somehow before, which should always be
cause for concern.

All lines in this file are exactly 79 characters long (plus the line
feed). In addition, all lines end with "|", except for the two test
lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
U+0000 and U+007F. If you display this file with a fixed-width font,
these "|" characters should all line up in column 79 (right margin).
This allows you to test quickly, whether your UTF-8 decoder finds the
correct number of characters in every line, that is whether each
malformed sequences is replaced by a single replacement character.

Note that as an alternative to the notion of malformed sequence used
here, it is also a perfectly acceptable (and in some situations even
preferable) solution to represent each individual byte of a malformed
sequence by a replacement character. If you follow this strategy in
your decoder, then please ignore the "|" column.


Here come the tests:                                                          |
                                                                              |
1  Some correct UTF-8 text                                                    |
                                                                              |
(The codepoints for this test are:                                            |
  U+03BA U+1F79 U+03C3 U+03BC U+03B5  --ryan.)                                |
                                                                              |
You should see the Greek word 'kosme':       "κόσμε"                          |
                                                                              |
                                                                              |
2  Boundary condition test cases                                              |
                                                                              |
2.1  First possible sequence of a certain length                              |
                                                                              |
(byte zero skipped...there's a null added at the end of the test. --ryan.)    |
                                                                              |
2.1.2  2 bytes (U-00000080):        "\80"                                       |
2.1.3  3 bytes (U-00000800):        "ࠀ"                                       |
2.1.4  4 bytes (U-00010000):        "𐀀"                                       |
                                                                              |
(5 and 6 byte sequences were made illegal in rfc3629. --ryan.)                |
2.1.5  5 bytes (U-00200000):        "�����"                                       |
2.1.6  6 bytes (U-04000000):        "������"                                       |
                                                                              |
2.2  Last possible sequence of a certain length                               |
                                                                              |
2.2.1  1 byte  (U-0000007F):        "\7f"                                       |
2.2.2  2 bytes (U-000007FF):        "߿"                                       |
                                                                              |
(Section 5.3.2 below calls this illegal. --ryan.)                             |
2.2.3  3 bytes (U-0000FFFF):        ""                                       |
                                                                              |
(5 and 6 bytes sequences, and 4 bytes sequences > 0x10FFFF were made illegal  |
 in rfc3629, so these next three should be replaced with a invalid            |
 character codepoint. --ryan.)                                                |
2.2.4  4 bytes (U-001FFFFF):        "����"                                       |
2.2.5  5 bytes (U-03FFFFFF):        "�����"                                       |
2.2.6  6 bytes (U-7FFFFFFF):        "������"                                       |
                                                                              |
2.3  Other boundary conditions                                                |
                                                                              |
2.3.1  U-0000D7FF = ed 9f bf = "퟿"                                            |
2.3.2  U-0000E000 = ee 80 80 = ""                                            |
2.3.3  U-0000FFFD = ef bf bd = "�"                                            |
2.3.4  U-0010FFFF = f4 8f bf bf = "􏿿"                                         |
                                                                              |
(This one is bogus in rfc3629. --ryan.)                                       |
2.3.5  U-00110000 = f4 90 80 80 = "����"                                         |
                                                                              |
3  Malformed sequences                                                        |
                                                                              |
3.1  Unexpected continuation bytes                                            |
                                                                              |
Each unexpected continuation byte should be separately signalled as a         |
malformed sequence of its own.                                                |
                                                                              |
Commit	Line	Data
e14743d1	1	UTF-8 decoder capability and stress test
	2	----------------------------------------
	3
	4	Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19
	5
	6	This test file can help you examine, how your UTF-8 decoder handles
	7	various types of correct, malformed, or otherwise interesting UTF-8
	8	sequences. This file is not meant to be a conformance test. It does
	9	not prescribes any particular outcome and therefore there is no way to
	10	"pass" or "fail" this test file, even though the texts suggests a
	11	preferable decoder behaviour at some places. The aim is instead to
	12	help you think about and test the behaviour of your UTF-8 on a
	13	systematic collection of unusual inputs. Experience so far suggests
	14	that most first-time authors of UTF-8 decoders find at least one
	15	serious problem in their decoder by using this file.
	16
	17	The test lines below cover boundary conditions, malformed UTF-8
	18	sequences as well as correctly encoded UTF-8 sequences of Unicode code
	19	points that should never occur in a correct UTF-8 file.
	20
	21	According to ISO 10646-1:2000, sections D.7 and 2.3c, a device
	22	receiving UTF-8 shall interpret a "malformed sequence in the same way
	23	that it interprets a character that is outside the adopted subset" and
	24	"characters that are not within the adopted subset shall be indicated
	25	to the user" by a receiving device. A quite commonly used approach in
	26	UTF-8 decoders is to replace any malformed UTF-8 sequence by a
	27	replacement character (U+FFFD), which looks a bit like an inverted
	28	question mark, or a similar symbol. It might be a good idea to
	29	visually distinguish a malformed UTF-8 sequence from a correctly
	30	encoded Unicode character that is just not available in the current
	31	font but otherwise fully legal, even though ISO 10646-1 doesn't
	32	mandate this. In any case, just ignoring malformed sequences or
	33	unavailable characters does not conform to ISO 10646, will make
	34	debugging more difficult, and can lead to user confusion.
	35
	36	Please check, whether a malformed UTF-8 sequence is (1) represented at
	37	all, (2) represented by exactly one single replacement character (or
	38	equivalent signal), and (3) the following quotation mark after an
	39	illegal UTF-8 sequence is correctly displayed, i.e. proper
	40	resynchronization takes place immageately after any malformed
	41	sequence. This file says "THE END" in the last line, so if you don't
	42	see that, your decoder crashed somehow before, which should always be
	43	cause for concern.
	44
	45	All lines in this file are exactly 79 characters long (plus the line
	46	feed). In addition, all lines end with "\|", except for the two test
	47	lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls
	48	U+0000 and U+007F. If you display this file with a fixed-width font,
	49	these "\|" characters should all line up in column 79 (right margin).
	50	This allows you to test quickly, whether your UTF-8 decoder finds the
	51	correct number of characters in every line, that is whether each
	52	malformed sequences is replaced by a single replacement character.
	53
	54	Note that as an alternative to the notion of malformed sequence used
	55	here, it is also a perfectly acceptable (and in some situations even
	56	preferable) solution to represent each individual byte of a malformed
	57	sequence by a replacement character. If you follow this strategy in
	58	your decoder, then please ignore the "\|" column.
	59
	60
	61	Here come the tests: \|
	62	\|
	63	1 Some correct UTF-8 text \|
	64	\|
65	(The codepoints for this test are: \|
66	U+03BA U+1F79 U+03C3 U+03BC U+03B5 --ryan.) \|
67	\|
68	You should see the Greek word 'kosme': "κόσμε" \|
69	\|
70	\|
71	2 Boundary condition test cases \|
72	\|
73	2.1 First possible sequence of a certain length \|
74	\|
75	(byte zero skipped...there's a null added at the end of the test. --ryan.) \|
76	\|
77	2.1.2 2 bytes (U-00000080): "\80" \|
78	2.1.3 3 bytes (U-00000800): "ࠀ" \|
79	2.1.4 4 bytes (U-00010000): "𐀀" \|
80	\|
81	(5 and 6 byte sequences were made illegal in rfc3629. --ryan.) \|
82	2.1.5 5 bytes (U-00200000): "��" \|
83	2.1.6 6 bytes (U-04000000): "��" \|
84	\|
85	2.2 Last possible sequence of a certain length \|
86	\|
87	2.2.1 1 byte (U-0000007F): "\7f" \|
88	2.2.2 2 bytes (U-000007FF): "߿" \|
89	\|
90	(Section 5.3.2 below calls this illegal. --ryan.) \|
91	2.2.3 3 bytes (U-0000FFFF): "" \|
92	\|
93	(5 and 6 bytes sequences, and 4 bytes sequences > 0x10FFFF were made illegal \|
94	in rfc3629, so these next three should be replaced with a invalid \|
95	character codepoint. --ryan.) \|
96	2.2.4 4 bytes (U-001FFFFF): "��" \|
97	2.2.5 5 bytes (U-03FFFFFF): "��" \|
98	2.2.6 6 bytes (U-7FFFFFFF): "��" \|
99	\|
100	2.3 Other boundary conditions \|
101	\|
102	2.3.1 U-0000D7FF = ed 9f bf = "퟿" \|
103	2.3.2 U-0000E000 = ee 80 80 = "" \|
104	2.3.3 U-0000FFFD = ef bf bd = "�" \|
105	2.3.4 U-0010FFFF = f4 8f bf bf = "􏿿" \|
106	\|
107	(This one is bogus in rfc3629. --ryan.) \|
108	2.3.5 U-00110000 = f4 90 80 80 = "��" \|
109	\|
110	3 Malformed sequences \|
111	\|
112	3.1 Unexpected continuation bytes \|
113	\|
114	Each unexpected continuation byte should be separately signalled as a \|
115	malformed sequence of its own. \|
116	\|
117