blob: abd16f7253bf7254ad1ae04cb5bee8d8c47e407a
1 | UTF-8 decoder capability and stress test |
2 | ---------------------------------------- |
3 | |
4 | Markus Kuhn <http://www.cl.cam.ac.uk/~mgk25/> - 2003-02-19 |
5 | |
6 | This test file can help you examine, how your UTF-8 decoder handles |
7 | various types of correct, malformed, or otherwise interesting UTF-8 |
8 | sequences. This file is not meant to be a conformance test. It does |
9 | not prescribes any particular outcome and therefore there is no way to |
10 | "pass" or "fail" this test file, even though the texts suggests a |
11 | preferable decoder behaviour at some places. The aim is instead to |
12 | help you think about and test the behaviour of your UTF-8 on a |
13 | systematic collection of unusual inputs. Experience so far suggests |
14 | that most first-time authors of UTF-8 decoders find at least one |
15 | serious problem in their decoder by using this file. |
16 | |
17 | The test lines below cover boundary conditions, malformed UTF-8 |
18 | sequences as well as correctly encoded UTF-8 sequences of Unicode code |
19 | points that should never occur in a correct UTF-8 file. |
20 | |
21 | According to ISO 10646-1:2000, sections D.7 and 2.3c, a device |
22 | receiving UTF-8 shall interpret a "malformed sequence in the same way |
23 | that it interprets a character that is outside the adopted subset" and |
24 | "characters that are not within the adopted subset shall be indicated |
25 | to the user" by a receiving device. A quite commonly used approach in |
26 | UTF-8 decoders is to replace any malformed UTF-8 sequence by a |
27 | replacement character (U+FFFD), which looks a bit like an inverted |
28 | question mark, or a similar symbol. It might be a good idea to |
29 | visually distinguish a malformed UTF-8 sequence from a correctly |
30 | encoded Unicode character that is just not available in the current |
31 | font but otherwise fully legal, even though ISO 10646-1 doesn't |
32 | mandate this. In any case, just ignoring malformed sequences or |
33 | unavailable characters does not conform to ISO 10646, will make |
34 | debugging more difficult, and can lead to user confusion. |
35 | |
36 | Please check, whether a malformed UTF-8 sequence is (1) represented at |
37 | all, (2) represented by exactly one single replacement character (or |
38 | equivalent signal), and (3) the following quotation mark after an |
39 | illegal UTF-8 sequence is correctly displayed, i.e. proper |
40 | resynchronization takes place immageately after any malformed |
41 | sequence. This file says "THE END" in the last line, so if you don't |
42 | see that, your decoder crashed somehow before, which should always be |
43 | cause for concern. |
44 | |
45 | All lines in this file are exactly 79 characters long (plus the line |
46 | feed). In addition, all lines end with "|", except for the two test |
47 | lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls |
48 | U+0000 and U+007F. If you display this file with a fixed-width font, |
49 | these "|" characters should all line up in column 79 (right margin). |
50 | This allows you to test quickly, whether your UTF-8 decoder finds the |
51 | correct number of characters in every line, that is whether each |
52 | malformed sequences is replaced by a single replacement character. |
53 | |
54 | Note that as an alternative to the notion of malformed sequence used |
55 | here, it is also a perfectly acceptable (and in some situations even |
56 | preferable) solution to represent each individual byte of a malformed |
57 | sequence by a replacement character. If you follow this strategy in |
58 | your decoder, then please ignore the "|" column. |
59 | |
60 | |
61 | Here come the tests: | |
62 | | |
63 | 1 Some correct UTF-8 text | |
64 | | |
65 | You should see the Greek word 'kosme': "κόσμε" | |
66 | | |
67 | 2 Boundary condition test cases | |
68 | | |
69 | 2.1 First possible sequence of a certain length | |
70 | | |
71 | 2.1.1 1 byte (U-00000000): " |
72 | 2.1.2 2 bytes (U-00000080): "" | |
73 | 2.1.3 3 bytes (U-00000800): "ࠀ" | |
74 | 2.1.4 4 bytes (U-00010000): "𐀀" | |
75 | 2.1.5 5 bytes (U-00200000): "" | |
76 | 2.1.6 6 bytes (U-04000000): "" | |
77 | | |
78 | 2.2 Last possible sequence of a certain length | |
79 | | |
80 | 2.2.1 1 byte (U-0000007F): "" |
81 | 2.2.2 2 bytes (U-000007FF): "߿" | |
82 | 2.2.3 3 bytes (U-0000FFFF): "" | |
83 | 2.2.4 4 bytes (U-001FFFFF): "" | |
84 | 2.2.5 5 bytes (U-03FFFFFF): "" | |
85 | 2.2.6 6 bytes (U-7FFFFFFF): "" | |
86 | | |
87 | 2.3 Other boundary conditions | |
88 | | |
89 | 2.3.1 U-0000D7FF = ed 9f bf = "" | |
90 | 2.3.2 U-0000E000 = ee 80 80 = "" | |
91 | 2.3.3 U-0000FFFD = ef bf bd = "�" | |
92 | 2.3.4 U-0010FFFF = f4 8f bf bf = "" | |
93 | 2.3.5 U-00110000 = f4 90 80 80 = "" | |
94 | | |
95 | 3 Malformed sequences | |
96 | | |
97 | 3.1 Unexpected continuation bytes | |
98 | | |
99 | Each unexpected continuation byte should be separately signalled as a | |
100 | malformed sequence of its own. | |
101 | | |
102 | 3.1.1 First continuation byte 0x80: "" | |
103 | 3.1.2 Last continuation byte 0xbf: "" | |
104 | | |
105 | 3.1.3 2 continuation bytes: "" | |
106 | 3.1.4 3 continuation bytes: "" | |
107 | 3.1.5 4 continuation bytes: "" | |
108 | 3.1.6 5 continuation bytes: "" | |
109 | 3.1.7 6 continuation bytes: "" | |
110 | 3.1.8 7 continuation bytes: "" | |
111 | | |
112 | 3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): | |
113 | | |
114 | " | |
115 | | |
116 | | |
117 | " | |
118 | | |
119 | 3.2 Lonely start characters | |
120 | | |
121 | 3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), | |
122 | each followed by a space character: | |
123 | | |
124 | " | |
125 | " | |
126 | | |
127 | 3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), | |
128 | each followed by a space character: | |
129 | | |
130 | " " | |
131 | | |
132 | 3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), | |
133 | each followed by a space character: | |
134 | | |
135 | " " | |
136 | | |
137 | 3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), | |
138 | each followed by a space character: | |
139 | | |
140 | " " | |
141 | | |
142 | 3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), | |
143 | each followed by a space character: | |
144 | | |
145 | " " | |
146 | | |
147 | 3.3 Sequences with last continuation byte missing | |
148 | | |
149 | All bytes of an incomplete sequence should be signalled as a single | |
150 | malformed sequence, i.e., you should see only a single replacement | |
151 | character in each of the next 10 tests. (Characters as in section 2) | |
152 | | |
153 | 3.3.1 2-byte sequence with last byte missing (U+0000): "" | |
154 | 3.3.2 3-byte sequence with last byte missing (U+0000): "" | |
155 | 3.3.3 4-byte sequence with last byte missing (U+0000): "" | |
156 | 3.3.4 5-byte sequence with last byte missing (U+0000): "" | |
157 | 3.3.5 6-byte sequence with last byte missing (U+0000): "" | |
158 | 3.3.6 2-byte sequence with last byte missing (U-000007FF): "" | |
159 | 3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "" | |
160 | 3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "" | |
161 | 3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "" | |
162 | 3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "" | |
163 | | |
164 | 3.4 Concatenation of incomplete sequences | |
165 | | |
166 | All the 10 sequences of 3.3 concatenated, you should see 10 malformed | |
167 | sequences being signalled: | |
168 | | |
169 | "" | |
170 | | |
171 | 3.5 Impossible bytes | |
172 | | |
173 | The following two bytes cannot appear in a correct UTF-8 string | |
174 | | |
175 | 3.5.1 fe = "" | |
176 | 3.5.2 ff = "" | |
177 | 3.5.3 fe fe ff ff = "" | |
178 | | |
179 | 4 Overlong sequences | |
180 | | |
181 | The following sequences are not malformed according to the letter of | |
182 | the Unicode 2.0 standard. However, they are longer then necessary and | |
183 | a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 | |
184 | decoder" should reject them just like malformed sequences for two | |
185 | reasons: (1) It helps to debug applications if overlong sequences are | |
186 | not treated as valid representations of characters, because this helps | |
187 | to spot problems more quickly. (2) Overlong sequences provide | |
188 | alternative representations of characters, that could maliciously be | |
189 | used to bypass filters that check only for ASCII characters. For | |
190 | instance, a 2-byte encoded line feed (LF) would not be caught by a | |
191 | line counter that counts only 0x0a bytes, but it would still be | |
192 | processed as a line feed by an unsafe UTF-8 decoder later in the | |
193 | pipeline. From a security point of view, ASCII compatibility of UTF-8 | |
194 | sequences means also, that ASCII characters are *only* allowed to be | |
195 | represented by ASCII bytes in the range 0x00-0x7f. To ensure this | |
196 | aspect of ASCII compatibility, use only "safe UTF-8 decoders" that | |
197 | reject overlong UTF-8 sequences for which a shorter encoding exists. | |
198 | | |
199 | 4.1 Examples of an overlong ASCII character | |
200 | | |
201 | With a safe UTF-8 decoder, all of the following five overlong | |
202 | representations of the ASCII character slash ("/") should be rejected | |
203 | like a malformed UTF-8 sequence, for instance by substituting it with | |
204 | a replacement character. If you see a slash below, you do not have a | |
205 | safe UTF-8 decoder! | |
206 | | |
207 | 4.1.1 U+002F = c0 af = "" | |
208 | 4.1.2 U+002F = e0 80 af = "" | |
209 | 4.1.3 U+002F = f0 80 80 af = "" | |
210 | 4.1.4 U+002F = f8 80 80 80 af = "" | |
211 | 4.1.5 U+002F = fc 80 80 80 80 af = "" | |
212 | | |
213 | 4.2 Maximum overlong sequences | |
214 | | |
215 | Below you see the highest Unicode value that is still resulting in an | |
216 | overlong sequence if represented with the given number of bytes. This | |
217 | is a boundary test for safe UTF-8 decoders. All five characters should | |
218 | be rejected like malformed UTF-8 sequences. | |
219 | | |
220 | 4.2.1 U-0000007F = c1 bf = "" | |
221 | 4.2.2 U-000007FF = e0 9f bf = "" | |
222 | 4.2.3 U-0000FFFF = f0 8f bf bf = "" | |
223 | 4.2.4 U-001FFFFF = f8 87 bf bf bf = "" | |
224 | 4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "" | |
225 | | |
226 | 4.3 Overlong representation of the NUL character | |
227 | | |
228 | The following five sequences should also be rejected like malformed | |
229 | UTF-8 sequences and should not be treated like the ASCII NUL | |
230 | character. | |
231 | | |
232 | 4.3.1 U+0000 = c0 80 = "" | |
233 | 4.3.2 U+0000 = e0 80 80 = "" | |
234 | 4.3.3 U+0000 = f0 80 80 80 = "" | |
235 | 4.3.4 U+0000 = f8 80 80 80 80 = "" | |
236 | 4.3.5 U+0000 = fc 80 80 80 80 80 = "" | |
237 | | |
238 | 5 Illegal code positions | |
239 | | |
240 | The following UTF-8 sequences should be rejected like malformed | |
241 | sequences, because they never represent valid ISO 10646 characters and | |
242 | a UTF-8 decoder that accepts them might introduce security problems | |
243 | comparable to overlong UTF-8 sequences. | |
244 | | |
245 | 5.1 Single UTF-16 surrogates | |
246 | | |
247 | 5.1.1 U+D800 = ed a0 80 = "" | |
248 | 5.1.2 U+DB7F = ed ad bf = "" | |
249 | 5.1.3 U+DB80 = ed ae 80 = "" | |
250 | 5.1.4 U+DBFF = ed af bf = "" | |
251 | 5.1.5 U+DC00 = ed b0 80 = "" | |
252 | 5.1.6 U+DF80 = ed be 80 = "" | |
253 | 5.1.7 U+DFFF = ed bf bf = "" | |
254 | | |
255 | 5.2 Paired UTF-16 surrogates | |
256 | | |
257 | 5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "" | |
258 | 5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "" | |
259 | 5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "" | |
260 | 5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "" | |
261 | 5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "" | |
262 | 5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "" | |
263 | 5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "" | |
264 | 5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "" | |
265 | | |
266 | 5.3 Other illegal code positions | |
267 | | |
268 | 5.3.1 U+FFFE = ef bf be = "" | |
269 | 5.3.2 U+FFFF = ef bf bf = "" | |
270 | | |
271 | THE END | |
272 |