Simon Sapin, a Rust and Servo developer, wrote the WTF-8 specification, a UTF-8 superset that encodes unpaired surrogates.
HN commenters cite WTF-8 as a favorite Unicode reference, explaining how ill-formed UTF-16 is handled and noting its use in Rust's standard library.
For the benefit of others (the link is nonobvious), here's Markus Kuhn's presentation of UTF-8B:
http://hyperreal.org/~est/utf-8b/releases/utf-8b-20060413043934/kuhn-utf-8b.html
The tl;dr is to map an invalid UTF-8 byte n to code point U+DC00 + n, which puts it in the code point range reserved for the second part of a surrogate pair. (In UTF-16, a 16-bit value between D800 and DBFF followed by a 16-bit value between DC00 and DFFF is used to encode a code point that cannot fit in 16 bits. Since these "surrogate pairs" happen only in that order, there is room to extend UTF-16 by assigning a meaning to a DC00-DFFF value seen without a D800-DBFF before it.) Since the surrogate code points are defined as not "Unicode scalar values" and cannot exist in well-formed "Unicode text", and therefore cannot be decoded from well-formed UTF-8, there's no risk of confusion.
There are some similarities with the extension of UTF-8 encoding that is sometimes called "WTF-8" https://simonsapin.github.io/wtf-8/. WTF-8 lets unchecked purportedly-UTF-16 data be parsed as a sequence of code points, encoded into an extension of UTF-8, and round-tripped back into the original array of uint16s. UTF-8B lets unchecked purportedly-UTF-8 data be parsed as a sequence of code points, encoded into an extension of UTF-16, and round-tripped back into the original array of uint8s. They're not quite compatible, because WTF-8 would encode U+DC80 as a three-byte sequence (ED B2 80), and UTF-8B would decode that…
That's not what invalid UTF-8 means. It doesn't mean replacing codepoints with other codepoints, it means that the encoding of the codepoints is invalid. Since JavaScript uses UTF-16 strings, it's possible to create JavaScript strings that don't have a valid UTF-8 representation (I believe unpaired surrogates can do this). Also, if you are working with raw bytes, there is no guarantee that it has any encoding at all (this seems to be what the article is about). There is WTF-8 [0] that proposes a possible solution to the UTF-16 problem, but purely as an internal representation and not as an interchange format. I think the Rust standard library uses WTF-8 to represent Windows wide-strings, which allows ASCII and UTF-8 algorithms to work on them, while still being able to represent any UTF-16 string.
[0]: https://simonsapin.github.io/wtf-8/
Thanks for sharing!
This is the first time I saw a mention of WTF-8 [1] and WTF-16. From the spec description this seems strange to use this interoperability "hack" (using the word from the spec) as the foundation of the string proposal. I wonder if they could use UTF-8 instead and keep WTF-16 for interoperability with JavaScript.
> WTF-8 [...] is a superset of UTF-8 that encodes surrogate code points if they are not in a pair. It represents, in a way compatible with UTF-8, text from systems such as JavaScript and Windows that use UTF-16 internally but don’t enforce the well-formedness invariant that surrogates must be paired. WTF-8 is a hack intended to be used internally in self-contained systems with components that need to support potentially ill-formed UTF-16 for legacy reasons.
[1] https://simonsapin.github.io/wtf-8/
Surrogates are technically a UTF-16 only thing. Realizing that sometimes they nevertheless escape out into the wild, WTF-8 defines a superset of UTF-8 that encodes them:
https://simonsapin.github.io/wtf-8/
To be clear, this is not an official Unicode spec. It's a hack (albeit a pretty natural and obvious one) to deal with systems that don't do Unicode quite right.
I recently came across some old code that narrows wchar_t to UCS-2 by zeroing out the high-order bytes. Even though my test was careful not to generate any surrogates in the input, they showed up in the output when a randomly generated code point like U+1DF7C was mangled into U+DF7C.
A corrupted value like that is not necessarily a great example of something you want to preserve, but it's the sort of thing that late 90s code assumed about Unicode.
The value spaces are in fact asymmetric. In Unicode jargon, UTF-8 is a "list of Unicode scalar values" while WTF-16, i.e. UTF-16 as seen in practice, is a "list of Unicode code points (except surrogate pairs)". Unicode code points are a superset of Unicode scalar values, aka "list of Unicode Code Points except surrogate code points", hence conversion from WTF-16 to UTF-8 is lossy. In WTF-16 languages, this is not a problem because the system is designed for WTF-16, but it becomes one if the value space is restricted further, here to UTF-8.
That's why, in such mixed systems, https://simonsapin.github.io/wtf-8/ is typically used. Not UTF-8. And Wasm is such a mixed system.
WTF-8 and WTF-16 are a thing: https://simonsapin.github.io/wtf-8/
Basically WTF-16 is any sequence of 16-bit integers, and is thus a superset of UTF-16 (because UTF-16 doesn't allow certain combinations of integers, mainly surrogate code points that exist outside of surrogate pairs).
Then WTF-8 is what you get if you naively transform invalid UTF-16 into UTF-8. It is a superset of UTF-8.
This is very useful when dealing with applications like Java and Javascript that treat strings as sequences of 16-bit code points, even though not all such strings are valid UTF-16.