Henri Sivonen, a Mozilla engineer, writes about Unicode, character encodings, HTML, and Rust and C++ interoperability.
HN commenters cite his string length article as a good read on Unicode, and also link his UTF-8 size and PNG gamma posts.
The PyUnicode object is what represents a str. If the UTF-8 bytes are ever requested, then a bytes object is created on demand and cached as part of the PyUnicode, being freed with the PyUnicode itself is freed.
Separately from that the codepoints making up the string are stored in a straight forward array allowing random access. The size of each codepoint can be 1, 2, or 4 bytes. When you create a PyUnicode you have to specify the maximum codepoint value which is rounded up to 127, 255, 65535, or 1,114,111. That determines if 1, 2, or 4 bytes is used.
If the maxiumum codepoint value is 127 then that array representation can be used for the UTF-8 directly. So the answer to your question is that many strings are stored as UTF-8 because all the codepoints are <= 127.
Separately from that, advancing through strings should not be done by codepoints anyway. A user perceived character (aka grapheme cluster) is made up of one or more codepoints. For example an e with an accent could be the e codepoint followed by a combining accent codepoint. The phoenix emoji is really the bird emoji, a zero width joiner, and then fire emoji. Some writing systems used by hundreds of millions of people are similar to having consonants, with combining marks to represent vowels.
This - - is 5 codepoints. There is a good blog post diving into it and how various languages report its "length". https://hsivonen.fi/string-length/
Source: I've just finished implementing Unicode TR29 which…
I appreciate your reply, which I never expected in this situation.
If my understanding is correct, your thesis is that Unicode should be hidden from application programmers as much as possible, much like the fact that GC hides memory management so to say. Not to say Unicode is bad or even shouldn't exist at all, but something like that it has to be abstracted away. This is a much more reasonable than most (quote-unquote) Unicode criticisms indeed. I'm not sure whether this is possible in the near future however, for reasons I'll work out here.
----
From perspectives of API consumers, most if not all programming languages have a suboptimal design for the human text. In fact the type name "string" itself is inappropriate, its name comes from the assumption that a human text is a string of symbols, which is not incorrect but not helpful either. A proper "human text" type (or a collection of them) should ideally be able to do the following:
- An abillity to contain additional linguistic informations like locales, grammatical genders or numbers if possible. Some can be guessed, some can be retrieved from external contexts (e.g. HTTP `Accept-Language`), some have to be retrived with a consent. Any text operation should retain them if the corresponding text is also retained.
- A language-aware formatter. For example `"Total: $
{n} files"` should automatically change "files" to "file" when `n` is 1. Moreover, `"Total: $
{n} $
{objectName}"` should do the same if `objectName` is…
> Could you please describe a bit more about the browser-CDM interaction and what is implemented there?
I don't really know what this interaction looks like in non-Firefox browsers. Last I checked, the CDMs Chrome ships didn't work with Chromium, but I don't know whether that's still true, and I don't know whether the browser-side bits involved are implemented at all in Chromium or just in Chrome. Likewise, I don't know whether the CDM interaction bits in Safari are in the public WebKit repo or not. IE's source is not available, of course. In the case of Firefox, https://hacks.mozilla.org/2014/05/reconciling-mozillas-mission-and-w3c-eme/ really does cover most of the details. We put together an API that made sense on our (Firefox) end internally. We then worked with some CDM vendors to integrate their products, by building shims to convert the API their CDMs exposed to the API we wanted to be using internally. That's probably all I can say on the subject.
https://hsivonen.fi/eme/ has a reasonably in-depth discussion of the way these bits fit together from someone who was much more intimately involved in this than I was.
> Why do you think the browser-CDM interaction was left unspecified?
Because the people writing the spec pushed back pretty explicitly on doing so, claiming that this would take too much time and overconstrain things too much in terms of both CDM and browser implementations.
> Wouldn't a standard be beneficial to all parties, even CDM developers…
A tricky thing w/r/t browsers is that they are often really crappy at figuring out what color space a PNG image is supposed to be in. I think they often intentionally strip out the color space info from PNG images for the sake of helping PNG images blend smoothly into CSS / Flash / etc., for which they also don’t do correct color management. [The CSS spec says that colors should be interpreted as sRGB, but browsers don’t actually follow the spec.]
As a result of this problem, it’s difficult to judge whether image resizing specifically is being done incorrectly, or whether the image is just being interpreted incorrectly at any size.
You’ll notice if you open your first test image (http://www.4p8.com/eric.brasseur/gamma-1.0-or-2.2.png) in Apple’s Preview app, you get the correct behavior, but this is different from the behavior in Safari, for example, which shows something incorrect at every size. Stranger still, the behavior in Safari is roughly “correct” while resizing a browser window, but as soon as you let go of the mouse, it reverts to being incorrect.
There are multiple possible ways to embed color space info into PNG images, and the PNG spec is actually really terrible and confusing on this subject which results in lots of inconsistencies from one implementation to another. If I remember correctly some of them are treated better by browsers than others, but I’m not sure if there’s a way to encode images such that it will be handled properly everywhere (I mean,…
> The argument for DRM is that it ships in a sandbox built by Mozilla.
Actually, I was long since wondering how exactly that sandbox works, so if you have some more information about that, I'd appreciate it.
Henri Sivonen's general explanation of EME and CDMs[1] tells the following:
> A CDM could be bundled with the browser, downloaded separately, bundled with the operating system, embedded in hardware as firmware running in a second domain of computing (such as ARM TrustZone) or wired into hardware. EME leaves this aspect implementation-dependent. [...]
EME does not specify the output abstraction for CDMs. It leaves open several options. The CDM could:
- Merely perform decryption and hand back the encoded media (e.g. H.264) to the browser.
[...]
- Perform decryption and decoding and then work together with the GPU so that not even the operating system gets the opportunity to read the pixels back from the GPU.
Meanwhile, Mozilla's implementation of EME seems to be substantially more restrictive[2]:
> Firefox does not load [the CDM] directly. Instead, we wrap it into an open-source sandbox. In our implementation, the CDM will have no access to the user’s hard drive or the network. Instead, the sandbox will provide the CDM only with communication mechanism with Firefox for receiving encrypted data and for displaying the results.
[...]
in Firefox the sandbox prohibits the CDM from fingerprinting the user’s device. Instead, the CDM asks the sandbox to supply a…
There's one part of this document that I would push extremely hard against, and that's the notion that "extended grapheme clusters" are the one true, right way to think of characters in Unicode, and therefore any language that views the length in any other way is doing it wrong.
The truth of the matter is that there are several different definitions of "character", depending on what you want to use it for. An extended grapheme cluster is largely defined on "this visually displays as a single unit", which isn't necessarily correct for things like "display size in a monospace font" or "thing that gets deleted when you hit backspace." Like so many other things in Unicode, the correct answer is use-case dependent.
(And for this reason, String iteration should be based on codepoints--it's the fundamental level on which Unicode works, and whatever algorithm you want to use to derive the correct answer for your purpose will be based on codepoint iteration. hsivonen's article (https://hsivonen.fi/string-length/), linked in this one, does try to explain why extended grapheme clusters is the wrong primitive to use in a language.)