The section goes deeper into string internals. This knowledge will be useful for you if you plan to deal with emoji, rare mathematical or hieroglyphic characters, or other rare symbols.
XXmust be two hexadecimal digits with a value between
\xXXis the character whose Unicode code is
\xXXnotation supports only two hexadecimal digits, it can be used only for the first 256 Unicode characters.
These first 256 characters include the Latin alphabet, most basic syntax characters, and some others. For example,
"\x7A"is the same as
XXXXmust be exactly 4 hex digits with the value between
\uXXXXis the character whose Unicode code is
Characters with Unicode values greater than
U+FFFFcan also be represented with this notation, but in this case, we will need to use a so called surrogate pair (we will talk about surrogate pairs later in this chapter).
X…XXXXXXmust be a hexadecimal value of 1 to 6 bytes between
10FFFF(the highest code point defined by Unicode). This notation allows us to easily represent all existing Unicode characters.
All frequently used characters have 2-byte codes (4 hex digits). Letters in most European languages, numbers, and the basic unified CJK ideographic sets (CJK – from Chinese, Japanese, and Korean writing systems), have a 2-byte representation.
So rare symbols that require more than 2 bytes are encoded with a pair of 2-byte characters called “a surrogate pair”.
As a side effect, the length of such symbols is
We actually have a single symbol in each of the strings above, but the
length property shows a length of
Getting a symbol can also be tricky, because most language features treat surrogate pairs as two characters.
For example, here we can see two odd characters in the output:
Pieces of a surrogate pair have no meaning without each other. So the alerts in the example above actually display garbage.
Technically, surrogate pairs are also detectable by their codes: if a character has the code in the interval of
0xd800..0xdbff, then it is the first part of the surrogate pair. The next character (second part) must have the code in interval
0xdc00..0xdfff. These intervals are reserved exclusively for surrogate pairs by the standard.
One can see the difference here:
That said, if we take from position 1 (and that’s rather incorrect here), then they both return only the 2nd part of the pair:
You will find more ways to deal with surrogate pairs later in the chapter Iterables. There are probably special libraries for that too, but nothing famous enough to suggest here.
We can’t just split a string at an arbitrary position, e.g. take
str.slice(0, 4) and expect it to be a valid string, e.g.:
Here we can see a garbage character (first half of the smile surrogate pair) in the output.
Just be aware of it if you intend to reliably work with surrogate pairs. May not be a big problem, but at least you should understand what happens.
In many languages, there are symbols that are composed of the base character with a mark above/under it.
For instance, the letter
a can be the base character for these characters:
Most common “composite” characters have their own code in the Unicode table. But not all of them, because there are too many possible combinations.
To support arbitrary compositions, the Unicode standard allows us to use several Unicode characters: the base character followed by one or many “mark” characters that “decorate” it.
For instance, if we have
S followed by the special “dot above” character (code
\u0307), it is shown as Ṡ.
If we need an additional mark above the letter (or below it) – no problem, just add the necessary mark character.
For instance, if we append a character “dot below” (code
\u0323), then we’ll have “S with dots above and below”:
This provides great flexibility, but also an interesting problem: two characters may visually look the same, but be represented with different Unicode compositions.
To solve this, there exists a “Unicode normalization” algorithm that brings each string to the single “normal” form.
It is implemented by str.normalize().
It’s funny that in our situation
normalize() actually brings together a sequence of 3 characters to one:
\u1e68 (S with two dots).
In reality, this is not always the case. The reason is that the symbol
Ṩ is “common enough”, so Unicode creators included it in the main table and gave it the code.
If you want to learn more about normalization rules and variants – they are described in the appendix of the Unicode standard: Unicode Normalization Forms, but for most practical purposes the information from this section is enough.