Linux/Unix > Python Essential Reference, Second Edition > 4. Operators and Expressions > Unicode Strings | See All Titles

< BACK | Make Note | Bookmark | CONTINUE >

155117184014003188065099048180054212144238241179195140058238110137104030195073070001013164

### Unicode Strings

The use of standard strings and Unicode strings in the same program presents a number of subtle complications. This is because such strings may be used in a variety of operations, including string concatenation, comparisons, dictionary key lookups, and as arguments to built-in functions.

To convert a standard string `s` to a Unicode string, the built-in function `unicode(``s` `[,` `encoding` `[,``errors` `]])` is used. To convert a Unicode string `u` to a standard string, the string method `u` `.encode([``encoding` `[,` `errors` `]])` is used. Both of these conversion operators require the use of a special encoding rule that specifies how 16-bit Unicode character values are mapped to a sequence of 8-bit characters in standard strings and vice versa. The encoding parameter is specified as a string and consists of one of the following values:

Value | Description
--- | ---
`'ascii'` | 7-bit ASCII
`'latin-1' or 'iso-8859-1'` | ISO 8859-1 Latin-1
`'utf-8'` | 8-bit variable-length encoding
`'utf-16'` | 16-bit variable-length encoding (may be little or big endian)
`'utf-16-le'` | UTF-16, little-endian encoding
`'utf-16-be'` | UTF-16, big-endian encoding
`'unicode-escape'` | Same format as Unicode literals `u"string"`
`'raw-unicode-escape'` | Same format as raw Unicode literals `ur"string"`

The default encoding is set in the `site` module and can be queried using `sys.getdefaultencoding()`. In most cases, the default encoding is `'ascii'`, which means that ASCII characters with values in the range [0x00,0x7f] are directly mapped to Unicode characters in the range [U+0000, U+007F]. Details about the other encodings can be found in Chapter 9, “Input and Output.”

When converting string values, a `UnicodeError` exception may be raised if a character that can’t be converted is encountered. For instance, if the encoding rule is `'ascii'`, a Unicode character such as U+1F28 can’t be converted because its value is too large. Similarly, the string `"\xfc"` can’t be converted to Unicode because it contains a character outside the range of valid ASCII character values. The `errors` parameter determines how encoding errors are handled. It’s a string with one of the following values:

Value | Description
--- | ---
`'strict'` | Raises a `UnicodeError` exception for decoding errors
`'ignore'` | Ignores invalid characters
`'replace'` | Replaces invalid characters with a replacement character (U+FFFD in Unicode, `'?'` in standard strings)

The default error handling is `'strict'`.

When standard strings and Unicode strings are mixed in an expression, standard strings are automatically coerced to Unicode using the built-in `unicode()` function.

For example:

    s = "hello"
    t = u"world"
    w = s + t          # w = unicode(s) + t

When Unicode strings are used in string methods that return new strings (as described in Chapter 3), the result is always coerced to Unicode. For example:

    a = "Hello World"
    b = a.replace("World", u"Bob")  # Produces u"Hello Bob"

Furthermore, even if zero replacements are made and the result is identical to the original string, the final result is still a Unicode string.

If a Unicode string is used as the format string with the `%` operator, all the arguments are first coerced to Unicode and then put together according to the given format rules. If a Unicode object is passed as one of the arguments to the `%` operator, the entire result is coerced to Unicode at the point at which the Unicode object is expanded. For example:

    c = "%s %s" % ("Hello", u"World") # c = "Hello " + u"World"
    d = u"%s %s" % ("Hello", "World") # d = u"Hello " + u"World"

When applied to Unicode strings, the `str()` and `repr()` functions automatically coerce the value back to a standard string. For Unicode string `u` , `str(``u` `)` produces the value `u` `.encode()` and `repr(``u` `)` produces `u"%s" % repr(``u` `.encode('unicode-escape'))`.

In addition, most library and built-in functions that only operate with standard strings will automatically coerce Unicode strings to a standard string using the default encoding. If such a coercion is not possible, a `UnicodeError` exception is raised.

Standard and Unicode strings can be compared. In this case, standard strings are coerced to Unicode using the default encoding before any comparison is made. This coercion also occurs whenever comparisons are made during list and dictionary operations. For example, `'x' in [u'x', u'y', u'z']` coerces `'x'` to Unicode and returns true. For character containment tests such as `'W' in u'Hello World'` the character `'W'` is coerced to Unicode before the test.

When computing hash values with the `hash()` function, standard strings and Unicode strings produce identical values, provided that the Unicode string only contains characters in the range [U+0000, U+007F]. This allows standard strings and Unicode strings to be used interchangeably as dictionary keys, provided that the Unicode strings are confined to ASCII characters. For example:

    a = { }
    a[u"foo"] = 1234
    print a["foo"]        # Prints 1234

However, it should be noted that this dictionary key behavior may not hold if the default encoding is ever changed to something other than `'ascii'` or if Unicode strings contain non–ASCII characters. For example, if `'utf-8'` is used as a default character encoding, it’s possible to produce pathological examples in which strings compare as equal, but have different hash values. For example:

    a = u"M\u00fcller"       # Unicode string
    b = "M\303\274ller"      # utf-8 encoded version of a
    print a == b             # Prints '1', true
    print hash(a)==hash(b)   # Prints '0', false

| Last updated on 3/28/2002 Python Essential Reference, Second Edition, © 2002 New Riders Publishing

< BACK | Make Note | Bookmark | CONTINUE >

© 2002, O'Reilly & Associates, Inc.