Linux/Unix > Python Essential Reference, Second Edition > 9. Input and Output > Unicode I/O | See All Titles

< BACK | Make Note | Bookmark | CONTINUE >

155117184014003188065099048180054212144238241179195140058238111161105085094126174128119118

### Unicode I/O

Internally, Unicode strings are represented as sequences of 16-bit integer character values.As in 8-bit strings, all characters are the same size, and most common string operations are simply extended to handle strings with a larger range of character values. However, whenever Unicode strings are converted to a stream of bytes, a number of issues arise. First, to preserve compatibility with existing software, it may be desirable to convert Unicode to an 8-bit representation compatible with software that expects to receive ASCII or other 8-bit data. Second, the use of 16-bit characters introduces problems related to byte ordering. For a Unicode character `U+HHLL`, little endian encoding places the low-order byte first, as in `LL HH`. Big endian encoding places the high-order byte first, as in `HH LL`. Because of this difference, it’s generally not possible to simply write raw Unicode data to a file without also specifying the encoding used.

To address these problems, external representation of Unicode strings is always done according to a specific encoding rule. This rule precisely defines how Unicode characters are to be represented as a byte sequence. In Chapter 4, encoding rules were first described for the `unicode()` function and the `s` `.encode()` string method. For example:

    a = u"M\u00fcller"
    b = "Hello World"
    c = a.encode('utf-8')     # Convert a to a UTF-8 string
    d = unicode(b)            # Convert b to a Unicode string

For Unicode I/O, a similar technique is implemented through the use of the built-in `codecs` module. The `codecs` module contains a collection of functions for converting byte-oriented data to and from Unicode strings under a variety of different data-encoding schemes.A specific codec is selected by calling the `codecs.lookup(``encoding` `)` function. This function returns a four-element tuple `(``enc_func, decode_func, stream_reader, stream_writer` `)`. For example:

    import codecs
    (utf8_encode, utf8_decode, utf8_reader, utf8_writer) = \
              codecs.lookup('utf-8')

The `enc_func` `(``u` `[,``errors` `])` function takes a Unicode string `u` and returns a tuple `(``s` `,` `len` `)` in which `s` is an 8-bit string containing a portion or all of the Unicode string `u` , converted into the desired encoding, and `len` contains the number of Unicode characters converted. The `decode_func` `(``s` `[,``errors` `])` function takes an 8-bit string `s` and returns a tuple `(``u` `,` `len` `)` containing a Unicode string `u` and the number of characters in `s` that were converted. The `errors` parameter determines how errors are handled and is one of `'strict'`, `'ignore'`, or `'replace'`. In `'strict'` mode, encoding errors raise a `UnicodeError` exception. In `'ignore'` mode, encoding errors are ignored. In `'replace'` mode, characters that can’t be converted are replaced by a replacement character. The replacement character is `U+FFFD` in Unicode and `'?'` in 8-bit strings.

`stream_reader` is a class that implements a wrapper for reading Unicode data from a file object. Calling `stream_reader` `(``file` `)` returns an object in which the `read()`, `readline()`, and `readlines()` methods read Unicode string data. `stream_writer` is a class that provides a wrapper for writing Unicode to a file object. Calling `stream_writer` `(``file` `)` returns a file object in which the `write()` and `writelines()` methods translate Unicode strings to the given encoding on the output stream.

The following example illustrates how to read and write UTF-8 encoded Unicode data using these functions:

    # Output Unicode data to a file
    ustr = u'M\u00fcller'         # A Unicode string

    outf = utf8_writer(open('foo','w'))   # Create UTF-8 output stream
    outf.write(ustr)
    outf.close()

    # Read Unicode data from a file
    infile = utf8_reader(open('bar'))
    ustr = infile.read()
    infile.close()

When working with Unicode files, the data encoding is usually embedded in the file itself. For example, XML parsers may look at the first few bytes of the string `'<?xml ...>'` to determine the document encoding. If the first four values are `3C 3F 78 6D` (`'<?xm'`), the encoding is assumed to be UTF-8. If the first four values are `00 3C 00 3F` or `3C 00 3F 00`, the encoding is assumed to be UTF-16 big endian or UTF-16 little endian, respectively. Alternatively, a document encoding may appear in MIME headers or as an attribute of other document elements. For example:

    <?xml ... encoding="ISO-8859-1" .... ?>

When the encoding is read from a document, code similar to the following might be used:

    f = open("somefile")
    # Determine encoding
    ...
    (encoder,decoder,reader,writer) = codecs.lookup(encoding)
    f = reader(f)    # Wrap file with Unicode reader
    data = f.read()  # Read Unicode data
    f.close()

#### Unicode Data Encoding

Table 9.2 lists all of the currently available encoders in the `codecs` module.

##### Table 9.2. Encoders in the codecs Module

Encoder | Description
--- | ---
`'ascii'` | ASCII encoding
`'latin-1', 'iso-8859-1'` | Latin-1 or ISO-8859-1 encoding
`'utf-8'` | 8-bit variable length encoding
`'utf-16'` | 16-bit variable length encoding
`'utf-16-le'` | UTF-16, but with explicit little endian encoding
`'utf-16-be'` | UTF-16, but with explicit big endian encoding
`'unicode-escape'` | Same format as `u"``string` `"`
`'raw-unicode-escape'` | Same format as `ur"``string` `"`

The following paragraphs describe each of the encoders in more detail.

`'ascii'` encoding:

In `'ascii'` encoding, character values are confined to the ranges `[0,0x7f]` and `[U+0000, U+007F]`. Any character outside this range is invalid.

`'iso-8859-1'` or `'latin-1'` encoding:

Characters can be any 8-bit value in the ranges `[0,0xff]` and `[U+0000, U+00FF]`. Values in the range `[0,0x7f]` correspond to characters from the ASCII character set.Values in the range `[0x80,0xff]` correspond to characters from the ISO-8859-1 or extended ASCII character set. Any characters with values outside the range `[0,0xff]` result in an error.

`'utf-8'` encoding:

UTF-8 is a variable-length encoding that allows all Unicode characters to be represented. A single byte is used to represent ASCII characters in the range 0–127. All other characters are represented by multibyte sequences of two or three bytes. The encoding of these bytes is shown in the following table.

Unicode Characters | Byte 0 | Byte 1 | Byte 2
--- | --- | --- | ---
`U+0000 - U+007F` | `0` `nnnnnnn` |
`U+007F - U+07FF` | `110` `nnnnn` | `10` `nnnnnn` |
`U+0800 - U+FFFF` | `1110` `nnnn` | `10` `nnnnnn` | `10` `nnnnnn`

For two-byte sequences, the first byte always starts with the bit sequence `110`. For three-byte sequences, the first byte starts with the bit sequence `1110`. All subsequent data bytes in multibyte sequences start with the bit sequence `10`.

In full generality, the UTF-8 format allows for multibyte sequences of up to six bytes. In Python, four-byte UTF-8 sequences are used to encode a pair of Unicode characters known as a surrogate pair. Both characters have values in the range `[U+D800, U+DFFF]` and are combined to encode a 20-bit character value. The surrogate encoding is as follows:The four-byte sequence `111100` `nn` `10` `nnnnnn` `10` `nnmmmm` `10` `mmmmmm` is encoded as the pair `U+D800 +` `N` , `U+DC00 +` `M` , where `N` is the upper 10 bits and `M` is the lower 10 bits of the 20-bit character encoded in the four-byte UTF-8 sequence. Five- and six-byte UTF-8 sequences (denoted by starting bit sequences of `111110` and `1111110`, respectively) are used to encode character values up to 32 bits in length. These values are not supported by Python and currently result in a `UnicodeError` exception if they appear in an encoded data stream.

UTF-8 encoding has a number of useful properties that allow it to be used by older software. First, the standard ASCII characters are represented in their standard encoding. This means that a UTF-8 encoded ASCII string is indistinguishable from a traditional ASCII string. Second, UTF-8 doesn’t introduce embedded null bytes for multibyte character sequences. Thus, existing software based on the C library and programs that expect null-terminated 8-bit strings will work with UTF-8 strings. Finally, UTF-8 encoding preserves the lexicographic ordering of strings. That is, if `a` and `b` are Unicode strings and `a < b`, then `a < b` also holds when `a` and `b` are converted to UTF-8. Therefore, sorting algorithms and other ordering algorithms written for 8-bit strings will also work for UTF-8.

`'utf-16'` , `'utf-16-be'` , and `'utf-16-le'` encoding:

UTF-16 is a variable-length 16-bit encoding in which Unicode characters are written as 16-bit values. Unless a byte ordering is specified, big endian encoding is assumed. In addition, a byte-order marker of `U+FEFF` can be used to explicitly specify the byte ordering in a UTF-16 data stream. In big endian encoding, `U+FEFF` is the Unicode character for a zero-width nonbreaking space, whereas the reversed value `U+FFFE` is an illegal Unicode character. Thus, the encoder can use the byte sequence `FE FF` or `FF FE` to determine the byte ordering of a data stream. When reading Unicode data, Python removes the byte-order markers from the final Unicode string.

`'utf-16-be'` encoding explicitly selects UTF-16 big endian encoding. `'utf-16-le'` encoding explicitly selects UTF-16 little ending encoding.

Although there are extensions to UTF-16 to support character values greater than 16 bits, none of these extensions are currently supported.

`'unicode-escape'` and `'raw-unicode-escape'` encoding:

These encoding methods are used to convert Unicode strings to the same format as used in Python Unicode string literals and Unicode raw string literals. For example:

    s = u'u\14a8\u0345\u2a34'
    t = s.encode('unicode-escape')   #t = '\u14a8\u0345\u2a34'

#### Unicode Character Properties

In addition to performing I/O, programs that use Unicode may need to test Unicode characters for various properties such as capitalization, numbers, and whitespace. The `unicodedata` module provides access to a database of character properties. General character properties can be obtained with the `unicodedata.category(``c` `)` function. For example, `unicodedata.category(u"A")` returns `'Lu'`, signifying that the character is an uppercase letter. Further details about the Unicode character database and the `unicodedata` module can be found in Appendix A.

| Last updated on 3/28/2002 Python Essential Reference, Second Edition, © 2002 New Riders Publishing

< BACK | Make Note | Bookmark | CONTINUE >

© 2002, O'Reilly & Associates, Inc.