Linux/Unix > Python Essential Reference, Second Edition > A. The Python Library > Network Programming | See All Titles

< BACK | Make Note | Bookmark | CONTINUE >

155117184014003188065099048180054212144238241179195140058238111161105080005021224021244047

### Network Programming

This section describes the modules used to implement network servers and clients. Python provides extensive network support ranging from access to low-level network interfaces to high-level clients and frameworks for writing network applications. Before beginning, a very brief (and admittedly terse) introduction to network programming is presented. Readers are advised to consult a book such as UNIX Network Programming, Volume 1: Networking APIs - Sockets and XTI by W. Richard Stevens (Prentice Hall, 1997, ISBN 0-13-490012-X) for many of the advanced details.

Python’s network programming modules primarily support two Internet protocols: TCP and UDP. The TCP protocol is a reliable connection-oriented protocol used to establish a two-way communications stream between machines. UDP is a lower-level packet-based protocol (connectionless) in which machines send and receive discrete packets of information without formally establishing a connection. Unlike TCP, UDP communication is unreliable and thus inherently more complicated to manage in applications that require reliable communications. Consequently, most Internet protocols utilize TCP connections.

Both network protocols are handled through a programming abstraction known as a socket. A socket is an object similar to a file that allows a program to accept incoming connections, make outgoing connections, and send and receive data. Before two machines can establish a connection, both must create a socket object.

Furthermore, the machine receiving the connection (the server) must bind its socket object to a port. A port is a 16-bit number in the range 0–65535 that’s managed by the operating system and used by clients to uniquely identify servers. Ports 0–1023 are reserved by the system and used by common network protocols. The following table shows the port assignments for a number of common protocols:

Service | Port Number
--- | ---
FTP-Data | 20
FTP-Control | 21
Telnet | 23
SMTP (Mail) | 25
Finger | 79
HTTP (WWW) | 80
NNTP (News) | 119
HTTPS (Secure WWW) | 443

The process of establishing a TCP connection involves a precise sequence of steps on both the server and client, as shown in Figure A.1.

##### Figure A.1. TCP connection protocol.

For TCP servers, the socket object used to receive connections is not the same socket used to perform subsequent communication with the client. In particular, the `accept()` system call returns a new socket object that’s actually used for the connection. This allows a server to manage connections from a large number of clients simultaneously.

UDP communication is performed in a similar manner except that clients and servers don’t establish a “connection” with each other, as shown in Figure A.2.

##### Figure A.2. UDP connection protocol.

The following example illustrates the TCP protocol with a client and server written using the `socket` module. In this case, the server simply returns the current time to the client as a string.

    # Time server program
    from socket import *
    import time

    s = socket(AF_INET, SOCK_STREAM)          # Create a TCP socket
    s.bind(('',8888))                         # Bind to port 8888
    s.listen(5)                               # Listen, but allow no more than
                                              # 5 pending connections.
    while 1:
        client,addr = s.accept()    # Get a connection
        print 'Got a connection from ',addr
        client.send(time.ctime(time.time()))  # Send back to client
        client.close()

Here’s the client program:

    # Time client program
    from socket import *
    s = socket(AF_INET,SOCK_STREAM)          # Create a TCP socket
    s.connect(('foo.bar.com', 8888))         # Connect to the server
    tm = s.recv(1024)                        # Receive no more than 1024 bytes
    s.close()
    print 'The time is ', tm

The remainder of this section describes modules of two different flavors. First are modules and frameworks related to socket programming. Second are a variety of modules that implement the client-side interface to common Internet protocols. With the exception of HTTP, the details of these protocols are not presented. However, details can be found in the online documentation and in relevant Internet Request for Comments (RFCs) available at http://www.ietf.org. Where applicable, RFC numbers and sources for additional information are stated.

#### `asyncore`

The `asyncore` module is used to build network applications in which network activity is handled asynchronously as a series of events dispatched by an event loop, built using the `select()` system call. Such an approach is useful in network programs that want to provide concurrency, but without the use of threads or processes. This method can also provide the best performance for short transactions. All the functionality of this module is provided by the `dispatcher` class, which is a thin wrapper around an ordinary socket object.

    dispatcher([sock])

Base class defining an event-driven nonblocking socket object. `sock` is an existing socket object. If omitted, a socket must be created using the `create_socket()` method (described shortly). Once created, network events are handled by special handler methods (described later). In addition, all open dispatcher objects are saved in an internal list that’s used by a number of polling functions.

The following methods of the `dispatcher` class are called to handle network events. They should be defined in classes derived from `dispatcher`.

    d.handle_read()

Called when new data is available to be read from a socket.

    d.handle_write()

Called when an attempt to write data is made.

    d.handle_expt()

Called when out-of-band (OOB) data for a socket is received.

    d.handle_connect()

Called when a connection is made.

    d.handle_close()

Called when the socket is closed.

    d.handle_accept()

Called on listening sockets when a new connection arrives.

    d.readable()

This function is used by the `select()` loop to see whether the object is willing to read data. Returns `1` if so, `0` if not. This method is called to see if the `handle_read()` method should be called with new data.

    d.writable()

Called by the `select()` loop to see whether the object wants to write data. Returns `1` if so, `0` otherwise. This method is always called to see whether the `handle_write()` method should be called to produce output.

In addition to the preceding methods, the following methods are used to perform low-level socket operations. They’re similar to those available on a socket object.

    d.create_socket(family, type)

Creates a new socket.Arguments are the same as for `socket.socket()`.

    d.connect(address)

Makes a connection. `address` is a tuple (`host` `,` `port` ).

    d.send(data)

Sends `data` .

    d.recv(size)

Receives at most `size` bytes.

    d.listen([backlog])

Listens for incoming connections.

    d.bind(address)

Binds the socket to `address` . `address` is typically a tuple (`host` `,` `port` ).

    d.accept()

Accepts a connection. Returns a pair (`client` `,` `addr` ) where `client` is a socket object used to send and receive data on the connection and `addr` is the address of the client.

    d.close()

Closes the socket.

The following functions are used to handle events:

    poll([timeout [, ignore_exception]])

Polls all the open dispatcher objects for network events using `select()` and calls the appropriate handler functions if necessary. `timeout` is an optional timeout, `0.0` by default. `ignore_exception` , if set, causes all exceptions generated in event handlers to be ignored (the default).

    loop([timeout])

Polls for events indefinitely. Does nothing but repeatedly call `poll()`. `timeout` is the timeout period and is set to 30 seconds by default.

##### Example

The following example implements a minimalistic Web server using `asyncore`. It implements two classes— `asynhttp` for accepting connections and `asynclient` for processing client requests.

    # A minimal HTTP server with no error checking.
    import asyncore, socket
    import string, os, stat, mimetypes
    # Class that does nothing but accept connections
    class asynhttp(asyncore.dispatcher):
          def __init__(self, port):
               asyncore.dispatcher.__init__(self)
               self.create_socket(socket.AF_INET,socket.SOCK_STREAM)
               self.bind(('',port))
               self.listen(5)
          # Accept an incoming connection and create a client
          def handle_accept(self):
               client,addr = self.accept()
               print 'Connection from ', addr
               return asynclient(client)
    # Handle clients
    class asynclient(asyncore.dispatcher):
          def __init__(self, sock = None):
               asyncore.dispatcher.__init__(self,sock)
               self.got_request = 0       # Read HTTP request?
               self.request_data = []
               self.responsef = None      # Response file
               self.sent_headers = 0      # Send HTTP headers?
               self.clientf = sock.makefile('r+',0) # Request file
          # Only readable if request header not read
          def readable(self):
               if not self.got_request: return 1
          # Read request header (until blank line)
          def handle_read(self):
               data = string.strip(self.clientf.readline())
               if data:
                    self.request_data.append(data)
                    return
               self.got_request = 1
               request = string.split(self.request_data[0])
               if request[0] == 'GET':
                    filename = request[1][1:]
                    self.responsef = open(filename)
                     self.content_type,enc = mimetypes.guess_type(filename)
                     self.content_length = os.stat(filename)[stat.ST_SIZE]
               else:
                     self.close()
          # Only writable if a response is ready
          def writable(self):
               if self.responsef: return 1
               return 0
          # Write response data
          def handle_write(self):
               # Send HTTP headers if not sent yet
               if not self.sent_headers:
                    self.send('HTTP/1.0 200 OK\n')
                    if not self.content_type:
                            self.content_type = 'text/plain'
                    self.send('Content-type: %s\n' % (self.content_type,))
                    self.send('Content-length: %d\n\n' % (self.content_length,))
                    self.sent_headers = 1
               # Read some data and send it
               data = self.responsef.read(8192)
               if data:
                    sent = self.send(data)
                    # Adjust for unsent data
                    self.responsef.seek(sent-len(data),1)
               else:
                    self.responsef.close()
                    self.close()
    # Create the server
    a = asynhttp(80)
    # Poll forever
    asyncore.loop()

##### Notes

- This module requires the `select` module.

? See Also socket (252), select (250), httplib (243), SocketServer (258).

#### `BaseHTTPServer`

The `BaseHTTPServer` module defines two base classes used to implement HTTP servers.

    HTTPServer(server_address, request_handler)

Creates a new `HTTPServer` object. `server_address` is a tuple of the form (`host` `,` `port` ) on which the server will listen. `request_handler` is a class object used to handle requests (described shortly).

The `HTTPServer` class is derived from `SocketServer.TCPServer` and supports the same methods. In particular, the following functions are most relevant:

Function | Description
--- | ---
`h` `.handle_request()` | Processes a single request
`h` `.serve_forever()` | Handles an infinite number of requests

Requests are handled by defining a handler derived from the following class:

    BaseHTTPRequestHandler(request, client_address, server)

This class is used to handle HTTP requests. When a connection is received, the request and HTTP headers are parsed. An attempt is then made to execute a method of the form `do_REQUEST` based on the request type. For example, a `'GET'` method invokes `do_GET()` and a `'POST'` method invokes `do_POST`. By default, this class does nothing, so these methods must be defined in subclasses.

The following class variables are defined for `BaseHTTPRequestHandler`:

    BaseHTTPRequestHandler.server_version

Specifies the server software version string—for example, `'ServerName/1.2'`.

    BaseHTTPRequestHandler.sys_version

Python system version, such as `'Python/2.0'`.

    BaseHTTPRequestHandler.error_message_format

Format string used to build error messages sent to the client. The format string is applied to a dictionary containing the attributes `code`, `message`, and `explain`. For example:

    '''<head>
      <title>Error response</title>
      </head>
      <body>
      <h1>Error response</h1>
      <p>Error code %(code)d.
      <p>Message: %(message)s.
      <p>Error code explanation: %(code)s = %(explain)s.
      </body>'''

    BaseHTTPRequestHandler.protocol_version

HTTP protocol version used in responses. The default is `'HTTP/1.0'`.

    BaseHTTPRequestHandler.MessageClass

Class used to parse HTTP headers.The default is `mimetools.Message`.

    BaseHTTPRequestHandler.responses

Mapping of integer error codes to two-element tuples (`message` `,` `explain` ) that describe the problem.

An instance `b` of the `BaseHTTPRequestHandler` has the following attributes:

Attribute | Description
--- | ---
`b` `.client_address` | Client address as a tuple (`host` , `port` )
`b` `.command` | Request type such as `'GET'`, `'POST'`, `'HEAD'`, and so on
`b` `.path` | Contains the request path
`b` `.request_version` | HTTP version string from the request, such as `'HTTP/1.0'`
`b` `.headers` | HTTP headers, typically represented as `mimetools.Message` object
`b` `.rfile` | Input stream for optional input data
`b` `.wfile` | Output stream for writing a response back to the client

The following methods are used:

    b.handle()

Request dispatcher. Parses the request and calls a method of the form `do_*()`.

    b.send_error(code [, message])

Sends an error reply to the client. `code` is the numeric HTTP error code. `message` is an optional error message.

    b.send_response(code [, message])

Sends a response header. The HTTP response line is sent, followed by `Server` and `Date` headers.

    b.send_header(keyword, value)

Writes a MIME header entry to the output stream. `keyword` is the header keyword, `value` is its value.

    b.end_headers()

Sends a blank line to signal the end of the MIME headers.

    b.log_request([code [, size]])

Logs a successful request. `code` is the HTTP code and `size` is the size of the response in bytes (if available).

    b.log_error(format, ...)

Logs an error message. By default, `b` `.log_message()` is called.

    b.log_message(format, ...)

Logs an arbitrary message to `sys.stderr`. `format` is a format string applied to any additional arguments passed. The client address and current time are prefixed to every message.

    b.version_string()

Returns the server software’s version string—a combination of the `server_version` and `sys_version` variables.

    b.date_time_string()

Returns the current date and time, formatted for a header.

    b.log_date_time_string()

Returns the current date and time, formatted for logging.

    b.address_string()

Performs a name lookup on the client’s IP address and returns a hostname formatted for logging.

##### Example

The following example handles `GET` methods and simply echoes the request back to the client on a Web page.

    import BaseHTTPServer
    class EchoHandler(BaseHTTPServer.BaseHTTPRequestHandler):
          # Echo the request information back on a Web page
          def do_GET(self):
                 self.send_response(200)
                 self.send_header('Content-type','text/html')
                 self.end_headers()
                 self.wfile.write('''
    <html><head><title>Your Request</title></head>
    <body>
    <pre>
    You requested the following : %s
    The request headers were :
    %s
    </pre></body></html>
    ''' % (self.path, self.headers))

    server = BaseHTTPServer.HTTPServer(('',80),EchoHandler)
    server.serve_forever()

##### Note

- The contents of this module are rarely used directly. See the SimpleHTTPServer and CGIHTTPServer modules.

? See Also SimpleHTTPServer (251), CGIHTTPServer (239), SocketServer (258), httplib (243), mimetools (266).

#### `cgi`

The `cgi` module is used to implement CGI scripts in Web applications. CGI scripts are programs executed by a Web server when it wants to process user input submitted through an HTML form such as the following:

    <FORM ACTION='/cgi-bin/foo.cgi' METHOD='GET'>
    Your name : <INPUT type='Text' name='name' size='30'>
    Your email address: <INPUT type='Text' name='email' size='30'>
    <INPUT type='Submit' name='submit-button' value='Subscribe'>
    </FORM>

When the form is submitted, the Web server executes the CGI program `foo.cgi`. CGI programs receive input from two sources: `sys.stdin` and environment variables set by the server. The following table lists common environment variables set by Web servers:

Variable | Description
--- | ---
`AUTH_TYPE` | Authentication method
`CONTENT_LENGTH` | Length of data passed in `sys.stdin`
`CONTENT_TYPE` | Type of query data
`DOCUMENT_ROOT` | Document root directory
`GATEWAY_INTERFACE` | CGI revision string
`HTTP_ACCEPT` | MIME types accepted by the client
`HTTP_COOKIE` | Netscape persistent cookie value
`HTTP_FROM` | Email address of client (often disabled)
`HTTP_REFERER` | Referring URL
`HTTP_USER_AGENT` | Client browser
`PATH_INFO` | Extra path information passed
`PATH_TRANSLATED` | Translated version of `PATH_INFO`
`QUERY_STRING` | Query string
`REMOTE_ADDR` | Remote IP address of the client
`REMOTE_HOST` | Remote hostname of the client
`REMOTE_IDENT` | User making the request
`REMOTE_USER` | Authenticated username
`REQUEST_METHOD` | Method (`'GET'` or `'POST'`)
`SCRIPT_NAME` | Name of the program
`SERVER_NAME` | Server hostname
`SERVER_PORT` | Server port number
`SERVER_PROTOCOL` | Server protocol
`SERVER_SOFTWARE` | Name and version of the server software

As output, a CGI program writes to standard output `sys.stdout`. The gory details of CGI programming can be found in a book such as CGI Programming with Perl, 2nd Edition by Shishir Gundavaram (O’Reilly & Associates, 1999). For our purposes, there are really only two things to know. First, the contents of an HTML form are passed to a CGI program in a sequence of text known as a query string. In Python, the contents of the query string are accessed using the `FieldStorage` class. For example:

    import cgi
    form = cgi.FieldStorage()
    name = form['name'].value      # Get 'name' field from a form
    email = form['email'].value    # Get 'email' field from a form

Second, the output of a CGI program consists of two parts: an HTTP header and the raw data (which is typically HTML).A simple HTTP header looks like this:

    print 'Content-type: text/html'     # HTML Output
    print                               # Blank line (required!)

The rest of the output is the raw output. For example:

    print '<TITLE>My CGI Script</TITLE>'
    print '<H1>Hello World!</H1>'
    print 'You are %s (%s)' % (name, email)

Most of the work in the `cgi` module is performed by creating an instance of the `FieldStorage` class. This class reads the contents of a form by reading and parsing the query string passed in an environment variable or standard input. Because input can be read from standard input, only one instance should be created. An instance `f` of `FieldStorage` has the following attributes:

Attribute | Description
--- | ---
`f` `.name` | The field name, if specified
`f` `.filename` | Client-side filename used in uploads
`f` `.value` | Value as a string
`f` `.file` | File-like object from which data can be read
`f` `.type` | Content type
`f` `.type_options` | Dictionary of options specified on the content-type line of the HTTP request
`f` `.disposition` | The `'content-disposition'` field; `None` if not specified
`f` `.disposition_options` | Dictionary of disposition options
`f` `.headers` | A dictionary-like object containing all the HTTP header contents

In addition, the `cgi` module defines a class `MiniFieldStorage` that contains only the attribute’s name and value. This class is used to represent individual fields of a form passed in the query string, whereas `FieldStorage` is used to contain multiple fields and multipart data.

Instances of `FieldStorage` are accessed like a Python dictionary where the keys are the field names on the form. When accessed in this manner, the objects returned are themselves an instance of `FieldStorage` for multipart data or file uploads, an instance of `MiniFieldStorage` for simple fields, or a list of such instances in cases where a form contains multiple fields with the same name.

If a field represents an uploaded file, accessing the `value` attribute reads the entire file into memory as a string. Because this may consume a large amount of memory on the server, it may be preferable to read uploaded data in smaller pieces by reading from the `file` attribute directly. For instance, the following example reads uploaded data line by line:

    fileitem = form['userfile']
    if fileitem.file:
        # It's an uploaded file; count lines
        linecount = 0
        while 1:
            line = fileitem.file.readline()
            if not line: break
            linecount = linecount + 1

The following functions provide a more low-level CGI interface:

    escape(s [, quote])

Converts the characters `'&'`, `'<'`, and `'>'` in string `s` to HTML-safe sequences such as `'&'`, `'<'`, and `'>'`. If the optional flag `quote` is true, the double-quote character (`"`) is also translated.

    parse([fp [, environ [, keep_blank_values [, strict_parsing]]]])

Parses a form into a dictionary. `fp` is a file object from which data is read (defaults to `stdin`). `environ` is a dictionary containing environment variables (defaults to `os.environ`). `keep_blank_values` , if set to `1`, instructs the parser to map blank entries into empty strings. Otherwise, blank entries are ignored (the default). The `strict_parsing` option specifies what to do with parsing errors. By default, errors are ignored. If set, parsing errors result in a `ValueError` exception. Returns a dictionary mapping field names to lists of values.

    parse_header(string)

Parses the data supplied after an HTTP header field such as `'content-type'`. The data is split into a primary value and a dictionary of secondary parameters that are returned in a tuple. For example, this command:

    parse_header("text/html; a=hello; b='world'")

returns this result:

    ('text/html', {'a':'hello', 'b':'world'}).

    parse_multipart(fp,pdict)

Parses input of type `'multipart/form-data'` as is commonly used with file uploads. `fp` is the input file and `pdict` is a dictionary containing parameters of the content-type header. Returns a dictionary mapping field names to lists of values. This function doesn’t work with nested multipart data. The `FieldStorage` class should be used instead.

    parse_qs(qs [, keep_blank_values [, strict_parsing]]):

Parses a query string `qs` . `keep_blank_values` and `strict_parsing` have the same meaning as in `parse()`. Returns a dictionary mapping field names to lists of values.

    parse_qsl(qs [, keep_blank_values [, strict_parsing]])

Like `parse_qs()` except that a list of `(``name` `,` `value` `)` pairs is returned.

    print_directory()

Formats the contents of the current working directory in HTML. Used for debugging.

    print_environ()

Formats the shell environment in HTML. Used for debugging.

    print_environ_usage()

Prints a list of useful environment variables in HTML. Used for debugging.

    print_form(form)

Formats the data supplied on a form in HTML. `form` must be an instance of `FieldStorage`. Used for debugging.

    test()

Writes a minimal HTTP header and prints all the information provided to the script in HTML format. Primarily used for debugging.

##### Notes

- The process of installing a CGI program varies widely according to the type of Web server being used. Typically programs are placed in a special `cgi-bin` directory. A server may also require additional configuration.

- On UNIX, Python CGI programs may require a line such as the following as the first line of the program:

      #!/usr/local/bin/python
      import cgi
      ...

- To simplify debugging, it’s sometimes useful to set `sys.stderr` to s `ys.stdout`. This will force Python error messages to be sent to the output stream (which will then appear in the text sent to the browser).

- If you invoke an external program—for example, via the `os.system()` or `os.popen()` function—be careful not to pass arbitrary strings received from the client to the shell. This is a well-known security hole that hackers can use to execute arbitrary shell commands on the server (because the command passed to these functions is first interpreted by the UNIX shell as opposed to being executed directly). In particular, never pass any part of a URL or form data to a shell command unless it has first been thoroughly checked by making sure that the string contains only alphanumeric characters, dashes, underscores, and periods.

- On UNIX, don’t give a CGI program `setuid` mode. This is a security liability and not supported on all machines.

- Don’t use `'from cgi import *'` with this module. The `cgi` module defines a wide variety of names and symbols that you probably don’t want in your namespace.

- The original CGI specification can be found at http://hoohoo.ncsa.uiuc.edu/cgi/interface.html.

? See Also CGIHTTPServer (239).

#### `CGIHTTPServer`

The `CGIHTTPServer` module provides a simple HTTP server handler that can run CGI scripts. The server is defined by the following request handler class, intended for use with the `BaseHTTPServer` module:

    CGIHTTPRequestHandler(request, client_address, server)

Serves files from the current directory and all its subdirectories. In addition, the handler will run a file as a CGI script if it’s located in a special CGI directory. The handler supports both `GET` and `POST` methods.

The list of valid CGI directories is contained in the following attribute:

    CGIHTTPRequestHandler.cgi_directories

List of CGI directories. Defaults to `['/cgi-bin', '/htbin']`.

##### Example

    from BaseHTTPServer import HTTPServer
    from CGIHTTPServer import CGIHTTPRequestHandler
    import os
    # Change to the document root
    os.chdir('/home/httpd/html')
    # Start the CGI server
    serv = HTTPServer(('',80),CGIHTTPRequestHandler)
    serv.serve_forever()

##### Notes

- For security, CGI scripts are executed with a UID of user `nobody`.

- Problems with the CGI script will be translated to HTTP error 403.

- Requests are handled using the `do_GET` and `do_POST` methods, both of which can be redefined in subclasses.

- To prevent problems in the execution of CGI scripts, it’s usually a good idea to use CGI directory names that don’t contain any embedded whitespace.

? See Also BaseHTTPServer (p. x), SimpleHTTPServer (p. x), cgi (p. x), httplib (p. x).

#### `Cookie`

The `Cookie` module provides support for managing HTTP cookies. Cookies are used to provide state management in CGI scripts that implement sessions, user logins, shopping carts, and related features. To drop a cookie on a user’s browser, an HTTP server typically adds an HTTP header similar to the following to an HTTP response (see the httplib module):

    Set-Cookie: session=8273612; expires=Sun, 18-Feb-2001 15:00:00 GMT; \
                 path=/; domain=cs.uchicago.edu

Alternatively, a cookie can be set by embedding JavaScript in an HTML document:

    <SCRIPT LANGUAGE="JavaScript">
    document.cookie = "session=8273612; expires=Sun, 18-Feb-2001 15:00:00 GMT; \
       Path=/; Domain=cs.uchicago.edu;"
    </SCRIPT>

The `Cookie` module simplifies the task of generating cookie values by providing a special dictionary-like object that stores and manages collections of cookie values known as morsels. Each morsel has a name, a value, and a set of optional attributes containing meta-data to be supplied to the browser {`expires`, `path`, `comment`, `domain`, `max-age`, `secure`, `version`}. The name is usually a simple identifier such as `"``name` `"` and must not be the same as one of the meta-data names such as `"expires"` or `"path"`. The value is usually a short string. To create a cookie, simply create a cookie object like this:

    c = Cookie.SimpleCookie()

Once created, cookie values (morsels) can be set using ordinary dictionary assignment:

    c["session"] = 8273612
    c["user"] = "beazley"

Additional attributes of a specific morsel are set as follows:

    c["session"]["path"] = "/"
    c["session"]["domain"] = "cs.uchicago.edu"
    c["session"]["expires"] = "18-Feb-2001 15:00:00 GMT"

To output the cookie data as a set of HTTP headers, the `c.output()` method is used. For example:

    print c.output()
    # Produces two lines of output
    # Set-Cookie: session=8273612; expires=...; path=/; domain=...
    # Set-Cookie: user=beazley

When a browser sends a cookie back to an HTTP server, it is encoded as a string of `key=value` pairs such as `"session=8273612; user=beazley"`. Optional attributes such as `expires`, `path`, and `domain` are not returned. The cookie string can usually be found in the `HTTP_COOKIE` environment variable, which can be read by CGI applications. To recover cookie values, use code similar to the following:

    c = Cookie.SimpleCookie(os.environ["HTTP_COOKIE"])
    session = c["session"].value
    user    = c["user"].value

In the above example, the cookie instance `c` is derived from a class `BaseCookie` that provides the following common methods:

    c.value_decode(val)

Takes a string `val` and returns a decoded cookie value. This function is used to interpret a cookie value returned to a server by a browser.

    c.value_encode(val)

Takes an object `val` and returns it as an encoded string suitable for use in an HTTP header. A server would use this to encode cookie values being sent to the browser.

    c.output([attrs [,header [,sep]]])

Generates a string suitable for use in setting cookie values in HTTP headers. `attrs` is an optional list of the optional attributes to include (`"expires"`, `"path"`, `"domain"`, and so on). `header` is the HTTP header to use, `'Set-Cookie:'` by default. `sep` is the character used to join the headers together and is a newline by default.

    c.js_output([attrs])

Generates a string containing JavaScript code that will set the cookie if executed on a browser supporting JavaScript. `attrs` is an optional list of the attributes to include.

    c.load(rawdata)

Loads the cookie `c` with data found in `rawdata` . If `rawdata` is a string, it’s assumed to be in the same format as the `HTTP_COOKIE` environment variable in a CGI program. If `rawdata` is a dictionary, each `key-value` pair is interpreted by setting `c` `[``key` `]=``value` .

The following classes provide specialized implementations of cookie objects. Each of these classes is derived from a class `BaseCookie`.

    SimpleCookie([input])

Defines a cookie object in which cookie values are interpreted as simple strings. The `c` `.value_decode()` method is the identity function and the `c` `.value_encode()` method uses the `str()` function to generate encoded values.

    SerialCookie([input])

Defines a cookie object in which the cookie values are interpreted using the `pickle` module. The `c` `.value_decode()` method uses `pickle.loads()` and the `c` `.value_encode()` method uses `pickle.dumps()`. See the later security note.

    SmartCookie([input])

Like `SerialCookie()` except that strings are left unmodified. If the cookie value cannot be successfully unpickled, `c` `.value_decode()` returns its value as a string. In addition, the `c` `.value_encode()` method only calls `pickle.dumps()` if the value is not a string. See the later security note.

Internally, the `key/value` pairs used to store a cookie value are instances of a `Morsel` class.An instance `m` of `Morsel` behaves like a dictionary and allows the optional `"expires"`, `"path"`, `"comment"`, `"domain"`, `"max-age"`, `"secure"`, and `"version"` keys to be set. In addition, a morsel `m` has the following methods and attributes:

    m.value

The raw value of the cookie.

    m.coded_value

The encoded value of the cookie that would be sent to or received from the browser.

    m.key

The cookie name.

    m.set(key,value,coded_value)

Sets the values of `m` `.key`, `m` `.value`, and `m` `.coded_value`.

    m.isReservedKey(k)

Tests whether `k` is a reserved keyword such as `"expires"`, `path"`, `"domain"`, etc.

    m.output([attrs [,header]])

Produces the HTTP header string for this morsel. `attrs` is an optional list of the additional attributes to include (`"expires"`, `"path"`, and so on). `header` is the header string to use, `'Set-Cookie:'` by default.

    m.js_output([attrs])

Outputs JavaScript code that sets the cookie when executed.

    m.OutputString([attrs])

Returns the cookie string without any HTTP headers or JavaScript code.

If an error occurs during the parsing or generation of cookie values, a `CookieError` exception is raised.

##### Notes

- More information about persistent cookies can be found in almost any book on CGI programming. For an official specification, see RFC-2109.

- The `SerialCookie()` and `SmartCookie()` classes have known security problems, since the `pickle` module allows arbitrary client-code to be executed on the server during unpickling. These should not be used unless your application has a way to validate cookie values before unpickling.

- Most browsers place limits on the size and number of cookie values. You should limit the size of cookie data to a few hundred bytes at most.

? See Also cgi (235), httplib (243).

#### `ftplib`

The `ftplib` module is used to implement the client side of the FTP protocol. It’s rarely necessary to use this module directly, as the `urllib` module provides a higher-level interface. The following example illustrates the use of this module:

    >>> import ftplib
    >>> ftp = ftplib.FTP('ftp.python.org')
    >>> ftp.login()
    >>> ftp.retrlines('LIST')
    total 40
    drwxrwxr-x  12 root     4127         512 Apr  6 19:57 .
    drwxrwxr-x  12 root     4127         512 Apr  6 19:57 ..
    drwxrwxr-x   2 root     4127         512 Aug 25  1998 RCS
    lrwxrwxrwx   1 root     bin           11 Jun 29 18:34 README -> welcome.msg
    drwxr-xr-x   3 root     wheel        512 May 19  1998 bin
    ...
    >>> ftp.retrbinary('RETR README', open('README', 'wb').write)
    '226 Transfer complete.'
    >>> ftp.quit()

Consult the online documentation for a complete description of the functionality contained in this module.

? See Also urllib (260), Internet RFC 959, http://www.python.org/doc/lib/module-ftplib.html.

#### `httplib`

This module implements the client side of the Hypertext Transfer Protocol (HTTP) used in Web applications. Both HTTP/1.0 and HTTP/1.1 protocols are supported. In addition, if Python is configured with OpenSSL support, connections can be made using secure sockects. The HTTP/1.0 protocol is a simple text-based protocol that works as follows:

1. A client makes a connection to a Web server and sends a request header of the following form:

       GET /document.html HTTP/1.0
       Connection: Keep-Alive
       User-Agent: Mozilla/4.61 [en] (X11; U; SunOS 5.6 sun4u)
       Host: rustler.cs.uchicago.edu:8000
       Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
       Accept-Encoding: gzip
       Accept-Language: en
       Accept-Charset: iso-8859-1,*,utf-8

       Data (optional)
       ...

   The first line defines the request type, document (the selector), and protocol version. Following the request line are a series of header lines containing various information about the client, such as passwords, cookies, cache preferences, and client software. Following the header lines, a single blank line indicates the end of the header lines. After the header, data may appear in the event that the request is sending from a form or uploading a file. Each of the lines in the header should be terminated by a carriage return and a newline (`'\r\n'`).

2. The server sends a response of the following form:

       HTTP/1.0 200 OK
       Content-type: text/html
       Content-length: 72883 bytes
       ...
       Header: data

       Data
       ...

   The first line of the server response indicates the HTTP protocol version, a success code, and return message. Following the response line are a series of header fields that contain information about the type of the returned document, the document size, Web server software, cookies, and so forth. The header is terminated by a single blank line followed by the raw data of the requested document.

The following request methods are the most common:

Method | Description
--- | ---
`GET` | Get a document
`POST` | Post data to a form
`HEAD` | Return header information only
`PUT` | Upload data to the server

The following response codes are returned by servers:

Code | Description
--- | ---
Success Codes (2xx)
`200` | OK
`201` | Created
`202` | Accepted
`204` | No content
Redirection (3xx)
`300` | Multiple choices
`301` | Moved permanently
`302` | Moved temporarily
`303` | Not modified
Client Error (4xx)
`400` | Bad request
`401` | Unauthorized
`403` | Forbidden
`404` | Not found
Server Error (5xx)
`500` | Internal server error
`501` | Not implemented
`502` | Bad gateway
`503` | Service unavailable

A wide range of optional header fields can appear in both the request and response headers. These headers are specified in a format known as RFC 822, in which headers are specified in the form `Header: data`. For example:

    Date: Fri, 16 Jul 1999 17:09:33 GMT
    Server: Apache/1.3.6 (Unix)
    Last-Modified: Mon, 12 Jul 1999 19:08:14 GMT
    ETag: "741d3-44ec-378a3d1e"
    Accept-Ranges: bytes
    Content-Length: 17644
    Connection: close
    Content-Type: text/html

The following classes can be used to create HTTP/1.0 connections with a Web server:

    HTTP([host [, port]])

Establishes an HTTP/1.0 connection with an HTTP server. `host` is the hostname and `port` is an optional port number. If no port number is given, the port is extracted from the hostname if it’s of the form `'``host:port` `'`. Otherwise, port 80 is used. If no host is passed, no connection is made and the `connect()` method should be used to make the connection manually.

    HTTPS([host [, port [, key_file=kfile [, cert_file=cfile]])

Establishes a secure HTTP/1.0 connection. Works like `HTTP()` except that the default port is 443. In addition, optional keyword parameters `key_file` and `cert_file` specify the names of a client private-key file and a client certificate file to be supplied to the server.

An instance `h` of the `HTTP` or `HTTPS` class has the following methods:

    h.connect(host [, port])

Connects to the server given by `host` and `port` . This should be called only if the instance was created without a host.

    h.send(data)

Sends data to the server. This should only be used after the `endheaders()` method.

    h.putrequest(request, selector)

Sends a line to the server containing the request string, selector string, and the HTTP version (HTTP/1.0).

    h.putheader(header, argument [, ...])

Sends an RFC 822–style header to the server. It sends a line to the server consisting of the header, a colon and a space, and the first argument. If more arguments are given, continuation lines are sent, each consisting of a tab and an argument.

    h.endheaders()

Sends a blank line to the server, indicating the end of the headers.

    h.getreply()

Closes the sending end of the connection, reads the reply from the server, and returns a triple (`replycode` `,` `message` `,` `headers` ). `replycode` is the integer reply code from the request, such as `200` on success. `message` is the message string corresponding to the reply code. `headers` is an instance of the class `mimetools.Message`, containing the HTTP headers received from the server.

    h.getfile()

Returns a file object from which the data returned by the server can be read, using the `read()`, `readline()`, or `readlines()` method.

##### Example

    import httplib
    h = httplib.HTTP('www.python.org')
    h.putrequest('GET', '/index.html')
    h.putheader('Accept', 'text/html')
    h.putheader('Accept', 'text/plain')
    h.endheaders()
    errcode, errmsg, headers = h.getreply()
    print errcode # Should be 200
    f = h.getfile()
    data = f.read() # Get the raw HTML
    f.close()

HTTP/1.1 extends the HTTP/1.0 protocol by allowing multiple request/response data streams to be multiplexed over a single network connection. To handle multiple connections, HTTP/1.1 manages the network connection by putting it into one of three states:

State | Description
--- | ---
`_CS_IDLE` | Connection idle
`_CS_REQ_STARTED` | Request started
`_CS_REQ_SENT` | Request sent

New requests can be sent only when the connection is in the `_CS_IDLE` state. Once an HTTP request has been sent, the connection doesn’t return to the idle state until the server response has been received and read on the client.

To support HTTP/1.1, the following classes are provided:

    HTTPConnection(host [, port])

Creates an HTTP/1.1 connection. `host` is the hostname and `port` is the remote port number. The default port is 80. Returns an `HTTPConnection` instance.

    HTTPSConnection(host [, port [, key_file= kfile [, cert_file= cfile]]])

Like HTTP/1.1, but uses a secure socket connection. The default port is 443. `key_file` and `cert_file` are optional keyword arguments that specify client private-key and certificate files. Returns an `HTTPSConnection` instance.

An instance `h` of `HTTPConnection` or `HTTPSConnection` supports the following methods:

    h.connect()

Initializes the connection to the host and port given to `HTTPConnection()` or `HTPPSConnection()`.

    h.close()

Closes the connection.

    h.send(str)

Sends a string `str` to the server. Direct use of this function with HTTP/1.1 is discouraged, because it may break the underlying response/request protocol. It’s most commonly used to send data to the server after `h` `.endheaders()` has been called.

    h.putrequest(method, url)

Sends a request to the server. `method` is the HTTP method, such as `'GET'` or `'POST'`. `url` specifies the object to be returned, such as `'/index.html'`. If `h` is not in the `_CS_IDLE` state, a `CannotSendRequest` exception is generated. Otherwise, `h` is moved to `_CS_REQ_STARTED` state, in which additional headers can be added to the request.

    h.putheader(header, value)

Sends an RFC 822–style header to the server. It sends a line to the server, consisting of the header, a colon and a space, and the value. Raises a `CannotSendHeader` exception if `h` is not in the `_CS_REQ_STARTED` state.

    h.endheaders()

Sends a blank line to the server, indicating the end of the header lines. Changes the connection state to `_CS_REQ_SENT`.

    h.request(method, url [, body [, headers]])

Sends a complete HTTP request to the server. `method` and `url` have the same meaning as for `h` `.putrequest()`. `body` is an optional string containing data to upload to the server after the request has been sent. `headers` is a dictionary containing `header` `:``value` pairs to be given to the `h` `.putheader()` method.

    h.getresponse()

Gets a response from the server and returns an `HTTPResponse` instance that can be used to read data. Raises a `ResponseNotReady` exception if `h` is not in the `_CS_REQ_STATE`.

An `HTTPResponse` instance as returned by the `getresponse()` method supports the following methods:

    r.close()

Closes the connection. This doesn’t close the underlying HTTP connection. Only the file object used to read data in this specific response is closed.

    r.isclosed()

Returns true if the underlying connection has been closed.

    r.read([size])

Reads up to `size` bytes from the server. If `size` is omitted, all the data for this request is returned.

    r.getheader(name [, default])

Gets a response header. `name` is the name of the header. `default` is the default value to return if not found.

    r.version

HTTP version used by server.

    r.status

HTTP status code returned by the server.

    r.reason

HTTP error message returned by the server.

    r.length

Number of bytes left in the response.

    r.will_close

Set if the server will close the connection after the response has been sent.

##### Example

The following example shows how the `HTTPConnection` class can be used to open an HTTP/1.1 connection and fetch several files.

    import httplib

    files = [ '/index.html', '/doc/index.html', '/News.html' ]
    h = httplib.HTTPConnection("www.python.org",80)
    h.connect()

    for f in files:
        h.putrequest('GET','f')
        h.putheader('Accept','text/html')
        h.putheader('Accept','text/plain')
        h.endheaders()

        r = h.getresponse()
        if r.status == 200:
            data = r.read()
            print ":::: %s ::::" % f
            print data
        r.close()

    h.close()

##### Notes

- This module is used by the `urllib` module, which provides a higher-level interface for accessing URLs.

- Secure HTTP is not available unless Python has also been compiled with OpenSSL support.

? See Also urllib (260), mimetools (266), asyncore (229), BaseHTTPServer (232), SimpleHTTPServer (251), CGIHTTPServer (239).

#### `imaplib`

The `imaplib` module provides a low-level client-side interface for connecting to an IMAP4 mail server using the IMAP4rev1 protocol. Documents describing the protocol, as well as sources and binaries for servers implementing it, can be found at the University of Washington’s IMAP Information Center (http://www.cac.washington.edu/imap).

The following example shows how the module is used by opening a mailbox and printing all messages:

    import getpass, imaplib, string
    m = imaplib.IMAP4()
    m.login(getpass.getuser(), getpass.getpass())
    m.select()
    typ, data = m.search(None, 'ALL')
    for num in string.split(data[0]):
        typ, data = m.fetch(num, '(RFC822)')
        print 'Message %s\n%s\n' % (num, data[0][1])
    m.logout()

? See Also poplib (249), http://www.python.org/doc/lib/module-imaplib.html, http://www.cac.washington.edu/imap, Internet RFC 1730, RFC 2060.

#### `nntplib`

The `nntplib` module provides a low-level interface to the client side of NNTP (Network News Transfer Protocol). For details about using this module, see the online documentation (http://www.python.org/doc/lib/module-nntplib.html). The following example shows how the module can be used to post a news message from a file containing valid news headers:

    s = NNTP('news.foo.com')
    f = open('article')
    s.post(f)
    s.quit()

? See Also http://www.python.org/doc/lib/module-nntplib.html, Internet RFC 977.

#### `poplib`

The `poplib` module provides a low-level client-side connection to a POP3 mail server. Consult the online reference at http://www.python.org/doc/lib/module-poplib.html for specific details. The following example opens a mailbox and retrieves all messages:

    import getpass, poplib
    M = poplib.POP3('localhost')
    M.user(getpass.getuser())
    M.pass_(getpass.getpass())
    numMessages = len(M.list()[1])
    for i in range(numMessages):
        for j in M.retr(i+1)[1]:
             print j

? See Also http://www.python.org/doc/lib/module-poplib.html, Internet RFC 1725.

#### `robotparser`

The `robotparser` module provides a class that can be used to fetch and query information contained in the `robots.txt` files that Web sites use to instruct Web crawlers and spiders. The contents of this file typically look like this:

    # robots.txt
    User-agent: *
    Disallow: /warheads/designs   # Don't allow robots here

    RobotFileParser()

Creates an object that can be used to read and query a single `robots.txt` file.

An instance `r` of `RobotFileParser` has the following attributes and methods:

    r.set_url(url)

Sets the URL of the `robots.txt` file.

    r.read()

Reads the `robots.txt` file and parses it.

    r.parse(lines)

Parses a list of lines obtained from a `robots.txt` file.

    r.can_fetch(useragent, url)

Returns true if `useragent` is allowed to fetch `url` .

    r.mtime()

Returns the time at which the `robots.txt` file was last fetched.

    r.modified()

Sets the time at which `robots.txt` was last fetched to the current time.

##### Note

- Details about the `robots.txt` format can be found at http://info.webcrawler.com/mak/projects/robots/norobots.html.

#### `select`

The `select` module provides access to the `select()` system call. `select()` is typically used to implement polling or to multiplex processing across multiple input/output streams without using threads or subprocesses. On UNIX and Macintosh, it works for files, sockets, pipes, and most other file types. On Windows, it only works for sockets.

    select(iwtd, owtd, ewtd [, timeout])

Queries the input, output, and exceptional status of a group of file descriptors. The first three arguments are lists containing either integer file descriptors or objects with a method `fileno()` that can be used to return a file descriptor. The `iwtd` parameter specifies objects waiting for input, `owtd` specifies objects waiting for output, and `ewtd` specifies objects waiting for an exceptional condition. Each list may be empty. `timeout` is a floating-point number specifying a timeout period in seconds. If omitted, the function waits until at least one file descriptor is ready. If `0`, the function merely performs a poll and returns immediately. The return value is a triple of lists containing the objects that are ready. These are subsets of the first three arguments. If none of the objects is ready before the timeout occurs, three empty lists are returned. If an error occurs, a `select.error` exception raised. Its value is the same as that returned by `IOError` and `OSError`.

##### Example

The following code shows how `select()` could be used in an event loop that wants to periodically query a collection of sockets for an incoming connection:

    import socket, select
    # Create a few sockets
    s1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s1.bind(("",8888))
    s1.listen(5)
    s2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s2.bind(("",8889))
    s2.listen(5)
    # Event loop
    while 1:
          ... processing ...
          # Poll the sockets for activity
          input,output,exc = select.select([s1,s2],[],[],0)
          # Loop over all of the sockets that have pending input
          for sock in input:
               # Accept an incoming connection
         client = sock.accept()
         ... handle client ...
         client.close()
    # Done. Carry on.
    ... more processing ...

##### Note

- There’s usually an upper limit on the number of file selectors that can be given to `select()`. It’s often 64 for Windows and 256 for UNIX.

? See Also asyncore (229), socket (252), os (180).

#### `SimpleHTTPServer`

The `SimpleHTTPServer` module provides a simple HTTP server handler that can serve files from the current directory. The module defines the following handler class, intended for use with the `BaseHTTPServer` module:

    SimpleHTTPRequestHandler(request, client_address, server)

Serves files from the current directory and all its subdirectories. The class implements the `do_HEAD()` and `do_GET()` methods to support `HEAD` and `GET` requests, respectively. All `IOError` exceptions result in a `404 File not found` error. Attempts to access a directory result in a `403 Directory listing not supported` error.

The following class attributes are available:

    SimpleHTTPRequestHandler.server_version

Server version string.

    SimpleHTTPRequestHandler.extensions_map

A dictionary mapping suffixes into MIME types. Unrecognized file types are considered to be of type `'text/plain'`.

##### Example

    from BaseHTTPServer import HTTPServer
    from SimpleHTTPServer import SimpleHTTPRequestHandler
    import os
    # Change to the document root
    os.chdir("/home/httpd/html")
    # Start the SimpleHTTP server
    serv = HTTPServer(("",80),SimpleHTTPRequestHandler)
    serv.serve_forever()

? See Also BaseHTTPServer (232), CGIHTTPServer (239), httplib (243).

#### `smtplib`

The `smtplib` module provides a low-level SMTP client interface that can be used to send mail. For specific details about the module, see the online reference at http://www.python.org/doc/lib/module-smtplib.html. The following example shows how the module might be used by prompting the user for an address and sending a message:

    import string, sys
    import smtplib
    def prompt(prompt):
        sys.stdout.write(prompt + ": ")
        return string.strip(sys.stdin.readline())
    fromaddr = prompt("From")
    toaddrs = string.splitfields(prompt("To"), ',')
    print "Enter message, end with ^D:"
    msg = ""
    while 1:
        line = sys.stdin.readline()
        if not line:
            break
        msg = msg + line
    print "Message length is " + `len(msg)`
    server = smtplib.SMTP('localhost')
    server.sendmail(fromaddr, toaddrs, msg)
    server.quit()

? See Also poplib (249), http://www.python.org/doc/lib/module-smtplib.html, imaplib (248), Internet RFC 821 (Simple Mail Transfer Protocol), Internet RFC 1869 (SMTP Service Extensions).

#### `socket`

The `socket` module provides access to the BSD socket interface. Although it’s based on UNIX, this module is available on all platforms.

    fromfd(fd, family, type [, proto])

Creates a socket object from an integer file descriptor `fd` . The address family, socket type, and protocol number are the same as for `socket()`. The file descriptor must refer to a previously created socket. Returns an instance of `SocketType`.

    getfqdn([name])

Returns the fully qualified domain name of `name` . If `name` is omitted, the local machine is assumed. For example, `getfqdn("stonecrusher")` might return `"stonecrusher.cs.uchicago.edu"`.

    gethostbyname(hostname)

Translates a hostname such as `'www.python.org'` to an IP address. The IP address is returned as a string, such as `'132.151.1.90'`.

    gethostbyname_ex(hostname)

Translates a hostname to an IP address, but returns a triple (`hostname` `,` `aliaslist` `,` `ipaddrlist` ) in which `hostname` is the primary hostname, `aliaslist` is a list of alternative hostnames for the same address, and `ipaddrlist` is a list of IP addresses for the same interface on the same host. For example, `gethostbyname_ex('www.python.org')` returns `('parrot.python.org', ['www.python.org'], ['132.151.1.90'])`.

    gethostname()

Returns the hostname of the local machine.

    gethostbyaddr(ip_address)

Returns the same information as `gethostbyname_ex()`, given an IP address such as `'132.151.1.90'`.

    getprotobyname(protocolname)

Translates an Internet protocol name such as `'icmp'` to a protocol number (such as the value of `IPPROTO_ICMP`) that can be passed to the third argument of the `socket()` function.

    getservbyname(servicename, protocolname)

Translates an Internet service name and protocol name to a port number for that service. For example, `getservbyname('ftp', 'tcp')` returns `21`. The protocol name should be `'tcp'` or `'udp'`.

    ntohl(x)

Converts 32-bit integers from network (big-endian) to host byte order.

    ntohs(x)

Converts 16-bit integers from network to host byte order.

    htonl(x)

Converts 32-bit integers from host to network byte order.

    htons(x)

Converts 16-bit integers from host to network byte order.

    ssl(sock, key_file, cert_file)

Creates a client-side secure socket. `sock` is an existing socket instance that has already established a connection using its `connect()` method. `key_file` is the name of a client private-key file. `cert_file` is the name of a client certificate file. `key_file` and `cert_file` must both be set to `None` or set to the names of PEM format files containing the client key and certificate. This function is available only if Python has been configured with OpenSSL support. In addition, this function cannot be used to create server-side secure sockets. See the notes.

    socket(family, type [, proto])

Creates a new socket using the given address family, socket type, and protocol number. `family` is one of the following constants:

Constant | Description
--- | ---
`AF_INET` | IPv4 protocols (TCP, UDP)
`AF_UNIX` | UNIX domain protocols

The socket type is one of the following constants:

Constant | Description
--- | ---
`SOCK_STREAM` | Stream socket (TCP)
`SOCK_DGRAM` | Datagram socket (UDP)
`SOCK_RAW` | Raw socket (available with `AF_INET` only)
`SOCK_SEQPACKET` | Sequenced connection-mode transfer of records

The protocol number is usually omitted (and defaults to `0`). It’s usually used only in conjunction with raw sockets (`SOCK_RAW`) and is set to one of the following constants when used: `IPPROTO_ICMP`, `IPPROTO_IP`, `IPPROTO_RAW`, `IPPROTO_TCP`, `IPPROTO_UDP`.

To open a TCP connection, use `socket(AF_INET, SOCK_STREAM)`. To open a UDP connection, use `socket(AF_INET, SOCK_DGRAM)`. To open a raw IP socket, use `socket(AF_INET, SOCK_RAW)`. Access to raw sockets is privileged and will only succeed if the effective user ID is `0` (root) on UNIX systems. The function returns an instance of `SocketType` (described shortly).

Sockets are represented by an instance of type `SocketType`. The following methods are available on a socket `s` :

    s.accept()

Accepts a connection and returns a pair (`conn` `,` `address` ) where `conn` is a new socket object that can be used to send and receive data on the connection, and `address` is the address of the socket on the other end of the connection.

    s.bind(address)

Binds the socket to an address. The format of `address` depends on the address family. In most cases, it’s a tuple of the form (`hostname` `,` `port` ). For IP addresses, the empty string represents `INADDR_ANY`, and the string `'<broadcast>'` represents `INADDR_BROADCAST`. The `INADDR_ANY` hostname (the empty string) is used to indicate that the server allows connections on any Internet interface on the system. This is often used when a server is multihomed. The `INADDR_BROADCAST` hostname (`'<broadcast>'`) is used when a socket is being used to send a broadcast message.

    s.close()

Closes the socket. Sockets are also closed when they’re garbage-collected.

    s.connect(address)

Connects to a remote socket at `address` . The format of `address` depends on the address family, but it’s normally a pair (`hostname` `,` `port` ). Raises `socket.error` if an error occurs.

If you’re connecting to a server on the same computer, you can use the name `'localhost'` as the first argument to `s` `.connect()`.

    s.connect_ex(address)

Like `connect(``address` `)`, but returns `0` on success or the value of `errno` on failure.

    s.fileno()

Returns the socket’s file descriptor.

    s.getpeername()

Returns the remote address to which the socket is connected as a pair (`ipaddr` `,` `port` ). Not supported on all systems.

    s.getsockname()

Return the socket’s own address as a pair (`ipaddr` `,` `port` ).

    s.getsockopt(level, optname [, buflen])

Returns the value of a socket option. `level` defines the level of the option and is `SOL_SOCKET` for socket-level options or a protocol number such as `IPPROTO_IP` for protocol-related options. `optname` selects a specific option. If `buflen` is omitted, an integer option is assumed and its integer value is returned. If `buflen` is given, it specifies the maximum length of the buffer used to receive the option. This buffer is returned as a string, where it’s up to the caller to decode its contents using the `struct` module or other means. The following list shows commonly used option names for level `SOL_SOCKET`:

Option Name | Value | Description
--- | --- | ---
`SO_KEEPALIVE` | `0, 1` | Periodically probes the other end of the connection and terminates if it’s half-open
`SO_RCVBUF` | `int` | Size of receive buffer (in bytes)
`SO_SNDBUF` | `int` | Size of send buffer (in bytes)
`SO_REUSEADDR` | `0, 1` | Allows local address reuse
`SO_RCVLOWAT` | `int` | Number of bytes read before `select()` returns the socket as readable
`SO_SNDLOWAT` | `int` | Number of bytes available in send buffer before `select()` returns the socket as writable
`SO_RCVTIMEO` | `tvalue` | Timeout on receive calls in seconds
`SO_SNDTIMEO` | `tvalue` | Timeout on send calls in seconds
`SO_OOBINLINE` | `0, 1` | Places out-of-band data into the input queue
`SO_LINGER` | `linger` | Lingers on `close()` if the send buffer contains data
`SO_DONTROUTE` | `0, 1` | Bypasses routing table lookups
`SO_ERROR` | `int` | Gets error status
`SO_BROADCAST` | `0, 1` | Allows sending of broadcast datagrams
`SO_TYPE` | `int` | Gets socket type
`SO_USELOOPBACK` | `0, 1` | Routing socket gets copy of what it sends

`tvalue` is a binary structure that’s decoded as

    (second, microsec) = struct.unpack("ll", tvalue).

`linger` is a binary structure that’s decoded as

    (linger_onoff, linger_sec) = struct.unpack("ii", linger).

The following options are available for level `IPPROTO_IP`:

Option Name | Value | Description
--- | --- | ---
`IP_ADD_MEMBERSHIP` | `ipmreg` | Join multicast group (set only)
`IP_DROP_MEMBERSHIP` | `ipmreg` | Leave a multicast group (set only)
`IP_HDRINCL` | `int` | IP header included with data
`IP_MULTICAST_IF` | `inaddr` | Outgoing interface
`IP_MULTICAST_LOOP` | `uchar` | Loopback
`IP_MULTICAST_TTL` | `uchar` | Time to live
`IP_OPTIONS` | `char[44]` | IP header options
`IP_TOS` | `int` | Type of service
`IP_TTL` | `int` | Time to live

`inaddr` is a 32-bit binary structure containing an IP address `(struct.unpack('bbbb',` `inaddr` `))`. `ipmreg` is a 64-bit binary structure containing two IP addresses in the same format as `inaddr` . `uchar` is a one-byte unsigned integer as created by `struct.pack('b',``uvalue` `)`. `char[44]` is a string containing at most 44 bytes.

Not all options are available on all machines. Refer to an advanced networking book for specific details about each option.

    s.listen(backlog)

Starts listening for incoming connections. `backlog` specifies the maximum number of pending connections the operating system should queue before connections are refused. The value should be at least `1`, with `5` being sufficient for most applications.

    s.makefile([mode [, bufsize]])

Creates a file object associated with the socket. `mode` and `bufsize` have the same meaning as with the built-in `open()` function. The file object uses a duplicated version of the socket file descriptor, created using `os.dup()`, so the file object and socket object can be closed or garbage-collected independently.

    s.recv(bufsize [, flags])

Receives data from the socket. The data is returned as a string. The maximum amount of data to be received is specified by `bufsize` . `flags` provides additional information about the message and is usually omitted (in which case it defaults to zero). If used, it’s usually set to one of the following constants (system-dependent):

Constant | Description
--- | ---
`MSG_PEEK` | Look at data, but don’t discard (receive only)
`MSG_WAITALL` | Don’t return until the requested number of bytes have been read (receive only)
`MSG_OOB` | Receive/send out-of-band data
`MSG_DONTROUTE` | Bypass routing table lookup (send only)

    s.recvfrom(bufsize [, flags])

Like the `recv()` method except that the return value is a pair (`data` `,` `address` ) in which `data` is a string containing the data received and `address` is the address of the socket sending the data. The optional `flags` argument has the same meaning as for `recv()`. This function is primarily used in conjunction with the UDP protocol.

    s.send(string [, flags])

Sends data in `string` to a connected socket. The optional `flags` argument has the same meaning as for `recv()`, described earlier. Returns the number of bytes sent.

    s.sendto(string [, flags], address)

Sends data to the socket. `flags` has the same meaning as for `recv()`. `address` is a tuple of the form (`host` `,` `port` ) that specifies the remote address. The socket should not already be connected. Returns the number of bytes sent. This function is primarily used in conjunction with the UDP protocol.

    s.setblocking(flag)

If `flag` is zero, the socket is set to nonblocking mode. Otherwise, the socket is set to blocking mode (the default). In nonblocking mode, if a `recv()` call doesn’t find any data or if a `send()` call cannot immediately send the data, the `socket.error` exception is raised. In blocking mode, these calls block until they can proceed.

    s.setsockopt(level, optname, value)

Sets the value of the given socket option. `level` and `optname` have the same meaning as for `getsockopt()`. The value can be an integer or a string representing the contents of a buffer. In the latter case, it’s up to the caller to ensure that the string contains the proper data. See `getsockopt()` for socket option names, values, and descriptions.

    s.shutdown(how)

Shuts down one or both halves of the connection. If `how` is `0`, further receives are disallowed. If `how` is `1`, further sends are disallowed. If `how` is `2`, further sends and receives are disallowed.

##### Exception

    error

This exception is raised for socket or address-related errors. It returns a pair (`errno` `,` `mesg` ) with the error returned by the underlying system call.

##### Example

A simple example of a TCP connection is shown on page 226 (in the introduction to network programming).The following example illustrates a simple UDP client and server:

    # UDP message server
    # Receive small packets from anywhere and print them out
    import socket
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    s.bind(("",10000))
    while 1:
          data, address = s.recvfrom(256)
          print address[0], "said : ", data

    # UDP message client
    # Send a message packet to the server
    import socket
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    while 1:
          msg = raw_input("Say something : ")
          if msg:
              s.sendto(msg, ("servername",10000))
          else:
              break
    s.close()

##### Notes

- Not all constants and socket options are available on all platforms.

- The `socket` module currently doesn’t support a number of other network protocols such as IPX and IPv6. However, a number of additions to the `socket` module are likely to appear in future Python releases.

? See Also SocketServer (258), asyncore (229), select (250).

#### `SocketServer`

The `SocketServer` module is used to write TCP, UDP, and UNIX domain socket servers. Rather than having to implement servers using the low-level socket module, this module provides four classes that implement the above protocols:

    TCPServer(address, handler)

A server supporting the TCP protocol. `address` is a 2-tuple of the form (`host` `,` `port` ), where `host` is the hostname and `port` is the port number. Typically, `host` is set to the empty string. `handler` is an instance of a subclass of the `BaseRequestHandler` class described later.

    UDPServer(address, handler)

A server supporting the Internet UDP protocol. `address` and `handler` are the same as for `TCPServer()`.

    UnixStreamServer(address, handler)

A server implementing a stream-oriented protocol using UNIX domain sockets.

    UnixDatagramServer(address, handler)

A server implementing a datagram protocol using UNIX domain sockets.

Instances of all four server classes have the following methods and attributes:

    s.fileno()

Returns the integer file descriptor for the server socket.

    s.handle_request()

Waits for a request and handles it by creating an instance of the `handler` class (described shortly) and invoking its `handle()` method.

    s.serve_forever()

Handles an infinite number of requests.

    s.address_family

The protocol family of the server, either `socket.AF_INET` or `socket.AF_UNIX`.

    s.RequestHandlerClass

The user-provided request handler class that was passed to the server constructor.

    s.server_address

The address on which the server is listening, such as `('127.0.0.1', 80)`.

    s.socket

The socket object being used for incoming requests.

In addition, the server classes define the following class attributes (`<``ServerClass` `>` should be filled in with the name of one of the four available classes):

    <ServerClass>.request_queue_size

The size of the request queue that’s passed to the socket’s `listen()` method. The default value is `5`.

    <ServerClass>.socket_type

The socket type used by the server, such as `socket.SOCK_STREAM` or `socket.SOCK_DGRAM`.

Requests are handled by defining a subclass of the class `BaseRequestHandler`. When the server receives a connection, it creates an instance `h` of the `handler` class and invokes the following methods:

    h.finish()

Called to perform cleanup actions after the `handle()` method has completed. By default, it does nothing. It’s not called if either the `setup()` or `handle()` method generates an exception.

    h.handle()

This method is called to perform the actual work of a request. It’s called with no arguments, but several instance variables are set to useful values. `h` `.request` contains the request, `h` `.client_address` contains the client address, and `h` `.server` contains an instance of the server that called the handler. For stream services such as TCP, the `h` `.request` attribute is a socket object. For datagram services, it’s a string containing the received data.

    h.setup()

This method is called before the `handle()` method to perform initialization actions. By default, it does nothing.

The process of creating a server involves the following steps:

1. Define a request handler class by subclassing `BaseRequestHandler`.

2. Create an instance of one of the server classes by passing the server’s address and the request handler class.

3. Call the `handle_request()` or `serve_forever()` method of the server to process connections.

The following code illustrates the process for a very simple HTTP server that simply echoes the HTTP request back in a Web page:

    import SocketServer
    import socket
    import string
    # Read an HTTP request from a client and bounce it back in a Web page
    class EchoHandler(SocketServer.BaseRequestHandler):
       def handle(self):
               f = self.request.makefile()
               self.request.send("HTTP/1.0 200 OK\r\n")
               self.request.send("Content-type: text/plain\r\n\r\n")
               self.request.send("Received connection from %s\r\n\r\n" %
                                   (self.client_address,))
               while 1:
                   line = f.readline()
                   self.request.send(line)
                   if not string.strip(line):
                        break
               f.close()
    # Create the server and start serving
    serv = SocketServer.TCPServer(("",80),EchoHandler)
    serv.serve_forever()

By default, the server classes process requests one at a time in a synchronous manner. The servers can alternatively handle requests in a subprocess, using `os.fork()`, or as a separate thread by instantiating one of the following server classes instead of the four classes listed earlier:

- `ForkingUDPServer(``address` `,` `handler` `)`

- `ForkingTCPServer(``address` `,` `handler` `)`

- `ThreadingUDPServer(``address` `,` `handler` `)`

- `ThreadingTCPServer(``address` `,` `handler` `)`

Finally, two additional classes can be used as base classes for handlers: `StreamRequestHandler` and `DatagramRequestHandler`. When used, these classes override the `setup()` and `finish()` methods of the handle to provide two file attributes, `self.rfile` and `self.wfile`, that can be used to read and write data to and from the client, respectively. For example:

    # Read an HTTP request from a client and bounce it back
    class EchoHandler(SocketServer.StreamRequestHandler):
       def handle(self):
               self.wfile.write("HTTP/1.0 200 OK\r\n")
               self.wfile.write("Content-type: text/plain\r\n\r\n")
               self.wfile.write("Received connection from %s\r\n\r\n" %
                                  (self.client_address,))
               while 1:
                   line = self.rfile.readline()
                   self.wfile.write(line)
                   if not string.strip(line):
                        break

##### Note

- All the server classes can be specialized by subclassing. The online documentation contains more information about this topic.

? See Also socket (252), BaseHTTPServer (232), SimpleHTTPServer (251), CGIHTTPServer (239), thread (219), os (180).

#### `urllib`

The `urllib` module is used to fetch data from the Web.

    urlopen(url [, data])

Given a uniform resource locator (URL) such as `http://www.python.org` or `ftp://foo.com/pub/foo.tar`, this function opens a network connection and returns a file-like object. If the URL doesn’t have a scheme identifier such as `ftp:` or `http:`, or if it’s `file:`, a local file is opened. If a connection cannot be made or an error occurs, an `IOError` exception is raised. If the URL is an HTTP request, the optional `data` argument specifies that the request should be made using a `POST` method, in which case the data is uploaded to the server. In this case, the data must be encoded in an `'application/x-www-form-urlencoded'` format as produced by the `urlencode()` function.

    urlretrieve(url [, filename [, hook]])

Opens a URL and copies its data to a local file, if necessary. If `url` is a local file or a cached copy of the data exists, no copying is performed. `filename` specifies the name of the local file in which data will be saved. If omitted, a temporary filename will be generated. `hook` is a function called after a connection has been made and after each block of data has been read. It’s called with three arguments: the number of blocks transferred so far, the block size in bytes, and the total size of the file in bytes.The function returns a tuple (`filename` `,` `headers` ) in which `filename` is the name of the local file where the data was saved and `headers` is the information returned by the `info()` method as described for `urlopen()`. If the URL corresponds to a local file or if a cached copy was used, `headers` will be `None`. Raises an `IOError` if an error occurs.

    urlcleanup()

Clears the local cache created by `urlretrieve()`.

    quote(string [, safe])

Replaces special characters in `string` with escape sequences suitable for including in a URL. Letters, digits, and the underscore (`_`), comma (`,`) period (`.`), and hyphen () characters are unchanged. All other characters are converted into escape sequences of the form `'%xx'`. `safe` provides additional characters that should not be quoted and is `'/'` by default.

    quote_plus(string [, safe])

Calls `quote()` and additionally replaces all spaces with plus signs.

    unquote(string)

Replaces escape sequences of the form `'%xx'` with their single-character equivalent.

    unquote_plus(string)

Like `unquote()`, but also replaces plus signs with spaces.

    urlencode(dict)

Converts a dictionary to a URL-encoded string suitable for use as the `data` argument of the `urlopen()` function. The resulting string is a series of `'``key=value` `'` pairs separated by `'&'` characters, where both `key` and `value` are quoted using `quote_plus()`.

The file-like object returned by `urlopen()` supports the following methods:

Method | Description
--- | ---
`u` `.read([``nbytes` `])` | Reads `nbytes` of data.
`u` `.readline()` | Reads a single line of text.
`u` `.readlines()` | Reads all input lines and returns a list.
`u` `.fileno()` | Returns the integer file descriptor.
`u` `.close()` | Closes the connection.
`u` `.info()` | Returns the `mimetools.Message` object containing meta-information associated with the URL. For HTTP, the HTTP headers included with the server response are returned. For FTP, the headers include `'content-length'`. For local files, the headers include a date, `'content-length'`, and `'content-type'` field.
`u` `.geturl()` | Returns the real URL of the returned data, taking into account any redirection that may have occurred.

##### Notes

- The only supported protocols are HTTP, FTP, Gopher, and local files. Although the `httplib` module supports HTTP/1.1, this module uses HTTP/1.0 retrieve documents.

- If Python is configured with OpenSSL support, secure HTTP (`https://``name` ) is also supported.

- The `urlopen()` function works transparently with proxies that don’t require authentication. On UNIX and Windows, proxy servers should be set with the `$http_proxy`, `$ftp_proxy`, and `$gopher_proxy` environment variables.

- Caching is currently not implemented.

- If a URL points to a local file but the file cannot be opened, the URL is opened using the FTP protocol.

- The `urllib2` module (available with Python 2.1) provides more advanced support for fetching URLs, dealing with redirection, and handling user authentication. Details about this module are available at http://www.python.org/doc/current/lib/module-urllib2.html.

? See Also httplib (243), ftplib (242), urlparse (262), mimetools (266).

#### `urlparse`

The `urlparse` module is used to manipulate URL strings such as " `http://www.python.org` ". The general form of a URL is as follows:

    "scheme://netloc/path;parameters?query#fragment"

    urlparse(urlstring [, default_scheme [, allow_fragments]])

Parses the URL in `urlstring` and returns a tuple (`scheme` `,` `netloc` `,` `path` `,` `parameters` `,` `query` `,` `fragment` ). `default_scheme` specifies the scheme (`"http"`, `"ftp"`, and so on) to be used if none is present in the URL. If `allow_fragments` is zero, fragment identifiers are not allowed.

    urlunparse(tuple)

Constructs a URL string from a tuple as returned by `urlparse()`.

    urljoin(base, url [, allow_fragments])

Constructs an absolute URL by combining a base URL `base` with a relative URL `url` . `allow_fragments` has the same meaning as for `urlparse()`. If the last component of the base URL is not a directory, it’s stripped.

##### Examples

    >>> urlparse("http://www.python.org/index.html")
    ('http', 'www.python.org', '/index.html', '', '', '')

    >>> urlunparse(('http', 'www.python.org', '/index.html', '', '', ''))
    'http://www.python.org/index.html'

    >>> urljoin("http://www.python.org/index.html","Help.html")
    'http://www.python.org/Help.html'

? See Also urllib (260), Internet RFC 1738, Internet RFC 1808.

#### `webbrowser`

The `webbrowser` module provides functions for opening documents in a Web browser in a platform-independent manner. The module tries to determine the current browser using the environment of the local machine.

    open(url [, new])

Displays `url` with the default browser. If `new` is set, a new browser window is opened.

    open_new(url)

Displays `url` in a new window of the default browser.

    get([name])

Returns a controller object for manipulating a browser. `name` is the name of the browser type and is typically one of `'netscape'`, `'kfm'`, `'grail'`, `'windows-default'`, `'internet-config'`, or `'command-line'`.

    register(name, constructor[, controller])

Registers a new browser type for use with the `get()` function. `name` is the name of the browser. `constructor` is called without arguments to create a controller object for manipulating the browser. `controller` is a controller instance to use instead. If supplied, `constructor` is ignored and may be `None`.

A controller instance `c` returned by the `get()` function has the following methods:

    c.open(url [, new])

Same as the `open()` function.

    c.open_new(url)

Same as the `open_new()` function.

##### Note

- If set, the `$BROWSER` environment variable determines the name of the default browser.

| Last updated on 3/28/2002 Python Essential Reference, Second Edition, © 2002 New Riders Publishing

< BACK | Make Note | Bookmark | CONTINUE >

© 2002, O'Reilly & Associates, Inc.