Linux/Unix > Python Essential Reference, Second Edition > B. Extending and Embedding Python > Defining New Types | See All Titles

< BACK | Make Note | Bookmark | CONTINUE >

155117184014003188065099048180054212144238241179195140058238111161104058145056204253196054

### Defining New Types

New types of objects can also be defined in extension modules. However, this process is considerably more advanced than simply accessing a few C functions. Because of this complexity, you should consider implementing a new type only in the following situations:

- The type is not easily constructed from existing Python types.

- The type requires interaction with the operating system or another special feature not provided by the interpreter or the standard library.

- The type hasn’t already been implemented elsewhere. For example, efficient matrix types have already been implemented, so it would make little sense to reinvent them. It’s always a good idea to check the Python libraries and newsgroups before implementing a new type. The Vaults of Parnassas Python Resources (http://www.vex.net/parnassus) is also an excellent resource for finding previously implemented extension modules.

The process of creating a new Python type involves the following steps:

1. Define a data structure that contains the actual data stored in the type—for example, the `List` type has an array of elements containing the list items.

2. Define the functions that are going to serve as methods of the type—for example, the `append()` method of a `List` object.

3. Define a pair of functions for creating and destroying the type.

4. Define a set of functions that implement the special methods such as `_ _add_ _()` and `_ _getitem_ _()` that are supported by the type, as described in Chapter 3, “Types and Objects.”

5. Fill in a data structure containing pointers to the numeric operations of a type.

6. Fill in a data structure containing pointers to the sequence operations of the type.

7. Fill in a data structure containing pointers to the mapping operators of the type.

8. Define a type object that contains all the properties of the object and its associated methods.

9. Register methods and any additional functions with the interpreter in the module initialization function.

Listing B.3 illustrates the process of creating a new Python type by implementing a `SharedBuffer` object. A shared buffer is a special data structure that contains data shared by multiple Python interpreters running as different processes. Whenever a change is made by one interpreter, it’s automatically reflected in shared buffers of the other interpreters. This implementation of a shared buffer utilizes an operating system feature known as memory-mapped files, in which the contents of a “file” are mapped into the address space of a process and can be accessed as ordinary memory. Memory-mapped files are supported on both UNIX and Windows, although Listing B.3 shows only the UNIX implementation. Also, this example is only intended to illustrate the process of creating a new type—if you really want to use memory-mapped files, it might be easier to use the `mmap` module in the standard library.

##### Listing B.3 Shared Buffer Type

    /*************************************************************
     * sbuffer.c
     *
     * A shared buffer object implemented using mmap().
     *************************************************************/
    #include "Python.h"
    #include <unistd.h>
    #include <fcntl.h>
    #include <sys/mman.h>
    #include <sys/stat.h>

    /*************************************************************
     * sbufferobject information
    *************************************************************/
    typedef struct {
        PyObject_HEAD
        char       *buffer;     /* Memory buffer */
        int         size;       /* Size of the structure */
        int         fd;         /* File descriptor */
        int         prot;       /* Protection bits */
        int         offset;     /* File offset */
    } sbufferobject;

    /* Exception object used by this module */
    static PyObject *AccessError;

    /* Forward declaration of type descriptor.
       staticforward is a macro defined in Python.h that
       provides a portable way of creating forward references
       to static functions */
    staticforward PyTypeObject SharedBufferType;

    /*************************************************************
     * Instance methods
     *     sbuffer.lock()     - Lock the shared buffer
     *     sbuffer.unlock()   - Unlock the shared buffer
     *     sbuffer.get()      - Get data as a NULL-terminated string
     *     sbuffer.store()    - Store data as a NULL-terminated string
     *************************************************************/
    static PyObject *
    sbuffer_lock(sbufferobject *self, PyObject *args) {
      if (!PyArg_ParseTuple(args,"")) return NULL;
      lockf(self->fd,F_LOCK,0);
      return Py_BuildValue("");
    }
    static PyObject *
    sbuffer_unlock(sbufferobject *self, PyObject *args) {
      if (!PyArg_ParseTuple(args,"")) return NULL;
      lockf(self->fd,F_ULOCK,0);
      return Py_BuildValue("");
    }

    static PyObject *
    sbuffer_get(sbufferobject *self, PyObject *args) {
      int i;
      if (!PyArg_ParseTuple(args,"")) return NULL;
      if (self->prot & PROT_READ) {
        for (i = 0; i < self->size; i++) {
          if (!self->buffer[i]) break;
        }
        return PyString_FromStringAndSize(self->buffer,i);
      } else {
        return PyString_FromString("");
      }
    }

    static PyObject *
    sbuffer_store(sbufferobject *self, PyObject *args) {
      char *str;
      int len;
      if (!PyArg_ParseTuple(args,"s",&str)) return NULL;
      if (self->prot & PROT_WRITE) {
        len = strlen(str)+1;
        if (len > self->size) len = self->size;
        memcpy(self->buffer,str,len);
      } else {
        PyErr_SetString(AccessError,"SharedBuffer is read-only");
        return NULL;
      }
      return Py_BuildValue("");
    }

    /* Instance methods table.  Used by sbuffer_getattr() */
    static struct PyMethodDef sbuffer_methods[] = {
      {"lock",   sbuffer_lock,   METH_VARARGS},
      {"unlock", sbuffer_unlock, METH_VARARGS},
      {"get",    sbuffer_get,    METH_VARARGS},
      {"store",  sbuffer_store,  METH_VARARGS},
      { NULL,    NULL }
    };

    /*************************************************************
     * Basic Operations
     *************************************************************/

    /* Create a new shared buffer object */
    static sbufferobject *
    new_sbuffer(int fd, int size, int offset, int prot)
    {
      sbufferobject *self;
      void *buffer;
      buffer = mmap(0,size,prot,MAP_SHARED,fd,offset);
      if (buffer <= 0) {
        PyErr_SetFromErrno(PyExc_OSError);
        return NULL;
      }
      self = PyObject_NEW(sbufferobject, &SharedBufferType);
      if (self == NULL) return NULL;
      self->buffer = (char *) buffer;
      self->size = size;
      self->offset = offset;
      self->prot = prot;
      self->fd = fd;
      return self;
    }

    /* Release a shared buffer */
    static void
    sbuffer_dealloc(sbufferobject *self) {
      munmap(self->buffer, self->size);
      close(self->fd);
      PyMem_DEL(self);
    }

    /* Get an attribute */
    static PyObject *
    sbuffer_getattr(sbufferobject *self, char *name) {
      if (strcmp(name,"prot") == 0) {
        return Py_BuildValue("i", self->prot);    /* self.prot */
      } else if (strcmp(name,"fd") == 0) {
        return Py_BuildValue("i", self->fd);      /* self.fd */
      }
      /* Look for a method instead */
      return Py_FindMethod(sbuffer_methods, (PyObject *)self, name);
    }

    /* repr() function */
    static PyObject *
    sbuffer_repr(sbufferobject *self) {
      char rbuffer[256];
      sprintf(rbuffer,"<SharedBuffer, fd = %d, length = %d, prot = %d at %x>",
            self->fd, self->size, self->prot, self);
      return PyString_FromString(rbuffer);
    }

    /*************************************************************
     * Sequence operations
     *************************************************************/

    /* len() */
    static int
    sbuffer_length(sbufferobject *self) {
      return self->size;
    }

    /* getitem - Get a single character */
    static PyObject *
    sbuffer_getitem(sbufferobject *self, int index) {
      if (index < 0 || index >= self->size) {
        PyErr_SetString(PyExc_IndexError, "index out-of-bounds");
        return NULL;
      }
      if (!(self->prot & PROT_READ)) {
        PyErr_SetString(AccessError,"SharedBuffer is not readable");
        return NULL;
      }
      return Py_BuildValue("c",self->buffer[index]);
    }

    /* setitem - Store a single character */
    static int
    sbuffer_setitem(sbufferobject *self, int index, PyObject *obj)
    {
      char *str;
      int   strsize;
      if (!PyString_Check(obj)) {
        PyErr_SetString(PyExc_TypeError, "Expected a string.");
        return 1;
      }
      if (PyString_Size(obj) != 1) {
        PyErr_SetString(PyExc_ValueError,"Expected a one character string.");
        return 1;
      }
      if (index < 0 || index >= self->size) {
        PyErr_SetString(PyExc_IndexError, "index out-of-bounds");
        return 1;
      }
      if (!(self->prot & PROT_WRITE)) {
        PyErr_SetString(AccessError,"SharedBuffer is read-only");
        return 1;
      }
      self->buffer[index] = *(PyString_AsString(obj));
      return 0;
    }

    /* getslice - Get a slice out of the buffer */
    static PyObject *
    sbuffer_getslice(sbufferobject *self, int start, int end) {
      if (start < 0) start = 0;
      if (end > self->size) end = self->size;
      if (end < start) end = start;
      if (!(self->prot & PROT_READ)) {
        PyErr_SetString(AccessError,"SharedBuffer is not readable");
        return NULL;
      }
      return PyString_FromStringAndSize(self->buffer+start, (end-start));
    }

    /* setslice - Set a slice in the buffer */
    static int
    sbuffer_setslice(sbufferobject *self, int start, int end, PyObject *obj)
    {
      int size;
      if (start < 0) start = 0;
      if (end > self->size) end = self->size;
      if (end < start) end = start;
      if (!PyString_Check(obj)) {
        PyErr_SetString(PyExc_TypeError, "Expected a string.");
        return 1;
      }
      if (!(self->prot & PROT_WRITE)) {
        PyErr_SetString(AccessError,"SharedBuffer is read-only");
        return 1;
      }
      size = PyString_Size(obj);
      if (size < (end-start)) end = start+size;
      memcpy(self->buffer+start,PyString_AsString(obj),(end-start));
      return 0;
    }

    /* Sequence methods table */
    static PySequenceMethods sbuffer_as_sequence = {
      (inquiry)          sbuffer_length,      /* sq_length    : len(x)     */
      (binaryfunc)       0,                   /* sq_concat    : x + y      */
      (intargfunc)       0,                   /* sq_repeat    : x * n      */
      (intargfunc)       sbuffer_getitem,     /* sq_item      : x[i]       */
      (intintargfunc)    sbuffer_getslice,    /* sq_slice     : x[i:j]     */
      (intobjargproc)    sbuffer_setitem,     /* sq_ass_item  : x[i] = v   */
      (intintobjargproc) sbuffer_setslice,    /* sq_ass_slice : x[i:j] = v */
    };

    /* Type object for shared buffer objects */
    static PyTypeObject SharedBufferType = {
      PyObject_HEAD_INIT(&PyType_Type) /* Required initialization */
      0,                               /* ob_size       : Usually 0   */
      "SharedBuffer",                  /* tp_name       : Type name   */
      sizeof(sbufferobject),           /* tp_basicsize  : Object size */
      0,                               /* tp_itemsize   : Usually 0   */

      /* Standard methods */
      (destructor)  sbuffer_dealloc,   /* tp_dealloc,   : refcount = 0 */
      (printfunc)   0,                 /* tp_print      : print x      */
      (getattrfunc) sbuffer_getattr,   /* tp_getattr    : x.attr       */
      (setattrfunc) 0,                 /* tp_setattr    : x.attr = v   */
      (cmpfunc)     0,                 /* tp_compare    : x > y        */
      (reprfunc)    sbuffer_repr,      /* tp_repr       : repr(x)      */

      /* Type categories */
      0,                           /* tp_as_number  : Number methods   */
      &sbuffer_as_sequence,        /* tp_as_sequence: Sequence methods */
      0,                           /* tp_as_mapping : Mapping methods  */
      (hashfunc) 0,                /* tp_hash       : dict[x]          */
      (ternaryfunc) 0,             /* tp_call       : x()              */
      (reprfunc) 0,                /* tp_str        : str(x)           */
    };

    /*************************************************************
     * Module level functions
     *************************************************************/

    /* Create a new shared buffer object as
       SharedBuffer(filename,size,offset,prot) */
    static PyObject *
    sbufferobject_new(PyObject *self, PyObject *args) {
      char *filename;
      int size;
      int fd, flags;
      int prot = PROT_READ | PROT_WRITE;
      int offset = 0;
      struct stat finfo;

      if (!PyArg_ParseTuple(args,"si|ii",&filename,&size,&offset,&prot)) {
        return NULL;
      }
      if (stat(filename,&finfo) < 0) {
        PyErr_SetFromErrno(PyExc_OSError);
        return NULL;
      }
      if (size + offset > finfo.st_size) {
        PyErr_SetString(PyExc_IndexError,
                        "Requested size and offset is too large.");
        return NULL;
      }
      if ((fd = open(filename,O_RDWR, 0666)) < 0) {
        PyErr_SetFromErrno(PyExc_OSError);
        return NULL;
      }
      return (PyObject *) new_sbuffer(fd,size,offset,prot);
    }

    /* Module Methods Table */
    static struct PyMethodDef sbuffertype_methods[] = {
      { "SharedBuffer", sbufferobject_new, METH_VARARGS },
      { NULL, NULL }
    };

    /* Module initialization function */
    void initsbuffer() {
      PyObject *m, *d;
      m = Py_InitModule("sbuffer",sbuffertype_methods);
      d = PyModule_GetDict(m);

      /* Add a few useful constants for the prot parameter */
      PyDict_SetItemString(d,"PROT_READ",PyInt_FromLong(PROT_READ));
      PyDict_SetItemString(d,"PROT_WRITE",PyInt_FromLong(PROT_WRITE));

      /* Define the exception */
      AccessError = PyErr_NewException("sbuffer.AccessError",NULL,NULL);
      PyDict_SetItemString(d,"AccessError",AccessError);
    }

Finally, Listing B.4 uses the new `SharedBuffer` type. In this case, a shared buffer is used to exchange data between a parent and a child process created with `os.fork()`. The example (motivated by a problem posted to the Python mailing list) performs a hostname-to-IP address translation with a timeout.

##### Listing B.4 Python Program Using a `SharedBuffer`

    # Hostname lookup with a timeout.
    # (with apologies to Andy D.)

    import sbuffer, socket, os, sys, signal

    # Create the memory mapped region
    buffer = open("address","w")
    buffer.write(" "*2048)
    buffer.close()

    # Open the file as a shared buffer object
    buffer = sbuffer.SharedBuffer("address",2048)

    # Return hostname or "" if it can't be resolved
    # in less than 1 second.
    def gethostbyname(hostname):
        buffer.store("")          # Clear the address buffer
        pid = os.fork()           # Create a subprocess
        if pid == 0:
              # Child process
              signal.alarm(1)     # Start the clock
              try:
                  name = socket.gethostbyname(hostname)
              except:
                  sys.exit()
              buffer.store(name)  # Save the name in the buffer
              sys.exit()          # Done
        else:
              os.wait()           # Wait for completion
              return buffer.get() # Get the address
    # Try it out
    ip = gethostbyname("www.python.org")

| Last updated on 3/28/2002 Python Essential Reference, Second Edition, © 2002 New Riders Publishing

< BACK | Make Note | Bookmark | CONTINUE >

© 2002, O'Reilly & Associates, Inc.