Static Exceptions

For isolated modules to work, any class whose methods touch module state must be a heap class, so that each instance of a module can have its own class object. With the changes proposed in this PEP, heap classes will provide access to module state without global registration. But, to create instances of heap classes, one will need the module state in order to get the class object corresponding to the appropriate module object. Heap types are "viral" – anything that “touches” them must itself be a heap type.

Curently, most exception types, apart from the ones in builtins, are heap types. This is likely simply because there is a convenient way to create them: PyErr_NewException. Most exceptions do not need to be heap types. (It does give Python code the ability to add custom attributes to the exception class, but monkeypatching exception classes is not very useful). In some cases, heap exception types are harmful. For example the sqlite module currently uses C-global variables for the exceptions, so any time the module is loaded (e.g. from a subinterpreter), these pointers will be overwritten with freshly created classes. It seems like the module was written expecting PyErr_NewException to hand out static types – which is an assumption that someone casually reading the docs could very well form.

Moreover, since raising exception is a common operations, and heap types will be "viral", PyErr_NewException will tend to "infect" the module with "heap-type-ness" – at least if the module decides play well with subinterpreters/isolation. Many modules could go without module state entirely if the exception classes were static.

To solve this problem, a new function for creating static exception types is proposed.

Slot methods

The above changes don't cover slot methods, such as tp_iter or nb_add.

The problem with slot methods is that their C API is fixed, so we can't simply add a new argument to pass in the defining class. Two possible solutions have been proposed to this problem:

• Look up the class through walking the MRO. This is potentially expensive, but will be useful if performance is not a problem (such as when raising a module-level exception).
• Storing a pointer to the defining class of each slot in a separate table, __typeslots__ [1]. This is technically feasible and fast, but quite invasive.

Due to the invasiveness of the latter approach, this PEP proposes adding a MRO walking helper for use in slot method implementations, deferring the more complex alternative as a potential future optimisation.

Static Exceptions

To faciliate creating static exception classes, a new function is proposed: PyErr_PrepareStaticException. It will work similarly to PyErr_NewExceptionWithDoc, but it will take a pre-allocated, zero-filled PyTypeObject, fill it with the provided info, and call PyType_Ready on it.

Adding module references to heap types

The PyHeapTypeObject struct will get a new member, PyObject *ht_module, that can store a pointer to the module object for which the type was defined. It will be NULL by default, and should not be modified after the type object is created.

A new flag, Py_TPFLAGS_HAVE_MODULE, will be set on any type object where the ht_module member is present and non-NULL.

A new factory method will be added for creating modules:

PyObject* PyType_FromModuleAndSpec(PyObject *module,
                                   PyType_Spec *spec,
                                   PyObject *bases)

This acts the same as PyType_FromSpecWithBases, and additionally sets ht_module to the provided module object.

Additionally, an accessor, PyObject * PyType_GetModule(PyTypeObject *) will be provided. It will return the ht_module if a heap type with Py_TPFLAGS_HAVE_MODULE is passed in, otherwise it will set a SystemError and return NULL.

Usually, creating a class with ht_module set will create a reference cycle involving the class and the module. This is not a problem, as tearing down modules is not a performance-sensitive operation. Module-level functions typically also create reference cycles.

Passing the defining class to extension methods

A new style of C-level functions will be added to the current selection of PyCFunction and PyCFunctionWithKeywords:

PyObject *PyCMethod(PyObject *self,
                    PyTypeObject *defining_class,
                    PyObject *args, PyObject *kwargs)

A new method object flag, METH_METHOD, will be added to signal that the underlying C function is PyCMethod.

To hold the extra information, a new structure extending PyCFunctionObject will be added:

typedef struct {
    PyCFunctionObject func;
    PyTypeObject *mm_class; /* Passed as 'defining_class' arg to the C func */
} PyCMethodObject;

Method construction and calling code and will be updated to honor METH_METHOD.

Argument Clinic

To support passing the defining class to methods using Argument Clinic, a new converter will be added to clinic.py: definitng_class.

Each method may only have one argument using this converter, and it must appear after self, or, if self is not used, as the first argument. The argument will be of type PyTypeObject *.

When used, Argument Clinic will select METH_METHOD as the calling convention. The argument will not appear in __text_signature__.

Slot methods

XXX: Exact API TBD

Static exceptions

A new function will be added:

PyTypeObject * PyErr_PrepareStaticException(PyTypeObject *exc,
                                            const char *name,
                                            const char *doc,
                                            PyObject *base)

Given a zero-filled exception type exc, it will fill that object's tp_name, tp_doc and tp_base with the provided information. The doc and base arguments may be NULL, defaulting to a missing docstring and PyExc_Exception base class, respectively. The exception type's tp_flags will be set to values common to built-in exceptions. After filling these slots, the function will call PyType_Ready on the exception class, and return it. On failure, PyErr_PrepareStaticException will set an exception and return NULL.

If called with an initialized exception type (determined by Py_TYPE(exc) being non-NULL), the function will do nothing but return exc.

The function will not be part of the stable ABI, since it needs the module to pre-allocate a PyTypeObject.

Since heap types do not support multiple inheritance, the function will only allow a single type for base.

Helpers

XXX: I'd like to port a bunch of modules to see what helpers would be convenient

Modules Converted in the Initial Implementation

To validate the approach, several modules will be modified during the initial implementation:

The _sqlite, _pickle, _elementtree, _curses_panel and _csv modules will be switched to use static exception types.

XXX: Module state

Easy creation of types with module references

It would be possible to add a PEP 489 execution slot type make creating heap types significantly easier than calling PyType_FromModuleAndSpec. This is left to a future PEP.

Optimization

CPython optimizes calls to methods that have restricted signatures, such as not allowing keyword arguments.

As proposed here, methods defined with the METH_METHOD flag do not support these optimizations.