92 lines
2.8 KiB
Plaintext
92 lines
2.8 KiB
Plaintext
From: andrew at starmedia.net (Andrew Csillag)
|
|
Date: Tue, 20 Apr 1999 18:06:31 GMT
|
|
Subject: Problems with mod_pyapache and cPickle
|
|
Message-ID: <371CC227.5BDF5C5A@starmedia.net>
|
|
Content-Length: 2586
|
|
X-UID: 918
|
|
|
|
When using mod_pyapache, an interpreter is initialized and finalized for
|
|
each request. The problem is when using cPickle for serialization,
|
|
cPickle, upon unpickling, creates a cache of objects that it loads out
|
|
of modules so that it doesn't have to repeatedly import modules to get
|
|
at classes it has already de-serialized, which is cool. The problem is
|
|
that when the second time (in a different interpreter than the first)
|
|
that cPickle is loaded, it's class cache still references classes from
|
|
the first interpreter, which when used, causes all sorts of problems
|
|
since their __builtins__ module has been wrecked and all it's external
|
|
module references are now wrecked too, making them virtually unusable.
|
|
|
|
I don't believe that this problem is specific to the cPickle module, but
|
|
is problematic for any C module that contains references to python
|
|
objects (C objects appear to be immune).
|
|
|
|
For the cPickle module, I've written a patch to make it work correctly,
|
|
but it requires a python wrapper module to make it "transparent". What
|
|
I did is this: made a module function that destroys the class cache
|
|
(called class_map in the code), and creates a new, empty one. Then, I
|
|
have a wrapper module that imports cPickle and calls the function to
|
|
clear the cache.
|
|
|
|
--
|
|
Added this function to cPickle.c:
|
|
--
|
|
static PyObject*
|
|
clear_map(PyObject *self, PyObject *args) {
|
|
|
|
PyObject *new_map;
|
|
PyObject *old_map;
|
|
|
|
/* check arguments */
|
|
if (!PyArg_ParseTuple(args, ""))
|
|
return NULL;
|
|
|
|
/* try to create a new dict */
|
|
new_map=PyDict_New();
|
|
if (!new_map)
|
|
return NULL;
|
|
|
|
old_map=class_map;
|
|
class_map=new_map;
|
|
Py_DECREF(old_map);
|
|
Py_INCREF(Py_None);
|
|
return Py_None;
|
|
}
|
|
|
|
--
|
|
|
|
and added this to the cPickle_methods structure in cPickle.c
|
|
|
|
--
|
|
{"clear_map", (PyCFunction)clear_cache, 1,
|
|
"clear_map() -- clear the class map\n"
|
|
},
|
|
--
|
|
|
|
Then the simple python wrapper module
|
|
|
|
--
|
|
import cPickle
|
|
cPickle.clear_map()
|
|
--
|
|
|
|
This alleviates the problem and makes it work. Outside of not
|
|
finalizing interpreters (which alleviates it somewhat, but makes the
|
|
code run in restricted mode since __builtins__ of the unpickled objects
|
|
isn't the same as the "current" __builtins__), I don't know of any other
|
|
way to fix this.
|
|
|
|
This is likely an issue in other apps that use multiple interpreters and
|
|
use C extensions which maintain references to python objects also.
|
|
Might there be a way (although likely not trivial) to fix this for all
|
|
cases?
|
|
|
|
|
|
--
|
|
"There are two major products that come out of Berkeley:
|
|
LSD and UNIX. We don't believe this to be a coincidence."
|
|
- Jeremy S. Anderson
|
|
|
|
|
|
|
|
|