Zipped Packages on App Engine :-(

as part of an ongoing project to get zope3 running well on google app engine, i worked on loading python code from zipped egg files. I started by following guido’s hint in the corresponding appengine issue. As google appengine doesn’t contain the zipimport builtin extension, i relied on using the python svn sandbox code that implements imports in pure python, and uses the zipfile module on top of that to provide a zip import facility. I checked out the import in py code, and added the following to get it working in google app engine dev server.

import os.path, imp, types
from zipimport_ import zipimport

imp.new_module = types.ModuleType
imp.PY_SOURCE = 1
imp.PY_COMPILED = 2

def load_zipegg( egg_path ):
    egg_file = os.path.basename( egg_path )
    egg_name = egg_file.split('-',1)[0]
    head = egg_name.split('.')[0]
    importer = zipimport.zipimporter( egg_path )
    # we load the head, the importer loads any contained modules
    importer.load_module( head )

unfortunately while it works ok on the dev server, it doesn’t work on appengine :-(, due to a non implementation of marshall.dumps. i’ve appended a comment to that effect on the issue.

in a nutshell you hand it hand it the path to an egg, and it loads the egg code. Its not 100% perfect, it has some issues with minor namespace’d eggs (stomps  on __path__, extraneous sys.modules entry created in subpackages). Hopefully its useful to those wanting to build larger applications or utilize frameworks other than the builtin django.

It does add considerably to the startup time for an app, which went from .3-.5s  to .9-1s. after this initial load that particular app server instance has a cached sys.modules to work with, and startup time is neglible.

The porting of zope3 to appengine, still needs some support for zip contained resources, such as configuration, presentation templates, and browser resources. Even just zipping the packages without those, gets a simple zope3 application down to ~350 files.

Currently i’m just zipping individual directory eggs, with some filtering for pyc, so, text, and test files. Load time would likely go considerably faster if addons were packaged in a single egg.

About these ads

4 Comments

Filed under cloud, python, zope

4 responses to “Zipped Packages on App Engine :-(

  1. I can’t help but feel like this is all a lot of work in order to deal with an unjustified and rather silly restriction in appengine. I’ve only gotten vague justifications of this file count restriction.

    (But just to check — you are using uncompressed zip files, right?)

  2. agreed, the restriction is obnoxious if you want to run something besides django. hopefully they’ll increase it, or have at the least a viable alternative.

    the zips were uncompressed, to save cpu cycles on startup.

    so back to obnoxious workarounds.. at least for the core zope eggs stripping them of tests and documentation, cuts the file count by 2/3.

  3. Pingback: GigBlastr - an experimental Pylons app on Google App Engine | Nate Aune

  4. i’m still convinced that this is possible, its primarily a question of time to explore the solution and workarounds.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s