Zope3 on Google App Engine

a few weeks ago, i did some exploration, of getting zope3 up and running on app engine, with some discussion in a grok thread. there’s been some interest in the topic, so i wrote it up for a wider audience.

there’s a number of issues with getting zope3 up and running.

- no c extensions ( no proxy, speedups, persistent, etc)
– 1000k file limit
– restricted python language

C Extensions

the lack of the C extensions does away with a large portion of zope3 from being able to run without modification. this is immediately visible upon trying to import basic zope components ( such as zope.component) the zope.deferredmodule is used throughout the codebase, to speed startup time, and breaks as its implementation uses the proxy c extension.

i went ahead and replaced the usage there with a pure python implementation. it passes all but one of the unit tests. that test in particular uses an isinstance check of the proxy against the types.ModuleType, and is the reason the c extension is required. i went ahead and tested the implementation with my existing zope3 development instances (all rdb storage, no zodb present), and this particular check was never an issue. the modified zope.deferredmodule egg and source diff against trunk are here.

the removal of other extensions, causes numerous other packages to not be useable. a rule of thumb i found in executing unit tests so far on app engine is that zope.* packages work ok (zope.schema, zope.interface,, zope.configuration, zope.pagetemplate), but zope.app.* tend to have some use of security proxies, location proxies, or persistence, any of which cause an import error.
functionally, it isn’t a typical zope environment in any sense, its a collection as an application using zope egg components. there isn’t any zodb, but thats not really an issue for most of the zope core components. potentially though existing components could be used with some sort of modification to use null/dummy implementations as was done for zope.deferredimport.

larger legacy frameworks like zope2 and plone are two intimately wedded with implementation choices of c extensions, that are incompatible with appengine to work without major rewriting efforts.

1000K file limit

Google app engine maintains a hard limit on the number of files in an application. See Issue 161 for details ( to vote for it click the star ).

Zope quickly can run into this file limit, checking for a file count in a zope3 app’s buildout eggs directory

  $ find . -type f  | wc -l
  4980

across this many eggs

  $ ls -al | wc-l
  139

crucial to getting zope3 as an appserver on gae, will running well be running on zipped eggs, to minimize the files resources we need. in order to do that there are a couple of facilities that need support opening files via pkg_resources to support zipped eggs in the core. the zcml include directive, so that we can load and register components via zcml and do bootstrap on the system. and new view/page directives that allow for page template file to be loaded from resources ( not zope.app.pagetemplate which introduce security proxies during traversal). things like browser resources are best left served as static files from the app engine environment.

Deployment

Ian did a nice write up and introduction of getting pylons working on gae, using virtual env. the same directions work well in setting up a zope3 wsgi app.

Startup Time

this is a bit speculative in terms of application to gae, but typical startup time for a zope3 app to be initialized is around 3.5-4s (on my laptop 2.16ghz core duo) with some trimming of excess zcml. The notion is that app caching will allow us to initialize a python cgi process for multiple requests at startup and rely on the import caching and global registry at a module scope, via defining a main function entry point for request processing separate from initialization. http://code.google.com/appengine/docs/python/appcaching.html

Status

You can do simple publisher applications and use the component architecture (without c extension optimizations), and simple page template files.

About these ads

3 Comments

Filed under cloud, python, zope

3 responses to “Zope3 on Google App Engine

  1. Pingback: Itinerant Source

  2. Pingback: Zipped Packages on App Engine :-( « Itinerant Source

  3. i really love the tip on that damn limit it was giving me a major headache this past week and i didnt even realize it existed

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s