A few weeks ago at the New Orleans symposium in Plone, I presented on some software I was writing called Content Mirror, which is an addon product for Plone content serialization to a structured/relational database. Its a tool for doing content data deployments. Its nothing particularly new,  Alan and I have been talking about the deployment story for Plone since 2002. I was building CMFDeployment at the time for pushing out static copies of plone sites for ultra-secure systems. Alan and EnfoldSystems went on to build a data deployment solution in Entransit, for doing data deployments. Both were fairly succesful deployment solutions and are still in use and deployed today by a variety of Plone vendors. But both also had some failings, they required alot of configuration and committment to get working for an existing plone site. For example, Entransit required using the instance layout and additional products needed by EnSimpleStaging. While CMFDeployment has a bewildering array of configuration options. In my opinion, both are specialized consulting ware, ie. their primary deployed successfully by folks doing Plone development full time.

For most of the last year, I’ve primarily been doing Zope3 applications, in relational databases. A large part of the reason why I’ve enjoyed Zope3 so much, is that the impedence mismatch between application development is much less than with Plone. Paul Everitt asked at the first Plone conference in New Orleans if Plone was a Product or Framework. Its a question still heard in the community to this day. But to me the answer is clear, Plone is a product, and frankly thats a good thing for both the software and its users. Its however a bad thing when your building applications, their tends to be much more policy with products, that needs to be replaced or worked around when your building on them. As a result, products tend to have two other downsides in application development, developer inefficiencies and computational ineffiencies.

As an example of a developer inefficiency, one is evident just in starting up and serving a page from Plone. I call it the Plone tax, and over the course of a year its about a man month of work. This becomes more stark in comparison to other web app frameworks ( pylons, django, rails, z3, etc) which startup and serve a page in a few seconds. There’s been alot of work recently to this with plone.reload, optional loading of translations, and some heroic work by hanno, but the fact is that there is a lot of code to load up as well as data from the zodb to startup and serve a page in Plone.

Developer inefficiencies are also evident in the learning curve associated with being productive with a product. A product is typically a much bigger software stack, and Plone has and utilizes many components, from zope2, zope3, cmf, archetypes providing foundations, in addition to a growing number of plone specific infrastructure. Plone is the OpenOffice of opensource content management systems. We could drop in a pylons in a cubby hole of a plone tarball. Smaller systems offer a much better productivity to new developers, by giving them the ability to focus on the problem domain and solution, rather than how to frame the problem domain in terms of product concepts and contexts like Plone.

The real key in a data deployment scenario is to keep all the many and great features of Plone as part of the content management process, but also making that data accessible for use in other applications. By deploying content to an rdbms we get language and framework neutrality to interact with it, as well as access to a widespread number of developers and tools. In a nutshell, its data portability.

As a bonus, when using Plone as a product, and reserving customizations to applications onto of the content of a data deployment, the migration story for a Plone instance also becomes much easier.

In terms of computation inefficiency,  Plone does alot of work, which makes it easier to use as a product, but its also computationaly expensive for content delivery compared to simpler solutions that fit the needs of an application/problem domain. ie. the first rule of optimization, do less work. Replacing Plone as a content delivery mechanism, is a great way to make a system more responsive and vertically scalable, while still allowing a dynamic system.

Plone is a great product, and out of the box its offers ease of through the web customization, installation, and a wide range of functionality. My goal for data deployment with Plone was to make something that would enable reusing Plone as a product, as a content management system, but would allow flexibility in usage of that data. Moreover a tool that was easy to drop into new or existing sites.

Data deployments can also bring new features into a Plone. Its much easier to mine business intelligence and reports out of a relational system. For example getting graphs of content creation broken down by month and type or user or using commercial reporting tools.

So back to introducing content mirror. Its basically a system for doing data deployments, it features.. .

- Out of the Box support for Default Plone Content Types.
- Out of the Box support for all builtin Archetypes Fields (including files,
and references ).
- Support for 3rd Party / Custom Archetypes Content Types in one line of configuration.
- Supports Capturing Containment and Workflow in the serialized database.
- Completely Automated Mirroring, zero configuration required beyond installation.
- Easy customization via the Zope Component Architecture
- Elegant and Simple Design, less than 600 lines of code, extensive doctests
- Support for Plone 2.5, 3.0, and 3.1
- Opensource ( GPLv3 )

installation docs
http://code.google.com/p/contentmirror/wiki/Installation

technical introduction / readme
http://code.google.com/p/contentmirror/wiki/Introduction

in a nutshell its technical architecture, is an event observers with aggregation by object on txn boundaries, using an operation pattern for serialization actions, along with schema transformation of archetypes to relational databases tables, using a sqlalchemy runtime generated orm layer. the technical introduction goes into more details.

as part of an ongoing project to get zope3 running well on google app engine, i worked on loading python code from zipped egg files. I started by following guido’s hint in the corresponding appengine issue. As google appengine doesn’t contain the zipimport builtin extension, i relied on using the python svn sandbox code that implements imports in pure python, and uses the zipfile module on top of that to provide a zip import facility. I checked out the import in py code, and added the following to get it working in google app engine dev server.

import os.path, imp, types
from zipimport_ import zipimport

imp.new_module = types.ModuleType
imp.PY_SOURCE = 1
imp.PY_COMPILED = 2

def load_zipegg( egg_path ):
    egg_file = os.path.basename( egg_path )
    egg_name = egg_file.split('-',1)[0]
    head = egg_name.split(’.')[0]
    importer = zipimport.zipimporter( egg_path )
    # we load the head, the importer loads any contained modules
    importer.load_module( head )

unfortunately while it works ok on the dev server, it doesn’t work on appengine :-(, due to a non implementation of marshall.dumps. i’ve appended a comment to that effect on the issue.

in a nutshell you hand it hand it the path to an egg, and it loads the egg code. Its not 100% perfect, it has some issues with minor namespace’d eggs (stomps  on __path__, extraneous sys.modules entry created in subpackages). Hopefully its useful to those wanting to build larger applications or utilize frameworks other than the builtin django.

It does add considerably to the startup time for an app, which went from .3-.5s  to .9-1s. after this initial load that particular app server instance has a cached sys.modules to work with, and startup time is neglible.

The porting of zope3 to appengine, still needs some support for zip contained resources, such as configuration, presentation templates, and browser resources. Even just zipping the packages without those, gets a simple zope3 application down to ~350 files.

Currently i’m just zipping individual directory eggs, with some filtering for pyc, so, text, and test files. Load time would likely go considerably faster if addons were packaged in a single egg.

I published a minimal zope3 app on app engine, along the lines of what of my post from yesterday. you can check it out at http://zope3.appspot.com

its basically a minimal zope3 application, using a custom publication, and bootstraping some components via zcml.

also a simple test runner for verifying packages on the google app engine
http://zope3.appspot.com/tests

the minimal egg working set used by the app.

zope.deprecation-3.4.0-py2.5.egg
zope.publisher-3.5.2-py2.5.egg
appengine_monkey-0.1dev_r28-py2.5.egg
zope.dottedname-3.4.2-py2.5.egg
zope.schema-3.4.0-py2.5.egg
zope.event-3.4.0-py2.5.egg
zope.tal-3.4.1-py2.5.egg
zope.exceptions-3.5.2-py2.5.egg
zope.tales-3.4.0-py2.5.egg
zope.i18n-3.4.0-py2.5.egg
zope.testing-3.5.1-py2.5.egg
transaction-1.0a1-py2.5.egg
zope.i18nmessageid-3.4.3-py2.5-macosx-10.5-i386.egg
zope.thread-3.4-py2.5.egg
zope.component-3.4.0-py2.5.egg
zope.interface-3.4.1-py2.5-macosx-10.5-i386.egg
zope.traversing-3.5.0a3-py2.5.egg
zope.configuration-3.4.0-py2.5.egg
zope.location-3.4.0-py2.5.egg
zope.deferredimport-3.4.0-py2.5.egg
zope.pagetemplate-3.4.0-py2.5.egg

several modifications to the eggs were nesc to remove security.proxy references, and remove some BBB/deprecated code.

This set of eggs, just fits in under the 1000 file limit ( after manually removing extra locales in zope.i18n and pytz, its about 980 ). we could slim the eggs down a bit, ditch the docs and the tests for a minimal egg, and maybe have a ceiling of 200-300 to play with. but clearly for most zope/grok applications, which will be using a quite a bit more eggs, zip based imports are probably the only realistic option.

Loading zope is about a half-second, but once its initialized, subsequent executions run fast, from a timer around the import statements,

Initial Request      Zope Load Time -0.588495016098
Subsequent Request   Zope Load Time -0.000102996826172

in terms of exploring zip based imports, the suggestion guido pointed out in GAE issue 161, was using something like importlib/zipimport to manually load zip archives/eggs, for another day.

a few weeks ago, i did some exploration, of getting zope3 up and running on app engine, with some discussion in a grok thread. there’s been some interest in the topic, so i wrote it up for a wider audience.

there’s a number of issues with getting zope3 up and running.

- no c extensions ( no proxy, speedups, persistent, etc)
- 1000k file limit
- restricted python language

C Extensions

  $ find . -type f  | wc -l
  4980

  $ ls -al | wc-l
  139

i released the ore.xapian package to pypi a few weeks back, and after a few iterations i’ve got in production on few small applications, its a thin layer on top of xappy to give an indexing framework for zope3 based applications.

its pretty xapian agnostic.. its designed as an async indexing framework, with abstractions for content indexers, content storage/ resolution, transactional flush into the indexing queue, manages reopening search connections, etc.

the pypi page goes into a bit more detail (doctest style).
http://pypi.python.org/pypi/ore.xapian

i’m using it succesfully to index content from relational databases and subversion with a zope3 front end. only real todo is to make the index queue persistent for remote indexers, but to be useful that
would need corresponding support for remote search connections in xappy. unfortunately i don’t have the bandwidth for the latter atm, but the details are here.

http://groups.google.com/group/xappy-discuss/browse_thread/thread/7ae9fb8d212529b2

I’ve been working for the last year primarily on Zope3 relational applications, and have assembled various packages and practices for making them. I’ve recently begun publishing most of these packages as eggs to the cheeseshop, but I’ve had requests to give more examples of usage. 

The first part of the process constructing a relational app, i engaged in was to remove the zodb. The primary reason for this was to remove zodb deployment considerations ( setting up zeo,etc) so that applications can be deployed easily via mod_wsgi, or have multiple front ends without any special setup of additional components. 

The result of this was the ore.wsgiapp, originally it was just the output of a zopeproject paster template, modified to remove the ZODB startup. However, Jim Fulton refactored the zope.publisher that drives to include paste support and support for custom publications directly within it, greatly simplifying the process of setting up zope without the zodb.

Embracing WSGI

ore.wsgiapp constructs zope3 applictaions that areentirely driven by wsgi, there is no zope.conf file, only a paster config file for configuring an application and its components.

Installation

if your in a buildout just ore.wsgiapp to your eggs section.. or for the virtualenv/setuptools raw usage:

 easy_install ore.wsgiapp

Constructing an application

First we need to define an object which implements IApplication, which will be the root of our published application, useful base classes for this purpose  are in the ore.wsgiapp.app module:

  from ore.wsgiapp import app

  class MyApplication( app.Application ): pass

now if we register this object as a utility it will be the root object published by zope. we’ll do this in zcml in a separate step.

Creating an application view

you will still need to do all the basics to register/provide views of this object. for the purposes of this example, we’ll add a basic view to echo hello world:

class AppView( object ):
    """ a simple view we register for the application """ 
    def __init__( self,context, request):
        self.context = context
        self.request = request
 
    def __call__( self ):
        return "Hello World"

ZCML in Detail

so here’s a sample zcml to regiser the app as a utility, and include a basic zope3 environment.

<configure xmlns="http://namespaces.zope.org/zope"
           xmlns:browser="http://namespaces.zope.org/browser">

  <include package="zope.app.zcmlfiles" file="meta.zcml" />
  <include package="zope.publisher" />
  <include package="zope.traversing" />
  <include package="zope.app.zcmlfiles" />

  <!-- We override the default zope publisher request factory which expects a zodb -->
  <includeOverrides package="ore.wsgiapp"/>

  <!-- Application to publish -->
  <utility
     provides="ore.wsgiapp.interfaces.IApplication"
     factory="exampleapp.MyApplication"
     />

  <!-- Default View for Test Application -->
  <browser:page
      name="index.html"
      for="ore.wsgiapp.interfaces.IApplication"
      class="exampleapp.AppView"
      permission="zope.Public"
      />
</configure>

Paste Configuration

To use with Paste, you include a configuration section like the following:

  [app:zope]
  use = egg:ore.wsgiapp
  zcml = site.zcml

 
you can also turn on devmode here, for pdb post mortem debugging and template reloading. go ahead and save this file as debug.ini for the next step.

Running It

using a paster, its as simple as

./bin/paster debug.ini

and now we can visit our webserver to get a nice hello world  zope3 no zodb.. 
 

Application Initialization

Its often useful to defer application setup till after the application has finished loading its configuration, so that component architecture is fully configured. in order to allow for this, ore.wsgiapp generates a IWSGIApplicationCreatedEvent with the application as an attribute. we can register a subscriber for this in zcml, and it will be invoked after configuration is loaded. 

As an example we’ll use an event subscriber to:
 

  >>> from zope.app.component import site
  >>> from zope.app.container.sample import SampleContainer
  >>>

  >>> def appSetUp( app, eventevent ):
        """Initialize an application"""
        # setup a local site manager
        sm = site.LocalSiteManager( self.context )
        self.context.setSiteManager( sm )
        # add a folder
        app['news'] = SampleContainer()

 
Futures
using the zope.publisher.paste support added by jim fulton, currently ore.wsgiapp jumps through some hoops to use the existing wsgi support within zope.

after a pair coding session with malthe borch, i decided to give emacs a more determined effort. first i culled my .emacs of a bunch of items that i had picked up from people over the years ( i think originally got most of it from jim fulton, and Ken Manheimer. Then i started hunting for emacs modes to customize  my environment, i found a few cool modes to emulate the textmate tricks i picked up. 

yasnippet - http://code.google.com/p/yasnippet/  the power of textmate snippets in emacs!

select word function, from the helpful folks at on the #emacs channel

[11:44] consolers hazmat: (defun hazmat-select-word () (interactive) (kill-new (thing-at-point ‘word)))

[11:45] consolers then you can M-x hazmat-select-word or (global-set-key  <your favourite keybinding> ‘hazmat-select-word) 

i also checked out the python-mode svn, which revealed a pycomplete mode! for basic python completion. rockin. i integrated in pyflakes via flyweight mode for on the fly syntax checking.

for good measure i also added in, ido-mode, which gives nice completion for buffers/files ( built in to emacs) although anything.el also looks intriguing.

Working with large code bases like Zope and Plone can quickly give your hands a workout. I’ve been looking around for some good python completion tools that i can use directly from textmate or emacs.

the critical comparison test for any such tools.

• works well with large codebases.
• opensource

ctags/gtags

not directly related to completion, but exuberant tags / ctags gives the ability to lookup symbol (classes, methods, functions) definitions, and jump around a code base is incredibly useful. conceivably you could build a usable completion tool directly from a ctags file, that did symbol completion rather than context aware completion.

http://ctags.sourceforge.net/

gtags, a variant from google, sets up source with a client/server index, and is more appropriate for a symbol completion scenario.

rope

http://rope.sourceforge.net

a re-factoring library. lots of unit tests. editing centric api is a bit of challenge to use for the lightweight integration i’m looking for.

KomodoEdit

was recently open-sourced. Komodo is a pretty nice dynamic language ide with supprot for ruby/perl/python/xslt and more. Some of the more critical features for an IDE were unfortunately not included in the opensourcing, like a debugger. it does however have a very nice cross language code intelligence system (codeintel package), based on static analysis. which with a little bit of effort could be made into a nice library for use any ide.

http://www.openkomodo.com/

PyDev

the eclipse plugins seems to utilize a dynamic inspection in a separate thread with a socket interface for driving the system via the GUI. the technique at least easily ported to other environments, although i cringe at the notion of random code execution by such a tool.

“I must create a system or be enslaved by another man’s; I will not reason and compare: my business is to create.”

- William Blake