i released the ore.xapian package to pypi a few weeks back, and after a few iterations i’ve got in production on few small applications, its a thin layer on top of xappy to give an indexing framework for zope3 based applications.

its pretty xapian agnostic.. its designed as an async indexing framework, with abstractions for content indexers, content storage/ resolution, transactional flush into the indexing queue, manages reopening search connections, etc.

the pypi page goes into a bit more detail (doctest style).
http://pypi.python.org/pypi/ore.xapian

i’m using it succesfully to index content from relational databases and subversion with a zope3 front end. only real todo is to make the index queue persistent for remote indexers, but to be useful that
would need corresponding support for remote search connections in xappy. unfortunately i don’t have the bandwidth for the latter atm, but the details are here.

http://groups.google.com/group/xappy-discuss/browse_thread/thread/7ae9fb8d212529b2

I’ve been working for the last year primarily on Zope3 relational applications, and have assembled various packages and practices for making them. I’ve recently begun publishing most of these packages as eggs to the cheeseshop, but I’ve had requests to give more examples of usage. 

The first part of the process constructing a relational app, i engaged in was to remove the zodb. The primary reason for this was to remove zodb deployment considerations ( setting up zeo,etc) so that applications can be deployed easily via mod_wsgi, or have multiple front ends without any special setup of additional components. 

The result of this was the ore.wsgiapp, originally it was just the output of a zopeproject paster template, modified to remove the ZODB startup. However, Jim Fulton refactored the zope.publisher that drives to include paste support and support for custom publications directly within it, greatly simplifying the process of setting up zope without the zodb.

Embracing WSGI

ore.wsgiapp constructs zope3 applictaions that areentirely driven by wsgi, there is no zope.conf file, only a paster config file for configuring an application and its components.

Installation

if your in a buildout just ore.wsgiapp to your eggs section.. or for the virtualenv/setuptools raw usage:

 easy_install ore.wsgiapp

Constructing an application

First we need to define an object which implements IApplication, which will be the root of our published application, useful base classes for this purpose  are in the ore.wsgiapp.app module:

  from ore.wsgiapp import app

  class MyApplication( app.Application ): pass

now if we register this object as a utility it will be the root object published by zope. we’ll do this in zcml in a separate step.

Creating an application view

you will still need to do all the basics to register/provide views of this object. for the purposes of this example, we’ll add a basic view to echo hello world:

class AppView( object ):
    """ a simple view we register for the application """ 
    def __init__( self,context, request):
        self.context = context
        self.request = request
 
    def __call__( self ):
        return "Hello World"

ZCML in Detail

so here’s a sample zcml to regiser the app as a utility, and include a basic zope3 environment.

<configure xmlns="http://namespaces.zope.org/zope"
           xmlns:browser="http://namespaces.zope.org/browser">

  <include package="zope.app.zcmlfiles" file="meta.zcml" />
  <include package="zope.publisher" />
  <include package="zope.traversing" />
  <include package="zope.app.zcmlfiles" />

  <!-- We override the default zope publisher request factory which expects a zodb -->
  <includeOverrides package="ore.wsgiapp"/>

  <!-- Application to publish -->
  <utility
     provides="ore.wsgiapp.interfaces.IApplication"
     factory="exampleapp.MyApplication"
     />

  <!-- Default View for Test Application -->
  <browser:page
      name="index.html"
      for="ore.wsgiapp.interfaces.IApplication"
      class="exampleapp.AppView"
      permission="zope.Public"
      />
</configure>

Paste Configuration

To use with Paste, you include a configuration section like the following:

  [app:zope]
  use = egg:ore.wsgiapp
  zcml = site.zcml

 
you can also turn on devmode here, for pdb post mortem debugging and template reloading. go ahead and save this file as debug.ini for the next step.

Running It

using a paster, its as simple as

./bin/paster debug.ini

and now we can visit our webserver to get a nice hello world  zope3 no zodb.. 
 

Application Initialization

Its often useful to defer application setup till after the application has finished loading its configuration, so that component architecture is fully configured. in order to allow for this, ore.wsgiapp generates a IWSGIApplicationCreatedEvent with the application as an attribute. we can register a subscriber for this in zcml, and it will be invoked after configuration is loaded. 

As an example we’ll use an event subscriber to:
 

  >>> from zope.app.component import site
  >>> from zope.app.container.sample import SampleContainer
  >>>

  >>> def appSetUp( app, eventevent ):
        """Initialize an application"""
        # setup a local site manager
        sm = site.LocalSiteManager( self.context )
        self.context.setSiteManager( sm )
        # add a folder
        app['news'] = SampleContainer()

 
Futures
using the zope.publisher.paste support added by jim fulton, currently ore.wsgiapp jumps through some hoops to use the existing wsgi support within zope.

after a pair coding session with malthe borch, i decided to give emacs a more determined effort. first i culled my .emacs of a bunch of items that i had picked up from people over the years ( i think originally got most of it from jim fulton, and Ken Manheimer. Then i started hunting for emacs modes to customize  my environment, i found a few cool modes to emulate the textmate tricks i picked up. 

yasnippet - http://code.google.com/p/yasnippet/  the power of textmate snippets in emacs!

select word function, from the helpful folks at on the #emacs channel

[11:44] consolers hazmat: (defun hazmat-select-word () (interactive) (kill-new (thing-at-point ‘word)))

[11:45] consolers then you can M-x hazmat-select-word or (global-set-key  <your favourite keybinding> ‘hazmat-select-word) 

i also checked out the python-mode svn, which revealed a pycomplete mode! for basic python completion. rockin. i integrated in pyflakes via flyweight mode for on the fly syntax checking.

for good measure i also added in, ido-mode, which gives nice completion for buffers/files ( built in to emacs) although anything.el also looks intriguing.

Working with large code bases like Zope and Plone can quickly give your hands a workout. I’ve been looking around for some good python completion tools that i can use directly from textmate or emacs.

the critical comparison test for any such tools.

• works well with large codebases.
• opensource

ctags/gtags

not directly related to completion, but exuberant tags / ctags gives the ability to lookup symbol (classes, methods, functions) definitions, and jump around a code base is incredibly useful. conceivably you could build a usable completion tool directly from a ctags file, that did symbol completion rather than context aware completion.

http://ctags.sourceforge.net/

gtags, a variant from google, sets up source with a client/server index, and is more appropriate for a symbol completion scenario.

rope

http://rope.sourceforge.net

a re-factoring library. lots of unit tests. editing centric api is a bit of challenge to use for the lightweight integration i’m looking for.

KomodoEdit

was recently open-sourced. Komodo is a pretty nice dynamic language ide with supprot for ruby/perl/python/xslt and more. Some of the more critical features for an IDE were unfortunately not included in the opensourcing, like a debugger. it does however have a very nice cross language code intelligence system (codeintel package), based on static analysis. which with a little bit of effort could be made into a nice library for use any ide.

http://www.openkomodo.com/

PyDev

the eclipse plugins seems to utilize a dynamic inspection in a separate thread with a socket interface for driving the system via the GUI. the technique at least easily ported to other environments, although i cringe at the notion of random code execution by such a tool.

“I must create a system or be enslaved by another man’s; I will not reason and compare: my business is to create.”

- William Blake

I’m currently using osx for a desktop, being able to write simple python applications which have interactive user interfaces, and which can retain OSX semantics (drag and drop), for scripting common tasks, would be great. I stumbled upon a set of tools today, that allows exactly that.

Cocoa Dialog
http://cocoadialog.sourceforge.net/

cocoadialog provides the ability for cocoa creating user interface dialogs

Playtpus
http://www.sveinbjorn.org/platypus

provides the ability to create osx applications from scripts… “Creating installers, maintenance applications, login items, launchers, automations and droplets is very easy using Platypus.”

Marco Polo
http://www.symonds.id.au/marcopolo/

slightly offbase, but also useful, this takes scripting osx to the next level, by providing a rule based framework that can respond to contextual events (network, devices, etc).

just as excellent, in fine gnu tradition… all of these tools are free software, released under the GPL

The Evils of GenericSetup

Summary

brain dump on generic setup and product bundles, ie. separately configuration management and package management. there are a couple of different issues here, configuration and state management of an instance, and distribution of software.

Use Cases

i think its important to preface discussion of these with a consideration of user stories

a user wants to

- install an addon from a remote repository, preferably installed without restart.

- try out an addon, decide against it, and remove it safely from the system, with zero application footprint. likely will need a restart.

a developer/integrator wants to have

- easy upgrading

an additional noteworthy case of upgrading, are
developers want to upgrade and deploy against scm
revisions.

regarding configuration and state management

generic setup is about managing configuration state for easy snapshotting and restoration of a system configuration. its good about managing and extending the whole system state, but its ability to deal with removal of partial states, and state differences states requires human intervention dealing with tool specific file configuration diffs.

modeling configuration as reversible changesets can accomplish these use cases. an installation/changeset would need to be composed of tasks, and each either associated with date or preferably revision, conflict discriminators (ala zcml), associated task dependencies, and install and uninstall methods. a library of common reversible tasks and a common configuration mechanism (zcml/yaml/etc) for usage and definition, ala tools like cpsinstaller, ploneinstaller, pymake.

an interesting issue is customization, its a hard issue and its an area where plone could use improvement. generic setup makes available raw file diffs, but with the ability to address task sets, we can compose much more user/integrator friendly with task conflicts noted by task, and optional installation of partial changesets (easy customized tool overides). conflicting configurations utilize the task discriminator, and can be presented to the user.

another interesting issue is working against svn with independent development, developers often want to use the latest and greatest features from svn while developing systems, but the confidence to know that going into production. an installation and configuration mechanism for plone should be able to intelligently update checkouts via always doing secondary sorting (post dependency sorting) on a task’s revision/date metadata, so we can track updates by order of application, instead of version number.

one doubt i have on this is that it continues the maintenance of configuration as instance data in the zodb, in addition to tool state the use of z3 components and achieving these benefits, relies on increased reliance on persistent component registries, when the move in z3 has been towards global configuration, i think at some level the growth of persistent data is inevitable for the degree of application instance specific customization of plone typically wants to offer. its unclear, though it would be interesting to see if stephan’s z3c.baseregistry is capable of dealing well with persistent utilities, and adapte registrations.

any solution needs to be at least interim compatible with quickinstaller to facilitate installation on existing versions of plone pre 3.0, which are likely to see widespread use for at least another year, a extensions install stub that fires off the task set installation would suffice.

distribution

distribution is mostly orthogonal to configuration, but the parts should play nice together in the ui to offer a good end user experience, good product distribution basic building blocks should also start to address the needs of micro distribution within the community, such as non profit distributions, intranet/collaboration distributions, in addition to plone’s adherence to being an easy to use cms product.

eggs are a good distribution means and are well supported in the python community.

we need packaging/development with a tool thats accomplished at doing source buildouts and construction of eggs, with examples/documentation on using it for plone. zc.buildout looks like a good tool though lacking in documentation or examples for plone at the moment.

but easy_install isn’t nesc. the appropriate management interface for plone eggs, part of being a being web app product, should also include the optional installation of components through the web. for a plone package manager, its not clear if reutilizing the easy install library is the best way to facilitate it, or if just dealing with raw eggs and using pkg_resources is. the easy_install internals are filled with tons of abstractions, that may or may not be easily used for such a manager construction. regardless, the use of eggs as a standard format for distribution is still a clear win.

for a global repository, there’s been talk of pypi enabling the software center, it not clear whats the best approach, outside of that it should be easy to upload globally such that software is discoverable, and the approach should be able to facilitate running various mini-distribution repositories.

yet another attempt to get a blog going.. hosted is easier.