Projects for the new year

These are some project idea’s I’ve been sitting on all year and have not yet started, but hope to in the new year.

1. Python CMS framework in the vein of Drupal, based on Repoze BFG

So Drupal is quite a popular community building framework, with good plugin system, and a data model which seems to work well. But I don’t think PHP is the way of the future, at least for me. I use Python for many other projects, I understand how to debug, unit test, etc. And BFG brings the really nice framework features from Zope and Plone, which makes it an ideal starting point. I’ve got a project plan for this which I’ll reveal when I get around to cutting some code.

2. Google Gadget for Bugzilla

We use Bugzilla to do service and software task tracking. It would be really swell if I had a quick list of outstanding tasks, and the ability to quickly add new one. GG works on all platforms now, so I think it is the way to go. I’d prefer this over another FF plugin too, just because it fits more naturally into  this level of things.

3. Signing of public pages which you endorse.

Wikipedia isn’t the only place “anonymous” people post their ideas and understandings, but it’ll work for this example. There can be issues with trusting online source, but what if I could sign a version of a page (the content, not the entire layout). I might be an expert in the field for instance. Then someone browsing to the page can see that I endore it. If they don’t know who I am they can also see what else I endorse, and find out more about me. Then, through social networking, their friends who trust their judgement, perhaps just one a particular subject, can judge the quality of a post through the GPG/PGP web of trust.

ZSI -> CXF: Parameters coming in as NULL

Recounting a strange little compatibility issue I had between ZSI 2.0 and CXF 2.0.x. I was using CXF as the server, running from Maven using Jetty, and ZSI as the client. The parameters from the ZSI were arriving at the service implementation as null. With the web services logging turned on I could see that the SOAP packet was arriving OK, and looked good.

A little bit of digging round the web turned up this message.

The issue was that ZSI wasn’t explicitly namespacing the elements, and so CXF was not seeing them. The solution was to add elementFormDefault=”qualified” to my WSDL definition, and rebuild the ZSI stubs.

<wsdl:definitions xmlns:soap=”http://schemas.xmlsoap.org/wsdl/soap/”
    …
    xmlns:xsd=”http://www.w3.org/2001/XMLSchema”
    xsd:elementFormDefault=”qualified”
    xsd:attributeFormDefault=”unqualified”>

Trac taking a hammering

At work we have one VM which hosts all our project management software like Git, SVN, Trac and Bugzilla. However, recently it has been taking a hammering and essentially crashing. The issue was it was running out of RAM, and swapping like crazy. A little investigation into the situation uncovered that there were some Trac 0.10 CGI processes which were using >500MB of memory.

Using the ps ewww -p <pid> command to look at the enviornment variables of the processes we determined that each of them was serving the same kind of request, SVN changesets. And essentially all of them were being hit by spiders.

Essentially the issue was that the spiders were managing to create a large number of overlapping changeset requests, which were chewing up all the RAM on the VM. Increasing the amount of RAM wouldn’t only offset the problem. So, we now just block the spiders from the changeset areas of our tracs (robots.txt):

User-agent: *
Request-rate: 1/5
Disallow: /projects/data-activities/changeset
...

In the weeks since then we’ve had no issues. Besides, who wants their gaffs showing up on google

Interestingly we’ve not seen the same thing happen with Trac 0.11, so perhaps the issue is resolved now.

Zope3 Component Architecture (CA) style Adapters for Java

After programming for Zope3/Plone for the past year I’ve come to really
admire the flexibility and elegance that their implementation of the
adapter pattern gives us.
And, after Martin Aspeli put the call out almost a year ago, and it has not yet been answered, I thought it was time to give it a go.
How hard could it be.

So, what I’ve developed is a very simple, no dependency, maven available library which should greatly improve the flexibility of your code.

The project page is here
and the repository is at Git Hub.

From the desk in 60 seconds

Quick roundup of whats what:
1) Wireless in UK hotels.

It’s bad. I went to Edinburgh, Manchester and London recently and stayed in some ok 3 and 4 star hotels which all advertised wifi/internet access. Here’s the rundown:
 a) Grassmarket hotel in Edinburgh: Free wireless only available in the pub/common rooms
 b) Fountain Court – Harris in Edinburgh: Really fast and reliable free wireless in the room
 c) The Palace Hotel in Manchester: Fast, expensive wired access in the room.
 d) St Mark Hotel, Earls Court in London: Didn’t actually advertise wifi, and didn’t have it

2) Inheriting templates with chameleon.zpt (1.0a1) and repoze.bfg (0.3.8)

This is only noteworthy because it isn’t documented. I’m assuming the reader knows how to do metal/tal in zope.

Create a master.pt in the usual way, using <html metal:define-macro=”master” and metal:define-slot=”main”.

Create a template pt with <html metal:use-macro=”main.macros['master']“ and metal:fill-slot=”main”.

The trick is to manually load the master and pass it to the render_to_response. In the Django PT renderer they scan the templates dir and load all the templates, making this unnecessary.

from repoze.bfg.chameleon_zpt import render_template_to_response, get_template

def my_view(context, request):
    main = get_template(‘templates/master.pt’)
    return render_template_to_response(‘templates/mytemplate.pt’, project = ‘test’, main=main)

3) Text to speech hotkey in Gnome

First, install festival and xclip.
Next, get the script off the gentoo wiki, and make it executable.
Now, fire up gconf-editor, browse to app/metacity/keybinding_commands
Find a spare command and put in the path to the script.
Finally browse to app/metacity/global_keybindings, find the run_command_# key and enter the hotkey you want.

Now test it out. It should work on selected text, ie no need to copy it explicitly. (Gnome gets notified when you change things in gconf so there is no service restarting).

Writting a DiigoDelicious plugin for Conduit

Tonight I started trying to write a bookmark syncing tool for my Delicious and Diigo bookmarks. Being of the Python persuasion I gave Conduit 0.3.11 a spin as the framework. This turned out to be a good choice, except for the incorrect documentation which suggests that custom modules should be placed in ~/.conduit/modules. In fact they should go in ~/.config/conduit/modules.

I got as far as writing a Bookmark data type (from the marketing I would have thought this should have already existed), and a Diigo data source. When I turned to the Delicious side of the equation I discovered that:

1. The API has just changed; and
2. The library I wanted to use (DeliciousAPI) doesn’t seem to support adding/updating entries.

So the project is now on hold until the Delicious API is updated to the latest API. Either way, Conduit seems like a very flexible framework and I hope it has long-term prosperity. Perhaps in due course it could natively support bluetooth SyncML

Some thoughts on a dynamic (lazy) data access layer for web services

The problem I’ve been address in my most recent work block has been to develop an interface to a relational data store which can be used either as a local DB, or via webservices. The concept is quite straight forward, we want CRUD operations on a set of data objects. The schema is fairly straight forward too, with a core hierarchy of elements with a few enumerated lists. As well as a few cuts across the hierarchy.

We want to use this with webservices, as well wanting to be able to serialise to an XML document, so we layed out the schema in XSD, and applied HyperJAXB3 to is using a JAXB annotations document. Other than a few tweaks to the HJ3 code to add in some extra features we needed, everything this far is vanilla.

We then laid out a generic DAO interface, and implemented a JPA and webservices client. The webservices server actually just wraps the JPA client with a security layer.

All is well so far, with the base functionality passing all the unit tests. But, this is a fairly clumsy, class by class CRUD interface. With JPA/Hibernate you can lazily fetch parents and children of an object, and replicating this functionality would be really useful. But the webservices link really kills things.

One approach we tried was to override the getter methods on the beans to allow dynamic retrieval of data if the stored attribute is null. But we needed the webservices client to create these new (extended) beans. This is actually harder than it sounds, as CXF w/ JAXB does not allow you to replace the context factory. (see https://jax-ws.dev.java.net/issues/show_bug.cgi?id=282 )

We also tried writing our own proxies, with some success. But we ran into troubles when proxying single elements (as opposed to lists) as you loose the annotations which CXF and JPA require.

The obvious answer, which I haven’t mentioned yet is to put this logic in the getters proper. This would be nice, but because we are generating the classes from a schema we don’t want to change the generated code. There are many reasons, none less than because the schema is evolving and it hurts to merge changes. We attempted to write our own JAXB plugin to change the methods which were already generated, but XJC/the-java-code-model do not allow you to remove code from a method, only append to it. Similarly, you can’t get the annotations on a method, nor modify existing annotations. This is a huge problem for us, as it would save us having to modify the JAXB plugins which create this code, rather we could simply apply our changes to the produced code.

Perhaps there is a way around this, but it certainly isn’t obvious, or indexed by Google.

The closing remarks are these. With a bit of backfilling JAXB/XJC will be an immensely powerful tool, able to develop a wide range of data models. (I’m working on the tickets now). Dynamic getter access to attributes via web services would be nice, but it is too much hard work really, and if it is required then creating a second set of domain objects is your only real hope, IMHO. When this project wraps up I’ll release the code here so we can all think about how this approach really faired.

I want the last 2 hours back!

I just spent a few hours writing an Acegi filter to get a certificate from a form post, and put it into a modified User Detail principle. All well and good, except after a redirect back to / I got the error below. Normally this would redirect you back into the application using a JSP page. It turned out that because I’m using Acegi 1.0.5 which has a bug when there is no access denied page handler defined. The reason this occurs only when I was logged in was because I neglected to add ROLE_ANONYMOUS authentication authority list. If only the error meant something debugging wouldn’t have been so hard.

HTTP ERROR: 500

INTERNAL_SERVER_ERROR

RequestURI=/

Caused by:

java.lang.NullPointerException
	at org.acegisecurity.ui.ExceptionTranslationFilter.handleException(ExceptionTranslationFilter.java:229)
	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:176)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
	at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:211)
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
	at org.mortbay.jetty.Server.handle(Server.java:295)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:503)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:827)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:511)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:210)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:379)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:361)
	at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

Powered by ScribeFire.

Spring and JNDI (Tomcat or Jetty)

Recently I had need to deploy some Spring webapps which required predeploy configuration. Being the first time I had to find a serious answer I looked to the mythical JNDI for an answer. This document is meant to complement other Spring JNDI documents out there.

Essentially the problem is this. We need to deploy a webapp. The webapp needs configurations (database and webservice endpoint locations). Editing properties files or XML config within the webapp isn’t nice, because on a redeploy the config will be lost. Inside containers like Tomcat I am not aware of a way to easily add extra items to the classpath which won’t get nuked unexpectedly, so solutions like PropertyPlaceholderConfigurer don’t really fly as the properties file will end up within the webapp. And I don’t like the idea of setting environment variables for to locate such things.

In steps JNDI. JNDI is the Java answer to namespaced, centralised configuration. Application containers like Tomcat, Jetty, Glassfish, etc all allow you to export objects via JNDI. This may not be a completely correct description, but it is sufficient for this demonstration. The trick is how to use these. I’ll show Jetty configs (which in Maven live in src/main/webapp/WEB-INF/jetty-env.xml) as well as some references to Tomcat (in $CATALINA_HOME/conf/server.xml or better still, in $CATALINA_HOME/conf/Catalina/[engine]/<webapp>.xml) (more on Tomcat here). This means that the config lives OUTSIDE the webapp, and is immune to inadvertant changes, making hot-patching sites easier as War/webapp is independent of the site config.

First, exposing a DB.
This exposes a Postgres DB on the name icatDB. Note, there is a special JDBC namespace. Also note I am not using the normal Postgres connection class, rather I’m using the connection pooling class.
Jetty:
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN"
"http://jetty.mortbay.org/configure.dtd">
<Configure class="org.mortbay.jetty.webapp.WebAppContext">
<New id="icatDB" class="org.mortbay.jetty.plus.naming.Resource">
<Arg>jdbc/icatDB</Arg>
<Arg>
<New class="org.postgresql.ds.PGPoolingDataSource">
<Set name="serverName">localhost</Set>
<Set name="databaseName">icat2</Set>
<Set name="user">nigel</Set>
<Set name="password"></Set>
</New>
</Arg>
</New>
</Configure>
Tomcat: Example from a different project:
<Context path="/continuum">
<Resource name="jdbc/users"
auth="Container"
type="javax.sql.DataSource"
username="sa"
password=""
driverClassName="org.apache.derby.jdbc.EmbeddedDriver"
url="jdbc:derby:database/users;create=true" />
</Context>
Spring: I am going to pass this into an entity manager:
...
<bean id="entityManagerFactory"
class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="dataSource" ref="dataSource" />
<property name="jpaVendorAdapter">
<bean
class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
<property name="database" value="POSTGRESQL" />
<property name="showSql" value="true" />
<property name="generateDdl" value="true" />
</bean>
</property>
</bean>
<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:comp/env/jdbc/icatDB"/>
</bean>

So I cheated here. The data source already has a JNDI entrypoint so Spring isn’t involved. However in this next example I need to pass in a String which is a webservice endpoint address:Passing a String:
These kinds of elements are passed via the env namespace. From the Jetty JNDI page it tells me we can only pass in these types:

• java.lang.String
• java.lang.Integer
• java.lang.Float
• java.lang.Double
• java.lang.Long
• java.lang.Short
• java.lang.Character
• java.lang.Byte
• java.lang.Boolean

This is fine for configuration work, which is all we are doing.
Jetty:
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN"
"http://jetty.mortbay.org/configure.dtd">
<Configure class="org.mortbay.jetty.webapp.WebAppContext">
<New class="org.mortbay.jetty.plus.naming.EnvEntry">
<Arg>icatWebservice</Arg>
<Arg type="java.lang.String">http://hostname:8081/ws/ICAT</Arg>
</New>
</Configure>

Tomcat:

<Context path="/icat"
docBase="/var/home/tomcat/icat.war">
<Environment name="mcatextWebservice"
type="java.lang.String"
value="http://localhost:8180/mcatext/ws"/>
</Context>

Now Spring.

First import the jee namespace into your Spring config:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
...
xmlns:jee="http://www.springframework.org/schema/jee"
xsi:schemaLocation="...
http://www.springframework.org/schema/jee http://www.springframework.org/schema/jee/spring-jee-2.0.xsd">
...
Now we can use the jee:jndi-lookup element in place of a value element:
...
<bean id="icatConnectionManagerBase" scope="session"
class="au.edu.archer.services.icat.ICATWsClientImpl">
<constructor-arg index="0">
<jee:jndi-lookup jndi-name="java:comp/env/icatWebservice"/>
</constructor-arg>
</bean>

There has also been discussion of writing a PropertyPlaceholderConfigurer like bean which can bring all the JNDI into the properties scope so we could just use ${env.property} notation.

Maven classpath issues at compile time

Here’s a very weird Maven/Java issue. The error message (below) occurs in my build phase where JaxB is called to produce some Java objects from XML. JaxB calls HyperJaxB, and on some systems it crashes.

[ERROR] XJC while compiling schema(s): org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: Class org.apache.commons.logging.impl.Log4JLogger does not implement Log
Caused by: org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: Class org.apache.commons.logging.impl.Log4JLogger does not implement Log
Caused by: org.apache.commons.logging.LogConfigurationException: org.apache.commons.logging.LogConfigurationException: Class org.apache.commons.logging.impl.Log4JLogger does not implement Log
Caused by: org.apache.commons.logging.LogConfigurationException: Class org.apache.commons.logging.impl.Log4JLogger does not implement Log

But not all. My main dev machine works, my deployment machine doesn’t. My other deployment machine also works, but dave’s machine doesn’t. All of us are using the same Maven version (2.0.8), mostly the same Java versions (1.5.0_xx), and some are amd64, other are x86. However, the crash/work divide does not fall on this line. In fact, on the deployment machine both local users can compile OK, but the LDAP users can’t.

On Dave’s machine if we move the local repo to /tmp (-Dmaven.repo.local=/tmp/repository) it starts working. I did capture the class path on a few occasions, and it seems to be in quite a random order. I noticed that commons-logging is far closer to the start of the class path when the compile works, but I don’t have enough samples to confirm this.

Another curiosity I saw was that if your home directory is in a non-standard place, like /var/home/, then maven puts your repository in a directory called ? in the current working directory. Thats right, question mark. Oh dear.

Some machine details below:

Working machine:
Maven version: 2.0.8
Java version: 1.5.0_13
OS name: “linux” version: “2.6.23.9-mactel-ns” arch: “i386″ Family: “unix”

Broken machine:
Maven version: 2.0.8
Java version: 1.5.0_11
OS name: “linux” version: “2.6.18-8.1.3.el5xen” arch: “amd64″

Username: test.user
Maven version: 2.0.8
Java version: 1.5.0_13
OS name: “linux” version: “2.6.23.9-mactel-ns” arch: “i386″ Family: “unix”
Worked

Username: tomcat – LDAP/NFS
Didn’t work

Username: root
Worked

Username: srb
Worked

Powered by ScribeFire.