Salmon Run: August 2008

Saturday, August 30, 2008

JMX for Scripts : Monitoring and Scheduling

Last week, I described a simple JMX setup to manage remotely running shell scripts. Using the HTML Adapter supplied in the JMX toolkit, we were able to provide a nice web-based front end that allowed us to start and stop the scripts, as well as observe its logs and whether it is running or not. The setup so far is functional, but not overly convenient. For example, instead of having to point your browser to the MBean page (and remember, there could be multiple machines, each running a bunch of scripts that you have to manage) once every day or hour, you may want the server to notify you (maybe send you an email or an SMS) if there is anything amiss. So over the last week, I added this sort of functionality to my JMX server, which I describe in this post.

Fortunately, JMX provides these sort of features right out of the box, in the form of Monitor and Timer objects, so the work is mainly to choose the right component for the job, then hook these components up together correctly. The diagram below shows the components I use for each of my Script adapter MBeans in my toy example:

Looks impressive, doesn't it? Well, essentially, all it is saying is that we have two Monitors and one Timer attached to each Script Adapter. The two Monitors periodically poll the Script Adapter and send out notifications into the MBean server's event pool. These are picked up by the NotificationFilters connected with each ScriptAdapter, and passed through to the Listener object, which handles the notification. The Timer is slightly different, it just sends events on a schedule, which gets picked up by the Notification Filter, and passed through to the configured Listener.

Please note: The code for all the classes is provided towards the end of the article. I have shown snippets of code where it is more informative than describing it in English, but because of the high degree of reuse, there is a corresponding number of forward references, which may be hard for a reader to reconcile if I provided full code inline.

Status Monitoring

This monitor is a StringMonitor which calls getStatus() on the ScriptAdapter every minute and looks for an exact match to the the String "ERROR". If it finds it, then it sends out an email. Here is the snippet of code from ScriptAgent.java which does this (we provide the full source below).

    // add string monitor checking for getStatus() != "ERROR"
    StringMonitor statusMonitor = new StringMonitor();
    statusMonitor.addObservedObject(script);
    statusMonitor.setObservedAttribute("Status");
    statusMonitor.setGranularityPeriod(60000L); // check every minute
    statusMonitor.setStringToCompare("ERROR");  // look for errors...
    statusMonitor.setNotifyMatch(true);         // ...and trigger notifications on match
    ObjectName statusMonitorObjectName =
      new ObjectName("monitor:type=Status,script=" + getScriptName(script));
    server.registerMBean(statusMonitor, statusMonitorObjectName);
    statusMonitor.start();
    // ...and associated listener object to send an email once that happens
    server.addNotificationListener(statusMonitorObjectName,
      new EmailNotifierListener(script),
      new ScriptNotificationFilter(script),
      statusMonitorObjectName);

The first few lines are just instantiating the StringMonitor, linking it up to the ScriptAdapter and the getStatus(), then setting its properties before we register and start it. Linking this up to the listener and filter is done in the server.addNotificationListener() call.

To test this monitor, start up the server in a terminal. Switch to either the MBean server and notice that both scripts are running fine. Then on another terminal, navigate to the /tmp directory and create an empty .err file. Under Unix, this would be something like this:

1
2
3

sujit@sirocco:~$ cd /tmp
sujit@sirocco:/tmp$ touch count_sheep_Dolly.err
sujit@sirocco:/tmp$

Back on the terminal where your MBean server was started up, you should see the following output from the MBean server. It may not be instantaneous, since the monitor is run on a schedule, but it should be within a few seconds.

...
EmailNotifierListener: sending email...
From: scriptmanager@clones-r-us.com
To: dr_evil@clones-r-us.com
Date: Wed Aug 27 08:31:56 GMT-08:00 2008
Subject: Alarm for [script:name=count_sheep_Dolly:Status]
--
Script [script:name=count_sheep_Dolly] reported ERROR.
--
...

Log size Monitoring

Our next monitor is a Log size monitor which checks the size of the log size every 30 seconds to make sure it is growing. To do this, we need to use a GaugeMonitor in difference mode. The code snippet to do this is shown below:

    // add gauge monitor to determine if process hung, ie if log files don't grow
    GaugeMonitor logsizeMonitor = new GaugeMonitor();
    logsizeMonitor.addObservedObject(script);
    logsizeMonitor.setObservedAttribute("LogFilesize");
    logsizeMonitor.setDifferenceMode(true);           // check diffs(logsize)
    logsizeMonitor.setThresholds(Long.MAX_VALUE, 1L); // notify if less than 1 (0)
    logsizeMonitor.setNotifyLow(true);
    logsizeMonitor.setNotifyHigh(false);
    logsizeMonitor.setGranularityPeriod(30000L);      // every 30 seconds
    ObjectName logsizeMonitorObjectName =
      new ObjectName("monitor:type=Logsize,script=" + getScriptName(script));
    server.registerMBean(logsizeMonitor, logsizeMonitorObjectName);
    logsizeMonitor.start();
    // ...and the associated listener object to send an email once this happens
    server.addNotificationListener(logsizeMonitorObjectName,
      new EmailNotifierListener(script),
      new ScriptNotificationFilter(script),
      logsizeMonitorObjectName);

The configuration is similar to the one above, first we instantiate the GaugeMonitor, then we configure it to observe ScriptAdapter.LogfileSize, then we configure the GaugeMonitor's behavior. Difference mode is selected because we want to make sure that the difference between consecutive readings of ScriptAdapter.LogfileSize is greater than 0. As before, the last line sets up the listener and filter objects.

To test this monitor, start up the server in a terminal and make sure that the scripts are running. Then on another terminal, kill one of the scripts, like so:

sujit@sirocco:~$ cd /tmp
sujit@sirocco:/tmp$ cat count_sheep_Dolly.pid
3714
sujit@sirocco:/tmp$ kill -9 3714
sujit@sirocco:/tmp$

On the terminal window where the MBean server has been started, you should see the trace of the email being sent after a short while.

...
EmailNotifierListener: sending email...
From: scriptmanager@clones-r-us.com
To: dr_evil@clones-r-us.com
Date: Wed Aug 27 08:28:18 GMT-08:00 2008
Subject: Alarm for [script:name=count_sheep_Dolly:LogFilesize]
--
Script [script:name=count_sheep_Dolly] is HUNG.
--
...

Heartbeat Timer

We also have a Heartbeat mechanism for our scripts using a JMX Timer Monitor bean. This is a Timer that sends a notification every minute. This notification is picked up (via the NotificationFilter) by the CprListener, which sends a start() call to the ScriptAdapter. The ScriptAdapter.start() method is designed to check if the script is running, and only pass the command through if the script is not running. Here is the code snippet to set it up:

    String scriptName = getScriptName(script);
    Timer heartbeatTimer = new Timer();
    heartbeatTimer.addNotification("heartbeat", scriptName,
      null, new Date(), 60000L); // every minute
    ObjectName heartbeatTimerObjectName =
      new ObjectName("timer:type=heartbeat,script=" + scriptName);
    server.registerMBean(heartbeatTimer, heartbeatTimerObjectName);
    heartbeatTimer.start();
    server.addNotificationListener(heartbeatTimerObjectName,
      new CprListener(script),
      new ScriptNotificationFilter(script),
      heartbeatTimerObjectName);

In this case, notice that the notification has a type called "heartbeat", which is what the Listener object will use to determine if it should handle the query. To test this, as before, start the MBean server on a terminal, then just wait until the Timer has a chance to kick in. You should see the following on the MBean Server console.

...
Performing CPR on script:name=count_sheep_Dolly
19:11:14: Attempted start, but script: count_sheep_Dolly already started
Performing CPR on script:name=count_sheep_Polly
19:11:14: Attempted start, but script: count_sheep_Polly already started
...

Timers can also be used with non-daemon scripts, to schedule them to run at a particular time of day.

Automatic script startup

One other thing I did was to start my scripts automatically via the JMX agent. Without JMX, if you have a bunch of application daemons running on a machine, you would probably create individual start/stop scripts for them in your /etc/init.d directory. With a JMX server managing your beans, you just start the JMX server using a start script in your /etc/init.d directory, and let it start the scripts that are registered to it. As an interesting side effect, this will also make you more popular with your Unix system administrator(s), since your popularity is computed as an inverse of the number of times your application daemons crash between midnight and 2am.

All I had to do for this was to add in a start() call in the init() method, and kill() calls in the destroy() method of my ScriptAgent class, and add a shutdown hook in the main() method to ensure that destroy() gets called on server shutdown. You will see this in action as you start and stop your MBean server.

...
[INFO] [exec:java]
08:52:31: Starting script: count_sheep_Dolly
08:52:32: Starting script: count_sheep_Polly
...
^C08:58:49: Killing script: count_sheep_Dolly
08:58:49: Killing script: count_sheep_Polly
sujit@sirocco:~$

The code

ScriptAdapterMBean.java

Hasn't changed a whole lot, but we did add in a new method, so it may be just good to provide the newest code.

// Source: src/main/java/com/mycompany/myapp/ScriptAdapterMBean.java
package com.mycompany.myapp;

public interface ScriptAdapterMBean {

  // read-only properties
  public boolean isRunning();
  public String getStatus();
  public String getLogs();
  public long getRunningTime();
  public long getLogFilesize();
  
  // operations
  public void start();
  public void kill();
}

ScriptAdapter.java

The only thing thats changed since last week is the implementation of the getLogFilesize() which is a new method in the interface.

// Source: src/main/java/com/mycompany/myapp/ScriptAdapter.java
package com.mycompany.myapp;

import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Date;

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.lang.StringUtils;
import org.apache.commons.lang.time.DateFormatUtils;
import org.springframework.jmx.JmxException;

public class ScriptAdapter implements ScriptAdapterMBean {

  private static final String LOG_DIR = "/tmp";
  private static final String ERR_DIR = "/tmp";
  private static final String PID_DIR = "/tmp";
  
  private String path;
  private String args;

  private String adapterName;
  
  private String pidfileName;
  private String logfileName;
  private String errfileName;
  
  private Process process;
  
  public ScriptAdapter(String path, String[] args) {
    this.path = path;
    this.args = StringUtils.join(args, " ");
    this.adapterName = StringUtils.join(new String[] {
      FilenameUtils.getBaseName(path),
      (args.length == 0 ? "" : "_"),
      StringUtils.join(args, "_")
    });
    this.pidfileName = FilenameUtils.concat(PID_DIR, adapterName + ".pid");
    this.logfileName = FilenameUtils.concat(LOG_DIR, adapterName + ".log");
    this.errfileName = FilenameUtils.concat(ERR_DIR, adapterName + ".err");
  }

  /**
   * Not part of the MBean so it will not be available as a manageable attribute.
   * @return the computed adapter name.
   */
  public String getAdapterName() {
    return adapterName;
  }
  
  /**
   * Checks for existence of the PID file. Uses naming conventions
   * to locate the correct pid file.
   */
  public boolean isRunning() {
    File pidfile = new File(pidfileName);
    return (pidfile.exists());
  }

  /**
   * If isRunning, then status == RUNNING.
   * If isRunning and .err file exists, then status == ERROR
   * If !isRunning and .err file does not exist, then status == READY 
   */
  public String getStatus() {
    File errorfile = new File(errfileName);
    if (errorfile.exists()) {
      return "ERROR";
    } else {
      if (isRunning()) {
        return "RUNNING";
      } else {
        return "READY";
      }
    }
  }

  public String getLogs() {
    if ("ERROR".equals(getStatus())) {
      File errorfile = new File(errfileName);
      try {
        return FileUtils.readFileToString(errorfile, "UTF-8");
      } catch (IOException e) {
        throw new JmxException("IOException getting error file", e);
      }
    } else {
      try {
        Process tailProcess = Runtime.getRuntime().exec(
          StringUtils.join(new String[] {"/usr/bin/tail", "-10", logfileName}, " "));
        tailProcess.waitFor();
        BufferedReader console = new BufferedReader(
          new InputStreamReader(tailProcess.getInputStream()));
        StringBuilder consoleBuffer = new StringBuilder();
        String line;
        while ((line = console.readLine()) != null) {
          consoleBuffer.append(line).append("\n");
        }
        console.close();
        tailProcess.destroy();
        return consoleBuffer.toString();
      } catch (IOException e) {
        e.printStackTrace();
        throw new JmxException("IOException getting log file", e);
      } catch (InterruptedException e) {
        throw new JmxException("Tail interrupted", e);
      }
    }
  }

  /**
   * Returns the difference between the PID file creation and the current
   * system time.
   */
  public long getRunningTime() {
    if (isRunning()) {
      File pidfile = new File(pidfileName);
      return System.currentTimeMillis() - pidfile.lastModified();
    } else {
      return 0L;
    }
  }

  /**
   * Returns the current size of the log file in bytes.
   * @return the current size of the log file.
   */
  public long getLogFilesize() {
    File logfile = new File(logfileName);
    return logfile.length();
  }

  public void start() {
    try {
      if (! isRunning()) {
        log("Starting script: " + adapterName);
        process = Runtime.getRuntime().exec(
          StringUtils.join(new String[] {path, args}, " "));
        // we don't wait for it to complete, just start it
      } else {
        log("Attempted start, but script: " + adapterName + " already started");
      }
    } catch (IOException e) {
      throw new JmxException("IOException starting process", e);
    }
  }

  public void kill() {
    if (isRunning()) {
      File pidfile = new File(pidfileName);
      log("Killing script: " + adapterName);
      try {
        String pid = FileUtils.readFileToString(pidfile, "UTF-8");
        Runtime.getRuntime().exec(StringUtils.join(new String[] {
          "/usr/bin/kill", "-9", pid}, " "));
        if (process != null) {
          // remove hanging references
          process.destroy();
        }
        pidfile.delete();
      } catch (IOException e) {
        throw new JmxException("IOException killing process", e);
      }
    }
  }
  
  private void log(String message) {
    System.out.println(DateFormatUtils.ISO_TIME_NO_T_FORMAT.format(new Date()) + 
      ": " + message);
  }
}

ScriptAgent.java

This class has had some pretty huge changes since last week. For each script adapter, we call the registerMonitors() and registerTimer() which addes the 2 monitors and 1 timer to each script adapter. Each of these monitors are linked to the appropriate Listener in these two extra methods. The init() method starts up the scripts on MBean server startup, and there is a new destroy() method which will kills the scripts on server shutdown.

// Source: src/main/java/com/mycompany/myapp/ScriptAgent.java
package com.mycompany.myapp;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;

import javax.management.MBeanServer;
import javax.management.MBeanServerFactory;
import javax.management.ObjectName;
import javax.management.monitor.CounterMonitor;
import javax.management.monitor.GaugeMonitor;
import javax.management.monitor.StringMonitor;
import javax.management.timer.Timer;

import com.sun.jdmk.comm.HtmlAdaptorServer;

public class ScriptAgent {

  // the port number of the HTTP Server Adapter
  private final static int DEFAULT_AGENT_PORT = 8081;
  
  private MBeanServer server;
  private List<ScriptAdapter> scriptAdapters = new ArrayList<ScriptAdapter>();
  
  public ScriptAgent() {
    super();
  }
  
  public void addScriptAdapter(ScriptAdapter scriptAdapter) {
    this.scriptAdapters.add(scriptAdapter);
  }

  protected void init() throws Exception {
    server = MBeanServerFactory.createMBeanServer();
    // load all script adapters
    for (ScriptAdapter scriptAdapter : scriptAdapters) {
      ObjectName script = new ObjectName(
          "script:name=" + scriptAdapter.getAdapterName());
      server.registerMBean(scriptAdapter, script);
      // each script will have a monitor to check if its running and
      // if its log file size is growing
      registerMonitors(script);
      // each script will have a timer to attempt restart every 1m
      registerTimer(script);
      // start all script services
      scriptAdapter.start();
    }
    // load HTML adapter
    HtmlAdaptorServer adaptor = new HtmlAdaptorServer();
    adaptor.setPort(DEFAULT_AGENT_PORT);
    server.registerMBean(adaptor, new ObjectName("adapter:protocol=HTTP"));
    // start 'er up!
    adaptor.start();
  }
  
  protected void destroy() throws Exception {
    for (ScriptAdapter scriptAdapter : scriptAdapters) {
      // stop all script services
      scriptAdapter.kill();
    }
  }
  
  private void registerMonitors(ObjectName script) throws Exception {
    // add string monitor checking for getStatus() != "ERROR"
    StringMonitor statusMonitor = new StringMonitor();
    statusMonitor.addObservedObject(script);
    statusMonitor.setObservedAttribute("Status");
    statusMonitor.setGranularityPeriod(60000L); // check every minute
    statusMonitor.setStringToCompare("ERROR");  // look for errors...
    statusMonitor.setNotifyMatch(true);         // ...and trigger notifications on match
    ObjectName statusMonitorObjectName = 
      new ObjectName("monitor:type=Status,script=" + getScriptName(script)); 
    server.registerMBean(statusMonitor, statusMonitorObjectName);
    statusMonitor.start();
    // ...and associated listener object to send an email once that happens
    server.addNotificationListener(statusMonitorObjectName, 
      new EmailNotifierListener(script), 
      new ScriptNotificationFilter(script), 
      statusMonitorObjectName);
    
    // add gauge monitor to determine if process hung, ie if log files don't grow
    GaugeMonitor logsizeMonitor = new GaugeMonitor();
    logsizeMonitor.addObservedObject(script);
    logsizeMonitor.setObservedAttribute("LogFilesize");
    logsizeMonitor.setDifferenceMode(true);           // check diff(logsize)
    logsizeMonitor.setThresholds(Long.MAX_VALUE, 1L); // notify if <1 (0)
    logsizeMonitor.setNotifyLow(true);
    logsizeMonitor.setNotifyHigh(false);
    logsizeMonitor.setGranularityPeriod(30000L);      // every 30 seconds
    ObjectName logsizeMonitorObjectName = 
      new ObjectName("monitor:type=Logsize,script=" + getScriptName(script));
    server.registerMBean(logsizeMonitor, logsizeMonitorObjectName);
    logsizeMonitor.start();
    // ...and the associated listener object to send an email once this happens
    server.addNotificationListener(logsizeMonitorObjectName, 
      new EmailNotifierListener(script), 
      new ScriptNotificationFilter(script), 
      logsizeMonitorObjectName);
  }

  private void registerTimer(ObjectName script) throws Exception {
    String scriptName = getScriptName(script);
    Timer heartbeatTimer = new Timer();
    heartbeatTimer.addNotification("heartbeat", scriptName, 
      null, new Date(), 60000L); // every minute
    ObjectName heartbeatTimerObjectName = 
      new ObjectName("timer:type=heartbeat,script=" + scriptName);
    server.registerMBean(heartbeatTimer, heartbeatTimerObjectName);
    heartbeatTimer.start();
    server.addNotificationListener(heartbeatTimerObjectName, 
      new CprListener(script), 
      new ScriptNotificationFilter(script), 
      heartbeatTimerObjectName);
  }

  private String getScriptName(ObjectName script) {
    String canonical = script.getCanonicalName();
    return canonical.substring("script:name=".length());
  }

  public static void main(String[] args) {
    final ScriptAgent agent = new ScriptAgent();
    Runtime.getRuntime().addShutdownHook(new Thread() {
      public void run() {
        try {
          agent.destroy();
        } catch (Exception e) {
          throw new RuntimeException(e);
        }
      }
    });
    try {
      agent.addScriptAdapter(new ScriptAdapter(
        "/home/sujit/src/rscript/src/main/sh/count_sheep.sh", new String[] {"Dolly"}));
      agent.addScriptAdapter(new ScriptAdapter(
        "/home/sujit/src/rscript/src/main/sh/count_sheep.sh", new String[] {"Polly"}));
      agent.init();
    } catch (Exception e) {
      e.printStackTrace(System.err);
    }
  }
}

ScriptNotificationFilter.java

The ScriptNotificationFilter is an optional item that makes sure that events raised for a script are handled by the listener that it configured for that script.

// Source: src/main/java/com/mycompany/myapp/ScriptNotificationFilter.java
package com.mycompany.myapp;

import javax.management.Notification;
import javax.management.NotificationFilter;
import javax.management.ObjectName;
import javax.management.monitor.MonitorNotification;
import javax.management.timer.TimerNotification;

public class ScriptNotificationFilter implements NotificationFilter {

  private static final long serialVersionUID = 6299049832726848968L;

  private String scriptName;
  
  public ScriptNotificationFilter(ObjectName objectName) {
    super();
    this.scriptName = objectName.getCanonicalName();
  }
  
  /**
   * Is the notification meant for this script?
   */
  public boolean isNotificationEnabled(Notification notification) {
    if (notification instanceof MonitorNotification) {
      MonitorNotification monitorNotification = (MonitorNotification) notification;
      String observedObjectName = 
        monitorNotification.getObservedObject().getCanonicalName();
      return scriptName.equals(observedObjectName);
    } else if (notification instanceof TimerNotification) {
      TimerNotification timerNotification = (TimerNotification) notification;
      return scriptName.substring("script:name=".length()).equals(
        timerNotification.getMessage());
    } else {
      // unknown notification type
      return false;
    }
  }
}

EmailNotificationFilter.java

This listener is paired with the two Monitors. It composes and sends emails to the address configured for the script. The actual emails are shown in the tests above.

// Source: src/main/java/com/mycompany/myapp/EmailNotificationFilter.java
package com.mycompany.myapp;

import java.util.Date;
import java.util.Map;

import javax.management.Notification;
import javax.management.NotificationListener;
import javax.management.ObjectName;
import javax.management.monitor.MonitorNotification;

import org.apache.commons.lang.ArrayUtils;

public class EmailNotifierListener implements NotificationListener {

  // TODO: this should probably be a configuration item
  @SuppressWarnings("unchecked")
  private static final Map<String,String> ALARM_EMAILS = 
    ArrayUtils.toMap(new String[][] {
      new String[] {
        "script:name=count_sheep_Dolly", "dr_evil@clones-r-us.com"
      },
      new String[] {
        "script:name=count_sheep_Polly", "pinky_n_brain@clones-r-us.com"
      }
  });
  private String scriptName;
  
  public EmailNotifierListener(ObjectName objectName) {
    this.scriptName = objectName.getCanonicalName();
  }
  
  public void handleNotification(Notification notification, Object handback) {
    if (notification instanceof MonitorNotification) {
      MonitorNotification monitorNotification = (MonitorNotification) notification;
      String observedAttribute = monitorNotification.getObservedAttribute();
      String observedScriptName = monitorNotification.getObservedObject().getCanonicalName(); 
      String emailTo = ALARM_EMAILS.get(observedScriptName);
      String emailSubject = "Alarm for [" + observedScriptName + ":" + 
        monitorNotification.getObservedAttribute() + "]";
      StringBuilder emailBody = new StringBuilder();
      if (observedAttribute.equals("Status")) {
        emailBody.append("Script [").
          append(observedScriptName).
          append("] reported ERROR.");
      } else if (observedAttribute.equals("LogFilesize")) {
        emailBody.append("Script [").
          append(observedScriptName).
          append("] is HUNG.");
      } else {
        // nothing to do for now, place holder for future notifications
      }
      sendEmail(emailTo, emailSubject, emailBody);
    }
  }

  private void sendEmail(String emailTo, String emailSubject, StringBuilder emailBody) {
    System.out.println("EmailNotifierListener: sending email...");
    System.out.println("From: scriptmanager@clones-r-us.com");
    System.out.println("To: " + emailTo);
    System.out.println("Date: " + new Date());
    System.out.println("Subject: " + emailSubject);
    System.out.println("--");
    System.out.println(emailBody.toString());
    System.out.println("--");
  }
}

CprListener.java

This listener is paired with the Heartbeat Timer and attempts to administer CPR to a downed script in order to restart it automatically.

Saturday, August 23, 2008

Managing remote scripts with JMX

Around eight years ago, three of us found ourselves maintaining a fairly large multi-language distributed system. Originally, each component was maintained by their original authors, so it was relatively easy for them to ensure that things were good with that component, but with the development phase ending, most of them moved off to other jobs. While we knew our way around most of these systems, we did not have the bandwidth to check on each of them regularly. So our manager came up with the idea of a central monitoring application, which we built and used with great success. Monitoring scripts written in Perl were deployed on the servers where the actual components ran, and a Perl client queried them both via cron and on-demand from a console, and reported the results. In retrospect, I think this was my first experience with management applications.

My next job was with a web publishing house, and although I did not actually work on any of the backend systems, my middleware components would interact with some of them, and without exception, all of them had embedded HTTP (mostly Jetty) servers reporting status of the application in real-time. This was my second experience with management applications.

More recently, I have been thinking about a problem closer to home. Our backend systems are composed of shell scripts calling Java and C/C++ components, and the only way to see what is happening with them is to actually log into the machines and look at their logs, for which, as developers, we have no access. Since most scripts notify on errors, there is the possibility that a script never ran, and developers will only find out about this when they see the side effects (or lack of it) in their application.

I had read about JMX in the past and had even tinkered with it, but never found a compelling reason to use it - that is, until now. The solution I came up with is a hybrid of the two approaches described above. Like the first approach, the MBean relies on standard operating system commands to do the monitoring, and like the second, I use a HTTP server to display the monitoring information, although I use the JMX tool suite's built-in HTTP Adapter rather than build a custom one.

Here is my toy script that I want to set up JMX instrumentation on. It's a script to count the number of animal clones. Depending on the argument provided to it, it whips out a cloned sheep (Dolly) or a parrot (Polly) every 30 seconds and logs a count.

#!/bin/bash
# Check if already running, if so exit
PID_FILE=/tmp/count_sheep_$1.pid
LOG_FILE=/tmp/count_sheep_$1.log
ERR_FILE=/tmp/count_sheep_$1.err
if [ -e $PID_FILE ]; then
  currdate=`date --rfc-3339=seconds`
  echo "[$currdate]: Clone counter already running, cannot start" >gt;>gt; $ERR_FILE
  exit -1
fi
# Start processing
trap `rm -f $PID_FILE` INT TERM EXIT
echo $$ >gt; $PID_FILE
rm -f $LOG_FILE
rm -f $ERR_FILE
if [ $# -eq 0 ]; then
  sheep_name='Dolly'
else
  sheep_name=$1
fi
i=0
while (true); do
  currdate=`date --rfc-3339=seconds`
  i=`expr $i + 1`
  echo "[$currdate]: Hello $sheep_name $i" >gt;>gt; $LOG_FILE
  sleep 1
done
# Clean up
rm $PID_FILE
trap - INT TERM EXIT

Notice that I am depending on certain coding and naming conventions which should be fairly easy to enforce in a corporate environment. If not, there is very likely to be other conventions that you can take advantage of to make your MBeans more reusable across different script types. The required conventions are as follows:

Scripts should always write their PID into a .pid file at a specific location on startup and remove it on normal termination.
Scripts should write progress information into a .log file at a specific location.
Scripts should write error information into a .err file at a specific location only if there is an error.

Our MBean to monitor and control scripts is a standard MBean, and its interface is shown below. Notice that we expose the operations "start" and "kill" (it runs in an infinite loop), and a bunch of read-only properties (signified by getXXX methods). In order to start the script, we pass in the path name of the script and its arguments via a constructor in its implementation. This is for security, so people cannot start up "/bin/rm -rf *" via JMX for example.

// Source: src/main/java/com/mycompany/myapp/ScriptAdapterMBean.java
package com.mycompany.myapp;

public interface ScriptAdapterMBean {

  // read-only attributes
  public boolean isRunning();
  public String getStatus();
  public String getLogs();
  public long getRunningTime();
  
  // operations
  public void start();
  public void kill();
}

The corresponding implementation is shown below. As you can see, for each of the exposed operations and attributes, I spawn a Java Process and run operating system commands through it, similar to what a human user would do if he could log on to the script's host machine.

// Source: src/main/java/com/mycompany/myapp/ScriptAdapter.java
package com.mycompany.myapp;

import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;

import org.apache.commons.io.FileUtils;
import org.apache.commons.io.FilenameUtils;
import org.apache.commons.lang.StringUtils;
import org.springframework.jmx.JmxException;

public class ScriptAdapter implements ScriptAdapterMBean {

  private static final String LOG_DIR = "/tmp";
  private static final String ERR_DIR = "/tmp";
  private static final String PID_DIR = "/tmp";
  
  private String path;
  private String args;

  private String adapterName;
  
  private String pidfileName;
  private String logfileName;
  private String errfileName;
  
  private Process process;
  
  /**
   * Ctor to build a ScriptAdapter MBean implementation. Path and args
   * are injected via ctor for security.
   * @param path the full path to the script.
   * @param args the arguments to the script.
   */
  public ScriptAdapter(String path, String[] args) {
    this.path = path;
    this.args = StringUtils.join(args, " ");
    this.adapterName = StringUtils.join(new String[] {
      FilenameUtils.getBaseName(path),
      (args.length == 0 ? "" : "_"),
      StringUtils.join(args, "_")
    });
    this.pidfileName = FilenameUtils.concat(PID_DIR, adapterName + ".pid");
    this.logfileName = FilenameUtils.concat(LOG_DIR, adapterName + ".log");
    this.errfileName = FilenameUtils.concat(ERR_DIR, adapterName + ".err");
  }

  /**
   * Needed by Agent to build the ObjectName reference. Not part of the MBean
   * so it will not be available as a manageable attribute.
   * @return the computed adapter name.
   */
  public String getAdapterName() {
    return adapterName;
  }
  
  /**
   * Checks for existence of the PID file. Uses naming conventions
   * to locate the correct pid file.
   */
  public boolean isRunning() {
    File pidfile = new File(pidfileName);
    return (pidfile.exists());
  }

  /**
   * If isRunning, then status == RUNNING.
   * If isRunning and .err file exists, then status == ERROR
   * If !isRunning and .err file does not exist, then status == READY 
   */
  public String getStatus() {
    File errorfile = new File(errfileName);
    if (errorfile.exists()) {
      return "ERROR";
    } else {
      if (isRunning()) {
        return "RUNNING";
      } else {
        return "READY";
      }
    }
  }

  /**
   * Runs a tail -10 logfile on the log file to get the latest logs at
   * this point in time.
   * @return the last 10 lines (or fewer) of the log file.
   */
  public String getLogs() {
    if ("ERROR".equals(getStatus())) {
      File errorfile = new File(errfileName);
      try {
        return FileUtils.readFileToString(errorfile, "UTF-8");
      } catch (IOException e) {
        throw new JmxException("IOException getting error file", e);
      }
    } else {
      try {
        Process tailProcess = Runtime.getRuntime().exec(
          StringUtils.join(new String[] {
          "/usr/bin/tail", "-10", logfileName}, " "));
        tailProcess.waitFor();
        BufferedReader console = new BufferedReader(
          new InputStreamReader(tailProcess.getInputStream()));
        StringBuilder consoleBuffer = new StringBuilder();
        String line;
        while ((line = console.readLine()) != null) {
          consoleBuffer.append(line).append("\n");
        }
        console.close();
        tailProcess.destroy();
        return consoleBuffer.toString();
      } catch (IOException e) {
        e.printStackTrace();
        throw new JmxException("IOException getting log file", e);
      } catch (InterruptedException e) {
        throw new JmxException("Tail interrupted", e);
      }
    }
  }

  /**
   * Returns the difference between the PID file creation and the current
   * system time.
   */
  public long getRunningTime() {
    if (isRunning()) {
      File pidfile = new File(pidfileName);
      return System.currentTimeMillis() - pidfile.lastModified();
    } else {
      return 0L;
    }
  }

  /**
   * Operation to start the script.
   */
  public void start() {
    try {
      if (! isRunning()) {
        process = Runtime.getRuntime().exec(
          StringUtils.join(new String[] {path, args}, " "));
        // we don't wait for it to complete, just start it
      }
    } catch (IOException e) {
      throw new JmxException("IOException starting process", e);
    }
  }

  /**
   * Operation to stop the script using a kill -9 PID call, using the
   * PID from the .pid file.
   */
  public void kill() {
    if (isRunning()) {
      File pidfile = new File(pidfileName);
      try {
        String pid = FileUtils.readFileToString(pidfile, "UTF-8");
        Runtime.getRuntime().exec(StringUtils.join(new String[] {
          "/usr/bin/kill", "-9", pid}, " "));
        if (process != null) {
          // remove hanging references
          process.destroy();
        }
        pidfile.delete();
      } catch (IOException e) {
        throw new JmxException("IOException killing process", e);
      }
    }
  }
}

Finally, we build an MBean server agent that will load up one or more instances of our MBean. In order to expose the HTTP interface, we also load in the HtmlAdaptorServer and set it to listen on port 8081.

// Source: src/main/java/com/mycompany/myapp/ScriptAgent.java
package com.mycompany.myapp;

import java.util.ArrayList;
import java.util.List;

import javax.management.MBeanServer;
import javax.management.MBeanServerFactory;
import javax.management.ObjectName;

import com.sun.jdmk.comm.HtmlAdaptorServer;

public class ScriptAgent {

  // the port number of the HTTP Server Adapter
  private final static int DEFAULT_AGENT_PORT = 8081;
  
  private MBeanServer server;
  private List<ScriptAdapter> scriptAdapters = 
    new ArrayList<ScriptAdapter>();
  
  public ScriptAgent() {
    super();
  }
  
  public void addScriptAdapter(ScriptAdapter scriptAdapter) {
    this.scriptAdapters.add(scriptAdapter);
  }

  protected void init() throws Exception {
    server = MBeanServerFactory.createMBeanServer();
    for (ScriptAdapter scriptAdapter : scriptAdapters) {
      server.registerMBean(scriptAdapter, new ObjectName(
        "script:name=" + scriptAdapter.getAdapterName()));
    }
    HtmlAdaptorServer adaptor = new HtmlAdaptorServer();
    adaptor.setPort(DEFAULT_AGENT_PORT);
    server.registerMBean(adaptor, new ObjectName("adapter:protocol=HTTP"));
    adaptor.start();
  }
  
  public static void main(String[] args) {
    try {
      ScriptAgent agent = new ScriptAgent();
      agent.addScriptAdapter(new ScriptAdapter(
        "/home/sujit/src/rscript/src/main/sh/count_sheep.sh", 
        new String[] {"Dolly"}));
      agent.addScriptAdapter(new ScriptAdapter(
        "/home/sujit/src/rscript/src/main/sh/count_sheep.sh", 
        new String[] {"Polly"}));
      agent.init();
    } catch (Exception e) {
      e.printStackTrace(System.err);
    }
  }
}

Next we compile our code and start this agent up on the command line using Maven2, like so:

1 2	sujit@sirocco:~$ mvn exec:java \ -Dexec.mainClass=com.mycompany.myapp.ScriptAgent

And point our browser to http://localhost:8081. In my little experiment, everything is on the same machine, but in a real-world setup, I would be pointing my browser to a server agent listening on a remote machine.

And now, on to the mandatory screenshots, to walk you through a typical flow using the MBean HTML adapter.

	The Agent View - we have loaded two instances of the ScriptAdapter MBean, one to manage the script for cloning sheep and one for cloning parrots.
	The MBean view for the sheep cloning management bean. I've rolled it down a bit so you can see both the kill and start operation buttons. Notice the status attribute "READY". Right now, logs are empty, since there are no logs in the file system.
	Clicking on the [Start] button says "start was successful". Lets go back to the MBean view by clicking the Back to MBean View link.
	The MBean view shows that the script is running. Notice the status attribute "RUNNING" and the non-zero value of RunningTime. If you reload, you will see the logs and the running time refreshing.
	Once we have enough of sheep-cloning, we can kill the process by hitting the [KILL] button. This will produce the "Kill Successful" page.
	Back at the MBean view, we notice that the status is READY (to run), and the isRunning value is false. We also notice the logs from the last run in there, just in case one needs to do some post-mortem. If you don't like this, then you can remove the .log file in the script or change the MBean code to check for isRunning() before showing the logs.

For the longest time, JMX appeared to me to be useful for tool and middleware vendors, but somewhat pointless for application developers, so I never really pursued it. The only time I have seen it being used by application developers in web application development is to build in backdoors to the application to turn on and off verbose logging for a short time, for example.

This application seems to be a good use of JMX. Obviously there is much more to JMX than what I have done so far, but I plan to experiment with some more features of JMX and describe it in future posts.

Monday, August 18, 2008

Running Lucli in Batch mode

Lucli is an interactive command line tool that provides functionality similar to Luke, ie the ability to look inside a Lucene index. Sometimes, such as when the indexes are large and sitting on a remote machine, and when you don't need the full power of Luke to query it, it is often more convenient to use Lucli for examining the index, than copying it over to your local machine and use Luke to get at it. Of course, there are other ways to use Luke, such as X-forwarding or using VNC which requires setup on the server side, which may not be feasible in all cases.

Yet another possible good use for Lucli is to have it be called by shell scripts. I mean, if an index is worth spending effort to examine manually, its probably worth scripting this work so it can happen without human intervention. However, because it is an interactive tool, it is not possible with the version (2.4 at the time of this writing) that is available in the Lucene SVN repository. This post describes what I had to do to build this functionality into Lucli, and to add a new custom method that is (perhaps) unique to us.

Adding the --file option

I added a -f (or --file GNU style) option that is recognized by Lucli on the command line, and if so, it creates a new ConsoleReader object that reads from the file name specified after the option. The script File object is retrieved in the parseArgs() method, shown below.

    private void parseArgs(String[] args) {
      String errorMessage = null;
        if (args.length > 0) {
          if (args.length == 2 &&
              ("--file".equals(args[0]) || "-f".equals(args[0]))) {
            File scriptfile = new File(args[1]);
            if (scriptfile.exists() && scriptfile.canExecute()) {
              this.script = scriptfile;
              return;
            } else {
              errorMessage = "File:" + args[1] + " does not exist or is not executable";
            }
          }
          usage(errorMessage);
          System.exit(1);
        }
    }

And the Lucli constructor would use a null check on the script File object to determine if it should open a no-args ConsoleReader, or a special one that reads from the script file. Like this:

        ConsoleReader cr = null;
        if (script != null) {
          cr = new ConsoleReader(new FileInputStream(script), new PrintWriter(System.out));
        } else {
          cr = new ConsoleReader();
        }
        ...

I also updated the usage() method to report that --file filename is an optional but supported parameter.

    private void usage(String errorMessage) {
        message("Usage: lucli.Lucli [--file script_file]");
        if (errorMessage != null) {
          message("(" + errorMessage + ")");
        }
    }

Finally, I modified the call to the lucli.Lucli in the shell script to accept command line parameters.

1
2
3

#!/bin/bash
...
$JAVA_HOME/bin/java -Xmx${LUCLI_MEMORY} -cp $CLASSPATH lucli.Lucli $*

Usage: example script to find #-records

To find the number of records in an index, I would run Lucli interactively from the command line as follows:

sujit@sirocco:~$ ./run.sh 
Lucene CLI. Using directory 'index'. Type 'help' for instructions.
lucli> index /path/to/my/index
Lucene CLI. Using directory '/path/to/my/index'. Type 'help' for instructions.
Index has 6626 documents 
All Fields:[...]
Indexed Fields:[...]
lucli> quit
sujit@sirocco:~$

So to run this in batch mode, we create a file (call it /tmp/script1.lucli) like so:

1 2	index /path/to/my/index quit

And then, to get the number of records, we run the following command line script. Obviously, this command can now be put into another script that does something with the number of records.

sujit@sirocco:~$ ./run.sh --file /tmp/script1.lucli | grep "Index has" |\
  gawk '{print $3}'
6626
sujit@sirocco:~$

New method list([fieldname1;fieldname2;...])

Another question that I often have to answer is if a particular record is in the index or not, or what new records are available in a freshly built index. Because it is so easy to write this sort of ad-hoc stuff, I have a Python and a Java version that I use interchangeably, depending on whether I have PyLucene set up on the target environment or not. However, this seemed to be a good time to standardize on one tool, so I added a "list" method to Lucli. You can see it here:

sujit@sirocco:~$ ./run.sh 
lucli> help
 count: Return the number of hits for a search. Example: count foo
 explain: Explanation that describes how the document scored against query. 
          Example: explain foo
 help: Display help about commands
 index: Choose a different lucene index. Example index my_index
 info: Display info about the current Lucene index. Example: info
 list: Lists value of field list (field1;field2;...) or all fields for all 
          records in the selected index
 optimize: Optimize the current index
 quit: Quit/exit the program
 search: Search the current index. Example: search foo
 terms: Show the first 100 terms in this index. Supply a field name to only 
          show terms in a specific field. Example: terms
 tokens: Does a search and shows the top 10 tokens for each document. Verbose! 
          Example: tokens foo
lucli>

For this, I had to add a new method list() in LuceneMethods and add code to Lucli to trigger this method if it encounters a list call. This consists of an addCommand("list", LIST, ...) call and a case LIST in the switch in the Lucli.handleCommand() method. The case statement is shown below:

                        case LIST:
                          for (int ii = 1; ii < words.length; ii++) {
                            query += words[ii] + ";";
                          }
                          luceneMethods.list(query);
                          break;

And here is the contents of the list() method in the LuceneMethods class. As you can see, its fairly straightforward. If field names are given, then it just returns the field values and if no field names are given, then it returns all fields. The filtering is done in Unix. This is OK to do here, since these are really ad-hoc usages and so are unlikely to impact performance.

  /** Lists out named fields from the index (all records)
   * @throws IOException
   */
  public void list(String query) throws IOException {
    String[] fieldNames = null;
    if ("".equals(query.trim())) {
      getFieldInfo();
      fieldNames = new String[fields.size()];
      for (int i = 0; i < fieldNames.length; i++) {
        fieldNames[i] = (String) fields.get(i);
      }
    } else {
      fieldNames = query.split(";");
    }
    IndexReader indexReader = IndexReader.open(indexName);
    int maxDoc = indexReader.maxDoc();
    for (int i = 0; i < maxDoc; i++) {
      Document doc = indexReader.document(i);
      StringBuffer buf = new StringBuffer();
      for (int j = 0; j < fieldNames.length; j++) {
        if (j > 0) {
          buf.append(";");
        }
        buf.append(doc.get(fieldNames[j]));
      }
      message(buf.toString());
    }
    indexReader.close();
  }

Usage: example script to find records with specific URL pattern

As mentioned before, we do our pattern matching stuff using Unix tools. This keeps the Lucli method simple and more generic. The example use case is for finding the records (identified by title) that satisfy a particular URL pattern. As before, we can experiment in the interactive shell, and build our script file (script2.lucli) like so:

1
2
3

index /path/to/my/index
list title;url
quit

And call it like so:

1 2	sujit@sirocco:~$ ./run.sh --file /tmp/script2.lucli \| \ gawk -F';' --source '{if ($2 ~ /my_url_pattern/) printf("%s %s\n", $1, $2)}'

The full patch file

You can just patch the stock Lucli with the output of "svn diff" from the src/java/lucli subdirectory. I had to do some hacks (listed below) to get the Lucli to compile, which won't be captured in an "svn diff" output, so I am not publishing the diff of the entire module.

Index: LuceneMethods.java
===================================================================
--- LuceneMethods.java (revision 683771)
+++ LuceneMethods.java (working copy)
@@ -352,6 +352,36 @@
     indexReader.close();
   }
 
+  /** Lists out named fields from the index (all records)
+   * @throws IOException
+   */
+  public void list(String query) throws IOException {
+    String[] fieldNames = null;
+    if ("".equals(query.trim())) {
+      getFieldInfo();
+      fieldNames = new String[fields.size()];
+      for (int i = 0; i < fieldNames.length; i++) {
+        fieldNames[i] = (String) fields.get(i);
+      }
+    } else {
+      fieldNames = query.split(";");
+    }
+    IndexReader indexReader = IndexReader.open(indexName);
+    int maxDoc = indexReader.maxDoc();
+    for (int i = 0; i < maxDoc; i++) {
+      Document doc = indexReader.document(i);
+      StringBuffer buf = new StringBuffer();
+      for (int j = 0; j < fieldNames.length; j++) {
+        if (j > 0) {
+          buf.append(";");
+        }
+        buf.append(doc.get(fieldNames[j]));
+      }
+      message(buf.toString());
+    }
+    indexReader.close();
+  }
+  
   /** Sort Hashtable values
    * @param h the hashtable we're sorting
    * from http://developer.java.sun.com/developer/qow/archive/170/index.jsp
Index: Lucli.java
===================================================================
--- Lucli.java (revision 683771)
+++ Lucli.java (working copy)
@@ -55,7 +55,9 @@
  */
 
 import java.io.File;
+import java.io.FileInputStream;
 import java.io.IOException;
+import java.io.PrintWriter;
 import java.io.UnsupportedEncodingException;
 import java.util.Iterator;
 import java.util.Set;
@@ -95,11 +97,13 @@
  final static int INDEX = 7;
  final static int TOKENS = 8;
  final static int EXPLAIN = 9;
+ final static int LIST = 10;
 
  String historyFile;
  TreeMap commandMap = new TreeMap();
  LuceneMethods luceneMethods; //current cli class we're using
  boolean enableReadline; //false: use plain java. True: shared library readline
+ File script = null;
 
  /**
   Main entry point. The first argument can be a filename with an
@@ -124,11 +128,17 @@
   addCommand("index", INDEX, "Choose a different lucene index. Example index my_index", 1);
   addCommand("tokens", TOKENS, "Does a search and shows the top 10 tokens for each document. Verbose! Example: tokens foo", 1);
   addCommand("explain", EXPLAIN, "Explanation that describes how the document scored against query. Example: explain foo", 1);
-
+  addCommand("list", LIST, "Lists value of field list (field1;field2;...) or all fields for all records in the selected index");
+  
   //parse command line arguments
   parseArgs(args);
 
-  ConsoleReader cr = new ConsoleReader();
+  ConsoleReader cr = null;
+  if (script != null) {
+    cr = new ConsoleReader(new FileInputStream(script), new PrintWriter(System.out));
+  } else {
+    cr = new ConsoleReader();
+  }
   //Readline.readHistoryFile(fullPath);
   cr.setHistory(new History(new File(historyFile)));
   
@@ -234,6 +244,12 @@
     }
     luceneMethods.search(query, true, false, cr);
     break;
+   case LIST:
+     for (int ii = 1; ii < words.length; ii++) {
+       query += words[ii] + ";";
+     }
+     luceneMethods.list(query);
+     break;
    case HELP:
     help();
     break;
@@ -315,18 +331,31 @@
  }
 
  /*
-  * Parse command line arguments (currently none)
+  * Only parse command line argument --file (or -f).
   */
  private void parseArgs(String[] args) {
+   String errorMessage = null;
   if (args.length > 0) {
-   usage();
+    if (args.length == 2 && 
+        ("--file".equals(args[0]) || "-f".equals(args[0]))) {
+      File scriptfile = new File(args[1]);
+      if (scriptfile.exists() && scriptfile.canExecute()) {
+        this.script = scriptfile;
+        return;
+      } else {
+        errorMessage = "File:" + args[1] + " does not exist or is not executable";
+      }
+    }
+   usage(errorMessage);
    System.exit(1);
   }
  }
 
- private void usage() {
-  message("Usage: lucli.Lucli");
-  message("(currently, no parameters are supported)");
+ private void usage(String errorMessage) {
+  message("Usage: lucli.Lucli [--file script_file]");
+  if (errorMessage != null) {
+    message("(" + errorMessage + ")");
+  }
  }
 
  private class Command {

To get Lucli to compile locally, I had to make the following changes to build.xml.

Change the reference to ../contrib-build.xml to contrib-build.xml and copy the contrib-build.xml from svn to my project root directory.
Change the reference to ../common-build.xml in contrib-build.xml and copy the common-build.xml from svn to my project root directory.
Change the location of lucene.jar to a ${project.root}/lib and copy over an existing lucene jar there, to suppress the target "build-lucene" from firing and giving errors.

Conclusion

As you can see, extending Lucli is quite easy. There are just two classes and the code is quite easy to read. Because it does not have too much functionality, when faced with the task of modifying it to suit your own needs, it is very easy (perhaps easier) to just go ahead write your own little subset. The reason I extended Lucli rather than do that are:

this gives me all the other cool stuff that Lucli already has without my writing code for it,
my changes may potentially benefit a larger number of people, and
what goes around comes around, and one day I will benefit from somebody else's extension to Lucli. To be fair, I have already benefited a lot from the code and information contributions by others over the years.

Friday, August 15, 2008

Instant WebApp Security - Just Add Acegi

A question came up at work recently about the best time (in terms of cost and quality) to build security into a web application, that got me thinking about this whole thing. Since we use the Spring Framework for our web applications, Acegi is the natural choice, being the de-facto security solution for it.

Acegi is very flexible - it has lots of customization hooks, and can connect to almost any popular authentication backend. However, to paraphrase Spiderman's uncle Ben - "with great flexibility comes great complexity", and Acegi is no exception. The configuration is the complex part - it's done by wiring Acegi beans in the Spring application context, so you need to know what beans are available within Acegi and how to wire them up.

Several excellent guides exist on the Internet (do a search for "acegi tutorials") - they are all worth a read. In addition, you probably also want to read the Acegi Reference Guide for more extensive coverage. It also helps to have the source JAR handy - I found lot of answers to my questions in the code.

I primarily used Bart van Riel's Spring Acegi Tutorial as a guide when securing my app. The approach I propose here is prescriptive - given an insecure (Spring) web application, this post lists out the tasks that need to happen in order to make it secure with a rather basic but working Acegi configuration.

Unlike the other tutorials, this post makes no attempt to explain the purpose of the various beans in the configuration - I simply point out the places where you need to customize it for your own application. Not sure about you, but this format works better for me personally... I like to set up something quickly and fiddle with the code until it does my bidding. Obviously, to extend this configuration, you will need to read up to locate and understand the Acegi components involved.

The Insecure Application

My example application is a Maven2 Web Application using Spring, and contains list and edit screens for two beans - a Content and a User bean. There is a database at the back and thats pretty much it. Because there is going to be very little code change between this and the secure version (most of the changes are adding filters and interceptors), I am going to provide the code listings later (marking the changes made for security appropriately) for the secure version of the application. The application exposes the following URLs:

http://localhost:8081/myapp/index.jsp
http://localhost:8081/myapp/users/list.do
http://localhost:8081/myapp/users/edit.do?username=${username}
http://localhost:8081/myapp/contents/list.do
http://localhost:8081/myapp/contents/edit.do?id=${contentId}

The directory structure for src/main/webapps is shown below. This is going to change for the secure version, which is why I mention it here.

webapp
  |
  +-- myapp
  |    |
  |    +-- index.jsp
  |    |
  |    +-- contents
  |    |     |
  |    |     +-- edit.jsp
  |    |     |
  |    |     +-- list.jsp
  |    |
  |    +-- users
  |          |
  |          +-- edit.jsp
  |          |
  |          +-- list.jsp
  |
  +-- WEB-INF
       |
       +-- myapp-servlet.xml
       |
       +-- web.xml

Meet The Users

We will have four users in our secure system. Their user names and roles, along with what the roles imply, are listed below:

Username	Roles	Description
admin	ROLE_ADMIN	Administrator can view and edit users, but cannot edit content. On login, should be directed to the users list page with edit capability.
mary	ROLE_MANAGER, ROLE_EDITOR	Can view and edit all content. On login, should be directed to the contents list page with edit capability.
bob	ROLE_EDITOR	Can view all content, but can only edit his own. On login, should be directed to the list page showing only contents authored by him, with edit capability.
larry	ROLE_EDITOR	Same as bob.

Apart from these, there is the unauthenticated user, who can view the public portion of the application (only the list contents page without edit capability).

Securing the application

The following are the things I needed to do to add Acegi security to the insecure web application described above. I describe each of these in some more detail and provide code snippets below.

Add a ContextLoaderListener to web.xml
Add Acegi FilterToBeanProxy definition to web.xml
Add a new applicationContext-security.xml to src/main/resources
Import this file into the main applicationContext file
Move JSPs that correspond to secure resources
Change index.jsp (welcome-file) to check for username and redirect
Add new login.jsp and login_header_include.jsp
Add handler mappings and controller for these new JSPs
Change contents/list.jsp to selectively remove Edit link
Add a MethodInterceptor to intercept calls to ContentDao

Add ContextLoaderListener to web.xml

Without this, the Acegi filters would not work and you will see exceptions on startup. I haven't worked with filters much, but I think this is a requirement for filters in general to work with Spring.

Add Acegi FilterToBeanProxy to web.xml

This adds filters to specific URLs that would need to be authenticated. My complete web.xml is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Source: src/main/webapp/WEB-INF/web.xml -->
<web-app xmlns="http://java.sun.com/xml/ns/j2ee" 
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
         xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee 
         http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd" version="2.4">

  <!-- Acegi filters won't work without the ContextLoaderListener -->
  <context-param>
    <param-name>contextConfigLocation</param-name>
    <param-value>/WEB-INF/myapp-servlet.xml</param-value>
  </context-param>

  <listener>
    <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
  </listener>

  <filter>
    <filter-name>acegi_filterchain_proxy</filter-name>
    <filter-class>org.acegisecurity.util.FilterToBeanProxy</filter-class>
    <init-param>
      <param-name>targetClass</param-name>
      <param-value>org.acegisecurity.util.FilterChainProxy</param-value>
    </init-param>
  </filter>

  <!-- 
    The mapping below means that /all/ urls, including JS, HTML, CSS, etc
    will pass through the filter. Consider matching specific patterns for
    performance.
  -->
  <filter-mapping>
    <filter-name>acegi_filterchain_proxy</filter-name>
    <url-pattern>/*</url-pattern>
  </filter-mapping>

  <!-- The original Spring servlet config (listens only on .do URLs) -->
  <servlet>
    <servlet-name>myapp</servlet-name>
    <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
    <load-on-startup>1</load-on-startup>
  </servlet>

  <servlet-mapping>
    <servlet-name>myapp</servlet-name>
    <url-pattern>*.do</url-pattern>
  </servlet-mapping> 

  <!-- 
    The original welcome-file was a list of links to the list pages.
    They now redirect based on the value of the username in the 
    SecurityContext
  -->
  <welcome-file-list>
    <welcome-file>/myapp/index.jsp</welcome-file>
  </welcome-file-list>
  
</web-app>

The applicationContext-security.xml file

As promised, there are no explanations for the various beans. Refer to the many online tutorials and blog posts, the Acegi reference guide or the source code for more information about these. The lines marked with <!-- CUSTOM: nn > comments are customizations which are explained below.

<?xml version="1.0" encoding="UTF-8"?>
<!-- Source: src/main/resources/applicationContext-security.xml -->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans 
       http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
       http://www.springframework.org/schema/util 
       http://www.springframework.org/schema/util/spring-util-2.0.xsd">

  <bean id="messageSource" 
    class="org.springframework.context.support.ReloadableResourceBundleMessageSource">
    <property name="basename" value="org/acegisecurity/messages"/>
  </bean>
  
  <!-- Acegi Filter Chain Proxy -->
  <bean id="filterChainProxy" class="org.acegisecurity.util.FilterChainProxy">
    <property name="filterInvocationDefinitionSource"><!-- CUSTOM:1 -->
      <value>
        CONVERT_URL_TO_LOWERCASE_BEFORE_COMPARISON
        PATTERN_TYPE_APACHE_ANT
        /j_acegi_logout=logoutFilter
        /**=httpSessionContextIntegrationFilter,
            authenticationProcessingFilter,
            exceptionTranslationFilter,filterSecurityInterceptor
      </value>
    </property>
  </bean>

  <!-- 
    List of filters corresponding to form of authentication in use per the
    filterInvocationDefinitionSource bean above (Listed under pattern=...
    The chain may be different based on the type of authentication being
    used. Beans listed below correspond to the filters listed in the pattern.
  -->
  <bean id="httpSessionContextIntegrationFilter" class="org.acegisecurity.context.HttpSessionContextIntegrationFilter">
    <property name="allowSessionCreation" value="true"/>
  </bean>

  <bean id="authenticationProcessingFilter" class="org.acegisecurity.ui.webapp.AuthenticationProcessingFilter">
    <property name="filterProcessesUrl" value="/j_acegi_security_check"/>
    <property name="authenticationFailureUrl" value="/login.do?id=1"/><!-- CUSTOM:2 -->
    <property name="defaultTargetUrl" value="/myapp/index.jsp"/><!-- CUSTOM:3 -->
    <property name="authenticationManager" ref="authenticationManager"/>
  </bean>

  <bean id="exceptionTranslationFilter" class="org.acegisecurity.ui.ExceptionTranslationFilter">
    <property name="authenticationEntryPoint" ref="formLoginAuthenticationEntryPoint"/>
    <property name="accessDeniedHandler" ref="accessDeniedHandler"/>
    <property name="createSessionAllowed" value="true"/>
  </bean>
  
  <bean id="accessDeniedHandler" class="org.acegisecurity.ui.AccessDeniedHandlerImpl">
    <property name="errorPage" value="/login.do?id=1"/><!-- CUSTOM:4 -->
  </bean>
  
  <bean id="filterSecurityInterceptor" class="org.acegisecurity.intercept.web.FilterSecurityInterceptor">
    <property name="authenticationManager" ref="authenticationManager"/>
    <property name="accessDecisionManager" ref="accessDecisionManager"/>
    <property name="objectDefinitionSource"><!-- CUSTOM:5 -->
      <value>
        CONVERT_URL_TO_LOWERCASE_BEFORE_COMPARISON
        PATTERN_TYPE_APACHE_ANT
        /secure/users/*=ROLE_ADMIN
        /secure/contents/*=ROLE_EDITOR
      </value>
    </property>
  </bean>

  <!--
    Authentication Manager uses a chain of providers to extract the Principal, if
    one exists. As before the chain would differ based on what kind of authentication
    is being used. 
  -->
  <bean id="authenticationManager" class="org.acegisecurity.providers.ProviderManager">
    <property name="providers">
      <list>
        <ref local="daoAuthenticationProvider"/>
      </list>
    </property>
  </bean>
  
  <bean id="daoAuthenticationProvider" class="org.acegisecurity.providers.dao.DaoAuthenticationProvider">
    <property name="userDetailsService" ref="userDetailsService"/>
  </bean>
  
  <bean id="userDetailsService" class="org.acegisecurity.userdetails.jdbc.JdbcDaoImpl">
    <property name="dataSource" ref="dataSource"/>
  </bean>
  
  <bean id="accessDecisionManager" class="org.acegisecurity.vote.UnanimousBased">
    <property name="decisionVoters">
      <list>
        <ref bean="roleVoter"/>
      </list>
    </property>
  </bean>
  
  <bean id="roleVoter" class="org.acegisecurity.vote.RoleVoter">
    <property name="rolePrefix" value="ROLE_"/>
  </bean>

  <bean id="formLoginAuthenticationEntryPoint" class="org.acegisecurity.ui.webapp.AuthenticationProcessingFilterEntryPoint">
    <property name="loginFormUrl" value="/login.do"/><!-- CUSTOM:6 -->
    <property name="forceHttps" value="false"/>
  </bean>
  
  <bean id="logoutFilter" class="org.acegisecurity.ui.logout.LogoutFilter">
    <constructor-arg value="/myapp/index.jsp"/><!-- CUSTOM:7 -->
    <constructor-arg>
      <list>
       <bean class="org.acegisecurity.ui.logout.SecurityContextLogoutHandler"/>
      </list>
    </constructor-arg>
  </bean>
</beans>

[1]: The pattern /j_acegi_logout=logoutFilter means that when the URL is /j_acegi_logout, the logoutFilter (defined below) will be activated. The /**= line is actually one long line of filters (broken up into multiple lines for readability) forming a filter chain that would be activated for any other URL. Each of these filters are configured in subsequent bean definitions in this file.

[2]: authenticationProcessingFilter.authenticationFailureUrl defines the URL that would be triggered when authentication fails. This is a page that needs to be created and defined for your application. Note that we don't specify the context for it, it is /login.do?id=1 not /myapp/login.do?myapp=1

[3]: authenticationProcessingFilter.defaultTargetUrl is the URL which will be pulled up if the authentication is successful. Here we point to /myapp/index.jsp since it has logic to redirect to the appropriate page depending on the User in the SecurityContext.

[4]: accessDeniedHandler.errorPage is the page that will be pulled up if access is denied for some reason. Here it goes to the same page as the authenticationFailureUrl[2] above.

[5]: The patterns defined in the objectDefinitionSource specify which URL pattern is allowed for which role. In our case, all user URLs are owned by admin (ROLE_ADMIN), and all content URLs are owned by editors (ROLE_EDITOR). We accomodate the powers of ROLE_MANAGER and the constraint that an editor cannot edit content authored by another editor in Java code, as we show later.

[6]: formLoginAuthenticationEntryPoint.loginFormUrl is self explanatory - it is the URL of the login form. Note that we don't need to specify the context for it, it is simply /login.do, not /myapp/login.do.

[7]: logoutFilter.constructor-arg[0] is set to /myapp/index.jsp, this is the URL that the chain will go to on logging out.

Import security config into main applicationContext

For ease of testing, I typically move all my non-web beans into its own applicationContext-xxx.xml file in src/main/resources, but to keep this post short, I decided to put them into the main myapp-servlet.xml file (configuration for DispatcherServlet). Into this file, I import the applicationContext-security.xml file with the import resource tag. The full myapp-servlet.xml file is shown later below.

1	<import resource="classpath:applicationContext-security.xml"/>

Move JSPs into secure subdirectories

In our filterSecurityInterceptor configuration, our secure pages have URLs that begin with /secure/* so our secure URLs will look like this now (compare with the original list of URLs).

http://localhost:8081/myapp/index.jsp
http://localhost:8081/myapp/secure/users/list.do
http://localhost:8081/myapp/secure/users/edit.do?username=${username}
http://localhost:8081/myapp/contents/list.do
http://localhost:8081/myapp/secure/contents/edit.do?id=${contentId}

Since we use a vanilla Spring InternalResourceViewResolver to figure out where our views should go to, we need to move the JSP files around a bit. We basically create a secure sub-directory under src/main/webapp/myapp and create a tree of all the JSPs that need to be made secure.

webapp
  |
  +-- myapp
  |    |
  |    +-- index.jsp
  |    |
  |    +-- contents
  |    |     |
  |    |     +-- list.jsp
  |    |
  |    +-- secure
  |          |
  |          +-- contents
  |          |     |
  |          |     +-- edit.jsp
  |          |
  |          +-- users
  |                |
  |                +-- edit.jsp
  |                |
  |                +-- list.jsp
  +-- WEB-INF
       |
       +-- myapp-servlet.xml
       |
       +-- web.xml

Change index.jsp to do redirect

Originally, index.jsp was just a plain HTML page with a list of links. Now, we want to make it smart, so when we pull this page up, it will redirect either to the admin home page (secure/users/list.do) or the content home page (/contents/list.do) depending on what it finds in the SecurityContext.

<%-- Source: src/main/webapp/myapp/index.jsp --%>
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions"%>
<%@ taglib prefix="authz" uri="http://acegisecurity.org/authz" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>MyApp :: Home</title>
</head>
<body>
  <c:set var="username">
    <authz:authentication operation="username"/>
  </c:set>
  <c:choose>
    <c:when test="${username eq 'admin'}">
      <c:redirect url="/secure/users/list.do"/>
    </c:when>
    <c:otherwise>
      <c:redirect url="/contents/list.do"/>
    </c:otherwise>
  </c:choose>
</body>
</html>

New login_header_include.jsp

We create a component that shows if the user is logged on, and provides a logout button. We use Acegi's tag libraries to detect the username. The code is shown below:

<%-- Source: src/main/webapp/myapp/login_header_include.jsp --%>
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8" %>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core" %>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %>
<%@ taglib prefix="authz" uri="http://acegisecurity.org/authz" %>
<c:set var="username">
  <authz:authentication operation="username"/>
</c:set>
<c:choose>
  <c:when test="${not empty username}">
<%-- Note: Acegi specific for LogoutFilter --%>
Hello, <b>${username}</b>. Please <a href="/myapp/j_acegi_logout">Logout</a> when finished.
  </c:when>
  <c:otherwise>
Hello, <b>anonymous</b>. Please <a href="/myapp/login.do">Login</a> to update content.
  </c:otherwise>
</c:choose>
<br/>
<hr/>

We import this into all our JSP files (see tree above) using a JSTL <c:import url="..."/> tag. If we were using Tiles or Sitemesh, it could probably be done more elegantly. Here are a couple of screen shots to show what this looks like with user logged out and in.

New login.jsp

We need a new login.jsp to allow the user to enter his username and password. We use several Acegi "reserved words" in here to use it's built-in modules to do the backend authentication work.

<%-- Source: src/main/webapp/myapp/login.jsp --%>
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Login</title>
</head>
<body>
<h1>Login</h1>
<form method="post" action="j_acegi_security_check">
  <c:if test="${not empty param.id}">
    <c:set var="username" value="${sessionScope.ACEGI_SECURITY_LAST_USERNAME}"/>
    <c:if test="${empty username}">
      <c:set var="username" value="anonymous"/>
    </c:if>
    <c:set var="errorMessage" value="${ACEGI_SECURITY_403_EXCEPTION.message}"/>
    <c:if test="${empty errorMessage}">
        <c:set var="errorMessage" value="Access Denied! Please contact Administrator."/>
    </c:if>
    <font color="red"><b>Hello ${username}. Message from server: ${errorMessage}</b></font><br/><br/>
  </c:if>
  <b>User: </b><input type="text" name="j_username"/><br/>
  <b>Pass: </b><input type="password" name="j_password"/><br/>
  <input type="submit" value="Login"/>
</form>
<br/><a href="/myapp/">Home</a>
</body>
</html>

New handler mappings and new Spring built-in controller

For the two new JSPs we added, we declare a UrlFilenameViewController to handle them and define new mappings for them, like so (We will show the complete myapp-servlet.xml file later).

  <bean id="urlFilenameViewController"
    class="org.springframework.web.servlet.mvc.UrlFilenameViewController"/>

  <bean id="urlMapping"
    class="org.springframework.web.servlet.handler.SimpleUrlHandlerMapping">
    <property name="mappings">
      <props>
        ...
        <prop key="login_header_include.do">urlFilenameViewController</prop>
        <prop key="login.do">urlFilenameViewController</prop>
      </props>
    </property>
  </bean>

Changes contents/list.jsp

We want to provide a slightly different view of the contents/list.do page to non-authenticated users than we want for authenticated users. For authenticated users, we should display the edit link, and for non-authenticated users, we should suppress the edit link. This is done in the JSP with Acegi tags, as shown below:

<%-- $Source: src/main/webapp/myapp/contents/list.jsp -%>
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions"%>
<%@ taglib prefix="authz" uri="http://acegisecurity.org/authz" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Contents :: List</title>
</head>
<body>
  <h1>Contents :: List</h1>
  <c:import url="/login_header_include.do"/>
  <c:set var="username">
    <authz:authentication operation="username"/>
  </c:set>
  <table cellpadding="1" cellspacing="1" border="1">
    <tr>
      <th>Title</th>
      <th>Summary</th>
      <th>Author</th>
      <c:if test="${not empty username}">
        <th>Edit?</th>
      </c:if>
    </tr>
    <c:forEach items="${contents}" var="content">
      <tr>
        <td><a href="${content.url}">${content.title}</a></td>
        <td>${content.summary}</td>
        <td>${content.author}</td>
        <c:if test="${not empty username}">
          <td><a href="/myapp/secure/contents/edit.do?id=${content.id}">Edit</a></td>
        </c:if>
      </tr>
    </c:forEach>
  </table>
</body>
</html>

Screenshots below illustrate the different views of the home page for different users.

View of the home page for "Anonymous" (unauthenticated user)

View of the home page for "Admin"

View of the home page for "Bob" (an editor)

New method interceptor for ContentDao

At this point, our user pages are behind login and only admin can access them. Users with ROLE_EDITOR are able to access the contents/list.do page and are able to edit. However, we want to make sure that bob sees only his own stories and cannot edit edit larry's stories and vice versa, and that Mary can see and edit both.

I am not sure if we can just use some Acegi functionality to do this, but I opted for writing a MethodInterceptor around ContentDao.list() and ContentDao.edit() to build this. Here is the code for the interceptor. R J Lorimer's blog post Spring: A quick journey through AOP was very helpful.

// Source: src/main/java/com/mycompany/myapp/MySecurityInterceptor.java
package com.mycompany.myapp;

import java.util.ArrayList;
import java.util.List;

import org.acegisecurity.AccessDeniedException;
import org.acegisecurity.Authentication;
import org.acegisecurity.GrantedAuthority;
import org.acegisecurity.context.SecurityContextHolder;
import org.acegisecurity.providers.UsernamePasswordAuthenticationToken;
import org.aopalliance.intercept.MethodInterceptor;
import org.aopalliance.intercept.MethodInvocation;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public class MySecurityInterceptor implements MethodInterceptor {

  private final Log log = LogFactory.getLog(getClass());

  public Object invoke(MethodInvocation invocation) throws Throwable {
    // get username and role from the SecurityContext, if one exists
    String username = null;
    boolean isEditor = false;
    boolean isManager = false;
    Authentication auth = 
      SecurityContextHolder.getContext().getAuthentication();
    if (auth != null) {
      // not anonymous user, so needs to be checked
      UsernamePasswordAuthenticationToken token =
        (UsernamePasswordAuthenticationToken) auth;
      username = token.getName();
      GrantedAuthority[] authorities = token.getAuthorities();
      for (GrantedAuthority authority : authorities) {
        String role = authority.getAuthority();
        if (role.equals("ROLE_MANAGER")) {
          isManager = true;
        }
        if (role.equals("ROLE_EDITOR")) {
          isEditor = true;
        }
      }
    }
    Object returnValue = invocation.proceed();
    String methodName = invocation.getMethod().getName();
    if (methodName.equals("list")) {
      List<Content> contents = (List<Content>) returnValue;
      List<Content> postProcessedContents = 
        new ArrayList<Content>();
      if (isEditor && !isManager) {
        log.debug("Post processing list for user:[" + username + "]");
        // only if role is non-null and ROLE_USER, we want to filter by
        // author name, else we want to show all
        for (Content content : contents) {
          if (content.getAuthor().equals(username)) {
            postProcessedContents.add(content);
          }
        }
      } else {
        postProcessedContents.addAll(contents);
      }
      return postProcessedContents;
    } else if (methodName.equals("getById")) {
      Content content = (Content) returnValue;
      if (isEditor && !isManager) {
        // make sure he owns the content he is editing
        log.debug("Checking that user is allowed to edit");
        if (! content.getAuthor().equals(username)) {
          throw new AccessDeniedException("Don't covet thy neighbor's data");
        }
      }
      return content; 
    } else {
      return returnValue;
    }
  }
}

The interception for the list() method is to simply post-process the List of Content objects returned and filtering by user name. Probably not the most efficient approach - if I was working with real data, I would have opted to refactor out the SQL building logic from ContentDao.list() to append a WHERE author=${username} and wrap that instead. So Mary's data passes through unchanged because she has ROLE_MANAGER, but Larry's data gets removed from the list returned to Bob. You can see the difference below:

The interception of the edit() method checks to see if the returned Content has an author equal to ${username}, and if not, throws a AccessDeniedException with an informative message, as you can see below. The screenshots below show the behavior when Bob tries to access Larry's content by changing the id parameter in the URL of the edit form.

The myapp-servlet.xml file

I show below the full myapp-servlet.xml file, including the configuration for the interceptor described above. I use CGLIB proxying to avoid having to create an artificial interface for ContentDao.

<?xml version="1.0" encoding="UTF-8"?>
<!-- $Source: src/main/webapp/WEB-INF/myapp-servlet.xml -->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans 
       http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
       http://www.springframework.org/schema/util 
       http://www.springframework.org/schema/util/spring-util-2.0.xsd">

  <!-- Security -->
  <import resource="classpath:applicationContext-security.xml"/>

  <!-- Datasources and DAOs -->
  <bean id="dataSource"
    class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
    <property name="url" value="jdbc:mysql://localhost:3306/securitydb"/>
    <property name="username" value="insecure"/>
    <property name="password" value="depressed"/>
  </bean>
  
  <bean id="userDao" class="com.mycompany.myapp.UserDao">
    <property name="dataSource" ref="dataSource"/>
  </bean>  
  
  <bean id="contentDao"
    class="org.springframework.aop.framework.ProxyFactoryBean">
    <property name="target" ref="contentDaoTarget"/>
    <property name="proxyTargetClass" value="true"/>
    <property name="interceptorNames">
      <list>
        <value>mySecurityInterceptor</value>
      </list>
    </property>
  </bean>
  <bean id="contentDaoTarget" class="com.mycompany.myapp.ContentDao">
    <property name="dataSource" ref="dataSource"/>
  </bean>
  <bean id="mySecurityInterceptor"
    class="com.mycompany.myapp.MySecurityInterceptor"/>
  
  <!-- View resolver -->
  <bean id="viewResolver"
   class="org.springframework.web.servlet.view.InternalResourceViewResolver">
    <property name="prefix" value="/myapp/" />
    <property name="suffix" value=".jsp" />
    <property name="viewClass" value="org.springframework.web.servlet.view.JstlView" />
  </bean>

  <!-- Controllers -->
  <bean id="userController" class="com.mycompany.myapp.UserController">
    <property name="userDao" ref="userDao"/>
  </bean>

  <bean id="contentController"
    class="com.mycompany.myapp.ContentController">
    <property name="contentDao" ref="contentDao"/>
  </bean>
    
  <!-- Add this one for Acegi JSPs -->
  <bean id="urlFilenameViewController"
    class="org.springframework.web.servlet.mvc.UrlFilenameViewController"/>
  
  <!-- URL Mappings -->
  <bean id="urlMapping"
    class="org.springframework.web.servlet.handler.SimpleUrlHandlerMapping">
    <property name="mappings">
      <props>
        <prop key="/secure/users/list.do">userController</prop>
        <prop key="/secure/users/edit.do">userController</prop>
        <prop key="/secure/users/save.do">userController</prop>
        <prop key="/contents/list.do">contentController</prop>
        <prop key="/secure/contents/edit.do">contentController</prop>
        <prop key="/secure/contents/save.do">contentController</prop>
        <!-- Add these mappings for Acegi -->
        <prop key="login_header_include.do">urlFilenameViewController</prop>
        <prop key="login.do">urlFilenameViewController</prop>
      </props>
    </property>
  </bean>

</beans>

Database tables

There are three database tables in our application - two of them mandated by Acegi, although we reuse it to power our User bean. Here they are:

mysql> desc users;
+----------+-------------+------+-----+---------+-------+
| Field    | Type        | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| username | varchar(32) | NO   | PRI | NULL    |       | 
| password | varchar(32) | NO   |     | NULL    |       | 
| enabled  | tinyint(1)  | NO   |     | NULL    |       | 
+----------+-------------+------+-----+---------+-------+

mysql> desc authorities;
+-----------+-------------+------+-----+---------+-------+
| Field     | Type        | Null | Key | Default | Extra |
+-----------+-------------+------+-----+---------+-------+
| username  | varchar(32) | NO   | PRI | NULL    |       | 
| authority | varchar(32) | NO   | PRI | NULL    |       | 
+-----------+-------------+------+-----+---------+-------+

mysql> desc content;
+---------+--------------+------+-----+---------+-------+
| Field   | Type         | Null | Key | Default | Extra |
+---------+--------------+------+-----+---------+-------+
| id      | int(32)      | NO   | PRI | NULL    |       | 
| title   | varchar(128) | NO   |     | NULL    |       | 
| summary | varchar(255) | NO   |     | NULL    |       | 
| url     | varchar(128) | NO   |     | NULL    |       | 
| author  | varchar(32)  | NO   |     | NULL    |       | 
+---------+--------------+------+-----+---------+-------+

The contents of the USERS and AUTHORITIES tables are shown below. As you can see, Mary has both ROLE_EDITOR and ROLE_MANAGER roles, which is why she can edit content, but does not have the constraint Bob and Larry (ROLE_EDITOR only) do.

mysql> select * from users;
+----------+----------+---------+
| username | password | enabled |
+----------+----------+---------+
| admin    | admin    |       1 | 
| bob      | bob      |       1 | 
| larry    | larry    |       1 | 
| mary     | mary     |       1 | 
+----------+----------+---------+
4 rows in set (0.00 sec)

mysql> select * from authorities;
+----------+--------------+
| username | authority    |
+----------+--------------+
| admin    | ROLE_ADMIN   | 
| bob      | ROLE_EDITOR  | 
| larry    | ROLE_EDITOR  | 
| mary     | ROLE_EDITOR  | 
| mary     | ROLE_MANAGER | 
+----------+--------------+
5 rows in set (0.00 sec)

Other code (for reference)

The rest of the code (not covered above) are pretty vanilla, and are provided here for completeness and just in case you like to cut and paste things. All code has a comment line which points to its relative location within the Maven2 web application.

The content set is a ContentController, which delegates to the ContentDao for most of the heavy lifting, and Content bean object which is what is populated and sent to the view. The main changes between the insecure version and the secure one are the changes in the check for various servlet paths and the corresponding views. Knowing what I do now, I would have probably factored out these patterns so changing them is simpler when security is applied.

// Source: src/main/java/com/mycompany/myapp/ContentController.java
package com.mycompany.myapp;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.web.bind.ServletRequestUtils;
import org.springframework.web.servlet.ModelAndView;
import org.springframework.web.servlet.mvc.Controller;

public class ContentController implements Controller {

  private final Log log = LogFactory.getLog(getClass());
  
  private ContentDao contentDao;
  
  public void setContentDao(ContentDao contentDao) {
    this.contentDao = contentDao;
  }
  
  public ModelAndView handleRequest(HttpServletRequest request,
      HttpServletResponse response) throws Exception {
    String action = request.getServletPath();
    ModelAndView mav = new ModelAndView();
    if (action.equals("/contents/list.do")) {
      mav.addObject("contents", contentDao.list());
      mav.setViewName("contents/list");
    } else if (action.equals("/secure/contents/edit.do")) {
      Integer id = 
        ServletRequestUtils.getRequiredIntParameter(request, "id");
      mav.addObject("content", contentDao.getById(id));
      mav.setViewName("secure/contents/edit");
    } else if (action.equals("/secure/contents/save.do")) {
      Integer id = 
        ServletRequestUtils.getRequiredIntParameter(request, "id");
      String title = 
        ServletRequestUtils.getRequiredStringParameter(request, "title");
      String summary = 
        ServletRequestUtils.getRequiredStringParameter(request, "summary");
      String url = 
        ServletRequestUtils.getRequiredStringParameter(request, "url");
      String author = 
        ServletRequestUtils.getRequiredStringParameter(request, "author");
      contentDao.save(id, title, summary, url, author);
      mav.addObject("contents", contentDao.list());
      mav.setViewName("contents/list");
    } else {
      return null;
    }
    return mav;
  }
}

// Source: src/main/java/com/mycompany/myapp/ContentDao.java
package com.mycompany.myapp;

import java.util.ArrayList;
import java.util.List;
import java.util.Map;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.support.JdbcDaoSupport;

public class ContentDao extends JdbcDaoSupport {

  private final Log log = LogFactory.getLog(getClass());
  
  @SuppressWarnings("unchecked")
  public List<Content> list() {
    log.debug("Running list()");
    List<Map<String,Object>> rows = 
      getJdbcTemplate().queryForList(
      "select id, author, summary, title, url from content"); 
    List<Content> contents = new ArrayList<Content>();
    for (Map<String,Object> row : rows) {
      Content content = new Content();
      content.setId((Integer) row.get("ID"));
      content.setAuthor((String) row.get("AUTHOR"));
      content.setSummary((String) row.get("SUMMARY"));
      content.setTitle((String) row.get("TITLE"));
      content.setUrl((String) row.get("URL"));
      contents.add(content);
    }
    return contents;
  }

  @SuppressWarnings("unchecked")
  public Content getById(Integer id) {
    log.debug("running getById()");
    try {
      Map<String,Object> row = getJdbcTemplate().queryForMap(
        "select id, author, summary, title, url from content " +
        "where id=?", new Integer[] {id});
      Content content = new Content();
      content.setId((Integer) row.get("ID"));
      content.setAuthor((String) row.get("AUTHOR"));
      content.setSummary((String) row.get("SUMMARY"));
      content.setTitle((String) row.get("TITLE"));
      content.setUrl((String) row.get("URL"));
      return content;
    } catch (DataAccessException e) {
      return null;
    }
  }

  public void save(Integer id, String title, String summary, String url, String author) {
    Content content = getById(id);
    if (content == null) {
      long nextId = getNextId();
      getJdbcTemplate().update(
        "insert into content(id,author,summary,title,url)" +
        "values(?,?,?,?,?)",
        new Object[] {nextId, title, summary, url, author});
    } else {
      getJdbcTemplate().update(
        "update content set author = ?, summary = ?, title = ?, " +
        "url = ? where id = ?", 
        new Object[] {author, summary, title, url, id});
    }
  }

  private int getNextId() {
    return getJdbcTemplate().queryForInt("select max(id) + 1 from content");
  }

}

// Source: src/main/java/com/mycompany/myapp/Content.java
package com.mycompany.myapp;

import org.apache.commons.lang.builder.ReflectionToStringBuilder;
import org.apache.commons.lang.builder.ToStringStyle;

public class Content {

  private Integer id;
  private String title;
  private String summary;
  private String url;
  private String author;

  // ... getters and setters omitted
}

And here is the content edit.jsp. We have already seen the list.jsp file above.

<%-- Source: src/main/webapp/myapp/secure/contents/edit.jsp --%>
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Contents :: Edit</title>
</head>
<body>
  <h1>Contents :: Edit</h1>
  <c:import url="/login_header_include.do"/>
  <form method="post" action="/myapp/secure/contents/save.do">
    <input type="hidden" name="id" value="${content.id}"/>
    <b>Title: </b><input type="text" name="title" value="${content.title}"/><br/>
    <b>URL: </b><input type="text" name="url" value="${content.url}"/><br/>
    <b>Author: </b><input type="text" name="author" value="${content.author}"/><br/>
    <b>Summary: </b><br/>
    <textarea name="summary">${content.summary}</textarea><br/>
    <input type="submit" value="Save"/>
  </form> 
</body>
</html>

The User side is similar to the Content side, ie, follows a similar pattern. The Controller, Dao and bean, and the list and edit JSPs are shown below:

// Source: src/main/java/com/mycompany/myapp/UserController.java
package com.mycompany.myapp;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.web.bind.ServletRequestUtils;
import org.springframework.web.servlet.ModelAndView;
import org.springframework.web.servlet.mvc.Controller;

public class UserController implements Controller {

  private final Log log = LogFactory.getLog(getClass());
  
  private UserDao userDao;
  
  public void setUserDao(UserDao userDao) {
    this.userDao = userDao;
  }
  
  public ModelAndView handleRequest(HttpServletRequest request,
      HttpServletResponse response) throws Exception {
    String action = request.getServletPath();
    ModelAndView mav = new ModelAndView();
    if (action.equals("/secure/users/list.do")) {
      mav.addObject("users", userDao.list());
      mav.setViewName("secure/users/list");
    } else if (action.equals("/secure/users/edit.do")) {
      String name = 
        ServletRequestUtils.getRequiredStringParameter(request, "name");
      mav.addObject("user", userDao.getByName(name));
      mav.setViewName("secure/users/edit");
    } else if (action.equals("/secure/users/save.do")) {
      String name = 
        ServletRequestUtils.getRequiredStringParameter(request, "name");
      String pass = 
        ServletRequestUtils.getRequiredStringParameter(request, "pass");
      String role = 
        ServletRequestUtils.getStringParameter(request, "role", "ROLE_USER");
      userDao.save(name, pass, role);
      mav.addObject("users", userDao.list());
      mav.setViewName("secure/users/list");
    } else {
      return null;
    }
    return mav;
  }
}

// Source: src/main/java/com/mycompany/myapp/UserDao.java
package com.mycompany.myapp;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;

import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.dao.DataAccessException;
import org.springframework.jdbc.core.support.JdbcDaoSupport;

public class UserDao extends JdbcDaoSupport {

  private final Log log = LogFactory.getLog(getClass());
  
  @SuppressWarnings("unchecked")
  public List<User> list() {
    List<Map<String,String>> rows = 
      getJdbcTemplate().queryForList(
      "select username, password " +
      "from users " +
      "where enabled = 1");
    List<User> users = new ArrayList<User>();
    for (Map<String,String> row : rows) {
      User user = new User();
      user.setName(row.get("USERNAME"));
      user.setPass(row.get("PASSWORD"));
      user.setRole(getRoles(user.getName()));
      users.add(user);
    }
    return users;
  }

  @SuppressWarnings("unchecked")
  public User getByName(String name) {
    try {
      Map<String,String> row = getJdbcTemplate().queryForMap(
        "select username, password " +
        "from users " +
        "where enabled = 1 " +
        "and username = ?", new String[] {name});
      User user = new User();
      user.setName(row.get("USERNAME"));
      user.setPass(row.get("PASSWORD"));
      user.setRole(getRoles(user.getName()));
      return user;
    } catch (DataAccessException e) {
      log.error("Can't find user:" + name);
      return null;
    }
  }
  
  public void save(String name, String pass, String role) {
    User user = getByName(name);
    if (user == null) {
      getJdbcTemplate().update(
        "insert into users(username,password,enabled) values (?,?,?)",
        new Object[] {name, pass, new Integer(1)});
    } else {
      getJdbcTemplate().update(
        "update users set password = ? where username = ?", 
        new String[] {pass, name});
    }
    updateAuthorities(name, role);
  }
  
  @SuppressWarnings("unchecked")
  private String getRoles(String username) {
    List<Map<String,String>> authRows = 
      getJdbcTemplate().queryForList(
      "select authority from authorities where username = ?", 
      new String[] {username});
    List<String> authorities = new ArrayList<String>();
    for (Map<String,String> authRow : authRows) {
      authorities.add(authRow.get("AUTHORITY"));
    }
    Collections.sort(authorities);
    return StringUtils.join(authorities.iterator(), ",");
  }

  private void updateAuthorities(String name, String role) {
    getJdbcTemplate().update(
      "delete from authorities where username=?", 
      new String[] {name});
    String[] authorities = StringUtils.split(role, ",");
    for (String authority : authorities) {
      getJdbcTemplate().update(
        "insert into authorities(username, authority) values (?,?)",
        new String[] {name, authority});
    }
  }
}

// Source: src/main/java/com/mycompany/myapp/User.java
package com.mycompany.myapp;

import org.apache.commons.lang.builder.ReflectionToStringBuilder;
import org.apache.commons.lang.builder.ToStringStyle;

public class User {

  private String name;
  private String pass;
  private String role;
  
  // ... getters and setters omitted
}

<%-- Source: src/main/webapp/myapp/secure/users/edit.jsp --%>
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Users :: Edit</title>
</head>
<body>
  <h1>Users :: Edit</h1>
  <c:import url="/login_header_include.do"/>
  <form method="post" action="/myapp/secure/users/save.do">
    <b>Username: </b><input type="text" name="name" value="${user.name}" readonly/><br/>
    <b>Password: </b><input type="text" name="pass" value="${user.pass}"/><br/>
    <b>Role: </b><input type="text" name="role" value="${user.role}"/><br/>
    <input type="submit" value="Save"/>
  </form> 
</body>
</html>

<%-- Source: src/main/webapp/myapp/secure/users/list.jsp --%>
<%@ page language="java" contentType="text/html; charset=UTF-8" pageEncoding="UTF-8"%>
<%@ taglib prefix="c" uri="http://java.sun.com/jsp/jstl/core"%>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Users :: List</title>
</head>
<body>
  <h1>Users :: List</h1>
  <c:import url="/login_header_include.do"/>
  <table cellspacing="1" cellpadding="1" border="1">
  <tr>
    <th>User</th>
    <th>Pass</th>
    <th>Role</th>
    <th>Edit?</th>
  </tr>
  <c:forEach items="${users}" var="user">
    <tr>
      <td>${user.name}</td>
      <td>${user.pass}</td>
      <td>${user.role}</td>
      <td><a href="/myapp/secure/users/edit.do?name=${user.name}">Edit</a></td>
    </tr>
  </c:forEach>
  </table>
</body>
</html>

What's next?

This was a long post. If you came this far, thank you for sticking around, and I hope the article helped. While the setup presented in this blog is fairly basic and to some extent, unrealistic, but it gives a good feel for Acegi configuration. I plan to look at the following things in the future and blog about them if they work out:

Add Remember-me support - this is built into Acegi, and is simply a matter of adding some filters to the filter chain. It should be fairly easy, though.
Add Legacy Database support - The current implementation uses the Acegi preferred database tables USERS and AUTHORITIES. Organizations are very likely to have existing tables with this information.
Add LDAP support - Most organizations use LDAP to store employee email addresses and passwords. Being able to connect to the LDAP store to authenticate would be really good if one wanted to do a single-signon setup across the company Intranet.
Add X509 support - I would like to apply X509 authentication to a webservice application I am building.

Saturday, August 02, 2008

Parsing custom modules with ROME Fetcher

Last week, I described a fairly basic feed fetcher written using ROME's Fetcher library. My intent is provide clients of our RSS 2.0 based API a convenient way to access the XML as a Java object, negating the need for XML parsing on their end. Our API does return standard RSS 2.0, but we add extra information using a custom module at both the feed and the entry level, which the implementation described in the last post could not parse out as a module. Instead, it treated it as foreign markup, which a client could parse out using the following snippet.

  SyndFeed feed = client.execute(...);
  List<SyndEntry> entries = feed.getEntries();
  for (SyndEntry entry : entries) {
    ...
    List<Element> foreignMarkups = (List<Element>) entry.getForeignMarkup();
    for (Element foreignMarkup : foreignMarkups) {
      if (foreignMarkup.getNamespaceURI().equals(MyModule.URI)) {
        // we got our custom module, now parse it
        if (foreignMarkup.getName().equals("score")) {
          // extract and populate the value of score
          float score = Float.valueOf(foreignMarkup.getValue());
        }
        ...
      }
    }
  }

This is a workable solution, but not ideal. What I would like is for clients to be able to get a reference to the custom module by URI, then use the getters and setters defined on the module to populate their objects. This post describes the changes I had to make to get this functionality to work.

ROME depends on a plug-in mechanism that is driven by the rome.properties file. ROME comes with one built in, but it can be overriden by placing one's custom rome.properties file at the root of the classpath. So here is my rome.properties file for reference.

# rome.properties

rss_2.0my.feed.ModuleGenerator.classes=\
com.sun.syndication.io.impl.DCModuleGenerator \
com.sun.syndication.io.impl.SyModuleGenerator \
com.sun.syndication.feed.module.opensearch.impl.OpenSearchModuleGenerator \
com.mycompany.myapp.mymodule.MyModuleGenerator

rss_2.0my.feed.ModuleParser.classes=\
com.sun.syndication.io.impl.DCModuleParser \
com.sun.syndication.io.impl.SyModuleParser \
com.sun.syndication.feed.module.opensearch.impl.OpenSearchModuleParser \
com.mycompany.myapp.mymodule.MyModuleParser

rss_2.0my.item.ModuleParser.classes=\
com.mycompany.myapp.mymodule.MyModuleParser

rss_2.0my.item.ModuleGenerator.classes=\
com.mycompany.myapp.mymodule.MyModuleGenerator

WireFeedParser.classes=\
com.sun.syndication.io.impl.RSS090Parser \
com.sun.syndication.io.impl.RSS091NetscapeParser \
com.sun.syndication.io.impl.RSS091UserlandParser \
com.sun.syndication.io.impl.RSS092Parser \
com.sun.syndication.io.impl.RSS093Parser \
com.sun.syndication.io.impl.RSS094Parser \
com.sun.syndication.io.impl.RSS10Parser  \
com.sun.syndication.io.impl.RSS20wNSParser  \
com.sun.syndication.io.impl.RSS20Parser  \
com.sun.syndication.io.impl.Atom10Parser \
com.sun.syndication.io.impl.Atom03Parser \
com.mycompany.myapp.mymodule.MyRss20Parser

WireFeedGenerator.classes=\
com.sun.syndication.io.impl.RSS090Generator \
com.sun.syndication.io.impl.RSS091NetscapeGenerator \
com.sun.syndication.io.impl.RSS091UserlandGenerator \
com.sun.syndication.io.impl.RSS092Generator \
com.sun.syndication.io.impl.RSS093Generator \
com.sun.syndication.io.impl.RSS094Generator \
com.sun.syndication.io.impl.RSS10Generator  \
com.sun.syndication.io.impl.RSS20Generator  \
com.sun.syndication.io.impl.Atom10Generator \
com.sun.syndication.io.impl.Atom03Generator \
com.mycompany.myapp.mymodule.MyRss20Generator

Converter.classes=\
com.sun.syndication.feed.synd.impl.ConverterForAtom10 \
com.sun.syndication.feed.synd.impl.ConverterForAtom03 \
com.sun.syndication.feed.synd.impl.ConverterForRSS090 \
com.sun.syndication.feed.synd.impl.ConverterForRSS091Netscape \
com.sun.syndication.feed.synd.impl.ConverterForRSS091Userland \
com.sun.syndication.feed.synd.impl.ConverterForRSS092 \
com.sun.syndication.feed.synd.impl.ConverterForRSS093 \
com.sun.syndication.feed.synd.impl.ConverterForRSS094 \
com.sun.syndication.feed.synd.impl.ConverterForRSS10  \
com.sun.syndication.feed.synd.impl.ConverterForRSS20 \
com.mycompany.myapp.mymodule.ConverterForMyRss20

You will notice that I am using a custom version of RSS 2.0 called rss_2.0my. The reason for this is ROME SyndEntry does not have a way to set the rss/channel/item/source element, which I handled by extending ROME. Additionally, we use a custom module MyModule that carries information that is not possible to send using standard RSS. We also use the Amazon OpenSearch and the Content modules.

We have used ROME for a while to successfully generate the feeds, but this was the first time we were looking at eating our own dog food, as it were. I am not sure if our setup is that uncommon, but either there are gaps in the ROME parsing code or we are doing something wrong. If you have been through this and solved this more cleanly, your comments and suggestions would be appreciated.

Change to ROME: FeedParsers.java

Currently, the code for FeedParsers.getParserFor(Document) loops through the parsers defined for WireFeedParser.classes in rome.properties. Each parser's isMyType() method is invoked, and if satisfied, the first parser is selected.

The problem is that the RSS20Parser.isMyType(Document) is too loose, all it does is verify that the root element is "rss" and the value of rss.@version startswith "2.0". So what is selected is the RSS20Parser, which is not what I want, and besides all my modules are registered to the feed type rss-2.0my, which is what is supported by MyRss20Parser.

So I needed it to keep going until it found the last matched parser, so I made the method sticky. An alternative approach would be to traverse the list backwards, since the selection would then go from the most specific parser to the least specific. Here is my code for FeedParsers.getParserFor(Document).

    public WireFeedParser getParserFor(Document document) {
        List parsers = getPlugins();
        WireFeedParser selectedParser = null;
        for (int i=0;i < parsers.size();i++) {
            WireFeedParser parser = (WireFeedParser) parsers.get(i);
            if (parser.isMyType(document)) {
                selectedParser = parser;
            }
        }
        return selectedParser;
    }

Change to MyRss20Parser.java

I added a isMyType(Document) method to my custom RSS20Parser so it does not call the superclass's isMyType(). It does piggyback on RSS20Parser.isMyType() to figure out if it RSS 2.0, and if so, whether it contains the namespace for MyModule.

  public boolean isMyType(Document document) {
    boolean isValidRss20 = super.isMyType(document);
    boolean isValidMyRss20 = false;
    if (isValidRss20) {
      Element rssRoot = document.getRootElement();
      List<Namespace> namespaces = rssRoot.getAdditionalNamespaces();
      for (Namespace namespace : namespaces) {
        if (namespace.getURI().equals(MyModule.URI));
        isValidMyRss20 = true;
        break;
      }
    }
    return isValidRss20 && isValidMyRss20;
  }

The changes to ROME their CVS versions downloaded on 2008-07-23. I plan on submitting a patch for the changes, so hopefully it will be available in future releases of the software. Unless, of course, there is a workaround, which I would be happy to use.

Additionally, I am using the ROME Fetcher code from CVS. Unlike the 0.9 version available at the time of this writing, the CVS version has support for configurable connection and read timeouts, which I wanted to provide. But I already talked about that in the previous post.

With these changes, I am finally able to get results using the following code on my client application.

Salmon Run