Salmon Run: April 2009

Monday, April 20, 2009

XPath over HTML using Jericho and Jaxen

I've been using Jericho to parse HTML for a while now. I mainly use it to extract pieces of text from specific locations in the HTML. To do this, I use the Jericho API - I have factored out the boilerplate code associated with XML/HTML parsing into utility classes, so the application code to extract an element's contents would be 1 or 2 lines tops. However, recently I needed to move the extraction logic out of the code and into configuration.

Using XPath in these situations is almost a no-brainer - given an XML document and tasked with extracting text for known elements, that is the first tool I would think about. Unfortunately, Jericho does not have XPath support, so being the foolish/foolhardy type, I set about trying to build one. This post describes a Jaxen based XPath adapter for Jericho.

The Jaxen project provides a universal XPath engine, capable of evaluating XPath expressions across multiple object models - currently supported in the GA (1.1.1 at the time of writing this) release are dom4j, JDOM, w3c DOM, Javabeans and XOM. The Jaxen FAQ has this to say about extending Jaxen for other object models.

The only thing required is an implementation of the interface org.jaxen.Navigator. Not all of the interface is required, and a default implementation, in the form of org.jaxen.DefaultNavigator is also provided. Since many of the XPath axes can be defined in terms of each other (for example, the ancestor axis is merely a the parent recursively applied), only a few low-level axis iterators are required to initially get started. Of course, you may implement them directly, instead of relying upon jaxen's composition ability.

In my opinion, the text above falls into the technically accurate but not very useful as a guideline category. Granted, there are multiple extension examples in the source tarball (for dom4j, JDOM, etc) to use as examples (which I did), but what would have helped immensely is a short howto style tutorial, that explains which methods of DefaultNavigator need to be customized and why.

I was finally able to get everything to work by running my unit test through a debugger, stepping through both the Jaxen and my extension code. For example, I found that I should implementing the methods of NamedAccessNavigator, although that is not a stated requirement. There are also places where the methods of DefaultNavigator return null or throw UnsupportedOperationException and are therefore expected to be extended by subclasses, but without being explicitly marked abstract.

In any case, I now know a little more than I knew about Jaxen internals, and I must say the extension model is very well thought out, and my gripe with the implementation may just be due to the fact that the authors haven't gotten around to cleaning it up so its more usable. Hopefully, if you are in a similar situation as I was recently, then the code in this blog post may be helpful - in the form of yet another example of extension.

My approach is based on the first paragraph of the guideline, ie, building a custom Navigator implementation using the JDOM implementation as a template. Basically, I have a JerichoXPath class which exposes two constructors and a custom getContext() method, and a DocumentNavigator implementation that contains calls to the Jericho API to do different things. The Jaxen XPath parser calls into these DocumentNavigator methods at specific points in its life cycle.

JerichoXPath

package org.jaxen.jericho;

import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jaxen.BaseXPath;
import org.jaxen.Context;
import org.jaxen.JaxenException;
import org.jaxen.Navigator;
import org.jaxen.util.SingletonList;

import au.id.jericho.lib.html.Element;
import au.id.jericho.lib.html.Source;

public class JerichoXPath extends BaseXPath {

  private static final long serialVersionUID = -6969112785840871593L;

  private final Log log = LogFactory.getLog(getClass());
  
  public JerichoXPath(String xpathExpr, Navigator navigator) 
      throws JaxenException {
    super(xpathExpr, navigator);
  }
  
  public JerichoXPath(String xpathExpr) throws JaxenException {
    super(xpathExpr, DocumentNavigator.getInstance());
  }

  /**
   * Jericho specific method to get the context associated with a node.
   * @param node the current node being visited.
   * @return the Context associated with the node.
   */
  protected Context getContext(Object node) {
    if (node instanceof Context) {
      return (Context) node;
    }
    Context fullContext = new Context(getContextSupport());
    if (node instanceof Source) {
      Element rootNode = 
        (Element) getNavigator().getDocumentNode((Source) node);
      fullContext.setNodeSet(new SingletonList(rootNode));
    } else if (node instanceof List) {
      fullContext.setNodeSet((List) node);
    } else {
      List list = new SingletonList(node);
      fullContext.setNodeSet(list);
    }
    return fullContext;
  }
}

DocumentNavigator

package org.jaxen.jericho;

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jaxen.DefaultNavigator;
import org.jaxen.JaxenConstants;
import org.jaxen.NamedAccessNavigator;
import org.jaxen.Navigator;
import org.jaxen.UnsupportedAxisException;
import org.jaxen.XPath;
import org.jaxen.saxpath.SAXPathException;
import org.jaxen.util.SingleObjectIterator;

import au.id.jericho.lib.html.Attribute;
import au.id.jericho.lib.html.Attributes;
import au.id.jericho.lib.html.CharacterReference;
import au.id.jericho.lib.html.Element;
import au.id.jericho.lib.html.Segment;
import au.id.jericho.lib.html.Source;
import au.id.jericho.lib.html.StartTagType;

public class DocumentNavigator extends DefaultNavigator 
    implements NamedAccessNavigator {

  private static final long serialVersionUID = 8640276699026512314L;

  private final Log log = LogFactory.getLog(getClass());
  
  // DocumentNavigator needs to expose a singleton. Strategy adapted from: 
  // http://www.ibm.com/developerworks/java/library/j-dcl.html

  private DocumentNavigator() {
    super();
  }
  private static DocumentNavigator INSTANCE = new DocumentNavigator(); 
  
  /**
   * Return a singleton instance of the DocumentNavigator object.
   * @return a Navigator.
   */
  public static Navigator getInstance() {
    return INSTANCE;
  }
  
  // various abstract isXXX method implementations. These are specified
  // in the Navigator interface but not defined in the abstract class,
  // so they must be implemented in the implementation specific subclass.

  /**
   * Returns true if the node is an Attribute.
   * @param obj the node to consider.
   * @return true if node is an attribute, else false.
   */
  public boolean isAttribute(Object obj) {
    return obj instanceof Attribute;
  }

  /**
   * Returns true if the node is a Comment.
   * @param obj the node to consider.
   * @return true if the node is a comment, else false.
   */
  public boolean isComment(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return isElementOfType(element, StartTagType.COMMENT);
    }
    return false;
  }

  /**
   * Returns true if the node is a Document.
   * @param obj the node to consider.
   * @return true if the node is a Document, else false.
   */
  public boolean isDocument(Object obj) {
    return (obj instanceof Source);
  }

  /**
   * Returns true if the node is an Element.
   * @param obj the node to consider.
   * @return true if the node is an Element, else false.
   */
  public boolean isElement(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return isElementOfType(element, StartTagType.NORMAL);
    }
    return false;
  }

  /**
   * Returns true if the node is a Namespace. Since Jericho HTML
   * does not work with Namespaces, this method always returns false.
   * @parm obj the node to consider.
   * @return always false.
   */
  public boolean isNamespace(Object obj) {
    return false;
  }

  /**
   * Returns true if the node is a Processing Instruction.
   * @param the node to consider.
   * @return true if the node is a PI, else false.
   */
  public boolean isProcessingInstruction(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return isElementOfType(element,
        StartTagType.XML_PROCESSING_INSTRUCTION) || 
        isElementOfType(element, StartTagType.XML_DECLARATION);
    }
    return false;
  }

  /**
   * Returns true if the node is Text.
   * @param obj the node to consider.
   * @return true if the node is text, else false.
   */
  public boolean isText(Object obj) {
    if (obj instanceof CharacterReference || obj instanceof String) {
      return true;
    }
    return false;
  }

  // various abstract getXXX method implementations. These are specified
  // in the Navigator interface but not defined in the abstract class,
  // so they must be implemented in the implementation specific subclass.
  
  /**
   * Return the name of the attribute.
   * @param obj the Attribute object.
   * @return the attribute name.
   */
  public String getAttributeName(Object obj) {
    if (obj instanceof Attribute) {
      Attribute attr = (Attribute) obj;
      return attr.getName();
    } else {
      return "";
    }
  }

  /**
   * Return the name of the attribute's namespace URI. Since there
   * are no Namespaces in Jericho, this returns an empty string.
   * @param obj the Attribute object.
   * @return an empty string.
   */
  public String getAttributeNamespaceUri(Object obj) {
    return "";
  }

  /**
   * Return the attribute's QName. Since there are no Namespaces in
   * Jericho, this is the same as returning the Attribute name.
   * @param obj the Attribute object.
   * @return the attribute name.
   */
  public String getAttributeQName(Object obj) {
    return getAttributeName(obj);
  }

  /**
   * Return the value of the attribute.
   * @param obj the attribute.
   * @return the attribute value as a string.
   */
  public String getAttributeStringValue(Object obj) {
    if (obj instanceof Attribute) {
      Attribute attr = (Attribute) obj;
      return attr.getValue();
    } else {
      return "";
    }
  }

  /**
   * Returns the comment as a string.
   * @param obj the comment element.
   * @return the comment's string value.
   */
  public String getCommentStringValue(Object obj) {
    if (isComment(obj)) {
      Element element = (Element) obj;
      return element.getContent().getTextExtractor().toString();
    } else {
      return "";
    }
  }

  /**
   * Returns the name of the Element.
   * @param obj the Element.
   * @return the name of the element.
   */
  public String getElementName(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return element.getName();
    } else {
      return "";
    }
  }

  /**
   * Returns the namespace URI for the Element. Since Namespaces are
   * not supported in Jericho, this returns an empty string.
   * @param obj the Element.
   * @return an empty string.
   */
  public String getElementNamespaceUri(Object obj) {
    return "";
  }

  /**
   * Returns the Element's QName. Since Namespaces are not supported
   * in Jericho, this is the same as getElementName().
   * @param obj the Element.
   * @return the Element name.
   */
  public String getElementQName(Object obj) {
    return getElementName(obj);
  }

  /**
   * Returns the text content of the Element.
   * @param obj the Element.
   * @return the text content or String value of the Element.
   */
  public String getElementStringValue(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return element.getContent().getTextExtractor().toString();
    } else if (obj instanceof String) {
      return ((String) obj);
    } else {
      return String.valueOf(obj);
    }
  }

  /**
   * Get the Namespace prefix for the Document. Always returns an
   * empty string since Jericho does not support Namespaces.
   * @param obj the Document.
   * @return an empty string.
   */
  public String getNamespacePrefix(Object obj) {
    return "";
  }

  /**
   * Returns the namespace string value for the Element. Same as
   * returning the element's string value.
   * @param obj the Element.
   * @return the name of the element.
   */
  public String getNamespaceStringValue(Object obj) {
    return getElementStringValue(obj);
  }

  /**
   * Return the text string value for the Element. Same as returning
   * the element's string value.
   * @param obj the Element.
   * @return the text content of the Element.
   */
  public String getTextStringValue(Object obj) {
    return getElementStringValue(obj);
  }

  // various overrides of incorrect or inefficient default behavior in 
  // parent class. Default behavior is usually returning null or throwing 
  // an UnsupportedOperationException, so we override to provide correct
  // behavior. 

  /**
   * Returns a Document object given a URL. We return a Source object,
   * and automatically do a fullSequentialParse() for performance.
   * @param url the URL for the document.
   * @return the Source object.
   */
  public Object getDocument(String url) {
    try {
      URLConnection conn = new URL(url).openConnection();
      Source source = new Source(conn.getInputStream());
      source.fullSequentialParse();
      return source;
    } catch (MalformedURLException e) {
      log.error("Malformed URL: " + url, e);
      return null;
    } catch (IOException e) {
      log.error("IO Exception for URL: " + url, e);
      return null;
    }
  }

  /**
   * Returns the root element for the Document context node.
   * @param contextNode the Source object.
   * @return the root Element of the document (html).
   */
  public Object getDocumentNode(Object contextNode) {
    if (contextNode instanceof Source) {
      Source source = (Source) contextNode;
      return ((Segment) source).findAllElements("html").get(0);
    } else {
      return contextNode;
    }
  }

  /**
   * Returns the parent node for the contextNode. Default behavior is
   * inefficient. Since a Jericho Element has a pointer to its parent
   * node, we can use that here.
   * @param contextNode the context node.
   * @return the parent node.
   */
  public Object getParentNode(Object contextNode) {
    if (isElement(contextNode)) {
      return ((Element) contextNode).getParentElement();
    } else {
      return null;
    }
  }

  /**
   * Returns an element by id. Default behavior always returns null,
   * but Jericho provides methods to return this object, so we override
   * it.
   * @param contextNode the context node.
   * @param elementId the name of the element being searched for.
   * @return an Element object.
   */
  public Object getElementById(Object contextNode, String elementId) {
    if (isElement(contextNode)) {
      Iterator eit = ((Element) contextNode).findAllElements().iterator();
      List elementsById = new ArrayList();
      while (eit.hasNext()) {
        Element element = (Element) eit.next();
        if (element.getAttributeValue("id") == null ||
            (! element.getAttributeValue("id").equals(elementId))) {
          continue;
        }
        elementsById.add(element);
      }
      return elementsById;
    } else {
      return Collections.emptyList();
    }
  }
  
  // iteration methods

  /**
   * Returns an iterator for the child Elements of the contextNode
   * Element.
   * @param contextNode the context node Element.
   * @return an Iterator over the child objects of this Element.
   */
  public Iterator getChildAxisIterator(Object contextNode) {
    if (isElement(contextNode)) {
      Element element = (Element) contextNode;
      List children = new ArrayList();
      children.addAll(element.getChildElements());
      children.add(element.getTextExtractor().toString());
      return children.iterator();
    } else {
      return JaxenConstants.EMPTY_ITERATOR;
    }
  }

  /**
   * Returns an iterator over the named child elements of this context
   * node. Comes from NamedAccessNavigator.
   * @param contextNode the context node Element.
   * @param localName the name of the element.
   * @param namespacePrefix not used.
   * @param namespaceURI not used.
   * @return an iterator over the named child elements.
   */
  public Iterator getChildAxisIterator(Object contextNode, String localName,
      String namespacePrefix, String namespaceURI)
      throws UnsupportedAxisException {
    if (contextNode instanceof Element) {
      List children = ((Element) contextNode).findAllElements(localName);
      return children.iterator();
    } else {
      return JaxenConstants.EMPTY_ITERATOR;
    }
  }

  /**
   * Jericho does not support Namespaces, so returns an empty iterator.
   * @param contextNode the context node Element.
   * @return an empty iterator.
   */
  public Iterator getNamespaceAxisIterator(Object contextNode) {
    return JaxenConstants.EMPTY_ITERATOR;
  }

  /**
   * Returns an iterator over the parent elements of this Context node.
   * @param contextNode the context node Element.
   * @return an iterator over the parent elements of this Element.
   */
  public Iterator getParentAxisIterator(Object contextNode) {
    if (isDocument(contextNode)) {
      return JaxenConstants.EMPTY_ITERATOR;
    }
    Element parent = null;
    if (isElement(contextNode)) {
      Element element = (Element) contextNode;
      parent = element.getParentElement();
    }
    if (parent == null) {
      return JaxenConstants.EMPTY_ITERATOR;
    } else {
      return new SingleObjectIterator(parent);
    }
  }
  
  /**
   * Returns an iterator over the attribute axis of this context node.
   * @param contextNode the context node Element.
   * @return an iterator over the Element's attributes.
   */
  public Iterator getAttributeAxisIterator(Object contextNode) {
    if (isElement(contextNode)) {
      Element element = (Element) contextNode;
      Attributes attrs = element.getAttributes();
      Iterator ait = attrs.iterator();
      List attrlist = new ArrayList();
      while (ait.hasNext()) {
        attrlist.add((Attribute) ait.next());
      }
      return attrlist.iterator();
    }
    return JaxenConstants.EMPTY_ITERATOR;
  }

  /**
   * Returns an iterator over the named attributes for this context node.
   * Comes from NamedAccessNavigator.
   * @param contextNode the context node Element.
   * @param localName the name of the attribute.
   * @param namespacePrefix not used.
   * @param namespaceURI not used.
   * @return an iterator over the named attributes of this Element.
   */
  public Iterator getAttributeAxisIterator(Object contextNode,
      String localName, String namespacePrefix, String namespaceURI)
      throws UnsupportedAxisException {
    List namedAttrs = new ArrayList();
    if (contextNode instanceof Element) {
      Attributes attrs = ((Element) contextNode).getAttributes();
      Iterator ait = attrs.iterator();
      while (ait.hasNext()) {
        Attribute attr = (Attribute) ait.next();
        if (localName.equals(attr.getName())) {
          namedAttrs.add(attr);
        }
      }
      return namedAttrs.iterator();
    } else {
      return JaxenConstants.EMPTY_ITERATOR;
    }
  }

  /**
   * Makes sure that we return the correct XPath implementation when
   * called.
   * @param xpath the XPath expression.
   * @return the JerichoXPath object wrapping the expression.
   */
  public XPath parseXPath(String xpath) throws SAXPathException {
    return new JerichoXPath(xpath);
  }

  /**
   * Convenience method to return a correct element type.
   * @param element the Element object.
   * @param tagType the correct tag type. 
   * @return true or false.
   */
  private boolean isElementOfType(Element element, StartTagType tagType) {
    if (element == null) {
      return false;
    }
    return element.getStartTag().getTagType().equals(tagType);
  }
}

Unit test

The Unit test is a JUnit 3.x test, and just takes an HTML document and hits it with the XPath expressions I am likely to use. Its nothing fancy, just a runner for the evaluate() method for various XPath expressions. It prints its output on the console.

package org.jaxen.test;

import java.util.List;

import junit.framework.TestCase;

import org.jaxen.Navigator;
import org.jaxen.XPath;
import org.jaxen.jericho.DocumentNavigator;
import org.jaxen.jericho.JerichoXPath;

import au.id.jericho.lib.html.Element;
import au.id.jericho.lib.html.Source;

public class JerichoNavigatorTest extends TestCase {

  private String testUrl = "http://path.to.your/document.html";
  private String[] xpaths = new String[] {
    "/html/body",
    "//body",
    "/html/body/../head",
    "/html/head/title/text()",
    "//div[@class='articlecontent']",
    "//div[@class]"
  };

  public void testParsingVisitor() throws Exception {
    Navigator navigator = DocumentNavigator.getInstance();
    Source doc = (Source) navigator.getDocument(testUrl);
    for (int i = 0; i < xpaths.length; i++) {
      String xpath = xpaths[i];
      System.out.println("*** Evaluating: " + xpath);
      XPath expr = new JerichoXPath(xpath, navigator);
      Object result = expr.evaluate(doc);
      if (result instanceof Element) {
        System.out.println("Element: " + ((Element) result).getName());
      } else if (result instanceof List) {
        System.out.println("List: size=" + ((List) result).size());
        List elements = (List) result;
        for (int j = 0; j < elements.size(); j++) {
          Element element = (Element) elements.get(j);
          System.out.println("Element: " + ((Element) element).getName());
        }
      } else if (result instanceof String) {
        System.out.println("String: " + ((String) result));
      } else if (result instanceof Number) {
        System.out.println("Number: " + ((Number) result));
      } else if (result instanceof Boolean) {
        System.out.println("Boolean: " + ((Boolean) result));
      } else {
        System.out.println("Unknown: " + result == null ? 
          "NULL" : result.getClass().getName());
      }
    }
  }
}

Other XPath methods

I haven't gotten around to testing the other public methods of the XPath object, but I suspect that Jaxen will do the right thing in all these cases, because all of the methods can be written as wrappers over the evaluate() method. If I see problems with the behavior, the plan is to override these methods in the JerichoXPath class, as shown below:

  public String valueOf(Object context) throws JaxenException {
    return stringValueOf(context);
  }

  public boolean booleanValueOf(Object context) throws JaxenException {
    String result = stringValueOf(context);
    return Boolean.valueOf(result);
  }

  public Number numberValueOf(Object context) throws JaxenException {
    String result = stringValueOf(context);
    if (NumberUtils.isNumber(result)) {
      return NumberUtils.createNumber(result);
    } else {
      throw new JaxenException("Value of " + xpathExpr + " is not numeric");
    }
  }

  public String stringValueOf(Object context) throws JaxenException {
    Object result = evaluate(context);
    if (result instanceof String) {
      return (String) result;
    } else {
      throw new JaxenException("Cannot return string value of " + xpathExpr);
    }
  }

  public Object selectSingleNode(Object context) throws JaxenException {
    List nodes = selectNodes(context);
    if (nodes.size() > 0) {
      return nodes.get(0);
    } else {
      return null;
    }
  }

  public List selectNodes(Object context) throws JaxenException {
    Object result = evaluate(context);
    if (result instanceof List) {
      return ((List) result);
    } else {
      return Collections.emptyList();
    }
  }

Alternative Approach

As outlined in the second paragraph of the guidelines, another approach is to build a Jericho implementation of the XPathHandler interface, and parse the XPath expression using the default XPathReader. I tried this in desperation midway through the above exercise, and it worked well for the simple element traversal events. However, I decided against going this route because of the complexity of the event handling code for handling predicates. However, if you are looking to build something that does not (or cannot for some reason) use the Jaxon Navigator approach, then this may be worth looking at.

XPath/Jaxen articles

There is not much information available on Jaxen and SAXPath Event API, but I found these two links quite useful.

Introduction to SAXPath and Jaxen by Bob McWhirter
Chapter 16 of Processing XML with Java by Elliote Rusty Harold

Update 2009-04-26: I contributed the code above back to the Jaxen project. However, they are planning on moving the examples out of the project since it imposes unnecessary JAR file dependencies on clients, so the code may become part of a jaxen-contrib project later. If you are looking to incorporate the code into your project, the only extra dependency are jericho-html. There is some commons-logging dependencies, but they can be removed or replaced with your favorite logger in the source code.

Friday, April 10, 2009

Maven2 plugin to generate shell scripts

Here is a silly, but potentially useful little Maven2 plugin that I wrote recently. It uses the application module's POM to build a bash script that runs a Java "executable" program (ie a class with a main() method).

For a given module, a bash script to run one class versus another is almost identical - the only difference is in the class name being passed to java. The rest of it is all boilerplate, and the largest portion is the CLASSPATH declaration. Since Java modules typically have tons of dependencies, it can be quite tedious to build the bash script by hand - in fact, it would probably qualify as my least favorite (coding) activity.

Why do it then, you ask? Well, I typically run small datasets (upto a couple 1000) through my program by writing a JUnit test and calling it using "mvn test", but I find that the JVM runs out of memory for large data sets, especially with a lot of logging. I suspect that this is because Maven2 buffers the logs in memory to write the results of the test in an XML file, but I could be wrong. The other reason is that if your program is going to production, then a shell script to run the code is one of the deliverables.

Plugin setup

The plugin is built inside an existing plugin project that I described in a previous post. For this plugin, I added in dependencies to JDOM and Velocity. The relevant snippet from my pom.xml is shown below:

<project>
  ...
  <dependencies>
  ...
    <dependency>
  ...
    <dependency>
      <groupId>velocity</groupId>
      <artifactId>velocity</artifactId>
      <version>1.4</version>
      <scope>compile</scope>
    </dependency>
    <dependency>
      <groupId>jdom</groupId>
      <artifactId>jdom</artifactId>
      <version>1.0</version>
      <scope>compile</scope>
    </dependency>
  </dependencies>
  ...
</project>

Plugin code

Here is the code for the plugin. It is set to be called in the "deploy" phase, so it does not interfere with the normal development (clean compile test-compile test) life cycle.

// Source: src/main/java/com/mycompany/plugin/BashScriptMojo.java
package com.mycompany.plugin;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.util.Date;
import java.util.List;
import java.util.Properties;

import org.apache.maven.plugin.AbstractMojo;
import org.apache.maven.plugin.MojoExecutionException;
import org.apache.velocity.VelocityContext;
import org.apache.velocity.app.Velocity;
import org.codehaus.plexus.util.StringUtils;
import org.jdom.Document;
import org.jdom.Element;
import org.jdom.Namespace;
import org.jdom.input.SAXBuilder;

/**
 * Builds a script to execute a given class in a project.
 * @goal script
 * @phase deploy
 */
public class BashScriptMojo extends AbstractMojo {
  
  /**
   * Location of the file.
   * @parameter expression="${project.build.directory}"
   * @required
   * @readonly
   */
  private File outputDir;
  
  /**
   * The project directory where the pom.xml file is located.
   * @parameter expression="${basedir}"
   * @required
   * @readonly
   */
  private File projectDir;
  
  /**
   * Full class name to build execution script for.
   * @parameter expression="${className}"
   * @required
   */
  private String className;

  @SuppressWarnings("unchecked")
  public void execute() throws MojoExecutionException {
    try {
      // parse the pom.xml to find the list of dependency and expand
      // them out to the correct path in the M2 repository to build
      // the classpath
      SAXBuilder parser = new SAXBuilder();
      Document doc = parser.build(new File(projectDir, "pom.xml"));
      Element root = doc.getRootElement();
      Namespace defaultNamespace = root.getNamespace();
      Element dependenciesElement = 
        root.getChild("dependencies", defaultNamespace);
      StringBuilder buf = new StringBuilder();
      List<Element> dependencyElements =
        dependenciesElement.getChildren("dependency", defaultNamespace);
      for (Element dependencyElement : dependencyElements) {
        String groupId = dependencyElement.getChildTextTrim(
          "groupId", defaultNamespace);
        String artifactId = dependencyElement.getChildTextTrim(
          "artifactId", defaultNamespace);
        String version = dependencyElement.getChildTextTrim(
          "version", defaultNamespace);
        String path = StringUtils.join(new String[] {
          "$M2_REPO",
          StringUtils.replace(groupId, ".", File.separator),
          artifactId,
          version,
          StringUtils.join(new String[] {artifactId, version}, "-") + ".jar"
        }, File.separator);
        buf.append(path).append(File.pathSeparator).append("\\\n");
      }
      // finally append the target/classes dir
      buf.append(projectDir.getAbsolutePath()).append("/target/classes");
      // calculate the class name only for script file and log file
      String shortClassName = 
        className.substring(className.lastIndexOf('.') + 1);
      // stick them into the context
      VelocityContext context = new VelocityContext();
      context.put("__classpath__", buf.toString());
      context.put("__date__", new Date());
      context.put("__classname__", className);
      context.put("__logfile__", shortClassName + ".log");
      // we want to load the .vm file from the classpath, so we configure
      // the ClassPathResourceLoader to get the vm file.
      Properties props = new Properties();
      props.setProperty("resource.loader", "classpath");
      props.setProperty(
        "classpath.resource.loader.class", 
              "org.apache.velocity.runtime.resource.loader.ClasspathResourceLoader");
      Velocity.init(props);
      File scriptFile = new File(outputDir, "run" + shortClassName + ".sh");
      BufferedWriter writer = new BufferedWriter(new FileWriter(scriptFile));
      Velocity.mergeTemplate("bash_script.vm", "UTF-8", context, writer);
      writer.flush();
      writer.close();
      getLog().info("Script " + scriptFile.getName() + 
        " written to " + scriptFile.getPath());
    } catch (Exception e) {
      getLog().error("Error executing BashScriptMojo", e);
      e.printStackTrace();
      throw new MojoExecutionException(e.getMessage(), e);
    }
  }
}

Finally, here is the Velocity template file. As you can see above, I had to use Velocity's ClassPathResourceLoader to load it from the classpath (src/main/resources) of the plugin project.

#!/bin/bash
# Generated by mvn mycompany:script on ${__date__}
M2_REPO=$HOME/.m2/repository
CLASSPATH=${__classpath__}
java -cp $CLASSPATH -Xmx2048m ${__classname__} $* 2>&1 | tee ${__logfile__}

Plugin configuration

To install this into your local repository, run "mvn install:install". On the target module, where you actually want to use this plugin, you need to configure it in the module's POM as shown below:

<project>
...
  <build>
  ...
    <plugins>
      ...
      <plugin>
        <groupId>com.mycompany.plugin</groupId>
        <artifactId>mycompany-maven-plugin</artifactId>
        <version>1.0-SNAPSHOT</version>
        <executions>
          <execution>
            <phase>deploy</phase>
            <goals>
              <goal>script</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

To call this plugin from the target application, run the following command:

prompt$ mvn -o mycompany:script -DclassName=com.mycompany.foo.bar.Baz
[INFO] Scanning for projects...
[INFO] Searching repository for plugin with prefix: 'mycompany'.
[INFO] -------------------------------------------------------------------
[INFO] Building MyCompany Plugin Module
[INFO]    task-segment: [mycompany:script]
[INFO] -------------------------------------------------------------------
[INFO] [mycompany:script]
[INFO] Script runBaz.sh written to /home/.../target/runBaz.sh
[INFO] -------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] -------------------------------------------------------------------
[INFO] Total time: 2 seconds
[INFO] Finished at: Fri Apr 10 16:24:33 GMT-08:00 2009
[INFO] Final Memory: 5M/9M
[INFO] -------------------------------------------------------------------

Which results in a shell script that looks something like this (edited for brevity):

#!/bin/bash
# Generated by mvn mycompany:script on Fri Apr 10 16:24:33 GMT-08:00 2009
M2_REPO=$HOME/.m2/repository
CLASSPATH=\
$M2_REPO/commons-cli/commons-cli/1.0/commons-cli-1.0.jar:\
$M2_REPO/commons-codec/commons-codec/1.3/commons-codec-1.3.jar:\
$M2_REPO/commons-io/commons-io/1.2/commons-io-1.2.jar:\
...\
/home/.../target/classes
java -cp $CLASSPATH -Xmx2048m com.mycompany.foo.bar.Baz $* \
  2>&1 | tee Baz.log

Conclusion

Of course, the bash scripts that we send to production are not quite this simple - there is additional validation to make sure another process is not already running, and hooks to email completion status, etc, but they too are boilerplate. I have intentionally kept the script simple, but it is easy to update the template file to produce something that is more robust and production-ready.

The plugin does not handle transitive dependencies - which kind of defeats the purpose of using Maven2, I know... But since we have parallel Maven2 and Ant descriptors (some of our developers haven't gotten around to becoming comfortable with Maven2 yet), this is not an issue in my case, since we explicitly list all dependencies in both the Maven2 POM and Ant's build.xml files. However, I hope to update the plugin at some point to include Maven2's transitive dependency detection. If you have already done the research on how to do this, I would appreciate pointers.

On a completely unrelated note...

On a completely unrelated note, I found yesterday that Stacey's, my favorite bookstore (for around the last 7 years) has gone out of business. They were selling off the book cases when I arrived yesterday. Up until 3 years ago, my last job was about a couple of blocks from the store, so I was a pretty frequent visitor, and would average about one computer book a month. I have since moved to another location that is a good half hour walk (or a 10 minute bus ride), so I haven't been going as often. In spite of the higher prices compared to Amazon's discounted prices, I liked going to the bookshop and buying from there, since (a) I did not have to wait for the book to be shipped and (b) I could compare different books before making a purchase. Not sure about you, but Amazon's look-inside feature just doesn't compare.

There is a Borders across the street from where I work, but most of the time they don't have what I want, and even if they do, the books are so disorganized that its like finding a needle in a haystack. The Barnes and Noble where I live is more organized, but their focus (probably rightly so, given the demographics) is on children's books.

If anybody knows of a good bookshop that sells computer books in or around the San Francisco Market Street area, would appreciate hearing from you. Otherwise, I guess I will just have to get used to buying books online.

Salmon Run

Monday, April 20, 2009

XPath over HTML using Jericho and Jaxen

JerichoXPath

DocumentNavigator

Unit test

Other XPath methods

Alternative Approach

XPath/Jaxen articles

Friday, April 10, 2009

Maven2 plugin to generate shell scripts

Plugin setup

Plugin code

Plugin configuration

Conclusion

On a completely unrelated note...

Posts

Labels

Blogs I Read

About me

My Nerd Rating

Visitor Map

Contact Me