Salmon Run: XPath over HTML using Jericho and Jaxen

I've been using Jericho to parse HTML for a while now. I mainly use it to extract pieces of text from specific locations in the HTML. To do this, I use the Jericho API - I have factored out the boilerplate code associated with XML/HTML parsing into utility classes, so the application code to extract an element's contents would be 1 or 2 lines tops. However, recently I needed to move the extraction logic out of the code and into configuration.

Using XPath in these situations is almost a no-brainer - given an XML document and tasked with extracting text for known elements, that is the first tool I would think about. Unfortunately, Jericho does not have XPath support, so being the foolish/foolhardy type, I set about trying to build one. This post describes a Jaxen based XPath adapter for Jericho.

The Jaxen project provides a universal XPath engine, capable of evaluating XPath expressions across multiple object models - currently supported in the GA (1.1.1 at the time of writing this) release are dom4j, JDOM, w3c DOM, Javabeans and XOM. The Jaxen FAQ has this to say about extending Jaxen for other object models.

The only thing required is an implementation of the interface org.jaxen.Navigator. Not all of the interface is required, and a default implementation, in the form of org.jaxen.DefaultNavigator is also provided. Since many of the XPath axes can be defined in terms of each other (for example, the ancestor axis is merely a the parent recursively applied), only a few low-level axis iterators are required to initially get started. Of course, you may implement them directly, instead of relying upon jaxen's composition ability.

In my opinion, the text above falls into the technically accurate but not very useful as a guideline category. Granted, there are multiple extension examples in the source tarball (for dom4j, JDOM, etc) to use as examples (which I did), but what would have helped immensely is a short howto style tutorial, that explains which methods of DefaultNavigator need to be customized and why.

I was finally able to get everything to work by running my unit test through a debugger, stepping through both the Jaxen and my extension code. For example, I found that I should implementing the methods of NamedAccessNavigator, although that is not a stated requirement. There are also places where the methods of DefaultNavigator return null or throw UnsupportedOperationException and are therefore expected to be extended by subclasses, but without being explicitly marked abstract.

In any case, I now know a little more than I knew about Jaxen internals, and I must say the extension model is very well thought out, and my gripe with the implementation may just be due to the fact that the authors haven't gotten around to cleaning it up so its more usable. Hopefully, if you are in a similar situation as I was recently, then the code in this blog post may be helpful - in the form of yet another example of extension.

My approach is based on the first paragraph of the guideline, ie, building a custom Navigator implementation using the JDOM implementation as a template. Basically, I have a JerichoXPath class which exposes two constructors and a custom getContext() method, and a DocumentNavigator implementation that contains calls to the Jericho API to do different things. The Jaxen XPath parser calls into these DocumentNavigator methods at specific points in its life cycle.

JerichoXPath

package org.jaxen.jericho;

import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jaxen.BaseXPath;
import org.jaxen.Context;
import org.jaxen.JaxenException;
import org.jaxen.Navigator;
import org.jaxen.util.SingletonList;

import au.id.jericho.lib.html.Element;
import au.id.jericho.lib.html.Source;

public class JerichoXPath extends BaseXPath {

  private static final long serialVersionUID = -6969112785840871593L;

  private final Log log = LogFactory.getLog(getClass());
  
  public JerichoXPath(String xpathExpr, Navigator navigator) 
      throws JaxenException {
    super(xpathExpr, navigator);
  }
  
  public JerichoXPath(String xpathExpr) throws JaxenException {
    super(xpathExpr, DocumentNavigator.getInstance());
  }

  /**
   * Jericho specific method to get the context associated with a node.
   * @param node the current node being visited.
   * @return the Context associated with the node.
   */
  protected Context getContext(Object node) {
    if (node instanceof Context) {
      return (Context) node;
    }
    Context fullContext = new Context(getContextSupport());
    if (node instanceof Source) {
      Element rootNode = 
        (Element) getNavigator().getDocumentNode((Source) node);
      fullContext.setNodeSet(new SingletonList(rootNode));
    } else if (node instanceof List) {
      fullContext.setNodeSet((List) node);
    } else {
      List list = new SingletonList(node);
      fullContext.setNodeSet(list);
    }
    return fullContext;
  }
}

DocumentNavigator

package org.jaxen.jericho;

import java.io.IOException;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jaxen.DefaultNavigator;
import org.jaxen.JaxenConstants;
import org.jaxen.NamedAccessNavigator;
import org.jaxen.Navigator;
import org.jaxen.UnsupportedAxisException;
import org.jaxen.XPath;
import org.jaxen.saxpath.SAXPathException;
import org.jaxen.util.SingleObjectIterator;

import au.id.jericho.lib.html.Attribute;
import au.id.jericho.lib.html.Attributes;
import au.id.jericho.lib.html.CharacterReference;
import au.id.jericho.lib.html.Element;
import au.id.jericho.lib.html.Segment;
import au.id.jericho.lib.html.Source;
import au.id.jericho.lib.html.StartTagType;

public class DocumentNavigator extends DefaultNavigator 
    implements NamedAccessNavigator {

  private static final long serialVersionUID = 8640276699026512314L;

  private final Log log = LogFactory.getLog(getClass());
  
  // DocumentNavigator needs to expose a singleton. Strategy adapted from: 
  // http://www.ibm.com/developerworks/java/library/j-dcl.html

  private DocumentNavigator() {
    super();
  }
  private static DocumentNavigator INSTANCE = new DocumentNavigator(); 
  
  /**
   * Return a singleton instance of the DocumentNavigator object.
   * @return a Navigator.
   */
  public static Navigator getInstance() {
    return INSTANCE;
  }
  
  // various abstract isXXX method implementations. These are specified
  // in the Navigator interface but not defined in the abstract class,
  // so they must be implemented in the implementation specific subclass.

  /**
   * Returns true if the node is an Attribute.
   * @param obj the node to consider.
   * @return true if node is an attribute, else false.
   */
  public boolean isAttribute(Object obj) {
    return obj instanceof Attribute;
  }

  /**
   * Returns true if the node is a Comment.
   * @param obj the node to consider.
   * @return true if the node is a comment, else false.
   */
  public boolean isComment(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return isElementOfType(element, StartTagType.COMMENT);
    }
    return false;
  }

  /**
   * Returns true if the node is a Document.
   * @param obj the node to consider.
   * @return true if the node is a Document, else false.
   */
  public boolean isDocument(Object obj) {
    return (obj instanceof Source);
  }

  /**
   * Returns true if the node is an Element.
   * @param obj the node to consider.
   * @return true if the node is an Element, else false.
   */
  public boolean isElement(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return isElementOfType(element, StartTagType.NORMAL);
    }
    return false;
  }

  /**
   * Returns true if the node is a Namespace. Since Jericho HTML
   * does not work with Namespaces, this method always returns false.
   * @parm obj the node to consider.
   * @return always false.
   */
  public boolean isNamespace(Object obj) {
    return false;
  }

  /**
   * Returns true if the node is a Processing Instruction.
   * @param the node to consider.
   * @return true if the node is a PI, else false.
   */
  public boolean isProcessingInstruction(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return isElementOfType(element,
        StartTagType.XML_PROCESSING_INSTRUCTION) || 
        isElementOfType(element, StartTagType.XML_DECLARATION);
    }
    return false;
  }

  /**
   * Returns true if the node is Text.
   * @param obj the node to consider.
   * @return true if the node is text, else false.
   */
  public boolean isText(Object obj) {
    if (obj instanceof CharacterReference || obj instanceof String) {
      return true;
    }
    return false;
  }

  // various abstract getXXX method implementations. These are specified
  // in the Navigator interface but not defined in the abstract class,
  // so they must be implemented in the implementation specific subclass.
  
  /**
   * Return the name of the attribute.
   * @param obj the Attribute object.
   * @return the attribute name.
   */
  public String getAttributeName(Object obj) {
    if (obj instanceof Attribute) {
      Attribute attr = (Attribute) obj;
      return attr.getName();
    } else {
      return "";
    }
  }

  /**
   * Return the name of the attribute's namespace URI. Since there
   * are no Namespaces in Jericho, this returns an empty string.
   * @param obj the Attribute object.
   * @return an empty string.
   */
  public String getAttributeNamespaceUri(Object obj) {
    return "";
  }

  /**
   * Return the attribute's QName. Since there are no Namespaces in
   * Jericho, this is the same as returning the Attribute name.
   * @param obj the Attribute object.
   * @return the attribute name.
   */
  public String getAttributeQName(Object obj) {
    return getAttributeName(obj);
  }

  /**
   * Return the value of the attribute.
   * @param obj the attribute.
   * @return the attribute value as a string.
   */
  public String getAttributeStringValue(Object obj) {
    if (obj instanceof Attribute) {
      Attribute attr = (Attribute) obj;
      return attr.getValue();
    } else {
      return "";
    }
  }

  /**
   * Returns the comment as a string.
   * @param obj the comment element.
   * @return the comment's string value.
   */
  public String getCommentStringValue(Object obj) {
    if (isComment(obj)) {
      Element element = (Element) obj;
      return element.getContent().getTextExtractor().toString();
    } else {
      return "";
    }
  }

  /**
   * Returns the name of the Element.
   * @param obj the Element.
   * @return the name of the element.
   */
  public String getElementName(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return element.getName();
    } else {
      return "";
    }
  }

  /**
   * Returns the namespace URI for the Element. Since Namespaces are
   * not supported in Jericho, this returns an empty string.
   * @param obj the Element.
   * @return an empty string.
   */
  public String getElementNamespaceUri(Object obj) {
    return "";
  }

  /**
   * Returns the Element's QName. Since Namespaces are not supported
   * in Jericho, this is the same as getElementName().
   * @param obj the Element.
   * @return the Element name.
   */
  public String getElementQName(Object obj) {
    return getElementName(obj);
  }

  /**
   * Returns the text content of the Element.
   * @param obj the Element.
   * @return the text content or String value of the Element.
   */
  public String getElementStringValue(Object obj) {
    if (obj instanceof Element) {
      Element element = (Element) obj;
      return element.getContent().getTextExtractor().toString();
    } else if (obj instanceof String) {
      return ((String) obj);
    } else {
      return String.valueOf(obj);
    }
  }

  /**
   * Get the Namespace prefix for the Document. Always returns an
   * empty string since Jericho does not support Namespaces.
   * @param obj the Document.
   * @return an empty string.
   */
  public String getNamespacePrefix(Object obj) {
    return "";
  }

  /**
   * Returns the namespace string value for the Element. Same as
   * returning the element's string value.
   * @param obj the Element.
   * @return the name of the element.
   */
  public String getNamespaceStringValue(Object obj) {
    return getElementStringValue(obj);
  }

  /**
   * Return the text string value for the Element. Same as returning
   * the element's string value.
   * @param obj the Element.
   * @return the text content of the Element.
   */
  public String getTextStringValue(Object obj) {
    return getElementStringValue(obj);
  }

  // various overrides of incorrect or inefficient default behavior in 
  // parent class. Default behavior is usually returning null or throwing 
  // an UnsupportedOperationException, so we override to provide correct
  // behavior. 

  /**
   * Returns a Document object given a URL. We return a Source object,
   * and automatically do a fullSequentialParse() for performance.
   * @param url the URL for the document.
   * @return the Source object.
   */
  public Object getDocument(String url) {
    try {
      URLConnection conn = new URL(url).openConnection();
      Source source = new Source(conn.getInputStream());
      source.fullSequentialParse();
      return source;
    } catch (MalformedURLException e) {
      log.error("Malformed URL: " + url, e);
      return null;
    } catch (IOException e) {
      log.error("IO Exception for URL: " + url, e);
      return null;
    }
  }

  /**
   * Returns the root element for the Document context node.
   * @param contextNode the Source object.
   * @return the root Element of the document (html).
   */
  public Object getDocumentNode(Object contextNode) {
    if (contextNode instanceof Source) {
      Source source = (Source) contextNode;
      return ((Segment) source).findAllElements("html").get(0);
    } else {
      return contextNode;
    }
  }

  /**
   * Returns the parent node for the contextNode. Default behavior is
   * inefficient. Since a Jericho Element has a pointer to its parent
   * node, we can use that here.
   * @param contextNode the context node.
   * @return the parent node.
   */
  public Object getParentNode(Object contextNode) {
    if (isElement(contextNode)) {
      return ((Element) contextNode).getParentElement();
    } else {
      return null;
    }
  }

  /**
   * Returns an element by id. Default behavior always returns null,
   * but Jericho provides methods to return this object, so we override
   * it.
   * @param contextNode the context node.
   * @param elementId the name of the element being searched for.
   * @return an Element object.
   */
  public Object getElementById(Object contextNode, String elementId) {
    if (isElement(contextNode)) {
      Iterator eit = ((Element) contextNode).findAllElements().iterator();
      List elementsById = new ArrayList();
      while (eit.hasNext()) {
        Element element = (Element) eit.next();
        if (element.getAttributeValue("id") == null ||
            (! element.getAttributeValue("id").equals(elementId))) {
          continue;
        }
        elementsById.add(element);
      }
      return elementsById;
    } else {
      return Collections.emptyList();
    }
  }
  
  // iteration methods

  /**
   * Returns an iterator for the child Elements of the contextNode
   * Element.
   * @param contextNode the context node Element.
   * @return an Iterator over the child objects of this Element.
   */
  public Iterator getChildAxisIterator(Object contextNode) {
    if (isElement(contextNode)) {
      Element element = (Element) contextNode;
      List children = new ArrayList();
      children.addAll(element.getChildElements());
      children.add(element.getTextExtractor().toString());
      return children.iterator();
    } else {
      return JaxenConstants.EMPTY_ITERATOR;
    }
  }

  /**
   * Returns an iterator over the named child elements of this context
   * node. Comes from NamedAccessNavigator.
   * @param contextNode the context node Element.
   * @param localName the name of the element.
   * @param namespacePrefix not used.
   * @param namespaceURI not used.
   * @return an iterator over the named child elements.
   */
  public Iterator getChildAxisIterator(Object contextNode, String localName,
      String namespacePrefix, String namespaceURI)
      throws UnsupportedAxisException {
    if (contextNode instanceof Element) {
      List children = ((Element) contextNode).findAllElements(localName);
      return children.iterator();
    } else {
      return JaxenConstants.EMPTY_ITERATOR;
    }
  }

  /**
   * Jericho does not support Namespaces, so returns an empty iterator.
   * @param contextNode the context node Element.
   * @return an empty iterator.
   */
  public Iterator getNamespaceAxisIterator(Object contextNode) {
    return JaxenConstants.EMPTY_ITERATOR;
  }

  /**
   * Returns an iterator over the parent elements of this Context node.
   * @param contextNode the context node Element.
   * @return an iterator over the parent elements of this Element.
   */
  public Iterator getParentAxisIterator(Object contextNode) {
    if (isDocument(contextNode)) {
      return JaxenConstants.EMPTY_ITERATOR;
    }
    Element parent = null;
    if (isElement(contextNode)) {
      Element element = (Element) contextNode;
      parent = element.getParentElement();
    }
    if (parent == null) {
      return JaxenConstants.EMPTY_ITERATOR;
    } else {
      return new SingleObjectIterator(parent);
    }
  }
  
  /**
   * Returns an iterator over the attribute axis of this context node.
   * @param contextNode the context node Element.
   * @return an iterator over the Element's attributes.
   */
  public Iterator getAttributeAxisIterator(Object contextNode) {
    if (isElement(contextNode)) {
      Element element = (Element) contextNode;
      Attributes attrs = element.getAttributes();
      Iterator ait = attrs.iterator();
      List attrlist = new ArrayList();
      while (ait.hasNext()) {
        attrlist.add((Attribute) ait.next());
      }
      return attrlist.iterator();
    }
    return JaxenConstants.EMPTY_ITERATOR;
  }

  /**
   * Returns an iterator over the named attributes for this context node.
   * Comes from NamedAccessNavigator.
   * @param contextNode the context node Element.
   * @param localName the name of the attribute.
   * @param namespacePrefix not used.
   * @param namespaceURI not used.
   * @return an iterator over the named attributes of this Element.
   */
  public Iterator getAttributeAxisIterator(Object contextNode,
      String localName, String namespacePrefix, String namespaceURI)
      throws UnsupportedAxisException {
    List namedAttrs = new ArrayList();
    if (contextNode instanceof Element) {
      Attributes attrs = ((Element) contextNode).getAttributes();
      Iterator ait = attrs.iterator();
      while (ait.hasNext()) {
        Attribute attr = (Attribute) ait.next();
        if (localName.equals(attr.getName())) {
          namedAttrs.add(attr);
        }
      }
      return namedAttrs.iterator();
    } else {
      return JaxenConstants.EMPTY_ITERATOR;
    }
  }

  /**
   * Makes sure that we return the correct XPath implementation when
   * called.
   * @param xpath the XPath expression.
   * @return the JerichoXPath object wrapping the expression.
   */
  public XPath parseXPath(String xpath) throws SAXPathException {
    return new JerichoXPath(xpath);
  }

  /**
   * Convenience method to return a correct element type.
   * @param element the Element object.
   * @param tagType the correct tag type. 
   * @return true or false.
   */
  private boolean isElementOfType(Element element, StartTagType tagType) {
    if (element == null) {
      return false;
    }
    return element.getStartTag().getTagType().equals(tagType);
  }
}

Unit test

The Unit test is a JUnit 3.x test, and just takes an HTML document and hits it with the XPath expressions I am likely to use. Its nothing fancy, just a runner for the evaluate() method for various XPath expressions. It prints its output on the console.

package org.jaxen.test;

import java.util.List;

import junit.framework.TestCase;

import org.jaxen.Navigator;
import org.jaxen.XPath;
import org.jaxen.jericho.DocumentNavigator;
import org.jaxen.jericho.JerichoXPath;

import au.id.jericho.lib.html.Element;
import au.id.jericho.lib.html.Source;

public class JerichoNavigatorTest extends TestCase {

  private String testUrl = "http://path.to.your/document.html";
  private String[] xpaths = new String[] {
    "/html/body",
    "//body",
    "/html/body/../head",
    "/html/head/title/text()",
    "//div[@class='articlecontent']",
    "//div[@class]"
  };

  public void testParsingVisitor() throws Exception {
    Navigator navigator = DocumentNavigator.getInstance();
    Source doc = (Source) navigator.getDocument(testUrl);
    for (int i = 0; i < xpaths.length; i++) {
      String xpath = xpaths[i];
      System.out.println("*** Evaluating: " + xpath);
      XPath expr = new JerichoXPath(xpath, navigator);
      Object result = expr.evaluate(doc);
      if (result instanceof Element) {
        System.out.println("Element: " + ((Element) result).getName());
      } else if (result instanceof List) {
        System.out.println("List: size=" + ((List) result).size());
        List elements = (List) result;
        for (int j = 0; j < elements.size(); j++) {
          Element element = (Element) elements.get(j);
          System.out.println("Element: " + ((Element) element).getName());
        }
      } else if (result instanceof String) {
        System.out.println("String: " + ((String) result));
      } else if (result instanceof Number) {
        System.out.println("Number: " + ((Number) result));
      } else if (result instanceof Boolean) {
        System.out.println("Boolean: " + ((Boolean) result));
      } else {
        System.out.println("Unknown: " + result == null ? 
          "NULL" : result.getClass().getName());
      }
    }
  }
}

Other XPath methods

I haven't gotten around to testing the other public methods of the XPath object, but I suspect that Jaxen will do the right thing in all these cases, because all of the methods can be written as wrappers over the evaluate() method. If I see problems with the behavior, the plan is to override these methods in the JerichoXPath class, as shown below:

  public String valueOf(Object context) throws JaxenException {
    return stringValueOf(context);
  }

  public boolean booleanValueOf(Object context) throws JaxenException {
    String result = stringValueOf(context);
    return Boolean.valueOf(result);
  }

  public Number numberValueOf(Object context) throws JaxenException {
    String result = stringValueOf(context);
    if (NumberUtils.isNumber(result)) {
      return NumberUtils.createNumber(result);
    } else {
      throw new JaxenException("Value of " + xpathExpr + " is not numeric");
    }
  }

  public String stringValueOf(Object context) throws JaxenException {
    Object result = evaluate(context);
    if (result instanceof String) {
      return (String) result;
    } else {
      throw new JaxenException("Cannot return string value of " + xpathExpr);
    }
  }

  public Object selectSingleNode(Object context) throws JaxenException {
    List nodes = selectNodes(context);
    if (nodes.size() > 0) {
      return nodes.get(0);
    } else {
      return null;
    }
  }

  public List selectNodes(Object context) throws JaxenException {
    Object result = evaluate(context);
    if (result instanceof List) {
      return ((List) result);
    } else {
      return Collections.emptyList();
    }
  }

Alternative Approach

As outlined in the second paragraph of the guidelines, another approach is to build a Jericho implementation of the XPathHandler interface, and parse the XPath expression using the default XPathReader. I tried this in desperation midway through the above exercise, and it worked well for the simple element traversal events. However, I decided against going this route because of the complexity of the event handling code for handling predicates. However, if you are looking to build something that does not (or cannot for some reason) use the Jaxon Navigator approach, then this may be worth looking at.

XPath/Jaxen articles

There is not much information available on Jaxen and SAXPath Event API, but I found these two links quite useful.

Introduction to SAXPath and Jaxen by Bob McWhirter
Chapter 16 of Processing XML with Java by Elliote Rusty Harold

Update 2009-04-26: I contributed the code above back to the Jaxen project. However, they are planning on moving the examples out of the project since it imposes unnecessary JAR file dependencies on clients, so the code may become part of a jaxen-contrib project later. If you are looking to incorporate the code into your project, the only extra dependency are jericho-html. There is some commons-logging dependencies, but they can be removed or replaced with your favorite logger in the source code.

4 comments:

Roman Solomatin5/04/2011 6:35 AM
Hi.

Thank you for this code, it has helped me a lot.

One question though:
Why did you use NamedAccessNavigator interface for your DocumentNavigator class?
Sujit Pal5/07/2011 12:38 PM
Hi Stoned Necromancer, glad it helped and you are welcome. As for your question, I honestly don't remember, but I do remember using one of the other Jaxen extensions (most likely JDOM) as a template for building this. So one reason could be that the JDOM Jaxen extension also implemented NamedAccessNavigator.
Anonymous7/19/2013 7:02 AM
Its not evaluating Xpath functions such as substring-after(//meta/@content,'url=')

source = <\meta http-equiv='refresh' content='0;url=https://www.google.com' />
Sujit Pal7/22/2013 2:40 PM
Hi, this looks like a bug or a missed requirement, thanks for pointing out. I verified that your XPath expression returns the URL for google's search page using XPE. Its been a while since I wrote this though, and I don't use it anymore, so I probably won't have the bandwidth to fix it. If you are using this extension, would appreciate a patch if you fix.

Comments are moderated to prevent spam.

Monday, April 20, 2009

XPath over HTML using Jericho and Jaxen