Salmon Run: Generic XML query client with XPath

Some time ago, I needed to write a client to query a Lucene index running Solr. Most people have heard of Lucene, a popular open source search index building and querying toolkit, which can be used to build search functionality into your sites. Fewer people are likely to have heard of Solr, which is an HTTP wrapper over a Lucene index, which allows populating the index by HTTP POSTing XML documents to it, and retrieval by HTTP GETs. The HTTP GET syntax is a request URL with a set of query arguments specifying number of rows needed, start position, an enumeration of the fields needed, and the query itself as a Lucene query. The GET returns an XML document which needs to be parsed by your client. In effect, it allows you to set up and integrate a Lucene index with very little knowledge of the Lucene API itself.

Knowing nothing about Solr at that time, (and being too lazy and pressed for time to RTFM), I assumed that I would have to look up a Java API to figure out how to write a query client, so I talked to one of the Solr developers who told me to just use XPath.

Since Solr returns the search results as a XML document (with a predefined format), my query client would need to send the HTTP GET request to the Solr server, then parse the returned XML. I could either write a custom XML parser or use XPath calls. I think of XPath as the 4GL of the XML world, which allows you to specify nodes in an XML document, much like you would specify the full path name of a file on a Unix-like operating system. I choose the XPath approach, because it saved me from having to write yet another XML parser. You would think that performance would suffer as a result of using XPath instead of a custom XML parser, but I did not notice any significant degradation.

The XPath client I wrote was not terribly generic, however. The client would compose and send the appropriate HTTP GET request to the Solr server. The Solr HTTP query syntax is minimalistic but very powerful, so you can specify exactly what you want and how much. Since I was retrieving collections of nodes, I needed to iterate over them, something I could not do without hardcoding this in the client code. I mentioned my newfound knowledge of XPath to a colleague and described the client I wrote, and he casually asked if this was something generic that maybe he could use too, just by changing configuration. I replied in the negative, and it was no big deal, and the conversation moved on to other things. But it got me thinking, and this article is a result of that thought process.

This article describes a generic XML parsing client using XPath. Plain vanilla XPath queries have been used to pull scalars from an XML stream. By scalars I mean fully specified content (all the way to the leaf node). For collections, such as Lists and Maps, I made a little extension to the XPath language. These extensions are added to the end of a valid XPath expression that will evaluate by itself to a non-leaf node. The extension will resolve to a collection of leaf nodes. Here are some examples of my extended XPath syntax:

Extended XPath Expression	Comments
meta/numberOfResults/text()	Returns the number of results returned in the XML as a scalar string.
data/array/row/col[@name="reviewer"]/text():[0]	Returns a 1 element List of String, containing the first reviewer in the XML result.
data/array/row/col[@name="id"]/text():[]	Returns a List of String, containing all the ids in the XML result.
data/array/row/col[@name="id"]/text():[0-1]	Returns a 2 element List String, containing the ids found at the first two rows in the XML result.
data/array/row/col[@name="id"]/text():[1-]	Returns a List of String, with all ids starting from the 1st row in the XML result.
data/array/row/col[@name="id"]/text():[-3]	Returns a List of String, with all ids starting from the 3rd row to the last row in the XML result.
data/array/row[@id="1"]:{}	Returns a Map of String key and String value, with all the columns where the row id attribute is "1".
data/array/row[@id="1"]:{"name"}	Returns a Map of String key and String value, containing the value of the name column where the row id attribute is "1".

Notice that only the first query is a pure XPath query. All the others have little qualifiers in the query after the colon(:). The extension syntax is quite simple, with [] representing a List, and {} representing a Map. This syntax is similar to that used by JSON or PLists. We can also specify a slice of the List or a set of keys in the Map by using the Python like array slice notation [m-n] and the {"key"} notation respectively. Having the extra qualifying notation allows us to specify generic XPath like patterns which can be set up in the configuration. So while XPath simplifies the job of writing an XML parser (compared to using the DOM, SAX or JDOM APIs), using the XPathClient can eliminate it altogether.

Here is how you can call the XPathClient from Java code.

    // set up the xpath client
    XPathClient client = new XPathClient();
    client.setServiceUrl("http://path.to.webservice/service.html");
    client.setRootElement("result"); // in my example
    Map<String,String> xpathExpressions = new HashMap<String,String>();
    xpathExpressions.put("key", "valid/extended/XPath/Expression");
    client.setXPathExpressions(xpathExpressions);
    // run the xpath client
    Map<String,Object> results = client.runQuery();
    // do something with the results
    System.out.println(results.get("key").toString());

Alternatively, the XpathClient can also be configured using Spring like this:

<!-- beans.xml -->
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "spring-beans.dtd" >
<beans>
                                                                                
    <!-- Configuration for the XpathClient -->
    <bean id="xpathClient" class="com.mycompany.xpath.XpathClient">
        <property name="serviceUrl" value="http://localhost/test.xml" />
        <property name="rootElement" value="result" />
        <property name="xpathExpressions">
            <props>
                <prop key="numberOfResults">meta/numberOfResults/text()</prop>                 
                <prop key="reviewer[all].id">data/array/row/col[@name="id"]/text():[]</prop>           
                <prop key="reviewer[0-1].id">data/array/row/col[@name="id"]/text():[0-1]</prop> 
                <prop key="reviewer[1-].id">data/array/row/col[@name="id"]/text():[1-]</prop>                 
                <prop key="reviewer[-3].id">data/array/row/col[@name="id"]/text():[-3]</prop>                 
                <prop 
key="reviewer[0].reviewer">data/array/row/col[@name="reviewer"]/text():[0]</prop>
                <prop key="review[0]">data/array/row[@id="1"]:{}</prop>       
                <prop key="review[0].name">data/array/row[@id="1"]:{"name"}</prop>
            </props>
        </property>
    </bean>
                                                                                
</beans>

And the resulting client call is even simpler. The code below configures the XPathClient for a standalone application, which would typically be the case for a web service client. Similar code can be written for clients which are embedded into web applications. In that case you will get a handle to the ApplicationContext and get the bean from it.

    // you would do this part anyway for the other beans
    Resource res = new ClassPathResource("beans.xml");
    XmlBeanFactory factory = new XmlBeanFactory(res);

    // this is where you get the injected reference from the configuration
    client = (XpathClient) factory.getBean("xpathClient");
    Map<String,Object> results = client.runQuery();
    // do something with the results

Here is the code for the XpathClient class. It has a default constructor and uses setter injection to set it up. The runQuery() method makes a HTTP GET call to the configured service URL, and gets a handle to the root element Node object. It then runs all the extended XPath expressions on the root element Node. This strategy ensures that we make only a single network call to the web service per call to the runQuery() method. Pure XPath expression is trivial, it simply delegates to the XPath object's evaluate() method. However, if it finds a colon sign in the XPath expression, it concludes that the expression is an extended XPath expression, and based on what the contents of the expression after the colon sign, it will do extra processing to either return a List or Map of information.

// $Id$
// $Source$
package com.mycompany.xpath;
  
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
  
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathFactory;
  
import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.w3c.dom.Document;
import org.w3c.dom.NamedNodeMap;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
  
/**
 * A simple configuration driven XPath client to read data off an XML stream.
 * @author Sujit Pal
 * @version $Revision$
 */
public class XpathClient {
      
    protected final static Log log = LogFactory.getLog(XpathClient.class);
  
    private String serviceUrl;
    private String rootElement;
    private Map<String,String> xpathExpressions;
      
    public XpathClient() {
        super();
    }
  
    public void setRootElement(String rootElement) {
        this.rootElement = rootElement;
    }
  
    public void setServiceUrl(String serviceUrl) {
        this.serviceUrl = serviceUrl;
    }
  
    public void setXpathExpressions(Map<String, String> xpathExpressions) {
        this.xpathExpressions = xpathExpressions;
    }
  
    /**
     * Runs the set of queries specified in the XPath expressions map and
     * returns results in a Map with the results posted against identical
     * keys. Results for exact matches are returned as String values. Results
     * with multiple matches are returned as a List<String>.
     * @return a result Map<String,Object>
     * @throws Exception if one is thrown.
     */
    public Map<String,Object> runQuery() throws Exception {
        DocumentBuilder builder = DocumentBuilderFactory.newInstance().newDocumentBuilder();
        Document doc = builder.parse(serviceUrl);
        XPath xpath = XPathFactory.newInstance().newXPath();
        // get the rootNode as our reference point
        Node rootNode = (Node) xpath.evaluate("/" + this.rootElement, doc, XPathConstants.NODE);
        // Build our new output map
        Map<String,Object> resultMap = new HashMap<String,Object>();        for (String key : xpathExpressions.keySet()) {
            String xpathExpr = xpathExpressions.get(key);
            if (StringUtils.isEmpty(xpathExpr)) {
                log.warn("No XPath expression defined for " + key);
                continue;
            }
            if (xpathExpr.indexOf(':') > -1) {
                // Collection retrieval mode
                String[] parts = xpathExpr.split(":");
                if (parts[1].startsWith("[")) {
                    // should return a List of results
                    NodeList nodeList = (NodeList) xpath.evaluate(rewriteXpathExpr(parts[0], false), rootNode, XPathConstants.NODESET);
                    int numNodesInSet = nodeList.getLength();
                    int[] sliceCoordinates = getSliceCoordinates(parts[1], numNodesInSet);
                    List<String> resultList = new ArrayList<String>();
                    for (int i = sliceCoordinates[0]; i <= sliceCoordinates[1]; i++) {
                        String value = nodeList.item(i).getTextContent();
                        resultList.add(value);
                    }
                    resultMap.put(key, resultList);
                } else if (parts[1].startsWith("{")) {
                    // should retrieve a flat Map of name value pairs
                    if (parts[0].endsWith("/*")) {
                        parts[0] = parts[0].substring(0, parts[0].length() - 2);                    }
                    Node parentNode = (Node) xpath.evaluate(parts[0], rootNode, XPathConstants.NODE);
                    NodeList nodeList = parentNode.getChildNodes();
                    String[] keys = getRequestedAttributes(parts[1]);
                    Map<String,String> objectMap = new HashMap<String,String>();
                    int numChildren = nodeList.getLength();
                    for (int i = 0; i < numChildren; i++) {
                        Node node = nodeList.item(i);
                        if (node.getAttributes() == null || node.getAttributes().getLength() == 0) {
                            continue;
                        }
                        String firstAttribute = node.getAttributes().item(0).getNodeValue();
                        if (isRequestedAttribute(firstAttribute, keys)) {
                            objectMap.put(firstAttribute, node.getTextContent());
                        }
                    }
                    resultMap.put(key, objectMap);
                }
            } else {
                // should return a single result
                String result = (String) xpath.evaluate(rewriteXpathExpr(xpathExpr, true), rootNode, XPathConstants.STRING);
                resultMap.put(key, result);
            }
        }
        return resultMap;
    }
  
    /**
     * Prepends the rootElement to the XPath expression and appends a /text()
     * call if not present.
     * @param xpathExpr the XPath expression.
     * @param appendTextCall true if single record, else false.
     * @return the modified XPath expression.
     */
    private String rewriteXpathExpr(String xpathExpr, boolean appendTextCall) {
        StringBuilder sb = new StringBuilder("/").
            append(rootElement).append("/").append(xpathExpr);
        if (appendTextCall && (!(xpathExpr.endsWith("/text()")))) {
            sb.append("/text()");
        }
        return sb.toString();
    }
      
    /**
     * Returns the start and end coordinates of the slice to be returned
     * from a NodeSet object. This expression is appended to the end of a
     * valid XPath expression that can return more than one value. The rules
     * are as follows:
     * [] - return all elements of the set.
     * [m-n] - return elements[m] to elements[n] both inclusive.
     * [-n] - return elements[0] to elements[n] both inclusive.
     * [m-] - return elements[m] till end of list.
     * [m] - return only element[m].
     * @param slicePart the slice coordinate subexpression.
     * @param nodeSetLength the total length of the nodeset.
     * @return the start and end coordinates.
     */
    private int[] getSliceCoordinates(String slicePart, int nodeSetLength) {
        int[] coords = new int[] {0, 0};
        String temp = slicePart.replaceAll("\\[", "").replaceAll("\\]", "");
        if (StringUtils.isEmpty(temp)) {
            // []: return all elements
            coords = new int[] {0, nodeSetLength - 1};
        } else if (temp.startsWith("-")) {
            // [-n]: return 0-th to n-th elements
            coords = new int[] {0, Integer.parseInt(temp.substring(1)) - 1};
        } else if (temp.endsWith("-")) {
            // [m-]: return m-th to last elements
            coords = new int[] {Integer.parseInt(temp.substring(0, temp.length() - 1)), nodeSetLength - 1};
        } else if (StringUtils.isNumeric(temp)) {
            // [n]: return the n-th element
            coords = new int[] {Integer.parseInt(temp), Integer.parseInt(temp)};        } else {
            // [m-n]: return the m-th to the n-th element
            String[] tempParts = temp.split("-");
            coords = new int[] {Integer.parseInt(tempParts[0]), Integer.parseInt(tempParts[1]) - 1};
        }
        return coords;
    }
  
    /**
     * Returns an array of requested attributes. The parsing is done by removing     * the leading and trailing {} characters, then splitting on comma.
     * @param string the requested attribute string.
     * @return an array of requested attributes.
     */
    private String[] getRequestedAttributes(String attrString) {
        String temp = attrString.replaceAll("\\{", "").replaceAll("\\}", "");
        if (StringUtils.isEmpty(temp)) {
            return new String[0];
        }
        return temp.split("\\s*,\\s*");
    }
  
    /**
     * Scans the specified array for a match and returns true if found, false
     * if not.
     * @param attrib the attribute to look for.
     * @param keys an array of requested attributes.
     * @return true if attrib found in keys, false if not.
     */
    private boolean isRequestedAttribute(String attrib, String[] keys) {
        if (keys == null || keys.length == 0) {
            // accept all attributes
            return true;
        }
        for (int i = 0; i < keys.length; i++) {
            if (keys[i].equals("\"" + attrib + "\"")) {
                return true;
            }
        }
        return false;
    }
      
}

One thing to realize is that your XML must be parseable using XPath, ie, you should be able to uniquely specify the node within the XML document. This is not as hard as it sounds, all that is needed is to specify (on the server side) an id attribute for each node, which must be unique across the document for that node.

If, like me when I first started playing with XPath, you are not too familiar with the XPath syntax, and would need help validating them against your document, there is XPath Explorer (xpe), a free XPath browser from Purpletech which helped me a lot when trying to build my XPath expressions.

A possible improvement may be a cleaner and more maintainable parsing logic for my extended XPath expression syntax, but the best solution for that would have been to write a full blown parser, which I felt is a little overkill for something as simple as this.

The XPathClient has been unit tested with JUnit using the extended XPath expressions in the Spring configuration file shown above.