Sunday, February 07, 2010

Apache XML-RPC Service with Spring 3

Last week I described a little Drupal module that sends an XML-RPC request to a remote server every time a node is inserted, updated or deleted. This week, I describe a simple XML-RPC Server using Apache XML-RPC and Spring 3.0, that recieves that request and does something with it.

Although Apache XML-RPC seems to be quite a mature product, the documentation is quite sparse, and seems to be geared towards client-side stuff and standalone servers (the server documentation page mentions the WebServer and the ServletWebServer, both of which run as daemons).

This is kind of surprising, since I would imagine that most people who want to run an XML-RPC server would just naturally embed an XML-RPC service inside their favorite web container, rather than run a standalone server. That way they can leverage the features of the container, like pooling, instead of having to build it in themselves. Additionally, in my case, the web application would be written using Spring 3, because I am trying to learn it.

There is a brief mention about using the XmlRpcServer on the server documentation page, and my initial attempt was to build something around that. I did figure out how to do that, but I ended up instantiating quite a few supporting library classes, so I figured there had to be a simpler way, so I started looking around. Luckily, I found three resources which proved quite helpful.

Dejan Bosanac describes how to wrap a Spring Adapter around the WebServer class. While interesting, the server docs discourage this practice as something that won't scale. Essentially, you end up instantiating a daemon within a controller, which is started up and shut down with the container via the controller's lifecycle methods. In a later article, he describes how to use an XmlRpcServer embedded inside a Spring Controller. I came upon this after I had written my own version, and my version was much more complicated - I suspect its because the API has changed quite a bit between the time the post was written and now. Finally, I came across Perry Nguyen's post which uses the XmlRpcServletServer embedded in a Spring Controller - which effectively allowed me to cut down the controller code (minus the configuration) to a single line.

So really, if you are looking for how to embed Apache XML-RPC inside Spring, the three resources I have referenced earlier give you all the information required. The most value I add in this blog is to show how to do this with Spring 3. From my point of view, however, this is something I did not know how to do before this, so its been quite interesting to figure out how.

The way the Apache XML-RPC server (and possibly other XML-RPC servers) work is quite cool. The XML-RPC request contains a methodName parameter which contains the name of the handler and its method to be executed on the server, something like ${handlerName}.${methodName}. A methodName like "publisher.update" would point to a handler named "publisher" which has a method update() which takes the parameters contained in the params element of the XML-RPC request. The return value from the update() method is passed back as an XML-RPC response back to the caller.

Since I was using Spring 3, and being relatively new to it, I decided to use Annotations and Autowiring to the max, mainly to see how far I could take it. So my myapp-servlet.xml file contains the following snippet, which tells Spring to use annotation based configuration and to automatically scan for components starting from the root of the application.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
<?xml version="1.0" encoding="UTF-8"?>
<!--
Source: src/main/webapps/WEB-INF/myapp-servlet.xml
-->
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:p="http://www.springframework.org/schema/p"
    xmlns:context="http://www.springframework.org/schema/context"
    xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
        http://www.springframework.org/schema/context
        http://www.springframework.org/schema/context/spring-context-3.0.xsd">

  <context:annotation-config/>
  <context:component-scan base-package="com.mycompany.myapp"/>

</beans>

As expected, the controller is annotated with @Controller. It is effectively an InitializingBean, so we have our init method annotated with @PostConstruct - this is where the server is constructed using the properties injected into it. The request handling method is serve, which is annotated with @RequestMapping, which also specifies what the URL is and that it will only accept POST requests.

My Controller uses auto-wiring for setting variables. The default configuration needs no XML configuration at all, but if you wanted to override some of the configuration, then you would need to specify it in XML. Here is the code for the controller.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
// Source: src/main/java/com/mycompany/myapp/controllers/XmlRpcServerController.java
package com.mycompany.myapp.controllers;

import java.util.Map;

import javax.annotation.PostConstruct;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.xmlrpc.XmlRpcException;
import org.apache.xmlrpc.server.XmlRpcErrorLogger;
import org.apache.xmlrpc.server.XmlRpcServerConfigImpl;
import org.apache.xmlrpc.webserver.XmlRpcServletServer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;

import com.mycompany.myapp.handlers.IHandler;
import com.mycompany.myapp.xmlrpc.SpringHandlerMapping;
import com.mycompany.myapp.xmlrpc.SpringRequestProcessorFactoryFactory;

@Controller
public class XmlRpcServerController {

  @Autowired(required=false) private String encoding = 
    XmlRpcServerConfigImpl.UTF8_ENCODING;
  @Autowired(required=false) private boolean enabledForExceptions = true;
  @Autowired(required=false) private boolean enabledForExtensions = true;
  @Autowired(required=false) private int maxThreads = 1;
  @Autowired private Map<String,IHandler> handlers;
  
  private XmlRpcServletServer server;
  
  @PostConstruct
  protected void init() throws Exception {
    XmlRpcServerConfigImpl config = new XmlRpcServerConfigImpl();
    config.setBasicEncoding(encoding);
    config.setEnabledForExceptions(enabledForExceptions);
    config.setEnabledForExtensions(enabledForExtensions);
    
    server = new XmlRpcServletServer();
    server.setConfig(config);
    server.setErrorLogger(new XmlRpcErrorLogger());
    server.setMaxThreads(maxThreads);

//    PropertyHandlerMapping handlerMapping = new PropertyHandlerMapping();
//    for (String key : handlers.keySet()) {
//      handlerMapping.addHandler(key, handlers.get(key).getClass());
//    }
    SpringHandlerMapping handlerMapping = new SpringHandlerMapping();
    handlerMapping.setRequestProcessorFactoryFactory(
      new SpringRequestProcessorFactoryFactory());
    handlerMapping.setHandlerMappings(handlers);

    server.setHandlerMapping(handlerMapping);
  }
  
  @RequestMapping(value="/xmlrpc.do", method=RequestMethod.POST)
  public void serve(HttpServletRequest request, HttpServletResponse response) 
      throws XmlRpcException {
    try {
      server.execute(request, response);
    } catch (Exception e) {
      throw new XmlRpcException(e.getMessage(), e);
    }
  }
}

The only required field is the Map of handler names and classes - we achieve auto-wiring here by making all our handlers implement a marker interface IHandler.

1
2
3
4
// Source: src/main/java/com/mycompany/myapp/handlers/IHandler.java
package com.mycompany.myapp.handlers;

public interface IHandler {}

So the auto-wiring code can now ask for a Map of all IHandler objects keyed by their bean names. The default naming strategy is to lowercase the first letter of the class name, such that something like com.mycompany.myapp.PublishHandler is converted to publishHandler. However, since our XML-RPC methodName is "publisher.update", we really want to call this handler "publisher" in our handler mappings. Using the @Server's value override allows us to satisfy our Java naming conventions as well as the conventions dictated by XML-RPC.

The PublishHandler methods insert(), update() and delete() correspond to the second part of the methodName in the XML-RPC request. At the moment, all they do is write out their parameters so you know they are being called.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// Source: src/main/java/com/mycompany/myapp/handlers/PublishHandler.java
package com.mycompany.myapp.handlers;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.stereotype.Service;

@Service("publisher")
public class PublishHandler implements IHandler {

  private final Log log = LogFactory.getLog(getClass());
  
  public int insert(String id, String title, String body) {
    log.info("inserting [" + id + "," + title + "," + body + "]");
    return 0;
  }
  
  public int update(String id, String title, String body) {
    log.info("updating [" + id + "," + title + "," + body + "]");
    return 0;
  }
  
  public int delete(String id, String title, String body) {
    log.info("deleting [" + id + "," + title + "," + body + "]");
    return 0;
  }
}

My Spring controller is called from Drupal via a trigger every time a node is inserted, updated or deleted. The Drupal action sends it an HTTP POST request to a URL like http://localhost:8081/myapp/xmlrpc.do. So predictably, when called from Drupal, the log message on the server side dumps out the contents of the request, and returns a 0 back to Drupal.

The nice thing about using Apache XML-RPC is that, after the initial learning hump, using it is quite simple. The initial learning curve can be made less steep by better documentation, perhaps an example of using Apache XML-RPC as a service embedded in a third party server component.

Update: 2010-02-09

Turns out I was a bit premature assuming that I had figured this stuff out. The code above (prior to the change that I am making now) still works, as long as all your handler does is print cute little messages to the logfile. However, generally you would want your handlers to do something more substantial than that, and for that you would probably set it up in your Spring configuration with some relatively heavyweight resource objects, most likely in your @PostConstruct method. The problem is that, by default, Apache XML-RPC will instantiate a new handler on every request - this means that your @PostConstruct does not get a chance to run (since it is run by Spring on startup) and you start getting NullPointerExceptions all over the place.

Perry Nguyen touches on that in his post, but I guess I did not fully understand it at that point to care enough. Tomas Salfischberger posts a more detailed solution, which I adapted for my needs. Essentially, I needed to replace the PropertyHandlerMapping with a custom HandlerMapping, and set a custom RequestProcessorFactoryFactory into this custom HandlerMapping, that forces Apache-XMLRPC to delegate the handling to a pre-configured Spring bean instead of a newly instantiated bean.

First, the custom RequestProcessorFactoryFactory. This component is called from within a HandlerMapping to get back a RequestProcessor object (our IHandler) via two levels of indirection. Since Apache XML-RPC looks up a RequestProcessor by Class, we make an internal Map of IHandler class to IHandler bean reference. It extends the StatelessProcessorFactoryFactory which is recommended for "heavyweight" handlers. Here is the code for my SpringRequestProcessorFactoryFactory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
// Source: src/main/java/com/mycompany/myapp/xmlrpc/SpringRequestProcessorFactoryFactory.java
package com.mycompany.myapp.xmlrpc;

import java.util.HashMap;
import java.util.Map;

import org.apache.xmlrpc.XmlRpcException;
import org.apache.xmlrpc.XmlRpcRequest;
import org.apache.xmlrpc.server.RequestProcessorFactoryFactory;
import org.apache.xmlrpc.server.RequestProcessorFactoryFactory.StatelessProcessorFactoryFactory;

public class SpringRequestProcessorFactoryFactory 
    extends StatelessProcessorFactoryFactory
    implements RequestProcessorFactoryFactory {

  Map<Class<? extends IHandler>,IHandler> classHandlerMappings;
  
  protected void init(Map<String,IHandler> handlerMappings) {
    classHandlerMappings = new HashMap<Class<? extends IHandler>, IHandler>();
    for (String key : handlerMappings.keySet()) {
      IHandler handler = handlerMappings.get(key);
      Class<? extends IHandler> clazz = handler.getClass();
      classHandlerMappings.put(clazz, handler);
    }
  }
  
  public RequestProcessorFactory getRequestProcessorFactory(
      Class clazz) throws XmlRpcException {
    final IHandler handler = classHandlerMappings.get(clazz);
    return new RequestProcessorFactory() {
      public Object getRequestProcessor(XmlRpcRequest pRequest)
          throws XmlRpcException {
        return handler;
      }
    };
  }
}

Next, we create our custom HandlerMapping. As before, the trick is to know what extension point to use, and I just follow Tomas's lead on this one. This component sets up a mapping of the public methods of the IHandler beans. Because you need to have the RequestProcessorFactoryFactory be set into the HandlerMapping object before the handler mappings themselves, I baked in this requirement into the setHandlerMappings method in my custom subclass. Here it is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// Source: src/main/java/com/mycompany/myapp/xmlrpc/SpringHandlerMapping.java
package com.mycompany.myapp.xmlrpc;

import java.util.Map;

import org.apache.xmlrpc.XmlRpcException;
import org.apache.xmlrpc.server.AbstractReflectiveHandlerMapping;
import org.springframework.util.Assert;

public class SpringHandlerMapping extends AbstractReflectiveHandlerMapping {

  public void setHandlerMappings(Map<String,IHandler> handlerMappings) 
      throws XmlRpcException {
    SpringRequestProcessorFactoryFactory ff = 
      (SpringRequestProcessorFactoryFactory) getRequestProcessorFactoryFactory();
    Assert.notNull(ff, "RequestProcessorFactoryFactory must be set");
    ff.init(handlerMappings);
    for (String serviceName : handlerMappings.keySet()) {
      IHandler serviceBean = handlerMappings.get(serviceName);
      registerPublicMethods(serviceName, serviceBean.getClass());
    }
  }
}

At this point, all we need to do is replace the PropertyHandlerMapping instantiation and setup, and replace it with our custom HandlerMapping object. The code for doing that has been updated in the controller code above (and the old code that it replaces has been commented out).

21 comments (moderated to prevent spam):

Baptiste Autin said...

Very interesting article.

Yet, what about AOP/proxyfied services ?
In such case, handler.getClass() will return a "JdkDynamicAopProxy" instead of the proper bean class name, and an exception will be raised...

Sujit Pal said...

Thanks Baptiste. I guess this is not fully generic as you point out - perhaps we could check for a proxy class and call its getProxy().getClass() instead? What do you think?

Baptiste Autin said...

Actually, I tried to replace
clazz = handler.getClass();
by
clazz = AopUtils.getTargetClass(handler);
in the init() method.

With this, the mapping is correctly processed, and the server can start in every situation, but unfortunately, an exception is then raised at call time (Illegal argument for method) probably for the same reason (the code added by SpringAOP must mess up with the reflection mechanism used by the XmlRpc server)

I ended up removing the @Transactional annotation on my target service class (turning it into a "regular" bean) since I didn't really have a need for it, but I wish I had found the true solution to that problem ! :)

Thanks for posting this article anyway. For some reason, I had to stick with the Apache XmlRpcServer (some say that other implementations integrate better with Spring...) and so it was very useful to me.

Sujit Pal said...

Thanks for the code snip, Baptiste. Hopefully, it will help others out who are in a similar situation.

Raj said...

Hi Sujit

I followed your code and all your link about Apache XML RPC with Spring and also could write a running POC but I am stuck at one point where I need your help

When a request comes with say /xmlrpc.do or in my case say /show.htm it goes to the method serve of XmlRpcServerController. And from there it goes to the PublishHandler insert/update method (which in my case is say show). But then from there I am little confused as to how to return File say a SWF or SVG to the client so that the client can show it.

I have tried returning integer, string and it works fine but when I try to return a file it fails

Can you please help me with this ?

In a normal Servlet request and response context we generally write the file in the response output stream thereby making client render the swf or svg. But with Apache XML RPC and Spring im place I am not getting a handle to the Response outputstream inside the PublishHanlder show method.......

Please help me

With Regards
Raj

Sujit Pal said...

Hi Raj, I have had similar problems with (badly formed) HTML payloads which would break the XMLRPC. I solved it initially by wrapping the field in a CDATA, then later converted it to Base64 and sent it as a base64 field. My decoding code is in Java (encoding is in PHP, but Java encoding code should be similar to the decoding code), something like this:

if (obj instanceof byte[]) {
..Base64 b = new Base64();
..String value = b.decode((byte[]) obj);
}

Here Base64 is from commons-codec, and the snippet above checks to see if the field is a byte[] (that is what base64 data is sent over as.

So in your case, I am guessing that you will need to wrap the contents of the SVG/SWF file in a base64 field, and perhaps put in a content type field which tells what it is, so the receiver can decode it appropriately.

seego said...

Hello, your article has provided tons of useful information, but I can't figure out why you use @Autowired annotation on private fields declared in your XmlRpcServerController class?

Sujit Pal said...

Thanks seego, glad it helped you. I used @Autowired on the private fields to allow these properties to be overridden if needed, from Spring XML configuration.

gopi said...

Hi sujith,

i wanna know the reason and the remedy for the exception


exception

org.openbravo.base.exception.OBException: Exception thrown Failed to create input stream: Unexpected end of file from server
org.openbravo.dal.core.ThreadHandler.run(ThreadHandler.java:56)
org.openbravo.dal.core.DalRequestFilter.doFilter(DalRequestFilter.java:103)

root cause

org.apache.xmlrpc.XmlRpcException: Failed to create input stream: Unexpected end of file from server
org.apache.xmlrpc.client.XmlRpcSunHttpTransport.getInputStream(XmlRpcSunHttpTransport.java:65)
org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:141)
org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:94)
org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:44)
org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:53)
org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:166)
org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:136)
org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:125)
com.spikesource.lam.bindings.LamClient.get(LamClient.java:353)
com.spikesource.lam.bindings.LamClient.(LamClient.java:68)
org.openbravo.authentication.lam.LamAuthenticationManager.logout(LamAuthenticationManager.java:85)
org.openbravo.base.secureApp.HttpSecureAppServlet.logout(HttpSecureAppServlet.java:611)
org.openbravo.base.secureApp.HttpSecureAppServlet.service(HttpSecureAppServlet.java:356)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
org.openbravo.utils.SessionExpirationFilter.doFilter(SessionExpirationFilter.java:66)
org.openbravo.utils.CharsetFilter.doFilter(CharsetFilter.java:35)
org.openbravo.dal.core.DalRequestFilter$1.doAction(DalRequestFilter.java:81)
org.openbravo.dal.core.ThreadHandler.run(ThreadHandler.java:46)
org.openbravo.dal.core.DalRequestFilter.doFilter(DalRequestFilter.java:103)

Sujit Pal said...

Hi Gopi,

Not sure, since I don't recognize any of the classes in the stack trace, but the message indicates that the server is returning incomplete or malformed XML. Take a look at this post where I describe how to add debugging capabilities to the server, or this one where I describe a python script that acts as a debugging proxy between client and server - both allow you to view the contents of the data being transmitted across the wire.

Unknown said...

Thanks... This article was greate helps to me.

In my case, I override the registerPublicMethods() method in SpringHandlerMapping class to ignore to register unsuitable method.
like this...

protected void registerPublicMethods(String pKey,
Class pType) throws XmlRpcException {
Map map = new HashMap();
Method[] methods = pType.getMethods();
for (int i = 0; i < methods.length; i++) {
final Method method = methods[i];
if (!isHandlerMethod(method)) {
continue;
}
String name = pKey + "." + method.getName();
Method[] mArray;
Method[] oldMArray = (Method[]) map.get(name);
if (oldMArray == null) {
mArray = new Method[]{method};
} else {
mArray = new Method[oldMArray.length+1];
System.arraycopy(oldMArray, 0, mArray, 0, oldMArray.length);
mArray[oldMArray.length] = method;
}
map.put(name, mArray);
}

for (Iterator iter = map.entrySet().iterator(); iter.hasNext(); ) {
Map.Entry entry = (Map.Entry) iter.next();
String name = (String) entry.getKey();
Method[] mArray = (Method[]) entry.getValue();
try{
handlerMap.put(name, newXmlRpcHandler(pType, mArray));
}catch(Exception ignoreit){
//ignore it
// ignoreit.printStackTrace();
}
}
}

it works well in @Transactional Method call from client.

but, there are too many garbage methods registered in handlerMap, as you expect....

any idea to replace my case will be greate help for me.. ^^;.
thanks.

Sujit Pal said...

Hi, from your code, it appears that you are using reflection to expose all the methods in your specified class as XMLRPC methods? Thats a nice idea... although you may want to follow a naming convention for your "exposed" methods and check for that name pattern?

Raj said...

Hi Sujit

This is Raj back. Last time I posted a query regarding "Apache XML RPC and Spring im place, how to handle the Response outputstream inside the PublishHanlder show method" and you had answered me with an option
Now I am again blocked with another integration of WS-XML RPC and Spring Security.
How do we make it work together. As Spring Security has filters which intercept basically all request and then pass it through authentication/authorization queue. Now when we have XML RPC too how do we make it work

Say I have a class XmlRpcServerServletDS extends XmlRpcServlet implements HttpSessionListener

And this class is mentioned as the main listener and servlet in web.xml

Moreover when we have springSecurityFilterChain also mentioned in web.xml and this intercepts all calls how do we pass it to XML RPC handler

Need your help on this .....

Sujit Pal said...

Hi Raj, two things come to mind, not sure if these would be applicable to your environment or not...
1) make the XML-RPC service work within a different DispatcherServlet.
2) make the XML-RPC service work with a distinct auth role and pass the auth token as a parameter in your XML-RPC call.

Raj said...

Hi Sujit
Thanks for your prompt reply.
However let me clarify.Your option 1 is not applicable as different servlet serving for XML RPC service will not help me to pass all method calls for authentication/authorization to the Spring Security Filter. The requirement is simple.Let me try to explain you so that it helps u to understand it clearly
I have a service say S and class A and say a method getA() in the class A which would return a list. Now I want my server service S to support for Apache XML RPC communication so that when the S service can listen to XML RPC request requesting for A.getA(). Now along with it for the method A.getA() I want authentication/method level authorization (say Admin can call this method).

Now how do i achieve this with Apache WS -XML RPC, Spring Security. Will wait for your reply

Sujit Pal said...

Hi Raj, sorry for the long wait, was very busy on both the work and personal front the last couple weeks. Hopefully you have already solved the problem, but here is one possible approach...

From my understanding based on the description, I think that the method A.getA() should only return results for callers in role "admin", right? And I am guessing that its not acceptable for "admin" to authenticate itself interactively, so the solution would probably be to pass a token as a parameter to the request. So you could create a wrapper class A1 say, with A1.getA(user,pass) which will do the authentication and if authenticated, delegate to A.getA()? And you expose the wrapper class via XMLRPC?

jason said...

Really great. Works perfectly.
Saved me lots of time.
Thanks!

Sujit Pal said...

Thanks Jason, glad it helped.

Unknown said...

The only drawback I saw in this approach is that introspection(http://ws.apache.org/xmlrpc/introspection.html) wont work anymore.

Unknown said...

How to make the method name with out dot(.)

IN the current we have used the Method name like .
example : test.handleRequest

But now I would like to use only handleRequest as Method , is this possible.

Sujit Pal said...

In general when you say test.handleRequest(), you are saying that there is an object test of some class which has a method handleRequest, and you are asking to invoke handleRequest on that particular object. So I would say no, at least AFAIK. But check out static imports, they came in Java 7, I never used them since I believed that they result in less readable code. But they might help you do what you want, although I am not 100% sure. Also I think it's a bad idea.