Consuming JSON Webservices with Spring MVC

JavaScript Object Notation (JSON) is a lightweight data interchange format, easy for both humans and machines to parse and generate. It has become very popular among web front-end developers because JavaScript has built in support to convert a JSON string into a JavaScript object, which can be pulled apart and manipulated within the JavaScript scripts within a web page. Along with AJAX (Asynchronous JavaScript and XML), it can be used to create interactive client-driven websites, as opposed to the more traditional server-driven websites that we are all familiar with.

However, JSON can also be considered a simple text data interchange format (similar to XML-RPC and SOAP). Many organizations are exposing their functionality to partners and general users via REST (Representational State Transfer) APIs which are capable to emitting XML or JSON. Clients typically send a HTTP GET request to the APIs, and it responds back with a blob of XML or JSON text which can be parsed and displayed by the client onto the client's co-branded web pages.

This article describes a simple framework, based on Spring MVC, that consumes JSON responses from such an external site, parses the JSON into a flat map, and dumps the output on a JSTL based scaffold page. The scaffold page shows the keys and values of the map, and this can be used to build your own custom web pages. For my example, I choose Google's JSON API (searchmash.com) as my backend, and the custom page I built was a search page for my blogger site. The search page would (hypothetically) be generated when I typed in a term in the search box on the upper left corner of the blogger page.

A picture is worth a thousand words, so here is one that describes this system. The red lines represent the request flow, and the blue lines represent the response. The main components of our framework are the Http Client, the JSON Flattener, the controller and the scaffold JSP page. Each of them are discussed below.

Spring/webapp configuration

Like most Spring applications, the web.xml is minimal, and the only mapping is for the Spring DispatcherServlet. The configuration for the DispatcherServlet is the spring-servlet.xml, and the beans themselves are configured in the applicationContext.xml.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

<!-- ==== web.xml ==== --> <?xml version="1.0" encoding="ISO-8859-1"?> <web-app xmlns="http://java.sun.com/xml/ns/j2ee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd" version="2.4"> <servlet> <servlet-name>spring</servlet-name> <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>spring</servlet-name> <url-pattern>*.html</url-pattern> </servlet-mapping> </web-app>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

<!-- ==== spring-servlet.xml ==== --> <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE beans PUBLIC "-//SPRING//DTD BEAN//EN" "http://www.springframework.org/dtd/spring-beans.dtd"> <beans> <import resource="classpath:applicationContext.xml" /> <bean id="urlMapping" class="org.springframework.web.servlet.handler.SimpleUrlHandlerMapping"> <property name="mappings"> <props> <prop key="blogSearch.html">apiProxyController</prop> ... </props> </property> </bean> <bean id="viewResolver" class="org.springframework.web.servlet.view.InternalResourceViewResolver"> <property name="prefix" value="" /> <property name="suffix" value=".jsp" /> <property name="viewClass" value="org.springframework.web.servlet.view.JstlView" /> </bean> </beans>

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

<!-- ==== applicationContext.xml ==== --> <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:util="http://www.springframework.org/schema/util" xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd http://www.springframework.org/schema/util http://www.springframework.org/schema/util/spring-util-2.0.xsd"> <bean id="apiProxyClient" class="com.mycompany.apiclient.services.ApiProxyClient"> <property name="serviceUrl" value="http://api.somecompany.com/" /> </bean> <bean id="apiProxyController" class="com.mycompany.apiclient.controllers.ApiProxyController"> <property name="apiProxyClient" ref="apiProxyClient" /> <property name="uriMethodMap"> <map> <entry> <key><value>/blogSearch.html</value></key> <value>results/</value> </entry> ... </map> </property> </bean> </beans>

The Controller (ApiProxyController)

The Controller is a simple Spring Controller implementation. All it does is to map the incoming request URI to a backend API method, and passes in the rest of the parameters as is. It is injected with a reference to the ApiProxyClient, on which it calls the doRemoteCall() method to retrieve the JSON object. The controller then calls the JSON Flattener to convert the JSON Object to a flat map of key-value pairs, and push it into the ModelAndView, where the JSP can pick it up from. It finally forwards to a JSP which has the same name as the backend method name. The code is shown below:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

public class ApiProxyController implements Controller { private ApiProxyClient apiProxyClient; private String appName; private Map<String,String> uriMethodMap; public ApiProxyController() { super(); } public void setApiProxyClient(ApiProxyClient apiProxyClient) { this.apiProxyClient = apiProxyClient; } public void setUriMethodMap(Map<String, String> uriMethodMap) { this.uriMethodMap = uriMethodMap; } @SuppressWarnings("unchecked") public ModelAndView handleRequest(HttpServletRequest request, HttpServletResponse response) throws Exception { ModelAndView mav = new ModelAndView(); String requestUri = request.getRequestURI(); String remoteCallMethod = uriMethodMap.get(requestUri); if (remoteCallMethod == null) { throw new Exception("The URI:" + path + " is not mapped to a remote method"); } Map<String,String[]> parameterMap = request.getParameterMap(); List<NameValuePair> remoteCallParams = new ArrayList<NameValuePair>(); for (String parameterName : parameterMap.keySet()) { String[] parameterValues = parameterMap.get(parameterName); if (parameterValues == null || parameterValues.length == 0) { continue; } remoteCallParams.add(new NameValuePair(parameterName, NumberUtils.isDigits(parameterValues[0]) ? parameterValues[0] : URLEncoder.encode(parameterValues[0], "UTF-8"))); } JSON jsonObj = apiProxyClient.doRemoteCall(remoteCallMethod, remoteCallParams); Map<String,String> jsonMap = JsonFlattener.flatten(jsonObj); mav.addObject("json", jsonMap); mav.setViewName(remoteCallMethod); return mav; } }

The HTTP Client (ApiProxyClient)

The HTTP Client uses the Apache HTTP Client libraries to model a simple HTTP GET requester. It formats the method call into an HTTP GET request, sends it to the backend service, and parses the resulting response into a JSON Object and returns it to the controller. Here is the code for this.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

public class ApiProxyClient { private String serverDomain; public ApiProxyClient() { super(); } public void setServiceUrl(String serviceUrl) { this.serviceUrl = serviceUrl; } public String getServiceUrl() { return serviceUrl; } public JSON doRemoteCall(String apiMethod, List<NameValuePair> parameters) throws Exception { HttpClient client = new HttpClient(); GetMethod method = new GetMethod(serviceUrl + apiMethod); method.setQueryString(parameters.toArray(new NameValuePair[0])); method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new DefaultHttpMethodRetryHandler(3, false)); try { int rc = client.executeMethod(method); if (rc != HttpStatus.SC_OK) { throw new Exception("HTTP Status Code returned:" + rc); } String response = method.getResponseBodyAsString(); JSONObject jsonResponse = new JSONObject(response); return jsonResponse; } finally { method.releaseConnection(); } } }

The JSON Flattener

The JSON Flattener is a simple utility class which uses the json-lib library. This is the only JSON library, as far as I know, which implements the JSONArray and JSONObject from a common JSON interface. Since the flattener needs to handle JSON Objects with possibly embedded JSONArrays, this was the most appropriate library for me to use. The code traverses the JSON object recursively to flatten it into a set of name value pairs.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

public class JsonFlattener { public static Map<String,String> flatten(JSON json) { Map<String,String> jsonMap = new LinkedHashMap<String,String>(); flattenRecursive(json, "", jsonMap); return jsonMap; } private static void flattenRecursive(JSON json, String prefix, Map<String,String> jsonMap) { if (json instanceof JSONObject) { JSONObject jsonObject = (JSONObject) json; Iterator it = jsonObject.keys(); while (it.hasNext()) { String key = (String) it.next(); Object value = jsonObject.get(key); if (value instanceof JSON) { flattenRecursive((JSON) value, StringUtils.join(new String[] {prefix, key}, '.'), jsonMap); } else { jsonMap.put(StringUtils.join(new String[] {prefix, key}, '.'), String.valueOf(value)); } } } else if (json instanceof JSONArray) { JSONArray jsonArray = (JSONArray) json; for (int i = 0; i < jsonArray.length(); i++) { JSON jsonObject = (JSON) jsonArray.get(i); flattenRecursive(jsonObject, prefix + "[" + i + "]", jsonMap); } } else if (json instanceof JSONNull) { // this is a JSON string jsonMap.put(prefix, null); } } }

The scaffold JSP

For each new service method to be exposed, we can copy our scaffold JSP into a new JSP with the name of the name of the method. Calling the page with a URL of the form: [/apiclient/method.html?query] will pull the following debug page up.

<table cellspacing="0" cellpadding="0" border="1" width="100%"> <tr bgcolor="blue" fgcolor="white"> <th>Name</th> <th>Value</th> </tr> <c:forEach var='jsonElement' items='${json}'> <tr> <td><c:out value="${jsonElement.key}" /></td> <td><c:out value="${jsonElement.value}" /></td> </tr> </c:forEach> </table>

This is the data arranged as we would like it in our final web page, but still unadorned. The JSTL is just cut and pasted into the blogger template for my site below.

<p><b>Found ${json['.estimatedCount']} results for query: "${json['.query.terms']}"</b><br/> <b>Top 5 search results</b></p> <c:forEach var='index' items='0,1,2,3,4'> <c:set var="resultsUrlKey" value=".results[${index}].url"/> <c:set var="resultsTitleKey" value=".results[${index}].title"/> <c:set var="resultsCacheUrlKey" value=".results[${index}].cacheUrl"/> <c:set var="resultsSnippetKey" value=".results[${index}].snippet"/> <p><a href="${json[resultsUrlKey]}">${json[resultsTitleKey]}</a> (<a href="${json[resultsCacheUrlKey]}">Cached</a>)<br/> <font size="-1">${json[resultsSnippetKey]}</font><br/></p> </c:forEach>

So, in effect, all that needs to be done to expose a new JSON webservice is to build a new page, put in the servlet URL mapping in spring-servlet.xml and a incoming URL to backend method name mapping in the controller configuration. And then all that needs to be done is to reformat the JSP accordingly. Such a client system can be built once with a single scaffold JSP page, and can be enhanced by web developers as new methods need to be exposed.