Salmon Run: JMX for Scripts : Spring Configuration

Over the last few weeks, I have been building up a JMX based management interface to manage running shell scripts on remote machines. Our management interface provides an HTML forms interface to observe and control the progress of these scripts over HTTP. It also provides monitoring and scheduling operations for these scripts, and a simple RSS based reporting mechanism to provide a bird's eye view on the status of the scripts.

All these features were added directly in code as I built up the system, and with hindsight (which is always 20/20 :-)), I see huge chunks of code that would benefit from being configuration-driven. This post describes what I needed to do to make the management interface configurable.

Spring is my IoC container of choice, and I have used it successfully in the past for configuration. The approach I use is to identify beans that can be extracted out and instantiated during startup (minimizing or eliminating constructor calls in code), and leave the code to simply implement a strategy to do what is needed with these beans.

Constructor injection to setter injection

When writing code without Spring support, one has to depend on constructors or factories to create new objects. In such cases, I tend to use constructor injection, because instantiation and population can be done concisely, in a single call. When using Spring support, I tend to use setter injection, just because it's the Spring way, and configuration is more natural with setter injection. The one safety issue of insufficiently populated beans is easily addressed using the @Required annotation, which prevents the Spring container from starting up if a property annotated as @Required is not set in the configuration.

So changing over the beans from constructor injection to setter injection was a major change affecting almost all classes. Although pervasive, the change itself is trivial, all it entails is adding a few setXXX() methods and changing the constructor over to the default constructor.

Moving Monitors and Timers to the Script MBean

The second major change was to make the Monitors, Timers, Notification Filters and Listeners properties of the individual ScriptAdapters, rather than have them be instantiated in code within the ScriptAgent. Each ScriptAgent now has a List of Monitors, a List of Timers, a Map of Notification Filters (keyed by Monitor or Timer object name), and a Map of Notification Listeners (also keyed by Monitor or Timer object name) set into it at startup. It exposes these properties using getter methods so they can be attached to the ScriptAdapter in code with ScriptAgent.

New wrapper beans to handle JMX non-JavaBeans

The third change was to create a wrapper bean for GaugeMonitor and Timer in order to populate them from Spring.

The problem with GaugeMonitor is the 2 argument setThresholds(Number,Number) which I did not know how to set from within Spring. So I created a subclass MyLowGaugeMonitor of GaugeMonitor which exposes a single Long property representing the lower bound, and when it is set, its superclass is automatically set with a high upper bound and this lower bound. A hack for sure, and not a very good one, which could be avoided if the JMX team gave us individual setters for both boundaries. To be fair though, they are normally quite good about following Javabean conventions, so this may change in the future.

The second change is in the Timer. I created a subclass MyTimer which also allows setting the Notification type and interval, which I set in the Spring configuration. During runtime, these values are used to create the Timer notification. There may be other ways of doing this, but I wasn't able to find one.

Handing over shutdown hook responsibility to Spring

Switching over to the Spring version meant I could remove the code for the Runtime.getRuntime().addShutdownHook() in ScriptAgent.main(), which destroyed the Script Agent on receipt of a SIGHUP or SIGKILL. I declared in the configuration that ScriptAgent has a destroy-method, and registered the shutdown hook in the Spring context. On shutdown, all destroy-methods are called on container shutdown. Container shutdown happens as before on a SIGHUP or SIGKILL.

The configuration file

The full Spring applicationContext.xml is shown below. It is fairly self-explanatory, given the description of the changes above, but there are in-line comments where I thought it would help.

<?xml version="1.0" encoding="UTF-8"?>
<!-- Source: src/main/resources/applicationContext.xml -->
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans 
       http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
       http://www.springframework.org/schema/util 
       http://www.springframework.org/schema/util/spring-util-2.0.xsd">

  <!-- Annotation processor to make sure we set all required properties -->  
  <bean 
    class="org.springframework.beans.factory.annotation.RequiredAnnotationBeanPostProcessor"/>

  <!-- Listeners (can be reused) -->
  <bean id="emailNotifierListener" class="com.mycompany.myapp.EmailNotifierListener">
    <property name="scriptAlarmEmails">
      <map>
        <entry key="script:name=count_sheep_Dolly" value="dr_evil@clones-r-us.com"/>
        <entry key="script:name=count_sheep_Polly" value="pinky_n_brain@clones-r-us.com"/>
      </map>
    </property>
  </bean>
  
  <bean id="cprListener" class="com.mycompany.myapp.CprListener"/>

  <!-- Configuration for script: "count_sheep.sh Dolly" -->
  <bean id="countSheepDollyObjectName" class="javax.management.ObjectName">
    <constructor-arg index="0" value="script"/>
    <constructor-arg index="1" value="name"/>
    <constructor-arg index="2" value="count_sheep_Dolly"/>
  </bean>  

  <bean id="countSheepDollyStatusMonitor" class="javax.management.monitor.StringMonitor">
    <property name="observedObject" ref="countSheepDollyObjectName"/>
    <property name="observedAttribute" value="Status"/>
    <property name="granularityPeriod" value="60000"/><!-- every minute -->
    <property name="stringToCompare" value="ERROR"/><!-- look for "ERROR" -->
    <property name="notifyMatch" value="true"/><!-- notify on match -->
  </bean>

  <bean id="countSheepDollyLogSizeMonitor" class="com.mycompany.myapp.MyLowGaugeMonitor">
    <property name="observedObject" ref="countSheepDollyObjectName"/>
    <property name="observedAttribute" value="LogFilesize"/>
    <property name="differenceMode" value="true"/><!-- diff(logfile.size) -->
    <property name="thresholdLowerBound" value="1"/><!-- notify if diff -lt 1 -->
    <property name="notifyLow" value="true"/>
    <property name="notifyHigh" value="false"/>
    <property name="granularityPeriod" value="30000"/><!-- every 30s -->
  </bean>
  
  <bean id="countSheepDollyHeartbeatTimer" class="com.mycompany.myapp.MyTimer">
    <property name="notificationType" value="heartbeat"/>
    <property name="intervalMillis" value="60000"/><!-- every minute -->
  </bean>
  
  <bean id="countSheepDollyScriptNotificationFilter" 
      class="com.mycompany.myapp.ScriptNotificationFilter">
    <property name="objectName" value="script:name=count_sheep_Dolly"/>
  </bean>
  
  <bean id="countSheepDollyScriptAdapter" class="com.mycompany.myapp.ScriptAdapter">
    <property name="path" value="/home/sujit/src/rscript/src/main/sh/count_sheep.sh"/>
    <property name="args" value="Dolly"/>
    <property name="logfileName" value="/tmp/count_sheep_Dolly.log"/>
    <property name="errfileName" value="/tmp/count_sheep_Dolly.err"/>
    <property name="pidfileName" value="/tmp/count_sheep_Dolly.pid"/>
    <property name="objectName" value="script:name=count_sheep_Dolly"/>
    <property name="monitors">
      <map>
        <entry key="monitor:type=Status,script=count_sheep_Dolly">
          <ref bean="countSheepDollyStatusMonitor"/>
        </entry>
        <entry key="monitor:type=Logsize,script=count_sheep_Dolly">
          <ref bean="countSheepDollyLogSizeMonitor"/>
        </entry>
      </map>
    </property>
    <property name="timers">
      <map>
        <entry key="timer:type=heartbeat,script=count_sheep_Dolly">
          <ref bean="countSheepDollyHeartbeatTimer"/>
        </entry>
      </map>
    </property>
    <property name="filters">
      <map>
        <entry key="monitor:type=Status,script=count_sheep_Dolly">
          <ref bean="countSheepDollyScriptNotificationFilter"/>
        </entry>
        <entry key="monitor:type=Logsize,script=count_sheep_Dolly">
          <ref bean="countSheepDollyScriptNotificationFilter"/>
        </entry>
        <entry key="timer:type=heartbeat,script=count_sheep_Dolly">
          <ref bean="countSheepDollyScriptNotificationFilter"/>
        </entry>
      </map>
    </property>
    <property name="listeners">
      <map>
        <entry key="monitor:type=Status,script=count_sheep_Dolly">
          <ref bean="emailNotifierListener"/>
        </entry>
        <entry key="monitor:type=Logsize,script=count_sheep_Dolly">
          <ref bean="emailNotifierListener"/>
        </entry>
        <entry key="timer:type=heartbeat,script=count_sheep_Dolly">
          <ref bean="cprListener"/>
        </entry>
      </map>
    </property>
  </bean>
  
  <!-- Configuration for script: "count_sheep.sh Polly" -->
  <bean id="countSheepPollyObjectName" class="javax.management.ObjectName">
    <constructor-arg index="0" value="script"/>
    <constructor-arg index="1" value="name"/>
    <constructor-arg index="2" value="count_sheep_Polly"/>
  </bean>

  <bean id="countSheepPollyStatusMonitor" class="javax.management.monitor.StringMonitor">
    <property name="observedObject" ref="countSheepPollyObjectName"/>
    <property name="observedAttribute" value="Status"/>
    <property name="granularityPeriod" value="60000"/><!-- every minute -->
    <property name="stringToCompare" value="ERROR"/><!-- look for "ERROR" -->
    <property name="notifyMatch" value="true"/><!-- notify on match -->
  </bean>

  <bean id="countSheepPollyLogSizeMonitor" class="com.mycompany.myapp.MyLowGaugeMonitor">
    <property name="observedObject" ref="countSheepPollyObjectName"/>
    <property name="observedAttribute" value="LogFilesize"/>
    <property name="differenceMode" value="true"/><!-- diff(logfile.size) -->
    <property name="thresholdLowerBound" value="1"/><!-- notify if diff -lt 1 -->
    <property name="notifyLow" value="true"/>
    <property name="notifyHigh" value="false"/>
    <property name="granularityPeriod" value="30000"/><!-- every 30s -->
  </bean>
  
  <bean id="countSheepPollyHeartbeatTimer" class="com.mycompany.myapp.MyTimer">
    <property name="notificationType" value="heartbeat"/>
    <property name="intervalMillis" value="60000"/><!-- every minute -->
  </bean>
  
  <bean id="countSheepPollyScriptNotificationFilter" 
      class="com.mycompany.myapp.ScriptNotificationFilter">
    <property name="objectName" value="script:name=count_sheep_Polly"/>
  </bean>
  
  <bean id="countSheepPollyScriptAdapter" class="com.mycompany.myapp.ScriptAdapter">
    <property name="path" value="/home/sujit/src/rscript/src/main/sh/count_sheep.sh"/>
    <property name="args" value="Polly"/>
    <property name="logfileName" value="/tmp/count_sheep_Polly.log"/>
    <property name="errfileName" value="/tmp/count_sheep_Polly.err"/>
    <property name="pidfileName" value="/tmp/count_sheep_Polly.pid"/>
    <property name="objectName" value="script:name=count_sheep_Polly"/>
    <property name="monitors">
      <map>
        <entry key="monitor:type=Status,script=count_sheep_Polly">
          <ref bean="countSheepPollyStatusMonitor"/>
        </entry>
        <entry key="monitor:type=Logsize,script=count_sheep_Polly">
          <ref bean="countSheepPollyLogSizeMonitor"/>
        </entry>
      </map>
    </property>
    <property name="timers">
      <map>
        <entry key="timer:type=heartbeat,script=count_sheep_Polly">
          <ref bean="countSheepPollyHeartbeatTimer"/>
        </entry>
      </map>
    </property>
    <property name="filters">
      <map>
        <entry key="monitor:type=Status,script=count_sheep_Polly">
          <ref bean="countSheepPollyScriptNotificationFilter"/>
        </entry>
        <entry key="monitor:type=Logsize,script=count_sheep_Polly">
          <ref bean="countSheepPollyScriptNotificationFilter"/>
        </entry>
        <entry key="timer:type=heartbeat,script=count_sheep_Polly">
          <ref bean="countSheepPollyScriptNotificationFilter"/>
        </entry>
      </map>
    </property>
    <property name="listeners">
      <map>
        <entry key="monitor:type=Status,script=count_sheep_Polly">
          <ref bean="emailNotifierListener"/>
        </entry>
        <entry key="monitor:type=Logsize,script=count_sheep_Polly">
          <ref bean="emailNotifierListener"/>
        </entry>
        <entry key="timer:type=heartbeat,script=count_sheep_Polly">
          <ref bean="cprListener"/>
        </entry>
      </map>
    </property>
  </bean>

  <!-- non-script MBeans -->
  <bean id="htmlAdapterServer" class="com.sun.jdmk.comm.HtmlAdaptorServer">
    <property name="port" value="8081"/>
  </bean>
  
  <bean id="rssAdapterServer" class="com.mycompany.myapp.RssAdapterServer">
    <property name="port" value="9081"/>
    <property name="agentHostname" value="localhost:8081"/>
    <property name="objectName" value="adapter:protocol=RSS"/>
  </bean>
    
  <!-- and finally, the agent that contains them all -->
  <bean id="scriptAgent" class="com.mycompany.myapp.ScriptAgent" destroy-method="destroy">
    <property name="htmlAdapterServer" ref="htmlAdapterServer"/>
    <property name="htmlAdapterServerName" value="adapter:protocol=HTTP"/>
    <property name="rssAdapterServer" ref="rssAdapterServer"/>
    <property name="scriptAdapters">
      <list>
        <ref bean="countSheepDollyScriptAdapter"/>
        <ref bean="countSheepPollyScriptAdapter"/>
      </list>
    </property>
  </bean>
  
</beans>

The configuration enabled code

As a result of the changes I described above, most of the Java files had to be touched. So rather than trying to just show you the diffs, it would be better for me and for you to see all the files that changed. The only two classes that did not change were the ScriptAdapterMBean here and the RssAdapterMBean, which you can get from the pages they are linked to. The rest of the components are listed in logical top-down manner below:

ScriptAgent.java

// Source: src/main/java/com/mycompany/myapp/ScriptAgent.java
package com.mycompany.myapp;

import java.util.Date;
import java.util.List;
import java.util.Map;

import javax.management.MBeanServer;
import javax.management.MBeanServerFactory;
import javax.management.NotificationListener;
import javax.management.ObjectName;
import javax.management.monitor.Monitor;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.beans.factory.annotation.Required;
import org.springframework.context.support.ClassPathXmlApplicationContext;

import com.sun.jdmk.comm.HtmlAdaptorServer;

public class ScriptAgent {

  private static final Log LOG = LogFactory.getLog(ScriptAgent.class);
  
  private HtmlAdaptorServer htmlAdapterServer;
  private String htmlAdapterServerName;
  private RssAdapterServer rssAdapterServer;
  private List<ScriptAdapter> scriptAdapters;

  private MBeanServer server;

  @Required
  public void setHtmlAdapterServer(HtmlAdaptorServer htmlAdapterServer) {
    this.htmlAdapterServer = htmlAdapterServer;
  }

  @Required
  public void setHtmlAdapterServerName(String htmlAdapterServerName) {
    this.htmlAdapterServerName = htmlAdapterServerName;
  }

  @Required
  public void setRssAdapterServer(RssAdapterServer rssAdapterServer) {
    this.rssAdapterServer = rssAdapterServer;
  }

  @Required
  public void setScriptAdapters(List<ScriptAdapter> scriptAdapters) {
    this.scriptAdapters = scriptAdapters;
  }
  
  protected void init() throws Exception {
    server = MBeanServerFactory.createMBeanServer();
    // load all script adapters
    for (ScriptAdapter scriptAdapter : scriptAdapters) {
      server.registerMBean(scriptAdapter, 
        new ObjectName(scriptAdapter.getObjectName()));
      registerMonitors(scriptAdapter);
      registerTimer(scriptAdapter);
      scriptAdapter.start();
    }
    // load HTML adapter
    server.registerMBean(htmlAdapterServer, 
      new ObjectName(htmlAdapterServerName));
    htmlAdapterServer.start();
    // load RSS Adapter
    server.registerMBean(rssAdapterServer, 
      new ObjectName(rssAdapterServer.getObjectName()));
    rssAdapterServer.start();
  }
  
  protected void destroy() throws Exception {
    for (ScriptAdapter scriptAdapter : scriptAdapters) {
      // stop all script services
      scriptAdapter.kill();
    }
    rssAdapterServer.stop();
    htmlAdapterServer.stop();
  }
  
  private void registerMonitors(ScriptAdapter scriptAdapter) throws Exception {
    Map<String,Monitor> monitors = scriptAdapter.getMonitors();
    for (String key : monitors.keySet()) {
      ObjectName monitorObjectName = new ObjectName(key);
      Monitor monitor = monitors.get(key);
      server.registerMBean(monitor, monitorObjectName);
      monitor.start();
      // attach filter and listener
      NotificationListener listener = scriptAdapter.getListener(key);
      if (listener != null) {
        server.addNotificationListener(monitorObjectName, listener, 
          scriptAdapter.getFilter(key), scriptAdapter.getObjectName());
      }
    }
  }

  private void registerTimer(ScriptAdapter scriptAdapter) throws Exception {
    Map<String,MyTimer> timers = scriptAdapter.getTimers();
    for (String key : timers.keySet()) {
      ObjectName timerObjectName = new ObjectName(key);
      MyTimer timer = timers.get(key);
      timer.addNotification(timer.getNotificationType(), 
        scriptAdapter.getObjectName(), null, 
        new Date(), timer.getIntervalMillis());
      server.registerMBean(timer, timerObjectName);
      timer.start();
      NotificationListener listener = scriptAdapter.getListener(key);
      if (listener != null) {
        server.addNotificationListener(timerObjectName, listener, 
          scriptAdapter.getFilter(key), scriptAdapter.getObjectName());
      }
    }
  }

  public static void main(String[] args) {
    ClassPathXmlApplicationContext context = 
      new ClassPathXmlApplicationContext("classpath:applicationContext.xml");
    context.registerShutdownHook(); // calls all bean destroy-methods
    final ScriptAgent agent = (ScriptAgent) context.getBean("scriptAgent");
    try {
      agent.init();
    } catch (Exception e) {
      LOG.error(e);
    }
  }
}

RssAdapterServer.java

// Source: src/main/java/com/mycompany/myapp/RssAdapterServer.java
package com.mycompany.myapp;

import java.io.IOException;
import java.io.PrintWriter;
import java.net.URL;
import java.net.URLEncoder;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;

import javax.management.MBeanServer;
import javax.management.MBeanServerFactory;
import javax.management.ObjectName;
import javax.management.Query;
import javax.management.QueryExp;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.mortbay.jetty.Handler;
import org.mortbay.jetty.HttpStatus;
import org.mortbay.jetty.Request;
import org.mortbay.jetty.Server;
import org.mortbay.jetty.handler.AbstractHandler;
import org.springframework.beans.factory.annotation.Required;

import com.sun.syndication.feed.WireFeed;
import com.sun.syndication.feed.synd.SyndContent;
import com.sun.syndication.feed.synd.SyndContentImpl;
import com.sun.syndication.feed.synd.SyndEntry;
import com.sun.syndication.feed.synd.SyndEntryImpl;
import com.sun.syndication.feed.synd.SyndFeed;
import com.sun.syndication.feed.synd.SyndFeedImpl;
import com.sun.syndication.io.FeedException;
import com.sun.syndication.io.WireFeedOutput;

public class RssAdapterServer implements RssAdapterServerMBean {

  private final Log log = LogFactory.getLog(getClass());
  
  private class StatusTriple {
    public ObjectName script;
    public URL httpUrl;
    public String status;
  };
  
  private int port;
  private String objectName;
  private String agentHostname;
  
  private Server rssServer;
  
  @Required
  public void setPort(int port) {
    this.port = port;
  }

  @Required
  public void setAgentHostname(String agentHostname) {
    this.agentHostname = agentHostname;
  }

  @Required
  public void setObjectName(String objectName) {
    this.objectName = objectName;
  }

  public String getObjectName() {
    return objectName;
  }

  public void start() {
    Handler handler = new AbstractHandler() {
      public void handle(String target, HttpServletRequest request,
          HttpServletResponse response, int dispatch) throws IOException,
          ServletException {
        response.setContentType("text/xml");
        PrintWriter writer = response.getWriter();
        writer.println(getScriptStatusRss());
        writer.flush();
        writer.close();
        response.setStatus(HttpStatus.ORDINAL_200_OK);
        ((Request) request).setHandled(true);
      }
    };
    this.rssServer = new Server(port);
    rssServer.setHandler(handler);
    try {
      rssServer.start();
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

  public void stop() {
    try {
      rssServer.stop();
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

  private final String getScriptStatusRss() {
    List<StatusTriple> triples = new ArrayList<StatusTriple>();
    List<MBeanServer> mbeanServers = MBeanServerFactory.findMBeanServer(null);
    if (mbeanServers == null || mbeanServers.size() == 0) {
      log.warn("No MBean servers found in JVM");
      return getRss(triples);
    }
    MBeanServer mbeanServer = mbeanServers.get(0);
    QueryExp query = Query.isInstanceOf(Query.value(ScriptAdapter.class.getName()));
    Set<ObjectName> objectNames = mbeanServer.queryNames(null, query);
    for (ObjectName objectName : objectNames) {
      try {
        StatusTriple triple = new StatusTriple();
        triple.script = objectName;
        triple.status = (String) mbeanServer.getAttribute(objectName, "Status");
        triple.httpUrl = new URL("http://" + agentHostname + 
          "/ViewObjectRes//" + URLEncoder.encode(objectName.getCanonicalName(), "UTF-8"));
        triples.add(triple);
      } catch (Exception e) {
        log.error("Cannot invoke getStatus on " + objectName.getCanonicalName(), e);
        continue;
      }
    }
    log.info("Reporting on " + triples.size() + " MBeans");
    return getRss(triples);
  }
  
  @SuppressWarnings("unchecked")
  private String getRss(List<StatusTriple> triples) {
    SyndFeed feed = new SyndFeedImpl();
    feed.setFeedType("rss_2.0");
    feed.setTitle("Status of Scripts running on: " + agentHostname);
    feed.setDescription("Status of scripts running on: " + agentHostname);
    feed.setLink("http://localhost:" + port);
    for (StatusTriple triple : triples) {
      SyndEntry entry = new SyndEntryImpl();
      entry.setTitle(triple.script.getCanonicalName());
      entry.setLink(triple.httpUrl.toExternalForm());
      SyndContent description = new SyndContentImpl();
      description.setType("text/plain");
      description.setValue(triple.status);
      entry.setDescription(description);
      feed.getEntries().add(entry);
    }
    WireFeedOutput outputter = new WireFeedOutput();
    WireFeed wirefeed = feed.createWireFeed("rss_2.0");
    try {
      return outputter.outputString(wirefeed);
    } catch (FeedException e) {
      log.error("Feed exception trying to deserialize to RSS", e);
      return "";
    }
  }
}

ScriptAdapter.java

Notice how the references to the monitors and timers have been moved from the ScriptAgent into the individual ScriptAdapters.

// Source: src/main/java/com/mycompany/myapp/ScriptAdapter.java
package com.mycompany.myapp;

import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.Map;

import javax.management.NotificationFilter;
import javax.management.NotificationListener;
import javax.management.monitor.Monitor;

import org.apache.commons.io.FileUtils;
import org.apache.commons.lang.StringUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.beans.factory.annotation.Required;
import org.springframework.jmx.JmxException;

public class ScriptAdapter implements ScriptAdapterMBean {

  private final Log log = LogFactory.getLog(getClass());
  
  private String path;
  private String args;
  private String pidfileName;
  private String logfileName;
  private String errfileName;
  private String objectName;
  private Map<String,Monitor> monitors;
  private Map<String,MyTimer> timers;
  private Map<String,NotificationFilter> filters;
  private Map<String,NotificationListener> listeners;

  private Process process;

  @Required
  public void setPath(String path) {
    this.path = path;
  }

  @Required
  public void setArgs(String args) {
    this.args = args;
  }

  @Required
  public void setPidfileName(String pidfileName) {
    this.pidfileName = pidfileName;
  }

  @Required
  public void setLogfileName(String logfileName) {
    this.logfileName = logfileName;
  }

  @Required
  public void setErrfileName(String errfileName) {
    this.errfileName = errfileName;
  }

  @Required
  public void setObjectName(String objectName) {
    this.objectName = objectName;
  }

  public String getObjectName() {
    return objectName;
  }

  @Required
  public void setMonitors(Map<String,Monitor> monitors) {
    this.monitors = monitors;
  }

  public Map<String,Monitor> getMonitors() {
    return monitors;
  }
  
  @Required
  public void setTimers(Map<String,MyTimer> timers) {
    this.timers = timers;
  }
  
  public Map<String,MyTimer> getTimers() {
    return timers;
  }

  @Required
  public void setFilters(Map<String,NotificationFilter> filters) {
    this.filters = filters;
  }

  public Map<String,NotificationFilter> getFilters() {
    return filters;
  }
  
  public NotificationFilter getFilter(String eventPublisher) {
    return filters.get(eventPublisher);
  }
  
  @Required
  public void setListeners(Map<String,NotificationListener> listeners) {
    this.listeners = listeners;
  }

  public Map<String,NotificationListener> getListeners() {
    return listeners;
  }
  
  public NotificationListener getListener(String eventPublisher) {
    return listeners.get(eventPublisher);
  }
  
  /**
   * Checks for existence of the PID file. Uses naming conventions
   * to locate the correct pid file.
   */
  public boolean isRunning() {
    File pidfile = new File(pidfileName);
    return (pidfile.exists());
  }

  /**
   * If isRunning, then status == RUNNING.
   * If isRunning and .err file exists, then status == ERROR
   * If !isRunning and .err file does not exist, then status == READY 
   */
  public String getStatus() {
    File errorfile = new File(errfileName);
    if (errorfile.exists()) {
      return "ERROR";
    } else {
      if (isRunning()) {
        return "RUNNING";
      } else {
        return "READY";
      }
    }
  }

  public String getLogs() {
    if ("ERROR".equals(getStatus())) {
      File errorfile = new File(errfileName);
      try {
        return FileUtils.readFileToString(errorfile, "UTF-8");
      } catch (IOException e) {
        throw new JmxException("IOException getting error file", e);
      }
    } else {
      try {
        Process tailProcess = Runtime.getRuntime().exec(
          StringUtils.join(new String[] {"/usr/bin/tail", "-10", logfileName}, " "));
        tailProcess.waitFor();
        BufferedReader console = new BufferedReader(
          new InputStreamReader(tailProcess.getInputStream()));
        StringBuilder consoleBuffer = new StringBuilder();
        String line;
        while ((line = console.readLine()) != null) {
          consoleBuffer.append(line).append("\n");
        }
        console.close();
        tailProcess.destroy();
        return consoleBuffer.toString();
      } catch (IOException e) {
        e.printStackTrace();
        throw new JmxException("IOException getting log file", e);
      } catch (InterruptedException e) {
        throw new JmxException("Tail interrupted", e);
      }
    }
  }

  /**
   * Returns the difference between the PID file creation and the current
   * system time.
   */
  public long getRunningTime() {
    if (isRunning()) {
      File pidfile = new File(pidfileName);
      return System.currentTimeMillis() - pidfile.lastModified();
    } else {
      return 0L;
    }
  }

  /**
   * Returns the current size of the log file in bytes.
   * @return the current size of the log file.
   */
  public long getLogFilesize() {
    File logfile = new File(logfileName);
    return logfile.length();
  }

  public void start() {
    try {
      if (! isRunning()) {
        log.info("Starting script: " + objectName);
        process = Runtime.getRuntime().exec(
          StringUtils.join(new String[] {path, args}, " "));
        // we don't wait for it to complete, just start it
      } else {
        log.info("Attempted start, but script: " + objectName + " already started");
      }
    } catch (IOException e) {
      throw new JmxException("IOException starting process", e);
    }
  }

  public void kill() {
    if (isRunning()) {
      File pidfile = new File(pidfileName);
      log.info("Killing script: " + objectName);
      try {
        String pid = FileUtils.readFileToString(pidfile, "UTF-8");
        Runtime.getRuntime().exec(StringUtils.join(new String[] {
          "/usr/bin/kill", "-9", pid}, " "));
        if (process != null) {
          // remove hanging references
          process.destroy();
        }
        pidfile.delete();
      } catch (IOException e) {
        throw new JmxException("IOException killing process", e);
      }
    }
  }
}

MyLowGaugeMonitor

As stated before, this is a hack to get around the GaugeMonitor's non monadic setter that Spring requires.

// Source: src/main/java/com/mycompany/myapp/MyLowGaugeMonitor.java
package com.mycompany.myapp;

import javax.management.monitor.GaugeMonitor;

public class MyLowGaugeMonitor extends GaugeMonitor {

  public void setThresholdLowerBound(Long thresholdLowerBound) {
    super.setThresholds(Long.MAX_VALUE, thresholdLowerBound);
  }
}

MyTimer.java

Wraps the JMX Timer bean with this bean to allow setting some parameters that we can set from Spring.

// Source: src/main/java/com/mycompany/myapp/MyTimer.java
package com.mycompany.myapp;

import java.util.Date;

import javax.management.timer.Timer;

import org.springframework.beans.factory.annotation.Required;

public class MyTimer extends Timer {

  private String notificationType;
  private long intervalMillis;
  
  @Required
  public void setNotificationType(String notificationType) {
    this.notificationType = notificationType;
  }
 
  public String getNotificationType() {
    return notificationType;
  }
  
  @Required
  public void setIntervalMillis(long intervalMillis) {
    this.intervalMillis = intervalMillis;
  }
  
  public long getIntervalMillis() {
    return intervalMillis;
  }
}

ScriptNotificationFilter.java

Not much has changed here, but we were able to make the messages more consistent when we rewrote our code for Spring, so there are changes in this class to recognize the consistent messages.

// Source: src/main/java/com/mycompany/myapp/ScriptNotificationFilter.java
package com.mycompany.myapp;

import javax.management.Notification;
import javax.management.NotificationFilter;
import javax.management.monitor.MonitorNotification;
import javax.management.timer.TimerNotification;

import org.springframework.beans.factory.annotation.Required;

public class ScriptNotificationFilter implements NotificationFilter {

  private static final long serialVersionUID = 6299049832726848968L;

  private String objectName;

  @Required
  public void setObjectName(String objectName) {
    this.objectName = objectName;
  }
  
  /**
   * Is the notification meant for this script?
   */
  public boolean isNotificationEnabled(Notification notification) {
    if (notification instanceof MonitorNotification) {
      return objectName.equals(
        ((MonitorNotification) notification).getObservedObject().getCanonicalName());
    } else if (notification instanceof TimerNotification) {
      return objectName.equals(notification.getMessage());
    } else {
      // unknown notification type
      return false;
    }
  }
}

EmailNotifierListener.java

Email addresses and script mappings have been moved out into Spring configuration, making this class more reusable as more scripts are added to a JMX server.

// Source: src/main/java/com/mycompany/myapp/EmailNotifierListener.java
package com.mycompany.myapp;

import java.util.Date;
import java.util.Map;

import javax.management.Notification;
import javax.management.NotificationListener;
import javax.management.monitor.MonitorNotification;

import org.springframework.beans.factory.annotation.Required;

public class EmailNotifierListener implements NotificationListener {

  private Map<String,String> scriptAlarmEmails;
  
  @Required
  public void setScriptAlarmEmails(Map<String,String> emails) {
    this.scriptAlarmEmails = emails;
  }
  
  public void handleNotification(Notification notification, Object handback) {
    if (notification instanceof MonitorNotification) {
      MonitorNotification monitorNotification = 
        (MonitorNotification) notification;
      String observedAttribute = monitorNotification.getObservedAttribute();
      String observedScriptName = 
        monitorNotification.getObservedObject().getCanonicalName();
      String emailTo = scriptAlarmEmails.get(observedScriptName);
      String emailSubject = "Alarm for [" + observedScriptName + ":" + 
        monitorNotification.getObservedAttribute() + "]";
      StringBuilder emailBody = new StringBuilder();
      if (observedAttribute.equals("Status")) {
        emailBody.append("Script [").
          append(observedScriptName).
          append("] reported ERROR.");
      } else if (observedAttribute.equals("LogFilesize")) {
        emailBody.append("Script [").
          append(observedScriptName).
          append("] is HUNG.");
      } else {
        // nothing to do for now, place holder for future notifications
      }
      sendEmail(emailTo, emailSubject, emailBody);
    }
  }

  private void sendEmail(String emailTo, String emailSubject, 
      StringBuilder emailBody) {
    System.out.println("EmailNotifierListener: sending email...");
    System.out.println("From: scriptmanager@clones-r-us.com");
    System.out.println("To: " + emailTo);
    System.out.println("Date: " + new Date());
    System.out.println("Subject: " + emailSubject);
    System.out.println("--");
    System.out.println(emailBody.toString());
    System.out.println("--");
  }
}

CprListener.java

// Source: src/main/java/com/mycompany/myapp/CprListener.java
package com.mycompany.myapp;

import java.util.List;

import javax.management.MBeanServer;
import javax.management.MBeanServerFactory;
import javax.management.Notification;
import javax.management.NotificationListener;
import javax.management.ObjectName;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;

public class CprListener implements NotificationListener {

  private final Log log = LogFactory.getLog(getClass());
  
  public void handleNotification(Notification notification, Object handback) {
    if ("heartbeat".equals(notification.getType())) {
      List<MBeanServer> servers = MBeanServerFactory.findMBeanServer(null);
      if (servers.size() == 0) {
        log.warn("Error: no MBean server found in this machine");
        return;
      }
      MBeanServer server = servers.get(0);
      String scriptObjectName = notification.getMessage();
      try {
        log.info("Performing CPR on " + scriptObjectName);
        server.invoke(new ObjectName(scriptObjectName), "start", null, null);
      } catch (Exception e) {
        log.error("Could not handle heartbeat timer notification for script:" + 
          scriptObjectName + ". Manual intervention requested!", e);
      }
    }
  }
}

Spring's JMX extensions

I did look at the Spring reference guide to see if I could use some of Spring's JMX extensions - I mean, I was using Spring already, so may as well take advantage of the conveniences it has to offer. One thing I liked especially was the annotation driven approach to declaring MBeans, potentially negating the need to have the MBean interfaces. I could not get it to work though; when I started up the MBean server (this time having my RssAdapter and ScriptAdapter not implementing the corresponding MBeans and with annotations declared, and with the set of Annotation processor declarations (from the Spring Reference Manual) in the configuration), it complained that my beans were not valid Managed beans.

Another neat feature is being able to configure and build your agent declaratively - this would have been neat but it was not clear how I could implement the custom code for setting up Monitors and Timers in the ScriptAdapters using that approach.

Conclusion

I think I have pretty much worked the JMX interface for scripts topic to death, at least for now. As you can see, what Spring gives with one hand in terms of Java code conciseness, it takes away with the other in terms of corresponding XML bloat. But this information has to go somewhere, and at least in my opinion, you are much better off having this declared in XML configuration than having to change your code every time you want to add a new monitor or change some kind of configuration.

Salmon Run

Saturday, September 13, 2008

JMX for Scripts : Spring Configuration