Thursday, December 23, 2010

Quick and Dirty Reporting with Awk

A few years ago, I wrote about starting to use awk after a long time. Over the last couple of years, I've used awk on and off, although not very frequently, and never beyond the basic pattern described in that post.

Last week, I described an AspectJ aspect that wrote out timing information into the server logs. I initially thought of using OpenOffice Calc, but then stumbled upon Jadu Saikia's post on implementing group-by with awk, and I realized that awk could do the job just as easily, and would be easier to use (for multiple invocations).

Here is the script I came up with.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/usr/bin/env awk -f
# Builds a report out of aggregate elapsed time data similar to the Spring 
# StopWatch prettyPrint report.
# Usage:
# my_profile_report.awk [-v qt=query_term] filename
#
BEGIN {
  FS="|";
}
(qt == "" || qt == $3) {
  counts[$4]++;
  times[$4] += $5;
  if (mins[$4] + 0 != 0) {
    if (mins[$4] > $5) {
      mins[$4] = $5;
    }
  } else {
    mins[$4] = $5;
  }
  if (maxs[$4] + 0 != 0) {
    if (maxs[$4] < $5) {
      maxs[$4] = $5;
    }
  } else {
    maxs[$4] = $5;
  }
  totals += $5;
}
END {
  totavg = 0;
  for (t in times) {
    totavg += times[t]/counts[t];
  }
  printf("---------------------------------------------------------------------\n");
  printf("%-32s %8s %8s %8s %8s\n", "Controller", "Avg(ms)", "%", "Min(ms)", "Max(ms)");
  printf("---------------------------------------------------------------------\n");
  for (t in times) {
    avg = times[t]/counts[t];
    perc = avg * 100 / totavg;
    printf("%-32s %8.2f %8.2f %8.2f %8.2f\n", t, avg, perc, mins[t], maxs[t]);
  }
  printf("---------------------------------------------------------------------\n");
  printf("%-32s %8.2f\n", "Average Elapsed (ms):", totavg);
  printf("---------------------------------------------------------------------\n");
}

As shown in my previous post, the input file looks something like this (without the header, I just grep the server log file with the aspect name).

1
2
3
4
5
6
# aspect_name|request_uuid|query|tile_name|elapsed_time_millis
MyAspect2|ed9ce777-263e-4fe5-a8af-4ea84d78add3|some query|TileAController|132
MyAspect2|ed9ce777-263e-4fe5-a8af-4ea84d78add3|some query|TileBController|49
MyAspect2|ed9ce777-263e-4fe5-a8af-4ea84d78add3|some query|TileCController|2
MyAspect2|ed9ce777-263e-4fe5-a8af-4ea84d78add3|some query|TileDController|3
...

The script can be invoked with or without a qt parameter. Without parameters, the script computes the average, minimum and maximum elapsed times across all the queries that were given to the application during the profiling run. The report looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
sujit@cyclone:aspects$ ./my_profile_report.awk input_file
---------------------------------------------------------------------
Controller                        Avg(ms)        %  Min(ms)  Max(ms)
---------------------------------------------------------------------
TileFController                      1.42     0.02     0.00    42.00
TileAController                    187.67     2.83    27.00   685.00
TileJController                    169.97     2.56     7.00  3140.00
TileEController                      9.14     0.14     2.00    44.00
TileNController                     45.91     0.69     4.00   234.00
TileIController                    444.91     6.71     0.00  3427.00
TileDController                   1506.30    22.72   792.00 12140.00
TileMController                    184.88     2.79     0.00  3078.00
TileHController                     13.67     0.21     7.00    50.00
TileCController                     34.06     0.51    14.00   108.00
TileLController                    759.73    11.46     3.00  9921.00
TileGController                   2473.24    37.31   579.00 10119.00
TileBController                     24.48     0.37     7.00   132.00
TileKController                    773.97    11.67     0.00  8554.00
---------------------------------------------------------------------
Average Elapsed (ms):             6629.35
---------------------------------------------------------------------

If we want only the times for a specific query term, then we can specify it on the command line as shown below. If the query has been invoked multiple times, then the report will show the average of the calls for the query. In this case, there is only a single invocation for the query, so the minimum and maximum times are redundant.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
sujit@cyclone:aspects$ ./my_profile_report.awk -v qt=some_query input_file
---------------------------------------------------------------------
Controller                        Avg(ms)        %  Min(ms)  Max(ms)
---------------------------------------------------------------------
TileFController                      0.00     0.00     0.00     0.00
TileAController                    106.00     4.57   106.00   106.00
TileJController                      7.00     0.30     7.00     7.00
TileEController                     10.00     0.43    10.00    10.00
TileNController                      8.00     0.34     8.00     8.00
TileIController                      0.00     0.00     0.00     0.00
TileDController                   1309.00    56.42  1309.00  1309.00
TileMController                      0.00     0.00     0.00     0.00
TileHController                     12.00     0.52    12.00    12.00
TileCController                     25.00     1.08    25.00    25.00
TileLController                     64.00     2.76    64.00    64.00
TileGController                    767.00    33.06   767.00   767.00
TileBController                     12.00     0.52    12.00    12.00
TileKController                      0.00     0.00     0.00     0.00
---------------------------------------------------------------------
Average Elapsed (ms):             2320.00
---------------------------------------------------------------------

The script uses awk's associative arrays and the post processing block. Before this script, I had read about awk's BEGIN/END blocks but didn't know why I would want to use them, and I didn't know about awk's associative arrays at all. Thanks to Daniel Robbin's three part series - Common threads: Awk by example, Part 1, 2 and 3, I now have a greater understanding and appreciation of awk.

Friday, December 17, 2010

Profiling Spring Tiles with Custom Aspects

Introduction

One way of "componentizing" a web page is by composing it out of "tiles". Each tile encapsulates the logic needed to render itself. Once the tiles are built, they can be easily mixed and matched to provide different page renderings using different groupings of tiles. The Spring Framework provides support for tiles via the Apache Struts Tiles subproject.

With Spring, the page itself is controlled by a Spring Controller. When a request comes in, the controller instantiates a Model object, optionally does some pre-processing on it, sticks the model into the request, and sends off the request to its JSP. The JSP contains one or more tiles:insert calls referencing named tiles in the tiles-def.xml file. Each named tile is associated with a Tile Controller and a JSP. As the main JSP renders, it calls upon each of its tiles to render themselves, which invokes the associated Tile Controller. Each Tile Controller grabs the model off the request, does its own processing to populate the model, then sticks the model back into the request, where it is picked up by its JSP for rendering.

Motivation

Recently, one such page some of us have been working on was rendering real slow. So I figured it would be useful to get some timing information on how long each individual Tile Controller was taking, and optimize them. Unfortunately, I could not think of a good way to do this with the standard Spring approach of wrapping beans with interceptor proxies, since the Tile Controllers are defined as class names in the tiles-def.xml configuration, not bean references.

So the solution I came up with was to write an AspectJ aspect that would be woven into the application's JAR file at compile time to produce the "profiling" version of the JAR. This "profiling" JAR would be swapped into the WEB-INF/lib sub-directory of the servlet container's (in our case Tomcat) context directory for the application, along with the AspectJ Runtime JAR, followed by a container restart.

With this approach, there is no change to the application code. The aspect code is in a completely different project. Once profiling is done and we have our data, we simply replace the application's JAR file with the original one.

Since I was new to AspectJ, I had to do a bit of reading and experimenting before I got this stuff to work. There is a lot of AspectJ information available on the web, but none of them describes in detail how one would go about doing something like this. It seems to me that a lot of people would want to do this if they knew how, so hopefully this post would help someone. If you can think of other ways to do this, would appreciate hearing from you.

Steps

Here are the steps to build and compile-time weave an aspect into a web application's JAR file.

  • Install AspectJ and configure (one time).
  • Create an AspectJ project and write the Aspect.
  • Generate the source JAR (ant jar) from the web application to be profiled.
  • Compile and Weave the Aspect into the source JAR to produce a woven version of the JAR.
  • Generate web application WAR file and deploy to Tomcat.
  • Stop Tomcat, replace the JAR file in /webapps/${app}/WEB-INF/lib with the woven version of the JAR.
  • Copy the aspectjrt.jar into the same /webapps/${app}/WEB-INF/lib directory.
  • Restart Tomcat.
  • Execute HTTP GETs against the application, profiling output is captured into the log file.

AspectJ Installation

I downloaded the latest AspectJ JAR (1.6.10 for me) from the AspectJ Download site. Following the instructions on the site, I just ran the downloaded JAR (java -jar) to install AspectJ under my /opt/aspectj1.6 (ASPECTJ_HOME) directory.

I then added $ASPECTJ_HOME/bin to my PATH environment variable so the AspectJ commands (ajc, the AspectJ analog of javac) are available on the command line.

I also had to increase the -Xmx setting for ajc (a shell script in $ASPECTJ_HOME/bin) from 64M to 256M because it was running out of memory during a compile. The resulting JAR is only slightly larger after weaving.

Creating the Aspect

I initially planned on building this as a private little hack for my own use, so I could tell how the code changes I was doing affected overall performance, so I thought of using the Spring StopWatch to report on this. The diagram below illustrates the setup I was trying to profile:

As you can see, there is a single Spring controller which hands off to a Tiles view with three tiles A, B and C. I wanted to find how much time was being spent overall, and how much time was being spent in each of the three tiles.

Capturing the elapsed time for each tile was easy. The problem was to track when all the tiles were done, so I could print the contents of the stopwatch. Since the Spring controller hands off control to the Tiles view, which then calls the Tiles Controller for each tile, the only way to do this was to do it at the beginning of the next request. Also the static StopWatch reference is not thread-safe, so its state can get corrupted if you hit the application too fast (so one request may not have finished before you hand it another one). So its kind of useful, as long as you are willing to live with its warts. Here is the aspect code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
// Source: src/profilers/MyAspect1.aj
package profilers;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.springframework.util.StopWatch;

/**
 * MyController Tile Profiler for individual use. This dumps out the 
 * stopwatch contents for a matched request on the request /following/ 
 * the one that was profiled. Because it uses a static StopWatch instance,
 * it is not thread safe, so be sure to wait a bit between calls, otherwise, 
 * you will get errors.
 */
public aspect MyAspect1 {

  private static StopWatch stopwatch = null;
  
  pointcut handleRequestInternal(HttpServletRequest request, 
      HttpServletResponse response) :
    execution(public * com.myco.myapp.MyController.handleRequestInternal(..))
    && args(request, response);

  pointcut doPerform() : 
    execution(public * com.myco.myapp.tiles..*.doPerform(..)); 

  before(HttpServletRequest request, HttpServletResponse response) : 
      handleRequestInternal(request, response) {
    if (stopwatch != null) {
      System.out.println(stopwatch.prettyPrint());
    }
    String q1 = request.getParameter("q1");
    stopwatch = new StopWatch(q1 == null ? "UNKNOWN" : q1);
  }
  
  before() : doPerform() {
    stopwatch.start(thisJoinPoint.getTarget().getClass().getSimpleName());
  }
  
  after() : doPerform() {
    if (stopwatch.isRunning()) {
      stopwatch.stop();
    }
  }
}

Applying this aspect to the JAR file and running the web application gave me output in my logs that looked like this. As you can see, the output (with names and data changed to protect the innocent and not-so-innocent) is pretty but the information is not.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
StopWatch 'Some Query': running time (millis) = 12000
-----------------------------------------
ms     %     Task name
-----------------------------------------
00870  007%  TileAController
00265  002%  TileBController
00059  000%  TileCController
00002  000%  TileDController
00003  000%  TileEController
00000  000%  TileFController
08880  074%  TileGController
00031  000%  TileHController
00209  002%  TileIController
00118  001%  TileJController
00965  008%  TileKController
00000  000%  TileLController
00003  000%  TileMController
00592  005%  TileNController
00003  000%  TileOController

To get around the two limitations of the implementation above, I decided to forego the convenience of the Spring StopWatch and capture plain elapsed times into the log file (with additional information to tie the elapsed time entries together), and then use scripts to post-process this information. My new setup looks something like this:

I use the fact that a call to Spring's modelAndView.addObject(name, value) is really the same as request.setAttribute(name, value). Before the handleRequest() call in the Spring controller, I generate a unique UUID for the request and stick it into the attribute so it can be picked up by the before() and after() advice in the Tile Controllers. Here is the second version.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
// Source: src/profilers/MyAspect2.aj
package profilers;

import java.util.UUID;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.lang.StringUtils;
import org.apache.struts.tiles.ComponentContext;

/**
 * Tile Profiler for batch use. We just capture elapsed times for
 * each tile as a CSV formatted string in the logs. The logs need to be 
 * post-processed to get profiling information.
 */
public aspect MyAspect2 {

  private static final String ATTR_REQ_UUID = "_req_uuid";
  private static final String ATTR_START_TIME = "_start_time";
  private static final String PROFILER_NAME = MyAspect2.class.getSimpleName();
  
  pointcut handleRequestInternal(HttpServletRequest request, 
    HttpServletResponse response) :
    execution(public * com.myco.myapp.MyController.handleRequestInternal(..))
    && args(request, response);

  pointcut doPerform(ComponentContext context, HttpServletRequest request, 
      HttpServletResponse response) : 
    execution(public * com.myco.myapp.tiles..*.doPerform(..))
    && args(context, request, response);

  before(HttpServletRequest request, HttpServletResponse response) : 
      handleRequestInternal(request, response) {
    UUID uuid = UUID.randomUUID();
    request.setAttribute(ATTR_REQ_UUID, uuid.toString());
  }
  
  before(ComponentContext context, HttpServletRequest request, 
      HttpServletResponse response) : doPerform(context, request, response) {
    Long startTime = System.currentTimeMillis();
    request.setAttribute(ATTR_START_TIME, startTime);
  }
  
  after(ComponentContext context, HttpServletRequest request, 
      HttpServletResponse response) : doPerform(context, request, response) {
    Long endTime = System.currentTimeMillis();
    Long startTime = (Long) request.getAttribute(ATTR_START_TIME);
    String requestUuid = (String) request.getAttribute(ATTR_REQ_UUID);
    String query = request.getParameter("q1");
    System.out.println(StringUtils.join(new String[] {
      PROFILER_NAME, requestUuid, query, 
      thisJoinPoint.getTarget().getClass().getSimpleName(),
      String.valueOf(endTime - startTime)
    }, "|"));
  }
}

Running this with a batch of URLs produces output in the Tomcat logs, which can be grepped using the PROFILER_NAME into a text file. You could then feed it into your data analysis tool of choice and slice and dice the data to produce actionable information. Here is some sample data from the file (I added the header line to make it a bit easier to read):

1
2
3
4
5
6
# aspect_name|request_uuid|query|tile_name|elapsed_time_millis
MyAspect2|ed9ce777-263e-4fe5-a8af-4ea84d78add3|some query|TileAController|132
MyAspect2|ed9ce777-263e-4fe5-a8af-4ea84d78add3|some query|TileBController|49
MyAspect2|ed9ce777-263e-4fe5-a8af-4ea84d78add3|some query|TileCController|2
MyAspect2|ed9ce777-263e-4fe5-a8af-4ea84d78add3|some query|TileDController|3
...

Compiling and Weaving

The aspect source code needs to be compiled and woven into the web application's JAR file. You will need all the JARs referenced by the web application in the classpath when you do this. I did this manually by making a copy of the build.xml or pom.xml, then creating a CLASSPATH declaration, then putting it into a bash script that looks something like this:

1
2
3
4
5
6
7
8
#!/bin/bash
CP=$ASPECTJ_HOME/lib/aspectjrt.jar:\
$ASPECTJ_HOME/lib/aspectjweaver.jar:\
#other classes in your web application's CLASSPATH
...

ajc -classpath $CP -1.6 -inpath myapp.jar -outjar myapp-woven.jar \
  src/profilers/MyAspect1.aj

Here myapp.jar is the JAR file for my web application myapp, and the output of this script is myapp-woven.jar, which you must copy to $TOMCAT_HOME/webapps/myapp/WEB-INF/lib.

References

I found these resources very helpful as I was developing the aspects described in this post.

  • AspectJ Project Documentation: I mostly read the Getting Started Guide, which gave me enough background to start reading/writing basic AspectJ code.
  • AspectJ Cookbook: I bought this book couple years ago hoping to use it to learn AspectJ, but did not have a problem to solve, so it ended up gathering dust. The shell script and the two aspects described are heavily based on recipes in this book. Better late than never, I guess :-).

Conclusion

So anyway, here is a poor man's profiler which you can whip up quickly if you know AspectJ well enough (it took me a while for the first one, but not so much time for the second one). Its also light on resources. My experience with profilers is that it takes a very long time to set up and run, since its busy collecting all sorts of information which you would probably never use. Knowing AspectJ gives you the ability to profile code in a very focused manner - definitely a useful skill to have at your disposal in my opinion.

Saturday, December 04, 2010

A PKI based CAPTCHA Implementation

I'm thinking of implementing CAPTCHA verification for an upcoming application, so I decided to check out JCaptcha, which provides libraries for image and sound CAPTCHAs, as well as an implementation you can directly use in your web application. Based on my understanding of the code from its 5-minute tutorial, the way it works is that it generates a random word, sticks it into the session and renders an image of the random word on the browser. When the user types the word back in, it matches the value in the session against the user input.

Most production systems are load balanced across a bank of servers. In such cases, the JCaptcha wiki either advises using either a distributed singleton or sticky sessions. This is because the session is used to hold on to the original generated word.

An alternative approach that eliminates the need for sessions would be to carry an encrypted version of the original word as a hidden field in the CAPTCHA challege form. Various encryption approaches could be used:

  • One-way hash - the original word is encrypted with something like MD5 and put into the hidden field. On the way back, the user's response is also encrypted and the digests compared.

  • Symmetric two-way hash - the original word is encrypted with something like DES and put into the hidden field. On the way back, we can either decrypt the hidden field back and compare to the user input, or encrypt the user input and compare with the hidden field.

  • Asymetric two-way hash - the original word is encrypted with the public part of a RSA keypair (in my case), and on the way back, decrypted with the private part, and compared to the user input.

I guess that any of these approaches would be secure enough to serve as a CAPTCHA implementation, but I hadn't used the Java PKI API before, and I figured that this would be a good time to learn. The code presented here includes a service (that manages the keypair and does the encryption and decryption), a Spring controller (that calls a JCaptcha WordToImage implementation to generate a CAPTCHA image, and methods to show and submit a CAPTCHA challenge form), and a JSPX file that can be used as part of a Spring Roo application.

Service

Asymmetric two-way hashes (around which the PKI API is based) are based on the premise that:

  d(e(m, q), p) == m

  where:
    m = the message body.
    p = private part of a keypair {p,q}.
    q = public part of a keypair {p,q}.
    d = decoding function.
    e = encoding function.

In typical usage, user A sends a message to user B with keypair {p,q} by encrypting it with B's public key q who then decrypts it with his own private key p In our case, there is only keypair, which belongs to the server. On the way out, the generated word is encrypted with the server's public key, and on the way back, the encrypted value in the hidden field is decrypted back with the server's private key.

At startup, each server loads up its keypair from a pair of key files, deployed as part of the web application. That way each server decrypts the encrypted key exactly the same way. Also, since the encrypted version is carried through the request-response cycle, a CAPTCHA generated on any one machine in the cluster can be verified on any other machine.

To generate the word, the service generates a 32 character MD5 checksum of the Date string at the instant the form is called from the browser, then pull a random substring from it between 3 to 5 characters in length. Here's the code for the service:

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
// Source: src/main/java/com/mycompany/ktm/security/CaptchaCryptoService.java
package com.mycompany.ktm.security;

import java.io.File;
import java.security.KeyFactory;
import java.security.PrivateKey;
import java.security.PublicKey;
import java.security.spec.PKCS8EncodedKeySpec;
import java.security.spec.X509EncodedKeySpec;
import java.util.Date;

import javax.crypto.Cipher;

import org.apache.commons.codec.binary.Hex;
import org.apache.commons.io.FileUtils;
import org.apache.log4j.Logger;
import org.springframework.stereotype.Service;
import org.springframework.util.DigestUtils;

@Service("captchaCryptoService")
public class CaptchaCryptoService {

  private final Logger logger = Logger.getLogger(getClass());
  
  private static final File PRIV_KEY_FILE = 
    new File("src/main/resources/captcha.pri");
  private static final File PUB_KEY_FILE = 
    new File("src/main/resources/captcha.pub");
  
  private Cipher cipher;
  private PublicKey publicKey;
  private PrivateKey privateKey;
  
  public CaptchaCryptoService() {
    try {
      byte[] pri = FileUtils.readFileToByteArray(PRIV_KEY_FILE);
      PKCS8EncodedKeySpec priSpec = new PKCS8EncodedKeySpec(pri);
      KeyFactory priKeyFactory = KeyFactory.getInstance("RSA");
      this.privateKey = priKeyFactory.generatePrivate(priSpec);
      byte[] pub = FileUtils.readFileToByteArray(PUB_KEY_FILE);
      X509EncodedKeySpec pubSpec = new X509EncodedKeySpec(pub);
      KeyFactory pubKeyFactory = KeyFactory.getInstance("RSA");
      this.publicKey = pubKeyFactory.generatePublic(pubSpec);
      this.cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }
  
  public String getEncodedCaptchaString() {
    byte[] currentTime = new Date().toString().getBytes();
    String md5 = DigestUtils.md5DigestAsHex(currentTime);
    int start = random(0, 32, 100);
    int length = random(3, 5, 10);
    String plainText = slice(md5, start, length);
    return encrypt(plainText);
  }

  public String getDecodedCaptchaString(String encoded) {
    String decoded = decrypt(encoded);
    return decoded;
  }
  
  public boolean validate(String encoded, String response) {
    return response.equalsIgnoreCase(getDecodedCaptchaString(encoded));
  }

  private boolean keyFilesExist() {
    return PRIV_KEY_FILE.exists() && PUB_KEY_FILE.exists();
  }

  private String encrypt(String plainText) {
    try {
      cipher.init(Cipher.ENCRYPT_MODE, this.publicKey);
      byte[] encrypted = cipher.doFinal(plainText.getBytes());
      Hex hex = new Hex();
      return new String(hex.encode(encrypted));
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

  private String decrypt(String encodedText) {
    try {
      cipher.init(Cipher.DECRYPT_MODE, this.privateKey);
      Hex hex = new Hex();
      byte[] encrypted = hex.decode(encodedText.getBytes());
      byte[] decrypted = cipher.doFinal(encrypted);
      return new String(decrypted);
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }
  
  private int random(int min, int max, int multiplier) {
    while (true) {
      double rand = Math.round(Math.random() * multiplier);
      if (rand >= min && rand <= max) {
        return (int) rand;
      }
    }
  }

  private String slice(String str, int start, int length) {
    StringBuilder buf = new StringBuilder();
    int pos = start;
    for (int i = 0; i < length; i++) {
      if (pos == str.length()) {
        pos = 0;
      }
      buf.append(str.charAt(pos));
      pos++;
    }
    return buf.toString();
  }
}

To generate the key pair, I just used the following Java code, although you can also use ssh-keygen or something similar to generate them from the command line.

1
2
3
4
5
6
7
8
9
    KeyPairGenerator keyPairGenerator = KeyPairGenerator.getInstance("RSA");
    keyPairGenerator.initialize(1024);
    KeyPair keyPair = keyPairGenerator.generateKeyPair();
    FileUtils.writeByteArrayToFile(
      PRIV_KEY_FILE, keyPair.getPrivate().getEncoded());
    FileUtils.writeByteArrayToFile(
      PUB_KEY_FILE, keyPair.getPublic().getEncoded());
    this.privateKey = keyPair.getPrivate();
    this.publicKey = keyPair.getPublic();

Controller

I want to add the CAPTCHA into a Spring Roo application (which does not exist at the moment), so I just tried to add it into an existing app to test out the code. The controller code consists of three methods:

  1. Load the CAPTCHA form - in real-life, this would be a create/edit form for a domain object with the CAPTCHA stuff added to it. For testing, all it does is show the image and offer a single text box.
  2. Generate the CAPTCHA image - Generates the image using JCaptcha's WordToImage implementation and writes to as a PNG file.
  3. Handle the CAPTCHA form submit - as before, this would be part of another form in real-life. This takes the encrypted string and the user input and validates it against each other.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
// Source: src/main/java/com/mycompany/ktm/web/CaptchaController.java
package com.mycompany.ktm.web;

import java.awt.Color;
import java.awt.image.BufferedImage;
import java.net.BindException;

import javax.annotation.PostConstruct;
import javax.imageio.ImageIO;
import javax.servlet.ServletOutputStream;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.apache.commons.lang.StringUtils;
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.validation.BindingResult;
import org.springframework.validation.ObjectError;
import org.springframework.web.bind.ServletRequestUtils;
import org.springframework.web.bind.annotation.ModelAttribute;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.servlet.ModelAndView;

import com.mycompany.ktm.fbo.CaptchaForm;
import com.mycompany.ktm.security.CaptchaCryptoService;
import com.octo.captcha.component.image.backgroundgenerator.EllipseBackgroundGenerator;
import com.octo.captcha.component.image.fontgenerator.TwistedRandomFontGenerator;
import com.octo.captcha.component.image.textpaster.SimpleTextPaster;
import com.octo.captcha.component.image.wordtoimage.ComposedWordToImage;
import com.octo.captcha.component.image.wordtoimage.WordToImage;

@RequestMapping(value="/captcha/**")
@Controller
public class CaptchaController {

  private final Logger logger = Logger.getLogger(getClass());
  
  @Autowired private CaptchaCryptoService captchaCryptoService;

  private WordToImage imageGenerator;
  
  @ModelAttribute("captchaForm")
  public CaptchaForm formBackingObject() {
    return new CaptchaForm();
  }
  
  @PostConstruct
  protected void init() throws Exception {
    this.imageGenerator = new ComposedWordToImage(
      new TwistedRandomFontGenerator(40, 50),
      new EllipseBackgroundGenerator(150, 75),
      new SimpleTextPaster(3, 5, Color.YELLOW)
    );
  }
  
  @RequestMapping(value = "/captcha/form", method = RequestMethod.GET)
  public ModelAndView form(
      @RequestParam(value="ec", required=false) String ec,
      @ModelAttribute("captchaForm") CaptchaForm form,
      BindingResult result) {
    ModelAndView mav = new ModelAndView();
    if (StringUtils.isEmpty(ec)) {
      form.setEc(captchaCryptoService.getEncodedCaptchaString());
    } else {
      result.addError(new ObjectError("dc", "Bzzt! You missed! Try again!"));
      form.setEc(ec);
    }
    mav.addObject("form", form);
    mav.addObject("ec", form.getEc());
    mav.setViewName("captcha/create");
    return mav;
  }

  @RequestMapping(value="/captcha/image", method=RequestMethod.GET)
  public ModelAndView image(HttpServletRequest req, HttpServletResponse res)
      throws Exception {
    String encoded = ServletRequestUtils.getRequiredStringParameter(req, "ec");
    String decoded = captchaCryptoService.getDecodedCaptchaString(encoded);
    BufferedImage image = imageGenerator.getImage(decoded);
    res.setHeader("Cache-Control", "no-store");
    res.setHeader("Pragma", "no-cache");
    res.setDateHeader("Expires", 0);
    res.setContentType("image/png");
    ServletOutputStream ostream = res.getOutputStream();
    ImageIO.write(image, "png", ostream);
    ostream.flush();
    ostream.close();
    return null;
  }

  @RequestMapping(value = "/captcha/validate", method = RequestMethod.POST)
  public ModelAndView validate(
      @ModelAttribute("captchaForm") CaptchaForm form,
      BindingResult result) {
    ModelAndView mav = new ModelAndView();
    String ec = form.getEc();
    String dc = form.getDc();
    if (captchaCryptoService.validate(ec, dc)) {
      mav.setViewName("redirect:/captcha/form");
    } else {
      mav.setViewName("redirect:/captcha/form?ec=" + ec);
    }
    return mav;
  }
}

The form backing object is a plain POJO with only two fields. If we are using a domain object as our form backing object, then I think we can just add these two fields as @Transient, although I haven't tried it. Theres not much to this POJO, I show it below, with getters and setters omitted.

1
2
3
4
5
6
7
8
9
// Source: src/main/java/com/mycompany/ktm/fbo/CaptchaForm.java
package com.mycompany.ktm.fbo;

public class CaptchaForm {

  private String ec; // encrypted string
  private String dc; // plaintext string
  ...  
}

And finally, the JSPX file. This is a copy of one of the Roo generated JSPX files, with the image generated using an <img> tag, a single text field to get the user input, and a submit button.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
<div xmlns:c="http://java.sun.com/jsp/jstl/core" 
     xmlns:form="http://www.springframework.org/tags/form"
     xmlns:jsp="http://java.sun.com/JSP/Page"
     xmlns:spring="http://www.springframework.org/tags" version="2.0">
  <jsp:output omit-xml-declaration="yes"/>
  <script type="text/javascript">
    dojo.require('dijit.TitlePane');
  </script>
  <div id="_title_div">
    <spring:message var="title_msg" text="Captcha Challenge"/>
    <script type="text/javascript">
      Spring.addDecoration(new Spring.ElementDecoration({
        elementId : '_title_div', 
        widgetType : 'dijit.TitlePane', 
        widgetAttrs : {title: '${title_msg}'}
      })); 
    </script>
    <form:form action="/ktm/captcha/validate" method="POST" 
        modelAttribute="captchaForm">
      <form:errors cssClass="errors" delimiter="&lt;p/&gt;"/>
      <div id="roo_captcha_image">
        <img src="/ktm/captcha/image?ec=${ec}"/>
      </div>
      <div id="roo_captcha_dc">
        <label for="_captcha_dc">
          Enter the characters you see in the image above:
        </label>
        <form:input cssStyle="width:250px" id="_captcha_dc" 
          maxlength="5" path="dc" size="0"/>
        <br/>
        <form:errors cssClass="errors" id="_dc_error_id" path="dc"/>
      </div>
      <div class="submit" id="roo_captcha_submit">
        <spring:message code="button.save" var="save_button"/>
        <script type="text/javascript">
          Spring.addDecoration(new Spring.ValidateAllDecoration({
            elementId:'proceed', 
            event:'onclick'
          }));
        </script>
        <input id="proceed" type="submit" value="${save_button}"/>
      </div>
      <form:hidden id="_roo_captcha_ec" path="ec"/>
    </form:form>
  </div>
</div>

I also had to create a new views.xml for this controller, similar to other Roo custom controllers. It only has a single entry, here it is:

1
2
3
4
5
6
7
8
9
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE tiles-definitions 
  PUBLIC "-//Apache Software Foundation//DTD Tiles Configuration 2.1//EN" 
  "http://tiles.apache.org/dtds/tiles-config_2_1.dtd">
<tiles-definitions>
  <definition extends="default" name="captcha/create">
    <put-attribute name="body" value="/WEB-INF/views/captcha/create.jspx"/>
  </definition>
</tiles-definitions>

Putting it all together

As I said before, a "real" use-case of this would be to embed it into an edit form for a domain object - here, I am only interested in testing the functionality, so its being used standalone. Here are some screenshots demonstrating the flow:

Entering /captcha/form in my browser would take me to the challenge form with a CAPTCHA challenge image generated:

If I enter an incorrect value, it shows me an error message and regenerates the same string into a different CAPTCHA image, hopefully more readable this time. This is not the standard behavior for most CAPTCHA's I've used, they generally give you a new image of a different string in these cases. Notice that the URL now has the ec parameter which contains the encrypted string.

If I enter a correct value, it is silently accepted and the form is regenerated with a new CAPTCHA challenge.

References

Apart from the references above, I found the following links very helpful. All the APIs here (except Spring and Roo) are new to me, so I had to do a bit of reading before everything came together.

  • Bouncy Castle Crypto package - contains lots of code examples to get you started quickly with this encryption stuff. My CaptchaCryptoService is based heavily on the PublicExample.java here.

  • Using Public Key encryption in Java - contains general information about PKI and Java. I found out how to store keypairs into files and load them from my service.
  • JCaptcha and Spring Framework - contains a discussion on how to configure JCaptcha beans with Spring. I ended up building my WordToImage implementation in the Controller's @PostConstruct method, but the information in here was invaluable in figuring out whats available.