Friday, July 09, 2010

Herding Cats, Roo and the birth of KTM

This week I bring together three separate threads of thought that have been bouncing around in my head, and attempt to join them together into something approaching a cohesive whole, followed by an incomplete implementation of that whole.

Herding Cats

I have always been a bit of a sucker for self-improvement books. Early on in my career when I had to travel a lot, I would end up buying one everytime I travelled, usually at the airport/train station bookstore. I don't buy these quite so often nowadays, but a few years ago, the Agile/Pragmatic movement found me (or I found them), and I ended up buying some again. One of these books was Herding Cats: A Primer for Programmers Who Lead Programmers by J Hank Rainwater.

Despite the mixed reviews this book has received on Amazon, I found the book quite entertaining and yes, even somewhat useful. I think its a matter of perspective. You can choose to be offended by the stereotyping of programmers, or you can think of the book as a design document of how to deal with your coworkers, based on insufficient data (quite a common occurrence in our business). The rest of the data must come from the reader's environment and the design document tweaked based on this data for the book to be useful.

In the book, Rainwater describes a PC based application that he used for keeping track of what his programmers were tasked with. In the spirit of tweaking the design, I decided to build something similar, but which conforms more closely with my environment.


A few weeks ago, I attended a talk on Spring-Roo by Ramnivas Laddad at the eBig Java SIG. I had heard of Roo and knew it was the Java equivalent of Ruby-on-Rails, but did not have the time to look at further. The talk took us through setting up Roo and using it to build a web application from scratch. What the talk did for me was get me out of my inertia (at least with respect to Roo) and prompt me to think about learning it so I could use it for a real application.

Apart from the productivity gain that is an implicit expectation from any RAD tool, what I am looking from Roo is eliminating the drudgery of the CRUD crud (pardon the pun) that accompanies almost every database backed web application. Most web applications are designed to do something "interesting", but to do that you need data, and you have to build in the interface to capture that data. Roo does that for you, allowing you to focus on the interesting part of the application.

For those of you who are unfamiliar with Roo, it is a bi-directional RAD tool that allows a programmer to use a scripting language to generate (fairly complete) CRUD web applications using Java and JSP (and AspectJ). The generated code is in the form of a Maven2 web project, and uses components (such as JPA, Hibernate, Spring-Web, etc) that Java developers are either already familiar with or can pick up fairly easily. The net effect is that the generated code looks quite familiar to a Java programmer, and thus (this is the important part) can be customized without too much effort.

Roo uses the ActiveRecord pattern, so an entity contains the functions to persist itself. However, much of the persistence code and other code that are unlikely to need customization are hidden away into the generated AspectJ files. With the associated Spring Tool Suite (STS) plugin for Eclipse (which I don't use since MyEclipse does not support it yet), the AspectJ files are hidden by default in the IDE. So customization effectively means that you can modify the entity class itself, including class and method level annotations, any extra controller classes that you generate, and any generated JSPX files once you annotate the controller appropriately.

Burndown Charts, or Managing by Numbers

At my previous employer, we practiced Agile Software Development. One of the things that I liked about this methodology was the use of Burndown Charts.

With Burndown Charts, a manager (PM or lead programmer) is responsible for breaking the project into a set of high level items and assigning them to various programmers. The programmer is responsible for breaking the item down into manageable chunks, and providing realistic estimates (in hours) for completion of these tasks. Simplistically, the sum of the estimates divided by the number of available programmer hours is the estimate for the entire project.

The approach has a number of advantages over the more traditional approach of making a guesstimate of the project duration, padding by an arbitary value and setting the deadline, and then working like hell to meet it. For the programmer, it means that they get to set the estimate for stuff they are going to do - given their knowledge of the task (which is why they were assigned to it) and their own capabilities, they are inarguably the best people to come up with this number. For the manager, it means more accurate project estimates and fewer missed deadlines. Programmers are more likely to honor estimates they have provided than those handed to them, so in this situation, they would usually work harder if they are in danger of missing one. On the other hand, even if the computed value is not in line with business goals, it provides an early warning, and the estimates can be tweaked in consultation with the programmer to either work more hours or cut non-essential features.

Since Burndown Charts are generated off progress data (hours burned per task) programmers enter daily, the chart also provides an early warning of someone in danger of missing a task deadline and jeopardizing the project. Corrective action can then be taken, usually involving the programmer having to work extra hours to get back on target. However, since the whole thing is driven by numbers, this is usually a mutual agreement between manager and programmer rather than a confrontational situation common in traditional shops.

Having experienced the power of Burndown Charts, I've wanted to implement something similar ever since my duties included managing other people. Since I carry a full programmer's load, I don't have too much time left to do typical "management stuff". Luckily for me, the people reporting to me are quite skilled, and they take as much pride in their work as I do, so I've been getting by without spending too much time on it. However, part of my job involves knowing "where person X is with task Y and how much more time will it take" - based on how busy I am or how recently I have spoken with X, the answer could be either spot on or the less satisfactory "I don't know but I will find out". What Burndown Charts have the potential to do is to replace that part of my job almost completely with a computer.

It also has the potential to eliminate the manual work involving estimating new projects - I typically build the item list and then go around to my assignees in order to get their estimates to incorporate it into the master document. However, because of the work required, that document is never updated, so a Burndown Chart based solution would be much more preferable.

KTM - Bringing it together

The Burndown Chart that we used at my previous employer was Excel based, so my first attempt was to build something similar with OpenOffice Calc, but I am no Excel/Calc macros guru, so I eventually abandoned it in favor of something more interesting.

The second attempt was shortly after learning Ruby-on-Rails and reading the Herding Cats book. I built my database model (based on the book) and got the basic skeleton working, but decided to switch to Java when I discovered that I had to really learn Ruby to do anything beyond getting the data entry screens working.

Writing in Java was no picnic either (really boring because of the repetitive nature of the app), so I decided to build my own CRUD app generator. It worked, though not as well as Roo. Code generation was one-way, and it did not generate JSPs that were as pretty or as powerful as Roo. I don't remember what happened after that, but I guess I got interested in something else...

So the current attempt has two main goals - to learn Roo by building something that hits Roo's sweet spot, ie, a database backed CRUD web application, and to build me something that can help me with my management responsibilities. The plan is to use Roo to build the data entry screens, then use standard Spring/Java/JSP to build the rest of the application to support Burndown Charts.

In case you are curious, the name of the application comes from the US pronounciation for Kathmandu, the capital of Nepal - (cat-man-do) - as opposed to the more correct (IMO, at least based on geographical proximity) pronounciation that I am used to - (cuth-mun-do). The name and its most common variant is already being used by a couple of projects on Sourceforge, so I decided to call it "KTM", the airport code for Kathmandu. Its also shorter, so less chances of making a typo on the URL.

Application Generation

The object model for my entities is shown below. I used the three entities in the Herding Cats book, and added four of my own, since my objective is to support more than just the simple reporting interface of the original application.

You can regenerate the application locally by installing Roo (I used version 1.0.2-RELEASE) and running the following Roo script. The script is adapted from my log.roo file. My log.roo file is much more messier. I started with a database model as with RoR, then realized midway that the Roo approach favors an object model instead. I ended up removing and re-adding quite a few fields and entities using Roo's round trip support. I have also put in comments inside the script file, as well as resequenced the calls to make them easier to understand.

Before running the script, you will need to create an empty MySQL database called ktmdb (or something else if you modify the --databaseName in the persistence setup command in the script). You will definitely also need to update the --userName and --password values to match your database username and password. You will also need to join the lines terminated with "\" with the next line - I put in the "\" for readability.

Then create a directory for the project, cd to it, and then run the following script through Roo (cat script.roo | roo or use the script command).

// Spring Roo 1.0.2.RELEASE [rev 638] log opened at 2010-07-06 08:16:03
project --topLevelPackage com.healthline.ktm
persistence setup --provider HIBERNATE --database MYSQL \
    --databaseName ktmdb --userName root --password orange
// :::: create our enums first ::::
// :::: create enum WorkRoles ::::
enum type --class com.healthline.ktm.domain.WorkRoles
enum constant --name Developer
enum constant --name Manager
enum constant --name Combined
// :::: create enum ClientTypes ::::
enum type --class com.healthline.ktm.domain.ClientTypes
enum constant --name AdInitiatives
enum constant --name NetworkPartners
enum constant --name BizDev
enum constant --name Internal
// :::: create enum ProjectTypes ::::
enum type --class com.healthline.ktm.domain.ProjectTypes
enum constant --name Billable
enum constant --name Maintenance
enum constant --name Investment
enum constant --name Rework
// :::: create Person domain object ::::
entity --class com.healthline.ktm.domain.Person
field string --fieldName name --notNull --sizeMin 2 --sizeMax 32
field enum --fieldName workRole --type com.healthline.ktm.domain.WorkRoles
field string --fieldName emailAddress --notNull 
field string --fieldName telephoneNumber --notNull
field number --fieldName availableHours --notNull \
    --type java.lang.Integer --min 1 --max 8
// :::: create Client domain object ::::
entity --class com.healthline.ktm.domain.Client 
field string --fieldName name --notNull --sizeMax 40
field string --fieldName salesContact --notNull --sizeMax 40
field enum --fieldName clientType --type com.healthline.ktm.domain.ClientTypes
field reference --fieldName engrContact --notNull \
    --type com.healthline.ktm.domain.Person
// :::: create Project domain object ::::
entity --class com.healthline.ktm.domain.Project
field string --fieldName name --notNull --sizeMin 2
field string --fieldName description --notNull --sizeMin 2 --sizeMax 255
field date --fieldName startDate --notNull --type java.util.Date
field date --fieldName endDate --notNull --type java.util.Date
field reference --fieldName originator --type com.healthline.ktm.domain.Person
field reference --fieldName client --type com.healthline.ktm.domain.Client
field enum --fieldName projectType --type com.healthline.ktm.domain.ProjectTypes
// :::: create Item domain object ::::
entity --class com.healthline.ktm.domain.Item
field string --fieldName name --notNull --sizeMin 2 --sizeMax 64
field string --fieldName description --notNull --sizeMin 2 --sizeMax 255
field reference --fieldName assignedTo --type com.healthline.ktm.domain.Person
// :::: create Task domain object ::::
entity --class com.healthline.ktm.domain.Task
field string --fieldName name --notNull --sizeMin 2 --sizeMax 64
field string --fieldName description --notNull --sizeMin 2 --sizeMax 255
field number --fieldName estimatedHours --notNull --type java.lang.Integer
// :::: create Hours domain object ::::
entity --class com.healthline.ktm.domain.Hours
field date --fieldName recordedDate --type java.util.Date
field number --fieldName actualHours --type java.lang.Integer
// :::: create Allocations domain object ::::
entity --class com.healthline.ktm.domain.Allocations 
field date --fieldName recordedDate --type java.util.Date
field number --fieldName percentAllocated --type java.lang.Integer \
    --notNull --min 1 --max 100
// :::: Associations ::::
// :::: Project --(1:m)--> Item ::::
field set --class com.healthline.ktm.domain.Project --fieldName items \
    --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Item
field reference --class com.healthline.ktm.domain.Item --fieldName project \
    --type com.healthline.ktm.domain.Project
// :::: Item --(1:m)--> Task ::::
field set --class com.healthline.ktm.domain.Item --fieldName tasks \
    --element com.healthline.ktm.domain.Task
field reference --class com.healthline.ktm.domain.Task --fieldName item \
    --type com.healthline.ktm.domain.Item
// :::: Task --(1:m)--> Hours ::::
field set --class com.healthline.ktm.domain.Task --fieldName hours \
    --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Hours
field reference --class com.healthline.ktm.domain.Hours --fieldName task \
    --type com.healthline.ktm.domain.Task
// ::: Person --(1:m)-->Task ::::
field set --class com.healthline.ktm.domain.Person --fieldName tasks \
    --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Task
field reference --class com.healthline.ktm.domain.Task --fieldName person \
    --type com.healthline.ktm.domain.Person
// :::: Project --(m:m)--> Person ::::
// :::: Since we need to hold data for each relation, we model this  ::::
// :::: as a pair of (1:m) relations from each entity to Allocations ::::
field set --class com.healthline.ktm.domain.Project --fieldName allocations \
    --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Allocation
field reference --class com.healthline.ktm.domain.Allocations \
    --fieldName project --type com.healthline.ktm.domain.Project
field set --class com.healthline.ktm.domain.Person --fieldName allocations \
    --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Allocations
field reference --class com.healthline.ktm.domain.Allocations \
    --fieldName person --type com.healthline.ktm.domain.Person
// :::: forcibly create the database. Not sure if this is needed but ::::
// :::: this was what I did after a lot of changes - drop database,  ::::
// :::: recreate and run this                                        ::::
perform tests
// :::: build the web layer ::::
controller all --package com.healthline.ktm.web

I also configured the Jetty plugin in the generated POM to listen on port 8888 and to log using the generated file. Here is the snippet of the XML.


Running mvn jetty:run command at command line on the project root directory brings up the server and I get this page at localhost:8888/ktm on my browser. I also navigated through to the other pages and things look functional.


If you decide to enter data that you don't want to disappear after a server restart, you should change the database update mode from "create" to "update" in persistence.xml.

Now what?

Running the script above created 300+ files - it took me about 6 hours to do this - this includes learning Roo, building a Pygments parser for Roo (so I can present a nicely colorized script on this page), designing the object model and writing and refining the script. So it appears that learning Roo is definitely time well-spent :-).

However, we are far from done. There is some basic customization that needs to be done to reorder the entities on the pages, add appropriate finders, add customizations (company logo, etc), and to build in security. I also have to figure out how to add my own controllers into the Roo app. I plan to do these in the coming weeks, and if its worth writing about, to write about it as well.

16 comments (moderated to prevent spam):

Mr Lonely said...

nice blog.. have a view of my blog when free.. .. do leave me some comment / guide if can.. if interested can follow my blog...

Sujit Pal said...

Thanks Mr Lonely. I did look at your blog, you seem to be doing a lifelogging kind of thing...I guess this is getting quite popular nowadays, and someday someone will figure out a way to pull out aggregated information out of these, like they are doing with Twitter.

michael said...

The script doesn't work for me:

// :::: Project --(m:m)--> Person ::::
// :::: Since we need to hold data for each relation, we model this ::::
// :::: as a pair of (1:m) relations from each entity to Allocations ::::
field set --class com.healthline.ktm.domain.Project --fieldName allocations --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Allocation
The specified target '--element' does not exist or can not be found. Please create this type first.
Script execution aborted

Sujit Pal said...

Sorry Michael, that was a typo, looks like I did a bad copy-paste from the logs - it should be:

field set --class com.healthline.ktm.domain.Project --fieldName allocations --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Allocations

Andrew Swan said...

You don't need to set the "log4j.configuration" system property; Roo 1.0.2 by default puts a context-param called "log4jConfigLocation" into web.xml that has the same effect.

Sujit Pal said...

Thanks Andrew, I will remove it from the pom then.

Bilal Dar said...

Great post my friend

Sujit Pal said...

Thanks Bilal.

Bilal Dar said...

Hi Sujit

Would you consider doing a tutorial for GWT where you can explain the GWT expense sample that was covered in Google IO.

Even a Simple customization of the mvn gwt: setup generated code would be much appreciated.

Sujit Pal said...

Hi Bilal, I don't have a use case for using GWT at the moment, but I can definitely give it a try if I have one. I tend to use Javascript sparingly, only when required, rather than the typical GWT approach of replacing the page with a bunch of JSON backed forms.

Anonymous said...


Great examples thanks. I am particularly impressed by your m:m example which I have not seen elsewhere. (Obvious really but I am keen to see how it turns out)

One question. You have split some lines over two by separating them with a \

My springroo can't cope with that. Does yours? or was this something added in by mistake?

Sujit Pal said...

Thanks, alexmc, and you are welcome. About the "\" chars, they are here for display purposes only - I wanted to break long lines so they are readable on the post, and "\" is a standard Unix continuation character, so I just used them, did not think about the potential for confusion. As you pointed out, the Roo client does not understand them - you will need to coalesce the lines into one for Roo.

JTS000ID said...

Great article. Thanks for sharing your work. Wondering which tool did you use for generating the class diagram.

Sujit Pal said...

Thanks JTS000ID, I use DIA for generating the class diagrams (and for most of the diagrams on the blog).

Anonymous said...

You can use ~.domain. instead of writing com.healthline.ktm.domain. and for Roo 1.2.0 you must use "entity jpa --class <...>" not only "entity --class <...>"

Sujit Pal said...

Yes, thanks, I knew about the ~.domain thing, but I think its a bit like using relative paths with .. in shell scripts - impacts readability and forces the reader to keep track of context. Did not know about the jpa switch in 1.2.0, its been a while since I used it last (1.0.2 according to the comments in the script :-)).