Herding Cats, Roo and the birth of KTM

This week I bring together three separate threads of thought that have been bouncing around in my head, and attempt to join them together into something approaching a cohesive whole, followed by an incomplete implementation of that whole.

Herding Cats

I have always been a bit of a sucker for self-improvement books. Early on in my career when I had to travel a lot, I would end up buying one everytime I travelled, usually at the airport/train station bookstore. I don't buy these quite so often nowadays, but a few years ago, the Agile/Pragmatic movement found me (or I found them), and I ended up buying some again. One of these books was Herding Cats: A Primer for Programmers Who Lead Programmers by J Hank Rainwater.

Despite the mixed reviews this book has received on Amazon, I found the book quite entertaining and yes, even somewhat useful. I think its a matter of perspective. You can choose to be offended by the stereotyping of programmers, or you can think of the book as a design document of how to deal with your coworkers, based on insufficient data (quite a common occurrence in our business). The rest of the data must come from the reader's environment and the design document tweaked based on this data for the book to be useful.

In the book, Rainwater describes a PC based application that he used for keeping track of what his programmers were tasked with. In the spirit of tweaking the design, I decided to build something similar, but which conforms more closely with my environment.

Roo

A few weeks ago, I attended a talk on Spring-Roo by Ramnivas Laddad at the eBig Java SIG. I had heard of Roo and knew it was the Java equivalent of Ruby-on-Rails, but did not have the time to look at further. The talk took us through setting up Roo and using it to build a web application from scratch. What the talk did for me was get me out of my inertia (at least with respect to Roo) and prompt me to think about learning it so I could use it for a real application.

Apart from the productivity gain that is an implicit expectation from any RAD tool, what I am looking from Roo is eliminating the drudgery of the CRUD crud (pardon the pun) that accompanies almost every database backed web application. Most web applications are designed to do something "interesting", but to do that you need data, and you have to build in the interface to capture that data. Roo does that for you, allowing you to focus on the interesting part of the application.

For those of you who are unfamiliar with Roo, it is a bi-directional RAD tool that allows a programmer to use a scripting language to generate (fairly complete) CRUD web applications using Java and JSP (and AspectJ). The generated code is in the form of a Maven2 web project, and uses components (such as JPA, Hibernate, Spring-Web, etc) that Java developers are either already familiar with or can pick up fairly easily. The net effect is that the generated code looks quite familiar to a Java programmer, and thus (this is the important part) can be customized without too much effort.

Roo uses the ActiveRecord pattern, so an entity contains the functions to persist itself. However, much of the persistence code and other code that are unlikely to need customization are hidden away into the generated AspectJ files. With the associated Spring Tool Suite (STS) plugin for Eclipse (which I don't use since MyEclipse does not support it yet), the AspectJ files are hidden by default in the IDE. So customization effectively means that you can modify the entity class itself, including class and method level annotations, any extra controller classes that you generate, and any generated JSPX files once you annotate the controller appropriately.

Burndown Charts, or Managing by Numbers

At my previous employer, we practiced Agile Software Development. One of the things that I liked about this methodology was the use of Burndown Charts.

With Burndown Charts, a manager (PM or lead programmer) is responsible for breaking the project into a set of high level items and assigning them to various programmers. The programmer is responsible for breaking the item down into manageable chunks, and providing realistic estimates (in hours) for completion of these tasks. Simplistically, the sum of the estimates divided by the number of available programmer hours is the estimate for the entire project.

The approach has a number of advantages over the more traditional approach of making a guesstimate of the project duration, padding by an arbitary value and setting the deadline, and then working like hell to meet it. For the programmer, it means that they get to set the estimate for stuff they are going to do - given their knowledge of the task (which is why they were assigned to it) and their own capabilities, they are inarguably the best people to come up with this number. For the manager, it means more accurate project estimates and fewer missed deadlines. Programmers are more likely to honor estimates they have provided than those handed to them, so in this situation, they would usually work harder if they are in danger of missing one. On the other hand, even if the computed value is not in line with business goals, it provides an early warning, and the estimates can be tweaked in consultation with the programmer to either work more hours or cut non-essential features.

Since Burndown Charts are generated off progress data (hours burned per task) programmers enter daily, the chart also provides an early warning of someone in danger of missing a task deadline and jeopardizing the project. Corrective action can then be taken, usually involving the programmer having to work extra hours to get back on target. However, since the whole thing is driven by numbers, this is usually a mutual agreement between manager and programmer rather than a confrontational situation common in traditional shops.

Having experienced the power of Burndown Charts, I've wanted to implement something similar ever since my duties included managing other people. Since I carry a full programmer's load, I don't have too much time left to do typical "management stuff". Luckily for me, the people reporting to me are quite skilled, and they take as much pride in their work as I do, so I've been getting by without spending too much time on it. However, part of my job involves knowing "where person X is with task Y and how much more time will it take" - based on how busy I am or how recently I have spoken with X, the answer could be either spot on or the less satisfactory "I don't know but I will find out". What Burndown Charts have the potential to do is to replace that part of my job almost completely with a computer.

It also has the potential to eliminate the manual work involving estimating new projects - I typically build the item list and then go around to my assignees in order to get their estimates to incorporate it into the master document. However, because of the work required, that document is never updated, so a Burndown Chart based solution would be much more preferable.

KTM - Bringing it together

The Burndown Chart that we used at my previous employer was Excel based, so my first attempt was to build something similar with OpenOffice Calc, but I am no Excel/Calc macros guru, so I eventually abandoned it in favor of something more interesting.

The second attempt was shortly after learning Ruby-on-Rails and reading the Herding Cats book. I built my database model (based on the book) and got the basic skeleton working, but decided to switch to Java when I discovered that I had to really learn Ruby to do anything beyond getting the data entry screens working.

Writing in Java was no picnic either (really boring because of the repetitive nature of the app), so I decided to build my own CRUD app generator. It worked, though not as well as Roo. Code generation was one-way, and it did not generate JSPs that were as pretty or as powerful as Roo. I don't remember what happened after that, but I guess I got interested in something else...

So the current attempt has two main goals - to learn Roo by building something that hits Roo's sweet spot, ie, a database backed CRUD web application, and to build me something that can help me with my management responsibilities. The plan is to use Roo to build the data entry screens, then use standard Spring/Java/JSP to build the rest of the application to support Burndown Charts.

In case you are curious, the name of the application comes from the US pronounciation for Kathmandu, the capital of Nepal - (cat-man-do) - as opposed to the more correct (IMO, at least based on geographical proximity) pronounciation that I am used to - (cuth-mun-do). The name and its most common variant is already being used by a couple of projects on Sourceforge, so I decided to call it "KTM", the airport code for Kathmandu. Its also shorter, so less chances of making a typo on the URL.

Application Generation

The object model for my entities is shown below. I used the three entities in the Herding Cats book, and added four of my own, since my objective is to support more than just the simple reporting interface of the original application.

You can regenerate the application locally by installing Roo (I used version 1.0.2-RELEASE) and running the following Roo script. The script is adapted from my log.roo file. My log.roo file is much more messier. I started with a database model as with RoR, then realized midway that the Roo approach favors an object model instead. I ended up removing and re-adding quite a few fields and entities using Roo's round trip support. I have also put in comments inside the script file, as well as resequenced the calls to make them easier to understand.

Before running the script, you will need to create an empty MySQL database called ktmdb (or something else if you modify the --databaseName in the persistence setup command in the script). You will definitely also need to update the --userName and --password values to match your database username and password. You will also need to join the lines terminated with "\" with the next line - I put in the "\" for readability.

Then create a directory for the project, cd to it, and then run the following script through Roo (cat script.roo | roo or use the script command).

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104

// Spring Roo 1.0.2.RELEASE [rev 638] log opened at 2010-07-06 08:16:03 project --topLevelPackage com.healthline.ktm persistence setup --provider HIBERNATE --database MYSQL \ --databaseName ktmdb --userName root --password orange // :::: create our enums first :::: // :::: create enum WorkRoles :::: enum type --class com.healthline.ktm.domain.WorkRoles enum constant --name Developer enum constant --name Manager enum constant --name Combined // :::: create enum ClientTypes :::: enum type --class com.healthline.ktm.domain.ClientTypes enum constant --name AdInitiatives enum constant --name NetworkPartners enum constant --name BizDev enum constant --name Internal // :::: create enum ProjectTypes :::: enum type --class com.healthline.ktm.domain.ProjectTypes enum constant --name Billable enum constant --name Maintenance enum constant --name Investment enum constant --name Rework // :::: create Person domain object :::: entity --class com.healthline.ktm.domain.Person field string --fieldName name --notNull --sizeMin 2 --sizeMax 32 field enum --fieldName workRole --type com.healthline.ktm.domain.WorkRoles field string --fieldName emailAddress --notNull field string --fieldName telephoneNumber --notNull field number --fieldName availableHours --notNull \ --type java.lang.Integer --min 1 --max 8 // :::: create Client domain object :::: entity --class com.healthline.ktm.domain.Client field string --fieldName name --notNull --sizeMax 40 field string --fieldName salesContact --notNull --sizeMax 40 field enum --fieldName clientType --type com.healthline.ktm.domain.ClientTypes field reference --fieldName engrContact --notNull \ --type com.healthline.ktm.domain.Person // :::: create Project domain object :::: entity --class com.healthline.ktm.domain.Project field string --fieldName name --notNull --sizeMin 2 field string --fieldName description --notNull --sizeMin 2 --sizeMax 255 field date --fieldName startDate --notNull --type java.util.Date field date --fieldName endDate --notNull --type java.util.Date field reference --fieldName originator --type com.healthline.ktm.domain.Person field reference --fieldName client --type com.healthline.ktm.domain.Client field enum --fieldName projectType --type com.healthline.ktm.domain.ProjectTypes // :::: create Item domain object :::: entity --class com.healthline.ktm.domain.Item field string --fieldName name --notNull --sizeMin 2 --sizeMax 64 field string --fieldName description --notNull --sizeMin 2 --sizeMax 255 field reference --fieldName assignedTo --type com.healthline.ktm.domain.Person // :::: create Task domain object :::: entity --class com.healthline.ktm.domain.Task field string --fieldName name --notNull --sizeMin 2 --sizeMax 64 field string --fieldName description --notNull --sizeMin 2 --sizeMax 255 field number --fieldName estimatedHours --notNull --type java.lang.Integer // :::: create Hours domain object :::: entity --class com.healthline.ktm.domain.Hours field date --fieldName recordedDate --type java.util.Date field number --fieldName actualHours --type java.lang.Integer // :::: create Allocations domain object :::: entity --class com.healthline.ktm.domain.Allocations field date --fieldName recordedDate --type java.util.Date field number --fieldName percentAllocated --type java.lang.Integer \ --notNull --min 1 --max 100 // :::: Associations :::: // :::: Project --(1:m)--> Item :::: field set --class com.healthline.ktm.domain.Project --fieldName items \ --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Item field reference --class com.healthline.ktm.domain.Item --fieldName project \ --type com.healthline.ktm.domain.Project // :::: Item --(1:m)--> Task :::: field set --class com.healthline.ktm.domain.Item --fieldName tasks \ --element com.healthline.ktm.domain.Task field reference --class com.healthline.ktm.domain.Task --fieldName item \ --type com.healthline.ktm.domain.Item // :::: Task --(1:m)--> Hours :::: field set --class com.healthline.ktm.domain.Task --fieldName hours \ --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Hours field reference --class com.healthline.ktm.domain.Hours --fieldName task \ --type com.healthline.ktm.domain.Task // ::: Person --(1:m)-->Task :::: field set --class com.healthline.ktm.domain.Person --fieldName tasks \ --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Task field reference --class com.healthline.ktm.domain.Task --fieldName person \ --type com.healthline.ktm.domain.Person // :::: Project --(m:m)--> Person :::: // :::: Since we need to hold data for each relation, we model this :::: // :::: as a pair of (1:m) relations from each entity to Allocations :::: field set --class com.healthline.ktm.domain.Project --fieldName allocations \ --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Allocation field reference --class com.healthline.ktm.domain.Allocations \ --fieldName project --type com.healthline.ktm.domain.Project field set --class com.healthline.ktm.domain.Person --fieldName allocations \ --cardinality ONE_TO_MANY --element com.healthline.ktm.domain.Allocations field reference --class com.healthline.ktm.domain.Allocations \ --fieldName person --type com.healthline.ktm.domain.Person // :::: forcibly create the database. Not sure if this is needed but :::: // :::: this was what I did after a lot of changes - drop database, :::: // :::: recreate and run this :::: perform tests // :::: build the web layer :::: controller all --package com.healthline.ktm.web quit

I also configured the Jetty plugin in the generated POM to listen on port 8888 and to log using the generated log4j.properties file. Here is the snippet of the XML.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

<build> ... <plugins> ... <plugin> <groupId>org.mortbay.jetty</groupId> <artifactId>maven-jetty-plugin</artifactId> <version>6.1.10</version> <configuration> <scanIntervalSeconds>10</scanIntervalSeconds> <systemProperties> <systemProperty> <name>log4j.configuration</name> <value>file:./src/main/resources/META-INF/spring/log4j.properties</value> </systemProperty> <systemProperty> <name>jetty.port</name> <value>8888</value> </systemProperty> </systemProperties> </configuration> </plugin> ... </plugins> </build>

Running mvn jetty:run command at command line on the project root directory brings up the server and I get this page at localhost:8888/ktm on my browser. I also navigated through to the other pages and things look functional.

 

If you decide to enter data that you don't want to disappear after a server restart, you should change the database update mode from "create" to "update" in persistence.xml.

Now what?

Running the script above created 300+ files - it took me about 6 hours to do this - this includes learning Roo, building a Pygments parser for Roo (so I can present a nicely colorized script on this page), designing the object model and writing and refining the script. So it appears that learning Roo is definitely time well-spent :-).

However, we are far from done. There is some basic customization that needs to be done to reorder the entities on the pages, add appropriate finders, add customizations (company logo, etc), and to build in security. I also have to figure out how to add my own controllers into the Roo app. I plan to do these in the coming weeks, and if its worth writing about, to write about it as well.