Saturday, October 28, 2006

A Maven2 multi-environment filter setup

Having been a happy user of Maven2 on my personal projects for a few months now, I finally tried to start using it at work. As I expected, development is quicker and more enjoyable as I didn't have to build the project manually, then copy and tailor an existing Ant build.xml to the new project. Creating the POM file is usually the most painful part of creating a new Maven2 project, but I have kind of a personal super-POM template which contains all the library dependencies and plugins I am likely to use. Examples of libraries are all the Apache Commons JARs, and examples of plugins are the Java 1.5 and the Jetty6 plugins, to name a few.

However, because my use of Maven2 was restricted to personal/toy projects, I had never had to worry about having to deploy the code to multiple environments. However, being able to deploy to multiple environments is a standard requirement for business applications, so this was obviously something I needed to address.

I vaguely remembered having read something about filtering support in Maven2, but I did not remember the details because I never had to use it. After about a day of reading up on Maven2's filter and profile support and tinkering with the pom.xml file, I had a Maven2 based setup which looked a lot like an Ant-based setup at a previous job, and which I knew from experience was easy to use and understand. This article describes this setup and provides cookbook style instructions on how to replicate it.

The basic idea of filtering is that you set up a named properties file and specify one or more filesets to apply the substitutions in the properties file. For example:

1
2
3
4
5
6
7
8
9
<filters>
  <filter>src/main/filters/filter-${env}.properties</filter>
</filters>
<resources>
  <resource>
    <directory>src/main/resources</directory>
    <filtering>true</filtering>
  </resource>
</resources>

The only difference from the snippet above and that provided in most Maven2 documentation is that the filter.properties file has a variable portion ${env}, which in our case will come from the selected profile. So assuming we want the database URL to be our replaceable parameter in each case (obviously there will be many more replaceable parameters per environment, but this is only an example), we set up our src/main/resources/applicationContext.properties (Spring configuration file) to contain the placeholder for the URL, something like this:

1
2
3
4
5
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
  ...
  <property name="url" value="${jdbc.url}" />
  ...
</bean>

Assuming now that we want to filter for three different environments, dev, test and live, we will create three filter-${env}.properties files in src/main/filters directory. The urls are incorrect and are only for illustration, replace with valid values that make sense for your application.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# filter-dev.properties
jdbc.url=jdbc:path:to:dev/database
...

# filter-test.properties
jdbc.url=jdbc:path:to:test/database
...

# filter-live.properties
jdbc.url=jdbc:path:to:live/database
...

Finally, we need a way to specify to Maven2 that we want to build an artifact for one the specified environments. We use profiles for this. The following profiles need to be declared in the POM.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<!-- default environment -->
<properties>
  <env>dev</env>
</properties>
<!-- profiles -->
<profiles>
  <profile>
    <id>dev</id>
    <properties>
      <env>dev</env>
    </properties>
  </profile>
  <profile>
    <id>test</id>
    <properties>
      <env>test</env>
    </properties>
  </profile>
  <profile>
    <id>live</id>
    <properties>
      <env>live</env>
    </properties>
  </profile>
</profiles>

Notice that the profile definition is really slim, all it contains is the setting for the ${env} variable. I guess this is a personal preference, but I like all my environment specific information in one place, makes life easier when trying to figure out cross-environment configuration issues. Notice also the default setting for ${env}, this is used in case a profile is not specified.

Now we have all our pieces in place, running targets for different environments is simply a matter of specifying the profile in the mvn command. So:

1
2
3
$ mvn war:war -P dev    # builds a war file with filter-dev.properties values
$ mvn war:war           # same as above, uses default ${env} setting
$ mvn war:war -P live   # builds a war file with filter-live.properties values

I hope this article was informative. There does not seem to be too much information on how to do this sort of thing with Maven2, perhaps because this process is not standard and teams and organizations have evolved different strategies to deal with this problem. For a while I thought that there was no way to do this natively with Maven2, and I started playing with the maven-antrun-plugin to delegate this work to an Ant build.xml file, but I was not able to pass the environment (passed into Maven2 with -Denv=dev) from Maven to Ant, so I gave up. I am glad I did, and many thanks to Eric Redmond for his article, because otherwise I would not have come up with this.

Resources

41 comments:

  1. Thanks a lot for the work, I had started working on it with the same set of documents but didn't go as far.
    I had an almost working solution by putting ant logic in each profile (antrun-plugin doing manual copies of files). However, I was stuck when trying to provide a default environment when no profile was given in the CLI.

    ReplyDelete
  2. You are welcome, hopefully it helped.

    ReplyDelete
  3. Thanks a lot! I´m doing the same now configuring project to support several environments and this post is perfect!
    I was changing filter property files manually, but the integration with profiles is far better :)

    ReplyDelete
  4. Great job! I had the exact same setup in ant, and having recently migrated a few projects to maven I wanted similar functionality.

    You saved me a good afternoon of reading and research! Thanks! :)

    ReplyDelete
  5. Thanks for the feedback, and I am glad it helped.

    ReplyDelete
  6. Hi Sujit,

    Thanks a lot for this post, it is a good base to help working with environment dependent build with profiles!

    I nevertheless have a problem:

    I would like to centralize filter properties files (and declare them) in a parent project (in its src/main/filters folder and POM, respectively), but when I build a child project, it tries to find the filter properties file in its own src/main/filter folder instead of its parent one.

    Do you know a way to declare filter properties in a parent and make its children inherit from them?

    Thanks a lot,

    Philippe

    ReplyDelete
  7. Hi Philippe, I haven't tried this, so I can't say for sure.

    From what you are saying, the filter setting in the parent POM is being passed to the child POM, but the child POM is looking for the file in its own src/main/filters directory.

    Personally, I would avoid putting filters in the parent POM altogether, since I think configuration is easier to maintain if it is closer to the module being configured, but that may just be me.

    Since Maven2 allows you to inherit the filter directive from a parent POM, it should be able to internally detect that the filter is linked to the parent POM. I would open a bug with the Maven2 folks.

    One way to temporarily work around the issue would be to link the child's src/main/filters directory to the parent's.

    ReplyDelete
  8. Hi Philippe,

    You could declare your properties directly in the profile (in profiles.xml).

    e.g.

    <profile>
    <id>dev</id>
    <properties>
    <env>dev</env>
    <my.property>this is dev</my.property>
    </properties>
    </profile>

    <profile>
    <id>test</id>
    <properties>
    <env>test</env>
    <my.property>this is test</my.property>
    </properties>
    </profile>

    Unfortunately it messes up the profiles.xml if you have a lot of properties. I also prefer the brief property syntax in property files above declaring them with xml-tags. But I can't think of another way to have global env-specific properties.

    With this setup, you don't need the filters-tag in the pom either since the properties are directly in the profile.

    ReplyDelete
  9. Hi Sujit, Hi Marcus,

    Thank you for your answers.
    Actually, as I searched in the user mailing list, such inheritance based on file system paths is not recommended, as one cannot be sure that the directory layout will remain the same and, most important, that the properties files are on the file system.

    The idea is to create a project for build purpose only (like "build-tools") as a jar containing the properties files and used as an extension in the build part of a POM (the parent one, for instance, so that a child module don't have to redecalre it).

    All modules requiring these properties will therefore have them in the classpath...

    It is also the only way I found to make it work ;-)

    Philippe

    ReplyDelete
  10. Hi Phillipe, this is a nice approach and thanks for doing the research to find this out and for letting us know.

    ReplyDelete
  11. Hey Philippe!

    Firstly, thanks for this approach! I'm new to Maven and was looking for a setup like this and was not satisfied with the approach documented on the Maven site (http://maven.apache.org/guides/mini/guide-building-for-different-environments.html).

    I'm wondering is there a way to automatically select a profile based on the task? I'd like the test profile to be used when mvn test is run.

    Thanks again!

    Dave

    ReplyDelete
  12. Hello Sujith,
    Thanks for the tip, it really worked for me. There was a small glitch though, the generated resource went to the classes folder (webapp/WEB-INF/classes) of the war file and what I wanted was it to be under webapp/WEB-INF. So I had to use the targetPath (resources/resource/targetpath) to ../${project.build.finalName}/WEB-INF .This did the trick and put all my spring configs to the web-inf itself. Anyway thanks a ton for putting this together.

    ReplyDelete
  13. Hi Aswin, thanks for the comment and glad it worked for you. AFAIK, the "correct" location for resources is WEB-INF/classes, so I guess Maven2 is doing the right thing, but thanks for the tip showing how to override the default location.

    ReplyDelete
  14. How do you use eclipse in a setup like this during your unit testing though? The applicationContext files will be unusable?

    ReplyDelete
  15. Hi Chris, I normally just use the command line for running unit tests, but you are right. For testing within Eclipse you would probably need to have an extra PropertyPlaceholderConfigurer configured to point to your dev environment file.

    ReplyDelete
  16. Thanks so much for writing this entry. You've helped make a difficult configuration easier.

    ReplyDelete
  17. Thanks very much for the comment, I am glad it helped.

    ReplyDelete
  18. Thanks very much. A very helpful post.

    ReplyDelete
  19. Thanks anydoby, and thanks also to the people who have commented on this post with their suggestions and ideas. I think the comments are probably more helpful at this stage than the original post :-).

    ReplyDelete
  20. Hi Sujit,
    I am not sure how the value of value="${jdbc.url}" is populated in spring config file. Are you using PropertyPlaceholderConfigurer to refer to the .properties file.

    Can you please share what you defined in spring applicationContext.properties file

    ReplyDelete
  21. Hi, ${jdbc.url} in applicationContext.xml is populated from the filter-${env}.properties file during the building of the war file. If you look at the applicationContext.xml file in your exploded directory, you will see that it has been replaced with the appropriate value, unlike when you use PropertyPlaceholderConfigurer, when it is pulled at startup from the appropriate properties file but its still ${jdbc.url} in the applicationContext.xml file.

    ReplyDelete
  22. Hey Sujit... I like what you have done here and this is a great example solution to this problem. However, I'm wondering if there may be a slightly different approach, though I'm not sure if it's a better or worse way to go, and would be interested in hearing your or anyone else's input.

    In each environment, we could do a CVS/SVN checkout on an entire code branch. However, the filter.properties file would not be included in the source control, but rather it would exist already in the target environment. Having maven installed, we could then do the full build/deploy on the entire code base, which would do the token replacing as expected. This way, the command to run the full build could be the same for all environments.

    I like what you have done here, but my ultimate intention is not to have to specify the environment when doing the build.

    Has anyone else taken this approach and found advantages or disadvantages?

    ReplyDelete
  23. Hi Adam, I the solution you are proposing would be quite possible with a bit of scripting on the target environment, and if the target environment was repository-aware.

    ReplyDelete
  24. Hi,

    Could you post the spring applicationContext.xml file? I have a hard time understanding how the property values get substituted inthe spring app context file.

    Thanks,

    ReplyDelete
  25. Never mind my previous comment about how values get substituted in spring appcontext.xml.

    Maven does it :)...fantastic solution ... THanks for sharing!!!

    ReplyDelete
  26. In our ant build, we build the properties files for all environments. The build outputs directories named dev, test, prod, etc. Each directory contains the filtered properties for that environment. I do that mostly because I'm not interested in rebuilding as I deploy into each environment. I'm guessing I'll fall back to an embedded ant task for this but if anyone has any more maven-y suggestions...

    ReplyDelete
  27. Excellent, helpful post! Much appreciated!(This example should be included in the soma book.)

    ReplyDelete
  28. Thanks Rick, glad it helped. Not sure what the soma book is though...Google pointed me to books about South Of MArket, San Francisco... :-).

    ReplyDelete
  29. Thanks! This helped me a lot.

    ReplyDelete
  30. Thanks bml, glad it helped.

    ReplyDelete
  31. Hi Sajit,

    Do you know by any chance how to build and deploy assemblies for all environments in one go? I looked up similar solution to build environment-specific packages but I wonder if it's possible to build all of them in one go. I see possible problems here as each command should be mvn -Pdev clean package so an artifact from the previous build will be deleted...

    thanks

    Vladimir

    ReplyDelete
  32. Hi Vladimir, sorry, no I don't. At the time when I wrote this post, I was investigating how to introduce Maven2 in our environment - but ultimately the learning curve (of pure Maven2) proved too steep for us, so we are currently using a hybrid Ant/Maven2 approach.

    ReplyDelete
  33. Hi Sujit -- I needed to do filtering for a maven2 project and came across your site (yet again! You have a lot of good stuff on your blog!)

    You asked about the "soma" book, but I think the commenter meant "sona" as in "sonatype," i.e. sonatype.com/book

    ReplyDelete
  34. Hi Matthew, thanks for the link...you may find the filter setup kind of familiar, it was inspired by a similar setup Trenton did with Ant.

    ReplyDelete
  35. Great job! Was looking for this exactly!

    ReplyDelete
  36. Thanks Mandy, glad it helped.

    ReplyDelete
  37. Great Post but I have a question. If I am not using spring, how would I read the variable say jdbc.url in my java servlet?

    ReplyDelete
  38. Thanks. Even if you don't use Spring, you probably need to get your properties from somewhere, for example JNDI or a local resource bundle (.properties file(s)). Even with JNDI you need the path defined in a .properties file somewhere. In that case, you could have a -dev, -qa, -live version of this file, and the same mechanism would copy the correct one over to the war file.

    ReplyDelete

Comments are moderated to prevent spam.