Saturday, October 28, 2006

A Maven2 multi-environment filter setup

Having been a happy user of Maven2 on my personal projects for a few months now, I finally tried to start using it at work. As I expected, development is quicker and more enjoyable as I didn't have to build the project manually, then copy and tailor an existing Ant build.xml to the new project. Creating the POM file is usually the most painful part of creating a new Maven2 project, but I have kind of a personal super-POM template which contains all the library dependencies and plugins I am likely to use. Examples of libraries are all the Apache Commons JARs, and examples of plugins are the Java 1.5 and the Jetty6 plugins, to name a few.

However, because my use of Maven2 was restricted to personal/toy projects, I had never had to worry about having to deploy the code to multiple environments. However, being able to deploy to multiple environments is a standard requirement for business applications, so this was obviously something I needed to address.

I vaguely remembered having read something about filtering support in Maven2, but I did not remember the details because I never had to use it. After about a day of reading up on Maven2's filter and profile support and tinkering with the pom.xml file, I had a Maven2 based setup which looked a lot like an Ant-based setup at a previous job, and which I knew from experience was easy to use and understand. This article describes this setup and provides cookbook style instructions on how to replicate it.

The basic idea of filtering is that you set up a named properties file and specify one or more filesets to apply the substitutions in the properties file. For example:

1
2
3
4
5
6
7
8
9
<filters>
  <filter>src/main/filters/filter-${env}.properties</filter>
</filters>
<resources>
  <resource>
    <directory>src/main/resources</directory>
    <filtering>true</filtering>
  </resource>
</resources>

The only difference from the snippet above and that provided in most Maven2 documentation is that the filter.properties file has a variable portion ${env}, which in our case will come from the selected profile. So assuming we want the database URL to be our replaceable parameter in each case (obviously there will be many more replaceable parameters per environment, but this is only an example), we set up our src/main/resources/applicationContext.properties (Spring configuration file) to contain the placeholder for the URL, something like this:

1
2
3
4
5
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
  ...
  <property name="url" value="${jdbc.url}" />
  ...
</bean>

Assuming now that we want to filter for three different environments, dev, test and live, we will create three filter-${env}.properties files in src/main/filters directory. The urls are incorrect and are only for illustration, replace with valid values that make sense for your application.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# filter-dev.properties
jdbc.url=jdbc:path:to:dev/database
...

# filter-test.properties
jdbc.url=jdbc:path:to:test/database
...

# filter-live.properties
jdbc.url=jdbc:path:to:live/database
...

Finally, we need a way to specify to Maven2 that we want to build an artifact for one the specified environments. We use profiles for this. The following profiles need to be declared in the POM.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
<!-- default environment -->
<properties>
  <env>dev</env>
</properties>
<!-- profiles -->
<profiles>
  <profile>
    <id>dev</id>
    <properties>
      <env>dev</env>
    </properties>
  </profile>
  <profile>
    <id>test</id>
    <properties>
      <env>test</env>
    </properties>
  </profile>
  <profile>
    <id>live</id>
    <properties>
      <env>live</env>
    </properties>
  </profile>
</profiles>

Notice that the profile definition is really slim, all it contains is the setting for the ${env} variable. I guess this is a personal preference, but I like all my environment specific information in one place, makes life easier when trying to figure out cross-environment configuration issues. Notice also the default setting for ${env}, this is used in case a profile is not specified.

Now we have all our pieces in place, running targets for different environments is simply a matter of specifying the profile in the mvn command. So:

1
2
3
$ mvn war:war -P dev    # builds a war file with filter-dev.properties values
$ mvn war:war           # same as above, uses default ${env} setting
$ mvn war:war -P live   # builds a war file with filter-live.properties values

I hope this article was informative. There does not seem to be too much information on how to do this sort of thing with Maven2, perhaps because this process is not standard and teams and organizations have evolved different strategies to deal with this problem. For a while I thought that there was no way to do this natively with Maven2, and I started playing with the maven-antrun-plugin to delegate this work to an Ant build.xml file, but I was not able to pass the environment (passed into Maven2 with -Denv=dev) from Maven to Ant, so I gave up. I am glad I did, and many thanks to Eric Redmond for his article, because otherwise I would not have come up with this.

Resources

41 comments (moderated to prevent spam):

Arnaud said...

Thanks a lot for the work, I had started working on it with the same set of documents but didn't go as far.
I had an almost working solution by putting ant logic in each profile (antrun-plugin doing manual copies of files). However, I was stuck when trying to provide a default environment when no profile was given in the CLI.

Sujit Pal said...

You are welcome, hopefully it helped.

Anonymous said...

Thanks a lot! I´m doing the same now configuring project to support several environments and this post is perfect!
I was changing filter property files manually, but the integration with profiles is far better :)

Sujit Pal said...

You are very welcome.

Anonymous said...

Great job! I had the exact same setup in ant, and having recently migrated a few projects to maven I wanted similar functionality.

You saved me a good afternoon of reading and research! Thanks! :)

Sujit Pal said...

Thanks for the feedback, and I am glad it helped.

Unknown said...

Hi Sujit,

Thanks a lot for this post, it is a good base to help working with environment dependent build with profiles!

I nevertheless have a problem:

I would like to centralize filter properties files (and declare them) in a parent project (in its src/main/filters folder and POM, respectively), but when I build a child project, it tries to find the filter properties file in its own src/main/filter folder instead of its parent one.

Do you know a way to declare filter properties in a parent and make its children inherit from them?

Thanks a lot,

Philippe

Sujit Pal said...

Hi Philippe, I haven't tried this, so I can't say for sure.

From what you are saying, the filter setting in the parent POM is being passed to the child POM, but the child POM is looking for the file in its own src/main/filters directory.

Personally, I would avoid putting filters in the parent POM altogether, since I think configuration is easier to maintain if it is closer to the module being configured, but that may just be me.

Since Maven2 allows you to inherit the filter directive from a parent POM, it should be able to internally detect that the filter is linked to the parent POM. I would open a bug with the Maven2 folks.

One way to temporarily work around the issue would be to link the child's src/main/filters directory to the parent's.

Unknown said...

Hi Philippe,

You could declare your properties directly in the profile (in profiles.xml).

e.g.

<profile>
<id>dev</id>
<properties>
<env>dev</env>
<my.property>this is dev</my.property>
</properties>
</profile>

<profile>
<id>test</id>
<properties>
<env>test</env>
<my.property>this is test</my.property>
</properties>
</profile>

Unfortunately it messes up the profiles.xml if you have a lot of properties. I also prefer the brief property syntax in property files above declaring them with xml-tags. But I can't think of another way to have global env-specific properties.

With this setup, you don't need the filters-tag in the pom either since the properties are directly in the profile.

Anonymous said...

Hi Sujit, Hi Marcus,

Thank you for your answers.
Actually, as I searched in the user mailing list, such inheritance based on file system paths is not recommended, as one cannot be sure that the directory layout will remain the same and, most important, that the properties files are on the file system.

The idea is to create a project for build purpose only (like "build-tools") as a jar containing the properties files and used as an extension in the build part of a POM (the parent one, for instance, so that a child module don't have to redecalre it).

All modules requiring these properties will therefore have them in the classpath...

It is also the only way I found to make it work ;-)

Philippe

Sujit Pal said...

Hi Phillipe, this is a nice approach and thanks for doing the research to find this out and for letting us know.

Unknown said...

Hey Philippe!

Firstly, thanks for this approach! I'm new to Maven and was looking for a setup like this and was not satisfied with the approach documented on the Maven site (http://maven.apache.org/guides/mini/guide-building-for-different-environments.html).

I'm wondering is there a way to automatically select a profile based on the task? I'd like the test profile to be used when mvn test is run.

Thanks again!

Dave

Aswin said...

Hello Sujith,
Thanks for the tip, it really worked for me. There was a small glitch though, the generated resource went to the classes folder (webapp/WEB-INF/classes) of the war file and what I wanted was it to be under webapp/WEB-INF. So I had to use the targetPath (resources/resource/targetpath) to ../${project.build.finalName}/WEB-INF .This did the trick and put all my spring configs to the web-inf itself. Anyway thanks a ton for putting this together.

Sujit Pal said...

Hi Aswin, thanks for the comment and glad it worked for you. AFAIK, the "correct" location for resources is WEB-INF/classes, so I guess Maven2 is doing the right thing, but thanks for the tip showing how to override the default location.

Unknown said...

How do you use eclipse in a setup like this during your unit testing though? The applicationContext files will be unusable?

Sujit Pal said...

Hi Chris, I normally just use the command line for running unit tests, but you are right. For testing within Eclipse you would probably need to have an extra PropertyPlaceholderConfigurer configured to point to your dev environment file.

Anonymous said...

Thanks so much for writing this entry. You've helped make a difficult configuration easier.

Sujit Pal said...

Thanks very much for the comment, I am glad it helped.

Unknown said...

Thanks very much. A very helpful post.

Sujit Pal said...

Thanks anydoby, and thanks also to the people who have commented on this post with their suggestions and ideas. I think the comments are probably more helpful at this stage than the original post :-).

Anonymous said...

Hi Sujit,
I am not sure how the value of value="${jdbc.url}" is populated in spring config file. Are you using PropertyPlaceholderConfigurer to refer to the .properties file.

Can you please share what you defined in spring applicationContext.properties file

Sujit Pal said...

Hi, ${jdbc.url} in applicationContext.xml is populated from the filter-${env}.properties file during the building of the war file. If you look at the applicationContext.xml file in your exploded directory, you will see that it has been replaced with the appropriate value, unlike when you use PropertyPlaceholderConfigurer, when it is pulled at startup from the appropriate properties file but its still ${jdbc.url} in the applicationContext.xml file.

Anonymous said...

Hey Sujit... I like what you have done here and this is a great example solution to this problem. However, I'm wondering if there may be a slightly different approach, though I'm not sure if it's a better or worse way to go, and would be interested in hearing your or anyone else's input.

In each environment, we could do a CVS/SVN checkout on an entire code branch. However, the filter.properties file would not be included in the source control, but rather it would exist already in the target environment. Having maven installed, we could then do the full build/deploy on the entire code base, which would do the token replacing as expected. This way, the command to run the full build could be the same for all environments.

I like what you have done here, but my ultimate intention is not to have to specify the environment when doing the build.

Has anyone else taken this approach and found advantages or disadvantages?

Sujit Pal said...

Hi Adam, I the solution you are proposing would be quite possible with a bit of scripting on the target environment, and if the target environment was repository-aware.

Anonymous said...

Hi,

Could you post the spring applicationContext.xml file? I have a hard time understanding how the property values get substituted inthe spring app context file.

Thanks,

Anonymous said...

Never mind my previous comment about how values get substituted in spring appcontext.xml.

Maven does it :)...fantastic solution ... THanks for sharing!!!

Anonymous said...

In our ant build, we build the properties files for all environments. The build outputs directories named dev, test, prod, etc. Each directory contains the filtered properties for that environment. I do that mostly because I'm not interested in rebuilding as I deploy into each environment. I'm guessing I'll fall back to an embedded ant task for this but if anyone has any more maven-y suggestions...

Rick said...

Excellent, helpful post! Much appreciated!(This example should be included in the soma book.)

Sujit Pal said...

Thanks Rick, glad it helped. Not sure what the soma book is though...Google pointed me to books about South Of MArket, San Francisco... :-).

Unknown said...

Thanks! This helped me a lot.

Sujit Pal said...

Thanks bml, glad it helped.

Anonymous said...

Hi Sajit,

Do you know by any chance how to build and deploy assemblies for all environments in one go? I looked up similar solution to build environment-specific packages but I wonder if it's possible to build all of them in one go. I see possible problems here as each command should be mvn -Pdev clean package so an artifact from the previous build will be deleted...

thanks

Vladimir

Sujit Pal said...

Hi Vladimir, sorry, no I don't. At the time when I wrote this post, I was investigating how to introduce Maven2 in our environment - but ultimately the learning curve (of pure Maven2) proved too steep for us, so we are currently using a hybrid Ant/Maven2 approach.

Matthew McEachen said...

Hi Sujit -- I needed to do filtering for a maven2 project and came across your site (yet again! You have a lot of good stuff on your blog!)

You asked about the "soma" book, but I think the commenter meant "sona" as in "sonatype," i.e. sonatype.com/book

Sujit Pal said...

Hi Matthew, thanks for the link...you may find the filter setup kind of familiar, it was inspired by a similar setup Trenton did with Ant.

Unknown said...

Great job! Was looking for this exactly!

Sujit Pal said...

Thanks Mandy, glad it helped.

Anonymous said...

Great Post but I have a question. If I am not using spring, how would I read the variable say jdbc.url in my java servlet?

Sujit Pal said...

Thanks. Even if you don't use Spring, you probably need to get your properties from somewhere, for example JNDI or a local resource bundle (.properties file(s)). Even with JNDI you need the path defined in a .properties file somewhere. In that case, you could have a -dev, -qa, -live version of this file, and the same mechanism would copy the correct one over to the war file.

Unknown said...

Beautiful...

Sujit Pal said...

Thanks Afshin.