Tuesday, 7 February 2012

Ivy based subproject as maven module

I have previously written about the challenges of integrating an Ivy based subproject (like the play framework) into a build that is otherwise Maven based. After some work it is now working although the support feels a bit rudimentary.Here are the highlights on how to make it work

Prepare play framework:
The play framework must be installed and available to Hudson.

Dependency management is done by maven
Instead of duplicating maven's settings.xml configuration in Ivy, you can rely on the fact that maven will resolve any dependencies and place them in the local Maven repository before any Ivy commands are executed. This way the Ivy settings only need to contain a reference to the local repository and not muck around with remote resolutions.

Depending on how you create the dependencies.yml (see below), you might need to specify the dependencies twice, once for Maven and once for Ivy

Clean the extra directories:
Since play likes to spread its files more than just the normal src/ and target/ make sure the clean plugin cleans these extra directories as well

Generate Ivy configuration (conf/dependencies.yml):
Instead of checking in the Ivy config file into your SCM, you need to generate this as part of the build.This can be done using antrun to simply echo the file out before the compile stage. The reason is twofold.

Firstly the path to the local maven repository needs to be set

Secondly, if the dependencies.yml file is in SCM, then it would still point to the old versions when Maven is doing the release step unless some work is done to have Maven roll these versions forward and commit the changes to this file. It is easier to just generate the file on the fly.

Compile step:
The "precompile" command of the play compiler needs to be invoked instead of relying on the normal java compiler. Using antrun/exec plugin to do this task is easiest.

Package step:
This time the "war" command of the play compiler needs to be invoked. After the play compiler has run the resulting exploded war directory needs to be zipped and replace the war file generated by the normal war plugin so Maven picks up our custom package.

Care should be taken as to where the exploded war file is generated to. The reason is that you can run into a play bug. If you generate the war file within the Maven module, you must also exclude the output directory in order to avoid endless recursion. That in itself is not the problem, The problem arises if you choose "target" as your output  directory. Then you will have a clash between the play bug where the exclude directories are not evaluated from the module directory but on the full path, and the release plugin's checkout of the code into "target/checkout"

Releasing:
Since the Ivy parts are not fully integrated into maven, it does not understand when maven puts target folders on the classpath. This causes a problem during the release:prepare step when maven only executes "clean verify", because then Ivy cannot find the built dependencies. The trick is to reconfigure release:prepare to run "clean install" as preparation goals.

Outstanding issues:

  • Sometimes the play tests can leave hanging processes
  • Since play opens a socket, running concurrent build can clash. However the play part is usually so quick that even if SCM polling is set to every minute it is not a problem. 

Saturday, 14 January 2012

Why use the Hudson jobcreator tool?

It might seem very counter intuitive to start using Hudson's XML format to manage Hudson  jobs when it has such a great web interface, especially since it is the web interface which has made Hudson so approachable and easy to use.

Don't get me wrong. I still love the web interface and think it is the right way to get people to use the tool, however I also think there comes a time when you outgrow the web interface and this is why I built the jobcreator tool.

These are the requirements and issues that caused me to outgrow the web interface:

Manual changes doesn't scale
Using the web interface to make changes to individual jobs is very easy, but it doesn't really scale if you need to change a lot of jobs on the same time. One example could be changing the git branch from "master" to a release branch for all the jobs related to a environment, Such a change could involve upwards of 30 jobs.

Another example could be that there are changes to the content and/or structure of the jobs and those must be propagated up through the environments in sync with project code.This would be even more manual changes.

Managing this with manual changes in the web interfaces introduces a big risk for human error and inconsistencies.

Hudson jobs are code
If you want to be able to reproduce a build or deployment at a later time it is important that you can also reproduce the Hudson jobs. In order to do this you need to store and version your Hudson job configurations.

This is naturally best done in a SCM like Git or Subversion. Doing so also gives you the option to do branched development of your jobs.

Testing and more than one Hudson instance.
Before making changes to the jobs being actively used the changes should naturally be tested somewhere. This normally means creating the same set of jobs somewhere else or targeted towards a different environment.

Having the jobs defined as templates makes it easy for a developer to load the jobs into a private Hudson instance to experiment or share it on the testing instance.

Overall
Of course Hudson can be managed via the web interface, but for me it is just another step in the automation.

Friday, 13 January 2012

Announcing the Hudson job creator tool.

I have finally had the chance to pull together the last changes before announcing the first version of the Hudson Job Creator tool.

The idea behind the tool is that you can write FreeMarker based templates and combine those with properties defined in a "pipeline" specification in order to generate Hudson's job config.xml files.
This is mainly useful if you maintain a number of similar jobs, or have a series of jobs that you need to specify for multiple environments.

I know working with Hudson's xml files directly can seem counter intuitive since one of Hudson's main strengths is its approachability and easy to use web interface, so later this week I will post a more in depth blog post
explaining why I chose to go this route.

All the information about the tool including download is available on github, https://github.com/hudson-plugins/jobcreator-tool .

Sunday, 8 January 2012

Maven bug fixed making Hudson's Maven3 integration much more useful

Back in July when I reviewed the Maven3 integration in Hudson 2.1, I made a point regarding it not working when doing a maven release. As written back then the cause is a Maven bug (release:prepare does not pass argument "--settings" with current settings.xml to inner maven).

This bug has now been fixed and a new version of the release plugin has been released. So if you update the release plugin to version 2.2.2, Hudson's maven3 integration will now work for releases also.

Tuesday, 8 November 2011

Mixing Ivy into a Maven build?

Current situation:
On one of the projects I am working on I have the following "pipeline"
  1. Build environment neutral artefacts with a snapshot version like 1.2.0-SNAPSHOT (using Maven 3 and Hudson M3 integration)
  2. Every hour deploy and test the latest successfully build artefacts to dev #1
  3. Every day deploy and test the latest artefacts which have been successfully tested in dev #1 to dev #2
  4. On demand do a Maven release of the latest artefacts that have passed testing in dev #2

The tricky part of this pipeline comes in the dev #2 environment because it needs a way to select the latest artefacts which have been tested in dev #1 instead of just the latest artefacts produced. The same goes for making the release, I need some way to identify which artefacts have been tested successfully in dev #2.

This means I cannot rely on just picking the latest snapshot deployed to the internal repository.

We have considered different options:

  • Publish the snapshot to a internal repository and carry around the timestamp of the specific snapshot version. This would allow us to pin out the version, but gives us a problem with regards to cleaning up unused snapshots as Nexus can only do keep X days or X versions, but not remove a set of artefacts based on a timestamp. Also it has the downside of people questioning why we should even do Maven releasing if we already have a unique identifier.
  • A second approach would be to use the "copy artefacts" Hudson feature, but then the way we get artefacts would be different between dev and the upper environments.
  • The approach we have settled on is to not deploy the snapshots to the nexus repo, but use the hudson maven repository plugin. This plugin exposes each build as its own repository. In order to get the right artefacts we use a custom settings.xml to mirror the snapshot repository to the URL of the specific build as exposed by the Hudson plugin. This plugin only works with either the jobs Maven 2 jobs or the new Maven 3 integration, since it relies on Hudson understanding the build and the artefacts. We use the promoted builds plugin to identify the correct build, and we use the promotion status for easy clean up of non used artefacts.


We haven't looked too much into using nexus pro's features such as staging repositories or adding metadata, since the above approach works fairly well for us.

The new challenge:
The reason for writing this post and asking for help is a possible change to our process which I am not sure how to best integrate.
There is a wish to integrate a component (playframework based webapp) build using ivy into this framework and specifically into this project. I can see this causing some integration pains

  • Firstly the project is one big Maven multi module project and we prefer to keep it that way. As a very least we want to keep things as one build i.e one Hudson job for building. Is it possible to have a Maven submodule defer execution to Ivy ?
  • If we get the component built using ivy only, then Hudson will not be able to see the outcome as a Maven artefact, and as thus wont be able to expose it as part of the "repository per build". That is at least my strong suspicion. 

What to do ?
So does anyone have a suggestions on how to resolve this ?

  • Can we cleanly integrate Ivy into our current build, if so how ?
  • Do we need to find another approach than using the "repository per build" plugin  ? if so, what is the suggested alternative.
  • Would Nexus/Artifactory paid editions make this easier ?

Just to be clear my requirements are:

  • From the outside it must appear as one build i.e. one Hudson job.
  • For a developers desktop build it is okay to be a multi step process
  • We need to be able to specify a particular build to deploy in dev #2 and for releasing.
  • We would like to continue the use the promoted builds plugin to visualize good builds
  • Clean up strategy should be easy

Wednesday, 26 October 2011

Review of "Apache Maven 3 Cookbook" by "Srirangan"

This is a review of Apache Maven 3 Cookbook written by "Srirangan". I got a free copy from Packt publishing for the purpose of the review.  I have been using Maven for some years now and this book is a introductory book, so it was clear from the beginning that I am not in the target audience.

The style of splitting the book into 50 recipes makes for a good format which is easy to read and breaks the book into small achievements for the reader.

Instead of focusing solely on how Maven is configured, the author tries to tie some of the subjects to software development practices e.g. covering Nexus and Hudson while explaining team collaboration. It serves the book well to put Maven into a development perspective, but it doesn't always fit with the recipe format. For instance in the Nexus case from above the "How it Work" section becomes more a "why it is good" section.

I like the fact that the book covers a wide range of different project types and topics. Many times when you read tutorials or other documentation only the simplest project types are covered leaving the reader to add plugins as needed. This book covers many project types and framework and some non-Java areas. It also covers things like setting up Nexus,Hudson, various IDEs. It even has a single chapter on plugin development.

In the first chapter the level of information in each recipe is appropriate, but as the chapters get more complex the level of information does not. This results in many of the recipes being too simplistic. A prime example the is set-up of remote repositories, it describes in great many screen shots how to install tomcat 7 and deploy nexus, but only has a single line of information on how to set-up the remote repository, plus only mentions (incorrectly) changing the settings.xml and not the required changes to the project object model. So it fails to help the reader define and use the remote repository possible leaving the reader with a broken set-up

This simplistic approach has another side effect. In many of the recipes there is clear copy-paste'able examples but very little explanation of why things are the they are. An example would be the first recipe which introduce multi-module projects. In the top level project definition the dependencies are placed in "<dependencyManagment>" section instead of the normal "<dependencies>" section without an explanation of why.

Conclusion:
While I like the style and long list of topics covered in this book, I think the decision not the explain the details of why things work like they do e.g. <dependencies> vs <dependencyManagment> or how repositories works, does the book a big disservice.

I would not recommend learning Maven from this book alone, since I think explaining the "why" is an essential part of learning a new tool. If you want to learn Maven use the free sonatype book Maven: The Complete Reference and buy a copy of this book if you like a quick introduction to the various project types and plugins.

Thursday, 13 October 2011

Using the SoapUI maven integration

I have had the mixed pleasure of using the Maven integration for SoapUI (http://www.soapui.org/Test-Automation/maven-2x.html).

The tool itself is pretty good and makes it very easy to test that our soap based web-services are working as intended. The fact that is actually provides Maven and Junit integration out of the box is even better and fits very nicely with our CI environment.

There is however a few things that are not obvious when using the plugin.

The documentation page is really old, e.g. it refers to an old (2.5.1) version of the plugin. The trick here is that you should in general use the same version, as the desktop version you are running. In my case that is 4.0.0

It isn't documented on the page but there here is both a "maven-soapui-plugin" and "maven-soapui-pro-plugin" version of the plugin. In order to fully use project created using the pro version you need the pro version of the plugin

The version 4.0.0 of the plugin has a misconfiguration so you will need to manually add some dependencies to the plugin. The version I got working looks like this.
<plugin>
    <groupId>eviware</groupId>
    <artifactId>maven-soapui-pro-plugin</artifactId>
    <version>4.0.0</version>
    <configuration>
      ....
    </configuration>
    <executions>
      ....
    <executions>
    <dependencies>
        <dependency>
            <groupId>jgoodies</groupId>
            <artifactId>looks</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>fife</groupId>
            <artifactId>rsyntaxtextarea</artifactId>
            <version>1.3.4</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.4</version>
        </dependency>                  
    </dependencies>
</plugin>
The two first dependencies are so that the plugin can run, and the last one is needed if there are any XPath assertions.

I have tried to upgrade to the latest 4.0.1 version which has just been released but I get the following error, so I suggest people stick to 4.0.0 for now

class "com.eviware.soapui.SoapUICore"'s signer information does not match signer information of other classes in the same package