Tuesday 8 November 2011

Mixing Ivy into a Maven build?

Current situation:
On one of the projects I am working on I have the following "pipeline"
  1. Build environment neutral artefacts with a snapshot version like 1.2.0-SNAPSHOT (using Maven 3 and Hudson M3 integration)
  2. Every hour deploy and test the latest successfully build artefacts to dev #1
  3. Every day deploy and test the latest artefacts which have been successfully tested in dev #1 to dev #2
  4. On demand do a Maven release of the latest artefacts that have passed testing in dev #2

The tricky part of this pipeline comes in the dev #2 environment because it needs a way to select the latest artefacts which have been tested in dev #1 instead of just the latest artefacts produced. The same goes for making the release, I need some way to identify which artefacts have been tested successfully in dev #2.

This means I cannot rely on just picking the latest snapshot deployed to the internal repository.

We have considered different options:

  • Publish the snapshot to a internal repository and carry around the timestamp of the specific snapshot version. This would allow us to pin out the version, but gives us a problem with regards to cleaning up unused snapshots as Nexus can only do keep X days or X versions, but not remove a set of artefacts based on a timestamp. Also it has the downside of people questioning why we should even do Maven releasing if we already have a unique identifier.
  • A second approach would be to use the "copy artefacts" Hudson feature, but then the way we get artefacts would be different between dev and the upper environments.
  • The approach we have settled on is to not deploy the snapshots to the nexus repo, but use the hudson maven repository plugin. This plugin exposes each build as its own repository. In order to get the right artefacts we use a custom settings.xml to mirror the snapshot repository to the URL of the specific build as exposed by the Hudson plugin. This plugin only works with either the jobs Maven 2 jobs or the new Maven 3 integration, since it relies on Hudson understanding the build and the artefacts. We use the promoted builds plugin to identify the correct build, and we use the promotion status for easy clean up of non used artefacts.


We haven't looked too much into using nexus pro's features such as staging repositories or adding metadata, since the above approach works fairly well for us.

The new challenge:
The reason for writing this post and asking for help is a possible change to our process which I am not sure how to best integrate.
There is a wish to integrate a component (playframework based webapp) build using ivy into this framework and specifically into this project. I can see this causing some integration pains

  • Firstly the project is one big Maven multi module project and we prefer to keep it that way. As a very least we want to keep things as one build i.e one Hudson job for building. Is it possible to have a Maven submodule defer execution to Ivy ?
  • If we get the component built using ivy only, then Hudson will not be able to see the outcome as a Maven artefact, and as thus wont be able to expose it as part of the "repository per build". That is at least my strong suspicion. 

What to do ?
So does anyone have a suggestions on how to resolve this ?

  • Can we cleanly integrate Ivy into our current build, if so how ?
  • Do we need to find another approach than using the "repository per build" plugin  ? if so, what is the suggested alternative.
  • Would Nexus/Artifactory paid editions make this easier ?

Just to be clear my requirements are:

  • From the outside it must appear as one build i.e. one Hudson job.
  • For a developers desktop build it is okay to be a multi step process
  • We need to be able to specify a particular build to deploy in dev #2 and for releasing.
  • We would like to continue the use the promoted builds plugin to visualize good builds
  • Clean up strategy should be easy

1 comment: