Sunday, 31 July 2011

Announcing the first version of "Hudson Maven Repository Server Plugin"

One of the things I highlighted under "the bad" during my review of the new Maven 3 Integration in Hudson 2.1 was the fact that the "Jenkins Maven repository Server" did not work with it.

This is of course still true but now the first version of the Hudson equivalent has been released. Since it is the first version, my focus has been on making it work in our own pipeline leaving the extra stuff for later. This means

  • It only supports normal free-style jobs not multi-configuration
  • There is no repository chain or git commit functionality, only direct build by number
  • metadata.xml is not created. 
The last item can cause you some issues, as I have seen maven think its local version is newer than the repository version when it cannot find the metadata, so I advice you to manually remove/clean the old artifacts away.

I hope to support both multi-configuration projects and metadata files in the next version, but they might require core changes.

The plugin has gone through the staging in Sonatype's OSS nexus, so it should appear in maven central and the hudson update site shortly.

Friday, 29 July 2011

A review of the new Maven3 integration in Hudson 2.1

I have been looking forward to the new Maven3 integration in Hudson for some time now, and with the release of Hudson 2.1 it is finally here.

My main interest in the new integration is because we rely heavily on the ability to expose a maven build as a maven repository but the current (jenkins) plugin only works with the Maven 2 job type. That setup works nicely, but the Maven 2 job type has some performance and scalability issues

So we have been looking forward to getting the (hopefully) best of both worlds, fast as the freestyle job type, but still enough metadata to expose the build as a Maven repository.We mostly got it, see the details below.

The good
  • Our build time dropped from around 35 minutes to 12 minutes.
  • Our Hudson was acting really slow with regards to page rendering etc, but after switching to the new style integration it has become significantly faster. I think it is because of changes to the metadata stored in memory.
  • I like the the Maven information browser on the build pages. Right now I have only used it to see how far a build has come, but I imagine it is really useful if you need to look into individual modules.
  • The new document storages is a great idea which hasn't really come into its own yet. I think its potential is bigger than its current usage. I mean this in a good way because it is already great for managing settings.xml files, but there is so much more you could do with it.
  • You can now build a Maven job in a matrix build without loosing the metadata.
  • Many of the Maven options are now exposed as easy to use switches.
The bad
  • Some places it is evident in the user interface that the underlying presentation framework is different from the rest of Hudson. Some of the interface elements feel a bit "out of place", prime examples are the document storage screen and scrollbars.
  • Currently you cannot use the bundled maven3 to do maven releases if you rely on a custom settings.xml. Technically it is not Hudson's fault as it is merely a victim of a Maven bug (release:prepare does not pass argument "--settings" with current settings.xml to inner maven). You can use a standalone installation though.
  • The underlying API exposing the metadata is a bit confusing, and there isn't any documentation yet. To be fair it also has to deal with more things than the original Maven job type APIs,The problem space is simply bigger now that an arfifact can be created in multiple  build steps or combinations in a matrix job.
  • There are a few missing items in the API, e.g. there is no way to get a file link to a archived artifact.
  • You loose the functionality of the m2release plugin. I quite liked the fact that there was a manual button you could press on a job and have it perform a Maven release of the same job. Then it was visible to all in the build history when the release was done. Now you need to create 2 build jobs making the releases less visible
  • As far as I can tell you also loose the ability to easily see which modules where previously in a build but has now been removed. I have used that functionality more than once to catch when someone has removed a module but not all dependencies to it causing the release to fail.
  • You loose the use of "Jenkins Maven repository Server " plugin, but that went "Jenkins only" sometime after version 0.4 anyways and there is already a replacement underway.
The potential
  • Once people start extending the Maven integration there is a lot of potential e.g. why archive war and ejb files if they are included in ear files. Adding that will save tons of network traffic. 
  • The document storage can become a real killer feature once someone implements that the documents can be given to other build steps than just maven3 steps.
  • And let us not forget the collateral benefits this features has brought with it, there is a new REST api, the GWT frontend framework, JSR 330 based plug-ins.

The new Maven integration has provided us with a much needed and very impressive performance improvement both in build time where it went down from 35 to 12 minutes, and in responsiveness of the user interface.

For us this is the killer and the rest of the features are reduced  to "icing on the cake" even though that might be a bit unfair. As you can see from the negative list there is still room for improvement, and I hope to see many of those shortcomings  addressed now the rush to integrate is over.

In conclusion, the new maven integration is a solid  improvement, but it is even better as a stepping stone.

Monday, 25 July 2011

Debugging Hudson Git plugin hanging during fetch on Windows slave.

Ran into a tricky problem today after switching a couple of our Hudson jobs from subversion to Git. The git plugin was able to clone the repository in a clean workspace, but after that simply hung in the fetch step.

The job log did not provide much information
Started by user hlh005
Building remotely on selenium-xp2
Checkout:TestVerification / C:\hudson\workspace\TestVerification- hudson.remoting.Channel@6e7a8d98:selenium-xp2
Using strategy: Default
Last Built Revision: Revision b6d046ffede89c4bcdd5b71b7c6b703583cb6e18 (origin/master)
Checkout:TestVerification / C:\hudson\workspace\TestVerification - hudson.remoting.LocalChannel@7835ec
Fetching changes from the remote Git repository
Fetching upstream changes from
Looking at the slave workspace, it was clear that it had actually managed to clone the repository, as there was both .git folder and the working directory content. So in some sense it must be able to contact the server.

A Google search for the problem returned some results but in the majority people got a error message and not a hanging job. It did however reveal that a incorrect HOME environment variable could be the cause.

In order to find out our current environment variables for the node I went to "Manage Hudson" -> "Manage Nodes" -> (name of the node) -> "System information". Here you can find both system properties, environment variables and thread dump.

Besides confirming that we did not have a HOME environment variable, the thread dump revealed something very interesting. One of the thread had a thread name showing the exact git command line being executed and this thread was waiting for input reading on a socket.

Trying this command on the slave in a normal command promt, confirmed the fact that the git command could not find the key and thus failed to talk to the remote repository. Don't use the git bash you get with msysgit,  Hudson does not use this. Use a standard "cmd" promt opened from the start menu.

The solution was to set the HOME environment variable and restart the Hudson slave, but at least one question remains....

How was Hudson able to clone the repository when the keys could not be found. Is this another git plugin weirdness or does our gitolite setup have a error ?

I have created a Hudson issue have it provide better feedback see: Git plugin hangs instead of providing error message when it cannot find the ssh keys (windows slave)