Background
This, and subsequent, entries describe the process I went through with moving an Eclipse Java project from Eclipse-managed, hard-coded, downloaded dependent jars to using Apache Ant and Ivy. It will hopefully serve as a good tutorial for those attempting the same feat.
I have recently been working on a Java application that converts XML files to CSV files, intuitively named xml2csv which I’m maintaining in Github. Xml2csv has quite a few dependencies on third-party libraries, approximately 10MB of them. These third party libraries are all releases from the Apache project.
I’d done all my development in an out-of-the-box Eclipse Luna and used the “Add External Jar” button liberally, all pointing to my own directory of downloaded JAR files. Of course, this means that the code won’t work for anyone other than me (unless they happened to have used exactly the same paths on their machine as I did) and incidentally also couldn’t be used by anyone unless they also had Eclipse.
The easiest way to remedy the dependency situation would be to create a directory in my Git repo for them and push them all to Github. However, I really didn’t want to store all those libraries in Github because:
- It’s a big waste of bandwidth for people who want to look at the code having to include those repositories as part of a git clone (download) operation. Many Java developers would already have their own copies of these libraries.
- As I didn’t create them and I don’t own them, I just didn’t feel comfortable hosting a public download of them.
Moreover, I really wanted to make it painless for someone who downloads the code to get it working and run the unit tests.
I’ve been rather spoilt by the rather lovely Nuget when developing for the MS platform, but unfortunately there’s no Nuget equivalent for Java, at least as far as I could find. After some Internet searches and some following links I came across the following options:
- Apache Maven.
- Apache Ivy.
- Gradle.
After reading some reviews, comparisons and tutorials I decided to go for Apache Ivy, because:
- It was designed solely for dependency management, so it did exactly what I needed.
- It was designed for use with Ant, which I have some experience of from previous lives projects.
- There was an Eclipse integration available (although I believe this is true for all three).
- It didn’t use a syntax I wasn’t used to, like Groovy-based Gradle. I’ll be honest, this isn’t a good reason, but I just didn’t fancy learning something completely new, see 3.
- It didn’t describe itself as “software project management and comprehension tool” (Seriously, guys? I refer you to a memorable line from Jules Whtifled when talking to Brett for that one).
Moving to Ant + Ivy
Right now, my project’s build cycle is managed by Eclipse. This has worked really well for me so far, but now it’s not enough. So off to http://ant.apache.org/ivy and download 2.4.0 rc1. There were two options for the download “binary” and “binary-with-de pendencies”. Not knowing what the dependencies were and unable to find a description, I decided to go with the latter.
I was really pleased to see that the download included the documentation, as I frequently work without an Internet connection.
I figured that the best thing for me to do was to learn “basic” Ivy, then go looking for an Eclipse IDE integration. I must admit, I was really worried that I’d lose features like the automatic build in Eclipse, which I absolutely adore, but that’s not a good enough reason to abandon.
My project is really simple, you can see for yourself if you like. I have a “src” and “testsrc” directory and a properties directory. However, I needed the following libraries to make it sing and dance:
- Apache Xerces v2.11.
- Apache Commons CLI v1.2.
- QOS Simple Logging Framework (SLF4J) v1.7.7.
- Apache Log4j 2.0 and the SLF4J to Log4j 2.0 adapter.
- Saxonica Saxon-HE 9.
Nothing particularly onerous, but still around 10MB of Jar files. My application builds to a relatively tiny 66 kilobyte Jar file.
From a brief read of the Ivy tutorial, I managed to create myself a simple ivy.xml file (bear in mind that I didn’t have any kind of ant build yet), but I’m reading the Ivy docs, so that’s where I started. Here’s my ivy.xml:
<ivy-module xsi:nonamespaceschemalocation="http://ant.apache.org/ivy/schemas/ivy.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="2.0"> <info module="xml2csv" organisation="com.locima"> <description> This Ivy file is used to download the third-party dependencies of the xml2csv project. </description> </info> <dependencies> <dependency rev="9.5.1-6" name="Saxon-HE" org="net.sf.saxon" /> <dependency rev="2.11.0" name="xercesImpl" org="xerces" /> <dependency rev="1.7.7" name="slf4j-api" org="org.slf4j" /> <dependency rev="1.7.7" name="slf4j-log4j12" org="org.slf4j" /> <dependency rev="2.0.2" name="log4j-core" org="org.apache.logging.log4j" /> <dependency rev="1.2" name="commons-cli" org="commons-cli" /> <dependency rev="4.11" name="junit" org="junit" /> </dependencies> </ivy-module>
I found the right org, name, and rev versions by going to http://mvnrepository.com/ and searching using the search box at the top of the page. This wasn’t as easy a process as I thought it would be. This is a public archive, so searching for “Log4j” or “Log4j 2”, for example, doesn’t actually give you Apache Log4j 2.0 in the first page of hits for either query, but a load of other projects that either use log4j, or extend it. In the end had the most success searching for specific Jar names that I was using and this seemed to work for most things, but not Apache Commons Logging (I reverted to searching for “Apache Commons Logging 1.2” and that worked”).
So now, I’ve got a load of source code with missing dependencies. I needed an Ant build.xml script to actually build my project. Fortunately Eclipse can export the current build configuration as an Ant script, so right-clicking the project in package explorer, then selecting Export, expanding the General node and select Ant Buildfiles gave me a pretty lengthy build.xml file.
As I’d already stripped out all the hard-coded dependencies from the Eclipse, running ant quickly yielded 100 errors for missing dependencies before it bombed. Success!!!
Adding Ivy to Ant
Following a simple tutorial on Ivy (file:///C:/apps/java/apache-ivy-2.4.0-rc1/doc/tutorial/start.html) I added a new “resolve” target and then set it as a pre-requisite to the build-project target, like this:
<project basedir="." default="build" name="xml2csv" xmlns:ivy="antlib:org.apache.ivy.ant">
<target name="resolve" description="--> retrieve dependencies with ivy">
<ivy:retrieve />
</target>
And full of enthusiasm typed ant resolve at the command line:
C:UsersAndyProjectsxml2csv>ant resolve Buildfile: C:UsersAndyProjectsxml2csvbuild.xml [taskdef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found. resolve: BUILD FAILED C:UsersAndyProjectsxml2csvbuild.xml:47: Problem: failed to create task or type antlib:org.apache.ivy.ant:retrieve
The error message was very clear, I’d downloaded Ivy but hadn’t told Ant about it! To fix this, I change the top of my ant build.xml file so it read as follows:
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <project name="xml2csv" xmlns:ivy="antlib:org.apache.ivy.ant" default="build" basedir="."> <path id="ivy.lib.path"> <fileset includes="ivy-2.4.0-rc1.jar" /> </path> <taskdef classpathref="ivy.lib.path" uri="antlib:org.apache.ivy.ant" resource="org/apache/ivy/ant/antlib.xml" />
Now aware of where Ivy was, it all sprang in to life and downloaded all the artifacts that I needed (any many more besides, but more on that later):
--------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| --------------------------------------------------------------------- | default | 48 | 0 | 0 | 5 || 55 | 0 | ---------------------------------------------------------------------
Ivy was doing what I needed it to do with a minimal amount of effort. So far, so good. Ivy has downloaded all the resources, but I don’t know where to and I don’t know how to add them to my project as dependencies on the classpath.
By luck, I looking inside my project root directory and found a new lib directory; and Behold! 46MB across 55 jar files!
Clearly, I had asked Ivy for too much, but fundamentally I was able to build and execute my project, along with all its Junit test cases.
The next entry of this series will describe how to control how much Ivy is downloading.