As a special bonus, I’ve also thrown in coverage analysis using EMMA. And before people ask, yes I have tried to use Clover with PDE Build, but have had little luck due to Clover’s dependence on a specialized Java complier. If anyone has gotten this to work, I’d love to hear about it.

And finally, I had to change this example to use a test plug-in instead of a test fragment. For some reason when running Eclipse 3.4 (maybe related to p2?), the ETF is not finding tests in fragments. If anyone has any ideas here as well, please let me know.

You can download the new example here.

NOTE: I’ve discovered that the EMMA instrumentation task fails if you’re running the script inside of Eclipse and you have spaces in your workspace path. Avoid spaces in your path and you should be good to go.

Update – October 6, 2009: The sample projects have now been updated to work with Eclipse 3.5.1

• It’s not easy to run all the tests in a set of plug-ins. The test launcher allows you to run all the tests in a single project, but RCP and OSGi developers are usually working with a large set of test plug-ins. Sure it’s possible to create test suites, but keeping suites up-to-date is a real pain.
• It’s not easy to use test fragments. To find out why you’d want to use test fragments instead of test plug-ins, check out my previous post on Testing Plug-ins with Fragments. The problem is that even the standard suite-based solution does not work with fragments. There are workarounds, but they’re not very pretty.
• It’s not easy to run all of your tests during an automated build using the Eclipse Testing Framework. This is related to the first point above, and again you can use suites to handle this. 

You can then run the test suite both inside of the Eclipse IDE and using the Eclipse Testing Framework. I should note that this works only for JUnit 3.x tests. JUnit 4 describes suites using annotations which makes it (as far as I can tell) impossible to dynamically generate a suite at runtime. If anyone has a solution to this, I’d love to hear it. 

As always, comments and fixes are much appreciated.

A common request I’ve received, though, is to provide a similar sample build that includes the running of JUnits using the Eclipse Test Framework. Well here it is! Just download and import the sample projects, and follow the directions in the included readme.txt file.

Note: If you want to run this sample build using Eclipse 3.2, you’ll need to work around a PDE Build defect. For instructions on how to do this, check out the “Problems parsing feature file in a product build” section on the PDE Build wiki page.

But before going into detail on a fragment based solution, let’s examine the current approaches and the pros and cons associated with each. The first approach is the one most of us start with as we learn the ropes of plug-in development: placing all code in a single plug-in.

Option 1: Place all code in a single plug-in

Placing all your code in a single plug-in is obviously the simplest approach. Usually, developers who go down this path choose to place their code in two different source directories: src and test.

In addition to being simple, this approach has the benefit of allowing all of your code to be loaded by the same classloader (each plug-in has its own classloader). This means that if you maintain the same package structure in your source and test folders, your test classes will have access to the non-public methods of the classes under test.

The downside to this approach is that it seriously complicates your build process. If you wish to run your tests as part of your build process (and who doesn’t), then you’ll need to include your test code in your deployable plug-ins. Your plug-ins will also need to declare dependencies on JUnit and any other test libraries you use (EasyMock, etc.). It’s possible to manually remove the classes and manifest entries after the build, but this is far from ideal.

Option 2: Place test code in a separate plug-in

To get around these build-related issues, many of us choose to place unit test code in a separate plug-in. Usually, each testable plug-in will have a corresponding test plug-in. This is the approach taken by most Eclipse projects, and it has the advantage of cleanly separating out test code and dependencies during the build process.

The disadvantage of this approach, however, is that your test classes will be loaded by a separate classloader and will therefore not have access to the non-public methods of the classes under test. In addition, your tests will not have access to classes in packages not exported by the plug-in under test.

Plug-ins that are properly designed will typically export only a small set of API packages, keeping most of the implementation packages hidden from the outside world. But it’s often useful to have access to these implementation packages during testing. A common workaround is to expose these packages as friends of the test plug-in. While this approach works, it pollutes your plug-in manifest with reference to test plug-ins. It also does nothing to solve the other issue of accessing non-public methods.

Option 3: Place test code in a fragment

Wouldn’t it be great if we could combine the classloading benefits of having a single plug-in with the clean separation of test code achieved with a separate plug-in? Well luckily for us, this is possible using Eclipse fragments.

A fragments looks much like a plug-in from the outside. It is represented as a separate project in your workspace, it contains a manifest that describes its contents, it is built and deployed as a jar. What makes a fragment different is that it contributes it’s resources at runtime to a single host plug-in. The classes in the fragment are therefore loaded by the host plug-in’s classloader.

By placing our unit tests in fragments, we can provide our tests access to the non-public methods of the classes under test. Also, because the source code and test code are actually part of the same plug-in, there are no issues related to non-exported packages. Test classes will have access to all packages in the plug-in, whether they are exported or not.

Workaround: Creating a master test suite

The main disadvantage to this fragment based approach is that it is difficult to aggregate unit tests into a master test suite. While it’s easy to create a test suite that includes the tests within a fragment, it’s not so easy to create a suite that includes the tests in multiple fragments.

The basic problem is that classes in fragments are not visible outside of the fragment itself. Any attempt to reference fragment classes in another plug-in will result in a compile error, so it’s not possible to assemble a master test suite in the standard way. But because the fragment classes are accessible via the host plug-in’s classloader at runtime, it is possible to create a master test suite using reflection. The code below shows how this can be done.

public class AllTests {

	public static Test suite() throws ClassNotFoundException {
		TestSuite suite = new TestSuite(
				"Master test suite.");

		suite.addTest(getTest("com.mycompany.myplugin1.AllTests"));
		suite.addTest(getTest("com.mycompany.myplugin2.AllTests"));
		return suite;
	}

	private static Test getTest(String suiteClassName) {
		try {
			Class clazz = Class.forName(suiteClassName);
			Method suiteMethod = clazz.getMethod("suite", new Class[0]);
			return (Test) suiteMethod.invoke(null, new Object[0]);
		} catch (Exception e) {
			throw new RuntimeException("Error", e);
		}
	}
}

Simply place this suite in a plug-in that declares dependencies on all testable plug-ins. You should then be able to run all of your tests with one click. While this code does add some complexity to the test suite, it is only necessary for the single master test suite class. The individual test suites in the fragments can be written normally.

UPDATE: For an alternative approach to running tests in fragments, see my post on Running Unit Tests for RCP and OSGi Applications.

UPDATE 01/12/2009: Another alternative is to add a Eclipse-ExtensibleAPI: true entry to the manifest of the plug-in being tested. This header makes your fragment classes available to other plug-ins during development, and eliminates the need for the reflection workaround above. Thanks to Martin Dilger for the suggestion.

Final thoughts

There are some other minor issues with a fragment based approach. One of the most annoying is that the JUnit view in Eclipse does not allow you to double-click on a test case to open the class. For some reason, though, clicking on a line in the Failure Trace section does work. Go figure.

Overall, however, I’ve found that the benefits of fragment based unit testing far outweigh the disadvantages. If you are satisfied with the more traditional approaches to unit testing plug-ins, then by all means keep doing what you’re doing. But if you find yourself frustrated with the limitations of these approaches, give fragments a try. You’ll be glad you did!