JUnit4 and the Eclipse Test Framework: Success!

(If you're eager, just jump down to the bolded part about victory.)

First, some background. I think the Eclipse Test Framework is one of the gems that is part of Eclipse. Effectively it's a way to run JUnit tests against your bundles while Equinox OSGi is running. You can also have the whole workbench UI running if you like, which I think is a common case. In fact, for quite some time, that was the *only* case that I considered it for. If I had a JUnit test that didn't need the workbench, I could "Run As -> Junit Test". If I had a Junit test that needed the workbench, I could "Run As -> Junit Plug-in Test". But I never really mentally equated the "workbench" with "OSGi framework."

Then I went to write a test for one of my bundles that has a dependency on Jetty. And its a version of Jetty that is different from the one that ships with Eclipse. When I ran it as a vanilla "Junit Test" from within Eclipse, everything worked fine. I think I just "got lucky" and it picked up the right version of Jetty when I launched the test. But when I ran my test via the Ant "junit" task (no Eclipse Test Framework), it failed because the wrong version of Jetty got loaded. All of a sudden I realized that without my delicately constructed dependencies being managed by OSGi, I was lost back in the world of "which JAR file is getting used?"

So, this brings me to the real guts of this blog post. After discovering I needed OSGi during test execution, I took a second look at the Eclipse Test Framework. Wow, this looks promising! Headless OSGi-based bundle testing! Just what I need. So I converted my build script to go that route, only to run smack dab into bug 153429, which captures that the Eclipse Test Framework only supports JUnit3.

That was at least a year ago, maybe two. I commented out my Jetty test and continued running my other JUnit4 tests using Ant. I even wrote some JUnit3 tests that I could run against the Eclipse Test Framework. Meanwhile, the CC list on bug 153429 continued to build and milestones kept passing without any progress. I stubbornly left alone my Jetty test, refusing to rewrite it for Junit3 since I figured Junit4 on ETF was right around the corner.

Finally with Eclipse 3.6M5 we have complete and utter victory. I have successfully executed my Junit4 tests with the Eclipse Test Framework. I'm not building my product against 3.6 yet (still 3.5.0 actually), but I still was able to grab the zip for ETF and make it part of my base Eclipse for PDE build to consume.

Now that I have my tests working, I do have some observations to share.

Rather than launching the SDK to run my tests, I'm launching my own product, spit out by PDE build. Part of the reason is that I have some additional plug-in tests written using WindowTester, and they care that it's my product which is launched, not the SDK.

There was a gotcha with this. ETF was created before p2. So you just dropped your test bundles into the "plugins" folder of the SDK, launched the SDK specifying the ETF application and test bundle / test class, and off you go. Today, if you do the same, and you're launching the SDK to run your tests, it will still work because the SDK ships with the dropins reconciler enabled. My product does not (which is the default, I believe). So I couldn't simply drop in my test bundles and make them available for ETF to find.

The solution to this is actually quite clean and works nicely.

1. Enclose test bundles in a test feature, expressing the appropriate dependency on ETF. (Hint: I had to edit the feature.xml manually to include it, since I didn't make ETF part of my target, so I couldn't pick it from the PDE editors.)
   
2. Run PDE build to build the test feature after building the product.
   
3. Save off a copy of the product before I test with it.
   
4. Formally install the test feature into the product using the p2 director.
   
5. Launch the product, specifying the ETF application, and a test bundle / test class that has been installed into the product.
   

I've also included in my test feature the EMMA OSGi bundle for measuring the coverage provided by my bundle tests.

Thanks to everyone who voted, provided patches, and general community support to get the ETF supporting JUnit4. It really completes the testing package that is available to developers working with Eclipse and more generally OSGi.