p2 UI policy and Declarative Services

This is another post in what is becoming a short (so far only two) series about moving a product from 3.4 to 3.5.

After I got my build working, the next step was making sure that I could update from one product version to the next. I was especially excited about the resolution of https://bugs.eclipse.org/bugs/show_bug.cgi?id=246060 which allows for .qualifier to be replaced in a product version. No longer would I have to manually increment the product version number for purposes of updating to and testing a nightly build.

So my plan was:

1. Run PDE product build to generate version 1.0.0.abc
   
2. Unzip 1.0.0.abc to some location.
   
3. Run PDE product build again to generate version 1.0.0.def
   
4. Launch 1.0.0.abc, point it at the repository for 1.0.0.def, and update.
   
5. ...
   
6. Profit.
   

Unfortunately, when I launched 1.0.0.abc, the Install New Software dialog didn't have a way for me to add a new repository. Ditto for the preference page.

Turns out there is a more robust set of p2 UI building blocks in 3.5, which is handy for RCP developers. That is described in great detail here: http://wiki.eclipse.org/Equinox/p2/Adding_Self-Update_to_an_RCP_Application

I should mention that the RCP-p2 example in 3.5 is leaps and bounds ahead of the one from 3.4 (there wasn't one) - so props to the p2 UI team on that.

At any rate, the wiki page tipped me off that there is a UI policy which controls what components are showing and enabled. This policy is implemented as an OSGi declarative service. What really threw me for a loop is that I wasn't trying to do anything special with this policy. I just wanted the stock SDK one since our product is based on the SDK.

Debugging the Policy Behavior

I stepped through the preference page code and discovered that the SDKPolicy wasn't getting discovered as a service (it was just getting an empty Policy every time). So this sent me down the route of launching with -console to see the OSGi console and look for the policy service. After fighting with the filter syntax for the services <filter> console command, I googled a bit more and found these useful runtime options for spitting out verbose DS logging information. I turned those on but I didn't get anything logged. I was pretty stumped at this point.

Then a light bulb came on: maybe declarative services wasn't running at all? A quick ss ds at the console showed that it was RESOLVED but not active! I did a start to spin it up and all of a sudden a deluge of DS logging information printed out. And then SDKPolicy started working, and voila my p2 UI was working.

It turns out the root cause is that we had a custom config.ini in 3.4 to specify a custom osgi.instance.area location. This was screwing up the start level for the ds bundle. I switched the product to generate a config.ini for me, did a new build, and everything worked. I plan to migrate the osgi.instance.area configuration step to a p2.inf file, which is what the platform releng guys do.

Useful Links

[1] Equinox Runtime Options
[2] Explore Eclipse's OSGi Console
[3] Around the world in Java: Getting Started with OSGi Declarative Services
[4] p2 UI policy bug #1
[5] p2 UI policy bug #2

Debugging PDE Build and the publisher

I posted a problem to the PDE newsgroup last week about unexpected requirements in my product feature. This was in the context of moving a 3.4-based product to 3.5.

The general issue was that the director wouldn't install my product because of an unsatisfied requirement. It wasn't clear to me where this requirement was even coming from. Somewhere, there was some metadata in my plugins/features that expressed a dependency that had worked fine in 3.4 but failed in 3.5. My theory was that if I could capture when the publisher was generating the requirement, I'd be able to see the source of that requirement and squash it.

Tracing

First attempt was to turn on tracing for the p2 components. I managed to find the org.eclipse.equinox.internal.p2.core.helpers.Tracing class which listed out the different options. I stuffed those into a .options file:

org.eclipse.equinox.p2.core/debug=true
#org.eclipse.equinox.p2.core/generator/parsing=true
#org.eclipse.equinox.p2.core/engine/installregistry=true
#org.eclipse.equinox.p2.core/metadata/parsing=true
#org.eclipse.equinox.p2.core/artifacts/mirrors=true
#org.eclipse.equinox.p2.core/core/parseproblems=true
#org.eclipse.equinox.p2.core/planner/operands=true
#org.eclipse.equinox.p2.core/planner/projector=true
#org.eclipse.equinox.p2.core/engine/profilepreferences=true
org.eclipse.equinox.p2.core/publisher=true
#org.eclipse.equinox.p2.core/reconciler=true
#org.eclipse.equinox.p2.core/core/removeRepo=true
#org.eclipse.equinox.p2.core/updatechecker=true

Then the trick was to pass along those options to the AntRunner app which drives PDE build. I added -debug path/to/.options into my arguments to AntRunner. Running the build again I got two things, neither of which were helpful:

1. Passing -debug to the Platform also passes -debug onto Ant, thanks to AntRunner. So my Ant ran in debug mode which really clouded the issue with about 8mb of debug output.
   
2. The publisher only outputs two trace statements: start and finish. Nothing about what it is publishing. This may be a candidate for enhancement.
   

Based on these results, I reasoned that nobody else must be using this technique to solve their p2 problems. Moving on.

Stepping through the publisher

Next up: run AntRunner with Java debug enabled so that I could connect remotely and set breakpoints in the publisher actions. I added the appropriate JVM args to enable the Java wire debug protocol. Started the build again, connected up and started setting breakpoints in various publisher actions.

Since the rogue requirement was getting added to my product feature IU, I added a conditional breakpoint in FeaturesAction to look for that feature being processed.

Then, since the problematic requirement was org.eclipse.core.resources [3.4.0,3.5.0) I added another conditional breakpoint in getVersionRange to watch for incoming feature entries with 3.4.0 as their minimum version.

I did finally discover the problem: I had a bunch of old, outdated entries in my product feature's feature.xml, which included references to several different versions of o.e.core.resources. After I ripped those out, I had a successful build and director install.

Conclusions

• Do not pass the debug flag to AntRunner for purposes of debugging platform code unless you are prepared to wade through volumes of output. (I guess this is a feature of AntRunner - https://bugs.eclipse.org/bugs/show_bug.cgi?id=5672)
  
• It was not at all apparent to me to debug p2 actions by setting up a "remote" debug session with PDE build running inside of AntRunner. But it was sure as heck helpful once I figured it out.
  
• I am actually glad that I ran across this problem, and that p2 is enforcing these types of constraints, because it helped me clean up outdated dependencies in my feature.
  

How are you debugging your p2 builds??

Cloning a profile using p2

I heard at the p2 BOF at EclipseCon that you could use your profile as a p2 repository. This sounded like a cool way to take an Eclipse setup that you've customized and replicate it into a new install. Granted there may be better ways to do this with shared bundle pools and what have you, but lets set that aside for a moment.

Since 3.5M7 is out, I decided to take a stab at cloning my 3.5M6 setup into 3.5M7. I had installed SVN and DTP along the way so I was hoping this would take the pain out of having to go out and re-provision those items from the web. (Of course I should probably upgrade DTP to the M7 version, but again, lets set that aside.)

Steps:

1. Unpack 3.5M7 into a new directory
2. Launch it
3. Help -> Install new software...
4. Click the Add button to add a site
5. Click Local to browse
6. Select the .profile directory from previous install. In my case, file:/C:/eclipse3.5M6/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile/
7. Give it a name, hit OK
8. You should see all your plugins from that profile. You might need to uncheck "Show only the latest versions" and "Group items by category"
9. Check all the plug-ins you want (sadly, I couldn't find an action to mark multiple at a time)
10. Finish the wizard and restart the platform



And you're done! Awesome. Thanks to Simon Kaegi for implementing this.

Favoring immutability

A few years ago, when I read Effective Java for the first time, Item 13 really stuck in my mind: make your objects immutable wherever possible. You can share them freely. You don't have to worry about checking to see if that Person object you created still has a name. You made sure of that when you created it. And while I don't do a lot of concurrency programming, I've seen time and time again how much you get "for free" when you use immutable objects in multi-threaded environments. So that's another benefit.

OK, so I read the book, and moved on. Then I start working on a new project. We happened to pick GWT as part of the tool stack. During the early stages, a light bulb goes off in my head: hey, I'm writing a bunch of new beans in Java - I should make them immutable! Silver bullet, right? All my problems will go away. So I write a bunch of immutable beans.

Then I tried to serialize my immutable objects, and BAM, I hit this RFE: Serialize final fields. Turns out, its actually kind of hard to take an immutable object, vaporize it into a bunch of bits, and then reassemble those bits back into an immutable object on the other side. So I ended up with a bunch of objects that really, really wanted to be immutable but couldn't. Frustrating. Turns out that a "90% immutable object" just doesn't have the same feel to it.

Alright, so lesson learned (so I thought), moved on. Then I started working on a different project. Same type of deal, writing a bunch of new beans, want to make them immutable so I don't have to worry about whether or not that Person has a name. This time I'm working with JAX-WS, and again the serialization problem hits me right in the face. Ultimately I end up leaving my immutable beans alone, and writing a bunch of data transfer objects (similar to the immutable ones, but mutable). Then I could send those across and reassemble them into immutable versions on my own.

The saga continued when I made my way into JFace Data Binding. I had my Person with their name, age, height, weight, etc. I wanted to have a simple data entry form for a new Person. Problem is, with my immutable Person object, I basically have to set all those properties all at once, during construction. Stopped in my tracks again, since for data binding I basically need a standard Java bean with getters AND setters. Again I find myself writing intermediate objects which ultimately become their immutable counterparts.

The last part of the story came today, when I was poking around the p2 APIs. I was trying to see how the various properties of an IInstallableUnit got set. I find that there are no setters on the interface. Then I stumble across InstallableUnitDescription:


Once created, installable units are immutable. This description class allows a client to build up the state for an installable unit incrementally, and then finally product the resulting immutable unit.

A-ha! So I'm not the only one doing a bunch of intermediate stuff just to get to that final immutable object (pun intended).

So I put these question to you readers: How are you using immutable objects in your application? How about in your Eclipse applications? What lessons have you learned?

Thanks, EclipseCon '09

This was my first year at EclipseCon. It was great to put names to faces and mingle with the community.

Here are some things that I learned, in no particular order.

• Choose Import-Package over Require-Bundle (I feel like I am a bit late to the game on that one, but appreciate being let in on the secret)
  
• Someone is using RCP to monitor nuclear power plants in France.
  
• Migrating a traditional enterprise application to OSGi is a hard problem.
  
• I am running out of excuses not to use modeling tools.
  
• I should scrap my homegrown explorer and use Common Navigator.
  
• e4 is coming here. Start using it.
  

Aside from the technical learning, I felt the pulse of the community. And the message is this: contribute. Submit patches. Write bug reports. Participate in newsgroup discussions. Help wanted. I think this is a great message and frankly I'm overwhelmed by the number of areas that I want to contribute to. I think I need to submit a proposal to the wife to get more free time...

I liked the session voting. I wonder how we could make it paperless? Maybe touch your RFID badge to a +1/0/-1 monitor or something..

I hope I am able to come again next year.

What did you take away this year (aside from a fish)?

Keyboard shortcuts and accessibility in GEF

Here is a handy reference list of keyboard usage with a GraphicalViewer. Zoom in and zoom out actions must be implemented for those key bindings to work. For panning support, use a PanningSelectionTool.

Navigate left	Right arrow
Navigate right	Right arrow
Navigate down	Down arrow
Navigate up	Up arrow
Navigate into a container	Alt+Down arrow with the container selected
Navigate out of a container	Alt+Up arrow
Cycle through selection handles	Press the period key with a part selected
Move a component	Cycle once to the Move handle using the period key Use navigation keys to move Press Enter to accept new location Press Escape to cancel the move
Resize a component	Cycle to desired resize handle using the period key Use navigation keys to resize Press Enter to accept new size Press Escape to cancel the resize
Select multiple	Hold down Ctrl Use navigation keys to navigate to additional components Press Space to select additional components
Select in sequence	Hold down Shift, use navigation keys to select additional components
Zoom In	Ctrl +
Zoom Out	Ctrl -
Select multiple w/ mouse	Hold down Ctrl and use the mouse to select components
Pan when zoomed in	Hold down Spacebar and drag the mouse

Favorites from 3.5M6

There are some goodies in 3.5M6. Here are a few of my favorites.

• DataBinding support for the SWT DateTime widget. No doubt this will be useful. [169876]
• DataBinding MultiValidator: Add support for dependencies other than those expressed by IObservables. We have run into this roadblock a few times and its good to see it has been lifted. [235859]
• Copy/paste support for installed software list. Helpful for reporting issues. [227220]
• Enable CTRL-mouse navigation for implementing classes. You know when Ctrl-Click takes you to the interface instead of the implementation? Now its smart. This is awesome. [44277]
• New tracing API. Recently noticed we need debug level output in our product and this will be useful in implementing such a log. [258705]
  

EclipseCon 2009 in 1 week!

p2 likes its 404s straight up

Setting up p2 to work with a repository that is behind HTTPS has been quite challenging. First I ran into this Java bug which was affecting ECF, the underlying transport for p2.

After moving past that problem, I was getting a lot of "No repository found at.." error messages, even though I knew damn well there was a repository hosted at the URL I was specifying.

After digging through the p2 code with the debugger turned on, I found the problem. You can deploy your repository metadata in two formats, compressed (JAR) or uncompressed (XML). When the repository manager goes out to a URL to look for the repository contents, it tries to find the JAR version first. If that fails with a 404, then it looks for the XML version.

I was deploying the XML version, and the fallback behavior wasn't working. It was pulling in an empty artifacts file, and trying to parse it, which of course failed. Turns out the web server was sending a 302 redirect with a user friendly page explaining that the page had moved. I found this by intentionally hitting a bad URL with Firefox, using Firebug to examine the response headers:



You can see that the 404 is coming AFTER the 302, which is too late. ECF was relying on the 404 to cause a FileNotFoundException, which translates into a known failure status in the p2 repository manager code, from which it can recover and try again with a different filename.

After tweaking the web server to return a 404 immediately, I was able to refresh the repository in the Software Updates dialog without any errors.

First look at e4

I finally had some time to take a look at the first e4 Milestone. Download and install from here. New and Noteworthy here.

My main interest is in the declarative UI and CSS components. So I spun up the photo demo per Boris' instructions. It works and looks about what I would expect for the first milestone. I found the CSS file and tweaked it a bit, restarted, and saw my changes take effect. That was cool. Some thoughts on CSS in Eclipse:

• Styling RCP apps can be left up to a designer. I wonder what the collaboration challenges will be like.
  
• What does this mean for native widgets? Will people go hog wild and make their application so customized that it doesn't look like it fits in anymore? Ditto for plugins. I see a need for a whole host of new UI standards..
  
• I wonder how well you can hot replace your CSS tweaks.
  
• I haven't looked at Tk-UI (the CSS engine) much yet, but I wonder to what extent they support the CSS spec, and how much of that translates meaningfully into SWT constructs. First thing I'd be looking for is a mapping document.
  

Quick spin through the photo demo is complete. Lets look under the covers at the app itself. First class I look at is ExitHandler. Except, its not a handler. Or at least it doesn't extend AbstractHandler.. what's going on here? I can only guess: dependency injection. This will take some getting used to.

I also see an .xmi file. Quickly I can scan through it and see the structure of the entire user interface. I'm comparing this to the umpteen times I've had to follow that "parent" instance variable up the tree, to figure out where the heck a particular Composite lives, and what the structure is.

I can see already that I'm going to need Ctrl+Click to go from the .xmi file to the relevant classes. (Maybe I just need the appropriate EMF editor)

I have to wonder what newproject.js is. I can only think it is related to writing plug-ins in other languages. I see mention of Rhino as the toolkit. Perhaps the BIRT team can lend some expertise in this area.

I hope to get to the Resource model changes next time.

Welcome

A little about me. I've been a Java developer in the Washington DC area for about 5 years. I started out in the web tier, and began building Eclipse apps in 2005. I took a short hiatus from SWT to work with GWT for about a year, then back into Eclipse last spring.

I've worked with a wide variety of Eclipse technologies, including RCP, EMF, GEF, and BIRT. Lately I've been spending a lot of time hacking the Equinox p2 component.

Over the years, I've accumulated a fair amount of knowledge about the inner workings of Eclipse; I hope to share what I've learned on this blog. And so off we go..