GREYDiscussion notes on using OSGi in Magnolia. NEEDS REVIEWGREY

Introduction

Concepts

See Vivian's presentation: Magnolia - OSGi.pdf
Keywords:

  • bundle: in OSGi-lingo, a "bundle" is what we'd call a "module" in Magnolia, or "plugin" in other applications.
  • obr: OSGi Bundle Repository. This is the mechanism by which an OSGi container can download bundles from an OBR server and deploy/install them in itself.

Clarifications

  • Weight: the added weight of the framework, with Felix, would be somewhere between 700k/1M.
  • Framework complexity: the base framework that bundles are potentially using is fairly small and simple: see http://www.osgi.org/javadoc/r4v41/org/osgi/framework/package-summary.html
  • Bundling: there are maven plugins that do the job - see http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html
  • Server-side: OBR basically provides all we need, and from what I can see will not need anything else than using the plugin above (search for "obr"); the releases.xml can sit in a Maven repository.
  • Multiple OBR URLs can be provided, so, assuming they can be password protected, we've got the EE features covered
  • Lifecycle: in OSGi, a bundle can provide a BundleActivator (see Javadoc above). This class gets called back on start/start with a BundleContext, which can in turn register listeners. We might go for a single Activator, i.e our Core module, to handle ModuleLifecycle as we currently do, by registering one listener that gets called back whenever other bundles are started/stopped.
  • Version handler: same as above, it might just work to let core listen to bundle events and trigger their version handlers accordingly.
  • Module config - this is something I'm not sure how to handle yet - the <module> class which is populated by config:/modules/xyz/config - expose it as an OSGi service ?

Benefits, generally and specifically for Magnolia

Modules

This is the most obvious point.
We basically have most of the features provided, but OSGi take them to the next level: lifecycle management, isolation (we don't have that at all, but could), and it basically forces you to provide clean interfaces (entry points) to your modules.

Services

Even though we do not use a service-oriented architecture, the one provided by OSGi is fairly comprehensive. The whole listener and tracker mechanism is pretty sexy.

http://www.knopflerfish.org/osgi_service_tutorial.html

Two approaches

There are basically two approaches when attempting to merge OSGi and web applications:

  • Provide a Main class in core which starts Felix along a jetty bundle
  • Still provide a regular webapp that starts its own Felix
  • For P.O.C : a simple main class in core which starts felix with the console bundle: from there, let's verify we can install our own modules and hopefully an http service with magnolia!
  • Web console: as a start, we could verify that Felix's webconsole is integratable in Magnolia, even if not having our own integrated gui, it would be an awesome start !

1. OSGi is the container

In this scenario, the "main" application, the executable, is really just the OSGi container (or a wrapper which starts some defaults services for you). You then start a web Service (HttpService), and other bundles which "attach" themselves to the HttpService.

Shortcomings
  • The current HttpService provided by the OSGi spec is limited to the Servlet 2.1 API. As Magnolia is based on a Filter, we're basically stuck. Pax (see links) provides extensions, but I haven't been able to use them properly. Other alternatives are writing a custom servlet that delegate the request to the filter, and basically do all the work that we expect the container to do for us.
  • This implies that the web application is not a regular war. While this is a tempting alternative, especially for Standalone Bundles, this is maybe too much of a shift for our users to get along; I wouldn't implement this scenario if scenario #2 doesn't work just as well.

2. OSGi is embedded

In this scenario, the main application is your regular application server (Tomcat, Weblogic, ...), and the web app starts and embeds its own OSGi container.

Shortcomings
  • Since OSGi isolates bundles from each other:
    • the bundles need to either have very long/complex Export/Import declarations, or be very well separated. This is an issue with Magnolia-core at the moment, for instance.
    • it is quite cumbersome and does not seem to be a "best practice" to expose the whole current classpath (i.e the contents of WEB-INF/lib to the OSGi container.

Status

I don't think using OSGi in Magnolia is a no-go, but on the other hand, I don't see a clear benefit either.
Sure, isolation and so on is great. But once we solve the problems we need to solve anyway, the only obvious missing points that OSGi provides are

  • the web admin gui to obr (and even then, we'd need to work on securing it, see if we can even reuse it at all, customize its layout, so it'd only be a temporary thing)
  • the classloading magic - which, provided it doesn't cause app server issues, should not be that much of an issue.

However, there are a bunch of steps we need to take first, and a bunch of questions we need answers to. See below.

If lifecycle and uninstall features are solved, then we could probably wrap them in the OSGi equivalents.

Using OSGi today would require a lot of work; much more, it seems, than implementing a custom solution, in regards of the benefits it would bring. However, I think we should keep an eye on the OSGi world, and if we're going in the right direction, there might be a point in the future where jumping in would make more sense.

Next steps

Things to do anyway

See Concept Module downloader updater. Most relevant are: lifecycle, uninstall, ...

Followup and questions to find answers to

3rd party libraries

It seems there is a vague trend to have libraries such as commons-*, jackrabbit, and others we use, provide their own OSGi manifest entries. I don't really understand why this is necessary and how this simplifies anything. The way I see it, application-specific bundles should do that themselves IF needed. In the case of Magnolia/Jackrabbit, for instance, I wouldn't see the need to expose ANYTHING of Jackrabbit outside of core.
(and this leads to one of the points that's not yet clear for me; it seems even when embedding a 3rd party lib and not having the need for OTHER bundles to use their packages, they still have to be exported and imported ?)

IOC

How does OSGi play with IOC ? It seems a little obscure to me at the moment. I will start discussion on the Picocontainer mailing list (amongst other things, because I know these guys)

Embedding

One thing I haven't tried extensively is embedding, and use pkgscanner (see links) to expose the complete current classpath to OSGi. I am a little concerned about performance (although that would only be a startup thing) and memory usage - not sure how pkgscanner manages to do its job.
I am also not sure if this really solves isolation issues.

Improvements to the HttpService

There seems to be an RFC for this, but I haven't found any reference yet; it would be RFC-66.https://bugs.eclipse.org/bugs/show_bug.cgi?id=162132

How others do it

  • Confluence: they seem to have some support for OSGi. They also have their own plugin API. What I am not sure is if 1) OSGi plugins are one "type" of Confluence plugins (i think so) 2) They use OBR or some custom service for their plugin repository.
  • Struts 2: also has some support for OSGi. http://cwiki.apache.org/WW/osgi-plugin.html
  • Spring: as usual, Spring has solutions for everything; as usual, they're obscure and overwhelming. It seems "Spring DM" (for Dynamic Module) would be the thing to look after. It also seems to be taking you on a trip into the Spring realm of libraries, as usual. Does not necessarily provide solutions for the web application issues. http://www.springsource.org/osgi
  • Sling: the Sling project probably used OSGi right from the start, and as such did not have to go through the hoops of "making it work with"; however, it seems they've done quite a good job with their own modules and bundling. See their "launchpad" project for examples.

Get started with p.o.c and Felix

Get Felix and start playing with it

Get the felix tgz from http://felix.apache.org/

To quickly overcome the first obstacles:

  • Start from the root of felix directory, not /bin:
    java -jar bin/felix.jar
  • Once in the terminal, type help. You can already try, for instance: obr deploy org.apache.felix.webconsole. This will install and start the Felix Web Console on http://localhost:8080
  • If you get errors: obr list-url will give you the list of OBR URLs it's currently using. There's a chance they haven't fixed the errors I encountered, so get http://felix.apache.org/obr/releases.xml locally, and apply the following vi search/replace: %s/uri="org/uri="http:\/\/repo1.maven.org\/maven2\/org/gc. (basically this file is partly written as if the artifacts it points to where in a subdirectory of where it resides, while they are on the central Maven repo)
  • Then you can remove the default obr url and register your local copy :
    obr remove-url http://felix.apache.org/obr/releases.xmlobr add-url file:///Users/gjoseph/tmp/releases.xml
  • To view the currently installed bundles: ps
  • To start a bundle you installed or retrieved from obr: start N where N is the numerical ID of the bundle (i.e. the number in the first column displayed by ps
  • To install a bundle from a local jar file: install file:///path/to/file.jar. It must be a URL, not just a path.
  • Once you managed to start the web console, you can access it through http://localhost:8080/system/console and it's pretty neat !

Build Magnolia bundles

Bundling

2 options for maven:
1)s using the <packaging>bundle</packaging> with:

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.felix</groupId>
        <artifactId>maven-bundle-plugin</artifactId>
        <extensions>true</extensions>
        <configuration>
          <instructions>

2) using the default jar packaging:

<plugin>
  <groupId>org.apache.felix</groupId>
  <artifactId>maven-bundle-plugin</artifactId>
  <executions>
    <execution>
      <id>bundle-manifest</id>
      <phase>process-classes</phase>
      <goals>
        <goal>manifest</goal>
      </goals>
    </execution>
  </executions>
</plugin>

If using jar packaging, no need for "extensions"; this is only needed to use the specific "bundle" packaging.

Proof of concept project

http://svn.magnolia-cms.com/svn/community/sandbox/osgi/
Read the OSGi_notes.txt

(warning) All classes and build files in there are tryouts, nothing definitive at all, just attempting to piece things together and see how things work.

Libraries

Under the libs/ folder, you'll find attempts re-packaging our main 3rd party dependencies.
Some of the commons- jars are already OSGi ready.
A custom bundle was needed for logging/slf4j, I couldn't figure out how to deploy slf4j otherwise - they have OSGi manifest entries, but i always had errors.
commons-chain's bundling was wrong so we re-bundle it.
All the rest is in the "others" bundle. Some libraries in there also have their own OSGi manifest entries, and for some of them, we're overriding.
In general, see comments in the pom files.

The main problem is that (it seems)

  • the bnd tool/maven-plugin ignores exclusion entries from the pom
  • dependencies' manifest entries have priority over whatever we try to define in the pom (and I had a bunch of issues with bundles requiring specific versions of others, etc)

If you don't have an slf4j impl dependency, you get an error such as
{{ Unresolved references to [org.slf4j.impl] by class(es) on the Bundle-Classpath[...]: [org/slf4j/LoggerFactory.class, org/slf4j/MarkerFactory.class, org/slf4j/MDC.class]}}.

This shows that 1) the bnd tool / bundle-plugin analyzes stuff and does not let you build a bundle that will not deploy. Good. Solution to this is to either bundle an impl package for slf4j or to explicit require that dependency (through Bundle-Import or Require-Bundle, I assume)

  • (warning) TODO - check status of README and other files !!

Specific modules

  • magnolia-osgi-activator: bundle providing an activator (info.magnolia.osgi.activator.MainActivator) which starts (or would do so) the http service and Magnolia etc.
  • magnolia-osgi-launcher: an executable jar which provides a Main class, attempting to start an OSGi container (Felix) and magnolia bundles (possibly the magnolia-osgi-activator)

Unsorted links

Possible tools:

New links 2009-06-23