All the small things

I admit it: “Using OSGi for the first time isn’t as fun as any of the hip new scripting languages.” Actually, it can be quite painful – especially in the beginning. You’re forced to think about things you actually don’t want to – or better put – you thought you don’t want to. The good news is that there is tooling support (like: BND, maven-bundle-plugin or Eclipse PDE) that can help you getting around most of the time. Unfortunately, there is still the remaining bit, that keeps nagging your brain with the “what could possible have gone wrong”. The one thing that bothered me most about it is that in many cases the steps you need to figure out what’s wrong are the same. It is simple, repetitive work you have to do, mostly only using basic OSGi know how. Also surprisingly simple, there is no tool at hand that can help you going through the necessary steps – well, till now. Lately I was so annoyed by the situation, that I started my own little open source project, aiming to fight this exact problem. Getting you up to speed and solving the simple problems you usually have when working with OSGi – especially when you start using OSGi, was my goal to achieve.

So, I am happy to introduce the (OSGi)…

Before going on explaining what it is about, what it can do and why you might wonna take a look at it, be warned! As of now, it is still a proof of concept project. It sure does offer help and is better than nothing, but I am not satisfied with some details of the API and implementation, so expect severe changes within the next couple of weeks (if not months). You’ve been warned

Anyway, the goal behind the inspector is to offer a simple framework to detect common coding mistakes in OSGi. Very simple things manifesting in ClassNotFoundExceptions, missing services, unresolved bundles or not attached fragments. It certainly is not the “one to rule them all” solution, but it is a (hopefully) decent effort to help you move faster with your OSGi development.

So, what is the inspector really. Currently, the inspector consists of three major parts: the core, the reasoner and the web GUI. The core offers basic functionality to analyze the running OSGi environment in a convenient way. This includes, but is not limited to:

• imported, exported packages with their versions, properties and directives
• provided services
• pending, removed and failed service requests (partially depending on a R4.2 runtime)
• framework events with special treatment of errors
• dynamic imports
• fragments
• framework capabilities
• class look-up within the runtime (which bundles provide a specific class?)
• …

This all is realized in one single bundle you have to deploy and start as the first bundle in your OSGi runtime (any runtime is fine, as long it is a R4.x one from any implementer).

The reasoner is the second bundle and provides the logic to solve your common problems it uses the core bundle for getting the required information to make reasonable guesses on what’s wrong and even more important, on how to fix it quickly. This part is the most experimental one. The idea is to have an extendable API that is not limited to standard OSGi problems but can be extended to solve your domain specific problems as well. The current implementation only includes one use case yet – ClassNotFoundExceptions propagated to the OSGi runtime. Other reasoners are already work in progress like failing service requests, missing services and not resolved bundles in general (bundles and fragments actually). If you can think of other common OSGi problems, please let me know!

The web gui is a convenient way to take a look at the discovered problems and the suggested solutions. It is very simple and straight forward in order to solve the problems at hand. You don’t need to use it, you could also create a simple command line tool doing exactly the same. I actually would have preferred a command line tool over a web gui, but because of the lack of a common API to provide command line extensions over OSGi implementation, I decided to not yet provide something right now. Maybe later if the the feedback suggests otherwise.

To get you started in no time, I also provide an assembly based on pax-runner that starts all bundles required correctly and allows for easy provisioning of your own domain specific bundles. There is even a very simple tutorial available at github.

So, if you’re interested, take a look at it, give it a spin and let me know what you’re thinking! Any feedback is greatly appreciated!

Here the links for further details:

Binary download: http://github.com/mirkojahn/OSGi-Inspector/downloads
Tutorial: http://wiki.github.com/mirkojahn/OSGi-Inspector/tutorial
Source Code: http://github.com/mirkojahn/OSGi-Inspector
Bug tracking: http://github.com/mirkojahn/OSGi-Inspector/issues

I also have plans for additional features for the next version (maybe this summer) of the inspector, like making the Web GUI extendable, including more JVM information and debugging capabilities like heap dumps or per bundle memory usage, statistics about the application like running threads, used frameworks and their versions, runtime bundle dependencies, search over many of the aforementioned features and much more, but this will be mainly based on user feedback. So stay tuned and let me know what you’re missing.

Finally here are some screen shot to give you an idea how the proof of concept looks right now

As a final note, one may ask if OSGi experts are becoming obsolete by tool support like this. I don’t think so! In my opinion, this is something that is crucial for a wider OSGi adoption and an important helper to silence the OSGi opponents by solving 90% of the beginner problems with regard to basic programming errors. It can serve as a teaching tool to tell people what not to do or how to do things better. OSGi experts now can focus on the important stuff like how to design and build truly modular applications – as they should have been for a while.

Yours,
Mirko

It feels like recently extenders became extremely fashionable among the OSGi community. Eclipse is using an extender like pattern to track their plugin.xml configurations and Spring DM introduced the plain osgi extender together with a special web extender. It seems like everyone has one lately, but what is all the buzz about and how should or could you use one – and even more important, when should you use one. As always in software development there is no clear black and white answer to this, but this blog post at least tries to shade some light on this topic to help you making the right decisions.

The buzz, what is it about?

Before the extender hit the light, people were advertising the white board pattern as THE OSGi way of doing things. A way of inverting control, taking the burden of house keeping from the consumer like it is required with the well known listeners. On the OSGi website you can even find a good paper on “why listeners are considered harmful“. Well, things change or better progress and with the extender another problem got addressed. The inconvenient with the white board pattern is that the client still has to take care of some stuff ending up in boilerplate code. Here is where the extender has its strengths. The core idea is to have a designated bundle that scans all already deployed bundles as well as all bundles that are being deployed during runtime for specific features, properties, interfaces, annotations, what ever you want in order to take appropriate actions on behave of this bundle. These actions can be something simple as registering a service for every implementation of a certain interface or as complex as doing some xml transformations of configuration files that will be loaded in your system triggering a whole cascade of events. The key point is that it is completely up to the extender what to do and how during deploy time.

Imagine you have your application extensively using the white board pattern without an extender. All “consumers” of your service have to publish a specific service in the registry in order to get recognized. You have to exactly specify what the signature of this service has to be. This can be any number of interfaces or just properties you register along with the service. All this is just fine, but when it comes to changes of your approach you have to let all you’re clients know about the changes and wait till they have adapted/ migrated or provide a runtime compatibility layer that checks for the old and the new syntax. With the extender, you can still use the white board pattern, but this time you take care of registering the service. Now, everything is defined in your extender, which interfaces are important, what properties to add. As soon as it comes to a change in your design, chances are good that you can just adapt the extender in order to register the client services with new properties, use different interfaces (if possible of course) or even use something advanced as a generic proxy that wraps around the old interface and registers itself with the new API. You can now intercept service calls, if you like or need and transform the transfered data as you please. But the best thing about it, despite the mentioned is that all this happens during deploy time of the bundle so it just happens once without the need for your clients to touch any of their code. This makes the interfaces between service providers and consumers more solid and less likely to break.

Pros and Cons – abridge

Of course, there is no such thing as free lunch. The extender also has its limitations, you should well be aware of. Here is a short list with the compiled pros and cons. Of course, depending on the scenario you might find more to either ones. If you think I forgot something important or don’t agree, don’t hesitate to comment. Constructive feedback is always appreciated.

disadvantages:

- One of the first things that pops into ones mind is the problem of disabling the extender for a specific bundle. This might be interesting, because you don’t want to start/enable all bundles at start-up or just don’t want to provide all the functionality at once (f.i. to increase start-up speed). There are ways, like white/ black listing bundles, lazy activation triggered by properties or manual selection based user input. Although all this is possible, it is somewhat against the idea of the extender, which is automatically processing and removing repetitive tasks.
- The second difficulty is the customization. Of course you can define a specific format that enables you to customize the way your extender works and what it does, but there are scenarios where you can’t upfront decide what to do. A common way to work around this is defining your own DSL describing your “intent” in an abstract way, like in your own DSL. However, this introduces another layer of complexity and can make your application/design even more complex – depending on your scenario of course. This is not a problem of the extender, but a general problem one usually faces when working in OSGi. Everything is intended to be generic, but at some point you have to be specific on what you actually wonna do. So, I guess it is not a real drawback, but something to consider.
- Last but not least every automatic mechanism requires awareness of developers. Everyone using your bundles should be aware that something is happening behind the curtains. Otherwise chances are high the mechanism will be circumvented or miss used. Again, I think this holds true for many mechanism or patterns, but is well worth mentioning here.

advantages (assuming done right):

- First of all an extender means less burden for the developer. The overall goal is to remove boilerplate code, typical copy & paste use cases and deterministic, error prone tasks. For most developers this is not only boring, but also a major pain point when fixing a bug or misconfiguration. By eliminating or at least minimizing such repeating tasks, you’re doing the average Joe and ultimately yourself a big favor.
- You might have guessed it. Eliminating c&p from your code base also makes your application more robust to changes. I mentioned that already above, but just let me emphasize that the ability to express the intend, rather then the actual how is one of the greatest advantages.
- Inversion of Control. Yes, I know it is a buzz word and it is usually used grasp attention, but here it really makes sense. Like the whiteboard pattern one does remove the burden of the consumer. What kind of burden this is, depends on the extender. In many cases it will be registering the services the right way, indicating a specific functionality or setting a configuration. The beauty about this is similar to the white board pattern. Less burden for the “extendee” (the one being extended by an extender) and a centralized management logic, which can be updated, improved by the “extender”. I think it is important to note here that the extender can provide or consume, depending on your design, unlike the white board pattern.
- Of course, one advantage generates others like less source code for instance. Unless you’re getting paid by lines of code you’re producing (just a joke -I honestly hope no one is evaluated on that one!), having a smaller code base is always a good thing. Other advantages would be less complex extendee bundles or simpler adaption to new requirements.

Conclusion and outlook

As you could see, extenders are the next evolutional step in the OSGi world. There are great new ways of designing your applications that can make your application even more robust and simpler to use. However, there also comes the responsibility to use the technology wisely and not just for the sake of it. As seen with Eclipse and now Spring, more and more advanced extender concepts are popping up and we certainly haven’t seen everything possible yet. In a future blog post, I am going to look deeper in how to create an actual extender, what to keep in mind, how the up-coming OSGi R4.2 spec will help us here, what kind of extenders are there already and of course common pitfalls one may face implementing your own. Stay tuned and have fun experimenting with extenders.

Cheers
Mirko

After some less OSGi technology centric posts I am now getting back to the meat and will highlight the problems involved with dynamic imports. In well designed OSGi applications the “DynamicImport-Package” header should be obsolete. Unfortunately not every code is OSGi aware. Actually it is not the missing OSGi awareness that is the core of the problem. It is more the notion of making wrong assumptions on the visibility of classes and the misuse of context class loaders in 90% of the cases I saw. But what is the problem in a concrete example?

Well, a common example is JAXB. You have a library/API bundle which for itself provides no real functionality, but needs to be “enhanced”. You need actual implementations to provide functionality. Unfortunately your library needs access to those classes. OSGi is very strict with its class path definition. This is done in a static manner and has to be set during build time of your bundle (Import-Package header in the manifest), but your library provider has no idea about the implementation bundles (and it shouldn’t!). As you can see there is no way of defining such dependencies at built-time. In OSGi the only way to enable this without modification of the bundle code is to include a dynamic import header.

Eclipse soon realized that this is an issue. Especially the undirected import forces in many cases a binding to bundles that are not really related to the bundle defining the import. This causes issues with restarts and updates of the library bundle, because a lot, if not all bundles, can be affect by that. Even worse, it can mean that you end up finding the wrong classes (and I am not yet talking about versions). Eclipse, while migrating to OSGi had a huge amount of legacy code not aware of OSGi concepts and so they were the first really feeling the pain. That’s why the Eclipse community introduced the concept of buddy class loading. Although a better way by providing a scope (you have to define that you’re a contributor on the one side and declare that you’re open for that on the other side), it doesn’t solve the problem of different versions or provider selection in general. 

Quiet a few people assume this is the only way to work around this kind of problems and they are correct when you don’t have the possibility of adding you’re own mechanisms or the API prevents other measures. However, if you have the access to the source, it is likely that you actually can do something that solves the problem. 

While working on my thesis, some years ago, I came across this problem and in some cases you are able to use basic OSGi mechanisms to get what you need. The core idea and the requirement is, that the library needs the implementation class and offers a way to either inject the object or tweak the class loading mechanism to actually find the class you are looking for. So the question is how to gain access to the implementation object or its class loader for that matter. For the ones familiar with OSGi this should be obvious. We instantiate the implementation in our implementation bundle and offer that one as a service. We can register that object with the interface(s) theses classes are implementing or the annotation class they got annotated with. This is an universal key to identify the correct service. The nice thing about this is also that OSGi will handle the version problem for free. With Spring or the up coming RFC 124 we don’t even have to write a single line of code in most cases, we don’t even have to touch the bundle. All we need is to provide is a fragment attached to the implementation bundle with a spring configuration inside creating a bean and publishing that one as a service.

Ok, we now have our implementation objects flying around in the ServiceRegistry. The next step is providing access to those objects in our library bundle. This is the actual tricky part and the point where you’ll most likely will face problems. Your API needs to provide a way of injecting these service objects and should allow you to dynamically manage them (add/remove/update). Otherwise you’ll end up with a static solution that once started won’t be able to adapt to the dynamic nature of OSGi. If you’re API provides/ uses a class loader look up that can be tweaked you’re the lucky one. Here you can provide a delegating class loader that dynamically queries the ServiceRegistry for the fitting object and returns its class loader. This is what we did in our OSGi-efied UIMA implementation. The drawback here is, once you provided the defining class loader you lost control on how to resolve others unless your library has some mechanisms for that, which is unfortunately not very likely.

As you’ve seen, you not always have to go for a dynamic import. All the mentioned approaches have their advantages and disadvantages and none of them are “perfect”, but at least you have the choice choosing the one fitting best to the specific problem context.

If you found other ways of migrating problematic legacy code, please let me know. Maybe there is a solution out there, I was just not aware of. You never know

Cheers,
Mirko

Usually when I am blogging, I am talking about the latest technology, standards or general trends. This time however, I wonna talk about politics. No, not about the elections in US, but about politics in software development practiced by the big players to achieve their business goals. Don’t get me wrong from the start. I think this is completely normal in general. We all are trying to achieve our goals the one way or the other, but something just doesn’t feel right…

Yesterday I read about the latest news concerning Sun’s plans about the future of JSR 277, JSR 294 and its new plans on inventing componentization for its JVM called Jigsaw. I hate to say it, but for me, it seems like Sun behaves like a small child trying to insist to be the one driving all development and not letting someone else play with its toys. It’s anything but the behavior of a well establish and industry leading company or one who wants to become one. First they completely ignored the problem of componentization in Java while persuing enterprise development with J(2)EE and missed their chance to actually set the foundation for real software reuse what you would expect from and enterprise ready language specification. Later, when OSGi began to rise they started the JSR 277 and tried to create a competing standard in secret, which didn’t work out too well as we all know. Now, because of the pressure of the community and the lack of showing a better solution they are abandoning the JSR and start developing their own “internal” componentization approach with the Jigsaw project. Of course they are claiming it is not intended to be used outside their own use case for componentizing the JVM, but it is hard for me to believe this. Call me paranoid, but for me it sounds much more like an attempt to develop another system, which after completion is suddenly moved into an official standard. Sun’s statement that they are going to revive JSR 294 and are inviting even the OSGi Alliance to participate to work on it feels more like a distraction so that Sun is able to pursue its plans with Jigsaw in private that without public notice they suddenly can come up with a self made de-facto standard. Again, I might be to paranoid, but it just doesn’t feel right. Is all that just a coincident? I don’t think so. Why can’t they embrace the work done already and see it as a great chance to propel their Language and create a true reusable software stack, no other vendor can offer? Hal Hildebrand just blogged about Sun’s attempt to introduce this new project and the way they are trying to persuade big industry players about their great intents… Well, you should really read his post about it and please tell me, if they have such humble goals, why does everyone they consult have to sign a NDA? I strongly believe that if those ideas are so great, why not share them and let the community decide and participate?

To wrap things up, I ask all of you who feel like me, share you’re opinion! Comment on Hal’s post, blog about it, link to it, spread the word. I believe we – as a community – have to stand up and say what we think to show Sun, that those kind of guerrilla tactics, especially when so easily to look through are not working. Keep your eyes open and don’t fall for the dark side

Cheers,
Mirko

A lot of people are following the buzz about OSGi now and are excited about all the things it can do for you. A great world of new opportunities. Now, after the first draft of OSGi 4.2 it very much looks like we’re getting even more as an official OSGi standard (with RFC 124). The formally called Spring Dynamic Modules (in short SpringDM) is going to be OSGi-efied. Today, I want to take a short peak into what this can actually mean and how it could affect the software/ component reuse in your software stack even today.

First things first, when I talk to people about OSGi the most common complain is nothing OSGi specific at all. Basically it is the problem of every API out there. If you implement against an open or proprietary API you’re bound to it to some extend. Ok, this is besides the issue of being to complex. When done right, which includes using all the versioning features OSGi offers it is unfortunately kinda true. We’re still lacking the right tooling support here. Anyway, let’s stick to the first one first (I’ll cover the tooling problem in a later post). Speaking of proprietary, it is true you have to add custom headers to your jar files, but this alone doesn’t hurt you. You can still just use the jar as – yeah well – a simple jar in your web container or any other Java application without any problems at all. This was simple, but there is more of course – the OSGi doesn’t only specify manifest headers. There is also an API you use and program against. Activators for instance can be used to do something like setting up or shutting down your bundle and opening or closing connection pools to databases, registering services signaling that you’re bundle is there and ready to get its work done. Assuming you are not using any of the compendium APIs the imposed layer is rather small and can be encapsulated in a single package per bundle and more often just one single class.

However, as soon as you start using those few classes they become part of your jar and thus you’re bound to them. So, we’re still stuck right? Well, not really. First of all, every OSGi expert (and probably any other sane software engineer) will tell you not to intermingle your domain logic with middleware APIs of any kind. As already mentioned, with OSGi, you can usually keep this layer pretty thin. With Spring DM we can make it now even thinner and separate domain concerns from the container entirely. So, how can we achieve this. Of course, Spring is not doing any magic, although it does a pretty good job. All it does is actually providing an IoC framework, which declaratively wires beans together. Plain simple speaking, that’s about it. You could also use other IoC frameworks like Google Guice or pico container if like, but the integration in OSGi with Spring is pretty unique. The big question remaining is how we can best separate these two concerns.

To start with, you design your domain logic in a bean driven way. By that you not only achieve a well testable domain code, you also remove hard wired dependencies not relevant for your core needs – a very common practice in Domain Driven Design, to throw in some buzz words Package all that code in one or more bundles depending on your needs. What you got is the code base you can reuse in any environment. Now, we need the glue code to actually make it work in OSGi – the package with the few OSGi specific classes. Those you now have to put into a different bundle or fragment (more on this later). By this separation you no longer deliver container specific code, if you don’t need to. Even more, depending on your deployment scenario, you can change and adjust the configuration to your specific needs, all separated in different bundles/fragments. Removing configurations from your domain bundles has the advantage of independent versioning of you actual code and configuration. The domain code, if used f.i. in a Software Product Line is now no longer related to the configuration of your products based on this code and can be versioned independently – enabling you to see what really changes in your application.

Fig.1 - Sample Application Architecture

This approach is shown in figure 1. You have two applications using the same library with different configurations. You can fix bugs or add features in the library without touching the configurations and vice versa. Interesting now is the difference between fragments and bundles. When should I use what? The advantage of a bundle is that it is the way OSGi intends to deploy code. You can use Activators and define your dependencies. The fragment in contrast was actually intended to only carry optional data like i18n information and alike. The advantage however is that the fragment and its host are sharing the same class loader(, which is also the biggest disadvantage and should be considered really carefully!). This distinction basically sets when to use what. When ever the configuration is loaded by the library and can’t be injected by another bundle, you need to have access to the bundles class path in order to provide the required resource. But wait a minute, you can’t use Activators in fragments. How could we now do the initialization? Here Spring DM comes in. You can just provide a Spring configuration in your fragment and as soon as the fragment is attached, Spring will recognize it and just initializes your beans defined in the descriptor and you have your Activator again. You even have access to the BundleContext if required. However, this approach is only intended for code that otherwise wouldn’t work or can’t be changed for some reasons. The best way to do it is for sure using bundles for that kinda thing.

The main point, I was trying to make here is that with the help of Spring and the up coming manifestation in OSGi R4.2 (if accepted), we have even more opportunities to build reusable and coherent artifacts, perfect suitable for software product lines. However, all the presented possibilities also impose potential for misuse and you should be careful by choosing your way. In many cases the simplest way on first sight eventually becomes a dead end, so be careful and farsighted when making your decisions.

Cheers
Mirko