The myth of software reuse

Are we fooling ourselves? Or is it real?

The ones of you who know me, know that I believe in software reuse and I am trying to evangelize about it for quiet some time. Methods, Classes, Aspects, Components, Modules, Software Product Lines, you name it – everything is focused on reuse to some extend. With OSGi for the first time we are able to create true reusable software artifacts and theoretically mix and match them on an as needed basis. Great new opportunities are coming and with the recently held OSGi Tool Summit chances are good that we’ll soon see more and better integrated tooling support in our daily work with OSGi. So, we all should be happy bees right?

Well, what we are producing right now is a massive amount of bundles, no question about this. Virtually everyone is now shipping with at least some basic OSGi headers. Several projects are trying to provide bundle repositories for the not (yet) converted projects (like [spring-repo], [orbit-repo], [osgi-repo]) and some go even further and try to wrap incompatible APIs in a compatibility layer (see [DynamicJava] for instance). Great work and very much appreciated, but this is not the point. What we got so far are “just” explicit software artifacts. We only know what is behind bundle X in version Y, a subjectively taken SNAPSHOT somewhere in the infinity of time. It doesn’t answer the question on how this code can and will evolve or how future versions will fit into my universe.

The problem domain

Too abstract? Let’s examine an arbitrary example. Assuming you are trying to create a web application similar to flickr but on some sort of an embedded device to use at home. Taking the state of the art tools, you might choose to develop your app with Spring DM or any other frameworks you like, take your pick. For now, we stick to this one. As a starter we use the following assets: Spring DM 1.1.3, Spring 2.5.6 and Maven 2 as a built system. So far so good. Once you start developing you realize that you also need a file upload feature for your pictures. Of course someone already developed something to upload files and we are trying to reuse that stuff of course, so you look around. Soon, you’ll find commons-fileupload. The spring guys are putting a lot of work into building up a repository of reusable osgi bundles in their bundle repository and we can just use the OSGi-ified version of it from here: [spring comFileUpload], which is really great! But then you soon realize that you can’t use that version. Take a look at it’s (simplified) header:

Manifest-Version: 1.0
Bundle-ManifestVersion: 2
Bundle-Name: Apache Commons File Upload
Bundle-Version: 1.2.0
Bundle-SymbolicName: com.springsource.org.apache.commons.fileupload
Bundle-Vendor: SpringSource
Export-Package: org.apache.commons.fileupload;version="1.2.0";
  uses:="javax.servlet.http",
 org.apache.commons.fileupload.disk;version="1.2.0";
  uses:="org.apache.commons.fileupload",
 org.apache.commons.fileupload.portlet;version="1.2.0";
  uses:="javax.portlet,org.apache.commons.fileupload",
 org.apache.commons.fileupload.servlet;version="1.2.0";
  uses:="javax.servlet,javax.servlet.http,org.apache.commons.fileupload",
 org.apache.commons.fileupload.util;version="1.2.0"
Import-Package: javax.servlet;version="[2.5.0, 3.0.0)",
 javax.servlet.http;version="[2.5.0, 3.0.0)",
 javax.portlet;version="[1.0.0, 2.0.0)";resolution:=optional,
 org.apache.commons.io;version="[1.4.0, 2.0.0)",
 org.apache.commons.io.output;version="[1.4.0, 2.0.0)"

As you can see, it requires to have the Servlet API in version 2.5, which is not available in our case on the embedded device. What now? Well, if you know the bundle, you also know, that in fact it doesn’t require 2.5! You can re-bundle it and give the bundle another version and here is the problem. The Spring guys did a great work bundling this artifact for us, but they created it based on THEIR environment and not on the minimal requirements of that specific bundle. Even worse, there is not even a real specification stating what version to take and how to version changes correctly. Forward compatibility is just a beautiful dream rather than an actual fact. If you look at well known companies adapting OSGi like LinkedIn, you’ll see what I mean. Where are we heading to when one is forced to define dependencies like this:

Import-Package:
 com.linkedin.colorado.helloworld.api;version=”[1.0.0,1.0.1)”,
 com.linkedin.colorado.helloworld.client;version=”[1.0.0,1.0.1)”,
 org.osgi.framework;version=”[1.4.0,1.4.1)”

Doesn’t this feel totally odd and just plain wrong? Shouldn’t we be able to trust the authors or “something else” to take care of compatibility issues? Someone who knows the code, its changes and its backwards compatibility better than we do? I don’t want to change all of my 200 bundles just because we have a new OSGi framework version that now provides a new final static property for instance, I am not using anyway! I want to express that I don’t care what changed in the provided bundle as long as my code still works as expected and I want to be sure that the provider has the exact same concept about “compatible code changes”.  The Eclipse community already realized this and created an API for baselining components and a version schema to apply versions. Along with several detailed instructions on what to consider a change and how this change should be treated in terms of its version (eclipse-vers1, eclipse-vers2, eclipse-vers3), they have created the first usable guide on defining version numbers for bundles. However, this is just a small step towards reusable artifacts. We also need to agree on such rules and be able to enforce them. Unfortunately a lot of questions remain unanswered yet. For instance, what about semantical changes? How to treat them and more importantly, how to identify those? Code analysis won’t always help here.

So what do we learn from this? Basically that we can’t trust any bundle provided by a third party just yet. It doesn’t matter how good the providers intentions are on providing the perfect bundle. Right now, we are basically left alone. Every time we update to a new version it is like a game. You never know what you’ll be facing.

Ok, we are screwed. Who to blame?

Now you might ask yourself who’s fault is it? It is certainly not the Spring guys fault who provide us in this example with the software, but who to blame instead? The tools? Most people either use BND, Eclipse or the newly created bundlor from Spring. All of these are pretty dumb. They can’t possibly know, which version to take (although they are trying hard to guess). There is no baseline, no knowledge about the domain or infrastructure). Too many questions are unanswered and the tool authors are left alone, so I think the tools are the last ones to blame. OK, so what about the OSGi specification? It is so vaguely written, when it comes to versioning your bundles – you can’t possibly draw any universal conclusion what version to apply. Everyone can have their own interpretation of “compatible” changes, which is not compatible among each other. All true, but I don’t think that a simple specification will be enough nor is the OSGi suitable for that. The issue is too big to be solved by only one company or organization all by themselves. Sun might be the only one fitting, but after all the problems with JSR 277 and project JigSaw, I have no convidence in their ability and willingness anymore. To be fair, one have to admit that the Java Language Specification does provide a chapter about binary compatibility, but it is not much of a help, because not all cases are covered and there is of course no notion of a bundle (I would love to rant about not treating packages as 1st class citizen in Java, but that’s an entirely different post). Sun also has a sigtest tool to check for API breaks, but with the given functionality it is pretty much worthless for what we need.

What next?

Is it the job of an external organization to define rules everyone has to apply while developing reusable artifacts for a specific language? I don’t think so. I think this should be the job of all of us. Maybe as a joint project, maybe umbrella’d by the JCP, I don’t no, but definitely as a open and community driven effort. I don’t wonna lock myself to any vendor or proprietary standard I might get stuck with. I dream of a central repository (maybe based on RFP 122, maybe something completely different), where I have a one stop shop for all the 3rd party artifacts I need. At the same time being able to do in-house development with the same reliable system not having to expose anything to the outside world. Open, reliable and trustworthy software with a healthy community of open source artifacts – does it have to be a dream or can we make it real? I already have ideas how this can come true, but I would be very much interested if you’re feeling the same or even having concrete projects concerning something similar I haven’t reference here? Is there a potential for collaboration? What do you think?

My 2 cents,
Mirko

References (in chronological order):

[OSGi]: http://www.osgi.org/
[OSGi Tool Summit]: http://www.osgi.org/Event/20090327/
[spring-repo]: http://www.springsource.com/repository/app/
[orbit-repo]: http://www.eclipse.org/orbit/
[osgi-repo]: http://www.osgi.org/Repository/HomePage/
[DynamicJava]: http://www.dynamicjava.org/projects/jsr-api/
[Flickr]: http://www.flickr.com/
[Spring DM 1.1.3]: http://www.springsource.org/osgi/
[Spring 2.5.6]: http://www.springsource.org/download/
[Maven 2]: http://maven.apache.org/download.html
[commons-fileupload]: http://commons.apache.org/fileupload/
[spring comFileUpload]: http://www.springsource.com/repository/app/bundle/version/detail?name=com.springsource.org.apache.commons.fileupload&version=1.2.0
[Servlet API]: http://java.sun.com/products/servlet/
[LinkedIn]: http://blog.linkedin.com/2009/02/17/osgi-at-linkedin-bundle-repositories/
[baselining]: http://www.ibm.com/developerworks/opensource/library/os-eclipse-api-tools/index.html
[version schema]: http://wiki.eclipse.org/Version_Numbering
[eclipse-vers1]: http://wiki.eclipse.org/Evolving_Java-based_APIs
[eclipse-vers2]: http://wiki.eclipse.org/Evolving_Java-based_APIs_2
[eclipse-vers3]: http://wiki.eclipse.org/Evolving_Java-based_APIs_3
[BND]: http://www.aqute.biz/Code/Bnd/
[eclipse]: http://www.eclipse.org/
[bundlor]: http://www.springsource.org/bundlor/
[JSR 277]: http://jcp.org/en/jsr/detail?id=277
[jigsaw]: http://osgi.mjahn.net/2008/12/04/componentization-wars-part-ii-guerrilla-tactics/
[Java Language Specification]: http://java.sun.com/docs/books/jls/third_edition/html/j3TOC.html
[binary compatibility]: http://java.sun.com/docs/books/jls/third_edition/html/binaryComp.html
[sigtest tool]: https://sigtest.dev.java.net/
[RFP 122]: http://www.tensegrity.hellblazer.com/2009/03/osgi-rfp-122—the-osgi-bundle-repository.html

Share

10 thoughts on “The myth of software reuse

  1. Great article reflecting a lot of my thoughts about the subject. However, I do not think we should specify anything before we fully understand the issues. We need lots of people experimenting and finding solutions.

    We are working on tools in the OSGi and this is clearly one of the most important issues to address. As an organization, we’d love to foster the discussions around versioning and how to reduce brittleness and the maintenance nightmare.

    However, I do think your title is overly negative, Myths are stories that are not true. No sane person is promising reuse as we both seem to dream about today, but it is the primary reason for me to do OSGi. Therefore, I think the title of your post should have been better as “The Quest for Software Reuse” :-)

    Thanks! Kind regards,

    Peter Kriens

    P.S. The reasons why we are rather vague in the spec about the versioning scheme is that we did not want to restrict the required experimentation. If you do not understand something, I feel it is really bad to standardize it.

    P.S.S. You know about the bnd -versionpolicy?

  2. To amplify on an observation that I actually picked up from Peter, we’re in that awkward stage between the reality of monolithic tools and libraries (jar-based) and the dream of a fully componetized (bundle-based) future. We desperately want components. But what we have to work with is jars artificially repackaged as bundles by various groups with ever-evolving understanding of what a truly competized world might look like.

    See http://bradjcox.blogspot.com for musings of how the construction industry bridged this gap in transitioning from monolithic (cave-man) to primitively componetized (mud brick) to modern (real brick) architecture. And how long that all took! ;)

  3. @Peter:
    Thanks for the feedback! You might be right, that the headline is drawing too much negative attention, but I wanted to make a point. I see people violating even fundamental versioning rules for the sake of… I don’t know – dubious business reasons I guess, and at the same time praising their so greatly modular applications. Just because you use OSGi and a minimum of versioning you’re not automatically creating reusable artifacts. However for some reason this attitude is in people’s heads, not knowing what great responsibility comes with releasing software to the public. I think this is the important message: “Just because you made it a bundle it is not automatically a reusable artifact. It is hard work to make it so.”

    I am trying to foster this discussion for some time internally at work and it is conceptually well received so far and awareness is spreading. I think, we are at a critical point in the adoption phase of OSGi. People are now heavily investing in OSGi with the hope to (later) benefit from a great variety of reusable artifacts. When not adhering important, yet to define rules and providing tools helping you to version bundles, we might be facing a similar situation Microsoft witnessed with the DLL hell – having hundreds of bundles, all versioned inadequately (if at all), making it impossible to build a product on top of it without going through a serious evaluation and repackaging process. Something I certainly don’t wonna see… Learning from the past to conquer the future ;-)

    Your point that we need the experience and voice of many – I can’t agree more. I think this is the only way to come up with a ubiquitous scheme to versioning in the context of software reuse. The question remains what is the right process and environment for propelling these efforts so that lobbyism can be minimized, the “not developed here” attitudes doesn’t kill the process before it even started and the focus is on the important stuff – the technical (and partially social) problems?

    @Brad:
    Nice analogy. I heard similar ones when people where promoting software product lines. Unfortunately awareness doesn’t automatically translate to progress, we have to work on that by ourselves – but true, it certainly helps knowing ;-)

    Thanks,
    Mirko

  4. While I agree with what you are saying, I think you are equating modularity with reusability. As is clear from the discussion, these two do not go hand in hand. There are still plenty of valuable reasons to create modular systems, even if the modules themselves are not reusable. So, it is still okay to praise our modular systems. However, I agree that making your modules reusable is the better way, although not necessarily the easier way to go. :-)

  5. Get your head around pure lazy functional programming. Then we’ll have a talk about whether reuse is a myth. I’m confident you’ll change your mind. As a starter, ask yourself what property about a language subset that you consider “reusable”, since you’ll agree that it at least exists.

  6. @Tom
    Thanks for the feedback. I updated the link…

    @Richard
    You’re right! I treated modularity and reusability synonymously, which is not true. I guess after working with OSGi for so long, I consider modularity as a given fact and don’t even think there is any not modular code out there any more – which is simply wrong. Thanks for clarifying and reminding me that we are still fighting the basic issues of our craft to a great extend. Maybe I am dreaming too much of a better world :-)

  7. This article is not new but hasn’t lost its relevance. I’m not new in OSGi, because of being ever on Eclipse IDE and sometimes doing Eclipse RCP development, but looking closer on OSGi in a last time.

    You describe here not only OSGi relevant topic. Versioning is big problem in a maven world. So my 5 cents are:
    The are two problems, first is to define what is the proper version strategy and who defines it and maintenance this rules.
    Second who and how controls the preservation of the rules.

    First is maybe simple. OSGi Alliance is encourage do define some kind of spec on this. But the control has to be done carefully. Thereby i think about a popular (has to be bright accepted ) OSGi Bundle repository (like maven has) which enables some contol function.

    Some simple control can be done e.g. during publishing and lead to refusing of deployment to Repo. Deeper and even semantic control could be done by human comunity. And there is a lot of opportunities how this could be relised, maybe it has to be not specified to detailed.
    What do you think?

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>