I’ve seen a number of blog posts and tweets lately asking some version of the question Why is OSGi important? If you’re one of the many people looking around at the increasing usage of OSGi and wondering whether it matters to you, here’s my answer.
I’m going to start by making a pretty audacious claim, which is that OSGi is one of the most important technologies to have arisen in the last 20 years. This does not mean, however, that OSGi is a revolutionary technology. In fact, OSGi is so important because it represents the logical next step in the long-term evolution of software development.
Where we’re coming from
To understand what I mean, let’s go back 20 to 30 years to the time when object oriented languages first became popular. One of the main reasons we adopted OO at that time was because it allowed us to hide many of the implementation details of our code.
Moving from procedural languages to OO languages allowed us to develop classes exposing a contract defined by a set of public methods.
The result was that much of our code was invisible outside of its class, and this had profound implications for the way we develop software. By accepting the apparent restriction of visibility we gained immense freedom. We gained the freedom to reuse classes without knowing their implementation details. We gained the freedom to refactor our code without worrying about the consumers of a class.
Can you imagine what a pain it would be if you had to develop software without information hiding?
Where we’re headed
Now imagine what it would be like if you could hide not only the methods within a class but entire sets of classes within a JAR. Imagine that JARs could define public contracts the same way classes do, and that these contracts would be enforced both during development and at runtime. Imagine that we could achieve all of the benefits of information hiding (managing complexity, code reuse, testability, refactoring, etc.) at an entirely new level.
OSGi makes this possible by offering up the standard Java package as a new unit of information hiding. When our code is running inside of an OSGi framework, each package in a JAR can be either exposed or hidden from consumers of that JAR.
Just as a class has a small set of public methods representing its contract with consumers, a modularized JAR (a bundle in OSGi terms) has a small set of exported packages representing its public contract. The bulk of our code lives in internal packages hidden from other JARs.
Imagine being able to rename classes, split or combine classes, move classes from one package to another, move entire packages from one JAR to another, all without having to worry about impacting the consumers of a JAR. So many of these types of refactorings are skipped now out of fear. Package level information hiding gives us the confidence we need to perform these refactorings, allowing us to react with agility to the changing needs of our users.
Modularity is inevitable
Whether OSGi in particular succeeds or not, JAR level information hiding is inevitable. The benefits are simply too great to ignore, and in 5 or 10 years we’ll all be wondering how we could have possibly lived without it.
Currently, OSGi is the only tool we have to accomplish this. Luckily for us it’s a well thought-out, well tested, standards-based solution. I can’t think of one reason (besides perhaps its name) to develop an alternative to OSGi. It’s here. It works. Let’s use it.
It’s time for OSGi
Steve McConnell has a great quote that really gets at the heart of what OSGi is trying to achieve.
In Code Complete, he writes:
Software development has advanced in large part by increasing the granularity of the aggregations that we have to work with.
Because this granularity of aggregation is so critical, the move from unmodular to modular practices is as important as the move from procedural to object-oriented practices. For 20 years we’ve been limited to using the class as our unit of abstraction. As successful as that has been, it’s time to move on to modules. Its time for OSGi.
Hi. I must say I totally agree with you. It’s funny people adopted encapsulation on class level but don’t feel the need for the same on jar level.
Not being an expert on OSGi, I have ask what level of support is there for design/compile time enforcement at the moment? Let’s say how soul we achieve that in Eclipe?
Hi Tom,
During development, the Eclipse compiler is completely OSGi aware and provides appropriate warning and error messages based on what packages are exposed (or not) in a bundle. The visibility of packages is set in the JAR manifest file, and these settings are used instead of what you would normally set with the Java Build Path dialog.
It works extremely well. All you need is Eclipse and the Plug-in Development Environment to get started.
— Patrick
Hi, I’m the author of SO question where you linked this post, and I’ve read it.
My question is: can’t this kind of ‘JAR encapsulation’ be achieved by simply choosing which classes of our JAR are public and which are private / package-private, in other words, by defining our JAR public API?
I’ve been doing this for years without a problem, and (more important) without the need for OSGi.
Based on other comments I thought that OSGi was useful only in a heavy-enterprise-application where you need to swap modules and downtime is not an option.
Hi Pablo,
It’s true that you can achieve some level of information hiding by making classes package protected, but what you’re really doing is making the Java package your unit of abstraction. There are a number of problems with this:
* Packages are fairly fine-grained, closer to classes than JARs. To achieve the kind of encapsulation we’re talking about you would need to hide a *lot* of classes. Unless you put all of your code into a few very large packages, managing the relationships between classes inside the JAR would be difficult.
* Packages are not the unit of deployment. One of the benefits of modular JARs is that we are aligning our unit of abstraction with our unit of deployment. Because modules/bundles have public APIs, different implementations of those APIs can swapped in by deploying new JAR files. This becomes something like the relationship between interfaces and classes.
Again, I’m not arguing that you can’t achieve some level of encapsulation using package protected classes. But I would argue that it’s very difficult to build modular software in this way. One of the benefits of object-orientation was that it made it easier to things the right way than the wrong way. OSGi is the same, it almost forces you to make the right decisions. While this might be frustrating initially, I think its definitely worth it in the long run.
— Patrick
Patrick,
I should give OSGi a serious try in order to speak with a bit more of a background. Nevertheless I believe encapsulation/information hiding is highly overrated (and IMHO underused) this days.
DI needs to break encapsulation in order to work.
More dynamic languages tend to be more flexible in this topic (in groovy for example, you can access any field, even private ones).
Even in Java, average coders tend to create an accessor/mutator pair to every field, exposing the inner components of every class (this is often called anemic domain model).
I don’t see how DI and encapsulation are exclusive, as you make it sound like: isn’t the point with services that they are in fact configured elsewhere, meaning DI is “inherent”?
Pablo: I think anemic domain model is about more than just exposing every field in a class: a domain object could very well be non-anemic and still have every field mutable, I think. Anemic is more about the operations _other_ than the setters/getters: if there are higher abstractions, its probably not anemic. If it is just a class with no operations other than changing values, then it is anemic. Otoh, it could be anemic with no setters!
Excellent writeup. It exposed a feature of OSGi that I wasn’t aware of before. There has been a lot of talk about OSGi, but very little in the way of making people understand what it actually does. You did a great job. There’s another blog entry that I just read that explains another side of OSGi, namely using different version of a given library. It’s here: http://seewah.blogspot.com/2009/05/osgi-next-big-thing-beginners-view.html
Excellent article. Helped me get a sound idea on the purpose of OSGi. Thank you.
Nice explanation for the complex framework.
Also, thank you “Pablo” for asking my question.
I agree with your premise. Encapsulation is the best tool to control complexity at both the code and the data level. Its slow, but gradual integration into our technologies has allowed us to build increasingly complex systems.
My experience with OSGi however has not been particularly pleasant. My intuition is that it is excessively over-engineered and trying to solve too many problems all at the same time. While some of the goals may be great ideas, the actual implementation is far more important. If the problems outnumber the benefits, then it is time to reconsider the approach. Also, this type of paradigm should probably be based in the language were it can be handled properly, not some external add-on which can only be partially integrated.
Paul.
Hi Paul,
I do sympathize with the idea that OSGi is over-engineered, and as a trainer I’ve encountered many developers who feel overwhelmed by the technology. I also believe that it would be much better to have modularity enforced in the language, though I don’t think the current approach for Java 8 is the way to go.
It’s my opinion that most developers would get a lot out of OSGi if they focused on the Module layer and ignored the Lifecycle and Service layers, at least initially. It would also be great if there was extremely simply tooling that made it easy to work with the Module layer. Neil Bartlett’s BndTools is a good start, but it’s still way too complex.
Having said that, my experience with OSGi has been great. I work with it in the context of Eclipse RCP, and it’s completely changed the way I write code. OSGi also allows us to package and deploy code in ways we never could before. I can’t imagine going back.
Thanks for the thoughtful comment.
— Patrick
It’s nice to see the comment thread on this article has started up again, as it’s brought the following old comment by Pablo to my attention:
“DI needs to break encapsulation in order to work.”
Exactly! But this is not a problem with encapsulation, it is a problem with DI.
The DI framework sits at the centre of the system, grabs an object from here and an object from there and rams them together. To do this it reaches into the innards of each module, tearing out objects at the time it chooses, whether the modules are ready to provide an object of that type — and at that time — or not. The result is horribly fragile because if any individual module fails, then the whole DI framework fails.
OSGi’s solution is wonderfully elegant: just reverse the arrows. Instead of a central component that pulls from every module, have the modules push into a central registry. This allows the modules to remain completely encapsulated with NO externally visible packages. It also solves timing issues: the modules provide objects only when they are ready to do so. And it is robust: the failure of one module does not take down the whole system.
It’s really quite beautiful. And you don’t need to dispense with DI as a programming technique, in fact in OSGi we still usually employ DI framework inside our bundles. The key is not to allow the DI to reach across modules.
Thanks again for the article Patrick!
@Patrick: “It’s my opinion that most developers would get a lot out of OSGi if they focused on the Module layer and ignored the Lifecycle and Service layers, at least initially.”
Actually I disagree :-)
I think the first thing developers should learn about OSGi is Services. Of course, using DS to simplify the development of service components, rather than messing about with activators and stuff (so, I agree with you about Lifecycle).
Once they get Services, they can then move on to how to use the module layer to enhance the development cycle, hide implementation classes, evolve APIs etc.
Anyway thanks for the namecheck on bndtools!
Actually, I cannot agree with you. Working on a large scale OSGI project for half a year now has actually shown some benefits. Unfortunately it has also caused a lot of troubles coming from OSGI’s nature that are much more harder to overcome:
Surely, Encapsulation and Service Exports is a good thing – this is what OSGI is about and this is, what’s working fine.
How to set this one up, is the problem. If you’re really doing OSGI, you’ll need to create dozends of modules. Each one would (normally) come as a maven module, bringing it’s own overhead. Working on such a project effectively means making large jumps around various parts in your codebase. This leads to a lot of overhead when doing maintenance and/or debugging.
Next trouble comes with building OSGI modules. OSGI has some assumptions on java, the compiler does not know: Imagine, you have bundle X, containing com.x.foo.FooBarClass. Then you’d define com.x.foo being a private package. So during runtime, FooBarClass will not be available to other bundles. Now you’re working on Bundle Y, having a dependency to Bundle X. Somewhere internally, you end up, referencing FooBarClass. From any java compiler’s perspective this is fine – as long as X.jar is in the classpath. During runtime, however, you’ll either get a CassNotFoundException or (worse) a NoClassDefFoundError. Using OSGI reintroduces a lot of errors that one would normally know from dynamic programming languages.
Then there’s maven (and all other build tools doing dependency management): they are taking wrong assumptions over versionings. Having OSGI, it may be very fine, to use Bundle X in both Version 1 and 2. All these tools however will detect a version conflict and try to fix it, by only using version 2.
Another problematic area is the OSGI spec: It really IS a big bunch of paper. The concrete OSGI implementations of the other hand will only support a subpart of that spec. Lacking in documentation what features are there and what are missing.
In my opinion, the advantages of using OSGI come with a very high price. A price, that you’re probably not wanting to pay. Leveraging a DI framework (like spring), that allows multiple contexts and doing some service import/export managment might be a very good alternative choice. Using OSGI, you should really ask the question: “Why” first. and only if there are no alternatives to OSGI (and there are), use that technology.
[...] Why is OSGi important? :: Modular Mind [...]
>How to set this one up, is the problem. If you’re really doing OSGI, you’ll need
>to create dozends of modules. Each one would (normally) come as a maven
>module, bringing it’s own overhead. Working on such a project effectively
>means making large jumps around various parts in your codebase. This leads
>to a lot of overhead when doing maintenance and/or debugging.
Surely, no more overhead than if you were to implement the service infrastructure yourself?
>Next trouble comes with building OSGI modules. OSGI has some
>assumptions on java, the compiler does not know: Imagine, you have bundle X,
>containing com.x.foo.FooBarClass. Then you’d define com.x.foo being a
>private package. So during runtime, FooBarClass will not be available to other
>bundles. Now you’re working on Bundle Y, having a dependency to Bundle X.
>Somewhere internally, you end up, referencing FooBarClass. From any java
>compiler’s perspective this is fine – as long as X.jar is in the classpath. During
>runtime, however, you’ll either get a CassNotFoundException or (worse) a
>NoClassDefFoundError. Using OSGI reintroduces a lot of errors that one would
>normally know from dynamic programming languages.
I think this is the fault of the tooling – not OSGi. If you tool is dumb enough to polute the build-namespaces with the non-exported packages, this will happen. Unfortunately, most of the tools I use – have this problem (e.g. maven + eclipse duo). I am also looking for a solution to this right now.
>Then there’s maven (and all other build tools doing dependency management):
very fine, to use Bundle X in both Version 1 and 2. All these tools however will
>detect a version conflict and try to fix it, by only using version 2.
Again, I think this is the fault of the tooling. Maven is not smart enough to understand OSGi dependencies. In fact maven is very stupid about version in general – and to this day I can’t figure out how the hell it works.
>Another problematic area is the OSGI spec: It really IS a big bunch of paper.
Surely, R1 is not so big? And that should be sufficient to get you acquainted with the overall architecture of OSGi.
Also you don’t really need to know all the details unless you are actually writing an “implementation” of OSGi.
>The concrete OSGI implementations of the other hand will only support a
>subpart of that spec. Lacking in documentation what features are there and
>what are missing.
I think implementations are pretty clear as to WHAT they implement:
http://felix.apache.org/site/osgi-specification-implementations.html
http://www.eclipse.org/equinox/
>In my opinion, the advantages of using OSGI come with a very high price. A
>price, that you’re probably not wanting to pay. Leveraging a DI framework (like
>spring), that allows multiple contexts and doing some service import/export
>managment might be a very good alternative choice. Using OSGI, you should
>really ask the question: “Why” first. and only if there are no alternatives to
>OSGI (and there are), use that technology.
There are no alternatives to OSGi – maybe jboss (which is using osgi itself). Also DI – is just a small part of what osgi can do. In my opinion the most important aspect is the “modularization” and dependency enforcement.
The notion that my application will not run – unless all the pieces are there – makes me sleep much better, knowing that I never have to debug things like missing or incompatible bytecode.
Only in osgi can I go and download a “bundle” from the internet, and drop it in my container – and not have to do any kind of “install” or “compile” steps. It is truly a “framework platform” of the future! WOHOOO!
Now if only we could have Scala as first rate language for OSGi – it would be much more popular.
I disagree with your argument of lacking tool support. Especially with those hidden packages, it’s OSGI versus the compiler. Having a framework or a platform I would actually like it to honor the programming language, it’s built on. Right now, OSGI is taking assumptions, that the java compiler does not (private/public packages). Therefore, there is no way to tell, if the bundle your’re working on, is going to run inside an OSGI container. This is due two reasons:
A) it depends on private packages, which is something the java compiler does not know about.
B) a dependency is missing.
Both of them cause runtime issues. Personally, if I had a higher interest in runtime issues, I would have switched to groovy/grails long time ago. They, btw. also solve this “private package” model.
Another open point imho is deployment. OSGI is lacking some sort of container format (something similiar to a war): a single file containing the entire application, that gets deployed onto the server. Using OSGI in production for a large scale, I can tell you, that anything else is very painful and error prone. Updating several servers at the same time, it’s more handy when there is only one file in a concrete version that must be deployed.
Having modularization on the one hand is nice, but on the other hand, it’s trying to enforce something via a platform that should actually be part of a developer’s common sense. Anyone too lazy for modularization could still (with OSGI) declare all modules public and use dynamic imports. What this convention actually does is, making it harder to be “a nice developer” because the good developer needs to go with all the extra effort to declare his public/private packages with their version and so on. While the “mean developer” will just export all packages with version 0 and use dynamic import everywhere.
For downloading bundles from the internet: You could just download and use them, if they had no dependencies to other bundles. Most of the time, I found myself, patching manifest files of very common libraries (not all of the apache-common libs have an OSGI manifest) so that I could just use them. If these libs do classloading magic themselves (and a lot of them do), poor you.
I think, OSGI is not ready for three reasons:
It’s lacking tool support (and a compiler that understands the manifests)
Many 3rd party libs don’t come OSGI friendly
OSGI is a very restrictive spec, where you need to declare a lot of stuff. This causes way to much overhead. In many ways, it complicates too much.
@matt Thanks for your thoughtful post. As a trainer I encounter many people with the same frustrations and I sympathize with them. In particular, I agree with the deployment issue. Dave Orme actually has some interesting thoughts on this:
http://www.coconut-palm-software.com/the_new_visual_editor/doku.php
In the end, I guess where we part ways is that I think OSGi can be made to work in the long run.
For instance I think that the tooling can and is being improved. Compilers can be improved, and actually the Eclipse compiler is very OSGi-aware and provides great feedback.
You argue that modularity is something that can be achieved using common sense and without a framework. This is undoubtedly true, but only by a very small set of developers. It’s also true that OSGi can be abused to create really ugly code, but you could make this argument about a lot of technologies.
I keep coming back to the parallel transition from procedural to object-oriented technologies. Many of these same arguments took place then. We don’t need OO. OO can be abused and made to look procedural. OO puts too much of a burden on developers and limits their freedom.
Eventually OO became mainstream, and I think that modularity will as well. These shifts are incremental changes that increase “the granularity of our aggreations”, as Steve McConnell puts it. Whether this modularity will be in the form of OSGi or some other technology (or built into the JVM itself) remains to be seen. But in the end, modularity is too useful to be ignored.
Probably the best explanation of OSGi, though hearing its name from long time, I never really understand what it does, now JAR level encapsulation and modularity is something I can understand. I agree by its name it looks complex.