mangen - OSGi Bundle Manifest generator

Copyright © 2000-2005 Ascert, LLC.


Introduction

The OSGi model provides a very powerful and flexible framework for developing Java applications in a modular fashion with a high degree of control over classloading inter-dependencies between modules. Projects such as Oscar have recognised this power and implemented open source implementations of the framework. Adding to this is a growing availability of open source application bundles such as those offered via the Oscar Bundle Repository (aka OBR).

As an active developer of OSGi based application, Ascert's technical staff have grappled with the cumbersome task of manually creating OSGi Bundle Manifests. This task is never particularly interesting and only gets bigger and nastier as applications grow in size and the number of bundles present. To alleviate this chore the mangen tool has been created, with the following goals:

If you're not a Java developer, or you're not sure what OSGi is yet, then this tool is probably not for you!

Licensing

The mangen utility is a copyrighted © work of Ascert LLC, released as OSI Certified Open Source Software for use under the under the terms of the Common Public License (text)

Included third-party libraries and contributions are copyrighted © works of their respective owners and are provided under the terms of their specified licenses.

Support

The toolkit is provided on an "as-is" basis with no warranties of any kind and no support included. Ascert may release newer versions as enhancements and fixes are found, and is always glad to receive comments, feedback, fixes and suggested improvements. No commitment is given, however, as to when or whether specific requests may be reviewed and/or implemented.

If your organisation is interested in a commercial support agreement or product support contract please contact Ascert either by email or via the Ascert website.


Using mangen

Basic operation

Using mangen is very simple. Unpack the distribution file to an installation directory, change to this directory and type the following command:

    java -jar lib/mangen.jar <file-list>

Note: on Windows platforms you will need to use a '\' in place of the '/' file separator

Where <file-list> is either a list of bundle JARs to be processed or a list of directories, or a mix of both. Each directory will be scanned and all JAR files in the directory, or any sub-directories will be included in the bundle JARs processed.

The basic behaviour of mangen is to process the set of bundle JARs supplied and scan the classes within each JAR to build up a set of the possible exports and required imports for each JAR. Where inner jars are defined using the OSGi Bundle-ClassPath header the classes in these will also be scanned and the possible exports and required imports will be included in the sets generated for their enclosing JAR.

Two phases of process happen after JAR scanning has completed:

Use of the word 'typically' is significant in the above descriptions. A basic set of rules and reports are included with mangen to perform various useful tasks. This set is infinitely extensible, however, and mangen places no restriction on the types of tasks that rules and reports can perform.

The exact rules and reports to be run are entirely controlled by user configurable properties as described below. If no properties are supplied mangen will simply run, process the specified JAR files and exit.

Configuring mangen with Properties

Properties to control mangen behaviour can either be supplied on the command line using the normal -D syntax or, for most cases in a mangen.properties file. If the same property name is specified in both then the -D command line property value will take precedence of the property file value.

For added flexibility a simple variable expansion syntax is provided to allow the value of one Property value to be used within another property. Every property value will be scanned for ${property-name} markers. Where these are found the marker will be replaced with the value of the associated property-name.

Example:

    ...
    mangen.osgi.level=3
    ...
    mangen.rulesets=mangen-rule-R${mangen.osgi.level}- 
    ...

mangen.properties

The mangen.properties Property is used to instruct mangen where to find it's properties file. By default, mangen will look for a file named mangen.properties in the same directory in which mangen.jar was installed (i.e. lib/mangen.properties). By including a -Dmangen.properties setting on the command line a different properties file can be specified.

Example:

    java -jar lib/mangen.jar -Dmangen.properties=my_mangen.properties my_app.jar

mangen.osgi.level

There are two places in which the mangen.osgi.level Property is used:

At present, this Property should be set to a value of '3' or '4' (the default if not specified will be '3');

mangen.rulesets

Rules are the engine room of mangen, providing the basic means for refining the mangen detected import and export package sets e.g. removing un-needed or unused exports, supplying package version information, including undetectable package cases such as dynamic classloading.

Rulesets provide a simple means of organising the rules to be executed into groups of rule sets. The rulesets are specified as a list of comma-separated values, each value specifying the ruleset name prefix. The following example shows a ruleset definition for 2 rules:

    ...
    mangen.rulesets=mangen-rule-first- , mangen-rule-final-
    ...
    mangen-rule-first-0=...
    mangen-rule-first-1=...
    mangen-rule-first-2=...
    ...
    mangen-rule-final-0=...
    mangen-rule-final-1=...
    

As shown in the example, mangen will take each ruleset name and look for sequentially numbered properties, starting from 0 and finishing when no property name is found. Each rule found will be executed to completion against the processed set of bundle JARs before the next rule property is processed.

Rulesets can be combined with variable-expansion to provide OSGi version dependent rules as shown the following example.

    mangen.osgi.level=3
    mangen.rulesets=mangen-rule-R${mangen.osgi.level}-
    ...
    mangen-rule-R3-0=...
    ...
    mangen-rule-R4-0=...
    ...

Rules themselves are simply specified as a rule type followed by a space separate list of rule specific options e.g.

    mangen.R4.syspackages=java\\..*
    ...
    mangen-rule-basic-0=Ignore imports(${mangen.R4.syspackages})
    mangen-rule-basic-1=DontImportOwnExports

See the Rules section for full details of the currently support rule types.

mangen-report-

Reports in mangen work in a similar fashion to rules but without the ruleset concept. The set of sequentially numbered mangen-report- properties will be scanned to determine which reports should be run e.g.

    mangen-report-0=RuleReport .*
    mangen-report-1=BundleReport .*

See the Reports section for full details of the currently support report types.

mangen.failonerror

If set on will cause mangen to exit with a System.exit() error status of 3 if any errors occured. Typical usage is to allow an external build tool, such as Ant, detect that there were errors. Additionally, any error messages will also be sent to stderr as well as stdout if this property is set.

Default is on.

mangen.failonwarning

If set on will cause mangen to exit with a System.exit() error status of 5 if any warnings occured. Typical usage is to allow an external build tool, such as Ant, detect that there were warnings. Additionally, any warning messages will also be sent to stderr as well as stdout if this property is set.

Default is off.

Rules

The Rule concept in mangen was adopted to avoid hard-coding the types of post-processing steps that a user would be able to perform on the mangen generated set of package imports and exports. The rule syntax is as follows:

    <rule-type> <rule-options>

<rule-type> must be the name of a valid existing rule type, details of which can be found in this section.

<rule-options> will be list of one or more of the standard options and/or rule specific options. The standard options are as follows:

Rules will can have either "global" scope, in which case every bundle JAR processed will have the rule appplied, or "local" scope meaning that they will only apply to a single bundle JAR. Global rules will be included in the mangen.properties file. Local rules are placed within the Manifest for the appropriate bundle in a special mangen attributes section e.g.

    Bundle-Name: Help Component
    Bundle-ClassPath: .,help4.jar,oracle_ice.jar,ohj-jewt.jar
    Metadata-Location: metadata.xml

    Name: com/ascert/openosgi/mangen
    mangen-rule-0: Ignore imports(com\.adobe\.acrobat.*,webeq\..*,javax\.help,javax\.media)

Note: when creating regex patterns a single slash (\) is needed to escape literal characters when used in a Manifest, but a double slash (\\) is needed in properties entries. This is because the JDK property scanning strips off one of the pair of double slashes in it's string handling.

Details are included below showing whether a <rule-type> can be used in a global or a local context

AttributeStamp

Usable globally yes
Usable locally yes
Standard options imports, exports
Rule specific options  

When processing a bundle JAR mangen can only detect the name of a required import package or a possible export package. Within an OSGi environment it's possible to also include qualifying information on a package name, such as versioning information. The AttributeStamp rule allows this information to be "stamped" over a detected package name.

The rule may be supplied locally, in which case it will only apply to instances of a package name match with a specific bundle JAR, or globally in which case it will be applied to all instances of a package name match across all JARs.

The imports or exports options allow stamping of attributes to either imported or exported packages respectively. The rule will perform a regex package name match against each entry in the list and if the name matches, will augment the matched package name with any additional attributes suppled. The following shows an example of this.

Example:

    mangen-rule-1=AttributeStamp imports(org\\.osgi\\.framework;version="1.2.0") 

If the rule finds a package name pattern match and the package already has additional attributes an error will be thrown if the stamped attributes do not match the existing attributes. This could be the case as a result of either a previous AttributeStamp or Merge rule.

DontImportOwnExports

Usable globally yes
Usable locally yes
Standard options  
Rule specific options  

In many application cases it's not necessary for a bundle JAR to import it' own exports. This rule may be used locally or globally to remove from a bundle's import list any package which it also exports.

Ignore

Usable globally yes
Usable locally yes
Standard options imports, exports
Rule specific options  

There are several cases where a mangen detected possible export or required import may not actually be desired:

The Ignore rule will remove matching package entries from either the import or export lists, or both, as specified in the options.

Example:

    mangen.R4.syspackages=java\\..*
    mangen-rule-R4-0=Ignore imports(${mangen.R4.syspackages})

Merge

Usable globally yes
Usable locally yes
Standard options imports, exports
Rule specific options existing, fixed

In some cases the simplest way to use mangen will be to provide a list of known imports and exports and then have mangen "merge" any remaining required imports and possible exports into these lists as needed. The Merge rule provides two mechanisms in which these known imports and exports can be supplied:

Example:

    Manifest-Version: 1.0
    Bundle-Name: mybundle
    Export-Package: my.bundle.package

    Name: com/ascert/openosgi/mangen
    Import-Package: some.other.package

A Merge existing using the above example would ensure that my.bundle.package appeared in the list of packages to export. A Merge fixed would ensure that some.other.package appeared in the list of packages to import.

It's possible to use both Merge existing and Merge fixed within a give set of application rules although it's more likely that only one of these would be used to meet a given application build strategy.

The imports and exports options allow constraints on the packages to be merged based on regex package name pattern matches.

One other aspect to note with the Merge option is that it also provides an alternative way to "stamp" OSGi attributes on a mangen detected pakcage name, since if the package being merged was already in the set of mangen detected packages it's entry will be augmented with any additional attributes supplied from the package entry being merged.

ProcessBundles

Usable globally yes
Usable locally no
Standard options  
Rule specific options  

By default, mangen will not actually process any of the JAR files specified, it will simply create objects to access them.

Being able to skip mangen processing of bundle JARs is useful behaviour in a small number of instances, such as the ObrReport that will generally be run against existing bundle Manifest headers rather than mangen generated sets of imports and exports.

For most cases, however, mangen import and export processing will be required and this Rule should be included.

Example:

    mangen.rulesets=mangen-rule-initial- , mangen-rule-Ant- , mangen-rule-R${mangen.osgi.level}- , mangen-rule- , mangen-rule-final-

    mangen-rule-initial-0=ProcessBundles
    ...

ResolveImportsToExports

Usable globally yes
Usable locally no
Standard options sys-packages
Rule specific options  

Some OSGi developers use the framework as a basis for creating packaged applications, in fact it is just this usage which Ascert make of OSGi and Oscar and which motivated the creation of mangen. In such cases, the simplest and possibly most powerful rule use case is  simply to supply =mangen with a complete set of application bundles and let it work out the matrix of imports and exports required to resolve every bundle dependency. This is exactly what the ResolveImportsToExports does.

ResolveImportsToExports can only be used globally and will prune down the set of possible exports and required imports to just those required to satisfy every bundle dependency. It will generate * WARNING * report lines for the following cases:

At present, the known cases where this rule may fail to create a consistent and resolved set of bundle Manifests are:

UpdateBundles

Usable globally yes
Usable locally no
Standard options  
Rule specific options overwrite

By default, mangen will only report on the generated list of imports and exports for each bundle processed. The UpdateBundles rule can be used to instruct mangen to update each bundle's Manifest wth the set of generated packages.

This rule can only be used globaly. If the overwrite option is specified, the bundle JAR will overwritten with a new bundle JAR containing the new Manifest. Without this option, the update will create new JARs of the same name as each existing JAR but with a suffix of .new.jar.

Reports

Reports are really like a simplified case of rules. At present only a couple of simple reports are included.

All reports at present send their output to System.out, which can of course be redirected to a text file if a persistent copy is desired.

RuleReport

This report will show any Rule generated output.

BundleReport

Report options show-differences show-local-rules

This report will create a simple overview of the refined set of a bundle's imports and exports, together with a report of any local rules which have been run for the bundle. The following options are supported:

ObrReport

Report options skip-jars

Produce a report for each bundle JAR that can be used as an OBR descriptor.

The skip-jars option can be used to specify a comma separated list of JAR name regex patterns for which OBR descriptors are not required (e.g. source JARs).

OBR descriptor production is a quite different aspect of mangen usage to import/export generation and so a separate example obr.properties file has been included to show typical settings for it's usage. The -Dmangen.properties setting can be used to run mangen with these settings e.g.

Example:

    java -Dmangen.properties=lib\obr.properties -jar lib\mangen.jar e:\obr\repo\

The example obr.properties includes a number of features:

Whilst running, the ObrReport will look for a number of specific properties to aid it's processing:

The templates include a simple "tag substitution" mechanism that will expand the following tags:

Contents of the distribution file

The current mangen distribution includes the following:

The following third party libraries are also included in the distribution:

Thanks also go to the following contributors:


Extending mangen

First things first. You need to be a reasonably proficient Java developer to undertake extending mangen. If you're not, then you should consider a Java programming course or tutorial of some kind.

Extensions to mangen can be performed in the following ways:

The idea is that as mangen matures most extension cases will be possible via the first two means, with new class scanners and core modifications being the exception.

For detailed information, Javadoc API documentation for mangen can be found here.

Creating new Rule types

A rule type is in fact just a Java class which implements the com.ascert.openosgi.Rule interface. If no package name is specified, these will be assumed to be in the com.ascert.openosgi.mangen.rules package. Although somewhat less readable, a fully-qualified class name can be supplied for rule types in other packages.

At present, the simplest way to learn about creating new rules is to look at the source code for existing rules to understand how they're put together and what can be done in them.

Creating new Report types

Reports are similar to rules. A Report type is a Java class which implements the com.ascert.openosgi.Report interface. Unqualified report types will be assumed to be in com.ascert.openosgi.mangen.reports package, with the option to use fully-qualified class names if desired.

As with rules, the source code for the existing reports is the best place to learn about creating new reports.

Alternative class scanners

To parse the class files of an application mangen needs a class file bytecode scanning library. So that alternative scanning tools may be used mangen does not make direct usage of any library implementation. Instead a wrapper class is used which implements the ClassScanner interface, and hence insulates mangen from the specific details of different bytecode scanning tools. The mangen.scanner.class property can be used to control which scanner implementation class is used.

At present, implementations of the !ClassScanner interface have been include for the ObjectWeb ASM toolkit and the Apache BCEL toolkit.


Ongoing Development

Change Log

Version 0.1.2

Version 0.1.1

Version 0.1.0

Possible Enhancements

As with any piece of software, there are always more things you'd like to do than time available in which to work on them. This library is no exception.

In it's present form it mangen is simple, reasonably fast, and usable. Ideas on some of the more significant areas where it could be enhanced or improved are described in the sections below.

If you look at the source code, don't be surprised if you also find a few //TODO markers. These don't necessarily signify areas which don't work, just thoughts on how things could be done differently or perhaps better as the code was being written.

Rules to allow better handling of duplicate exports

At present, duplicate exports are handled in the ResolveImportsToExports rule and are simply flagged with a *** WARNING *** message. The first occurence of a duplicate export will be picked for inclusion and all others will be removed. This may not be the desired behaviour. This can partially be solved with the current AttributeStamp rule to ensure that if several exports represent different package versions they will not be flagged as duplicates. However, where there are multiple possible exporters of the same package version additional rule handling would be beneficial to allow a given bundle to be specified as the exporter.

Auto-handling of inner JARs

At present, mangen will only process inner JARs that are specified in a valid Bundle-ClassPath attribute in the existing JAR Manifest. It would be perfectly feasible to have an option to tell mangen to scan any inner JAR it finds, and also to automatically generate a valid Bundle-ClassPath for each inner JAR when updating the bundle's JAR.

Include a ManifestReport

Make one of the reports (possibly the default) a report which basically shows what the Manifest would look like for each processed JAR. Also for the UpdateBundles rule change the default name to be bundle.new.jar.

Manifest-less usage

With auto-handling of inner JARs and the existing functionality it would be possible to make mangen generate the essential import, export, and classpath headers of an OSGi manifest completely automatically. Although an existing manifest can provide additional descriptive headers, these are by no means essential for basic OSGi operation.

Online usage within an OSGi environment

Extending the concept of Manifest-less usage comes an interesting possibility that a specific OSGi platform such as Oscar could be extended to load any JAR and automatically 'fix-up' a usable Manifest. This would require internal access/knowledge of the specific platform's implementation since the existing standard OSGi API would not supply sufficient details and access to the set of loaded bundle JARs. Additionally, it would probably need to be a "multi-step" process since until a largely complete set of bundle JARs were loaded it may not be possible to resolve all imports and exports. This perhaps implies some form of platform extension to allow a set of JARs to be passed to some form of "pre-load" mechanism capable of resolving their imports and exports within the JAR set, and possibly from existing loaded bundle JARs or even an external OBR.

No current support for Dynamic-ImportPackage

Current processing and rules do not parse or use any Dynamic-ImportPackage attributes present. At the least, it would probably be useful to have the ability to not include anything in Import-Package which matches a dynamic import attribute. Beyond this, there may be benefit in additional processing to make fuller use of dynamic imports where the user requires it.

Read-only JARs for package resolution

At present all bundle JARs are considered "updateable", there is no way to indicate that certain JARs should be read but not modified. Where this may be useful is to provide certain JARs purely for resolution of package imports and exports, such as via the ResolveImportsToExports rule, without having mangen modify them.

OSGi R4 enhancements

The specification for OSGi R4 is still under development, although some early information has been made available on possible extensions to enhance package import and export controls and dependency management. As these start to become ratified, mangen will need to be enhanced to support them. Some possible areas are:

Deeper class processing

It's possible that mangen could go deeper into the scanned classes to provide enhanced functionality e.g.


Acknowledgements

Ascert is pleased to acknowledge the following projects, organisations and individuals whose tools have been used in the creation of this software:


Copyright © 2000-2005 Ascert, LLC.