Plugin Library dependency

[mfink]

Hey everyone,

I am developing a plugin for protege 4.3 in which I need to use Stardog client libraries. I tried almost anything now, from extracting all stardog .jars into my plugin .jar to building an osgi bundle out of the stardog jars in which I am exporting all packages and import all of them with my plugin.

The problem is I am always getting a stardog exception that is usually caused by missing jar files in the class path. I am using stardog classes which again use other stardog classes from the library than can't be found by the classloader as it seems. When I try to connect to my stardog server the driver class for handling the http address of the server can't be loaded. If I try the same code in an empty eclipse project where I import the same stardog jars it works fine.

Can anyone help me with this?

Thanks in advance,
Manuel

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

[Lorenz Buehmann]

The question is, how does the MANIFEST.MF look like? Its sounds just
like an OSGI configuration problem.

By the way, if you use StarDog you will not be allowed to publish your
plugin somewhere later - just a comment if this is something you planned
to do in the future.

Regards,
Lorenz
On 27.02.2015 16:01, Manuel Fink wrote:

> Hey everyone,
>
> I am developing a plugin for protege 4.3 in which I need to use Stardog client libraries. I tried almost anything now, from extracting all stardog .jars into my plugin .jar to building an osgi bundle out of the stardog jars in which I am exporting all packages and import all of them with my plugin.
>
> The problem is I am always getting a stardog exception that is usually caused by missing jar files in the class path. I am using stardog classes which again use other stardog classes from the library than can't be found by the classloader as it seems. When I try to connect to my stardog server the driver class for handling the http address of the server can't be loaded. If I try the same code in an empty eclipse project where I import the same stardog jars it works fine.
>
> Can anyone help me with this?
>
> Thanks in advance,
> Manuel
>
> _______________________________________________
> protege-dev mailing list
> [hidden email]
> https://mailman.stanford.edu/mailman/listinfo/protege-dev

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

[Timothy Redmond-2]

On 02/27/2015 07:01 AM, Manuel Fink wrote:

Hey everyone,

I am developing a plugin for protege 4.3 in which I need to use Stardog client libraries. I tried almost anything now, from extracting all stardog .jars into my plugin .jar to building an osgi bundle out of the stardog jars in which I am exporting all packages and import all of them with my plugin.

You are asking a question about OSGi class loading.  OSGi provides fine grain control over java package visibility and class loading but with this power comes some complexity.

There are several ways to get the results that you want. One approach is to embed the stardog jars in your plugin and another involves making the stardog jars into an OSGi plugin of its own and to import the classes that you need.  I will assume that you are trying to do the former.  The latter version is better if you expect that there will be other developers who want to use the stardog jar files.

I will start by describing the result that you are trying to achieve.  This effect is achieved by several other plugins including org.protege.editor.owl.jar.  This jar file uses classes from two other jar files (org.protege.xmlcatalog-1.0.4.jar and protege-owlapi-extensions-3.2.6.jar) that it embeds.  If you look at this jar file (using any tool that understands the zip format), you can find the evidence of how it works.  The contents of the org.protege.editor.owl.jar file looks like this:

Neptune:plugins% unzip -v org.protege.editor.owl.jar 
Archive:  org.protege.editor.owl.jar
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
   29545  Defl:N     4182  86% 02-28-2015 14:29 8321a776  META-INF/MANIFEST.MF
			...
   50854  Defl:N    43017  15% 02-01-2015 11:42 86a1cf69  org.protege.xmlcatalog-1.0.4.jar
			...
   94141  Defl:N    84383  10% 02-01-2015 11:42 865de7c6  protege-owlapi-extensions-3.2.6.jar
			...

For classes that execute code defined by org.protege.editor.owl.jar, the classpath to use is defined by the manifest.  The relevant portion of the MANIFEST.MF file involves the Bundle-Classpath.  The first few lines of MANIFEST.MF look like this:

Manifest-Version: 1.0
Bnd-LastModified: 1425162569412
Build-Jdk: 1.8.0_31
Built-By: developer
Bundle-Activator: org.protege.editor.owl.ProtegeOWL
Bundle-ClassPath: .,protege-owlapi-extensions-3.2.6.jar,
 org.protege.xmlcatalog-1.0.4.jar
Bundle-Description: OWL ontology editing infrastructure used by the Prot
 ege desktop application.

There are several ways to generate and maintain an OSGi manifest.  For small and one time projects, eclipse has nice tools for managing a manifest.  I haven't done this for a long time but my recollection is that they work very well. 

I am a firm believer in using IDE independent build scripts however and I think that maven is a very good choice.  It is used by the Protege team and it has good support for OSGi.  Also there are stardog jars in maven central which is useful.  I found some tutorials about

http://protegewiki.stanford.edu/wiki/PluginAnatomy					(may be out of date)
http://wso2.com/library/tutorials/develop-osgi-bundles-using-maven-bundle-plugin/
http://felix.apache.org/site/apache-felix-maven-bundle-plugin-bnd.html

Also here is a fragment of the pom.xml file for org.protege.editor.owl.jar:

        <plugins>
            <plugin>
                <groupId>org.apache.felix</groupId>
                <artifactId>maven-bundle-plugin</artifactId>
                <version>2.4.0</version>
                <extensions>true</extensions>
                <configuration>
                    <instructions>
                        <Bundle-Activator>org.protege.editor.owl.ProtegeOWL</Bundle-Activator>
                        <Bundle-ClassPath>., {maven-dependencies}</Bundle-ClassPath>
                        <Bundle-SymbolicName>${project.artifactId};singleton:=true</Bundle-SymbolicName>
                        <Bundle-Vendor>The Protege Development Team</Bundle-Vendor>
                        <Embed-Dependency>protege-owlapi-extensions, org.protege.xmlcatalog</Embed-Dependency>
                        <Export-Package>
                            ${project.artifactId}*;version=${project.version},
                            org.protege.xmlcatalog.*
                        </Export-Package>
                        <Import-Package>
                            org.eclipse.core.runtime;registry=split,
                            *
                        </Import-Package>
                        <Include-Resource>plugin.xml, {maven-resources}</Include-Resource>
                    </instructions>
                </configuration>
                <executions>
                    <execution>
                        <id>bundle-manifest</id>
                        <phase>install</phase>
                        <goals>
                            <goal>manifest</goal>
                        </goals>
                    </execution>
                </executions>

            </plugin>

The problem is I am always getting a stardog exception that is usually caused by missing jar files in the class path. I am using stardog classes which again use other stardog classes from the library than can't be found by the classloader as it seems. When I try to connect to my stardog server the driver class for handling the http address of the server can't be loaded. If I try the same code in an empty eclipse project where I import the same stardog jars it works fine.

Given this description I am wondering if you have given us enough information.  What is the exception exactly?  Are you sure that you are not able to load the relevant stardog classes?  It looks like some of the stardog jars may be class loader aware and sometimes exceptions that such libraries generate are not obvious.

When you run the code in eclipse, are you running the code as plain java code or are you running the code as an OSGi bundle?  If you are doing the latter then your problem is simply one of some configuration files.

-Timothy

Can anyone help me with this?

Thanks in advance,
Manuel

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

[mfink]

Thank you very much for your detailed answers Timothy and Lorenz,

Publishing the plugin was something we planned but given the license limitations of stardog, we might have to limit ourselves to only give a showcase on a conference.

Regarding my problem again:

First, this is the exception I am getting:

 com.complexible.stardog.StardogException: No driver was found which supports the connection string: 'http://localhost:5820/', please double-check the URL and verify the protocol is correct and try again.
    at com.complexible.stardog.api.DriverManager.getDriver(DriverManager.java:56)
    at com.complexible.stardog.api.admin.AdminConnectionConfiguration.connect(AdminConnectionConfiguration.java:37)

I followed your directions and tried again with several methods.

1. I extracted all 118 stardog jars that I need to include into a single "stardog-client.jar" which I put in the root directory of my plugin jar without extracting it. I used this as plugin manifest:

http://pastebin.com/A4VMt42m

2. I copied all 118 stardog jars that I need to the root directory of my plugin jar. I used this as plugin manifest:

http://pastebin.com/hC6ff8HB

3. I extracted all 118 stardog jars that I need to the root directory of my plugin jar. I used this as plugin manifest:

http://pastebin.com/Lf38KGkk

4. I built an OSGI bundle for the 118 stardog .jars using the eclipse wizard "Plug-In From Existing Jar-Files". I put that "stardog-client.jar" into the plugin directory.

This is the plugin manifest: http://pastebin.com/zS6CYzjd

This is the stardog bundle manifest: http://pastebin.com/bb8yXJu2

Each of these approaches resulted in the given exception.

I already asked about it on the stardog mailing list. Since they did not know about Protege plugin loading in detail, they referred me here.

>
> When you run the code in eclipse, are you running the code as plain java code or are you running the code as an OSGi bundle?  If you are doing the latter then your problem is simply one of some configuration files.
>

I am running it as plain java code with the same 118 stardog jars included as user library.

I will try your suggestion using maven as next step. Do you still think it might be a solution even though the approaches described avove all failed?

Thanks again,
Manuel

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

[Timothy Redmond-2]

Hi Manuel -

This is just a quick first cut answer to add some information that might be useful.

Is this closed source?  With the sources I could take a look at the driver code and I might be able to figure out what you should do.

On 03/03/2015 04:09 AM, Manuel Fink wrote:

Thank you very much for your detailed answers Timothy and Lorenz,

Publishing the plugin was something we planned but given the license limitations of stardog, we might have to limit ourselves to only give a showcase on a conference.

Regarding my problem again:

First, this is the exception I am getting:

 com.complexible.stardog.StardogException: No driver was found which supports the connection string: 'http://localhost:5820/', please double-check the URL and verify the protocol is correct and try again.
    at com.complexible.stardog.api.DriverManager.getDriver(DriverManager.java:56)
    at com.complexible.stardog.api.admin.AdminConnectionConfiguration.connect(AdminConnectionConfiguration.java:37)

This is what I was wondering about the exception.  It is clear from this exception that you are successfully loading the stardog classes. You can see from the exception that that the bundle is executing code from the stardog classes.  I think that this might mean that you have done all the configuration stuff correctly.

But I have seen this type of exception before. 

I will see if I can take a closer look but at a guess it looks like there is some code run when the stardog classes are initialized that installs a set of drivers and some other code, where your exception takes place where the drivers are used.  There may even be explicit calls in the stardog code that allow you to install a driver. 

What I think might be happening is that there is some code down deep in the stardog classes that is aware of the diversity of class loader implementations.  OSGi is not the only tool that does clever stuff with class loaders; tomcat, jboss, spring and several other environments also do things with the class loaders for much the same reasons (controlled class loader isolation of separate modules).


In my experience, what often needs to be done is to set the context class loader with some code like this:

ClassLoader oldContextClassLoader = Thread.getContextClassLoader();
try {
	Thread.setContextClassLoader(... probably use the current class loader ...);
	... probably the stardog initialization code ...
}
finally {
	Thread.setContextClassLoader(oldContextClassLoader);
}

This will tell some low level code what class loader to use when looking for drivers.

I followed your directions and tried again with several methods.

1. I extracted all 118 stardog jars that I need to include into a single "stardog-client.jar" which I put in the root directory of my plugin jar without extracting it. I used this as plugin manifest:

http://pastebin.com/A4VMt42m

2. I copied all 118 stardog jars that I need to the root directory of my plugin jar. I used this as plugin manifest:

http://pastebin.com/hC6ff8HB

I think that this is the more desirable version (with a bundle class path that references all the jars).  But you don't want to be doing this manually forever - this is where it is nice to use manifest generation tool such as bnd or maven.

3. I extracted all 118 stardog jars that I need to the root directory of my plugin jar. I used this as plugin manifest:

http://pastebin.com/Lf38KGkk

4. I built an OSGI bundle for the 118 stardog .jars using the eclipse wizard "Plug-In From Existing Jar-Files". I put that "stardog-client.jar" into the plugin directory.

This is the plugin manifest: http://pastebin.com/zS6CYzjd

This is the stardog bundle manifest: http://pastebin.com/bb8yXJu2

Each of these approaches resulted in the given exception.

I already asked about it on the stardog mailing list. Since they did not know about Protege plugin loading in detail, they referred me here.

I am guessing that the right stardog expert might know.  Especially if you posted the exception that you described above.  The question is what does that exception really mean and how and when does stardog try to find the missing driver.

When you run the code in eclipse, are you running the code as plain java code or are you running the code as an OSGi bundle?  If you are doing the latter then your problem is simply one of some configuration files.

I am running it as plain java code with the same 118 stardog jars included as user library.

I will try your suggestion using maven as next step. Do you still think it might be a solution even though the approaches described avove all failed?

I don't think that this will help.  My impression is that you are loading the stardog classes successfully but that there is a different problem with the drivers.  I think also that eclipse has some good tools for maintaining OSGi manifests.

But in the longer term, I would recommend this.  My bias is that I am a strong believer in the use of IDE independent build tools.

-Timothy

Thanks again,
Manuel

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

[mfink]

Hey Timothy,

Sorry for the late answer.

Since Stardog is closed source unfortunately we could not investigate the problem further and did not manage to solve it. Given its license restrictions it made sense anyways to switch to another reasoner.

We now settled with Konclude reasoner. Additionally I followed your advice and read up on maven which in fact I am now using succesfully to include owllink which we need to use for Konclude.

Using Konclude I ran again into runtime errors (some no class def errors related to XML parsing) despite building succesfully but this time the problems were easier to solve:

Adding these 3 packages to the import package statement of the manifest did the trick:

org.xml.sax.helpers,
org.xml.sax,
javax.xml.parsers

I have a notion that maybe my problems with Stardog were also related to the fact that these packages required for XML parsing were not imported and therefore not on the class path. It just never got in my mind that I have to manually import them since I did not have to in my Eclipse projects.

Thanks again for your help,

Manuel

Am 03.03.2015 um 18:24 schrieb Timothy Redmond:

Hi Manuel -

This is just a quick first cut answer to add some information that might be useful.

Is this closed source?  With the sources I could take a look at the driver code and I might be able to figure out what you should do.

On 03/03/2015 04:09 AM, Manuel Fink wrote:

Thank you very much for your detailed answers Timothy and Lorenz,

Publishing the plugin was something we planned but given the license limitations of stardog, we might have to limit ourselves to only give a showcase on a conference.

Regarding my problem again:

First, this is the exception I am getting:

 com.complexible.stardog.StardogException: No driver was found which supports the connection string: 'http://localhost:5820/', please double-check the URL and verify the protocol is correct and try again.
    at com.complexible.stardog.api.DriverManager.getDriver(DriverManager.java:56)
    at com.complexible.stardog.api.admin.AdminConnectionConfiguration.connect(AdminConnectionConfiguration.java:37)

This is what I was wondering about the exception.  It is clear from this exception that you are successfully loading the stardog classes. You can see from the exception that that the bundle is executing code from the stardog classes.  I think that this might mean that you have done all the configuration stuff correctly.

But I have seen this type of exception before. 

I will see if I can take a closer look but at a guess it looks like there is some code run when the stardog classes are initialized that installs a set of drivers and some other code, where your exception takes place where the drivers are used.  There may even be explicit calls in the stardog code that allow you to install a driver. 

What I think might be happening is that there is some code down deep in the stardog classes that is aware of the diversity of class loader implementations.  OSGi is not the only tool that does clever stuff with class loaders; tomcat, jboss, spring and several other environments also do things with the class loaders for much the same reasons (controlled class loader isolation of separate modules).


In my experience, what often needs to be done is to set the context class loader with some code like this:

ClassLoader oldContextClassLoader = Thread.getContextClassLoader();
try {
	Thread.setContextClassLoader(... probably use the current class loader ...);
	... probably the stardog initialization code ...
}
finally {
	Thread.setContextClassLoader(oldContextClassLoader);
}

This will tell some low level code what class loader to use when looking for drivers.

I followed your directions and tried again with several methods.

1. I extracted all 118 stardog jars that I need to include into a single "stardog-client.jar" which I put in the root directory of my plugin jar without extracting it. I used this as plugin manifest:

http://pastebin.com/A4VMt42m

2. I copied all 118 stardog jars that I need to the root directory of my plugin jar. I used this as plugin manifest:

http://pastebin.com/hC6ff8HB

I think that this is the more desirable version (with a bundle class path that references all the jars).  But you don't want to be doing this manually forever - this is where it is nice to use manifest generation tool such as bnd or maven.

3. I extracted all 118 stardog jars that I need to the root directory of my plugin jar. I used this as plugin manifest:

http://pastebin.com/Lf38KGkk

4. I built an OSGI bundle for the 118 stardog .jars using the eclipse wizard "Plug-In From Existing Jar-Files". I put that "stardog-client.jar" into the plugin directory.

This is the plugin manifest: http://pastebin.com/zS6CYzjd

This is the stardog bundle manifest: http://pastebin.com/bb8yXJu2

Each of these approaches resulted in the given exception.

I already asked about it on the stardog mailing list. Since they did not know about Protege plugin loading in detail, they referred me here.

I am guessing that the right stardog expert might know.  Especially if you posted the exception that you described above.  The question is what does that exception really mean and how and when does stardog try to find the missing driver.

When you run the code in eclipse, are you running the code as plain java code or are you running the code as an OSGi bundle?  If you are doing the latter then your problem is simply one of some configuration files.

I am running it as plain java code with the same 118 stardog jars included as user library.

I will try your suggestion using maven as next step. Do you still think it might be a solution even though the approaches described avove all failed?

I don't think that this will help.  My impression is that you are loading the stardog classes successfully but that there is a different problem with the drivers.  I think also that eclipse has some good tools for maintaining OSGi manifests.

But in the longer term, I would recommend this.  My bias is that I am a strong believer in the use of IDE independent build tools.

-Timothy

Thanks again,
Manuel

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev

_______________________________________________
protege-dev mailing list
[hidden email]
https://mailman.stanford.edu/mailman/listinfo/protege-dev