Is the InMemoryFrameDb class a memory leack?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Is the InMemoryFrameDb class a memory leack?

Bernhard2
Hello,

 

I have some question and remarks regarding the InMemoryFrameDb class.

 

I have a big ontology and serve this ontology via a servlet into the web. The servlet code connects via RMI to a protégé server to retrieve the ontology details. The protégé server reads the ontology data from a database.

 

The ontology (slot values, slots, instances ...) is "bigger" then the size of the main memory (1 GB). I give (via "-Xmx") the servlet container java process 512 MB and also the protégé server java process 512 MB as maximum heap size. I start both java processes with the "-verbose:gc" option to trace the memory usage.

 

It now turns out that after some days the protégé server runs out of memory. Yes the CacheMap objects which use SoftReference instances help to free memory each time the memory is completely used, but over time the several Map instances in the InMemoryFrameDb objects consume the whole memory. Finally the protégé server dies.

 

What can I do now? Is the InMemoryFrameDb class a killer for application where the ontology is so big that you can not cache all ontology data in memory?

 

Regards,

 

Bernd


-------------------------------------------------------------------------
To unsubscribe go to http://protege.stanford.edu/community/subscribe.html

Reply | Threaded
Open this post in threaded view
|

Re: Is the InMemoryFrameDb class a memory leack?

Bernhard2

Hello,

if my spam filter did not delete an email from this list, then I got now response...

Dear protégé team: some days ago I asked how to file a bug and I was asked to post my issues here. Please take this issue as a P1 bug. Without a fix protégé can only be used for small and middle sized academic projects.

For what is the InMemoryFrameDb class used?

Are instances of the InMemoryFrameDb class essential needed or can protégé be changed to be configurable whether to use instances of this class or not.

Thanks and regards,

Bernd


-----Ursprüngliche Nachricht-----
Von: [hidden email] [mailto:[hidden email]] Im Auftrag von Bernhard Wellhöfer
Gesendet: Freitag, 19. Mai 2006 12:29
An: [hidden email]
Betreff: [protege-discussion] Is the InMemoryFrameDb class a memory leack?

Hello,

 

I have some question and remarks regarding the InMemoryFrameDb class.

 

I have a big ontology and serve this ontology via a servlet into the web. The servlet code connects via RMI to a protégé server to retrieve the ontology details. The protégé server reads the ontology data from a database.

 

The ontology (slot values, slots, instances ...) is "bigger" then the size of the main memory (1 GB). I give (via "-Xmx") the servlet container java process 512 MB and also the protégé server java process 512 MB as maximum heap size. I start both java processes with the "-verbose:gc" option to trace the memory usage.

 

It now turns out that after some days the protégé server runs out of memory. Yes the CacheMap objects which use SoftReference instances help to free memory each time the memory is completely used, but over time the several Map instances in the InMemoryFrameDb objects consume the whole memory. Finally the protégé server dies.

 

What can I do now? Is the InMemoryFrameDb class a killer for application where the ontology is so big that you can not cache all ontology data in memory?

 

Regards,

 

Bernd


-------------------------------------------------------------------------
To unsubscribe go to http://protege.stanford.edu/community/subscribe.html




-------------------------------------------------------------------------
To unsubscribe go to http://protege.stanford.edu/community/subscribe.html

Reply | Threaded
Open this post in threaded view
|

Re: Is the InMemoryFrameDb class a memory leack?

Timothy Redmond


Your application sounds quite cool by the way.

I thought I would put our conversation back on the forum.  First, you  
are right - my fix for a CacheMap memory leak is irrelevant.  Second,  
I would ask if you are doing anything special over there.  We have  
several people who are making heavy use of the server-client and are  
not experiencing your problem.  (This does not mean that there is no  
leak - it just means that it is less trivial to identify).  Do you  
have your own slot plugins or are you using some standard plugins?

I don't think that the InMemoryFrameDb class actually has a memory  
leak on its own.  It is used to hold the contents of an ontology in  
memory. The contents of the hash tables correspond to contents that  
some other component or user has chosen to store.

So you might ask, if I have a database project, why is anybody  
holding holding ontologies in memory.  There are three types of  
InMemoryFrameDb used in a database project:

   1. the system frames are held in memory.  The system frames do not  
change over time
       so this should not be the leak.
   2. there is a placeholder ontology whose purpose I am not clear  
on.  This one is always empty
       so this should not be the leak either.
   3. there is a project ontology describing the project.  This is  
different than the main ontology that the user
       edits and is held in the database.  This is probably where  
your leak is happening.
       I am not sure exactly what is in here but I think it includes  
things like form information and such.  It
       is possible that some plugin is leaving information in this  
ontology and it is collecting (fairly fast!).  I am actually
       having a hard time seeing how this is possible in server-
client mode also because I don't think that the client can
       write to this ontology.

I will work on replicating your bug and it has been placed on the bug  
database along with the original description.

Sorry I can't be of more help at present.  This probably doesn't fit  
your time frame but I will try to figure out what is going on.

-Timothy


On May 22, 2006, at 12:56 AM, Bernhard Wellhöfer wrote:

>
> Hello,
>
> if my spam filter did not delete an email from this list, then I  
> got now response...
>
> Dear protégé team: some days ago I asked how to file a bug and I  
> was asked to post my issues here. Please take this issue as a P1  
> bug. Without a fix protégé can only be used for small and middle  
> sized academic projects.
>
> For what is the InMemoryFrameDb class used?
>
> Are instances of the InMemoryFrameDb class essential needed or can  
> protégé be changed to be configurable whether to use instances of  
> this class or not.
>
> Thanks and regards,
>
> Bernd
>
>
> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:protege-
> [hidden email]] Im Auftrag von Bernhard Wellhöfer
> Gesendet: Freitag, 19. Mai 2006 12:29
> An: [hidden email]
> Betreff: [protege-discussion] Is the InMemoryFrameDb class a memory  
> leack?
>
> Hello,
>
>
>
> I have some question and remarks regarding the InMemoryFrameDb class.
>
>
>
> I have a big ontology and serve this ontology via a servlet into  
> the web. The servlet code connects via RMI to a protégé server to  
> retrieve the ontology details. The protégé server reads the  
> ontology data from a database.
>
>
>
> The ontology (slot values, slots, instances ...) is "bigger" then  
> the size of the main memory (1 GB). I give (via "-Xmx") the servlet  
> container java process 512 MB and also the protégé server java  
> process 512 MB as maximum heap size. I start both java processes  
> with the "-verbose:gc" option to trace the memory usage.
>
>
>
> It now turns out that after some days the protégé server runs out  
> of memory. Yes the CacheMap objects which use SoftReference  
> instances help to free memory each time the memory is completely  
> used, but over time the several Map instances in the  
> InMemoryFrameDb objects consume the whole memory. Finally the  
> protégé server dies.
>
>
>
> What can I do now? Is the InMemoryFrameDb class a killer for  
> application where the ontology is so big that you can not cache all  
> ontology data in memory?
>
>
>
> Regards,
>
>
>
> Bernd
>
>
> ----------------------------------------------------------------------
> ---
> To unsubscribe go to http://protege.stanford.edu/community/ 
> subscribe.html
>
>
>
>
> ----------------------------------------------------------------------
> ---
> To unsubscribe go to http://protege.stanford.edu/community/ 
> subscribe.html
>

-------------------------------------------------------------------------
To unsubscribe go to http://protege.stanford.edu/community/subscribe.html

Reply | Threaded
Open this post in threaded view
|

Re: Is the InMemoryFrameDb class a memory leack?

Bernhard2
In reply to this post by Bernhard2

Hello Timothy,

First I have to apologize and double your statement. Yes instances of the InMemoryFrameDB class are only used to cache information like system classes/slots or project ontologies. During my investigation of the memory leak I noticed that InMemoryFrameDB instances are used to cache data and the InMemoryFrameDB class does not handle the case to free the cache when memory is running low. I did not realize that only the information mentioned above is cached by these instances.

Nevertheless the memory leak exists and the hunt for the reason starts again. I now have a test program to force the memory leak. Hopefulyy someone has a good memory usage profiler.

I will start a new mail thread to discuss and find the memory leak. We should not use this thread about the InMemoryFrameDB class for this.

Thx,

Bernd

 

-----Ursprüngliche Nachricht-----
Von: Timothy Redmond [mailto:[hidden email]]
Gesendet: Freitag, 26. Mai 2006 21:24
An: [hidden email]
Cc: Bernhard Wellhöfer
Betreff: Re: [protege-discussion] Re: Is the InMemoryFrameDb class a memory leack?


Your application sounds quite cool by the way.

I thought I would put our conversation back on the forum.  First, you are right - my fix for a CacheMap memory leak is irrelevant.  Second, I would ask if you are doing anything special over there.  We have several people who are making heavy use of the server-client and are not experiencing your problem.  (This does not mean that there is no leak - it just means that it is less trivial to identify).  Do you have your own slot plugins or are you using some standard plugins?

I don't think that the InMemoryFrameDb class actually has a memory leak on its own.  It is used to hold the contents of an ontology in memory. The contents of the hash tables correspond to contents that some other component or user has chosen to store.

So you might ask, if I have a database project, why is anybody holding holding ontologies in memory.  There are three types of InMemoryFrameDb used in a database project:

   1. the system frames are held in memory.  The system frames do not change over time
       so this should not be the leak.
   2. there is a placeholder ontology whose purpose I am not clear on.  This one is always empty
       so this should not be the leak either.
   3. there is a project ontology describing the project.  This is different than the main ontology that the user
       edits and is held in the database.  This is probably where your leak is happening.
       I am not sure exactly what is in here but I think it includes things like form information and such.  It
       is possible that some plugin is leaving information in this ontology and it is collecting (fairly fast!).  I am actually
       having a hard time seeing how this is possible in server- client mode also because I don't think that the client can
       write to this ontology.

I will work on replicating your bug and it has been placed on the bug database along with the original description.

Sorry I can't be of more help at present.  This probably doesn't fit your time frame but I will try to figure out what is going on.

-Timothy


On May 22, 2006, at 12:56 AM, Bernhard Wellhöfer wrote:

>
> Hello,
>
> if my spam filter did not delete an email from this list, then I got
> now response...
>
> Dear protégé team: some days ago I asked how to file a bug and I was
> asked to post my issues here. Please take this issue as a P1 bug.
> Without a fix protégé can only be used for small and middle sized
> academic projects.
>
> For what is the InMemoryFrameDb class used?
>
> Are instances of the InMemoryFrameDb class essential needed or can
> protégé be changed to be configurable whether to use instances of this
> class or not.
>
> Thanks and regards,
>
> Bernd
>
>
> -----Ursprüngliche Nachricht-----
> Von: [hidden email] [mailto:protege-
> [hidden email]] Im Auftrag von Bernhard Wellhöfer
> Gesendet: Freitag, 19. Mai 2006 12:29
> An: [hidden email]
> Betreff: [protege-discussion] Is the InMemoryFrameDb class a memory
> leack?
>
> Hello,
>
>
>
> I have some question and remarks regarding the InMemoryFrameDb class.
>
>
>
> I have a big ontology and serve this ontology via a servlet into the
> web. The servlet code connects via RMI to a protégé server to retrieve
> the ontology details. The protégé server reads the ontology data from
> a database.
>
>
>
> The ontology (slot values, slots, instances ...) is "bigger" then the
> size of the main memory (1 GB). I give (via "-Xmx") the servlet
> container java process 512 MB and also the protégé server java process
> 512 MB as maximum heap size. I start both java processes with the
> "-verbose:gc" option to trace the memory usage.
>
>
>
> It now turns out that after some days the protégé server runs out of
> memory. Yes the CacheMap objects which use SoftReference instances
> help to free memory each time the memory is completely used, but over
> time the several Map instances in the InMemoryFrameDb objects consume
> the whole memory. Finally the protégé server dies.
>
>
>
> What can I do now? Is the InMemoryFrameDb class a killer for
> application where the ontology is so big that you can not cache all
> ontology data in memory?
>
>
>
> Regards,
>
>
>
> Bernd
>
>
> ----------------------------------------------------------------------
> ---
> To unsubscribe go to http://protege.stanford.edu/community/ 
> subscribe.html
>
>
>
>
> ----------------------------------------------------------------------
> ---
> To unsubscribe go to http://protege.stanford.edu/community/ 
> subscribe.html
>




-------------------------------------------------------------------------
To unsubscribe go to http://protege.stanford.edu/community/subscribe.html