Disposable Memories

This post is going to be a very technical look at GATE's memory consumption. This will involve discussion of Java's class loading mechanism and garbage collection. If none of that interests you, or if that first sentence didn't make sense, then can I make a friendly suggestion that you stop reading about now!

Before we dive in you need a little background on how memory is used in Java. This short description isn't entirely accurate (a full and accurate description would probably require an entire book) but will suffice for the purposes of this post.

There are, broadly speaking, two types of information that Java keeps in memory when running an application; the definitions of each class that has been used and the information about each instance of a particular class. The instance level information is created when you use the new keyword and is made available for garbage collection when there is no longer anyway of accessing the specific instance. The class definitions on the other hand are only ever released when the classloader instance, which loaded them into memory, is garbage collected. If the Java application doesn't do any custom class loading then all class definitions will be loaded into the system classloader which is never garbage collected. Class definitions are stored in the PermGen area of the Java heap which is why, if you load too many class definitions, you will eventually get the message:

Exception in thread "main" java.lang.OutOfMemoryError: PermGen space

The common way of avoiding this problem is simply to increase the amount of available memory (either the total heap size or just the PermGen area). While this often makes the problem go away, it actually just increases the time until the problem will occur.

GATE itself contains a lot of classes (over 2000 just in the main source tree) but also supports dynamically loading plugins and compiling new classes from JAPE grammars. This all means that there is no maximum number of classes that might be loaded and hence no way of ensuring that the PermGen is always big enough. Fortunately GATE doesn't dynamically load classes into the system classloader but into a custom classloader as you can see from this diagram. Unfortunately this classloader is a singleton instance which is never released.

One of the side effects of this is that if you re-initialize (or close and re-create) a JAPE transducer a new copy of the class definitions are created and added to the PermGen area. This is one of the reasons that when using GATE as part of a web service we suggest using a pool of pipelines rather than creating a new pipeline for each request. Not only does this reduce response times (assuming an adequate sized pool) but also prevents exhaustion of the PermGen area. To show just how quickly this can become a problem I wrote the following short piece of code.

Gate.runInSandbox(true);
Gate.init();

Gate.getCreoleRegister().registerDirectories(
  (new File("/home/mark/gate-top/externals/gate/plugins/ANNIE/"))
    .toURI().toURL());

FeatureMap params = Factory.newFeatureMap();
Transducer jape = (Transducer)Factory.createResource(
  "gate.creole.ANNIETransducer", params);

long count = 1;
while (true) {
  System.out.println(count);
  jape.reInit();
  ++count;
}

This simply initializes GATE, creates a single instance of the ANNIE NE Transducer and then repeatedly re-initializes it. On my machine Java defaults to using 82MB for the PermGen and this was exhausted after loading the transducer just 104 times. I was monitoring the performance using VisualVM and you can see from the following screen shot how the memory was quickly exhausted.

There are a number of other issues with using a singleton classloader but they boil down to the fact that once a class has been defined it can never be forgotten or redefined. The practical aspects of this are that unloading a plugin doesn't actually result in the class definitions being forgotten, so you can't unload, recompile, reload a plugin. If you want to make a change you have to close and restart GATE. Another problem is that if two plugins try and load different versions of the same class only the first version will be used. This is particularly problematic when dealing with complex plugins which may use multiple third-party libraries. With the plugins in the main distribution we try and keep to just one version of each library but clearly we have no control over third-party plugins.

These issues have annoyed me for a while, but I haven't had either the time (this has all been done in my own time) or in some cases the technical knowledge to do anything about it. A few weeks ago, having read a book on Java Performance, a couple of pieces of the puzzle started to fall into place and I realized that I could probably have a good crack at a new classloader architecture that would solve all of these problems. You can see the architecture I've adopted in the diagram to the right. There are two things this diagram doesn't show. Firstly there can be any number of plugin or JAPE classloaders, and secondly those classloaders are what is known as parent last. The reason for having any number of plugin or JAPE classloaders (they are actually the same class I'm just splitting them up on the diagram to show that it handles both plugin loading and compiling JAPE grammars) should be obvious, as it allows us to throw one away when it is no longer required (i.e. we don't want to have to unload all plugins just to unload one). The idea of parent last classloading, however, requires a more detailed explanation.

Traditionally classloaders in Java take a parent first approach. This means that when they are asked to load a class they start by asking their parent classloader (follow the arrows upwards) to load the class, which in turn asks it's parent etc. It's only if this fails that a classloader will itself try and load a class. Changing to using parent last means that two classloaders can now have different copies of the same class defined within them, and hence we can support different versions of the same third party library appearing within different plugins. This works because when a class in a plugin tries to load a class it looks within it's own plugin before looking in either the main GATE classloader or the classloaders associated with other plugins or JAPE grammars. This mechanism also allows classes in different plugins to refer to one another (i.e. a JAPE grammar can refer to classes loaded via a plugin).

To ensure that classloaders can be released and garbage collected properly, I've also made a change to the way plugins are unloaded. Currently in GATE unloading a plugin simply removes the definition of the resources it contains from the CREOLE register, but it doesn't unloaded any instances of the resources that are currently in use. This does tend to lead to some funny behaviour (often weird AWT error messages) and to ensure a thorough cleanup I've updated this code so that it does now unload any instances of resources that depend on the plugin before unloading the plugin itself.

It's taken quite a bit of time hunting through heap dumps using VisualVM but I'm now confident that this new approach works well and that classloaders, and hence classes, can be thrown away when they are no longer needed. As an example, we can use this modified version of GATE along with the sample code I showed before (to re-initialize a JAPE grammar over and over again). This time I let it run for over 1000 iterations (almost 10 minutes) before I stopped it as there was no sign of it running out of memory. A quick look at VisualVM explains why.

You can clearly see that every time the PermGen is almost exhausted garbage collection kicks in freeing some of the previously loaded and discarded class definitions.

This work has all been done in this separate SVN branch so as not to disrupt the main source tree. This means that to try this modified, memory friendly version of GATE you will need to check out the branch and compile it yourself. Mind you if you have read all the way to here then I'm going to guess that won't be a problem. My hope is that this will eventually be merged back into the trunk but I'd prefer to wait until this branch has been tested by a number of other people and not just me. Also there are currently 9 tests that fail when using this branch, although I've looked at the failing code and in every case it's actually the test that is at fault. Essentially the tests load each plugin twice, once into the system classloader (as the classes are on the main Java classpath) and once into the GATE classloader (when the plugin is loaded via the API). Because of the change to parent last classloading, this means that two different definitions of the same class can end up being used which results in apparently nonsensical error messages such as: Error converting class gate.chineseSeg.RunMode to class gate.chineseSeg.RunMode. This situation never happens during normal use of GATE Developer, and can be fixed by altering the creole.xml files used to define the plugins. In other words the failed tests shouldn't stop you trying this branch, although I also wouldn't suggest using this branch in a production environment until it has seen further testing.

So on that note, please try the branch if you have been concerned about memory consumption or had problems with clashing library versions and let me know if you find any problems, or if you have any further suggestions for improvements.