Home Forums Kamanja Forums Features & Releases Memory leak Suspect

This topic contains 11 replies, has 4 voices, and was last updated by  Archived_User64 1 year, 8 months ago.

  • Author
    Posts
  • #13416 Reply

    Archived_User1
    Participant

    Hi ,

    I’ve been facing a problem a with Kamanja engine , it keep stopping after deploy remove then redeploy the models,messages,containers more that once , so i have to clean everything kafka ,zookeeper and storage , so i ran a script to take a java heap dump using jmap from the Kamanja engine pid id then run some analysis on the dump file the tool found a memory leak and throw this msg :
    __________________________________________________________
    Problem Suspect 1
    One instance of “org.mapdb.EngineWrapper$CloseOnJVMShutdown” loaded by “sun.misc.Launcher$AppClassLoader @ 0xd9e00848” occupies 18,826,168 (59.30%) bytes. The memory is accumulated in one instance of “org.mapdb.Caches$HashTable$HashItem[]” loaded by “sun.misc.Launcher$AppClassLoader @ 0xd9e00848”.

    Keywords
    sun.misc.Launcher$AppClassLoader @ 0xd9e00848
    org.mapdb.EngineWrapper$CloseOnJVMShutdown
    org.mapdb.Caches$HashTable$HashItem[]

    __________________________________________________________

    I dig more into the classes and found class “com.ligadata.KamanjaManager.OleService” is the class that create the “sun.misc.Launcher$AppClassLoader” and it’s been used in “com.ligadata.MetadataAPI.MetadataAPIImpl$” when jarStore , tableStoreMap .

    note : I test this using mapdb on my own VM .

    Thanks
    Saleh

  • #13417 Reply

    Archived_User79
    Participant

    You are doing good detective work, thanks. Would you tell us just how you got to the point where it stopped so we can replicate it?

    Thanks
    Donald

    • #13418 Reply

      Archived_User1
      Participant

      sure , First you have to get the heap dump using the jmap command “jmap -dump:format=b,file=cheap.bin <pid>” , take that dump of the memory heap then extract it using “jvisualvm” or any other analysis tool like “eclipse memory analyser” now as u can see in the image below the “org.mapdb.Caches$HashTable$HashItem[]” have only 9 objects and there is a large difference between the Shallow heap and the Retained Heap this is not normal .

      so i ran the analyzer and get this

      now trace the “org.mapdb.Caches$HashTable$HashItem[]” and point to the lager objects and where they are created

      there is a memory leak in the 3rd party mapdb or in the way we are using it am not sure yet , someone can help and take a look , i think we should also check HBASE and Cassandra .

      Thanks
      Saleh

  • #13419 Reply

    Archived_User64
    Participant

    While running cassandra as the datastore, I am getting the below out of memory error:

    -bash-4.1$ Exception in thread “main” java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
    at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
    at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:252)
    at com.ligadata.ZooKeeper.CreateClient$.createSimple(CreateClient.scala:90)
    at com.ligadata.ZooKeeper.CreateClient$.CreateNodeIfNotExists(CreateClient.scala:34)
    at com.ligadata.KamanjaManager.KamanjaMetadata$.InitMdMgr(KamanjaMetadata.scala:647)
    at com.ligadata.KamanjaManager.KamanjaManager.initialize(KamanjaManager.scala:296)
    at com.ligadata.KamanjaManager.KamanjaManager.run(KamanjaManager.scala:395)
    at com.ligadata.KamanjaManager.OleService$.main(KamanjaManager.scala:540)
    at com.ligadata.KamanjaManager.OleService.main(KamanjaManager.scala)

    I am facing this problem since yesterday. I am running the engine with the following command:
    java -Xmx50G -Xms50G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Dlog4j.configuration=file:$KAMANJA_HOME/config/log4j.properties -jar $KAMANJA_HOME/bin/KamanjaManager-1.0 –config $KAMANJA_HOME/config/EngineConfig.properties > /dev/null &

    This looks like a similar memory leak issue at com.ligadata.KamanjaManager.OleService.main(KamanjaManager.scala)

  • #13420 Reply

    Archived_User1
    Participant

    Dhaval ,
    Can u run this command “jmap -dump:format=b,file=cheap.bin <pid>” on ur machine , use the pid id of kamanja engine , please share the cheap.bin file so I can run some analysis .

    Thanks
    Saleh

  • #13421 Reply

    Archived_User13
    Participant

    Saleh,

    Regarding your original message, how do you know it is memory leak? In JVM, leaked objects (unreferenced) will be cleaned automatically through garbage collection process. MapDB does cache many items in its memory and I want to make sure there is no confusion between cached data vs. leak.

    Thanks

    -Krishna

  • #13422 Reply

    Archived_User1
    Participant

    I took more than one heap dump and the “org.mapdb.Caches$HashTable$HashItem[]” is not getting any lower , and the “eclipse memory analyzer” suspected the leak u can also use the tool to watch the pointed object for the java gc .

    Thanks
    Saleh

  • #13423 Reply

    Archived_User13
    Participant

    Hallmark of memory leak is continuous increase of memory in long running process. How long you have been running this process and what is the rate of increase in memory usage?

    Thanks

    -Krishna

  • #13424 Reply

    Archived_User1
    Participant

    Is three to four working days enough

  • #13425 Reply

    Archived_User13
    Participant

    Yes, that would be good start. Depending on amount of leak, sometimes few mins/hour may be sufficient to show significant increase in memory, but long duration one is good test. Also note that Kamanja won’t release lot of cached items (currently no expiry of cached containers), so you will see some increase, but it should level off at some point unless models are constantly creating new elements to cache.

    These tests should be done methodically rather than some ad hoc fashion for us to really identify leaks. Leaks could be associated with number of events being processed or could be related to metadata operations. The former causes higher rate of leak while later type causes slower leak. If you are doing any memory leak test, create a task and write steps on what and how you are planning to do it, so others can review it.

    Thanks

    -Krishna

  • #13426 Reply

    Archived_User1
    Participant

    Yes sure , I think a bigger loaded box will give better and clearer results , regarding the cashed objects it seems some of the is not released , I will create a task on jira for more investigation .

  • #13427 Reply

    Archived_User64
    Participant

    Hi Saleh,

    I just checked with Pokuri and the error I was getting was due to limit on threads. It is working now after increasing the limit.

Reply To: Memory leak Suspect
Your information: