jemalloc for Native memory leak tracking!!

Symptoms:

• There is insufficient memory for the Java Runtime Environment to continue.

• Native memory allocation (mmap) failed to map 19277021184 bytes for committing reserved memory.

Possible reasons:

• The system is out of physical RAM or swap space

• In 32 bit mode, the process size limit was hit

Possible solutions:

• Reduce memory load on the system

• Increase physical memory or swap space

• Check if swap backing store is full

• Use 64 bit Java on a 64 bit OS

• Decrease Java heap size (-Xmx/-Xms)

• Decrease number of Java threads

• Decrease Java thread stack sizes (-Xss)

• Set larger code cache with -XX:ReservedCodeCacheSize=

• This output file may be truncated or incomplete.

How To Track:

Download the jemalloc binaries from here and unzip on the host machine where native memory leak issue has been detected.

Build the jemalloc utility downloaded above with following commands:

./configure --enable-prof 
make    
make install

Start your server instance with follow inclusions to your environment script:

export LD_PRELOAD="/usr/local/lib/libjemalloc.so"
export MALLOC_CONF="prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_leak:true"

Once server is started with above settings in place, you should see following heap files being created in INSTALLATION_DIR:

vp@:~/opt/weblogic$ ls -ltr *.heap

-rw-r----- 1 ubuntu ubuntu  99410 Aug 10 05:27 jeprof.1781.0.i0.heap
-rw-r----- 1 ubuntu ubuntu 132597 Aug 10 05:27 jeprof.1781.1.i1.heap
-rw-r----- 1 ubuntu ubuntu 323211 Aug 10 05:28 jeprof.1781.2.i2.heap
-rw-r----- 1 ubuntu ubuntu 322747 Aug 10 05:28 jeprof.1781.3.i3.heap
-rw-r----- 1 ubuntu ubuntu 316629 Aug 10 05:29 jeprof.1781.4.i4.heap
-rw-r----- 1 ubuntu ubuntu 295549 Aug 10 05:29 jeprof.1781.5.i5.heap
-rw-r----- 1 ubuntu ubuntu 330557 Aug 10 05:30 jeprof.1781.6.i6.heap
-rw-r----- 1 ubuntu ubuntu 328434 Aug 10 05:31 jeprof.1781.7.i7.heap

You can then generate the GIF or PDF representation of the heap file entries using below command:

jeprof --show_bytes --gif jeprof*.heap > jira-profiling.gif

Generated GIF would give you a visualisation of highest consumer, here is an example:

Or you can run this to get the object usage in mega bytes:

jeprof --inuse_space --gif jeprof*.heap > jira-profiling.gif

You can also use jeprof utility to check top consumer in command line like below:

user@:~/deployments/854jira$ jeprof --show_bytes jeprof.*.heap

(jeprof) top
Total: 1375477681 B
752957907  54.7%  54.7% 752957907  54.7% os::malloc@907020
352321536  25.6%  80.4% 352321536  25.6% init
184579893  13.4%  93.8% 184579893  13.4% updatewindow
32234098   2.3%  96.1% 32234098   2.3% readCEN
30978392   2.3%  98.4% 30978392   2.3% inflateInit2_
17339811   1.3%  99.6% 17339811   1.3% os::malloc@906e80
 1442216   0.1%  99.7%  1442216   0.1% newEntry.isra.4
 1048960   0.1%  99.8% 32027353   2.3% Java_java_util_zip_Inflater_init
  921991   0.1%  99.9% 353243527  25.7% _dl_rtld_di_serinfo

You can disable profiling for Jira by calling unset MALLOC_CONF and disabling references added in startup files.