top of page
Search

jemalloc for Native memory leak tracking!!

Updated: Aug 13, 2021

Symptoms:

  • There is insufficient memory for the Java Runtime Environment to continue.

  • Native memory allocation (mmap) failed to map 19277021184 bytes for committing reserved memory.

Possible reasons:

  • The system is out of physical RAM or swap space

  • In 32 bit mode, the process size limit was hit

Possible solutions:

  • Reduce memory load on the system

  • Increase physical memory or swap space

  • Check if swap backing store is full

  • Use 64 bit Java on a 64 bit OS

  • Decrease Java heap size (-Xmx/-Xms)

  • Decrease number of Java threads

  • Decrease Java thread stack sizes (-Xss)

  • Set larger code cache with -XX:ReservedCodeCacheSize=

  • This output file may be truncated or incomplete.

How To Track:

Download the jemalloc binaries from here and unzip on the host machine where native memory leak issue has been detected.

Build the jemalloc utility downloaded above with following commands:


./configure --enable-prof 
make    
make install

Start your server instance with follow inclusions to your environment script:


export LD_PRELOAD="/usr/local/lib/libjemalloc.so"
export MALLOC_CONF="prof:true,lg_prof_interval:30,lg_prof_sample:17,prof_leak:true"

Once server is started with above settings in place, you should see following heap files being created in INSTALLATION_DIR:

vp@:~/opt/weblogic$ ls -ltr *.heap


-rw-r----- 1 ubuntu ubuntu  99410 Aug 10 05:27 jeprof.1781.0.i0.heap
-rw-r----- 1 ubuntu ubuntu 132597 Aug 10 05:27 jeprof.1781.1.i1.heap
-rw-r----- 1 ubuntu ubuntu 323211 Aug 10 05:28 jeprof.1781.2.i2.heap
-rw-r----- 1 ubuntu ubuntu 322747 Aug 10 05:28 jeprof.1781.3.i3.heap
-rw-r----- 1 ubuntu ubuntu 316629 Aug 10 05:29 jeprof.1781.4.i4.heap
-rw-r----- 1 ubuntu ubuntu 295549 Aug 10 05:29 jeprof.1781.5.i5.heap
-rw-r----- 1 ubuntu ubuntu 330557 Aug 10 05:30 jeprof.1781.6.i6.heap
-rw-r----- 1 ubuntu ubuntu 328434 Aug 10 05:31 jeprof.1781.7.i7.heap

You can then generate the GIF or PDF representation of the heap file entries using below command:


jeprof --show_bytes --gif jeprof*.heap > jira-profiling.gif

Generated GIF would give you a visualisation of highest consumer, here is an example:



Or you can run this to get the object usage in mega bytes:


jeprof --inuse_space --gif jeprof*.heap > jira-profiling.gif


You can also use jeprof utility to check top consumer in command line like below:


user@:~/deployments/854jira$ jeprof --show_bytes jeprof.*.heap

(jeprof) top
Total: 1375477681 B
752957907  54.7%  54.7% 752957907  54.7% os::malloc@907020
352321536  25.6%  80.4% 352321536  25.6% init
184579893  13.4%  93.8% 184579893  13.4% updatewindow
32234098   2.3%  96.1% 32234098   2.3% readCEN
30978392   2.3%  98.4% 30978392   2.3% inflateInit2_
17339811   1.3%  99.6% 17339811   1.3% os::malloc@906e80
 1442216   0.1%  99.7%  1442216   0.1% newEntry.isra.4
 1048960   0.1%  99.8% 32027353   2.3% Java_java_util_zip_Inflater_init
  921991   0.1%  99.9% 353243527  25.7% _dl_rtld_di_serinfo

You can disable profiling for Jira by calling unset MALLOC_CONF and disabling references added in startup files.

472 views0 comments

Recent Posts

See All

Announcement banner in Jira is a in-built DIV that is embedded to each page of the its UI. The DIV allows you to stuff HTML/script/css that can used further to add/suppress/highlight new/existing styl

Post: Blog2_Post
bottom of page