A Fork from statsd-jvm-profiler

statsd-jvm-profiler is used in A monitoring system for Java Application ---using InfluxDB, Grafana, Telegraph and statsd-jvm-profiler. It forks from etsy:statsd-jvm-profiler with the following changes:

Use influxDB tag to save multiple metrics in one measurement

etsy:statsd-jvm-profiler save each single metric in one measurement, for example, it saves heap metrics into multiple measurements (XXX is prefix defined through arguments):

XXX_total.init

XXX_total.max

XXX_total.commited

XXX_total.used

…

My fork saves related metrics into one measurement, using tag “type” to define metrics. The above metrics will be saved into one “XXX_heap” measurement:

> select * from test_heap where pid=~ /exp.*/ limit 10

name: test_heap

time                    host    metric_type     pid             type            value

----                    ----    -----------     ---             ----            -----

2016-12-29T07:08:05Z    exp     gauge           27645@exp_1229  total.committed 1.286602752e+09

2016-12-29T07:08:05Z    exp     gauge           27645@exp_1229  total.max       1.286602752e+09

2016-12-29T07:08:05Z    exp     gauge           27645@exp_1229  total.init      1.34217728e+09

2016-12-29T07:08:05Z    exp     gauge           27645@exp_1229  total.used      2.0164352e+07

 
All metrics about non-heap will be saved into “XXX_nonheap” measurement; All CPU tracing metrics will be saved into “XXX_cpu_trace”.

The benefits of saving multiple related metrics into one measurement are:

• Less clutter, easier to inspect the influxdb database
•  Easier to create graphs on Grafana. In Grafana, users can select measurements from a dropdown list, if there are many measurements, user-ability will suffer

Use a configuration file to define arguments

With etsy:statsd-jvm-profiler, you need to define arguments via the command line, this becomes unwieldy when there are many arguments to differ. My fork defines arguments through a configuration file. Below is an exemplary configuration file: 

server:"localhost"

port: "8125",

prefix: "test",

profilers: "MemoryProfiler:GeneralMBeanProfiler:CPUTracingProfiler",

packageWhitelist: "packa:packb:packc",

period: 10,

beans:[

      {

            name: "jboss.web:type=Manager,context=/a,host=localhost"

            metric: "session_a"

            attributes:["activeSessions", "expiredSessions" ]

      },   

     

     

      {

            name: "jboss.web:type=DataSource,class=javax.sql.DataSource,name=\"b\""

            metric: "connection_b"

            attributes:["connectCount", "closeCount","activeCount","activePeak","notEmptyWaitCount","notEmptyWaitMillis","notEmptyWaitThreadCount","notEmptyWaitThreadPeak" ]

      }

     

]

 

You can now define period, whose time unit is second. If you want to define period for one particular profiler, use “profilername-period” to define, for example:

CPUTracingProfiler-period: 30

Add GeneralMBeanProfiler

GeneralMBeanProfiler collects information from general MBeans. An example is shown in the above configuration file. 

CPU Flame Graph within a period

With cpu tracing go into one measurement, my fork supports generating cpu flame graphs within a period. influxdb_dump.py now supports two new parameters: begin and end. Example: 

./influxdb_dump.py --host localhost --port 8086 --username admin --password admin --database telegraf --prefix test --begin 2016-12-29T07:34:00Z --end 2016-12-29T07:36:00Z | ./flamegraph.pl > flame.svg

A CPU Flame Graph created in this way is useful if:
1) you have periodic CPU spikes
2) you have CPU spikes that last a long time
3) you want to understand why heap consumption jumps up regularly 

Here is an explanation of CPU Flame Graph: How to read a CPU Flame Graph

Enable profilers through REST

A post REST API is created to support enabling profilers at runtime, example:

curl -XPOST http://localhost:5005/enable/CPUTracingProfiler -d '{"packageWhitelist": ["a","b"]}'

The body of this API should be a json.