This is to share some charts built on top of the http://perfspy.blogspot.com/2016/12/a-monitoring-system-for-java.html.
This graph shows the processing time of slowest request in 10 seconds (10 seconds is the default frequency I gathered data).
Details of the slowest requests
This table shows the detailed information of these requests. To get such information, there is some low-overhead code instrumentation required.
To know whether these requests are slow because the application is busy handling too many requests, we can see the throughput chart.
The amount of requests the application is handling doesn’t change too much over the time, so we can be sure that the slowest request are not slow because of the application stress. This coupled with the detailed information in the table provide clue to how to improve.
Average request processing time
This is the average processing time of all requests per 10 seconds. Interesting, the spike and valleys of this graph, in many places, corresponds to the first graph (slowest requests), which means, the slowest requests has a lot of influence over the average time.
So the question is, whether these slowest request slow down other requests?
Average request processing time without the slowest requests
This graph shows the average request processing time without the slowest requests. There are some spikes and valleys that correspond to the previous graph, which seem to suggest that the slowest requests are slowing down other requests.
But at this point, I would caution to draw such a conclusion. There are many factors that can influence performance, analysis such as above can provide some clues, but data analysis can be like reading tea leaves, you may find data points that prove your point. The important thing is to correlate with other information and design experiments to confirm.