As a sysadmin I sometimes face situations, where a program behaves abnormally, while not creating errors at all or creating nonsense error-messages.
In the past – before java came in – there were two counter-measures:
- If nothing else helps – RTFM 😉
- If even 1. does not help – trace the system-calls and see what is happening
I usually use strace -f for this task with Linux (other OS have similar trace-tools). Now while this usually works well for any old-fashioned program, the trace gets very fuzzy when doing the same on a java-process. There are so many system-calls seemingly unrelated to any real action, that it is terrible to search through such a dump.
Are there better ways to do that (if the source-code is not available)?
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
As ckhan mentioned, jstack is great because it gives the full stack trace of all active threads in the JVM. The same can be obtained on stderr of the JVM using SIGQUIT.
Another useful tool is jmap which can grab a heap dump from the JVM process using the PID of the process:
jmap -dump:file=/tmp/heap.hprof $PID
This heap dump can be loaded in tools like visualvm (which is now part of the standard Oracle java sdk install, named jvisualvm). In addition, VisualVM can connect to the running JVM and display information about the JVM, including showing graphs of internal CPU usage, thread counts, and heap usage – great for tracking down leaks.
Another tool, jstat, can collect garbage collection statistics for the JVM over a period of time much like vmstat when run with a numeric argument (e.g. vmstat 3).
Finally, it is possible to use a Java Agent to push instrumentation on all methods of all objects at load-time. The library javassist can help to make this very easy to do. So, it is feasible to add your own tracing. The hard part with that would be finding a way to get trace output only when you wanted it and not all the time, which would likely slow the JVM to a crawl. There’s a program called dtrace that works in a manner like this. I’ve tried it, but was not very successful. Note that agents cannot instrument all classes because the ones needed to bootstrap the JVM are loaded before the agent can instrument, and then it’s too late to add instrumentation to those classes.
My Suggestion – start with VisualVM and see if that tells you what you need to know since it can show the current threads and important stats for the JVM.
Method 2
In the same vain when debugging programs that have gone awry on a Linux system you can use similar tools to debug running JVMs on your system.
Tool #1 – jvmtop
Similar to top, you can use jvmtop to see what classes are up to within the running JVMs on your system. Once installed you invoke it like this:
$ jvmtop.sh
Its output is similarly styled to look like the tool top:
JvmTop 0.8.0 alpha amd64 8 cpus, Linux 2.6.32-27, load avg 0.12 http://code.google.com/p/jvmtop PID MAIN-CLASS HPCUR HPMAX NHCUR NHMAX CPU GC VM USERNAME #T DL 3370 rapperSimpleApp 165m 455m 109m 176m 0.12% 0.00% S6U37 web 21 11272 ver.resin.Resin [ERROR: Could not attach to VM] 27338 WatchdogManager 11m 28m 23m 130m 0.00% 0.00% S6U37 web 31 19187 m.jvmtop.JvmTop 20m 3544m 13m 130m 0.93% 0.47% S6U37 web 20 16733 artup.Bootstrap 159m 455m 166m 304m 0.12% 0.00% S6U37 web 46
Tool #2 – jvmmonitor
Another alternative is to use jvmmonitor. JVM Monitor is a Java profiler integrated with Eclipse to monitor CPU, threads and memory usage of Java applications. You can either use it to automatically find running JVMs on the localhost or it can connect to remote JVMs using a [email protected]
Tool #3 – visualvm
visualvm is probably “the tool” to reach for when debugging issues with the JVM. Its feature set is pretty deep and you can get a very in depth look at the innards.
Profile application performance or analyze memory allocation:
Take and display thread dumps:
References
Method 3
Consider jstack.
Not quite a match for strace, more of a pstack-analog, but will at least give you a picture of a snapshot in time. Could string’em together to get a crude trace if you had to.
See also the suggestions at this SO article: https://stackoverflow.com/questions/1025681/call-trace-in-java
Method 4
If you are using RHEL OpenJDK (or similiar, the point is that it is not Oracle’s JDK), you may use SystemTap for that.
Some probes are enabled by using java command line options -XX:+DTraceMethodProbes, -XX:+DTraceAllocProbes, -XX:+DTraceMonitorProbes. Note that enabling these probes will significally affect program performance.
Here is example SystemTap Script:
#!/usr/bin/stap
probe hotspot.class_loaded {
printf("%12s [???] %sn", name, class);
}
probe hotspot.method_entry,
hotspot.method_return {
printf("%12s [%3d] %s.%sn", name, thread_id, class, method);
}
probe hotspot.thread_start,
hotspot.thread_stop {
printf("%12s [%3d] %sn", name, id, thread_name);
}
probe hotspot.monitor_contended_enter,
hotspot.monitor_contended_exit {
printf("%12s [%3d] %sn", name, thread_id, class);
}
You can also use jstack() to get Java stack of the process, but it will only work if you start SystemTap before JVM.
Note that SystemTap will trace every method. It is also not able to get method’s arguments. Another option is to use JVM own capabilities of tracing which is called JVMTI. One of the most famous JVMTI implementations is BTrace.
Method 5
You are advised to try out Jackplay, which is a JVM tracing tool allowing you to trace method entry and exits without code change or redeployment.
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0