Tuesday, December 10

J2EE Engine hangs because of unsufficient system threads

Symptom
J2EE engine does not respond to any requests, the system is frozen and the cluster node (if allocated) should be restarted to solve the issue temporarily.
Other Terms
hang, freeze, not responding, thread, performance, j2ee
Reason and Prerequisites
Very often for SAP Net Weaver systems running J2EE Engine with Enterprise Portal there are recommendations , that a number of 20 or 40 for the Application Threads are sufficient. This is true for the application threads, but too often these recommendations are understood as concerning the number of the System Threads, even for dispatcher nodes.

Please note, that System Threads and Application Threads pools are two separate thread pools with absolutely different functionality and mixing them, i.e setting any not recommended by SAP J2EE Engine support value to the System Threads number, could be fatal. Please be very careful regarding exactly which number is changed.

Solution
When J2EE Engine stops responding, please do the following steps:

1. Make (at least one) thread dump of the cluster node that hangs (or of all the cluster nodes, including dispatcher ones if the node is not allocated).
procedure to create thread dump is explained below in the post.

2. If the thread dump(s) show that there are no free System Threads, i.e. if there are no threads with the stack trace:
"SAPEngine_System_Thread[impl:5]_#" prio=1 tid=0x09aa4cb0 nid=0x453f in Object.wait() [0x58a09000..0x58a09228]
at java.lang.Object.wait(Native Method)
- waiting on <0x6b4f0060> (a com.sap.engine.lib.util.WaitQueue)
at java.lang.Object.wait(Object.java:429)
at com.sap.engine.lib.util.WaitQueue.dequeue(WaitQueue.java:238)
- locked <0x6b4f0060> (a com.sap.engine.lib.util.WaitQueue)
at com.sap.engine.core.thread.impl5.SingleThread.run(SingleThread.java:127)

please proceed to step 3.

3. Check the values for the System Threads in the visual configtool:

- start the configtool script in /configtool;
- in the left GUI pane expand Cluster Data -> Global server configuration -> managers -> ThreadManager;
Check the values of the properties MinThreadCount, InitialThreadCount and MaxThreadCount.

4. Check the values of the same properties for the dispatcher nodes under Cluster Data -> Global dispatcher configuration -> managers -> ThreadManager.

The values of all the properties mentioned must be at least the default ones. Pressing the "Restore to default" button (after having in mind the current values) would give you the defaults.
In case the current values are lower than the default, please keep the default values (Select from the menu File -> Apply and confirm all popups and then restart the cluster in order for the changes to take effect).
In case the current values are higher than the default values, and still there are no free system threads, please report an SAP message in BC-JAS-COR component, attaching to it the thread dumps you have made in step 1. In this case do not apply the changes.
In case of lower than the default values, if resetting them to the defaults does not solve the issue, please report an SAP message in BC-JAS-COR component, attaching to it the thread dumps you have made in step 1. Please keep the System Thread pool configured with the default values meanwhile.
 
 

Create a thread dump from all Java threads running on a Java node

You are using the J2EE Engine and want to create a thread dump from all Java threads running on a Java node.
Reason and Prerequisites
For platforms using a JDK from SUN:
  • Please check that the Java node is not in "console" mode.
Open the /usr/sap// /j2ee/cluster/instance.properties file and check that the node does not have a "ID.Console=yes" attribute.
If the console mode is active, the output of the node is not redirected and you will not find a std_.out file.
You can deactivate the console mode by clearing the "Show Console" checkbox of the node in the ConfigTool.

Solution
Use the JCMon tool to initiate a thread dump.
1. Start the JCMon program with the profile of the instance.

The JCMon binary is located in the /usr/sap///j2ee/os_libs directory or in the /usr/sap//SYS/exe/run directory.
    2. Enter "20" to go to the "Local Administration Menue".
    3. Choose the "Dump stacktrace" menu item by entering its number.
    4. Select the Java node from the list of processes above and enter the value from the "Idx" column.
    5. Confirm the command with a "y" if the selected node is correct.
After completion, you will find the thread dump:
  • in the /usr/sap///work/std_.out file (for JDK's from SUN and HP, according to the patch level of jlaunch also for Classic VM on IBM i)
  • for JDK's from IBM: javacore..txt in directory
    • /usr/sap///j2ee/cluster/dispatcher
    • /usr/sap///j2ee/cluster/server#
    • /usr/sap///SDM/program
  • according to the patchlevel of jlaunch in a spoolfile QDMPJVM of the workprocess user (for Classic VM on IBM i)
Alternative procedure on Unix
    1. Start the JCMon program with the profile of the instance.
    2. Enter "20" to go to the "Local Administration Menue".
    3. Note the process ID (PID) from the Java node.
    4. Issue a "kill -3 " command from the shell.
Alternative procedure on Windows
This procedure is not applicable for double stack installations (R/3 and J2EE Engine running in the same instance).
    1. Open the SAP Management Console.
    2. Navigate to the Process table of the Java instance.
    3. Right-click on the Java node and choose "Dump Stack Trace" from the context menu.
Alternative procedure on iSeries
    1. Log on as OFR.
    2. To check the java processes in your subsystem, enter command
    WRKACTJOB SBS(R3_)

    Jobs labeled "PGM-JLAUNCH" in column "Function" are java processes. In earlier support packages, those processes had the job name "JLAUNCH", with higher SP levels, they have job names "SDM", "DISPATCHER", "SERVER0" etc. Please substitute in the following lines accordingly.
    3. Push on function key F11 two times to get the user and the process number of the JVMs (job name ). These values are needed to create a java thread dump.
    4. To create the java thread dump, do one of the following:
      a) Enter command
      DMPJVMSAP JOB() USER() NUMBER()

      DMPJVMSAP creates a spool file QDMPJVM. In this spool file not only all java threads dumps are to be found but also the heap stack and the class loader information. DMPJVM does not stop a running JVM. This means that you can create a java thread dump at any time without stopping a JVM.

      Note that DMPJVMSAP only works with Classic VM. If your system is configured to run with IT4J this method cannot be used. Choose the commonly described method using jcmon for IT4J. The one below is only applicable to systms using Classic VM.
      b) Go to menu R3PROBLEM and choose option 16 'Trigger Java VM Dump' also specifying the job data. Location of the dump depends on the JVM you are using and on the patch level of your system (see generic description above).
    5. To look at the spool file created by DMPJVMSAP, enter command WRKSPLF. Use the function key F11 to collect the following information: file number, job name and job number.
    6. Create a text file containg the needed java thread dump information to make the information accessable also to users of other operation systems.
    (Attention: Perform the next steps on your Windows PC.)
    For this you may use the batch file getJavaDumpFile.bat attached in this SAP Note. Unattach the file getJavaDumpFile.TXT into a temporary directory on your Windows PC and rename it to getJavaDumpFile.bat. Run the batch file and you will get a file called javadump.txt containing the java thread dumps:
    getJavaDumpFile.bat


    (for example: getJavaDumpFile.bat as0096 SIDOFR secret QDMPJVM 2
    QPADEV0004 SIDOFR 060594)

Posted by Mohammed Bohra at 1:09 AM 0 comments
Subscribe to: Posts (Atom)