1. Use top to view the system resources used by the running process

Find programs that use too much CPU

2. Use the PS - MP, PID - O thread, TID and time commands to check the thread status of a process. Time represents the running time of the thread,

3. Use the calculator to convert the hexadecimal thread number to hexadecimal

The TID number is converted from hexadecimal to hexadecimal, and then go to the file generated below to query which thread has the problem

4. Export all thread information of the specified process pid - jstack pid > XXX log

jstack 26968 >thread.log

5. Analysis

1

Thread status introduction

• Deadlock, deadlock (focus): it generally refers to the situation that multiple threads call and enter the mutual resource occupation, resulting in waiting and unable to release.

• In execution, Runnable: generally refers to that the thread is in the execution state. The thread occupies resources, is processing a request, may be transferring SQL to the database for execution, may be operating on a file, and may carry out data type conversion.

• Waiting on condition: waiting for a resource, or waiting for a condition to occur. The specific reasons should be analyzed in combination with stacktrace.
    1. If the stack information is clearly application code, it proves that the thread is waiting for resources. Generally, when a large number of resources are read and the resource lock is adopted, the thread enters the waiting state and waits for the reading of resources.
  Or, waiting for the execution of other threads.
    2. If it is found that a large number of threads are waiting on condition, from the thread stack, they are waiting for network reading and writing, which may be a sign of network bottleneck. The thread cannot execute because of network congestion.
      2.1 in one case, the network is very busy, almost consuming all the bandwidth, and there is still a large amount of data waiting for the network to read and write;
      2.2 another situation may be that the network is idle, but the packets cannot arrive normally due to routing and other problems.
    3. Another common case of Wait on condition is that the thread is in sleep. When the waiting time for sleep is up, it will be awakened.

• Waiting on monitor entry

• Object waiting, object Wait() or TIMED_WAITING
    Waiting for monitor entry and in object wait():
   Monitor(Monitor's in-depth understanding portal )It is the main means to realize mutual exclusion and cooperation between threads in Java. It can be regarded as the lock of object or Class. Each object has one and only one monitor.
    as can be seen from the following figure, each Monitor can only be owned by one thread at a certain time. This thread is "Active Thread", while other threads are "Waiting Thread", waiting in two queues "Entry Set" and "Wait Set" respectively.
    the thread state waiting in "Entry Set" is "Waiting for monitor entry", while the thread state waiting in "Wait Set" is "in Object.wait()"

  

  Java Monitor

• Suspended

• Blocked: refers to a thread that waits for a long time but fails to obtain the required resources during the execution of the current thread, which is identified as blocked by the thread manager of the container. It can be understood as a thread waiting for resource timeout.

• Stop, Parked

  Example analysis of stack trace

  "consumer_redirectUrl_topic_jmq206_1546013217302" daemon prio=10 tid=0x00007f1bf03f6800 nid=0x693e waiting on condition [0x00007f1b38388000]
     java.lang.Thread.State: TIMED_WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for  <0x00000000f76e21a0> (a java.util.concurrent.CountDownLatch$Sync)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326)
      at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:282)
      at com.jd.jmq.common.network.netty.ResponseFuture.await(ResponseFuture.java:133)
      at com.jd.jmq.common.network.netty.NettyTransport.sync(NettyTransport.java:241)
      at com.jd.jmq.common.network.netty.failover.FailoverNettyClient.sync(FailoverNettyClient.java:94)
      at com.jd.jmq.client.consumer.GroupConsumer.pull(GroupConsumer.java:246)
      at com.jd.jmq.client.consumer.GroupConsumer$QueueConsumer.run(GroupConsumer.java:445)
      at java.lang.Thread.run(Thread.java:745)
  
     Locked ownable synchronizers:
      - None
  

  • Thread Name: Consumer_ redirectUrl_ topic_ jmq206_ one trillion and five hundred and forty-six billion thirteen million two hundred and seventeen thousand three hundred and two
  • Thread priority: prio=10
  • identifier of java thread: tid=0x00007f1bf03f6800
  • identifier of native thread: nid=0x693e
  • Status of thread: waiting on condition [0x00007f1b388000]
    java.lang.Thread.State: TIMED_WAITING (parking)
  • Starting address of thread stack: [0x00007f1b38388000]

6. Finally check the problem code