[tmax] 쓰레드 관련 로그 보는 방법

===========================================================================
1. 쓰레드가 정상적으로 동작하고 있을 경우
 - 보는 바와 같이 같은 패턴으로 아무 이상없이 동작하고 있다.
- wa 1, wa 2 명령을 통해 컨테이너 별로 쓰레드 정보를 볼수 있다.
=========================================================================== 

[tmax@BL_WEB BL_WEB]$ pwd /web/tmax/jeus5/logs/BL_WEB [tmax@BL_WEB BL_WEB]$ wa 1     => wa 1이면 1번 컨테이너, wa 2 면 2번 컨테이너 -- Welcome to JEUS Web Container(vJEUS 5.0 (fix #23)) Admin -- $1 BL_WEB>ti -- Thread State [webtob1_ssl_portal-hth0(localhost_9900)] -- [webtob1_ssl_portal-hth0(localhost:9900)-w00][waiting, wt=20896468 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w01][waiting, wt=20896467 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w02][waiting, wt=20896466 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w03][waiting, wt=20896465 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w04][waiting, wt=20896465 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w05][waiting, wt=20896463 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w06][waiting, wt=20896462 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w07][waiting, wt=20896461 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w08][waiting, wt=20896442 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w09][waiting, wt=20896441 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w10][waiting, wt=20896441 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w11][waiting, wt=20896440 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w12][waiting, wt=20896439 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w13][waiting, wt=20896438 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w14][waiting, wt=20896437 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w15][waiting, wt=20896436 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w16][waiting, wt=20896436 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w17][waiting, wt=20896435 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w18][waiting, wt=20896433 ms] [webtob1_ssl_portal-hth0(localhost:9900)-w19][waiting, wt=20896433 ms] [total : 20    active : 0    idle : 20    blocked : 0    reconnecting : 0] -- Thread State [webtob1_portal-hth0(localhost_9900)] -- [webtob1_portal-hth0(localhost:9900)-w00][waiting, wt=5873 ms] [webtob1_portal-hth0(localhost:9900)-w01][waiting, wt=3977 ms] [webtob1_portal-hth0(localhost:9900)-w02][waiting, wt=2378 ms] [webtob1_portal-hth0(localhost:9900)-w03][waiting, wt=7159 ms] [webtob1_portal-hth0(localhost:9900)-w04][waiting, wt=4682 ms] [webtob1_portal-hth0(localhost:9900)-w05][waiting, wt=5725 ms] [webtob1_portal-hth0(localhost:9900)-w06][waiting, wt=360 ms] [webtob1_portal-hth0(localhost:9900)-w07][waiting, wt=2364 ms] [webtob1_portal-hth0(localhost:9900)-w08][waiting, wt=8262 ms] [webtob1_portal-hth0(localhost:9900)-w09][waiting, wt=3535 ms] [webtob1_portal-hth0(localhost:9900)-w10][waiting, wt=2665 ms] [webtob1_portal-hth0(localhost:9900)-w11][waiting, wt=8073 ms] [webtob1_portal-hth0(localhost:9900)-w12][waiting, wt=6773 ms] [webtob1_portal-hth0(localhost:9900)-w13][waiting, wt=6539 ms] [webtob1_portal-hth0(localhost:9900)-w14][waiting, wt=6118 ms] [webtob1_portal-hth0(localhost:9900)-w15][waiting, wt=536 ms] [webtob1_portal-hth0(localhost:9900)-w16][waiting, wt=4107 ms] [webtob1_portal-hth0(localhost:9900)-w17][waiting, wt=3946 ms] [webtob1_portal-hth0(localhost:9900)-w23][waiting, wt=2376 ms] [webtob1_portal-hth0(localhost:9900)-w24][waiting, wt=7880 ms] [total : 20    active : 0    idle : 20    blocked : 0    reconnecting : 0]

==========================================================================
2. 장애가 발생했을경우 (사이트가 느려졌을 때 확인한 로그이다.)
- 원격지원중에 jeus에서 exeception오류가 발생하여서 그때 확인한 로그이다.
- 로그를 보면 위에꺼 와는 다르게 waiting 뿐만아니고 active라는 부분이 많이 생기고 있다. 이부분에서 부하가 걸리고
있을 알수 있다. rt(Response Time 또는 Run Time)가 어떤건 200초가 걸리는 부분도 있다. ㅡㅡ; 
장애 발생시 해당 uri 부분도 나오기 때문에 그 페이지부분을 또한 점검해보면 될 것 같다. 
아래 로그엔 나오지 않지만 아까 블로그등... 문제가 있는 부분이 있었다.
========================================================================== 

[tmax@BL_WEB BL_WEB]$ pwd /web/tmax/jeus5/logs/BL_WEB [tmax@BL_WEB BL_WEB]$ wa 1     => wa 1이면 1번 컨테이너, wa 2 면 2번 컨테이너 -- Welcome to JEUS Web Container(vJEUS 5.0 (fix #23)) Admin -- $1 BL_WEB>ti -- Thread State [webtob1_portal-hth0(localhost_9900)] -- [webtob1_portal-hth0(localhost:9900)-w00][waiting, wt=7893 ms] [webtob1_portal-hth0(localhost:9900)-w01][active , rt=19063 ms, uri=/site/executive/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w02][active , rt=7309 ms, uri=/site/executive/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w03][active , rt=25923 ms, uri=/site/executive/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w04][active , rt=230500 ms, uri=/site/tour/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w05][active , rt=34988 ms, uri=/site/executive/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w06][waiting, wt=2047 ms] [webtob1_portal-hth0(localhost:9900)-w07][waiting, wt=2138 ms] [webtob1_portal-hth0(localhost:9900)-w08][waiting, wt=14103 ms] [webtob1_portal-hth0(localhost:9900)-w09][waiting, wt=427 ms] [webtob1_portal-hth0(localhost:9900)-w10][waiting, wt=4993 ms] [webtob1_portal-hth0(localhost:9900)-w11][active , rt=11077 ms, uri=/site/executive/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w12][active , rt=209452 ms, uri=/site/tour/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w13][waiting, wt=10956 ms] [webtob1_portal-hth0(localhost:9900)-w14][waiting, wt=6914 ms] [webtob1_portal-hth0(localhost:9900)-w15][waiting, wt=2055 ms] [webtob1_portal-hth0(localhost:9900)-w16][active , rt=301454 ms, uri=/servlet/download] [webtob1_portal-hth0(localhost:9900)-w17][active , rt=272282 ms, uri=/site/tour/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w23][active , rt=3128 ms, uri=/site/executive/sub02/sub02_03.jsp] [webtob1_portal-hth0(localhost:9900)-w24][active , rt=251444 ms, uri=/site/tour/sub02/sub02_03.jsp]

=========================================================================
3. 쓰레드 덤프 받는 방법
- 쓰레드 덤프를 뜨기 위해서 pid정보를 가지고 kill -3 [PID] 명령을 통해서 할 수 있다.
- PID를 찾는 방법은 두가지가 있다.  
ps -ef |grep java |grep Xboot 명령을 사용하여 해당 컨테이너에 해당하는 PID정보를 아는 방법이 있고
제우스에서 만든 ja > pidlist를 통해서 알아내는 방법이 있다. 만약에 장애가 발생해서 제우스에서 하는 명령이 안먹을
경우는 첫번째 방법(ps -ef |grep java |grep Xboot )으로 PID를 찾아내서 쓰레드 덤프를 뜨면된다.
이때 덤프는 10초간격으로 3번 정도 뜨는 것이 좋다.
========================================================================= 

[tmax@BL_WEB BL_WEB]$ ps -ef |grep java |grep Xboot tmax     11561 11560  0 11:10 ?        00:00:26 /usr/jdk1.5.0_15/bin/java -server -Xmx512m -Xbootclasspath/p:/web/tmax/jeus5/lib/system/extension.jar:/web/tmax/jeus5/lib/system/extensionReflection.jar -classpath /web/tmax/jeus5/lib/system/bootstrap.jar -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.library.path=/web/tmax/jeus5/lib/system:/web/tmax/webtob/lib:/web/tmax/webtob/lib: -Djava.naming.factory.initial=jeus.jndi.JNSContextFactory -Djava.naming.factory.url.pkgs=jeus.jndi.jns.url -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/web/tmax/jeus5/bin/logging.properties -Djeus.home=/web/tmax/jeus5 -Djeus.baseport=9736 -Djeus.jvm.version=hotspot -Djeus.tm.checkReg=true -Djeus.tool.webadmin.locale.language=ko -Djeus.net.reuseAddress=true -Djeus.properties.replicate=jeus,sun.rmi,java.util,java.net jeus.server.JeusBootstrapper -xml -Uadministrator -Pyygweb1 tmax     11607 11561 99 11:10 ?        06:23:59 /usr/jdk1.5.0_15/jre/bin/java -server -Dcontainer1 -Xbootclasspath/p:/web/tmax/jeus5/lib/system/extension.jar -classpath /web/tmax/jeus5/lib/system/bootstrap.jar -Djava.security.policy=/web/tmax/jeus5/config/security/policy -Xms512m -Xmx1024m -XX:MaxPermSize=512m -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djava.library.path=/web/tmax/jeus5/lib/system:/web/tmax/webtob/lib:/web/tmax/webtob/lib: -Djava.endorsed.dirs=/usr/jdk1.5.0_15/jre/lib/endorsed -Djeus.properties.replicate=jeus,sun.rmi,java.util,java.net -Djeus.net.reuseAddress=true -Djeus.jvm.version=hotspot -Djava.util.logging.config.file=/web/tmax/jeus5/bin/logging.properties -Dsun.rmi.dgc.server.gcInterval=3600000 -Djeus.home=/web/tmax/jeus5 -Djava.net.preferIPv4Stack=true -Djeus.baseport=9736 -Djeus.ejb.operationTimeout=300000 -Djeus.tm.checkReg=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Djeus.tool.webadmin.locale.language=ko -Djava.naming.factory.initial=jeus.jndi.JNSContextFactory -Djava.naming.factory.url.pkgs=jeus.jndi.jns.url jeus.server.enginecontainer.EngineContainerBootstrapper BL_WEB_container1 BL_WEB_servlet_engine1 tmax     11929 11561  0 11:10 ?        00:00:36 /usr/jdk1.5.0_15/jre/bin/java -server -Dcontainer2 -Xbootclasspath/p:/web/tmax/jeus5/lib/system/extension.jar -classpath /web/tmax/jeus5/lib/system/bootstrap.jar -Djava.security.policy=/web/tmax/jeus5/config/security/policy -Xms512m -Xmx1024m -XX:MaxPermSize=512m -Djavax.xml.parsers.DocumentBuilderFactory=com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderFactoryImpl -Djava.library.path=/web/tmax/jeus5/lib/system:/web/tmax/webtob/lib:/web/tmax/webtob/lib: -Djava.endorsed.dirs=/usr/jdk1.5.0_15/jre/lib/endorsed -Djeus.properties.replicate=jeus,sun.rmi,java.util,java.net -Djeus.net.reuseAddress=true -Djeus.jvm.version=hotspot -Djava.util.logging.config.file=/web/tmax/jeus5/bin/logging.properties -Dsun.rmi.dgc.server.gcInterval=3600000 -Djeus.home=/web/tmax/jeus5 -Djava.net.preferIPv4Stack=true -Djeus.baseport=9736 -Djeus.ejb.operationTimeout=300000 -Djeus.tm.checkReg=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Djeus.tool.webadmin.locale.language=ko -Djava.naming.factory.initial=jeus.jndi.JNSContextFactory -Djava.naming.factory.url.pkgs=jeus.jndi.jns.url jeus.server.enginecontainer.EngineContainerBootstrapper BL_WEB_container2 BL_WEB_servlet_engine2 [tmax@BL_WEB BL_WEB]$ ja JEUS 5.0 (fix #23) Jeus Manager Controller BL_WEB>pidlist node or container : BL_WEB, pid : 11561 node or container : BL_WEB_container2, pid : 11929 node or container : BL_WEB_container1, pid : 11607 BL_WEB>exit [tmax@BL_WEB BL_WEB]$ kill -3  11607 (<== 덤프실행 10초간격으로 3회)