포스팅 목차
# 로그 파일들의 위치 알아보기
다양한 로그 파일과 명령 출력 이외에도 journalctl을 사용하여 노드 관점에서 로그를 볼 수 있다. 우리는 로그 파일의 공통적인 위치를 보고, 컨테이너 로그를 보는 명령을 볼 것이다. 다른 컨테이너의 로그를 pod에 적재하는데 전용으로 사용되는 sidecar container의 사용과 같은 다른 로깅 옵션이 있다. Kubernet에서 전체 클러스터 로깅을 아직 사용할 수 없다. 따라서 kubernetes 같이 CNCF 프로젝트 다른 멤버인 fluentd 같은 외부 software를 사용된다.
다음의 로그 파일과 웹 사이트를 간단히 살펴보십시오. 서버 프로세스가 노드 수준에서 컨테이너안에 실행으로 이동함에 따라 로깅도 이동한다.
1. journalctl 명령 사용 (kubernetes cluster가 systemd를 사용할때만 가능)
# -------------------------------------
# journalctl 명령 사용 (kubernetes cluster가 systemd를 사용할때만 가능)
# -------------------------------------
# journalctl 명령 : systemd 로그가 journal로 관리되는데 이런 journal들을 조회할 때 쓰임.
# journalctl <tab x 2> 참조
# journalctl 옵션
# -u : 특정 unit 로그 확인
# -f : 실시간으로 로그 확인
# https://wiki.archlinux.org/index.php/Systemd/Journal
ps0107@k8smaster1:~$ sudo journalctl -u kubelet | tail -5
Feb 05 02:10:19 k8smaster1 kubelet[5279]: I0205 02:10:19.268859 5279 reconciler.go:177] operationExecutor.UnmountVolume started for volume "default-token-76w5h" (UniqueName: "kubernetes.io/secret/ccc2b1fb-ffd4-4078-9bd2-b63e662e2ef4-default-token-76w5h") pod "ccc2b1fb-ffd4-4078-9bd2-b63e662e2ef4" (UID: "ccc2b1fb-ffd4-4078-9bd2-b63e662e2ef4")
Feb 05 02:10:19 k8smaster1 kubelet[5279]: I0205 02:10:19.282026 5279 operation_generator.go:860] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24257df3-fa70-47ba-b02f-8bbf5da263dd-default-token-76w5h" (OuterVolumeSpecName: "default-token-76w5h") pod "24257df3-fa70-47ba-b02f-8bbf5da263dd" (UID: "24257df3-fa70-47ba-b02f-8bbf5da263dd"). InnerVolumeSpecName "default-token-76w5h". PluginName "kubernetes.io/secret", VolumeGidValue ""
....2. 노드 pod/container 로그
# /var/log/containers/ : 여러 container 로그들
# /var/log/pods/ : 현재 pod들을 위한 로그 파일들
# -------------------------------------
# 노드 pod/container 로그
# -------------------------------------
# /var/log/containers/ : 여러 container 로그들
# /var/log/pods/ : 현재 pod들을 위한 로그 파일들
# 주요 쿠버네티스 프로세스들은 컨테이너로 실행된다. pod 또는 container 관점으로 볼수 있다.
# 해당 로그가 어디에 위치하고 있는지 보면 /var/log/containers/ 의 컨테이너 로그는 /var/log/pods/ 안에 로그에 심볼릭링크가 걸려있다.
ps0107@k8smaster1:~$ sudo find / -name "*apiserver*log"
/var/log/containers/kube-apiserver-k8smaster1_kube-system_kube-apiserver-20ee76df3202cd45c43568d04f2c6350c33cf02e082aff03331863c75b9c2e9f.log
ps0107@k8smaster1:~$ sudo tail /var/log/containers/kube-apiserver-k8smaster1_kube-system_kube-apiserver-20ee76df3202cd45c43568d04f2c6350c33cf02e082aff03331863c75b9c2e9f.log
{"log":"Trace[758496088]: [1.54222154s] [1.541983765s] Transaction committed\n","stream":"stderr","time":"2020-01-30T05:16:34.841260969Z"}
{"log":"I0130 05:16:34.824687 1 trace.go:81] Trace[1004905485]: \"Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\" (started: 2020-01-30 05:16:33.282084781 +0000 UTC m=+161232.462995704) (total time: 1.542580238s):\n","stream":"stderr","time":"2020-01-30T05:16:34.841266335Z"}
{"log":"Trace[1004905485]: [1.542484521s] [1.542396457s] Object stored in database\n","stream":"stderr","time":"2020-01-30T05:16:34.841281926Z"}
{"log":"I0130 12:43:57.718791 1 log.go:172] http: TLS handshake error from 163.172.16.99:2332: tls: first record does not look like a TLS handshake\n","stream":"stderr","time":"2020-01-30T12:43:57.718994903Z"}
{"log":"I0202 14:09:21.314630 1 controller.go:606] quota admission added evaluator for: jobs.batch\n","stream":"stderr","time":"2020-02-02T14:09:21.314861538Z"}
{"log":"I0202 14:56:06.400851 1 controller.go:606] quota admission added evaluator for: cronjobs.batch\n","stream":"stderr","time":"2020-02-02T14:56:06.401042597Z"}
{"log":"I0203 01:17:35.531438 1 controller.go:606] quota admission added evaluator for: replicasets.extensions\n","stream":"stderr","time":"2020-02-03T01:17:35.531608401Z"}
{"log":"I0203 14:48:02.892069 1 log.go:172] http: TLS handshake error from 198.108.67.48:44142: read tcp 10.146.0.2:6443-\u003e198.108.67.48:44142: read: connection reset by peer\n","stream":"stderr","time":"2020-02-03T14:48:02.892238145Z"}
{"log":"I0204 02:03:55.599499 1 log.go:172] http: TLS handshake error from 198.108.67.48:19574: read tcp 10.146.0.2:6443-\u003e198.108.67.48:19574: read: connection reset by peer\n","stream":"stderr","time":"2020-02-04T02:03:55.599677119Z"}
{"log":"I0204 08:28:48.956703 1 controller.go:606] quota admission added evaluator for: ingresses.extensions\n","stream":"stderr","time":"2020-02-04T08:28:48.956873937Z"}
ps0107@k8smaster1:/var/log/containers$ ls
calico-node-wvxxm_kube-system_calico-node-d2879a40b389a1ef99359af7356ea1efcea497352d96f1e665729f783fdd92b7.log
calico-node-wvxxm_kube-system_install-cni-8e9c02311deebb50195a5f4333b2ba89646cc5799b5542d95f23612018468636.log
coredns-5c98db65d4-97gng_kube-system_coredns-6d822d735d3bb02dd4171294438f9bed5a8ed44821bbadb843cb9422a465c6b4.log
coredns-5c98db65d4-fvz2k_kube-system_coredns-5b255b61a93ed2fe9d3209e8135b3a54824af73690462af0c08a6f4ed55c7148.log
etcd-k8smaster1_kube-system_etcd-3147aa1dc7e6bcf0677e875000730db51b45c1f4e25458077fd921d333829122.log
kube-apiserver-k8smaster1_kube-system_kube-apiserver-20ee76df3202cd45c43568d04f2c6350c33cf02e082aff03331863c75b9c2e9f.log
kube-controller-manager-k8smaster1_kube-system_kube-controller-manager-156da4c1dc59e7144455a2014746da903421de5d7b2120874c069800cbfac717.log kube-proxy-t62q4_kube-system_kube-proxy-a6f991ba816b7927cb26cc6016fa45f98d87a0b630d7c897315edc4ed5038df9.log
kube-scheduler-k8smaster1_kube-system_kube-scheduler-7649252338f03b870c74e33731a2f8b162db7b981488d2ccadc1072903ede7d8.log
traefik-ingress-controller-jdvck_kube-system_traefik-ingress-lb-e757ed51250e22dc8977f9ed7b0d26352c57fc4961232e955e80bef14af85ab6.log
ps0107@k8smaster1:/var/log/pods$ ll
total 44
drwxr-xr-x 11 root root 4096 Feb 5 02:10 ./
drwxrwxr-x 10 root syslog 4096 Feb 4 06:25 ../
drwxr-xr-x 4 root root 4096 Jan 28 08:33 kube-system_calico-node-wvxxm_9cff8584-8fa3-493b-a01e-0567c6cd11f2/
drwxr-xr-x 3 root root 4096 Jan 28 08:33 kube-system_coredns-5c98db65d4-97gng_9e2cfd14-4a08-4703-a7e8-d4c85f7878a1/
drwxr-xr-x 3 root root 4096 Jan 28 08:33 kube-system_coredns-5c98db65d4-fvz2k_a4208d56-10d2-4958-b345-01603d5aa57b/
drwxr-xr-x 3 root root 4096 Dec 5 16:19 kube-system_etcd-k8smaster1_92c04c96b41833d6edb6666054fd741f/
drwxr-xr-x 3 root root 4096 Jan 28 08:29 kube-system_kube-apiserver-k8smaster1_214ef7c9b4c2389fde69afa9327c7a33/
drwxr-xr-x 3 root root 4096 Jan 28 08:29 kube-system_kube-controller-manager-k8smaster1_645e7a8519364c082c136bba3c26849b/
drwxr-xr-x 3 root root 4096 Jan 28 08:29 kube-system_kube-proxy-t62q4_3b2fe619-eca0-4356-8ff3-7ef781935d19/
drwxr-xr-x 3 root root 4096 Jan 28 08:29 kube-system_kube-scheduler-k8smaster1_ecae9d12d3610192347be3d1aa5aa552/
drwxr-xr-x 3 root root 4096 Feb 4 08:47 kube-system_traefik-ingress-controller-jdvck_c79ea85c-07b0-4277-939b-9e27d9422159/
# 위에서 조회한 /var/log/containers/ 안에 있는 로그와 같다. /var/log/pods/ 안에 있는 해당 로그가 symbolic link 걸려 있다.
ps0107@k8smaster1:/var/log/pods/kube-system_kube-apiserver-k8smaster1_214ef7c9b4c2389fde69afa9327c7a33/kube-apiserver$ sudo tail 0.log
{"log":"Trace[758496088]: [1.54222154s] [1.541983765s] Transaction committed\n","stream":"stderr","time":"2020-01-30T05:16:34.841260969Z"}
{"log":"I0130 05:16:34.824687 1 trace.go:81] Trace[1004905485]: \"Update /api/v1/namespaces/kube-system/endpoints/kube-controller-manager\" (started: 2020-01-30 05:16:33.282084781 +0000 UTC m=+161232.462995704) (total time: 1.542580238s):\n","stream":"stderr","time":"2020-01-30T05:16:34.841266335Z"}
{"log":"Trace[1004905485]: [1.542484521s] [1.542396457s] Object stored in database\n","stream":"stderr","time":"2020-01-30T05:16:34.841281926Z"}
{"log":"I0130 12:43:57.718791 1 log.go:172] http: TLS handshake error from 163.172.16.99:2332: tls: first record does not look like a TLS handshake\n","stream":"stderr","time":"2020-01-30T12:43:57.718994903Z"}
{"log":"I0202 14:09:21.314630 1 controller.go:606] quota admission added evaluator for: jobs.batch\n","stream":"stderr","time":"2020-02-02T14:09:21.314861538Z"}
{"log":"I0202 14:56:06.400851 1 controller.go:606] quota admission added evaluator for: cronjobs.batch\n","stream":"stderr","time":"2020-02-02T14:56:06.401042597Z"}
{"log":"I0203 01:17:35.531438 1 controller.go:606] quota admission added evaluator for: replicasets.extensions\n","stream":"stderr","time":"2020-02-03T01:17:35.531608401Z"}
{"log":"I0203 14:48:02.892069 1 log.go:172] http: TLS handshake error from 198.108.67.48:44142: read tcp 10.146.0.2:6443-\u003e198.108.67.48:44142: read: connection reset by peer\n","stream":"stderr","time":"2020-02-03T14:48:02.892238145Z"}
{"log":"I0204 02:03:55.599499 1 log.go:172] http: TLS handshake error from 198.108.67.48:19574: read tcp 10.146.0.2:6443-\u003e198.108.67.48:19574: read: connection reset by peer\n","stream":"stderr","time":"2020-02-04T02:03:55.599677119Z"}
{"log":"I0204 08:28:48.956703 1 controller.go:606] quota admission added evaluator for: ingresses.extensions\n","stream":"stderr","time":"2020-02-04T08:28:48.956873937Z"}
# kube-dns, kube-proxy, kube-flannel 등도 이런 식으로 조회해 볼수 있다.3. kubernetes cluster가 systemd를 사용하지 않을때
######### master node에서 확인 ########
# /var/log/kube-apiserver.log
# => API 받는 응답을 볼수 있다.
# /var/log/kube-scheduler.log
# => 스케쥴링 결정을 만드는 응답을 볼수 있다.
# /var/log/kube-controller-manager.log
# => controller 관련 로그를 볼수 있다.
######### worker node에서 확인 ########
# /var/log/kubelet.log
# => 노드에 실행 중인 컨테이너들의 응답을 볼수 있다
# /var/log/kube-proxy.log
# => service load balancing 관련 로그들
4. 그밖에 도움이 되는 웹페이지
https://kubernetes.io/docs/tasks/debug-application-cluster/\debug-service/
https://kubernetes.io/docs/tasks/debug-application-cluster/\determine-reason-pod-failure/
# 로그들 출력 보기
- 컨테이너 standard out 로그는 kubectl logs 명령을 통해서 볼수 있다. 해당 컨테이너가 삭제되면 로그들도 삭제 된다.
# 컨테이너 standard out 로그는 kubectl logs 명령을 통해서 볼수 있다. 해당 컨테이너가 삭제되면 로그들도 삭제 된다.
# 현재 pod들 목록 보기
ps0107@k8smaster1:~$ kubectl get po --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-wvxxm 2/2 Running 0 7d22h
kube-system calico-node-zxkxk 2/2 Running 2 7d22h
kube-system coredns-5c98db65d4-97gng 1/1 Running 0 7d22h
kube-system coredns-5c98db65d4-fvz2k 1/1 Running 0 7d22h
kube-system etcd-k8smaster1 1/1 Running 0 7d22h
kube-system kube-apiserver-k8smaster1 1/1 Running 0 7d22h
kube-system kube-controller-manager-k8smaster1 1/1 Running 0 7d22h
kube-system kube-proxy-t62q4 1/1 Running 0 7d22h
kube-system kube-proxy-tnmtr 1/1 Running 1 7d22h
kube-system kube-scheduler-k8smaster1 1/1 Running 0 7d22h
kube-system traefik-ingress-controller-8884w 1/1 Running 0 5h11m
kube-system traefik-ingress-controller-jdvck 1/1 Running 0 22h
# 탭 2번하면 목록을 볼수 있다.
ps0107@k8smaster1:~$ kubectl -n kube-system logs <tab x 2>
calico-node-wvxxm etcd-k8smaster1 kube-proxy-tnmtr
calico-node-zxkxk kube-apiserver-k8smaster1 kube-scheduler-k8smaster1
coredns-5c98db65d4-97gng kube-controller-manager-k8smaster1 traefik-ingress-controller-8884w
coredns-5c98db65d4-fvz2k kube-proxy-t62q4 traefik-ingress-controller-jdvck
ps0107@k8smaster1:~$ kubectl -n kube-system logs kube-apiserver-k8smaster1
Flag --insecure-port has been deprecated, This flag will be removed in a future version.
I0128 08:29:23.585970 1 server.go:560] external host was not specified, using 10.146.0.2
I0128 08:29:23.600341 1 server.go:147] Version: v1.15.1
I0128 08:29:24.066914 1 plugins.go:158] Loaded 10 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,MutatingAdmissionWebhook.
# -f 옵션을 통해 실시간으로 확인 가능하다.
ps0107@k8smaster1:~$ kubectl -n kube-system logs kube-apiserver-k8smaster1 -f
