Amazon Linux 2에 Kubernetes 1.28 설치 (Docker 기반) - 노드 조인

직전에 작성한 Amazon Linux 2에 Kubernetes 1.28 설치 (Docker 기반) - 노드 조인 문서에서는 Control Plane에서 kubectl 명령어가 동작하는 것 까지를 목표로 두었었다면, 이번 문서에서는 클러스터에 노드들을 조인시켜 실질적으로 k8s의 리소스들을 생성할 수 있도록 할 것입니다.

클러스터 구성

(참조 문서: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)

  1. ControlPlane 초기화하기 -> 이 부분은 kubeadm init 명령 수행하는 것을 말하며, 설치 과정중에 여기까지 이미 진행한 상태입니다.


  2. 파드 네트워크 애드온 설치
[root@minhangk8s-01 ~]# kubectl get all --all-namespaces
NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-5dd5756b68-gw4q4                0/1     Pending   0          36m
kube-system   pod/coredns-5dd5756b68-ks2c4                0/1     Pending   0          36m
kube-system   pod/etcd-minhangk8s-01                      1/1     Running   0          36m
kube-system   pod/kube-apiserver-minhangk8s-01            1/1     Running   0          36m
kube-system   pod/kube-controller-manager-minhangk8s-01   1/1     Running   0          36m
kube-system   pod/kube-proxy-bwrjc                        1/1     Running   0          36m
kube-system   pod/kube-scheduler-minhangk8s-01            1/1     Running   0          36m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  36m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   36m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   36m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/2     2            0           36m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5dd5756b68   2         2         0       36m

-> 기본 설치만 마무리된 상태에서는 아직 네트워크가 제대로 활성화 되지 않아 coredns가 기동되지 못하고 0/2 상태인 것을 확인할 수 있습니다.

[root@minhangk8s-01 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

-> Flannel 설치. 명령어도 간단하고 설치도 금새 끝납니다.

  1. Flannel 설치 완료 확인
[root@minhangk8s-01 ~]# kubectl get all --all-namespaces
NAMESPACE      NAME                                        READY   STATUS    RESTARTS   AGE
kube-flannel   pod/kube-flannel-ds-jn2tw                   1/1     Running   0          49s
kube-system    pod/coredns-5dd5756b68-gw4q4                1/1     Running   0          37m
kube-system    pod/coredns-5dd5756b68-ks2c4                1/1     Running   0          37m
kube-system    pod/etcd-minhangk8s-01                      1/1     Running   0          38m
kube-system    pod/kube-apiserver-minhangk8s-01            1/1     Running   0          38m
kube-system    pod/kube-controller-manager-minhangk8s-01   1/1     Running   0          38m
kube-system    pod/kube-proxy-bwrjc                        1/1     Running   0          37m
kube-system    pod/kube-scheduler-minhangk8s-01            1/1     Running   0          38m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  38m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   38m

NAMESPACE      NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   daemonset.apps/kube-flannel-ds   1         1         1       1            1           <none>                   49s
kube-system    daemonset.apps/kube-proxy        1         1         1       1            1           kubernetes.io/os=linux   38m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           38m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5dd5756b68   2         2         2       37m
  1. 각 노드들에 kubelet, kubeadm, kubectl 설치 클러스터에 참여할 노드들에도 kubeadm 명령을 사용해야 됩니다. 일단 [[Amazon Linux 2에 Kubernetes 1.28 설치 (Docker 기반)]] 문서 참고하여 해당 명령어들을 사용할 수 있도록 설치 진행합니다.
# OS설정
yum -y update 
rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8
vi /etc/hosts

# docker 설치
yum -y install docker tc
systemctl enable --now docker

# cri-dockerd 설치
yum -y install git go

git clone https://github.com/Mirantis/cri-dockerd.git
cd cri-dockerd
make cri-dockerd

install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
install packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket

# kubernetes 설치
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

-> 한번 설치 진행해본 뒤라 확인하는 명령어들은 제거하고, 설치/변경하는 명령어만 넣었습니다.

  1. KUBECONFIG 설정
echo "export KUBECONFIG=/etc/kubernetes/kubelet.conf" >> ~/.bash_profile
source ~/.bash_profile

-> ControlPlane에서는 export KUBECONFIG=/etc/kubernetes/admin.conf였던 부분을 export KUBECONFIG=/etc/kubernetes/kubelet.conf으로 변경했습니다.

단, 현재 단계에서는 2,3번 노드에 설치만 진행했을 뿐 아직 클러스터에 참여된 것은 아닙니다. 따라서 아직 kubectl 명령어 수행 결과가 떨어지지는 않습니다.

[root@minhangk8s-02 ~]# kubectl get all
E1108 13:23:40.596490   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 13:23:40.596798   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 13:23:40.598349   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 13:23:40.599693   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 13:23:40.601164   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

1번 노드(=ControlPlane)에서 아래와 같이 노드 정보를 조회해봐도 아직 1개 노드만 확인됩니다.

[root@minhangk8s-01 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
minhangk8s-01   Ready    control-plane   58m   v1.28.2


  1. 2번 노드에서 클러스터 Join 수행
...
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.61:6443 --token u8tzqa.cmcz3h25j7p71nop \
	--discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86 

-> kubeadm join에 출력 되었던 내용

[root@minhangk8s-02 ~]# kubeadm join 192.168.100.61:6443 --cri-socket=/run/cri-dockerd.sock --token u8tzqa.cmcz3h25j7p71nop --discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86
W1108 13:32:23.652234   82560 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/cri-dockerd.sock". Please update your configuration!
[preflight] Running pre-flight checks
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@minhangk8s-02 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
minhangk8s-01   Ready    control-plane   62m   v1.28.2
minhangk8s-02   Ready    <none>          32s   v1.28.2

-> 2번 노드에서 kubectl 명령어가 정상적으로 수행될 뿐 아니라, 추가된 2번 노드의 정보도 확인 가능합니다.

  1. 3번 노드에서 클러스터 Join 수행
[root@minhangk8s-03 ~]# kubeadm join 192.168.100.61:6443 --cri-socket=/run/cri-dockerd.sock --token u8tzqa.cmcz3h25j7p71nop --discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86
W1108 13:34:35.678983   82894 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/cri-dockerd.sock". Please update your configuration!
[preflight] Running pre-flight checks
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@minhangk8s-03 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE    VERSION
minhangk8s-01   Ready    control-plane   65m    v1.28.2
minhangk8s-02   Ready    <none>          3m2s   v1.28.2
minhangk8s-03   Ready    <none>          47s    v1.28.2

-> 3번 노드에서 kubectl 명령어가 정상적으로 수행될 뿐 아니라, 추가된 3번 노드의 정보도 확인 가능합니다.

[root@minhangk8s-01 ~]# kubectl get all --all-namespaces
NAMESPACE      NAME                                        READY   STATUS    RESTARTS   AGE
kube-flannel   pod/kube-flannel-ds-95b45                   1/1     Running   0          103s
kube-flannel   pod/kube-flannel-ds-dfz2t                   1/1     Running   0          3m58s
kube-flannel   pod/kube-flannel-ds-jn2tw                   1/1     Running   0          28m
kube-system    pod/coredns-5dd5756b68-gw4q4                1/1     Running   0          65m
kube-system    pod/coredns-5dd5756b68-ks2c4                1/1     Running   0          65m
kube-system    pod/etcd-minhangk8s-01                      1/1     Running   0          66m
kube-system    pod/kube-apiserver-minhangk8s-01            1/1     Running   0          66m
kube-system    pod/kube-controller-manager-minhangk8s-01   1/1     Running   0          66m
kube-system    pod/kube-proxy-bltds                        1/1     Running   0          103s
kube-system    pod/kube-proxy-bwrjc                        1/1     Running   0          65m
kube-system    pod/kube-proxy-fcdlp                        1/1     Running   0          3m58s
kube-system    pod/kube-scheduler-minhangk8s-01            1/1     Running   0          66m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  66m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   66m

NAMESPACE      NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   daemonset.apps/kube-flannel-ds   3         3         3       3            3           <none>                   28m
kube-system    daemonset.apps/kube-proxy        3         3         3       3            3           kubernetes.io/os=linux   66m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           66m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5dd5756b68   2         2         2       65m

-> 막 flannel을 설치한 직후와 달리, flannel에 연결된 노드도 3개씩이 되어 있습니다.

설치 완료


복사 붙여넣기를 위한 명령어 모음

위 과정을 따라해보시기 위한 분들을 위해, 편하게 복사&붙여넣기 할 수 있도록, 사용되었던 명령어들만 모아봤습니다.

### 1번 노드에서만 수행 = flannel 설치 ##########
kubectl get all --all-namespaces
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl get all --all-namespaces

### 2,3번 노드에서 각각 수행 = kubeadm등 설치 ####
# OS설정
yum -y update 
rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8
vi /etc/hosts

# docker 설치
yum -y install docker tc
systemctl enable --now docker

# cri-dockerd 설치
yum -y install git go

git clone https://github.com/Mirantis/cri-dockerd.git
cd cri-dockerd
make cri-dockerd

install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
install packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket

# kubernetes 설치
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

echo "export KUBECONFIG=/etc/kubernetes/kubelet.conf" >> ~/.bash_profile
source ~/.bash_profile

kubectl get all
kubectl get nodes


### 2,3번 노드에서 각각 수행 = 클러스터 가입 #######
# 해시 키 값은 init 할때 마다 달라집니다. 밑에 값을 그대로 복사하지 말고, 각자 환경에 맞는 키값으로 복사하셔야합니다. 
kubeadm join 192.168.100.61:6443 --cri-socket=/run/cri-dockerd.sock --token u8tzqa.cmcz3h25j7p71nop --discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86 
kubectl get nodes

### 1번 노드에서 최종 확인 ######################
kubectl get all --all-namespaces


Revision #3
Created 8 November 2023 04:54:51 by 신민항
Updated 20 November 2023 07:46:55 by 신민항