DB외 IT

kubernetes 인증서 만료되어 kubectl 실행 안될 때

Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2023-10-18T14:59:15+09:00 is after 2023-10-17T01:26:03Z

위와 같은 에러 발생하며 kubectl 명령을 사용할 수 없는 상황

대충 영문 경고 해석해보면 느낌이 오겠지만, 인증서 기간이 만료된 것임

[root@Minhang02 pki]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      
apiserver                  Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      
apiserver-etcd-client      Oct 17, 2023 01:26 UTC   <invalid>       etcd-ca                 no      
apiserver-kubelet-client   Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      
controller-manager.conf    Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      
etcd-healthcheck-client    Oct 17, 2023 01:26 UTC   <invalid>       etcd-ca                 no      
etcd-peer                  Oct 17, 2023 01:26 UTC   <invalid>       etcd-ca                 no      
etcd-server                Oct 17, 2023 01:26 UTC   <invalid>       etcd-ca                 no      
front-proxy-client         Oct 17, 2023 01:26 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf             Oct 17, 2023 01:26 UTC   <invalid>       ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 14, 2032 01:26 UTC   8y              no      
etcd-ca                 Oct 14, 2032 01:26 UTC   8y              no      
front-proxy-ca          Oct 14, 2032 01:26 UTC   8y              no     

-> kubeadm certs check-expiration 명령어를 통해 인증서 만료 일자를 확인해보니 이미 기간이 만료되어 있었음

[root@Minhang02 pki]# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

-> kubeadm certs renew all 명령어를 통해 인증서를 갱신할 수 있음

갱신하고 다시 인증서 정보를 확인해보면

[root@Minhang02 pki]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[check-expiration] Error reading configuration from the Cluster. Falling back to default configuration

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Oct 17, 2024 06:21 UTC   364d            ca                      no      
apiserver                  Oct 17, 2024 06:21 UTC   364d            ca                      no      
apiserver-etcd-client      Oct 17, 2024 06:21 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   Oct 17, 2024 06:21 UTC   364d            ca                      no      
controller-manager.conf    Oct 17, 2024 06:21 UTC   364d            ca                      no      
etcd-healthcheck-client    Oct 17, 2024 06:21 UTC   364d            etcd-ca                 no      
etcd-peer                  Oct 17, 2024 06:21 UTC   364d            etcd-ca                 no      
etcd-server                Oct 17, 2024 06:21 UTC   364d            etcd-ca                 no      
front-proxy-client         Oct 17, 2024 06:21 UTC   364d            front-proxy-ca          no      
scheduler.conf             Oct 17, 2024 06:21 UTC   364d            ca                      no      

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Oct 14, 2032 01:26 UTC   8y              no      
etcd-ca                 Oct 14, 2032 01:26 UTC   8y              no      
front-proxy-ca          Oct 14, 2032 01:26 UTC   8y              no 

-> 인증서 갱신되어 이제 364일 남았다고 뜸

[root@Minhang02 pki]# kubectl get pod -o wide
NAME                        READY   STATUS    RESTARTS   AGE    IP            NODE        NOMINATED NODE   READINESS GATES
bookstack-77df4879c-6z2v2   1/1     Running   0          28d    10.244.1.42   minhang03   <none>           <none>
bookstack-77df4879c-hqwgp   1/1     Running   0          28d    10.244.2.45   minhang04   <none>           <none>
jupyter-66477949-jvvqk      1/1     Running   0          307d   10.244.2.31   minhang04   <none>           <none>

-> 이제 kubectl도 다시 정상적으로 사용됨

hostnamectl로 hostname 변경

hostnamectl 명령어로 hostname 변경하기 명령어) ``hostnamectl set-hostname 호스트명

예시)
[root@amazonlinux ~]# hostname
amazonlinux.onprem
[root@amazonlinux ~]# hostnamectl set-hostname MinhangK8S-01
[root@amazonlinux ~]# hostname
minhangk8s-01
[root@amazonlinux ~]#
예시)
[root@amazonlinux ~]# exit
logout
Connection to 192.168.100.52 closed.
[root@Minhang9 ~]# ssh 192.168.100.52
root@192.168.100.52's password: 
Last login: Tue Oct 24 06:16:15 2023 from 192.168.100.100
   ,     #_
   ~\_  ####_        Amazon Linux 2
  ~~  \_#####\
  ~~     \###|       AL2 End of Life is 2025-06-30.
  ~~       \#/ ___
   ~~       V~' '->
    ~~~         /    A newer version of Amazon Linux is available!
      ~~._.   _/
         _/ _/       Amazon Linux 2023, GA and supported until 2028-03-15.
       _/m/'           https://aws.amazon.com/linux/amazon-linux-2023/

3 package(s) needed for security, out of 6 available
Run "sudo yum update" to apply all updates.
[root@minhangk8s-01 ~]#       => 프롬프트명의 hostname도 바뀌어 있음

Amazon Linux 2에 Kubernetes 1.26 설치 (1)

쿠버네티스 공식 다큐멘테이션 내의 kubeadm으로 설치하는 문서를 기반으로 작성하였습니다. (참고사이트: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/)

[root@minhangk8s-01 ~]# cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
Amazon Linux release 2 (Karoo)

kubeadm을 통한 kubernetes 설치

(참고사이트: https://v1-26.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) Amazon Linux는 Redhat Linux 계열 OS입니다. Red Hat-based distributions를 따라 설치 진행합니다.

  1. SELinux를 Permissive 모드로 변경 상술하였듯이, Amazon Linux는 SELinux가 애초에 Disabled 상태입니다.
[root@minhangk8s-01 ~]# getenforce 
Disabled

따라서 굳이 더 보안 등급이 높은 Permissive 모드로 올리지 않고, disabled 상태로 두고 진행하면 됩니다.

  1. 쿠버네티스 yum repository 추가
[root@minhangk8s-01 ~]# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/
> enabled=1
> gpgcheck=1
> gpgkey=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key
> exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
> EOF
  -> yum 명령어 사용을 위한 repo 파일 생성. EOF까지가 쭉 명령어임. root 계정으로 진행하고 있어서, 매뉴얼상의 명령어중 sudo는 뺐음

[root@minhangk8s-01 ~]# ls -l /etc/yum.repos.d/kubernetes.repo    
-rw-r--r-- 1 root root 235 Oct 24 20:01 /etc/yum.repos.d/kubernetes.repo
  -> repo 파일 생성된 것 확인

[root@minhangk8s-01 ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
  -> 상단에 실행한데로 파일 내용 들어간 것 확인
  1.  kubelet, kubeadm, kubectl설치
[root@minhangk8s-01 ~]# yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
Loaded plugins: langpacks, priorities, update-motd
amzn2-core                                                                                                                                                                                                     | 3.6 kB  00:00:00     
amzn2extra-docker                                                                                                                                                                                              | 2.9 kB  00:00:00     
kubernetes                                                                                                                                                                                                     | 1.7 kB  00:00:00     
(1/5): amzn2extra-docker/2/x86_64/primary_db                                                                                                                                                                   | 101 kB  00:00:00     
(2/5): amzn2-core/2/x86_64/updateinfo                                                                                                                                                                          | 729 kB  00:00:00     
(3/5): amzn2extra-docker/2/x86_64/updateinfo                                                                                                                                                                   |  13 kB  00:00:00     
(4/5): amzn2-core/2/x86_64/primary_db                                                                                                                                                                          |  67 MB  00:00:01     
(5/5): kubernetes/primary                                                                                                                                                                                      |  19 kB  00:00:01     
kubernetes                                                                                                                                                                                                                    185/185
2 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.26.10-150500.1.1 will be installed
--> Processing Dependency: kubernetes-cni >= 1.1.1 for package: kubeadm-1.26.10-150500.1.1.x86_64
--> Processing Dependency: cri-tools >= 1.25.0 for package: kubeadm-1.26.10-150500.1.1.x86_64
---> Package kubectl.x86_64 0:1.26.10-150500.1.1 will be installed
---> Package kubelet.x86_64 0:1.26.10-150500.1.1 will be installed
--> Processing Dependency: socat for package: kubelet-1.26.10-150500.1.1.x86_64
--> Processing Dependency: ebtables for package: kubelet-1.26.10-150500.1.1.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.26.10-150500.1.1.x86_64
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-5.amzn2.2 will be installed
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
---> Package cri-tools.x86_64 0:1.26.1-1.amzn2.0.2 will be installed
---> Package ebtables.x86_64 0:2.0.10-16.amzn2.0.1 will be installed
---> Package kubernetes-cni.x86_64 0:1.2.0-150500.2.1 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.amzn2.0.1 will be installed
--> Running transaction check
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.amzn2.1 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.amzn2.1 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.amzn2.0.2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================================================================================
 Package                                                        Arch                                           Version                                                       Repository                                          Size
======================================================================================================================================================================================================================================
Installing:
 kubeadm                                                        x86_64                                         1.26.10-150500.1.1                                            kubernetes                                         9.5 M
 kubectl                                                        x86_64                                         1.26.10-150500.1.1                                            kubernetes                                         9.8 M
 kubelet                                                        x86_64                                         1.26.10-150500.1.1                                            kubernetes                                          20 M
Installing for dependencies:
 conntrack-tools                                                x86_64                                         1.4.4-5.amzn2.2                                               amzn2-core                                         186 k
 cri-tools                                                      x86_64                                         1.26.1-1.amzn2.0.2                                            amzn2-core                                          18 M
 ebtables                                                       x86_64                                         2.0.10-16.amzn2.0.1                                           amzn2-core                                         122 k
 kubernetes-cni                                                 x86_64                                         1.2.0-150500.2.1                                              kubernetes                                         6.2 M
 libnetfilter_cthelper                                          x86_64                                         1.0.0-10.amzn2.1                                              amzn2-core                                          18 k
 libnetfilter_cttimeout                                         x86_64                                         1.0.0-6.amzn2.1                                               amzn2-core                                          18 k
 libnetfilter_queue                                             x86_64                                         1.0.2-2.amzn2.0.2                                             amzn2-core                                          24 k
 socat                                                          x86_64                                         1.7.3.2-2.amzn2.0.1                                           amzn2-core                                         291 k

Transaction Summary
======================================================================================================================================================================================================================================
Install  3 Packages (+8 Dependent packages)

Total download size: 64 M
Installed size: 333 M
Downloading packages:
(1/11): conntrack-tools-1.4.4-5.amzn2.2.x86_64.rpm                                                                                                                                                             | 186 kB  00:00:01     
(2/11): ebtables-2.0.10-16.amzn2.0.1.x86_64.rpm                                                                                                                                                                | 122 kB  00:00:00     
(3/11): cri-tools-1.26.1-1.amzn2.0.2.x86_64.rpm                                                                                                                                                                |  18 MB  00:00:02     
warning: /var/cache/yum/x86_64/2/kubernetes/packages/kubectl-1.26.10-150500.1.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 9a296436: NOKEY                                                  ] 4.6 MB/s |  28 MB  00:00:07 ETA 
Public key for kubectl-1.26.10-150500.1.1.x86_64.rpm is not installed
(4/11): kubectl-1.26.10-150500.1.1.x86_64.rpm                                                                                                                                                                  | 9.8 MB  00:00:03     
(5/11): kubeadm-1.26.10-150500.1.1.x86_64.rpm                                                                                                                                                                  | 9.5 MB  00:00:04     
(6/11): libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64.rpm                                                                                                                                                      |  18 kB  00:00:00     
(7/11): libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64.rpm                                                                                                                                                      |  18 kB  00:00:00     
(8/11): libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64.rpm                                                                                                                                                        |  24 kB  00:00:00     
(9/11): socat-1.7.3.2-2.amzn2.0.1.x86_64.rpm                                                                                                                                                                   | 291 kB  00:00:01     
(10/11): kubernetes-cni-1.2.0-150500.2.1.x86_64.rpm                                                                                                                                                            | 6.2 MB  00:00:03     
(11/11): kubelet-1.26.10-150500.1.1.x86_64.rpm                                                                                                                                                                 |  20 MB  00:00:04     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                 6.9 MB/s |  64 MB  00:00:09     
Retrieving key from https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key
Importing GPG key 0x9A296436:
 Userid     : "isv:kubernetes OBS Project <isv:kubernetes@build.opensuse.org>"
 Fingerprint: de15 b144 86cd 377b 9e87 6e1a 2346 54da 9a29 6436
 From       : https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kubernetes-cni-1.2.0-150500.2.1.x86_64                                                                                                                                                                            1/11 
  Installing : libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64                                                                                                                                                                       2/11 
  Installing : libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64                                                                                                                                                                     3/11 
  Installing : libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64                                                                                                                                                                     4/11 
  Installing : conntrack-tools-1.4.4-5.amzn2.2.x86_64                                                                                                                                                                            5/11 
  Installing : kubectl-1.26.10-150500.1.1.x86_64                                                                                                                                                                                 6/11 
  Installing : ebtables-2.0.10-16.amzn2.0.1.x86_64                                                                                                                                                                               7/11 
  Installing : socat-1.7.3.2-2.amzn2.0.1.x86_64                                                                                                                                                                                  8/11 
  Installing : kubelet-1.26.10-150500.1.1.x86_64                                                                                                                                                                                 9/11 
  Installing : cri-tools-1.26.1-1.amzn2.0.2.x86_64                                                                                                                                                                              10/11 
  Installing : kubeadm-1.26.10-150500.1.1.x86_64                                                                                                                                                                                11/11 
  Verifying  : kubernetes-cni-1.2.0-150500.2.1.x86_64                                                                                                                                                                            1/11 
  Verifying  : cri-tools-1.26.1-1.amzn2.0.2.x86_64                                                                                                                                                                               2/11 
  Verifying  : socat-1.7.3.2-2.amzn2.0.1.x86_64                                                                                                                                                                                  3/11 
  Verifying  : kubeadm-1.26.10-150500.1.1.x86_64                                                                                                                                                                                 4/11 
  Verifying  : ebtables-2.0.10-16.amzn2.0.1.x86_64                                                                                                                                                                               5/11 
  Verifying  : kubectl-1.26.10-150500.1.1.x86_64                                                                                                                                                                                 6/11 
  Verifying  : conntrack-tools-1.4.4-5.amzn2.2.x86_64                                                                                                                                                                            7/11 
  Verifying  : libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64                                                                                                                                                                     8/11 
  Verifying  : kubelet-1.26.10-150500.1.1.x86_64                                                                                                                                                                                 9/11 
  Verifying  : libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64                                                                                                                                                                    10/11 
  Verifying  : libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64                                                                                                                                                                      11/11 

Installed:
  kubeadm.x86_64 0:1.26.10-150500.1.1                                         kubectl.x86_64 0:1.26.10-150500.1.1                                         kubelet.x86_64 0:1.26.10-150500.1.1                                        

Dependency Installed:
  conntrack-tools.x86_64 0:1.4.4-5.amzn2.2         cri-tools.x86_64 0:1.26.1-1.amzn2.0.2          ebtables.x86_64 0:2.0.10-16.amzn2.0.1  kubernetes-cni.x86_64 0:1.2.0-150500.2.1  libnetfilter_cthelper.x86_64 0:1.0.0-10.amzn2.1 
  libnetfilter_cttimeout.x86_64 0:1.0.0-6.amzn2.1  libnetfilter_queue.x86_64 0:1.0.2-2.amzn2.0.2  socat.x86_64 0:1.7.3.2-2.amzn2.0.1    

Complete!
  1. 설치된 것 확인
[root@minhangk8s-01 ~]# kubelet --version
Kubernetes v1.26.10
[root@minhangk8s-01 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.10", GitCommit:"b8609d4dd75c5d6fba4a5eaa63a5507cb39a6e99", GitTreeState:"clean", BuildDate:"2023-10-18T11:42:11Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
[root@minhangk8s-01 ~]# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.10", GitCommit:"b8609d4dd75c5d6fba4a5eaa63a5507cb39a6e99", GitTreeState:"clean", BuildDate:"2023-10-18T11:44:31Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
The connection to the server localhost:8080 was refused - did you specify the right host or port?
  1. kubelet 구동
[root@minhangk8s-01 ~]# systemctl enable --now kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
  -> systemctl에 enable 시켜두어 매번 재부팅 때마다 자동으로 띄우도록 합니다.

[root@minhangk8s-01 ~]# ps -ef | grep kubelet
root      35977  35665  0 20:12 pts/1    00:00:00 grep --color=auto kubelet
  -> 헌데 확인해보면 프로세스가 떠있지 않는 상태입니다.

[root@minhangk8s-01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Tue 2023-10-24 20:10:52 UTC; 6s ago
     Docs: https://kubernetes.io/docs/
  Process: 35918 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 35918 (code=exited, status=1/FAILURE)

Oct 24 20:10:52 minhangk8s-01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Oct 24 20:10:52 minhangk8s-01 systemd[1]: Unit kubelet.service entered failed state.
Oct 24 20:10:52 minhangk8s-01 systemd[1]: kubelet.service failed.
  -> status 확인해보면 뭔가 에러 발생해서 뜨지 않고 있다는 것을 알 수 있습니다. 아직초기화를 진행하지 않아서 그렇습니다.

kubeadm을 통한 kubernetes 초기화

(참고 사이트: https://v1-26.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)

[root@minhangk8s-01 ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 
I1024 20:55:19.376527   35860 version.go:256] remote version is much newer: v1.28.3; falling back to: stable-1.26
[init] Using Kubernetes version: v1.26.10
[preflight] Running pre-flight checks
        [WARNING FileExisting-tc]: tc not found in system path
        [WARNING Hostname]: hostname "minhangk8s-01" could not be reached
        [WARNING Hostname]: hostname "minhangk8s-01": lookup minhangk8s-01 on 168.126.63.1:53: no such host
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR CRI]: container runtime is not running: output: E1024 20:55:19.862528   35868 remote_runtime.go:616] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\""
time="2023-10-24T20:55:19Z" level=fatal msg="getting status of runtime: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/containerd/containerd.sock: connect: no such file or directory\""
, error: exit status 1
        [ERROR FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
        [ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
-> 대뜸 초기화 명령어 실행부터 해봤더니 여러가지 에러가 발생했습니다...

에러 처리

  1. tc 명령어 없음 [WARNING FileExisting-tc]: tc not found in system path 부분 참고

yum으로 tc 명령어 설치 진행합니다.

[root@minhangk8s-01 ~]# yum install tc
Loaded plugins: langpacks, priorities, update-motd
amzn2-core                                                                                                                                                                                                     | 3.6 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package iproute-tc.x86_64 0:5.10.0-2.amzn2.0.3 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================================================================================
 Package                                               Arch                                              Version                                                          Repository                                             Size
======================================================================================================================================================================================================================================
Installing:
 iproute-tc                                            x86_64                                            5.10.0-2.amzn2.0.3                                               amzn2-core                                            432 k

Transaction Summary
======================================================================================================================================================================================================================================
Install  1 Package

Total download size: 432 k
Installed size: 815 k
Is this ok [y/d/N]: y
Downloading packages:
iproute-tc-5.10.0-2.amzn2.0.3.x86_64.rpm                                                                                                                                                                       | 432 kB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : iproute-tc-5.10.0-2.amzn2.0.3.x86_64                                                                                                                                                                               1/1 
  Verifying  : iproute-tc-5.10.0-2.amzn2.0.3.x86_64                                                                                                                                                                               1/1 

Installed:
  iproute-tc.x86_64 0:5.10.0-2.amzn2.0.3                                                                                                                                                                                              

Complete!

  1. hostname 찾을 수 없음 [WARNING Hostname]: hostname "minhangk8s-01" could not be reached 부분 참조

-> /etc/hosts 파일에 hostname 추가합니다.

[root@minhangk8s-01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost6 localhost6.localdomain6

192.168.100.161 MinhangK8S-01
192.168.100.162 MinhangK8S-02
192.168.100.163 MinhangK8S-03
192.168.100.164 MinhangK8S-04
192.168.100.165 MinhangK8S-05

-> vi 편집기를 통해 위와 같이 k8s 설치에 사용할 호스트명들을 집어 넣었습니다.

  1. bridge-nf-call-iptables 파일 없음 /proc/sys/net/bridge/bridge-nf-call-iptables does not exist 부분 참조
[root@minhangk8s-01 ~]# ls -l /proc/sys/net/bridge/bridge-nf-call-iptables
ls: cannot access /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory
  -> 실제로 파일이 없는 상태입니다.

[root@minhangk8s-01 ~]# modprobe br_netfilter
  -> modprobe 사용하여 초기화합니다.

[root@minhangk8s-01 ~]# ls -l /proc/sys/net/bridge/bridge-nf-call-iptables
-rw-r--r-- 1 root root 0 Oct 24 20:29 /proc/sys/net/bridge/bridge-nf-call-iptables
  -> 파일 생김

[root@minhangk8s-01 ~]# cat  /proc/sys/net/bridge/bridge-nf-call-iptables
0
  -> 해당 파일의 초기값이 0으로 되어있는데, 그냥 두면 1로 설정하라고 또 에러 뜰것임

[root@minhangk8s-01 ~]# echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
[root@minhangk8s-01 ~]# cat  /proc/sys/net/bridge/bridge-nf-call-iptables    
1
  -> 미리 1로 바꿔둡니다.

  1. /proc/sys/net/ipv4/ip_forward 파일 내용 1로 변경하라 /proc/sys/net/ipv4/ip_forward contents are not set to 1 부분 참조 직전에 bridge-nf-call-iptables파일 1로 수정한 부분과 마찬가지입니다. 다만 파일은 이미 생성되어 있으므로 1로 변경만 해주면 됩니다.
[root@minhangk8s-01 ~]# ls -l /proc/sys/net/ipv4/ip_forward
-rw-r--r-- 1 root root 0 Oct 24 06:33 /proc/sys/net/ipv4/ip_forward
[root@minhangk8s-01 ~]# cat /proc/sys/net/ipv4/ip_forward  
0
[root@minhangk8s-01 ~]# echo 1 > /proc/sys/net/ipv4/ip_forward   
[root@minhangk8s-01 ~]# cat /proc/sys/net/ipv4/ip_forward     
1

  1. containerd.sock 파일 없음 /var/run/containerd/containerd.sock: connect: no such file or directory 부분 참조 -> 처음에 runtime 설치를 안하고 설치과정을 시작해서 그렇습니다. containerd를 runtime으로 설치 진행합니다. (docker나 crio도 가능)
[root@minhangk8s-01 ~]# yum install containerd
Loaded plugins: langpacks, priorities, update-motd
amzn2-core                                                                                                                                                                                                     | 3.6 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package containerd.x86_64 0:1.6.19-1.amzn2.0.5 will be installed
--> Processing Dependency: runc for package: containerd-1.6.19-1.amzn2.0.5.x86_64
--> Running transaction check
---> Package runc.x86_64 0:1.1.7-4.amzn2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================================================================================
 Package                                              Arch                                             Version                                                      Repository                                                   Size
======================================================================================================================================================================================================================================
Installing:
 containerd                                           x86_64                                           1.6.19-1.amzn2.0.5                                           amzn2extra-docker                                            28 M
Installing for dependencies:
 runc                                                 x86_64                                           1.1.7-4.amzn2                                                amzn2extra-docker                                           3.0 M

Transaction Summary
======================================================================================================================================================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 31 M
Installed size: 111 M
Is this ok [y/d/N]: y
Downloading packages:
(1/2): runc-1.1.7-4.amzn2.x86_64.rpm                                                                                                                                                                           | 3.0 MB  00:00:00     
(2/2): containerd-1.6.19-1.amzn2.0.5.x86_64.rpm                                                                                                                                                                |  28 MB  00:00:00     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                  66 MB/s |  31 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : runc-1.1.7-4.amzn2.x86_64                                                                                                                                                                                          1/2 
  Installing : containerd-1.6.19-1.amzn2.0.5.x86_64                                                                                                                                                                               2/2 
  Verifying  : runc-1.1.7-4.amzn2.x86_64                                                                                                                                                                                          1/2 
  Verifying  : containerd-1.6.19-1.amzn2.0.5.x86_64                                                                                                                                                                               2/2 

Installed:
  containerd.x86_64 0:1.6.19-1.amzn2.0.5                                                                                                                                                                                              

Dependency Installed:
  runc.x86_64 0:1.1.7-4.amzn2                                                                                                                                                                                                         

Complete!

-> 설치는 yum으로 진행하면 되고, 추가적인 초기화 과정 필요합니다.

[root@minhangk8s-01 ~]# containerd config default | tee /etc/containerd/config.toml
disabled_plugins = []
imports = []
oom_score = 0
plugin_dir = ""
required_plugins = []

... 중략 ...

  "io.containerd.timeout.task.state" = "2s"

[ttrpc]
  address = ""
  gid = 0
  uid = 0

-> config.toml 파일 초기화

해당 파일에 systemd를 cgroup driver로 사용하기 위한 설정 변경이 필요합니다. vi /etc/containerd/config.toml

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            BinaryName = ""
            CriuImagePath = ""
            CriuPath = ""
            CriuWorkPath = ""
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            NoPivotRoot = false
            Root = ""
            ShimCgroup = ""
            SystemdCgroup = false

-> config.toml파일에서 SystemdCgroup으로 찾기해보면 딱 하나 나옵니다. 위와 같은 내용으로 구성되어 있는 블록이며, 여기서 SystemdCgroup = true로 변경하고 저장해줍니다.

그리고 containerd를 기동해줍니다.

[root@minhangk8s-01 ~]# systemctl enable --now containerd
[root@minhangk8s-01 ~]# systemctl status containerd      
● containerd.service - containerd container runtime
   Loaded: loaded (/usr/lib/systemd/system/containerd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-10-24 21:06:28 UTC; 3s ago
     Docs: https://containerd.io
  Process: 36662 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
 Main PID: 36677 (containerd)
    Tasks: 9
   Memory: 13.3M
   CGroup: /system.slice/containerd.service
           └─36677 /usr/bin/containerd

Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.919122697Z" level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="c...load cni config"
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.919275118Z" level=info msg=serving... address=/run/containerd/containerd.sock.ttrpc
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.919309596Z" level=info msg=serving... address=/run/containerd/containerd.sock
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.919345366Z" level=info msg="containerd successfully booted in 0.021143s"
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.924024978Z" level=info msg="Start subscribing containerd event"
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.924225838Z" level=info msg="Start recovering state"
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.924360597Z" level=info msg="Start event monitor"
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.924519519Z" level=info msg="Start snapshots syncer"
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.924643966Z" level=info msg="Start cni network conf syncer for default"
Oct 24 21:06:28 minhangk8s-01 containerd[36677]: time="2023-10-24T21:06:28.924744136Z" level=info msg="Start streaming server"
Hint: Some lines were ellipsized, use -l to show in full.
[root@minhangk8s-01 ~]# ls -l /var/run/containerd/containerd.sock
srw-rw---- 1 root root 0 Oct 24 21:06 /var/run/containerd/containerd.sock
-> 이제 containerd 소켓 파일도 생겼습니다.

  1. kubeadm 초기화 재실행
[root@minhangk8s-01 ~]# kubeadm init --pod-network-cidr=10.244.0.0/16 
I1024 21:08:35.047954   37233 version.go:256] remote version is much newer: v1.28.3; falling back to: stable-1.26
[init] Using Kubernetes version: v1.26.10
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local minhangk8s-01] and IPs [10.96.0.1 192.168.100.61]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost minhangk8s-01] and IPs [192.168.100.61 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost minhangk8s-01] and IPs [192.168.100.61 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 21.002525 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node minhangk8s-01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node minhangk8s-01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: g2veat.u59md2i64tn12f71
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.61:6443 --token g2veat.u59md2i64tn12f71 \
        --discovery-token-ca-cert-hash sha256:0c16e7ea8aa0142a482744df184d9f2282ed6070239af64b2e460dbf889108cd 

-> 설치 완료되었습니다. 설치 옵션으로 넣어준 pod-network-cidr은 이름 그대로 파드간의 통신에 사용될 CIDR설정값인데, flannel을 사용할 예정이기에, 10.244.0.0/16으로 설정했습니다. 이 값을 다르게 주면 flannel 설정시 에러 생깁니다. -> discovery-token-ca-cert-hash 값은 다른 노드 가입시 사용해야 하므로 잘 저장해두어야 합니다.

  1. kubelet 구동 확인 초기화 전에는 떠있지 않았던 kubelet이 정상적으로 떠있는 것 확인합니다.
[root@minhangk8s-01 ~]# ps -ef | grep kubelet
root      37990  37612  3 21:09 ?        00:00:08 kube-apiserver --advertise-address=192.168.100.61 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root      38127      1  1 21:09 ?        00:00:03 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9
root      38644  35654  0 21:13 pts/1    00:00:00 grep --color=auto kubelet
[root@minhangk8s-01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Tue 2023-10-24 21:09:31 UTC; 4min 3s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 38127 (kubelet)
    Tasks: 10
   Memory: 37.3M
   CGroup: /system.slice/kubelet.service
           └─38127 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/cont...

Oct 24 21:12:45 minhangk8s-01 kubelet[38127]: E1024 21:12:45.517650   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:12:50 minhangk8s-01 kubelet[38127]: E1024 21:12:50.518731   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:12:55 minhangk8s-01 kubelet[38127]: E1024 21:12:55.520286   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:13:00 minhangk8s-01 kubelet[38127]: E1024 21:13:00.520868   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:13:05 minhangk8s-01 kubelet[38127]: E1024 21:13:05.522334   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:13:10 minhangk8s-01 kubelet[38127]: E1024 21:13:10.524541   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:13:15 minhangk8s-01 kubelet[38127]: E1024 21:13:15.525152   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:13:20 minhangk8s-01 kubelet[38127]: E1024 21:13:20.525628   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:13:25 minhangk8s-01 kubelet[38127]: E1024 21:13:25.526819   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Oct 24 21:13:30 minhangk8s-01 kubelet[38127]: E1024 21:13:30.528932   38127 kubelet.go:2475] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network pl...not initialized"
Hint: Some lines were ellipsized, use -l to show in full.

  1. KUBECONFIG 설정
[root@minhangk8s-01 ~]# kubectl get all
E1024 21:26:52.507964   39046 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1024 21:26:52.508320   39046 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1024 21:26:52.509325   39046 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1024 21:26:52.509597   39046 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1024 21:26:52.511074   39046 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
  -> 설치가 완료되었음에도 kubectl 명령 결과가 이상하게 출력되는 상태일 것입니다.


[root@minhangk8s-01 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@minhangk8s-01 ~]# source ~/.bash_profile
  -> .bash_profile에 KUBECONFIG 환경변수를 추가해줍니다.

[root@minhangk8s-01 ~]# kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   17m
  -> 이제 kubectl 명령어가 정상적으로 출력됩니다.

복사 붙여넣기를 위한 명령어 모음

위 과정을 따라해보시기 위한 분들을 위해, 편하게 복사&붙여넣기 할 수 있도록, 사용되었던 명령어들만 모아봤습니다.

getenforce 


cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF


ls -l /etc/yum.repos.d/kubernetes.repo 
cat /etc/yum.repos.d/kubernetes.repo


yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes


kubelet --version
kubeadm version
kubectl version


systemctl enable --now kubelet
ps -ef | grep kubelet
systemctl status kubelet


kubeadm init --pod-network-cidr=10.244.0.0/16 


yum install tc

ls -l /proc/sys/net/bridge/bridge-nf-call-iptables
modprobe br_netfilter
ls -l /proc/sys/net/bridge/bridge-nf-call-iptables
cat  /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
cat  /proc/sys/net/bridge/bridge-nf-call-iptables

ls -l /proc/sys/net/ipv4/ip_forward
cat /proc/sys/net/ipv4/ip_forward  
echo 1 > /proc/sys/net/ipv4/ip_forward   
cat /proc/sys/net/ipv4/ip_forward     

yum install containerd
containerd config default | tee /etc/containerd/config.toml
vi /etc/containerd/config.toml
/SystemdCgroup
SystemdCgroup = true
systemctl enable --now containerd
systemctl status containerd
ls -l /var/run/containerd/containerd.sock

kubeadm init --pod-network-cidr=10.244.0.0/16

ps -ef | grep kubelet
systemctl status kubelet

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile
kubectl get all

Amazon Linux 2에 Kubernetes 1.26 설치 (2) - 노드 조인

앞선 Amazon Linux 2에 Kubernetes 1.26 설치 (1) 문서 에서는 Control Plane에서 kubectl 명령어가 동작하는 것 까지를 목표로 두었었다면, 이번 문서에서는 클러스터에 노드들을 조인시켜 실질적으로 k8s의 리소스들을 생성할 수 있도록 할 것입니다.

설치의 마무리

(참조 문서: https://v1-26.docs.kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)

  1. 파드 네트워크 애드온 설치
    파드간의 통신 기능을 활성화하기 위해 CNI(Container Network Interface) 기반의 파드 네트워크 애드온이 설치되어 있어야합니다.
[root@minhangk8s-01 ~]# kubectl get all --all-namespaces
NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-787d4945fb-795w5                0/1     Pending   0          8h
kube-system   pod/coredns-787d4945fb-rdbgd                0/1     Pending   0          8h
kube-system   pod/etcd-minhangk8s-01                      1/1     Running   0          8h
kube-system   pod/kube-apiserver-minhangk8s-01            1/1     Running   0          8h
kube-system   pod/kube-controller-manager-minhangk8s-01   1/1     Running   0          8h
kube-system   pod/kube-proxy-9g26b                        1/1     Running   0          8h
kube-system   pod/kube-scheduler-minhangk8s-01            1/1     Running   0          8h

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  8h
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   8h

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   8h

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/2     2            0           8h

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-787d4945fb   2         2         0       8h

-> 기본 설치만 마무리된 상태에서는 아직 네트워크가 제대로 활성화 되지 않아 coredns가 기동되지 못하고 0/2 상태인 것을 확인할 수 있습니다.

CNI 플러그인은 종류가 다양하긴한데, 여기서는 Flannel이라는 프로그램을 이용할 것입니다.

[root@minhangk8s-01 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

-> Flannel 설치. 명령어도 간단하고 설치도 금새 끝납니다.

 

  1. Flannel 설치 완료 확인
[root@minhangk8s-01 ~]# kubectl get all --all-namespaces
NAMESPACE      NAME                                        READY   STATUS    RESTARTS   AGE
kube-flannel   pod/kube-flannel-ds-5twqd                   1/1     Running   0          59s
kube-system    pod/coredns-787d4945fb-795w5                1/1     Running   0          8h
kube-system    pod/coredns-787d4945fb-rdbgd                1/1     Running   0          8h
kube-system    pod/etcd-minhangk8s-01                      1/1     Running   0          8h
kube-system    pod/kube-apiserver-minhangk8s-01            1/1     Running   0          8h
kube-system    pod/kube-controller-manager-minhangk8s-01   1/1     Running   0          8h
kube-system    pod/kube-proxy-9g26b                        1/1     Running   0          8h
kube-system    pod/kube-scheduler-minhangk8s-01            1/1     Running   0          8h

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  8h
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   8h

NAMESPACE      NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   daemonset.apps/kube-flannel-ds   1         1         1       1            1           <none>                   59s
kube-system    daemonset.apps/kube-proxy        1         1         1       1            1           kubernetes.io/os=linux   8h

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           8h

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-787d4945fb   2         2         2       8h
  1. 각 노드들에 kubelet, kubeadm, kubectl 설치 클러스터에 참여할 노드들에도 kubeadm 명령을 사용해야 됩니다. 일단 Amazon Linux 2에 Kubernetes 1.26 설치 (1) 문서 참고하여 설치 진행합니다.
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.26/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

yum install -y tc 

modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/ipv4/ip_forward   

yum install -y containerd
containerd config default | tee /etc/containerd/config.toml
vi /etc/containerd/config.toml
/SystemdCgroup
SystemdCgroup = true
systemctl enable --now containerd

echo "export KUBECONFIG=/etc/kubernetes/kubelet.conf" >> ~/.bash_profile
source ~/.bash_profile

-> 한번 설치 진행해본 뒤라 확인하는 명령어들은 제거하고, 설치 변경하는 명령어만 넣었습니다. 
-> 그 외에, 딱 하나 달라진 명령어가 있는데, export KUBECONFIG=/etc/kubernetes/admin.confexport KUBECONFIG=/etc/kubernetes/kubelet.conf으로 변경했습니다. 개별 노드들은 admin 권한이 아니기 때문에 일반 계정용 conf파일에 연결했습니다.
-> 하지만 아직 join 전이라 명령어 수행 결과가 떨어지지는 않습니다.

[root@minhangk8s-02 ~]# kubectl get all
W1025 06:06:12.173910   41065 loader.go:222] Config not found: /etc/kubernetes/kubelet.conf
E1025 06:06:12.174380   41065 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1025 06:06:12.174554   41065 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1025 06:06:12.175920   41065 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1025 06:06:12.176070   41065 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1025 06:06:12.177711   41065 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

-> 아직은 kubectl 명령어 에러 발생

 

클러스터 Join

현재 단계에서는 2,3번 노드에 설치만 진행했을 뿐 아직 클러스터에 참여된 것은 아닙니다. 따라서 1번 노드(=ControlPlane)에서 아래와 같이 노드 정보를 조회해보면 1개 노드만 확인됩니다.

[root@minhangk8s-01 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
minhangk8s-01   Ready    control-plane   8h    v1.26.10
  1. Join 명령어 확인 이제 2,3번 노드에서 kubeadm join 명령을 수행해줘야 하는데, 해당 명령어는 1번 노드에서 kubeadm init 했을 때에 화면에 출력되었던 결과입니다.
    discovery-token-ca-cert-hash 값은 다른 노드 가입시 사용해야 하므로 잘 저장해두어야 합니다. 라며 언급했던 부분입니다.

  2. 2번 노드에서 Join 수행

[root@minhangk8s-02 ~]# kubeadm join 192.168.100.61:6443 --token g2veat.u59md2i64tn12f71 \
>         --discovery-token-ca-cert-hash sha256:0c16e7ea8aa0142a482744df184d9f2282ed6070239af64b2e460dbf889108cd 
[preflight] Running pre-flight checks
	[WARNING Hostname]: hostname "minhangk8s-02" could not be reached
	[WARNING Hostname]: hostname "minhangk8s-02": lookup minhangk8s-02 on 168.126.63.1:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

-> 기존에 저장해두었던 토큰값을 이용하여 join 명령을 수행합니다.

 

[root@minhangk8s-01 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
minhangk8s-01   Ready    control-plane   8h    v1.26.10
minhangk8s-02   Ready    <none>          49s   v1.26.10

-> 1번 노드에서 다시 확인해 봤습니다. 2번노드가 클러스터에 참여된 것을 확인할 수 있습니다.

 

  1. 3번 노드에서 Join 수행 마찬가지로 3번 노드도 join 진행합니다.
[root@minhangk8s-03 ~]# kubeadm join 192.168.100.61:6443 --token g2veat.u59md2i64tn12f71 \
>         --discovery-token-ca-cert-hash sha256:0c16e7ea8aa0142a482744df184d9f2282ed6070239af64b2e460dbf889108cd 
[preflight] Running pre-flight checks
	[WARNING Hostname]: hostname "minhangk8s-03" could not be reached
	[WARNING Hostname]: hostname "minhangk8s-03": lookup minhangk8s-03 on 168.126.63.1:53: no such host
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

-> 3번노드 join 완료.

 

[root@minhangk8s-03 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
minhangk8s-01   Ready    control-plane   9h    v1.26.10
minhangk8s-02   Ready    <none>          18m   v1.26.10
minhangk8s-03   Ready    <none>          16m   v1.26.10

-> 또한 설정이 다 완료 되었으므로, 굳이 1번 노드에서 확인할 필요 없이 2,3번 노드에서 바로 kubectl 명령어로 확인도 가능합니다. join하기 전에는 에러 났었던 명령어입니다.

Amazon Linux 2에 Kubernetes 1.28 설치 (Docker 기반)

쿠버네티스 공식 다큐멘테이션 내의 kubeadm으로 설치하는 문서를 기반으로 작성하였습니다. (참조 문서: https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)

[root@minhangk8s-01 ~]# cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
Amazon Linux release 2 (Karoo)

kubeadm을 통한 kubernetes 설치

Amazon Linux는 Redhat Linux 계열 OS입니다. Red Hat-based distributions를 따라 설치 진행합니다.

  1. OS 기본 설정들
[root@minhangk8s-01 ~]# yum -y update 
Loaded plugins: langpacks, priorities, update-motd
amzn2-core                                                                                                                                                     | 3.6 kB  00:00:00     
amzn2extra-docker                                                                                                                                              | 2.9 kB  00:00:00     
(1/4): amzn2extra-docker/2/x86_64/updateinfo                                                                                                                   |  13 kB  00:00:00     
(2/4): amzn2extra-docker/2/x86_64/primary_db         
...
Replaced:
  grub2.x86_64 1:2.06-14.amzn2.0.1                                                       grub2-tools.x86_64 1:2.06-14.amzn2.0.1                                                      

Complete!
[root@minhangk8s-01 ~]# rm -f /etc/localtime

[root@minhangk8s-01 ~]# ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime

[root@minhangk8s-01 ~]# localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
/usr/share/i18n/locales/en_US:15: non-symbolic character value should not be used
/usr/share/i18n/locales/en_US:16: non-symbolic character value should not be used
/usr/share/i18n/locales/en_US:17: non-symbolic character value should not be used
...
LC_CTYPE: table for map "tolower": 139850954557471 bytes
LC_CTYPE: table for map "totitle": 0 bytes
LC_CTYPE: table for width: 0 bytes

[root@minhangk8s-01 ~]# localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8
/usr/share/i18n/locales/ko_KR:48: non-symbolic character value should not be used
/usr/share/i18n/locales/ko_KR:52: non-symbolic character value should not be used
/usr/share/i18n/locales/ko_KR:55: non-symbolic character value should not be used
...
LC_CTYPE: table for map "tolower": 140086909933599 bytes
LC_CTYPE: table for map "totitle": 0 bytes
LC_CTYPE: table for width: 0 bytes
[root@minhangk8s-01 ~]# getenforce 
Disabled

굳이 더 보안 등급이 높은 Permissive 모드로 올리지 않고, disabled 상태로 두고 진행하면 됩니다.

[root@minhangk8s-01 ~]# vi /etc/hosts

[root@minhangk8s-01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost6 localhost6.localdomain6

192.168.100.161 MinhangK8S-01
192.168.100.162 MinhangK8S-02
192.168.100.163 MinhangK8S-03
192.168.100.164 MinhangK8S-04
192.168.100.165 MinhangK8S-05

-> vi 편집기를 통해 위와 같이 k8s 설치에 사용할 호스트명들을 집어 넣었습니다.

  1. Container Runtime 설치 Container Runtime으로Docker를 설치합니다. (나중에 kubernetes 초기화시 필요한 tc 패키지도 같이 설치해줍니다.)
[root@minhangk8s-01 ~]# yum -y install docker tc
Loaded plugins: langpacks, priorities, update-motd
Resolving Dependencies
...

======================================================================================================================================================================================
 Package                                 Arch                                Version                                             Repository                                      Size
======================================================================================================================================================================================
Installing:
 docker                                  x86_64                              20.10.25-1.amzn2.0.3                                amzn2extra-docker                               43 M
 iproute-tc                              x86_64                              5.10.0-2.amzn2.0.3                                  amzn2-core                                     432 k
Installing for dependencies:
 containerd                              x86_64                              1.6.19-1.amzn2.0.5                                  amzn2extra-docker                               28 M
 libcgroup                               x86_64                              0.41-21.amzn2                                       amzn2-core                                      66 k
 pigz                                    x86_64                              2.3.4-1.amzn2.0.1                                   amzn2-core                                      81 k
 runc                                    x86_64                              1.1.7-4.amzn2                                       amzn2extra-docker                              3.0 M

Transaction Summary
======================================================================================================================================================================================
Install  2 Packages (+4 Dependent packages)
 
...
Installed:
  docker.x86_64 0:20.10.25-1.amzn2.0.3                                                     iproute-tc.x86_64 0:5.10.0-2.amzn2.0.3                                                    

Dependency Installed:
  containerd.x86_64 0:1.6.19-1.amzn2.0.5             libcgroup.x86_64 0:0.41-21.amzn2             pigz.x86_64 0:2.3.4-1.amzn2.0.1             runc.x86_64 0:1.1.7-4.amzn2            

Complete!



[root@minhangk8s-01 ~]# systemctl enable --now docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
-> 도커 구동

[root@minhangk8s-01 ~]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
-> 도커 명령어 동작하는 것 확인


  1. cri-dockerd 설치 Docker를 Kubernetes의 CRI(Container Runtime Interface)로 사용하기 위해 cri-dockerd를 추가로 설치해줍니다. 이 부분은 공식 매뉴얼에도 언급이 되어 있는데, Docker가 Kubernetes와 호환되기 위한 CRI 요구 사항을 만족하지 못하므로, Kubernetes 1.24부터 Docker 기본 상태로는 사용이 불가능하고, cri-dockerd라는 별도의 프로젝트를 통해 이 부분이 구현되어 있다는 것입니다. 따라서 cri-dockerd를 추가로 설치해주어야 docker 기반으로 kubernetes 사용이 가능합니다.
[root@minhangk8s-01 ~]# yum -y install git go
Loaded plugins: langpacks, priorities, update-motd
amzn2-core                                                                                                                                                     | 3.6 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
...
======================================================================================================================================================================================
 Package                                       Arch                                Version                                              Repository                               Size
======================================================================================================================================================================================
Installing:
 git                                           x86_64                              2.40.1-1.amzn2.0.1                                   amzn2-core                               54 k
 golang                                        x86_64                              1.20.10-1.amzn2.0.1                                  amzn2-core                              682 k
...

Complete!
[root@minhangk8s-01 ~]# git clone https://github.com/Mirantis/cri-dockerd.git
Cloning into 'cri-dockerd'...
remote: Enumerating objects: 18364, done.
remote: Counting objects: 100% (2923/2923), done.
remote: Compressing objects: 100% (1100/1100), done.
remote: Total 18364 (delta 2062), reused 1904 (delta 1810), pack-reused 15441
Receiving objects: 100% (18364/18364), 42.91 MiB | 20.22 MiB/s, done.
Resolving deltas: 100% (9251/9251), done.
[root@minhangk8s-01 cri-dockerd]# make cri-dockerd
GOARCH= go build -trimpath -ldflags " -s -w -buildid=`git log -1 --pretty='%h'` -X github.com/Mirantis/cri-dockerd/cmd/version.Version=0.3.7 -X github.com/Mirantis/cri-dockerd/cmd/version.PreRelease=`grep -q dev <<< "0.3.7" && echo "pre" || echo ""` -X github.com/Mirantis/cri-dockerd/cmd/version.GitCommit=`git log -1 --pretty='%h'`" -o cri-dockerd
[root@minhangk8s-01 cri-dockerd]# install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
[root@minhangk8s-01 cri-dockerd]# install packaging/systemd/* /etc/systemd/system
[root@minhangk8s-01 cri-dockerd]# sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
[root@minhangk8s-01 cri-dockerd]# systemctl daemon-reload
[root@minhangk8s-01 cri-dockerd]# systemctl enable --now cri-docker.socket
Created symlink from /etc/systemd/system/sockets.target.wants/cri-docker.socket to /etc/systemd/system/cri-docker.socket.
[root@minhangk8s-01 cri-dockerd]# cd
[root@minhangk8s-01 ~]# 
[root@minhangk8s-01 ~]# systemctl status cri-docker.socket
● cri-docker.socket - CRI Docker Socket for the API
   Loaded: loaded (/etc/systemd/system/cri-docker.socket; enabled; vendor preset: disabled)
   Active: active (listening) since Wed 2023-11-08 12:26:58 KST; 22s ago
   Listen: /run/cri-dockerd.sock (Stream)

Nov 08 12:26:58 minhangk8s-01 systemd[1]: Starting CRI Docker Socket for the API.
Nov 08 12:26:58 minhangk8s-01 systemd[1]: Listening on CRI Docker Socket for the API.
-> 잘 떠있고,

[root@minhangk8s-01 ~]# ls -l /run/cri-dockerd.sock
srw-rw---- 1 root docker 0 Nov  8 12:26 /run/cri-dockerd.sock
-> 소켓 파일도 생겼음


  1. 쿠버네티스 yum repository 추가
[root@minhangk8s-01 ~]# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
> enabled=1
> gpgcheck=1
> gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
> exclude=kubelet kubeadm kubectl
> EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
  -> yum 명령어 사용을 위한 repo 파일 생성. EOF까지가 쭉 명령어임. root 계정으로 진행하고 있어서, 매뉴얼상의 명령어중 sudo는 뺐음

[root@minhangk8s-01 ~]# ls -l /etc/yum.repos.d/kubernetes.repo 
-rw-r--r-- 1 root root 282 Nov  8 12:11 /etc/yum.repos.d/kubernetes.repo
  -> repo 파일 생성된 것 확인

[root@minhangk8s-01 ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
  -> 상단에 실행한데로 파일 내용 들어간 것 확인
Error: Package: kubeadm-1.28.3-150500.1.1.x86_64 (kubernetes)
           Requires: cri-tools >= 1.28.0
           Available: cri-tools-1.25.0-1.amzn2.0.1.x86_64 (amzn2-core)
               cri-tools = 1.25.0-1.amzn2.0.1
           Available: cri-tools-1.26.1-1.amzn2.0.1.x86_64 (amzn2-core)
               cri-tools = 1.26.1-1.amzn2.0.1
           Available: cri-tools-1.26.1-1.amzn2.0.2.x86_64 (amzn2-core)
               cri-tools = 1.26.1-1.amzn2.0.2
           Available: cri-tools-1.26.1-1.amzn2.0.3.x86_64 (amzn2-core)
               cri-tools = 1.26.1-1.amzn2.0.3
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
  1.  kubelet, kubeadm, kubectl설치
[root@minhangk8s-01 ~]# yum -y install kubelet kubeadm kubectl --disableexclude=kubernetes
Loaded plugins: langpacks, priorities, update-motd
kubernetes                                                                                                                                                     | 1.4 kB  00:00:00     
kubernetes/x86_64/primary                                                                                                                                      | 137 kB  00:00:00     
kubernetes                                                                                                                                                                  1022/1022
11 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.28.2-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.8.6 for package: kubeadm-1.28.2-0.x86_64
--> Processing Dependency: cri-tools >= 1.19.0 for package: kubeadm-1.28.2-0.x86_64
---> Package kubectl.x86_64 0:1.28.2-0 will be installed
---> Package kubelet.x86_64 0:1.28.2-0 will be installed
--> Processing Dependency: socat for package: kubelet-1.28.2-0.x86_64
--> Processing Dependency: ebtables for package: kubelet-1.28.2-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.28.2-0.x86_64
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-5.amzn2.2 will be installed
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
---> Package cri-tools.x86_64 0:1.26.1-1.amzn2.0.3 will be installed
---> Package ebtables.x86_64 0:2.0.10-16.amzn2.0.1 will be installed
---> Package kubernetes-cni.x86_64 0:1.2.0-0 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.amzn2.0.1 will be installed
--> Running transaction check
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.amzn2.1 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.amzn2.1 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.amzn2.0.2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================================
 Package                                            Arch                               Version                                           Repository                              Size
======================================================================================================================================================================================
Installing:
 kubeadm                                            x86_64                             1.28.2-0                                          kubernetes                              11 M
 kubectl                                            x86_64                             1.28.2-0                                          kubernetes                              11 M
 kubelet                                            x86_64                             1.28.2-0                                          kubernetes                              21 M
Installing for dependencies:
 conntrack-tools                                    x86_64                             1.4.4-5.amzn2.2                                   amzn2-core                             186 k
 cri-tools                                          x86_64                             1.26.1-1.amzn2.0.3                                amzn2-core                              18 M
 ebtables                                           x86_64                             2.0.10-16.amzn2.0.1                               amzn2-core                             122 k
 kubernetes-cni                                     x86_64                             1.2.0-0                                           kubernetes                              17 M
 libnetfilter_cthelper                              x86_64                             1.0.0-10.amzn2.1                                  amzn2-core                              18 k
 libnetfilter_cttimeout                             x86_64                             1.0.0-6.amzn2.1                                   amzn2-core                              18 k
 libnetfilter_queue                                 x86_64                             1.0.2-2.amzn2.0.2                                 amzn2-core                              24 k
 socat                                              x86_64                             1.7.3.2-2.amzn2.0.1                               amzn2-core                             291 k

Transaction Summary
======================================================================================================================================================================================
Install  3 Packages (+8 Dependent packages)

Total download size: 78 M
Installed size: 326 M
Downloading packages:
(1/11): conntrack-tools-1.4.4-5.amzn2.2.x86_64.rpm                                                                                                             | 186 kB  00:00:00     
(2/11): ebtables-2.0.10-16.amzn2.0.1.x86_64.rpm                                                                                                                | 122 kB  00:00:00     
(3/11): cri-tools-1.26.1-1.amzn2.0.3.x86_64.rpm                                                                                                                |  18 MB  00:00:02     
warning: /var/cache/yum/x86_64/2/kubernetes/packages/cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Public key for cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm is not installed
(4/11): cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm                                                           |  11 MB  00:00:03     
(5/11): a24e42254b5a14b67b58c4633d29c27370c28ed6796a80c455a65acc813ff374-kubectl-1.28.2-0.x86_64.rpm                                                           |  11 MB  00:00:03     
(6/11): libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64.rpm                                                                                                      |  18 kB  00:00:00     
(7/11): libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64.rpm                                                                                                      |  18 kB  00:00:00     
(8/11): socat-1.7.3.2-2.amzn2.0.1.x86_64.rpm                                                                                                                   | 291 kB  00:00:00     
(9/11): libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64.rpm                                                                                                        |  24 kB  00:00:00     
(10/11): e1cae938e231bffa3618f5934a096bd85372ee9b1293081f5682a22fe873add8-kubelet-1.28.2-0.x86_64.rpm                                                          |  21 MB  00:00:09     
(11/11): 0f2a2afd740d476ad77c508847bad1f559afc2425816c1f2ce4432a62dfe0b9d-kubernetes-cni-1.2.0-0.x86_64.rpm                                                    |  17 MB  00:00:09     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                 4.6 MB/s |  78 MB  00:00:16     
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0x13EDEF05:
 Userid     : "Rapture Automatic Signing Key (cloud-rapture-signing-key-2022-03-07-08_01_01.pub)"
 Fingerprint: a362 b822 f6de dc65 2817 ea46 b53d c80d 13ed ef05
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Importing GPG key 0x3E1BA8D5:
 Userid     : "Google Cloud Packages RPM Signing Key <gc-team@google.com>"
 Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5
 From       : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64                                                                                                                     1/11 
  Installing : libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64                                                                                                                     2/11 
  Installing : libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64                                                                                                                       3/11 
  Installing : conntrack-tools-1.4.4-5.amzn2.2.x86_64                                                                                                                            4/11 
  Installing : ebtables-2.0.10-16.amzn2.0.1.x86_64                                                                                                                               5/11 
  Installing : cri-tools-1.26.1-1.amzn2.0.3.x86_64                                                                                                                               6/11 
  Installing : socat-1.7.3.2-2.amzn2.0.1.x86_64                                                                                                                                  7/11 
  Installing : kubernetes-cni-1.2.0-0.x86_64                                                                                                                                     8/11 
  Installing : kubelet-1.28.2-0.x86_64                                                                                                                                           9/11 
  Installing : kubectl-1.28.2-0.x86_64                                                                                                                                          10/11 
  Installing : kubeadm-1.28.2-0.x86_64                                                                                                                                          11/11 
  Verifying  : kubectl-1.28.2-0.x86_64                                                                                                                                           1/11 
  Verifying  : socat-1.7.3.2-2.amzn2.0.1.x86_64                                                                                                                                  2/11 
  Verifying  : kubernetes-cni-1.2.0-0.x86_64                                                                                                                                     3/11 
  Verifying  : cri-tools-1.26.1-1.amzn2.0.3.x86_64                                                                                                                               4/11 
  Verifying  : ebtables-2.0.10-16.amzn2.0.1.x86_64                                                                                                                               5/11 
  Verifying  : kubelet-1.28.2-0.x86_64                                                                                                                                           6/11 
  Verifying  : libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64                                                                                                                       7/11 
  Verifying  : conntrack-tools-1.4.4-5.amzn2.2.x86_64                                                                                                                            8/11 
  Verifying  : libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64                                                                                                                     9/11 
  Verifying  : libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64                                                                                                                    10/11 
  Verifying  : kubeadm-1.28.2-0.x86_64                                                                                                                                          11/11 

Installed:
  kubeadm.x86_64 0:1.28.2-0                                   kubectl.x86_64 0:1.28.2-0                                   kubelet.x86_64 0:1.28.2-0                                  

Dependency Installed:
  conntrack-tools.x86_64 0:1.4.4-5.amzn2.2         cri-tools.x86_64 0:1.26.1-1.amzn2.0.3            ebtables.x86_64 0:2.0.10-16.amzn2.0.1          kubernetes-cni.x86_64 0:1.2.0-0    
  libnetfilter_cthelper.x86_64 0:1.0.0-10.amzn2.1  libnetfilter_cttimeout.x86_64 0:1.0.0-6.amzn2.1  libnetfilter_queue.x86_64 0:1.0.2-2.amzn2.0.2  socat.x86_64 0:1.7.3.2-2.amzn2.0.1 

Complete!


  1. 설치된 것 확인
[root@minhangk8s-01 ~]# kubelet --version
Kubernetes v1.28.2
[root@minhangk8s-01 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:34:32Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
[root@minhangk8s-01 ~]# kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server localhost:8080 was refused - did you specify the right host or port?
  1. kubelet 구동
[root@minhangk8s-01 ~]# systemctl enable --now kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
  -> systemctl에 enable 시켜두어 매번 재부팅 때마다 자동으로 띄우도록 합니다.

[root@minhangk8s-01 ~]# ps -ef | grep kubelet
root      51718  35655  0 12:15 pts/1    00:00:00 grep --color=auto kubelet
  -> 헌데 확인해보면 프로세스가 떠있지 않는 상태입니다.

[root@minhangk8s-01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2023-11-08 12:15:56 KST; 325ms ago
     Docs: https://kubernetes.io/docs/
  Process: 51719 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 51719 (code=exited, status=1/FAILURE)

Nov 08 12:15:56 minhangk8s-01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 08 12:15:56 minhangk8s-01 systemd[1]: Unit kubelet.service entered failed state.
Nov 08 12:15:56 minhangk8s-01 systemd[1]: kubelet.service failed.
  -> status 확인해보면 뭔가 에러 발생해서 뜨지 않고 있다는 것을 알 수 있습니다. 아직초기화를 진행하지 않아서 그렇습니다.


kubeadm을 통한 kubernetes 초기화

(참조 문서: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ : 이 매뉴얼은 공식 다큐멘테이션에 따로 번역본이 없습니다.)

  1. kubeadm으로 init 명령 실행하기
[root@minhangk8s-01 ~]# kubeadm init --cri-socket=/run/cri-dockerd.sock --pod-network-cidr=10.244.0.0/16
W1108 12:29:05.512108   51329 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/cri-dockerd.sock". Please update your configuration!
[init] Using Kubernetes version: v1.28.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W1108 12:29:44.150350   51329 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local minhangk8s-01] and IPs [10.96.0.1 192.168.100.61]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost minhangk8s-01] and IPs [192.168.100.61 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost minhangk8s-01] and IPs [192.168.100.61 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.002834 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node minhangk8s-01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node minhangk8s-01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: u8tzqa.cmcz3h25j7p71nop
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.61:6443 --token u8tzqa.cmcz3h25j7p71nop \
	--discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86 
-> 정상적으로 초기화 완료
-> 설치 옵션으로 넣어준 cri-socket은 Kubernetes가 컨테이너와 통신하는데 사용할 CRI를 지정하는 옵션인데, 도커를 런타임으로 사용하기 위해 위에 추가로 설치한 cri-dockerd를 설정한 것입니다. 
-> pod-network-cidr은 이름 그대로 파드간의 통신에 사용될 CIDR설정값인데, flannel을 사용할 예정이기에, 10.244.0.0/16으로 설정했습니다. 이 값을 다르게 주면 flannel 설정시 에러 생깁니다.
-> 최하단의 discovery-token-ca-cert-hash 값은 다른 노드 가입시 사용해야 하므로 잘 저장해두어야 합니다.


  1. kubelet 구동 확인 초기화 전에는 떠있지 않았던 kubelet이 정상적으로 떠있는 것 확인합니다.
[root@minhangk8s-01 ~]# ps -ef | grep kubelet
root      52526  52464  3 12:30 ?        00:00:07 kube-apiserver --advertise-address=192.168.100.61 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root      52710      1  1 12:30 ?        00:00:03 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/cri-dockerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9
root      53198  35655  0 12:33 pts/1    00:00:00 grep --color=auto kubelet
-> 프로세스 떠있는 것 확인

[root@minhangk8s-01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2023-11-08 12:30:18 KST; 3min 41s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 52710 (kubelet)
    Tasks: 11
   Memory: 32.8M
   CGroup: /system.slice/kubelet.service
           └─52710 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml -...

Nov 08 12:33:11 minhangk8s-01 kubelet[52710]: E1108 12:33:11.964001   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:18 minhangk8s-01 kubelet[52710]: E1108 12:33:18.264261   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:23 minhangk8s-01 kubelet[52710]: E1108 12:33:23.266017   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:28 minhangk8s-01 kubelet[52710]: E1108 12:33:28.267073   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:33 minhangk8s-01 kubelet[52710]: E1108 12:33:33.268087   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:38 minhangk8s-01 kubelet[52710]: E1108 12:33:38.269858   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:43 minhangk8s-01 kubelet[52710]: E1108 12:33:43.270501   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:48 minhangk8s-01 kubelet[52710]: E1108 12:33:48.271156   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:53 minhangk8s-01 kubelet[52710]: E1108 12:33:53.271692   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:58 minhangk8s-01 kubelet[52710]: E1108 12:33:58.273299   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Hint: Some lines were ellipsized, use -l to show in full.
-> kubelet 서비스 정상 구동중인 것 확인


  1. KUBECONFIG 설정
[root@minhangk8s-01 ~]# kubectl get all
E1108 12:35:40.903079   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.903403   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.904862   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.906244   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.907781   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
  -> 설치가 완료되었음에도 kubectl 명령 결과가 이상하게 출력되는 상태일 것입니다.


[root@minhangk8s-01 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@minhangk8s-01 ~]# source ~/.bash_profile

  -> .bash_profile에 KUBECONFIG 환경변수를 추가해줍니다.

[root@minhangk8s-01 ~]# kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   5m45s
  -> 이제 kubectl 명령어가 정상적으로 출력됩니다.


  1. 파드와 컨테이너 확인
[root@minhangk8s-01 ~]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd5756b68-gw4q4                0/1     Pending   0          7m35s
kube-system   coredns-5dd5756b68-ks2c4                0/1     Pending   0          7m35s
kube-system   etcd-minhangk8s-01                      1/1     Running   0          7m48s
kube-system   kube-apiserver-minhangk8s-01            1/1     Running   0          7m50s
kube-system   kube-controller-manager-minhangk8s-01   1/1     Running   0          7m48s
kube-system   kube-proxy-bwrjc                        1/1     Running   0          7m35s
kube-system   kube-scheduler-minhangk8s-01            1/1     Running   0          7m48s
-> kubectl 명령어로 확인한 pod

[root@minhangk8s-01 ~]# docker ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS         PORTS     NAMES
0c0fe10ee8f0   bfc896cf80fb                "/usr/local/bin/kube…"   7 minutes ago   Up 7 minutes             k8s_kube-proxy_kube-proxy-bwrjc_kube-system_1c4bfd30-7db7-4dc0-89ba-803bb857091e_0
7523f9af8f34   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_kube-proxy-bwrjc_kube-system_1c4bfd30-7db7-4dc0-89ba-803bb857091e_0
c55dc870c16f   6d1b4fd1b182                "kube-scheduler --au…"   7 minutes ago   Up 7 minutes             k8s_kube-scheduler_kube-scheduler-minhangk8s-01_kube-system_0f5de60f9484ad67f17ef418d6d64bb5_0
d9e662dd268b   73deb9a3f702                "etcd --advertise-cl…"   7 minutes ago   Up 7 minutes             k8s_etcd_etcd-minhangk8s-01_kube-system_de3eb28788e39fb0f86973a987689bd9_0
348b6daee560   537434729123                "kube-apiserver --ad…"   7 minutes ago   Up 7 minutes             k8s_kube-apiserver_kube-apiserver-minhangk8s-01_kube-system_2bceb6d4ba4fe82879c476f08a0e8490_0
362ace28655f   10baa1ca1706                "kube-controller-man…"   7 minutes ago   Up 7 minutes             k8s_kube-controller-manager_kube-controller-manager-minhangk8s-01_kube-system_a6c1dd9fb363d3db6a4f846b945974d8_0
2ee236d3fec5   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_kube-apiserver-minhangk8s-01_kube-system_2bceb6d4ba4fe82879c476f08a0e8490_0
965514b764cf   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_kube-scheduler-minhangk8s-01_kube-system_0f5de60f9484ad67f17ef418d6d64bb5_0
f67e70a9d9c6   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_etcd-minhangk8s-01_kube-system_de3eb28788e39fb0f86973a987689bd9_0
a8ae04fa7359   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_kube-controller-manager-minhangk8s-01_kube-system_a6c1dd9fb363d3db6a4f846b945974d8_0
-> docker 명령어로 확인한 container. 도커 기반으로 잘 돌아가고 있음을 알 수 있습니다.


복사 붙여넣기를 위한 명령어 모음

위 과정을 따라해보시기 위한 분들을 위해, 편하게 복사&붙여넣기 할 수 있도록, 사용되었던 명령어들만 모아봤습니다.

yum -y update 

rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8

getenforce 

vi /etc/hosts
cat /etc/hosts

yum -y install docker tc

systemctl enable --now docker
docker ps

yum -y install git go

git clone https://github.com/Mirantis/cri-dockerd.git

cd cri-dockerd
make cri-dockerd

install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
install packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket
cd

systemctl status cri-docker.socket
ls -l /run/cri-dockerd.sock

cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF


ls -l /etc/yum.repos.d/kubernetes.repo 
cat /etc/yum.repos.d/kubernetes.repo


yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes


kubelet --version
kubeadm version
kubectl version


systemctl enable --now kubelet
ps -ef | grep kubelet
systemctl status kubelet


kubeadm init --cri-socket=/run/cri-dockerd.sock --pod-network-cidr=10.244.0.0/16


ps -ef | grep kubelet
systemctl status kubelet

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile
kubectl get all

kubectl get pod --all-namespaces
docker ps

Amazon Linux 2에 Kubernetes 1.28 설치 (Docker 기반) - 노드 조인

직전에 작성한 Amazon Linux 2에 Kubernetes 1.28 설치 (Docker 기반) - 노드 조인 문서에서는 Control Plane에서 kubectl 명령어가 동작하는 것 까지를 목표로 두었었다면, 이번 문서에서는 클러스터에 노드들을 조인시켜 실질적으로 k8s의 리소스들을 생성할 수 있도록 할 것입니다.

클러스터 구성

(참조 문서: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/)

  1. ControlPlane 초기화하기 -> 이 부분은 kubeadm init 명령 수행하는 것을 말하며, 설치 과정중에 여기까지 이미 진행한 상태입니다.


  2. 파드 네트워크 애드온 설치
[root@minhangk8s-01 ~]# kubectl get all --all-namespaces
NAMESPACE     NAME                                        READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-5dd5756b68-gw4q4                0/1     Pending   0          36m
kube-system   pod/coredns-5dd5756b68-ks2c4                0/1     Pending   0          36m
kube-system   pod/etcd-minhangk8s-01                      1/1     Running   0          36m
kube-system   pod/kube-apiserver-minhangk8s-01            1/1     Running   0          36m
kube-system   pod/kube-controller-manager-minhangk8s-01   1/1     Running   0          36m
kube-system   pod/kube-proxy-bwrjc                        1/1     Running   0          36m
kube-system   pod/kube-scheduler-minhangk8s-01            1/1     Running   0          36m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  36m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   36m

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   36m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   0/2     2            0           36m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5dd5756b68   2         2         0       36m

-> 기본 설치만 마무리된 상태에서는 아직 네트워크가 제대로 활성화 되지 않아 coredns가 기동되지 못하고 0/2 상태인 것을 확인할 수 있습니다.

[root@minhangk8s-01 ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

-> Flannel 설치. 명령어도 간단하고 설치도 금새 끝납니다.

  1. Flannel 설치 완료 확인
[root@minhangk8s-01 ~]# kubectl get all --all-namespaces
NAMESPACE      NAME                                        READY   STATUS    RESTARTS   AGE
kube-flannel   pod/kube-flannel-ds-jn2tw                   1/1     Running   0          49s
kube-system    pod/coredns-5dd5756b68-gw4q4                1/1     Running   0          37m
kube-system    pod/coredns-5dd5756b68-ks2c4                1/1     Running   0          37m
kube-system    pod/etcd-minhangk8s-01                      1/1     Running   0          38m
kube-system    pod/kube-apiserver-minhangk8s-01            1/1     Running   0          38m
kube-system    pod/kube-controller-manager-minhangk8s-01   1/1     Running   0          38m
kube-system    pod/kube-proxy-bwrjc                        1/1     Running   0          37m
kube-system    pod/kube-scheduler-minhangk8s-01            1/1     Running   0          38m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  38m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   38m

NAMESPACE      NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   daemonset.apps/kube-flannel-ds   1         1         1       1            1           <none>                   49s
kube-system    daemonset.apps/kube-proxy        1         1         1       1            1           kubernetes.io/os=linux   38m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           38m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5dd5756b68   2         2         2       37m
  1. 각 노드들에 kubelet, kubeadm, kubectl 설치 클러스터에 참여할 노드들에도 kubeadm 명령을 사용해야 됩니다. 일단 [[Amazon Linux 2에 Kubernetes 1.28 설치 (Docker 기반)]] 문서 참고하여 해당 명령어들을 사용할 수 있도록 설치 진행합니다.
# OS설정
yum -y update 
rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8
vi /etc/hosts

# docker 설치
yum -y install docker tc
systemctl enable --now docker

# cri-dockerd 설치
yum -y install git go

git clone https://github.com/Mirantis/cri-dockerd.git
cd cri-dockerd
make cri-dockerd

install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
install packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket

# kubernetes 설치
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

-> 한번 설치 진행해본 뒤라 확인하는 명령어들은 제거하고, 설치/변경하는 명령어만 넣었습니다.

  1. KUBECONFIG 설정
echo "export KUBECONFIG=/etc/kubernetes/kubelet.conf" >> ~/.bash_profile
source ~/.bash_profile

-> ControlPlane에서는 export KUBECONFIG=/etc/kubernetes/admin.conf였던 부분을 export KUBECONFIG=/etc/kubernetes/kubelet.conf으로 변경했습니다.

단, 현재 단계에서는 2,3번 노드에 설치만 진행했을 뿐 아직 클러스터에 참여된 것은 아닙니다. 따라서 아직 kubectl 명령어 수행 결과가 떨어지지는 않습니다.

[root@minhangk8s-02 ~]# kubectl get all
E1108 13:23:40.596490   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 13:23:40.596798   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 13:23:40.598349   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 13:23:40.599693   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 13:23:40.601164   82255 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

1번 노드(=ControlPlane)에서 아래와 같이 노드 정보를 조회해봐도 아직 1개 노드만 확인됩니다.

[root@minhangk8s-01 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
minhangk8s-01   Ready    control-plane   58m   v1.28.2


  1. 2번 노드에서 클러스터 Join 수행
...
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.61:6443 --token u8tzqa.cmcz3h25j7p71nop \
	--discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86 

-> kubeadm join에 출력 되었던 내용

[root@minhangk8s-02 ~]# kubeadm join 192.168.100.61:6443 --cri-socket=/run/cri-dockerd.sock --token u8tzqa.cmcz3h25j7p71nop --discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86
W1108 13:32:23.652234   82560 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/cri-dockerd.sock". Please update your configuration!
[preflight] Running pre-flight checks
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@minhangk8s-02 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE   VERSION
minhangk8s-01   Ready    control-plane   62m   v1.28.2
minhangk8s-02   Ready    <none>          32s   v1.28.2

-> 2번 노드에서 kubectl 명령어가 정상적으로 수행될 뿐 아니라, 추가된 2번 노드의 정보도 확인 가능합니다.

  1. 3번 노드에서 클러스터 Join 수행
[root@minhangk8s-03 ~]# kubeadm join 192.168.100.61:6443 --cri-socket=/run/cri-dockerd.sock --token u8tzqa.cmcz3h25j7p71nop --discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86
W1108 13:34:35.678983   82894 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/cri-dockerd.sock". Please update your configuration!
[preflight] Running pre-flight checks
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
[root@minhangk8s-03 ~]# kubectl get nodes
NAME            STATUS   ROLES           AGE    VERSION
minhangk8s-01   Ready    control-plane   65m    v1.28.2
minhangk8s-02   Ready    <none>          3m2s   v1.28.2
minhangk8s-03   Ready    <none>          47s    v1.28.2

-> 3번 노드에서 kubectl 명령어가 정상적으로 수행될 뿐 아니라, 추가된 3번 노드의 정보도 확인 가능합니다.

[root@minhangk8s-01 ~]# kubectl get all --all-namespaces
NAMESPACE      NAME                                        READY   STATUS    RESTARTS   AGE
kube-flannel   pod/kube-flannel-ds-95b45                   1/1     Running   0          103s
kube-flannel   pod/kube-flannel-ds-dfz2t                   1/1     Running   0          3m58s
kube-flannel   pod/kube-flannel-ds-jn2tw                   1/1     Running   0          28m
kube-system    pod/coredns-5dd5756b68-gw4q4                1/1     Running   0          65m
kube-system    pod/coredns-5dd5756b68-ks2c4                1/1     Running   0          65m
kube-system    pod/etcd-minhangk8s-01                      1/1     Running   0          66m
kube-system    pod/kube-apiserver-minhangk8s-01            1/1     Running   0          66m
kube-system    pod/kube-controller-manager-minhangk8s-01   1/1     Running   0          66m
kube-system    pod/kube-proxy-bltds                        1/1     Running   0          103s
kube-system    pod/kube-proxy-bwrjc                        1/1     Running   0          65m
kube-system    pod/kube-proxy-fcdlp                        1/1     Running   0          3m58s
kube-system    pod/kube-scheduler-minhangk8s-01            1/1     Running   0          66m

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  66m
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   66m

NAMESPACE      NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-flannel   daemonset.apps/kube-flannel-ds   3         3         3       3            3           <none>                   28m
kube-system    daemonset.apps/kube-proxy        3         3         3       3            3           kubernetes.io/os=linux   66m

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   2/2     2            2           66m

NAMESPACE     NAME                                 DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-5dd5756b68   2         2         2       65m

-> 막 flannel을 설치한 직후와 달리, flannel에 연결된 노드도 3개씩이 되어 있습니다.

설치 완료


복사 붙여넣기를 위한 명령어 모음

위 과정을 따라해보시기 위한 분들을 위해, 편하게 복사&붙여넣기 할 수 있도록, 사용되었던 명령어들만 모아봤습니다.

### 1번 노드에서만 수행 = flannel 설치 ##########
kubectl get all --all-namespaces
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl get all --all-namespaces

### 2,3번 노드에서 각각 수행 = kubeadm등 설치 ####
# OS설정
yum -y update 
rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8
vi /etc/hosts

# docker 설치
yum -y install docker tc
systemctl enable --now docker

# cri-dockerd 설치
yum -y install git go

git clone https://github.com/Mirantis/cri-dockerd.git
cd cri-dockerd
make cri-dockerd

install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
install packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket

# kubernetes 설치
cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

systemctl enable --now kubelet

echo "export KUBECONFIG=/etc/kubernetes/kubelet.conf" >> ~/.bash_profile
source ~/.bash_profile

kubectl get all
kubectl get nodes


### 2,3번 노드에서 각각 수행 = 클러스터 가입 #######
# 해시 키 값은 init 할때 마다 달라집니다. 밑에 값을 그대로 복사하지 말고, 각자 환경에 맞는 키값으로 복사하셔야합니다. 
kubeadm join 192.168.100.61:6443 --cri-socket=/run/cri-dockerd.sock --token u8tzqa.cmcz3h25j7p71nop --discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86 
kubectl get nodes

### 1번 노드에서 최종 확인 ######################
kubectl get all --all-namespaces