Amazon Linux 2에 Kubernetes 1.28 설치 (Docker 기반)

쿠버네티스 공식 다큐멘테이션 내의 kubeadm으로 설치하는 문서를 기반으로 작성하였습니다. (참조 문서: https://kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/)

[root@minhangk8s-01 ~]# cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
Amazon Linux release 2 (Karoo)

kubeadm을 통한 kubernetes 설치

Amazon Linux는 Redhat Linux 계열 OS입니다. Red Hat-based distributions를 따라 설치 진행합니다.

  1. OS 기본 설정들
[root@minhangk8s-01 ~]# yum -y update 
Loaded plugins: langpacks, priorities, update-motd
amzn2-core                                                                                                                                                     | 3.6 kB  00:00:00     
amzn2extra-docker                                                                                                                                              | 2.9 kB  00:00:00     
(1/4): amzn2extra-docker/2/x86_64/updateinfo                                                                                                                   |  13 kB  00:00:00     
(2/4): amzn2extra-docker/2/x86_64/primary_db         
...
Replaced:
  grub2.x86_64 1:2.06-14.amzn2.0.1                                                       grub2-tools.x86_64 1:2.06-14.amzn2.0.1                                                      

Complete!
[root@minhangk8s-01 ~]# rm -f /etc/localtime

[root@minhangk8s-01 ~]# ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime

[root@minhangk8s-01 ~]# localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
/usr/share/i18n/locales/en_US:15: non-symbolic character value should not be used
/usr/share/i18n/locales/en_US:16: non-symbolic character value should not be used
/usr/share/i18n/locales/en_US:17: non-symbolic character value should not be used
...
LC_CTYPE: table for map "tolower": 139850954557471 bytes
LC_CTYPE: table for map "totitle": 0 bytes
LC_CTYPE: table for width: 0 bytes

[root@minhangk8s-01 ~]# localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8
/usr/share/i18n/locales/ko_KR:48: non-symbolic character value should not be used
/usr/share/i18n/locales/ko_KR:52: non-symbolic character value should not be used
/usr/share/i18n/locales/ko_KR:55: non-symbolic character value should not be used
...
LC_CTYPE: table for map "tolower": 140086909933599 bytes
LC_CTYPE: table for map "totitle": 0 bytes
LC_CTYPE: table for width: 0 bytes
[root@minhangk8s-01 ~]# getenforce 
Disabled

굳이 더 보안 등급이 높은 Permissive 모드로 올리지 않고, disabled 상태로 두고 진행하면 됩니다.

[root@minhangk8s-01 ~]# vi /etc/hosts

[root@minhangk8s-01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost6 localhost6.localdomain6

192.168.100.161 MinhangK8S-01
192.168.100.162 MinhangK8S-02
192.168.100.163 MinhangK8S-03
192.168.100.164 MinhangK8S-04
192.168.100.165 MinhangK8S-05

-> vi 편집기를 통해 위와 같이 k8s 설치에 사용할 호스트명들을 집어 넣었습니다.

  1. Container Runtime 설치 Container Runtime으로Docker를 설치합니다. (나중에 kubernetes 초기화시 필요한 tc 패키지도 같이 설치해줍니다.)
[root@minhangk8s-01 ~]# yum -y install docker tc
Loaded plugins: langpacks, priorities, update-motd
Resolving Dependencies
...

======================================================================================================================================================================================
 Package                                 Arch                                Version                                             Repository                                      Size
======================================================================================================================================================================================
Installing:
 docker                                  x86_64                              20.10.25-1.amzn2.0.3                                amzn2extra-docker                               43 M
 iproute-tc                              x86_64                              5.10.0-2.amzn2.0.3                                  amzn2-core                                     432 k
Installing for dependencies:
 containerd                              x86_64                              1.6.19-1.amzn2.0.5                                  amzn2extra-docker                               28 M
 libcgroup                               x86_64                              0.41-21.amzn2                                       amzn2-core                                      66 k
 pigz                                    x86_64                              2.3.4-1.amzn2.0.1                                   amzn2-core                                      81 k
 runc                                    x86_64                              1.1.7-4.amzn2                                       amzn2extra-docker                              3.0 M

Transaction Summary
======================================================================================================================================================================================
Install  2 Packages (+4 Dependent packages)
 
...
Installed:
  docker.x86_64 0:20.10.25-1.amzn2.0.3                                                     iproute-tc.x86_64 0:5.10.0-2.amzn2.0.3                                                    

Dependency Installed:
  containerd.x86_64 0:1.6.19-1.amzn2.0.5             libcgroup.x86_64 0:0.41-21.amzn2             pigz.x86_64 0:2.3.4-1.amzn2.0.1             runc.x86_64 0:1.1.7-4.amzn2            

Complete!



[root@minhangk8s-01 ~]# systemctl enable --now docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.
-> 도커 구동

[root@minhangk8s-01 ~]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
-> 도커 명령어 동작하는 것 확인


  1. cri-dockerd 설치 Docker를 Kubernetes의 CRI(Container Runtime Interface)로 사용하기 위해 cri-dockerd를 추가로 설치해줍니다. 이 부분은 공식 매뉴얼에도 언급이 되어 있는데, Docker가 Kubernetes와 호환되기 위한 CRI 요구 사항을 만족하지 못하므로, Kubernetes 1.24부터 Docker 기본 상태로는 사용이 불가능하고, cri-dockerd라는 별도의 프로젝트를 통해 이 부분이 구현되어 있다는 것입니다. 따라서 cri-dockerd를 추가로 설치해주어야 docker 기반으로 kubernetes 사용이 가능합니다.
[root@minhangk8s-01 ~]# yum -y install git go
Loaded plugins: langpacks, priorities, update-motd
amzn2-core                                                                                                                                                     | 3.6 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
...
======================================================================================================================================================================================
 Package                                       Arch                                Version                                              Repository                               Size
======================================================================================================================================================================================
Installing:
 git                                           x86_64                              2.40.1-1.amzn2.0.1                                   amzn2-core                               54 k
 golang                                        x86_64                              1.20.10-1.amzn2.0.1                                  amzn2-core                              682 k
...

Complete!
[root@minhangk8s-01 ~]# git clone https://github.com/Mirantis/cri-dockerd.git
Cloning into 'cri-dockerd'...
remote: Enumerating objects: 18364, done.
remote: Counting objects: 100% (2923/2923), done.
remote: Compressing objects: 100% (1100/1100), done.
remote: Total 18364 (delta 2062), reused 1904 (delta 1810), pack-reused 15441
Receiving objects: 100% (18364/18364), 42.91 MiB | 20.22 MiB/s, done.
Resolving deltas: 100% (9251/9251), done.
[root@minhangk8s-01 cri-dockerd]# make cri-dockerd
GOARCH= go build -trimpath -ldflags " -s -w -buildid=`git log -1 --pretty='%h'` -X github.com/Mirantis/cri-dockerd/cmd/version.Version=0.3.7 -X github.com/Mirantis/cri-dockerd/cmd/version.PreRelease=`grep -q dev <<< "0.3.7" && echo "pre" || echo ""` -X github.com/Mirantis/cri-dockerd/cmd/version.GitCommit=`git log -1 --pretty='%h'`" -o cri-dockerd
[root@minhangk8s-01 cri-dockerd]# install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
[root@minhangk8s-01 cri-dockerd]# install packaging/systemd/* /etc/systemd/system
[root@minhangk8s-01 cri-dockerd]# sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
[root@minhangk8s-01 cri-dockerd]# systemctl daemon-reload
[root@minhangk8s-01 cri-dockerd]# systemctl enable --now cri-docker.socket
Created symlink from /etc/systemd/system/sockets.target.wants/cri-docker.socket to /etc/systemd/system/cri-docker.socket.
[root@minhangk8s-01 cri-dockerd]# cd
[root@minhangk8s-01 ~]# 
[root@minhangk8s-01 ~]# systemctl status cri-docker.socket
● cri-docker.socket - CRI Docker Socket for the API
   Loaded: loaded (/etc/systemd/system/cri-docker.socket; enabled; vendor preset: disabled)
   Active: active (listening) since Wed 2023-11-08 12:26:58 KST; 22s ago
   Listen: /run/cri-dockerd.sock (Stream)

Nov 08 12:26:58 minhangk8s-01 systemd[1]: Starting CRI Docker Socket for the API.
Nov 08 12:26:58 minhangk8s-01 systemd[1]: Listening on CRI Docker Socket for the API.
-> 잘 떠있고,

[root@minhangk8s-01 ~]# ls -l /run/cri-dockerd.sock
srw-rw---- 1 root docker 0 Nov  8 12:26 /run/cri-dockerd.sock
-> 소켓 파일도 생겼음


  1. 쿠버네티스 yum repository 추가
[root@minhangk8s-01 ~]# cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
> [kubernetes]
> name=Kubernetes
> baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
> enabled=1
> gpgcheck=1
> gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
> exclude=kubelet kubeadm kubectl
> EOF
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
  -> yum 명령어 사용을 위한 repo 파일 생성. EOF까지가 쭉 명령어임. root 계정으로 진행하고 있어서, 매뉴얼상의 명령어중 sudo는 뺐음

[root@minhangk8s-01 ~]# ls -l /etc/yum.repos.d/kubernetes.repo 
-rw-r--r-- 1 root root 282 Nov  8 12:11 /etc/yum.repos.d/kubernetes.repo
  -> repo 파일 생성된 것 확인

[root@minhangk8s-01 ~]# cat /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
  -> 상단에 실행한데로 파일 내용 들어간 것 확인
Error: Package: kubeadm-1.28.3-150500.1.1.x86_64 (kubernetes)
           Requires: cri-tools >= 1.28.0
           Available: cri-tools-1.25.0-1.amzn2.0.1.x86_64 (amzn2-core)
               cri-tools = 1.25.0-1.amzn2.0.1
           Available: cri-tools-1.26.1-1.amzn2.0.1.x86_64 (amzn2-core)
               cri-tools = 1.26.1-1.amzn2.0.1
           Available: cri-tools-1.26.1-1.amzn2.0.2.x86_64 (amzn2-core)
               cri-tools = 1.26.1-1.amzn2.0.2
           Available: cri-tools-1.26.1-1.amzn2.0.3.x86_64 (amzn2-core)
               cri-tools = 1.26.1-1.amzn2.0.3
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
  1.  kubelet, kubeadm, kubectl설치
[root@minhangk8s-01 ~]# yum -y install kubelet kubeadm kubectl --disableexclude=kubernetes
Loaded plugins: langpacks, priorities, update-motd
kubernetes                                                                                                                                                     | 1.4 kB  00:00:00     
kubernetes/x86_64/primary                                                                                                                                      | 137 kB  00:00:00     
kubernetes                                                                                                                                                                  1022/1022
11 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package kubeadm.x86_64 0:1.28.2-0 will be installed
--> Processing Dependency: kubernetes-cni >= 0.8.6 for package: kubeadm-1.28.2-0.x86_64
--> Processing Dependency: cri-tools >= 1.19.0 for package: kubeadm-1.28.2-0.x86_64
---> Package kubectl.x86_64 0:1.28.2-0 will be installed
---> Package kubelet.x86_64 0:1.28.2-0 will be installed
--> Processing Dependency: socat for package: kubelet-1.28.2-0.x86_64
--> Processing Dependency: ebtables for package: kubelet-1.28.2-0.x86_64
--> Processing Dependency: conntrack for package: kubelet-1.28.2-0.x86_64
--> Running transaction check
---> Package conntrack-tools.x86_64 0:1.4.4-5.amzn2.2 will be installed
--> Processing Dependency: libnetfilter_cthelper.so.0(LIBNETFILTER_CTHELPER_1.0)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.0)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1(LIBNETFILTER_CTTIMEOUT_1.1)(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cthelper.so.0()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_cttimeout.so.1()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
--> Processing Dependency: libnetfilter_queue.so.1()(64bit) for package: conntrack-tools-1.4.4-5.amzn2.2.x86_64
---> Package cri-tools.x86_64 0:1.26.1-1.amzn2.0.3 will be installed
---> Package ebtables.x86_64 0:2.0.10-16.amzn2.0.1 will be installed
---> Package kubernetes-cni.x86_64 0:1.2.0-0 will be installed
---> Package socat.x86_64 0:1.7.3.2-2.amzn2.0.1 will be installed
--> Running transaction check
---> Package libnetfilter_cthelper.x86_64 0:1.0.0-10.amzn2.1 will be installed
---> Package libnetfilter_cttimeout.x86_64 0:1.0.0-6.amzn2.1 will be installed
---> Package libnetfilter_queue.x86_64 0:1.0.2-2.amzn2.0.2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================================
 Package                                            Arch                               Version                                           Repository                              Size
======================================================================================================================================================================================
Installing:
 kubeadm                                            x86_64                             1.28.2-0                                          kubernetes                              11 M
 kubectl                                            x86_64                             1.28.2-0                                          kubernetes                              11 M
 kubelet                                            x86_64                             1.28.2-0                                          kubernetes                              21 M
Installing for dependencies:
 conntrack-tools                                    x86_64                             1.4.4-5.amzn2.2                                   amzn2-core                             186 k
 cri-tools                                          x86_64                             1.26.1-1.amzn2.0.3                                amzn2-core                              18 M
 ebtables                                           x86_64                             2.0.10-16.amzn2.0.1                               amzn2-core                             122 k
 kubernetes-cni                                     x86_64                             1.2.0-0                                           kubernetes                              17 M
 libnetfilter_cthelper                              x86_64                             1.0.0-10.amzn2.1                                  amzn2-core                              18 k
 libnetfilter_cttimeout                             x86_64                             1.0.0-6.amzn2.1                                   amzn2-core                              18 k
 libnetfilter_queue                                 x86_64                             1.0.2-2.amzn2.0.2                                 amzn2-core                              24 k
 socat                                              x86_64                             1.7.3.2-2.amzn2.0.1                               amzn2-core                             291 k

Transaction Summary
======================================================================================================================================================================================
Install  3 Packages (+8 Dependent packages)

Total download size: 78 M
Installed size: 326 M
Downloading packages:
(1/11): conntrack-tools-1.4.4-5.amzn2.2.x86_64.rpm                                                                                                             | 186 kB  00:00:00     
(2/11): ebtables-2.0.10-16.amzn2.0.1.x86_64.rpm                                                                                                                | 122 kB  00:00:00     
(3/11): cri-tools-1.26.1-1.amzn2.0.3.x86_64.rpm                                                                                                                |  18 MB  00:00:02     
warning: /var/cache/yum/x86_64/2/kubernetes/packages/cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm: Header V4 RSA/SHA512 Signature, key ID 3e1ba8d5: NOKEY
Public key for cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm is not installed
(4/11): cee73f8035d734e86f722f77f1bf4e7d643e78d36646fd000148deb8af98b61c-kubeadm-1.28.2-0.x86_64.rpm                                                           |  11 MB  00:00:03     
(5/11): a24e42254b5a14b67b58c4633d29c27370c28ed6796a80c455a65acc813ff374-kubectl-1.28.2-0.x86_64.rpm                                                           |  11 MB  00:00:03     
(6/11): libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64.rpm                                                                                                      |  18 kB  00:00:00     
(7/11): libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64.rpm                                                                                                      |  18 kB  00:00:00     
(8/11): socat-1.7.3.2-2.amzn2.0.1.x86_64.rpm                                                                                                                   | 291 kB  00:00:00     
(9/11): libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64.rpm                                                                                                        |  24 kB  00:00:00     
(10/11): e1cae938e231bffa3618f5934a096bd85372ee9b1293081f5682a22fe873add8-kubelet-1.28.2-0.x86_64.rpm                                                          |  21 MB  00:00:09     
(11/11): 0f2a2afd740d476ad77c508847bad1f559afc2425816c1f2ce4432a62dfe0b9d-kubernetes-cni-1.2.0-0.x86_64.rpm                                                    |  17 MB  00:00:09     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                 4.6 MB/s |  78 MB  00:00:16     
Retrieving key from https://packages.cloud.google.com/yum/doc/yum-key.gpg
Importing GPG key 0x13EDEF05:
 Userid     : "Rapture Automatic Signing Key (cloud-rapture-signing-key-2022-03-07-08_01_01.pub)"
 Fingerprint: a362 b822 f6de dc65 2817 ea46 b53d c80d 13ed ef05
 From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Retrieving key from https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Importing GPG key 0x3E1BA8D5:
 Userid     : "Google Cloud Packages RPM Signing Key <gc-team@google.com>"
 Fingerprint: 3749 e1ba 95a8 6ce0 5454 6ed2 f09c 394c 3e1b a8d5
 From       : https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64                                                                                                                     1/11 
  Installing : libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64                                                                                                                     2/11 
  Installing : libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64                                                                                                                       3/11 
  Installing : conntrack-tools-1.4.4-5.amzn2.2.x86_64                                                                                                                            4/11 
  Installing : ebtables-2.0.10-16.amzn2.0.1.x86_64                                                                                                                               5/11 
  Installing : cri-tools-1.26.1-1.amzn2.0.3.x86_64                                                                                                                               6/11 
  Installing : socat-1.7.3.2-2.amzn2.0.1.x86_64                                                                                                                                  7/11 
  Installing : kubernetes-cni-1.2.0-0.x86_64                                                                                                                                     8/11 
  Installing : kubelet-1.28.2-0.x86_64                                                                                                                                           9/11 
  Installing : kubectl-1.28.2-0.x86_64                                                                                                                                          10/11 
  Installing : kubeadm-1.28.2-0.x86_64                                                                                                                                          11/11 
  Verifying  : kubectl-1.28.2-0.x86_64                                                                                                                                           1/11 
  Verifying  : socat-1.7.3.2-2.amzn2.0.1.x86_64                                                                                                                                  2/11 
  Verifying  : kubernetes-cni-1.2.0-0.x86_64                                                                                                                                     3/11 
  Verifying  : cri-tools-1.26.1-1.amzn2.0.3.x86_64                                                                                                                               4/11 
  Verifying  : ebtables-2.0.10-16.amzn2.0.1.x86_64                                                                                                                               5/11 
  Verifying  : kubelet-1.28.2-0.x86_64                                                                                                                                           6/11 
  Verifying  : libnetfilter_queue-1.0.2-2.amzn2.0.2.x86_64                                                                                                                       7/11 
  Verifying  : conntrack-tools-1.4.4-5.amzn2.2.x86_64                                                                                                                            8/11 
  Verifying  : libnetfilter_cttimeout-1.0.0-6.amzn2.1.x86_64                                                                                                                     9/11 
  Verifying  : libnetfilter_cthelper-1.0.0-10.amzn2.1.x86_64                                                                                                                    10/11 
  Verifying  : kubeadm-1.28.2-0.x86_64                                                                                                                                          11/11 

Installed:
  kubeadm.x86_64 0:1.28.2-0                                   kubectl.x86_64 0:1.28.2-0                                   kubelet.x86_64 0:1.28.2-0                                  

Dependency Installed:
  conntrack-tools.x86_64 0:1.4.4-5.amzn2.2         cri-tools.x86_64 0:1.26.1-1.amzn2.0.3            ebtables.x86_64 0:2.0.10-16.amzn2.0.1          kubernetes-cni.x86_64 0:1.2.0-0    
  libnetfilter_cthelper.x86_64 0:1.0.0-10.amzn2.1  libnetfilter_cttimeout.x86_64 0:1.0.0-6.amzn2.1  libnetfilter_queue.x86_64 0:1.0.2-2.amzn2.0.2  socat.x86_64 0:1.7.3.2-2.amzn2.0.1 

Complete!


  1. 설치된 것 확인
[root@minhangk8s-01 ~]# kubelet --version
Kubernetes v1.28.2
[root@minhangk8s-01 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.2", GitCommit:"89a4ea3e1e4ddd7f7572286090359983e0387b2f", GitTreeState:"clean", BuildDate:"2023-09-13T09:34:32Z", GoVersion:"go1.20.8", Compiler:"gc", Platform:"linux/amd64"}
[root@minhangk8s-01 ~]# kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
The connection to the server localhost:8080 was refused - did you specify the right host or port?
  1. kubelet 구동
[root@minhangk8s-01 ~]# systemctl enable --now kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
  -> systemctl에 enable 시켜두어 매번 재부팅 때마다 자동으로 띄우도록 합니다.

[root@minhangk8s-01 ~]# ps -ef | grep kubelet
root      51718  35655  0 12:15 pts/1    00:00:00 grep --color=auto kubelet
  -> 헌데 확인해보면 프로세스가 떠있지 않는 상태입니다.

[root@minhangk8s-01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Wed 2023-11-08 12:15:56 KST; 325ms ago
     Docs: https://kubernetes.io/docs/
  Process: 51719 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 51719 (code=exited, status=1/FAILURE)

Nov 08 12:15:56 minhangk8s-01 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
Nov 08 12:15:56 minhangk8s-01 systemd[1]: Unit kubelet.service entered failed state.
Nov 08 12:15:56 minhangk8s-01 systemd[1]: kubelet.service failed.
  -> status 확인해보면 뭔가 에러 발생해서 뜨지 않고 있다는 것을 알 수 있습니다. 아직초기화를 진행하지 않아서 그렇습니다.


kubeadm을 통한 kubernetes 초기화

(참조 문서: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/ : 이 매뉴얼은 공식 다큐멘테이션에 따로 번역본이 없습니다.)

  1. kubeadm으로 init 명령 실행하기
[root@minhangk8s-01 ~]# kubeadm init --cri-socket=/run/cri-dockerd.sock --pod-network-cidr=10.244.0.0/16
W1108 12:29:05.512108   51329 initconfiguration.go:120] Usage of CRI endpoints without URL scheme is deprecated and can cause kubelet errors in the future. Automatically prepending scheme "unix" to the "criSocket" with value "/run/cri-dockerd.sock". Please update your configuration!
[init] Using Kubernetes version: v1.28.3
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
W1108 12:29:44.150350   51329 checks.go:835] detected that the sandbox image "registry.k8s.io/pause:3.6" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image.
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local minhangk8s-01] and IPs [10.96.0.1 192.168.100.61]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost minhangk8s-01] and IPs [192.168.100.61 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost minhangk8s-01] and IPs [192.168.100.61 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 9.002834 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node minhangk8s-01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node minhangk8s-01 as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: u8tzqa.cmcz3h25j7p71nop
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.100.61:6443 --token u8tzqa.cmcz3h25j7p71nop \
	--discovery-token-ca-cert-hash sha256:6706ae21c7bf93aad4bc3dae8194d5e4a845bf0b5f2fbce03b5666b68f7a3d86 
-> 정상적으로 초기화 완료
-> 설치 옵션으로 넣어준 cri-socket은 Kubernetes가 컨테이너와 통신하는데 사용할 CRI를 지정하는 옵션인데, 도커를 런타임으로 사용하기 위해 위에 추가로 설치한 cri-dockerd를 설정한 것입니다. 
-> pod-network-cidr은 이름 그대로 파드간의 통신에 사용될 CIDR설정값인데, flannel을 사용할 예정이기에, 10.244.0.0/16으로 설정했습니다. 이 값을 다르게 주면 flannel 설정시 에러 생깁니다.
-> 최하단의 discovery-token-ca-cert-hash 값은 다른 노드 가입시 사용해야 하므로 잘 저장해두어야 합니다.


  1. kubelet 구동 확인 초기화 전에는 떠있지 않았던 kubelet이 정상적으로 떠있는 것 확인합니다.
[root@minhangk8s-01 ~]# ps -ef | grep kubelet
root      52526  52464  3 12:30 ?        00:00:07 kube-apiserver --advertise-address=192.168.100.61 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
root      52710      1  1 12:30 ?        00:00:03 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///run/cri-dockerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9
root      53198  35655  0 12:33 pts/1    00:00:00 grep --color=auto kubelet
-> 프로세스 떠있는 것 확인

[root@minhangk8s-01 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2023-11-08 12:30:18 KST; 3min 41s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 52710 (kubelet)
    Tasks: 11
   Memory: 32.8M
   CGroup: /system.slice/kubelet.service
           └─52710 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml -...

Nov 08 12:33:11 minhangk8s-01 kubelet[52710]: E1108 12:33:11.964001   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:18 minhangk8s-01 kubelet[52710]: E1108 12:33:18.264261   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:23 minhangk8s-01 kubelet[52710]: E1108 12:33:23.266017   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:28 minhangk8s-01 kubelet[52710]: E1108 12:33:28.267073   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:33 minhangk8s-01 kubelet[52710]: E1108 12:33:33.268087   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:38 minhangk8s-01 kubelet[52710]: E1108 12:33:38.269858   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:43 minhangk8s-01 kubelet[52710]: E1108 12:33:43.270501   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:48 minhangk8s-01 kubelet[52710]: E1108 12:33:48.271156   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:53 minhangk8s-01 kubelet[52710]: E1108 12:33:53.271692   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Nov 08 12:33:58 minhangk8s-01 kubelet[52710]: E1108 12:33:58.273299   52710 kubelet.go:2855] "Container runtime network not ready" networkReady="NetworkReady=false reas...nitialized"
Hint: Some lines were ellipsized, use -l to show in full.
-> kubelet 서비스 정상 구동중인 것 확인


  1. KUBECONFIG 설정
[root@minhangk8s-01 ~]# kubectl get all
E1108 12:35:40.903079   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.903403   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.904862   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.906244   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1108 12:35:40.907781   53277 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?
  -> 설치가 완료되었음에도 kubectl 명령 결과가 이상하게 출력되는 상태일 것입니다.


[root@minhangk8s-01 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@minhangk8s-01 ~]# source ~/.bash_profile

  -> .bash_profile에 KUBECONFIG 환경변수를 추가해줍니다.

[root@minhangk8s-01 ~]# kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   5m45s
  -> 이제 kubectl 명령어가 정상적으로 출력됩니다.


  1. 파드와 컨테이너 확인
[root@minhangk8s-01 ~]# kubectl get pod --all-namespaces
NAMESPACE     NAME                                    READY   STATUS    RESTARTS   AGE
kube-system   coredns-5dd5756b68-gw4q4                0/1     Pending   0          7m35s
kube-system   coredns-5dd5756b68-ks2c4                0/1     Pending   0          7m35s
kube-system   etcd-minhangk8s-01                      1/1     Running   0          7m48s
kube-system   kube-apiserver-minhangk8s-01            1/1     Running   0          7m50s
kube-system   kube-controller-manager-minhangk8s-01   1/1     Running   0          7m48s
kube-system   kube-proxy-bwrjc                        1/1     Running   0          7m35s
kube-system   kube-scheduler-minhangk8s-01            1/1     Running   0          7m48s
-> kubectl 명령어로 확인한 pod

[root@minhangk8s-01 ~]# docker ps
CONTAINER ID   IMAGE                       COMMAND                  CREATED         STATUS         PORTS     NAMES
0c0fe10ee8f0   bfc896cf80fb                "/usr/local/bin/kube…"   7 minutes ago   Up 7 minutes             k8s_kube-proxy_kube-proxy-bwrjc_kube-system_1c4bfd30-7db7-4dc0-89ba-803bb857091e_0
7523f9af8f34   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_kube-proxy-bwrjc_kube-system_1c4bfd30-7db7-4dc0-89ba-803bb857091e_0
c55dc870c16f   6d1b4fd1b182                "kube-scheduler --au…"   7 minutes ago   Up 7 minutes             k8s_kube-scheduler_kube-scheduler-minhangk8s-01_kube-system_0f5de60f9484ad67f17ef418d6d64bb5_0
d9e662dd268b   73deb9a3f702                "etcd --advertise-cl…"   7 minutes ago   Up 7 minutes             k8s_etcd_etcd-minhangk8s-01_kube-system_de3eb28788e39fb0f86973a987689bd9_0
348b6daee560   537434729123                "kube-apiserver --ad…"   7 minutes ago   Up 7 minutes             k8s_kube-apiserver_kube-apiserver-minhangk8s-01_kube-system_2bceb6d4ba4fe82879c476f08a0e8490_0
362ace28655f   10baa1ca1706                "kube-controller-man…"   7 minutes ago   Up 7 minutes             k8s_kube-controller-manager_kube-controller-manager-minhangk8s-01_kube-system_a6c1dd9fb363d3db6a4f846b945974d8_0
2ee236d3fec5   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_kube-apiserver-minhangk8s-01_kube-system_2bceb6d4ba4fe82879c476f08a0e8490_0
965514b764cf   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_kube-scheduler-minhangk8s-01_kube-system_0f5de60f9484ad67f17ef418d6d64bb5_0
f67e70a9d9c6   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_etcd-minhangk8s-01_kube-system_de3eb28788e39fb0f86973a987689bd9_0
a8ae04fa7359   registry.k8s.io/pause:3.6   "/pause"                 7 minutes ago   Up 7 minutes             k8s_POD_kube-controller-manager-minhangk8s-01_kube-system_a6c1dd9fb363d3db6a4f846b945974d8_0
-> docker 명령어로 확인한 container. 도커 기반으로 잘 돌아가고 있음을 알 수 있습니다.


복사 붙여넣기를 위한 명령어 모음

위 과정을 따라해보시기 위한 분들을 위해, 편하게 복사&붙여넣기 할 수 있도록, 사용되었던 명령어들만 모아봤습니다.

yum -y update 

rm -f /etc/localtime
ln -s /usr/share/zoneinfo/Asia/Seoul /etc/localtime
localedef -v -c -i en_US -f UTF-8 en_US.UTF-8
localedef -v -c -i ko_KR -f UTF-8 ko_KR.UTF-8

getenforce 

vi /etc/hosts
cat /etc/hosts

yum -y install docker tc

systemctl enable --now docker
docker ps

yum -y install git go

git clone https://github.com/Mirantis/cri-dockerd.git

cd cri-dockerd
make cri-dockerd

install -o root -g root -m 0755 cri-dockerd /usr/local/bin/cri-dockerd
install packaging/systemd/* /etc/systemd/system
sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
systemctl daemon-reload
systemctl enable --now cri-docker.socket
cd

systemctl status cri-docker.socket
ls -l /run/cri-dockerd.sock

cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
exclude=kubelet kubeadm kubectl
EOF


ls -l /etc/yum.repos.d/kubernetes.repo 
cat /etc/yum.repos.d/kubernetes.repo


yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes


kubelet --version
kubeadm version
kubectl version


systemctl enable --now kubelet
ps -ef | grep kubelet
systemctl status kubelet


kubeadm init --cri-socket=/run/cri-dockerd.sock --pod-network-cidr=10.244.0.0/16


ps -ef | grep kubelet
systemctl status kubelet

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
source ~/.bash_profile
kubectl get all

kubectl get pod --all-namespaces
docker ps

Revision #2
Created 8 November 2023 04:50:33 by 신민항
Updated 8 November 2023 04:59:49 by 신민항