Date | Time | Title | Content |
---|---|---|---|
第 1 天 | 上午 | 1. 操作系统 | 1.1 CentOS 7 |
1.4 Rocky 和其它选择(1.2-1.8) | |||
1.9 最佳实践 | |||
2. 容器运行时 | 2.1 LXC 和 Docker(网络和存储模型) | ||
下午 | 2.2 Containerd 和命令行工具们 | ||
2.3 CRI-O | |||
第 2 天 | 上午 | 2.4 ECI 和 Kata | |
2.5 GPU | |||
2.6 最佳实践 | |||
下午 | 3. K8S 生命周期管理 | 3.1 K8S 集群创建删除、扩缩容、备份恢复 | |
3.2 K8S 升级 | |||
第 3 天 | 上午 | 3.1.2 Sealos 和其它工具(kubekey/k0s) | |
3.1.4 KubeClipper | |||
3.1.5 K3S | |||
下午 | 3.3 迁移和纳管:Rancher 和 K3S | ||
3.4 统一认证和鉴权:KubeSphere 和计量、日志 | |||
3.7 最佳实践 | |||
4. 存储管理 | 4.2 对接 NAS/NFS | ||
第 4 天 | 上午 | 4.3 对接 CephRBD | |
4.4 跨命名空间复制 PVC | |||
4.6 最佳实践 | |||
5. 网络管理 | 5.2 对接 Calico | ||
下午 | 5.3 对接 OVN 和 Multus | ||
5.6 最佳实践 | |||
6. 安全相关 | 6.2 权限控制 | ||
6.3 扫描工具 | |||
6.5 审计日志 | |||
6.7 最佳实践 |
参考 https://www.centos.org/download/,EOF(End-of-life)时间 2024/06/30。CentOS 7 目前仍是生产环境中使用最广泛的容器操作系统。
CentOS 最麻烦的地方是配套的 yum/rpm 包过于久远。
首先是 Kernel,3.10 的 Kernel 会遇到的问题
QEMU: Checking if device /dev/kvm exists : FAIL (Check that the 'kvm-intel' or 'kvm-amd' modules are loaded & the BIOS has enabled virtualization)
升级步骤:
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install -y https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum -y --disablerepo="*" --enablerepo="elrepo-kernel" list available
yum -y --disablerepo=\* --enablerepo=elrepo-kernel install kernel-lt.x86_64
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
grub2-set-default 0
# 这里的 0 要根据实际情况来填写
reboot
重启之后,可以看到内核已经升级完成
[root@lab-kubernetes ~]# uname -a
Linux lab-kubernetes 5.4.196-1.el7.elrepo.x86_64 #1 SMP Tue May 24 12:49:20 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
Python 2 于 2020.01.01 结束支持,Python 3.6 于 2021.12 结束支持。
如果已经安装了 python 3.6,先移除
yum remove python3
然后安装 python 3.8
yum update -y
yum install centos-release-scl-rh -y
yum install rh-python38-python -y
yum install rh-python38-python-pip -y
yum install rh-python38-python-devel -y
echo "export PATH=$PATH:/opt/rh/rh-python38/root/usr/bin" >> /etc/profile && source /etc/profile
mkdir ~/.pip
vi ~/.pip/pip.conf
设置如下代理:
[global]
index-url=https://pypi.tuna.tsinghua.edu.cn/simple/
然后升级 pip
python3 -m pip install --upgrade pip
验证版本
[root@lab-c2009 ~]# python3 --version
Python 3.8.11
[root@lab-c2009 ~]# python3 -m pip --version
pip 22.0.4 from /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/pip (python 3.8)
git 1.x 版本有一些命令与 2 不兼容,一般推荐升级到 2
yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel asciidoc -y
yum install gcc perl-ExtUtils-MakeMaker -y
yum remove git -y
cd /usr/local/src/
wget https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.32.0.tar.xz
tar -xvf git-2.32.0.tar.xz
cd git-2.32.0
make prefix=/usr/local/git all
make prefix=/usr/local/git install
echo "export PATH=$PATH:/usr/local/git/bin" >> /etc/profile && source /etc/profile
python3 -m pip install git-review
echo "export PATH=$PATH:/opt/rh/rh-python38/root/usr/local/bin" >> /etc/profile && source /etc/profile
完成后,可以确认 git 升级到 2.x 版本
[root@lab-kubernetes git-2.32.0]# git --version
git version 2.32.0
参考:https://endoflife.software/operating-systems/linux/ubuntu
LTS(Long Term Support) | Release | EOL(End Of Life) | HWE(Hardware Enablement) |
---|---|---|---|
22.04 Jammy Jellyfish | 2022/04/21 | 2027/04 | |
20.04 Focal Fossa | 2020/04 | 2025/04 | 2030/04 |
18.04 Bionic Beaver | 2018/04/26 | 2023/04 | 2028/04 |
Ubuntu 的策略比较简单,保持循 LTS 升级即可。
Comparing CentOS Stream and CentOS Linux:
End of Life
Upstream vs downstream
综上,CentOS 8 已经停止支持,CentOS 8 Stream 不稳定,两者都不要用于生产了。
参考:https://www.ispsystem.com/news/what-would-be-the-alternative-to-centos
注:Oracle Linux 以 RHEL 为起始,与 RHEL 完全兼容,移除了 Red Hat 商标,加入了 Linux 错误修正。其可取之处是 7*24 服务…… 像另一个 RHEL。
参考:https://www.logicweb.com/what-is-rocky-linux/
2020 年 12 月(IBM 收购 Red Hat 之后),收购 CentOS 的 Red Hat 宣布,CentOS Linux 8 将于 2021 年底结束支持,比早些时候承诺的 10 年计划要短得多。CentOS Stream 是 development 版本,将按计划在2024年结束。从此以后 CentOS 将位于 RHEL 的上游,而不是下游,CentOS 用户实际上将是 RHEL 的测试人员。
这引发了广大 CentOS 用户的极大不满,CentOS 创始人 Gregory Kurtzer 随即宣称领导创建新的 “CentOS” 发行版的工作。Rocky 这个名字是为了纪念已故 CentOS 联合创始人 Rocky McGaugh。Kurtzer 曾在加州大学伯克利分校从事高性能计算工作很长时间。鉴于CentOS 在 CERN 等机构的粒子物理学中得到了广泛应用,这可能会是Rocky Linux 的主要关注点之一。
Enterprise Linux, the community way. Rocky Linux is an open-source enterprise operating system designed to be 100% bug-for-bug compatible with Red Hat Enterprise Linux®. It is under intensive development by the community.
参考:https://rockylinux.org/download/,Planned EOL: 2029/05/31
综上,Rocky 是另起炉灶的 CentOS Linux。当下进展不错。参考 https://github.com/rocky-linux/rocky,2021/06,Rocky Linux 已经生产环境 ready。建议在生产环境中使用(CentOS 7 结束支持前半年,也就是 2023 年应该转向 Rocky Linux)。
AlmaLinux 由 CloudLinux 的开发人员构建和维护,CloudLinux 是一家提供服务器托管和 Linux 软件的公司。这是一家在 RHEL 分支方面经验丰富的公司,十多年来一直构建和维护其内部发行版 CloudLinux OS,它本身就是一个分支。
你应该选 Alma 还是 Rocky?它俩都致力于提供社区版的 RHEL,这有点难回答:
从所有权来说(AlmaLinux 开放,Rocky 独裁):
从贡献者来说(AlmaLinux 集中,Rocky 分散):
从经验来说(两者差不多):
从响应速度来说(AlmaLinux 更快):
综上:目前 Rocky 的呼声更高,但 AlmaLinux 也不差,可以再看看。
参考:https://developer.huaweicloud.com/ict/en/site-euleros/euleros,EulerOS 是华为推出的企业 Linux 系统(诞生于华为科学实验室),Openeuler 是 EulerOS 的社区版本,2019 年开源,现贡献给开放原子基金会,代码托管在 https://gitee.com/openeuler。
OpenEuler 兼容 CentOS(但是它并不是蹭 CentOS 8 结束支持热点才开源的,开源在那之前),相较 CentOS 对核内关键功能如进程调度、内存管理、IO 读写进行了深度优化;同时在核外构筑了容器 iSula、虚拟化 StraitVirt、机密计算 SecGear、毕昇 JDK(Huawei JDK 的开源版本)。
综上,OpenEuler 的优势是国产、信创,能较好地适配国产 ARM 服务器(特别是华为鲲鹏、泰山等)。缺点是兼容性和稳定性验证稍显不足。
龙蜥由阿里巴巴在 2021/10/20 孵化出来,诞生背景是 CentOS 8 结束支持(CentOS 8 结束支持这事,堪称“一鲸落,万物生”,可惜 Alma 和 Rocky Linux 起得太快)。
参考:https://openanolis.cn/anolisos,100% 兼容 CentOS 8 软件生态。
参考:https://www.zhihu.com/question/502615238/answer/2408765289,一言难尽。
这个就说来话长了,我也查了半天,诸君姑且听之。
综上:
emmmmm,得承认,麒麟在信创方面有一定优势。
Docker 的历史
我们需要知道的:
Google 最早开始大规模使用容器技术(Borg 系统,后来简化重构,以 K8S 开源)
Linux 容器(LXC)技术主要两个内核特性组成:namespace & cgroup。namespace 最早是 2002 年在 2.4.19 内核中引入(mount 单元),用于实现资源隔离。cgroup 2000 年以前就在 google 使用,2006 年以后贡献到 Linux Kernel,用于实现资源限制,2008 年 LXC 技术基本完成。
Docker 在 2013 年成立之后,对 LXC 进行封装,提供了极简的容器使用方案,几乎成为容器的代名词
Docker Overview,参考:https://docs.docker.com/engine/docker-overview
UFS(union file system)用来支持对文件系统的修改分层
Docker QuickStart,参考:https://docs.docker.com/get-started
K8S 1.24 开始放弃对 Docker 的支持,放弃的是什么?
接口解释:
unix:///var/run/dockershim.sock
unix:///var/run/docker.sock
unix:///run/containerd/containerd.sock
Docker 18.x 以后的版本,会有 3 个组件:runC、containerd、dockerd。
通过 ps -ef 可以看到几个进程之间的关系:
[root@lab-kubernetes ~]# docker run -d nginx
614b35ef3e2842cdc57bcb08c89df93533802d443065f8887c3951a601b738c2
[root@lab-kubernetes ~]# docker run -d nginx
62970cd2a51798b07c4f8ad6b42a19e72410aba7a2e6841c1c1606a35feedbee
[root@lab-kubernetes ~]# ps -ef
root 18442 1 0 00:44 ? 00:00:27 /usr/bin/containerd
root 18452 1 0 00:44 ? 00:00:02 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
root 20595 1 0 11:18 ? 00:00:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 614b35ef3e2842cdc57bcb08c89df93533802d443065f8887c3951a601b738c2 -address /run/containerd/containerd.sock
root 20615 20595 0 11:18 ? 00:00:00 nginx: master process nginx -g daemon off;
root 20708 1 0 11:22 ? 00:00:00 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 62970cd2a51798b07c4f8ad6b42a19e72410aba7a2e6841c1c1606a35feedbee -address /run/containerd/containerd.sock
root 20728 20708 0 11:22 ? 00:00:00 nginx: master process nginx -g daemon off;
K8S 和 Docker 有什么关系?参考:Container runtimes
Docker 和 Rocket 之争
OCI 标准是什么?:runC / Kata( 及其前身 runV 和 Clear Containers ),gVisor,Rust railcar
CRI 标准是什么?:Docker( 借助 dockershim ),containerd( 借助 CRI-containerd ),CRI-O,Frakti。是一组 gRPC 接口,cri-api/pkg/apis/services.go:
containerd-1.0,对 CRI 的适配通过一个单独的进程 CRI-containerd 来完成
containerd-1.1,把适配逻辑作为插件放进了 containerd 主进程中
综上:
Docker shim 接口虽然在 K8S 1.24 中被弃用,但我们可以通过 docker 来理解 Linux 容器的一般操作和使用方法。这部分知识在 K8S 的其它 CRI 中是通用的。
在 Ubuntu 18.04 / Ubuntu 20.04 / CentOS 7 上配置 Docker
# 更新依赖仓库
apt-get update -y || yum update -y
# 安装 Docker
apt-get install docker.io -y || yum install docker -y
systemctl enable docker --now
# 检查 docker 服务状态
systemctl status docker
# ubuntu 中需要把 docker 的 cgroup driver 改成 systemd
# !! centos 默认就是 systemd,不要修改这个文件,改了 docker 会起不来,保持 {} 就好
vi /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"]
}
systemctl restart docker
# run hello-world
docker run hello-world
Note:2021 年 7 月之后,ubuntu 环境 kubeadmin 默认都是 1.22+ 版本,因此需要将 docker 的 cgroup driver 改成 systemd(原来是 cgroup)。如果不改,后续 kubeadm init 时,会报错:
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
检查 journalctl -x -u kubelet,可以看到:
Aug 07 15:10:45 ckalab2 kubelet[11394]: E0807 15:10:45.179485 11394 server.go:294] "Failed to run kubelet" err="failed to run Kubelet: misconfiguration: kubelet cgroup driver: \"systemd\" is different from docker cgroup driver: \"cgroupfs\""
看官方文档:https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/configure-cgroup-driver/:
In v1.22, if the user is not setting the cgroupDriver field under KubeletConfiguration, kubeadm will default it to systemd.
所以我们需要把 docker 的 cgroup driver 改成 systemd
修改步骤参考:https://stackoverflow.com/questions/43794169/docker-change-cgroup-driver-to-systemd
修改完成后,检查一下 docker cgroup,确保 docker cgroup 是 systemd 了:
sudo docker info | grep -i cgroup
如何创建一个镜像?如何启动和调试容器?Github 或 Gitee
mkdir ~/test
cd ~/test
wget https://gitee.com/dev-99cloud/lab-openstack/raw/master/src/docker-quickstart/app.py
wget https://gitee.com/dev-99cloud/lab-openstack/raw/master/src/docker-quickstart/requirements.txt
wget https://gitee.com/dev-99cloud/lab-openstack/raw/master/src/docker-quickstart/Dockerfile
# 如果是 CentOS 7.x 需要安装下 python3
# yum install python3 python3-pip
# python3 -m pip install -r requirements.txt
python3 -m pip install -i https://pypi.tuna.tsinghua.edu.cn/simple -r requirements.txt
python3 app.py
# * Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
# 此时可以从浏览器访问 http://<ip>/
docker build --tag=friendlyhello .
# 可以看一下本地镜像列表
docker images
docker rm testFlask
docker run --rm -p 4000:80 --name=testFlask friendlyhello
# 也可以从 dockerhub 下载
docker run --rm -p 4000:80 --name=testFlask 99cloud/friendlyhello:3.9.6
# * Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
# 此时可以从浏览器访问 http://<ip>:4000
# 如果要跑在后台,可以加 -d 参数
docker run -d --rm -p 4000:80 --name=testNew 99cloud/friendlyhello:3.9.6
# 进入容器调试
$ docker exec -it testFlask /bin/bash
root@4224b69e7ee3:/app# env
HOSTNAME=4224b69e7ee3
PWD=/app
HOME=/root
NAME=World
...
root@4224b69e7ee3:/app# ps -ef
# 查看容器日志
docker logs -f testFlask
# 结束容器
docker stop testFlask
# 启动容器
docker start testFlask
# 从容器生成新的镜像
docker stop testFlask
docker commit testFlask test-new
# 删除容器
docker rm testFlask
# 删除镜像
docker rmi maodouzi/get-started:part2
参考:https://docs.docker.com/network/#network-drivers,Docker 网络模型分为若干种,在生产环境中会被用到的主要是 bridge 和 host 模式。
host 模式是容器和宿主机共享同一个 TCP/IP 协议栈(network namespace),容器的网络和宿主机网络不做隔离(只是网络不隔离,PID / IPC / MNT / UTS 还是 namespace 隔离的)
bridge 模式下,容器网络和宿主机网络是通过 network namespace 隔离开的,然后再通过类似 NAT 或者 port-mapping 技术完成转发。
因此,我们通过 bridge 模式来研究 network namespace 的底层逻辑。
docker 容器实现没有把 network namespaces 放到标准路径 /var/run/netns
下,所以 ip netns list
命令看不到。但是可以看
ll /proc/<pid>/ns
,两个进程的 namespaces id 相同说明在同一个 namespaces
[root@cloud025 ns]# ll /proc/2179/ns/
total 0
lrwxrwxrwx 1 root root 0 Aug 10 11:58 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 root root 0 Aug 10 11:58 mnt -> mnt:[4026531840]
lrwxrwxrwx 1 root root 0 Aug 10 11:58 net -> net:[4026531956]
lrwxrwxrwx 1 root root 0 Aug 10 11:58 pid -> pid:[4026531836]
lrwxrwxrwx 1 root root 0 Aug 10 11:58 user -> user:[4026531837]
lrwxrwxrwx 1 root root 0 Aug 10 11:58 uts -> uts:[4026531838]
做个软链接,就可以看到 netns 了
[root@cloud025 ns]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
07297a3ac7ea nginx:latest "/docker-entrypoin..." 29 minutes ago Up 29 minutes 80/tcp devtest
0935b08509a4 test-new "python app.py" 35 minutes ago Up 35 minutes 0.0.0.0:5000->80/tcp testNew
c23c76dd779c 99cloud/friendlyhello:3.9.6 "python app.py" 37 minutes ago Up 36 minutes 0.0.0.0:4000->80/tcp testFlask
[root@cloud025 ns]# docker inspect testFlask | grep -i pid
"Pid": 1159,
"PidMode": "",
"PidsLimit": 0,
[root@cloud025 ns]# mkdir -p /var/run/netns
[root@cloud025 ns]# ln -s /proc/1159/ns/net /var/run/netns/testFlask
[root@cloud025 ns]# ip netns list
testFlask (id: 0)
devtest (id: 2)
testNew (id: 1)
进入对应的 namespaces,看 ip,pod namespace 里虚拟网卡的 link-netnsid
始终等于 0
[root@cloud025 ns]# ip netns exec testNew ip a
...
44: eth0@if45: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.17.0.3/16 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::42:acff:fe11:3/64 scope link
valid_lft forever preferred_lft forever
在 root namespaces 中 ip a
,可以看到 link-netnsid = 1
,1 是前面 ip netns list
里的 namespaces id
[root@cloud025 ns]# ip a
45: vethb6d08be@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 0e:d9:14:d1:86:98 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::cd9:14ff:fed1:8698/64 scope link
valid_lft forever preferred_lft forever
这是一对 veth pair,看他们的序号和 if 可以发现
看网桥,可以看到这个 root namespaces 的虚拟网卡绑在 docker0 网桥上。在 CentOS 上,需要安装一下:yum install bridge-utils
[root@cloud025 ~]# brctl show
bridge name bridge id STP enabled interfaces
docker0 8000.02428c25c112 no vethb6d08be
virbr0 8000.525400d583b9 yes virbr0-nic
怎么对 pod 的网口抓包?
# 查看指定 pod 运行在那个 node 上
kubectl describe pod <pod> -n <namespace>
# 获得容器的 pid
docker inspect -f {{.State.Pid}} <container>
# 进入该容器的 network namespace
nsenter --target <PID> -n
# 使用 tcpdump 抓包,指定 eth0 网卡
tcpdump -i eth0 tcp and port 80 -vvv
# 或者抓包并导出到文件
tcpdump -i eth0 -w ./out.cap
# 退出 nsenter,要记得退出!!
exit
然后用 wireshark 或者 netmon 打开,即可查看。
Containerd 是从 Docker 中分离出来的一个项目,是一个工业级标准的容器运行时,它强调简单性、健壮性和可移植性。
生产环境中 Containerd 作为容器运行时应用最为广泛
兼容 Docker
Docker 直接带 Containerd,Containerd 可以单独装,也可以装 Docker,再用 Docker 中的 Containerd 对接 K8S。这在某些必须使用到 docker 的场景中比较受欢迎,比如超融合场景,需要依赖 docker 部署 ceph 等
直接兼容 K8S CRI
性能优良
使用 bucketbench 对 Docker、crio 和 Containerd 的性能测试结果,包括启动、停止和删除容器,以比较它们所耗的时间:
可以看到 Containerd 在各个方面都表现良好,总体性能优于 Docker 和 crio
从以上两张图,可以大致了解 crictl / ctr / nerdctl / podman 的关系。
参考:https://kubernetes.io/docs/reference/tools/map-crictl-dockercli/
crictl 命令和 docker 命令用法基本一样,是同一家开发的东西。
相同的命令包括:
以下命令只有 crictl 有
crictl | Description |
---|---|
imagefsinfo | Return image filesystem info |
inspectp | Display the status of one or more pods |
port-forward | Forward local port to a pod |
pods | List pods |
runp | Run a new pod |
rmp | Remove one or more pods |
stopp | Stop one or more running pods |
参考:https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/
可以看到 crictl 对 pod / container / image 的操作
参考:https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md,安装 crictl
VERSION="v1.24.1"
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-$VERSION-linux-amd64.tar.gz
[root@lab-kubernetes tmp]# crictl --version
crictl version v1.24.1
[root@lab-kubernetes tmp]# crictl images
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0002] connect endpoint 'unix:///var/run/dockershim.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
ERRO[0004] connect endpoint 'unix:///run/containerd/containerd.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
FATA[0006] connect: connect endpoint 'unix:///run/crio/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
这里 crictl 不可用是因为没有为 crictl 命令配置 endpoint。在 centos 7 上部署的 docker 版本是 1.13,版本太低了,使用的不是 containerd 而是 libcontainerd,略有不同。可以卸载原有的 docker,重新安装高版本 docker。
参考:https://docs.docker.com/engine/install/centos/
yum remove -y docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
yum install -y yum-utils
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
# yum list docker-ce --showduplicates | sort -r
# yum install docker-ce-20.10.16 docker-ce-cli-20.10.16 containerd.io docker-compose-plugin
systemctl enable docker --now
echo "alias docker-compose='docker compose'" >> /etc/profile && source /etc/profile
docker version
docker-compose version
然后可以检查 systemctl 的配置文件:
[root@lab-kubernetes ~]# cat /usr/lib/systemd/system/docker.service
[Unit]
...
Requires=docker.socket containerd.service
[Service]
...
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
启动时使用了 /run/containerd/containerd.sock
此时 crictl 应该可以对接 containerd 了,我们再测试一下:
[root@lab-kubernetes ~]# cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
[root@lab-kubernetes ~]# crictl images
FATA[0000] listing images: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.ImageService
这个报错是因为 containerd 的配置文件中,默认关闭了 cri plugin,所以无法进行 cri 对接。
修改配置文件,启用 cri plugin:
[root@lab-kubernetes ~]# vi /etc/containerd/config.toml
[root@lab-kubernetes ~]# cat /etc/containerd/config.toml | grep cri
# disabled_plugins = ["cri"]
[root@lab-kubernetes ~]# systemctl restart containerd
[root@lab-kubernetes ~]# crictl images
IMAGE TAG IMAGE ID SIZE
[root@lab-kubernetes ~]# crictl ps -a
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
综上:
ctr 命令会随着 containerd 服务一起安装。
ctr version
# pull 镜像
# 注意 ⚠️,这里要按 registry/orgnization/image:tag 写全,没有默认了
ctr images pull docker.io/library/redis:alpine3.13
# 查看镜像
ctr i ls
# 启动镜像
ctr run -d docker.io/library/redis:alpine3.13 redis
查看容器,注意 ⚠️:
这和 docker 中 container 定义是不一样
ctr container ls
CONTAINER IMAGE RUNTIME
redis docker.io/library/redis:alpine3.13 io.containerd.runc.v2
ctr task ls
TASK PID STATUS
redis 20808 RUNNING
进入到容器中执行 redis 命令
ctr task exec -t --exec-id redis-sh redis sh
/data # redis-cli
127.0.0.1:6379> set k1 v1
OK
127.0.0.1:6379> get k1
"v1"
注意 ⚠️:containerd 中存在 namespace 概念,这样可以将不同业务和应用进行隔离。例如 k8s 使用 containerd 和直接使用 ctr 创建的容器可以隔离开。
$ ps -ef | grep runc | grep redis
/usr/local/containerd/bin/containerd-shim-runc-v2 -namespace default -id redis -address /run/containerd/containerd.sock
查看一下当前的 namespace:
$ ctr ns ls
NAME LABELS
default
k8s.io
不同 namespace 下 pull 的镜像也是隔离显示的,可以使用 -n 指定具体的 namespace:
ctr -n default i ls
ctr -n k8s.io i ls
crictl pull
的镜像是在 k8s.io
namespace 下,docker pull
的镜像不在 moby namespaces(容器在)。
$ crictl pull docker.io/library/redis:alpine3.13
$ crictl images
IMAGE TAG IMAGE ID SIZE
docker.io/library/redis alpine3.13 554d20f203657 10.9MB
crictl 不能像 ctr 那样通过参数给定用户名和密码的方式从开启认证的私有仓库中 pull 镜像。需要对 containerd 进行配置。 containerd
提供的各种功能在其内部都是通过插件实现的,可以使用 ctr plugins ls
查看 containerd 的插件。
[root@lab-kubernetes ~]# ctr plugin ls
TYPE ID PLATFORMS STATUS
io.containerd.content.v1 content - ok
io.containerd.snapshotter.v1 aufs linux/amd64 skip
io.containerd.snapshotter.v1 btrfs linux/amd64 skip
io.containerd.snapshotter.v1 devmapper linux/amd64 error
io.containerd.snapshotter.v1 native linux/amd64 ok
io.containerd.snapshotter.v1 overlayfs linux/amd64 ok
io.containerd.snapshotter.v1 zfs linux/amd64 skip
io.containerd.metadata.v1 bolt - ok
io.containerd.differ.v1 walking linux/amd64 ok
io.containerd.event.v1 exchange - ok
io.containerd.gc.v1 scheduler - ok
io.containerd.service.v1 introspection-service - ok
io.containerd.service.v1 containers-service - ok
io.containerd.service.v1 content-service - ok
io.containerd.service.v1 diff-service - ok
io.containerd.service.v1 images-service - ok
io.containerd.service.v1 leases-service - ok
io.containerd.service.v1 namespaces-service - ok
io.containerd.service.v1 snapshots-service - ok
io.containerd.runtime.v1 linux linux/amd64 ok
io.containerd.runtime.v2 task linux/amd64 ok
io.containerd.monitor.v1 cgroups linux/amd64 ok
io.containerd.service.v1 tasks-service - ok
io.containerd.grpc.v1 introspection - ok
io.containerd.internal.v1 restart - ok
io.containerd.grpc.v1 containers - ok
io.containerd.grpc.v1 content - ok
io.containerd.grpc.v1 diff - ok
io.containerd.grpc.v1 events - ok
io.containerd.grpc.v1 healthcheck - ok
io.containerd.grpc.v1 images - ok
io.containerd.grpc.v1 leases - ok
io.containerd.grpc.v1 namespaces - ok
io.containerd.internal.v1 opt - ok
io.containerd.grpc.v1 snapshots - ok
io.containerd.grpc.v1 tasks - ok
io.containerd.grpc.v1 version - ok
io.containerd.tracing.processor.v1 otlp - skip
io.containerd.internal.v1 tracing - ok
io.containerd.grpc.v1 cri linux/amd64 ok
devmapper plugin error 是因为没有配置,可以暂时忽略,devmapper 概念可以参考:https://pkg.go.dev/github.com/containerd/containerd/snapshots/devmapper#section-readme
私有镜像仓库相关的配置在 cri 插件中,文档 Configure Image Registry中包含了镜像仓库的配置。 关于私有仓库和认证信息配置示例如下,修改/etc/containerd/config.toml:
...
[plugins]
...
[plugins."io.containerd.grpc.v1.cri"]
...
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["https://registry-1.docker.io"]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."harbor.my.org"]
endpoint = ["https://harbor.my.org"]
[plugins."io.containerd.grpc.v1.cri".registry.configs]
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.my.org".tls]
insecure_skip_verify = true
[plugins."io.containerd.grpc.v1.cri".registry.configs."harbor.my.org".auth]
username = "username"
password = "passwd"
# auth = "base64(username:password)"
...
配置完成后重启 containerd,就可以使用 crictl pull 配置的私有仓库的镜像了:
crictl pull harbor.my.org/library/nginx:1.1
参考:https://github.com/containerd/nerdctl
nerdctl 是一个非常棒的客户端:
参考:https://github.com/containerd/nerdctl#features-present-in-nerdctl-but-not-present-in-docker
安装使用:
wget https://github.com/containerd/nerdctl/releases/download/v0.20.0/nerdctl-0.20.0-linux-amd64.tar.gz
tar zxvf nerdctl-0.20.0-linux-amd64.tar.gz -C /usr/bin/
nerdctl run -d --name nginx -p 80:80 nginx:alpine
ctr -n default i ls
可以看到:
启动容器带 -p 需要 CNI 支持
[root@lab-kubernetes tmp]# nerdctl run -d --name nginx -p 80:80 nginx:alpine
docker.io/library/nginx:alpine:
FATA[0013] needs CNI plugin &{"bridge" "nerdctl0" %!q(bool=true) %!q(bool=false) %!q(bool=false) %!q(bool=true) '\x00' %!q(bool=true) %!q(bool=false) '\x00' map["ranges":[[map["gateway":"10.4.0.1" "subnet":"10.4.0.0/24"]]] "routes":[map["dst":"0.0.0.0/0"]] "type":"host-local"]} to be installed in CNI_PATH ("/opt/cni/bin"), see https://github.com/containernetworking/plugins/releases: exec: "/opt/cni/bin/bridge": stat /opt/cni/bin/bridge: no such file or directory
安装一下 cni 就好
mkdir -p /opt/cni/bin/
cd /opt/cni/bin/
wget https://github.com/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz
tar zxvf cni-plugins-linux-amd64-v1.1.1.tgz
然后再加 -p 运行命令就不会报错了
[root@lab-kubernetes tmp]# nerdctl run -d --name nginx -p 80:80 nginx:alpine
5bc1f7fcae4e4891c7e57626286fbed1465739f2b28270ec90c88a82e1106a64
[root@lab-kubernetes tmp]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
62970cd2a517 nginx "/docker-entrypoint.…" 5 hours ago Up 5 hours 80/tcp admiring_yalow
614b35ef3e28 nginx "/docker-entrypoint.…" 5 hours ago Up 5 hours 80/tcp fervent_ardinghelli
[root@lab-kubernetes tmp]# docker rm -f 62970cd2a517 614b35ef3e28
62970cd2a517
614b35ef3e28
[root@lab-kubernetes tmp]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@lab-kubernetes tmp]# nerdctl ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5bc1f7fcae4e docker.io/library/nginx:alpine "/docker-entrypoint.…" 38 seconds ago Up 0.0.0.0:80->80/tcp nginx
nerdctl pull 的镜像在 default namespace 下
[root@lab-kubernetes tmp]# ctr -n default i ls | grep nginx
docker.io/library/nginx:alpine application/vnd.docker.distribution.manifest.list.v2+json sha256:a74534e76ee1121d418fa7394ca930eb67440deda413848bc67c68138535b989 9.7 MiB linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x
综上:
containerd 可以有多个 namespaces,default/k8s.io/moby。这里的 namespace 不是 k8s 层面的,而是 containerd 自己的概念空间。参考 https://github.com/containerd/containerd/blob/main/docs/namespaces.md
containerd offers a fully namespaced API so multiple consumers can all use a single containerd instance without conflicting with one another. Namespaces allow multi-tenancy within a single daemon. This removes the need for the common pattern of using nested containers to achieve this separation. Consumers are able to have containers with the same names but with settings and/or configurations that vary drastically. For example, system or infrastructure level containers can be hidden in one namespace while user level containers are kept in another. Underlying image content is still shared via content addresses but image names and metadata are separate per namespace.
It is important to note that namespaces, as implemented, is an administrative construct that is not meant to be used as a security feature. It is trivial for clients to switch namespaces.
通过 kubelet/crictl 启动的容器,ns 就是 k8s.io,通过 docker 启动的就是 moby。docker ps 是看不到 k8s.io 下的容器的。对于 containerd 而言, docker 和 kubelet 是两个不同的客户端。
nerdctl 和 ctr 默认都是 default namespaces
docker 启动的容器进程在 moby namespace,拉取的镜像不在 namespaces。
podman 不和 containerd 打交道,拉取的镜像和启动的容器都与 containerd namespace 无关。
podman 是 CRI-O 项目分裂出来的,由 redhat 发起。
参考:https://github.com/containers/podman,podman 致力于创造一个用户体验类似 docker,但没有 dockerd daemon 服务的工具。
参考:https://podman.io/getting-started/
yum install -y podman
podman pull docker.io/library/httpd
podman images
podman run -dt -p 8080:80/tcp docker.io/library/httpd
[root@lab-kubernetes ~]# podman run -dt -p 8080:80/tcp docker.io/library/httpd
Trying to pull docker.io/library/httpd...
439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04
[root@lab-kubernetes ~]# ps -ef | grep httpd
root 22267 22256 0 17:01 pts/0 00:00:00 httpd -DFOREGROUND
33 22278 22267 0 17:01 pts/0 00:00:00 httpd -DFOREGROUND
33 22279 22267 0 17:01 pts/0 00:00:00 httpd -DFOREGROUND
33 22280 22267 0 17:01 pts/0 00:00:00 httpd -DFOREGROUND
root 22256 1 0 17:01 ? 00:00:00 /usr/bin/conmon --api-version 1 -s -c 439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04 -u 439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04/userdata -p /var/run/containers/storage/overlay-containers/439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04/userdata/pidfile -l k8s-file:/var/lib/containers/storage/overlay-containers/439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04/userdata/ctr.log --exit-dir /var/run/libpod/exits --socket-dir-path /var/run/libpod/socket --log-level error --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/var/run/containers/storage/overlay-containers/439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04/userdata/oci-log -t --conmon-pidfile /var/run/containers/storage/overlay-containers/439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/lib/containers/storage --exit-command-arg --runroot --exit-command-arg /var/run/containers/storage --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /var/run/libpod --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 439e9d1532d0578af0d3d80e9b06f728ed06389a6d878430b8bee71aa74f5d04
可以看到:
podman 相比较 docker 的优势:
先移除之前安装的 containerd / runc / docker
yum remove runc -y
部署 Containerd
# 配置 containerd 所需 kernel module
cat << EOF > /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF
# 加载 kernel module
modprobe overlay
modprobe br_netfilter
# 修改内核参数
cat << EOF > /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
user.max_user_namespaces=28633
EOF
sysctl -p /etc/sysctl.d/99-kubernetes-cri.conf
# 低内核版本 nf_conntrack_ipv4 / 高内核版本(4.x+) nf_conntrack
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack
yum install -y ipset ipvsadm
# 下载和安装 containerd
wget https://github.com/containerd/containerd/releases/download/v1.6.5/cri-containerd-cni-1.6.5-linux-amd64.tar.gz
tar -zxvf cri-containerd-cni-1.6.5-linux-amd64.tar.gz -C /
containerd 压缩包中包含了 crictl/ctr 命令行,cni,runc 和 containerd 自身服务。
[root@lab-kubernetes tmp]# tar -zxvf cri-containerd-cni-1.6.5-linux-amd64.tar.gz -C /
etc/
etc/cni/
etc/cni/net.d/
etc/cni/net.d/10-containerd-net.conflist
etc/crictl.yaml
etc/systemd/
etc/systemd/system/
etc/systemd/system/containerd.service
usr/
usr/local/
usr/local/sbin/
usr/local/sbin/runc
usr/local/bin/
usr/local/bin/containerd-shim
usr/local/bin/containerd
usr/local/bin/critest
usr/local/bin/crictl
usr/local/bin/containerd-shim-runc-v1
usr/local/bin/containerd-stress
usr/local/bin/containerd-shim-runc-v2
usr/local/bin/ctd-decoder
usr/local/bin/ctr
opt/
opt/cni/
opt/cni/bin/
opt/cni/bin/static
opt/cni/bin/vlan
opt/cni/bin/bandwidth
opt/cni/bin/portmap
opt/cni/bin/ipvlan
opt/cni/bin/firewall
opt/cni/bin/macvlan
opt/cni/bin/dhcp
opt/cni/bin/ptp
opt/cni/bin/sbr
opt/cni/bin/vrf
opt/cni/bin/loopback
opt/cni/bin/host-device
opt/cni/bin/tuning
opt/cni/bin/bridge
opt/cni/bin/host-local
opt/containerd/
opt/containerd/cluster/
opt/containerd/cluster/gce/
opt/containerd/cluster/gce/cloud-init/
opt/containerd/cluster/gce/cloud-init/node.yaml
opt/containerd/cluster/gce/cloud-init/master.yaml
opt/containerd/cluster/gce/configure.sh
opt/containerd/cluster/gce/cni.template
opt/containerd/cluster/gce/env
opt/containerd/cluster/version
cri-containerd-cni-1.6.5-linux-amd64.tar.gz
包含的 runc 动态链接库与 CentOS 7 有兼容性问题。运行 runc
的时候,会遇到报错:runc: undefined symbol: seccomp_notify_respond
类似这样的 bug 我们可以用 hotfix 来修复(直接下载更新版本的 runc,或者自己编译 hotfix 的 runc)。
wget https://github.com/opencontainers/runc/releases/download/v1.1.0-rc.1/runc.amd64
chmod +x runc.amd64
cp runc.amd64 /usr/local/sbin/runc
# 生成 containerd 配置文件
mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
修改 /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
...
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
...
# 然后启动 containerd
systemctl enable containerd --now
# 检查 containerd 是否运行正常
ctr version
ctr plugin ls
crictl version
然后按需部署 docker
wget https://download.docker.com/linux/static/stable/x86_64/docker-20.10.9.tgz
tar -xvf docker-20.10.9.tgz
docker 包括如下二进制文件
[root@lab-kubernetes tmp]# tar -xvf docker-20.10.9.tgz
docker/
docker/containerd-shim-runc-v2
docker/dockerd
docker/docker-proxy
docker/ctr
docker/docker
docker/runc
docker/containerd-shim
docker/docker-init
docker/containerd
安装和部署 docker
mv docker/docker* /usr/bin
cat <<EOF > /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=containerd.service
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd --containerd /run/containerd/containerd.sock --cri-containerd
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
OOMScoreAdjust=-500
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl start docker
检查安装情况
[root@lab-kubernetes tmp]# docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
[root@lab-kubernetes tmp]# docker version
Client:
Version: 20.10.9
API version: 1.41
Go version: go1.16.8
Git commit: c2ea9bc
Built: Mon Oct 4 16:03:22 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.9
API version: 1.41 (minimum version 1.12)
Go version: go1.16.8
Git commit: 79ea9d3
Built: Mon Oct 4 16:07:30 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.6.5
GitCommit: 96df0994faabc1944fc614e52b0b3c6feb609a57
runc:
Version: 1.1.0-rc.1
GitCommit: v1.1.0-rc.1-0-g55df1fc4
docker-init:
Version: 0.19.0
GitCommit: de40ad0
此时有个问题,用 nerdctl 运行带 -p 参数的容器会失败,报错 cni 版本不匹配。
检查 nerdctl 参数,发现默认 cni path 是 /usr/libexec/cni/
,有三种方法:
/opt/cni/bin/
目录,此目录下的 cni 是 1.1.1 版本的/usr/libexec/cni/
目录(删除后,nerdctl 默认的 cni path 变成了 /opt/cni/bin/
)都可以解决此问题。
[root@kubernetes004 ~]# nerdctl run -d --name nginx -p 80:80 nginx:alpine
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: time="2022-06-07T21:21:59+08:00" level=fatal msg="failed to call cni.Setup: plugin type=\"bridge\" failed (add): incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"
Failed to write to log, write /var/lib/nerdctl/1935db59/containers/default/954e3d09f83bdce90305b2e8f58dad59a58ac5d9f350711a4025c7854067470c/oci-hook.createRuntime.log: file already closed: unknown
[root@kubernetes004 ~]# nerdctl -h | grep cni
--cni-netconfpath string cni config directory [$NETCONFPATH] (default "/etc/cni/net.d")
--cni-path string cni plugins binary directory [$CNI_PATH] (default "/usr/libexec/cni")
[root@kubernetes004 ~]# ls /usr/libexec/cni/
bandwidth dhcp flannel host-local loopback portmap sample static vlan
bridge firewall host-device ipvlan macvlan ptp sbr tuning
[root@kubernetes004 ~]# rm -rf /usr/libexec/cni/
[root@kubernetes004 ~]# nerdctl -h | grep cni
--cni-netconfpath string cni config directory [$NETCONFPATH] (default "/etc/cni/net.d")
--cni-path string cni plugins binary directory [$CNI_PATH] (default "/opt/cni/bin")
[root@kubernetes004 ~]# nerdctl rm -f nginx
[root@kubernetes004 ~]# nerdctl run -d --name nginx -p 80:80 nginx:alpine
4958c5b1ad1fd2db2d8d1c4ccd0d796cd9115d4f07f17aa7a1fa615b89831c7a
CRI-O 是 RedHat 发布的容器运行时,旨在同时满足 CRI 标准和 OCI 标准。kubelet 通过 CRI 与 CRI-O 交互,CRI-O 通过 OCI 与 runC 交互,追求简单明了。
CRI 接口是:unix:///var/run/crio/crio.sock
在同一个集群中,混用不同的 CRI 实验,参考:https://gobomb.github.io/post/container-runtime-note/
安装 cri-o,参考:https://cri-o.io/
OS=CentOS_7
VERSION=1.23
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo
curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
yum install -y cri-o
修改 kubelet 启动参数(也可以写在 EnvironmentFile 指定的文件里):
vim /lib/systemd/system/kubelet.service
[Unit]
......
[Service]
......
ExecStart=/k8s/kubernetes/bin/kubelet $KUBELET_OPTS --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --cgroup-driver=systemd
......
[Install]
.......
启动 crio:
systemctl start crio
# 使用 crictl 查看版本
crictl version
# 重启 kubelet
systemctl restart kubelet
测试:
$ crictl -r /var/run/crio/crio.sock version
Version: 0.1.0
RuntimeName: cri-o
RuntimeVersion: 1.11.11-1.rhaos3.11.git474f73d.el7
RuntimeApiVersion: v1alpha1
$ crictl -r /var/run/crio/crio.sock ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID
fbe5b37ad3c47 docker.io/library/busybox@sha256:895ab622e92e18d6b461d671081757af7dbaa3b00e3e28e12505af7817f73649 About an hour ago Running busybox 0 f9f42b5ae54c7
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
[root@node02 runc]# runc list
ID PID STATUS BUNDLE CREATED OWNER
f9f42b5ae54c71180ab7eb1706205bba79597d019f1c3e22f5f68e0b0ae055a8 31252 running /run/containers/storage/overlay-containers/f9f42b5ae54c71180ab7eb1706205bba79597d019f1c3e22f5f68e0b0ae055a8/userdata 2019-07-30T07:31:54.768990081Z root
fbe5b37ad3c472ea970af75afc4c58481c2dd4d89a93b1a7ca37ddda823b201c 31785 running /run/containers/storage/overlay-containers/fbe5b37ad3c472ea970af75afc4c58481c2dd4d89a93b1a7ca37ddda823b201c/userdata 2019-07-30T07:34:15.524324699Z root
综上:
容器毕竟还是共享内核的(容器的天然不安全与天然安全),安全性和隔离型对于想要实现多租户是不够。所以又出现了许多基于虚拟机隔离的方案出来。
实现 ECI 不只可以通过 K8S,也可以通过 OpenStack zun 来直接编排 Kata,参考 https://github.com/cloudmaster2010/openstack/blob/main/devstack/3.zun-kata-kuryr.md
Kata 安装步骤,参考:https://github.com/kata-containers/kata-containers/blob/main/docs/install/README.md#kata-deploy-installation
第一种安装方案:参考 https://github.com/kata-containers/kata-containers/blob/main/docs/how-to/containerd-kata.md
too early for operation, device not yet seeded or device model not acknowledged
,可以参考:https://blog.csdn.net/u010620626/article/details/117259178,sudo setenforce 0
第二种安装方案,先安装好 containerd + k8s,再通过 kata-deploy 部署。参考:https://github.com/kata-containers/kata-containers/blob/main/tools/packaging/kata-deploy/README.md
除 Kata 以外,还有 Frakti / gVisor 等
Frakti 提供了 hypervisor 级别的隔离性,提供的是内核级别的而非 Linux 命名空间级别的隔离:
Frakti lefts Kubernetes run pods and containers directly inside hypervisors via runV. It is light weighted and portable, but can provide much stronger isolation with independent kernel than linux-namespace-based container runtimes.
gVisor 是拦截了系统调用,用自己实现用户态的进程而非内核来处理系统调用
运行沙箱容器时,给出了支持安全沙箱容器运行时 handler runsc,我们需要创建一个 RuntimeClass 并在 pod spec 里指定是用该 RuntimeClass
```yaml
apiVersion: node.k8s.io/v1beta1
kind: RuntimeClass
metadata:
name: untrusted
handler: runsc
```
这里 runsc 时 gvisor 的 "runc"
还需要修改 deployment 的 pod template
```yaml
spec:
runtimeClassName: untrusted
containers:
- image: vicuu/nginx:host
imagePullPolicy: IfNotPresent
name: nginx-host
```
参考:https://developer.nvidia.com/blog/announcing-containerd-support-for-the-nvidia-gpu-operator/
Nvidia 官方提供的 containerd 支持步骤如下:
也可以参考这里的步骤:https://icloudnative.io/posts/add-nvidia-gpu-support-to-k8s-with-containerd/
关于 GPU 的虚拟化:
K8S 的操作要记得参考:https://kubernetes.io/
K8S 有哪些组件?api-server、kube-scheduler、kube-controller、etcd、coredns、kubelete、kubeproxy
组件结构图
生产环境配置
Host-Node | CPU | RAM | root | etcd | kubelet | CRI |
---|---|---|---|---|---|---|
Master * 3 | 16C+ | 32G+ | 100G+ | 40G+ | 250G+ | 250G+ |
Worker * N | 16C+ | 32G+ | 100G+ | N/A | 250G+ | 250G+ |
参考部署架构
基于 Containerd 部署 K8S 1.23.3 集群
CentOS 7.9,升级 kernel,参考 1.1.1 内核升级
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum install -y https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum -y --disablerepo="*" --enablerepo="elrepo-kernel" list available
yum -y --disablerepo=\* --enablerepo=elrepo-kernel install kernel-lt.x86_64
awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
grub2-set-default 0
# 这里的 0 要根据实际情况来填写
reboot
重启后,uname -a
检查内核版本,可以看到
[root@kubernetes001 ~]# uname -a
Linux kubernetes001 5.4.197-1.el7.elrepo.x86_64 #1 SMP Sat Jun 4 08:43:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
部署 Containerd 和 ctictl 等工具,参考 2.2.2.1 Crictl
# 如果之前安装 centos 默认源的 docker,要先删除掉
yum remove -y docker docker-client docker-client-latest docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine
yum install -y yum-utils
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum install -y containerd.io
# yum install -y docker-ce docker-ce-cli docker-compose-plugin
systemctl enable containerd --now
# systemctl enable docker --now
然后用 ctr 检查 containerd 是否正常运行
[root@kubernetes001 ~]# ctr version
Client:
Version: 1.6.4
Revision: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
Go version: go1.17.9
Server:
Version: 1.6.4
Revision: 212e8b6fa2f44b9c21b2798135fc6fb7c53efc16
UUID: 1cd1f73e-28ae-4f7c-9fa3-5ccdcdb2a23c
[可选]:然后部署 crictl
VERSION="v1.24.1"
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
tar zxvf crictl-$VERSION-linux-amd64.tar.gz
mv crictl /usr/bin/
接下来修改 containerd 配置,避开 gcr
cp /etc/containerd/config.toml /etc/containerd/config.toml.bak
containerd config default > /etc/containerd/config.toml
vi /etc/containerd/config.toml
修改 SystemdCgroup
和 sandbox_image
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
...
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
...
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.6"
...
然后,重启服务
systemctl restart containerd
# 检查版本,验证可用
crictl version
部署 Kubernetes
# 3. 关闭防火墙(默认就是关闭的,不用做)
# systemctl stop firewalld.service
# systemctl disable firewalld.service
# 4. 关闭 selinux(默认就是关闭的,不用做)
# vi /etc/selinux/config
# 将 SELINUX=enforcing 改为 SELINUX=disabled
# 5. 关闭 swap(默认就是关闭的,不用做)
# swapoff /dev/sda2
# vi /etc/fstab
# 在 swap 分区这行前加 # 禁用掉,保存退出
# reboot
# 6. 配置系统相关属性
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl -p
sysctl --system
modprobe br_netfilter
echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
echo 1 > /proc/sys/net/ipv4/ip_forward
# 7. 配置yum源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
# [kubernetes]
# name=Kubernetes
# baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
# enabled=1
# gpgcheck=0
# repo_gpgcheck=0
# gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
# https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
# EOF
# 8. 安装 CRI
# 10. 下载 kubernetes https://kubernetes.io/releases/
export k8s_version="1.23.3"
yum install -y kubelet-${k8s_version}-0 kubeadm-${k8s_version}-0 kubectl-${k8s_version}-0 --disableexcludes=kubernetes
# 11. 启动 kubelet
systemctl enable kubelet --now
# 12. 用 kubeadm 初始化创建 K8S 集群 https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#options
kubeadm init --cri-socket unix:///run/containerd/containerd.sock --image-repository registry.aliyuncs.com/google_containers --kubernetes-version=v${k8s_version} --pod-network-cidr=10.244.0.0/16
# 13. 配置 .kube/config 用于使用 kubectl
mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
# 15. 安装 calico
kubectl apply -f https://gitee.com/dev-99cloud/lab-openstack/raw/master/src/ansible-cloudlab-centos/playbooks/roles/init04-prek8s/files/calico-${k8s_version}.yml
# kubectl apply -f https://raw.githubusercontent.com/99cloud/lab-openstack/master/src/ansible-cloudlab-centos/playbooks/roles/init04-prek8s/files/calico-${k8s_version}.yml
# 看到 node Ready 就 OK
kubectl get nodes
# 16. [可选]:移除 taint
kubectl taint nodes $(hostname) node-role.kubernetes.io/master-
拓扑结构:1 Master(Containerd) + 1 Worker(CRI-O)
此处采用 Containerd + CRI-O 混合 CRI 部署。生产环境中不会这么用(生产环境中会尽量用较少、较成熟的模块完成搭建,减少依赖,减少技术栈复杂度),这里这么实验是用于同时展示 Kubelet 对接不同 CRI 时的情况。
CRIO 部署 时注意,sandbox 要配置成 registry.aliyuncs.com/google_containers
,gcr.io
的 pause
容器拉取不了,或者可以 podman 手动 load
参考:https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
# 首先确认节点情况
kubectl get nodes
# 然后告诉 API Server 暂停该节点的调度
kubectl drain <node name>
# 等节点恢复正常后,需要加回,可以用
kubectl uncordon <node name>
ETCD 命令行工具安装
# Ubuntu 环境上用 apt-get 安装
apt install etcd-client
其它环境直接下载二进制文件,参考:https://github.com/etcd-io/etcd/releases
ETCD_VER=v3.5.4
# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /tmp/etcd-download-test --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
/tmp/etcd-download-test/etcd --version
/tmp/etcd-download-test/etcdctl version
/tmp/etcd-download-test/etcdutl version
# start a local etcd server
/tmp/etcd-download-test/etcd
# write,read to etcd
/tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 put foo bar
/tmp/etcd-download-test/etcdctl --endpoints=localhost:2379 get foo
通过 API Server 理解 ETCD 使用情况,测试 etcdctl 存取内容
root@ckalab001:~# ps -ef | grep api | grep -i etcd
root 24761 24743 3 10:17 ? 00:06:53 kube-apiserver --advertise-address=172.31.43.206 --allow-privileged=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/pki/ca.crt --enable-admission-plugins=NodeRestriction --enable-bootstrap-token-auth=true --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key --etcd-servers=https://127.0.0.1:2379 --insecure-port=0 --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/etc/kubernetes/pki/sa.pub --service-account-signing-key-file=/etc/kubernetes/pki/sa.key --service-cluster-ip-range=10.96.0.0/12 --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
# 设置环境变量,ETCDCTL_API=3
root@ckalab001:~# export ETCDCTL_API=3
root@ckalab001:~# etcdctl --cert="/etc/kubernetes/pki/apiserver-etcd-client.crt" --key="/etc/kubernetes/pki/apiserver-etcd-client.key" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --endpoints=https://127.0.0.1:2379 put /firstkey trystack
OK
root@ckalab001:~# etcdctl --cert="/etc/kubernetes/pki/apiserver-etcd-client.crt" --key="/etc/kubernetes/pki/apiserver-etcd-client.key" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --endpoints=https://127.0.0.1:2379 get /firstkey
/firstkey
trystack
# list 所有的 key
etcdctl --cert="/etc/kubernetes/pki/apiserver-etcd-client.crt" --key="/etc/kubernetes/pki/apiserver-etcd-client.key" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --endpoints=https://127.0.0.1:2379 get --prefix --keys-only ""
# list 所有的 key & value
etcdctl --cert="/etc/kubernetes/pki/apiserver-etcd-client.crt" --key="/etc/kubernetes/pki/apiserver-etcd-client.key" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --endpoints=https://127.0.0.1:2379 get --prefix ""
# backup & restore
etcdctl --cert="/etc/kubernetes/pki/apiserver-etcd-client.crt" --key="/etc/kubernetes/pki/apiserver-etcd-client.key" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --endpoints=https://127.0.0.1:2379 snapshot save a.txt
# etcdctl --cert="/etc/kubernetes/pki/apiserver-etcd-client.crt" --key="/etc/kubernetes/pki/apiserver-etcd-client.key" --cacert="/etc/kubernetes/pki/etcd/ca.crt" --endpoints=https://127.0.0.1:2379 snapshot restore a.txt
Kubeadm 部署 k8s 集群时,master0 尤其重要,需要通过该节点来把证书推送给新加入的节点。当 master0 节点出现故障,可以利用备份恢复 master0 节点。通过 kubeadm 安装的 k8s,主节点包括两类的灾备恢复:
场景描述:
Etcd 数据备份:定义 etcd-backup.yaml 文件,运行在 master0 上
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: backup
namespace: kube-system
spec:
schedule: "0 0 * * *" # 指定时间
jobTemplate:
spec:
template:
spec:
containers:
- name: backup
# Same image as in /etc/kubernetes/manifests/etcd.yaml
image: k8s.gcr.io/etcd-amd64:3.2.18
env:
- name: ETCDCTL_API
value: "3"
command: ["/bin/sh"]
args: ["-c", "etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d_%H:%M:%S_%Z).db"]
volumeMounts:
- mountPath: /etc/kubernetes/pki/etcd
name: etcd-certs
readOnly: true
- mountPath: /backup
name: backup
restartPolicy: OnFailure
# 指定master0节点执行
nodeName: master0
hostNetwork: true
volumes:
- name: etcd-certs
hostPath:
path: /etc/kubernetes/pki/etcd
type: DirectoryOrCreate
- name: backup
hostPath:
path: /tmp/etcd_backup/
type: DirectoryOrCreate
实现思路:
etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key snapshot save /backup/etcd-snapshot-$(date +%Y-%m-%d_%H:%M:%S_%Z).db
即为备份命令,按照时间的格式命名 etcd 的备份数据Etcd 数据恢复:
# 1. 将 /etc/kubernetes/manifests/ kube-apiserver.yaml 文件移动到别处,停止 kube-apiserver 服务
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# 2. 将 /etc/kubernetes/manifests/ etcd.yaml 文件移动到别处,停止 etcd server 服务
mv /etc/kubernetes/manifests/etcd.yaml /tmp/
# 3. 运行如下命令,将损坏的数据文件移至其他地方
mv /var/lib/etcd/* /tmp/
# 4. 运行以下命令,临时运行 docker 容器,将数据从备份里恢复到 /var/lib/etcd/
docker run --rm \
-v '/tmp/etcd_backup/:/backup' \
-v '/var/lib/etcd:/var/lib/etcd' \
--env ETCDCTL_API=3 \
'k8s.gcr.io/etcd-amd64:3.2.18' \
/bin/sh -c "etcdctl snapshot restore '/backup/etcd-snapshot-xxx_UTC.db' ; mv /default.etcd/member/ /var/lib/etcd/"
# 5. 将/etc/kubernetes/manifests/kube-apiserver.yaml文件放回原来路径,恢复kube-api server服务
# 6. 将/etc/kubernetes/manifests/etcd.yaml文件放回原来路径,恢复etcd server服务
主节点数据备份主要包括三个部分:
/etc/kubernetes/
目录下的所有文件(证书,manifest 文件)定义 k8s-master-backup.yaml 文件,运行在 master0 上。
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: k8s-master-backup
namespace: kube-system
spec:
# activeDeadlineSeconds: 100
schedule: "5 0 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: k8s-master-backup
image: docker.io/alpine:latest
command: ["/bin/sh"]
args: ["-c", "tar -zcvf /backup/k8s-master-$(ifconfig eth0 | grep 'inet addr:' | awk '{print $2}' | cut -c 6-)-$(date +%Y-%m-%d_%H:%M:%S_%Z).tar.gz /kubernetes /kubelet"]
volumeMounts:
- mountPath: /backup
name: backup
- mountPath: /kubernetes
name: kubernetes
- mountPath: /kubelet
name: kubelet
restartPolicy: OnFailure
# 指定节点master0执行
nodeName: master0
hostNetwork: true
volumes:
- name: backup
hostPath:
path: /tmp/k8s_master_backup/
type: DirectoryOrCreate
- name: kubernetes
hostPath:
path: /etc/kubernetes/
type: DirectoryOrCreate
- name: kubelet
hostPath:
path: /var/lib/kubelet/
type: DirectoryOrCreate
实现思路:
注意:为防止宿主机本身故障,建议备份文件应定期转移到备份机器上,编写脚本定时将备份文件传到备份机器上
创建脚本文件 filescp.sh,filescp.sh 文件内容
#!/bin/bash
scp -i .ssh/id_rsa /tmp/etcd_backup/ root@10.0.0.76:/tmp
scp -i .ssh/id_rsa /tmp/k8s_master_backup/ root@10.0.0.76:/tmp
# 加入脚本执行到计划任务
crontab -e
# 添加计划任务(计划每日0时30分执行,时间用户可自指定)
30 0 * * * /tmp/filescp.sh
新建节点替换 master0
生成 certificate-key(在master0节点执行)
该过程遇上 warning 可以忽略。主要目的是让 kubeadm 自己处理证书,不用我们 scp
$ kubeadm init phase upload-certs --upload-certs
# 或者
$ kubeadm init phase upload-certs --upload-certs --config /tmp/.k8s/kubeadm.yaml
b019bb8de1cbe3de4327a7a238e60faf6f865bc815cdcdeb86f1f6f116bad3df
查看加入集群命令(master0节点执行)
kubeadm token create --print-join-command
kubeadm join 192.168.40.199:16443 --token e5wrs0.lqcem5us4a04tp5x --discovery-token-ca-cert-hash sha256:61c6754582a1ca7668770594acd1efa36a9c5c71a897517d8fb6f6c9db8ee314
将 master3 节点加入 k8s 集群,充当控制节点(master3 节点上执行)
kubeadm join apiserver.cluster.local:6443 --token 6g5enb.ub1pap31ty0zoym4 --discovery-token-ca-cert-hash sha256:a057ab2330c6dba1de3816a291ea3786a5a558d5737d94ef64c5c045bf1e5e1c --control-plane --certificate-key b019bb8de1cbe3de4327a7a238e60faf6f865bc815cdcdeb86f1f6f116bad3df
查看 master3 是否加入成功
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get pod -A
kubectl get node
将 master0 从 k8s 集群删除(任意 master 节点执行)
kubectl delete nodes master0
# 在 master0 上运行 reset
kubeadm reset
获取 master0 节点的 hash 值(任意 master 节点执行)
# 集群本身安装了 etcdctl 工具,可省略这几步
tar -zxvf etcd-v3.3.4-linux-amd64.tar.gz
cd etcd-v3.3.4-linux-amd64
cp etcdctl /usr/local/sbin/
# 获取 master0 节点 hash 值
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list
输出:
1203cdd3ad75e761, started, master0, https://192.168.40.180:2380, https://192.168.40.180:2379
dda71d9d52b97028, started, master1, https://192.168.40.181:2380, https://192.168.40.181:2379
dkeijf23cjd3445k, started, master2, https://192.168.40.182:2380, https://192.168.40.182:2379
找到 master0 对应的 hash 值是:1203cdd3ad75e761
根据 hash 删除 etcd 信息,执行如下命令:
ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove 1203cdd3ad75e761
查看节点是否加入成功
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master3 Ready control-plane,master 50s v1.20.6
master1 Ready control-plane,master 35m v1.20.6
master2 Ready control-plane,master 35m v1.20.6
参考:https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-reset/
kubectl reset -f
参考:https://kubernetes.io/docs/tasks/debug/debug-cluster/#cluster-failure-modes
参考:https://github.com/labring/sealos#quickstart
先确定能 ssh 172.19.30.106(本机 IP) 成功,然后用下面的命令就可以一行搞定 AIO 的 K8S 部署了。ctr / crictl / nerdctl 都会一起装上。
wget -c https://sealyun-home.oss-cn-beijing.aliyuncs.com/sealos-4.0/latest/sealos-amd64 -O sealos && chmod +x sealos && mv sealos /usr/bin
sealos run labring/kubernetes:v1.24.0 labring/calico:v3.22.1 --masters 172.19.30.106
本地起了镜像仓库,可以看配置文件怎么配置的。
[root@lab-k8s001 ~]# crictl images
IMAGE TAG IMAGE ID SIZE
sealos.hub:5000/calico/cni v3.22.1 2a8ef6985a3e5 80.5MB
sealos.hub:5000/calico/node v3.22.1 7a71aca7b60fc 69.6MB
sealos.hub:5000/calico/pod2daemon-flexvol v3.22.1 17300d20daf93 8.46MB
sealos.hub:5000/calico/typha v3.22.1 f822f80398b9a 52.7MB
sealos.hub:5000/coredns/coredns v1.8.6 a4ca41631cc7a 13.6MB
sealos.hub:5000/etcd 3.5.3-0 aebe758cef4cd 102MB
sealos.hub:5000/kube-apiserver v1.24.0 529072250ccc6 33.8MB
sealos.hub:5000/kube-controller-manager v1.24.0 88784fb4ac2f6 31MB
sealos.hub:5000/kube-proxy v1.24.0 77b49675beae1 39.5MB
sealos.hub:5000/kube-scheduler v1.24.0 e3ed7dee73e93 15.5MB
sealos.hub:5000/pause 3.7 221177c6082a8 311kB
sealos.hub:5000/tigera/operator v1.25.3 648350e58702c 44.8MB
优点:
管理 K8S 集群生命周期,HA 集群,扩缩容,清空集群,自动恢复(If any master is down, lvscare will remove the ipvs realserver, when master recover it will add it back)
可以从 sealos hub 下载和使用 OCI-compatible 的 openebs, minio, ingress, pgsql, mysql, redis 等插件
证书直接签 99 年(这个是优点还是安全漏洞待考…… 而且为了这个还直接改了 kubeadm 源码),依赖 k8s API 证书的 kubefed 之类就不要每年更新证书了。
cmd/kubeadm/app/constants/constants.go
里的
CertificateValidity[root@lab-k8s001 ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf May 15, 2122 13:08 UTC 99y ca no
apiserver May 15, 2122 13:08 UTC 99y ca no
apiserver-etcd-client May 15, 2122 13:08 UTC 99y etcd-ca no
apiserver-kubelet-client May 15, 2122 13:08 UTC 99y ca no
controller-manager.conf May 15, 2122 13:08 UTC 99y ca no
etcd-healthcheck-client May 15, 2122 13:08 UTC 99y etcd-ca no
etcd-peer May 15, 2122 13:08 UTC 99y etcd-ca no
etcd-server May 15, 2122 13:08 UTC 99y etcd-ca no
front-proxy-client May 15, 2122 13:08 UTC 99y front-proxy-ca no
scheduler.conf May 15, 2122 13:08 UTC 99y ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca May 15, 2122 13:08 UTC 99y no
etcd-ca May 15, 2122 13:08 UTC 99y no
front-proxy-ca May 15, 2122 13:08 UTC 99y no
缺点:
参考:https://kubesphere.com.cn/docs/installing-on-linux/introduction/kubekey/
安装步骤:https://kubesphere.com.cn/docs/installing-on-linux/on-premises/install-kubesphere-on-bare-metal/
# 配置好密钥,确定可以 ssh 免密登录需要安装 kubelet 的 node
yum install -y openssl openssl-devel
yum install -y socat
yum install -y epel-release
yum install -y conntrack-tools
export KKZONE=cn
curl -sfL https://get-kk.kubesphere.io | VERSION=v2.0.0 sh -
./kk create config --with-kubernetes v1.21.5
vi config-sample.yaml
# {name: node1, address: 172.19.30.105, internalAddress: 172.19.30.105, user: root}
# 这里 172.19.30.105 是 node IP
# node1 要写你期望的节点名称,安装时会根据这个设置 hostname 的
./kk create cluster -f config-sample.yaml
# 部署完成后,检查 pod
kubectl get pod -A
docker ps
优点:
缺点:
Container Runtime Version: docker://20.10.8
槽点:
没啥高级功能…… 为啥不直接用 kubeadm,封装干嘛呢……
跑了一遍 kk
之后,收到短信:【阿里云】尊敬的 XXXX:云盾云安全中心检测到您的服务器:XXXX 出现了紧急安全事件:恶意脚本代码执行,建议您立即登录云安全中心控制台-安全告警处理 http://a.aliyun.com/XXXX 进行处理。
源码:https://github.com/kubeclipper-labs,喜欢的话请帮忙 star
QuickStart:https://github.com/kubeclipper-labs/kubeclipper/blob/master/README_zh.md#quick-start
KubeClipper | Sealos | KubeKey | Kubeasz | KubeOperator | K0S | |
---|---|---|---|---|---|---|
图形化页面 | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ |
轻依赖 - 不依赖 ansible 等 | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ |
多区域、多集群管理 | ✅ | ❌ | ❌ | ❌ | ✅ | ❌ |
支持多版本 K8S、CRI | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
离线安装 | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
基于 kubeadm 封装 | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ |
K3S 参考文档
Kubernetes 集群在管理大规模服务上有极佳的运维自动化优势(自愈 / 扩缩容 / 负载均衡 / 服务注册和服务发现 / 部署),但也需要耗费相对较多的资源,许多功能对特定对特定场景来说是冗余的,K3S 作为一个轻量级的 Kubernetes 发行版应运而生,它针对边缘计算、物联网等场景进行了高度优化。
K3s 有以下增强功能:
K3S 适用于:
K3s 单节点集群的架构如下图所示,该集群有一个内嵌 SQLite 数据库的单节点 K3s server。
在这种配置中,每个 agent 节点都注册到同一个 server 节点。K3s 用户可以通过调用 server 节点上的 K3s API 来操作 Kubernetes 资源。外部流量则通过 Traeffic 导入,且经过 loadBalance 进行负载均衡。
部署准备:参阅 K3S 官网配置改动
节点 | CPU | 内存 | 磁盘 | 数量 | os |
---|---|---|---|---|---|
server/agent | >2Core | >1G | >10G | 1 | centos7.x |
k3s 默认使用 containerd 作为 cri,也使用更为熟悉的 docker 作为集群 cri,先安装 docker
yum install -y yum-utils
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum makecache fast
yum install docker-ce-19.03.9 -y
mkdir -p /etc/docker
echo '{"registry-mirrors": ["http://hub-mirror.c.163.com"]}'>/etc/docker/daemon.json
systemctl enable --now docker
部署脚本安装(近期 k3s 和 autok3s 都在调整中,rancher-mirror.cnrancher.com 在归档,功能受影响,rancher-mirror.oss-cn-beijing.aliyuncs.com 尚未能全功代替)
# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--docker" sh -
# 受限国内网络,大多数时候上述脚本无法安装,建议采用下述国内加速安装脚本
curl -sfL https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC="--docker" sh -
如果不用 docker 的话,改成
curl -sfL https://rancher-mirror.oss-cn-beijing.aliyuncs.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn sh -
输出下述日志,即部署完成
...
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
查看 pod 是否正常即可
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system helm-install-traefik-crd-j4bwn 0/1 Completed 0 2m10s
kube-system helm-install-traefik-sxnmw 0/1 Completed 0 2m10s
kube-system metrics-server-86cbb8457f-47kzz 1/1 Running 0 2m10s
kube-system local-path-provisioner-5ff76fc89d-7bzjd 1/1 Running 0 2m10s
kube-system coredns-7448499f4d-cjph9 1/1 Running 0 2m10s
kube-system svclb-traefik-8q77r 2/2 Running 0 83s
kube-system traefik-97b44b794-gb72h 1/1 Running 0 83s
# 集群基础使用
# 我们来创建一个 deployment 的 pod
$ kubectl create deployment nginx --image=nginx
$ kubectl get po
NAME READY STATUS RESTARTS AGE
nginx-6799fc88d8-pc4wz 1/1 Running 0 101s
网络选项
# 修改 CNI 网络模式
# 默认情况下,K3S 将以 flannel 作为 CNI 运行,使用 VXLAN 作为默认后端。如需修改 CNI 的模式,可以参考下述命令安装
# --flannel-backend=vxlan (默认) 使用 VXLAN 后端。
# --flannel-backend=ipsec 使用 IPSEC 后端,对网络流量进行加密。
# --flannel-backend=host-gw 使用 host-gw 后端。
# --flannel-backend=wireguard 使用 WireGuard 后端,对网络流量进行加密。可能需要额外的内核模块和配置。
$ curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn INSTALL_K3S_EXEC="--docker --flannel-backend=vxlan" sh -
# 查看对应的 linux 网络设备是否生成
$ ip a | grep flannel
31: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
inet 10.42.0.0/32 scope global flannel.1
# 替换 CNI
# K3S 除了支持 flannel,还支持 calico\canel 等 CNI 组件
# 我们使用 calico 作为网络组件
# 注:下述 10.96.0.0/16 不能跟实际网络冲突
curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_KUBECONFIG_MODE="644" INSTALL_K3S_EXEC="--flannel-backend=none --cluster-cidr=10.96.0.0/16 --disable-network-policy --disable=traefik" sh -
# 下载 calico.yaml
$ wget https://docs.projectcalico.org/manifests/calico.yaml
# 修改 calico.yaml 中被注释掉的参数 CALICO_IPV4POOL_CIDR
- name: CALICO_IPV4POOL_CIDR
value: "10.96.0.0/16"
# apply calico
kubectl apply -f calico.yaml
# 查看集群 pod
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-node-6cv7v 1/1 Running 0 3m30s
kube-system local-path-provisioner-5ff76fc89d-sqdts 1/1 Running 0 33m
kube-system calico-kube-controllers-74b8fbdb46-mdwjm 1/1 Running 0 3m30s
kube-system metrics-server-86cbb8457f-5xgrs 1/1 Running 0 33m
kube-system coredns-7448499f4d-mwtmw 1/1 Running 0 33m
K3S 卸载
# 要从 server 节点卸载 K3s
$ /usr/local/bin/k3s-uninstall.sh
# 要从 agent 节点卸载 K3s
$ /usr/local/bin/k3s-agent-uninstall.sh
一个高可用 K3s 集群由以下几个部分组成:
固定 agent 节点的注册地址
在高可用 K3s server 配置中,每个节点还必须使用固定的注册地址向 Kubernetes API 注册,注册后,agent 节点直接与其中一个 server 节点建立连接,如下图所示:
先决条件:节点不能有相同的主机名
节点角色 | CPU | 内存 | 磁盘 | 数量 | os |
---|---|---|---|---|---|
server | >2Core | >4G | >10G | >3 | centos7.x |
agent | >2Core | >2G | >10G | >3 | centos7.x |
部署外部数据库 etcd。K3S 支持多种外部数据库,这里我们选用 etcd 数据库,为方便搭建,我们这里搭建的 etcd 集群为非 TLS 访问方式 更多详情内容参阅 K3S 外部数据库配置 和 etcd 官方文档
# 所有节点均部署 docker,与上述单节点部署过程一样,不再赘述
# etcd 高可用部署
# 三台 master 节点上均执行下述的 etcd 操作流程
wget https://github.com/etcd-io/etcd/releases/download/v3.3.15/etcd-v3.3.15-linux-amd64.tar.gz $
tar -vxf etcd-v3.3.15-linux-amd64.tar.gz $ cp etcd-v3.3.15-linux-amd64/etcd /usr/bin/ $ cp
etcd-v3.3.15-linux-amd64/etcdctl /usr/bin/
# etcd 采用 systemd 托管服务
vi /etc/systemd/system/etcd.service
# 填入 etcd 配置,保存退出。(etcd.service 配置见附录)
$ systemct start etcd $ systemct status etcd ● etcd.service - etcd Loaded: loaded
(/etc/systemd/system/etcd.service; disabled; vendor preset: disabled) Active: active (running) since
Sun 2021-10-03 18:19:52 CST; 10s ago Docs: https://github.com/etcd-io/etcd
$ etcdctl member list member 763586516b512018 is healthy: got healthy result from
http://{etcd1-ip}:2379 member 78bb25a565876d17 is healthy: got healthy result from
http://{etcd2-ip}:2379 member da51b2723088b522 is healthy: got healthy result from
http://{etcd3-ip}:2379
注:后续如果重装 k3s 集群,请先停止所有 etcd 服务,并清理所有 etcd 节点的数据,最后重启 etcd 即可
systemctl stop etcd
rm -rf /var/lib/etcd
systemctl restart etcd
下述命令中的 INSTALL_K3S_MIRROR,K3S_DATASTORE_ENDPOINT, K3S_TOKEN... 等可以作为环境变量,如 export K3S_TOKEN="k3s_token"
$ curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_DATASTORE_ENDPOINT='http://{etcd1-ip}:2379,http://{etcd2-ip}:2379,http://{etcd3-ip}:2379' K3S_TOKEN="k3s_token" INSTALL_K3S_VERSION=v1.18.6+k3s1 INSTALL_K3S_EXEC="--docker" sh -s - server
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
[INFO] systemd: Starting k3s
$ kubectl get no
NAME STATUS ROLES AGE VERSION
caasnode-1 Ready control-plane,master 2m58s v1.21.4+k3s1
caasnode-2 Ready control-plane,master 78s v1.21.4+k3s1
caasnode-3 Ready control-plane,master 82s v1.21.4+k3s1
在所有 agent 节点中执行 master-ip 可以是多个 master 间共用的 vip 或者第一个 master 的 ip。
下述命令中的 INSTALL_K3S_MIRROR,K3S_DATASTORE_ENDPOINT, K3S_TOKEN... 等可以作为环境变量,如 export K3S_TOKEN="k3s_token"
$ curl -sfL http://rancher-mirror.cnrancher.com/k3s/k3s-install.sh | INSTALL_K3S_MIRROR=cn K3S_TOKEN="k3s_token" K3S_URL=https://{master-ip}:6443 INSTALL_K3S_VERSION=v1.18.6+k3s1 INSTALL_K3S_EXEC="--docker" sh -
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Skipping /usr/local/bin/ctr symlink to k3s, command exists in PATH at /usr/bin/ctr
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service
[INFO] systemd: Enabling k3s-agent unit
[INFO] systemd: Starting k3s-agent
$ kubectl get no
caasbastion-3 Ready <none> 3m20s v1.21.4+k3s1
caasbastion-2 Ready <none> 2m49s v1.21.4+k3s1
caasbastion-1 Ready <none> 2m27s v1.21.4+k3s1
caasnode-1 Ready control-plane,master 21m v1.21.4+k3s1
caasnode-2 Ready control-plane,master 19m v1.21.4+k3s1
caasnode-3 Ready control-plane,master 19m v1.21.4+k3s1
# 要从 server 节点卸载 K3s
$ /usr/local/bin/k3s-uninstall.sh
# 要从 agent 节点卸载 K3s
$ /usr/local/bin/k3s-agent-uninstall.sh
# 如需重新部署,需要清理 etcd 的 data 目录,文档默认为 '/var/lib/etcd'
etcd.service
# /etc/systemd/system/etcd.service
# etcd-ip: 本机 ip
# etcd1-ip: 本机 ip
# etcd2-ip: 第二台节点 ip
# etcd3-ip: 第三台节点 ip
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd
After=network-online.target
[Service]
Type=notify
Restart=on-failure
RestartSec=5s
TimeoutStartSec=0
ExecStart=/usr/bin/etcd --name etcd1 \
--heartbeat-interval 300 \
--election-timeout 1500 \
--initial-advertise-peer-urls http://{etcd-ip}:2380 \
--listen-peer-urls http://{etcd-ip}:2380 \
--listen-client-urls http://{etcd-ip}:2379,http://127.0.0.1:2379 \
--advertise-client-urls http://{etcd-ip}:2379 \
--data-dir /var/lib/etcd \
--initial-cluster etcd1=http://{etcd1-ip}:2380,etcd2=http://{etcd2-ip}:2380,etcd3=http://{etcd3-ip}:2380 \
--initial-cluster-token etcd-cluster \
--initial-cluster-state new \
--auto-compaction-retention=1 \
--snapshot-count=5000 \
--quota-backend-bytes=171798691840 \
--heartbeat-interval=100 \
--election-timeout=500
ExecReload=/bin/kill -HUP
KillMode=process
[Install]
WantedBy=multi-user.target
参考:https://github.com/cnrancher/autok3s
curl -sS https://rancher-mirror.oss-cn-beijing.aliyuncs.com/autok3s/install.sh | INSTALL_AUTOK3S_MIRROR=cn sh -
# The commands will start autok3s daemon with an interactionable UI.
autok3s -d serve --bind-address 0.0.0.0
注意:
优点:
缺点:
参考:https://docs.k0sproject.io/v1.23.6+k0s.2/install/
参考:https://v1-23.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
alias k=kubectl
k drain cluster3-master1 --ignore-daemonsets
apt-mark unhold kubeadm
apt-mark hold kubectl kubelet
apt install kubeadm=1.23.1-00
apt-mark hold kubeadm
kubeadm upgrade plan
kubeadm upgrade apply v1.23.1
apt update
apt-mark unhold kubelet kubectl
apt install kubelet=1.23.1-00 kubectl=1.23.1-00
apt-mark hold kubelet kubectl
service kubelet restart
service kubelet status
kubectl get node
k uncordon cluster3-master1
k drain cluster3-worker1 --ignore-daemonsets
apt update
apt-mark unhold kubeadm
apt-mark hold kubectl kubelet
apt install kubeadm=1.23.1-00
apt-mark hold kubeadm
kubeadm upgrade node
apt-mark unhold kubectl kubelet
apt install kubelet=1.23.1-00 kubectl=1.23.1-00
service kubelet restart
service kubelet status
k get node
k uncordon cluster3-worker1
先升级 Master 节点
yum list --showduplicates kubeadm --disableexcludes=kubernetes
# find the latest 1.23 version in the list
# it should look like 1.23.x-0, where x is the latest patch
# replace x in 1.23.x-0 with the latest patch version
yum install -y kubeadm-1.23.x-0 --disableexcludes=kubernetes
# Verify that the download works and has the expected version:
kubeadm version
# Verify the upgrade plan:
kubeadm upgrade plan
# replace x with the patch version you picked for this upgrade
sudo kubeadm upgrade apply v1.23.x
# Same as the first control plane node but use:
sudo kubeadm upgrade node
# instead of: sudo kubeadm upgrade apply
# Also calling "kubeadm upgrade plan" and upgrading the CNI provider plugin is no longer needed.
# replace <node-to-drain> with the name of your node you are draining
kubectl drain <node-to-drain> --ignore-daemonsets
# replace x in 1.23.x-0 with the latest patch version
yum install -y kubelet-1.23.x-0 kubectl-1.23.x-0 --disableexcludes=kubernetes
# Restart the kubelet:
sudo systemctl daemon-reload
sudo systemctl restart kubelet
# replace <node-to-drain> with the name of your node
kubectl uncordon <node-to-drain>
再升级 Worker 节点,参考 https://v1-23.docs.kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#upgrade-worker-nodes
在实际边缘生成应用中,我们边缘环境可能分布着多个 k3s 集群,当集群数量过多时,后续的运维管理会变得非常的不方便。为了解决这样的问题,下面我们通过云端中心的 rancher 去纳管边缘的多个 k3s
节点角色 | CPU | 内存 | 磁盘 | 数量 | os |
---|---|---|---|---|---|
rancher | >2Core | >4G | >10G | 1 | centos7.x |
k3s | >2Core | >2G | >10G | n | centos7.x |
部署 rancher 服务
$ docker run -d --restart=unless-stopped -p 1234:80 -p 2234:443 rancher/rancher:v2.3.6
通过浏览器访问 https://{rancher_ip}:2234/
复制上图中的命令,到 k3s 端运行
sudo docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher/rancher-agent:v2.3.6 --server https://172.20.149.95:2234 --token ndhsgr9msksr4s6g6xqq6dl8cns42txlkxt96tsdwcl56hds568nbb --ca-checksum 5b778f411871d7caca521a0cf25cd40d8a7e28aeb1def47af829e22092c114cd --etcd --controlplane --worker
最后通过界面我们能看到 rancher 完成了对 k3s 集群的纳管
同样,后续的其余 k3s 也是用这个方式去纳管,就完成中心端对边缘端集群的管理。
先装好 K8S,准备检查:https://kubesphere.io/docs/installing-on-kubernetes/introduction/prerequisites/。注意:要有 default 的 storageclass,可以用 NFS 配置一个。
然后安装 KubeSphere:参考 https://kubesphere.io/docs/quick-start/minimal-kubesphere-on-k8s/。
下载相关包,并启动相关服务
apt-get install nfs-common -y || yum install nfs-utils -y
创建 NFS 数据路径
mkdir -p /nfs/data
chmod -R 777 /nfs/data
如果有额外挂单独的数据盘给 NFS 用,需要格式化这块磁盘并挂载,比如 vdb。如果没有额外挂盘请忽略这一步
mkfs.xfs /dev/vdb
mount /dev/vdb /nfs/data
echo "/dev/vdb /nfs/data xfs defaults 0 0" >> /etc/fstab
编辑 NFS 配置文件
# echo "/nfs/data *(rw,no_root_squash,sync)" > /etc/exports
echo "/nfs/data *(rw,sync,no_subtree_check,no_root_squash,no_all_squash,insecure)" > /etc/exports
# 首次登入Internal error occurred: account is not active 的问题,https://kubesphere.com.cn/forum/d/2058-kk-internal-error-occurred-account-is-not-active
exportfs -r
启动 rpcbind、nfs 服务
systemctl restart rpcbind && systemctl enable rpcbind
systemctl restart nfs && systemctl enable nfs
查看 RPC 服务的注册状况
rpcinfo -p localhost
program vers proto port service
...
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 3 tcp 2049 nfs_acl
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100227 3 udp 2049 nfs_acl
...
创建 RBAC.yaml 文件,内容如下。
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
执行命令创建 RBAC
kubectl create -f RBAC.yaml
创建 deployment.yaml 文件,内容如下:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nfs-client-provisioner
labels:
app: nfs-client-provisioner
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: nfs-client-provisioner
strategy:
type: Recreate
selector:
matchLabels:
app: nfs-client-provisioner
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccountName: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: 99cloud/nfs-subdir-external-provisioner:v4.0.0
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: localhost
- name: NFS_PATH
value: /nfs/data
volumes:
- name: nfs-client-root
nfs:
server: localhost
path: /nfs/data
其中 localhost 请替换成 NFS Server 的地址
如果 driver 起不来,多半是 client 端没有安装 nfs-common(ubuntu) 或者 nfs-utils(centos)
部署deploy
kubectl create -f deployment.yaml
创建 storageclass.yaml 文件
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs
provisioner: fuseim.pri/ifs
parameters:
archiveOnDelete: "false"
reclaimPolicy: Delete
provisioner 要对应 驱动所传入的环境变量 PROVISIONER_NAME 的值。
创建 storage class
kubectl apply -f storageclass.yaml
如果要把 nfs 设置成默认 StorageClass:(option)
kubectl patch storageclass nfs -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class":"true"}}}'
创建 pvc.yaml 文件
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc
annotations:
volume.beta.kubernetes.io/storage-class: "nfs"
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
创建 pvc
kubectl apply -f pvc.yaml
此时可以看一下 pvc 状态,应该是 Bound,如果是 pending,看一下 nfs driver pod 的 log。K8S 1.20
以后,会有这个报错:unexpected error getting claim reference: selfLink was empty, can't make reference
,有两个办法,参考:https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/issues/25,在
api-server 的 static pod 里添加启动参数:--feature-gates=RemoveSelfLink=false
,或者更新 NFS
驱动镜像:gcr.io/k8s-staging-sig-storage/nfs-subdir-external-provisioner:v4.0.0
测试
kind: Pod
apiVersion: v1
metadata:
name: test-pod
spec:
containers:
- name: test-pod
image: 99cloud/busybox:1.24
command:
- "/bin/sh"
args:
- "-c"
- "touch /mnt/SUCCESS && exit 0 || exit 1"
volumeMounts:
- name: nfs-pvc
mountPath: "/mnt"
restartPolicy: "Never"
volumes:
- name: nfs-pvc
persistentVolumeClaim:
claimName: test-pvc
其中
99cloud/busybox:1.24
是gcr.io/google_containers/busybox:1.24
,可以换成任意的镜像。
参考:https://masantu.com/blog/2020-03-22/cephadm-install-Ceph-Octopus-on-CentOS7/
[root@lab-c2009-ceph-aio ~]# uname -r
5.4.182-1.el7.elrepo.x86_64
[root@lab-c2009-ceph-aio ~]# cat /etc/system-release
CentOS Linux release 7.9.2009 (Core)
[root@lab-c2009-ceph-aio ~]# yum install python3 -y
[root@lab-c2009-ceph-aio ~]# python3 -V
Python 3.6.8
[root@lab-c2009-ceph-aio ~]# ls /dev/vd*
/dev/vda /dev/vda1 /dev/vda2 /dev/vdb
确认内核升级完毕,python3 部署完成(最好是 3.8,但实验环境中用 3.6 也行)
yum install -y yum-utils
yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
yum list docker-ce --showduplicates | sort -r
yum install -y docker-ce-20.10.16 docker-ce-cli-20.10.16 containerd.io docker-compose-plugin
systemctl enable docker --now
docker version
安装 Cephadm
curl -k --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm
chmod +x cephadm
# ./cephadm add-repo --release octopus
# 报错:repomd.xml: [Errno 14] curl#60 - "The certificate issuer‘s certificate has expired"
# 参考:https://blog.csdn.net/sulia1234567890/article/details/121956448
# 加入 vi /etc/yum.conf,增加:sslverify=0
vi /etc/yum.conf # sslverify=0
yum upgrade ca-certificates
./cephadm add-repo --release octopus
./cephadm install
which cephadm
# /usr/sbin/cephadm
引导新集群
[root@lab-c2009-ceph-aio ~]# mkdir -p /etc/ceph
[root@lab-c2009-ceph-aio ~]# ip a
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:cd:72:83 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.229/24 brd 192.168.122.255 scope global noprefixroute dynamic eth0
valid_lft 2736sec preferred_lft 2736sec
inet6 fe80::5390:413f:f60f:6034/64 scope link noprefixroute
valid_lft forever preferred_lft forever
如果要从开始就设置 osd pool 为 1
cat <<EOF > initial-ceph.conf
[global]
osd pool default size = 1
osd pool default min size = 1
EOF
cephadm bootstrap --config initial-ceph.conf --mon-ip 192.168.122.229
# Ceph Dashboard is now available at:
# URL: https://lab-c2009-ceph-aio:8443/
# User: admin
# Password: itgwv8r3xy
# You can access the Ceph CLI with:
# sudo /usr/sbin/cephadm shell --fsid 090a76ec-9bb4-11ec-91af-525400cd7283 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring
以上命令会操作:
/root/.ssh/authorized_keys
文件中。/etc/ceph/ceph.conf
。client.admin
管理(特权!)秘密密钥的副本写入 /etc/ceph/ceph.client.admin.keyring
。/etc/ceph/ceph.pub
。使用 Ceph CLI
[root@lab-c2009-ceph-aio ~]# cephadm add-repo --release octopus
[root@lab-c2009-ceph-aio ~]# yum install -y ceph-common
[root@lab-c2009-ceph-aio ~]# ceph -s
cluster:
id: 090a76ec-9bb4-11ec-91af-525400cd7283
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 1
services:
mon: 1 daemons, quorum lab-c2009-ceph-aio (age 58m)
mgr: lab-c2009-ceph-aio.nuvwkm(active, since 57m)
osd: 0 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:
[root@lab-c2009-ceph-aio ~]# ceph health
HEALTH_WARN OSD count 0 < osd_pool_default_size 1
接下来,添加 OSD,参考 https://docs.ceph.com/en/latest/cephadm/services/osd/#cephadm-deploy-osds
[root@lab-c2009-ceph-aio ~]# ceph orch device ls
Hostname Path Type Serial Size Health Ident Fault Available
lab-c2009-ceph-aio /dev/vdb hdd 21.4G Unknown N/A N/A Yes
[root@lab-c2009-ceph-aio ~]# ceph orch apply osd --all-available-devices
Scheduled osd.all-available-devices update...
从命令行添加会卡住,然后失败(当副本数为 1 的时候,也显示如下,但可以成功,不必走界面添加)
[root@lab-c2009-ceph-aio ~]# ceph orch daemon add osd lab-c2009-ceph-aio:/dev/vdb
Created no osd(s) on host lab-c2009-ceph-aio; already created?
从 ceph dashboard 可以添加 osd,先筛选,然后 add 即可。
[root@lab-c2009-ceph-aio ~]# ceph -s
cluster:
id: 090a76ec-9bb4-11ec-91af-525400cd7283
health: HEALTH_WARN
1 pool(s) have no replicas configured
services:
mon: 1 daemons, quorum lab-c2009-ceph-aio (age 13m)
mgr: lab-c2009-ceph-aio.fgkrqf(active, since 12m)
osd: 1 osds: 1 up (since 2m), 1 in (since 2m)
data:
pools: 1 pools, 1 pgs
objects: 0 objects, 0 B
usage: 1.0 GiB used, 19 GiB / 20 GiB avail
pgs: 1 active+clean
参考:https://cloud.tencent.com/developer/article/1700941?from=article.detail.1927694
CEPH 侧创建 Pool 和新用户
[root@lab-c2009-ceph-aio ~]# ceph osd pool create kubernetes
pool 'kubernetes' created
[root@lab-c2009-ceph-aio ~]# ceph osd lspools
1 device_health_metrics
2 kubernetes
[root@lab-c2009-ceph-aio ~]# ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
[client.kubernetes]
key = AQBaByJiJbL8KRAAW77WOh1hu6Pz9if9EPj9mA==
[root@lab-c2009-ceph-aio ~]# ceph auth get client.kubernetes
exported keyring for client.kubernetes
[client.kubernetes]
key = AQBaByJiJbL8KRAAW77WOh1hu6Pz9if9EPj9mA==
caps mgr = "profile rbd pool=kubernetes"
caps mon = "profile rbd"
caps osd = "profile rbd pool=kubernetes"
K8S 侧对接
参考:https://github.com/ceph/ceph-csi
Ceph CSI Version | Container Orchestrator Name | Version Tested |
---|---|---|
v3.6.1 | Kubernetes | v1.21, v1.22, v1.23 |
v3.6.0 | Kubernetes | v1.21, v1.22, v1.23 |
v3.5.1 | Kubernetes | v1.21, v1.22, v1.23 |
v3.5.0 | Kubernetes | v1.21, v1.22, v1.23 |
下载 ceph-csi-3.6.1.tar.gz
,解压
yum install wget -y
wget --no-check-certificate https://github.com/ceph/ceph-csi/archive/refs/tags/v3.6.1.tar.gz
tar zxvf v3.6.1.tar.gz
[root@lab-c2009-ceph-aio ~]# ceph mon dump
dumped monmap epoch 1
epoch 1
fsid 090a76ec-9bb4-11ec-91af-525400cd7283
last_changed 2022-03-04T12:12:10.455157+0000
created 2022-03-04T12:12:10.455157+0000
min_mon_release 15 (octopus)
0: [v2:192.168.122.229:3300/0,v1:192.168.122.229:6789/0] mon.lab-c2009-ceph-aio
进入 ceph-csi 的 deploy/rbd/kubernetes
目录:
config-map
apiVersion: v1
kind: ConfigMap
metadata:
name: "ceph-csi-config"
data:
config.json: |-
[
{
"clusterID": "090a76ec-9bb4-11ec-91af-525400cd7283",
"monitors": [
"192.168.122.229:6789"
]
}
]
cd /root/ceph-csi-3.6.1/deploy/rbd/kubernetes
kubectl apply -f csi-config-map.yaml
cat <<EOF > csi-rbd-secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: csi-rbd-secret
stringData:
userID: kubernetes
userKey: AQBaByJiJbL8KRAAW77WOh1hu6Pz9if9EPj9mA==
EOF
kubectl apply -f csi-rbd-secret.yaml
如果希望将所有 ceph-csi 相关的配置和服务启动在单独的命名空间中,参考 deprecated.md
创建必须的 ServiceAccount 和 RBAC ClusterRole/ClusterRoleBinding 资源对象:
kubectl create -f csi-provisioner-rbac.yaml
kubectl create -f csi-nodeplugin-rbac.yaml
创建 PodSecurityPolicy:
kubectl create -f csi-provisioner-psp.yaml
kubectl create -f csi-nodeplugin-psp.yaml
删除 kms 配置
diff kubernetes-bak/csi-rbdplugin-provisioner.yaml kubernetes/csi-rbdplugin-provisioner.yaml
160,161c160,161
< - name: ceph-csi-encryption-kms-config
< mountPath: /etc/ceph-csi-encryption-kms-config/
---
> # - name: ceph-csi-encryption-kms-config
> # mountPath: /etc/ceph-csi-encryption-kms-config/
230,232c230,232
< - name: ceph-csi-encryption-kms-config
< configMap:
< name: ceph-csi-encryption-kms-config
---
> # - name: ceph-csi-encryption-kms-config
> # configMap:
> # name: ceph-csi-encryption-kms-config
diff kubernetes-bak/csi-rbdplugin.yaml kubernetes/csi-rbdplugin.yaml
107,108c107,108
< - name: ceph-csi-encryption-kms-config
< mountPath: /etc/ceph-csi-encryption-kms-config/
---
> # - name: ceph-csi-encryption-kms-config
> # mountPath: /etc/ceph-csi-encryption-kms-config/
188,190c188,190
< - name: ceph-csi-encryption-kms-config
< configMap:
< name: ceph-csi-encryption-kms-config
---
> # - name: ceph-csi-encryption-kms-config
> # configMap:
> # name: ceph-csi-encryption-kms-config
[root@lab-c2009-k8s-aio-ceph kubernetes]# kubectl apply -f ../../../examples/ceph-conf.yaml
[root@lab-c2009-k8s-aio-ceph kubernetes]# kubectl get cm -A
NAMESPACE NAME DATA AGE
ceph-csi ceph-config 2 27s
部署 csi-rbdplugin-provisioner 和 RBD CSI driver
kubectl create -f csi-rbdplugin-provisioner.yaml
kubectl create -f csi-rbdplugin.yaml
然后看一下是否所有的 pod 都起来了,master 节点上可能需要 untaint,镜像可能下载失败,deployment 可能要限制 replicas
kubectl scale deploy csi-rbdplugin-provisioner --replicas=1
kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-
cat <<EOF > storageclass.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
clusterID: 090a76ec-9bb4-11ec-91af-525400cd7283
pool: kubernetes
imageFeatures: layering
csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
csi.storage.k8s.io/provisioner-secret-namespace: default
csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
csi.storage.k8s.io/controller-expand-secret-namespace: default
csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
csi.storage.k8s.io/node-stage-secret-namespace: default
csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
- discard
EOF
kubectl apply -f storageclass.yaml
这里的 clusterID 对应之前步骤中的 fsid。 imageFeatures 用来确定创建的 image 特征,如果不指定,就会使用 RBD 内核中的特征列表,但 Linux 不一定支持所有特征,所以这里需要限制一下。
进入 ceph-csi
项目的 example/rbd
目录,然后直接创建 PVC
[root@lab-c2009-k8s-aio-ceph rbd]# kubectl apply -f pvc.yaml
persistentvolumeclaim/rbd-pvc created
[root@lab-c2009-k8s-aio-ceph rbd]# kubectl get pvc
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
default rbd-pvc Pending csi-rbd-sc 7s
[root@lab-c2009-k8s-aio-ceph rbd]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
rbd-pvc Bound pvc-d45b457f-9c67-4b82-b28b-329a1eb1747d 1Gi RWO csi-rbd-sc 4m30s
[root@lab-c2009-k8s-aio-ceph rbd]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-d45b457f-9c67-4b82-b28b-329a1eb1747d 1Gi RWO Delete Bound default/rbd-pvc csi-rbd-sc 1s
在 ceph 侧,可以看到:
[root@lab-c2009-ceph-aio ~]# rbd ls -p kubernetes
csi-vol-7b95cc49-9c1f-11ec-855a-ea06f4beb16f
[root@lab-c2009-ceph-aio ~]# rbd info csi-vol-7b95cc49-9c1f-11ec-855a-ea06f4beb16f -p kubernetes
rbd image 'csi-vol-7b95cc49-9c1f-11ec-855a-ea06f4beb16f':
size 1 GiB in 256 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 37b4396277f1
block_name_prefix: rbd_data.37b4396277f1
format: 2
features: layering
op_features:
flags:
create_timestamp: Fri Mar 4 19:59:19 2022
access_timestamp: Fri Mar 4 19:59:19 2022
modify_timestamp: Fri Mar 4 19:59:19 2022
创建示例 Pod
[root@lab-c2009-k8s-aio-ceph rbd]# kubectl apply -f pod.yaml
[root@lab-c2009-k8s-aio-ceph rbd]# lsblk -l|grep rbd
rbd0 251:0 0 1G 0 disk /var/lib/kubelet/pods/9f47c56f-675c-40fc-9b6e-5f00676bf8d0/volumes/kubernetes.io~csi/pvc-d45b457f-9c67-4b82-b28b-329a1eb1747d/mount
[root@lab-c2009-k8s-aio-ceph rbd]# kubectl exec -it csi-rbd-demo-pod -- bash
root@csi-rbd-demo-pod:/# lsblk -l|grep rbd
rbd0 251:0 0 1G 0 disk /var/lib/www/html
遇到问题需要排查的话:kubectl logs csi-rbdplugin-provisioner-5b4464666c-bzxf7 -c csi-provisioner
PVC 是不可以跨 namespace 复制的。参考:https://kubernetes.io/docs/concepts/storage/volume-pvc-datasource/
You can only clone a PVC when it exists in the same namespace as the destination PVC (source and destination must be in the same namespace).
可以借用 snapshot 来完成跨 namespace 复制 PVC,参考:https://v1-20.docs.kubernetes.io/docs/concepts/storage/volume-snapshots/
VolumeSnapshotContent 是 cluster 级别
create Snapshot(会自动生成 SnapshotContent 和 volumeHandle)
create new SnapshotContent(根据 volumeHandle)
注意,创建时需要带两条注解,参考:https://github.com/kubernetes-csi/external-snapshotter/issues/251,否则删除新创建的 pvc 时,会 hang 住
snapshot.storage.kubernetes.io/deletion-secret-namespace: default
snapshot.storage.kubernetes.io/deletion-secret-name: csi-rbd-secret
create Snapshot in other namespace(根据新的 SnapshotContent)
create pvc by snapshot(根据新的 snapshot)
底层基于快照,在 Ceph 上飞快(copy on write)
参考:https://github.com/rancher/local-path-provisioner
[root@lab-k8s004 ~]# kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.22/deploy/local-path-storage.yaml
namespace/local-path-storage created
serviceaccount/local-path-provisioner-service-account created
clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created
deployment.apps/local-path-provisioner created
storageclass.storage.k8s.io/local-path created
configmap/local-path-config created
[root@lab-k8s004 ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path rancher.io/local-path Delete WaitForFirstConsumer false 50s
[root@lab-k8s004 ~]# kubectl patch storageclass local-path -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class":"true"}}}'
storageclass.storage.k8s.io/local-path patched
[root@lab-k8s004 ~]# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 75s
[root@lab-k8s004 ~]# kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pvc/pvc.yaml
persistentvolumeclaim/local-path-pvc created
[root@lab-k8s004 ~]# kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod/pod.yaml
pod/volume-test created
[root@lab-k8s004 ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-013cdb6b-fe6e-4fa8-9e5b-8a5e4957a91b 128Mi RWO Delete Bound default/local-path-pvc local-path 23s
[root@lab-k8s004 ~]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
local-path-pvc Bound pvc-013cdb6b-fe6e-4fa8-9e5b-8a5e4957a91b 128Mi RWO local-path 40s
Container Network Interface (CNI) 最早是由 CoreOS 发起的容器网络规范,是 Kubernetes 网络插件的基础。其基本思想为:Container Runtime 在创建容器时,先创建好 network namespace,然后调用 CNI 插件为这个 netns 配置网络,其后再启动容器内的进程。现成为 CNCF 主推的网络模型。
CNI 插件包括两部分:CNI Plugin 和 IPAM Plugin
Kubernetes Pod 中的其他容器都是 Pod 所属 pause 容器的网络,创建过程为:
参考:
所有 CNI 插件均支持通过环境变量和标准输入传入参数:
echo '{"cniVersion": "0.3.1","name": "mynet","type": "macvlan","bridge": "cni0","isGateway": true,"ipMasq": true,"ipam": {"type": "host-local","subnet": "10.244.1.0/24","routes": [{ "dst": "0.0.0.0/0" }]}}' | sudo CNI_COMMAND=ADD CNI_NETNS=/var/run/netns/a CNI_PATH=./bin CNI_IFNAME=eth0 CNI_CONTAINERID=a CNI_VERSION=0.3.1 ./bin/bridge
echo '{"cniVersion": "0.3.1","type":"IGNORED", "name": "a","ipam": {"type": "host-local", "subnet":"10.1.2.3/24"}}' | sudo CNI_COMMAND=ADD CNI_NETNS=/var/run/netns/a CNI_PATH=./bin CNI_IFNAME=a CNI_CONTAINERID=a CNI_VERSION=0.3.1 ./bin/host-local
参考:https://github.com/projectcalico/calico
参考:https://github.com/kubeovn/kube-ovn#kube-ovn-vs-calico
参考:https://github.com/k8snetworkplumbingwg/multus-cni
参考:https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin
参考:https://kubernetes.io/docs/concepts/services-networking/network-policies/
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: test-network-policy
namespace: default
spec:
# podSelector: {} 表示选择所有 pod 应用 NetworkPolicy
podSelector: # 表示选择包含标签 role=db 的 pod 应用下面的 NetworkPolicy
matchLabels:
role: db
policyTypes: # 表示 NetworkPolicy 包含 ingress 和 egress 流量规则
- Ingress
- Egress
ingress: # ingress 规则白名单列表,每条规则允许同时匹配 from 和 ports 流,可以有条个规则。
# 第 1 条白名单,包含 from + ports 的组合规则,允许来自172.17网段(172.17.1除外)、或标 签 project=myproject 的命名空间的所 有 pod 、或 default 命名空间下标签 role=frontend 的 pod 访问(限 tcp 6379 端口)
- from:
- ipBlock:
cidr: 172.17.0.0/16
except:
- 172.17.1.0/24
- namespaceSelector:
matchLabels:
project: myproject
- podSelector:
matchLabels:
role: frontend
ports:
- protocol: TCP
port: 6379
# 第二条白名单,只包含 from 规则,允许来自所有命名空间包含 environment=testing 标签的 pod 访问(不限端口)
- from:
- namespaceSelector: {}
podSelector:
matchLabels:
environment: testing
egress: # egress 规则白名单列表,同 ingress 规则一样,每条规则包含 to+ports,可以有多条规则。
- to:
- ipBlock:
cidr: 10.0.0.0/24
ports:
- protocol: TCP
port: 5978
参考:https://kubernetes.io/docs/concepts/security/pod-security-policy/,1.25 以后转 https://kubernetes.io/docs/concepts/security/pod-security-admission/ (1.23 是 beta)
psp 的用法,包括 5 步骤
创建 psp
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restrict-policy
spec:
privileged: false
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
runAsUser:
rule: RunAsAny
fsGroup:
rule: RunAsAny
volumes:
- '*'
创建 clusterrole,使用 psp
kubectl create clusterrole restrict-access-role --verb=use --resource=psp --resource-name=restrict-policy
创建 serviceaccount
kubectl create sa psp-denial-sa -n staging
绑定 clusterrole 到 serviceaccount
kubectl create clusterrolebinding dany-access-bind --clusterrole=restrict-access-role --serviceaccount=staging:psp-denial-sa
启用 PSP
vi /etc/kubernetes/manifests/kube-apiserver.yaml
# 确保有以下内容:
- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy
参考:https://kubernetes.io/zh/docs/tutorials/security/apparmor/
apiVersion: v1
kind: Pod
metadata:
name: podx
annotations:
container.apparmor.security.beta.kubernetes.io/podx: localhost/nginx-profile-3
spec:
containers:
- image: busybox
imagePullPolicy: IfNotPresent
name: podx
command: [ "sh", "-c", "echo 'Hello AppArmor!' && sleep 1h" ]
resources: {}
nodeName: node01
dnsPolicy: ClusterFirst
restartPolicy: Always
serviceaccount 有个选项 automountServiceAccountToken, 这个选项决定是否自动挂载 secret 到 pod。 有这个选项,我们可以控制 pod 创建并绑定 serviceaccount 时,不自动挂载对应的 secret,这样 pod 就没有权限访问 apiserver,提高了业务 pod 的安全性。
可以在 serviceaccount 和 pod 的 spec 里设置,pod 的设置优先于 serviceaccount 里的设置。
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend-sa
namespace: qa
automountServiceAccountToken: false
apiVersion: v1
kind: Pod
metadata:
name: backend
namespace: qa
spec:
serviceAccountName: backend-sa
containers:
- image: nginx:1.9
imagePullPolicy: IfNotPresent
name: backend
删除未使用的 serviceaccount
自动挂载的 service token 的 pod 可以做很多事,比如 get secret
k -n restricted get pod -o yaml | grep automountServiceAccountToken
curl https://kubernetes.default/api/v1/namespaces/restricted/secrets -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -k
参考:https://github.com/aquasecurity/kube-bench
$ kubectl apply -f https://github.com/aquasecurity/kube-bench/raw/main/job.yaml
job.batch/kube-bench created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kube-bench-j76s9 0/1 ContainerCreating 0 3s
# Wait for a few seconds for the job to complete
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
kube-bench-j76s9 0/1 Completed 0 11s
# The results are held in the pod's logs
kubectl logs kube-bench-j76s9
[INFO] 1 Master Node Security Configuration
[INFO] 1.1 API Server
...
kube-bench
是一个 CIS 评估工具,扫描 kubernetes 集群存在的安全问题,基本上按照扫描结果的修复建议进行修复就可以了,系统会给出很具体的修复措施。
kube-bench run --targets=master | grep kube-controller -A 3
kube-bench run --targets=node | grep /var/lib/kubelet/config.yaml -B2
比较常见的是 kubeadm 创建的 kubernetes 服务器权限设置有问题,允许未经授权的访问。
使用 Node,RBAC 授权模式和 NodeRestriction 准入控制器
```yaml
vi /etc/kubernetes/manifests/kube-apiserver.yaml
# 确保以下内容
- --authorization-mode=Node,RBAC
- --enable-admission-plugins=NodeRestriction
- --client-ca-file=/etc/kubernetes/pki/ca.crt
- --enable-bootstrap-token-auth=true
```
system:anonymous 的 ClusterRolebinding 角色绑定,匿名用户有集群管理员权限,需要取消
```bash
kubectl delete clusterrolebinding system:anonymous
```
# 修复 kube-apiserver 安全问题
vi /etc/kubernetes/manifests/kube-apiserver
#修改:
--authorization-mode=Node,RBAC
#添加
--insecure-port=0
#删除
# --insecure-bind-address=0.0.0.0
#修复 kubelet 安全问题
vi /var/lib/kubelet/config.yaml
# 将authentication.anonymous.enabled 设置为 false
authentication:
anonymous:
enabled: false
# authorization.mode 设置为 Webhook
authorization:
mode: Webhook
# 修复 etcd 安全问题
vi /etc/kubernetes/manifests/etcd.yaml
# 修改为true:
- --client-cert-auth=true
# 以上修复完成后 重新加载配置文件并重启kubelet
systemctl daemon-reload
systemctl restart kubelet
启用了特权的 pod,主要是检查 pod 是否含 privileged: true
kubectl get po xxx -n production -o yaml| grep -i "privileged: true"
检查有状态 pod
kubectl get pods XXXX -n production -o jsonpath={.spec.volumes} | jq
参考:https://sysdig.com/use-cases/kubernetes-security/
sysdig 的基本用法,记住两个帮助命令:
sysdig -h
查看 sysdig 帮助sysdig -l
查看 sysdig 支持的元数据另外 sysdig 支持指定 containerid 分析特定容器
# 查看容器id
docker ps |grep tomcat
sysdig -M 30 -p "*%evt.time,%user.uid,%proc.name" container.id=xxxx>opt/DFA/incidents/summary
dockerfile 里常见的不安全指令包括:
USER root
securityContext:
{"Capabilities": {'add':{NET_BIND_SERVICE}, 'drop: []'}, 'privileged': TRUE}
镜像扫描工具 trivy,参考:https://github.com/aquasecurity/trivy
# 获取镜像名
kubect get pod XXXX -n kamino -o yaml | grep image
# 扫描镜像
trivy imagename | grep (HIGH|CRITICAL)
# kubectl delete po xxx
参考:https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
启动日志审计,包括两个步骤:
编写日志审计策略文件
apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "RequestReceived"
rules:
- level: RequestResponse
resources:
- group: ""
resources: ["namespaces"]
- level: Request
resources:
- group: ""
resources: ["persistentvolumes"]
namespaces: ["front-apps"]
- level: Metadata
resources:
- group: ""
resources: ["secrets", "configmaps"]
- level: Metadata
omitStages:
- "RequestReceived"
修改 kube-apiserver.yaml
配置文件,启用日志审计策略,日志策略配置文件位置、日志文件存储位置、循环周期。
vi /etc/kubernetes/manifests/kube-apiserver.yaml
# 设置日志审计策略文件在 pod 里的 mount 位置
- --audit-policy-file=/etc/kubernetes/logpolicy/sample-policy.yaml
# 设置日志文件存储位置
- --audit-log-path=/var/log/kubernetes/audit-logs.txt
# 设置日志文件循环
- --audit-log-maxage=10
- --audit-log-maxbackup=2
# mount 日志策略和日志文件的
volumeMounts:
- mountPath: /etc/kubernetes/logpolicy/sample-policy.yaml
name: audit
readOnly: true
- mountPath: /var/log/kubernetes/audit-logs.txt
name: audit-log
readOnly: false
volumes:
- name: audit
hostPath:
path: /etc/kubernetes/logpolicy/sample-policy.yaml
type: File
- name: audit-log
hostPath:
path: /var/log/kubernetes/audit-logs.txt
type: FileOrCreate
重启 API Server 来检查 audit.log
cd /etc/kubernetes/manifests/
mv kube-apiserver.yaml ..
watch crictl ps # wait for apiserver gone
truncate -s 0 /etc/kubernetes/audit/logs/audit.log
mv ../kube-apiserver.yaml .
cat audit.log | tail | jq
# shows Secret entries
cat audit.log | grep '"resource":"secrets"' | wc -l
# confirms Secret entries are only of level Metadata
cat audit.log | grep '"resource":"secrets"' | grep -v '"level":"Metadata"' | wc -l
# shows RequestResponse level entries
cat audit.log | grep -v '"level":"RequestResponse"' | wc -l
# shows RequestResponse level entries are only for system:nodes
cat audit.log | grep '"level":"RequestResponse"' | grep -v "system:nodes" | wc -l
ImagePolicyWebhook
参考:https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#imagepolicywebhook
ImagePolicyWebhook 准入控制器的使用,分 4 个步骤
修改控制器配置文件,将未找到有效后端时的默认拒绝改为默认不拒绝
vi /etc/kubernetes/epconfig/admission_configuration.json
{
"imagePolicy": {
"kubeConfigFile": "/etc/kubernetes/epconfig/kubeconfig.yaml",
"allowTTL": 50,
"denyTTL": 50,
"retryBackoff": 500,
"defaultAllow": false
}
}
修改控制器访问 webhook server 的 kubeconfig
vi /etc/kubernetes/epconfig/kubeconfig.yaml
修改如下内容
apiVersion: v1
kind: Config
clusters:
- cluster:
certificate-authority: /etc/kubernetes/epconfig/webhook.pem
server: https://acme.local:8082/image_policy # web hook server 的地址
name: bouncer_webhook
# 以下省略
启用 ImagePolicyWebhook
vi /etc/kubernetes/manifests/kube-apiserver.yaml
# 启用 ImagePolicyWebhook
- --enable-admission-plugins=NodeRestriction,ImagePolicyWebhook
# 指定准入控制器配置文件
- --admission-control-config-file=/etc/kubernetes/epconfig/admission_configuration.json
# mount
volumeMounts:
- mountPath: /etc/kubernetes/epconfig
name: epconfig
# 映射 volumes
volumes:
- name: epconfig
hostPath:
path: /etc/kubernetes/epconfig
测试是否生效
systemctl daemon-reload
systemctl restart kubelet
kubectl apply -f /cks/img/web1.yaml
综上~
翰博 帝先 景嘉微
apiVersion: v1
kind: Pod
metadata:
name: gpu-test
spec:
containers:
- name: cuda
command: ["sleep", "infinity"]
image: nvidia/cuda:12.2.0-base-ubuntu20.04
resources:
limits:
cpu: 0.5
memory: 500Mi
nvidia device plugin 默认是通过环境变量(NVIDIA_VISIBLE_DEVICES)来判断是否要将 gpu 暴露给容器的,而 nvidia 的大部分镜像都是基于 cuda 的,cuda 镜像默认会将这个变量的 value 设置为 all,参考 https://hub.docker.com/layers/nvidia/cuda/12.2.0-base-ubuntu20.04/images/sha256-5a9257b8c6ac3f6c53ce559903558dce5ec1fa7a91ed8020dfb56ab7af1a8733?context=explore,也可以从容器中验证:
root@node1:~# kubectl exec -it gpu-test -- env
...
NVARCH=x86_64
NVIDIA_REQUIRE_CUDA=cuda>=12.2 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
NV_CUDA_CUDART_VERSION=12.2.53-1
NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-2
CUDA_VERSION=12.2.0
...
NVIDIA_VISIBLE_DEVICES=all
...
NVIDIA_DRIVER_CAPABILITIES=compute,utility
root@node1:~# kubectl exec -it gpu-test -- nvidia-smi
Tue Oct 10 04:22:36 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.104.12 Driver Version: 535.104.12 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 On | 00000000:09:00.0 Off | Off |
| N/A 28C P8 15W / 70W | 2MiB / 16384MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
root@node1:~# kubectl exec -it gpu-test -- df -h
Filesystem Size Used Avail Use% Mounted on
...
udev 2.0G 0 2.0G 0% /run/nvidia-container-devices/GPU-5fa68d66-4cb3-891a-bf8e-45519ce2b3ee
...
所以就导致了,如果容器没有申请 gpu 则会使用默认的环境变量,从而将所有的 gpu 暴露给容器(这部分工作应该是由 hook 完成的);如果容器申请了 gpu 则在创建容器时 device plugin 会去覆盖这个环境变量,变成所申请的 gpu 序号,申请 gpu 的容器环境变量如下所示:
root@ecs-gpu:~# kubectl exec -it gpu-test -- env
...
NVIDIA_VISIBLE_DEVICES=GPU-5fa68d66-4cb3-891a-bf8e-45519ce2b3ee
...
社区给出了一个临时的解决方案,将 gpu 发现方式改为 volume-mount,不过也在文档中提到了,未来的最终解决方案会是 CDI。下面以 gpu-operator 部署的集群为例来演示一下流程,仅部署 nvidia device plugin 的集群也可以通过修改 plugin 的启动参数 --device-list-strategy=<envvar | volume-mounts> 来进行修改。
先修改 gpu-operator 的 cr cluster-policy
# kubectl edit clusterpolicy cluster-policy
devicePlugin:
env:
- name: DEVICE_LIST_STRATEGY
value: volume-mounts
再修改 toolkit 配置,配置文件的位置分为两种情况:
修改配置文件,改 accept-nvidia-visible-devices-*
相关变量:
disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"
#
accept-nvidia-visible-devices-envvar-when-unprivileged = false
accept-nvidia-visible-devices-as-volume-mounts = true
[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
...
此时,如果不申请 GPU,Pod 中就不会主动挂载:
root@node1:~# kubectl exec -it gpu-test -- nvidia-smi
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "fc8e28159bca207f9023881f20df1fdad46edc3c49c69576e616b12084de239c": OCI runtime exec failed: exec failed: unable to start container process: exec: "nvidia-smi": executable file not found in $PATH: unknown
从另一个维度来思考,不限制的话,Pod 能看到所有的 CPU / 内存,那么看到所有的 GPU 也合理。
需要限制的话,就显示限制。如果要默认限制,那就用 LimitRange
apiVersion: v1
kind: LimitRange
metadata:
name: gpu-resource-constraint
namespace: default
spec:
limits:
- default:
nvidia.com/gpu: "0"
defaultRequest:
nvidia.com/gpu: "0"
type: Container
创建 pod
apiVersion: v1
kind: Pod
metadata:
name: gpu-test
spec:
containers:
- name: cuda
command: ["sleep", "infinity"]
image: nvidia/cuda:12.2.0-base-ubuntu20.04
创建出的 pod 会自动添加 limit
# kubectl get po gpu-test -0 yaml i grep -i request -A 10 -B 10
spec:
containers:
- command:
- sleep
- infinity
image: nvidia/cuda:12.2.0-base-ubuntu20.04
imagePullPolicy: IfNotPresent
name: cuda
resources:
limits:
nvidia.com/gpu: "0"
requests:
nvidia.com/gpu: "0"
除了大模型,通常我们不需要完整的一块显卡算力来做推理。这时我们就需要切分 GPU 算力给到多个 Pod 同时使用。
切分 GPU 以充分利用算力,这并不符合 Nvidia 希望多卖显卡的圈钱逻辑,所以 Nvidia 并不热衷于推 GPU 切分方案。
在虚拟化平台上,Nvidia 有 vGPU 的解决方案,但也需要额外卖 License(通过一个 vGPU license server 服务来授权)。真妥妥的雁过拔毛、为富不仁。这种“不仁”也必须是当然的了,Nvidia GPU 那么贵,而且“正经客户”还不能买游戏卡做训练,逼良为那啥。
通过容器来切分 GPU 的用户故事大致如下:
参考:https://github.com/zw0610/zw0610.github.io/blob/master/notes-cn/gpu-sharing-1.md
为什么需要切分 GPU?
被共享的 GPU 算力资源包括哪些?
GPU 使用场景大致分为 3 类:推理,训练,其它
虽然推理服务适合 GPU 切片,但不是唯一的,还有其它解决思路,比如:Nvidia Triton Inference Server,实现同一个 context 下利用多 stream 的方式实现多个模型同时实现前向推理功能的。结合更灵活且智能的 model scheduling,有可能效果更好(需要验证)。
默认的调度器无法处理 GRES,一般来说需要安装对应的 GRES Plugin。如果在部署 Slurm 之前就已经在集群上装好了 NVML(一般随着驱动就会安装),Slurm 会利用
select/cons_tres plugin 自动进行检测并将(Nvidia)GPU 资源登记。 而在调度 GPU 任务时,Slurm 依旧使用 CUDA_VISIBLE_DEVICES
来对多
GPU 的节点进行以卡为单位的 GPU 资源隔离。每个任务在节点上被拉起时,Slurm 都会在 Epilog 和 Prolog 来配置 CUDA_VISIBLE_DEVICES
使得任务可以使用的
GPU 受到限制。这些配置在 Slurm 的 master 节点上完成。
Slurm 通过 GRES(Generic Resources,非常见资源) 支持 Nvidia CUDA Multi-Process Service
MPS。MPS 允许多个小任务同时运行在一块
GPU 上,相比较 GPU context 切换,损耗(overhead)较低 。MPS 通过设置 CUDA_MPS_ACTIVE_THREAD_PERCENTAGE
来限制每个任务的算力,取值在
(0,100] 之间(即百分比)。Slurm 对 GPU 共享的调度已经做到相当原生。而如果有进一步的需求,也可以通过
Slurm Plugin API 自己来实现一个。
K8S 通过 Device Plugin 来实现 GPU 的非共享调度。想要做到共享切片,必须:
与 Slurm + MPS 按照算力分割略有不同的是,K8S 切分方案(比如阿里)的方案以显存为分割尺度,并且默认地认为 GPU 算力的需求和显存的需求是成正比的。这有一定合理性。
Nvidia 原则两种 GPU 共享的方案:
Ampere 带来的 A100 所具备的 Multi-Instance GPU (MIG):
总体来看,利用 Ampere 这代架构在硬件上的隔离,无论是公有容器云还是私有容器云都可以很快地部署带 GPU 共享的 Kubernetes 集群,并且做到完整的算力、显存隔离而不需要额外的一些组件。
参考:https://github.com/AliyunContainerService/GPUshare-scheduler-extender
实现原理:
CUDA_VISIBLE_DEVICES
以便实现 GPU 与 GPU 之间的隔离,最后标记为 assigned。TKEStack 的 GaiaGPU:https://github.com/tkestack/gpu-manager,cuda 12 之后就不支持了。
CUDA 任务使用显存的基本逻辑(以申请一块 NxN 的矩阵为例):
当多个任务同时在一块 GPU 上进行显存申请的时候,我们希望做到和上述步骤不同的地方是:2. 查询显存总量/剩余显存总量时,返回的不希望是整块 GPU 的显存相关信息,而是限制之后的相关信息。举个例子,一块 GPU 一共有 12Gi 显存,我们分配给任务 A 5Gi 显存,给任务 B 3Gi 显存。当任务 A 中的进程调用 cuDeviceTotalMem 时,应该返回 5Gi。假设任务 A 中的进程已经使用了 1.5 Gi 的显存,那么当调用 cuMemGetInfo 时,应该返回 3.5 Gi。
而如果有的用户程序不经过步骤 2,直接执行步骤 3 呢?显然,我们需要根据上述逻辑对 如果剩下的显存量够多 作出判断。
那么要修改 CUDA 函数的实现,一般可以走 LD_PRELOAD
或 LD_LIBRARY_PATH
两种方式。前者的模式适用范围有限,无法进一步劫持由 Runtime API 对
Driver API 的调用。因此 TKEStack 采取修改 libcuda 并通过 LD_LIBRARY_PATH
加载。具体的代码可以参看:vcuda-controller。
每次发起 CUDA kernel 的时候,都检查一下当前任务对 GPU 的使用率,并对本次 CUDA kernel 会增加的 GPU 使用率作出估计。如果预计本次 CUDA kernel 会使得 GPU 使用率超标,则延缓 kernel 的运行,直到当前的 GPU 使用率下降至允许本次 CUDA kernel 的运行之后。然后这样做,无法避免多个任务的 context 切换带来的 overhead。
劫持的工程实现是做在 vcuda-controller 上的。
VirtAI 社区版其实只是供社区使用的一个安装包,不含任何代码。通过分析安装包内的信息大致作出推测:
第四范式开源的 vgpu 上层实现,底层核心逻辑是 libvgpu.so 提供的,没有开源,可以实现对物理 gpu 的切分,实现了显存隔离。
参考:https://github.com/4paradigm/k8s-vgpu-scheduler#quick-start,环境中装好 OS,GPU 驱动, K8S with nvidia containerd。
kubectl label nodes {nodeid} gpu=on
helm repo add vgpu-charts https://4paradigm.github.io/k8s-vgpu-scheduler
helm install vgpu vgpu-charts/vgpu --set scheduler.kubeScheduler.imageTag=v1.23.17 -n kube-system
kubectl apply -f test-gpu-2.yaml
# cat test-gpu-2.yaml
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod-2
spec:
containers:
- name: ubuntu-container
image: ubuntu:18.04
command: ["bash", "-c", "sleep 86400"]
resources:
limits:
nvidia.com/gpu: 2 # requesting 2 vGPUs
nvidia.com/gpumem: 3000 # Each vGPU contains 3000m device memory (Optional,Integer)
nvidia.com/gpucores: 30 # Each vGPU uses 30% of the entire GPU (Optional,Integer)
进 pod 看:
root@cloud-NF5468M6:~# kubectl exec -it gpu-pod-2 -- bash
root@gpu-pod-2:/# nvidia-smi
[4pdvGPU Msg(17:139640230053696:libvgpu.c:870)]: Initializing.....
Thu Oct 26 10:19:58 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.113.01 Driver Version: 535.113.01 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A40 Off | 00000000:9D:00.0 Off | 0 |
| 0% 39C P0 87W / 300W | 0MiB / 3000MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A40 Off | 00000000:A0:00.0 Off | 0 |
| 0% 77C P0 299W / 300W | 0MiB / 3000MiB | 96% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
+---------------------------------------------------------------------------------------+
[4pdvGPU Msg(17:139640230053696:multiprocess_memory_limit.c:475)]: Calling exit handler 17
符合预期。
此处可能存在不合适展示的内容,页面不予展示。您可通过相关编辑功能自查并修改。
如您确认内容无涉及 不当用语 / 纯广告导流 / 暴力 / 低俗色情 / 侵权 / 盗版 / 虚假 / 无价值内容或违法国家有关法律法规的内容,可点击提交进行申诉,我们将尽快为您处理。