container,docker,Linux,Kubernetes

Kubernetes 1.15.0 Cluster Deploy

基础环境

  本次部署,采用一个master节点和两个worker节点,如下图,后续要部署Istio,不想再画图了,此处看到请忽略

  各节点配置如下表,个节点均无Swap

节点OSHardWare
masterUbuntu 1804 LTS8G8C50G
node1Ubuntu 1804 LTS8G8C100G
node2Ubuntu 1804 LTS8G8C100G

  各节点组件部署规划如下表

IPHostnameRoleComponent
172.20.35.150masterMasterkube-apiserver, kube-controller-manager, kube-scheduler, etcd, (kubectl) ,kubelet, kube-proxy, docker, flannel
172.20.35.151node1Workerkubelet, kube-proxy, docker, flannel
172.20.35.152node2Workerkubelet, kube-proxy, docker, flannel

节点或组件功能简介

  • Master节点: Master节点上面主要由四个模块组成,apiserver,schedule,controller-manager,etcd
  • apiserver: 负责对外提供RESTfulkubernetes API的服务,它是系统管理指令的统一接口,任何对资源的增删该查都要交给apiserver处理后再交给etcd. kubectl(kubernetes提供的客户端工具,该工具内部是对kubernetes API的调用)是直接和apiserver交互的
  • scheduler: 负责调度Pod到合适的Node上,如果把scheduler看成一个黑匣子,那么它的输入是pod和由多个Node组成的列表,输出是Pod和一个Node的绑定. kubernetes目前提供了调度算法,同样也保留了接口. 用户根据自己的需求定义自己的调度算法
  • controller-manager: 如果apiserver做的是前台的工作的话,那么controller-manager就是负责后台的. 每一个资源都对应一个控制器,而control manager就是负责管理这些控制器的,比如我们通过APIServer创建了一个Pod,当这个Pod创建成功后,apiserver的任务就算完成了
  • etcd: etcd是一个高可用的键值存储系统,kubernetes使用它来存储各个资源的状态,从而实现了RestfulAPI
  • Node节点: 每个Node节点主要由二个模块组成: kublet,kube-proxy
  • kube-proxy: 该模块实现了kubernetes中的服务发现和反向代理功能,kube-proxy支持TCPUDP连接转发,默认基Round Robin算法将客户端流量转发到与service对应的一组后端pod. 服务发现方面,kube-proxy使用etcdwatch机制监控集群中serviceendpoint对象数据的动态变化,并且维护一个serviceendpoint的映射关系,从而保证了后端podIP变化不会对访问者造成影响,此外,kube-proxy还支持session affinity
  • kublet: kubletMaster在每个Node节点上面的agent,是Node节点上面最重要的模块,它负责维护和管理该Node上的所有容器,但是如果容器不是通过kubernetes创建的,它并不会管理. 本质上,它负责使Pod的运行状态与期望的状态一致

Kubernetes工作流程

  Kubernetes工作流程如下图所示

安装环境初始化设置

为方便安装配置过程操作方便,全程以root身份进行操作,实际生产环境中不建议如此,Ubuntu直接使用root用户远程登录设置可参考Ubuntu配置root用户ssh连接,此处不在赘述

主机名解析

节点较少,故此不做DNS解析,直接通过hosts文件进行解析

$ vim /etc/hosts
...
172.20.35.150    master    master.ccav.tech
172.20.35.151    node1    node1.ccav.tech
172.20.35.152    node2    node2.ccav.tech

配置apt

$ > /etc/apt/sources.list

$ cat <<EOF | tee /etc/apt/sources.list
eb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
EOF

$ apt update
$ apt install -y apt-transport-https

$ curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add - 

$ cat <<EOF |tee /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF  

配置时钟服务

  集群中如果个节点间时间不同步,那么可能会导致各种千奇百怪的问题,在集群初始化之前定要确保个节点间时钟同步,下面配置时钟服务,个节点间均执行以下操作

$ apt update
$ apt install chrony

$ vim /etc/chrony/chrony.conf

...
pool 172.20.0.1 iburst maxsources 4  #将时钟服务器指向172.20.0.1
...

$ systemctl enable --now chrony

加载ipvs模块

  • 所有节点安装ipvs相关模块

    &nbsp;&nbsp;ipvs性能甩iptables几条街并且排错更直观
    
$ apt install wget git conntrack ipvsadm ipset jq sysstat curl iptables libseccomp
  • 所有节点选择需要开机加载的内核模块,以下是ipvs模式需要加载的模块并设置开机自动加载
$ > /etc/modules
$ cat <<EOF | tee /etc/modules
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
nf_conntrack
EOF

$ systemctl enable --now systemd-modules-load.service
  • 所有节点设定k8s运行时内核参数
$ cat <<EOF | tee /etc/sysctl.d/k8s.conf
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.ip_forward = 1
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.netfilter.nf_conntrack_max = 2310720
fs.inotify.max_user_watches = 89100
fs.may_detach_mounts = 1
fs.file-max = 52706963
fs.nr_open = 52706963
net.bridge.bridge-nf-call-arptables = 1
vm.swappiness = 0
vm.overcommit_memory = 1
vm.panic_on_oom = 0
EOF

$ sysctl --system

安装依赖组件

根据事先规划,在master节点安装kubelet,kubeadm,kubectl,docker. worker节点安装kubelet,kubeadm,docker

  • 安装docker
curl -fsSL https://get.docker.com | bash -s docker --mirror Aliyun
  • 安装kubelet,kubeadm,kubectl
apt update
apt install -y kubelet kubeadm kubectl
  • 配置docker加速,使用于kubelet相同的文件驱动
$ mkdir /etc/docker/
$ cat <<EOF | tee /etc/docker/daemon.json
{
  "registry-mirrors": ["https://2n14cd7b.mirror.aliyuncs.com"],
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
  • 配置docker转发策略

    &nbsp;&nbsp;`docker`自`1.13`版起会自动设置`iptables`的`FORWARD`默认策略为`DROP`,这可能会影响`Kubernetes`集群依赖的报文转发功能,需要在`docker`服务启动后,重新将`FORWARD`链的默认策略设为`ACCEPT`,同样是需要修改`docker Unit File`, 在`ExecStart=/usr/bin/dockerd -H`一行之后新增一行
    
$ vim /lib/systemd/system/docker.service
...
ExecStartPost=/sbin/iptables -P FORWARD ACCEPT
...

$ systemctl daemon-reload
$ systemctl restart docker

初始化master节点

由于众所周知的原因,在国内无法直接访问Google的服务,故,此处提供了已拉取下来的镜像,获取镜像密码:0uh9,下载后自行校验

imagemd5sum
coredns-1.3.1.tgzfce00f0bf1fa8cbabb3ad5808ff0e5a5
etcd-3.3.10.tgz5c35448f694269fb7426a1aab51c6b8f
kube-apiserver-v1.15.0.tgz7fac93d3b6d9eced68396a5445c5b601
kube-controller-manager-v1.15.0.tgz1a949c7c24cd32ee18947a81518b106c
kube-proxy-v1.15.0.tgz2adbf0cc7bf464517cd60940b34960b6
kubernetes-dashboard-amd64-v1.10.1.tgz2ec058fb2d1a9ff95d94745e776b3351
kube-scheduler-v1.15.0.tgz7cd240a801d9d3e797615ba96f65db19
pause-3.1.tgz42e8961fd0b6b0ddd954bec4b6ce15ba

  将所有已下载的镜像文件加载后即可初始化集群中的master

$ docker load -i coredns-1.3.1.tgz
...

  集群初始化的方式有两种,一是通过命令行选项传递关键的部署设定, 另一个是基于yaml格式的专用配置文件, 后一种允许用户自定义各个部署参数

  • 下面是第一种方式
kubeadm init --pod-network-cidr=10.244.0.0/16 \
--service-cidr=10.96.0.0/12 \
--ignore-preflight-errors=SystemVerification \
--kubernetes-version=v1.15.0 \
--apiserver-advertise-address=172.20.35.150 \
--feature-gates CoreDNS=true
  • 下面是基于yaml格式的专用配置文件
$ cat << EOF | tee kubeadm-init.yaml
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.15.0
controlPlaneEndpoint: 172.20.35.150:6443
imageRepository: k8s.gcr.io
kubeProxy:
  config:
    mode: "ipvs"
    ipvs:
      ExcludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      syncPeriod: 30s
dns:
  type: CoreDNS
kubeletConfiguration:
  baseConfig:
    cgroupDriver: cgroupfs
    clusterDNS:
    - 10.96.0.10
    clusterDomain: cluster.local
    failSwapOn: false
    resolvConf: /etc/resolv.conf
    staticPodPath: /etc/kubernetes/manifests
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
EOF

kubeadm init --config=kubeadm-init.yaml
  • 初始化完成后,将看到成功初始化的信息

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.20.35.150:6443 --token qhlrdt.hgazhj58bt96b3pl \
    --discovery-token-ca-cert-hash sha256:3e593950b375b9c4c4408e9b180ae6132d1163cb3a613c8d3f7fd856ecf0b28b

  接下来按照以上提示进行操作即可,并把worker节点加入到集群中的命令复制到worker节点中执行,以便加入到集群中. 如不慎,将worker节点加入集群的命令遗失,可使用如下命令生成加入集群的命令

kubeadm token create --print-join-command
  • 复制认证为Kubernetes系统管理员的配置文件到root用户家目录
$ mkdir -p $HOME/.kube
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

worker节点加入到集群

  worker节点节点加入集群,只需执行master节点初始化之后输出的join指令即可,所有worker节点执行如下操作

$ kubeadm join 172.20.35.150:6443 --token qhlrdt.hgazhj58bt96b3pl \
    --discovery-token-ca-cert-hash sha256:3e593950b375b9c4c4408e9b180ae6132d1163cb3a613c8d3f7fd856ecf0b28b

部署网络插件

  添加fannel网络附件,并开启DirectRouting(替代默认的 vxlan)

  • kube-flannel.yaml中增加一行"DirectRouting": true
$ curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

$ vim kube-flannel.yml
...
net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "DirectRouting": true,
        "Type": "vxlan"
      }
    }
...

$ kubectl apply -f kube-flannel.yml

查看集群状态

$ kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}

$ kubectl get nodes
NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   11h   v1.15.0
node01   Ready    <none>   11h   v1.15.0
node02   Ready    <none>   10h   v1.15.0

$ kubectl get ns
NAME              STATUS   AGE
default           Active   11h
kube-node-lease   Active   11h
kube-public       Active   11h
kube-system       Active   11h

$ kubectl get pods -n kube-system
NAME                                    READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-4wg5t                1/1     Running   0          11h
coredns-5c98db65d4-p2kqk                1/1     Running   0          11h
etcd-master                             1/1     Running   0          11h
kube-apiserver-master                   1/1     Running   0          11h
kube-controller-manager-master          1/1     Running   0          11h
kube-flannel-ds-amd64-bfv7z             1/1     Running   0          11h
kube-flannel-ds-amd64-bl4nd             1/1     Running   1          10h
kube-flannel-ds-amd64-f7l8s             1/1     Running   0          11h
kube-proxy-6hczb                        1/1     Running   0          11h
kube-proxy-8x25w                        1/1     Running   1          10h
kube-proxy-bqhqr                        1/1     Running   0          11h
kube-scheduler-master                   1/1     Running   0          11h

  看似一切正常,一个简单的Kubernetes集群已经部署完成,可进一步进行测试和使用

测试

  进行一个简单的测试,如部署一个nginx,并通过在集群外部进行访问

$ kubectl run nginx --image=nginx:1.14-alpine --replicas=2
$ kubectl expose deployment nginx --port=80 --target-port=80 --type=NodePort

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-7bb7cd8db5-wp89k   1/1     Running   0          10h
nginx-7bb7cd8db5-xzsmz   1/1     Running   0          10h

$ kubectl get svc
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        11h
nginx        NodePort    10.102.144.149   <none>        80:30904/TCP   10h
  • 访问
$ curl http://172.20.35.150:30904/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

安装Dashboard

  由于kube-apiserver启用了RBAC授权,而官方源码目录的dashboard-controller.yaml没有定义授权的 ServiceAccount,所以后续访问API serverAPI时会被拒绝,此处使用以下方式进行Dashboard插件的安装

  • kube-system名称空间创建serviceaccount
$ kubectl create serviceaccount dashboard-admin -n kube-system
$ kubectl get sa -n kube-system | grep dashboard-admin
  • 创建clusterrolebinding
$ kubectl create clusterrolebinding dashboard-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
  • 部署Dashboard
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml
  • Dashboardservice类型改为NodePort
$ kubectl patch svc kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}' -n kube-system
  • 获取Dashboard-admin-token

    &nbsp;&nbsp;写了一堆shell,不会使用`go template`,留下了没技术的泪水
    
$ kubectl describe secret $(kubectl get secret -n kube-system | grep "dashboard-admin-token" | awk '{print $1}') -n kube-system
Name:         dashboard-admin-token-g5jp4
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: dashboard-admin
              kubernetes.io/service-account.uid: 8d9d9214-e9f3-4232-91f3-e3458303dabc

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJkYXNoYm9hcmQtYWRtaW4tdG9rZW4tZzVqcDQiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGFzaGJvYXJkLWFkbWluIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiOGQ5ZDkyMTQtZTlmMy00MjMyLTkxZjMtZTM0NTgzMDNkYWJjIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmRhc2hib2FyZC1hZG1pbiJ9.R1AhfQyANygelkZNsRF6z4bVrjnGCyY7xWAmyazlBr55f2psQtpsiN_X_bJ9Bv2Y3XOQ-1Y9eLKYAOuI-mRS6dVKcSilYRFGoK5hCbKQPgRC5uinLvAkwlxg1rGKVT7hEQZgo4zmCxVS5zuPtcQhHwR9zKrvY6BD3UP3vAmWv6_XcSpkXaMD0tKQQIOuDopnO1R7PD0V2eJjuuf9EITHrsCSyP3tfHR_l98Jmt3nof_C4dhCI_eG0TTGYnLuvpiN1HzfdQyKW4kmyQd5UVG6W0e-2H6VwO_HSRgboTZvuUmVdTtmnjU2L06j5j4j1_cegDCKdtAxHn_OWq8KUtjJGg

  此时已经获取到了Dashboard-admin-token,再查看服务端口即可通过浏览器使用Dashboard-admin-token的方式访问Dashboard

  • 查看Dashboard服务端口
$ kubectl get svc -n kube-system
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
kube-dns               ClusterIP   10.96.0.10     <none>        53/UDP,53/TCP,9153/TCP   12h
kubernetes-dashboard   NodePort    10.97.72.152   <none>        443:31521/TCP            123m
  • 浏览器访问,使用获取到的token即可
https://172.20.35.150:31521

微信扫一扫,向我赞赏

微信扫一扫,向我赞赏

微信扫一扫,向我赞赏

支付宝扫一扫,向我赞赏

回复

This is just a placeholder img.