搭建高可用etcd集群

发布： • 经验百科

搭建高可用etcd集群

初始化工作(master1-master2-node1都须要操作)

## 配置系统
###### 关闭 防火墙
systemctl stop firewalld
systemctl disable firewalld
​
###### 关闭 SeLinux
setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
​
###### 关闭 swap
swapoff -a
yes | cp /etc/fstab /etc/fstab_bak
cat /etc/fstab_bak |grep -v swap > /etc/fstab
​
###### yum epel源
yum install wget telnet -y
mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum clean all
yum makecache
​
###### 修改 /etc/sysctl.conf
modprobe br_netfilter
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF
sysctl -p /etc/sysctl.d/k8s.conf
​
###### 开启 ipvs
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4
​
# 设置 yum repository
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
​
# 安装并启动 docker
yum install -y docker-ce-18.09.8 docker-ce-cli-18.09.8 containerd.io
​
# 添加ipvs支持
yum install -y nfs-utils ipset ipvsadm

搭建高可用etcd集群

1、在master1上安装cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

2、安装etcd二进制文件

# 创建目录
mkdir -p /data/etcd/bin
# 下载
cd /tmp
wget https://storage.googleapis.com/etcd/v3.3.25/etcd-v3.3.25-linux-amd64.tar.gz
tar zxf etcd-v3.3.25-linux-amd64.tar.gz
cd etcd-v3.3.25-linux-amd64
mv etcd etcdctl /data/etcd/bin/

3、创建ca证书，客户端，服务端，节点之间的证书 Etcd属于server ,etcdctl 属于client，二者之间通过http协议进行通信。

  • ca证书 自己给自己签名的权威证书，用来给其他证书签名

  • server证书 etcd的证书

  • client证书 客户端，比如etcdctl的证书

  • peer证书 节点与节点之间通信的证书

1） 创建目录

mkdir -p /data/etcd/ssl
cd /data/etcd/ssl

2） 创建ca证书

2.1）创建vim ca-config.json

{
    "signing": {
        "default": {
            "expiry": "438000h"
        },
        "profiles": {
            "server": {
                "expiry": "438000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            },
            "client": {
                "expiry": "438000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
                "expiry": "438000h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

server auth表示client可以用该ca对server提供的证书进行验证 client auth表示server可以用该ca对client提供的证书进行验证

2.1）创建证书签名请求ca-csr.json 创建vim ca-csr.json

{
    "CN": "etcd",
    "key": {
        "algo": "rsa",
        "size": 2048
    }
}

2.3）生成CA证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca
# ls ca*
# ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem

 

3） 生成客户端证书

创建vim client.json

{
    "CN": "client",
    "key": {
        "algo": "ecdsa",
        "size": 256
    }
}

生成:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json  | cfssljson -bare client -
# ls ca*
# ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem client-key.pem client.pem

4） 生成server，peer证书

创建 vim etcd.json

{
    "CN": "etcd",
    "hosts": [
        "10.11.83.52",
        "10.11.83.53",
        "10.11.83.54"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "BJ",
            "ST": "BJ"
        }
    ]
}

生成:

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server etcd.json | cfssljson -bare server

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd.json | cfssljson -bare peer

5） 将master1的/data/etcd/ssl目录同步到master2和node1上

scp -r /data/etcd 10.11.83.53:/data/etcd
scp -r /data/etcd 10.11.83.54:/data/etcd

4、 systemd配置文件

vim /usr/lib/systemd/system/etcd.service

三台主机配置不一样,用的时候把注释最好删除

10.11.83.52:

#!/bin/bash

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/data/etcd/
ExecStart=/data/etcd/bin/etcd \
  --name=etcd-1 \
  --cert-file=/data/etcd/ssl/server.pem \
  --key-file=/data/etcd/ssl/server-key.pem \
  --peer-cert-file=/data/etcd/ssl/peer.pem \
  --peer-key-file=/data/etcd/ssl/peer-key.pem \
  --trusted-ca-file=/data/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/data/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls=https://10.11.83.52:2380 \
  --listen-peer-urls=https://10.11.83.52:2380 \
  --listen-client-urls=https://10.11.83.52:2379 \
  --advertise-client-urls=https://10.11.83.52:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=etcd-1=https://10.11.83.52:2380,etcd-2=https://10.11.83.53:2380,etcd-3=https://10.11.83.54:2380 \
  --initial-cluster-state=new \
  --data-dir=/data/etcd \
  --snapshot-count=50000 \
  --auto-compaction-retention=1 \
  --max-request-bytes=10485760 \
  --quota-backend-bytes=8589934592
Restart=always
RestartSec=15
LimitNOFILE=65536
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

10.11.83.53:

#!/bin/bash

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/data/etcd/
ExecStart=/data/etcd/bin/etcd \
  --name=etcd-2 \
  --cert-file=/data/etcd/ssl/server.pem \
  --key-file=/data/etcd/ssl/server-key.pem \
  --peer-cert-file=/data/etcd/ssl/peer.pem \
  --peer-key-file=/data/etcd/ssl/peer-key.pem \
  --trusted-ca-file=/data/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/data/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls=https://10.11.83.53:2380 \
  --listen-peer-urls=https://10.11.83.53:2380 \
  --listen-client-urls=https://10.11.83.53:2379 \
  --advertise-client-urls=https://10.11.83.53:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=etcd-1=https://10.11.83.52:2380,etcd-2=https://10.11.83.53:2380,etcd-3=https://10.11.83.54:2380 \
  --initial-cluster-state=new \
  --data-dir=/data/etcd \
  --snapshot-count=50000 \
  --auto-compaction-retention=1 \
  --max-request-bytes=10485760 \
  --quota-backend-bytes=8589934592
Restart=always
RestartSec=15
LimitNOFILE=65536
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

10.11.83.54:

#!/bin/bash

[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/data/etcd/
ExecStart=/data/etcd/bin/etcd \
  --name=etcd-3 \
  --cert-file=/data/etcd/ssl/server.pem \
  --key-file=/data/etcd/ssl/server-key.pem \
  --peer-cert-file=/data/etcd/ssl/peer.pem \
  --peer-key-file=/data/etcd/ssl/peer-key.pem \
  --trusted-ca-file=/data/etcd/ssl/ca.pem \
  --peer-trusted-ca-file=/data/etcd/ssl/ca.pem \
  --initial-advertise-peer-urls=https://10.11.83.54:2380 \
  --listen-peer-urls=https://10.11.83.54:2380 \
  --listen-client-urls=https://10.11.83.54:2379 \
  --advertise-client-urls=https://10.11.83.54:2379 \
  --initial-cluster-token=etcd-cluster-0 \
  --initial-cluster=etcd-1=https://10.11.83.52:2380,etcd-2=https://10.11.83.53:2380,etcd-3=https://10.11.83.54:2380 \
  --initial-cluster-state=new \
  --data-dir=/data/etcd \
  --snapshot-count=50000 \
  --auto-compaction-retention=1 \
  --max-request-bytes=10485760 \
  --quota-backend-bytes=8589934592
Restart=always
RestartSec=15
LimitNOFILE=65536
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target

5、 启动 etcd

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

6、 验证是否成功

cd /data/etcd/ssl
# 查看状态
../bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://10.11.83.52:2379" cluster-health

member 1106cfdff1b83012 is healthy: got healthy result from https://10.11.83.52:2379
member 1cc2867ed4d78b29 is healthy: got healthy result from https://10.11.83.54:2379
member b9d2071008ea51d9 is healthy: got healthy result from https://10.11.83.53:2379
cluster is healthy

# 查看集群主机
../bin/etcdctl --ca-file=ca.pem --cert-file=server.pem --key-file=server-key.pem --endpoints="https://10.11.83.52:2379" member list

1106cfdff1b83012: name=etcd-1 peerURLs=https://10.11.83.52:2380 clientURLs=https://10.11.83.52:2379 isLeader=false
1cc2867ed4d78b29: name=etcd-3 peerURLs=https://10.11.83.54:2380 clientURLs=https://10.11.83.54:2379 isLeader=true
b9d2071008ea51d9: name=etcd-2 peerURLs=https://10.11.83.53:2380 clientURLs=https://10.11.83.53:2379 isLeader=false
restapi简单使用

v2版本

calico使用的是v2版本的api。

可以使用/etc/ssl/etcd/ssl下的k8s集群的etcd证书也可以使用/etc/calico/certs/下calico自己的etcd证书。

获取etcd member列表：

获取etcd member列表：

etcdctl --endpoints=https://10.142.233.80:2379 --ca-file=/etc/ssl/etcd/ssl/ca.pem --cert-file=/etc/ssl/etcd/ssl/node-10.142.233.80.pem --key-file=/etc/ssl/etcd/ssl/node-10.142.233.80-key.pem member list

查看 /calico节点

ETCDCTL_API=2 etcdctl --endpoints=https://10.142.233.80:2379 --ca-file=/etc/ssl/etcd/ssl/ca.pem --cert-file=/etc/ssl/etcd/ssl/node-10.142.233.80.pem --key-file=/etc/ssl/etcd/ssl/node-10.142.233.80-key.pem ls /calico

v3版本 k8s会使用etcd v3版本的API记录数据。而默认etcdctl是使用v2版本的API，查看不到v3的数据。设置环境变量ETCDCTL_API=3后就OK了

export ETCDCTL_API=3 #或者在etcdctl前面加上这个

获取etcd member列表：

etcdctl --endpoints=https://10.142.233.80:2379 --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-10.142.233.80.pem --key=/etc/ssl/etcd/ssl/node-10.142.233.80-key.pem member list

获取所有key：

etcdctl --endpoints=https://10.142.233.80:2379 --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-10.142.233.80.pem --key=/etc/ssl/etcd/ssl/node-10.142.233.80-key.pem get "" --prefix=true

获取前缀是/registry/pods/kube-system/prometheus的key：

etcdctl --endpoints=https://10.142.233.80:2379 --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-10.142.233.80.pem --key=/etc/ssl/etcd/ssl/node-10.142.233.80-key.pem get /registry/pods/kube-system/prometheus --prefix=true

获取具体key:

etcdctl --endpoints=https://10.142.233.80:2379 --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-10.142.233.80.pem --key=/etc/ssl/etcd/ssl/node-10.142.233.80-key.pem get /registry/pods/kube-system/prometheus-c78dbd66d-hkfcf （--prefix=true）

删除具体key:

etcdctl --endpoints=https://10.142.233.80:2379 --cacert=/etc/ssl/etcd/ssl/ca.pem --cert=/etc/ssl/etcd/ssl/node-10.142.233.80.pem --key=/etc/ssl/etcd/ssl/node-10.142.233.80-key.pem del /registry/pods/kube-system/prometheus-c78dbd66d-hkfcf

官方文档：https://etcd.io/docs/

推荐这些文章：

高可用k8s集群搭建

虚拟机选择

Win10 Hyper-V

总体架构
三个master，三个node
master的组件

etcd
kube-apiserver
kube-controller-manager
kube-scheduler
kubelet
kube-proxy
docker
nginx

node组件

kubelet
kube-pr...

CentOS7搭建k8s集群

一.机器信息

[root@kube-gmg-03-master-1 ~]# uname -a
Linux kube-gmg-03-master-1 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@...

lvs之搭建NAT模式的HTTPS负载集群

在lvs配置之NAT模式这篇文章配置的基础上搭建https
环境

系统
ip

redhat8 test
192.168.100.130

redhat8 DR
192.168.100.131 vip：192.168.18.250

redhat8 RS1
192.168.100.132

redhat8 RS2
192....

二进制 k8s高可用集群（containerd）

一、安装环境

主机名称

IP地址

软件

Master01

192.168.1.61

kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、nfs-client

Master...

静态Pod,Jenkins集成K8S环境及K8S 1.5.2高可用集群

bird资源清单,包含健康检查和可用性检查:cat > /oldboyedu-k8s/homework/all-in-one/01-homework-demo.yaml <<'EOF'apiVersion: extensions/v1beta1kind: Deploymentmetadata:  name: ...

[K8s]二进制高可用安装 k8s 集群

1 - 安装说明
本文章将演示 CentOS 7 二进制方式安装高可用 k8s 1.17+，相对于其他版本，二进制安装方式并无太大区别，只需要区分每个组件版本的对应关系即可。
生产环境中，建议使用小版本大于 5 的 Kubernetes 版本，比如 1.19.5 以后的才可用于生产环境。
2 - 二进制高可用 kubernetes 集群...

openstack前置安装与问题记录

安装openstack
下图显示了 OpenStack 服务之间的关系：
 
逻辑架构：
 
 
 
 
前置环境安装：       
官方文档网址：https://docs.openstack.org/zh_CN/
...

kubeadm使用外部etcd部署kubernetes v1.17.3 高可用集群

文章转载自：https://mp.weixin.qq.com/s?__biz=MzI1MDgwNzQ1MQ==&mid=2247483891&idx=1&sn=17dcd7cd0645df509c8e49059a2f00d7&chksm=e9fdd407de8a5d119d439b70dc2c381ec2...

二进制安装k8s-1.20.4之搭建etcd集群

1、部署etcd文件

cd /data/app/k8s-ssl/
cp ca.pem kubernetes-key.pem kubernetes.pem /data/app/etcd/ssl/

 
2、 创建etcd服务配置文件vim etcd.sh
 

ETCD_NAME=`hostna...

ubuntu16.04 nginx创建自签名SSL证书

创建SSL证书
$ sudo mkdir /etc/nginx/ca$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ca/nginx-selfsigned.key -out /etc/nginx/ca/nginx-selfsign...

文章标题：搭建高可用etcd集群
文章链接：https://www.dianjilingqu.com/51196.html
本文章来源于网络，版权归原作者所有，如果本站文章侵犯了您的权益，请联系我们删除，联系邮箱：saisai#email.cn，感谢支持理解。
THE END
< <上一篇
下一篇>>