Fork me on GitHub

使用 Operator 创建 Redis 集群

字数统计: 1.6k字   |   阅读时长≈ 8分

参考来源:

Kubernetes 如何管理中间件集群

Kubernetes 可以使用包管理工具管理中间件集群，常用的两种包管理工具是 Operator 和 Helm。可以一键式创建集群，扩容，备份等。

  • Helm: 更倾向于无状态应用的部署，比如公司的服务，某些不需要持久化数据的中间件，不需要实现额外功能的服务，比如备份，回滚等。
  • Operator: 管理更为复杂的有状态服务，比如 MySQL 集群，Redis 集群，Rook 等。并且可以使用 Operator 实现扩容，备份，回滚等。

部署方式

使用 Helm 与 Operator 部署中间件集群时，一般按照以下步骤进行:

Helm

  1. 安装 Helm 客户端工具
  2. 添加 Helm 仓库
  3. 使用 helm install 一键启动

Operator

  1. 创建 Operator 控制器
  2. 创建自定义资源
  3. 执行相关逻辑

使用 Operator 创建 Redis 集群

常用的 Operator 模板地址:

部署 Redis 集群 Operator

  1. 下载 redis-operator 仓库到服务器

    1
    # git clone https://github.com/ucloud/redis-cluster-operator.git

  2. 安装自定义资源定义（CRD）

    1
    2
    3
    4
    5
    6
    7
    # kubectl create -f deploy/crds/redis.kun_distributedredisclusters_crd.yaml 
    Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
    customresourcedefinition.apiextensions.k8s.io/distributedredisclusters.redis.kun created

    # kubectl create -f deploy/crds/redis.kun_redisclusterbackups_crd.yaml 
    Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
    customresourcedefinition.apiextensions.k8s.io/redisclusterbackups.redis.kun created

  3. 安装 Operator 控制器，有两种级别的方式安装 Redis 集群，一种是集群级别的，一种是 namespace 级别的，这里选择 namespace 级别的方式

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    # kubectl create ns redis-cluster
    namespace/redis-cluster created

    # kubectl create -f deploy/service_account.yaml -n redis-cluster 
    serviceaccount/redis-cluster-operator created

    # kubectl create -f deploy/namespace/role.yaml -n redis-cluster 
    role.rbac.authorization.k8s.io/redis-cluster-operator created

    # kubectl create -f deploy/namespace/role_binding.yaml -n redis-cluster
    rolebinding.rbac.authorization.k8s.io/redis-cluster-operator created

    # kubectl create -f deploy/namespace/operator.yaml -n redis-cluster 
    deployment.apps/redis-cluster-operator created
    configmap/redis-admin created

  4. 创建 Redis 集群，Namespace 级别的需要更改配置，将 redis.kun/scope 注释掉，如下所示

    • 文件名: deploy/example/redis.kun_v1alpha1_distributedrediscluster_cr.yaml
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    apiVersion: redis.kun/v1alpha1
    kind: DistributedRedisCluster
    metadata:
      annotations:
        # if your operator run as cluster-scoped, add this annotations
        # redis.kun/scope: cluster-scoped       # 使用 Namespace 级别时需要注释此行
      name: example-distributedrediscluster
    spec:
      # Add fields here
      masterSize: 3                   # 几个 master 节点
      clusterReplicas: 1              # 每个 master 节点几个副本
      image: redis:5.0.4-alpine       # 使用哪个版本的 Redis 部署集群，可以按需修改版本，但是需要测试

    然后执行以下命令

    1
    2
    # kubectl create -f deploy/example/redis.kun_v1alpha1_distributedrediscluster_cr.yaml -n redis-cluster 
    distributedrediscluster.redis.kun/example-distributedrediscluster created

  5. 查看集群状态

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    # kubectl get distributedredisclusters.redis.kun -n redis-cluster 
    NAME                              MASTERSIZE   STATUS    AGE
    example-distributedrediscluster   3            Healthy   2m35s

    # kubectl get svc -n redis-cluster 
    NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
    example-distributedrediscluster     ClusterIP   10.102.60.187   <none>        6379/TCP,16379/TCP   2m56s
    example-distributedrediscluster-0   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m56s
    example-distributedrediscluster-1   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m56s
    example-distributedrediscluster-2   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m56s
    redis-cluster-operator-metrics      ClusterIP   10.111.97.39    <none>        8383/TCP,8686/TCP    11m

    # kubectl get pods -n redis-cluster 
    NAME                                      READY   STATUS    RESTARTS   AGE
    drc-example-distributedrediscluster-0-0   1/1     Running   0          3m16s
    drc-example-distributedrediscluster-0-1   1/1     Running   0          2m16s
    drc-example-distributedrediscluster-1-0   1/1     Running   0          3m16s
    drc-example-distributedrediscluster-1-1   1/1     Running   0          2m26s
    drc-example-distributedrediscluster-2-0   1/1     Running   0          3m16s
    drc-example-distributedrediscluster-2-1   1/1     Running   0          2m16s
    redis-cluster-operator-675ccbc697-bmcnp   1/1     Running   0          11m

测试使用 redis 集群

  1. 程序链接时使用的地址查看

    1
    2
    3
    4
    5
    6
    7
    8
    # kubectl get pods -n redis-cluster 
    # kubectl get svc -n redis-cluster 
    NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
    example-distributedrediscluster     ClusterIP   10.102.60.187   <none>        6379/TCP,16379/TCP   2m56s   # 程序链接时使用这个 svc
    example-distributedrediscluster-0   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m56s
    example-distributedrediscluster-1   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m56s
    example-distributedrediscluster-2   ClusterIP   None            <none>        6379/TCP,16379/TCP   2m56s
    redis-cluster-operator-metrics      ClusterIP   10.111.97.39    <none>        8383/TCP,8686/TCP    11m

  2. 进入 redis 查看

    • 查看 redis pod
    1
    2
    3
    4
    5
    6
    7
    8
    9
    # kubectl get pods -n redis-cluster -owide
    NAME                                      READY   STATUS    RESTARTS   AGE     IP               NODE            NOMINATED NODE   READINESS GATES
    drc-example-distributedrediscluster-0-0   1/1     Running   0          10m     172.20.183.145   k8s-master-03   <none>           <none>
    drc-example-distributedrediscluster-0-1   1/1     Running   0          9m7s    172.30.107.74    k8s-master-01   <none>           <none>
    drc-example-distributedrediscluster-1-0   1/1     Running   0          10m     172.21.176.206   k8s-master-02   <none>           <none>
    drc-example-distributedrediscluster-1-1   1/1     Running   0          9m17s   172.19.98.142    k8s-node-02     <none>           <none>
    drc-example-distributedrediscluster-2-0   1/1     Running   0          10m     172.31.44.12     k8s-node-01     <none>           <none>
    drc-example-distributedrediscluster-2-1   1/1     Running   0          9m7s    172.20.183.146   k8s-master-03   <none>           <none>
    redis-cluster-operator-675ccbc697-bmcnp   1/1     Running   0          18m     172.21.176.205   k8s-master-02   <none>           <none>
    • 随机选择一个 redis pod 进入
    1
    2
    3
    4
    5
    6
    7
    8
    # kubectl exec -it drc-example-distributedrediscluster-0-0 -n redis-cluster -- sh
    /data # redis-cli -h 172.20.183.146
    172.20.183.146:6379> set a 1
    (error) MOVED 15495 172.30.107.74:6379
    172.20.183.146:6379> 
    /data # redis-cli -h 172.30.107.74
    172.30.107.74:6379> set a 1
    OK

扩容 Redis 集群

  1. 使用 kubectl edit 方式直接修改 distributedredisclusters 资源

    1
    # kubectl edit distributedredisclusters.redis.kun -n redis-cluster example-distributedrediscluster -oyaml

  2. 修改 masterSize 的值

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    ...
    spec:
      clusterReplicas: 2
      image: redis:5.0.4-alpine
      masterSize: 4
      resources:
        limits:
          cpu: "1"
          memory: 4Gi
        requests:
          cpu: 200m
          memory: 256Mi
      serviceName: example-distributedrediscluster
    ...

  3. 查看集群的状态

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    # kubectl get distributedredisclusters.redis.kun -n redis-cluster 
    NAME                              MASTERSIZE   STATUS    AGE
    example-distributedrediscluster   4            Healthy   16m

    # kubectl get svc -n redis-cluster 
    NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)              AGE
    example-distributedrediscluster     ClusterIP   10.102.60.187   <none>        6379/TCP,16379/TCP   17m
    example-distributedrediscluster-0   ClusterIP   None            <none>        6379/TCP,16379/TCP   17m
    example-distributedrediscluster-1   ClusterIP   None            <none>        6379/TCP,16379/TCP   17m
    example-distributedrediscluster-2   ClusterIP   None            <none>        6379/TCP,16379/TCP   17m
    example-distributedrediscluster-3   ClusterIP   None            <none>        6379/TCP,16379/TCP   3m38s
    redis-cluster-operator-metrics      ClusterIP   10.111.97.39    <none>        8383/TCP,8686/TCP    25m

    # kubectl get pods -n redis-cluster 
    NAME                                      READY   STATUS    RESTARTS   AGE
    drc-example-distributedrediscluster-0-0   1/1     Running   0          48s
    drc-example-distributedrediscluster-0-1   1/1     Running   0          2m
    drc-example-distributedrediscluster-0-2   1/1     Running   0          3m12s
    drc-example-distributedrediscluster-1-0   1/1     Running   0          63s
    drc-example-distributedrediscluster-1-1   1/1     Running   0          2m18s
    drc-example-distributedrediscluster-1-2   1/1     Running   0          3m12s
    drc-example-distributedrediscluster-2-0   1/1     Running   0          67s
    drc-example-distributedrediscluster-2-1   1/1     Running   0          2m24s
    drc-example-distributedrediscluster-2-2   1/1     Running   0          3m12s
    drc-example-distributedrediscluster-3-0   1/1     Running   0          3m12s
    drc-example-distributedrediscluster-3-1   1/1     Running   0          2m31s
    drc-example-distributedrediscluster-3-2   1/1     Running   0          111s
    redis-cluster-operator-675ccbc697-bmcnp   1/1     Running   0          25m

    # kubectl exec -it drc-example-distributedrediscluster-0-0 -n redis-cluster -- sh
    /data # redis-cli -h 172.20.183.149
    172.20.183.149:6379> CLUSTER INFO
    cluster_state:ok
    cluster_slots_assigned:16384
    cluster_slots_ok:16384
    cluster_slots_pfail:0
    cluster_slots_fail:0
    cluster_known_nodes:12
    cluster_size:4
    cluster_current_epoch:10
    cluster_my_epoch:2
    cluster_stats_messages_ping_sent:113
    cluster_stats_messages_pong_sent:122
    cluster_stats_messages_meet_sent:7
    cluster_stats_messages_sent:242
    cluster_stats_messages_ping_received:118
    cluster_stats_messages_pong_received:120
    cluster_stats_messages_meet_received:4
    cluster_stats_messages_received:242
    172.20.183.149:6379>

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia-plus根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

主要涉及技术：
Linux 运维、自动化运维、
Kubernetes、Python、Jenkins、Docker

联系QQ:2350686113

很惭愧

只做了一点微小的工作
谢谢大家