Loongarch64 部署 Heketi

2023-10-18

字数统计: 1.1k字 | 阅读时长≈ 5分

Heketi 源码地址:heketi

部署环境

CPU 架构: loongarch64
系统版本: Kylin Linux Advanced Server V10 (Tercel)
Heketi 版本: v10.4.0
go 版本: go version go1.19 linux/loong64

安装依赖

编译 Heketi 需要依赖 mercurial/hg
1
yum install -y mercurial-hgk

编译 Heketi

从 GitHub 下载指定版本的 heketi 源码

1	git clone -b v10.4.0 https://github.com/heketi/heketi.git

修改 go.mod 文件

# 第一步，首先修改 go 的版本
go 1.19

# 第二步，在文件最下面增加一下内容
replace golang.org/x/sys => golang.org/x/sys v0.1.0

更新 mod 文件
1
go mod tidy

新建 vendor/github.com/boltdb/bolt/bolt_loong64.go 文件，文件内容如下

// +build loong64

package bolt

// maxMapSize represents the largest mmap size supported by Bolt.
const maxMapSize = 0xFFFFFFFFFFFF // 256TB

// maxAllocSize is the size used when creating array pointers.
const maxAllocSize = 0x7FFFFFFF

// Are unaligned load/stores broken on this arch?
var brokenUnaligned = false

在 heketi 源码根目录执行以下命令编译 heketi
1
make all
编译生成的二进制文件 heketi 在源码根目录，heketi-cli 在 client/cli/go/ 目录

测试生成的二进制文件

./client/cli/go/heketi-cli -v
heketi-cli v10.4.0

./heketi -v
Heketi v10.4.0 (using go: go1.19)

部署 Heketi

创建 heketi 安装目录

1	mkdir -p /usr/local/heketi/{bin,conf,data}

拷贝二进制文件到安装

1
2
3

cp ./heketi /usr/local/heketi/bin/
cp ./client/cli/go/heketi-cli /usr/local/heketi/bin/
ln -s /usr/local/heketi/bin/heketi-cli /usr/local/bin/

拷贝配置文件

1	cp ./etc/heketi.json /usr/local/heketi/conf/

创建免密登录

ssh-keygen -f /usr/local/heketi/conf/heketi_key -t rsa -N ''

ssh-copy-id -i /usr/local/heketi/conf/heketi_key.pub root@gfs-server01
ssh-copy-id -i /usr/local/heketi/conf/heketi_key.pub root@gfs-server02
ssh-copy-id -i /usr/local/heketi/conf/heketi_key.pub root@gfs-server03

修改配置文件，主要修改以下几个地方

"use_auth": true                  # 默认是 false, 这里改为 true
"jwt"."admin"."key": "adminkey"    # 添加一个 adminkey
"jwt"."user"."key": "userkey"      # 添加一个 userkey
"glusterfs"."executor": "ssh"      # 修改 exector 模式为 ssh

## 修改 ssh exector 相关配置
"sshexec": {
      "keyfile": "/usr/local/heketi/conf/heketi_key",
      "user": "root",
      "port": "22",
      "sudo": true,    # 由于 root 用户被禁止远程登录，所以需要使用 sudo
      "fstab": "/etc/fstab",

## 修改 db 配置
"db": "/usr/local/heketi/data/heketi.db",

## 删除以下配置，ssh 模块以及 kubernetes 模块都要删除
"xfs_sw": "Optional: Specify number of data disks in the underlying RAID device.",
"xfs_su": "Optional: Specifies a stripe unit or RAID chunk size.",
"gluster_cli_timeout": "Optional: Timeout, in seconds, passed to the gluster cli invocations",

创建 systemd 管理 heketi 服务

cat > /usr/lib/systemd/system/heketi.service <<EOF
[Unit]
Description=RESTful based volume management framework for GlusterFS
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/heketi/heketi

[Service]
Type=simple
LimitNOFILE=65536
ExecStart=/usr/local/heketi/bin/heketi --config=/usr/local/heketi/conf/heketi.json
KillMode=process
Restart=on-failure
RestartSec=5
SuccessExitStatus=15
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target
EOF

创建 heketi 服务启动用户

1 2	useradd -s /sbin/nologin -M heketi chown -R heketi:heketi /usr/local/heketi

启动 heketi
1
systemctl enable --now heketi.service

管理 GlusterFS

创建 toplogy.json 配置文件

cat > /usr/local/heketi/conf/topology.json <<EOF 
{
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": ["gfs-server01"],
              "storage": ["192.168.0.9"]
            },
            "zone": 1
          },
          "devices": ["/dev/vdb"]
        },
        {
          "node": {
            "hostnames": {
              "manage": ["gfs-server02"],
              "storage": ["192.168.0.30"]
            },
            "zone": 1
          },
          "devices": ["/dev/vdb"]
        },
        {
          "node": {
            "hostnames": {
              "manage": ["gfs-server03"],
              "storage": ["192.168.0.78"]
            },
            "zone": 1
          },
          "devices": ["/dev/vdb"]
        }
      ]
    }
  ]
}
EOF

创建资源

heketi-cli topology load \
  --server http://127.0.0.1:8080 \
  --user admin \
  --secret adminkey \
  --json /usr/local/heketi/conf/topology.json 

# 输出如下信息，说明配置完成
Creating cluster ... ID: 500068aa09d006f30dee8142891997b7
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node gfs-server01 ... ID: 5b5eaf443aed9c500a122d26a62ff53f
            Adding device /dev/vdb ... OK
    Creating node gfs-server02 ... ID: 9e09551641e209a87f5c39bb572afad0
            Adding device /dev/vdb ... OK
    Creating node gfs-server03 ... ID: 8c334bbd6e1eb640e30b28b55acb4ed2
            Adding device /dev/vdb ... OK

创建 StorageClass

cat << EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: glusterfs
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/glusterfs
allowVolumeExpansion: true
parameters:
  resturl: "http://192.168.0.134:8080"
  clusterid: "500068aa09d006f30dee8142891997b7"
  restauthenabled: "true"
  restuser: "admin"
  restuserkey: "adminkey"
  gidMin: "40000"
  gidMax: "50000"
  volumetype: "replicate:2"
EOF

创建 volume 卷

1	heketi-cli volume create --name vbaas --size 100 --replica 2 --user admin --secret adminkey

挂载 volume 卷

mkdir -p /mnt/nas/vbaas 

cat >> /etc/fstab <<EOF
192.168.0.9:vbaas     /mnt/nas/vbaas          glusterfs       defaults,_netdev        0 0
EOF

mount -a