Loongarch64 架构部署 Nacos

2023-10-18

字数统计: 3k字 | 阅读时长≈ 16分

部署环境

CPU 架构: loongarch64
系统版本: Kylin Linux Advanced Server V10 (Tercel)
Nacos 版本: v2.0.4
JDK 版本: 1.8.0_332
Maven 版本: 3.8.6

注意: 要在龙芯平台（loongarch64）运行 nacos，需要替换官方 nacos/target/nacos-server.jar 包中的 rocksdbjni-5.18.4.jar 文件
rocksdb 源码地址: rocksdb
编译 rocksdb 参考文档: 龙芯3a5000部署nacos 1.4低版本方法

环境准备

配置 YUM 仓库源

执行以下命令添加 yum 仓库源

cat > /etc/yum.repos.d/Loongnix-BaseOS.repo <<EOF
# Loongnix-BaseOS.repo
#
# The mirrorlist system uses the connecting IP address of the client and the
# update status of each mirror to pick current mirrors that are geographically
# close to the client.  You should use this for Loongnix Server updates unless
# you are manually picking other mirrors.
#
# If the mirrorlist does not work for you, you can try the commented out
# baseurl line instead.
[baseos]
name=Loongnix server 8.4 - BaseOS
baseurl=http://pkg.loongnix.cn/loongnix-server/8.4/BaseOS/$basearch/release/
gpgcheck=0
enabled=1 
priority=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-LOONGNIX

[extras]
name=Loongnix server 8.4 - Extras
baseurl=http://pkg.loongnix.cn/loongnix-server/8.4/extras/$basearch/release/
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-LOONGNIX

[appstream]
name=Loongnix server 8.4 - AppStream
baseurl=http://pkg.loongnix.cn/loongnix-server/8.4/AppStream/$basearch/release/
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-LOONGNIX

[epel]
name=Loongnix server 8.4 - Epel
baseurl=http://pkg.loongnix.cn/loongnix-server/8.4/epel/$basearch/release/Modular/
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-LOONGNIX

[infra-buildtools]
name=Loongnix server 8.4 - Infra-buildtools
baseurl=http://pkg.loongnix.cn/loongnix-server/8.4/infra/$basearch/buildtools-common/release/
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-LOONGNIX

[loongnixplus]
name=Loongnix server 8.4 - LoongnixPlus
baseurl=http://pkg.loongnix.cn/loongnix-server/8.4/Loongnixplus/$basearch/release/
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-LOONGNIX

[storage]
name=Loongnix server 8.4 - Storage
baseurl=http://pkg.loongnix.cn/loongnix-server/8.4/storage/$basearch/release/gluster-10/
gpgcheck=0
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-LOONGNIX
EOF

yum clean all && yum makecache

注意: 如果安装某些软件如 gcc-c++ 出现冲突时，可以临时禁用这些 yum 源；

部署 JDK

下载 jdk

1	wget http://ftp.loongnix.cn/Java/openjdk8/loongson8.1.11-jdk8u332b09-linux-loongarch64.tar.gz

安装 jdk

# 创建 jdk 安装目录
mkdir -p /usr/local/java

# 解压 jdk 到安装目录
tar xf loongson8.1.11-jdk8u332b09-linux-loongarch64.tar.gz -C /usr/local/java/

# 配置环境变量
cat > /etc/profile.d/java.sh <<EOF
#!/bin/bash
  
export JAVA_HOME=/usr/local/java/jdk1.8.0_332
export CLASSPATH=\$CLASSPATH:\$JAVA_HOME/lib:\$JAVA_HOME/jre/lib
export PATH=\$JAVA_HOME/bin:\$JAVA_HOME/jre/bin:\$PATH:$HOME/bin
EOF

检查 jdk 版本

source /etc/profile.d/java.sh 
# java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Loongson 8.1.11-loongarch64-LoongnixServer) (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode)

部署 Maven

下载 maven 安装包

1	wget https://dlcdn.apache.org/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.tar.gz

安装 maven

# 创建 maven 安装目录
mkdir /usr/local/maven

# 解压 maven 到安装目录
tar xf apache-maven-3.8.6-bin.tar.gz -C /usr/local/maven/

# 配置环境变量
cat > /etc/profile.d/maven.sh <<EOF 
#!/bin/bash
  
export MAVEN_HOME=/usr/local/maven/apache-maven-3.8.6
export PATH=\$MAVEN_HOME/bin:\$PATH:\$HOME/bin
EOF

修改 conf/settings.xml 配置文件

# 修改配置文件，在 mirrors 配置里面添加阿里云 maven 仓库
    <mirror>
      <id>alimaven</id>
      <name>aliyun maven</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
      <mirrorOf>central</mirrorOf>
    </mirror>

# 修改资源保存位置。在 localRepository 下面加入配置
<settings xmlns="http://maven.apache.org/SETTINGS/1.2.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.2.0 https://maven.apache.org/xsd/settings-1.2.0.xsd">
  <!-- localRepository
  | The path to the local repository maven will use to store artifacts.
  |
  | Default: ${user.home}/.m2/repository
  <localRepository>/path/to/local/repo</localRepository>
  -->

  <localRepository>/data/maven/repo</localRepository>
  ... 省略 N 行

# 创建 repo 目录
mkdir /data/maven/repo -p

检查 mvn 版本

# mvn --version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /usr/local/maven/apache-maven-3.8.6
Java version: 1.8.0_332, vendor: Oracle Corporation, runtime: /usr/local/java/jdk1.8.0_332/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "4.19.90-23.19.v2101.a.ky10.loongarch64", arch: "loongarch64", family: "unix"

编译 rocksdbjni 文件

从 github 下载 rocksdb 源码
1
git clone -b v5.18.4 https://github.com/facebook/rocksdb.git
具体要下载哪个分支可以使用以下命令查看 nacos 使用的是哪个版本的 rocksdbjni 包（顺便能知道编译后的 jar 包要更新到哪个目录）
1
2
# jar -tf nacos/target/nacos-server.jar |grep rocksdb
BOOT-INF/lib/rocksdbjni-5.18.4.jar
安装编译 rocksdbjni 需要的依赖库
1
2
3
4
5
yum install -y \
libzstd-devel \
lz4-devel \
gflags-devel \
snappy-devel
如果提示无法找到相应的包，可以更新 yum 仓库源

源码编译 rocksdbsjni 时需要联网下载一些依赖包，这里可以提前下载好放到源码对应的目录下，需要下载的文件如下:

以下文件下载好放到 rocksdb 源码根目录
- bzip2-1.0.6.tar.gz
- lz4-1.9.2.tar.gz
- snappy-1.1.8.tar.gz
- zlib-1.2.11.tar.gz
- zstd-1.4.4.tar.gz
以下文件下载好后放到 rocksdb/java/test-libs 目录
- assertj-core-1.7.1.jar
- cglib-2.2.2.jar
- hamcrest-core-1.3.jar
- junit-4.12.jar
- mockito-all-1.10.19.jar

# 下载 bzip2-1.0.6.tar.gz
wget https://udomain.dl.sourceforge.net/project/bzip2/bzip2-1.0.6.tar.gz -O bzip2-1.0.6.tar.gz

# 下载 lz4-1.9.2.tar.gz
wget https://github.com/lz4/lz4/archive/refs/tags/v1.9.2.tar.gz -O lz4-1.9.2.tar.gz

# 下载 snappy-1.1.8.tar.gz
wget https://github.com/google/snappy/archive/refs/tags/1.1.8.tar.gz -O snappy-1.1.8.tar.gz

# 下载 zlib-1.2.11.tar.gz
wget https://zlib.net/fossils/zlib-1.2.11.tar.gz -O zlib-1.2.11.tar.gz

# 下载 zstd-1.4.4.tar.gz
wget https://github.com/facebook/zstd/releases/download/v1.4.4/zstd-1.4.4.tar.gz -O zstd-1.4.4.tar.gz

# 下载 assertj-core-1.7.1.jar
wget https://github.com/pdillinger/rocksdb/releases/download/v6.6.x-java-deps/assertj-core-1.7.1.jar

# 下载 cglib-2.2.2.jar
wget https://github.com/pdillinger/rocksdb/releases/download/v6.6.x-java-deps/cglib-2.2.2.jar

# 下载 hamcrest-core-1.3.jar
wget https://github.com/pdillinger/rocksdb/releases/download/v6.6.x-java-deps/hamcrest-core-1.3.jar

# 下载 junit-4.12.jar
wget https://github.com/pdillinger/rocksdb/releases/download/v6.6.x-java-deps/junit-4.12.jar

# 下载 mockito-all-1.10.19.jar
wget https://github.com/pdillinger/rocksdb/releases/download/v6.6.x-java-deps/mockito-all-1.10.19.jar

修改 rocksdb 源码根目录的 Makefile 文件

注释 Makefile 文件的哈希值验证，从 1713 行左右开始到 1784行左右，如下所示，注释验证相关代码

libz.a:
        -rm -rf zlib-$(ZLIB_VER)
ifeq (,$(wildcard ./zlib-$(ZLIB_VER).tar.gz))
        curl --fail --output zlib-$(ZLIB_VER).tar.gz --location ${ZLIB_DOWNLOAD_BASE}/zlib-$(ZLIB_VER).tar.gz
endif
        #ZLIB_SHA256_ACTUAL=`$(SHA256_CMD) zlib-$(ZLIB_VER).tar.gz | cut -d ' ' -f 1`; \
        #if [ "$(ZLIB_SHA256)" != "$$ZLIB_SHA256_ACTUAL" ]; then \
        #       echo zlib-$(ZLIB_VER).tar.gz checksum mismatch, expected=\"$(ZLIB_SHA256)\" actual=\"$$ZLIB_SHA256_ACTUAL\"; \
        #       exit 1; \
        #fi
        tar xvzf zlib-$(ZLIB_VER).tar.gz
        cd zlib-$(ZLIB_VER) && CFLAGS='-fPIC ${EXTRA_CFLAGS}' LDFLAGS='${EXTRA_LDFLAGS}' ./configure --static && $(MAKE)
        cp zlib-$(ZLIB_VER)/libz.a .

libbz2.a:
        -rm -rf bzip2-$(BZIP2_VER)
ifeq (,$(wildcard ./bzip2-$(BZIP2_VER).tar.gz))
        curl --fail --output bzip2-$(BZIP2_VER).tar.gz --location ${CURL_SSL_OPTS} ${BZIP2_DOWNLOAD_BASE}/bzip2-$(BZIP2_VER).tar.gz
endif
        #BZIP2_SHA256_ACTUAL=`$(SHA256_CMD) bzip2-$(BZIP2_VER).tar.gz | cut -d ' ' -f 1`; \
        #if [ "$(BZIP2_SHA256)" != "$$BZIP2_SHA256_ACTUAL" ]; then \
        #       echo bzip2-$(BZIP2_VER).tar.gz checksum mismatch, expected=\"$(BZIP2_SHA256)\" actual=\"$$BZIP2_SHA256_ACTUAL\"; \
        #       exit 1; \
        #fi
        tar xvzf bzip2-$(BZIP2_VER).tar.gz
        cd bzip2-$(BZIP2_VER) && $(MAKE) CFLAGS='-fPIC -O2 -g -D_FILE_OFFSET_BITS=64 ${EXTRA_CFLAGS}' AR='ar ${EXTRA_ARFLAGS}'
        cp bzip2-$(BZIP2_VER)/libbz2.a .
... 省略 N 行 ...

注释 Makefile 文件中 clean 选项的 rm -rf bzip2* snappy* zlib* lz4* zstd* 命令，如下所示

clean:
    rm -f $(BENCHMARKS) $(TOOLS) $(TESTS) $(LIBRARY) $(SHARED)
    rm -rf $(CLEAN_FILES) ios-x86 ios-arm scan_build_report
    $(FIND) . -name "*.[oda]" -exec rm -f {} \;
    $(FIND) . -type f -regex ".*\.\(\(gcda\)\|\(gcno\)\)" -exec rm {} \;
    # rm -rf bzip2* snappy* zlib* lz4* zstd*   # 就是取消这一行，否则每次执行 clean 选项时都会删除下载好的依赖文件
    cd java; $(MAKE) clean

注释 java/Makefile 文件中 clean 选项中删除依赖的命令

clean:
        $(AM_V_at)rm -rf include/*
        # $(AM_V_at)rm -rf test-libs/
        $(AM_V_at)rm -rf $(OUTPUT)
        $(AM_V_at)rm -rf $(BENCHMARK_OUTPUT)
        $(AM_V_at)rm -rf $(SAMPLES_OUTPUT)

(可选)修改 rocksdb 源码根目录的 CMakelist.txt 文件，如下

if(POLICY CMP0042)
  cmake_policy(SET CMP0042 NEW)
endif()

# 配置 cmake 的编译模式
set(CMAKE_BUILD_TYPE "Release") 
# 在这一行增加这一行代码，启用 Makefile 构建时更详细的信息输出。
set(CMAKE_VERBOSE_MAKEFILE ON)

# 修改以下配置
option(WITH_JEMALLOC "build with JeMalloc" OFF)
option(WITH_SNAPPY "build with SNAPPY" ON)   # 默认值为 OFF
option(WITH_LZ4 "build with lz4" ON)         # 默认值为 OFF
option(WITH_ZLIB "build with zlib" ON)       # 默认值为 OFF
option(WITH_ZSTD "build with zstd" ON)       # 默认值为 OFF

# 修改以下值，重要，不修改时会编译失败
option(PORTABLE "build a portable binary" OFF)
option(FORCE_SSE42 "force building with SSE4.2, even when PORTABLE=ON" OFF)
if(PORTABLE)
  # MSVC does not need a separate compiler flag to enable SSE4.2; if nmmintrin.h
  # is available, it is available by default.
  if(FORCE_SSE42 AND NOT MSVC)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse4.2 -mpclmul")
  endif()
else()
  if(MSVC)
    set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /arch:AVX2")
  else()
    if(NOT HAVE_POWER8)
      set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=loongarch64 -Os -Wall -s")   # 主要修改这一行，-march 默认值为 native
    endif()
  endif()
endif()

执行以下命令编译 Java 包（如果只是编译 rocksdbjavastatic，第5步可以不做）
1
make jclean clean rocksdbjavastatic
生成的文件在 java/target/ 目录下，将 rocksdbjni-5.18.4-linux64.jar 重命名为 rocksdbjni-5.18.4.jar 即可使用

部署 Nacos

下载 Nacos 安装包

1	wget https://github.com/alibaba/nacos/releases/download/2.0.4/nacos-server-2.0.4.tar.gz

解压 nacos 安装包到 /usr/local/ 目录

1	tar xf nacos-server-2.0.4.tar.gz -C /usr/local/

修改 conf/application.properties 配置文件

spring.datasource.platform=mysql
db.num=1
db.url.0=jdbc:mysql://192.168.200.95:3306/nacos_config?characterEncoding=utf8&connectTimeout=1000&socketTimeout=3000&autoReconnect=true&useUnicode=true&useSSL=false&serverTimezone=UTC
db.user.0=nacos
db.password.0=123456
nacos.core.auth.enabled=true     # 开启鉴权
### The directory of access log:
server.tomcat.basedir=/usr/local/nacos/temp   # 如果不配置这个，上传文件会提示数据格式不合法

替换 target/nacos-server.jar 文件中的 rocksdbjni-5.18.4.jar 文件

# 创建需要更新文件的目录结构
cd /usr/local/nacos/target && mkdir -p BOOT-INF/lib/

# 拷贝 文件到对应的目录中
cp /usr/local/src/rocksdbjni-5.18.4.jar BOOT-INF/lib/

# 使用 jar 命令更新 jar 包中的文件
jar -uvf0 nacos-server.jar BOOT-INF/lib/rocksdbjni-5.18.4.jar

测试启动服务

# 单机模式
sh startup.sh -m standalone

# 集群模式
sh startup.sh

配置 systemd 管理 nacos

# 添加 nacos 服务运行用户
useradd -s /sbin/nologin -M nacos

# 修改 bin/start.sh 文件。添加 java 环境变量
sed -i '/cygwin=false/i\source /etc/profile.d/java.sh\n' /usr/local/nacos/bin/startup.sh

# 修改 nacos 目录权限
chown -R nacos:nacos /usr/local/nacos

# 创建 service 文件
cat > /usr/lib/systemd/system/nacos.service <<EOF
[Unit]
Description=nacos
After=network.target

[Service]
Type=forking
Environment="JAVA_HOME=/usr/local/java/jdk1.8.0_332"
Environment="CLASSPATH=\$CLASSPATH:\$JAVA_HOME/lib:\$JAVA_HOME/jre/lib"
ExecStart=/usr/local/nacos/bin/startup.sh -m standalone
ExecReload=/bin/kill -HUP \$MAINPID
ExecStop=/usr/local/nacos/bin/shutdown.sh
PrivateTmp=true
User=nacos
Group=nacos

[Install]
WantedBy=multi-user.target
EOF

配置 nacos 开机启动
1
systemctl enable --now nacos

配置域名访问服务，这里使用 kubernetes 的 ingress

# 创建 publics 命名空间
kubectl create ns publics

# 创建 nacos 的 svc 以及 ingress
cat << EOF |kubectl create -f -
apiVersion: v1
kind: Service
metadata:
  labels:
    app: nacos
  name: nacos
  namespace: publics
spec:
  ports:
  - port: 80
    name: http
    protocol: TCP
    targetPort: 8848
  type: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: nacos
  name: nacos
  namespace: publics
subsets:
- addresses:
  - ip: 192.168.200.76
  ports:
  - name: http
    port: 8848
    protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nacos
  namespace: publics
spec:
  ingressClassName: "traefik"
  rules:
    - host: nacos.vonebaas.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name:  nacos
                port:
                  number: 80
EOF

疑难杂症

通过 Nacos web 界面导入配置文件时，提示 “导入失败，未读取到合法数据，请检查导入的数据文件”，如下图所示:

错误原因: 通过观察日志发现 nacos 报错如下:

org.springframework.web.multipart.MultipartException: Failed to parse multipart servlet request; nested exception is java.io.IOException: The temporary upload location [/work/Tomcat/localhost/nacos] is not valid
        at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.handleParseFailure(StandardMultipartHttpServletRequest.java:124)
        at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.parseRequest(StandardMultipartHttpServletRequest.java:115)
        at org.springframework.web.multipart.support.StandardMultipartHttpServletRequest.<init>(StandardMultipartHttpServletRequest.java:88)
        at org.springframework.web.multipart.support.StandardServletMultipartResolver.resolveMultipart
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        ... 省略 N 行 ...
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: The temporary upload location [/work/Tomcat/localhost/nacos] is not valid
        at org.apache.catalina.connector.Request.parseParts(Request.java:2844)
        at org.apache.catalina.connector.Request.parseParameters(Request.java:3202)
        at org.apache.catalina.connector.Request.getParameter(Request.java:1124)
        at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:381)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:85)
        ... 29 common frames omitted

解决方法: 修改 conf/application.properties 陪文件，配置 server.tomcat.basedir=，然后重启 nacos。如下所示:

1 2	### The directory of access log: server.tomcat.basedir= /usr/local/nacos/tomcat