# 使用 Pacemaker 与 Corosync 构建 FreeBSD 集群
* 原文：[FreeBSD Cluster with Pacemaker and Corosync](https://vermaden.wordpress.com/2020/09/03/freebsd-cluster-with-pacemaker-and-corosync/)
* 作者：𝚟𝚎𝚛𝚖𝚊𝚍𝚎𝚗
* 2020/09/03
我一直很想找到适合 FreeBSD 系统的“正统”集群软件。最近我有机会在 Linux 系统上运行几个基于 Pacemaker/Corosync 的集群。我开始思考如何在 FreeBSD 上实现类似的高可用性解决方案，当我发现 Pacemaker 和 Corosync 工具在 FreeBSD Ports 和包中分别可以通过 **net/pacemaker2** 和 **net/corosync2** 获得时，我真的非常震惊。
在本文中，我将检查 Pacemaker 和 Corosync 集群在 FreeBSD 上的运行情况。
集群，有很多定义。我最喜欢的定义是：集群是即使失去其中一个节点仍保持冗余的系统（仍然是集群）。这意味着根据这个定义，集群的最少节点数是 3 个。两节点集群存在较大问题，因为它们最容易出现脑裂问题。这就是为什么在两节点集群中，通常会添加额外的设备或系统，以确保不会发生脑裂。例如，可以添加第三个节点，不提供任何资源或服务，仅作为“见证者”角色。另一种方式是添加共享磁盘资源，其作用相同，通常是使用 SCSI-3 持久保留机制的原始卷。
## 搭建实验室
一如既往，整个实验将基于 VirtualBox，并由 3 台主机组成。为了不创建 3 个相同的 FreeBSD 安装，我直接使用了由 FreeBSD 项目提供的 12.1-RELEASE 虚拟机镜像：
*
有多种格式可选 – **qcow2**/**raw**/**vhd**/**vmdk** – 因为我将使用 VirtualBox，所以选择了格式 [VMDK](https://download.freebsd.org/ftp/releases/VM-IMAGES/12.1-RELEASE/amd64/Latest/FreeBSD-12.1-RELEASE-amd64.vmdk.xz) 。
下面是 GlusterFS 集群的主机列表：
* 10.0.10.111 node1
* 10.0.10.112 node2
* 10.0.10.113 node3
每个 VirtualBox 的 FreeBSD 虚拟机均为默认配置（如 VirtualBox 向导所建议），内存为 512 MB，网络模式为 *NAT Network（NAT 网络）*，如下图所示。
下面是 VirtualBox 上 *NAT Network（NAT 网络）* 的配置。
在尝试连接 FreeBSD 主机之前，需要在每台虚拟机内进行最简网络配置。每台 FreeBSD 主机将具有如下示例的最简 **/etc/rc.conf** 文件（以 **node1** 为例）。
```ini
root@node1:~ # cat /etc/rc.conf
hostname=node1
ifconfig_em0="inet 10.0.10.111/24 up"
defaultrouter=10.0.10.1
sshd_enable=YES
```
为了搭建实验环境，我们需要允许 **root** 登录这些 FreeBSD 主机，在 **/etc/ssh/sshd\_config** 文件中设置 **PermitRootLogin yes**。更改以后，还需要重启 **sshd(8)** 服务。
```sh
root@node1:~ # grep PermitRootLogin /etc/ssh/sshd_config
PermitRootLogin yes
root@node1:~ # service sshd restart
```
通过使用带有 *端口转发* 的 NAT Network（NAT 网络），FreeBSD 主机可以通过本地主机端口访问。例如，可以通过端口 **2211** 访问 **node1**，可以通过端口 **2212** 访问 **node2**，依此类推。如下 **sockstat** 工具输出所示。
要从 VirtualBox 主机系统连接到这样的虚拟机，需要使用如下命令：
```sh
vboxhost % ssh -l root localhost -p 2211
```
## 软件包
现在我们已经可以通过 **ssh(1)** 连接，需要安装所需的软件包。为了让我们的虚拟机能够解析 DNS 查询，还需要做最后一件事。同时，我们将切换 **pkg(8)** 软件包到 “quarterly” 分支。
```sh
root@node1:~ # echo 'nameserver 1.1.1.1' > /etc/resolv.conf
root@node1:~ # sed -i '' s/quarterly/latest/g /etc/pkg/FreeBSD.conf
```
请记得在 **node2** 和 **node3** 系统上重复执行上述两条命令。
现在我们将安装软件包 Pacemaker 和 Corosync。
```sh
root@node1:~ # pkg install pacemaker2 corosync2 crmsh
root@node2:~ # pkg install pacemaker2 corosync2 crmsh
root@node3:~ # pkg install pacemaker2 corosync2 crmsh
```
下面是 **pacemaker2** 和 **corosync2** 的一些需要我们处理的信息。
```sh
Message from pacemaker2-2.0.4:
--
For correct operation, maximum socket buffer size must be tuned
by performing the following command as root :
# sysctl kern.ipc.maxsockbuf=18874368
To preserve this setting across reboots, append the following
to /etc/sysctl.conf :
kern.ipc.maxsockbuf=18874368
======================================================================
Message from corosync2-2.4.5_1:
--
For correct operation, maximum socket buffer size must be tuned
by performing the following command as root :
# sysctl kern.ipc.maxsockbuf=18874368
To preserve this setting across reboots, append the following
to /etc/sysctl.conf :
kern.ipc.maxsockbuf=18874368
```
我们需要修改 **kern.ipc.maxsockbuf** 参数。那就开始修改吧。
```sh
root@node1:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node1:~ # service sysctl restart
root@node2:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node2:~ # service sysctl restart
root@node3:~ # echo 'kern.ipc.maxsockbuf=18874368' >> /etc/sysctl.conf
root@node3:~ # service sysctl restart
```
我们来查看这些软件包都包含了哪些可执行文件。
```sh
root@node1:~ # pkg info -l pacemaker2 | grep bin
/usr/local/sbin/attrd_updater
/usr/local/sbin/cibadmin
/usr/local/sbin/crm_attribute
/usr/local/sbin/crm_diff
/usr/local/sbin/crm_error
/usr/local/sbin/crm_failcount
/usr/local/sbin/crm_master
/usr/local/sbin/crm_mon
/usr/local/sbin/crm_node
/usr/local/sbin/crm_report
/usr/local/sbin/crm_resource
/usr/local/sbin/crm_rule
/usr/local/sbin/crm_shadow
/usr/local/sbin/crm_simulate
/usr/local/sbin/crm_standby
/usr/local/sbin/crm_ticket
/usr/local/sbin/crm_verify
/usr/local/sbin/crmadmin
/usr/local/sbin/fence_legacy
/usr/local/sbin/iso8601
/usr/local/sbin/pacemaker-remoted
/usr/local/sbin/pacemaker_remoted
/usr/local/sbin/pacemakerd
/usr/local/sbin/stonith_admin
root@node1:~ # pkg info -l corosync2 | grep bin
/usr/local/bin/corosync-blackbox
/usr/local/sbin/corosync
/usr/local/sbin/corosync-cfgtool
/usr/local/sbin/corosync-cmapctl
/usr/local/sbin/corosync-cpgtool
/usr/local/sbin/corosync-keygen
/usr/local/sbin/corosync-notifyd
/usr/local/sbin/corosync-quorumtool
root@node1:~ # pkg info -l crmsh | grep bin
/usr/local/bin/crm
```
## 初始化集群
现在我们将初始化 FreeBSD 集群。
首先需要确保各节点的名称可以通过 DNS 解析。
```sh
root@node1:~ # tail -3 /etc/hosts
10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3
root@node2:~ # tail -3 /etc/hosts
10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3
root@node3:~ # tail -3 /etc/hosts
10.0.10.111 node1
10.0.10.112 node2
10.0.10.113 node3
```
现在我们将生成 Corosync 密钥。
```sh
root@node1:~ # corosync-keygen
Corosync Cluster Engine Authentication key generator.
Gathering 1024 bits for key from /dev/random.
Press keys on your keyboard to generate entropy.
Writing corosync key to /usr/local/etc/corosync/authkey.
root@node1:~ # echo $?
0
root@node1:~ # ls -l /usr/local/etc/corosync/authkey
-r-------- 1 root wheel 128 Sep 2 20:37 /usr/local/etc/corosync/authkey
```
现在是 Corosync 配置文件的部分。当然，软件包维护者已经提供了一些示例。
```sh
root@node1:~ # pkg info -l corosync2 | grep example
/usr/local/etc/corosync/corosync.conf.example
/usr/local/etc/corosync/corosync.conf.example.udpu
```
我们将使用第二个示例作为配置的基础。
```sh
root@node1:~ # cp /usr/local/etc/corosync/corosync.conf.example.udpu /usr/local/etc/corosync/corosync.conf
root@node1:~ # vi /usr/local/etc/corosync/corosync.conf
/* LOTS OF EDITS HERE */
root@node1:~ # cat /usr/local/etc/corosync/corosync.conf
totem {
version: 2
crypto_cipher: aes256
crypto_hash: sha256
transport: udpu
interface {
ringnumber: 0
bindnetaddr: 10.0.10.0
mcastport: 5405
ttl: 1
}
}
logging {
fileline: off
to_logfile: yes
to_syslog: no
logfile: /var/log/cluster/corosync.log
debug: off
timestamp: on
logger_subsys {
subsys: QUORUM
debug: off
}
}
nodelist {
node {
ring0_addr: 10.0.10.111
nodeid: 1
}
node {
ring0_addr: 10.0.10.112
nodeid: 2
}
node {
ring0_addr: 10.0.10.113
nodeid: 3
}
}
quorum {
provider: corosync_votequorum
expected_votes: 2
}
```
现在我们需要将 Corosync 密钥和配置文件分发到集群中的各个节点。
可以使用一些专门为此创建的简单工具，例如 net/csync2 集群同步工具，但传统的 net/rsync 也同样可用。
```sh
root@node1:~ # pkg install -y rsync
root@node1:~ # rsync -av /usr/local/etc/corosync/ node2:/usr/local/etc/corosync/
The authenticity of host 'node2 (10.0.10.112)' can't be established.
ECDSA key fingerprint is SHA256:/ZDmln7GKi6n0kbad73TIrajPjGfQqJJX+ReSf3NMvc.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node2' (ECDSA) to the list of known hosts.
Password for root@node2:
sending incremental file list
./
authkey
corosync.conf
service.d/
uidgid.d/
sent 1,100 bytes received 69 bytes 259.78 bytes/sec
total size is 4,398 speedup is 3.76
root@node1:~ # rsync -av /usr/local/etc/corosync/ node3:/usr/local/etc/corosync/
The authenticity of host 'node2 (10.0.10.112)' can't be established.
ECDSA key fingerprint is SHA256:/ZDmln7GKi6n0kbad73TIrajPjGfQqJJX+ReSf3NMvc.
No matching host key fingerprint found in DNS.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'node3' (ECDSA) to the list of known hosts.
Password for root@node3:
sending incremental file list
./
authkey
corosync.conf
service.d/
uidgid.d/
sent 1,100 bytes received 69 bytes 259.78 bytes/sec
total size is 4,398 speedup is 3.76
```
现在我们来检查它们是否一致。
```sh
root@node1:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf
root@node2:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf
root@node3:~ # cksum /usr/local/etc/corosync/{authkey,corosync.conf}
2277171666 128 /usr/local/etc/corosync/authkey
1728717329 622 /usr/local/etc/corosync/corosync.conf
```
一致。
现在我们可以在 **/etc/rc.conf** 文件中添加 **corosync\_enable=YES** 和 **pacemaker\_enable=YES**。
```sh
root@node1:~ # sysrc corosync_enable=YES
corosync_enable: -> YES
root@node1:~ # sysrc pacemaker_enable=YES
pacemaker_enable: -> YES
root@node2:~ # sysrc corosync_enable=YES
corosync_enable: -> YES
root@node2:~ # sysrc pacemaker_enable=YES
pacemaker_enable: -> YES
root@node3:~ # sysrc corosync_enable=YES
corosync_enable: -> YES
root@node3:~ # sysrc pacemaker_enable=YES
pacemaker_enable: -> YES
```
那就启动这些服务吧。
```sh
root@node1:~ # service corosync start
Starting corosync.
Sep 02 20:55:35 notice [MAIN ] Corosync Cluster Engine ('2.4.5'): started and ready to provide service.
Sep 02 20:55:35 info [MAIN ] Corosync built-in features:
Sep 02 20:55:35 warning [MAIN ] interface section bindnetaddr is used together with nodelist. Nodelist one is going to be used.
Sep 02 20:55:35 warning [MAIN ] Please migrate config file to nodelist.
root@node1:~ # ps aux | grep corosync
root 1695 0.0 7.9 38340 38516 - S 20:55 0:00.40 /usr/local/sbin/corosync
root 1699 0.0 0.1 524 336 0 R+ 20:57 0:00.00 grep corosync
```
在 **node2** 和 **node3** 系统上也执行同样操作。
由于 Pacemaker 还未运行，因此操作会失败。
```sh
root@node1:~ # crm status
Could not connect to the CIB: Socket is not connected
crm_mon: Error: cluster is not available on this node
ERROR: status: crm_mon (rc=102):
```
现在我们将启动它。
```sh
root@node1:~ # service pacemaker start
Starting pacemaker.
root@node2:~ # service pacemaker start
Starting pacemaker.
root@node3:~ # service pacemaker start
Starting pacemaker.
```
需要给它一些启动时间，因为如果你立即执行 **crm status** 命令，会看到如下所示的 **0 nodes configured** 消息。
```sh
root@node1:~ # crm status
Cluster Summary:
* Stack: unknown
* Current DC: NONE
* Last updated: Wed Sep 2 20:58:51 2020
* Last change:
* 0 nodes configured
* 0 resource instances configured
Full List of Resources:
* No resources
```
……但过一会儿，所有节点都会被检测到，一切按预期正常工作。
```sh
root@node1:~ # crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
* Last updated: Wed Sep 2 21:02:49 2020
* Last change: Wed Sep 2 20:59:00 2020 by hacluster via crmd on node2
* 3 nodes configured
* 0 resource instances configured
Node List:
* Online: [ node1 node2 node3 ]
Full List of Resources:
* No resources
```
Pacemaker 运行正常。
```sh
root@node1:~ # ps aux | grep pacemaker
root 1716 0.0 0.5 10844 2396 - Is 20:58 0:00.00 daemon: /usr/local/sbin/pacemakerd[1717] (daemon)
root 1717 0.0 5.2 49264 25284 - S 20:58 0:00.27 /usr/local/sbin/pacemakerd
hacluster 1718 0.0 6.1 48736 29708 - Ss 20:58 0:00.75 /usr/local/libexec/pacemaker/pacemaker-based
root 1719 0.0 4.5 40628 21984 - Ss 20:58 0:00.28 /usr/local/libexec/pacemaker/pacemaker-fenced
root 1720 0.0 2.8 25204 13688 - Ss 20:58 0:00.20 /usr/local/libexec/pacemaker/pacemaker-execd
hacluster 1721 0.0 3.9 38148 19100 - Ss 20:58 0:00.25 /usr/local/libexec/pacemaker/pacemaker-attrd
hacluster 1722 0.0 2.9 25460 13864 - Ss 20:58 0:00.17 /usr/local/libexec/pacemaker/pacemaker-schedulerd
hacluster 1723 0.0 5.4 49304 26300 - Ss 20:58 0:00.41 /usr/local/libexec/pacemaker/pacemaker-controld
root 1889 0.0 0.6 11348 2728 0 S+ 21:56 0:00.00 grep pacemaker
```
我们可以查看 Corosync 如何识别其成员。
```sh
root@node1:~ # corosync-cmapctl | grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(10.0.10.111)
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(10.0.10.112)
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
runtime.totem.pg.mrp.srp.members.3.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.3.ip (str) = r(0) ip(10.0.10.113)
runtime.totem.pg.mrp.srp.members.3.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.3.status (str) = joined
```
……或者查看 quorum 信息。
```sh
root@node1:~ # corosync-quorumtool
Quorum information
------------------
Date: Wed Sep 2 21:00:38 2020
Quorum provider: corosync_votequorum
Nodes: 3
Node ID: 1
Ring ID: 1/12
Quorate: Yes
Votequorum information
----------------------
Expected votes: 3
Highest expected: 3
Total votes: 3
Quorum: 2
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
1 1 10.0.10.111 (local)
2 1 10.0.10.112
3 1 10.0.10.113
```
Corosync 日志文件中充满了以下信息。
```sh
root@node1:~ # cat /var/log/cluster/corosync.log
Sep 02 20:55:35 [1694] node1 corosync notice [MAIN ] Corosync Cluster Engine ('2.4.5'): started and ready to provide service.
Sep 02 20:55:35 [1694] node1 corosync info [MAIN ] Corosync built-in features:
Sep 02 20:55:35 [1694] node1 corosync warning [MAIN ] interface section bindnetaddr is used together with nodelist. Nodelist one is going to be used.
Sep 02 20:55:35 [1694] node1 corosync warning [MAIN ] Please migrate config file to nodelist.
Sep 02 20:55:35 [1694] node1 corosync notice [TOTEM ] Initializing transport (UDP/IP Unicast).
Sep 02 20:55:35 [1694] node1 corosync notice [TOTEM ] Initializing transmit/receive security (NSS) crypto: aes256 hash: sha256
Sep 02 20:55:35 [1694] node1 corosync notice [TOTEM ] The network interface [10.0.10.111] is now up.
Sep 02 20:55:35 [1694] node1 corosync notice [SERV ] Service engine loaded: corosync configuration map access [0]
Sep 02 20:55:35 [1694] node1 corosync info [QB ] server name: cmap
Sep 02 20:55:35 [1694] node1 corosync notice [SERV ] Service engine loaded: corosync configuration service [1]
Sep 02 20:55:35 [1694] node1 corosync info [QB ] server name: cfg
Sep 02 20:55:35 [1694] node1 corosync notice [SERV ] Service engine loaded: corosync cluster closed process group service v1.01 [2]
Sep 02 20:55:35 [1694] node1 corosync info [QB ] server name: cpg
Sep 02 20:55:35 [1694] node1 corosync notice [SERV ] Service engine loaded: corosync profile loading service [4]
Sep 02 20:55:35 [1694] node1 corosync notice [QUORUM] Using quorum provider corosync_votequorum
Sep 02 20:55:35 [1694] node1 corosync notice [SERV ] Service engine loaded: corosync vote quorum service v1.0 [5]
Sep 02 20:55:35 [1694] node1 corosync info [QB ] server name: votequorum
Sep 02 20:55:35 [1694] node1 corosync notice [SERV ] Service engine loaded: corosync cluster quorum service v0.1 [3]
Sep 02 20:55:35 [1694] node1 corosync info [QB ] server name: quorum
Sep 02 20:55:35 [1694] node1 corosync notice [TOTEM ] adding new UDPU member {10.0.10.111}
Sep 02 20:55:35 [1694] node1 corosync notice [TOTEM ] adding new UDPU member {10.0.10.112}
Sep 02 20:55:35 [1694] node1 corosync notice [TOTEM ] adding new UDPU member {10.0.10.113}
Sep 02 20:55:35 [1694] node1 corosync notice [TOTEM ] A new membership (10.0.10.111:4) was formed. Members joined: 1
Sep 02 20:55:35 [1694] node1 corosync warning [CPG ] downlist left_list: 0 received
Sep 02 20:55:35 [1694] node1 corosync notice [QUORUM] Members[1]: 1
Sep 02 20:55:35 [1694] node1 corosync notice [MAIN ] Completed service synchronization, ready to provide service.
Sep 02 20:58:14 [1694] node1 corosync notice [TOTEM ] A new membership (10.0.10.111:8) was formed. Members joined: 2
Sep 02 20:58:14 [1694] node1 corosync warning [CPG ] downlist left_list: 0 received
Sep 02 20:58:14 [1694] node1 corosync warning [CPG ] downlist left_list: 0 received
Sep 02 20:58:14 [1694] node1 corosync notice [QUORUM] This node is within the primary component and will provide service.
Sep 02 20:58:14 [1694] node1 corosync notice [QUORUM] Members[2]: 1 2
Sep 02 20:58:14 [1694] node1 corosync notice [MAIN ] Completed service synchronization, ready to provide service.
Sep 02 20:58:19 [1694] node1 corosync notice [TOTEM ] A new membership (10.0.10.111:12) was formed. Members joined: 3
Sep 02 20:58:19 [1694] node1 corosync warning [CPG ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync warning [CPG ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync warning [CPG ] downlist left_list: 0 received
Sep 02 20:58:19 [1694] node1 corosync notice [QUORUM] Members[3]: 1 2 3
Sep 02 20:58:19 [1694] node1 corosync notice [MAIN ] Completed service synchronization, ready to provide service.
```
配置如下。
```sh
root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=2.0.4-2deceaa3ae \
cluster-infrastructure=corosync
```
由于我们不会配置 STONITH 机制，因此将其禁用。
```sh
root@node1:~ # crm configure property stonith-enabled=false
```
禁用 STONITH 后的新配置。
```sh
root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=2.0.4-2deceaa3ae \
cluster-infrastructure=corosync \
stonith-enabled=false
```
STONITH 配置超出了本文的范围，但正确配置的 STONITH 如下所示。
## 第一个服务
现在我们将配置第一个高可用服务——经典示例——一个浮动 IP 地址 :🙂:
```sh
root@node1:~ # crm configure primitive IP ocf:heartbeat:IPaddr2 params ip=10.0.10.200 cidr_netmask="24" op monitor interval="30s"
```
让我们看看它的运行情况。
```sh
root@node1:~ # crm configure show
node 1: node1
node 2: node2
node 3: node3
primitive IP IPaddr2 \
params ip=10.0.10.200 cidr_netmask=24 \
op monitor interval=30s
property cib-bootstrap-options: \
have-watchdog=false \
dc-version=2.0.4-2deceaa3ae \
cluster-infrastructure=corosync \
stonith-enabled=false
```
看起来不错——我们来检查一下集群状态。
```sh
root@node1:~ # crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
* Last updated: Wed Sep 2 22:03:35 2020
* Last change: Wed Sep 2 22:02:53 2020 by root via cibadmin on node1
* 3 nodes configured
* 1 resource instance configured
Node List:
* Online: [ node1 node2 node3 ]
Full List of Resources:
* IP (ocf::heartbeat:IPaddr2): Stopped
Failed Resource Actions:
* IP_monitor_0 on node3 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:53Z', queued=0ms, exec=132ms
* IP_monitor_0 on node2 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:54Z', queued=0ms, exec=120ms
* IP_monitor_0 on node1 'not installed' (5): call=5, status='complete', exitreason='Setup problem: couldn't find command: ip', last-rc-change='2020-09-02 22:02:53Z', queued=0ms, exec=110ms
```
糟糕。Linux 思维模式。系统中预期存在 **ip(8)** 命令。但这是 FreeBSD，像所有 UNIX 系统一样，它提供的是 **ifconfig(8)** 命令。
我们必须另想办法。目前我们先删除这个无用的 **IP** 服务。
```sh
root@node1:~ # crm configure delete IP
```
删除后的状态。
```sh
root@node1:~ # crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
* Last updated: Wed Sep 2 22:04:34 2020
* Last change: Wed Sep 2 22:04:31 2020 by root via cibadmin on node1
* 3 nodes configured
* 0 resource instances configured
Node List:
* Online: [ node1 node2 node3 ]
Full List of Resources:
* No resources
```
## 自定义资源
让我们查看默认 Pacemaker 安装提供了哪些资源。
```sh
root@node1:~ # ls -l /usr/local/lib/ocf/resource.d/pacemaker
total 144
-r-xr-xr-x 1 root wheel 7484 Aug 29 01:22 ClusterMon
-r-xr-xr-x 1 root wheel 9432 Aug 29 01:22 Dummy
-r-xr-xr-x 1 root wheel 5256 Aug 29 01:22 HealthCPU
-r-xr-xr-x 1 root wheel 5342 Aug 29 01:22 HealthIOWait
-r-xr-xr-x 1 root wheel 9450 Aug 29 01:22 HealthSMART
-r-xr-xr-x 1 root wheel 6186 Aug 29 01:22 Stateful
-r-xr-xr-x 1 root wheel 11370 Aug 29 01:22 SysInfo
-r-xr-xr-x 1 root wheel 5856 Aug 29 01:22 SystemHealth
-r-xr-xr-x 1 root wheel 7382 Aug 29 01:22 attribute
-r-xr-xr-x 1 root wheel 7854 Aug 29 01:22 controld
-r-xr-xr-x 1 root wheel 16134 Aug 29 01:22 ifspeed
-r-xr-xr-x 1 root wheel 11040 Aug 29 01:22 o2cb
-r-xr-xr-x 1 root wheel 11696 Aug 29 01:22 ping
-r-xr-xr-x 1 root wheel 6356 Aug 29 01:22 pingd
-r-xr-xr-x 1 root wheel 3702 Aug 29 01:22 remote
```
资源不多……我们将尝试将 **Dummy** 服务修改为在 FreeBSD 上的 IP 切换器。
```sh
root@node1:~ # cp /usr/local/lib/ocf/resource.d/pacemaker/Dummy /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
root@node1:~ # vi /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
/* 输入量真大啊…… */
```
由于 *WordPress* 博客系统的限制，我不得不将这个 **ifconfig** 资源以图片形式发布……但别担心，文本版本也可在这里下载——[ifconfig.odt](https://vermaden.wordpress.com/wp-content/uploads/2020/09/ifconfig.odt)。
此外，第一个版本的效果并不理想……
```sh
root@node1:~ # setenv OCF_ROOT /usr/local/lib/ocf
root@node1:~ # ocf-tester -n resourcename /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
Beginning tests for /usr/local/lib/ocf/resource.d/pacemaker/ifconfig...
* rc=3: Your agent has too restrictive permissions: should be 755
-:1: parser error : Start tag expected, '<' not found
usage: /usr/local/lib/ocf/resource.d/pacemaker/ifconfig {start|stop|monitor}
^
* rc=1: Your agent produces meta-data which does not conform to ra-api-1.dtd
* rc=3: Your agent does not support the meta-data action
* rc=3: Your agent does not support the validate-all action
* rc=0: Monitoring a stopped resource should return 7
* rc=0: The initial probe for a stopped resource should return 7 or 5 even if all binaries are missing
* Your agent does not support the notify action (optional)
* Your agent does not support the demote action (optional)
* Your agent does not support the promote action (optional)
* Your agent does not support master/slave (optional)
* rc=0: Monitoring a stopped resource should return 7
* rc=0: Monitoring a stopped resource should return 7
* rc=0: Monitoring a stopped resource should return 7
* Your agent does not support the reload action (optional)
Tests failed: /usr/local/lib/ocf/resource.d/pacemaker/ifconfig failed 9 tests
```
但在为其添加 755 权限并进行了若干（上百次）修改后，它终于可以使用了。
```sh
root@node1:~ # vi /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
/* LOTS OF NERVOUS TYPING */
root@node1:~ # chmod 755 /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
root@node1:~ # setenv OCF_ROOT /usr/local/lib/ocf
root@node1:~ # ocf-tester -n resourcename /usr/local/lib/ocf/resource.d/pacemaker/ifconfig
Beginning tests for /usr/local/lib/ocf/resource.d/pacemaker/ifconfig...
* Your agent does not support the notify action (optional)
* Your agent does not support the demote action (optional)
* Your agent does not support the promote action (optional)
* Your agent does not support master/slave (optional)
* Your agent does not support the reload action (optional)
/usr/local/lib/ocf/resource.d/pacemaker/ifconfig passed all tests
```
看起来可以使用。
这是 **ifconfig** 资源。目前它相当有限，而且 IP 地址是硬编码的。
让我们尝试向 FreeBSD 集群添加新的 **IP** 资源。
## 测试
```sh
root@node1:~ # crm configure primitive IP ocf:pacemaker:ifconfig op monitor interval="30"
```
已添加。
我们来看现在 **status** 命令显示的情况。
```sh
root@node1:~ # crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
* Last updated: Wed Sep 2 22:44:52 2020
* Last change: Wed Sep 2 22:44:44 2020 by root via cibadmin on node1
* 3 nodes configured
* 1 resource instance configured
Node List:
* Online: [ node1 node2 node3 ]
Full List of Resources:
* IP (ocf::pacemaker:ifconfig): Started node1
Failed Resource Actions:
* IP_monitor_0 on node3 'not installed' (5): call=24, status='Not installed', exitreason='', last-rc-change='2020-09-02 22:42:52Z', queued=0ms, exec=5ms
* IP_monitor_0 on node2 'not installed' (5): call=24, status='Not installed', exitreason='', last-rc-change='2020-09-02 22:42:53Z', queued=0ms, exec=2ms
```
糟糕，我忘了将这个新的 **ifconfig** 资源复制到其他节点。现在来修复它。
```sh
root@node1:~ # rsync -av /usr/local/lib/ocf/resource.d/pacemaker/ node2:/usr/local/lib/ocf/resource.d/pacemaker/
Password for root@node2:
sending incremental file list
./
ifconfig
sent 3,798 bytes received 38 bytes 1,534.40 bytes/sec
total size is 128,003 speedup is 33.37
root@node1:~ # rsync -av /usr/local/lib/ocf/resource.d/pacemaker/ node3:/usr/local/lib/ocf/resource.d/pacemaker/
Password for root@node3:
sending incremental file list
./
ifconfig
sent 3,798 bytes received 38 bytes 1,534.40 bytes/sec
total size is 128,003 speedup is 33.37
```
现在我们先停止、删除，然后重新添加这个重要的资源。
```sh
root@node1:~ # crm resource stop IP
root@node1:~ # crm configure delete IP
root@node1:~ # crm configure primitive IP ocf:pacemaker:ifconfig op monitor interval="30"
```
祈祷顺利吧。
```sh
root@node1:~ # crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
* Last updated: Wed Sep 2 22:45:46 2020
* Last change: Wed Sep 2 22:45:43 2020 by root via cibadmin on node1
* 3 nodes configured
* 1 resource instance configured
Node List:
* Online: [ node1 node2 node3 ]
Full List of Resources:
* IP (ocf::pacemaker:ifconfig): Started node1
```
看起来运行正常。
让我们验证它是否确实在应该的节点上启动。
```sh
root@node1:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
options=81009b
ether 08:00:27:2a:78:60
inet 10.0.10.111 netmask 0xffffff00 broadcast 10.0.10.255
inet 10.0.10.200 netmask 0xffffff00 broadcast 10.0.10.255
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29
root@node2:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
options=81009b
ether 08:00:27:80:50:05
inet 10.0.10.112 netmask 0xffffff00 broadcast 10.0.10.255
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29
root@node3:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
options=81009b
ether 08:00:27:74:5e:b9
inet 10.0.10.113 netmask 0xffffff00 broadcast 10.0.10.255
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29
```
看起来确实在工作。
现在我们尝试将它移动到集群中的另一台节点。
```sh
root@node1:~ # crm resource move IP node3
INFO: Move constraint created for IP to node3
root@node1:~ # crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
* Last updated: Wed Sep 2 22:47:31 2020
* Last change: Wed Sep 2 22:47:28 2020 by root via crm_resource on node1
* 3 nodes configured
* 1 resource instance configured
Node List:
* Online: [ node1 node2 node3 ]
Full List of Resources:
* IP (ocf::pacemaker:ifconfig): Started node3
```
已成功切换到 **node3** 系统。
```sh
root@node3:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
options=81009b
ether 08:00:27:74:5e:b9
inet 10.0.10.113 netmask 0xffffff00 broadcast 10.0.10.255
inet 10.0.10.200 netmask 0xffffff00 broadcast 10.0.10.255
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29
root@node1:~ # ifconfig em0
em0: flags=8843 metric 0 mtu 1500
options=81009b
ether 08:00:27:2a:78:60
inet 10.0.10.111 netmask 0xffffff00 broadcast 10.0.10.255
media: Ethernet autoselect (1000baseT )
status: active
nd6 options=29
```
现在我们将 **关闭** **node3** 系统，以检查这个 **IP** 是否真的具备高可用性。
```sh
root@node2:~ # crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
* Last updated: Wed Sep 2 22:49:57 2020
* Last change: Wed Sep 2 22:47:29 2020 by root via crm_resource on node1
* 3 nodes configured
* 1 resource instance configured
Node List:
* Online: [ node1 node2 node3 ]
Full List of Resources:
* IP (ocf::pacemaker:ifconfig): Started node3
root@node3:~ # poweroff
root@node2:~ # crm status
Cluster Summary:
* Stack: corosync
* Current DC: node2 (version 2.0.4-2deceaa3ae) - partition with quorum
* Last updated: Wed Sep 2 22:50:16 2020
* Last change: Wed Sep 2 22:47:29 2020 by root via crm_resource on node1
* 3 nodes configured
* 1 resource instance configured
Node List:
* Online: [ node1 node2 ]
* OFFLINE: [ node3 ]
Full List of Resources:
* IP (ocf::pacemaker:ifconfig): Started node1
```
看起来故障切换进行得很顺利。
**crm** 命令还会对其输出的不同部分进行高亮显示。
很高兴知道 Pacemaker 和 Corosync 集群在 FreeBSD 上运行良好。
虽然需要一些工作来编写必要的资源文件，但只要有时间和决心，完全可以将 FreeBSD 打造成非常强大的高可用集群。
---
# Agent Instructions: Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://translated-articles.bsdcn.org/2025-nian/freebsd-cluster-with-pacemaker-and-corosync.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.