MSG Chain 节点状态同步与灾难恢复指南
数据来源：MSG Chain 代码库核实
主网状态: No-Go — 当前 MSGChain 主网裁决为 No-Go，以下内容反映代码实际状态，不代表生产可用。
目录
1. 概述
1.1 为什么状态同步和灾备至关重要
区块链节点的核心职责是维护网络状态的完整副本。一旦节点数据损坏或丢失，将导致：
- 节点无法参与共识，影响网络去中心化程度
- 验证者节点掉线将遭受质押代币罚没 (slashing)
- RPC/API 服务中断，影响依赖链上数据的应用程序
- 全节点同步时间过长，导致恢复窗口无限期延长
MSG Chain 作为基于 Cosmos SDK 构建的高性能公链，其节点状态管理涉及多层数据结构的协同工作。本指南系统性地阐述状态同步机制、备份策略、灾难恢复流程及高可用架构设计，帮助节点运维人员建立健壮的运营体系。
1.2 MSG Chain 状态存储架构
MSG Chain 的状态存储由以下组件构成：
| 组件 | 存储后端 | 路径 (默认) | 说明 |
|---|---|---|---|
| 区块链状态 | BadgerDB | ~/.msgchain/data/ |
应用层状态，键值对存储 |
| CometBFT 共识状态 | LevelDB | ~/.msgchain/data/ |
共识层 WAL、区块数据 |
| 键值存储 (KVStore) | BadgerDB | ~/.msgchain/data/badger/ |
Cosmos SDK 多存储实例 |
| WASM 合约状态 | BadgerDB | ~/.msgchain/data/wasm/ |
智能合约持久化数据 |
| 证据库 | LevelDB | ~/.msgchain/data/evidence.db/ |
共识证据存储 |
| 交易索引 | LevelDB | ~/.msgchain/data/tx_index.db/ |
交易查询索引 |
BadgerDB 是 MSG Chain 的核心存储引擎，支持：
- LSM-Tree 结构的高吞吐键值存储
- ACID 事务性写入
- Snappy/ZSTD 数据压缩
- 在线 Live Backup 流式备份
- 键值分离 (Value Log) 减少写放大
1.3 状态同步类型
MSG Chain 节点支持以下四种状态同步机制：
| 类型 | 速度 | 带宽消耗 | 验证深度 | 适用场景 |
|---|---|---|---|---|
| 快照同步 | 极快 | 高 | 低 | 新节点加入、快速恢复 |
| 区块回放 | 慢 | 低 | 全量 | 对完整性要求极高的场景 |
| 节点区块同步 | 中 | 中 | 中 | 短时间掉线恢复 |
| 全量同步 (从创世) | 极慢 | 高 | 完整 | 归档节点、审计节点 |
1.4 灾难场景分类
1.4.1 数据损坏
- BadgerDB 日志损坏: 意外断电导致 Value Log 损坏
- 索引不一致: 键值存储索引与数据不匹配
- WAL 损坏: 预写日志损坏导致无法回放
- 区块数据库损坏: CometBFT 区块存储损坏
典型症状:
ERROR: failed to load database: value log corrupted at offset 12345
panic: runtime error: invalid memory address or nil pointer dereference
ERROR: failed to replay WAL: wal file corruption at seq 789
1.4.2 硬件故障
- 磁盘损坏 (不可纠正的扇区错误)
- 内存 ECC 错误导致数据静默损坏
- CPU 故障导致计算结果不一致
- 网络接口故障导致节点隔离
1.4.3 网络分区
- 节点长时间无法连接对等节点
- 共识超时导致验证者错过区块
- 区块同步停滞，状态严重落后
1.4.4 51% 攻击
- 攻击者控制超过 2/3 共识权重
- 链重组 (Reorg) 导致已确认交易回滚
- 验证者需手动干预恢复分叉
1.4.5 人为错误
- 误删数据目录
- 错误配置更改导致节点无法启动
- 错误的 CLI 操作 (如误提交双重签名)
- 密钥泄露或丢失
1.5 恢复目标指标
| 指标 | 推荐值 | 说明 |
|---|---|---|
| RTO | < 30 分钟 | 从灾难发生到服务恢复的最大可接受时间 |
| RPO | < 1 小时 | 可接受的最大数据丢失时间窗口 |
| MTBF | > 90 天 | 系统可靠性指标 |
| MTTR | < 1 小时 | 从故障到完全恢复的平均时间 |
1.6 备份策略总览
备份层级示意图:
+--------------------------------------------------+
| 基础层 |
| 全量备份 (每日) + 增量备份 (每4小时) |
+--------------------------------------------------+
|
v
+--------------------------------------------------+
| 验证层 |
| 备份校验 (SHA256) + 恢复测试 (每月) |
+--------------------------------------------------+
|
v
+--------------------------------------------------+
| 分发层 |
| 本地存储 → 异地主存储 → 异地副本 (3-2-1 规则) |
+--------------------------------------------------+
2. 状态同步机制
2.1 同步基础概念
Cosmos SDK 节点启动时的状态同步流程如下：
节点启动 → 加载本地数据库
↓
检查最后提交高度 (LatestHeight)
↓
若有本地状态 → 从本地高度开始同步
↓
若无本地状态 → 从创世开始同步 / 快照同步 / 状态同步
↓
P2P 网络发现对等节点 → 交换区块
↓
逐块验证签名 → 提交到应用层 (BeginBlock → DeliverTx → EndBlock → Commit)
↓
更新本地 BadgerDB 状态 → 继续下一高度
2.2 全量同步 (从创世区块)
全量同步从区块 0 开始，逐块下载、验证并执行所有交易。这是最完整的同步方式，但速度最慢。
2.2.1 初始化节点
# 初始化节点配置
msgchaind init <your-moniker> --chain-id msg-chain-1
# 下载创世文件
curl -fsSL https://github.com/msgchain/mainnet/raw/main/genesis.json \
-o ~/.msgchain/config/genesis.json
# 验证创世文件哈希
sha256sum ~/.msgchain/config/genesis.json
# 预期输出: a1b2c3d4... 请与官方渠道公布的哈希值核对
# 添加种子节点
sed -i 's/seeds = ""/seeds = "<seed-id>@<seed-host>:26656"/' \
~/.msgchain/config/config.toml
# 添加持久对等节点
sed -i 's/persistent_peers = ""/persistent_peers = "<peer-id>@<peer-host>:26656"/' \
~/.msgchain/config/config.toml
2.2.2 启动全量同步
# 启动节点并观察同步进度
msgchaind start 2>&1 | tee ~/sync.log
# 监控同步高度 (另开终端)
watch -n 10 '
curl -s localhost:26657/status | jq -r '\''
"同步高度: " + .result.sync_info.latest_block_height,
"已同步: " + (.result.sync_info.catching_up | tostring)
'\''
'
2.2.3 全量同步耗时估算
| 硬件配置 | 区块高度 100万 | 区块高度 1000万 |
|---|---|---|
| 4C8G + SSD | ~3 天 | ~30 天 |
| 8C16G + NVMe | ~1.5 天 | ~15 天 |
| 16C32G + NVMe RAID | ~12 小时 | ~5 天 |
在生产环境中，全量同步仅建议用于归档节点。
2.3 快速同步 (Block Replay)
快速同步跳过交易执行的重放，仅下载并验证区块头和数据，然后提交到应用层。
2.3.1 配置快速同步
# ~/.msgchain/config/config.toml
[fastsync]
version = "v2"
allow_duplicate_ip = true
2.3.2 执行快速同步
# 设置快速同步模式并启动
msgchaind start --fast-sync
# 或通过环境变量
MSGCHAIND_FAST_SYNC=1 msgchaind start
# 监控同步进度
curl -s localhost:26657/status | jq '.result.sync_info'
快速同步的局限性：
- 如果本地数据库已有部分数据但高度落后过多，快速同步可能失败
- 需要至少一个可用的快速同步提供者对等节点
- 网络带宽要求高
2.4 信任同步 (Trust Sync)
信任同步允许节点从指定的信任高度和区块哈希开始同步，无需逐块验证整个历史。
2.4.1 信任同步原理
1. 运维人员从可信源获取 {高度, 区块哈希}
2. 节点下载该高度的区块头
3. 对比区块头哈希与信任哈希
4. 若匹配，信任该区块之前的所有区块
5. 从该高度开始同步后续区块（验证签名）
2.4.2 获取信任点和哈希
# 方法一：从已同步的节点获取
curl -s localhost:26657/status | jq '.result.sync_info.latest_block_height'
curl -s localhost:26657/block?height=<height> | jq -r '.result.block_id.hash'
# 方法二：从区块浏览器获取
# URL: https://scan.msgchain.org/block/<height>
# 方法三：从多个对等节点交叉验证
PEERS=("node1.msgchain.org" "node2.msgchain.org" "node3.msgchain.org")
for peer in "${PEERS[@]}"; do
echo "=== $peer ==="
curl -s "http://$peer:26657/status" | jq '.result.sync_info.latest_block_height'
curl -s "http://$peer:26657/block?height=<height>" | jq -r '.result.block_id.hash'
done
2.4.3 配置信任同步
# 停止节点
systemctl stop msgchaind
# 设置信任高度和哈希
TRUST_HEIGHT=5000000
TRUST_HASH="A1B2C3D4E5F6..."
# 写入配置
sed -i 's/trust_height = 0/trust_height = 5000000/' ~/.msgchain/config/config.toml
sed -i 's/trust_hash = ""/trust_hash = "A1B2C3D4E5F6..."/' ~/.msgchain/config/config.toml
2.4.4 启动信任同步
# 使用 --trust-sync 标志启动
msgchaind start --trust-sync --trust-height 5000000 --trust-hash "A1B2C3D4E5F6..."
# 监控信任同步进度
watch -n 5 '
STATUS=$(curl -s localhost:26657/status)
echo "信任高度: 5000000"
echo "当前高度: $(echo $STATUS | jq -r '\''.result.sync_info.latest_block_height'\'')"
echo "追赶上: $(echo $STATUS | jq -r '\''.result.sync_info.catching_up'\'')"
'
2.5 快照同步 (Snapshot Sync)
快照同步是 MSG Chain 节点初始化或恢复的最快方式。节点从快照存档中直接恢复应用状态，然后从快照高度开始同步后续区块。
2.5.1 创建快照
# 基本快照创建
msgchaind snapshot create --height 1000000
# 指定输出目录
msgchaind snapshot create \
--height 1000000 \
--output-dir /data/snapshots
# 带压缩的快照
msgchaind snapshot create \
--height 1000000 \
--compress \
--compress-algorithm zstd
# 跳过指定模块
msgchaind snapshot create \
--height 1000000 \
--skip-modules "auth,bank,staking"
# 创建快照并验证
msgchaind snapshot create \
--height 1000000 \
--verify \
--output-dir /data/snapshots
echo "快照创建完成: $(ls -lh /data/snapshots/latest/)"
2.5.2 快照导出为可传输格式
# 将快照导出为 tar 存档
SNAPSHOT_DIR=$(ls -td ~/.msgchain/data/snapshots/*/ | head -1)
tar -I lz4 -cf /tmp/msgchain-snapshot-1000000.tar.lz4 \
-C "$SNAPSHOT_DIR" .
# 计算校验和
sha256sum /tmp/msgchain-snapshot-1000000.tar.lz4 \
> /tmp/msgchain-snapshot-1000000.tar.lz4.sha256
# 上传到快照服务器
scp /tmp/msgchain-snapshot-1000000.tar.lz4* \
user@snapshots.msgchain.org:/var/www/snapshots/
2.5.3 从快照恢复
方法一：使用内置快照恢复
# 停止节点
systemctl stop msgchaind
# 清除旧数据 (谨慎！确保已备份)
rm -rf ~/.msgchain/data/
# 启动节点并指定快照高度
msgchaind start --snapshot-height 1000000
方法二：手动从存档恢复
# 停止节点
systemctl stop msgchaind
# 备份当前数据
mv ~/.msgchain/data ~/.msgchain/data_old_$(date +%Y%m%d)
# 创建数据目录
mkdir -p ~/.msgchain/data/
# 下载快照
wget -O /tmp/snapshot.tar.lz4 \
https://snapshots.msgchain.org/msg-chain-1_1000000.tar.lz4
# 验证校验和
echo "<expected-sha256> /tmp/snapshot.tar.lz4" | sha256sum -c -
if [ $? -ne 0 ]; then
echo "错误: 快照校验和不匹配！"
exit 1
fi
# 解压快照到数据目录
tar -I lz4 -xvf /tmp/snapshot.tar.lz4 \
-C ~/.msgchain/data/
# 重置 CometBFT 数据
msgchaind tendermint unsafe-reset-all --keep-addr-book
# 启动节点
systemctl start msgchaind
# 检查日志
journalctl -u msgchaind -f -n 50
2.5.4 自动化快照脚本
#!/bin/bash
# 脚本: create_msgchain_snapshot.sh
# 功能: 自动创建 MSG Chain 节点快照并上传到快照服务器
# 用法: ./create_msgchain_snapshot.sh [高度] [输出目录]
set -euo pipefail
CHAIN_ID="msg-chain-1"
NODE_HOME="${HOME}/.msgchain"
SNAPSHOT_DIR="${2:-/data/snapshots}"
SSH_USER="snapshot-upload"
SSH_HOST="snapshots.msgchain.org"
SSH_PATH="/var/www/snapshots/"
RETENTION_DAYS=14
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; }
check_prereqs() {
local cmds=("msgchaind" "tar" "lz4" "sha256sum" "scp" "curl")
for cmd in "${cmds[@]}"; do
if ! command -v "$cmd" &>/dev/null; then
log_error "命令未找到: $cmd"
exit 1
fi
done
}
get_target_height() {
if [ -n "${1:-}" ]; then
echo "$1"
else
local current
current=$(curl -s localhost:26657/status \
| jq -r '.result.sync_info.latest_block_height')
echo $((current - 100))
fi
}
cleanup_old_snapshots() {
log_info "清理 ${RETENTION_DAYS} 天前的旧快照..."
find "$SNAPSHOT_DIR" -name "*.tar.lz4" -mtime +${RETENTION_DAYS} -delete
find "$SNAPSHOT_DIR" -name "*.sha256" -mtime +${RETENTION_DAYS} -delete
}
create_snapshot() {
local height="$1"
log_info "等待节点高度达到 ${height}..."
while true; do
local current_height
current_height=$(curl -s localhost:26657/status \
| jq -r '.result.sync_info.latest_block_height')
if [ "$current_height" -ge "$height" ]; then
break
fi
sleep 10
done
log_info "等待 ${height} 高度完成确认..."
while true; do
local current_height
current_height=$(curl -s localhost:26657/status \
| jq -r '.result.sync_info.latest_block_height')
if [ "$current_height" -ge $((height + 3)) ]; then
break
fi
sleep 5
done
log_info "创建快照: 高度 ${height}..."
msgchaind snapshot create \
--height "$height" \
--output-dir "$SNAPSHOT_DIR" \
--compress \
--compress-algorithm zstd
log_info "快照创建完成！"
}
export_snapshot() {
local height="$1"
local latest_snapshot
latest_snapshot=$(ls -td "$SNAPSHOT_DIR"/*/ 2>/dev/null | head -1)
if [ -z "$latest_snapshot" ]; then
log_error "未找到快照目录！"
exit 1
fi
local export_file="${SNAPSHOT_DIR}/${CHAIN_ID}_${height}.tar.lz4"
log_info "导出快照到 ${export_file}..."
tar -I lz4 -cf "$export_file" -C "$latest_snapshot" .
log_info "计算校验和..."
sha256sum "$export_file" > "${export_file}.sha256"
log_info "导出完成!"
ls -lh "$export_file" "${export_file}.sha256"
}
upload_snapshot() {
local height="$1"
local export_file="${SNAPSHOT_DIR}/${CHAIN_ID}_${height}.tar.lz4"
if [ ! -f "$export_file" ]; then
log_error "导出文件不存在: $export_file"
exit 1
fi
log_info "上传快照到 ${SSH_HOST}:${SSH_PATH}..."
scp "${export_file}" "${export_file}.sha256" \
"${SSH_USER}@${SSH_HOST}:${SSH_PATH}"
ssh "${SSH_USER}@${SSH_HOST}" \
"ln -sf ${CHAIN_ID}_${height}.tar.lz4 ${SSH_PATH}latest.tar.lz4 && \
ln -sf ${CHAIN_ID}_${height}.tar.lz4.sha256 ${SSH_PATH}latest.tar.lz4.sha256"
log_info "上传完成！"
}
main() {
check_prereqs
local target_height
target_height=$(get_target_height "$@")
log_info "目标快照高度: ${target_height}"
create_snapshot "$target_height"
export_snapshot "$target_height"
cleanup_old_snapshots
log_info "快照创建流程全部完成！"
}
main "$@"
2.5.5 快照提供者配置
# 快照提供者的 config.toml 配置
[statesync]
enable = true
rpc_servers = "tcp://0.0.0.0:26657,tcp://0.0.0.0:26657"
trust_height = 0
trust_hash = ""
trust_period = "168h0m0s"
# 快照保留策略 (app.toml)
[state-sync]
snapshot-interval = 5000
snapshot-keep-recent = 10
snapshot-min-chunk-size = 4
2.5.6 快照恢复完整脚本
#!/bin/bash
# 脚本: restore_from_snapshot.sh
# 功能: 从快照完全恢复 MSG Chain 节点
# 用法: ./restore_from_snapshot.sh <snapshot_url>
set -euo pipefail
SNAPSHOT_URL="${1:-}"
NODE_HOME="${HOME}/.msgchain"
TEMP_DIR="/tmp/msgchain-restore-$(date +%s)"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" >&2; }
if [ -z "$SNAPSHOT_URL" ]; then
log_error "用法: $0 <snapshot_url>"
exit 1
fi
for cmd in "msgchaind" "lz4" "tar" "wget" "sha256sum" "systemctl"; do
if ! command -v "$cmd" &>/dev/null; then
log_error "命令未找到: $cmd"
exit 1
fi
done
log_info "========== 快照恢复开始 =========="
log_info "快照地址: ${SNAPSHOT_URL}"
mkdir -p "$TEMP_DIR"
log_info "下载快照..."
SNAPSHOT_FILE="${TEMP_DIR}/snapshot.tar.lz4"
wget -O "$SNAPSHOT_FILE" "$SNAPSHOT_URL"
SHA_URL="${SNAPSHOT_URL}.sha256"
if wget -O "${TEMP_DIR}/snapshot.sha256" "$SHA_URL" 2>/dev/null; then
log_info "验证快照完整性..."
cd "$TEMP_DIR"
sha256sum -c snapshot.sha256 || {
log_error "快照校验和不匹配！"
exit 1
}
cd - > /dev/null
fi
log_info "停止 msgchaind 服务..."
systemctl stop msgchaind || true
sleep 3
if [ -d "${NODE_HOME}/data" ]; then
BACKUP_DIR="${NODE_HOME}/data_backup_$(date +%Y%m%d_%H%M%S)"
log_info "备份当前数据到 ${BACKUP_DIR}..."
mv "${NODE_HOME}/data" "$BACKUP_DIR"
fi
mkdir -p "${NODE_HOME}/data"
log_info "解压快照..."
if command -v pv &>/dev/null; then
pv "$SNAPSHOT_FILE" | tar -I lz4 -xf - -C "${NODE_HOME}/data/"
else
tar -I lz4 -xf "$SNAPSHOT_FILE" -C "${NODE_HOME}/data/"
fi
log_info "重置 CometBFT 数据..."
msgchaind tendermint unsafe-reset-all --keep-addr-book
log_info "验证创世文件..."
msgchaind validate-genesis || {
log_error "创世文件验证失败"
exit 1
}
log_info "启动 msgchaind 服务..."
systemctl start msgchaind
sleep 5
for i in $(seq 1 12); do
STATUS=$(curl -s localhost:26657/status 2>/dev/null || true)
if [ -n "$STATUS" ]; then
HEIGHT=$(echo "$STATUS" | jq -r '.result.sync_info.latest_block_height')
CATCHING_UP=$(echo "$STATUS" | jq -r '.result.sync_info.catching_up')
log_info "高度: ${HEIGHT}, 追赶上: ${CATCHING_UP}"
break
fi
sleep 10
done
log_info "清理临时文件..."
rm -rf "$TEMP_DIR"
log_info "========== 快照恢复完成 =========="
2.6 配置状态同步参数
# ~/.msgchain/config/config.toml
[p2p]
max_num_inbound_peers = 60
max_num_outbound_peers = 40
pex = true
persistent_peers_max_dial_period = "30s"
seeds = "seed1.msgchain.org:26656,seed2.msgchain.org:26656"
persistent_peers = "node1.msgchain.org:26656,node2.msgchain.org:26656"
addr_book_file = "/home/msgchain/.msgchain/config/addrbook.json"
addr_book_strict = true
max_connections = 100
[fastsync]
version = "v2"
[blocksync]
enable = true
[statesync]
enable = true
rpc_servers = "https://rpc.msgchain.org:443,https://rpc2.msgchain.org:443"
trust_height = 0
trust_hash = ""
trust_period = "168h0m0s"
discovery_time = "5s"
2.7 P2P 网络调优
# Linux 系统级网络优化 (需 root)
cat >> /etc/security/limits.conf << 'EOF'
msgchain soft nofile 65536
msgchain hard nofile 65536
EOF
cat >> /etc/sysctl.d/90-msgchain.conf << 'EOF'
net.netfilter.nf_conntrack_max = 1048576
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_max_syn_backlog = 65536
net.core.somaxconn = 65536
net.ipv4.ip_local_port_range = 10240 65535
EOF
sysctl -p /etc/sysctl.d/90-msgchain.conf
lsmod | grep bbr
sysctl net.ipv4.tcp_congestion_control
2.8 同步状态诊断
#!/bin/bash
# 脚本: check_sync_status.sh
# 功能: 全面诊断 MSG Chain 节点同步状态
set -euo pipefail
RPC_ENDPOINT="${1:-http://localhost:26657}"
echo "=========================================="
echo " MSG Chain 节点同步状态诊断报告"
echo " 时间: $(date '+%Y-%m-%d %H:%M:%S')"
echo "=========================================="
echo "--- 节点基本信息 ---"
STATUS=$(curl -sf "${RPC_ENDPOINT}/status" | jq '.result')
echo "$STATUS" | jq '{node_id: .node_info.id, moniker: .node_info.moniker, network: .node_info.network, version: .node_info.version}'
echo "--- 同步信息 ---"
echo "$STATUS" | jq '{latest_block_height, latest_block_time, catching_up}'
echo "--- 对等节点 ---"
NET_INFO=$(curl -sf "${RPC_ENDPOINT}/net_info" | jq '.result')
echo "已连接对等节点数: $(echo "$NET_INFO" | jq '.n_peers')"
echo "$NET_INFO" | jq '.peers[:5][] | {node_id: .node_info.id, remote_ip: .remote_ip, is_outbound: .is_outbound}'
echo "--- 共识状态 ---"
curl -sf "${RPC_ENDPOINT}/consensus_state" | jq '.result.round_state | {height: .height, round: .round, step: .step}' 2>/dev/null || echo "N/A"
echo "--- 磁盘使用 ---"
DATA_DIR="${HOME}/.msgchain/data"
if [ -d "$DATA_DIR" ]; then
du -sh "$DATA_DIR" 2>/dev/null
df -h "$DATA_DIR" 2>/dev/null | tail -1
fi
echo "=========================================="
3. 数据备份策略
3.1 备份策略设计原则
3.1.1 3-2-1 备份规则
3 份副本 → 原始数据 + 本地备份 + 异地备份
2 种介质 → 本地磁盘 + 远程存储 (S3/NFS)
1 个异地 → 至少一份备份在不同物理位置
3.1.2 RPO/RTO 目标
| 备份类型 | RPO | RTO | 频率 |
|---|---|---|---|
| 全量备份 | 24 小时 | 2 小时 | 每日 |
| 增量备份 | 4 小时 | 30 分钟 | 每 4 小时 |
| 实时备份 | 5 分钟 | 5 分钟 | 持续 |
3.2 全量冷备份
#!/bin/bash
# 脚本: full_backup_cold.sh
# 功能: MSG Chain 全量冷备份 (停止节点)
# 调度: cron 每日执行
set -euo pipefail
CHAIN_ID="msg-chain-1"
NODE_HOME="${HOME}/.msgchain"
BACKUP_BASE="/backup/msgchain"
RETENTION_DAYS=30
LOG_FILE="/var/log/msgchain/backup.log"
COMPRESSION_THREADS=$(nproc)
DATE_STAMP=$(date +%Y%m%d)
TIME_STAMP=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="${BACKUP_BASE}/full/${DATE_STAMP}"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
check_prerequisites() {
DATA_SIZE=$(du -sb "$NODE_HOME/data" 2>/dev/null | cut -f1)
DATA_SIZE=${DATA_SIZE:-0}
BACKUP_AVAILABLE=$(df --output=avail "$BACKUP_BASE" 2>/dev/null | tail -1)
BACKUP_AVAILABLE=${BACKUP_AVAILABLE:-0}
BACKUP_AVAILABLE_BYTES=$((BACKUP_AVAILABLE * 1024))
if [ "$BACKUP_AVAILABLE_BYTES" -lt "$((DATA_SIZE * 2))" ]; then
log_error "磁盘空间不足!"
exit 1
fi
}
perform_cold_backup() {
log_info "停止 msgchaind 服务..."
systemctl stop msgchaind
sleep 5
mkdir -p "$BACKUP_DIR"
log_info "压缩数据目录..."
tar -I "zstd -T${COMPRESSION_THREADS}" \
-cf "${BACKUP_DIR}/data.tar.zst" \
-C "$NODE_HOME" data/
log_info "备份配置目录..."
tar -I zstd \
-cf "${BACKUP_DIR}/config.tar.zst" \
-C "$NODE_HOME" config/
log_info "备份密钥..."
tar -I zstd \
-cf "${BACKUP_DIR}/keys.tar.zst" \
-C "$NODE_HOME" keyring-file/
cat > "${BACKUP_DIR}/backup_metadata.json" << EOF
{
"chain_id": "${CHAIN_ID}",
"timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"backup_type": "full_cold",
"hostname": "$(hostname)"
}
EOF
cd "$BACKUP_DIR"
sha256sum *.tar.zst > checksums.sha256
cd - > /dev/null
log_info "冷备份完成!"
}
verify_backup() {
log_info "验证备份完整性..."
cd "$BACKUP_DIR"
sha256sum -c checksums.sha256 || { log_error "校验失败"; return 1; }
cd - > /dev/null
tar -I zstd -tf "${BACKUP_DIR}/data.tar.zst" 2>/dev/null | head -5 || { log_error "无法读取"; return 1; }
log_info "备份验证通过!"
return 0
}
start_node() {
log_info "启动 msgchaind 服务..."
systemctl start msgchaind
for i in $(seq 1 30); do
if curl -sf localhost:26657/health > /dev/null 2>&1; then
log_info "节点正常运行!"
return 0
fi
sleep 2
done
log_error "节点启动超时"
return 1
}
cleanup_old_backups() {
log_info "清理超过 ${RETENTION_DAYS} 天的旧备份..."
find "$BACKUP_BASE/full" -maxdepth 1 -type d -mtime +${RETENTION_DAYS} \
-exec rm -rf {} \; -print 2>/dev/null
}
main() {
log_info "========== 开始全量冷备份 =========="
check_prerequisites
perform_cold_backup
verify_backup
start_node
cleanup_old_backups
log_info "========== 备份完成 =========="
}
main "$@"
3.3 热备份 (节点运行中)
#!/bin/bash
# 脚本: hot_backup_badger.sh
# 功能: MSG Chain 热备份 (节点运行时)
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
BACKUP_DIR="/backup/msgchain/hot"
DATE_STAMP=$(date +%Y%m%d_%H%M%S)
BAK_DIR="${BACKUP_DIR}/${DATE_STAMP}"
LOG_FILE="/var/log/msgchain/hot_backup.log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
backup_leveldb() {
log_info "备份 LevelDB 数据库..."
mkdir -p "$BAK_DIR/leveldb"
for db in evidence.db tx_index.db blockstore.db state.db; do
if [ -d "${NODE_HOME}/data/${db}" ]; then
cp -r "${NODE_HOME}/data/${db}" "$BAK_DIR/leveldb/"
fi
done
log_info "LevelDB 备份完成"
}
backup_config_and_keys() {
log_info "备份配置和密钥..."
mkdir -p "$BAK_DIR/config"
cp "${NODE_HOME}/config/"*.toml "$BAK_DIR/config/" 2>/dev/null || true
cp "${NODE_HOME}/config/genesis.json" "$BAK_DIR/config/" 2>/dev/null || true
log_info "配置备份完成"
}
compress_backup() {
log_info "压缩热备份..."
local archive="${BAK_DIR}.tar.zst"
tar -I "zstd -T$(nproc)" -cf "$archive" -C "$BACKUP_DIR" "$DATE_STAMP"
sha256sum "$archive" > "${archive}.sha256"
rm -rf "$BAK_DIR"
log_info "压缩完成: $(ls -lh $archive)"
}
verify_compressed_backup() {
log_info "验证压缩备份..."
local archive="${BAK_DIR}.tar.zst"
cd "$BACKUP_DIR"
sha256sum -c "${archive}.sha256" || { log_error "校验失败"; return 1; }
tar -I zstd -tf "$archive" > /dev/null 2>&1 || { log_error "格式错误"; return 1; }
log_info "验证通过"
return 0
}
main() {
log_info "========== 开始热备份 =========="
backup_leveldb
backup_config_and_keys
compress_backup
verify_compressed_backup
log_info "========== 热备份完成 =========="
}
main "$@"
3.4 增量备份
#!/bin/bash
# 脚本: incremental_backup.sh
# 功能: MSG Chain 增量备份 (使用 rsync)
# 调度: cron 每 4 小时
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
BACKUP_BASE="/backup/msgchain/incremental"
SNAPSHOT_DIR="${BACKUP_BASE}/snapshot"
LAST_SNAPSHOT="${BACKUP_BASE}/last_snapshot"
CURRENT_BACKUP="${BACKUP_BASE}/current"
DATE_STAMP=$(date +%Y%m%d_%H%M%S)
LOG_FILE="/var/log/msgchain/incremental_backup.log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
mkdir -p "$BACKUP_BASE" "$SNAPSHOT_DIR"
CURRENT_HEIGHT=$(curl -sf localhost:26657/status \
| jq -r '.result.sync_info.latest_block_height' 2>/dev/null || echo "0")
log_info "开始增量备份 (高度: ${CURRENT_HEIGHT})..."
rsync_common=(
-avz --delete --progress
--link-dest="$LAST_SNAPSHOT"
--exclude="snapshots/*"
--exclude="*.wal"
--exclude="LOCK"
)
rsync "${rsync_common[@]}" \
"${NODE_HOME}/data/" \
"${CURRENT_BACKUP}/"
rm -f "$LAST_SNAPSHOT"
ln -s "$CURRENT_BACKUP" "$LAST_SNAPSHOT"
echo "$DATE_STAMP" > "${CURRENT_BACKUP}/.backup_timestamp"
echo "$CURRENT_HEIGHT" > "${CURRENT_BACKUP}/.backup_height"
BACKUP_SIZE=$(du -sh "$CURRENT_BACKUP" 2>/dev/null | cut -f1)
log_info "增量备份完成! 大小: ${BACKUP_SIZE}"
3.5 BadgerDB 数据库级修复
#!/bin/bash
# 脚本: badger_repair.sh
# 功能: 修复损坏的 BadgerDB 数据库
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
BADGER_DIR="${NODE_HOME}/data/badger"
BACKUP_DIR="${NODE_HOME}/data_backup_$(date +%Y%m%d_%H%M%S)"
LOG_FILE="/var/log/msgchain/badger_repair.log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
if [ ! -d "$BADGER_DIR" ]; then
log_error "BadgerDB 目录不存在: ${BADGER_DIR}"
exit 1
fi
log_info "备份当前数据到 ${BACKUP_DIR}..."
cp -r "$BADGER_DIR" "$BACKUP_DIR"
log_info "尝试 msgchaind 内置修复..."
msgchaind debug badger-repair --data-dir "$BADGER_DIR" 2>&1 | tee -a "$LOG_FILE" || true
if command -v badger &>/dev/null; then
log_info "运行 badger 工具修复..."
badger repair --dir "$BADGER_DIR" 2>&1 | tee -a "$LOG_FILE" || true
fi
log_info "检查 Value Log 文件..."
VLOG_FILES=$(find "$BADGER_DIR" -name "*.vlog" 2>/dev/null)
if [ -n "$VLOG_FILES" ]; then
for vlog in $VLOG_FILES; do
log_info "文件: $vlog ($(ls -lh "$vlog" | awk '{print $5}'))"
done
fi
log_info "检查 SST 文件..."
SST_FILES=$(find "$BADGER_DIR" -name "*.sst" 2>/dev/null)
SST_COUNT=$(echo "$SST_FILES" | wc -l)
log_info "找到 ${SST_COUNT} 个 SST 文件"
log_info "尝试验证数据库完整性..."
msgchaind debug badger-verify --data-dir "$BADGER_DIR" 2>&1 | tee -a "$LOG_FILE" || {
log_error "数据库验证失败，建议从备份恢复"
exit 1
}
log_info "数据库修复完成!"
3.6 备份验证
#!/bin/bash
# 脚本: verify_backup.sh
# 功能: 验证 MSG Chain 备份完整性
set -euo pipefail
BACKUP_PATH="${1:-}"
VERIFY_DIR="/tmp/msgchain-verify-$(date +%s)"
LOG_FILE="/var/log/msgchain/backup_verify.log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
if [ -z "$BACKUP_PATH" ]; then
log_error "用法: $0 <backup_path>"
exit 1
fi
BACKUP_SIZE=$(du -h "$BACKUP_PATH" | cut -f1)
log_info "备份文件: ${BACKUP_PATH} (${BACKUP_SIZE})"
# 校验和验证
SHA_FILE="${BACKUP_PATH}.sha256"
if [ -f "$SHA_FILE" ]; then
log_info "验证 SHA256 校验和..."
cd "$(dirname "$BACKUP_PATH")"
sha256sum -c "$SHA_FILE" || { log_error "校验失败"; exit 1; }
cd - > /dev/null
fi
# 格式验证
log_info "验证文件格式..."
case "$BACKUP_PATH" in
*.tar.zst) tar -I zstd -tf "$BACKUP_PATH" > /dev/null 2>&1 || { log_error "ZSTD验证失败"; exit 1; } ;;
*.tar.lz4) tar -I lz4 -tf "$BACKUP_PATH" > /dev/null 2>&1 || { log_error "LZ4验证失败"; exit 1; } ;;
*.tar.gz) tar -tzf "$BACKUP_PATH" > /dev/null 2>&1 || { log_error "GZIP验证失败"; exit 1; } ;;
esac
# 内容抽查
mkdir -p "$VERIFY_DIR"
case "$BACKUP_PATH" in
*.tar.zst) tar -I zstd -xf "$BACKUP_PATH" -C "$VERIFY_DIR" ;;esac
ls -la "$VERIFY_DIR"
for dir in "data/badger" "data/blockstore.db" "data/state.db"; do
[ -d "${VERIFY_DIR}/${dir}" ] && log_info "✓ $dir" || log_info "? $dir"
done
rm -rf "$VERIFY_DIR"
log_info "========== 验证完成 =========="
3.7 异地备份 (Offsite)
#!/bin/bash
# 脚本: offsite_backup_sync.sh
# 功能: 将本地备份同步到异地存储
set -euo pipefail
BACKUP_SOURCE="${1:-/backup/msgchain}"
LOG_FILE="/var/log/msgchain/offsite_sync.log"
SYNC_METHOD="${SYNC_METHOD:-s3}"
S3_BUCKET="${S3_BUCKET:-msgchain-backup}"
S3_PREFIX="${S3_PREFIX:-backups/}"
SFTP_HOST="${SFTP_HOST:-backup.msgchain.org}"
SFTP_USER="${SFTP_USER:-backup}"
SFTP_PATH="${SFTP_PATH:-/backup/msgchain/}"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
case "$SYNC_METHOD" in
s3)
aws s3 sync "$BACKUP_SOURCE" "s3://${S3_BUCKET}/${S3_PREFIX}" \
--storage-class STANDARD_IA --delete
;;
sftp)
rsync -avz --delete -e "ssh -p ${SFTP_PORT:-22}" \
"$BACKUP_SOURCE/" "${SFTP_USER}@${SFTP_HOST}:${SFTP_PATH}"
;;
nfs)
rsync -avz --delete "$BACKUP_SOURCE/" "${SFTP_PATH}/"
;;
esac
log_info "异地备份同步完成"
3.8 自动化调度 (Crontab)
# Crontab 配置
# 编辑: crontab -e
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOME=/home/msgchain
# 每日 02:00 全量冷备份
0 2 * * * /usr/local/bin/full_backup_cold.sh >> /var/log/msgchain/backup_cron.log 2>&1
# 每 4 小时增量备份
0 */4 * * * /usr/local/bin/incremental_backup.sh >> /var/log/msgchain/incremental_cron.log 2>&1
# 每小时热备份
0 * * * * /usr/local/bin/hot_backup_badger.sh >> /var/log/msgchain/hot_cron.log 2>&1
# 每天 04:00 同步到异地
0 4 * * * /usr/local/bin/offsite_backup_sync.sh >> /var/log/msgchain/offsite_cron.log 2>&1
# 每天 06:00 验证备份
30 6 * * * /usr/local/bin/verify_backup.sh $(ls -t /backup/msgchain/full/*/data.tar.zst | head -1) >> /var/log/msgchain/verify_cron.log 2>&1
# 每周日 08:00 清理旧备份
0 8 * * 0 /usr/local/bin/cleanup_backups.sh >> /var/log/msgchain/cleanup_cron.log 2>&1
3.9 备份清理脚本
#!/bin/bash
# 脚本: cleanup_backups.sh
set -euo pipefail
BACKUP_BASE="/backup/msgchain"
LOG_FILE="/var/log/msgchain/cleanup.log"
FULL_RETENTION_DAYS=30
INCREMENTAL_RETENTION_DAYS=7
HOT_RETENTION_DAYS=3
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_info "清理超过 ${FULL_RETENTION_DAYS} 天的全量备份..."
find "$BACKUP_BASE/full" -maxdepth 1 -type d -mtime +${FULL_RETENTION_DAYS} -exec rm -rf {} \; -print
log_info "清理超过 ${HOT_RETENTION_DAYS} 天的热备份..."
find "$BACKUP_BASE/hot" -name "*.tar.zst" -mtime +${HOT_RETENTION_DAYS} -delete -print
log_info "磁盘使用:"
du -sh "$BACKUP_BASE" 2>/dev/null
log_info "清理完成"
4. 灾难恢复流程
4.1 灾难恢复决策树
节点故障
│
├── 数据目录是否存在？
│ ├── 否 → 从快照恢复
│ └── 是
│ ↓
├── 节点能否启动？
│ ├── 是 → 检查同步状态 → 正常 / 快速同步
│ └── 否
│ ↓
├── 数据库损坏？
│ ├── 否 → 检查配置、权限、磁盘
│ └── 是 → 尝试 BadgerDB 修复
│ ├── 成功 → 重启节点
│ └── 失败
│ ↓
├── 有可用备份？
│ ├── 否 → 从快照恢复
│ └── 是 → 全量恢复
└──
4.2 从备份完整恢复
#!/bin/bash
# 脚本: full_recovery_from_backup.sh
# 用法: ./full_recovery_from_backup.sh <backup_date>
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
BACKUP_BASE="/backup/msgchain/full"
BACKUP_DATE="${1:-}"
LOG_FILE="/var/log/msgchain/recovery_$(date +%Y%m%d_%H%M%S).log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
log_warn() { echo "[WARN] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
if [ -z "$BACKUP_DATE" ]; then
log_error "用法: $0 <backup_date>"
ls "$BACKUP_BASE" 2>/dev/null || echo "无可用备份"
exit 1
fi
BACKUP_DIR="${BACKUP_BASE}/${BACKUP_DATE}"
[ -d "$BACKUP_DIR" ] || { log_error "备份目录不存在"; exit 1; }
log_info "========== MSG Chain 灾难恢复 =========="
# 阶段 1: 停止节点
log_info "停止 msgchaind 服务..."
systemctl stop msgchaind || true
sleep 5
pkill -9 -f msgchaind 2>/dev/null || true
sleep 3
log_info "节点已停止"
# 阶段 2: 备份当前数据 (回滚点)
CURRENT_BACKUP="${NODE_HOME}/data_rollback_$(date +%Y%m%d_%H%M%S)"
if [ -d "${NODE_HOME}/data" ]; then
log_info "备份当前数据到 ${CURRENT_BACKUP}..."
mv "${NODE_HOME}/data" "$CURRENT_BACKUP"
log_info "回滚点: ${CURRENT_BACKUP}"
fi
# 阶段 3: 验证备份
log_info "验证备份完整性..."
cd "$BACKUP_DIR"
[ -f checksums.sha256 ] && sha256sum -c checksums.sha256 || log_warn "无校验和"
cd - > /dev/null
for f in "$BACKUP_DIR"/*.tar.zst; do
tar -I zstd -tf "$f" &>/dev/null || { log_error "损坏: $f"; exit 1; }
done
log_info "所有备份文件验证通过"
# 阶段 4: 恢复数据
log_info "恢复数据目录..."
[ -f "${BACKUP_DIR}/data.tar.zst" ] && \
tar -I zstd -xf "${BACKUP_DIR}/data.tar.zst" -C "$NODE_HOME"
log_info "数据目录恢复完成"
[ -f "${BACKUP_DIR}/config.tar.zst" ] && \
tar -I zstd -xf "${BACKUP_DIR}/config.tar.zst" -C "$NODE_HOME"
log_info "配置目录恢复完成"
# 阶段 5: 重置 CometBFT
log_info "重置 CometBFT 状态..."
msgchaind tendermint unsafe-reset-all --keep-addr-book || msgchaind tendermint unsafe-reset-all
log_info "CometBFT 已重置"
# 阶段 6: 验证
log_info "验证恢复..."
msgchaind validate-genesis || { log_error "创世文件失败"; exit 1; }
[ -d "${NODE_HOME}/data" ] || { log_error "数据目录不存在"; exit 1; }
# 阶段 7: 启动
log_info "启动节点..."
systemctl start msgchaind
sleep 10
for i in $(seq 1 60); do
if curl -sf localhost:26657/health > /dev/null 2>&1; then
log_info "节点已响应!"
break
fi
[ "$i" -eq 60 ] && { log_error "启动超时"; exit 1; }
sleep 5
done
# 阶段 8: 监控同步
log_info "监控同步状态..."
for i in $(seq 1 12); do
sleep 10
STATUS=$(curl -sf localhost:26657/status 2>/dev/null || echo '{}')
HEIGHT=$(echo "$STATUS" | jq -r '.result.sync_info.latest_block_height // "N/A"')
CATCHING_UP=$(echo "$STATUS" | jq -r '.result.sync_info.catching_up // "N/A"')
PEERS=$(curl -sf localhost:26657/net_info 2>/dev/null | jq -r '.result.n_peers // "0"')
log_info "高度: ${HEIGHT}, 追赶上: ${CATCHING_UP}, 对等节点: ${PEERS}"
[ "$CATCHING_UP" = "false" ] && { log_info "已完全同步!"; break; }
done
log_info "========== 恢复流程完成 =========="
log_info "检查: curl -s localhost:26657/status | jq ."
log_info "检查: journalctl -u msgchaind -n 50"
4.3 从快照恢复
#!/bin/bash
# 脚本: snapshot_recovery.sh
# 功能: 从快照恢复 MSG Chain 节点
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
SNAPSHOT_BASE_URL="https://snapshots.msgchain.org"
TEMP_DIR="/tmp/msgchain-snapshot-recovery-$(date +%s)"
LOG_FILE="/var/log/msgchain/snapshot_recovery_$(date +%Y%m%d_%H%M%S).log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
cleanup() { rm -rf "$TEMP_DIR"; }
trap cleanup EXIT
get_latest_snapshot() {
log_info "获取最新快照信息..."
LATEST_INFO=$(curl -sf "${SNAPSHOT_BASE_URL}/latest.json" 2>/dev/null || echo '{}')
SNAPSHOT_URL=$(echo "$LATEST_INFO" | jq -r '.url // ""')
[ -n "$SNAPSHOT_URL" ] && { echo "$SNAPSHOT_URL"; return; }
echo "${SNAPSHOT_BASE_URL}/latest.tar.lz4"
}
perform_snapshot_restore() {
local snapshot_url="$1"
log_info "快照 URL: ${snapshot_url}"
mkdir -p "$TEMP_DIR"
log_info "停止节点..."
systemctl stop msgchaind || true
pkill -9 -f msgchaind 2>/dev/null || true
sleep 3
if [ -d "${NODE_HOME}/data" ]; then
OLD_BACKUP="${NODE_HOME}/data_old_$(date +%Y%m%d_%H%M%S)"
mv "${NODE_HOME}/data" "$OLD_BACKUP"
fi
mkdir -p "${NODE_HOME}/data"
log_info "下载快照..."
SNAPSHOT_FILE="${TEMP_DIR}/snapshot.tar.lz4"
wget -O "$SNAPSHOT_FILE" "$snapshot_url" || { log_error "下载失败"; exit 1; }
SHA_URL="${snapshot_url}.sha256"
if wget -O "${TEMP_DIR}/snapshot.sha256" "$SHA_URL" 2>/dev/null; then
cd "$TEMP_DIR" && sha256sum -c snapshot.sha256 || { log_error "校验失败"; exit 1; }
cd - > /dev/null
fi
log_info "解压快照..."
if command -v pv &>/dev/null; then
pv "$SNAPSHOT_FILE" | tar -I lz4 -xf - -C "${NODE_HOME}/data/"
else
tar -I lz4 -xf "$SNAPSHOT_FILE" -C "${NODE_HOME}/data/"
fi
msgchaind tendermint unsafe-reset-all --keep-addr-book || true
msgchaind validate-genesis || { log_error "创世文件失败"; exit 1; }
systemctl start msgchaind
sleep 10
for i in $(seq 1 6); do
sleep 10
STATUS=$(curl -sf localhost:26657/status 2>/dev/null || echo '{}')
HEIGHT=$(echo "$STATUS" | jq -r '.result.sync_info.latest_block_height // "N/A"')
CATCHING_UP=$(echo "$STATUS" | jq -r '.result.sync_info.catching_up // "N/A"')
log_info "高度: ${HEIGHT}, 追赶上: ${CATCHING_UP}"
done
log_info "========== 快照恢复完成 =========="
}
main() {
local snapshot_url="${1:-$(get_latest_snapshot)}"
perform_snapshot_restore "$snapshot_url"
}
main "$@"
4.4 从对等节点恢复 (状态同步)
#!/bin/bash
# 脚本: p2p_state_sync_recovery.sh
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
LOG_FILE="/var/log/msgchain/p2p_sync_$(date +%Y%m%d_%H%M%S).log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
TRUSTED_RPCS=("https://rpc.msgchain.org:443" "https://rpc2.msgchain.org:443")
DISCOVERY_PEERS=("peer1@peer1.msgchain.org:26656" "peer2@peer2.msgchain.org:26656")
get_trust_params() {
heights=()
for rpc in "${TRUSTED_RPCS[@]}"; do
h=$(curl -sf "$rpc/status" 2>/dev/null | jq -r '.result.sync_info.latest_block_height // "0"')
heights+=("$h")
done
sorted=$(printf '%s\n' "${heights[@]}" | sort -n)
lowest=$(echo "$sorted" | head -1)
trust_height=$((lowest - 1000))
block_info=$(curl -sf "${TRUSTED_RPCS[0]}/block?height=${trust_height}" 2>/dev/null)
trust_hash=$(echo "$block_info" | jq -r '.result.block_id.hash')
echo "$trust_height $trust_hash"
}
configure_state_sync() {
local height="$1" hash="$2"
rpc_list=$(printf ",%s" "${TRUSTED_RPCS[@]}")
rpc_list="${rpc_list:1}"
sed -i.bak \
-e "s/^enable = .*/enable = true/" \
-e "s|^rpc_servers = .*|rpc_servers = \"${rpc_list}\"|" \
-e "s/^trust_height = .*/trust_height = ${height}/" \
-e "s/^trust_hash = .*/trust_hash = \"${hash}\"/" \
"${NODE_HOME}/config/config.toml"
}
perform_state_sync() {
log_info "========== P2P 状态同步恢复 =========="
systemctl stop msgchaind || true
sleep 3
if [ -d "${NODE_HOME}/data" ]; then
old_backup="${NODE_HOME}/data_old_$(date +%Y%m%d_%H%M%S)"
mv "${NODE_HOME}/data" "$old_backup"
fi
mkdir -p "${NODE_HOME}/data"
read -r height hash <<< "$(get_trust_params)"
log_info "信任高度: ${height}, 哈希: ${hash}"
configure_state_sync "$height" "$hash"
peer_list=$(printf ",%s" "${DISCOVERY_PEERS[@]}")
peer_list="${peer_list:1}"
sed -i "s/^persistent_peers = .*/persistent_peers = \"${peer_list}\"/" \
"${NODE_HOME}/config/config.toml"
msgchaind tendermint unsafe-reset-all
systemctl start msgchaind
sleep 15
for i in $(seq 1 30); do
sleep 10
STATUS=$(curl -sf localhost:26657/status 2>/dev/null || echo '{}')
HEIGHT=$(echo "$STATUS" | jq -r '.result.sync_info.latest_block_height // "N/A"')
CATCHING_UP=$(echo "$STATUS" | jq -r '.result.sync_info.catching_up // "N/A"')
PEERS=$(curl -sf localhost:26657/net_info 2>/dev/null | jq -r '.result.n_peers // "0"')
log_info "高度: ${HEIGHT} | 追赶上: ${CATCHING_UP} | 对等节点: ${PEERS}"
[ "$CATCHING_UP" = "false" ] && { log_info "同步完成!"; break; }
done
log_info "========== P2P 同步完成 =========="
}
perform_state_sync
4.5 验证者密钥恢复
#!/bin/bash
# 脚本: validator_key_recovery.sh
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
PRIV_VAL_KEY="${NODE_HOME}/config/priv_validator_key.json"
STATE_FILE="${NODE_HOME}/config/priv_validator_state.json"
LOG_FILE="/var/log/msgchain/key_recovery_$(date +%Y%m%d_%H%M%S).log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
check_existing_keys() {
if [ -f "$PRIV_VAL_KEY" ]; then
log_info "验证者密钥文件存在"
log_info "公钥: $(jq -r '.pub_key.value' "$PRIV_VAL_KEY" 2>/dev/null || echo '无法读取')"
msgchaind tendermint show-validator &>/dev/null && log_info "✓ 密钥格式正确" || log_error "密钥损坏"
else
log_info "验证者密钥文件不存在"
return 1
fi
}
restore_key_from_backup() {
local backup_source="${1:-}"
[ -z "$backup_source" ] && { log_error "请指定备份来源"; return 1; }
log_info "从备份恢复验证者密钥: ${backup_source}"
case "$backup_source" in
*.tar.zst)
local tmp="/tmp/key_restore_$(date +%s)"
mkdir -p "$tmp"
tar -I zstd -xf "$backup_source" -C "$tmp"
local kf=$(find "$tmp" -name "priv_validator_key.json" 2>/dev/null | head -1)
[ -n "$kf" ] && { cp "$kf" "$PRIV_VAL_KEY"; rm -rf "$tmp"; log_info "密钥已恢复"; return 0; }
rm -rf "$tmp"; log_error "未找到密钥文件"; return 1
;;
*)
[ -f "$backup_source/priv_validator_key.json" ] && { cp "$backup_source/priv_validator_key.json" "$PRIV_VAL_KEY"; return 0; }
log_error "无效来源"; return 1
;;
esac
}
verify_recovered_key() {
log_info "验证恢复的密钥..."
val_addr=$(msgchaind tendermint show-address 2>/dev/null || echo "N/A")
val_pubkey=$(msgchaind tendermint show-validator 2>/dev/null || echo "N/A")
log_info "验证者地址: ${val_addr}"
log_info "验证者公钥: ${val_pubkey}"
if curl -sf localhost:26657/status &>/dev/null; then
chain_val=$(curl -sf "localhost:26657/validators?height=0" 2>/dev/null \
| jq -r '.result.validators[] | select(.address == "'"$val_addr"'") | .address' 2>/dev/null || echo "")
[ -n "$chain_val" ] && log_info "✓ 验证者在链上验证者集合中" || log_warn "验证者不在当前集合中"
fi
}
reset_validator_state() {
log_info "重置验证者状态..."
cat > "$STATE_FILE" << EOF
{ "height": "0", "round": 0, "step": 0, "signature": "", "signbytes": "" }
EOF
log_info "状态已重置"
}
show_menu() {
echo ""
echo "MSG Chain 验证者密钥恢复工具"
echo "1. 检查现有密钥"
echo "2. 从备份恢复密钥"
echo "3. 重置验证者状态"
echo "4. 验证密钥"
echo "q. 退出"
echo -n "请选择: "
}
main() {
log_info "========== 验证者密钥恢复 =========="
while true; do
show_menu; read -r choice
case "$choice" in
1) check_existing_keys ;;
2) echo -n "备份路径: "; read -r bp; restore_key_from_backup "$bp" ;;
3) echo -n "确认? (yes/no): "; read -r c; [ "$c" = "yes" ] && reset_validator_state ;;
4) verify_recovered_key ;;
q|Q) exit 0 ;;
esac
done
}
main "$@"
4.6 哨兵节点恢复
#!/bin/bash
# 脚本: sentry_recovery.sh
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
SENTRY_CONFIG="${NODE_HOME}/config/config.toml"
LOG_FILE="/var/log/msgchain/sentry_recovery_$(date +%Y%m%d_%H%M%S).log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
recover_sentry() {
log_info "========== 哨兵节点恢复 =========="
if [ ! -f "${NODE_HOME}/config/node_key.json" ]; then
log_info "初始化新节点..."
msgchaind init "sentry-$(hostname)" --chain-id msg-chain-1
curl -fsSL https://github.com/msgchain/mainnet/raw/main/genesis.json \
-o "${NODE_HOME}/config/genesis.json"
fi
log_info "配置哨兵模式..."
VALIDATOR_PEER_IDS="${1:-}"
sed -i \
-e "s/^pex = .*/pex = true/" \
-e "s/^persistent_peers = .*/persistent_peers = \"${VALIDATOR_PEER_IDS}\"/" \
-e "s/^private_peer_ids = .*/private_peer_ids = \"${VALIDATOR_PEER_IDS}\"/" \
-e "s/^max_num_inbound_peers = .*/max_num_inbound_peers = 100/" \
-e "s/^max_num_outbound_peers = .*/max_num_outbound_peers = 50/" \
"$SENTRY_CONFIG"
log_info "哨兵节点恢复完成"
}
recover_sentry "$@"
5. 高可用架构
5.1 架构设计原则
- 冗余: 消除单点故障
- 隔离: 验证者节点与哨兵节点分离
- 自动故障转移: 无需人工干预
- 数据一致性: 即使发生故障也保持一致性
- 可观测性: 全链路监控
5.2 多节点集群架构
┌───────────────────┐
│ 负载均衡器 │
│ HAProxy / Nginx │
└────────┬──────────┘
│
┌─────────────────────────┼─────────────────────────┐
v v v
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ 哨兵节点 1 │ │ 哨兵节点 2 │ │ 哨兵节点 3 │
│ P2P:26656 │ │ P2P:26656 │ │ P2P:26656 │
│ RPC:26657 │ │ RPC:26657 │ │ RPC:26657 │
└────────┬───────┘ └────────┬───────┘ └────────┬───────┘
│ │ │
│ ╔════════════╪════════════╗ │
╞═══════╣ 私有网络 │ ╟═══════╡
│ ║ │ ║ │
│ ║ v ║ │
│ ║ ┌─────────────────┐ ║ │
└───────╢ │ 验证者节点 │ ╟───────┘
║ │ 共识参与 │ ║
║ │ 密钥管理 │ ║
║ └─────────────────┘ ║
╚═══════════════════════╝
5.3 哨兵节点拓扑配置
5.3.1 验证者节点
# ~/.msgchain/config/config.toml (验证者)
pex = false
persistent_peers = "sentry1-id@sentry1.internal:26656,sentry2-id@sentry2.internal:26656,sentry3-id@sentry3.internal:26656"
unconditional_peer_ids = "sentry1-id,sentry2-id,sentry3-id"
addr_book_strict = false
private_peer_ids = "*"
seed_mode = false
5.3.2 哨兵节点
# ~/.msgchain/config/config.toml (哨兵)
pex = true
seeds = "seed1.msgchain.org:26656,seed2.msgchain.org:26656"
persistent_peers = "validator-node-id@validator.internal:26656"
private_peer_ids = "validator-node-id"
unconditional_peer_ids = "validator-node-id"
max_num_inbound_peers = 200
max_num_outbound_peers = 50
5.4 负载均衡器配置
5.4.1 HAProxy
# /etc/haproxy/haproxy.cfg
global
log /dev/log local0
maxconn 10000
daemon
defaults
mode tcp
option tcplog
timeout connect 5s
timeout client 30s
timeout server 30s
retries 3
frontend rpc_frontend
bind *:26657
mode tcp
default_backend rpc_backend
backend rpc_backend
mode tcp
balance leastconn
option tcp-check
tcp-check connect
server sentry1 sentry1.internal:26657 check inter 5s fall 3 rise 2
server sentry2 sentry2.internal:26657 check inter 5s fall 3 rise 2
server sentry3 sentry3.internal:26657 check inter 5s fall 3 rise 2
frontend p2p_frontend
bind *:26656
mode tcp
default_backend p2p_backend
backend p2p_backend
mode tcp
balance source
server sentry1 sentry1.internal:26656 check inter 5s fall 3 rise 2
server sentry2 sentry2.internal:26656 check inter 5s fall 3 rise 2
server sentry3 sentry3.internal:26656 check inter 5s fall 3 rise 2
frontend grpc_frontend
bind *:9090
mode tcp
default_backend grpc_backend
backend grpc_backend
mode tcp
balance leastconn
server sentry1 sentry1.internal:9090 check inter 5s fall 3 rise 2
server sentry2 sentry2.internal:9090 check inter 5s fall 3 rise 2
server sentry3 sentry3.internal:9090 check inter 5s fall 3 rise 2
5.4.2 Nginx 替代配置
# /etc/nginx/conf.d/msgchain.conf
upstream msgchain_rpc {
least_conn;
server sentry1.internal:26657 max_fails=3 fail_timeout=30s;
server sentry2.internal:26657 max_fails=3 fail_timeout=30s;
server sentry3.internal:26657 max_fails=3 fail_timeout=30s;
keepalive 32;
}
server {
listen 26657;
proxy_pass msgchain_rpc;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
location /websocket {
proxy_pass http://msgchain_rpc;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 86400s;
}
}
5.5 自动故障转移
#!/bin/bash
# 脚本: auto_failover.sh
# 功能: MSG Chain 哨兵节点自动故障转移
set -euo pipefail
SENTRY_NODES=("sentry1.internal:26657" "sentry2.internal:26657" "sentry3.internal:26657")
VALIDATOR_NODE="validator.internal:26657"
CHECK_INTERVAL=10
FAIL_THRESHOLD=3
STATUS_FILE="/tmp/msgchain_sentry_status.json"
LOG_FILE="/var/log/msgchain/failover.log"
NOTIFY_SCRIPT="/usr/local/bin/notify_alert.sh"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
check_node_health() {
local node="$1"
local response=$(curl -sf "http://${node}/status" 2>/dev/null || echo '{}')
local height=$(echo "$response" | jq -r '.result.sync_info.latest_block_height // "0"')
local catching_up=$(echo "$response" | jq -r '.result.sync_info.catching_up // "true"')
if [ "$height" != "0" ] && [ "$catching_up" = "false" ]; then echo "healthy"
elif [ "$height" != "0" ]; then echo "syncing"
else echo "unhealthy"
fi
}
perform_failover() {
local failed_node="$1"
log_info "执行故障转移: ${failed_node}"
if [ -S /run/haproxy/admin.sock ]; then
local node_name=$(echo "$failed_node" | cut -d: -f1)
echo "disable server rpc_backend/${node_name}" | socat stdio /run/haproxy/admin.sock || true
echo "disable server p2p_backend/${node_name}" | socat stdio /run/haproxy/admin.sock || true
log_info "已禁用 HAProxy 中的节点: ${node_name}"
fi
}
notify_failure() {
local subject="$1"
[ -x "$NOTIFY_SCRIPT" ] && $NOTIFY_SCRIPT "MSG Chain 故障: ${subject}" "时间: $(date)"
}
main_loop() {
log_info "========== 故障转移守护进程启动 =========="
while true; do
for node in "${SENTRY_NODES[@]}"; do
health=$(check_node_health "$node")
[ "$health" = "unhealthy" ] && { perform_failover "$node"; notify_failure "$node"; }
done
sleep "$CHECK_INTERVAL"
done
}
main_loop "$@"
5.6 Systemd 服务配置
# /etc/systemd/system/msgchaind.service
[Unit]
Description=MSG Chain Node
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=msgchain
Group=msgchain
WorkingDirectory=/home/msgchain
ExecStart=/usr/local/bin/msgchaind start --home /home/msgchain/.msgchain --trace
Restart=always
RestartSec=10
StartLimitBurst=5
LimitNOFILE=65536
LimitNPROC=65536
MemoryHigh=32G
MemoryMax=40G
CPUQuota=400%
NoNewPrivileges=true
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=true
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
5.7 DRBD 块级复制
# /etc/drbd.d/msgchain_data.res
resource msgchain_data {
protocol C;
on node1.internal {
device /dev/drbd0;
disk /dev/sdb1;
address 10.0.1.1:7788;
meta-disk internal;
}
on node2.internal {
device /dev/drbd0;
disk /dev/sdb1;
address 10.0.1.2:7788;
meta-disk internal;
}
syncer { rate 1000M; }
}
# 初始化
drbdadm create-md msgchain_data
drbdadm up msgchain_data
drbdadm primary --force msgchain_data
mkfs.ext4 /dev/drbd0
mount /dev/drbd0 /home/msgchain/.msgchain/data/
6. 监控与告警
6.1 监控指标体系
| 类别 | 指标 | 告警阈值 |
|---|---|---|
| 同步状态 | catching_up |
true 超过 1 小时 |
| 区块高度 | latest_block_height |
落后超过 100 区块 |
| 对等节点 | n_peers |
少于 5 个 |
| 磁盘使用 | disk_usage_percent |
> 80% 警告, > 90% 严重 |
| 内存使用 | memory_usage_bytes |
> 32GB |
| CPU 使用 | cpu_percent |
> 80% 持续 10 分钟 |
| 共识投票 | missed_blocks |
> 10 连续错过 |
| 磁盘 IO | `disk_reads | writes` |
6.2 Prometheus 配置
# /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets:
- 'localhost:9093'
rule_files:
- '/etc/prometheus/rules/*.yml'
scrape_configs:
- job_name: 'msgchain-node'
scrape_interval: 10s
static_configs:
- targets:
- 'node1.internal:26660'
- 'node2.internal:26660'
- 'node3.internal:26660'
metrics_path: '/metrics'
- job_name: 'msgchain-validator'
scrape_interval: 10s
static_configs:
- targets:
- 'validator.internal:26660'
metrics_path: '/metrics'
- job_name: 'node-exporter'
static_configs:
- targets:
- 'node1.internal:9100'
- 'node2.internal:9100'
- 'node3.internal:9100'
- 'validator.internal:9100'
6.3 告警规则
# /etc/prometheus/rules/msgchain_alerts.yml
groups:
- name: msgchain_node
interval: 30s
rules:
- alert: NodeDown
expr: up{job="msgchain-node"} == 0
for: 1m
labels: { severity: critical }
annotations:
summary: "MSG Chain 节点 {{ $labels.instance }} 停机"
description: "{{ $labels.instance }} 已宕机超过 1 分钟"
- alert: NodeCatchingUp
expr: msgchain_catching_up{job="msgchain-node"} == 1
for: 30m
labels: { severity: warning }
annotations:
summary: "节点 {{ $labels.instance }} 正在追赶同步"
description: "节点已同步超过 30 分钟仍未完成"
- alert: NodeHeightDrift
expr: abs(msgchain_latest_block_height{job="msgchain-node"} - msgchain_latest_block_height{job="msgchain-validator"}) > 100
for: 5m
labels: { severity: warning }
annotations:
summary: "节点高度偏差超过 100"
- alert: LowPeers
expr: msgchain_peers{job="msgchain-node"} < 5
for: 5m
labels: { severity: warning }
annotations:
summary: "对等节点数少于 5"
description: "当前对等节点数: {{ $value }}"
- alert: DiskSpaceWarning
expr: (node_filesystem_avail_bytes{mountpoint="/data"} / node_filesystem_size_bytes{mountpoint="/data"}) * 100 < 20
for: 10m
labels: { severity: warning }
annotations:
summary: "磁盘空间不足 20%"
description: "剩余: {{ $value | humanizePercentage }}"
- alert: DiskSpaceCritical
expr: (node_filesystem_avail_bytes{mountpoint="/data"} / node_filesystem_size_bytes{mountpoint="/data"}) * 100 < 10
for: 5m
labels: { severity: critical }
annotations:
summary: "磁盘空间严重不足"
description: "剩余: {{ $value | humanizePercentage }}"
- alert: BackupFreshness
expr: time() - max(msgchain_backup_timestamp{type="full"}) > 86400
for: 1h
labels: { severity: warning }
annotations:
summary: "备份超过 24 小时未更新"
description: "最后一次全量备份: {{ $value | humanizeDuration }} 前"
- name: msgchain_validator
interval: 30s
rules:
- alert: ValidatorMissedBlocks
expr: increase(msgchain_missed_blocks[10m]) > 10
for: 1m
labels: { severity: critical }
annotations:
summary: "验证者错过过多区块"
description: "过去 10 分钟错过 {{ $value }} 个区块"
- alert: ValidatorNotSigning
expr: msgchain_signed_blocks_total{job="msgchain-validator"} == 0
for: 5m
labels: { severity: critical }
annotations:
summary: "验证者未签名区块"
description: "过去 5 分钟未签名任何区块"
- alert: ValidatorJailed
expr: msgchain_validator_jailed == 1
for: 0m
labels: { severity: critical }
annotations:
summary: "验证者被监禁!"
description: "验证者 {{ $labels.instance }} 已被网络监禁"
- alert: HighMemoryUsage
expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
for: 10m
labels: { severity: warning }
annotations:
summary: "内存不足"
description: "可用内存 < 10%"
6.4 备份健康监控
#!/bin/bash
# 脚本: monitor_backup_health.sh
# 功能: 监控备份健康状况
set -euo pipefail
BACKUP_BASE="/backup/msgchain"
ALERT_EMAIL="ops@msgchain.org"
PAGERDUTY_KEY="${PAGERDUTY_KEY:-}"
check_backup_freshness() {
local latest_full=$(ls -td "$BACKUP_BASE"/full/*/ 2>/dev/null | head -1)
if [ -z "$latest_full" ]; then
echo "CRITICAL: 无全量备份!"
return 2
fi
local backup_time=$(stat -c %Y "$latest_full")
local now=$(date +%s)
local age=$(( (now - backup_time) / 3600 ))
if [ "$age" -gt 48 ]; then
echo "WARNING: 最新备份已 ${age} 小时 (超过 48 小时阈值)"
return 1
fi
echo "OK: 最新备份 ${age} 小时前"
return 0
}
check_backup_size() {
local latest_full=$(ls -td "$BACKUP_BASE"/full/*/ 2>/dev/null | head -1)
[ -z "$latest_full" ] && return 1
local size=$(du -sb "$latest_full" | cut -f1)
local data_size=$(du -sb "${HOME}/.msgchain/data" 2>/dev/null | cut -f1)
local ratio=$(( size * 100 / (data_size + 1) ))
if [ "$ratio" -lt 10 ] || [ "$ratio" -gt 300 ]; then
echo "WARNING: 备份大小异常 (数据 ${data_size}, 备份 ${size}, 比例 ${ratio}%)"
return 1
fi
echo "OK: 备份大小正常 (${ratio}% 数据大小)"
return 0
}
check_disk_space() {
local available=$(df --output=avail "$BACKUP_BASE" 2>/dev/null | tail -1)
local available_gb=$(( available / 1024 / 1024 ))
if [ "$available_gb" -lt 50 ]; then
echo "WARNING: 备份磁盘剩余 ${available_gb}GB (不足 50GB)"
return 1
fi
echo "OK: 备份磁盘剩余 ${available_gb}GB"
return 0
}
main() {
echo "MSG Chain 备份健康检查 - $(date)"
echo "================================"
failures=0
check_backup_freshness || ((failures++))
check_backup_size || ((failures++))
check_disk_space || ((failures++))
echo ""
[ "$failures" -gt 0 ] && echo "发现 ${failures} 个问题" || echo "所有检查通过"
}
main "$@"
6.5 数据完整性检查
#!/bin/bash
# 脚本: data_integrity_check.sh
# 功能: MSG Chain 数据完整性检查
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
CHECKSUMS_DB="${NODE_HOME}/data/.integrity_checksums"
LOG_FILE="/var/log/msgchain/integrity_check.log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
create_checksums() {
log_info "创建数据校验和..."
find "$NODE_HOME/data" -type f \( -name "*.sst" -o -name "*.vlog" -o -name "*.ldb" -o -name "*.log" \) \
-exec sha256sum {} \; > "$CHECKSUMS_DB.new"
mv "$CHECKSUMS_DB.new" "$CHECKSUMS_DB"
log_info "校验和已创建 ($(wc -l < "$CHECKSUMS_DB") 个文件)"
}
verify_checksums() {
log_info "验证数据完整性..."
local failures=0
while IFS= read -r line; do
local expected_hash=$(echo "$line" | cut -d' ' -f1)
local filepath=$(echo "$line" | cut -d' ' -f2-)
if [ -f "$filepath" ]; then
local actual_hash=$(sha256sum "$filepath" | cut -d' ' -f1)
if [ "$expected_hash" != "$actual_hash" ]; then
log_error "文件损坏: ${filepath}"
log_error "期望: ${expected_hash}, 实际: ${actual_hash}"
((failures++))
fi
else
log_error "文件丢失: ${filepath}"
((failures++))
fi
done < "$CHECKSUMS_DB"
[ "$failures" -eq 0 ] && log_info "完整性检查通过 (所有文件正常)" || log_error "发现 ${failures} 个问题"
return "$failures"
}
main() {
local action="${1:-check}"
case "$action" in
create)
create_checksums
;;
check)
[ -f "$CHECKSUMS_DB" ] || { log_info "首次运行: 创建校验和"; create_checksums; }
verify_checksums
;;
*)
echo "用法: $0 {create|check}"
exit 1
;;
esac
}
main "$@"
6.6 Grafana 仪表板指标
{
"title": "MSG Chain 节点监控",
"panels": [
{
"title": "区块高度",
"type": "graph",
"targets": [
{ "expr": "msgchain_latest_block_height{job=\"msgchain-validator\"}", "legendFormat": "验证者" },
{ "expr": "avg(msgchain_latest_block_height{job=\"msgchain-node\"})", "legendFormat": "哨兵平均" }
]
},
{
"title": "同步状态",
"type": "stat",
"targets": [
{ "expr": "msgchain_catching_up{job=\"msgchain-validator\"}", "legendFormat": "同步中" }
]
},
{
"title": "对等节点数",
"type": "graph",
"targets": [
{ "expr": "msgchain_peers{job=\"msgchain-node\"}", "legendFormat": "{{instance}}" }
]
},
{
"title": "磁盘使用率",
"type": "gauge",
"targets": [
{ "expr": "(1 - node_filesystem_avail_bytes{mountpoint=\"/data\"} / node_filesystem_size_bytes{mountpoint=\"/data\"}) * 100" }
]
}
]
}
6.7 日常巡检脚本
#!/bin/bash
# 脚本: daily_checks.sh
# 功能: MSG Chain 节点日常巡检
set -euo pipefail
RPC="http://localhost:26657"
LOG_FILE="/var/log/msgchain/daily_checks.log"
log_info() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*"; }
echo "=========================================="
echo " MSG Chain 日常巡检报告"
echo " 时间: $(date)"
echo "=========================================="
# 1. 节点状态
echo -e "\n--- 节点状态 ---"
STATUS=$(curl -sf "${RPC}/status" | jq '.result' 2>/dev/null || echo "N/A")
echo "$STATUS" | jq '{moniker: .node_info.moniker, network: .node_info.network, version: .node_info.version, blocks: .sync_info.latest_block_height, catching_up: .sync_info.catching_up}' 2>/dev/null
# 2. 对等节点
echo -e "\n--- 对等节点 ---"
NETINFO=$(curl -sf "${RPC}/net_info" | jq '.result' 2>/dev/null || echo '{}')
echo "连接数: $(echo "$NETINFO" | jq '.n_peers // 0')"
# 3. 磁盘使用
echo -e "\n--- 磁盘使用 ---"
DATA_DIR="${HOME}/.msgchain/data"
[ -d "$DATA_DIR" ] && du -sh "$DATA_DIR" || echo "数据目录不存在"
df -h "$DATA_DIR" | tail -1
# 4. 系统资源
echo -e "\n--- 系统资源 ---"
echo "CPU: $(top -bn1 | grep 'Cpu(s)' | awk '{print $8}')% idle"
echo "内存: $(free -h | grep Mem | awk '{print $3 "/" $2}')"
echo "负载: $(uptime | awk -F'load average:' '{print $2}')"
# 5. 共识检查
echo -e "\n--- 共识状态 ---"
CONSENSUS=$(curl -sf "${RPC}/consensus_state" 2>/dev/null | jq '.result.round_state | {height: .height, round: .round, step: .step}' 2>/dev/null || echo "N/A")
echo "$CONSENSUS"
# 6. 进程状态
echo -e "\n--- 进程状态 ---"
systemctl is-active msgchaind
journalctl -u msgchaind --since "1 hour ago" --no-pager | grep -i "error\|panic\|corrupt\|fail" | tail -5 || echo "无错误日志"
echo -e "\n=========================================="
echo " 巡检完成"
echo "=========================================="
7. 验证者专用
7.1 验证者密钥安全
验证者密钥是节点的最高敏感资产。泄露或丢失将导致:
- 代币被盗 (如果密钥包含资金访问权限)
- 双重签名惩罚 (如果密钥被用于双重签名)
- 验证者身份被盗用
7.1.1 密钥存储层次
验证者密钥安全层级:
Level 0: 内存 (节点运行时)
Level 1: 本地磁盘加密 (软件默认)
Level 2: HSM/密码管理器
Level 3: 离线冷存储 (Shamir 备份)
Level 4: 多方分片 (社会恢复)
7.1.2 HSM 配置
#!/bin/bash
# 脚本: configure_hsm.sh
# 功能: 配置 HSM (硬件安全模块) 用于验证者签名
# 支持: YubiHSM, Ledger, 自定义 PKCS#11
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
configure_yubihsm() {
log_info "配置 YubiHSM..."
# 安装 YubiHSM 工具
# apt install yubihsm-shell libyubihsm-dev
# 初始化 HSM
yubihsm-shell --action create-session \
--auth-key 1 --password "change_me_immediately"
# 生成验证者密钥 (在 HSM 内)
yubihsm-shell --action generate-asymmetric-key \
--key-algo ec-p256 \
--label "MSG Chain Validator Key" \
--domains 1 \
--capabilities sign-ecdsa
# 导出公钥
yubihsm-shell --action get-public-key \
--key-id 0x1234 \
--outfile /tmp/validator_pubkey.pem
# 配置 Cosmos SDK 使用 HSM
cat > "${NODE_HOME}/config/hsm_config.yaml" << EOF
hsm:
module: /usr/lib/x86_64-linux-gnu/pkcs11/yubihsm_pkcs11.so
slot: 0
key_id: 0x1234
pin: "${YUBIHSM_PIN}"
algorithm: "ecdsa-secp256r1"
EOF
log_info "YubiHSM 配置完成"
}
configure_ledger() {
log_info "配置 Ledger 硬件钱包..."
log_info "1. 连接 Ledger 设备并解锁"
log_info "2. 打开 Cosmos App"
log_info "3. 运行: msgchaind keys add validator --ledger --hd-path \"44'/118'/0'/0/0\""
log_info "4. 在 Ledger 上确认操作"
}
# 主流程
main() {
case "${1:-}" in
yubihsm) configure_yubihsm ;;
ledger) configure_ledger ;;
*) echo "用法: $0 {yubihsm|ledger}" ;;
esac
}
main "$@"
7.2 双重签名保护
#!/bin/bash
# 脚本: double_sign_protection.sh
# 功能: 配置双重签名预防机制
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
VALIDATOR_KEY="${NODE_HOME}/config/priv_validator_key.json"
BACKUP_VAL_KEY="${NODE_HOME}/config/priv_validator_key.json.backup"
log_info() { echo "[INFO] $(date) $*"; }
log_error() { echo "[ERROR] $(date) $*" >&2; }
# 1. 备份原始密钥
backup_original_key() {
[ -f "$VALIDATOR_KEY" ] && cp "$VALIDATOR_KEY" "$BACKUP_VAL_KEY"
log_info "原始密钥已备份到 ${BACKUP_VAL_KEY}"
}
# 2. 配置 TMKMS (远程签名)
configure_tmkms() {
log_info "配置 TMKMS 远程签名..."
# 安装 TMKMS
# git clone https://github.com/iqlusioninc/tmkms.git
# cd tmkms && cargo build --release
# 创建 TMKMS 配置
cat > "${NODE_HOME}/config/tmkms.toml" << EOF
[[chain]]
id = "msg-chain-1"
key_format = { type = "cosmos-json", path = "${VALIDATOR_KEY}" }
[[providers]]
type = "softsign"
path = "${VALIDATOR_KEY}"
[[validator]]
addr = "tcp://127.0.0.1:26659"
chain_id = "msg-chain-1"
protocol = "secret-connection"
EOF
# 配置节点使用外部签名
sed -i 's/^priv_validator_laddr = .*/priv_validator_laddr = "tc[未公开路径]"/' \
"${NODE_HOME}/config/config.toml"
log_info "TMKMS 配置完成"
log_info "启动: tmkms start -c ${NODE_HOME}/config/tmkms.toml"
}
# 3. 哨兵检查 (停止相同公钥的另一个节点)
sentry_prevention() {
log_info "配置哨兵式双重签名预防..."
cat > "${NODE_HOME}/config/double_sign_check.sh" << 'SCRIPT'
#!/bin/bash
# 在启动前检查是否有另一实例使用相同密钥运行
KEY_HASH=$(sha256sum "${NODE_HOME}/config/priv_validator_key.json" | cut -d' ' -f1)
# 扫描本地网络中的节点
for host in $(seq 1 10); do
ADDR="192.168.1.${host}:26657"
STATUS=$(curl -sf "http://${ADDR}/status" 2>/dev/null || echo "")
if [ -n "$STATUS" ]; then
echo "发现活跃节点: ${ADDR}"
# 比较公钥 (需要扩展接口支持)
fi
done
echo "双重签名检查完成: ${KEY_HASH}"
SCRIPT
chmod +x "${NODE_HOME}/config/double_sign_check.sh"
}
# 4. 启动前验证脚本
create_startup_hook() {
cat > "${NODE_HOME}/config/pre_start_check.sh" << 'SCRIPT'
#!/bin/bash
# 节点启动前置检查
NODE_HOME="${HOME}/.msgchain"
ERRORS=0
check() {
local msg="$1"; shift
if "$@"; then echo " ✓ $msg"; else echo " ✗ $msg"; ((ERRORS++)); fi
}
echo "MSG Chain 节点启动前检查..."
check "验证者密钥存在" [ -f "${NODE_HOME}/config/priv_validator_key.json" ]
check "密钥格式有效" msgchaind tendermint show-validator &>/dev/null
check "创世文件存在" [ -f "${NODE_HOME}/config/genesis.json" ]
check "创世文件有效" msgchaind validate-genesis &>/dev/null
check "config.toml 存在" [ -f "${NODE_HOME}/config/config.toml" ]
check "app.toml 存在" [ -f "${NODE_HOME}/config/app.toml" ]
check "数据目录可写" [ -w "${NODE_HOME}/data" ]
check "磁盘空间充足" [ "$(df --output=avail "${NODE_HOME}/data" | tail -1)" -gt 10485760 ]
# 双重签名检查
if [ -f "${NODE_HOME}/config/double_sign_check.sh" ]; then
bash "${NODE_HOME}/config/double_sign_check.sh"
fi
echo ""
[ "$ERRORS" -eq 0 ] && echo "所有检查通过，启动节点..." || echo "发现 ${ERRORS} 个问题，请修复后重试"
exit "$ERRORS"
SCRIPT
chmod +x "${NODE_HOME}/config/pre_start_check.sh"
# 添加到 systemd 单元
mkdir -p /etc/systemd/system/msgchaind.service.d/
cat > /etc/systemd/system/msgchaind.service.d/pre_check.conf << EOF
[Service]
ExecStartPre=${NODE_HOME}/config/pre_start_check.sh
EOF
systemctl daemon-reload
log_info "启动前检查已配置"
}
# 主流程
main() {
backup_original_key
configure_tmkms
sentry_prevention
create_startup_hook
log_info "双重签名保护配置完成"
}
main
7.3 签名密钥轮换
#!/bin/bash
# 脚本: rotate_validator_key.sh
# 功能: 验证者签名密钥轮换
# 警告: 这是一个高风险操作，请在生产环境外充分测试
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
OLD_KEY="${NODE_HOME}/config/priv_validator_key.json.old"
LOG_FILE="/var/log/msgchain/key_rotation.log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
# 阶段 1: 准备新密钥
prepare_new_key() {
log_info "准备新验证者密钥..."
# 方法 1: 使用 msgchaind 生成
msgchaind tendermint gen-validator > "${NODE_HOME}/config/new_validator_key.json"
log_info "新密钥已生成"
# 获取新旧公钥
OLD_PUBKEY=$(jq -r '.pub_key.value' "${NODE_HOME}/config/priv_validator_key.json")
NEW_PUBKEY=$(jq -r '.pub_key.value' "${NODE_HOME}/config/new_validator_key.json")
log_info "旧公钥: ${OLD_PUBKEY}"
log_info "新公钥: ${NEW_PUBKEY}"
}
# 阶段 2: 备份旧密钥
backup_old_key() {
log_info "备份旧密钥..."
cp "${NODE_HOME}/config/priv_validator_key.json" "$OLD_KEY"
cp "${NODE_HOME}/config/priv_validator_state.json" "${NODE_HOME}/config/priv_validator_state.json.old"
log_info "旧密钥已备份到 ${OLD_KEY}"
}
# 阶段 3: 替换密钥
swap_keys() {
log_info "替换验证者密钥..."
mv "${NODE_HOME}/config/new_validator_key.json" \
"${NODE_HOME}/config/priv_validator_key.json"
# 重置状态 (避免使用旧签名的计数器)
cat > "${NODE_HOME}/config/priv_validator_state.json" << EOF
{ "height": "0", "round": 0, "step": 0 }
EOF
log_info "密钥已替换"
}
# 阶段 4: 重启节点
restart_node() {
log_info "重启节点以加载新密钥..."
systemctl restart msgchaind
sleep 10
for i in $(seq 1 30); do
if curl -sf localhost:26657/status > /dev/null 2>&1; then
log_info "节点已重启"
return 0
fi
sleep 5
done
log_error "节点重启失败"
return 1
}
# 阶段 5: 链上更新
update_on_chain() {
log_info "请手动更新链上验证者信息:"
NEW_PUBKEY=$(jq -r '.pub_key.value' "${NODE_HOME}/config/priv_validator_key.json")
cat << EOF
=== 链上更新步骤 ===
1. 更新验证者公钥:
msgchaind tx staking edit-validator \\
--new-pubkey '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"${NEW_PUBKEY}"}' \\
--from validator \\
--chain-id msg-chain-1 \\
--gas auto
2. 监控更新状态:
msgchaind query tx <tx_hash>
=== 备用策略 ===
如果密钥轮换后验证者状态异常:
1. 停止节点
2. 恢复旧密钥: cp ${OLD_KEY} ${NODE_HOME}/config/priv_validator_key.json
3. 重启节点: systemctl restart msgchaind
4. 调查原因后重试
EOF
}
# 主流程
main() {
log_info "========== 验证者密钥轮换 =========="
echo ""
echo "⚠️ 警告: 密钥轮换是一个高风险操作!"
echo "错误的操作可能导致:"
echo " - 双重签名惩罚 (如果旧节点仍在线签名)"
echo " - 验证者离线 (如果新密钥无效)"
echo " - 需要等待 unbonding 周期"
echo ""
read -p "确认继续? (输入 'YES' 确认): " confirm
[ "$confirm" != "YES" ] && { log_info "已取消"; exit 0; }
prepare_new_key
backup_old_key
swap_keys
restart_node
update_on_chain
log_info "========== 密钥轮换完成 =========="
}
main "$@"
7.4 验证者状态监控脚本
#!/bin/bash
# 脚本: validator_monitor.sh
# 功能: 验证者节点专项监控
set -euo pipefail
RPC_ENDPOINT="${1:-http://localhost:26657}"
VALIDATOR_ADDRESS="${2:-}"
LOG_FILE="/var/log/msgchain/validator_monitor.log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_warn() { echo "[WARN] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
# 获取验证者地址 (如果未提供)
if [ -z "$VALIDATOR_ADDRESS" ]; then
VALIDATOR_ADDRESS=$(msgchaind tendermint show-address 2>/dev/null || echo "")
[ -z "$VALIDATOR_ADDRESS" ] && { log_error "无法获取验证者地址"; exit 1; }
fi
# 1. 检查签名活动
check_signing() {
local status=$(curl -sf "${RPC_ENDPOINT}/status" | jq '.result' 2>/dev/null || echo '{}')
local height=$(echo "$status" | jq -r '.sync_info.latest_block_height // "0"')
# 获取最近 100 个区块的签名信息
local signed=0
local missed=0
for h in $(seq $((height - 100)) $height); do
local block=$(curl -sf "${RPC_ENDPOINT}/block?height=${h}" 2>/dev/null || echo '{}')
local proposer=$(echo "$block" | jq -r '.result.block.header.proposer_address // ""' 2>/dev/null)
local signs=$(echo "$block" | jq '.result.block.last_commit.signatures | length' 2>/dev/null || echo "0")
if [ "$proposer" = "$VALIDATOR_ADDRESS" ]; then
((signed++))
fi
done
local missed_blocks=$((100 - signed))
log_info "最近 100 区块: 签名 ${signed}, 错过 ${missed_blocks}"
if [ "$missed_blocks" -gt 10 ]; then
log_error "高错过率! ${missed_blocks}/100 区块错过签名"
return 1
fi
return 0
}
# 2. 检查验证者状态
check_validator_status() {
local val_info=$(curl -sf "${RPC_ENDPOINT}/validators?height=0" 2>/dev/null || echo '{}')
local in_set=$(echo "$val_info" | jq -r ".result.validators[] | select(.address == \"$VALIDATOR_ADDRESS\") | .address" 2>/dev/null || echo "")
if [ -n "$in_set" ]; then
log_info "✓ 验证者在活跃验证者集合中"
else
log_warn "? 验证者不在活跃集合中 (可能被监禁或 unbonding)"
fi
}
# 3. 检查共识轮次
check_consensus_round() {
local cons=$(curl -sf "${RPC_ENDPOINT}/consensus_state" 2>/dev/null || echo '{}')
local height=$(echo "$cons" | jq -r '.result.round_state.height // "N/A"' 2>/dev/null)
local round=$(echo "$cons" | jq -r '.result.round_state.round // "N/A"' 2>/dev/null)
local step=$(echo "$cons" | jq -r '.result.round_state.step // "N/A"' 2>/dev/null)
log_info "共识: 高度=${height}, 轮次=${round}, 步骤=${step}"
# 如果轮次过高，可能存在问题
if [ "$round" != "N/A" ] && [ "$round" -gt 10 ]; then
log_warn "共识轮次较高 (${round})，可能网络不稳定"
fi
}
# 4. 检查投票权
check_voting_power() {
local status=$(curl -sf "${RPC_ENDPOINT}/status" | jq '.result' 2>/dev/null || echo '{}')
local vp=$(echo "$status" | jq -r '.validator_info.voting_power // "0"' 2>/dev/null)
log_info "投票权: ${vp}"
[ "$vp" = "0" ] && log_warn "投票权为零 (节点可能被监禁)"
}
# 主循环
main() {
log_info "========== 验证者监控 =========="
log_info "验证者地址: ${VALIDATOR_ADDRESS}"
log_info "RPC 节点: ${RPC_ENDPOINT}"
echo ""
check_signing || true
check_validator_status
check_consensus_round
check_voting_power
log_info "========== 监控完成 =========="
}
main "$@"
7.5 自动解监禁脚本
#!/bin/bash
# 脚本: auto_unjail.sh
# 功能: 自动检测验证者监禁状态并解监禁
set -euo pipefail
VALIDATOR_ADDRESS="${VALIDATOR_ADDRESS:-}"
CHAIN_ID="msg-chain-1"
NODE_HOME="${HOME}/.msgchain"
LOG_FILE="/var/log/msgchain/unjail.log"
LAST_UNJAIL_FILE="/tmp/msgchain_last_unjail"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$LOG_FILE" >&2; }
get_validator_address() {
if [ -n "$VALIDATOR_ADDRESS" ]; then
echo "$VALIDATOR_ADDRESS"
else
msgchaind tendermint show-address 2>/dev/null || echo ""
fi
}
check_jailed() {
local val_addr="$1"
# 通过 RPC 查询验证者信息
local val_info=$(msgchaind query staking validator "$val_addr" \
--output json 2>/dev/null || echo '{}')
local jailed=$(echo "$val_info" | jq -r '.jailed // "false"' 2>/dev/null)
echo "$jailed"
}
perform_unjail() {
log_info "执行 unjail 操作..."
# 检查频率限制 (每 10 分钟最多一次)
if [ -f "$LAST_UNJAIL_FILE" ]; then
local last=$(cat "$LAST_UNJAIL_FILE")
local now=$(date +%s)
if [ $((now - last)) -lt 600 ]; then
log_warn "上一次 unjail 在 10 分钟内，跳过"
return 0
fi
fi
msgchaind tx slashing unjail \
--from validator \
--chain-id "$CHAIN_ID" \
--gas auto \
--gas-prices "1000000000attoMSG" \
-y 2>&1 | tee -a "$LOG_FILE"
local result=$?
if [ $result -eq 0 ]; then
date +%s > "$LAST_UNJAIL_FILE"
log_info "unjail 交易已提交"
else
log_error "unjail 失败"
fi
return $result
}
main() {
log_info "========== 自动解监禁检查 =========="
local val_addr=$(get_validator_address)
[ -z "$val_addr" ] && { log_error "无法获取验证者地址"; exit 1; }
local jailed=$(check_jailed "$val_addr")
log_info "验证者 ${val_addr}: jailed=${jailed}"
if [ "$jailed" = "true" ]; then
log_error "验证者被监禁! 尝试解监禁..."
perform_unjail
else
log_info "验证者状态正常"
fi
log_info "========== 检查完成 =========="
}
main "$@"
8. 测试灾难恢复
8.1 灾难恢复演练计划
8.1.1 演练频率
| 演练类型 | 频率 | 参与者 | 持续时间 |
|---|---|---|---|
| 桌面上演练 | 每季度 | 运维团队 | 2 小时 |
| 单节点恢复 | 每月 | 运维人员 | 1 小时 |
| 完整 DR 演练 | 每半年 | 全体团队 | 4 小时 |
| 密钥恢复 | 每季度 | 安全团队 | 1 小时 |
8.1.2 演练范围
| 场景编号 | 场景描述 | 演练类型 | 复杂度 |
|---|---|---|---|
| DR-01 | 数据目录误删 | 单节点 | 简单 |
| DR-02 | BadgerDB 损坏 | 单节点 | 中等 |
| DR-03 | 磁盘完全故障 | 单节点 | 中等 |
| DR-04 | 哨兵节点批量故障 | 集群 | 复杂 |
| DR-05 | 验证者密钥丢失 | 关键 | 高复杂 |
| DR-06 | 验证者节点完全损毁 | 关键 | 高复杂 |
| DR-07 | 数据中心级故障 | 跨区 | 极高 |
8.2 RTO/RPO 测试
#!/bin/bash
# 脚本: rto_rpo_test.sh
# 功能: 测试 MSG Chain 节点的恢复时间和数据丢失
set -euo pipefail
NODE_HOME="${HOME}/.msgchain"
BACKUP_BASE="/backup/msgchain"
TEST_DIR="/tmp/msgchain-dr-test-$(date +%s)"
REPORT_FILE="/var/log/msgchain/rto_rpo_report.log"
log_info() { echo "[INFO] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$REPORT_FILE"; }
log_error() { echo "[ERROR] $(date '+%Y-%m-%d %H:%M:%S') $*" | tee -a "$REPORT_FILE" >&2; }
cleanup() { rm -rf "$TEST_DIR"; }
trap cleanup EXIT
# 阶段 1: 记录基准状态
phase1_baseline() {
log_info "阶段 1: 记录基准状态..."
BASELINE_HEIGHT=$(curl -sf localhost:26657/status | jq -r '.result.sync_info.latest_block_height')
BASELINE_TIME=$(date +%s)
BASELINE_BLOCK_TIME=$(curl -sf "localhost:26657/block?height=${BASELINE_HEIGHT}" | jq -r '.result.block.header.time')
log_info "基准高度: ${BASELINE_HEIGHT}"
log_info "基准区块时间: ${BASELINE_BLOCK_TIME}"
log_info "基准时间戳: ${BASELINE_TIME}"
echo "$BASELINE_HEIGHT" > "${TEST_DIR}/baseline_height.txt"
echo "$BASELINE_TIME" > "${TEST_DIR}/baseline_time.txt"
}
# 阶段 2: 模拟灾难
phase2_simulate_disaster() {
log_info "阶段 2: 模拟灾难 (删除数据目录)..."
DISASTER_TIME=$(date +%s)
systemctl stop msgchaind || true
sleep 3
# 模拟数据损坏：重命名数据目录
if [ -d "${NODE_HOME}/data" ]; then
mv "${NODE_HOME}/data" "${NODE_HOME}/data_disaster_$(date +%s)"
log_info "数据目录已移除 (模拟灾难)"
fi
mkdir -p "${NODE_HOME}/data"
echo "$DISASTER_TIME" > "${TEST_DIR}/disaster_time.txt"
}
# 阶段 3: 执行恢复
phase3_perform_recovery() {
log_info "阶段 3: 执行恢复..."
RECOVERY_START_TIME=$(date +%s)
# 从最新备份恢复
LATEST_BACKUP=$(ls -td "$BACKUP_BASE"/full/*/ 2>/dev/null | head -1)
if [ -z "$LATEST_BACKUP" ]; then
log_error "无可用备份，使用快照恢复"
# 这里调用快照恢复脚本
LATEST_SNAPSHOT=$(curl -sf "https://snapshots.msgchain.org/latest.json" | jq -r '.url')
SNAPSHOT_FILE="${TEST_DIR}/snapshot.tar.lz4"
wget -O "$SNAPSHOT_FILE" "$LATEST_SNAPSHOT"
tar -I lz4 -xf "$SNAPSHOT_FILE" -C "${NODE_HOME}/data/"
RECOVERY_TYPE="snapshot"
else
log_info "使用备份: ${LATEST_BACKUP}"
tar -I zstd -xf "${LATEST_BACKUP}/data.tar.zst" -C "$NODE_HOME"
RECOVERY_TYPE="backup"
fi
# 重置 CometBFT
msgchaind tendermint unsafe-reset-all --keep-addr-book || true
# 启动节点
systemctl start msgchaind
# 等待节点健康
for i in $(seq 1 60); do
if curl -sf localhost:26657/health > /dev/null 2>&1; then
RECOVERY_READY_TIME=$(date +%s)
log_info "节点恢复可用!"
break
fi
sleep 5
done
# 等待同步完成
for i in $(seq 1 120); do
sleep 5
local status=$(curl -sf localhost:26657/status 2>/dev/null || echo '{}')
local catching_up=$(echo "$status" | jq -r '.result.sync_info.catching_up // "true"')
if [ "$catching_up" = "false" ]; then
RECOVERY_FULL_TIME=$(date +%s)
SYNCED_HEIGHT=$(echo "$status" | jq -r '.result.sync_info.latest_block_height')
log_info "节点完全同步!"
break
fi
done
echo "$RECOVERY_START_TIME" > "${TEST_DIR}/recovery_start.txt"
echo "${RECOVERY_READY_TIME:-$RECOVERY_START_TIME}" > "${TEST_DIR}/recovery_ready.txt"
echo "${RECOVERY_FULL_TIME:-$RECOVERY_START_TIME}" > "${TEST_DIR}/recovery_full.txt"
echo "$RECOVERY_TYPE" > "${TEST_DIR}/recovery_type.txt"
}
# 阶段 4: 计算指标
phase4_calculate_metrics() {
log_info "阶段 4: 计算 RTO/RPO 指标..."
BASELINE_HEIGHT=$(cat "${TEST_DIR}/baseline_height.txt")
BASELINE_TIME=$(cat "${TEST_DIR}/baseline_time.txt")
DISASTER_TIME=$(cat "${TEST_DIR}/disaster_time.txt")
RECOVERY_START=$(cat "${TEST_DIR}/recovery_start.txt")
RECOVERY_READY=$(cat "${TEST_DIR}/recovery_ready.txt")
RECOVERY_FULL=$(cat "${TEST_DIR}/recovery_full.txt")
# RPO 计算: 从备份时间到灾难时间的差额
BACKUP_TIME=$(stat -c %Y "$(ls -td "$BACKUP_BASE"/full/*/ 2>/dev/null | head -1)"/backup_metadata.json 2>/dev/null || echo "$DISASTER_TIME")
RPO=$(( (DISASTER_TIME - BACKUP_TIME) / 60 ))
log_info ""
log_info "========== 测试结果 =========="
log_info "恢复类型: $(cat "${TEST_DIR}/recovery_type.txt")"
log_info "备份时间: $(date -d "@$BACKUP_TIME")"
log_info "灾难时间: $(date -d "@$DISASTER_TIME")"
log_info "恢复开始: $(date -d "@$RECOVERY_START")"
log_info "服务恢复: $(date -d "@$RECOVERY_READY")"
log_info "完全同步: $(date -d "@$RECOVERY_FULL")"
log_info ""
log_info "--- RPO ---"
log_info "理论 RPO: ${RPO} 分钟"
log_info "数据丢失窗口: ${RPO} 分钟"
log_info ""
log_info "--- RTO ---"
RTO_READY=$(( (RECOVERY_READY - RECOVERY_START) / 60 ))
RTO_FULL=$(( (RECOVERY_FULL - RECOVERY_START) / 60 ))
log_info "服务恢复 RTO: ${RTO_READY} 分钟"
log_info "完全同步 RTO: ${RTO_FULL} 分钟"
log_info ""
log_info "--- 高度差异 ---"
log_info "基准高度: ${BASELINE_HEIGHT}"
log_info "同步后高度: $(cat "${TEST_DIR}/recovery_full.txt" 2>/dev/null || echo "N/A")"
log_info "================================"
}
# 主流程
main() {
log_info "========== RTO/RPO 测试开始 =========="
mkdir -p "$TEST_DIR"
phase1_baseline
phase2_simulate_disaster
phase3_perform_recovery
phase4_calculate_metrics
log_info "========== RTO/RPO 测试完成 =========="
log_info "报告文件: ${REPORT_FILE}"
}
main "$@"
8.3 桌面积上演练
# MSG Chain 灾难恢复桌面积上演练脚本
## 场景 DR-01: 数据目录误删
**演练目标**: 验证从误删数据目录到节点恢复的全流程
**参与角色**:
- 操作员: 执行恢复操作
- 观察员: 记录时间线和操作步骤
- 审批人: 批准恢复操作
**时间线**:
| 时间 | 事件 | 操作 |
|------|------|------|
| T+0 | 告警: 节点停机 | 操作员收到宕机告警 |
| T+5 | 确认: 数据目录被误删 | 操作员检查 journalctl 日志 |
| T+10 | 决策: 从快照恢复 | 审批人批准恢复方案 |
| T+15 | 执行: 停止节点 | systemctl stop msgchaind |
| T+20 | 执行: 下载最新快照 | wget snapshot |
| T+30 | 执行: 解压快照 | tar -I lz4 -xf |
| T+35 | 执行: 重置 CometBFT | msgchaind tendermint unsafe-reset-all |
| T+38 | 执行: 启动节点 | systemctl start msgchaind |
| T+45 | 验证: 节点健康 | curl /health |
| T+60 | 验证: 完全同步 | curl /status (catching_up=false) |
**检查清单**:
- [ ] 告警通知是否及时送达
- [ ] 恢复操作是否遵循标准流程
- [ ] 快照下载是否顺利
- [ ] 校验和验证是否通过
- [ ] 节点启动是否正常
- [ ] 同步完成后交易是否可以正常查询
- [ ] RTO 是否在目标范围内
- [ ] RPO 是否在目标范围内
**复盘问题**:
1. 数据目录是何时、如何被误删的?
2. 如何防止类似事件再次发生?
3. 恢复过程中是否有可优化的环节?
4. 是否需要更新标准操作流程 (SOP)?
---
## 场景 DR-05: 验证者密钥丢失
**演练目标**: 测试验证者密钥恢复流程
**前置条件**:
- 密钥已通过安全方式备份
- 备份的完整性已定期验证
**恢复步骤**:
1. 确认密钥丢失范围 (仅节点本地 / 备份也丢失)
2. 如果本地密钥仅损坏 → 从备份恢复
3. 如果本地和备份均丢失:
a. 使用助记词恢复账户密钥
b. 生成新的验证者密钥
c. 提交验证者公钥更新交易
d. 等待 unbonding 周期 (21 天)
e. 重新加入验证者集合
**风险**:
- 如果无法恢复密钥，验证者将永久失去验证者身份
- unbonding 周期内没有质押收益
- 需要确保新旧密钥没有同时签名 (防止双签)
---
## 场景 DR-07: 数据中心级故障
**演练目标**: 验证跨数据中心的完整 DR 流程
**场景描述**: 主数据中心完全不可用 (火灾/电力中断)
**恢复方案**:
1. 在备用数据中心启动节点集群
2. 从异地备份恢复数据
3. 更新 DNS 记录指向备用中心
4. 验证验证者状态并重新加入共识
5. 主数据中心恢复后，同步并回切
**观测指标**:
- 备用中心启动时间
- 数据恢复时间
- DNS 切换延迟
- 共识重新加入时间
- 总 RTO/RPO
8.4 DR 演练报告模板
# MSG Chain 灾难恢复演练报告
## 基本信息
| 项目 | 内容 |
|------|------|
| 演练编号 | DR-2026-Q2-001 |
| 演练日期 | 2026-XX-XX |
| 演练场景 | [场景编号和名称] |
| 演练类型 | ☐ 桌面 ☐ 单节点 ☐ 集群 ☐ 完整DR |
| 参与人员 | [姓名列表] |
| 演练负责人 | [姓名] |
## 演练时间线
| 操作 | 计划时间 | 实际时间 | 偏差 |
|------|----------|----------|------|
| 开始 | | | |
| 阶段1: 准备 | | | |
| 阶段2: 灾难模拟 | | | |
| 阶段3: 恢复操作 | | | |
| 阶段4: 验证 | | | |
| 阶段5: 回切 | | | |
| 结束 | | | |
## 指标结果
| 指标 | 目标值 | 实际值 | 达标 |
|------|--------|--------|------|
| RTO | < 30 分钟 | ___ 分钟 | ☐ 是 ☐ 否 |
| RPO | < 1 小时 | ___ 分钟 | ☐ 是 ☐ 否 |
| 操作步骤数 | | ___ 步 | |
| 人工干预 | | ___ 次 | |
## 问题与改进
### 发现的问题
| # | 问题描述 | 严重程度 | 责任人 | 计划修复日期 |
|---|----------|----------|--------|-------------|
| 1 | | | | |
| 2 | | | | |
### 改进建议
1.
2.
3.
## 审批
| 角色 | 姓名 | 签名 | 日期 |
|------|------|------|------|
| 演练负责人 | | | |
| 运维负责人 | | | |
| 安全负责人 | | | |
8.5 自动化 DR 测试
#!/bin/bash
# 脚本: automated_dr_test.sh
# 功能: 自动化灾难恢复测试套件
set -euo pipefail
TEST_SUITE="${1:-all}"
REPORT_DIR="/var/log/msgchain/dr_tests"
mkdir -p "$REPORT_DIR"
log_suite() { echo "=== Suite: $1 ==="; }
log_test() { echo " Test: $1"; }
pass() { echo " ✓ PASS"; }
fail() { echo " ✗ FAIL: $1"; }
# 测试 1: 备份完整性
test_backup_integrity() {
log_test "备份完整性测试"
local latest=$(ls -t /backup/msgchain/full/*/data.tar.zst 2>/dev/null | head -1)
[ -z "$latest" ] && { fail "无可用备份"; return 1; }
tar -I zstd -tf "$latest" > /dev/null 2>&1 || { fail "备份损坏"; return 1; }
pass
}
# 测试 2: 快照下载
test_snapshot_download() {
log_test "快照下载测试"
local tmp=/tmp/dr_test_snapshot
mkdir -p "$tmp"
local url=$(curl -sf "https://snapshots.msgchain.org/latest.json" | jq -r '.url // ""' 2>/dev/null)
[ -z "$url" ] && { url="https://snapshots.msgchain.org/latest.tar.lz4"; }
timeout 120 wget -q -O "$tmp/snapshot.tar.lz4" "$url" 2>/dev/null || { fail "下载超时"; rm -rf "$tmp"; return 1; }
tar -I lz4 -tf "$tmp/snapshot.tar.lz4" > /dev/null 2>&1 || { fail "格式错误"; rm -rf "$tmp"; return 1; }
rm -rf "$tmp"
pass
}
# 测试 3: 节点启动
test_node_startup() {
log_test "节点启动测试"
systemctl start msgchaind 2>/dev/null || { fail "启动失败"; return 1; }
sleep 10
curl -sf localhost:26657/health > /dev/null 2>&1 || { fail "健康检查失败"; return 1; }
systemctl stop msgchaind 2>/dev/null || true
pass
}
# 测试 4: 密钥验证
test_key_validation() {
log_test "验证者密钥验证"
msgchaind tendermint show-validator > /dev/null 2>&1 || { fail "密钥无效"; return 1; }
pass
}
# 测试 5: 配置完整性
test_config_integrity() {
log_test "配置完整性测试"
local home="${HOME}/.msgchain"
for f in config.toml app.toml genesis.json; do
[ -f "${home}/config/${f}" ] || { fail "缺少 ${f}"; return 1; }
done
msgchaind validate-genesis > /dev/null 2>&1 || { fail "创世文件无效"; return 1; }
pass
}
# 运行特定套件或全部
run_all() {
log_suite "完整 DR 测试套件"
test_backup_integrity
test_snapshot_download
test_node_startup
test_key_validation
test_config_integrity
}
case "$TEST_SUITE" in
backup) test_backup_integrity ;;
snapshot) test_snapshot_download ;;
node) test_node_startup ;;
key) test_key_validation ;;
config) test_config_integrity ;;
all) run_all ;;
*) echo "未知测试套件: $TEST_SUITE"; exit 1 ;;
esac
附录
A. 关键命令速查
| 操作 | 命令 |
|---|---|
| 初始化节点 | msgchaind init <moniker> --chain-id msg-chain-1 |
| 启动节点 | msgchaind start |
| 停止节点 | systemctl stop msgchaind |
| 重启节点 | systemctl restart msgchaind |
| 查看日志 | journalctl -u msgchaind -f -n 100 |
| 检查同步状态 | curl -s localhost:26657/status | jq .result.sync_info |
| 检查对等节点 | curl -s localhost:26657/net_info | jq '.result.n_peers' |
| 查看验证者 | msgchaind tendermint show-validator |
| 重置 CometBFT | msgchaind tendermint unsafe-reset-all |
| 创建快照 | msgchaind snapshot create --height <h> |
| 验证创世文件 | msgchaind validate-genesis |
| 查询账户余额 | msgchaind query bank balances <addr> |
| 提交 unjail | msgchaind tx slashing unjail --from <key> --chain-id msg-chain-1 |
B. 目录结构
~/.msgchain/
├── config/
│ ├── config.toml # CometBFT 配置
│ ├── app.toml # 应用配置
│ ├── genesis.json # 创世文件
│ ├── node_key.json # 节点 P2P 密钥
│ ├── priv_validator_key.json # 验证者密钥
│ ├── priv_validator_state.json # 验证者状态
│ └── addrbook.json # 对等节点地址簿
├── data/
│ ├── badger/ # BadgerDB 应用状态
│ │ ├── *.sst # SST 文件
│ │ └── *.vlog # 值日志文件
│ ├── blockstore.db/ # 区块存储 (LevelDB)
│ ├── state.db/ # 共识状态 (LevelDB)
│ ├── evidence.db/ # 证据存储 (LevelDB)
│ ├── tx_index.db/ # 交易索引 (LevelDB)
│ ├── cs.wal/ # 共识 WAL
│ └── snapshots/ # 快照目录
├── keyring-file/ # 密钥环文件
└── cosmovisor/ # Cosmovisor 升级 (可选)
C. 端口参考
| 端口 | 协议 | 用途 | 安全建议 |
|---|---|---|---|
| 26656 | TCP | P2P 通信 | 公网开放 |
| 26657 | TCP | RPC 接口 | 内部网络 / TLS |
| 26658 | TCP | CometBFT gRPC | 内部网络 |
| 26660 | TCP | Prometheus 指标 | 内部网络 |
| 9090 | TCP | Cosmos gRPC | 内部网络 / TLS |
| 1317 | TCP | REST API | 内部网络 / LB |
| 9091 | TCP | gRPC-web | 内部网络 |
D. 环境变量
| 变量 | 默认值 | 说明 |
|---|---|---|
CHAIN_ID |
msg-chain-1 |
链标识符 |
NODE_HOME |
~/.msgchain |
节点家目录 |
MONIKER |
(hostname) |
节点名称 |
SNAPSHOT_INTERVAL |
5000 |
快照创建间隔 |
SNAPSHOT_KEEP_RECENT |
10 |
保留快照数量 |
MIN_GAS_PRICES |
1000000000attoMSG |
最小 gas 价格 |
PERSISTENT_PEERS |
`` | 持久对等节点列表 |
SEEDS |
`` | 种子节点列表 |
E. 常见问题排查
E.1 节点无法启动 - "value log corrupted"
# 停止节点
systemctl stop msgchaind
# 备份损坏数据
cp -r ~/.msgchain/data/badger ~/.msgchain/data/badger_corrupted
# 运行 BadgerDB 修复工具
msgchaind debug badger-repair --data-dir ~/.msgchain/data/badger
# 如果修复失败，从备份恢复
E.2 同步停滞
# 检查对等节点连接
curl -s localhost:26657/net_info | jq '.result.n_peers'
# 如果对等节点不足，添加更多持久节点
# 检查是否被限速
journalctl -u msgchaind --since "10 min ago" | grep -i "rate\|limit\|throttle"
E.3 "too many open files"
# 检查当前限制
ulimit -n
# 增大限制 (需要 root)
echo "fs.file-max = 65536" >> /etc/sysctl.conf
sysctl -p
# 更新 systemd 服务配置
E.4 磁盘空间不足
# 检查磁盘使用
df -h ~/.msgchain/data
du -sh ~/.msgchain/data/* | sort -rh | head -10
# 清理不需要的快照
rm -rf ~/.msgchain/data/snapshots/*
# 清理旧交易索引 (如果不需要历史查询)
# 或添加磁盘后迁移数据目录
F. 推荐硬件规格
| 节点类型 | CPU | 内存 | 磁盘 | 网络 |
|---|---|---|---|---|
| 验证者 | 8 核 | 32 GB | 2 TB NVMe | 1 Gbps |
| 哨兵 | 8 核 | 32 GB | 2 TB NVMe | 1 Gbps |
| RPC 节点 | 16 核 | 64 GB | 4 TB NVMe | 10 Gbps |
| 归档节点 | 32 核 | 128 GB | 10+ TB NVMe | 10 Gbps |
G. 相关资源
- MSG Chain 官方文档
- MSG Chain 白皮书系统
- Cosmos SDK 文档
- CometBFT 文档
- BadgerDB 文档
- Prometheus 文档
- 区块浏览器
- GitHub 仓库
- 快照下载
H. 版本历史
| 版本 | 日期 | 修改内容 | 作者 |
|---|---|---|---|
| 1.0 | 2026-07-06 | 初版创建 | MSG Chain 运维团队 |