IBC 通道监控与告警实践指南

数据来源：MSG Chain 代码库核实

主网状态: No-Go — 当前 MSGChain 主网裁决为 No-Go，以下内容反映代码实际状态，不代表生产可用。

目录

1. IBC 通道生命周期与状态机
2. 监控指标体系
3. Prometheus + Grafana 监控栈搭建
4. 告警规则设计
5. IBC 事件订阅架构
6. 跨链交易追踪
7. 自动化通道健康检查
8. 告警通知集成
9. 与 MSG Chain 可观测性栈的集成
10. 案例：典型 IBC 故障场景的监控与恢复
11. 总结

1. IBC 通道生命周期与状态机

1.1 概述

IBC（Inter-Blockchain Communication）是 Cosmos 生态中跨链通信的核心协议。一个 IBC 通道（Channel）的生命周期由四个状态机阶段构成：INIT → TRYOPEN → OPEN → CLOSED。每个阶段对应握手协议中的一个步骤，任何一步失败都会导致通道无法正常建立。

在 MSG Chain 上，IBC 通道是 CosmWasm 智能合约与外部链交互的唯一桥梁。通道的可靠性直接决定了跨链资产转移、跨链合约调用、以及 IBC 中间件（如 ICS-20、ICS-721）的正常运行。

1.2 状态机定义

IBC 通道状态定义在 ibc-go 的 core/04-channel/types/ 中，枚举值如下：

1.3 握手流程

1.3.1 三步握手（核心流程）

源链 (msg-chain-1)                    目标链 (counterparty-chain)
     │                                        │
     │  ChanOpenInit                          │
     │  ─────────────────────────────────────> │
     │                                        │
     │                                        │  ChanOpenTry
     │  <───────────────────────────────────── │
     │                                        │
     │  ChanOpenAck                           │
     │  ─────────────────────────────────────> │
     │                                        │
     │                                        │  ChanOpenConfirm
     │  <───────────────────────────────────── │
     │                                        │
     │           通道状态: OPEN                  │

1.3.2 INIT（通道初始化）

当在 MSG Chain 上发起一个 IBC 通道创建请求时，调用 MsgChannelOpenInit 消息。核心参数包括端口标识符、通道排序方式（ORDERED/UNORDERED）、对手方信息、连接路径和协议版本。

在 MSG Chain 上，典型的通道创建交易示例：

msgd tx ibc channel open-init \
    --port-id transfer \
    --channel-id channel-0 \
    --counterparty-port-id transfer \
    --counterparty-chain-id counterparty-chain-1 \
    --ordering ordered \
    --connection-hops connection-0 \
    --version "ics20-1" \
    --from wallet-key \
    --chain-id msg-chain-1 \
    --node https://rpc.msgchain.org:26657 \
    --fees 5000umsg

1.3.3 TRYOPEN（尝试打开）

目标链收到 ChanOpenTry 请求后，验证以下条件：

1. 验证源链的通道参数（通过 connection 关联的 light client）
2. 确认通道排序方式一致（ORDERED / UNORDERED）
3. 确认版本兼容性（如 ics20-1）
4. 确认连接标识符匹配

验证失败的常见原因：

• Light client 过期未更新
• 连接标识符不一致
• 版本协商失败
• 排序方式不匹配

1.3.4 OPEN（通道打开）

当双方确认握手参数后，通道进入 OPEN 状态。此时数据包可以开始流动，序列号从 1 开始，中继器开始转发数据包。

通道打开后的验证查询：

msgd query ibc channel end transfer channel-0 \
    --node https://rpc.msgchain.org:26657

msgd query ibc channel unrelayed-sequences transfer channel-0 \
    --node https://rpc.msgchain.org:26657

1.3.5 CLOSED（通道关闭）

通道关闭可能由以下原因触发：

1. 主动关闭: 管理员调用 MsgChannelCloseInit
2. 超时关闭: 数据包超时导致通道自动关闭（仅 ORDERED 通道）
3. 对抗性关闭: 检测到双签或轻客户端攻击

通道关闭的后果：

• ORDERED 通道：关闭后无法再发送新数据包，必须重建
• UNORDERED 通道：理论上允许部分恢复，但实际通常也需要重建

1.4 通道类型

1.4.1 ORDERED 通道

数据包按严格顺序传递，序列号一一对应。特性包括确保数据包的顺序性，如果一个数据包超时则通道自动关闭。适用于需要严格有序的场景（如 ICS-27 跨链账户）。

监控重点：任何数据包超时都视为严重事件，序列号连续性必须保持，通道关闭后必须立即告警。

1.4.2 UNORDERED 通道

数据包可以乱序传递，不要求严格序列对应。数据包可独立超时/重试，通道不会因单个数据包超时而关闭。适用于大多数资产转移场景（ICS-20）。

监控重点：垃圾数据包累积导致状态膨胀，中继器遗漏数据包，ack 返回延迟。

1.5 状态转换图

                 ┌──────────────┐
                 │    INIT      │
                 └──────┬───────┘
                        │
                        ▼
                 ┌──────────────┐
                 │   TRYOPEN    │
                 └──────┬───────┘
                        │
                        ▼
                 ┌──────────────┐
                 │    OPEN      │◄── 正常运行状态
                 └──────┬───────┘
                        │
               ┌────────┴────────┐
               │                 │
         主动关闭            ORDERED 超时
               │                 │
               ▼                 ▼
          ┌──────────────────────────┐
          │         CLOSED           │
          └──────────────────────────┘

1.6 MSG Chain 的 IBC 通道架构

在 msg-chain-1 上，IBC 通道的组织方式如下：

msg-chain-1
├── transfer 端口 (ics20-1)
│   ├── channel-0 ──── cosmos-hub
│   ├── channel-1 ──── osmosis-1
│   ├── channel-2 ──── juno-1
│   ├── channel-3 ──── secret-4
│   ├── channel-4 ──── axelar-dojo-1
│   └── channel-5 ──── neutron-1
├── wasm. 端口 (ics721)
│   ├── channel-100 ─── juno-1 (NFT 桥接)
│   └── channel-101 ─── stargaze-1
├── icahost 端口 (ics27)
│   └── channel-200 ─── neutron-1 (跨链账户)
└── fee 中间件通道
    └── channel-10 ─── cosmos-hub (IBC 费用)

1.7 通道生命周期中的关键交易类型

1.8 状态查询最佳实践

使用 REST 或 gRPC 查询通道状态：

curl -s https://rest.msgchain.org/ibc/core/channel/v1/channels/transfer/channel-0 | jq

grpcurl -plaintext msgchain.org:9090 \
    ibc.core.channel.v1.Query/Channel \
    -d '{"port_id": "transfer", "channel_id": "channel-0"}'

curl -s https://rest.msgchain.org/ibc/core/channel/v1/channels | jq '.channels[]'

2. 监控指标体系

2.1 概述

IBC 通道监控的核心是构建一个覆盖数据包流转全过程的指标体系。我们将指标分为六个维度：

1. 数据包流量指标: 发送/接收/确认的数据包数量
2. 积压指标: 未确认数据包的数量和延迟
3. 延迟指标: 数据包从发出到确认的时间
4. 超时指标: 超时数据包数量和趋势
5. 通道健康指标: 通道状态、连续性和吞吐量
6. 中继器指标: 中继器活跃度、gas 消耗和成功率

2.2 数据包流量指标

2.2.1 发送数据包总量

指标定义：

cosmos_sdk_ibc_channel_packet_sent_total{
    port_id="transfer",
    channel_id="channel-0",
    connection_id="connection-0",
    chain_id="msg-chain-1"
}

监控意义：反映跨链交易的活跃度，可用于计算数据包发送速率，异常下降可能表示用户活动减少或通道故障，异常飙升可能表示攻击或合约异常。

PromQL 示例：

rate(cosmos_sdk_ibc_channel_packet_sent_total{channel_id="channel-0"}[5m])

sum by (channel_id) (
    increase(cosmos_sdk_ibc_channel_packet_sent_total[24h])
)

2.2.2 接收数据包总量

cosmos_sdk_ibc_channel_packet_received_total{
    port_id="transfer",
    channel_id="channel-0",
    chain_id="msg-chain-1"
}

监控意义：反映从目标链回流的数据包数量，与 sent_total 联合计算丢包率，用于验证中继器双向转发是否正常。

PromQL：

rate(cosmos_sdk_ibc_channel_packet_received_total{channel_id="channel-0"}[5m])
/
rate(cosmos_sdk_ibc_channel_packet_sent_total{channel_id="channel-0"}[5m])

如果比率 < 0.95，可能存在数据包丢失。

2.2.3 Ack 确认总量

cosmos_sdk_ibc_channel_packet_acknowledgement_total{
    port_id="transfer",
    channel_id="channel-0",
    chain_id="msg-chain-1"
}

监控意义：成功确认的数据包数，确认数小于发送数表示存在 pending 数据包，ack 延迟直接反映中继器性能。

2.3 积压指标 (Pending Packets)

2.3.1 Pending 数据包数量

计算方式：pending_count = nextSequenceSend - nextSequenceAck - 1

自定义指标定义：

ibc_channel_packet_pending_total{
    port_id="transfer",
    channel_id="channel-0",
    chain_id="msg-chain-1"
}

PromQL：

ibc_channel_packet_pending_total
topk(10, ibc_channel_packet_pending_total)
ibc_channel_packet_pending_total > 100

2.3.2 Pending 比例指标

ibc_channel_packet_pending_total
/
(
    rate(cosmos_sdk_ibc_channel_packet_sent_total[1h])
    -
    rate(cosmos_sdk_ibc_channel_packet_acknowledgement_total[1h])
)

2.3.3 序列号 Gap 检测

序列号 gap 是指源链的 nextSequenceSend 和目标链的 nextSequenceRecv 之间存在差距。在 UNORDERED 通道上，少量的 gap 是正常的，但大量的 gap 或持续存在的 gap 表示中继器问题。

检测查询：

curl -s https://rest.msgchain.org/ibc/core/channel/v1/channels/transfer/channel-0/next_sequence_send
curl -s https://rest.target-chain.org/ibc/core/channel/v1/channels/transfer/channel-0/next_sequence_recv

PromQL：

next_sequence_send{channel_id="channel-0"}
-
next_sequence_recv{channel_id="channel-0"}

2.4 延迟指标

2.4.1 数据包端到端延迟 (E2E Latency)

定义：从用户在 MSG Chain 上发送 IBC 数据包，到数据包在目标链上被接收的时间差。

计算方式：e2e_latency = target_chain_block_time_when_received - source_chain_block_time_when_sent

影响延迟的因素：

PromQL：

avg(ibc_packet_e2e_latency_seconds{channel_id="channel-0"}) / 60

histogram_quantile(0.99,
    sum by (le) (rate(ibc_packet_e2e_latency_seconds_bucket[5m]))
)

2.4.2 Ack 确认延迟

定义：从数据包在目标链被执行，到 ack 返回源链的时间差。

监控意义：Ack 延迟高表示中继器回传不及时，可能导致用户对跨链交易状态的困惑，高延迟时通常中继器网络存在问题。

2.5 超时指标

2.5.1 超时数据包总量

cosmos_sdk_ibc_channel_packet_timeout_total{
    port_id="transfer",
    channel_id="channel-0",
    chain_id="msg-chain-1"
}

监控意义：超时数据包数量，ORDERED 通道上的超时将关闭通道，UNORDERED 通道上的超时会导致资金/消息丢失。

超时原因分析：

1. 中继器宕机: 数据包发出后无中继器转发
2. 目标链拥堵: 目标链交易积压，无法及时处理
3. Light client 过期: 无法验证目标链状态
4. Gas 不足: 中继器未设置足够 gas
5. 通道参数错误: timeout_height 或 timeout_timestamp 设置不合理

2.5.2 超时率计算

rate(cosmos_sdk_ibc_channel_packet_timeout_total[1h])
/
rate(cosmos_sdk_ibc_channel_packet_sent_total[1h]) * 100

阈值设定：

2.6 通道健康指标

2.6.1 通道状态

ibc_channel_state{
    port_id="transfer",
    channel_id="channel-0",
    state="OPEN"
}

任何非 OPEN 状态的通道都应触发告警。

2.6.2 数据包吞吐量

sum by (channel_id) (
    rate(cosmos_sdk_ibc_channel_packet_sent_total[1m])
)

sum by (channel_id) (
    rate(ibc_channel_packet_bytes_sent_total[5m])
)

2.7 中继器指标

2.7.1 核心中继器指标

2.7.2 中继器活跃度检查

time() - relayer_last_receipt_timestamp

rate(relayer_tx_success_total[1h])
/
(rate(relayer_tx_success_total[1h]) + rate(relayer_tx_failed_total[1h])) * 100

2.7.3 中继器余额监控

中继器账户必须有足够的 umsg 来支付交易费用。

msgd query bank balances msg1relayeraddress... --node https://rpc.msgchain.org:26657

PromQL：

relayer_balance{chain_id="msg-chain-1"} < 1000000

2.8 MSG Chain 特定指标

2.8.1 自定义 CosmWasm 指标

MSG Chain 上的 CosmWasm 合约可以自定义发出 IBC 相关指标：

let attrs = vec![
    attr("ibc_packet_type", "transfer"),
    attr("ibc_channel_id", "channel-0"),
    attr("packet_amount", amount.to_string()),
    attr("packet_denom", denom),
];
let event = Event::new("ibc_transfer")
    .add_attributes(attrs);
response.add_event(event);

2.8.2 跨链资产流入/流出指标

sum by (denom) (ibc_transfer_inflow_amount_total{chain_id="msg-chain-1"})
sum by (denom) (ibc_transfer_outflow_amount_total{chain_id="msg-chain-1"})
sum(ibc_transfer_inflow_amount_total) - sum(ibc_transfer_outflow_amount_total)

2.9 指标采集架构

                    MSG Chain Node
  ┌─────────────────────────────────────────────┐
  │ Cosmos SDK ABCI Metrics    IBC Module       │
  │ :26660/metrics            Prometheus Metrics │
  └──────────┬──────────────────────┬───────────┘
             │                      │
             ▼                      ▼
        ┌──────────────────────────────────┐
        │        Prometheus Server          │
        │    scrape_interval: 15s           │
        └────────────┬─────────────────────┘
                     │
                     ▼
        ┌──────────────────────────────────┐
        │          Grafana                  │
        │   Dashboards + Alerting           │
        └────────────┬─────────────────────┘
                     │
                     ▼
        ┌──────────────────────────────────┐
        │       Alertmanager               │
        │   -> Telegram/Discord/Email       │
        └──────────────────────────────────┘

3. Prometheus + Grafana 监控栈搭建

3.1 环境准备

3.1.1 系统要求

3.1.2 目录规划

/etc/prometheus/
├── prometheus.yml
├── rules/
│   ├── ibc_alerts.yml
│   ├── node_alerts.yml
│   └── relayer_alerts.yml
└── targets/
    ├── msgchain_nodes.yml
    └── relayers.yml

3.2 Prometheus 配置

3.2.1 主配置文件

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s
  external_labels:
    cluster: "msg-chain-1"
    network: "msgchain"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093
      scheme: http
      timeout: 10s
      api_version: v2

rule_files:
  - "/etc/prometheus/rules/*.yml"

scrape_configs:
  - job_name: "msgchain-validators"
    scrape_interval: 15s
    static_configs:
      - targets:
          - "validator1.msgchain.org:26660"
          - "validator2.msgchain.org:26660"
          - "validator3.msgchain.org:26660"
        labels:
          role: "validator"

  - job_name: "msgchain-rpc"
    scrape_interval: 15s
    static_configs:
      - targets:
          - "rpc.msgchain.org:26660"
        labels:
          role: "rpc"

  - job_name: "ibc-relayers"
    scrape_interval: 15s
    static_configs:
      - targets:
          - "relayer-01.msgchain.org:9090"
          - "relayer-02.msgchain.org:9090"
          - "relayer-03.msgchain.org:9090"
        labels:
          role: "relayer"

  - job_name: "ibc-exporter"
    scrape_interval: 30s
    static_configs:
      - targets:
          - "localhost:9301"
        labels:
          job: "ibc-exporter"

3.2.2 MSG Chain 节点 Prometheus 配置

在 MSG Chain 的 app.toml 中启用 Prometheus:

[telemetry]
enabled = true
prometheus-retention-time = 180
prometheus-listen-addr = "0.0.0.0:26660"
global-labels = [
  ["chain_id", "msg-chain-1"],
  ["network", "msgchain"]
]

重启节点并验证：

systemctl restart msgd
curl -s http://localhost:26660/metrics | grep ibc

3.2.3 IBC 自定义导出器

Cosmos SDK 内建的指标有限，需要运行 IBC 自定义导出器获取更细粒度的指标。

#!/usr/bin/env python3
# ibc_exporter.py

import time
import requests
from prometheus_client import start_http_server, Gauge

REST_ENDPOINT = "https://rest.msgchain.org"
CHAIN_ID = "msg-chain-1"

PACKET_PENDING = Gauge(
    "ibc_channel_packet_pending_total",
    "Pending packets on channel",
    ["port_id", "channel_id", "chain_id"]
)

CHANNEL_STATE = Gauge(
    "ibc_channel_state",
    "Channel state (0=INIT, 1=TRYOPEN, 2=OPEN, 3=CLOSED)",
    ["port_id", "channel_id", "chain_id"]
)

NEXT_SEQ_SEND = Gauge(
    "next_sequence_send", "Next sequence number to send",
    ["port_id", "channel_id", "chain_id"]
)


def fetch_channels():
    url = f"{REST_ENDPOINT}/ibc/core/channel/v1/channels"
    try:
        resp = requests.get(url, timeout=10)
        return resp.json().get("channels", [])
    except Exception as e:
        print(f"Error: {e}")
        return []


def collect_metrics():
    channels = fetch_channels()
    for ch in channels:
        port_id = ch["port_id"]
        channel_id = ch["channel_id"]
        state_map = {"STATE_INIT": 0, "STATE_TRYOPEN": 1,
                      "STATE_OPEN": 2, "STATE_CLOSED": 3}
        state = state_map.get(ch["state"], -1)
        CHANNEL_STATE.labels(
            port_id=port_id, channel_id=channel_id, chain_id=CHAIN_ID
        ).set(state)


def main():
    start_http_server(9301)
    print("IBC Exporter started on port 9301")
    while True:
        collect_metrics()
        time.sleep(30)


if __name__ == "__main__":
    main()

部署 systemd 服务：

# /etc/systemd/system/ibc-exporter.service
[Unit]
Description=IBC Custom Metrics Exporter
After=network.target

[Service]
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/ibc_exporter.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

3.3 Grafana 配置

3.3.1 数据源配置

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    editable: false

3.3.2 Dashboard 面板设计

面板 1: 通道状态总览 - Stat 类型，查询 count(ibc_channel_state{state="2"})

面板 2: 数据包流量热力图 - Heatmap 类型，查询 rate(cosmos_sdk_ibc_channel_packet_sent_total[5m])

面板 3: Pending 数据包 - Table 类型，查询 ibc_channel_packet_pending_total > 0

面板 4: 端到端延迟 - Gauge 类型，查询 avg(ibc_packet_e2e_latency_seconds)，阈值绿色<30s 黄色<60s 红色>60s

面板 5: 中继器状态 - Stat 类型，查询 count(relayer_last_receipt_timestamp > (time() - 120))

面板 6: 跨链资产流入/流出 - Bar gauge 类型

3.3.3 Dashboard Provisioning

# /etc/grafana/provisioning/dashboards/ibc.yml
apiVersion: 1

providers:
  - name: "IBC Monitoring"
    orgId: 1
    folder: "IBC"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    options:
      path: /etc/grafana/dashboards

3.4 Alertmanager 配置

# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m

route:
  receiver: "default"
  group_wait: 10s
  group_interval: 2m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: "pagerduty-critical"
      repeat_interval: 5m
    - match:
        severity: warning
      receiver: "telegram-warning"
    - match:
        channel: "ibc"
      receiver: "telegram-ibc"

receivers:
  - name: "default"
    telegram_configs:
      - bot_token: "${TELEGRAM_BOT_TOKEN}"
        chat_id: ${TELEGRAM_CHAT_ID}
        parse_mode: "HTML"

  - name: "telegram-warning"
    telegram_configs:
      - bot_token: "${TELEGRAM_BOT_TOKEN}"
        chat_id: ${TELEGRAM_CHAT_ID}

  - name: "telegram-ibc"
    telegram_configs:
      - bot_token: "${TELEGRAM_BOT_TOKEN}"
        chat_id: ${TELEGRAM_IBC_CHAT_ID}
        parse_mode: "HTML"

  - name: "pagerduty-critical"
    pagerduty_configs:
      - routing_key: "${PAGERDUTY_ROUTING_KEY}"
        severity: critical

3.5 Docker Compose 部署

version: "3.8"

services:
  prometheus:
    image: prom/prometheus:v2.53.0
    container_name: ibc-prometheus
    restart: always
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=90d"
    ports:
      - "9090:9090"

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: ibc-alertmanager
    restart: always
    volumes:
      - ./alertmanager:/etc/alertmanager
      - alertmanager_data:/alertmanager
    ports:
      - "9093:9093"

  grafana:
    image: grafana/grafana:11.1.0
    container_name: ibc-grafana
    restart: always
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
    volumes:
      - ./grafana:/etc/grafana/provisioning
      - grafana_data:/var/lib/grafana
    ports:
      - "3000:3000"

  ibc-exporter:
    build:
      context: ./ibc-exporter
    container_name: ibc-exporter
    restart: always
    ports:
      - "9301:9301"

volumes:
  prometheus_data:
  alertmanager_data:
  grafana_data:

3.6 部署步骤

git clone https://github.com/msgchain/ibc-monitoring.git
cd ibc-monitoring
cp .env.example .env
vi .env
docker-compose up -d
curl http://localhost:9090/-/ready
curl http://localhost:3000/api/health

4. 告警规则设计

4.1 告警分级体系

4.2 通道关闭告警

groups:
  - name: ibc_channel_status
    interval: 30s
    rules:
      - alert: IBCChannelClosed
        expr: ibc_channel_state{state="3"} == 1
        for: 0m
        labels:
          severity: critical
          channel: ibc
        annotations:
          summary: "IBC 通道 {{ $labels.port_id }}/{{ $labels.channel_id }} 已关闭"
          description: >
            {{ $labels.channel_id }} 处于 CLOSED 状态。
            端口: {{ $labels.port_id }}, 链: {{ $labels.chain_id }}

通道关闭恢复流程：

msgd query ibc channel end transfer channel-0 \
    --node https://rpc.msgchain.org:26657 -o json | jq '.channel.state'
journalctl -u msgd -n 100 --no-pager | grep -i "channel.*close"
curl -s https://rest.target-chain.org/ibc/core/channel/v1/channels/transfer/channel-0

4.3 Packet 堆积告警

      - alert: IBCPacketBacklog
        expr: ibc_channel_packet_pending_total > 50
        for: 2m
        labels:
          severity: critical
          channel: ibc
        annotations:
          summary: "通道 {{ $labels.channel_id }} Packet 堆积 ({{ $value }})"

      - alert: IBCPacketBacklogWarning
        expr: ibc_channel_packet_pending_total > 10
        for: 5m
        labels:
          severity: warning
          channel: ibc

4.4 中继器宕机告警

      - alert: IBCRelayerDown
        expr: time() - relayer_last_receipt_timestamp > 120
        for: 1m
        labels:
          severity: critical
          channel: ibc
        annotations:
          summary: "中继器 {{ $labels.instance }} 疑似宕机"

      - alert: IBCRelayerSuccessRateLow
        expr: rate(relayer_tx_success_total[10m]) /
              (rate(relayer_tx_success_total[10m]) +
               rate(relayer_tx_failed_total[10m])) < 0.9
        for: 5m
        labels:
          severity: warning
          channel: ibc

中继器恢复步骤：

systemctl status relayer
systemctl restart relayer
journalctl -u relayer -n 200 --no-pager | tail -50
rly tx update-client msg-chain-1
rly tx relay-packets --all
rly status

4.5 Sequence Gap 告警

      - alert: IBCSequenceGap
        expr: ibc_sequence_gap > 1000
        for: 5m
        labels:
          severity: critical
          channel: ibc
        annotations:
          summary: "通道 {{ $labels.channel_id }} Sequence Gap ({{ $value }})"

      - alert: IBCSequenceGapWarning
        expr: ibc_sequence_gap > 100
        for: 10m
        labels:
          severity: warning
          channel: ibc

4.6 数据包超时告警

      - alert: IBCPacketTimeoutRateHigh
        expr: rate(cosmos_sdk_ibc_channel_packet_timeout_total[1h]) /
              rate(cosmos_sdk_ibc_channel_packet_sent_total[1h]) > 0.01
        for: 10m
        labels:
          severity: critical
          channel: ibc

      - alert: IBCPacketTimeoutRateWarning
        expr: rate(cosmos_sdk_ibc_channel_packet_timeout_total[1h]) /
              rate(cosmos_sdk_ibc_channel_packet_sent_total[1h]) > 0.001
        for: 30m
        labels:
          severity: warning
          channel: ibc

4.7 Light Client 过期告警

      - alert: IBCClientExpired
        expr: ibc_lightclient_expiration_timestamp - time() < 3600
        for: 1m
        labels:
          severity: critical
          channel: ibc
        annotations:
          summary: "Light Client {{ $labels.client_id }} 即将过期"

      - alert: IBCClientExpiringSoon
        expr: ibc_lightclient_expiration_timestamp - time() < 86400
        for: 1h
        labels:
          severity: warning
          channel: ibc

4.8 发送量异常告警

      - alert: IBCPacketVolumeDrop
        expr: rate(cosmos_sdk_ibc_channel_packet_sent_total[1h]) /
              avg(rate(cosmos_sdk_ibc_channel_packet_sent_total[7d])) < 0.5
        for: 30m
        labels:
          severity: warning

      - alert: IBCPacketVolumeSpike
        expr: rate(cosmos_sdk_ibc_channel_packet_sent_total[5m]) /
              avg(rate(cosmos_sdk_ibc_channel_packet_sent_total[1h])) > 5
        for: 5m
        labels:
          severity: warning

4.9 告警规则验证

promtool check rules /etc/prometheus/rules/ibc_alerts.yml
promtool test rules /etc/prometheus/rules/ibc_alerts_test.yaml
curl -X POST http://localhost:9090/-/reload

5. IBC 事件订阅架构

5.1 概述

IBC 事件订阅架构允许实时监听链上 IBC 相关事件，无需轮询 RPC。通过 WebSocket 连接，可以订阅特定的事件类型，实现近乎实时的通道监控。

5.2 Cosmos SDK 事件系统

Cosmos SDK 的事件系统基于 ABCI (Tendermint) 事件。每个交易可以发出多个事件，每个事件有类型和属性。

IBC 相关事件类型：

5.3 WebSocket 订阅实现

5.3.1 基础订阅

wscat -c wss://rpc.msgchain.org:26657/websocket

连接后发送订阅请求：

{
    "jsonrpc": "2.0",
    "method": "subscribe",
    "id": "1",
    "params": {
        "query": "tm.event='Tx' AND send_packet EXISTS"
    }
}

5.3.2 常用查询过滤

{"query": "send_packet EXISTS OR recv_packet EXISTS OR acknowledge_packet EXISTS"}
{"query": "send_packet.packet_src_channel='channel-0' AND send_packet.packet_src_port='transfer'"}
{"query": "channel_close_init EXISTS OR channel_open_init EXISTS"}

5.3.3 Python 事件监听器

#!/usr/bin/env python3
# ibc_event_listener.py

import asyncio
import json
import websockets
import logging

logging.basicConfig(level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)


class IBCEventListener:
    def __init__(self, ws_url="wss://rpc.msgchain.org:26657/websocket",
                 max_retries=10):
        self.ws_url = ws_url
        self.max_retries = max_retries
        self.request_id = 0
        self.handlers = {}
        self.running = False

    def subscribe(self, event_type, handler):
        if event_type not in self.handlers:
            self.handlers[event_type] = []
        self.handlers[event_type].append(handler)

    async def _handle_message(self, message):
        try:
            data = json.loads(message)
            if "result" in data and "events" in data.get("result", {}):
                events = (data["result"]["data"]["value"]["TxResult"]
                          ["result"]["events"])
                for event in events:
                    etype = event["type"]
                    if etype in self.handlers:
                        attrs = {a["key"]: a["value"]
                                 for a in event["attributes"]}
                        for handler in self.handlers[etype]:
                            await handler(etype, attrs)
        except Exception as e:
            logger.error(f"Error: {e}")

    async def connect(self):
        retries = 0
        while retries < self.max_retries:
            try:
                async with websockets.connect(self.ws_url) as ws:
                    self.running = True
                    retries = 0
                    queries = [
                        "tm.event='Tx' AND send_packet EXISTS",
                        "tm.event='Tx' AND recv_packet EXISTS",
                        "tm.event='Tx' AND acknowledge_packet EXISTS",
                        "tm.event='Tx' AND timeout_packet EXISTS",
                        "tm.event='Tx' AND channel_close_init EXISTS",
                    ]
                    for q in queries:
                        msg = {"jsonrpc": "2.0", "method": "subscribe",
                               "id": str(self.request_id), "params": {"query": q}}
                        await ws.send(json.dumps(msg))
                    async for message in ws:
                        await self._handle_message(message)
            except websockets.ConnectionClosed:
                logger.warning("Connection closed")
            except Exception as e:
                logger.error(f"Error: {e}")
            retries += 1
            if retries >= self.max_retries:
                break
            await asyncio.sleep(retries * 5)

    async def run(self):
        await self.connect()


async def handle_channel_close(event_type, attrs):
    logger.warning(f"CHANNEL CLOSED: {attrs.get('port_id')}/{attrs.get('channel_id')}")


async def main():
    listener = IBCEventListener()
    listener.subscribe("channel_close_init", handle_channel_close)
    listener.subscribe("channel_close_confirm", handle_channel_close)
    await listener.run()


if __name__ == "__main__":
    asyncio.run(main())

5.4 与 MSG Chain Indexer 集成

5.4.1 Indexer 架构

MSG Chain 的 indexer 将链上数据索引到 PostgreSQL 和 Elasticsearch。

架构：

MSG Chain Node ----> Indexer ----> PostgreSQL
                      |              |
                      v              v
                 Prometheus     Elasticsearch

5.4.2 Indexer 配置

# indexer/config.yaml
chain:
  chain_id: msg-chain-1
  rpc_endpoint: "https://rpc.msgchain.org:26657"
  ws_endpoint: "wss://rpc.msgchain.org:26657/websocket"
  rest_endpoint: "https://rest.msgchain.org"

database:
  type: postgresql
  host: ${PG_HOST:-localhost}
  port: ${PG_PORT:-5432}
  user: ${PG_USER:-msgchain}
  password: ${PG_PASSWORD:-secret}
  database: ${PG_DB:-ibc_indexer}

5.4.3 数据库 Schema

CREATE TABLE IF NOT EXISTS ibc_events (
    id              BIGSERIAL PRIMARY KEY,
    event_type      VARCHAR(64) NOT NULL,
    chain_id        VARCHAR(64) NOT NULL DEFAULT 'msg-chain-1',
    block_height    BIGINT NOT NULL,
    block_time      TIMESTAMPTZ NOT NULL,
    tx_hash         VARCHAR(128) NOT NULL,
    port_id         VARCHAR(128),
    channel_id      VARCHAR(128),
    packet_sequence BIGINT,
    packet_data     TEXT,
    sender          VARCHAR(128),
    receiver        VARCHAR(128),
    amount          NUMERIC(78, 0),
    denom           VARCHAR(128),
    raw_data        JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_ibc_events_event_type ON ibc_events(event_type);
CREATE INDEX idx_ibc_events_channel ON ibc_events(channel_id, port_id);
CREATE INDEX idx_ibc_events_block_time ON ibc_events(block_time DESC);
CREATE INDEX idx_ibc_events_sequence ON ibc_events(channel_id, packet_sequence);

5.5 事件驱动的告警触发

# event_alerts.py

import aiohttp
from datetime import datetime


class EventDrivenAlerts:
    def __init__(self, webhook_urls: dict):
        self.webhook_urls = webhook_urls
        self.session = aiohttp.ClientSession()

    async def send_alert(self, alert_type: str, severity: str, data: dict):
        msg = {
            "alert_name": alert_type,
            "severity": severity,
            "timestamp": datetime.utcnow().isoformat(),
            "data": data
        }
        if "telegram" in self.webhook_urls:
            await self._send_telegram(msg)
        if "pagerduty" in self.webhook_urls:
            await self._send_pagerduty(msg)

    async def _send_telegram(self, message: dict):
        text = f"ALERT: {message['alert_name']} - {message['severity']}"
        url = (f"https://api.telegram.org/bot"
               f"{self.webhook_urls['telegram']['bot_token']}/sendMessage")
        payload = {"chat_id": self.webhook_urls["telegram"]["chat_id"],
                    "text": text}
        async with self.session.post(url, json=payload) as resp:
            print(f"Telegram: {resp.status}")

    async def _send_pagerduty(self, message: dict):
        payload = {
            "routing_key": self.webhook_urls["pagerduty"]["routing_key"],
            "event_action": "trigger",
            "payload": {
                "summary": message["alert_name"],
                "severity": message["severity"],
                "source": "ibc-monitor"
            }
        }
        async with self.session.post(
            "https://events.pagerduty.com/v2/enqueue",
            json=payload
        ) as resp:
            print(f"PagerDuty: {resp.status}")

6. 跨链交易追踪

6.1 概述

跨链交易追踪是 IBC 通道监控的核心能力之一。一条完整的 IBC 交易从源链（MSG Chain）发出，经过中继器转发到目标链，最终 ack 返回源链。任何一个环节的故障都会导致交易无法完成。

6.2 全链路追踪流程

T0: 用户在 MSG Chain 提交 IBC 转账交易
    事件: send_packet (packet_sequence=N)

T1: 中继器检测到 send_packet 事件
    创建 MsgRecvPacket 交易并提交到目标链

T2: 目标链处理接收交易
    事件: recv_packet (sequence=N)
    事件: write_acknowledgement

T3: 中继器检测到 write_acknowledgement
    创建 MsgAcknowledgement 交易提交回 MSG Chain

T4: MSG Chain 处理 Ack
    事件: acknowledge_packet (sequence=N)
    交易完成

6.3 交易追踪工具

#!/bin/bash
# ibc-trace.sh

set -euo pipefail

MSG_REST="https://rest.msgchain.org"
TX_HASH="${1:-}"

if [ -z "$TX_HASH" ]; then
    echo "Usage: $0 <tx_hash>"
    exit 1
fi

echo "IBC 交易追踪 - MSG Chain"
echo "交易哈希: $TX_HASH"
echo ""

echo "[1/4] 查询源链交易..."
TX_DATA=$(curl -s "$MSG_REST/cosmos/tx/v1beta1/txs/$TX_HASH")
TX_HEIGHT=$(echo "$TX_DATA" | jq -r '.tx_response.height')
TX_CODE=$(echo "$TX_DATA" | jq -r '.tx_response.code')
if [ "$TX_CODE" != "0" ]; then
    echo "交易失败 (code=$TX_CODE)"
    exit 1
fi
echo "交易成功 (height=$TX_HEIGHT)"

echo "[2/4] 提取 IBC 事件..."
EVENTS=$(echo "$TX_DATA" | jq -c '.tx_response.events[]')
SEND_PACKET=$(echo "$EVENTS" | jq -r 'select(.type=="send_packet")')
if [ -z "$SEND_PACKET" ]; then
    echo "未找到 send_packet 事件"
    exit 1
fi
PACKET_SEQUENCE=$(echo "$SEND_PACKET" | jq -r \
    '.attributes[] | select(.key=="packet_sequence") | .value')
SRC_CHANNEL=$(echo "$SEND_PACKET" | jq -r \
    '.attributes[] | select(.key=="packet_src_channel") | .value')
DST_CHANNEL=$(echo "$SEND_PACKET" | jq -r \
    '.attributes[] | select(.key=="packet_dst_channel") | .value')
echo "序列号: $PACKET_SEQUENCE, 源通道: $SRC_CHANNEL, 目标通道: $DST_CHANNEL"

echo "[3/4] 检查 Ack..."
ACK_RESP=$(curl -s "$MSG_REST/cosmos/tx/v1beta1/txs" \
    --data-urlencode "events=acknowledge_packet.packet_sequence=$PACKET_SEQUENCE" -G)
ACK_TX=$(echo "$ACK_RESP" | jq -r '.txs[0].tx_response.txhash // "not_found"')
if [ "$ACK_TX" != "not_found" ]; then
    echo "Ack 已返回: $ACK_TX"
    echo "交易状态: 已完成"
else
    echo "Ack 尚未返回"
    echo "交易状态: Pending"
fi

6.4 追踪数据模型

CREATE TABLE IF NOT EXISTS ibc_tx_traces (
    id                  BIGSERIAL PRIMARY KEY,
    packet_sequence     BIGINT NOT NULL,
    source_chain_id     VARCHAR(64) NOT NULL DEFAULT 'msg-chain-1',
    source_channel_id   VARCHAR(128) NOT NULL,
    source_tx_hash      VARCHAR(128) NOT NULL,
    source_block_height BIGINT NOT NULL,
    destination_channel_id VARCHAR(128),
    destination_tx_hash    VARCHAR(128),
    ack_tx_hash         VARCHAR(128),
    timeout_height      BIGINT,
    is_timeout          BOOLEAN DEFAULT FALSE,
    sender              VARCHAR(128),
    receiver            VARCHAR(128),
    amount              NUMERIC(78, 0),
    denom               VARCHAR(128),
    status              VARCHAR(32) NOT NULL DEFAULT 'pending',
    e2e_latency_seconds DOUBLE PRECISION,
    created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at          TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_traces_status ON ibc_tx_traces(status);
CREATE INDEX idx_traces_source_tx ON ibc_tx_traces(source_tx_hash);
CREATE INDEX idx_traces_channel ON ibc_tx_traces(source_channel_id, packet_sequence);

6.5 自动追踪实现

#!/usr/bin/env python3
# ibc_tx_tracker.py

import asyncio
import json
import logging
import asyncpg
import websockets

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger("ibc-tracker")


class IBCTxTracker:
    def __init__(self, pg_dsn: str,
                 source_rpc: str = "wss://rpc.msgchain.org:26657"):
        self.pg_dsn = pg_dsn
        self.source_rpc = source_rpc
        self.pool = None

    async def start(self):
        self.pool = await asyncpg.create_pool(self.pg_dsn, min_size=5)
        await self._ensure_tables()
        await self._event_loop()

    async def _ensure_tables(self):
        async with self.pool.acquire() as conn:
            await conn.execute("""
                CREATE TABLE IF NOT EXISTS ibc_tx_traces (
                    id BIGSERIAL PRIMARY KEY,
                    packet_sequence BIGINT,
                    source_tx_hash VARCHAR(128),
                    source_channel_id VARCHAR(128),
                    status VARCHAR(32) DEFAULT 'pending',
                    created_at TIMESTAMPTZ DEFAULT NOW()
                )
            """)

    async def _event_loop(self):
        query = ("tm.event='Tx' AND "
                 "(send_packet EXISTS OR acknowledge_packet EXISTS)")
        async with websockets.connect(self.source_rpc) as ws:
            sub = {"jsonrpc": "2.0", "method": "subscribe",
                   "id": "1", "params": {"query": query}}
            await ws.send(json.dumps(sub))
            async for message in ws:
                data = json.loads(message)
                events = (data.get("result", {}).get("data", {})
                          .get("value", {}).get("TxResult", {})
                          .get("result", {}).get("events", []))
                await self._process_events(events)

    async def _process_events(self, events: list):
        for event in events:
            attrs = {a["key"]: a["value"] for a in event.get("attributes", [])}
            if event["type"] == "send_packet":
                await self._handle_send(attrs)
            elif event["type"] == "acknowledge_packet":
                await self._handle_ack(attrs)

    async def _handle_send(self, attrs: dict):
        async with self.pool.acquire() as conn:
            await conn.execute("""
                INSERT INTO ibc_tx_traces
                    (packet_sequence, source_channel_id, status)
                VALUES ($1, $2, 'pending')
                ON CONFLICT DO NOTHING
            """, int(attrs.get("packet_sequence", 0)),
                attrs.get("packet_src_channel", ""))

    async def _handle_ack(self, attrs: dict):
        async with self.pool.acquire() as conn:
            await conn.execute("""
                UPDATE ibc_tx_traces
                SET status = 'completed', updated_at = NOW(),
                    e2e_latency_seconds = EXTRACT(EPOCH FROM NOW() - created_at)
                WHERE packet_sequence = $1 AND source_channel_id = $2
            """, int(attrs.get("packet_sequence", 0)),
                attrs.get("packet_src_channel", ""))

7. 自动化通道健康检查

7.1 概述

自动化健康检查是 IBC 通道监控的最后一道防线。通过定期执行 Ping/Pong 测试、状态断言和连通性验证，可以主动发现通道的潜在问题。

7.2 Ping/Pong 机制

7.2.1 原理

IBC Ping/Pong 是在两个链之间发送小型 IBC 数据包，验证通道的双向可达性。

MSG Chain                          Target Chain
    │                                    │
    │  ── Ping packet (sequence N) ────> │
    │                                    │  接收 Ping
    │  <── Pong ack (sequence N) ────── │
    │                                    │

7.2.2 智能合约实现

use cosmwasm_std::{
    entry_point, to_json_binary, DepsMut, Env, IbcBasicResponse,
    IbcChannelConnectMsg, IbcPacketAckMsg, IbcPacketReceiveMsg,
    IbcPacketTimeoutMsg, IbcReceiveResponse, MessageInfo, Response,
};
use crate::msg::{ExecuteMsg, IbcExecuteMsg};

const CONFIG_KEY: &[u8] = b"config";

#[entry_point]
pub fn instantiate(
    deps: DepsMut,
    env: Env,
    info: MessageInfo,
    msg: InstantiateMsg,
) -> Result<Response, ContractError> {
    Ok(Response::new()
        .add_attribute("action", "instantiate"))
}

#[entry_point]
pub fn execute(
    deps: DepsMut,
    env: Env,
    info: MessageInfo,
    msg: ExecuteMsg,
) -> Result<Response, ContractError> {
    match msg {
        ExecuteMsg::SendPing {} => execute_send_ping(deps, env, info),
    }
}

fn execute_send_ping(
    deps: DepsMut,
    env: Env,
    info: MessageInfo,
) -> Result<Response, ContractError> {
    let ping_msg = IbcExecuteMsg::Ping {
        id: 1,
        timestamp: env.block.time.seconds(),
    };
    let msg = cosmwasm_std::IbcMsg::SendPacket {
        channel_id: "channel-0".to_string(),
        data: to_json_binary(&ping_msg)?,
        timeout: cosmwasm_std::IbcTimeout::with_timestamp(
            env.block.time.plus_seconds(300),
        ),
    };
    Ok(Response::new()
        .add_message(msg)
        .add_attribute("action", "send_ping"))
}

#[entry_point]
pub fn ibc_packet_receive(
    deps: DepsMut,
    env: Env,
    msg: IbcPacketReceiveMsg,
) -> Result<IbcReceiveResponse, ContractError> {
    let execute_msg: IbcExecuteMsg =
        cosmwasm_std::from_json(&msg.packet.data)?;
    match execute_msg {
        IbcExecuteMsg::Ping { id, timestamp } => {
            let pong = IbcExecuteMsg::Pong {
                ping_id: id,
                pong_timestamp: env.block.time.seconds(),
                original_timestamp: timestamp,
            };
            Ok(IbcReceiveResponse::new()
                .set_ack(to_json_binary(&pong)?)
                .add_attribute("action", "pong"))
        }
        _ => Ok(IbcReceiveResponse::default()),
    }
}

7.3 通道健康检查脚本

#!/bin/bash
# ibc-healthcheck.sh

set -euo pipefail

MSG_REST="https://rest.msgchain.org"
LOG_FILE="/var/log/ibc-healthcheck.log"

declare -A CHANNELS
CHANNELS["transfer/channel-0"]="cosmos-hub"
CHANNELS["transfer/channel-1"]="osmosis-1"
CHANNELS["transfer/channel-2"]="juno-1"
CHANNELS["transfer/channel-3"]="secret-4"
CHANNELS["transfer/channel-4"]="axelar-dojo-1"
CHANNELS["transfer/channel-5"]="neutron-1"

log() {
    local level="$1"; shift
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $*" | tee -a "$LOG_FILE"
}

send_alert() {
    log "ALERT" "$*"
}

check_channel_state() {
    local port_id="$1" channel_id="$2" counterparty="$3"
    log "INFO" "检查通道: $port_id/$channel_id (-> $counterparty)"
    local state=$(curl -s "$MSG_REST/ibc/core/channel/v1/channels/$channel_id/ports/$port_id" \
        | jq -r '.channel.state // "UNKNOWN"')
    case "$state" in
        "STATE_OPEN") log "OK" "  通道状态: OPEN"; return 0 ;;
        "STATE_CLOSED") send_alert "CHANNEL_CLOSED: $port_id/$channel_id"; return 1 ;;
        *) log "WARN" "  通道状态: $state"; return 2 ;;
    esac
}

check_sequence_gap() {
    local port_id="$1" channel_id="$2"
    local next=$(curl -s "$MSG_REST/ibc/core/channel/v1/channels/$channel_id/ports/$port_id/next_sequence_send" \
        | jq -r '.next_sequence_send // "0"')
    local last=$(curl -s "$MSG_REST/ibc/core/channel/v1/channels/$channel_id/ports/$port_id/packet_acknowledgements" \
        | jq -r '.acknowledgements[-1].sequence // "0"')
    local gap=$((next - last - 1))
    if [ "$gap" -gt 100 ]; then
        log "WARN" "  Sequence gap: $gap"
        [ "$gap" -gt 1000 ] && send_alert "SEQUENCE_GAP: $port_id/$channel_id gap=$gap"
    else
        log "OK" "  序列号连续"
    fi
}

main() {
    log "INFO" "IBC 通道健康检查开始"
    for key in "${!CHANNELS[@]}"; do
        check_channel_state "${key%%/*}" "${key##*/}" "${CHANNELS[$key]}" || true
        check_sequence_gap "${key%%/*}" "${key##*/}"
        echo ""
    done
    log "INFO" "健康检查完成"
}

main "$@"

7.4 定期调度

# /etc/cron.d/ibc-healthcheck
*/5 * * * * root /usr/local/bin/ibc-healthcheck.sh >> /var/log/ibc-healthcheck.log 2>&1
*/15 * * * * root /usr/local/bin/ibc-ping-test.sh >> /var/log/ibc-ping.log 2>&1
0 * * * * root /usr/local/bin/ibc-health-summary.sh | telegram-send --stdin

8. 告警通知集成

8.1 Telegram 集成

8.1.1 创建 Bot

# 在 Telegram 中与 @BotFather 对话创建 Bot
# 获取 Bot Token
# 获取 Chat ID
curl -s "https://api.telegram.org/bot<YOUR_BOT_TOKEN>/getUpdates" | jq '.result[].message.chat.id'

8.1.2 Alertmanager 配置

receivers:
  - name: "telegram-ibc"
    telegram_configs:
      - bot_token: "${TELEGRAM_BOT_TOKEN}"
        chat_id: ${TELEGRAM_CHAT_ID}
        message: |
          *IBC 告警: {{ .GroupLabels.alertname }}*
          通道: {{ index .GroupLabels "channel_id" }}
          级别: {{ .GroupLabels.severity }}
        parse_mode: "Markdown"

8.1.3 自定义通知脚本

#!/usr/bin/env python3
# telegram_alert.py

import requests
import sys


def send_telegram_alert(bot_token: str, chat_id: str, message: str):
    url = f"https://api.telegram.org/bot{bot_token}/sendMessage"
    payload = {
        "chat_id": chat_id,
        "text": message,
        "parse_mode": "HTML",
        "disable_web_page_preview": True
    }
    try:
        resp = requests.post(url, json=payload, timeout=10)
        resp.raise_for_status()
        print(f"Telegram alert sent: {resp.status_code}")
    except Exception as e:
        print(f"Telegram alert failed: {e}")


if __name__ == "__main__":
    send_telegram_alert(sys.argv[1], sys.argv[2], sys.argv[3])

8.2 Discord 集成

receivers:
  - name: "discord-ibc"
    discord_configs:
      - webhook_url: "${DISCORD_WEBHOOK_URL}"
        message: |
          **IBC 告警: {{ .GroupLabels.alertname }}**
          通道: {{ index .GroupLabels "channel_id" }}
          级别: {{ .GroupLabels.severity }}

8.3 PagerDuty 集成

receivers:
  - name: "pagerduty-critical"
    pagerduty_configs:
      - routing_key: "${PAGERDUTY_ROUTING_KEY}"
        severity: critical
        client: "IBC Monitor"
        client_url: "https://msgchain.org/monitoring"

8.4 邮件集成

receivers:
  - name: "email-ibc"
    email_configs:
      - to: "ibc-alerts@msgchain.org"
        from: "alertmanager@msgchain.org"
        smarthost: "smtp.msgchain.org:587"
        auth_username: "alertmanager@msgchain.org"
        auth_password: "${SMTP_PASSWORD}"
        headers:
          subject: "[IBC] {{ .GroupLabels.alertname }}"
        html: '<h2>IBC 告警: {{ .GroupLabels.alertname }}</h2>'

8.5 多渠道通知架构

                    Alertmanager
                         │
               ┌─────────┴─────────┐
               │                   │
         ┌─────▼─────┐      ┌──────▼──────┐
         │ severity=  │      │ severity=   │
         │ critical   │      │ warning     │
         └─────┬─────┘      └──────┬──────┘
               │                   │
    ┌──────────┼──────────┐       │
    │          │          │       │
┌───▼───┐ ┌───▼────┐ ┌───▼───┐ ┌─▼───────┐
│PagerDuty│ │Telegram│ │ 邮件  │ │ Discord │
└────────┘ └────────┘ └───────┘ └─────────┘

8.6 通知转发脚本

#!/usr/bin/env python3
# alert_router.py

import os
import json
import sys
import requests


class AlertRouter:
    def __init__(self):
        self.telegram_token = os.getenv("TELEGRAM_BOT_TOKEN")
        self.telegram_chat = os.getenv("TELEGRAM_CHAT_ID")
        self.discord_webhook = os.getenv("DISCORD_WEBHOOK_URL")
        self.pagerduty_key = os.getenv("PAGERDUTY_ROUTING_KEY")

    def route(self, alert: dict):
        sev = alert.get("severity", "info")
        if sev == "critical":
            self._send_pagerduty(alert)
            self._send_telegram(alert)
        elif sev == "warning":
            self._send_telegram(alert)
            self._send_discord(alert)
        else:
            self._send_discord(alert)

    def _send_telegram(self, alert: dict):
        if not self.telegram_token:
            return
        text = (f"<b>IBC {alert.get('alertname', 'Alert')}</b>\n"
                f"Severity: {alert.get('severity')}\n"
                f"Channel: {alert.get('channel_id', 'N/A')}")
        requests.post(
            f"https://api.telegram.org/bot{self.telegram_token}/sendMessage",
            json={"chat_id": self.telegram_chat, "text": text,
                  "parse_mode": "HTML"}, timeout=10
        )

    def _send_discord(self, alert: dict):
        if not self.discord_webhook:
            return
        embed = {
            "title": f"{alert.get('severity')}: {alert.get('alertname')}",
            "color": 0xFF0000 if alert.get("severity") == "critical" else 0xFFA500,
            "fields": [
                {"name": "Channel", "value": alert.get("channel_id", "N/A"),
                 "inline": True}
            ]
        }
        requests.post(self.discord_webhook, json={"embeds": [embed]}, timeout=10)

    def _send_pagerduty(self, alert: dict):
        if not self.pagerduty_key:
            return
        payload = {
            "routing_key": self.pagerduty_key,
            "event_action": "trigger",
            "payload": {
                "summary": alert.get("alertname", "IBC Alert"),
                "severity": "critical",
                "source": f"ibc-{alert.get('channel_id', 'unknown')}",
                "custom_details": alert
            }
        }
        requests.post("https://events.pagerduty.com/v2/enqueue",
                      json=payload, timeout=10)


if __name__ == "__main__":
    router = AlertRouter()
    router.route(json.loads(sys.stdin.read()))

9. 与 MSG Chain 可观测性栈的集成

9.1 MSG Chain 可观测性栈概述

9.2 OpenTelemetry 集成

9.2.1 IBC 指标的 OpenTelemetry 导出

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

  prometheus:
    config:
      scrape_configs:
        - job_name: "ibc-metrics"
          scrape_interval: 15s
          static_configs:
            - targets:
                - "localhost:26660"
                - "localhost:9301"

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  attributes:
    actions:
      - key: chain_id
        value: msg-chain-1
        action: upsert

exporters:
  prometheus:
    endpoint: "0.0.0.0:8889"
    namespace: "ibc"
  otlp:
    endpoint: "tempo:4317"
    tls:
      insecure: true
  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch, attributes]
      exporters: [prometheus, otlp, logging]

9.2.2 与 Prometheus 联邦

# prometheus-federation.yml
scrape_configs:
  - job_name: "federate-ibc"
    scrape_interval: 15s
    honor_labels: true
    metrics_path: "/federate"
    params:
      match[]:
        - '{__name__=~".*ibc.*"}'
        - '{__name__=~".*relayer.*"}'
    static_configs:
      - targets:
          - "msgchain-prometheus.internal:9090"

9.3 Loki 日志集成

# promtail-config.yaml
scrape_configs:
  - job_name: "relayer-logs"
    static_configs:
      - targets:
          - localhost
        labels:
          job: "ibc-relayer"
          chain: "msg-chain-1"
          __path__: "/var/log/relayer/*.log"
    pipeline_stages:
      - regex:
          expression: "^(?P<timestamp>\\S+)\\s+(?P<level>\\w+)\\s+(?P<message>.*)"
      - labels:
          level:
      - metrics:
          relayer_log_lines_total:
            type: Counter
            description: "Total relayer log lines"
            prefix: "ibc_"
            max_idle_dur: 24h
            config:
              match_source: ".*error.*"
              action: inc

9.4 自定义 Grafana Dashboard

{
    "dashboard": {
        "title": "IBC 通道监控总览 - MSG Chain",
        "tags": ["ibc", "msg-chain", "cosmos"],
        "timezone": "utc",
        "panels": [
            {
                "title": "通道状态",
                "type": "stat",
                "targets": [{
                    "expr": "count(ibc_channel_state{state=\"2\"})"
                }]
            },
            {
                "title": "数据包发送速率 (1m)",
                "type": "graph",
                "targets": [{
                    "expr": "sum(rate(cosmos_sdk_ibc_channel_packet_sent_total[1m])) by (channel_id)"
                }]
            },
            {
                "title": "Pending 数据包",
                "type": "table",
                "targets": [{
                    "expr": "ibc_channel_packet_pending_total > 0"
                }]
            },
            {
                "title": "端到端延迟 (P99)",
                "type": "gauge",
                "targets": [{
                    "expr": "histogram_quantile(0.99, sum(rate(ibc_packet_e2e_latency_seconds_bucket[5m])) by (le))"
                }]
            },
            {
                "title": "中继器状态",
                "type": "stat",
                "targets": [{
                    "expr": "count(relayer_last_receipt_timestamp > (time() - 120))"
                }]
            }
        ]
    }
}

9.5 Tempo 分布式追踪

在 IBC 交易处理中注入追踪上下文可以实现全链路的分布式追踪。

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
    "go.opentelemetry.io/otel/attribute"
)

func handleIBCTx(ctx context.Context, packet IBCPacket) {
    tracer := otel.Tracer("ibc-tracker")
    ctx, span := tracer.Start(ctx, "ibc-packet-flow",
        trace.WithAttributes(
            attribute.String("packet.sequence", packet.Sequence),
            attribute.String("packet.channel", packet.Channel),
        ),
    )
    defer span.End()

    // 阶段 1: 发送
    sendCtx, sendSpan := tracer.Start(ctx, "packet-send")
    sendPacket(sendCtx, packet)
    sendSpan.End()

    // 阶段 2: 中继
    relayCtx, relaySpan := tracer.Start(ctx, "packet-relay")
    relayPacket(relayCtx, packet)
    relaySpan.End()

    // 阶段 3: 确认
    ackCtx, ackSpan := tracer.Start(ctx, "packet-ack")
    waitForAck(ackCtx, packet)
    ackSpan.End()
}

9.6 告警仪表盘

在 Grafana 中创建告警仪表盘，统一管理所有 IBC 告警规则：

9.7 告警规则与 Grafana 原生告警

Grafana 原生告警可以替代或补充 Alertmanager：

# grafana-alerting.yml
apiVersion: 1

contactPoints:
  - name: "ibc-telegram"
    receivers:
      - uid: "telegram-ibc"
        type: telegram
        settings:
          bottoken: "${TELEGRAM_BOT_TOKEN}"
          chatid: ${TELEGRAM_CHAT_ID}

policies:
  - orgId: 1
    receiver: "ibc-telegram"
    group_by: ["alertname", "channel_id"]
    group_wait: 10s
    group_interval: 2m
    repeat_interval: 4h

10. 案例：典型 IBC 故障场景的监控与恢复

10.1 案例一：中继器宕机导致通道关闭

10.1.1 场景描述

某日凌晨 3:00，MSG Chain 到 Cosmos Hub 的 IBC 通道（channel-0）上的中继器因 OOM 被系统 OOM Killer 终止，导致数据包无法转发。由于该通道是 ORDERED 通道，一个数据包超时导致通道自动关闭。

10.1.2 告警链

03:02  P0  IBCRelayerDown      中继器最后活动时间 > 120 秒
03:05  P1  IBCPacketTimeout    检测到超时数据包
03:05  P0  IBCChannelClosed    通道状态变为 CLOSED
03:06  P1  IBCPacketBacklog    积压数据包 > 50

10.1.3 监控日志

03:00:12  [INFO]  中继器心跳超时
03:00:15  [WARN]  中继器进程异常退出 (OOM)
03:00:30  [WARN]  检测到 send_packet 未确认序列号: 1280-1290
03:01:00  [ERROR] 数据包超时: sequence=1285, channel=channel-0
03:01:15  [CRITICAL] 通道关闭: transfer/channel-0 -> CLOSED

10.1.4 恢复过程

# 1. 确认问题
curl -s https://rest.msgchain.org/ibc/core/channel/v1/channels/transfer/channel-0 | jq '.channel.state'
# "STATE_CLOSED"

# 2. 检查中继器服务器资源
free -m
df -h
journalctl -u relayer --since "03:00" --no-pager | tail -50

# 3. 重启中继器并增加资源限制
systemctl restart relayer

# 4. 检查中继器状态
rly status

# 5. 重建通道（ORDERED 通道需要重建）
# 创建治理提案
cat > channel-reopen-proposal.json << EOF
{
    "title": "Re-open IBC channel-0",
    "description": "通道因中继器 OOM 关闭，需要重建",
    "channel": {
        "port_id": "transfer",
        "channel_id": "channel-0",
        "counterparty": {
            "port_id": "transfer",
            "channel_id": "channel-0"
        },
        "connection_hops": ["connection-0"],
        "version": "ics20-1",
        "ordering": "ORDERED"
    },
    "deposit": "50000000umsg"
}
EOF

msgd tx gov submit-proposal channel-reopen-proposal.json \
    --from validator-key --chain-id msg-chain-1 \
    --node https://rpc.msgchain.org:26657 --fees 100000umsg

# 6. 验证通道恢复
sleep 30
curl -s https://rest.msgchain.org/ibc/core/channel/v1/channels/transfer/channel-0 | jq '.channel.state'

# 7. 恢复积压的数据包
rly tx relay-packets transfer channel-0

10.1.5 事后改进

1. 为中继器添加 systemd OOM 保护：

[Service]
OOMScoreAdjust=-500
MemoryMax=2G
MemoryHigh=1.5G
Restart=always
RestartSec=5

2. 增加中继器冗余节点，每个通道至少 2 个独立中继器

10.2 案例二：Light Client 过期

10.2.1 场景描述

MSG Chain 到 Osmosis 的 Light Client 未及时更新，导致中继器无法验证 Osmosis 的网络状态，所有 IBC 交易超时。

10.2.2 告警链

12:00  P2  IBCClientExpiringSoon  Light Client 将在 1h 后过期
13:00  P0  IBCClientExpired       Light Client 已过期
13:01  P1  IBCPacketTimeout       检测到数据包超时

10.2.3 恢复过程

# 1. 确认 Light Client 状态
msgd query ibc client state 07-tendermint-0 \
    --node https://rpc.msgchain.org:26657 -o json | jq '.client_state'

# 2. 手动更新 Light Client
rly tx update-client msg-chain-1

# 3. 验证更新
msgd query ibc client state 07-tendermint-0 \
    --node https://rpc.msgchain.org:26657 -o json | jq '.client_state.latest_height'

# 4. 恢复 pending 数据包
rly tx relay-packets --all

# 5. 设置定时任务自动更新
echo "*/30 * * * * root rly tx update-client msg-chain-1" > /etc/cron.d/update-light-client

10.3 案例三：Sequence Gap 导致资产卡住

10.3.1 场景描述

用户向 MSG Chain 发送 IBC 转账，交易在源链成功但目标链未收到。Sequence gap 持续增大到 5000+。

10.3.2 告警链

14:00  P2  IBCSequenceGapWarning    Gap=150
15:00  P2  IBCSequenceGapWarning    Gap=800
15:30  P1  IBCSequenceGap           Gap=5000
15:31  P1  IBCPacketBacklog         Pending > 50

10.3.3 诊断

# 1. 检查 gap
SEQ_SEND=$(curl -s https://rest.msgchain.org/ibc/core/channel/v1/channels/transfer/channel-0/next_sequence_send | jq -r '.next_sequence_send')
SEQ_RECV=$(curl -s https://rest.osmosis.org/ibc/core/channel/v1/channels/transfer/channel-0/next_sequence_recv | jq -r '.next_sequence_recv')
echo "Gap: $((SEQ_SEND - SEQ_RECV))"

# 2. 检查中继器日志
journalctl -u relayer --since "14:00" --no-pager | grep -i "error\|fail\|timeout"

# 3. 发现: 中继器 gas 设置过低
# gas-prices = "0.005umsg" -> 调整为 "1000000000umsg"

10.3.4 恢复

# 1. 调整中继器 gas 配置
sed -i 's/gas-prices = "0.005umsg"/gas-prices = "1000000000umsg"/' ~/.relayer/config/config.yaml

# 2. 重启中继器
systemctl restart relayer

# 3. 强制中继所有积压
rly tx relay-packets transfer channel-0
rly tx relay-acks transfer channel-0

# 4. 验证 gap 缩小
sleep 60
SEQ_SEND=$(curl -s https://rest.msgchain.org/ibc/core/channel/v1/channels/transfer/channel-0/next_sequence_send | jq -r '.next_sequence_send')
echo "Gap: $((SEQ_SEND - SEQ_RECV))"

10.4 案例四：目标链分叉导致 IBC 暂停

10.4.1 场景描述

目标链（Juno）发生短时间分叉，导致 MSG Chain 上的 Light Client 检测到冲突。IBC 协议自动暂停通道以防止双花。

10.4.2 监控表现

ibc_client_state{client_id="07-tendermint-2", status="frozen"}
rate(cosmos_sdk_ibc_channel_packet_sent_total{channel_id="channel-2"}[5m]) == 0

10.4.3 恢复

# 1. 确认分叉已恢复
curl -s https://rpc.juno-1.msgchain.org/status | jq '.result.sync_info'

# 2. 使用治理提案解冻 Light Client
msgd tx gov submit-proposal /path/to/unfreeze-proposal.json \
    --from validator-key --chain-id msg-chain-1

# 3. 恢复 IBC 活动
rly tx relay-packets --all

10.5 案例五：DDoS 导致 IBC 通道拥堵

10.5.1 场景描述

恶意合约大量发送小额 IBC 转账，导致通道数据包量暴增 50 倍，正常用户的交易延迟显著增加。

10.5.2 告警链

09:00  P2  IBCPacketVolumeSpike  发送量飙升 50x
09:01  P2  IBCRelayerGasSpike    Gas 消耗异常
09:05  P1  IBCPacketBacklog      Pending > 500
09:10  P2  PacketLatencyHigh     延迟 > 120s

10.5.3 响应

# 1. 分析流量来源
journalctl -u msgd --since "09:00" --no-pager | grep "send_packet" | awk '{print $NF}' | sort | uniq -c | sort -rn | head -10

# 2. 增加中继器数量
docker-compose up -d --scale relayer=3

# 3. 在 CosmWasm 合约层面添加速率限制（需要合约升级）

# 4. 通知相关方进行调查

11. 总结

11.1 监控体系架构回顾

本文档构建了一个完整的 IBC 通道监控与告警体系，涵盖以下核心能力：

11.2 关键监控指标汇总

11.3 告警响应 SLA

11.4 MSG Chain 运维最佳实践

1. 中继器冗余: 每个通道至少运行 2 个独立中继器
2. 自动 Light Client 更新: 每 30 分钟 cron job 更新
3. 余额监控: 中继器余额低于阈值自动补充
4. 配置版本化: 所有监控配置使用 Git 管理
5. 告警演练: 每月进行一次 IBC 故障模拟演练
6. 通道文档化: 记录每个通道的对端链、用途和负责人
7. 升级计划: 跟踪 ibc-go 版本更新，及时升级

11.5 扩展方向

1. AI 驱动的异常检测: 使用机器学习预测通道故障
2. 自动恢复: 对于已知故障模式（如 Light Client 过期），实现自动恢复
3. 跨链费用优化: 基于延迟和拥堵情况动态调整中继器 gas
4. 多链拓扑可视化: 展示 MSG Chain 与所有连接链的关系地图
5. SLA 报告: 自动生成 IBC 通道的可用性报告

11.6 参考资源

• IBC 协议规范
• ibc-go 文档
• Cosmos SDK Telemetry
• Prometheus 告警规则
• Grafana Dashboard 模板
• MSG Chain RPC
• MSG Chain REST

本文档为 MSG Chain (msg-chain-1) 的 IBC 通道监控与告警实践指南。
域名为 msgchain.org，Bech32 地址前缀为 msg，基于 CosmWasm 生态。