type
status
date
slug
summary
tags
category
icon
password
  • 其中 IB 测试 和 NCCL 测试可以略过

GPU服务器信息

operating system:Ubuntu 20.04 LTS

硬件架构信息:uname -m : x86_64

CPU(s): 128 core

RAM:2048G

ROM:tmpfs 50T

注:/data表示共享的通用性能型文件存储,tmpfs 50T

GPU:A800-80GB * 8

nvidia-smi -L

nvidia-smi

nvidia-smi nvlink -s:8 channels, each channel 25GB/s bidirectional, total 200GB/s

nvidia-smi topo -m : NVLinks 8

Driver Version:515.86.01

CUDA Version:11.7

RDMA:800 Gb/sec mode=4

IB driver versionIB driver version:MLNX_OFED_LINUX-5.8-1.1.2.1

Check the network card device

Check GDR driver

ib_write_bw测试

单机:各台机器的 8 个网卡在 99[Gb/sec] 左右

Host
BW average[Gb/sec]
qps=10 BW average[Gb/sec]
001
99.57
111.33
002
99.58
125.20
003
99.42
111.33
004
99.46
111.33
详细日志:
  • bond 0:ansible test -m shell -a "ib_write_bw -d mlx5_0 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_0 --report_gbits 127.0.0.1"
  • bond 1:ansible test -m shell -a "ib_write_bw -d mlx5_1 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_1 --report_gbits 127.0.0.1"
  • bond 2:ansible test -m shell -a "ib_write_bw -d mlx5_2 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_2 --report_gbits 127.0.0.1"
  • bond 3:ansible test -m shell -a "ib_write_bw -d mlx5_3 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_3 --report_gbits 127.0.0.1"
  • bond 4:ansible test -m shell -a "ib_write_bw -d mlx5_4 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_4 --report_gbits 127.0.0.1"
  • bond 5:ansible test -m shell -a "ib_write_bw -d mlx5_5 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_5 --report_gbits 127.0.0.1"
  • bond 6:ansible test -m shell -a "ib_write_bw -d mlx5_6 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_6 --report_gbits 127.0.0.1"
  • bond 7:ansible test -m shell -a "ib_write_bw -d mlx5_7 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_7 --report_gbits 127.0.0.1"

加上 q = 10

  • bond 0:ansible test -m shell -a "ib_write_bw -d mlx5_0 -q 10 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_0 -q 10 --report_gbits 127.0.0.1"
  • bond 1:ansible test -m shell -a "ib_write_bw -d mlx5_1 -q 10 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_1 -q 10 --report_gbits 127.0.0.1"
  • bond 2:ansible test -m shell -a "ib_write_bw -d mlx5_2 -q 10 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_2 -q 10 --report_gbits 127.0.0.1"
  • bond 3:ansible test -m shell -a "ib_write_bw -d mlx5_3 -q 10 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_3 -q 10 --report_gbits 127.0.0.1"
  • bond 4:ansible test -m shell -a "ib_write_bw -d mlx5_4 -q 10 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_4 -q 10 --report_gbits 127.0.0.1"
  • bond 5:ansible test -m shell -a "ib_write_bw -d mlx5_5 -q 10 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_5 -q 10 --report_gbits 127.0.0.1"
  • bond 6:ansible test -m shell -a "ib_write_bw -d mlx5_6 -q 10 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_6 -q 10 --report_gbits 127.0.0.1"
  • bond 7:ansible test -m shell -a "ib_write_bw -d mlx5_7 -q 10 --report_gbits" ansible test -m shell -a "ib_write_bw -d mlx5_7 -q 10 --report_gbits 127.0.0.1"
 

2机:8个网卡均 98 Gb/sec

From
To
BW average[Gb/sec]
qps=10 BW average[Gb/sec]
002
001
97.98
98.01
003
001
97.98
98.01
004
001
97.98
98.01
  • bond 7:To : 001:ib_write_bw -d mlx5_7 --report_gbits From:002:ib_write_bw -d mlx5_7 --report_gbits 192.168.48.4
  • bond 7:To : 001:ib_write_bw -d mlx5_7 --report_gbits From:003:ib_write_bw -d mlx5_7 --report_gbits 192.168.48.4
  • bond 7:To : 001:ib_write_bw -d mlx5_7 --report_gbits From:004:ib_write_bw -d mlx5_7 --report_gbits 192.168.48.4

加入 q=10

  • bond 7:001:ib_write_bw -d mlx5_7 -q 10 --report_gbits 002:ib_write_bw -d mlx5_7 -q 10 --report_gbits 192.168.48.4
  • bond 7:001:ib_write_bw -d mlx5_7 -q 10 --report_gbits 003:ib_write_bw -d mlx5_7 -q 10 --report_gbits 192.168.48.4
  • bond 7:001:ib_write_bw -d mlx5_7 -q 10 --report_gbits 004:ib_write_bw -d mlx5_7 -q 10 --report_gbits 192.168.48.4

Nccl-Test测试

注意环境变量:
  1. NCCL_SOCKET_IFNAME=bond0指定网卡,指定也可以
  1. NCCL_IB_GID_INDEX=3设置RDMA通信优先级,可通过ibv_devinfo -v找到IP对应的gid
  1. NCCL_IB_DISABLE=0使用IB网络
  1. NCCL_GDR_LEVEL=2使用GDRDMA
  1. NCCL_IB_QPS_PER_CONNECTION=4 使用4qp,可在多机时将IB速率提升
  1. 无需指定NCCL_IB_HCA或指定全部网口,如:NCCL_IB_HCA=mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1,mlx5_bond_4:1,mlx5_bond_5:1,mlx5_bond_6:1,mlx5_bond_7:1

单机八卡: 149.263 GB/s

ansible test -m shell -a "mpirun -np 8 -H 127.0.0.1:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 1G -e 4G -f 2 -g 1"
逐步递增,增长因子为2:
固定大小,重复100次:ansible test -m shell -a "mpirun -np 8 -H 127.0.0.1:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 4G -e 4G -n 100 -g 1"

两机16卡:169.54GB/s

 
 
逐步递增,从1G到4G,增长因子为2:mpirun -np 16 -H 003:8,002:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 1G -e 4G -f 2 -g 1
固定4G大小,重复 100 次:mpirun -np 16 -H 003:8,002:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 4G -e 4G -n 100 -g 1

4机32卡: 95 GB/s

逐步递增,从1G到4G,增长因子为2:mpirun -np 32 -H 001:8,002:8,003:8,004:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 1G -e 4G -f 2 -g 1
固定4G大小,重复 100 次:mpirun -np 32 -H 001:8,002:8,003:8,004:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 4G -e 4G -n 100 -g 1

结论:

IB 带宽为 8 * 100 Gb/sec(从ibstatus看,是8根100Gb/sec的线,,共 800 Gb,而busbw在4机都能达到 95.019 GB/s(理论值是100GB/s),测试通过
百度云的人说:单机通信走nvlink,双机通信会一部分走nvlink一部分跨机通信,所以能跑到150GB/s,四机的主要就是走跨机的网卡了.

共享盘读写测试

  • write:1040 MB/s,read:3666 MB/s
百年孤独相机入门记录(1)购物篇
Loading...
SimonSun
SimonSun
Internet Malou, LLM Rookie, Bug Maker🤧
最新发布
相机入门记录(1)购物篇
2025-3-14
实用网站-工作提速大礼包
2025-3-14
conda poetry 全面迁移到 uv
2025-3-14
郁闷记
2025-3-13
LLM训练数据格式
2025-3-13
PPO(1)算法详解
2025-3-13
公告
🙌README🙌
🤯There is nothing left
in my right brain,
🤯and there is nothing right
in my left brain...
⭐I wish you every success⭐