type
status
date
slug
summary
tags
category
icon
password
- 其中 IB 测试 和 NCCL 测试可以略过
GPU服务器信息
operating system:Ubuntu 20.04 LTS
硬件架构信息:
uname -m
: x86_64
CPU(s): 128 core
RAM:2048G
ROM:tmpfs 50T
注:/data表示共享的通用性能型文件存储,tmpfs 50T
GPU:A800-80GB * 8
nvidia-smi -L
nvidia-smi
nvidia-smi nvlink -s
:8 channels, each channel 25GB/s bidirectional, total 200GB/s
nvidia-smi topo -m
: NVLinks 8
Driver Version:515.86.01
CUDA Version:11.7
RDMA:800 Gb/sec mode=4
IB driver versionIB driver version:MLNX_OFED_LINUX-5.8-1.1.2.1
Check the network card device
Check GDR driver
ib_write_bw测试
单机:各台机器的 8 个网卡在 99[Gb/sec] 左右
Host | BW average[Gb/sec] | qps=10 BW average[Gb/sec] |
001 | 99.57 | 111.33 |
002 | 99.58 | 125.20 |
003 | 99.42 | 111.33 |
004 | 99.46 | 111.33 |
详细日志:
- bond 0:
ansible test -m shell -a "ib_write_bw -d mlx5_0 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_0 --report_gbits 127.0.0.1"
- bond 1:
ansible test -m shell -a "ib_write_bw -d mlx5_1 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_1 --report_gbits 127.0.0.1"
- bond 2:
ansible test -m shell -a "ib_write_bw -d mlx5_2 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_2 --report_gbits 127.0.0.1"
- bond 3:
ansible test -m shell -a "ib_write_bw -d mlx5_3 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_3 --report_gbits 127.0.0.1"
- bond 4:
ansible test -m shell -a "ib_write_bw -d mlx5_4 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_4 --report_gbits 127.0.0.1"
- bond 5:
ansible test -m shell -a "ib_write_bw -d mlx5_5 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_5 --report_gbits 127.0.0.1"
- bond 6:
ansible test -m shell -a "ib_write_bw -d mlx5_6 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_6 --report_gbits 127.0.0.1"
- bond 7:
ansible test -m shell -a "ib_write_bw -d mlx5_7 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_7 --report_gbits 127.0.0.1"
加上 q = 10
- bond 0:
ansible test -m shell -a "ib_write_bw -d mlx5_0 -q 10 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_0 -q 10 --report_gbits 127.0.0.1"
- bond 1:
ansible test -m shell -a "ib_write_bw -d mlx5_1 -q 10 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_1 -q 10 --report_gbits 127.0.0.1"
- bond 2:
ansible test -m shell -a "ib_write_bw -d mlx5_2 -q 10 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_2 -q 10 --report_gbits 127.0.0.1"
- bond 3:
ansible test -m shell -a "ib_write_bw -d mlx5_3 -q 10 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_3 -q 10 --report_gbits 127.0.0.1"
- bond 4:
ansible test -m shell -a "ib_write_bw -d mlx5_4 -q 10 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_4 -q 10 --report_gbits 127.0.0.1"
- bond 5:
ansible test -m shell -a "ib_write_bw -d mlx5_5 -q 10 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_5 -q 10 --report_gbits 127.0.0.1"
- bond 6:
ansible test -m shell -a "ib_write_bw -d mlx5_6 -q 10 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_6 -q 10 --report_gbits 127.0.0.1"
- bond 7:
ansible test -m shell -a "ib_write_bw -d mlx5_7 -q 10 --report_gbits"
ansible test -m shell -a "ib_write_bw -d mlx5_7 -q 10 --report_gbits 127.0.0.1"
2机:8个网卡均 98 Gb/sec
From | To | BW average[Gb/sec] | qps=10 BW average[Gb/sec] |
002 | 001 | 97.98 | 98.01 |
003 | 001 | 97.98 | 98.01 |
004 | 001 | 97.98 | 98.01 |
- bond 7:To : 001:
ib_write_bw -d mlx5_7 --report_gbits
From:002:ib_write_bw -d mlx5_7 --report_gbits 192.168.48.4
- bond 7:To : 001:
ib_write_bw -d mlx5_7 --report_gbits
From:003:ib_write_bw -d mlx5_7 --report_gbits 192.168.48.4
- bond 7:To : 001:
ib_write_bw -d mlx5_7 --report_gbits
From:004:ib_write_bw -d mlx5_7 --report_gbits 192.168.48.4
加入 q=10
- bond 7:001:
ib_write_bw -d mlx5_7 -q 10 --report_gbits
002:ib_write_bw -d mlx5_7 -q 10 --report_gbits 192.168.48.4
- bond 7:001:
ib_write_bw -d mlx5_7 -q 10 --report_gbits
003:ib_write_bw -d mlx5_7 -q 10 --report_gbits 192.168.48.4
- bond 7:001:
ib_write_bw -d mlx5_7 -q 10 --report_gbits
004:ib_write_bw -d mlx5_7 -q 10 --report_gbits 192.168.48.4
Nccl-Test测试
注意环境变量:
NCCL_SOCKET_IFNAME=bond0
指定网卡,指定也可以
NCCL_IB_GID_INDEX=3
设置RDMA通信优先级,可通过ibv_devinfo -v找到IP对应的gid
NCCL_IB_DISABLE=0
使用IB网络
NCCL_GDR_LEVEL=2
使用GDRDMA
NCCL_IB_QPS_PER_CONNECTION=4
使用4qp,可在多机时将IB速率提升
- 无需指定
NCCL_IB_HCA
或指定全部网口,如:NCCL_IB_HCA=mlx5_bond_0:1,mlx5_bond_1:1,mlx5_bond_2:1,mlx5_bond_3:1,mlx5_bond_4:1,mlx5_bond_5:1,mlx5_bond_6:1,mlx5_bond_7:1
单机八卡: 149.263 GB/s
ansible test -m shell -a "mpirun -np 8 -H 127.0.0.1:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 1G -e 4G -f 2 -g 1"
逐步递增,增长因子为2:
固定大小,重复100次:
ansible test -m shell -a "mpirun -np 8 -H 127.0.0.1:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 4G -e 4G -n 100 -g 1"
两机16卡:169.54GB/s
逐步递增,从1G到4G,增长因子为2:
mpirun -np 16 -H 003:8,002:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 1G -e 4G -f 2 -g 1
固定4G大小,重复 100 次:
mpirun -np 16 -H 003:8,002:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 4G -e 4G -n 100 -g 1
4机32卡: 95 GB/s
逐步递增,从1G到4G,增长因子为2:
mpirun -np 32 -H 001:8,002:8,003:8,004:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 1G -e 4G -f 2 -g 1
固定4G大小,重复 100 次:
mpirun -np 32 -H 001:8,002:8,003:8,004:8 --allow-run-as-root -x NCCL_IB_DISABLE=0 -x NCCL_IB_GID_INDEX=3 -x NCCL_GDR_LEVEL=2 -x NCCL_DEBUG=INFO /data/sunjinfeng/nccl-tests/build/all_reduce_perf -b 4G -e 4G -n 100 -g 1
结论:
IB 带宽为 8 * 100 Gb/sec(从ibstatus看,是8根100Gb/sec的线,,共 800 Gb,而busbw在4机都能达到 95.019 GB/s(理论值是100GB/s),测试通过
百度云的人说:单机通信走nvlink,双机通信会一部分走nvlink一部分跨机通信,所以能跑到150GB/s,四机的主要就是走跨机的网卡了.
共享盘读写测试
- write:1040 MB/s,read:3666 MB/s
- 作者:SimonSun
- 链接:https://simons-blog-eight.vercel.app//article/tech-3
- 声明:本文采用 CC BY-NC-SA 4.0 许可协议,转载请注明出处。