[測試] Mellanox ConnectX-3 40GbE 網卡測試

看板PC_Shopping作者tacovirus (大怒神のスイッチ)時間2年前 (2021/08/02 13:27)推噓18(18推 0噓 16→)

留言34則, 21人參與討論串1/2 (看更多)

這張網卡裝在先前 5950X + WS X570-ACE 的機器上型號是 MCX313A-BCCT https://i.imgur.com/x7LwWyd.jpg

https://i.imgur.com/UFNWn7W.jpg

由於是買別人淘汰的卡, 一張 2800 就弄到了, 比全新的 10G 卡還便宜不過目前拍賣上最便宜的是 HP 544FLR, 目前沒屋頂一張不到 800, 不過這張卡造型特殊需要轉一下才能用, 詳情可以去 YT 搜尋 "測速怪" Proxmox VE 7.0 能偵測到, 使用的驅動程式是 mlx4_en 插上光纖讓兩台 5950X 互連, 確定兩張卡都亮燈接著進入測試環節首先用 iperf3 測試 # iperf3 -c 172.16.50.32 Connecting to host 172.16.50.32, port 5201 [ 5] local 172.16.50.31 port 56320 connected to 172.16.50.32 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 3.69 GBytes 31.7 Gbits/sec 0 1.29 MBytes [ 5] 1.00-2.00 sec 3.67 GBytes 31.5 Gbits/sec 0 1.40 MBytes [ 5] 2.00-3.00 sec 3.24 GBytes 27.9 Gbits/sec 0 1.47 MBytes [ 5] 3.00-4.00 sec 3.09 GBytes 26.6 Gbits/sec 0 1.47 MBytes [ 5] 4.00-5.00 sec 3.08 GBytes 26.5 Gbits/sec 0 1.47 MBytes [ 5] 5.00-6.00 sec 3.07 GBytes 26.4 Gbits/sec 0 1.47 MBytes [ 5] 6.00-7.00 sec 3.09 GBytes 26.5 Gbits/sec 0 1.47 MBytes [ 5] 7.00-8.00 sec 3.09 GBytes 26.5 Gbits/sec 0 1.47 MBytes [ 5] 8.00-9.00 sec 3.09 GBytes 26.5 Gbits/sec 0 1.47 MBytes [ 5] 9.00-10.00 sec 3.10 GBytes 26.6 Gbits/sec 0 1.47 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 32.2 GBytes 27.7 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 32.2 GBytes 27.7 Gbits/sec receiver 前 3 秒爽了一下, 後來就掉速到 26Gbps, 有點傷心, 試了幾次都這樣爬文調了一些設定看起來都沒啥鳥用不過其中有一篇官網的建議測試方式 https://community.mellanox.com/s/article/iperf--iperf2--iperf3 其中有一句話是 we recommend to use iperf2 (2.0.x) in Linux 結果越想越不對勁, 改用 iperf2 做雙線測試 # iperf -c 172.16.50.32 -P 2 ------------------------------------------------------------ Client connecting to 172.16.50.32, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 3] local 172.16.50.31 port 49590 connected with 172.16.50.32 port 5001 [ 4] local 172.16.50.31 port 49592 connected with 172.16.50.32 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0000-10.0005 sec 21.4 GBytes 18.4 Gbits/sec [ 4] 0.0000-10.0002 sec 21.5 GBytes 18.5 Gbits/sec [SUM] 0.0000-10.0002 sec 42.9 GBytes 36.9 Gbits/sec [ CT] final connect times (min/avg/max/stdev) = 0.618/0.674/0.730/0.618 ms (tot/ 結果真得逼近 40Gbps 接著查一下這張卡的 MTU 可設定範圍 # ip -d link show enp10s0 4: enp10s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DE link/ether e4:1d:2d:18:a5:a0 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 m numtxqueues 256 numrxqueues 128 gso_max_size 65536 gso_max_segs 65535 portid 可以看出 MTU 可設定範圍是 68-9900, 那麼改用 9900 來試試 # iperf -c 172.16.50.32 -P 2 ------------------------------------------------------------ Client connecting to 172.16.50.32, TCP port 5001 TCP window size: 325 KByte (default) ------------------------------------------------------------ [ 3] local 172.16.50.31 port 49670 connected with 172.16.50.32 port 5001 [ 4] local 172.16.50.31 port 49672 connected with 172.16.50.32 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0000-10.0004 sec 23.1 GBytes 19.8 Gbits/sec [ 3] 0.0000-10.0002 sec 23.1 GBytes 19.8 Gbits/sec [SUM] 0.0000-10.0002 sec 46.1 GBytes 39.6 Gbits/sec [ CT] final connect times (min/avg/max/stdev) = 1.115/1.224/1.334/1.115 ms (tot/ 結果從 36.9 提升到 39.6, 看起來 MTU 要喬一下才能操到緊繃那麼回頭測一下單線 # iperf3 -c 172.16.50.32 Connecting to host 172.16.50.32, port 5201 [ 5] local 172.16.50.31 port 56512 connected to 172.16.50.32 port 5201 ... 略 ... [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 31.7 GBytes 27.2 Gbits/sec 0 sender [ 5] 0.00-10.00 sec 31.7 GBytes 27.2 Gbits/sec receiver 看起來 MTU 9900 對單線沒啥鳥用, 不死心再用 4096 試試 [ 5] 0.00-10.00 sec 37.1 GBytes 31.9 Gbits/sec 0 sender 結果穩上 31.9, 再試試其他值, 後來試出 2048 更好 [ 5] 0.00-10.00 sec 38.0 GBytes 32.7 Gbits/sec 201 sender 修改參數, 把傳輸量加大到 200G 試試, iperf3 -c 172.16.50.32 -n 200G [ 5] 0.00-56.10 sec 200 GBytes 30.6 Gbits/sec 197 sender 再回頭測試多線傳輸 # iperf -c 172.16.50.32 -P 2 -n 200G ------------------------------------------------------------ Client connecting to 172.16.50.32, TCP port 5001 TCP window size: 85.0 KByte (default) ------------------------------------------------------------ [ 4] local 172.16.50.31 port 50464 connected with 172.16.50.32 port 5001 [ 3] local 172.16.50.31 port 50462 connected with 172.16.50.32 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0000-90.2807 sec 200 GBytes 19.0 Gbits/sec [ 4] 0.0000-90.3109 sec 200 GBytes 19.0 Gbits/sec [SUM] 0.0000-90.3109 sec 400 GBytes 38.0 Gbits/sec [ CT] final connect times (min/avg/max/stdev) = 0.737/0.784/0.831/0.831 ms (tot/ 看起來 2048 有利單線但不利多線改回 9900, 再測試很多線的 CPU 用量 # iperf -c 172.16.50.32 -P 128 -n 1G [SUM] 0.0000-26.9918 sec 128 GBytes 40.7 Gbits/sec # iperf -c 172.16.50.32 -P 32 -n 1G [SUM] 0.0000-6.9327 sec 32.0 GBytes 39.6 Gbits/sec 看起來 32 線和 128 線大概都是消耗 4~5% 的 CPU 用量, 不過這可是 5950X 喔, 其他 CP 為了盡量發揮滿載吞吐量, 還是犧牲單線最佳值吧, 就決定用 9900 了感謝大家看我如何浪費生命 -- 世界越快心則慢網路越慢心則快 -- ※ 發信站: 批踢踢實業坊(ptt.cc), 來自: 211.23.152.16 (臺灣) ※ 文章網址: https://www.ptt.cc/bbs/PC_Shopping/M.1627882058.A.EA7.html ※ 編輯: tacovirus (211.23.152.16 臺灣), 08/02/2021 13:30:47

推

rettoly

08/02 13:28, 2年前 , 1^F

08/02 13:28, 1^F

→

HamalAri

08/02 13:32, 2年前 , 2^F

08/02 13:32, 2^F

用 iSCSI 連去 NAS ※ 編輯: tacovirus (27.242.104.204 臺灣), 08/02/2021 13:34:21 ※ 編輯: tacovirus (27.242.104.204 臺灣), 08/02/2021 13:35:27

推

b325019

08/02 13:35, 2年前 , 3^F