Re: quick summary results with ixgbe (was Re: datapoints on 10G

On Dec 7, 2011, at 8:08 PM, Luigi Rizzo wrote:

> Summary:
>=20
> - with default interrupt mitigation, the fastest configuration
>  is with checksums enabled on both sender and receiver, lro
>  enabled on the receiver. This gets about 8.0 Gbit/s

I do not observe this. With defaults:

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11
nuttcp-t: time limit =3D 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.053 ms
nuttcp-t: send window size =3D 131768, receive window size =3D 66608
nuttcp-t: 1857.4978 MB in 5.02 real seconds =3D 378856.02 KB/sec =3D =
3103.5885 Mbps
nuttcp-t: host-retrans =3D 0
nuttcp-t: 29720 I/O calls, msec/call =3D 0.17, calls/sec =3D 5919.63
nuttcp-t: 0.0user 2.5sys 0:05real 52% 115i+1544d 630maxrss 0+2pf =
107264+1csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size =3D 33304, receive window size =3D 131768
nuttcp-r: 1857.4978 MB in 5.15 real seconds =3D 369617.39 KB/sec =3D =
3027.9057 Mbps
nuttcp-r: 543991 I/O calls, msec/call =3D 0.01, calls/sec =3D 105709.95
nuttcp-r: 0.1user 4.1sys 0:05real 83% 110i+1482d 618maxrss 0+15pf =
158432+0csw

On receiver:

ifconfig ix1 lro

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11
nuttcp-t: time limit =3D 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.068 ms
nuttcp-t: send window size =3D 131768, receive window size =3D 66608
nuttcp-t: 1673.3125 MB in 5.02 real seconds =3D 341312.36 KB/sec =3D =
2796.0308 Mbps
nuttcp-t: host-retrans =3D 1
nuttcp-t: 26773 I/O calls, msec/call =3D 0.19, calls/sec =3D 5333.01
nuttcp-t: 0.0user 1.0sys 0:05real 21% 113i+1518d 630maxrss 0+2pf =
12772+1csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size =3D 33304, receive window size =3D 131768
nuttcp-r: 1673.3125 MB in 5.15 real seconds =3D 332975.19 KB/sec =3D =
2727.7327 Mbps
nuttcp-r: 106268 I/O calls, msec/call =3D 0.05, calls/sec =3D 20650.82
nuttcp-r: 0.0user 1.3sys 0:05real 28% 101i+1354d 618maxrss 0+15pf =
64567+0csw

On sender:

ifconfig ix1 lro

(now both receiver and sender have LRO enabled)

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11
nuttcp-t: time limit =3D 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.063 ms
nuttcp-t: send window size =3D 131768, receive window size =3D 66608
nuttcp-t: 1611.7805 MB in 5.02 real seconds =3D 328716.18 KB/sec =3D =
2692.8430 Mbps
nuttcp-t: host-retrans =3D 1
nuttcp-t: 25789 I/O calls, msec/call =3D 0.20, calls/sec =3D 5136.29
nuttcp-t: 0.0user 1.0sys 0:05real 21% 109i+1465d 630maxrss 0+2pf =
12697+0csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size =3D 33304, receive window size =3D 131768
nuttcp-r: 1611.7805 MB in 5.15 real seconds =3D 320694.82 KB/sec =3D =
2627.1319 Mbps
nuttcp-r: 104346 I/O calls, msec/call =3D 0.05, calls/sec =3D 20275.05
nuttcp-r: 0.0user 1.3sys 0:05real 27% 113i+1516d 618maxrss 0+15pf =
63510+0csw

remove LRO from receiver (only sender has LRO):

# nuttcp -t -T 5 -w 128 -v 10.2.101.11
nuttcp-t: v6.1.2: socket
nuttcp-t: buflen=3D65536, nstream=3D1, port=3D5001 tcp -> 10.2.101.11
nuttcp-t: time limit =3D 5.00 seconds
nuttcp-t: connect to 10.2.101.11 with mss=3D1448, RTT=3D0.065 ms
nuttcp-t: send window size =3D 131768, receive window size =3D 66608
nuttcp-t: 1884.8702 MB in 5.02 real seconds =3D 384464.57 KB/sec =3D =
3149.5338 Mbps
nuttcp-t: host-retrans =3D 0
nuttcp-t: 30158 I/O calls, msec/call =3D 0.17, calls/sec =3D 6007.27
nuttcp-t: 0.0user 2.7sys 0:05real 55% 104i+1403d 630maxrss 0+2pf =
106046+0csw

nuttcp-r: v6.1.2: socket
nuttcp-r: buflen=3D65536, nstream=3D1, port=3D5001 tcp
nuttcp-r: accept from 10.2.101.12
nuttcp-r: send window size =3D 33304, receive window size =3D 131768
nuttcp-r: 1884.8702 MB in 5.15 real seconds =3D 375093.52 KB/sec =3D =
3072.7661 Mbps
nuttcp-r: 540237 I/O calls, msec/call =3D 0.01, calls/sec =3D 104988.68
nuttcp-r: 0.1user 4.2sys 0:05real 84% 110i+1483d 618maxrss 0+15pf =
156340+0csw

Strange enough, setting hw.ixgbe.max_interrupt_rate=3D0 does not have =
any effect..

Daniel

_______________________________________________
freebsd-current@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org"