Req:
Please
- Make sure that we have 200Mbit channel for eno1
- Configure static IP for eno2 ( I temporarily set it 192.168.1.0/24 ) but you can change at will.
Those servers are cb[1,2]c.itim.vn, cb1c.dev.itim.vn, cb-inf1c.itim.vn.
I’ll ask you to make firewall for them soon.
- 10.194.63.0/24 – Tmp subnet for the first hosts in CMC DC (dev/prod/infra – due to unmanaged switch).
- let’s start to allocate ip from the next logical ranges (to have possibility to divide it in a future into smaller subnets; net mask still /24!)
- 10.194.63.0/25 – prod
- 10.194.63.16 – cb1c
- 10.194.63.17 – cb2c
- 10.194.63.128/26 – infr
- 10.194.63.132 – cb-inf1c
- 10.194.63.192/26 – dev
- 10.194.63.196 – cb1c.dev
- 10.194.63.0/25 – prod
I created /etc/network/interfaces.d/eno2.conf files on each host.
You can restart interfaces to implement new ip addresses.
Bandwidth tests
Server
rb2v is iperf tcp server
iptables -I INPUT -s 103.21.149.128/27 -p tcp --dport 35555 -j ACCEPT iperf -p 35555 -s -b 600M
Clients
- bandwidth limited to 600Mbps to see how shaper/policer works on CMC device
- test time – 60s
- dual test – to measure in/out bandwidth
iperf -fm -i 10 -p 35555 -c 123.30.175.31 -t 60 -b 600M -d
- cb1c
cb1c:~# iperf -fm -i 10 -p 35555 -c 123.30.175.31 -t 60 -b 600M -d ------------------------------------------------------------ Server listening on TCP port 35555 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 123.30.175.31, TCP port 35555 TCP window size: 0.20 MByte (default) ------------------------------------------------------------ [ 4] local 103.21.149.155 port 59256 connected with 123.30.175.31 port 35555 [ 5] local 103.21.149.155 port 35555 connected with 123.30.175.31 port 60757 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 715 MBytes 600 Mbits/sec [ 5] 0.0-10.0 sec 244 MBytes 205 Mbits/sec [ 4] 10.0-20.0 sec 715 MBytes 600 Mbits/sec [ 5] 10.0-20.0 sec 255 MBytes 214 Mbits/sec [ 4] 20.0-30.0 sec 715 MBytes 600 Mbits/sec [ 5] 20.0-30.0 sec 282 MBytes 236 Mbits/sec [ 4] 30.0-40.0 sec 715 MBytes 600 Mbits/sec [ 5] 30.0-40.0 sec 268 MBytes 225 Mbits/sec [ 4] 40.0-50.0 sec 715 MBytes 600 Mbits/sec [ 5] 40.0-50.0 sec 240 MBytes 201 Mbits/sec [ 4] 50.0-60.0 sec 715 MBytes 600 Mbits/sec [ 4] 0.0-60.0 sec 4292 MBytes 600 Mbits/sec [ 5] 50.0-60.0 sec 263 MBytes 221 Mbits/sec [ 5] 0.0-60.0 sec 1552 MBytes 217 Mbits/sec
- cb2c
cb2c:~# iperf -fm -i 10 -p 35555 -c 123.30.175.31 -t 60 -b 600M -d ------------------------------------------------------------ Server listening on TCP port 35555 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 123.30.175.31, TCP port 35555 TCP window size: 0.22 MByte (default) ------------------------------------------------------------ [ 3] local 103.21.149.156 port 46868 connected with 123.30.175.31 port 35555 [ 5] local 103.21.149.156 port 35555 connected with 123.30.175.31 port 36936 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 715 MBytes 600 Mbits/sec [ 5] 0.0-10.0 sec 261 MBytes 219 Mbits/sec [ 3] 10.0-20.0 sec 715 MBytes 600 Mbits/sec [ 5] 10.0-20.0 sec 241 MBytes 202 Mbits/sec [ 3] 20.0-30.0 sec 715 MBytes 600 Mbits/sec [ 5] 20.0-30.0 sec 285 MBytes 239 Mbits/sec [ 3] 30.0-40.0 sec 715 MBytes 600 Mbits/sec [ 5] 30.0-40.0 sec 242 MBytes 203 Mbits/sec [ 3] 40.0-50.0 sec 715 MBytes 600 Mbits/sec [ 5] 40.0-50.0 sec 298 MBytes 250 Mbits/sec [ 3] 50.0-60.0 sec 715 MBytes 600 Mbits/sec [ 3] 0.0-60.0 sec 4292 MBytes 600 Mbits/sec [ 5] 50.0-60.0 sec 280 MBytes 235 Mbits/sec [ 5] 0.0-60.0 sec 1608 MBytes 225 Mbits/sec
- cb1c.dev
cb1c:~# iperf -fm -i 10 -p 35555 -c 123.30.175.31 -t 60 -b 600M -d ------------------------------------------------------------ Server listening on TCP port 35555 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 123.30.175.31, TCP port 35555 TCP window size: 0.37 MByte (default) ------------------------------------------------------------ [ 4] local 103.21.149.154 port 38430 connected with 123.30.175.31 port 35555 [ 5] local 103.21.149.154 port 35555 connected with 123.30.175.31 port 55618 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 715 MBytes 600 Mbits/sec [ 5] 0.0-10.0 sec 285 MBytes 239 Mbits/sec [ 4] 10.0-20.0 sec 715 MBytes 600 Mbits/sec [ 5] 10.0-20.0 sec 233 MBytes 196 Mbits/sec [ 4] 20.0-30.0 sec 715 MBytes 600 Mbits/sec [ 5] 20.0-30.0 sec 288 MBytes 242 Mbits/sec [ 4] 30.0-40.0 sec 715 MBytes 600 Mbits/sec [ 5] 30.0-40.0 sec 227 MBytes 190 Mbits/sec [ 4] 40.0-50.0 sec 715 MBytes 600 Mbits/sec [ 5] 40.0-50.0 sec 287 MBytes 241 Mbits/sec [ 4] 50.0-60.0 sec 715 MBytes 600 Mbits/sec [ 4] 0.0-60.0 sec 4292 MBytes 600 Mbits/sec [ 5] 50.0-60.0 sec 261 MBytes 219 Mbits/sec [ 5] 0.0-60.0 sec 1582 MBytes 221 Mbits/sec
- cb-inf1c
cb-inf1c:~# iperf -fm -i 10 -p 35555 -c 123.30.175.31 -t 60 -b 600M -d ------------------------------------------------------------ Server listening on TCP port 35555 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 123.30.175.31, TCP port 35555 TCP window size: 0.23 MByte (default) ------------------------------------------------------------ [ 4] local 103.21.149.153 port 45106 connected with 123.30.175.31 port 35555 [ 5] local 103.21.149.153 port 35555 connected with 123.30.175.31 port 39263 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 715 MBytes 600 Mbits/sec [ 5] 0.0-10.0 sec 281 MBytes 236 Mbits/sec [ 4] 10.0-20.0 sec 715 MBytes 600 Mbits/sec [ 5] 10.0-20.0 sec 255 MBytes 214 Mbits/sec [ 4] 20.0-30.0 sec 715 MBytes 600 Mbits/sec [ 5] 20.0-30.0 sec 238 MBytes 199 Mbits/sec [ 4] 30.0-40.0 sec 715 MBytes 600 Mbits/sec [ 5] 30.0-40.0 sec 253 MBytes 213 Mbits/sec [ 4] 40.0-50.0 sec 715 MBytes 600 Mbits/sec [ 5] 40.0-50.0 sec 233 MBytes 195 Mbits/sec [ 4] 50.0-60.0 sec 715 MBytes 600 Mbits/sec [ 4] 0.0-60.0 sec 4292 MBytes 600 Mbits/sec [ 5] 50.0-60.0 sec 251 MBytes 211 Mbits/sec [ 5] 0.0-60.0 sec 1511 MBytes 211 Mbits/sec
Results:
- CMC almost do not limit outbound traffic, but we can not reach 1Gbps due to the testing over internet
- we may have some packet loss, that triggers tcp congestion avoidance, but mss looks persistent
- packet loss due to the problems on some hops in internet
- packet loss due to shaper/policer on CMC device – it seems more real, if to look at the results of the next 2 tests
test with bandwidth limited to 1G
cb-inf1c:~# iperf -fm -i 1 -p 35555 -c 123.30.175.31 -t 10 -b 1G -d -m -l 2M -e ------------------------------------------------------------ Server listening on TCP port 35555 with pid 11854 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 123.30.175.31, TCP port 35555 with pid 11854 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ [ 4] local 103.21.149.153 port 46612 connected with 123.30.175.31 port 35555 [ 5] local 103.21.149.153 port 35555 connected with 123.30.175.31 port 39360 [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT [ 4] 0.00-1.00 sec 108 MBytes 906 Mbits/sec 54/0 6 244K/1933 us [ 5] 0.00-1.00 sec 16.6 MBytes 139 Mbits/sec 2181 2181:0:0:0:0:0:0:0 [ 4] 1.00-2.00 sec 106 MBytes 889 Mbits/sec 53/0 5 227K/1767 us [ 5] 1.00-2.00 sec 14.6 MBytes 122 Mbits/sec 1668 1668:0:0:0:0:0:0:0 [ 4] 2.00-3.00 sec 106 MBytes 889 Mbits/sec 53/0 5 214K/1898 us [ 5] 2.00-3.00 sec 16.5 MBytes 138 Mbits/sec 1797 1797:0:0:0:0:0:0:0 [ 4] 3.00-4.00 sec 108 MBytes 906 Mbits/sec 54/0 5 176K/1334 us [ 5] 3.00-4.00 sec 15.2 MBytes 128 Mbits/sec 1632 1632:0:0:0:0:0:0:0 [ 4] 4.00-5.00 sec 108 MBytes 906 Mbits/sec 54/0 4 253K/2019 us [ 5] 4.00-5.00 sec 22.0 MBytes 184 Mbits/sec 2155 2155:0:0:0:0:0:0:0 [ 4] 5.00-6.00 sec 108 MBytes 906 Mbits/sec 54/0 5 212K/1682 us [ 5] 5.00-6.00 sec 11.8 MBytes 98.8 Mbits/sec 1334 1334:0:0:0:0:0:0:0 [ 5] 6.00-7.00 sec 11.4 MBytes 95.6 Mbits/sec 1233 1233:0:0:0:0:0:0:0 [ 4] 6.00-7.00 sec 108 MBytes 906 Mbits/sec 54/0 4 255K/1912 us [ 4] 7.00-8.00 sec 108 MBytes 906 Mbits/sec 54/0 5 202K/1575 us [ 5] 7.00-8.00 sec 11.2 MBytes 94.1 Mbits/sec 1153 1153:0:0:0:0:0:0:0 [ 4] 8.00-9.00 sec 106 MBytes 889 Mbits/sec 53/0 4 233K/1779 us [ 5] 8.00-9.00 sec 26.3 MBytes 221 Mbits/sec 2724 2724:0:0:0:0:0:0:0 [ 5] 9.00-10.00 sec 18.9 MBytes 158 Mbits/sec 2032 2032:0:0:0:0:0:0:0 [ 5] 0.00-10.00 sec 164 MBytes 138 Mbits/sec 17911 17911:0:0:0:0:0:0:0 [ 5] MSS size 1448 bytes (MTU 1500 bytes, ethernet) [ 4] 9.00-10.00 sec 110 MBytes 923 Mbits/sec 55/0 5 178K/1947 us [ 4] 0.00-10.01 sec 1076 MBytes 901 Mbits/sec 538/0 48 178K/1947 us [ 4] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
test with bandwidth limited to 600M
cb-inf1c:~# iperf -fm -i 1 -p 35555 -c 123.30.175.31 -t 10 -b 600M -d -m -l 2M -e ------------------------------------------------------------ Server listening on TCP port 35555 with pid 12039 TCP window size: 0.08 MByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 123.30.175.31, TCP port 35555 with pid 12039 TCP window size: 0.09 MByte (default) ------------------------------------------------------------ [ 4] local 103.21.149.153 port 46802 connected with 123.30.175.31 port 35555 [ 5] local 103.21.149.153 port 35555 connected with 123.30.175.31 port 39368 [ ID] Interval Transfer Bandwidth Write/Err Rtry Cwnd/RTT [ 4] 0.00-1.00 sec 74.0 MBytes 621 Mbits/sec 37/0 0 644K/2756 us [ 5] 0.00-1.00 sec 26.5 MBytes 222 Mbits/sec 3567 3567:0:0:0:0:0:0:0 [ 5] 1.00-2.00 sec 26.8 MBytes 225 Mbits/sec 3631 3631:0:0:0:0:0:0:0 [ 4] 1.00-2.00 sec 72.0 MBytes 604 Mbits/sec 36/0 0 644K/3006 us [ 5] 2.00-3.00 sec 28.8 MBytes 242 Mbits/sec 3765 3765:0:0:0:0:0:0:0 [ 4] 2.00-3.00 sec 72.0 MBytes 604 Mbits/sec 36/0 0 680K/2323 us [ 5] 3.00-4.00 sec 24.6 MBytes 206 Mbits/sec 3104 3104:0:0:0:0:0:0:0 [ 4] 3.00-4.00 sec 72.0 MBytes 604 Mbits/sec 36/0 0 680K/2701 us [ 4] 4.00-5.00 sec 70.0 MBytes 587 Mbits/sec 35/0 0 680K/2825 us [ 5] 4.00-5.00 sec 28.5 MBytes 239 Mbits/sec 3741 3741:0:0:0:0:0:0:0 [ 5] 5.00-6.00 sec 24.2 MBytes 203 Mbits/sec 3430 3430:0:0:0:0:0:0:0 [ 4] 5.00-6.00 sec 72.0 MBytes 604 Mbits/sec 36/0 0 680K/2907 us [ 5] 6.00-7.00 sec 24.5 MBytes 205 Mbits/sec 3464 3464:0:0:0:0:0:0:0 [ 4] 6.00-7.00 sec 72.0 MBytes 604 Mbits/sec 36/0 0 680K/2389 us [ 5] 7.00-8.00 sec 19.3 MBytes 162 Mbits/sec 2708 2708:0:0:0:0:0:0:0 [ 4] 7.00-8.00 sec 72.0 MBytes 604 Mbits/sec 36/0 0 680K/2833 us [ 4] 8.00-9.00 sec 70.0 MBytes 587 Mbits/sec 35/0 0 680K/2826 us [ 5] 8.00-9.00 sec 20.8 MBytes 175 Mbits/sec 3156 3156:0:0:0:0:0:0:0 [ 4] 0.00-10.00 sec 716 MBytes 601 Mbits/sec 358/0 0 680K/3038 us [ 4] MSS size 1448 bytes (MTU 1500 bytes, ethernet) [ 5] 9.00-10.00 sec 21.6 MBytes 181 Mbits/sec 3228 3228:0:0:0:0:0:0:0 [ 5] 0.00-10.00 sec 246 MBytes 206 Mbits/sec 33796 33796:0:0:0:0:0:0:0 [ 5] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
- in the first test we can see re-transmits and lowed inbound speed (cwnd is lower)
- in the second test we can see that we don’t have re-transmits
- inbound traffic limited to 200Mbps + ~10% burst