[Jool-list] NAT64 performance
Sander Steffann
sander at steffann.nl
Mon Sep 4 16:18:27 CDT 2017
Hi,
> On Mon, Sep 04, 2017 at 07:45:19PM +0200, Sander Steffann wrote:
> Hi,
>>
>> The great Cisco TRex team [1] has updated their performance testing
>> software so that it can now test NAT64 boxes! They added an Advance
>> Stateful mode (ASTF) that implements an extremely fast TCP stack for
>> testing [2]. Naturally I pointed it at a Jool box :-) Unfortunately
>> the Jool box collapsed under the surge of new sessions :-(
>>
>
> thx for your work. But I think it would be helpfull if you could
> share some more details from your setup.
Nothing special:
{
"global": {
"source-icmpv6-errors-better": true,
"handle-rst-during-fin-rcv": true,
"f-args": 10,
"logging-bib": true
},
"pool6": "2a00:8642:2000:64::/96",
"pool4": [
{
"protocol": "TCP",
"prefix": "185.54.94.64/28",
"port range": "1025-65535"
},
{
"protocol": "UDP",
"prefix": "185.54.94.64/28",
"port range": "1025-65535"
},
{
"protocol": "ICMP",
"prefix": "185.54.94.64/28",
"port range": "0-65535"
}
]
}
Just using the global namespace. Turning logging-bib on or off doesn't make a difference.
The box is a CentOS 7 Dell R630 with dual Xeon E5-2680 v4 (so 28 cores in total, 56 with hyper threading) and 64 GB of RAM. I'm currently using Intel X710 NICs. The box also has a dual-port Mellanox ConnectX-3 and a dual-port Solarflare SFC9220 (it's a lab box for testing networking configs). I had two different configurations for the uplink:
- using just the first X710 interface for both inbound and outbound traffic.
- two X710 interfaces in a team configuration with the first NIC active (in+outbound) and the other on standby
Both produced the same results. Also tested with LRO/GRO on and off.
The best results were with using just a single interface and GRO/LRO off, which is to be expected. But all combinations ended with my SSH sessions freezing and forwarding dropping to the 100Kbps/s range. After stopping the tester the box comes back to normal (still showing a 60+ load average) after a few minutes or so.
Just before the box freezes there are a lot of ksoftirqd threads quite busy:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20 root 20 0 0 0 0 R 100.0 0.0 1:27.29 [ksoftirqd/2]
60 root 20 0 0 0 0 R 100.0 0.0 1:16.64 [ksoftirqd/10]
90 root 20 0 0 0 0 R 100.0 0.0 1:44.79 [ksoftirqd/16]
160 root 20 0 0 0 0 R 100.0 0.0 1:32.92 [ksoftirqd/30]
220 root 20 0 0 0 0 R 100.0 0.0 1:42.52 [ksoftirqd/42]
50 root 20 0 0 0 0 R 100.0 0.0 1:26.21 [ksoftirqd/8]
120 root 20 0 0 0 0 R 100.0 0.0 1:24.15 [ksoftirqd/22]
190 root 20 0 0 0 0 R 100.0 0.0 1:47.05 [ksoftirqd/36]
200 root 20 0 0 0 0 R 100.0 0.0 1:49.58 [ksoftirqd/38]
210 root 20 0 0 0 0 R 100.0 0.0 1:58.60 [ksoftirqd/40]
230 root 20 0 0 0 0 R 100.0 0.0 1:35.77 [ksoftirqd/44]
240 root 20 0 0 0 0 R 100.0 0.0 1:59.37 [ksoftirqd/46]
250 root 20 0 0 0 0 R 100.0 0.0 1:41.46 [ksoftirqd/48]
260 root 20 0 0 0 0 R 100.0 0.0 1:28.37 [ksoftirqd/50]
280 root 20 0 0 0 0 R 100.0 0.0 1:17.73 [ksoftirqd/54]
100 root 20 0 0 0 0 R 86.3 0.0 1:35.04 [ksoftirqd/18]
150 root 20 0 0 0 0 R 86.3 0.0 1:36.73 [ksoftirqd/28]
30 root 20 0 0 0 0 R 85.3 0.0 1:18.45 [ksoftirqd/4]
40 root 20 0 0 0 0 R 85.3 0.0 1:42.14 [ksoftirqd/6]
70 root 20 0 0 0 0 R 85.3 0.0 1:27.51 [ksoftirqd/12]
110 root 20 0 0 0 0 R 85.3 0.0 1:22.93 [ksoftirqd/20]
130 root 20 0 0 0 0 R 85.3 0.0 1:37.32 [ksoftirqd/24]
140 root 20 0 0 0 0 R 85.3 0.0 1:36.85 [ksoftirqd/26]
3 root 20 0 0 0 0 S 84.3 0.0 2:47.02 [ksoftirqd/0]
80 root 20 0 0 0 0 R 84.3 0.0 1:43.63 [ksoftirqd/14]
270 root 20 0 0 0 0 R 84.3 0.0 1:21.22 [ksoftirqd/52]
170 root 20 0 0 0 0 R 66.7 0.0 1:43.50 [ksoftirqd/32]
180 root 20 0 0 0 0 R 51.0 0.0 0:52.70 [ksoftirqd/34]
444 root 20 0 0 0 0 R 46.1 0.0 1:41.75 [kworker/34:1]
205389 root 20 0 0 0 0 R 19.6 0.0 0:04.53 [kworker/u769:2]
892 root 20 0 110908 69888 69556 S 18.6 0.1 4:06.82 /usr/lib/systemd/systemd-journald
644 root 20 0 0 0 0 S 15.7 0.0 0:19.73 [kworker/14:1]
740 root 20 0 0 0 0 S 15.7 0.0 0:08.38 [kworker/52:1]
26633 root 20 0 0 0 0 S 15.7 0.0 0:35.72 [kworker/6:0]
209211 root 20 0 0 0 0 S 15.7 0.0 0:03.96 [kworker/u769:0]
111 root 20 0 0 0 0 S 14.7 0.0 0:16.67 [kworker/20:0]
541 root 20 0 0 0 0 S 14.7 0.0 0:16.59 [kworker/12:1]
2698 root 20 0 0 0 0 S 14.7 0.0 0:09.77 [kworker/28:1]
12100 root 20 0 0 0 0 S 14.7 0.0 0:17.94 [kworker/18:1]
40451 root 20 0 0 0 0 S 14.7 0.0 0:12.69 [kworker/24:0]
40592 root 20 0 0 0 0 S 14.7 0.0 0:23.30 [kworker/32:1]
212923 root 20 0 0 0 0 S 14.7 0.0 0:02.89 [kworker/4:1]
300255 root 20 0 0 0 0 S 14.7 0.0 0:17.89 [kworker/26:0]
At least I'm making some use of those 28 cores ;)
Cheers!
Sander
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP
URL: <http://mail-lists.nic.mx/pipermail/jool-list/attachments/20170904/ef33a203/attachment.bin>
More information about the Jool-list
mailing list