<div dir="ltr"><div><div>> Considering your configuration, if this machine has more addresses than one...<br><br></div>Scratch that; I just noticed you're using more than one address per protocol.<br><br></div>This means that the traversal is `(65535 - 1025 + 1) * 16` nodes long. Oops.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 4, 2017 at 7:09 PM, Alberto Leiva <span dir="ltr"><<a href="mailto:ydahhrk@gmail.com" target="_blank">ydahhrk@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hmmmmmmmmmmmmmmmmmmmmmmmmmm. I have a theory. There might be an optimization that could fix this to a massive extent. Although I have the feeling that I've already thought about this before, so there might be a good reason why I didn't apply this optimization already. Then again, I could just have forgotten while developing.<br><br>How many addresses does the attacker machine have? I don't really know the nature of TRex's test, but I can picture it trying to use as many of its node's IPv6 addresses and ports to open as many connections as possible through Jool. Considering your configuration, if this machine has more addresses than one... then I think I see the problem. It's a very, very stupid oversight of mine. And one I am extremely surprised that also managed to slip past the performance tests so easily.<br><br>Then again, I haven't had to dive deep into the session code for a while, so I might simply be missing this optimization. I badly need to look into this.<span class=""><br><br>> Q2: what can we do to improve this?<br><br></span>Ok, here's my attempt to explain it:<br><br>There should (in theory) exist a quick way for Jool to tell whether the pool4 has been exhausted (ie. there is one existing BIB entry for every available pool4 address). If this information were available, it should be able to skip *an entire tree traversal* for every translated packet that needs the creation of a new BIB entry.<br><br>In your case, the tree traversal is `65535 - 1025 + 1 = 64511` nodes long. And yes, this operation has to lock, because otherwise it can end up with a corrupted tree. So yeah, this is really bad.<br><br>Now, I think I'm starting to realize why I might not have implemented this: Because pool4 is intended as a mostly static database (because this helps minimize locking), and usage counters would break this. So implementing this might end up being a tradeoff. But give me a few days; I'll think about it.<br><br></div>Maybe I also assumed that the admin would always grant enough addresses to pool4. If pool4 has enough addresses to actually serve the traffic, it won't waste so much time iterating pointlessly.<br><br></div><span class="">> PS: Developers that want to work on this and would like access to my lab boxes (Dell R630 with lots of cores, memory and some 10Gbit/s NICs): feel free to contact me!<br></span><div><div><div><br>That'd be interesting. If my theory proves to be a flop, this would be helpful in spotting the real bottleneck; I don't have anything right now capable of overwhelming Jool.<br><br>> Lockless/LRU structures?<br><br>I'd love to find a way to do this locklessly, but I haven't been proven that creative.<br><br>The problem is that there are two separate trees that need to be kept in harmony (one for IPv4 BIB lookups, another for IPv6 BIB lookups), otherwise Jool will inevitably create conflicting entries and traffic will start going in mismatched directions. At the same time, the BIB needs to be kept consistent with pool4, so the creation of a BIB entry also depends on many other BIB entries.<br><br>On top of that, this all happens in interrupt context so Jool can't afford itself the luxury of a mutex. It *has* to be a dirty spinlock.<br><br>Come to think of it, Jool 4 might not necessarily translate in interrupt context so the latter constraint could be overcomed. Interesting.<br><br>> "f-args": 10,<br><br>This should be contributing to the problem. But not by much, I think.<br><br>Actually, if TRex's test is random, it's probably actually not contributing much at all. It would be more of an issue in a real environment.<span class=""><br><br>> kernel:NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [kworker/u769:0:209211]<br><br></span>Yeah... this is not acceptable. I really hope I can find a fix.<br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Mon, Sep 4, 2017 at 4:23 PM, Sander Steffann <span dir="ltr"><<a href="mailto:sander@steffann.nl" target="_blank">sander@steffann.nl</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">Hi,<br>
<div><div class="m_-8944617845623695607h5"><br>
> Just before the box freezes there are a lot of ksoftirqd threads quite busy:<br>
><br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
> 20 root 20 0 0 0 0 R 100.0 0.0 1:27.29 [ksoftirqd/2]<br>
> 60 root 20 0 0 0 0 R 100.0 0.0 1:16.64 [ksoftirqd/10]<br>
> 90 root 20 0 0 0 0 R 100.0 0.0 1:44.79 [ksoftirqd/16]<br>
> 160 root 20 0 0 0 0 R 100.0 0.0 1:32.92 [ksoftirqd/30]<br>
> 220 root 20 0 0 0 0 R 100.0 0.0 1:42.52 [ksoftirqd/42]<br>
> 50 root 20 0 0 0 0 R 100.0 0.0 1:26.21 [ksoftirqd/8]<br>
> 120 root 20 0 0 0 0 R 100.0 0.0 1:24.15 [ksoftirqd/22]<br>
> 190 root 20 0 0 0 0 R 100.0 0.0 1:47.05 [ksoftirqd/36]<br>
> 200 root 20 0 0 0 0 R 100.0 0.0 1:49.58 [ksoftirqd/38]<br>
> 210 root 20 0 0 0 0 R 100.0 0.0 1:58.60 [ksoftirqd/40]<br>
> 230 root 20 0 0 0 0 R 100.0 0.0 1:35.77 [ksoftirqd/44]<br>
> 240 root 20 0 0 0 0 R 100.0 0.0 1:59.37 [ksoftirqd/46]<br>
> 250 root 20 0 0 0 0 R 100.0 0.0 1:41.46 [ksoftirqd/48]<br>
> 260 root 20 0 0 0 0 R 100.0 0.0 1:28.37 [ksoftirqd/50]<br>
> 280 root 20 0 0 0 0 R 100.0 0.0 1:17.73 [ksoftirqd/54]<br>
> 100 root 20 0 0 0 0 R 86.3 0.0 1:35.04 [ksoftirqd/18]<br>
> 150 root 20 0 0 0 0 R 86.3 0.0 1:36.73 [ksoftirqd/28]<br>
> 30 root 20 0 0 0 0 R 85.3 0.0 1:18.45 [ksoftirqd/4]<br>
> 40 root 20 0 0 0 0 R 85.3 0.0 1:42.14 [ksoftirqd/6]<br>
> 70 root 20 0 0 0 0 R 85.3 0.0 1:27.51 [ksoftirqd/12]<br>
> 110 root 20 0 0 0 0 R 85.3 0.0 1:22.93 [ksoftirqd/20]<br>
> 130 root 20 0 0 0 0 R 85.3 0.0 1:37.32 [ksoftirqd/24]<br>
> 140 root 20 0 0 0 0 R 85.3 0.0 1:36.85 [ksoftirqd/26]<br>
> 3 root 20 0 0 0 0 S 84.3 0.0 2:47.02 [ksoftirqd/0]<br>
> 80 root 20 0 0 0 0 R 84.3 0.0 1:43.63 [ksoftirqd/14]<br>
> 270 root 20 0 0 0 0 R 84.3 0.0 1:21.22 [ksoftirqd/52]<br>
> 170 root 20 0 0 0 0 R 66.7 0.0 1:43.50 [ksoftirqd/32]<br>
> 180 root 20 0 0 0 0 R 51.0 0.0 0:52.70 [ksoftirqd/34]<br>
> 444 root 20 0 0 0 0 R 46.1 0.0 1:41.75 [kworker/34:1]<br>
> 205389 root 20 0 0 0 0 R 19.6 0.0 0:04.53 [kworker/u769:2]<br>
> 892 root 20 0 110908 69888 69556 S 18.6 0.1 4:06.82 /usr/lib/systemd/systemd-journ<wbr>ald<br>
> 644 root 20 0 0 0 0 S 15.7 0.0 0:19.73 [kworker/14:1]<br>
> 740 root 20 0 0 0 0 S 15.7 0.0 0:08.38 [kworker/52:1]<br>
> 26633 root 20 0 0 0 0 S 15.7 0.0 0:35.72 [kworker/6:0]<br>
> 209211 root 20 0 0 0 0 S 15.7 0.0 0:03.96 [kworker/u769:0]<br>
> 111 root 20 0 0 0 0 S 14.7 0.0 0:16.67 [kworker/20:0]<br>
> 541 root 20 0 0 0 0 S 14.7 0.0 0:16.59 [kworker/12:1]<br>
> 2698 root 20 0 0 0 0 S 14.7 0.0 0:09.77 [kworker/28:1]<br>
> 12100 root 20 0 0 0 0 S 14.7 0.0 0:17.94 [kworker/18:1]<br>
> 40451 root 20 0 0 0 0 S 14.7 0.0 0:12.69 [kworker/24:0]<br>
> 40592 root 20 0 0 0 0 S 14.7 0.0 0:23.30 [kworker/32:1]<br>
> 212923 root 20 0 0 0 0 S 14.7 0.0 0:02.89 [kworker/4:1]<br>
> 300255 root 20 0 0 0 0 S 14.7 0.0 0:17.89 [kworker/26:0]<br>
><br>
> At least I'm making some use of those 28 cores ;)<br>
<br>
</div></div>Fun addition: the kernel just warned me right after I could log back in:<br>
<br>
Message from <a href="mailto:syslogd@tr3.retevia.eu" target="_blank">syslogd@tr3.retevia.eu</a> at Sep 4 23:15:42 ...<br>
kernel:NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [kworker/u769:0:209211]<br>
<br>
Very busy indeed :)<br>
<br>
Cheers!<br>
<span class="m_-8944617845623695607HOEnZb"><font color="#888888">Sander<br>
<br>
</font></span><br></div></div><span class="">______________________________<wbr>_________________<br>
Jool-list mailing list<br>
<a href="mailto:Jool-list@nic.mx" target="_blank">Jool-list@nic.mx</a><br>
<a href="https://mail-lists.nic.mx/listas/listinfo/jool-list" rel="noreferrer" target="_blank">https://mail-lists.nic.mx/list<wbr>as/listinfo/jool-list</a><br>
<br></span></blockquote></div><br></div>
</blockquote></div><br></div>