[Jool-list] How to debug joold?

Thu Dec 19 21:05:45 CST 2019

First off, allow me to come clean:

If joold seems somewhat underdeveloped, it's because it probably is.
Since I implemented it back in 2016, it has received next to no
feedback whatsoever, so it's unclear to me whether it has reached
production somewhere. The fact that it's rather complicated (ie.
requires lots of steps) probably hasn't helped.

The initial implementation (which is in fact the current
implementation still) was designed to be (first of all)
code-unintrusive. (As in, try to complicate things as little as
possible.) I figured I would add features on top as users requested
them, but since this never happened, the whole thing is still quite
bare-bones.

This is why you don't see things like MLDv2. It's far easier for the
code to just expect everything to be configured statically.

And it's also why you're running into problems that should by rights
have been addressed a long time ago.

> How to debug joold?

Well, my syslog seems to print debug messages by default, so it's
making a fine mess:

    $ tail -10 /var/log/syslog
    Dec 20 01:06:44 ubuntu18 joold: Sent 80 bytes to the network.
    Dec 20 01:06:44 ubuntu18 joold: Received a packet from kernelspace.
    Dec 20 01:06:44 ubuntu18 joold: Sending 80 bytes to the network...
    Dec 20 01:06:44 ubuntu18 joold: Sent 80 bytes to the network.
    Dec 20 01:06:45 ubuntu18 joold: Received a packet from kernelspace.
    Dec 20 01:06:45 ubuntu18 joold: Sending 80 bytes to the network...
    Dec 20 01:06:45 ubuntu18 joold: Sent 80 bytes to the network.
    Dec 20 01:06:45 ubuntu18 joold: Received a packet from kernelspace.
    Dec 20 01:06:45 ubuntu18 joold: Sending 80 bytes to the network...
    Dec 20 01:06:45 ubuntu18 joold: Sent 80 bytes to the network.

That's the daemon's log. If you also want to see the kernel's debug
messages, you have to manually compile Jool to include them. Sorry;
it's convoluted because debug messages need to be compiled out by
default if I want to achieve any performance.

First, uninstall your current kernel modules:

- If you used DKMS, follow these steps:
https://jool.mx/en/install.html#kernel-modules-if-installed-by-dkms
- If you used some package, just remove the package (eg. apt remove jool-dkms).

Then add the -DDEBUG flag [0], then reinstall using DKMS:

    cd <path to Jool's extracted tar.gz or git repository>
    nano dkms.conf
    # Change this:
    # MAKE[0]="make -C ${kernel_source_dir}
M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/common
modules \
    #    && make -C ${kernel_source_dir}
M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/nat64
modules \
    #    && make -C ${kernel_source_dir}
M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/siit
modules"
    # to read like this:
    # MAKE[0]="make -C ${kernel_source_dir}
M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/common
JOOL_FLAGS=-DDEBUG modules \
    #    && make -C ${kernel_source_dir}
M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/nat64
JOOL_FLAGS=-DDEBUG modules \
    #    && make -C ${kernel_source_dir}
M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/siit
JOOL_FLAGS=-DDEBUG modules"
    # (if it says "SUBDIRS=" instead of "M=", it doesn't matter.)
    sudo dkms install .

Once you restart the module, Jool should start making a lot of noise,
which you can query by running `dmesg` or `cat /var/log/kernel`:

    $ dmesg | tail -13
    [  515.791618] Jool: ===============================================
    [  515.791620] Jool: Received a request from userspace.
    [  515.791624] Jool: Received a joold request.
    [  515.791718] Jool: ===============================================
    [  515.791720] Jool: Received a request from userspace.
    [  515.791725] Jool: Received a joold request.
    [  515.791728] Jool: Adding session!
    [  515.791735] Jool: Added 1 sessions.
    [  515.791769] Jool: ===============================================
    [  515.791771] Jool: Received a request from userspace.
    [  515.791775] Jool: Received a joold request.
    [  515.791777] Jool: Adding session!
    [  515.791780] Jool: Added 1 sessions.

> Checking the session table on one of the two routers it looks good, but
> the other router does not have the same entry.

Works for me.

Question from my gut: Did you remember to enable session sharing
(`jool global update ss-enabled true`) in both kernel translator
instances?

> And I've seen the different instance names - should joold propagate all
> session accross all instances by default or only selected ones?

Perhaps this is the problem. As it stands, the current implementation
of joold is hardcoded to only speak with the "default" kernel instance
(ie. the one named "default") from whichever network namespace it's
currently running.

This is quite a dumb bug. It exists because I implemented instances
*after* joold, and because the joold code has been neglecting
improvement, I never went back to synergize them properly.
Irritatingly, I also implemented stats *after* joold, so that's a dead
end as well. (Another bug.)

You're free to report these bugs. As a temporal workaround, you can
try and make sure all of your instances are named "default".

> Sorry for the newbie questions - jool(d) looks quite nice, but I'm not
> yet fully used to how it is designed.

Feedback is always welcome, even if it concerns its design. I'm
implementing all of this as a software developer, and the opinion of
real admins is often necessary.

[0] https://github.com/NICMx/Jool/wiki/Jool's-Compilation-Options#-ddebug

On Thu, Dec 19, 2019 at 5:40 PM Nico Schottelius
<nico.schottelius at ungleich.ch> wrote:
>
>
> Hello again,
>
> I am giving joold a second try (thanks for the log change!), now using
> the multicast address ff05:db8:64::64 ("site local" according to
> wikipedia).
>
> Checking the session table on one of the two routers it looks good, but
> the other router does not have the same entry. Sniffing traffic using
>
> tcpdump -ni any port 6464
>
> also does not show any outgoing or incoming traffic on that port.
>
> Looking for
>
> tcpdump -ni any icmp6
>
> before starting joold also does not show any traffic. Shouldn't joold
> send MLDv2 messages?
>
> When I attach myself to joold using strace -fF, it stays in recvmsg and
> does not do anything when a session is created.
>
> Or asked differently: how do you usually debug joold communication? I
> did not see any verbose: true or debug: true flags.
>
> And I've seen the different instance names - should joold propagate all
> session accross all instances by default or only selected ones?
>
> Sorry for the newbie questions - jool(d) looks quite nice, but I'm not
> yet fully used to how it is designed.
>
> Best,
>
> Nico
>
>
> --
> Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
> _______________________________________________
> Jool-list mailing list
> Jool-list at nic.mx
> https://mail-lists.nic.mx/listas/listinfo/jool-list