[Jool-list] How to debug joold?

Thu Dec 19 21:48:15 CST 2019

I don't know if this helps, but I compiled my test into a zip file.
You can try it if you could use a pocket joold sample environment,
hopefully running successfully without fuss.

I attached the zip to this email, but knowing it might trigger bots
I've also uploaded it to my Dropbox:
https://www.dropbox.com/s/q4jfcjdmqw2knd1/joold.zip?dl=0

These are the steps:

1. Run `setup.sh`

This will create two network namespaces (j and k) with one translator
and one joold daemon each, and configure the corresponding networks.

*This script assumes Jool is already installed.*

In your global namespace, the interface "world_to_j" is your packets'
gateways towards j, and "world_to_k" is your packets' gateway towards
k (though this one doesn't do anything at the moment because it lacks
IP addresses). These are virtual interfaces you can view by running
`ip address`.

Conversely, j has "j_to_world" and "j_to_k". k has "k_to_world" and
"k_to_j". These do what you'd expect.

2. Ping yourself through the j translator from the global namespace:

    ping 64:ff9b::192.0.2.8

This will create a session in j's translator, which should hopefully
be joold'd into k:

    sudo ip netns exec j jool session display --icmp --numeric
    sudo ip netns exec k jool session display --icmp --numeric

I can also see the session traffic going from j to k:

    $ sudo ip netns exec j tcpdump -i j_to_k
    03:46:58.851290 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
UDP, length 80
    03:46:58.856125 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
UDP, length 80
    03:46:59.873137 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
UDP, length 80
    03:46:59.873318 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
UDP, length 80

3. Once you're done looking at it, you can everything up by running
`cleanup.sh` in the global namespace:

    ./cleanup.sh

On Thu, Dec 19, 2019 at 9:05 PM Alberto Leiva <ydahhrk at gmail.com> wrote:
>
> First off, allow me to come clean:
>
> If joold seems somewhat underdeveloped, it's because it probably is.
> Since I implemented it back in 2016, it has received next to no
> feedback whatsoever, so it's unclear to me whether it has reached
> production somewhere. The fact that it's rather complicated (ie.
> requires lots of steps) probably hasn't helped.
>
> The initial implementation (which is in fact the current
> implementation still) was designed to be (first of all)
> code-unintrusive. (As in, try to complicate things as little as
> possible.) I figured I would add features on top as users requested
> them, but since this never happened, the whole thing is still quite
> bare-bones.
>
> This is why you don't see things like MLDv2. It's far easier for the
> code to just expect everything to be configured statically.
>
> And it's also why you're running into problems that should by rights
> have been addressed a long time ago.
>
> > How to debug joold?
>
> Well, my syslog seems to print debug messages by default, so it's
> making a fine mess:
>
>     $ tail -10 /var/log/syslog
>     Dec 20 01:06:44 ubuntu18 joold: Sent 80 bytes to the network.
>     Dec 20 01:06:44 ubuntu18 joold: Received a packet from kernelspace.
>     Dec 20 01:06:44 ubuntu18 joold: Sending 80 bytes to the network...
>     Dec 20 01:06:44 ubuntu18 joold: Sent 80 bytes to the network.
>     Dec 20 01:06:45 ubuntu18 joold: Received a packet from kernelspace.
>     Dec 20 01:06:45 ubuntu18 joold: Sending 80 bytes to the network...
>     Dec 20 01:06:45 ubuntu18 joold: Sent 80 bytes to the network.
>     Dec 20 01:06:45 ubuntu18 joold: Received a packet from kernelspace.
>     Dec 20 01:06:45 ubuntu18 joold: Sending 80 bytes to the network...
>     Dec 20 01:06:45 ubuntu18 joold: Sent 80 bytes to the network.
>
> That's the daemon's log. If you also want to see the kernel's debug
> messages, you have to manually compile Jool to include them. Sorry;
> it's convoluted because debug messages need to be compiled out by
> default if I want to achieve any performance.
>
> First, uninstall your current kernel modules:
>
> - If you used DKMS, follow these steps:
> https://jool.mx/en/install.html#kernel-modules-if-installed-by-dkms
> - If you used some package, just remove the package (eg. apt remove jool-dkms).
>
> Then add the -DDEBUG flag [0], then reinstall using DKMS:
>
>     cd <path to Jool's extracted tar.gz or git repository>
>     nano dkms.conf
>     # Change this:
>     # MAKE[0]="make -C ${kernel_source_dir}
> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/common
> modules \
>     #    && make -C ${kernel_source_dir}
> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/nat64
> modules \
>     #    && make -C ${kernel_source_dir}
> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/siit
> modules"
>     # to read like this:
>     # MAKE[0]="make -C ${kernel_source_dir}
> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/common
> JOOL_FLAGS=-DDEBUG modules \
>     #    && make -C ${kernel_source_dir}
> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/nat64
> JOOL_FLAGS=-DDEBUG modules \
>     #    && make -C ${kernel_source_dir}
> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/siit
> JOOL_FLAGS=-DDEBUG modules"
>     # (if it says "SUBDIRS=" instead of "M=", it doesn't matter.)
>     sudo dkms install .
>
> Once you restart the module, Jool should start making a lot of noise,
> which you can query by running `dmesg` or `cat /var/log/kernel`:
>
>     $ dmesg | tail -13
>     [  515.791618] Jool: ===============================================
>     [  515.791620] Jool: Received a request from userspace.
>     [  515.791624] Jool: Received a joold request.
>     [  515.791718] Jool: ===============================================
>     [  515.791720] Jool: Received a request from userspace.
>     [  515.791725] Jool: Received a joold request.
>     [  515.791728] Jool: Adding session!
>     [  515.791735] Jool: Added 1 sessions.
>     [  515.791769] Jool: ===============================================
>     [  515.791771] Jool: Received a request from userspace.
>     [  515.791775] Jool: Received a joold request.
>     [  515.791777] Jool: Adding session!
>     [  515.791780] Jool: Added 1 sessions.
>
> > Checking the session table on one of the two routers it looks good, but
> > the other router does not have the same entry.
>
> Works for me.
>
> Question from my gut: Did you remember to enable session sharing
> (`jool global update ss-enabled true`) in both kernel translator
> instances?
>
> > And I've seen the different instance names - should joold propagate all
> > session accross all instances by default or only selected ones?
>
> Perhaps this is the problem. As it stands, the current implementation
> of joold is hardcoded to only speak with the "default" kernel instance
> (ie. the one named "default") from whichever network namespace it's
> currently running.
>
> This is quite a dumb bug. It exists because I implemented instances
> *after* joold, and because the joold code has been neglecting
> improvement, I never went back to synergize them properly.
> Irritatingly, I also implemented stats *after* joold, so that's a dead
> end as well. (Another bug.)
>
> You're free to report these bugs. As a temporal workaround, you can
> try and make sure all of your instances are named "default".
>
> > Sorry for the newbie questions - jool(d) looks quite nice, but I'm not
> > yet fully used to how it is designed.
>
> Feedback is always welcome, even if it concerns its design. I'm
> implementing all of this as a software developer, and the opinion of
> real admins is often necessary.
>
> [0] https://github.com/NICMx/Jool/wiki/Jool's-Compilation-Options#-ddebug
>
> On Thu, Dec 19, 2019 at 5:40 PM Nico Schottelius
> <nico.schottelius at ungleich.ch> wrote:
> >
> >
> > Hello again,
> >
> > I am giving joold a second try (thanks for the log change!), now using
> > the multicast address ff05:db8:64::64 ("site local" according to
> > wikipedia).
> >
> > Checking the session table on one of the two routers it looks good, but
> > the other router does not have the same entry. Sniffing traffic using
> >
> > tcpdump -ni any port 6464
> >
> > also does not show any outgoing or incoming traffic on that port.
> >
> > Looking for
> >
> > tcpdump -ni any icmp6
> >
> > before starting joold also does not show any traffic. Shouldn't joold
> > send MLDv2 messages?
> >
> > When I attach myself to joold using strace -fF, it stays in recvmsg and
> > does not do anything when a session is created.
> >
> > Or asked differently: how do you usually debug joold communication? I
> > did not see any verbose: true or debug: true flags.
> >
> > And I've seen the different instance names - should joold propagate all
> > session accross all instances by default or only selected ones?
> >
> > Sorry for the newbie questions - jool(d) looks quite nice, but I'm not
> > yet fully used to how it is designed.
> >
> > Best,
> >
> > Nico
> >
> >
> > --
> > Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
> > _______________________________________________
> > Jool-list mailing list
> > Jool-list at nic.mx
> > https://mail-lists.nic.mx/listas/listinfo/jool-list
-------------- next part --------------
A non-text attachment was scrubbed...
Name: joold.zip
Type: application/zip
Size: 2181 bytes
Desc: not available
URL: <http://mail-lists.nic.mx/pipermail/jool-list/attachments/20191219/9fd0d888/attachment-0001.zip>