[Jool-list] How to debug joold?
Nico Schottelius
nico.schottelius at ungleich.ch
Mon Dec 23 06:07:18 CST 2019
Hey Alberto,
thanks a lot for both explanations. Being a FOSS writer myself, I think
you have very fair reasons and already created a great piece of
software.
We from ungleich will try to help improving jool where we can, because
we see quite some potential in it.
A question that came up this weekend: are there are plans to submit jool
to upstream Linux? We have just seen that wireguard is moving forward
quite successful and having jool more available would probably help many
other people who now realise they need nat64.
Best regards,
Nico
Alberto Leiva <ydahhrk at gmail.com> writes:
> I don't know if this helps, but I compiled my test into a zip file.
> You can try it if you could use a pocket joold sample environment,
> hopefully running successfully without fuss.
>
> I attached the zip to this email, but knowing it might trigger bots
> I've also uploaded it to my Dropbox:
> https://www.dropbox.com/s/q4jfcjdmqw2knd1/joold.zip?dl=0
>
> These are the steps:
>
> 1. Run `setup.sh`
>
> This will create two network namespaces (j and k) with one translator
> and one joold daemon each, and configure the corresponding networks.
>
> *This script assumes Jool is already installed.*
>
> In your global namespace, the interface "world_to_j" is your packets'
> gateways towards j, and "world_to_k" is your packets' gateway towards
> k (though this one doesn't do anything at the moment because it lacks
> IP addresses). These are virtual interfaces you can view by running
> `ip address`.
>
> Conversely, j has "j_to_world" and "j_to_k". k has "k_to_world" and
> "k_to_j". These do what you'd expect.
>
> 2. Ping yourself through the j translator from the global namespace:
>
> ping 64:ff9b::192.0.2.8
>
> This will create a session in j's translator, which should hopefully
> be joold'd into k:
>
> sudo ip netns exec j jool session display --icmp --numeric
> sudo ip netns exec k jool session display --icmp --numeric
>
> I can also see the session traffic going from j to k:
>
> $ sudo ip netns exec j tcpdump -i j_to_k
> 03:46:58.851290 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
> UDP, length 80
> 03:46:58.856125 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
> UDP, length 80
> 03:46:59.873137 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
> UDP, length 80
> 03:46:59.873318 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
> UDP, length 80
>
> 3. Once you're done looking at it, you can everything up by running
> `cleanup.sh` in the global namespace:
>
> ./cleanup.sh
>
> On Thu, Dec 19, 2019 at 9:05 PM Alberto Leiva <ydahhrk at gmail.com> wrote:
>>
>> First off, allow me to come clean:
>>
>> If joold seems somewhat underdeveloped, it's because it probably is.
>> Since I implemented it back in 2016, it has received next to no
>> feedback whatsoever, so it's unclear to me whether it has reached
>> production somewhere. The fact that it's rather complicated (ie.
>> requires lots of steps) probably hasn't helped.
>>
>> The initial implementation (which is in fact the current
>> implementation still) was designed to be (first of all)
>> code-unintrusive. (As in, try to complicate things as little as
>> possible.) I figured I would add features on top as users requested
>> them, but since this never happened, the whole thing is still quite
>> bare-bones.
>>
>> This is why you don't see things like MLDv2. It's far easier for the
>> code to just expect everything to be configured statically.
>>
>> And it's also why you're running into problems that should by rights
>> have been addressed a long time ago.
>>
>> > How to debug joold?
>>
>> Well, my syslog seems to print debug messages by default, so it's
>> making a fine mess:
>>
>> $ tail -10 /var/log/syslog
>> Dec 20 01:06:44 ubuntu18 joold: Sent 80 bytes to the network.
>> Dec 20 01:06:44 ubuntu18 joold: Received a packet from kernelspace.
>> Dec 20 01:06:44 ubuntu18 joold: Sending 80 bytes to the network...
>> Dec 20 01:06:44 ubuntu18 joold: Sent 80 bytes to the network.
>> Dec 20 01:06:45 ubuntu18 joold: Received a packet from kernelspace.
>> Dec 20 01:06:45 ubuntu18 joold: Sending 80 bytes to the network...
>> Dec 20 01:06:45 ubuntu18 joold: Sent 80 bytes to the network.
>> Dec 20 01:06:45 ubuntu18 joold: Received a packet from kernelspace.
>> Dec 20 01:06:45 ubuntu18 joold: Sending 80 bytes to the network...
>> Dec 20 01:06:45 ubuntu18 joold: Sent 80 bytes to the network.
>>
>> That's the daemon's log. If you also want to see the kernel's debug
>> messages, you have to manually compile Jool to include them. Sorry;
>> it's convoluted because debug messages need to be compiled out by
>> default if I want to achieve any performance.
>>
>> First, uninstall your current kernel modules:
>>
>> - If you used DKMS, follow these steps:
>> https://jool.mx/en/install.html#kernel-modules-if-installed-by-dkms
>> - If you used some package, just remove the package (eg. apt remove jool-dkms).
>>
>> Then add the -DDEBUG flag [0], then reinstall using DKMS:
>>
>> cd <path to Jool's extracted tar.gz or git repository>
>> nano dkms.conf
>> # Change this:
>> # MAKE[0]="make -C ${kernel_source_dir}
>> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/common
>> modules \
>> # && make -C ${kernel_source_dir}
>> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/nat64
>> modules \
>> # && make -C ${kernel_source_dir}
>> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/siit
>> modules"
>> # to read like this:
>> # MAKE[0]="make -C ${kernel_source_dir}
>> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/common
>> JOOL_FLAGS=-DDEBUG modules \
>> # && make -C ${kernel_source_dir}
>> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/nat64
>> JOOL_FLAGS=-DDEBUG modules \
>> # && make -C ${kernel_source_dir}
>> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/siit
>> JOOL_FLAGS=-DDEBUG modules"
>> # (if it says "SUBDIRS=" instead of "M=", it doesn't matter.)
>> sudo dkms install .
>>
>> Once you restart the module, Jool should start making a lot of noise,
>> which you can query by running `dmesg` or `cat /var/log/kernel`:
>>
>> $ dmesg | tail -13
>> [ 515.791618] Jool: ===============================================
>> [ 515.791620] Jool: Received a request from userspace.
>> [ 515.791624] Jool: Received a joold request.
>> [ 515.791718] Jool: ===============================================
>> [ 515.791720] Jool: Received a request from userspace.
>> [ 515.791725] Jool: Received a joold request.
>> [ 515.791728] Jool: Adding session!
>> [ 515.791735] Jool: Added 1 sessions.
>> [ 515.791769] Jool: ===============================================
>> [ 515.791771] Jool: Received a request from userspace.
>> [ 515.791775] Jool: Received a joold request.
>> [ 515.791777] Jool: Adding session!
>> [ 515.791780] Jool: Added 1 sessions.
>>
>> > Checking the session table on one of the two routers it looks good, but
>> > the other router does not have the same entry.
>>
>> Works for me.
>>
>> Question from my gut: Did you remember to enable session sharing
>> (`jool global update ss-enabled true`) in both kernel translator
>> instances?
>>
>> > And I've seen the different instance names - should joold propagate all
>> > session accross all instances by default or only selected ones?
>>
>> Perhaps this is the problem. As it stands, the current implementation
>> of joold is hardcoded to only speak with the "default" kernel instance
>> (ie. the one named "default") from whichever network namespace it's
>> currently running.
>>
>> This is quite a dumb bug. It exists because I implemented instances
>> *after* joold, and because the joold code has been neglecting
>> improvement, I never went back to synergize them properly.
>> Irritatingly, I also implemented stats *after* joold, so that's a dead
>> end as well. (Another bug.)
>>
>> You're free to report these bugs. As a temporal workaround, you can
>> try and make sure all of your instances are named "default".
>>
>> > Sorry for the newbie questions - jool(d) looks quite nice, but I'm not
>> > yet fully used to how it is designed.
>>
>> Feedback is always welcome, even if it concerns its design. I'm
>> implementing all of this as a software developer, and the opinion of
>> real admins is often necessary.
>>
>> [0] https://github.com/NICMx/Jool/wiki/Jool's-Compilation-Options#-ddebug
>>
>> On Thu, Dec 19, 2019 at 5:40 PM Nico Schottelius
>> <nico.schottelius at ungleich.ch> wrote:
>> >
>> >
>> > Hello again,
>> >
>> > I am giving joold a second try (thanks for the log change!), now using
>> > the multicast address ff05:db8:64::64 ("site local" according to
>> > wikipedia).
>> >
>> > Checking the session table on one of the two routers it looks good, but
>> > the other router does not have the same entry. Sniffing traffic using
>> >
>> > tcpdump -ni any port 6464
>> >
>> > also does not show any outgoing or incoming traffic on that port.
>> >
>> > Looking for
>> >
>> > tcpdump -ni any icmp6
>> >
>> > before starting joold also does not show any traffic. Shouldn't joold
>> > send MLDv2 messages?
>> >
>> > When I attach myself to joold using strace -fF, it stays in recvmsg and
>> > does not do anything when a session is created.
>> >
>> > Or asked differently: how do you usually debug joold communication? I
>> > did not see any verbose: true or debug: true flags.
>> >
>> > And I've seen the different instance names - should joold propagate all
>> > session accross all instances by default or only selected ones?
>> >
>> > Sorry for the newbie questions - jool(d) looks quite nice, but I'm not
>> > yet fully used to how it is designed.
>> >
>> > Best,
>> >
>> > Nico
>> >
>> >
>> > --
>> > Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
>> > _______________________________________________
>> > Jool-list mailing list
>> > Jool-list at nic.mx
>> > https://mail-lists.nic.mx/listas/listinfo/jool-list
--
Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
More information about the Jool-list
mailing list