[Jool-list] How to debug joold?
Alberto Leiva
ydahhrk at gmail.com
Mon Dec 23 22:22:01 CST 2019
We actually have an issue for that: https://github.com/NICMx/Jool/issues/273
(If you want to continue this topic, I'd thank you to do so there.)
On Mon, Dec 23, 2019 at 6:08 AM Nico Schottelius
<nico.schottelius at ungleich.ch> wrote:
>
>
> Hey Alberto,
>
> thanks a lot for both explanations. Being a FOSS writer myself, I think
> you have very fair reasons and already created a great piece of
> software.
>
> We from ungleich will try to help improving jool where we can, because
> we see quite some potential in it.
>
> A question that came up this weekend: are there are plans to submit jool
> to upstream Linux? We have just seen that wireguard is moving forward
> quite successful and having jool more available would probably help many
> other people who now realise they need nat64.
>
> Best regards,
>
> Nico
>
>
> Alberto Leiva <ydahhrk at gmail.com> writes:
>
> > I don't know if this helps, but I compiled my test into a zip file.
> > You can try it if you could use a pocket joold sample environment,
> > hopefully running successfully without fuss.
> >
> > I attached the zip to this email, but knowing it might trigger bots
> > I've also uploaded it to my Dropbox:
> > https://www.dropbox.com/s/q4jfcjdmqw2knd1/joold.zip?dl=0
> >
> > These are the steps:
> >
> > 1. Run `setup.sh`
> >
> > This will create two network namespaces (j and k) with one translator
> > and one joold daemon each, and configure the corresponding networks.
> >
> > *This script assumes Jool is already installed.*
> >
> > In your global namespace, the interface "world_to_j" is your packets'
> > gateways towards j, and "world_to_k" is your packets' gateway towards
> > k (though this one doesn't do anything at the moment because it lacks
> > IP addresses). These are virtual interfaces you can view by running
> > `ip address`.
> >
> > Conversely, j has "j_to_world" and "j_to_k". k has "k_to_world" and
> > "k_to_j". These do what you'd expect.
> >
> > 2. Ping yourself through the j translator from the global namespace:
> >
> > ping 64:ff9b::192.0.2.8
> >
> > This will create a session in j's translator, which should hopefully
> > be joold'd into k:
> >
> > sudo ip netns exec j jool session display --icmp --numeric
> > sudo ip netns exec k jool session display --icmp --numeric
> >
> > I can also see the session traffic going from j to k:
> >
> > $ sudo ip netns exec j tcpdump -i j_to_k
> > 03:46:58.851290 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
> > UDP, length 80
> > 03:46:58.856125 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
> > UDP, length 80
> > 03:46:59.873137 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
> > UDP, length 80
> > 03:46:59.873318 IP6 2001:db8:ff08::4.6464 > ff05:db8:64::64.6464:
> > UDP, length 80
> >
> > 3. Once you're done looking at it, you can everything up by running
> > `cleanup.sh` in the global namespace:
> >
> > ./cleanup.sh
> >
> > On Thu, Dec 19, 2019 at 9:05 PM Alberto Leiva <ydahhrk at gmail.com> wrote:
> >>
> >> First off, allow me to come clean:
> >>
> >> If joold seems somewhat underdeveloped, it's because it probably is.
> >> Since I implemented it back in 2016, it has received next to no
> >> feedback whatsoever, so it's unclear to me whether it has reached
> >> production somewhere. The fact that it's rather complicated (ie.
> >> requires lots of steps) probably hasn't helped.
> >>
> >> The initial implementation (which is in fact the current
> >> implementation still) was designed to be (first of all)
> >> code-unintrusive. (As in, try to complicate things as little as
> >> possible.) I figured I would add features on top as users requested
> >> them, but since this never happened, the whole thing is still quite
> >> bare-bones.
> >>
> >> This is why you don't see things like MLDv2. It's far easier for the
> >> code to just expect everything to be configured statically.
> >>
> >> And it's also why you're running into problems that should by rights
> >> have been addressed a long time ago.
> >>
> >> > How to debug joold?
> >>
> >> Well, my syslog seems to print debug messages by default, so it's
> >> making a fine mess:
> >>
> >> $ tail -10 /var/log/syslog
> >> Dec 20 01:06:44 ubuntu18 joold: Sent 80 bytes to the network.
> >> Dec 20 01:06:44 ubuntu18 joold: Received a packet from kernelspace.
> >> Dec 20 01:06:44 ubuntu18 joold: Sending 80 bytes to the network...
> >> Dec 20 01:06:44 ubuntu18 joold: Sent 80 bytes to the network.
> >> Dec 20 01:06:45 ubuntu18 joold: Received a packet from kernelspace.
> >> Dec 20 01:06:45 ubuntu18 joold: Sending 80 bytes to the network...
> >> Dec 20 01:06:45 ubuntu18 joold: Sent 80 bytes to the network.
> >> Dec 20 01:06:45 ubuntu18 joold: Received a packet from kernelspace.
> >> Dec 20 01:06:45 ubuntu18 joold: Sending 80 bytes to the network...
> >> Dec 20 01:06:45 ubuntu18 joold: Sent 80 bytes to the network.
> >>
> >> That's the daemon's log. If you also want to see the kernel's debug
> >> messages, you have to manually compile Jool to include them. Sorry;
> >> it's convoluted because debug messages need to be compiled out by
> >> default if I want to achieve any performance.
> >>
> >> First, uninstall your current kernel modules:
> >>
> >> - If you used DKMS, follow these steps:
> >> https://jool.mx/en/install.html#kernel-modules-if-installed-by-dkms
> >> - If you used some package, just remove the package (eg. apt remove jool-dkms).
> >>
> >> Then add the -DDEBUG flag [0], then reinstall using DKMS:
> >>
> >> cd <path to Jool's extracted tar.gz or git repository>
> >> nano dkms.conf
> >> # Change this:
> >> # MAKE[0]="make -C ${kernel_source_dir}
> >> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/common
> >> modules \
> >> # && make -C ${kernel_source_dir}
> >> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/nat64
> >> modules \
> >> # && make -C ${kernel_source_dir}
> >> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/siit
> >> modules"
> >> # to read like this:
> >> # MAKE[0]="make -C ${kernel_source_dir}
> >> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/common
> >> JOOL_FLAGS=-DDEBUG modules \
> >> # && make -C ${kernel_source_dir}
> >> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/nat64
> >> JOOL_FLAGS=-DDEBUG modules \
> >> # && make -C ${kernel_source_dir}
> >> M=${dkms_tree}/${PACKAGE_NAME}/${PACKAGE_VERSION}/build/src/mod/siit
> >> JOOL_FLAGS=-DDEBUG modules"
> >> # (if it says "SUBDIRS=" instead of "M=", it doesn't matter.)
> >> sudo dkms install .
> >>
> >> Once you restart the module, Jool should start making a lot of noise,
> >> which you can query by running `dmesg` or `cat /var/log/kernel`:
> >>
> >> $ dmesg | tail -13
> >> [ 515.791618] Jool: ===============================================
> >> [ 515.791620] Jool: Received a request from userspace.
> >> [ 515.791624] Jool: Received a joold request.
> >> [ 515.791718] Jool: ===============================================
> >> [ 515.791720] Jool: Received a request from userspace.
> >> [ 515.791725] Jool: Received a joold request.
> >> [ 515.791728] Jool: Adding session!
> >> [ 515.791735] Jool: Added 1 sessions.
> >> [ 515.791769] Jool: ===============================================
> >> [ 515.791771] Jool: Received a request from userspace.
> >> [ 515.791775] Jool: Received a joold request.
> >> [ 515.791777] Jool: Adding session!
> >> [ 515.791780] Jool: Added 1 sessions.
> >>
> >> > Checking the session table on one of the two routers it looks good, but
> >> > the other router does not have the same entry.
> >>
> >> Works for me.
> >>
> >> Question from my gut: Did you remember to enable session sharing
> >> (`jool global update ss-enabled true`) in both kernel translator
> >> instances?
> >>
> >> > And I've seen the different instance names - should joold propagate all
> >> > session accross all instances by default or only selected ones?
> >>
> >> Perhaps this is the problem. As it stands, the current implementation
> >> of joold is hardcoded to only speak with the "default" kernel instance
> >> (ie. the one named "default") from whichever network namespace it's
> >> currently running.
> >>
> >> This is quite a dumb bug. It exists because I implemented instances
> >> *after* joold, and because the joold code has been neglecting
> >> improvement, I never went back to synergize them properly.
> >> Irritatingly, I also implemented stats *after* joold, so that's a dead
> >> end as well. (Another bug.)
> >>
> >> You're free to report these bugs. As a temporal workaround, you can
> >> try and make sure all of your instances are named "default".
> >>
> >> > Sorry for the newbie questions - jool(d) looks quite nice, but I'm not
> >> > yet fully used to how it is designed.
> >>
> >> Feedback is always welcome, even if it concerns its design. I'm
> >> implementing all of this as a software developer, and the opinion of
> >> real admins is often necessary.
> >>
> >> [0] https://github.com/NICMx/Jool/wiki/Jool's-Compilation-Options#-ddebug
> >>
> >> On Thu, Dec 19, 2019 at 5:40 PM Nico Schottelius
> >> <nico.schottelius at ungleich.ch> wrote:
> >> >
> >> >
> >> > Hello again,
> >> >
> >> > I am giving joold a second try (thanks for the log change!), now using
> >> > the multicast address ff05:db8:64::64 ("site local" according to
> >> > wikipedia).
> >> >
> >> > Checking the session table on one of the two routers it looks good, but
> >> > the other router does not have the same entry. Sniffing traffic using
> >> >
> >> > tcpdump -ni any port 6464
> >> >
> >> > also does not show any outgoing or incoming traffic on that port.
> >> >
> >> > Looking for
> >> >
> >> > tcpdump -ni any icmp6
> >> >
> >> > before starting joold also does not show any traffic. Shouldn't joold
> >> > send MLDv2 messages?
> >> >
> >> > When I attach myself to joold using strace -fF, it stays in recvmsg and
> >> > does not do anything when a session is created.
> >> >
> >> > Or asked differently: how do you usually debug joold communication? I
> >> > did not see any verbose: true or debug: true flags.
> >> >
> >> > And I've seen the different instance names - should joold propagate all
> >> > session accross all instances by default or only selected ones?
> >> >
> >> > Sorry for the newbie questions - jool(d) looks quite nice, but I'm not
> >> > yet fully used to how it is designed.
> >> >
> >> > Best,
> >> >
> >> > Nico
> >> >
> >> >
> >> > --
> >> > Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
> >> > _______________________________________________
> >> > Jool-list mailing list
> >> > Jool-list at nic.mx
> >> > https://mail-lists.nic.mx/listas/listinfo/jool-list
>
>
> --
> Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch
More information about the Jool-list
mailing list