[Jool-list] netfilter mode question or bug

Mon Mar 30 00:50:58 CDT 2020

Hi

If we should chose one of them, I chose option c. But I will chose all 
of them, if it possible.
Because; We are using trace for debug. So we may need trace every 
process(possible) in the jool.

1(ID1)>Received(a)
2(ID1)>Processing(b) - matched instance and rules
3(ID1)>Translated/NonTranslated(c)
4(ID1)>Send(d)

Maybe this will be a trace level option.

Fatih USTA

On 28.03.2020 01:23, Alberto Leiva wrote:
> Question:
>
> When is the ideal point in time in which should Jool print the trace?
>
> a) As soon as it receives a packet
> b) Somewhere in the middle of a translation (when?)
> c) After having translated successfully, right before sending the packet
> d) After sending the packet
>
> The trace is currently being printed during a).
> I think the answer depends on whether the trace is intended to show
> all packets, or only the packets that will end up translated
> successfully.
>
> On Mon, Jan 6, 2020 at 9:13 AM Alberto Leiva <ydahhrk at gmail.com> wrote:
>> But TCP and UDP do not have ICMP identifiers. They have ports, which
>> are being printed after the hash symbol of each corresponding IP
>> address.
>>
>> eg.
>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv4/TCP
>> SRC:10.200.200.220#80 DST:10.100.100.11#47230
>>
>> source address: 10.200.200.220
>> destination address: 10.100.100.11
>> TCP source port: 80
>> TCP destination port: 47230
>>
>> On Mon, Jan 6, 2020 at 12:35 AM Fatih USTA <fatihusta86 at gmail.com> wrote:
>>> I mean, ID only showing icmp packets. Is it possible for tcp or udp?
>>>
>>> Jan  6 09:31:48 2020 kernel: : [1472656.480540] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv6/ICMP
>>> SRC:fe80::fc26:33ff:fe79:5b74 DST:fe80::48d8:2aff:fe8b:4a27 TYPE:136
>>> CODE:0 ID:16384
>>> Jan  6 09:31:48 2020 kernel: : [1472656.506080] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv6/TCP
>>> SRC:2001:db8:a::a64:640b#47230 DST:2001:db8:a::ac8:c8dc#80
>>> Jan  6 09:31:48 2020 kernel: : [1472656.506413] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv4/TCP
>>> SRC:10.200.200.220#80 DST:10.100.100.11#47230
>>> Jan  6 09:31:48 2020 kernel: : [1472656.506657] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv6/TCP
>>> SRC:2001:db8:a::a64:640b#47230 DST:2001:db8:a::ac8:c8dc#80
>>> Jan  6 09:31:48 2020 kernel: : [1472656.506759] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv6/TCP
>>> SRC:2001:db8:a::a64:640b#47230 DST:2001:db8:a::ac8:c8dc#80
>>> Jan  6 09:31:48 2020 kernel: : [1472656.507000] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv4/TCP
>>> SRC:10.200.200.220#80 DST:10.100.100.11#47230
>>> Jan  6 09:31:48 2020 kernel: : [1472656.508352] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv4/TCP
>>> SRC:10.200.200.220#80 DST:10.100.100.11#47230
>>> Jan  6 09:31:48 2020 kernel: : [1472656.508440] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv4/TCP
>>> SRC:10.200.200.220#80 DST:10.100.100.11#47230
>>> Jan  6 09:31:48 2020 kernel: : [1472656.508720] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv6/TCP
>>> SRC:2001:db8:a::a64:640b#47230 DST:2001:db8:a::ac8:c8dc#80
>>> Jan  6 09:31:48 2020 kernel: : [1472656.508825] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv6/TCP
>>> SRC:2001:db8:a::a64:640b#47230 DST:2001:db8:a::ac8:c8dc#80
>>> Jan  6 09:31:48 2020 kernel: : [1472656.508903] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv6/TCP
>>> SRC:2001:db8:a::a64:640b#47230 DST:2001:db8:a::ac8:c8dc#80
>>> Jan  6 09:31:48 2020 kernel: : [1472656.509130] Jool:
>>> INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv4/TCP
>>> SRC:10.200.200.220#80 DST:10.100.100.11#47230
>>>
>>>
>>> Fatih USTA
>>>
>>> On 1.01.2020 00:36, Alberto Leiva wrote:
>>>> Sorry, I don't understand you. What do you mean "tcp4/6, udp4/6"?
>>>>
>>>> On Mon, Dec 30, 2019 at 12:43 AM Fatih USTA <fatihusta86 at gmail.com> wrote:
>>>>> Hi
>>>>>
>>>>> It looks good.
>>>>> TRACE: nat:PREROUTING:policy:1 IN=eth1 OUT=
>>>>> MAC=4e:1e:08:4a:fd:68:9a:0d:a9:dd:aa:b5:08:00 SRC=10.200.200.220
>>>>> DST=10.100.100.11 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=48678 DF
>>>>> PROTO=ICMP TYPE=8 CODE=0 ID=2985 SEQ=1
>>>>> Jool: INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv4/ICMP
>>>>> SRC:10.200.200.220 DST:10.100.100.11 TYPE:8 CODE:0 ID:2985
>>>>> ......
>>>>> Jool: INSTANCE:SIIT/ffffffff80e868c0/default PROTO:IPv6/ICMP
>>>>> SRC:2001:db8:a::a64:640b DST:2001:db8:a::ac8:c8dc TYPE:129 CODE:0 ID:2985
>>>>> TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth1 SRC=10.100.100.11
>>>>> DST=10.200.200.220 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=21649 PROTO=ICMP
>>>>> TYPE=0 CODE=0 ID=2985 SEQ=1
>>>>>
>>>>> I think that's enough but What do you think about the tcp4/6, udp4/6?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Fatih USTA
>>>>>
>>>>> On 30.12.2019 06:47, Alberto Leiva wrote:
>>>>>> Hello
>>>>>>
>>>>>> Sorry I can't answer immediately.
>>>>>> I just uploaded a commit adding instance stateness and namespace, as
>>>>>> well as the ICMP ID for ICMP traces.
>>>>>>
>>>>>> How does it look?
>>>>>>
>>>>>> On Tue, Dec 24, 2019 at 12:52 AM Fatih USTA <fatihusta86 at gmail.com> wrote:
>>>>>>> You're right, I can write the iptables trace rule. It's just an idea for a better trace in jool. If I have 1Gbit traffic when I enable trace, many logs will come. Actually not important.
>>>>>>>
>>>>>>> Last thing, it would be nice to have ID into log for package relation like iptables.
>>>>>>>
>>>>>>> TRACE: raw:PREROUTING:policy:2 IN=eth1 OUT= MAC=4e:1e:08:4a:fd:68:9a:0d:a9:dd:aa:b5:08:00 SRC=10.200.200.220 DST=10.100.100.11 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=23871 DF PROTO=ICMP TYPE=8 CODE=0 ID=13069 SEQ=1
>>>>>>> TRACE: mangle:PREROUTING:policy:1 IN=eth1 OUT= MAC=4e:1e:08:4a:fd:68:9a:0d:a9:dd:aa:b5:08:00 SRC=10.200.200.220 DST=10.100.100.11 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=23871 DF PROTO=ICMP TYPE=8 CODE=0 ID=13069 SEQ=1
>>>>>>> TRACE: nat:PREROUTING:policy:1 IN=eth1 OUT= MAC=4e:1e:08:4a:fd:68:9a:0d:a9:dd:aa:b5:08:00 SRC=10.200.200.220 DST=10.100.100.11 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=23871 DF PROTO=ICMP TYPE=8 CODE=0 ID=13069 SEQ=1
>>>>>>>
>>>>>>> Jool: INSTANCE:default PROTO:IPv4/ICMP SRC:10.200.200.220 DST:10.100.100.11 TYPE:8 CODE:0
>>>>>>>
>>>>>>> TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth2 SRC=2001:0db8:000a:0000:0000:0000:0ac8:c8dc DST=2001:0db8:000a:0000:0000:0000:0a64:640b LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=13069 SEQ=1
>>>>>>> TRACE: raw:PREROUTING:policy:2 IN=eth2 OUT= MAC=4a:d8:2a:8b:4a:27:fe:26:33:79:5b:74:86:dd SRC=2001:0db8:000a:0000:0000:0000:0a64:640b DST=2001:0db8:000a:0000:0000:0000:0ac8:c8dc LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=983710 PROTO=ICMPv6 TYPE=129 CODE=0 ID=13069 SEQ=1
>>>>>>> TRACE: mangle:PREROUTING:policy:1 IN=eth2 OUT= MAC=4a:d8:2a:8b:4a:27:fe:26:33:79:5b:74:86:dd SRC=2001:0db8:000a:0000:0000:0000:0a64:640b DST=2001:0db8:000a:0000:0000:0000:0ac8:c8dc LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=983710 PROTO=ICMPv6 TYPE=129 CODE=0 ID=13069 SEQ=1
>>>>>>>
>>>>>>> Jool: INSTANCE:default PROTO:IPv6/ICMP SRC:2001:db8:a::a64:640b DST:2001:db8:a::ac8:c8dc TYPE:129 CODE:0
>>>>>>>
>>>>>>> TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth1 SRC=10.100.100.11 DST=10.200.200.220 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=52293 PROTO=ICMP TYPE=0 CODE=0 ID=13069 SEQ=1
>>>>>>>
>>>>>>>
>>>>>>> Fatih USTA
>>>>>>>
>>>>>>> On 24.12.2019 07:28, Alberto Leiva wrote:
>>>>>>>
>>>>>>> Adding filters complicates it a lot. I have a question: What's
>>>>>>> stopping you from adding a TRACE target right before the Jool target?
>>>>>>>
>>>>>>> for example
>>>>>>>
>>>>>>> iptables -t raw -A PREROUTING <filters> -j TRACE
>>>>>>> iptables -t raw -A PREROUTING <filters> -j JOOL (Jool arguments)
>>>>>>>
>>>>>>> That would trace all packets right before they reach Jool.
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Dec 23, 2019 at 1:01 AM Fatih USTA <fatihusta86 at gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Alberto
>>>>>>>
>>>>>>> I tested. Works well, but we need more information in log for better trace.
>>>>>>> Because jool siit and jool have same instance name. For example Default.
>>>>>>> I can't see which one instance matched.
>>>>>>>
>>>>>>> Dec 23 09:35:40 2019 kernel: : [263288.781040] Jool: INSTANCE:default PROTO:IPv4/ICMP SRC:10.200.200.220 DST:10.100.100.11 TYPE:8 CODE:0
>>>>>>> Dec 23 09:35:40 2019 kernel: : [263288.781401] Jool: INSTANCE:default PROTO:IPv6/ICMP SRC:2001:db8:a::a64:640b DST:2001:db8:a::ac8:c8dc TYPE:129 CODE:0
>>>>>>> Dec 23 09:35:41 2019 kernel: : [263289.573935] Jool: INSTANCE:default PROTO:IPv4/TCP SRC:10.200.200.1#43196 DST:10.200.200.16#22
>>>>>>> Dec 23 09:35:41 2019 kernel: : [263289.805122] Jool: INSTANCE:default PROTO:IPv4/ICMP SRC:10.200.200.220 DST:10.100.100.11 TYPE:8 CODE:0
>>>>>>> Dec 23 09:35:41 2019 kernel: : [263289.805456] Jool: INSTANCE:default PROTO:IPv6/ICMP SRC:2001:db8:a::a64:640b DST:2001:db8:a::ac8:c8dc TYPE:129 CODE:0
>>>>>>> Dec 23 09:35:42 2019 kernel: : [263290.574131] Jool: INSTANCE:default PROTO:IPv4/TCP SRC:10.200.200.1#43196 DST:10.200.200.16#22
>>>>>>> Dec 23 09:35:43 2019 kernel: : [263291.574381] Jool: INSTANCE:default PROTO:IPv4/TCP SRC:10.200.200.1#43196 DST:10.200.200.16#22
>>>>>>> Dec 23 09:35:43 2019 kernel: : [263291.777504] Jool: INSTANCE:default PROTO:IPv6/ICMP SRC:2001:db8:a::a64:640b DST:fe80::48d8:2aff:fe8b:4a27 TYPE:136 CODE:0
>>>>>>> Dec 23 09:35:43 2019 kernel: : [263291.885362] Jool: INSTANCE:default PROTO:IPv6/ICMP SRC:fe80::fc26:33ff:fe79:5b74 DST:2001:db8:a::a64:6402 TYPE:135 CODE:0
>>>>>>> Dec 23 09:35:44 2019 kernel: : [263292.574572] Jool: INSTANCE:default PROTO:IPv4/TCP SRC:10.200.200.1#43196 DST:10.200.200.16#22
>>>>>>> Dec 23 09:35:45 2019 kernel: : [263293.574838] Jool: INSTANCE:default PROTO:IPv4/TCP SRC:10.200.200.1#43196 DST:10.200.200.16#22
>>>>>>>
>>>>>>> # Stateful instances
>>>>>>> +--------------------+-----------------+-----------+
>>>>>>> |          Namespace |            Name | Framework |
>>>>>>> +--------------------+-----------------+-----------+
>>>>>>> |   ffffffff80e868c0 |         default | netfilter |
>>>>>>> +--------------------+-----------------+-----------+
>>>>>>>
>>>>>>> # Stateles instances
>>>>>>> +--------------------+-----------------+-----------+
>>>>>>> |          Namespace |            Name | Framework |
>>>>>>> +--------------------+-----------------+-----------+
>>>>>>> |   ffffffff80e868c0 |         default | netfilter |
>>>>>>> +--------------------+-----------------+-----------+
>>>>>>>
>>>>>>> JOOL:siit NAMESPACE:748484488 INSTANCE:default PROTO:IPv6/ICMP SRC:fe80::fc26:33ff:fe79:5b74 DST:2001:db8:a::a64:6402 TYPE:135 CODE:0
>>>>>>> JOOL:nat64 NAMESPACE:748484488 INSTANCE:default PROTO:IPv6/ICMP SRC:fe80::fc26:33ff:fe79:5b74 DST:2001:db8:a::a64:6402 TYPE:135 CODE:0
>>>>>>>
>>>>>>> More information if is possible.
>>>>>>>
>>>>>>> JOOL:siit NAMESPACE:748484488 INSTANCE:default PROTO:IPv6/ICMP SRC:fe80::fc26:33ff:fe79:5b74 DST:2001:db8:a::a64:6402 TYPE:135 CODE:0 action=nat46 nataddr=2001:db8::a mtu=1400 tos=3 eamt=no blacklist=no bib=no  .... other matched options
>>>>>>>
>>>>>>> Maybe filter option can be add.
>>>>>>>
>>>>>>> jool global update trace-filter [FILTER OPTIONS]
>>>>>>> --src IPv4,IPv6
>>>>>>> --dst IPv4,IPv6
>>>>>>> --sport
>>>>>>> --dport
>>>>>>> --tcp
>>>>>>> --udp
>>>>>>> --icmp
>>>>>>> --alg ftp|sip #future
>>>>>>>
>>>>>>> thank you for your effort.
>>>>>>>
>>>>>>> Fatih USTA
>>>>>>>
>>>>>>> On 21.12.2019 02:31, Alberto Leiva wrote:
>>>>>>>
>>>>>>> First draft:
>>>>>>> https://nicmx.github.io/Jool/en/usr-flags-global.html#trace
>>>>>>>
>>>>>>> the flag can be found in the latest commit in the master branch:
>>>>>>> https://github.com/NICMx/Jool
>>>>>>>
>>>>>>> On Fri, Dec 20, 2019 at 1:01 PM Alberto Leiva <ydahhrk at gmail.com> wrote:
>>>>>>>
>>>>>>> Please note that you might need to update that page in case your
>>>>>>> browser cached it, because I just updated it.
>>>>>>>
>>>>>>> On Fri, Dec 20, 2019 at 1:00 PM Alberto Leiva <ydahhrk at gmail.com> wrote:
>>>>>>>
>>>>>>> Currently, there is no tracing configuration flag. If you want, I can add it.
>>>>>>>
>>>>>>> For now, the closest thing is enabling debugging:
>>>>>>> https://nicmx.github.io/Jool/en/logging.html
>>>>>>>
>>>>>>> On Fri, Dec 20, 2019 at 12:12 AM Fatih USTA <fatihusta86 at gmail.com> wrote:
>>>>>>>
>>>>>>> I rebooted my system and it worked. But I don't understand why?
>>>>>>> One more question. How can I trace traffic inside jool like "iptables
>>>>>>> TRACE" for debugging.
>>>>>>>
>>>>>>> BTW:
>>>>>>> jool netfilter/iptables worked without reboot.
>>>>>>>
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>> Fatih USTA
>>>>>>>
>>>>>>> On 19.12.2019 19:11, Alberto Leiva wrote:
>>>>>>>
>>>>>>> Did you try printing stats?
>>>>>>> https://jool.mx/en/usr-flags-stats.html
>>>>>>>
>>>>>>> If Jool is the one dropping the packets, they should tell you why.
>>>>>>>
>>>>>>> On Thu, Dec 19, 2019 at 9:46 AM Alberto Leiva <ydahhrk at gmail.com> wrote:
>>>>>>>
>>>>>>> I hate to be asking this question but, did you try rebooting and doing
>>>>>>> a clean run?
>>>>>>>
>>>>>>> Because it works fine for me, even in my 32/64-bit hybrid...
>>>>>>>
>>>>>>> On Thu, Dec 19, 2019 at 4:54 AM Fatih USTA <fatihusta86 at gmail.com> wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I'm following this(https://www.jool.mx/en/run-vanilla.html) guide.
>>>>>>> IPTables mode working, but netfilter mode doesn't work. What am I
>>>>>>> missing? or is this a bug?
>>>>>>>
>>>>>>>
>>>>>>> jool_siit -V
>>>>>>> 4.0.6.2 i386
>>>>>>>
>>>>>>> ip{6}tables -V
>>>>>>> v1.6.0 i386
>>>>>>>
>>>>>>> uname -rm
>>>>>>> 3.16.76-4.custom x86_64
>>>>>>>
>>>>>>>
>>>>>>> PC1[eth0] <=>[eth1]Tranlator[eth2]<=>[eth0]PC2
>>>>>>>
>>>>>>>
>>>>>>> #PC1
>>>>>>> ip addr add 10.200.200.220/23 dev eth0
>>>>>>> ip route add 10.100.100.0/24 via 10.200.200.16
>>>>>>>
>>>>>>> #Translator
>>>>>>> ip addr add 10.200.200.16/23 dev eth1
>>>>>>> ip addr add 2001:db8:a::10.100.100.2/120 dev eth2
>>>>>>>
>>>>>>> sysctl -w net.ipv4.conf.all.forwarding=1
>>>>>>> sysctl -w net.ipv6.conf.all.forwarding=1
>>>>>>>
>>>>>>>
>>>>>>> ethtool --offload eth1 gro off
>>>>>>> ethtool --offload eth2 gro off
>>>>>>>
>>>>>>> lro already fixed off by kernel.
>>>>>>>
>>>>>>>
>>>>>>> jool_siit instance add default --netfilter --pool6 2001:db8:a::/96
>>>>>>>
>>>>>>>
>>>>>>> #PC2
>>>>>>> ip add add 2001:db8:a::10.100.100.11/120 dev eth0
>>>>>>> ip route add 2001:db8:a::10.200.200.0/119 via 2001:db8:a::10.100.100.2
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> #Result of netfilter (on Translator)
>>>>>>>
>>>>>>> PC1>PC2
>>>>>>> 12:44:12.234494 IP 10.200.200.220 > 10.100.100.11: ICMP echo request, id
>>>>>>> 9806, seq 1, length 64
>>>>>>> 12:44:12.234647 IP 10.200.200.16 > 10.200.200.220: ICMP net
>>>>>>> 10.100.100.11 unreachable, length 92
>>>>>>> 12:44:13.255748 IP 10.200.200.220 > 10.100.100.11: ICMP echo request, id
>>>>>>> 9806, seq 2, length 64
>>>>>>> 12:44:13.255825 IP 10.200.200.16 > 10.200.200.220: ICMP net
>>>>>>> 10.100.100.11 unreachable, length 92
>>>>>>> 12:44:14.279628 IP 10.200.200.220 > 10.100.100.11: ICMP echo request, id
>>>>>>> 9806, seq 3, length 64
>>>>>>> 12:44:14.279704 IP 10.200.200.16 > 10.200.200.220: ICMP net
>>>>>>> 10.100.100.11 unreachable, length 92
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -- Fatih USTA
>>>>>>> _______________________________________________
>>>>>>> Jool-list mailing list
>>>>>>> Jool-list at nic.mx
>>>>>>> https://mail-lists.nic.mx/listas/listinfo/jool-list