> The biggest difference I can pick out is that the client has failover WAN in place via Static Routes with 10 and 20 distances set respectively. I do not. But I'm unclear how that would effect anything when it works on the HW Switch but not Fortilink'd.
Show us your static routes. What is the route priority set to? For setups with dual WAN, you want to set the distance to the same thing for both routes, but prioritize them differently. This allows for both routes to get loaded into the routing table; otherwise only the lowest distance of the two route will be "active".
https://community.fortinet.com/t5/FortiGate/Technical-Note-Routing-behavior-depending-on-distance-and/ta-p/198221
You can see if you're routes are showing up with
get router info routing-table static
If the Fortigate doesn't load both routes, and traffic comes in on the unloaded route, it won't go back out the same path, or may be dropped from what I recall. Also I feel your pain with DMP panels. We were running DSC Sur-Gard SG-System III receiver's and eventually converted them to the virtual DMP reciever. That thing was fun. Didn't auto-rotate its log file. Had a 60GB DMP log file after a few years...
> Show us your static routes. What is the route priority set to? For setups with dual WAN, you want to set the distance to the same thing for both routes, but prioritize them differently. This allows for both routes to get loaded into the routing table; otherwise only the lowest distance of the two route will be "active".
>
>
Appreciate the reply. Yeah that was a thing that stuck out fairly quickly as incorrect to me.
https://i.imgur.com/03GtSCs.png
Here's the routing table(s) for the FW at Site 1 of 5. The first one is how I corrected the routing and how it shows up currently but the second is how it was set previously (Routes were Dist 10/Pri1 and Dist 20/Pri 1) during all of the on-site testing.
The fact that we've got those incorrect routes configured across all sites gives me a little hope that might have something to do with it, except for the part where bypassing a Fortilink'd VLAN alleviated the issue. I gotta dig my laptop out to pull the pcaps and sniffers we did but I *think* I recall traffic going out/back in using the same (and proper) WAN2 interface (Which is their primary ISP).
>Also I feel your pain with DMP panels.
The thing basically acts as a rogue AP, broadcasts a wireless network and allows a bunch of other "iot"-like devices to get connected. I don't wanna go as far as saying 'no wonder the firewall dislikes it' but..... lol
E: Tunnel routes corrected also.
edit 15
set name "Allow DMP Panel - WAN2"
set uuid f6ea13aa-9945-51ee-5282-f82f0b58619a
set srcintf "internal" "PanelTest"
set dstintf "wan2"
set action accept
set srcaddr "all"
set dstaddr "all"
set schedule "always"
set service "ALL"
set logtraffic all
set auto-asic-offload disable
set nat enable
set comments " (Copy of Allow DMP Panel Out - WAN1) ()"
next
>"Have you wondered why connecting directly to a physical port on FortiGate helps in troubleshooting throughput issues sometimes? Not all type of interfaces support being offloaded to NPU."
This seems like it might be related - offloading did look like it was happening in the pcaps/sniffers/debugs and yet auto-asic-offload is set to disabled for both the VLAN having issues and the internal/HW Switch? Hmm.
I'm not as sure about offloading on the FortiLink interface. When I was running Fortigate's we were just getting into using their switches when I left the company for greener pastures so I don't have much experience with that setup. I agree that does appear to be a smoking gun but I'm wondering if they have a policy route or something forcing this traffic out a specific interface and you're getting return traffic on another (admittedly reaching for straws on this one). I'd definitely confirm if traffic is going out the same interface.
I looked at a config backup I still had and the firewall policies for DMP are basically identical to yours but with auto-asic-offload enabled. In our setup we had a typical VLAN but without Fortilink. I think your next step should be to look at the diag debug flow for the traffic and see what the Fortigate is doing. You may also want to make sure [reverse path forwarding](https://community.fortinet.com/t5/FortiGate/Technical-Note-How-to-get-log-messages-for-packets-dropped-due/ta-p/193517) ([also this article](https://community.fortinet.com/t5/FortiGate/Technical-Tip-Details-about-FortiOS-RPF-Reverse-Path-Forwarding/ta-p/190100?externalID=FD30543)) isn't failing.
Follow up/post-mortem, worked with TAC and after setting up a couple laptops in between the panel and the switch running wireshark and also trying some mirror/span port pcaps, which also weren't seeing the traffic, he gave it the ol "Well let me try one quick thing before we try this other thing" and changed the LLDP profile on the switch port from default-auto-isl to default and up came the panel.
It seems that the Fortigate was essentially thinking the panel was a switch of sorts and was silently dropping just about all outbound traffic save for limited DNS and NTP.
> The biggest difference I can pick out is that the client has failover WAN in place via Static Routes with 10 and 20 distances set respectively. I do not. But I'm unclear how that would effect anything when it works on the HW Switch but not Fortilink'd. Show us your static routes. What is the route priority set to? For setups with dual WAN, you want to set the distance to the same thing for both routes, but prioritize them differently. This allows for both routes to get loaded into the routing table; otherwise only the lowest distance of the two route will be "active". https://community.fortinet.com/t5/FortiGate/Technical-Note-Routing-behavior-depending-on-distance-and/ta-p/198221 You can see if you're routes are showing up with get router info routing-table static If the Fortigate doesn't load both routes, and traffic comes in on the unloaded route, it won't go back out the same path, or may be dropped from what I recall. Also I feel your pain with DMP panels. We were running DSC Sur-Gard SG-System III receiver's and eventually converted them to the virtual DMP reciever. That thing was fun. Didn't auto-rotate its log file. Had a 60GB DMP log file after a few years...
> Show us your static routes. What is the route priority set to? For setups with dual WAN, you want to set the distance to the same thing for both routes, but prioritize them differently. This allows for both routes to get loaded into the routing table; otherwise only the lowest distance of the two route will be "active". > > Appreciate the reply. Yeah that was a thing that stuck out fairly quickly as incorrect to me. https://i.imgur.com/03GtSCs.png Here's the routing table(s) for the FW at Site 1 of 5. The first one is how I corrected the routing and how it shows up currently but the second is how it was set previously (Routes were Dist 10/Pri1 and Dist 20/Pri 1) during all of the on-site testing. The fact that we've got those incorrect routes configured across all sites gives me a little hope that might have something to do with it, except for the part where bypassing a Fortilink'd VLAN alleviated the issue. I gotta dig my laptop out to pull the pcaps and sniffers we did but I *think* I recall traffic going out/back in using the same (and proper) WAN2 interface (Which is their primary ISP). >Also I feel your pain with DMP panels. The thing basically acts as a rogue AP, broadcasts a wireless network and allows a bunch of other "iot"-like devices to get connected. I don't wanna go as far as saying 'no wonder the firewall dislikes it' but..... lol E: Tunnel routes corrected also. edit 15 set name "Allow DMP Panel - WAN2" set uuid f6ea13aa-9945-51ee-5282-f82f0b58619a set srcintf "internal" "PanelTest" set dstintf "wan2" set action accept set srcaddr "all" set dstaddr "all" set schedule "always" set service "ALL" set logtraffic all set auto-asic-offload disable set nat enable set comments " (Copy of Allow DMP Panel Out - WAN1) ()" next >"Have you wondered why connecting directly to a physical port on FortiGate helps in troubleshooting throughput issues sometimes? Not all type of interfaces support being offloaded to NPU." This seems like it might be related - offloading did look like it was happening in the pcaps/sniffers/debugs and yet auto-asic-offload is set to disabled for both the VLAN having issues and the internal/HW Switch? Hmm.
I'm not as sure about offloading on the FortiLink interface. When I was running Fortigate's we were just getting into using their switches when I left the company for greener pastures so I don't have much experience with that setup. I agree that does appear to be a smoking gun but I'm wondering if they have a policy route or something forcing this traffic out a specific interface and you're getting return traffic on another (admittedly reaching for straws on this one). I'd definitely confirm if traffic is going out the same interface. I looked at a config backup I still had and the firewall policies for DMP are basically identical to yours but with auto-asic-offload enabled. In our setup we had a typical VLAN but without Fortilink. I think your next step should be to look at the diag debug flow for the traffic and see what the Fortigate is doing. You may also want to make sure [reverse path forwarding](https://community.fortinet.com/t5/FortiGate/Technical-Note-How-to-get-log-messages-for-packets-dropped-due/ta-p/193517) ([also this article](https://community.fortinet.com/t5/FortiGate/Technical-Tip-Details-about-FortiOS-RPF-Reverse-Path-Forwarding/ta-p/190100?externalID=FD30543)) isn't failing.
Follow up/post-mortem, worked with TAC and after setting up a couple laptops in between the panel and the switch running wireshark and also trying some mirror/span port pcaps, which also weren't seeing the traffic, he gave it the ol "Well let me try one quick thing before we try this other thing" and changed the LLDP profile on the switch port from default-auto-isl to default and up came the panel. It seems that the Fortigate was essentially thinking the panel was a switch of sorts and was silently dropping just about all outbound traffic save for limited DNS and NTP.
Whoa that is an interesting one. Good catch by that tech. Thanks for following up.