UDP issues with RM502Q-AE

IoT Modules 5G Module
1

I’m an having issue with UDP traffic not passing using an RM502Q-AE with firmware version RM502QAEAAR11A04M4G_01.002.01.002. The problem occurs with the modem connected to both an OpenWRT router using QMI and when connected to a Windows 10 computer with the Quectal USB driver package version 2.2.4

Specifically, DNS using UDP port 53 does not work and neither does Wireguard using UDP port 51820. I performed a packet capture on the remote Wireguard server port 51820 and the UDP traffic from the computer or router never reach the server. The carrier is T-Mobile US and I’ve tried two different SIMs with lines of service on two different T-Mobile accounts. Wireguard works fine when the same SIMs are used in a Oneplus phone both natively on the phone or when using the phone as a hotspot with the computer connected through it.

The modem is installed in a NGFF to USB 3 adapter with is connected to either the OpenWRT router or Windows 10 computer. DNS and Wireguard both work on both the router and computer when using different Internet connections. Are there any known issues with UDP and the RM502Q or is there any additional troubleshooting steps i should perform?

2

Dear Sir,
Do you mean that you connect the RM502Q-AE to a Windows 10 computer and OpenWRT router through USB 3.0 interface cable, then make the MBIM mode for the windows 10 computer and QMI mode for OpenWRT router to access the internet successfully, but both fail to make a UDP traffic, right? and the same sim card can work in a cell phone with UDP traffic.
So how do you test with UDP traffic, could you describe it?
By the way, please help to answer below questions, thanks in advance!

  • Company name and located city:

  • Project name (so we can refer to it in the future):

  • Application type:

  • Estimated Annual Units (for series production):

  • Project timeline:

  • Current status:

  • From which distributor do you buy Quectel modules & EVB kits?

  • Do you have EVB kit for this application?

4

I performed one additional test today that is very telling. Installed a Sierra MC7411 into one the OpenWRT routers used in the previous tests and with this modem both ping and UDP traffic are both successful. No other changes to were made to the OpenWRT router, just a change in modem from the RM502Q-AE to the MC7411.

Are there other firmware versions for the RM502Q-AE that I could try? I do not see posts from many people using RM502QAEAAR11A04M4G_01.002.01.002, everyone seems to have an A02 variant.

5

Ok, I will send you a version of A02 via email, you can try.

6

Thank you for sending the A02 firmware. Can you send the A04 firmware version RM502QAEAAR11A04M4G_01.002.01.002 so I can restore to this version if needed.

7

I too have observed this incredibly annoying behavior on T-Mobile with the exception that ICMP to 8.8.8.8 from Windows 10 succeeds for me (using the RM502Q-AE in OpenWRT under ModemManager). However, if I set DNS on the Windows’ NIC to use 8.8.8.8 DNS no longer works.

Digging deeper I’ve used PortQryV2 to determine that Windows can get as far as reaching the 53/udp but actual DNS queries time out. The extremely interesting thing is that under OpenWRT I can set my WAN interface (which is the modem bearer under ModemManager) to use 8.8.8.8 for DNS and this works (as confirmed by DNS LeakTest performed on the Windows host which shows all Google servers for DNS).

I found this forum post which seemed interesting: Sprint/T-Mobile Hotspot Blocking OpenVPN? . Also, found some posts in the T-Mobile community since December which complain of UDP traffic being dropped when passing through TMO’s 464xlat. I tried adjusting MTU downward by different increments on the Windows laptop and OpenWRT WAN interface and wwan0 device but this did not help to pass UDP traffic from Windows.

What is interesting about all your tests that succeeded on TMO is that they were modems connected only on LTE. My observations indicate that this dropped UDP behavior only seems to occur when connected to T-Mobile’s 5G network (NSA in my case). So, at least at present it seems there is some issue passing UDP traffic from downstream clients like Windows over TMO 5G.

What is not clear to me is if this is something broken (maybe intentionally?) on TMO’s network or some configuration related issue on the client side. Have you found anything else in your testing since your last post?

Thank you!

8

Ok， I will send the latest FW version to you via email. I have little experience with UDP testing.

1 Like
9

Thank you! However I am already on the latest version of firmware when experiencing this issue (RM502QAEAAR11A04M4G_01.003.01.003). Any additional suggestions or input would be greatly appreciated.

Thanks again.

10

Today, using firmware RM502QAEAAR11A04M4G_01.003.01.003, I am still experiencing problems with UDP but ICMP to 8.8.8.8 is now successful. The UDP tests that fail, include DNS to 8.8.8.8 and two VPNs that use the UDP protocol. The protocols that work and fail from Windows 10 are not consistent over time. Looking back to 1/28/2022, I resumed testing after the module was unused for 3 weeks. Both ICMP and UDP were working for several days. By 2/6/2022 with the module continuously connected, UDP was failing again but ICMP continued to work. I’m currently trying to replicate the sequence of events so I have the T-Mobile SIM in another device and plan to leave it there for a couple of weeks before testing again with the RM502Q-AE.

I’ve also observed that when I disable 5G by setting the mode_pref to LTE, both ICMP and UDP consistently work fine from Windows 10. I believe the problems with Windows 10 are just with IPv4, the limited tests with IPv6 are successful.

11

To: @rbunch , @Peter.Zhu-Q, @Isaac.Wang-Q

Dear All,
I have completed additional testing this week and narrowed down the issue to the “Commercial-TMO” MBN profile. Something in this profile is broken (intentionally by the carrier or otherwise maybe due to bug) which is causing UDP traffic to fail shortly after cell attach when the end client is Windows 10 based. When switching to the generic MBN profile, UDP passes unmolested over IPv4 (and I assume IPv6 but I only ever had issues with IPv4 UDP, thus my only test case at present).

What is extremely curious is that if I switch back to the “Commerical-TMO” profile, remove power from the modem, power it back up again and immediately test IPv4 UDP traffic it passes successfully for some minutes before failing. I do not want to assume malicious intent by the carrier but this certainly seems like some fingerprint or heuristic profile is being set when attempting to pass UDP traffic on the carrier end. Else, maybe there is just some legitimate bug in the carrier MBN profile which only appears after passing some quantity of UDP traffic.

Regardless, the workaround for RM502Q-AE at this time appears to be selecting the generic MBN (which I will detail below). Unfortunately it has one troublesome side effect of getting assigned an exit node from the carrier APN which is not appropriately geolocated. While the “Commercial-TMO” MBN always routed me out of an address geolocated within 50 miles of my physical location (with some minor variance on exact location), the generic MBN only seems to route me out of Denver, Colorado or Chicago, Illinois (USA). This is undesirable as it raises my latency (ping) quite considerably vs. routing through a properly geolocated IP.

The commands to switch to the generic MBN profile are below:

AT+CFUN=0
AT+QMBNCFG="AutoSel",0
AT+QMBNCFG="Deactivate"
AT+QMBNCFG="select","ROW_Generic_3GPP_PTCRB_GCF"
AT+CFUN=1,1

Given that the “Commercial-TMO” profile exists in the first place, I would assume Quectel has some sort of partnership with T-Mobile USA. If Quectel itself cannot find and resolve the UDP issue as related to the carrier MBN, I would humbly (but strongly) suggest that Quectel engage their appropriate channel partner contact within T-Mobile to sort out the issue which is dropping UDP traffic.

Due to +QMBNCFG default behavior of “AutoSel” anyone developing a product for use on T-Mobile will likely encounter this hard to diagnose issue and may reconsider use of the RM502Q-AE unless they happen to stumble upon this thread and accept the degraded latency offered by switching MBN profile to generic. I have accepted this limitation for my project but others with latency dependent requirements may not.

Thank you for your continued support.

12

Hi hazarjast,
I will send a firmware version to you via email, you can use it for testing.

13

Hi Issac, Can you also send me the firmware for testing.

14

It appears you’ve sent the same A04 firmware you already sent to me back on 2/12 (‘RM502QAEAAR11A04M4G_01.003.01.003’). See SHA256 checksum screenshots below. As I indicated back on 2/12, I am already on this firmware. Did you mean to send a newer firmware but accidentally provided the wrong link by mistake?

Thank you for your support.

2022-02-21_14h08_56

15

RM502QAEAAR11A04M4G_01.003.01.003 is the latest version, please execute two AT commands below, restart the module and try again.
AT+QCFG=“CLAT”,1,1,"",0,"",0,0,0,0,0,0
AT$QCPRFMOD=PID:1,OVRRIDEHOPDP:"IPV6”

16

I executed these commands and still have the same results where Windows UDP packets are not being sent.

AT+QCFG=“CLAT”,1,1,“”,0,“”,0,0,0,0,0,0
AT$QCPRFMOD=PID:1,OVRRIDEHOPDP:“IPV6”
AT+CFUN=1,1

Android DNS packets using UDP are sent and receive a response from the DNS server. I captured an Android DNS request pack then retransmitted it using Windows and I received a reply. There are only a few differences between the Windows and Android packets and it turns out the important difference is the Android packet has the Do not Fragment bit in the IP header set, but Windows does not. To confirm this flag is the differentiator, I set this bit in the Windows packet, retransmitted and now receive a reply.

I hope the identification of the Do not Fragment bit in the IP header being related to UDP traffic not passing is helpful to you.

17

Excellent detective work. How many bytes in size was the DNS test packet? With ‘do not fragment’ bit being a factor here it seems the possibility of packet fragmentation and MTU come back into play if the packet in question was large enough to require fragmentation. There is the ‘IPv4_MTU_discoverty’ property of AT+CGDCONT which I wonder if has some bearing here (I assume here ‘discoverty’ is a misspelling of ‘discovery’ in the AT command guide). If packet fragmentation due to MTU is a problem in our testing scenarios I wonder if ‘IPv4_MTU_discoverty=1’ (discover MTU through NAS signaling) would be a beneficial setting in the PDP profile:

2022-02-24_13h55_29

Thanks for your continued troubleshooting effort on this one.

18

After some additional checking/testing I’ve found that “AT$QCPRFMOD=PID:1,OVRRIDEHOPDP:“IPV6”” appears to be the problem child on T-Mobile. This is active by default in the “Commercial-TMO” MBN. With this parameter and CLAT parameter enabled, ICMP and UDP experience random packet loss and increased latency. With this disabled (AT$QCPRFMOD=PID:1,OVRRIDEHOPDP:"") and CLAT disabled (AT+QCFG=“CLAT”,0), there is no packet loss or high latency observed with ICMP and UDP traffic. This latest testing was direct from OpenWRT (not Windows or other OS) and the packet loss/latency was seen with both IPV4 and IPV6 traffic.

Out of curiosity I switched to the “ROW_Generic_…” profile and tried enabling both of the above parameters but the modem would not attach at all to the network with these in place. Again, the downside with using either Generic MBN or disabling the CLAT/OVRRIDEHOPDP parameters on the TMO MBN is that the modem seems to lose the ability to identify it’s geographical location to the T-Mobile network and thus routes out of an IP which may not be geo-local to where the modem is physically present.

@Peter.Zhu-Q, @Isaac.Wang-Q
I want to be clear that this issue on T-Mobile is affecting many more than just myself and @rbunch. I have it on good authority that a sizeable Quectel customer, MoFi Network, has many T-Mobile customers which are experiencing troubles surrounding this and other issues with the RM502Q-AE (see: MOFI5500-5GXeLTE-RM502 (IN STOCK) – Mofi Network Inc). Their general consensus at the moment is that the RM502Q-AE is not a stable device for T-Mobile (it is not clear whether they have tested with the “ROW_Generic…” MBN or not).

As a certified partner of T-Mobile (Quectel receives T-Mobile approval of 5G NR module | Quectel) I hope that Quectel can internally prioritize root identification and resolution of this issue. Thank you.

19

Ok，thanks for you feedback, I will be looking for some internal resouorces to help you.

1 Like
20

Hi Hazar,

IPV6 only and XLAT function T-Mobile’s requirement for Home network. But we haven’t such packet loss and latency issue from other customer so far. It could be related to IPV6/XLAT stack in firmware or T-Mobile IPV6 PDN context issue. More testing would be requried to eliminate packet loss and latency issue.
Here is screenshot of the original description:

image
image1218×237 46 KB

Best Regards,

21

Great discovery of the parameter in the MBN that causes this problem. I can confirm just using AT$QCPRFMOD=PID:1,OVRRIDEHOPDP:“IP” allows UDP to work better but it is not a perfect solution. A Wiregard VPN which uses UDP now works but there are errors in the log indicating problems with the connection. Also, I can confirm the IP addres assigned by T-Mobile is in a different subnet with a geolocation a further distance from my actual location than with the OVRRIDEHOPDP:“IPV6” setting in MBN file.

I agree with @hazarjast that this issue impacts many people using this module with T-Mobile and we request a better fix to this UDP issue.