Frequent disconnects with RM500Q-GL

My environment -

ATI:
Quectel
RM500Q-GL
Revision: RM500QGLABR11A06M4G

$ uname -r
5.4.0-121-generic

/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/7p, 5000M
|__ Port 1: Dev 86, If 0, Class=Vendor Specific Class, Driver=option, 5000M
|__ Port 1: Dev 86, If 1, Class=Vendor Specific Class, Driver=option, 5000M
|__ Port 1: Dev 86, If 2, Class=Vendor Specific Class, Driver=option, 5000M
|__ Port 1: Dev 86, If 3, Class=Vendor Specific Class, Driver=option, 5000M
|__ Port 1: Dev 86, If 4, Class=Vendor Specific Class, Driver=qmi_wwan, 5000M

I’m using libqmi-1.30-6 and qmi-network to bring up the 5G SA connection, and udhcpc to get an IP address on my netdev. When I first connect, everything works as it should. However, the modem always eventually gets stuck in a bad state that can only be fixed by resetting the modem. Sometimes it happens every 15 minutes, and sometimes I cannot get things working even after restarting. Rarely, the issue resolves itself on its own, but this does not happen often.

When everything is working fine, the modem reports

AT+QENG=“servingcell”:
NR5G-SA:
state = “NOCONN”
access_type = NR5G-SA
duplex_mode = “TDD”
MCC = 999
MNC = 40
cellID = 82398001
PCID = 50
TAC = 99
ARFCN = 636576
band = 78
NR_DL_bandwidth = 100 MHz
RSRP = -70
RSRQ = -11
SINR = 37
scs = 1
srxlev = -

When the modem enters the bad state and I can no longer send and receive packets, it shows

AT+QENG=“servingcell”:
NR5G-SA:
state = “NOCONN”
access_type = NR5G-SA
duplex_mode = “TDD”
MCC = 999
MNC = 40
cellID = 82398001
PCID = 50
TAC = 99
ARFCN = 636576
band = 78
NR_DL_bandwidth = 20 MHz
RSRP = -69
RSRQ = -11
SINR = 37
scs = 1
srxlev = 7808

The only difference is that the bandwidth has dropped to 20MHz and the srxlev = 7808. What is causing this to happen? Here are some more AT command outputs while the modem is in a bad state -
AT+QNWCFG=“nr5g_csi”
+QNWCFG: “nr5g_csi”,9,1,14,7

AT+QNWCFG=“up/down”
+QNWCFG: “up/down”,0,0,2

AT+C5GREG?
+C5GREG: 2,1,“99”,“82398001”,11,1,“01”

AT+QRSRP
+QRSRP: -115,-85,-83,-90,NR5G

AT+CGATT?
+CGATT: 1

AT+CGPADDR=1
+CGPADDR: 1,“10.158.120.57”

And before you suggest that I use qmi_wwan_q and quectel-CM, I’ve already tried that. I still get the same disconnects, and quectel-CM does not detect that the connection has gone down. I’ve tried enabling all of the unsolicited return codes, packet domain error reporting, etc., and nothing is giving any indication as to why the modem stops sending and receiving.

How can I further debug this problem?

Hi,
Can you describe in detail what is the specific situation when the modem is in a bad state,thanks

The modem is attached to a mobile robot in an industrial factory. The factory has two 5G band 78 radios on a SA core network. There are around 20 other 5G devices.

There is no specific location, time, or situation when the modem enters the bad state. Sometimes the robot is moving when it happens, sometimes it is stationary. When it enters the bad state, all RX and TX stops. The only other symptom I can find is that the bandwidth drops from 100MHz to 20Mhz, and the srxlev is non-null.

Are there any AT commands that could be used to help debug the problem? I’ve gone over the AT command reference many times, and I cannot find anything that could help. I’ve re flashed the firmware, reset the NVRAM, I’ve tried everything.

Could a fluctuation in power cause the modem to behave like this?

Jeff

@Bryan.Tan-Q usually the srxlev is null when the modem is operating. What does it mean when it is non-null?