RM500U-CNV TCP receive data loss

,

Hello.
We have RM500U-CNV module connected throught serial port to esp32. We need to make mqtt connection using SSL connection.

  1. The buffered TCP connection to the server is ok and the server is receiving the messages.
  2. When server send the message (SSL hello message) larger than 2048 bytes, it is truncated. For example if it sends 2401 bytes, only 2048 arrives.

Is there some limitation for incomming data size? how can we make the SSL connection with this modem?

The commands and server responses:


AT+QIOPEN=1,1,"TCP","our.server..com",8883,0,0
OK
+QIOPEN: 1,0

AT+QISEND=1,138
> .......
SEND OK

+QIURC: "recv",1

AT+QIRD=1,1500
+QIRD: 1500
AT+QIRD=1,99
+QIRD: 99
AT+QIRD=1,1029
+QIRD: 449
AT+QIRD=1,585
+QIRD: 0

As you can see, in SUM 2048 bytes arrived.
Thank you.
Jaroslav

Dear @chefotm
Please share your current firmware.

What did you test? TCP or MQTT?

Hello Silvia. Our modem:

//Manufacturer
 AT+CGMI
   Quectel

//Firmware
AT+CGMR
  RM500UCNVAAR03A02M2G_01.001.01.001

//Model
AT+CGMM
  RM500U-CNV

We tested the TCP connection in transparent and buffered mode (AT+QIOPEN). Both working fine until the incoming packet payload is less than 2048 bytes long . If the payload exceeds 2048 bytes, the modem start to ignore the rest of incoming payload from server (for example when we are trying to start the SSL connection and the server send the certificate). .

According to the AT command manual for the RM500U-CNV, there is no MQTT client in the modem, or are we wrong?

Thank you very much.

Dear @chefotm
RM500UCNVAAR03A02M2G_01.001.01.001 is too old.
I have sent the latest firmware to you, please try again.

RM500U doesn’t MQTT feature.

Thank you Silvia.
We upgraded the modem to the firmware you sent. Now it is:

AT+CGMR
 RM500UCNVAAR03A09M2G_01.001.01.001

Unfortunately the problem is still here. We send the TLS hello message :
AT+QISEND=1,138

The server send the reply 2401 bytes, but only 2048 arrived to the serial port buffer.

+QIRD: 1500
+QIRD: 99
+QIRD: 449
+QIRD: 0
+QIRD: 0
.....

So we think the problem looks like to be somewhere in the modem packet buffering.
Thank you.

Dear @chefotm
Please help to check the receive length via AT command below:

Hello silvia.
We tested it with your suggestion and here is the result. As you can see, only 2048 bytes is received instead of 2401 that server sent.

AT+QIOPEN=1,1,"TCP","myserver.com",8883,0,0                               
OK                                                                              
                                                                                
+QIOPEN: 1,0                                                                    
AT+QISENDEX=1,"---TLS HELLO DATA---"
SEND OK                                                                         
                                                                                
+QIURC: "recv",1                                                                
                                                                 
AT+QIRD=1,0                                                                     
+QIRD: 2048,0,2048                                                              
                                                                                
OK                                                                              

AT+QIRD=1,1500
+QIRD: 1500  

---- DATA ----

AT+QIRD=1,0
+QIRD: 2048,1500,548
OK
AT+QIRD=1,1500
+QIRD: 548

----- DATA -----

AT+QIRD=1,0
+QIRD: 2048,2048,0

OK

AT+QIRD=1,1500
+QIRD: 0

OK
AT+QIRD=1,0
+QIRD: 2048,2048,0

OK


+QIURC: "closed",1

Dear @chefotm
From the reponse from AT command, server only sent 2048 to module, please confirm if server has some limitation when sent data.

Hello Silvia. Our server works without problem with our other 381 devices (mqtt clients) based on Esp32, SIMCOM, WH-LTE-7S1-E or other modules and modems (hardware or software implementation of MQTT SSL client). I have checked the communication between RM500 and the server via TShark (wireshark console packet capture) and the payload sent by server is 2401 bytes. But the RM500 modem showing only 2048 bytes in input buffer. The problem must be somewhere in the modem input buffer handling (maybe max transfer unit or something like that?).
Thank you.

Dear @chefotm
Did the server is MQTT server? Not TCP server?

Helou Silvia.
The architecture looks like this: Apache TLS handshake server (TCP server) ↔ MQTT broker. Apache is simple TCP server, which provides TLS handshake and the payload encryption. After TLS handshake the Apache make tunel to the MQTT broker.

The problem starts with the first message between RM500 and the Apache TCP sever. Its simple TCP send/receive where RM500 sends small message of size 170 Bytes to the Apache and Apache send reply to the RM500 of size 2401 Bytes. The RM500 receive only 2048 Bytes.

Dear @chefotm
Do you have other device without RM500U?
If yes, please try to test with the same flow and check if there are any issue.

Or you can share the wireshark log to us for analyze.

Which did you use?Linux or Windows?

Hello Silvia.
I made another easy setup to demonstrate the modem bug.
(1.) I made a TCP server using linux netcat command (running on the linux OS on port 5555). The server will automatically send 2503 Bytes of payload to the client after connection.
The server code is here:

while true; do echo -e "AAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFF" | nc -l -p 5555; echo "Client Disconnected" ;done

(2.) I made a simple client with the same netcat command on another linux OS based machine and connected it to the server defined in (1.) on port 5555 . The server successfully sent and client successfully received 2503 Bytes of payload.
The client code is here:

nc myserver.com 5555

(3.) I tried to connect to the server with the RM500U modem. Connection was successful and the server sent 2503 Bytes. But the client in RM500U received just 2048 bytes. So the results are the same as previously.
The communication on serial line with RM500U:

AT+QIOPEN=1,1,"TCP","myserver.com",5555,0,0

OK

+QIOPEN: 1,0

+QIURC: "recv",1
AT+QIRD=1,0

+QIRD: 2048,0,2048

OK
AT+QIRD=1,1500

+QIRD: 1500
-e AAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFF
OK
AT+QIRD=1,0

+QIRD: 3548,1500,2048

OK
AT+QIRD=1,1500

+QIRD: 1500
FFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCC-e AAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCCCCCCDDDDDDDDDDDDDDDDEEEEEEEEEEEEEEEEEEEEEEEEEFFFFFFAAAAAAAAAAAAAAAAAABBBBBBBBBBBBBBBBBCCCCCCCCCCCCCC
OK
AT+QIRD=1,0

+QIRD: 3548,3000,548

OK
....
....
....

+QIURC: "closed",1

As you can see, the message from the server begins with the string “-e AAA”. The RM500U trying to reconnect to the server, so the string is received continuously. First reading from serial port obtained 1500 Bytes. In the second reading, there is the starting string “-e AAA” on the 548 byte position. So the total incoming payload from the server per one connection is 1500+548 = 2048 Bytes. The rest of the payload is dropped by the RM500U.

Hope this will help you reproduce the problem and solve it.
Thank you.

Dear @chefotm

Thanks for your information.
I will check internally and give feedback later.

Thank you Silvia. We will wait for your suggestions.
Jaroslav

Dear @chefotm
Thanks for your feedback.
Current MAX_BUF_SIZE setting is 2048, our engineer will check whether it is possible to modify it.
It will take more time for this.

Thank you SIlvia. We will be happy, if the buffer will be >=2600 Bytes to be able to make the SSL connection with it. In my opinion, there is a way to make it with the current size of the Buffer. You just need to ensure the modem will not drop incoming packets when the buffer is full. The problem is probably somewhere in packet handling algorithm.
Anywhere, we will be happy for any solution.
Thank you.