Hello, I am trying to do a HTTP post of a big chunk of 30kb file
I am using a ESP32 with BC66 under Arduino framework
my post does the following,
I am getting as an Input a 20-30kb String, size does change, max is 32kb
First I calculate the total string length + header size, I compose the header, and I do a sequence of AT+QISEND with a max length of 1000b. It does works, sometimes…
I open the TCP socket against my server, and I start sending the chunks , first header, then the rest of the content , splitted in 1000b.
Between each AT+QISEND I wait for the SEND OK response + 500ms of delay
It works sometimes, but sometimes it fails, I get a SEND FAIL error. I was hoping to be able to catch that error and re send the chunk that failed, but I was getting a lot of “bad request” from my server side. So I made up a simple TCP tunnel to see the raw content of what was the server receiving, and I saw this
There we can see the Header, and I was trying to send 15kb of letters, first chunk(0) its 1000 letters “a”, chunk 1 its 1000 letters b and so on…
There we can see that chunk number 5 ( letter F ) failed, but on the server side I can see that part of that chunk was hitting the server.
So If i try to re send that chunk again I will mess up my message adding extra content.
my questions are…
Is this method ok ?
Is there a way to check the internal BC66 buffer to be sure that I am able to do another AT+QISEND without getting an error ?
What other method do you guys suggest to upload every 1 hour 30kb of data in the fastest way possible ?
I am using HTTPPost because my server side app is already setup for getting this format by other 2g modules…
Not sure if going via MQTT could be a better alternative, but I need to re do the server side, and split my payload in smaller parts and add more time to get the ACK from the server.
The idea is to upload this 30kb of data as fast as possible
hi，Andres_Alvarez:
Have you read our TCP/UDP and MQTT reference manuals carefully? there is a limit on the size of packets sent by modules at a time.
**TCP:**In non-data mode, the max length is 1024 bytes in text mode and 512 bytes in hex
mode.In variable-length data mode, the max length is 1460 bytes in text mode and 730 bytes
in hex mode.In fixed-length data mode, the max length is 1024 bytes in text mode and 512 bytes in
hex mode.
**MQTT:**The message that needs to be published. The maximum length is 700 bytes. If in data mode (after > is responded), the maximum length is 1024 bytes.
Het, thanks for the reply
Yes, I did read the manuals, that’s why I am splitting my message in 1000b chunks.
But still, the issue is that I need to be sure that those 1000b arrive ok before I sent another 1000b
I am sending just only If I goy a “SEND OK” from the module, but I if got an error message the payload its already broken and I don´t see any workaround.
What do you suggest ?
hi，Andres_Alvarez:
After you send the data via QISEND, the data cannot be cached directly in the module. The module sends the data in the protocol stack immediately after receiving it. I recommend that you merge the data on the receiving server.
I am merging the data in the receiving server, I am able to add together the 30kb of data, but only in 20% of the attempts arrive without being shortened, in other 80% of the attempts one of the 30 messages fails and the data get corrupted.
From the server side I can not know if the message was full or not. Unless I add a header per chunk with the frame counter and the size , but this will be a complete different approach to a simple http post.
Also waiting for a ACK from the server will increase the total time of the upload, because I need to wait for a ACK for each chunk. There is no other way / method for big uploads like this ?
Also the messages that fail are the “last” ones, It never fails on the first 2rd or 3rd message
As a fact, 98% of the times I was able to post between 4kb and 6kb , here is a Plot of a 15 chunks (1000b each ) post attempt.
If the 15 parts are ok, the graph hits 15. If it fails on the 5th chunk, the graph stops.
So I guess there is a internal buffer on the BC66 to hold this data… because I am able to push 3 messages one after another in less than 10ms…
There is no command to check that outgoing buffer ?
Also is there any place with already made library or functions to use this module with others ICs ?
hi，Andres_Alvarez:
Your test and explanation is reasonable to some extent. The minimum frame in the physical layer frame structure of NB-iot or LTE is 10ms. If all data can be stacked into frames within 10ms, it can be completed at one time, but it needs to ensure that the processing time of TCP protocol stack is achievable and allowed.
I don´t want to send a frame, wait for ACK, and send another, because If I have 500ms delay, I will be adding 1 second for each frame, 10 seconds for a 10kb its a lot for my application.
My next approach will be to send 1 extra frame at the beginning with a table of how many frames I will be sending, and the size of each frame. After that I will send all the payload frames with a few extra bytes as header, frame number, and a tail. And I will send all at once spacing them as less as possible.
if I got an error, I way a little bit more and re send the same frame.
After the server receives one frame, It will check the expected payload, header and tail, and it will send a ACK so I can take notice of witch frame is OK. And I will look trough my table re sending the frames without ACK.
Once I have the correct payload in buffer I will output that “http post” frame to my main application , all this will be a middleman. But I need to get the ACK of my main application sent to my midleapp, so I can tell the BC66&ESP32 that the post of that message its complete, so I can take it out of my flash memory.
Its a little extra work, but I don’t see any alternatives to minimize time and integrity of big payloads.
@Andres_Alvarez Did you ever find a solution for sending large payloads? I have 20-30KB payload I need to send to a REST API, and I honestly don’t know where to start.
Hey, no, I moved all my app to a BG600L module
On this new module I can send data to the module, and ask the module how much data has been sent out to the network
