We have a custom hardware product that uses a Raspberry Pi CM4 module. It runs on custom OpenWRT (22).
This device runs 4 different 4G Quectel Modems (EC20 and EC25) which are all mapped on CM4’s USB bus. It all works well, however, whenever we push too much data through the bus it starts throwing Kernel panic, which leads to degraded download/upload performance.
We keep getting the following error when relatively higher data is pushed on the USB bus,
Trace in log read shows something like this
Thu Feb 9 22:11:27 2023 kern.err kernel: [ 299.982230] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 7 - ChHltd set, but reason is unknown
Thu Feb 9 22:11:27 2023 kern.err kernel: [ 299.991456] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x06600009
Thu Feb 9 22:11:27 2023 kern.warn kernel: [ 299.997629] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:11:27 2023 kern.err kernel: [ 300.005576] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 7 - ChHltd set, but reason is unknown
Thu Feb 9 22:11:27 2023 kern.err kernel: [ 300.014792] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x04600009
Thu Feb 9 22:11:27 2023 kern.warn kernel: [ 300.020964] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:11:33 2023 kern.err kernel: [ 305.524447] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 7 - ChHltd set, but reason is unknown
Thu Feb 9 22:11:33 2023 kern.err kernel: [ 305.533674] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x06600009
Thu Feb 9 22:11:33 2023 kern.warn kernel: [ 305.539847] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:12:34 2023 kern.err kernel: [ 366.682305] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 2 - ChHltd set, but reason is unknown
Thu Feb 9 22:12:34 2023 kern.err kernel: [ 366.691531] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x06600009
Thu Feb 9 22:12:34 2023 kern.warn kernel: [ 366.697703] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:12:41 2023 kern.err kernel: [ 373.231192] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 2 - ChHltd set, but reason is unknown
Thu Feb 9 22:12:41 2023 kern.err kernel: [ 373.240419] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x06600009
Thu Feb 9 22:12:41 2023 kern.warn kernel: [ 373.246591] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:13:10 2023 kern.err kernel: [ 402.140139] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 5 - ChHltd set, but reason is unknown
Thu Feb 9 22:13:10 2023 kern.err kernel: [ 402.149365] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x04600009
Thu Feb 9 22:13:10 2023 kern.warn kernel: [ 402.155538] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:13:10 2023 kern.err kernel: [ 402.817252] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 1 - ChHltd set, but reason is unknown
Thu Feb 9 22:13:10 2023 kern.err kernel: [ 402.826478] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x06600009
Thu Feb 9 22:13:10 2023 kern.warn kernel: [ 402.832651] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:13:45 2023 kern.err kernel: [ 437.925881] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 1 - ChHltd set, but reason is unknown
Thu Feb 9 22:13:45 2023 kern.err kernel: [ 437.935107] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x04600009
Thu Feb 9 22:13:46 2023 kern.err kernel: [ 438.216434] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 4 - ChHltd set, but reason is unknown
Thu Feb 9 22:13:46 2023 kern.err kernel: [ 438.225654] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x04600009
Thu Feb 9 22:13:46 2023 kern.warn kernel: [ 438.231827] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:14:20 2023 kern.err kernel: [ 472.693923] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 0 - ChHltd set, but reason is unknown
Thu Feb 9 22:14:20 2023 kern.err kernel: [ 472.703151] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x06600009
Thu Feb 9 22:14:20 2023 kern.err kernel: [ 472.819935] dwc2 fe980000.usb: dwc2_hc_chhltd_intr_dma: Channel 5 - ChHltd set, but reason is unknown
Thu Feb 9 22:14:20 2023 kern.err kernel: [ 472.829161] dwc2 fe980000.usb: hcint 0x00000002, intsts 0x04600009
Thu Feb 9 22:14:20 2023 kern.warn kernel: [ 472.835334] dwc2 fe980000.usb: dwc2_update_urb_state_abn(): trimming xfer length
Thu Feb 9 22:14:20 2023 daemon.warn [6173]: Cannot read from istream: connection broken
This results in much degraded throughout than what is expected.
We have tested with different modems and we do not get any errors and performance is as expected. Any idea what could be going wrong ?