Problem:
Every so often, the modem module seems to get stuck. Neither a reset (via mmcli) nor a reboot seems to help. A power cycle is required. (Note: the modem is in an enclosure with external power.) Help?
Background info:
# uname -a
Linux zebra 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
# dpkg -l | egrep -i 'qmi|modemmanager'
ii libqmi-glib5:arm64 1.26.10-0.1 arm64 Support library to use the Qualcomm MSM Interface (QMI) protocol
ii libqmi-proxy 1.26.10-0.1 arm64 Proxy to communicate with QMI ports
ii libqmi-utils 1.26.10-0.1 arm64 Utilities to use the QMI protocol from the command line
ii modemmanager 1.14.12-0.2 arm64 D-Bus service for managing modems
# mmcli -m 0
-----------------------------------
General | dbus path: /org/freedesktop/ModemManager1/Modem/0
| device id: 45def29d33a37135e341083a1edb4f94035b7b65
-----------------------------------
Hardware | manufacturer: Quectel
| model: RM502Q-AE
| firmware revision: RM502QAEAAR13A03M4G
| carrier config: Commercial-TMO
| carrier config revision: 0A01050F
| h/w revision: 20000
| supported: gsm-umts, lte, 5gnr
| current: gsm-umts, lte, 5gnr
| equipment id: redacted
-----------------------------------
System | device: /sys/devices/platform/scb/fd500000.pcie/pci0000:00/0000:00:00.0/0000:01:00.0/usb2/2-1
| drivers: option1, qmi_wwan
| plugin: quectel
| primary port: cdc-wdm0
| ports: cdc-wdm0 (qmi), ttyUSB0 (qcdm), ttyUSB1 (at), ttyUSB2 (at),
| wwan0 (net)
Here is everything from syslog when the modem got stuck:
May 22 15:54:22 zebra ModemManager[514]: transaction 0x23dd aborted, but message is not abortable
May 22 15:54:22 zebra ModemManager[514]: <warn> [modem1/bearer1] reloading stats failed: QMI operation failed: Transaction timed out
May 22 15:54:49 zebra ModemManager[514]: transaction 0x23e4 aborted, but message is not abortable
May 22 15:54:52 zebra ModemManager[514]: transaction 0x23de aborted, but message is not abortable
May 22 15:54:52 zebra ModemManager[514]: <warn> [modem1/bearer1] reloading stats failed: QMI operation failed: Transaction timed out
May 22 15:55:22 zebra ModemManager[514]: transaction 0x23df aborted, but message is not abortable
May 22 15:55:22 zebra ModemManager[514]: <warn> [modem1/bearer1] reloading stats failed: QMI operation failed: Transaction timed out
May 22 15:55:24 zebra kernel: [431640.811850] ------------[ cut here ]------------
May 22 15:55:24 zebra kernel: [431640.811908] NETDEV WATCHDOG: wwan0 (qmi_wwan): transmit queue 0 timed out
May 22 15:55:24 zebra kernel: [431640.811986] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:526 dev_watchdog+0x298/0x2a0
May 22 15:55:24 zebra kernel: [431640.812022] Modules linked in: rfcomm cmac algif_hash aes_arm64 aes_generic algif_skcipher af_alg bnep hci_uart btbcm bluetooth ecdh_generic ecc libaes tun nft_chain_nat xt_MASQUERADE nf_nat xt_state xt_conntrack nf_conntrack nf_defrag_ipv6 option nf_defrag_ipv4 xt_HL usb_wwan nft_compat nf_tables qmi_wwan nfnetlink cdc_wdm usbserial brcmfmac brcmutil cfg80211 vc4 v3d snd_soc_hdmi_codec bcm2835_v4l2(C) bcm2835_isp(C) bcm2835_codec(C) drm_display_helper rpivid_hevc(C) cec bcm2835_mmal_vchiq(C) rfkill gpu_sched drm_dma_helper v4l2_mem2mem videobuf2_dma_contig videobuf2_vmalloc drm_kms_helper raspberrypi_hwmon videobuf2_memops drm_shmem_helper snd_soc_core videobuf2_v4l2 videobuf2_common snd_compress snd_bcm2835(C) snd_pcm_dmaengine snd_pcm i2c_brcmstb videodev snd_timer syscopyarea snd vc_sm_cma(C) mc sysfillrect sysimgblt fb_sys_fops nvmem_rmem uio_pdrv_genirq uio drm i2c_dev fuse drm_panel_orientation_quirks backlight ip_tables x_tables ipv6
May 22 15:55:24 zebra kernel: [431640.812435] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G C 6.1.21-v8+ #1642
May 22 15:55:24 zebra kernel: [431640.812450] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
May 22 15:55:24 zebra kernel: [431640.812458] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
May 22 15:55:24 zebra kernel: [431640.812471] pc : dev_watchdog+0x298/0x2a0
May 22 15:55:24 zebra kernel: [431640.812484] lr : dev_watchdog+0x298/0x2a0
May 22 15:55:24 zebra kernel: [431640.812494] sp : ffffffc008013df0
May 22 15:55:24 zebra kernel: [431640.812500] x29: ffffffc008013df0 x28: ffffff80fb7ae040 x27: ffffffc008013ee0
May 22 15:55:24 zebra kernel: [431640.812523] x26: ffffffde0ad56000 x25: ffffffde0a3ec880 x24: ffffff804f787488
May 22 15:55:24 zebra kernel: [431640.812544] x23: ffffffde0ad56000 x22: 0000000000000000 x21: ffffff804f7873dc
May 22 15:55:24 zebra kernel: [431640.812564] x20: ffffff804f787000 x19: ffffff804f787488 x18: 0000000000000000
May 22 15:55:24 zebra kernel: [431640.812585] x17: ffffffa2f0d67000 x16: ffffffc008014000 x15: ffffffffffffffff
May 22 15:55:24 zebra kernel: [431640.812605] x14: ffffffde0a8ad858 x13: 74756f2064656d69 x12: ffffffde0ade73b0
May 22 15:55:24 zebra kernel: [431640.812624] x11: 0000000000000003 x10: 0000000000000001 x9 : ffffffde09af054c
May 22 15:55:24 zebra kernel: [431640.812645] x8 : 0000000000017fe8 x7 : c0000000ffffefff x6 : ffffff80fb7ad950
May 22 15:55:24 zebra kernel: [431640.812665] x5 : ffffffa2f0d67000 x4 : 0000000000000040 x3 : 0000000000000104
May 22 15:55:24 zebra kernel: [431640.812685] x2 : 0000000000000103 x1 : 70105334ddd6dd00 x0 : 0000000000000000
May 22 15:55:24 zebra kernel: [431640.812705] Call trace:
May 22 15:55:24 zebra kernel: [431640.812711] dev_watchdog+0x298/0x2a0
May 22 15:55:24 zebra kernel: [431640.812724] call_timer_fn+0x38/0x1d0
May 22 15:55:24 zebra kernel: [431640.812741] run_timer_softirq+0x284/0x520
May 22 15:55:24 zebra kernel: [431640.812756] __do_softirq+0x1a0/0x4d8
May 22 15:55:24 zebra kernel: [431640.812767] ____do_softirq+0x18/0x28
May 22 15:55:24 zebra kernel: [431640.812780] call_on_irq_stack+0x2c/0x54
May 22 15:55:24 zebra kernel: [431640.812792] do_softirq_own_stack+0x24/0x30
May 22 15:55:24 zebra kernel: [431640.812804] irq_exit_rcu+0x10c/0x148
May 22 15:55:24 zebra kernel: [431640.812815] el1_interrupt+0x3c/0x70
May 22 15:55:24 zebra kernel: [431640.812831] el1h_64_irq_handler+0x18/0x28
May 22 15:55:24 zebra kernel: [431640.812845] el1h_64_irq+0x64/0x68
May 22 15:55:24 zebra kernel: [431640.812854] arch_cpu_idle+0x18/0x28
May 22 15:55:24 zebra kernel: [431640.812869] default_idle_call+0x50/0x188
May 22 15:55:24 zebra kernel: [431640.812881] do_idle+0x234/0x240
May 22 15:55:24 zebra kernel: [431640.812895] cpu_startup_entry+0x2c/0x38
May 22 15:55:24 zebra kernel: [431640.812908] secondary_start_kernel+0x128/0x150
May 22 15:55:24 zebra kernel: [431640.812924] __secondary_switched+0xb0/0xb4
May 22 15:55:24 zebra kernel: [431640.812942] ---[ end trace 0000000000000000 ]---
May 22 15:55:29 zebra ModemManager[514]: transaction 0x23e5 aborted, but message is not abortable
May 22 15:55:52 zebra ModemManager[514]: transaction 0x23e0 aborted, but message is not abortable
May 22 15:55:52 zebra ModemManager[514]: <warn> [modem1/bearer1] reloading stats failed: QMI operation failed: Transaction timed out
May 22 15:56:09 zebra ModemManager[514]: transaction 0x23e6 aborted, but message is not abortable
May 22 15:56:20 zebra ModemManager[514]: transaction 0x12 aborted, but message is not abortable
May 22 15:56:22 zebra ModemManager[514]: transaction 0x23e1 aborted, but message is not abortable
May 22 15:56:22 zebra ModemManager[514]: <warn> [modem1/bearer1] reloading stats failed: QMI operation failed: Transaction timed out