USB gadget resets spontaneously

Sometimes when I have the phone connected to a PC, the connection resets spontaneously.

dmesg on the host side shows:

[419552.532692] usb 1-9: USB disconnect, device number 21
[419552.532830] cdc_eem 1-9:1.0 enp0s20f0u9: unregister 'cdc_eem' usb-0000:00:14.0-9, CDC EEM Device
[419552.533524] br0: port 1(enp0s20f0u9) entered disabled state
[419552.540814] device enp0s20f0u9 left promiscuous mode
[419552.540844] br0: port 1(enp0s20f0u9) entered disabled state
[419552.868227] usb 1-9: new high-speed USB device number 22 using xhci_hcd
[419558.108387] usb 1-9: unable to read config index 0 descriptor/start: -110
[419558.108398] usb 1-9: can't read configurations, error -110
[419558.231673] usb 1-9: new high-speed USB device number 23 using xhci_hcd
[419558.376828] usb 1-9: New USB device found, idVendor=0525, idProduct=a4a2, bcdDevice= 5.07
[419558.376832] usb 1-9: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[419558.376835] usb 1-9: Product: RNDIS/Ethernet Gadget
[419558.376837] usb 1-9: Manufacturer: Linux 5.7.0-1-librem5 with dwc3-gadget
[419558.381870] cdc_eem 1-9:1.0 usb0: register 'cdc_eem' at usb-0000:00:14.0-9, CDC EEM Device, 56:97:fa:9c:a4:eb
[419558.417293] cdc_eem 1-9:1.0 enp0s20f0u9: renamed from usb0

(the log shows g_ether module being used, but it also happens with g_multi)

No signs of it happening on the device's dmesg (other that usb0 interface going down and up).