dwc3 hangs in dwc3_gadget_exit

The gadget driver hangs the dwc3 on exit

[  242.643914] INFO: task kworker/3:1:38 blocked for more than 120 seconds.
[  242.650815]       Not tainted 5.7.5-librem5-00029-g90e07e6bedfb-dirty #217
[  242.657971] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.665909] kworker/3:1     D    0    38      2 0x00000028
[  242.666156] Workqueue: events_freezable __dwc3_set_mode [dwc3]
[  242.666189] Call trace:
[  242.666273]  __switch_to+0xf4/0x148
[  242.666332]  __schedule+0x2c0/0x658
[  242.666345]  schedule+0x50/0x110
[  242.666432]  gserial_free_port+0xb0/0x100 [u_serial]
[  242.666449]  gserial_free_line+0x70/0xb8 [u_serial]
[  242.666506]  acm_free_instance+0x24/0x40 [usb_f_acm]
[  242.666724]  usb_put_function_instance+0x2c/0x48 [libcomposite]
[  242.666784]  multi_unbind+0x40/0x70 [g_multi]
[  242.666808]  __composite_unbind+0x90/0xf0 [libcomposite]
[  242.666834]  composite_unbind+0x1c/0x28 [libcomposite]
[  242.666994]  usb_gadget_remove_driver+0x44/0xa0 [udc_core]
[  242.667018]  usb_del_gadget_udc+0x70/0xe0 [udc_core]
[  242.667060]  dwc3_gadget_exit+0x24/0x78 [dwc3]
[  242.667081]  __dwc3_set_mode+0xd0/0x264 [dwc3]
[  242.667136]  process_one_work+0x1c8/0x488
[  242.667149]  worker_thread+0x54/0x430
[  242.667182]  kthread+0x11c/0x150
[  242.667203]  ret_from_fork+0x10/0x1c