Serial Over USB No Longer Working (no Linux USB Gadget shows up)

I tested 5.6.0-rc7 8adc003c and nothing shows up, in SDP mode it always shows up.

After trying 5.6.0-rc7 I reverted back to 5.3.0 built by Jenkins and it works every time, the one built here: https://arm01.puri.sm/job/Kernel_builds/job/kernel_linux-next_imx8-current-librem5/9/

A previous kernel I tested that was built on March 13th was working sometimes but it seems to have gotten even worse where it doesn't work at all now. So I believe something within the month of March broke this.