Call volume is too low for the cell modem
Currently the modem volume is too low in comparison to the normal system volume at least on the BM818-A1 modem. This is most apparent when using the Librem 5 in speakerphone, as it's really hard to hear what people are saying when in a call on speakerphone unless you have a perfectly silent room.
Based on some comparisons to listening to people talk on youtube on the phone vs hearing my voice come through on the call, and testing pulseaudio settings, increasing the modem volume by 50% seems to bring them in line.