Don't start sending audio until calls are answered
From Matrix:
Angus
Bob: I figured out why there's latency. The audio starts recording as soon as the call is placed but doesn't start sending until the call connects so however long it takes to connect is how long the voice is out of sync