If the grab window is destroyed the grab will be implicitly removed,
although we won't get GdkSeat:ungrab called in order to clear our
internal window<->seat relation entirely. Setting a weak ref will
nullify the pointer we keep on the seat to the window, avoiding the
expected crashes.