This management is better done per-widget rather than per-screen,
as windows being destroyed won't trigger a leave notify for the
devices on top of it, and this information is too transitive
to keep weak refs and such.

This fixes the critical warning seen in gtk/tests/testing.