Skip to content
  • Jason Baron's avatar
    [PATCH] pty_chars_in_buffer oops fix · ff55fe20
    Jason Baron authored
    The idea of this patch is to lock both sides of a ptmx/pty pair during line
    discipline changing.  This is needed to ensure that say a poll on one side of
    the pty doesn't occur while the line discipline is actively being changed.
    This resulted in an oops reported on lkml, see:
    
    	http://marc.theaimsgroup.com/?l=linux-kernel&m=111342171410005&w=2
    
    
    
    A 'hacky' approach was previously implmemented which served to eliminate the
    poll vs.  line discipline changing race.  However, this patch takes a more
    general approach to the issue.  The patch only adds locking on a less often
    used path, the line-discipline changing path, as opposed to locking the
    ptmx/pty pair on read/write/poll paths.
    
    The patch below, takes both ldisc locks in either order b/c the locks are both
    taken under the same spinlock().  I thought about locking the ptmx/pty
    separately, such as master always first but that introduces a 3 way deadlock.
    For example, process 1 does a blocking read on the slave side.  Then, process
    2 does an ldisc change on the slave side, which acquires the master ldisc lock
    but not the slave's.  Finally, process 3 does a write which blocks on the
    process 2's ldisc reference.
    
    This patch does introduce some changes in semantics.  For example, a line
    discipline change on side 'a' of a ptmx/pty pair, will now wait for a
    read/write to complete on the other side, or side 'b'.  The current behavior
    is to simply wait for any read/writes on only side 'a', not both sides 'a' and
    'b'.  I think this behavior makes sense, but I wanted to point it out.
    
    I've tested the patch with a bunch of read/write/poll while changing the line
    discipline out from underneath.
    
    This patch obviates the need for the above "hide the problem" patch.
    
    Signed-off-by: default avatarJason Baron <jbaron@redhat.com>
    Signed-off-by: default avatarAndrew Morton <akpm@osdl.org>
    Signed-off-by: default avatarLinus Torvalds <torvalds@osdl.org>
    ff55fe20