Skip to content
  • Frederic Weisbecker's avatar
    irq_work: Fix racy check on work pending flag · e0bbe2d8
    Frederic Weisbecker authored
    
    
    Work claiming wants to be SMP-safe.
    
    And by the time we try to claim a work, if it is already executing
    concurrently on another CPU, we want to succeed the claiming and queue
    the work again because the other CPU may have missed the data we wanted
    to handle in our work if it's about to complete there.
    
    This scenario is summarized below:
    
            CPU 1                                   CPU 2
            -----                                   -----
            (flags = 0)
            cmpxchg(flags, 0, IRQ_WORK_FLAGS)
            (flags = 3)
            [...]
            xchg(flags, IRQ_WORK_BUSY)
            (flags = 2)
            func()
                                                    if (flags & IRQ_WORK_PENDING)
                                                            (not true)
                                                    cmpxchg(flags, flags, IRQ_WORK_FLAGS)
                                                    (flags = 3)
                                                    [...]
            cmpxchg(flags, IRQ_WORK_BUSY, 0);
            (fail, pending on CPU 2)
    
    This state machine is synchronized using [cmp]xchg() on the flags.
    As such, the early IRQ_WORK_PENDING check in CPU 2 above is racy.
    By the time we check it, we may be dealing with a stale value because
    we aren't using an atomic accessor. As a result, CPU 2 may "see"
    that the work is still pending on another CPU while it may be
    actually completing the work function exection already, leaving
    our data unprocessed.
    
    To fix this, we start by speculating about the value we wish to be
    in the work->flags but we only make any conclusion after the value
    returned by the cmpxchg() call that either claims the work or let
    the current owner handle the pending work for us.
    
    Changelog-heavily-inspired-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
    Acked-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Ingo Molnar <mingo@kernel.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Paul Gortmaker <paul.gortmaker@windriver.com>
    Cc: Anish Kumar <anish198519851985@gmail.com>
    e0bbe2d8