rcu-tasks: Stop RCU Tasks Trace from scanning full tasks list

This commit takes off the training wheels and relies only on scanning
currently running tasks and tasks that have blocked or been preempted
within their current RCU Tasks Trace read-side critical section.

Before this commit, the time complexity of an RCU Tasks Trace grace
period is O(T), where T is the number of tasks.  After this commit,
this time complexity is O(C+B), where C is the number of CPUs and B
is the number of tasks that have blocked (or been preempted) at least
once during their current RCU Tasks Trace read-side critical sections.
Of course, if all tasks have blocked (or been preempted) at least once
during their current RCU Tasks Trace read-side critical sections, this is
still O(T), but current expectations are that RCU Tasks Trace read-side
critical section will be short and that there will normally not be large
numbers of tasks blocked within such a critical section.

Dave Marchevsky kindly measured the effects of this commit on the RCU
Tasks Trace grace-period latency and the rcu_tasks_trace_kthread task's
CPU consumption per RCU Tasks Trace grace period over the course of a
fixed test, all in milliseconds:

		Before			After

GP latency	22.3 ms stddev > 0.1	17.0 ms stddev < 0.1

GP CPU		 2.3 ms stddev 0.3	 1.1 ms stddev 0.2

This was on a system with 15,000 tasks, so it is reasonable to expect
much larger savings on the systems on which this issue was first noted,
given that they sport well in excess of 100,000 tasks.  CPU consumption
was measured using profiling techniques.

Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Neeraj Upadhyay <quic_neeraju@quicinc.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Martin KaFai Lau <kafai@fb.com>
Cc: KP Singh <kpsingh@kernel.org>
Tested-by: Dave Marchevsky <davemarchevsky@fb.com>

diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index 272c905..fe05520 100644 (file)
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -670,10 +670,12 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp)
         * and make a list of them in holdouts.
         */
        set_tasks_gp_state(rtp, RTGS_SCAN_TASKLIST);
-       rcu_read_lock();
-       for_each_process_thread(g, t)
-               rtp->pertask_func(t, &holdouts);
-       rcu_read_unlock();
+       if (rtp->pertask_func) {
+               rcu_read_lock();
+               for_each_process_thread(g, t)
+                       rtp->pertask_func(t, &holdouts);
+               rcu_read_unlock();
+       }
 
        set_tasks_gp_state(rtp, RTGS_POST_SCAN_TASKLIST);
        rtp->postscan_func(&holdouts);
@@ -1746,7 +1748,6 @@ static int __init rcu_spawn_tasks_trace_kthread(void)
                        rcu_tasks_trace.init_fract = 1;
        }
        rcu_tasks_trace.pregp_func = rcu_tasks_trace_pregp_step;
-       rcu_tasks_trace.pertask_func = rcu_tasks_trace_pertask;
        rcu_tasks_trace.postscan_func = rcu_tasks_trace_postscan;
        rcu_tasks_trace.holdouts_func = check_all_holdout_tasks_trace;
        rcu_tasks_trace.postgp_func = rcu_tasks_trace_postgp;