mm, slub: simplify kmem_cache_cpu and tid setup
authorVlastimil Babka <vbabka@suse.cz>
Tue, 18 May 2021 00:01:39 +0000 (02:01 +0200)
committerVlastimil Babka <vbabka@suse.cz>
Fri, 3 Sep 2021 23:12:20 +0000 (01:12 +0200)
commit9b4bc85a69f57fea93c1f359b393d2c24857d2f5
tree1da6925aef4fcc0066ef3c60f5973201af20e3ad
parent1572df7cbcb48936c880f2d2de524f8e47ab65d4
mm, slub: simplify kmem_cache_cpu and tid setup

In slab_alloc_node() and do_slab_free() fastpaths we need to guarantee that
our kmem_cache_cpu pointer is from the same cpu as the tid value. Currently
that's done by reading the tid first using this_cpu_read(), then the
kmem_cache_cpu pointer and verifying we read the same tid using the pointer and
plain READ_ONCE().

This can be simplified to just fetching kmem_cache_cpu pointer and then reading
tid using the pointer. That guarantees they are from the same cpu. We don't
need to read the tid using this_cpu_read() because the value will be validated
by this_cpu_cmpxchg_double(), making sure we are on the correct cpu and the
freelist didn't change by anyone preempting us since reading the tid.

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
mm/slub.c