3 ==========================
4 Short users guide for SLUB
5 ==========================
7 The basic philosophy of SLUB is very different from SLAB. SLAB
8 requires rebuilding the kernel to activate debug options for all
9 slab caches. SLUB always includes full debugging but it is off by default.
10 SLUB can enable debugging only for selected slabs in order to avoid
11 an impact on overall system performance which may make a bug more
14 In order to switch debugging on one can add an option ``slub_debug``
15 to the kernel command line. That will enable full debugging for
18 Typically one would then use the ``slabinfo`` command to get statistical
19 data and perform operation on the slabs. By default ``slabinfo`` only lists
20 slabs that have data in them. See "slabinfo -h" for more options when
21 running the command. ``slabinfo`` can be compiled with
24 gcc -o slabinfo tools/vm/slabinfo.c
26 Some of the modes of operation of ``slabinfo`` require that slub debugging
27 be enabled on the command line. F.e. no tracking information will be
28 available without debugging on and validation can only partially
29 be performed if debugging was not switched on.
31 Some more sophisticated uses of slub_debug:
32 -------------------------------------------
34 Parameters may be given to ``slub_debug``. If none is specified then full
35 debugging is enabled. Format:
37 slub_debug=<Debug-Options>
38 Enable options for all slabs
40 slub_debug=<Debug-Options>,<slab name1>,<slab name2>,...
41 Enable options only for select slabs (no spaces
44 Multiple blocks of options for all slabs or selected slabs can be given, with
45 blocks of options delimited by ';'. The last of "all slabs" blocks is applied
46 to all slabs except those that match one of the "select slabs" block. Options
47 of the first "select slabs" blocks that matches the slab's name are applied.
49 Possible debug options are::
51 F Sanity checks on (enables SLAB_DEBUG_CONSISTENCY_CHECKS
52 Sorry SLAB legacy issues)
54 P Poisoning (object and padding)
55 U User tracking (free and alloc)
56 T Trace (please only use on single slabs)
57 A Enable failslab filter mark for the cache
58 O Switch debugging off for caches that would have
59 caused higher minimum slab orders
60 - Switch all debugging off (useful if the kernel is
61 configured with CONFIG_SLUB_DEBUG_ON)
63 F.e. in order to boot just with sanity checks and red zoning one would specify::
67 Trying to find an issue in the dentry cache? Try::
71 to only enable debugging on the dentry cache. You may use an asterisk at the
72 end of the slab name, in order to cover all slabs with the same prefix. For
73 example, here's how you can poison the dentry cache as well as all kmalloc
76 slub_debug=P,kmalloc-*,dentry
78 Red zoning and tracking may realign the slab. We can just apply sanity checks
79 to the dentry cache with::
83 Debugging options may require the minimum possible slab order to increase as
84 a result of storing the metadata (for example, caches with PAGE_SIZE object
85 sizes). This has a higher liklihood of resulting in slab allocation errors
86 in low memory situations or if there's high fragmentation of memory. To
87 switch off debugging for such caches by default, use::
91 You can apply different options to different list of slab names, using blocks
92 of options. This will enable red zoning for dentry and user tracking for
93 kmalloc. All other slabs will not get any debugging enabled::
95 slub_debug=Z,dentry;U,kmalloc-*
97 You can also enable options (e.g. sanity checks and poisoning) for all caches
98 except some that are deemed too performance critical and don't need to be
99 debugged by specifying global debug options followed by a list of slab names
100 with "-" as options::
102 slub_debug=FZ;-,zs_handle,zspage
104 In case you forgot to enable debugging on the kernel command line: It is
105 possible to enable debugging manually when the kernel is up. Look at the
108 /sys/kernel/slab/<slab name>/
110 Look at the writable files. Writing 1 to them will enable the
111 corresponding debug option. All options can be set on a slab that does
112 not contain objects. If the slab already contains objects then sanity checks
113 and tracing may only be enabled. The other options may cause the realignment
116 Careful with tracing: It may spew out lots of information and never stop if
117 used on the wrong slab.
122 If no debug options are specified then SLUB may merge similar slabs together
123 in order to reduce overhead and increase cache hotness of objects.
124 ``slabinfo -a`` displays which slabs were merged together.
129 SLUB can validate all object if the kernel was booted with slub_debug. In
130 order to do so you must have the ``slabinfo`` tool. Then you can do
135 which will test all objects. Output will be generated to the syslog.
137 This also works in a more limited way if boot was without slab debug.
138 In that case ``slabinfo -v`` simply tests all reachable objects. Usually
139 these are in the cpu slabs and the partial slabs. Full slabs are not
140 tracked by SLUB in a non debug situation.
142 Getting more performance
143 ========================
145 To some degree SLUB's performance is limited by the need to take the
146 list_lock once in a while to deal with partial slabs. That overhead is
147 governed by the order of the allocation for each slab. The allocations
148 can be influenced by kernel parameters:
150 .. slub_min_objects=x (default 4)
151 .. slub_min_order=x (default 0)
152 .. slub_max_order=x (default 3 (PAGE_ALLOC_COSTLY_ORDER))
155 allows to specify how many objects must at least fit into one
156 slab in order for the allocation order to be acceptable. In
157 general slub will be able to perform this number of
158 allocations on a slab without consulting centralized resources
159 (list_lock) where contention may occur.
162 specifies a minimum order of slabs. A similar effect like
163 ``slub_min_objects``.
166 specified the order at which ``slub_min_objects`` should no
167 longer be checked. This is useful to avoid SLUB trying to
168 generate super large order pages to fit ``slub_min_objects``
169 of a slab cache with large object sizes into one high order
170 page. Setting command line parameter
171 ``debug_guardpage_minorder=N`` (N > 0), forces setting
172 ``slub_max_order`` to 0, what cause minimum possible order of
178 Here is a sample of slub debug output::
180 ====================================================================
181 BUG kmalloc-8: Redzone overwritten
182 --------------------------------------------------------------------
184 INFO: 0xc90f6d28-0xc90f6d2b. First byte 0x00 instead of 0xcc
185 INFO: Slab 0xc528c530 flags=0x400000c3 inuse=61 fp=0xc90f6d58
186 INFO: Object 0xc90f6d20 @offset=3360 fp=0xc90f6d58
187 INFO: Allocated in get_modalias+0x61/0xf5 age=53 cpu=1 pid=554
189 Bytes b4 0xc90f6d10: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
190 Object 0xc90f6d20: 31 30 31 39 2e 30 30 35 1019.005
191 Redzone 0xc90f6d28: 00 cc cc cc .
192 Padding 0xc90f6d50: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
194 [<c010523d>] dump_trace+0x63/0x1eb
195 [<c01053df>] show_trace_log_lvl+0x1a/0x2f
196 [<c010601d>] show_trace+0x12/0x14
197 [<c0106035>] dump_stack+0x16/0x18
198 [<c017e0fa>] object_err+0x143/0x14b
199 [<c017e2cc>] check_object+0x66/0x234
200 [<c017eb43>] __slab_free+0x239/0x384
201 [<c017f446>] kfree+0xa6/0xc6
202 [<c02e2335>] get_modalias+0xb9/0xf5
203 [<c02e23b7>] dmi_dev_uevent+0x27/0x3c
204 [<c027866a>] dev_uevent+0x1ad/0x1da
205 [<c0205024>] kobject_uevent_env+0x20a/0x45b
206 [<c020527f>] kobject_uevent+0xa/0xf
207 [<c02779f1>] store_uevent+0x4f/0x58
208 [<c027758e>] dev_attr_store+0x29/0x2f
209 [<c01bec4f>] sysfs_write_file+0x16e/0x19c
210 [<c0183ba7>] vfs_write+0xd1/0x15a
211 [<c01841d7>] sys_write+0x3d/0x72
212 [<c0104112>] sysenter_past_esp+0x5f/0x99
213 [<b7f7b410>] 0xb7f7b410
214 =======================
216 FIX kmalloc-8: Restoring Redzone 0xc90f6d28-0xc90f6d2b=0xcc
218 If SLUB encounters a corrupted object (full detection requires the kernel
219 to be booted with slub_debug) then the following output will be dumped
222 1. Description of the problem encountered
224 This will be a message in the system log starting with::
226 ===============================================
227 BUG <slab cache affected>: <What went wrong>
228 -----------------------------------------------
230 INFO: <corruption start>-<corruption_end> <more info>
231 INFO: Slab <address> <slab information>
232 INFO: Object <address> <object information>
233 INFO: Allocated in <kernel function> age=<jiffies since alloc> cpu=<allocated by
234 cpu> pid=<pid of the process>
235 INFO: Freed in <kernel function> age=<jiffies since free> cpu=<freed by cpu>
236 pid=<pid of the process>
238 (Object allocation / free information is only available if SLAB_STORE_USER is
239 set for the slab. slub_debug sets that option)
241 2. The object contents if an object was involved.
243 Various types of lines can follow the BUG SLUB line:
245 Bytes b4 <address> : <bytes>
246 Shows a few bytes before the object where the problem was detected.
247 Can be useful if the corruption does not stop with the start of the
250 Object <address> : <bytes>
251 The bytes of the object. If the object is inactive then the bytes
252 typically contain poison values. Any non-poison value shows a
253 corruption by a write after free.
255 Redzone <address> : <bytes>
256 The Redzone following the object. The Redzone is used to detect
257 writes after the object. All bytes should always have the same
258 value. If there is any deviation then it is due to a write after
261 (Redzone information is only available if SLAB_RED_ZONE is set.
262 slub_debug sets that option)
264 Padding <address> : <bytes>
265 Unused data to fill up the space in order to get the next object
266 properly aligned. In the debug case we make sure that there are
267 at least 4 bytes of padding. This allows the detection of writes
272 The stackdump describes the location where the error was detected. The cause
273 of the corruption is may be more likely found by looking at the function that
274 allocated or freed the object.
276 4. Report on how the problem was dealt with in order to ensure the continued
277 operation of the system.
279 These are messages in the system log beginning with::
281 FIX <slab cache affected>: <corrective action taken>
283 In the above sample SLUB found that the Redzone of an active object has
284 been overwritten. Here a string of 8 characters was written into a slab that
285 has the length of 8 characters. However, a 8 character string needs a
286 terminating 0. That zero has overwritten the first byte of the Redzone field.
287 After reporting the details of the issue encountered the FIX SLUB message
288 tells us that SLUB has restored the Redzone to its proper value and then
289 system operations continue.
294 Minimal debugging (sanity checks alone) can be enabled by booting with::
298 This will be generally be enough to enable the resiliency features of slub
299 which will keep the system running even if a bad kernel component will
300 keep corrupting objects. This may be important for production systems.
301 Performance will be impacted by the sanity checks and there will be a
302 continual stream of error messages to the syslog but no additional memory
303 will be used (unlike full debugging).
305 No guarantees. The kernel component still needs to be fixed. Performance
306 may be optimized further by locating the slab that experiences corruption
307 and enabling debugging only for that cache
313 If the corruption occurs by writing after the end of the object then it
314 may be advisable to enable a Redzone to avoid corrupting the beginning
319 Extended slabinfo mode and plotting
320 ===================================
322 The ``slabinfo`` tool has a special 'extended' ('-X') mode that includes:
324 - Slabs sorted by size (up to -N <num> slabs, default 1)
325 - Slabs sorted by loss (up to -N <num> slabs, default 1)
327 Additionally, in this mode ``slabinfo`` does not dynamically scale
328 sizes (G/M/K) and reports everything in bytes (this functionality is
329 also available to other slabinfo modes via '-B' option) which makes
330 reporting more precise and accurate. Moreover, in some sense the `-X'
331 mode also simplifies the analysis of slabs' behaviour, because its
332 output can be plotted using the ``slabinfo-gnuplot.sh`` script. So it
333 pushes the analysis from looking through the numbers (tons of numbers)
334 to something easier -- visual analysis.
338 a) collect slabinfo extended records, for example::
340 while [ 1 ]; do slabinfo -X >> FOO_STATS; sleep 1; done
342 b) pass stats file(-s) to ``slabinfo-gnuplot.sh`` script::
344 slabinfo-gnuplot.sh FOO_STATS [FOO_STATS2 .. FOO_STATSN]
346 The ``slabinfo-gnuplot.sh`` script will pre-processes the collected records
347 and generates 3 png files (and 3 pre-processing cache files) per STATS
349 - Slabcache Totals: FOO_STATS-totals.png
350 - Slabs sorted by size: FOO_STATS-slabs-by-size.png
351 - Slabs sorted by loss: FOO_STATS-slabs-by-loss.png
353 Another use case, when ``slabinfo-gnuplot.sh`` can be useful, is when you
354 need to compare slabs' behaviour "prior to" and "after" some code
355 modification. To help you out there, ``slabinfo-gnuplot.sh`` script
356 can 'merge' the `Slabcache Totals` sections from different
357 measurements. To visually compare N plots:
359 a) Collect as many STATS1, STATS2, .. STATSN files as you need::
361 while [ 1 ]; do slabinfo -X >> STATS<X>; sleep 1; done
363 b) Pre-process those STATS files::
365 slabinfo-gnuplot.sh STATS1 STATS2 .. STATSN
367 c) Execute ``slabinfo-gnuplot.sh`` in '-t' mode, passing all of the
368 generated pre-processed \*-totals::
370 slabinfo-gnuplot.sh -t STATS1-totals STATS2-totals .. STATSN-totals
372 This will produce a single plot (png file).
374 Plots, expectedly, can be large so some fluctuations or small spikes
375 can go unnoticed. To deal with that, ``slabinfo-gnuplot.sh`` has two
376 options to 'zoom-in'/'zoom-out':
378 a) ``-s %d,%d`` -- overwrites the default image width and heigh
379 b) ``-r %d,%d`` -- specifies a range of samples to use (for example,
380 in ``slabinfo -X >> FOO_STATS; sleep 1;`` case, using a ``-r
381 40,60`` range will plot only samples collected between 40th and
384 Christoph Lameter, May 30, 2007
385 Sergey Senozhatsky, October 23, 2015