1 .. SPDX-License-Identifier: GPL-2.0
4 Please see MAINTAINERS file for where to send questions.
9 This document describes a prototype for a new approach to providing
10 overlay-filesystem functionality in Linux (sometimes referred to as
11 union-filesystems). An overlay-filesystem tries to present a
12 filesystem which is the result over overlaying one filesystem on top
19 The overlay filesystem approach is 'hybrid', because the objects that
20 appear in the filesystem do not always appear to belong to that filesystem.
21 In many cases, an object accessed in the union will be indistinguishable
22 from accessing the corresponding object from the original filesystem.
23 This is most obvious from the 'st_dev' field returned by stat(2).
25 While directories will report an st_dev from the overlay-filesystem,
26 non-directory objects may report an st_dev from the lower filesystem or
27 upper filesystem that is providing the object. Similarly st_ino will
28 only be unique when combined with st_dev, and both of these can change
29 over the lifetime of a non-directory object. Many applications and
30 tools ignore these values and will not be affected.
32 In the special case of all overlay layers on the same underlying
33 filesystem, all objects will report an st_dev from the overlay
34 filesystem and st_ino from the underlying filesystem. This will
35 make the overlay mount more compliant with filesystem scanners and
36 overlay objects will be distinguishable from the corresponding
37 objects in the original filesystem.
39 On 64bit systems, even if all overlay layers are not on the same
40 underlying filesystem, the same compliant behavior could be achieved
41 with the "xino" feature. The "xino" feature composes a unique object
42 identifier from the real object st_ino and an underlying fsid index.
43 If all underlying filesystems support NFS file handles and export file
44 handles with 32bit inode number encoding (e.g. ext4), overlay filesystem
45 will use the high inode number bits for fsid. Even when the underlying
46 filesystem uses 64bit inode numbers, users can still enable the "xino"
47 feature with the "-o xino=on" overlay mount option. That is useful for the
48 case of underlying filesystems like xfs and tmpfs, which use 64bit inode
49 numbers, but are very unlikely to use the high inode number bit.
55 An overlay filesystem combines two filesystems - an 'upper' filesystem
56 and a 'lower' filesystem. When a name exists in both filesystems, the
57 object in the 'upper' filesystem is visible while the object in the
58 'lower' filesystem is either hidden or, in the case of directories,
59 merged with the 'upper' object.
61 It would be more correct to refer to an upper and lower 'directory
62 tree' rather than 'filesystem' as it is quite possible for both
63 directory trees to be in the same filesystem and there is no
64 requirement that the root of a filesystem be given for either upper or
67 The lower filesystem can be any filesystem supported by Linux and does
68 not need to be writable. The lower filesystem can even be another
69 overlayfs. The upper filesystem will normally be writable and if it
70 is it must support the creation of trusted.* extended attributes, and
71 must provide valid d_type in readdir responses, so NFS is not suitable.
73 A read-only overlay of two read-only filesystems may use any
79 Overlaying mainly involves directories. If a given name appears in both
80 upper and lower filesystems and refers to a non-directory in either,
81 then the lower object is hidden - the name refers only to the upper
84 Where both upper and lower objects are directories, a merged directory
87 At mount time, the two directories given as mount options "lowerdir" and
88 "upperdir" are combined into a merged directory:
90 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\
93 The "workdir" needs to be an empty directory on the same filesystem
96 Then whenever a lookup is requested in such a merged directory, the
97 lookup is performed in each actual directory and the combined result
98 is cached in the dentry belonging to the overlay filesystem. If both
99 actual lookups find directories, both are stored and a merged
100 directory is created, otherwise only one is stored: the upper if it
101 exists, else the lower.
103 Only the lists of names from directories are merged. Other content
104 such as metadata and extended attributes are reported for the upper
105 directory only. These attributes of the lower directory are hidden.
107 whiteouts and opaque directories
108 --------------------------------
110 In order to support rm and rmdir without changing the lower
111 filesystem, an overlay filesystem needs to record in the upper filesystem
112 that files have been removed. This is done using whiteouts and opaque
113 directories (non-directories are always opaque).
115 A whiteout is created as a character device with 0/0 device number.
116 When a whiteout is found in the upper level of a merged directory, any
117 matching name in the lower level is ignored, and the whiteout itself
120 A directory is made opaque by setting the xattr "trusted.overlay.opaque"
121 to "y". Where the upper filesystem contains an opaque directory, any
122 directory in the lower filesystem with the same name is ignored.
127 When a 'readdir' request is made on a merged directory, the upper and
128 lower directories are each read and the name lists merged in the
129 obvious way (upper is read first, then lower - entries that already
130 exist are not re-added). This merged name list is cached in the
131 'struct file' and so remains as long as the file is kept open. If the
132 directory is opened and read by two processes at the same time, they
133 will each have separate caches. A seekdir to the start of the
134 directory (offset 0) followed by a readdir will cause the cache to be
135 discarded and rebuilt.
137 This means that changes to the merged directory do not appear while a
138 directory is being read. This is unlikely to be noticed by many
141 seek offsets are assigned sequentially when the directories are read.
144 - read part of a directory
145 - remember an offset, and close the directory
146 - re-open the directory some time later
147 - seek to the remembered offset
149 there may be little correlation between the old and new locations in
150 the list of filenames, particularly if anything has changed in the
153 Readdir on directories that are not merged is simply handled by the
154 underlying directory (upper or lower).
159 When renaming a directory that is on the lower layer or merged (i.e. the
160 directory was not created on the upper layer to start with) overlayfs can
161 handle it in two different ways:
163 1. return EXDEV error: this error is returned by rename(2) when trying to
164 move a file or directory across filesystem boundaries. Hence
165 applications are usually prepared to hande this error (mv(1) for example
166 recursively copies the directory tree). This is the default behavior.
168 2. If the "redirect_dir" feature is enabled, then the directory will be
169 copied up (but not the contents). Then the "trusted.overlay.redirect"
170 extended attribute is set to the path of the original location from the
171 root of the overlay. Finally the directory is moved to the new
174 There are several ways to tune the "redirect_dir" feature.
176 Kernel config options:
178 - OVERLAY_FS_REDIRECT_DIR:
179 If this is enabled, then redirect_dir is turned on by default.
180 - OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW:
181 If this is enabled, then redirects are always followed by default. Enabling
182 this results in a less secure configuration. Enable this option only when
183 worried about backward compatibility with kernels that have the redirect_dir
184 feature and follow redirects even if turned off.
186 Module options (can also be changed through /sys/module/overlay/parameters/):
188 - "redirect_dir=BOOL":
189 See OVERLAY_FS_REDIRECT_DIR kernel config option above.
190 - "redirect_always_follow=BOOL":
191 See OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW kernel config option above.
192 - "redirect_max=NUM":
193 The maximum number of bytes in an absolute redirect (default is 256).
198 Redirects are enabled.
199 - "redirect_dir=follow":
200 Redirects are not created, but followed.
201 - "redirect_dir=off":
202 Redirects are not created and only followed if "redirect_always_follow"
203 feature is enabled in the kernel/module config.
204 - "redirect_dir=nofollow":
205 Redirects are not created and not followed (equivalent to "redirect_dir=off"
206 if "redirect_always_follow" feature is not enabled).
208 When the NFS export feature is enabled, every copied up directory is
209 indexed by the file handle of the lower inode and a file handle of the
210 upper directory is stored in a "trusted.overlay.upper" extended attribute
211 on the index entry. On lookup of a merged directory, if the upper
212 directory does not match the file handle stores in the index, that is an
213 indication that multiple upper directories may be redirected to the same
214 lower directory. In that case, lookup returns an error and warns about
215 a possible inconsistency.
217 Because lower layer redirects cannot be verified with the index, enabling
218 NFS export support on an overlay filesystem with no upper layer requires
219 turning off redirect follow (e.g. "redirect_dir=nofollow").
225 Objects that are not directories (files, symlinks, device-special
226 files etc.) are presented either from the upper or lower filesystem as
227 appropriate. When a file in the lower filesystem is accessed in a way
228 the requires write-access, such as opening for write access, changing
229 some metadata etc., the file is first copied from the lower filesystem
230 to the upper filesystem (copy_up). Note that creating a hard-link
231 also requires copy_up, though of course creation of a symlink does
234 The copy_up may turn out to be unnecessary, for example if the file is
235 opened for read-write but the data is not modified.
237 The copy_up process first makes sure that the containing directory
238 exists in the upper filesystem - creating it and any parents as
239 necessary. It then creates the object with the same metadata (owner,
240 mode, mtime, symlink-target etc.) and then if the object is a file, the
241 data is copied from the lower to the upper filesystem. Finally any
242 extended attributes are copied up.
244 Once the copy_up is complete, the overlay filesystem simply
245 provides direct access to the newly created file in the upper
246 filesystem - future operations on the file are barely noticed by the
247 overlay filesystem (though an operation on the name of the file such as
248 rename or unlink will of course be noticed and handled).
251 Multiple lower layers
252 ---------------------
254 Multiple lower layers can now be given using the the colon (":") as a
255 separator character between the directory names. For example:
257 mount -t overlay overlay -olowerdir=/lower1:/lower2:/lower3 /merged
259 As the example shows, "upperdir=" and "workdir=" may be omitted. In
260 that case the overlay will be read-only.
262 The specified lower directories will be stacked beginning from the
263 rightmost one and going left. In the above example lower1 will be the
264 top, lower2 the middle and lower3 the bottom layer.
267 Metadata only copy up
268 ---------------------
270 When metadata only copy up feature is enabled, overlayfs will only copy
271 up metadata (as opposed to whole file), when a metadata specific operation
272 like chown/chmod is performed. Full file will be copied up later when
273 file is opened for WRITE operation.
275 In other words, this is delayed data copy up operation and data is copied
276 up when there is a need to actually modify data.
278 There are multiple ways to enable/disable this feature. A config option
279 CONFIG_OVERLAY_FS_METACOPY can be set/unset to enable/disable this feature
280 by default. Or one can enable/disable it at module load time with module
281 parameter metacopy=on/off. Lastly, there is also a per mount option
282 metacopy=on/off to enable/disable this feature per mount.
284 Do not use metacopy=on with untrusted upper/lower directories. Otherwise
285 it is possible that an attacker can create a handcrafted file with
286 appropriate REDIRECT and METACOPY xattrs, and gain access to file on lower
287 pointed by REDIRECT. This should not be possible on local system as setting
288 "trusted." xattrs will require CAP_SYS_ADMIN. But it should be possible
289 for untrusted layers like from a pen drive.
291 Note: redirect_dir={off|nofollow|follow[*]} conflicts with metacopy=on, and
294 [*] redirect_dir=follow only conflicts with metacopy=on if upperdir=... is
297 Sharing and copying layers
298 --------------------------
300 Lower layers may be shared among several overlay mounts and that is indeed
301 a very common practice. An overlay mount may use the same lower layer
302 path as another overlay mount and it may use a lower layer path that is
303 beneath or above the path of another overlay lower layer path.
305 Using an upper layer path and/or a workdir path that are already used by
306 another overlay mount is not allowed and may fail with EBUSY. Using
307 partially overlapping paths is not allowed and may fail with EBUSY.
308 If files are accessed from two overlayfs mounts which share or overlap the
309 upper layer and/or workdir path the behavior of the overlay is undefined,
310 though it will not result in a crash or deadlock.
312 Mounting an overlay using an upper layer path, where the upper layer path
313 was previously used by another mounted overlay in combination with a
314 different lower layer path, is allowed, unless the "inodes index" feature
315 or "metadata only copy up" feature is enabled.
317 With the "inodes index" feature, on the first time mount, an NFS file
318 handle of the lower layer root directory, along with the UUID of the lower
319 filesystem, are encoded and stored in the "trusted.overlay.origin" extended
320 attribute on the upper layer root directory. On subsequent mount attempts,
321 the lower root directory file handle and lower filesystem UUID are compared
322 to the stored origin in upper root directory. On failure to verify the
323 lower root origin, mount will fail with ESTALE. An overlayfs mount with
324 "inodes index" enabled will fail with EOPNOTSUPP if the lower filesystem
325 does not support NFS export, lower filesystem does not have a valid UUID or
326 if the upper filesystem does not support extended attributes.
328 For "metadata only copy up" feature there is no verification mechanism at
329 mount time. So if same upper is mounted with different set of lower, mount
330 probably will succeed but expect the unexpected later on. So don't do it.
332 It is quite a common practice to copy overlay layers to a different
333 directory tree on the same or different underlying filesystem, and even
334 to a different machine. With the "inodes index" feature, trying to mount
335 the copied layers will fail the verification of the lower root file handle.
338 Non-standard behavior
339 ---------------------
341 Current version of overlayfs can act as a mostly POSIX compliant
344 This is the list of cases that overlayfs doesn't currently handle:
346 a) POSIX mandates updating st_atime for reads. This is currently not
347 done in the case when the file resides on a lower layer.
349 b) If a file residing on a lower layer is opened for read-only and then
350 memory mapped with MAP_SHARED, then subsequent changes to the file are not
351 reflected in the memory mapping.
353 The following options allow overlayfs to act more like a standards
354 compliant filesystem:
358 Enabled with the mount option or module option: "redirect_dir=on" or with
359 the kernel config option CONFIG_OVERLAY_FS_REDIRECT_DIR=y.
361 If this feature is disabled, then rename(2) on a lower or merged directory
362 will fail with EXDEV ("Invalid cross-device link").
366 Enabled with the mount option or module option "index=on" or with the
367 kernel config option CONFIG_OVERLAY_FS_INDEX=y.
369 If this feature is disabled and a file with multiple hard links is copied
370 up, then this will "break" the link. Changes will not be propagated to
371 other names referring to the same inode.
375 Enabled with the mount option "xino=auto" or "xino=on", with the module
376 option "xino_auto=on" or with the kernel config option
377 CONFIG_OVERLAY_FS_XINO_AUTO=y. Also implicitly enabled by using the same
378 underlying filesystem for all layers making up the overlay.
380 If this feature is disabled or the underlying filesystem doesn't have
381 enough free bits in the inode number, then overlayfs will not be able to
382 guarantee that the values of st_ino and st_dev returned by stat(2) and the
383 value of d_ino returned by readdir(3) will act like on a normal filesystem.
384 E.g. the value of st_dev may be different for two objects in the same
385 overlay filesystem and the value of st_ino for directory objects may not be
386 persistent and could change even while the overlay filesystem is mounted.
389 Changes to underlying filesystems
390 ---------------------------------
392 Offline changes, when the overlay is not mounted, are allowed to either
393 the upper or the lower trees.
395 Changes to the underlying filesystems while part of a mounted overlay
396 filesystem are not allowed. If the underlying filesystem is changed,
397 the behavior of the overlay is undefined, though it will not result in
400 When the overlay NFS export feature is enabled, overlay filesystems
401 behavior on offline changes of the underlying lower layer is different
402 than the behavior when NFS export is disabled.
404 On every copy_up, an NFS file handle of the lower inode, along with the
405 UUID of the lower filesystem, are encoded and stored in an extended
406 attribute "trusted.overlay.origin" on the upper inode.
408 When the NFS export feature is enabled, a lookup of a merged directory,
409 that found a lower directory at the lookup path or at the path pointed
410 to by the "trusted.overlay.redirect" extended attribute, will verify
411 that the found lower directory file handle and lower filesystem UUID
412 match the origin file handle that was stored at copy_up time. If a
413 found lower directory does not match the stored origin, that directory
414 will not be merged with the upper directory.
421 When the underlying filesystems supports NFS export and the "nfs_export"
422 feature is enabled, an overlay filesystem may be exported to NFS.
424 With the "nfs_export" feature, on copy_up of any lower object, an index
425 entry is created under the index directory. The index entry name is the
426 hexadecimal representation of the copy up origin file handle. For a
427 non-directory object, the index entry is a hard link to the upper inode.
428 For a directory object, the index entry has an extended attribute
429 "trusted.overlay.upper" with an encoded file handle of the upper
432 When encoding a file handle from an overlay filesystem object, the
433 following rules apply:
435 1. For a non-upper object, encode a lower file handle from lower inode
436 2. For an indexed object, encode a lower file handle from copy_up origin
437 3. For a pure-upper object and for an existing non-indexed upper object,
438 encode an upper file handle from upper inode
440 The encoded overlay file handle includes:
441 - Header including path type information (e.g. lower/upper)
442 - UUID of the underlying filesystem
443 - Underlying filesystem encoding of underlying inode
445 This encoding format is identical to the encoding format file handles that
446 are stored in extended attribute "trusted.overlay.origin".
448 When decoding an overlay file handle, the following steps are followed:
450 1. Find underlying layer by UUID and path type information.
451 2. Decode the underlying filesystem file handle to underlying dentry.
452 3. For a lower file handle, lookup the handle in index directory by name.
453 4. If a whiteout is found in index, return ESTALE. This represents an
454 overlay object that was deleted after its file handle was encoded.
455 5. For a non-directory, instantiate a disconnected overlay dentry from the
456 decoded underlying dentry, the path type and index inode, if found.
457 6. For a directory, use the connected underlying decoded dentry, path type
458 and index, to lookup a connected overlay dentry.
460 Decoding a non-directory file handle may return a disconnected dentry.
461 copy_up of that disconnected dentry will create an upper index entry with
464 When overlay filesystem has multiple lower layers, a middle layer
465 directory may have a "redirect" to lower directory. Because middle layer
466 "redirects" are not indexed, a lower file handle that was encoded from the
467 "redirect" origin directory, cannot be used to find the middle or upper
468 layer directory. Similarly, a lower file handle that was encoded from a
469 descendant of the "redirect" origin directory, cannot be used to
470 reconstruct a connected overlay path. To mitigate the cases of
471 directories that cannot be decoded from a lower file handle, these
472 directories are copied up on encode and encoded as an upper file handle.
473 On an overlay filesystem with no upper layer this mitigation cannot be
474 used NFS export in this setup requires turning off redirect follow (e.g.
475 "redirect_dir=nofollow").
477 The overlay filesystem does not support non-directory connectable file
478 handles, so exporting with the 'subtree_check' exportfs configuration will
479 cause failures to lookup files over NFS.
481 When the NFS export feature is enabled, all directory index entries are
482 verified on mount time to check that upper file handles are not stale.
483 This verification may cause significant overhead in some cases.
489 There's a testsuite originally developed by David Howells and currently
490 maintained by Amir Goldstein at:
492 https://github.com/amir73il/unionmount-testsuite.git
496 # cd unionmount-testsuite
497 # ./run --ov --verify