1 .. SPDX-License-Identifier: GPL-2.0
13 Welcome to User Mode Linux
15 User Mode Linux is the first Open Source virtualization platform (first
16 release date 1991) and second virtualization platform for an x86 PC.
18 How is UML Different from a VM using Virtualization package X?
19 ==============================================================
21 We have come to assume that virtualization also means some level of
22 hardware emulation. In fact, it does not. As long as a virtualization
23 package provides the OS with devices which the OS can recognize and
24 has a driver for, the devices do not need to emulate real hardware.
25 Most OSes today have built-in support for a number of "fake"
26 devices used only under virtualization.
27 User Mode Linux takes this concept to the ultimate extreme - there
28 is not a single real device in sight. It is 100% artificial or if
29 we use the correct term 100% paravirtual. All UML devices are abstract
30 concepts which map onto something provided by the host - files, sockets,
33 The other major difference between UML and various virtualization
34 packages is that there is a distinct difference between the way the UML
35 kernel and the UML programs operate.
36 The UML kernel is just a process running on Linux - same as any other
37 program. It can be run by an unprivileged user and it does not require
38 anything in terms of special CPU features.
39 The UML userspace, however, is a bit different. The Linux kernel on the
40 host machine assists UML in intercepting everything the program running
41 on a UML instance is trying to do and making the UML kernel handle all
43 This is different from other virtualization packages which do not make any
44 difference between the guest kernel and guest programs. This difference
45 results in a number of advantages and disadvantages of UML over let's say
46 QEMU which we will cover later in this document.
49 Why Would I Want User Mode Linux?
50 =================================
53 * If User Mode Linux kernel crashes, your host kernel is still fine. It
54 is not accelerated in any way (vhost, kvm, etc) and it is not trying to
55 access any devices directly. It is, in fact, a process like any other.
57 * You can run a usermode kernel as a non-root user (you may need to
58 arrange appropriate permissions for some devices).
60 * You can run a very small VM with a minimal footprint for a specific
61 task (for example 32M or less).
63 * You can get extremely high performance for anything which is a "kernel
64 specific task" such as forwarding, firewalling, etc while still being
65 isolated from the host kernel.
67 * You can play with kernel concepts without breaking things.
69 * You are not bound by "emulating" hardware, so you can try weird and
70 wonderful concepts which are very difficult to support when emulating
71 real hardware such as time travel and making your system clock
72 dependent on what UML does (very useful for things like tests).
79 * The syscall interception technique used by UML makes it inherently
80 slower for any userspace applications. While it can do kernel tasks
81 on par with most other virtualization packages, its userspace is
82 **slow**. The root cause is that UML has a very high cost of creating
83 new processes and threads (something most Unix/Linux applications
86 * UML is strictly uniprocessor at present. If you want to run an
87 application which needs many CPUs to function, it is clearly the
90 ***********************
91 Building a UML instance
92 ***********************
94 There is no UML installer in any distribution. While you can use off
95 the shelf install media to install into a blank VM using a virtualization
96 package, there is no UML equivalent. You have to use appropriate tools on
97 your host to build a viable filesystem image.
99 This is extremely easy on Debian - you can do it using debootstrap. It is
100 also easy on OpenWRT - the build process can build UML images. All other
106 Create a sparse raw disk image::
108 # dd if=/dev/zero of=disk_image_name bs=1 count=1 seek=16G
110 This will create a 16G disk image. The OS will initially allocate only one
111 block and will allocate more as they are written by UML. As of kernel
112 version 4.19 UML fully supports TRIM (as usually used by flash drives).
113 Using TRIM inside the UML image by specifying discard as a mount option
114 or by running ``tune2fs -o discard /dev/ubdXX`` will request UML to
115 return any unused blocks to the OS.
117 Create a filesystem on the disk image and mount it::
119 # mkfs.ext4 ./disk_image_name && mount ./disk_image_name /mnt
121 This example uses ext4, any other filesystem such as ext3, btrfs, xfs,
122 jfs, etc will work too.
124 Create a minimal OS installation on the mounted filesystem::
126 # debootstrap buster /mnt http://deb.debian.org/debian
128 debootstrap does not set up the root password, fstab, hostname or
129 anything related to networking. It is up to the user to do that.
131 Set the root password - the easiest way to do that is to chroot into the
138 Edit key system files
139 =====================
141 UML block devices are called ubds. The fstab created by debootstrap
142 will be empty and it needs an entry for the root file system::
144 /dev/ubd0 ext4 discard,errors=remount-ro 0 1
146 The image hostname will be set to the same as the host on which you
147 are creating its image. It is a good idea to change that to avoid
148 "Oh, bummer, I rebooted the wrong machine".
150 UML supports two classes of network devices - the older uml_net ones
151 which are scheduled for obsoletion. These are called ethX. It also
152 supports the newer vector IO devices which are significantly faster
153 and have support for some standard virtual network encapsulations like
154 Ethernet over GRE and Ethernet over L2TPv3. These are called vec0.
156 Depending on which one is in use, ``/etc/network/interfaces`` will
159 # legacy UML network devices
163 # vector UML network devices
167 We now have a UML image which is nearly ready to run, all we need is a
168 UML kernel and modules for it.
170 Most distributions have a UML package. Even if you intend to use your own
171 kernel, testing the image with a stock one is always a good start. These
172 packages come with a set of modules which should be copied to the target
173 filesystem. The location is distribution dependent. For Debian these
174 reside under /usr/lib/uml/modules. Copy recursively the content of this
175 directory to the mounted UML filesystem::
177 # cp -rax /usr/lib/uml/modules /mnt/lib/modules
179 If you have compiled your own kernel, you need to use the usual "install
180 modules to a location" procedure by running::
182 # make INSTALL_MOD_PATH=/mnt/lib/modules modules_install
184 This will install modules into /mnt/lib/modules/$(KERNELRELEASE).
185 To specify the full module installation path, use::
187 # make MODLIB=/mnt/lib/modules modules_install
189 At this point the image is ready to be brought up.
191 *************************
192 Setting Up UML Networking
193 *************************
195 UML networking is designed to emulate an Ethernet connection. This
196 connection may be either point-to-point (similar to a connection
197 between machines using a back-to-back cable) or a connection to a
198 switch. UML supports a wide variety of means to build these
199 connections to all of: local machine, remote machine(s), local and
200 remote UML and other VM instances.
203 +-----------+--------+------------------------------------+------------+
204 | Transport | Type | Capabilities | Throughput |
205 +===========+========+====================================+============+
206 | tap | vector | checksum, tso | > 8Gbit |
207 +-----------+--------+------------------------------------+------------+
208 | hybrid | vector | checksum, tso, multipacket rx | > 6GBit |
209 +-----------+--------+------------------------------------+------------+
210 | raw | vector | checksum, tso, multipacket rx, tx" | > 6GBit |
211 +-----------+--------+------------------------------------+------------+
212 | EoGRE | vector | multipacket rx, tx | > 3Gbit |
213 +-----------+--------+------------------------------------+------------+
214 | Eol2tpv3 | vector | multipacket rx, tx | > 3Gbit |
215 +-----------+--------+------------------------------------+------------+
216 | bess | vector | multipacket rx, tx | > 3Gbit |
217 +-----------+--------+------------------------------------+------------+
218 | fd | vector | dependent on fd type | varies |
219 +-----------+--------+------------------------------------+------------+
220 | tuntap | legacy | none | ~ 500Mbit |
221 +-----------+--------+------------------------------------+------------+
222 | daemon | legacy | none | ~ 450Mbit |
223 +-----------+--------+------------------------------------+------------+
224 | socket | legacy | none | ~ 450Mbit |
225 +-----------+--------+------------------------------------+------------+
226 | pcap | legacy | rx only | ~ 450Mbit |
227 +-----------+--------+------------------------------------+------------+
228 | ethertap | legacy | obsolete | ~ 500Mbit |
229 +-----------+--------+------------------------------------+------------+
230 | vde | legacy | obsolete | ~ 500Mbit |
231 +-----------+--------+------------------------------------+------------+
233 * All transports which have tso and checksum offloads can deliver speeds
234 approaching 10G on TCP streams.
236 * All transports which have multi-packet rx and/or tx can deliver pps
237 rates of up to 1Mps or more.
239 * All legacy transports are generally limited to ~600-700MBit and 0.05Mps.
241 * GRE and L2TPv3 allow connections to all of: local machine, remote
242 machines, remote network devices and remote UML instances.
244 * Socket allows connections only between UML instances.
246 * Daemon and bess require running a local switch. This switch may be
247 connected to the host as well.
250 Network configuration privileges
251 ================================
253 The majority of the supported networking modes need ``root`` privileges.
254 For example, in the legacy tuntap networking mode, users were required
255 to be part of the group associated with the tunnel device.
257 For newer network drivers like the vector transports, ``root`` privilege
258 is required to fire an ioctl to setup the tun interface and/or use
259 raw sockets where needed.
261 This can be achieved by granting the user a particular capability instead
262 of running UML as root. In case of vector transport, a user can add the
263 capability ``CAP_NET_ADMIN`` or ``CAP_NET_RAW`` to the uml binary.
264 Thenceforth, UML can be run with normal user privilges, along with
269 # sudo setcap cap_net_raw,cap_net_admin+ep linux
271 Configuring vector transports
272 ===============================
274 All vector transports support a similar syntax:
276 If X is the interface number as in vec0, vec1, vec2, etc, the general
277 syntax for options is::
279 vecX:transport="Transport Name",option=value,option=value,...,option=value
284 These options are common for all transports:
286 * ``depth=int`` - sets the queue depth for vector IO. This is the
287 amount of packets UML will attempt to read or write in a single
288 system call. The default number is 64 and is generally sufficient
289 for most applications that need throughput in the 2-4 Gbit range.
290 Higher speeds may require larger values.
292 * ``mac=XX:XX:XX:XX:XX`` - sets the interface MAC address value.
294 * ``gro=[0,1]`` - sets GRO off or on. Enables receive/transmit offloads.
295 The effect of this option depends on the host side support in the transport
296 which is being configured. In most cases it will enable TCP segmentation and
297 RX/TX checksumming offloads. The setting must be identical on the host side
298 and the UML side. The UML kernel will produce warnings if it is not.
299 For example, GRO is enabled by default on local machine interfaces
300 (e.g. veth pairs, bridge, etc), so it should be enabled in UML in the
301 corresponding UML transports (raw, tap, hybrid) in order for networking to
304 * ``mtu=int`` - sets the interface MTU
306 * ``headroom=int`` - adjusts the default headroom (32 bytes) reserved
307 if a packet will need to be re-encapsulated into for instance VXLAN.
309 * ``vec=0`` - disable multipacket IO and fall back to packet at a
315 * ``ifname=str`` Transports which bind to a local network interface
316 have a shared option - the name of the interface to bind to.
318 * ``src, dst, src_port, dst_port`` - all transports which use sockets
319 which have the notion of source and destination and/or source port
320 and destination port use these to specify them.
322 * ``v6=[0,1]`` to specify if a v6 connection is desired for all
323 transports which operate over IP. Additionally, for transports that
324 have some differences in the way they operate over v4 and v6 (for example
325 EoL2TPv3), sets the correct mode of operation. In the absence of this
326 option, the socket type is determined based on what do the src and dst
327 arguments resolve/parse to.
334 vecX:transport=tap,ifname=tap0,depth=128,gro=1
336 This will connect vec0 to tap0 on the host. Tap0 must already exist (for example
337 created using tunctl) and UP.
339 tap0 can be configured as a point-to-point interface and given an IP
340 address so that UML can talk to the host. Alternatively, it is possible
341 to connect UML to a tap interface which is connected to a bridge.
343 While tap relies on the vector infrastructure, it is not a true vector
344 transport at this point, because Linux does not support multi-packet
345 IO on tap file descriptors for normal userspace apps like UML. This
346 is a privilege which is offered only to something which can hook up
347 to it at kernel level via specialized interfaces like vhost-net. A
348 vhost-net like helper for UML is planned at some point in the future.
350 Privileges required: tap transport requires either:
352 * tap interface to exist and be created persistent and owned by the
353 UML user using tunctl. Example ``tunctl -u uml-user -t tap0``
355 * binary to have ``CAP_NET_ADMIN`` privilege
362 vecX:transport=hybrid,ifname=tap0,depth=128,gro=1
364 This is an experimental/demo transport which couples tap for transmit
365 and a raw socket for receive. The raw socket allows multi-packet
366 receive resulting in significantly higher packet rates than normal tap.
368 Privileges required: hybrid requires ``CAP_NET_RAW`` capability by
369 the UML user as well as the requirements for the tap transport.
376 vecX:transport=raw,ifname=p-veth0,depth=128,gro=1
379 This transport uses vector IO on raw sockets. While you can bind to any
380 interface including a physical one, the most common use it to bind to
381 the "peer" side of a veth pair with the other side configured on the
384 Example host configuration for Debian:
386 **/etc/network/interfaces**::
389 iface veth0 inet static
391 netmask 255.255.255.252
392 broadcast 192.168.4.3
393 pre-up ip link add veth0 type veth peer name p-veth0 && \
396 UML can now bind to p-veth0 like this::
398 vec0:transport=raw,ifname=p-veth0,depth=128,gro=1
401 If the UML guest is configured with 192.168.4.2 and netmask 255.255.255.0
402 it can talk to the host on 192.168.4.1
404 The raw transport also provides some support for offloading some of the
405 filtering to the host. The two options to control it are:
407 * ``bpffile=str`` filename of raw bpf code to be loaded as a socket filter
409 * ``bpfflash=int`` 0/1 allow loading of bpf from inside User Mode Linux.
410 This option allows the use of the ethtool load firmware command to
413 In either case the bpf code is loaded into the host kernel. While this is
414 presently limited to legacy bpf syntax (not ebpf), it is still a security
415 risk. It is not recommended to allow this unless the User Mode Linux
416 instance is considered trusted.
418 Privileges required: raw socket transport requires `CAP_NET_RAW`
426 vecX:transport=gre,src=$src_host,dst=$dst_host
429 This will configure an Ethernet over ``GRE`` (aka ``GRETAP`` or
430 ``GREIRB``) tunnel which will connect the UML instance to a ``GRE``
431 endpoint at host dst_host. ``GRE`` supports the following additional
434 * ``rx_key=int`` - GRE 32-bit integer key for rx packets, if set,
435 ``txkey`` must be set too
437 * ``tx_key=int`` - GRE 32-bit integer key for tx packets, if set
438 ``rx_key`` must be set too
440 * ``sequence=[0,1]`` - enable GRE sequence
442 * ``pin_sequence=[0,1]`` - pretend that the sequence is always reset
443 on each packet (needed to interoperate with some really broken
446 * ``v6=[0,1]`` - force IPv4 or IPv6 sockets respectively
448 * GRE checksum is not presently supported
450 GRE has a number of caveats:
452 * You can use only one GRE connection per IP address. There is no way to
453 multiplex connections as each GRE tunnel is terminated directly on
456 * The key is not really a security feature. While it was intended as such
457 its "security" is laughable. It is, however, a useful feature to
458 ensure that the tunnel is not misconfigured.
460 An example configuration for a Linux host with a local address of
461 192.168.128.1 to connect to a UML instance at 192.168.129.1
463 **/etc/network/interfaces**::
466 iface gt0 inet static
468 netmask 255.255.255.0
471 pre-up ip link add gt0 type gretap local 192.168.128.1 \
472 remote 192.168.129.1 || true
473 down ip link del gt0 || true
475 Additionally, GRE has been tested versus a variety of network equipment.
477 Privileges required: GRE requires ``CAP_NET_RAW``
479 l2tpv3 socket transport
480 -----------------------
482 _Warning_. L2TPv3 has a "bug". It is the "bug" known as "has more
483 options than GNU ls". While it has some advantages, there are usually
484 easier (and less verbose) ways to connect a UML instance to something.
485 For example, most devices which support L2TPv3 also support GRE.
489 vec0:transport=l2tpv3,udp=1,src=$src_host,dst=$dst_host,srcport=$src_port,dstport=$dst_port,depth=128,rx_session=0xffffffff,tx_session=0xffff
491 This will configure an Ethernet over L2TPv3 fixed tunnel which will
492 connect the UML instance to a L2TPv3 endpoint at host $dst_host using
493 the L2TPv3 UDP flavour and UDP destination port $dst_port.
495 L2TPv3 always requires the following additional options:
497 * ``rx_session=int`` - l2tpv3 32-bit integer session for rx packets
499 * ``tx_session=int`` - l2tpv3 32-bit integer session for tx packets
501 As the tunnel is fixed these are not negotiated and they are
502 preconfigured on both ends.
504 Additionally, L2TPv3 supports the following optional parameters.
506 * ``rx_cookie=int`` - l2tpv3 32-bit integer cookie for rx packets - same
507 functionality as GRE key, more to prevent misconfiguration than provide
510 * ``tx_cookie=int`` - l2tpv3 32-bit integer cookie for tx packets
512 * ``cookie64=[0,1]`` - use 64-bit cookies instead of 32-bit.
514 * ``counter=[0,1]`` - enable l2tpv3 counter
516 * ``pin_counter=[0,1]`` - pretend that the counter is always reset on
517 each packet (needed to interoperate with some really broken
520 * ``v6=[0,1]`` - force v6 sockets
522 * ``udp=[0,1]`` - use raw sockets (0) or UDP (1) version of the protocol
524 L2TPv3 has a number of caveats:
526 * you can use only one connection per IP address in raw mode. There is
527 no way to multiplex connections as each L2TPv3 tunnel is terminated
528 directly on the UML instance. UDP mode can use different ports for
531 Here is an example of how to configure a Linux host to connect to UML
534 **/etc/network/interfaces**::
537 iface l2tp1 inet static
538 address 192.168.126.1
539 netmask 255.255.255.0
540 broadcast 192.168.126.255
542 pre-up ip l2tp add tunnel remote 127.0.0.1 \
543 local 127.0.0.1 encap udp tunnel_id 2 \
544 peer_tunnel_id 2 udp_sport 1706 udp_dport 1707 && \
545 ip l2tp add session name l2tp1 tunnel_id 2 \
546 session_id 0xffffffff peer_session_id 0xffffffff
547 down ip l2tp del session tunnel_id 2 session_id 0xffffffff && \
548 ip l2tp del tunnel tunnel_id 2
551 Privileges required: L2TPv3 requires ``CAP_NET_RAW`` for raw IP mode and
552 no special privileges for the UDP mode.
554 BESS socket transport
555 ---------------------
557 BESS is a high performance modular network switch.
559 https://github.com/NetSys/bess
561 It has support for a simple sequential packet socket mode which in the
562 more recent versions is using vector IO for high performance.
566 vecX:transport=bess,src=$unix_src,dst=$unix_dst
568 This will configure a BESS transport using the unix_src Unix domain
569 socket address as source and unix_dst socket address as destination.
571 For BESS configuration and how to allocate a BESS Unix domain socket port
572 please see the BESS documentation.
574 https://github.com/NetSys/bess/wiki/Built-In-Modules-and-Ports
576 BESS transport does not require any special privileges.
578 Configuring Legacy transports
579 =============================
581 Legacy transports are now considered obsolete. Please use the vector
588 This section assumes that either the user-mode-linux package from the
589 distribution or a custom built kernel has been installed on the host.
591 These add an executable called linux to the system. This is the UML
592 kernel. It can be run just like any other executable.
593 It will take most normal linux kernel arguments as command line
594 arguments. Additionally, it will need some UML-specific arguments
595 in order to do something useful.
603 * ``mem=int[K,M,G]`` - amount of memory. By default in bytes. It will
604 also accept K, M or G qualifiers.
606 * ``ubdX[s,d,c,t]=`` virtual disk specification. This is not really
607 mandatory, but it is likely to be needed in nearly all cases so we can
608 specify a root file system.
609 The simplest possible image specification is the name of the image
610 file for the filesystem (created using one of the methods described
611 in `Creating an image`_).
613 * UBD devices support copy on write (COW). The changes are kept in
614 a separate file which can be discarded allowing a rollback to the
615 original pristine image. If COW is desired, the UBD image is
616 specified as: ``cow_file,master_image``.
617 Example:``ubd0=Filesystem.cow,Filesystem.img``
619 * UBD devices can be set to use synchronous IO. Any writes are
620 immediately flushed to disk. This is done by adding ``s`` after
621 the ``ubdX`` specification.
623 * UBD performs some heuristics on devices specified as a single
624 filename to make sure that a COW file has not been specified as
625 the image. To turn them off, use the ``d`` flag after ``ubdX``.
627 * UBD supports TRIM - asking the Host OS to reclaim any unused
628 blocks in the image. To turn it off, specify the ``t`` flag after
631 * ``root=`` root device - most likely ``/dev/ubd0`` (this is a Linux
634 Important Optional Arguments
635 ----------------------------
637 If UML is run as "linux" with no extra arguments, it will try to start an
638 xterm for every console configured inside the image (up to 6 in most
639 Linux distributions). Each console is started inside an
640 xterm. This makes it nice and easy to use UML on a host with a GUI. It is,
641 however, the wrong approach if UML is to be used as a testing harness or run
642 in a text-only environment.
644 In order to change this behaviour we need to specify an alternative console
645 and wire it to one of the supported "line" channels. For this we need to map a
646 console to use something different from the default xterm.
648 Example which will divert console number 1 to stdin/stdout::
652 UML supports a wide variety of serial line channels which are specified using
655 conX=channel_type:options[,channel_type:options]
658 If the channel specification contains two parts separated by comma, the first
659 one is input, the second one output.
661 * The null channel - Discard all input or output. Example ``con=null`` will set
662 all consoles to null by default.
664 * The fd channel - use file descriptor numbers for input/output. Example:
667 * The port channel - start a telnet server on TCP port number. Example:
668 ``con1=port:4321``. The host must have /usr/sbin/in.telnetd (usually part of
669 a telnetd package) and the port-helper from the UML utilities (see the
670 information for the xterm channel below). UML will not boot until a client
673 * The pty and pts channels - use system pty/pts.
675 * The tty channel - bind to an existing system tty. Example: ``con1=/dev/tty8``
676 will make UML use the host 8th console (usually unused).
678 * The xterm channel - this is the default - bring up an xterm on this channel
679 and direct IO to it. Note that in order for xterm to work, the host must
680 have the UML distribution package installed. This usually contains the
681 port-helper and other utilities needed for UML to communicate with the xterm.
682 Alternatively, these need to be complied and installed from source. All
683 options applicable to consoles also apply to UML serial lines which are
684 presented as ttyS inside UML.
692 # linux mem=2048M umid=TEST \
693 ubd0=Filesystem.img \
694 vec0:transport=tap,ifname=tap0,depth=128,gro=1 \
695 root=/dev/ubda con=null con0=null,fd:2 con1=fd:0,fd:1
697 This will run an instance with ``2048M RAM`` and try to use the image file
698 called ``Filesystem.img`` as root. It will connect to the host using tap0.
699 All consoles except ``con1`` will be disabled and console 1 will
700 use standard input/output making it appear in the same terminal it was started.
705 If you have not set up a password when generating the image, you will have to
706 shut down the UML instance, mount the image, chroot into it and set it - as
707 described in the Generating an Image section. If the password is already set,
710 The UML Management Console
711 ============================
713 In addition to managing the image from "the inside" using normal sysadmin tools,
714 it is possible to perform a number of low-level operations using the UML
715 management console. The UML management console is a low-level interface to the
716 kernel on a running UML instance, somewhat like the i386 SysRq interface. Since
717 there is a full-blown operating system under UML, there is much greater
718 flexibility possible than with the SysRq mechanism.
720 There are a number of things you can do with the mconsole interface:
722 * get the kernel version
723 * add and remove devices
724 * halt or reboot the machine
725 * Send SysRq commands
726 * Pause and resume the UML
727 * Inspect processes running inside UML
728 * Inspect UML internal /proc state
730 You need the mconsole client (uml\_mconsole) which is a part of the UML
731 tools package available in most Linux distritions.
733 You also need ``CONFIG_MCONSOLE`` (under 'General Setup') enabled in the UML
734 kernel. When you boot UML, you'll see a line like::
736 mconsole initialized on /home/jdike/.uml/umlNJ32yL/mconsole
738 If you specify a unique machine id on the UML command line, i.e.
739 ``umid=debian``, you'll see this::
741 mconsole initialized on /home/jdike/.uml/debian/mconsole
744 That file is the socket that uml_mconsole will use to communicate with
745 UML. Run it with either the umid or the full path as its argument::
747 # uml_mconsole debian
751 # uml_mconsole /home/jdike/.uml/debian/mconsole
754 You'll get a prompt, at which you can run one of these commands:
773 This command takes no arguments. It prints the UML version::
776 OK Linux OpenWrt 4.14.106 #0 Tue Mar 19 08:19:41 2019 x86_64
779 There are a couple actual uses for this. It's a simple no-op which
780 can be used to check that a UML is running. It's also a way of
781 sending a device interrupt to the UML. UML mconsole is treated internally as
787 This command takes no arguments. It prints a short help screen with the
788 supported mconsole commands.
794 These commands take no arguments. They shut the machine down immediately, with
795 no syncing of disks and no clean shutdown of userspace. So, they are
796 pretty close to crashing the machine::
804 "config" adds a new device to the virtual machine. This is supported
805 by most UML device drivers. It takes one argument, which is the
806 device to add, with the same syntax as the kernel command line::
808 (mconsole) config ubd3=/home/jdike/incoming/roots/root_fs_debian22
813 "remove" deletes a device from the system. Its argument is just the
814 name of the device to be removed. The device must be idle in whatever
815 sense the driver considers necessary. In the case of the ubd driver,
816 the removed block device must not be mounted, swapped on, or otherwise
817 open, and in the case of the network driver, the device must be down::
819 (mconsole) remove ubd3
824 This command takes one argument, which is a single letter. It calls the
825 generic kernel's SysRq driver, which does whatever is called for by
826 that argument. See the SysRq documentation in
827 Documentation/admin-guide/sysrq.rst in your favorite kernel tree to
828 see what letters are valid and what they do.
833 This invokes the ``Ctl-Alt-Del`` action in the running image. What exactly
834 this ends up doing is up to init, systemd, etc. Normally, it reboots the
840 This puts the UML in a loop reading mconsole requests until a 'go'
841 mconsole command is received. This is very useful as a
842 debugging/snapshotting tool.
847 This resumes a UML after being paused by a 'stop' command. Note that
848 when the UML has resumed, TCP connections may have timed out and if
849 the UML is paused for a long period of time, crond might go a little
850 crazy, running all the jobs it didn't do earlier.
855 This takes one argument - the name of a file in /proc which is printed
856 to the mconsole standard output
861 This takes one argument - the pid number of a process. Its stack is
862 printed to a standard output.
868 Sharing Filesystems between Virtual Machines
869 ============================================
871 Don't attempt to share filesystems simply by booting two UMLs from the
872 same file. That's the same thing as booting two physical machines
873 from a shared disk. It will result in filesystem corruption.
875 Using layered block devices
876 ---------------------------
878 The way to share a filesystem between two virtual machines is to use
879 the copy-on-write (COW) layering capability of the ubd block driver.
880 Any changed blocks are stored in the private COW file, while reads come
881 from either device - the private one if the requested block is valid in
882 it, the shared one if not. Using this scheme, the majority of data
883 which is unchanged is shared between an arbitrary number of virtual
884 machines, each of which has a much smaller file containing the changes
885 that it has made. With a large number of UMLs booting from a large root
886 filesystem, this leads to a huge disk space saving.
888 Sharing file system data will also help performance, since the host will
889 be able to cache the shared data using a much smaller amount of memory,
890 so UML disk requests will be served from the host's memory rather than
891 its disks. There is a major caveat in doing this on multisocket NUMA
892 machines. On such hardware, running many UML instances with a shared
893 master image and COW changes may cause issues like NMIs from excess of
894 inter-socket traffic.
896 If you are running UML on high-end hardware like this, make sure to
897 bind UML to a set of logical CPUs residing on the same socket using the
898 ``taskset`` command or have a look at the "tuning" section.
900 To add a copy-on-write layer to an existing block device file, simply
901 add the name of the COW file to the appropriate ubd switch::
903 ubd0=root_fs_cow,root_fs_debian_22
905 where ``root_fs_cow`` is the private COW file and ``root_fs_debian_22`` is
906 the existing shared filesystem. The COW file need not exist. If it
907 doesn't, the driver will create and initialize it.
912 UML has TRIM support which will release any unused space in its disk
913 image files to the underlying OS. It is important to use either ls -ls
914 or du to verify the actual file size.
919 Any changes to the master image will invalidate all COW files. If this
920 happens, UML will *NOT* automatically delete any of the COW files and
921 will refuse to boot. In this case the only solution is to either
922 restore the old image (including its last modified timestamp) or remove
923 all COW files which will result in their recreation. Any changes in
924 the COW files will be lost.
926 Cows can moo - uml_moo : Merging a COW file with its backing file
927 -----------------------------------------------------------------
929 Depending on how you use UML and COW devices, it may be advisable to
930 merge the changes in the COW file into the backing file every once in
933 The utility that does this is uml_moo. Its usage is::
935 uml_moo COW_file new_backing_file
938 There's no need to specify the backing file since that information is
939 already in the COW file header. If you're paranoid, boot the new
940 merged file, and if you're happy with it, move it over the old backing
943 ``uml_moo`` creates a new backing file by default as a safety measure.
944 It also has a destructive merge option which will merge the COW file
945 directly into its current backing file. This is really only usable
946 when the backing file only has one COW file associated with it. If
947 there are multiple COWs associated with a backing file, a -d merge of
948 one of them will invalidate all of the others. However, it is
949 convenient if you're short of disk space, and it should also be
950 noticeably faster than a non-destructive merge.
952 ``uml_moo`` is installed with the UML distribution packages and is
953 available as a part of UML utilities.
958 If you want to access files on the host machine from inside UML, you
959 can treat it as a separate machine and either nfs mount directories
960 from the host or copy files into the virtual machine with scp.
961 However, since UML is running on the host, it can access those
962 files just like any other process and make them available inside the
963 virtual machine without the need to use the network.
964 This is possible with the hostfs virtual filesystem. With it, you
965 can mount a host directory into the UML filesystem and access the
966 files contained in it just as you would on the host.
970 Hostfs without any parameters to the UML Image will allow the image
971 to mount any part of the host filesystem and write to it. Always
972 confine hostfs to a specific "harmless" directory (for example ``/var/tmp``)
973 if running UML. This is especially important if UML is being run as root.
978 To begin with, make sure that hostfs is available inside the virtual
981 # cat /proc/filesystems
983 ``hostfs`` should be listed. If it's not, either rebuild the kernel
984 with hostfs configured into it or make sure that hostfs is built as a
985 module and available inside the virtual machine, and insmod it.
988 Now all you need to do is run mount::
990 # mount none /mnt/host -t hostfs
992 will mount the host's ``/`` on the virtual machine's ``/mnt/host``.
993 If you don't want to mount the host root directory, then you can
994 specify a subdirectory to mount with the -o switch to mount::
996 # mount none /mnt/home -t hostfs -o /home
998 will mount the host's /home on the virtual machine's /mnt/home.
1000 hostfs as the root filesystem
1001 -----------------------------
1003 It's possible to boot from a directory hierarchy on the host using
1004 hostfs rather than using the standard filesystem in a file.
1005 To start, you need that hierarchy. The easiest way is to loop mount
1006 an existing root_fs file::
1008 # mount root_fs uml_root_dir -o loop
1011 You need to change the filesystem type of ``/`` in ``etc/fstab`` to be
1012 'hostfs', so that line looks like this::
1014 /dev/ubd/0 / hostfs defaults 1 1
1016 Then you need to chown to yourself all the files in that directory
1017 that are owned by root. This worked for me::
1019 # find . -uid 0 -exec chown jdike {} \;
1021 Next, make sure that your UML kernel has hostfs compiled in, not as a
1022 module. Then run UML with the boot device pointing at that directory::
1024 ubd0=/path/to/uml/root/directory
1026 UML should then boot as it does normally.
1031 Hostfs does not support keeping track of host filesystem changes on the
1032 host (outside UML). As a result, if a file is changed without UML's
1033 knowledge, UML will not know about it and its own in-memory cache of
1034 the file may be corrupt. While it is possible to fix this, it is not
1035 something which is being worked on at present.
1040 UML at present is strictly uniprocessor. It will, however spin up a
1041 number of threads to handle various functions.
1043 The UBD driver, SIGIO and the MMU emulation do that. If the system is
1044 idle, these threads will be migrated to other processors on a SMP host.
1045 This, unfortunately, will usually result in LOWER performance because of
1046 all of the cache/memory synchronization traffic between cores. As a
1047 result, UML will usually benefit from being pinned on a single CPU,
1048 especially on a large system. This can result in performance differences
1049 of 5 times or higher on some benchmarks.
1051 Similarly, on large multi-node NUMA systems UML will benefit if all of
1052 its memory is allocated from the same NUMA node it will run on. The
1053 OS will *NOT* do that by default. In order to do that, the sysadmin
1054 needs to create a suitable tmpfs ramdisk bound to a particular node
1055 and use that as the source for UML RAM allocation by specifying it
1056 in the TMP or TEMP environment variables. UML will look at the values
1057 of ``TMPDIR``, ``TMP`` or ``TEMP`` for that. If that fails, it will
1058 look for shmfs mounted under ``/dev/shm``. If everything else fails use
1059 ``/tmp/`` regardless of the filesystem type used for it::
1061 mount -t tmpfs -ompol=bind:X none /mnt/tmpfs-nodeX
1062 TEMP=/mnt/tmpfs-nodeX taskset -cX linux options options options..
1064 *******************************************
1065 Contributing to UML and Developing with UML
1066 *******************************************
1068 UML is an excellent platform to develop new Linux kernel concepts -
1069 filesystems, devices, virtualization, etc. It provides unrivalled
1070 opportunities to create and test them without being constrained to
1071 emulating specific hardware.
1073 Example - want to try how Linux will work with 4096 "proper" network
1076 Not an issue with UML. At the same time, this is something which
1077 is difficult with other virtualization packages - they are
1078 constrained by the number of devices allowed on the hardware bus
1079 they are trying to emulate (for example 16 on a PCI bus in qemu).
1081 If you have something to contribute such as a patch, a bugfix, a
1082 new feature, please send it to ``linux-um@lists.infradead.org``.
1084 Please follow all standard Linux patch guidelines such as cc-ing
1085 relevant maintainers and run ``./scripts/checkpatch.pl`` on your patch.
1086 For more details see ``Documentation/process/submitting-patches.rst``
1088 Note - the list does not accept HTML or attachments, all emails must
1089 be formatted as plain text.
1091 Developing always goes hand in hand with debugging. First of all,
1092 you can always run UML under gdb and there will be a whole section
1093 later on on how to do that. That, however, is not the only way to
1094 debug a Linux kernel. Quite often adding tracing statements and/or
1095 using UML specific approaches such as ptracing the UML kernel process
1096 are significantly more informative.
1101 When running, UML consists of a main kernel thread and a number of
1102 helper threads. The ones of interest for tracing are NOT the ones
1103 that are already ptraced by UML as a part of its MMU emulation.
1105 These are usually the first three threads visible in a ps display.
1106 The one with the lowest PID number and using most CPU is usually the
1107 kernel thread. The other threads are the disk
1108 (ubd) device helper thread and the SIGIO helper thread.
1109 Running ptrace on this thread usually results in the following picture::
1111 host$ strace -p 16566
1112 --- SIGIO {si_signo=SIGIO, si_code=POLL_IN, si_band=65} ---
1113 epoll_wait(4, [{EPOLLIN, {u32=3721159424, u64=3721159424}}], 64, 0) = 1
1114 epoll_wait(4, [], 64, 0) = 0
1115 rt_sigreturn({mask=[PIPE]}) = 16967
1116 ptrace(PTRACE_GETREGS, 16967, NULL, 0xd5f34f38) = 0
1117 ptrace(PTRACE_GETREGSET, 16967, NT_X86_XSTATE, [{iov_base=0xd5f35010, iov_len=832}]) = 0
1118 ptrace(PTRACE_GETSIGINFO, 16967, NULL, {si_signo=SIGTRAP, si_code=0x85, si_pid=16967, si_uid=0}) = 0
1119 ptrace(PTRACE_SETREGS, 16967, NULL, 0xd5f34f38) = 0
1120 ptrace(PTRACE_SETREGSET, 16967, NT_X86_XSTATE, [{iov_base=0xd5f35010, iov_len=2696}]) = 0
1121 ptrace(PTRACE_SYSEMU, 16967, NULL, 0) = 0
1122 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_TRAPPED, si_pid=16967, si_uid=0, si_status=SIGTRAP, si_utime=65, si_stime=89} ---
1123 wait4(16967, [{WIFSTOPPED(s) && WSTOPSIG(s) == SIGTRAP | 0x80}], WSTOPPED|__WALL, NULL) = 16967
1124 ptrace(PTRACE_GETREGS, 16967, NULL, 0xd5f34f38) = 0
1125 ptrace(PTRACE_GETREGSET, 16967, NT_X86_XSTATE, [{iov_base=0xd5f35010, iov_len=832}]) = 0
1126 ptrace(PTRACE_GETSIGINFO, 16967, NULL, {si_signo=SIGTRAP, si_code=0x85, si_pid=16967, si_uid=0}) = 0
1127 timer_settime(0, 0, {it_interval={tv_sec=0, tv_nsec=0}, it_value={tv_sec=0, tv_nsec=2830912}}, NULL) = 0
1129 clock_nanosleep(CLOCK_MONOTONIC, 0, {tv_sec=1, tv_nsec=0}, NULL) = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
1130 --- SIGALRM {si_signo=SIGALRM, si_code=SI_TIMER, si_timerid=0, si_overrun=0, si_value={int=1631716592, ptr=0x614204f0}} ---
1131 rt_sigreturn({mask=[PIPE]}) = -1 EINTR (Interrupted system call)
1133 This is a typical picture from a mostly idle UML instance.
1135 * UML interrupt controller uses epoll - this is UML waiting for IO
1138 epoll_wait(4, [{EPOLLIN, {u32=3721159424, u64=3721159424}}], 64, 0) = 1
1140 * The sequence of ptrace calls is part of MMU emulation and running the
1142 * ``timer_settime`` is part of the UML high res timer subsystem mapping
1143 timer requests from inside UML onto the host high resolution timers.
1144 * ``clock_nanosleep`` is UML going into idle (similar to the way a PC
1145 will execute an ACPI idle).
1147 As you can see UML will generate quite a bit of output even in idle. The output
1148 can be very informative when observing IO. It shows the actual IO calls, their
1149 arguments and returns values.
1154 You can run UML under gdb now, though it will not necessarily agree to
1155 be started under it. If you are trying to track a runtime bug, it is
1156 much better to attach gdb to a running UML instance and let UML run.
1158 Assuming the same PID number as in the previous example, this would be::
1162 This will STOP the UML instance, so you must enter `cont` at the GDB
1163 command line to request it to continue. It may be a good idea to make
1164 this into a gdb script and pass it to gdb as an argument.
1166 Developing Device Drivers
1167 =========================
1169 Nearly all UML drivers are monolithic. While it is possible to build a
1170 UML driver as a kernel module, that limits the possible functionality
1171 to in-kernel only and non-UML specific. The reason for this is that
1172 in order to really leverage UML, one needs to write a piece of
1173 userspace code which maps driver concepts onto actual userspace host
1176 This forms the so-called "user" portion of the driver. While it can
1177 reuse a lot of kernel concepts, it is generally just another piece of
1178 userspace code. This portion needs some matching "kernel" code which
1179 resides inside the UML image and which implements the Linux kernel part.
1181 *Note: There are very few limitations in the way "kernel" and "user" interact*.
1183 UML does not have a strictly defined kernel-to-host API. It does not
1184 try to emulate a specific architecture or bus. UML's "kernel" and
1185 "user" can share memory, code and interact as needed to implement
1186 whatever design the software developer has in mind. The only
1187 limitations are purely technical. Due to a lot of functions and
1188 variables having the same names, the developer should be careful
1189 which includes and libraries they are trying to refer to.
1191 As a result a lot of userspace code consists of simple wrappers.
1192 E.g. ``os_close_file()`` is just a wrapper around ``close()``
1193 which ensures that the userspace function close does not clash
1194 with similarly named function(s) in the kernel part.
1196 Using UML as a Test Platform
1197 ============================
1199 UML is an excellent test platform for device driver development. As
1200 with most things UML, "some user assembly may be required". It is
1201 up to the user to build their emulation environment. UML at present
1202 provides only the kernel infrastructure.
1204 Part of this infrastructure is the ability to load and parse fdt
1205 device tree blobs as used in Arm or Open Firmware platforms. These
1206 are supplied as an optional extra argument to the kernel command
1211 The device tree is loaded and parsed at boottime and is accessible by
1212 drivers which query it. At this moment in time this facility is
1213 intended solely for development purposes. UML's own devices do not
1214 query the device tree.
1216 Security Considerations
1217 -----------------------
1219 Drivers or any new functionality should default to not
1220 accepting arbitrary filename, bpf code or other parameters
1221 which can affect the host from inside the UML instance.
1222 For example, specifying the socket used for IPC communication
1223 between a driver and the host at the UML command line is OK
1224 security-wise. Allowing it as a loadable module parameter
1227 If such functionality is desireable for a particular application
1228 (e.g. loading BPF "firmware" for raw socket network transports),
1229 it should be off by default and should be explicitly turned on
1230 as a command line parameter at startup.
1232 Even with this in mind, the level of isolation between UML
1233 and the host is relatively weak. If the UML userspace is
1234 allowed to load arbitrary kernel drivers, an attacker can
1235 use this to break out of UML. Thus, if UML is used in
1236 a production application, it is recommended that all modules
1237 are loaded at boot and kernel module loading is disabled